mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-09 04:00:52 +00:00
Sameer Kadam
07c74cc544
Adds native vector deletion support for FAISS Flat indices via remove_ids and IDSelectorBatch (Closes #1374): - Adds FAISSIndex.delete_vectors() and FAISSStore.delete_vectors() to translate external string IDs to sequential internal positions, remove vectors via remove_ids, and keep vector_ids and metadata parallel with the compacted index. - Auto-saves index and metadata sidecar after deletion when the store was loaded from disk via load_index(), skipping redundant disk writes for no-op deletions. - Rejects IVF and HNSW index deletions with NotImplementedError: IVF does not compact internal labels upon remove_ids (causing search/offset desynchronization), and HNSW lacks remove_ids entirely. Both cleanly map to STATUS_UNSUPPORTED in ErasureCoordinator. Robust ID Management & Invariant Hardening: - Generates default IDs with a monotonic candidate-check loop against existing vector_ids, eliminating collisions and metadata overwrites when explicit vec_N IDs coexist. - Persists next_id in the .meta.json sidecar and clamps on load() to max(persisted, max(vec_N) + 1), preventing stale sidecars from re-introducing collisions across restarts or legacy migrations. - Guards against FAISS -1 sentinels (0 <= idx < len(vector_ids)) in search results when k > ntotal, preventing negative index wrapping to vector_ids[-1]. - Replaces bare post-delete assert with an explicit ProcessingError check that survives python -O. Adds 43 comprehensive tests in test_faiss_delete_vectors.py covering deletion, search exclusion, metadata cleanup, persistence roundtrips, IVF/HNSW rejection, ErasureCoordinator receipts, and mock-spied deterministic no-op saves.