_get_candidate_embeddings()'s expand-and-retry loop widens the search
pool (up to limit*10) when post-filtering leaves too few candidates.
If the backend keeps returning a full page and filtered matches never
reach `limit`, the loop exited via the while condition instead of the
break branch, so the pre-loop empty embeddings/metadata/scores lists
were returned instead of the matches actually found in the final
iteration. This silently returned [] for filtered queries against
large persistent-backend stores even when matches existed - exactly
the scenario this PR adds support for.
Falls back to the last collected batch instead of discarding it.
Also documents this PR and #839 in the changelog.
- FAISSStore: get_metadata now correctly retrieves from self.metadata instead of raising NotImplementedError.
- MilvusStore:
- Changed schema to support String IDs (VARCHAR) instead of auto-generated INT64, preventing loss of IDs during insert.
- Added metadata storage using JSON.
- Replaced insert_vectors with add_vectors accepting ids and metadata (added insert_vectors alias for backward compatibility).
- Implemented get_vector and get_metadata with safe parameterized querying to prevent query injection.
- PgVectorStore & SQLiteVecStore:
- Fixed get_vector and get_metadata to call self.get([vector_id]) instead of the non-existent get_vectors([vector_id]), fixing the silent None return bug.
- VectorStore.get_vector() and get_metadata() were hardcoded to access
self.vectors and self.metadata dicts, which are only initialized for
the inmemory backend, causing AttributeError on all persistent backends
(FAISS, Qdrant, Pinecone, Milvus, Weaviate, PgVector, SQLiteVec).
Changes:
- Refactor VectorStore.get_vector() and get_metadata() to branch on
self.backend == 'inmemory' (zero behavior change) and delegate to
self._backend_store otherwise.
- Harden save() to use getattr(self, 'vectors', {}) / getattr(self,
'metadata', {}) to prevent crash when saving a persistent backend store.
- Add get_vector() and get_metadata() to all 7 backend wrappers:
- FAISSStore: get_vector uses index.reconstruct(); get_metadata raises
NotImplementedError (FAISS has no metadata storage natively).
- QdrantStore: uses client.retrieve() with with_vectors/with_payload.
- PineconeStore: wraps existing fetch_vectors() call.
- MilvusStore: raises NotImplementedError (auto_id=True schema discards
string IDs at insert time, making by-ID lookup impossible in this
wrapper's current schema).
- WeaviateStore: uses collection.query.fetch_object_by_id().
- PgVectorStore: wraps existing get_vectors() SQL method.
- SQLiteVecStore: wraps existing get_vectors() SQL method.
- Add TestVectorStoreRetrieval regression tests covering inmemory and
FAISS backends with real (non-mocked) assertions.
All 28 tests pass.
- Replace direct .vectors and .metadata access with VectorStore.search_vectors().
- Add a fallback in HybridSimilarityCalculator (via ind_similar_decisions) to use the search score when backend vector databases do not natively return the raw vector array.
- Fix get_decision_statistics to gracefully fall back when .metadata is not fully supported by the underlying DB.
- Add regression tests utilizing the real FAISS and inmemory backends directly without mocking.
QdrantStore.search_vectors() returned results keyed by "payload" while
HybridSearch and PineconeStore both expect/return "metadata". This silently
dropped metadata from Qdrant results and caused HybridSearch.filter_by_metadata
to reject every candidate when a filter was applied (empty result sets).
Fixes#840
* fix(vector_store): stop dropping metadata for add_vectors-only backends
VectorStore.store_vectors() previously discarded the metadata argument
whenever the backend only exposed add_vectors() (e.g. FAISSStore), even
though add_vectors() supports it. Now metadata is forwarded, and is only
passed when the backend's add_vectors() signature actually accepts it
(checked via inspect.signature), avoiding a TypeError for stricter
backend signatures.
Fixes#832
* fix(vector_store): guard signature introspection in store_vectors
inspect.signature() can raise ValueError/TypeError for some callables
(e.g. certain C-implemented or dynamically built methods). Wrap the
add_vectors() signature probe in try/except, consistent with the same
pattern already used in ProvenanceManager.trace_lineage(), defaulting
to attempting to pass metadata when introspection fails.
* docs(changelog): document VectorStore metadata-drop fix (#832, #835)
* test(vector_store): add regression coverage for metadata forwarding
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
- Add SQLITE_VEC_AVAILABLE flag via importlib.util.find_spec so the test
suite's skipif actually reflects whether sqlite-vec is installed; it was
previously undefined, causing all sqlite vector store tests to be
silently skipped regardless of installation state.
- Actually apply PRAGMA synchronous=NORMAL alongside journal_mode=WAL when
use_wal=True, matching the documented behavior; document use_wal as an
opt-in kwarg in the docstring and usage guide.
- Correct _is_safe_identifier error messages (regex never allowed hyphens).
- Batch get() and update() with IN(...)/executemany instead of per-id
round trips, consistent with add()/delete().
- Fix flaky test_update_vectors assertion that relied on list.index()
over dicts containing numpy arrays.
- Reorder sqlite_vec_store import alphabetically in vector_store/__init__.py.
Co-Authored-By: Luffy2208 <209925020+Luffy2208@users.noreply.github.com>
- Keep pgvector backend integration with _init_backend_store method
- Preserve decision-specific components from main branch
- Maintain both VectorStore backend support and decision pipeline functionality
- Fix duplicate initialization and proper component placement
- CRUD unit tests
- Similarity search tests with filters
- Index creation tests (HNSW, IVFFlat)
- Docker-based PostgreSQL + pgvector support
- Tests skip if DB unavailable
- Fix variable shadowing in fetch_vectors (use vector_id instead of id)
- Remove redundant PINECONE_AVAILABLE check in create_index
- Add Pinecone imports and exports to __init__.py
- Add 'pinecone' to SUPPORTED_BACKENDS in vector_store.py
- Add vectorstore-pinecone dependency group to pyproject.toml
- Create vectorstore-all optional dependency group
- Fix duplicate MagicMock import in test_pinecone_store.py
- Update test_pinecone_removal.py with explanatory comment
- Update all docstrings to include Pinecone in supported backends
All fixes address code review feedback and ensure proper integration.
- Robust ID extraction in CentralityCalculator, CommunityDetector, and ConnectivityAnalyzer
- Support for direct Entity objects and dictionaries as node identifiers
- Improved Entity hashability in utils/types.py
- Added integration test to verify fix and prevent regression
- Removed all Pinecone references, adapters, and documentation to align with open-source, self-hosted focus.
- Removed PineconeAdapter and related dependencies.
- Updated VectorStore to enforce supported backends (FAISS, Weaviate, Qdrant, Milvus, InMemory).
- Updated cookbooks (e.g., 13_Vector_Store.ipynb) to use Weaviate/FAISS examples instead of Pinecone.
- Updated core documentation (modules.md, rchitecture.md, etc.) to reflect backend changes.
- Added new tests ( est_pinecone_removal.py, est_vector_store_deepdive.py) to verify removal and validate remaining backends.
- Verified all vector store tests pass.
Sameer6305