vector_store_config.get_all() always includes a "dimension" key, so
forwarding it via **config into VectorIndexer(dimension=dimension, **config)
raised "got multiple values for keyword argument 'dimension'" any time the
default index-creation path ran with the default config — including
`semantica embed index`, which is exactly the second half of the #994
quick-start pipeline this PR fixes.
* fix(vector_store): make VectorManager methods work on persistent backends (#855)
maintain_store() and collect_statistics() reached into VectorStore
internals (.vectors/.metadata), which only exist for the inmemory
backend — any persistent backend (FAISS, Qdrant, Pinecone, Milvus,
...) crashed with AttributeError.
Add a public backend-agnostic VectorStore.count() accessor following
the get_vector()/get_metadata() precedent (#843) and the
NotImplementedError-on-unsupported-capability precedent of
_filter_by_metadata() (#848): inmemory counts its dict, persistent
backends delegate to count() when available, and raise
NotImplementedError otherwise. VectorManager methods now go through
count(); maintain_store() keeps the exact inmemory semantics (separate
vector/metadata dict counts) and reports a 1:1 count for persistent
backends, where metadata is stored alongside each vector.
Tests: 10 hermetic unit tests covering inmemory, delegation and the
NotImplementedError path. Core vector_store suite: 40 passed.
* fix(vector_store): raise NotImplementedError when count() unavailable
Address Qodo review findings on #914:
- Persistent backend with no wrapped store no longer silently returns 0
(which masked a missing initialization as an empty, healthy store);
it now raises NotImplementedError like get_vector()/get_metadata().
- A mis-shaped adapter exposing a non-callable 'count' attribute now
surfaces a clean NotImplementedError instead of a TypeError, via a
getattr + callable() capability check.
Adds regression tests for both cases.
* fix(vector_store): implement count() on FAISS/SQLite/PgVector backends (#914)
- FAISSStore.count(): returns len(index.vector_ids); 0 when no index exists yet
- SQLiteVecStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*))
- PgVectorStore.count(): delegates to get_stats()[vector_count] (SELECT COUNT(*))
- VectorStore.count(): fix misleading NotImplementedError message; now describes
how to add count() support to a backend adapter rather than claiming only the
inmemory backend can ever support counting
- VectorManager.maintain_store(): split inmemory and persistent paths:
* inmemory: independently reads len(vectors) and len(metadata) and compares
them as an integrity check (original semantics preserved)
* persistent: calls store.count(); returns metadata_count=None because
metadata is co-located with vectors in the backend and cannot be counted
independently; never manufactures metadata_count=vector_count as a vacuous
tautology (#914 Qodo review)
- Tests: rewrite test_vector_manager_persistent.py with 31 tests covering
dispatch logic, inmemory divergence detection, persistent metadata_count=None
invariant, FAISSStore/PgVectorStore via mocks, and SQLiteVecStore via real
in-memory SQLite (skipped when sqlite-vec absent)
* docs(changelog): document VectorManager persistent-backend count fix (#914, closes#855)
Records the VectorStore.count() accessor, the FAISS/SQLite/PgVector
implementations added during review, and the maintain_store()
metadata_count fix (no longer fabricates equality for persistent backends).
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
- pinecone_store: call self.index.describe_index_stats() instead of the
nonexistent self.describe_index_stats(), and use a unit query vector
instead of an all-zero vector so filter_by_metadata() works on
cosine-metric indexes (the library's own default)
- pgvector_store: apply the existing lowercase true/false bool handling
to the list-filter branch too, and use the jsonb ?| operator so
list-valued metadata fields match on intersection instead of being
compared as a single JSON-text blob
- sqlite_vec_store: use json_each() with a json_type guard so list-valued
metadata fields match on intersection, mirroring the in-memory
backend's set-intersection semantics
- faiss_store: filter_by_metadata(limit=0) now returns [] instead of one
result
- milvus_store: reject NaN/Infinity filter values up front with a clear
ValidationError instead of building an invalid expression that gets
silently swallowed
- update the #848 FAISS NotImplementedError test to reflect that FAISS
now implements real filter_by_metadata() (this PR's whole point)
- add regression tests for each fix; sqlite tests run against the real
sqlite-vec extension
- vector_store.save(): use v.tolist() instead of list(v) so numpy float32
vectors round-trip through JSON instead of raising TypeError.
- ontology._fetch_url_sync(): resolve relative Location headers via urljoin
before re-validating (previously any relative redirect was rejected
outright), and close every response instead of leaking the connection
across redirect hops.
- sparql.execute_sparql(): move _build_rdflib_graph inside the handler's
error handling so the graph-size cap returns a clean SparqlResponse
error instead of an unhandled 500.
- add regression tests for all three.
The 1.0 / (1.0 + max(0.0, 1.0 - score)) normalization added in the last
commit clamped every raw score >= 1.0 to an identical 1.0, collapsing
result ranking for dot-product-metric indexes (unbounded), which cosine
(bounded to [-1, 1]) never exercised. Replaced with x/(1+|x|) rescaled
to (0, 1), which is strictly monotonic for any real score.
Also adds regression tests for scores >= 1 and a CHANGELOG entry.
Adds similarity_unavailable marker and warning logs to build_decision_context and explain_decision when a persistent backend (like FAISS) fails to reconstruct a vector. Updates docstrings to explicitly state this degraded-path behavior and guarantees schema stability. Adds regression tests to test vector retrieval failure behavior via caplog and context assertions.
- build_decision_context() and explain_decision(include_paths=True) both
accessed self.vectors directly, which is only initialized for the
inmemory backend, crashing with AttributeError on any persistent
backend (FAISS, Qdrant, Pinecone, etc.). Replaced with self.get_vector()
(#843's backend-agnostic accessor) + an is-not-None check — a verified
1:1 behavioral equivalent for the old 'decision_id in self.vectors'
guard on the inmemory path.
- Found a third, undocumented instance of the same bug during
verification: _filter_by_metadata() also accessed self.metadata/
self.vectors directly. Initial fix silently returned [] for persistent
backends, which was itself a new silent-failure bug (indistinguishable
from a genuine zero-match result). Reconciled to raise
NotImplementedError instead, matching the established precedent from
get_vector()/get_metadata() (#843) for 'backend exists but doesn't
support this operation' — confirmed via full grep of all 7 backend
wrapper classes that none currently implement filter_by_metadata,
so this path was previously dead-code-masked-as-working.
Tests: 14 new tests across two rounds — inmemory behavioral equivalence,
real (non-mocked) FAISS backend regression tests for all three methods,
and explicit coverage proving the NotImplementedError fires with a clear
message rather than the old silent-[] behavior. Full suite: 53 passed,
0 failed, 0 regressions across the 39 pre-existing tests.
- Removed total=False from SearchResult TypedDict so all fields are strictly required
- Ensured distance: None is returned from backends that don't natively expose distance (Qdrant, Pinecone, SQLite, pgvector, in-memory)
- Standardized search result score to a consistent 0.0 - 1.0 similarity metric scale across all backend adapters
- Relaxed SearchResult id type to Union[str, int] to accommodate native integer IDs from Milvus and Qdrant without casting
- Updated schema verification tests
_get_candidate_embeddings()'s expand-and-retry loop widens the search
pool (up to limit*10) when post-filtering leaves too few candidates.
If the backend keeps returning a full page and filtered matches never
reach `limit`, the loop exited via the while condition instead of the
break branch, so the pre-loop empty embeddings/metadata/scores lists
were returned instead of the matches actually found in the final
iteration. This silently returned [] for filtered queries against
large persistent-backend stores even when matches existed - exactly
the scenario this PR adds support for.
Falls back to the last collected batch instead of discarding it.
Also documents this PR and #839 in the changelog.
- FAISSStore: get_metadata now correctly retrieves from self.metadata instead of raising NotImplementedError.
- MilvusStore:
- Changed schema to support String IDs (VARCHAR) instead of auto-generated INT64, preventing loss of IDs during insert.
- Added metadata storage using JSON.
- Replaced insert_vectors with add_vectors accepting ids and metadata (added insert_vectors alias for backward compatibility).
- Implemented get_vector and get_metadata with safe parameterized querying to prevent query injection.
- PgVectorStore & SQLiteVecStore:
- Fixed get_vector and get_metadata to call self.get([vector_id]) instead of the non-existent get_vectors([vector_id]), fixing the silent None return bug.
- VectorStore.get_vector() and get_metadata() were hardcoded to access
self.vectors and self.metadata dicts, which are only initialized for
the inmemory backend, causing AttributeError on all persistent backends
(FAISS, Qdrant, Pinecone, Milvus, Weaviate, PgVector, SQLiteVec).
Changes:
- Refactor VectorStore.get_vector() and get_metadata() to branch on
self.backend == 'inmemory' (zero behavior change) and delegate to
self._backend_store otherwise.
- Harden save() to use getattr(self, 'vectors', {}) / getattr(self,
'metadata', {}) to prevent crash when saving a persistent backend store.
- Add get_vector() and get_metadata() to all 7 backend wrappers:
- FAISSStore: get_vector uses index.reconstruct(); get_metadata raises
NotImplementedError (FAISS has no metadata storage natively).
- QdrantStore: uses client.retrieve() with with_vectors/with_payload.
- PineconeStore: wraps existing fetch_vectors() call.
- MilvusStore: raises NotImplementedError (auto_id=True schema discards
string IDs at insert time, making by-ID lookup impossible in this
wrapper's current schema).
- WeaviateStore: uses collection.query.fetch_object_by_id().
- PgVectorStore: wraps existing get_vectors() SQL method.
- SQLiteVecStore: wraps existing get_vectors() SQL method.
- Add TestVectorStoreRetrieval regression tests covering inmemory and
FAISS backends with real (non-mocked) assertions.
All 28 tests pass.
- Replace direct .vectors and .metadata access with VectorStore.search_vectors().
- Add a fallback in HybridSimilarityCalculator (via ind_similar_decisions) to use the search score when backend vector databases do not natively return the raw vector array.
- Fix get_decision_statistics to gracefully fall back when .metadata is not fully supported by the underlying DB.
- Add regression tests utilizing the real FAISS and inmemory backends directly without mocking.
QdrantStore.search_vectors() returned results keyed by "payload" while
HybridSearch and PineconeStore both expect/return "metadata". This silently
dropped metadata from Qdrant results and caused HybridSearch.filter_by_metadata
to reject every candidate when a filter was applied (empty result sets).
Fixes#840
* fix(vector_store): stop dropping metadata for add_vectors-only backends
VectorStore.store_vectors() previously discarded the metadata argument
whenever the backend only exposed add_vectors() (e.g. FAISSStore), even
though add_vectors() supports it. Now metadata is forwarded, and is only
passed when the backend's add_vectors() signature actually accepts it
(checked via inspect.signature), avoiding a TypeError for stricter
backend signatures.
Fixes#832
* fix(vector_store): guard signature introspection in store_vectors
inspect.signature() can raise ValueError/TypeError for some callables
(e.g. certain C-implemented or dynamically built methods). Wrap the
add_vectors() signature probe in try/except, consistent with the same
pattern already used in ProvenanceManager.trace_lineage(), defaulting
to attempting to pass metadata when introspection fails.
* docs(changelog): document VectorStore metadata-drop fix (#832, #835)
* test(vector_store): add regression coverage for metadata forwarding
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
- Add SQLITE_VEC_AVAILABLE flag via importlib.util.find_spec so the test
suite's skipif actually reflects whether sqlite-vec is installed; it was
previously undefined, causing all sqlite vector store tests to be
silently skipped regardless of installation state.
- Actually apply PRAGMA synchronous=NORMAL alongside journal_mode=WAL when
use_wal=True, matching the documented behavior; document use_wal as an
opt-in kwarg in the docstring and usage guide.
- Correct _is_safe_identifier error messages (regex never allowed hyphens).
- Batch get() and update() with IN(...)/executemany instead of per-id
round trips, consistent with add()/delete().
- Fix flaky test_update_vectors assertion that relied on list.index()
over dicts containing numpy arrays.
- Reorder sqlite_vec_store import alphabetically in vector_store/__init__.py.
Co-Authored-By: Luffy2208 <209925020+Luffy2208@users.noreply.github.com>
- Keep pgvector backend integration with _init_backend_store method
- Preserve decision-specific components from main branch
- Maintain both VectorStore backend support and decision pipeline functionality
- Fix duplicate initialization and proper component placement
- CRUD unit tests
- Similarity search tests with filters
- Index creation tests (HNSW, IVFFlat)
- Docker-based PostgreSQL + pgvector support
- Tests skip if DB unavailable
- Fix variable shadowing in fetch_vectors (use vector_id instead of id)
- Remove redundant PINECONE_AVAILABLE check in create_index
- Add Pinecone imports and exports to __init__.py
- Add 'pinecone' to SUPPORTED_BACKENDS in vector_store.py
- Add vectorstore-pinecone dependency group to pyproject.toml
- Create vectorstore-all optional dependency group
- Fix duplicate MagicMock import in test_pinecone_store.py
- Update test_pinecone_removal.py with explanatory comment
- Update all docstrings to include Pinecone in supported backends
All fixes address code review feedback and ensure proper integration.
- Robust ID extraction in CentralityCalculator, CommunityDetector, and ConnectivityAnalyzer
- Support for direct Entity objects and dictionaries as node identifiers
- Improved Entity hashability in utils/types.py
- Added integration test to verify fix and prevent regression
- Removed all Pinecone references, adapters, and documentation to align with open-source, self-hosted focus.
- Removed PineconeAdapter and related dependencies.
- Updated VectorStore to enforce supported backends (FAISS, Weaviate, Qdrant, Milvus, InMemory).
- Updated cookbooks (e.g., 13_Vector_Store.ipynb) to use Weaviate/FAISS examples instead of Pinecone.
- Updated core documentation (modules.md, rchitecture.md, etc.) to reflect backend changes.
- Added new tests ( est_pinecone_removal.py, est_vector_store_deepdive.py) to verify removal and validate remaining backends.
- Verified all vector store tests pass.
KaifAhmad1