_get_candidate_embeddings()'s expand-and-retry loop widens the search
pool (up to limit*10) when post-filtering leaves too few candidates.
If the backend keeps returning a full page and filtered matches never
reach `limit`, the loop exited via the while condition instead of the
break branch, so the pre-loop empty embeddings/metadata/scores lists
were returned instead of the matches actually found in the final
iteration. This silently returned [] for filtered queries against
large persistent-backend stores even when matches existed - exactly
the scenario this PR adds support for.
Falls back to the last collected batch instead of discarding it.
Also documents this PR and #839 in the changelog.
- Added insert_vectors alias to add_vectors for backward compatibility.
- Sanitized vector_id in get_vector and get_metadata to prevent query injection.
- FAISSStore: get_metadata now correctly retrieves from self.metadata instead of raising NotImplementedError.
- MilvusStore:
- Changed schema to support String IDs (VARCHAR) instead of auto-generated INT64, preventing loss of IDs during insert.
- Added metadata storage using JSON.
- Replaced insert_vectors with add_vectors accepting ids and metadata (added insert_vectors alias for backward compatibility).
- Implemented get_vector and get_metadata with safe parameterized querying to prevent query injection.
- PgVectorStore & SQLiteVecStore:
- Fixed get_vector and get_metadata to call self.get([vector_id]) instead of the non-existent get_vectors([vector_id]), fixing the silent None return bug.
- VectorStore.get_vector() and get_metadata() were hardcoded to access
self.vectors and self.metadata dicts, which are only initialized for
the inmemory backend, causing AttributeError on all persistent backends
(FAISS, Qdrant, Pinecone, Milvus, Weaviate, PgVector, SQLiteVec).
Changes:
- Refactor VectorStore.get_vector() and get_metadata() to branch on
self.backend == 'inmemory' (zero behavior change) and delegate to
self._backend_store otherwise.
- Harden save() to use getattr(self, 'vectors', {}) / getattr(self,
'metadata', {}) to prevent crash when saving a persistent backend store.
- Add get_vector() and get_metadata() to all 7 backend wrappers:
- FAISSStore: get_vector uses index.reconstruct(); get_metadata raises
NotImplementedError (FAISS has no metadata storage natively).
- QdrantStore: uses client.retrieve() with with_vectors/with_payload.
- PineconeStore: wraps existing fetch_vectors() call.
- MilvusStore: raises NotImplementedError (auto_id=True schema discards
string IDs at insert time, making by-ID lookup impossible in this
wrapper's current schema).
- WeaviateStore: uses collection.query.fetch_object_by_id().
- PgVectorStore: wraps existing get_vectors() SQL method.
- SQLiteVecStore: wraps existing get_vectors() SQL method.
- Add TestVectorStoreRetrieval regression tests covering inmemory and
FAISS backends with real (non-mocked) assertions.
All 28 tests pass.
- Replace direct .vectors and .metadata access with VectorStore.search_vectors().
- Add a fallback in HybridSimilarityCalculator (via ind_similar_decisions) to use the search score when backend vector databases do not natively return the raw vector array.
- Fix get_decision_statistics to gracefully fall back when .metadata is not fully supported by the underlying DB.
- Add regression tests utilizing the real FAISS and inmemory backends directly without mocking.
Adds an Unreleased/Fixed entry for #841 (closes#840) — QdrantStore
search results were keyed "payload" instead of "metadata", breaking
HybridSearch.filter_by_metadata() for Qdrant results.
Adds an Unreleased/Added entry for #838 (closes#834), including the
follow-up fix that preserves ImportError for a missing pyoxigraph
install instead of masking it as a generic ProcessingError.
Upstream moved the v4 tag to 5595ccaf912efad79be6eef63a5619ff05969be3
(v4.37.6), which the repo's own verify-action-pins.sh now (correctly)
flags as a mismatch against the previously-pinned commit. Pre-existing
drift unrelated to #830/#836, but it was failing this PR's required
"verify" check, so fixing it here.
- Wire the Explorer frontend's node --test suites (test:graph-store,
test:graph-workspace, and the new test:plugin-registry regression
test) into CI. Previously only `npm run build` ran, so none of the
frontend tests -- including this fix's own regression coverage --
executed anywhere except a contributor's local machine.
- Broaden the diagnostics dedup's structureLayer comparison to also
cover disabledReason/curveCount/bridgeCurveCount/backboneCurveCount,
not just cacheKey/lastDrawAt/enabled, so a disabledReason-only
transition doesn't leave the dev diagnostics panel stale.
QdrantStore.search_vectors() returned results keyed by "payload" while
HybridSearch and PineconeStore both expect/return "metadata". This silently
dropped metadata from Qdrant results and caused HybridSearch.filter_by_metadata
to reject every candidate when a filter was applied (empty result sets).
Fixes#840
* fix(vector_store): stop dropping metadata for add_vectors-only backends
VectorStore.store_vectors() previously discarded the metadata argument
whenever the backend only exposed add_vectors() (e.g. FAISSStore), even
though add_vectors() supports it. Now metadata is forwarded, and is only
passed when the backend's add_vectors() signature actually accepts it
(checked via inspect.signature), avoiding a TypeError for stricter
backend signatures.
Fixes#832
* fix(vector_store): guard signature introspection in store_vectors
inspect.signature() can raise ValueError/TypeError for some callables
(e.g. certain C-implemented or dynamically built methods). Wrap the
add_vectors() signature probe in try/except, consistent with the same
pattern already used in ProvenanceManager.trace_lineage(), defaulting
to attempting to pass metadata when introspection fails.
* docs(changelog): document VectorStore metadata-drop fix (#832, #835)
* test(vector_store): add regression coverage for metadata forwarding
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
- pluginRegistryPredicates.ts: consolidate 8-line JSDoc to 5 lines,
removing redundant detail that restated implementation mechanics
already obvious from the code.
- GraphWorkspace.tsx: shorten the lastScrubberMsRef comment from 5 lines
to 2; trim the handleDiagnosticsChange block comment by removing the
'rather than bailing out' implementation-alternative sentence; tighten
the distanceVisual inline comment.
- pluginRegistry.temporal.test.mjs: replace 17-line file-level JSDoc
with 9 lines focused on the invariant rather than the root-cause
narrative (already covered in pluginRegistryPredicates.ts); remove
two tsx loader implementation-detail comments; tighten two test-level
inline comments.
No logic, types, or test assertions changed. All 42 tests pass.
- Legacy top_k kwarg was read but not removed from options, so it got
forwarded via **options into VectorStore.search_vectors(), colliding
with backends that call search(..., top_k=k, **options) (e.g. sqlite,
pgvector) and raising "got multiple values for keyword argument
'top_k'". Now popped instead of just read.
- VectorStore.search_vectors()'s dispatch only recognized backend
methods named search/search_similar, so HybridSearch's delegation
still hit NotImplementedError for qdrant/milvus/pinecone, which name
their method search_vectors() with a differently-named count
parameter (limit vs k). Added a third dispatch branch that binds the
count positionally so it works regardless of the backend's parameter
name.
- Backend-delegated results defaulted a missing "distance" to the raw
score, silently reusing the local path's cosine-similarity convention
(distance = 1 - score) even for backends using unrelated metrics
(L2, inner product). A missing distance is now left as None instead
of a fabricated, metric-inconsistent value.
Two issues addressed:
1. Plugin-loading useEffect unnecessarily depended on temporalState.
After the #830 fix, no shouldLoad predicate reads temporalState, but
the effect's dep array still included it, causing extra re-runs on
every scrubber update. Removed temporalState from the dep array and
the shouldLoad call site. Made temporalState optional in the
LazyPluginRegistryEntry shouldLoad context type to match.
2. Regression test imported a local copy of shouldLoad instead of the
production predicate. Extracted all three shouldLoad predicates into
pluginRegistryPredicates.ts (pure module, no React/DOM dependencies),
wired GraphWorkspace.tsx to use the imported functions, and updated
the test to import and exercise the real production code via tsx.
Verified: introducing the old broken condition causes the test to fail;
the correct implementation passes all 7 assertions.
HybridSearch.search() directly accessed self.vector_store.vectors, a dict
that VectorStore only creates for backend="inmemory". Every other backend
(faiss, weaviate, qdrant, milvus, pinecone, pgvector, sqlite) raised
AttributeError. It now delegates to VectorStore.search_vectors() for
non-inmemory backends, applying metadata_filter as a post-filter and
normalizing results to a consistent {id, score, distance, metadata} shape.
Also fixes two related bugs surfaced while testing the backend-delegated
path end to end:
- vector_ids could remain None when explicit vectors/metadata were passed
without vector_ids, crashing downstream indexing.
- query_vector passed as a plain list crashed backend stores (e.g.
FAISSStore.search_similar) that call .ndim on it; now normalized to a
numpy array up front.
And in vector_store.py: VectorStore.store_vectors() silently dropped
metadata for FAISS (and any add_vectors-only backend) because it called
add_vectors(vectors, **options) without forwarding metadata, even though
FAISSStore.add_vectors() accepts it. This blocked HybridSearch's metadata
filtering from working at all against FAISS.
Fixes#833
temporalDiffState.ts belongs to feat/793-temporal-diff-ui and should not
appear in the #830 diff. Remove it from this branch's tracked files.
Add the pluginRegistry.temporal.test.mjs regression test that covers the
shouldLoad fix committed in the main #830 commit (it was never committed).
Add test:plugin-registry script to package.json so the regression test
can be run via npm run test:plugin-registry.
Two independent render loops were causing the Temporal panel to remain
stuck on 'Loading temporal...' in npm run dev:
Loop 1 — diagnostics state churn (GraphWorkspace.tsx):
handleDiagnosticsChange unconditionally called setGraphDiagnosticsState
with a new object on every invocation. buildEffectAvailability (called
inside GraphCanvas's diagnostics useEffect) always returns a new object,
so setGraphDiagnosticsState was called on every effect run, creating a
cycle: setGraphDiagnosticsState graphDiagnosticsState new
diagnosticsSnapshot new pluginContext new handleInteractionStateChange
new GraphCanvas re-renders diagnostics effect fires again.
Fix: before calling setGraphDiagnosticsState, compare the incoming
diagnostics field-by-field against the last accepted snapshot via a ref
(lastDiagnosticsRef). All effectAvailability entries, edgeClasses.updatedAt,
structureLayer.cacheKey/lastDrawAt/enabled, and distanceVisual identity
must differ for a state update to proceed. The ref approach avoids
scheduling a re-render at all, rather than bailing out inside a functional
updater after the render has already been committed.
Loop 2 — scrubberTime churn (GraphWorkspace.tsx + GraphWorkspaceShell.tsx):
TimelinePanel.tsx calls onTimeChange(defaultTime) whenever its useEffect
re-runs. React 18 concurrent mode re-runs effects with structurally-new
Date objects for the same timestamp when speculative renders discard
useMemo caches, causing setScrubberTime to be called repeatedly with a
new Date that has the same millisecond value — triggering temporalState
churn, the diagnostics effect, and eventually the same loop.
Fix: wrap setScrubberTime in an onTimeChange useCallback that compares the
incoming time's millisecond value against the last sent value (via
lastScrubberMsRef). Redundant calls with the same timestamp are dropped
before reaching setScrubberTime. Stable useCallback identity also prevents
TimelinePanel's useEffect from re-firing solely due to prop identity churn.
Both fixes applied to GraphWorkspace.tsx and identically to
GraphWorkspaceShell.tsx which has the same pattern.
Verified:
- npm run dev: 0 'Maximum update depth exceeded' errors
- Temporal panel renders with real data in dev mode
- Effects and Neighbors panels unaffected
- npm run build + preview: identical behavior, 0 errors
- All 42 frontend tests pass (34 graph-workspace, 1 graph-store, 7 plugin-registry)
fixed qodo review
applyDiffHighlight/clearDiffHighlight were writing baseColor only to
graphStore.graph (the store singleton), but Sigma is constructed with
displayGraphRef.current and the nodeReducer reads attributes from that
instance. When the display graph is a derived copy (aggregated,
focused, or grouped view), the store write has no effect on the
currently-rendered frame -- sigma.scheduleRefresh() flushes the
reducer over the display graph, which did not receive the mutation.
Fix: introduce writeBaseColor(context, nodeId, color) which writes to
BOTH the store graph (so the color propagates into the next display
graph rebuild via aggregateDisplayGraph's shallow attribute copy) AND
context.displayGraph (the live Graph instance currently bound to
Sigma, so the change is visible in the current frame immediately).
The dg !== graph guard skips the display-graph write when they happen
to be the same object (non-aggregated full view), avoiding a redundant
double-write in that case.
Original baseColor is still captured from the store graph (the
authoritative source, since aggregateDisplayGraph copies from there),
so restore remains correct across all view modes.
Adds a Compare section to the existing Temporal Context panel
(temporalOverlayPlugin.tsx) that lets a user pick two ISO timestamps
and diff the graph's node set between them via the existing, previously
UI-less GET /api/temporal/diff backend route.
- New temporalDiffState.ts: typed fetch wrapper (fetchTemporalDiff)
matching the route's added_nodes/removed_nodes response shape.
- Diff results recolor affected nodes via baseColor (not
ringColor/haloColor -- traced and confirmed those are only read by
the sigma reducer for hovered/selected/path-state nodes and are
silently discarded for default-state nodes).
- Validates both timestamps are present, parseable, and from < to
before firing a request.
- Distinct idle/loading/error/empty/success states -- an empty diff
(no changes) is rendered as its own state, not as an error.
- Cancels any in-flight request via AbortController on re-submission
and on unmount; restores each highlighted node's original baseColor
(captured before overwrite, not cleared to a fallback default) on
both paths.
- Reuses existing theme tokens (GRAPH_THEME.palette.semantic[2],
ui.control.dangerText) and existing button/input/loading/error
visual patterns already established in this same plugins directory
and in GraphInspectorPanel.tsx, rather than introducing new styling.
- SQLiteStorage now migrates an existing (pre-#825) provenance.db in place
via ALTER TABLE ADD COLUMN for any columns introduced since, instead of
only ever running CREATE TABLE IF NOT EXISTS. Without this, opening an
older database with the new code would break on the first insert/select
since the row width and _row_to_entry's fixed indices grew past the old
schema. Added test_migrates_pre_existing_old_schema_database.
- verify_chain() now also checks that sequence_id is exactly the
predecessor's plus one (no gap, no duplicate), in addition to the existing
previous_checksum comparison. Hardens against the narrow case where
compute_checksum()'s deliberate exclusion of entity_id could let two
distinct rows coincidentally share a checksum, which alone would let a
checksum-only comparison miss a gap. Added
test_verify_chain_detects_tampered_sequence_gap.
- Explorer provenance route: edge ids now include direction
(f"{src}-{eid}-{direction}") to match the seen_edges dedupe key, which
already included it. The same (src, target) pair can legitimately appear
in both the upstream and downstream chains (cycles/overlap), and without
this the two edges collided on the same id. Added
test_add_chain_edges_ids_distinguish_direction.
- Removed an unused `Any` import in parse_provenance.py.
Sameer6305