* security: require API-key auth on all Explorer API routes (GHSA-j4mq-hprp-987v)
Every Explorer route (bulk import/export, delete, LLM-backed ontology
generation, SPARQL, etc.) was mounted with no authentication, and both
server entrypoints bind 0.0.0.0 by default. Anyone reaching the port got
full read/write/delete on the graph.
- Add require_auth dependency (explorer/dependencies.py): checks
X-API-Key against SEMANTICA_API_KEY, fails closed with 503 if
unconfigured (not silently anonymous), 401 on wrong/missing key.
SEMANTICA_ALLOW_ANONYMOUS=true opts out explicitly for local dev.
- Wire dependencies=[Depends(require_auth)] into all 11 API routers in
both explorer/app.py and server.py. /health, /api/info, static assets,
and the SPA catch-all stay public.
- /ws/graph-updates handshake now checks the same key via header or
?api_key= query param (browsers can't set custom WS headers) before
accepting the connection.
- Default bind changed from 0.0.0.0 to 127.0.0.1 in server.py's main()
and cli.py's `server start`; the CLI warns if a non-loopback host is
passed explicitly without a key configured.
- Startup logging reports the resolved auth mode in both app factories.
- Document/generate SEMANTICA_API_KEY in the deploy recipes that expose
a public endpoint by default: docker-compose, Railway, Fly, Render.
Added tests/explorer/test_explorer_auth.py covering fail-closed default,
wrong/missing/correct key, anonymous opt-in, public-route exemptions, and
the WS handshake. Added tests/explorer/conftest.py defaulting the
pre-existing ~200 explorer tests to SEMANTICA_ALLOW_ANONYMOUS=true so
they keep exercising route logic without needing a key.
* fix CORS
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(ingest): add SSRF protection for WebIngestor and RESTIngestor
Block non-http(s) schemes and private/loopback/link-local targets before
outbound requests, with allow_private_ips opt-in for trusted deployments.
* fix(ingest): fail closed on SSRF DNS resolution errors
* fix(ingest): validate SSRF targets on every HTTP redirect hop
* fix(ingest): avoid blocking on SSRF DNS executor shutdown
* fix(ingest): parse allow_private_ips without truthy-string pitfalls
* docs(ingest): clarify robots.txt SSRF/validation comment
---------
Co-authored-by: Pravit Ampapathini
* test(deduplication): cover ClusterBuilder, MergeStrategyManager, and batch paths
Add focused coverage for union-find clustering, property merge rules,
merge_duplicates, embedding similarity, and incremental detection (#866).
* test(deduplication): assert unrelated clusters without conditional skip
Make cluster-separation coverage fail closed by using mocked pairs and
unconditional assertions for distinct Apple vs Microsoft cluster IDs.
* test(deduplication): tighten update_clusters attachment assertions
Require the incremental path to place the new near-duplicate in the
same rebuilt cluster instead of accepting a vacuous cluster-count check.
* test(deduplication): strengthen incremental detect_duplicates wrapper checks
Assert real DuplicateCandidate matches, score threshold, and new×existing
routing instead of only checking that the wrapper returns a list.
* test(deduplication): verify metadata provenance behavior
Assert that preserve_provenance writes metadata.provenance fields and add a disabled-path test so regressions do not pass through merge_entities metadata alone.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@users.noreply.github.com>
The 1.0 / (1.0 + max(0.0, 1.0 - score)) normalization added in the last
commit clamped every raw score >= 1.0 to an identical 1.0, collapsing
result ranking for dot-product-metric indexes (unbounded), which cosine
(bounded to [-1, 1]) never exercised. Replaced with x/(1+|x|) rescaled
to (0, 1), which is strictly monotonic for any real score.
Also adds regression tests for scores >= 1 and a CHANGELOG entry.
Adds similarity_unavailable marker and warning logs to build_decision_context and explain_decision when a persistent backend (like FAISS) fails to reconstruct a vector. Updates docstrings to explicitly state this degraded-path behavior and guarantees schema stability. Adds regression tests to test vector retrieval failure behavior via caplog and context assertions.
- build_decision_context() and explain_decision(include_paths=True) both
accessed self.vectors directly, which is only initialized for the
inmemory backend, crashing with AttributeError on any persistent
backend (FAISS, Qdrant, Pinecone, etc.). Replaced with self.get_vector()
(#843's backend-agnostic accessor) + an is-not-None check — a verified
1:1 behavioral equivalent for the old 'decision_id in self.vectors'
guard on the inmemory path.
- Found a third, undocumented instance of the same bug during
verification: _filter_by_metadata() also accessed self.metadata/
self.vectors directly. Initial fix silently returned [] for persistent
backends, which was itself a new silent-failure bug (indistinguishable
from a genuine zero-match result). Reconciled to raise
NotImplementedError instead, matching the established precedent from
get_vector()/get_metadata() (#843) for 'backend exists but doesn't
support this operation' — confirmed via full grep of all 7 backend
wrapper classes that none currently implement filter_by_metadata,
so this path was previously dead-code-masked-as-working.
Tests: 14 new tests across two rounds — inmemory behavioral equivalence,
real (non-mocked) FAISS backend regression tests for all three methods,
and explicit coverage proving the NotImplementedError fires with a clear
message rather than the old silent-[] behavior. Full suite: 53 passed,
0 failed, 0 regressions across the 39 pre-existing tests.
self.indexer is only set for backend="inmemory", so save()/load() still
raised AttributeError for persistent backends (faiss, qdrant, etc.) even
after this PR's getattr() guards on self.vectors/self.metadata, since the
unguarded `self.indexer` access happened first. Guard it the same way and
delegate to the backend store's native save_index/load_index (currently
only FAISSStore implements these) so persistent-backend saves actually
persist instead of silently no-oping.
- Removed total=False from SearchResult TypedDict so all fields are strictly required
- Ensured distance: None is returned from backends that don't natively expose distance (Qdrant, Pinecone, SQLite, pgvector, in-memory)
- Standardized search result score to a consistent 0.0 - 1.0 similarity metric scale across all backend adapters
- Relaxed SearchResult id type to Union[str, int] to accommodate native integer IDs from Milvus and Qdrant without casting
- Updated schema verification tests
_get_candidate_embeddings()'s expand-and-retry loop widens the search
pool (up to limit*10) when post-filtering leaves too few candidates.
If the backend keeps returning a full page and filtered matches never
reach `limit`, the loop exited via the while condition instead of the
break branch, so the pre-loop empty embeddings/metadata/scores lists
were returned instead of the matches actually found in the final
iteration. This silently returned [] for filtered queries against
large persistent-backend stores even when matches existed - exactly
the scenario this PR adds support for.
Falls back to the last collected batch instead of discarding it.
Also documents this PR and #839 in the changelog.
- Added insert_vectors alias to add_vectors for backward compatibility.
- Sanitized vector_id in get_vector and get_metadata to prevent query injection.
- FAISSStore: get_metadata now correctly retrieves from self.metadata instead of raising NotImplementedError.
- MilvusStore:
- Changed schema to support String IDs (VARCHAR) instead of auto-generated INT64, preventing loss of IDs during insert.
- Added metadata storage using JSON.
- Replaced insert_vectors with add_vectors accepting ids and metadata (added insert_vectors alias for backward compatibility).
- Implemented get_vector and get_metadata with safe parameterized querying to prevent query injection.
- PgVectorStore & SQLiteVecStore:
- Fixed get_vector and get_metadata to call self.get([vector_id]) instead of the non-existent get_vectors([vector_id]), fixing the silent None return bug.
- VectorStore.get_vector() and get_metadata() were hardcoded to access
self.vectors and self.metadata dicts, which are only initialized for
the inmemory backend, causing AttributeError on all persistent backends
(FAISS, Qdrant, Pinecone, Milvus, Weaviate, PgVector, SQLiteVec).
Changes:
- Refactor VectorStore.get_vector() and get_metadata() to branch on
self.backend == 'inmemory' (zero behavior change) and delegate to
self._backend_store otherwise.
- Harden save() to use getattr(self, 'vectors', {}) / getattr(self,
'metadata', {}) to prevent crash when saving a persistent backend store.
- Add get_vector() and get_metadata() to all 7 backend wrappers:
- FAISSStore: get_vector uses index.reconstruct(); get_metadata raises
NotImplementedError (FAISS has no metadata storage natively).
- QdrantStore: uses client.retrieve() with with_vectors/with_payload.
- PineconeStore: wraps existing fetch_vectors() call.
- MilvusStore: raises NotImplementedError (auto_id=True schema discards
string IDs at insert time, making by-ID lookup impossible in this
wrapper's current schema).
- WeaviateStore: uses collection.query.fetch_object_by_id().
- PgVectorStore: wraps existing get_vectors() SQL method.
- SQLiteVecStore: wraps existing get_vectors() SQL method.
- Add TestVectorStoreRetrieval regression tests covering inmemory and
FAISS backends with real (non-mocked) assertions.
All 28 tests pass.
- Replace direct .vectors and .metadata access with VectorStore.search_vectors().
- Add a fallback in HybridSimilarityCalculator (via ind_similar_decisions) to use the search score when backend vector databases do not natively return the raw vector array.
- Fix get_decision_statistics to gracefully fall back when .metadata is not fully supported by the underlying DB.
- Add regression tests utilizing the real FAISS and inmemory backends directly without mocking.
Adds an Unreleased/Fixed entry for #841 (closes#840) — QdrantStore
search results were keyed "payload" instead of "metadata", breaking
HybridSearch.filter_by_metadata() for Qdrant results.
Adds an Unreleased/Added entry for #838 (closes#834), including the
follow-up fix that preserves ImportError for a missing pyoxigraph
install instead of masking it as a generic ProcessingError.
Upstream moved the v4 tag to 5595ccaf912efad79be6eef63a5619ff05969be3
(v4.37.6), which the repo's own verify-action-pins.sh now (correctly)
flags as a mismatch against the previously-pinned commit. Pre-existing
drift unrelated to #830/#836, but it was failing this PR's required
"verify" check, so fixing it here.
- Wire the Explorer frontend's node --test suites (test:graph-store,
test:graph-workspace, and the new test:plugin-registry regression
test) into CI. Previously only `npm run build` ran, so none of the
frontend tests -- including this fix's own regression coverage --
executed anywhere except a contributor's local machine.
- Broaden the diagnostics dedup's structureLayer comparison to also
cover disabledReason/curveCount/bridgeCurveCount/backboneCurveCount,
not just cacheKey/lastDrawAt/enabled, so a disabledReason-only
transition doesn't leave the dev diagnostics panel stale.
QdrantStore.search_vectors() returned results keyed by "payload" while
HybridSearch and PineconeStore both expect/return "metadata". This silently
dropped metadata from Qdrant results and caused HybridSearch.filter_by_metadata
to reject every candidate when a filter was applied (empty result sets).
Fixes#840
