The previous rework of the redirect loop closed the response on each
redirect hop but dropped the try/finally around the success path, so the
terminal response (the one actually read and returned) was left
unclosed, leaking the connection back to the pool unclosed under load.
* security: require API-key auth on all Explorer API routes (GHSA-j4mq-hprp-987v)
Every Explorer route (bulk import/export, delete, LLM-backed ontology
generation, SPARQL, etc.) was mounted with no authentication, and both
server entrypoints bind 0.0.0.0 by default. Anyone reaching the port got
full read/write/delete on the graph.
- Add require_auth dependency (explorer/dependencies.py): checks
X-API-Key against SEMANTICA_API_KEY, fails closed with 503 if
unconfigured (not silently anonymous), 401 on wrong/missing key.
SEMANTICA_ALLOW_ANONYMOUS=true opts out explicitly for local dev.
- Wire dependencies=[Depends(require_auth)] into all 11 API routers in
both explorer/app.py and server.py. /health, /api/info, static assets,
and the SPA catch-all stay public.
- /ws/graph-updates handshake now checks the same key via header or
?api_key= query param (browsers can't set custom WS headers) before
accepting the connection.
- Default bind changed from 0.0.0.0 to 127.0.0.1 in server.py's main()
and cli.py's `server start`; the CLI warns if a non-loopback host is
passed explicitly without a key configured.
- Startup logging reports the resolved auth mode in both app factories.
- Document/generate SEMANTICA_API_KEY in the deploy recipes that expose
a public endpoint by default: docker-compose, Railway, Fly, Render.
Added tests/explorer/test_explorer_auth.py covering fail-closed default,
wrong/missing/correct key, anonymous opt-in, public-route exemptions, and
the WS handshake. Added tests/explorer/conftest.py defaulting the
pre-existing ~200 explorer tests to SEMANTICA_ALLOW_ANONYMOUS=true so
they keep exercising route logic without needing a key.
* fix CORS
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(ingest): add SSRF protection for WebIngestor and RESTIngestor
Block non-http(s) schemes and private/loopback/link-local targets before
outbound requests, with allow_private_ips opt-in for trusted deployments.
* fix(ingest): fail closed on SSRF DNS resolution errors
* fix(ingest): validate SSRF targets on every HTTP redirect hop
* fix(ingest): avoid blocking on SSRF DNS executor shutdown
* fix(ingest): parse allow_private_ips without truthy-string pitfalls
* docs(ingest): clarify robots.txt SSRF/validation comment
---------
Co-authored-by: Pravit Ampapathini
- vector_store.save(): use v.tolist() instead of list(v) so numpy float32
vectors round-trip through JSON instead of raising TypeError.
- ontology._fetch_url_sync(): resolve relative Location headers via urljoin
before re-validating (previously any relative redirect was rejected
outright), and close every response instead of leaking the connection
across redirect hops.
- sparql.execute_sparql(): move _build_rdflib_graph inside the handler's
error handling so the graph-size cap returns a clean SparqlResponse
error instead of an unhandled 500.
- add regression tests for all three.
* test(deduplication): cover ClusterBuilder, MergeStrategyManager, and batch paths
Add focused coverage for union-find clustering, property merge rules,
merge_duplicates, embedding similarity, and incremental detection (#866).
* test(deduplication): assert unrelated clusters without conditional skip
Make cluster-separation coverage fail closed by using mocked pairs and
unconditional assertions for distinct Apple vs Microsoft cluster IDs.
* test(deduplication): tighten update_clusters attachment assertions
Require the incremental path to place the new near-duplicate in the
same rebuilt cluster instead of accepting a vacuous cluster-count check.
* test(deduplication): strengthen incremental detect_duplicates wrapper checks
Assert real DuplicateCandidate matches, score threshold, and new×existing
routing instead of only checking that the wrapper returns a list.
* test(deduplication): verify metadata provenance behavior
Assert that preserve_provenance writes metadata.provenance fields and add a disabled-path test so regressions do not pass through merge_entities metadata alone.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@users.noreply.github.com>
The 1.0 / (1.0 + max(0.0, 1.0 - score)) normalization added in the last
commit clamped every raw score >= 1.0 to an identical 1.0, collapsing
result ranking for dot-product-metric indexes (unbounded), which cosine
(bounded to [-1, 1]) never exercised. Replaced with x/(1+|x|) rescaled
to (0, 1), which is strictly monotonic for any real score.
Also adds regression tests for scores >= 1 and a CHANGELOG entry.
Adds similarity_unavailable marker and warning logs to build_decision_context and explain_decision when a persistent backend (like FAISS) fails to reconstruct a vector. Updates docstrings to explicitly state this degraded-path behavior and guarantees schema stability. Adds regression tests to test vector retrieval failure behavior via caplog and context assertions.
- build_decision_context() and explain_decision(include_paths=True) both
accessed self.vectors directly, which is only initialized for the
inmemory backend, crashing with AttributeError on any persistent
backend (FAISS, Qdrant, Pinecone, etc.). Replaced with self.get_vector()
(#843's backend-agnostic accessor) + an is-not-None check — a verified
1:1 behavioral equivalent for the old 'decision_id in self.vectors'
guard on the inmemory path.
- Found a third, undocumented instance of the same bug during
verification: _filter_by_metadata() also accessed self.metadata/
self.vectors directly. Initial fix silently returned [] for persistent
backends, which was itself a new silent-failure bug (indistinguishable
from a genuine zero-match result). Reconciled to raise
NotImplementedError instead, matching the established precedent from
get_vector()/get_metadata() (#843) for 'backend exists but doesn't
support this operation' — confirmed via full grep of all 7 backend
wrapper classes that none currently implement filter_by_metadata,
so this path was previously dead-code-masked-as-working.
Tests: 14 new tests across two rounds — inmemory behavioral equivalence,
real (non-mocked) FAISS backend regression tests for all three methods,
and explicit coverage proving the NotImplementedError fires with a clear
message rather than the old silent-[] behavior. Full suite: 53 passed,
0 failed, 0 regressions across the 39 pre-existing tests.
self.indexer is only set for backend="inmemory", so save()/load() still
raised AttributeError for persistent backends (faiss, qdrant, etc.) even
after this PR's getattr() guards on self.vectors/self.metadata, since the
unguarded `self.indexer` access happened first. Guard it the same way and
delegate to the backend store's native save_index/load_index (currently
only FAISSStore implements these) so persistent-backend saves actually
persist instead of silently no-oping.
- Removed total=False from SearchResult TypedDict so all fields are strictly required
- Ensured distance: None is returned from backends that don't natively expose distance (Qdrant, Pinecone, SQLite, pgvector, in-memory)
- Standardized search result score to a consistent 0.0 - 1.0 similarity metric scale across all backend adapters
- Relaxed SearchResult id type to Union[str, int] to accommodate native integer IDs from Milvus and Qdrant without casting
- Updated schema verification tests
_get_candidate_embeddings()'s expand-and-retry loop widens the search
pool (up to limit*10) when post-filtering leaves too few candidates.
If the backend keeps returning a full page and filtered matches never
reach `limit`, the loop exited via the while condition instead of the
break branch, so the pre-loop empty embeddings/metadata/scores lists
were returned instead of the matches actually found in the final
iteration. This silently returned [] for filtered queries against
large persistent-backend stores even when matches existed - exactly
the scenario this PR adds support for.
Falls back to the last collected batch instead of discarding it.
Also documents this PR and #839 in the changelog.
Zohaib Hassnain