Merge branch 'main' into fix/mcp-server-version

This commit is contained in:
Sameer Kadam
2026-08-12 12:49:31 +05:30
committed by GitHub
57 changed files with 4842 additions and 251 deletions
+59
View File
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
## [0.6.5] - 2026-08-11
### Added ### Added
- **Embedded Oxigraph backend for `TripletStore`** (#838, closes #834) by @Linxiushen - **Embedded Oxigraph backend for `TripletStore`** (#838, closes #834) by @Linxiushen
@@ -69,6 +71,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed ### Fixed
- **`PipelineWithProvenance` raised `ModuleNotFoundError` on import and `AttributeError` on `.run()`** (#858, closes #858) by @Karunasagar12
- `from .pipeline import Pipeline` failed because `semantica/pipeline/pipeline.py` does not exist; corrected to `from .pipeline_builder import Pipeline`
- `.run()` called `self._pipeline.run()` on the `Pipeline` dataclass, which has no such method; replaced with `self._engine.execute_pipeline(self._pipeline, ...)` delegating to `ExecutionEngine`
- Constructor now accepts a built `Pipeline` instance (from `PipelineBuilder.build()`) instead of `**config`; the old `Pipeline(**config)` internal construction was invalid and never functional
- Replaced deprecated `datetime.utcnow()` with `datetime.now(timezone.utc)` in `run()`
- **`VectorStore.search_vectors()` returned inconsistent result shapes across backend implementations** (#853, closes #845) by @Sameer6305, reviewed by @KaifAhmad1 - **`VectorStore.search_vectors()` returned inconsistent result shapes across backend implementations** (#853, closes #845) by @Sameer6305, reviewed by @KaifAhmad1
- Every built-in backend (FAISS, Milvus, pgvector, Pinecone, Qdrant, SQLite-vec, Weaviate, in-memory) now returns the same canonical `SearchResult` shape (`id`, `score`, `metadata`, `vector`, `distance`), instead of some backends omitting `vector`/`metadata`/`distance` or, for Weaviate, returning a backend-specific `properties` key instead of `metadata` - Every built-in backend (FAISS, Milvus, pgvector, Pinecone, Qdrant, SQLite-vec, Weaviate, in-memory) now returns the same canonical `SearchResult` shape (`id`, `score`, `metadata`, `vector`, `distance`), instead of some backends omitting `vector`/`metadata`/`distance` or, for Weaviate, returning a backend-specific `properties` key instead of `metadata`
- Added a `SearchResult` `TypedDict` (`semantica/vector_store/vector_store.py`, exported from `semantica.vector_store`) documenting the contract; `metadata` now always defaults to `{}` rather than being absent, and `id` accepts `Union[str, int]` to accommodate Milvus/Qdrant's native integer IDs without casting - Added a `SearchResult` `TypedDict` (`semantica/vector_store/vector_store.py`, exported from `semantica.vector_store`) documenting the contract; `metadata` now always defaults to `{}` rather than being absent, and `id` accepts `Union[str, int]` to accommodate Milvus/Qdrant's native integer IDs without casting
@@ -244,6 +252,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Security ### Security
- **DNS check-then-use hardening for the ontology URL fetcher, and a remaining object-IRI validation gap** (#916, follow-up to GHSA-8c7v-62gr-hj6g and GHSA-8vgg-8mr4-r236) by @KaifAhmad1
- **DNS check-then-use (TOCTOU) window**: GHSA-8c7v-62gr-hj6g's own fix description flagged this as a secondary gap — `_validate_fetch_url()` resolved and validated a hostname once, but `_fetch_url_sync()` then let `requests` resolve the same hostname again independently at connect time. A low-TTL or rebinding DNS answer could differ between the two lookups, reopening the SSRF window the validation exists to close
- `_validate_fetch_url()` now returns the validated IP, and a new `_make_pinned_session()` builds a per-hop `requests.Session` whose connection pool is pinned directly to that IP — bypassing DNS resolution for the connection entirely — while explicitly restoring the real hostname as the outgoing HTTP `Host` header and, for HTTPS, the TLS SNI `server_hostname`/`assert_hostname`, so the connection reaches the validated IP but still presents (and is verified against) the real hostname's identity, keeping virtual hosting and certificate validation correct
- Caught during implementation: an earlier draft set urllib3's `_dns_host` post-construction, assuming (as in some urllib3 releases) that it was decoupled from `host`. In the version this project installs (2.7.0), `host` is a property that reads/writes `_dns_host` directly, so that approach would have silently changed the Host header too — caught by an end-to-end test against a real local server before landing, rather than shipping. Verified with real (non-mocked) local HTTP and HTTPS servers, the latter using a generated self-signed certificate to prove SNI/cert-hostname verification checks the real hostname rather than the pinned IP, plus a negative control confirming a hostname/cert mismatch is still correctly rejected, not silently bypassed
- **Object-IRI validation gap** (GHSA-8vgg-8mr4-r236 follow-up, distinct from the object-branch fix already shipped in #911): a triplet object already wrapped in `<...>` skipped `sparql_escaping.validate_uri()` in both `blazegraph_store.py` and `rdf4j_store.py`'s `_format_object_for_sparql`/`_format_object_for_ntriples`, only checking the inner content for a literal space or `>` — the pre-wrapped and unwrapped branches now validate identically
- **Fixed along the way** (caught in automated review across two follow-up rounds): `_validate_fetch_url()` originally pinned to only the first resolved IP, so a hostname with multiple A/AAAA records would fail outright if that specific address was unreachable — it now returns every validated IP and `_make_pinned_session()` falls back through all of them, verified by pinning to a genuinely unreachable address followed by a working one and confirming the fetch still succeeds; the test HTTPS server allowed TLSv1/TLSv1.1 by not setting a minimum version, now pinned to TLSv1.2; and when an HTTP(S) proxy applied, pinning was silently skipped in favor of the unpinned path — proxies are now disabled outright for this fetcher (`session.trust_env = False`, so `HTTP_PROXY`/`HTTPS_PROXY` env vars are never consulted) with a fail-closed 502 backstop if a proxy is ever forced onto the session some other way, verified by pointing `HTTP_PROXY` at an address that would fail if actually used and confirming the fetch still succeeds directly
- New `tests/explorer/test_ontology_dns_pinning.py` (12 tests: real local HTTP/HTTPS servers including 2 real-TLS checks, multi-IP fallback success/failure, and no-proxy-trust verification — gracefully skipped without the optional `cryptography` package where applicable); updated `tests/explorer/test_ontology_ssrf.py` for the new per-hop session construction; 4 new tests in `tests/triplet_store/test_sparql_injection.py` for the object-IRI fix. Full `explorer` + `triplet_store` suite: 572 passed
- **Missing Origin validation on the `/ws/graph-updates` WebSocket handshake** (#917, GHSA-4643-wpgq-w329) by @KaifAhmad1
- `CORSMiddleware` doesn't cover WebSocket handshakes at all (Starlette's CORS support only wraps HTTP), so under `SEMANTICA_ALLOW_ANONYMOUS=true` — the mode `docker-compose.dev.yml` ships — the anonymous-mode key bypass accepted a `/ws/graph-updates` connection from any origin. Loopback binding isn't a boundary against a browser: any page the operator has open can still reach `ws://localhost:8000/ws/graph-updates` directly, and `ConnectionManager.broadcast` sends every `graph_mutation` to every connected socket with no per-connection scoping. Combined with `/api/import` accepting `multipart/form-data` (a CORS-safelisted content type that skips preflight), a hostile page could write to the graph over REST and read the result back over the unauthenticated WebSocket
- Not affected: any deployment with `SEMANTICA_API_KEY` configured — the handshake already rejects without a valid key in that mode. This was an anonymous-mode-only, development-configuration exposure
- Fix: check the handshake's `Origin` header against `app.state.explorer_settings['allowed_origins']` — the same list `CORSMiddleware` already enforces for HTTP — before the key check. A missing `Origin` (native/CLI clients, which never set the header) is still allowed through, since the browser is the only threat this closes
- 4 new tests in `tests/explorer/test_explorer_auth.py`: hostile Origin rejected under anonymous mode; hostile Origin rejected even with a correct key (Origin is checked first, so a leaked key alone can't hijack the socket); an allowlisted Origin still connects; a missing Origin still connects. Full `explorer` suite: 226 passed
- **Polynomial-time ReDoS in the SPARQL route's `_PREFIX_DECL` regex** (#915, CodeQL `py/polynomial-redos`) by @Sameer6305
- The prior pattern's trailing `\s*` overlapped with the preceding `<[^>]*>` IRI-body match on inputs containing no closing `>` (e.g. `base<` followed by thousands of `!<` repetitions), forcing the regex engine to explore every possible split between the two quantifiers — O(n²) backtracking reachable from `req.query` via `_is_read_only_query()`
- Fixed by making the two quantifiers character-disjoint: horizontal whitespace only (`[ \t]`, never overlapping the IRI body) instead of `\s*`, and excluding CR/LF from the IRI body (`[^>\r\n]*`) so it can never span a line boundary. Independently verified: the exact pathological payload (`base<` + `!<` × 5,000/20,000) scales linearly (0.238ms → 0.841ms for 4x input, not the ~16x a surviving quadratic blowup would show)
- Added `_SPARQL_MAX_QUERY_LEN = 10_000` as defense-in-depth, checked in `execute_sparql()` before any regex work so a future pattern regression stays bounded regardless
- Two correctness regressions raised in review were checked and did not reproduce: comment-then-prefix stripping order means an inline comment after a `PREFIX` line (`PREFIX ex: <...> # comment`) is already gone by the time `_PREFIX_DECL` runs, verified directly against the pipeline; and the allowlist's `.sub()`-based cleaning only ever affects the yes/no decision, never the query actually sent to `graph.query()` — so even the narrow case of a multi-line string literal that happens to start a line with the literal text `PREFIX` or `BASE` can only cause a legitimate query to be wrongly rejected, never let something malicious through, since rdflib's parser still gates whatever actually executes
- 20 new/updated tests in `tests/explorer/test_sparql_route.py` and `tests/test_security_regression.py` (inline prologues, CRLF line endings, multi-line CRLF prefix chains, oversized-query rejection). 225 `explorer` + 82 SPARQL-specific tests passing
- **SPARQL injection via unvalidated triplet IRIs** (#911, GHSA-8vgg-8mr4-r236) by @KaifAhmad1
- `Triplet.subject`/`.predicate` (and, in some builders, `.object`) were interpolated directly into SPARQL update/query strings in the Blazegraph and RDF4J stores, and into a SELECT filter in the Jena store. A subject containing `>` closes the `<...>` IRI token early, so the rest of the value is parsed as more SPARQL. Entity names are document text in the normal ingest pipeline, so anyone whose content gets processed could append operations like `CLEAR ALL`, running with the application's store credentials
- Applied the existing `sparql_escaping.validate_uri` (already used by `anzo_store.py`, the one backend that was already hardened — this generalizes its approach rather than inventing a new one) at every subject/predicate/object interpolation site: `blazegraph_store.py`'s `_build_insert_data`, `_triplets_to_rdf`, `bulk_load`'s `graph` option, `get_triplets`'s filter, and `delete_triplet`; `rdf4j_store.py`'s `_triplets_to_ntriples`, `get_triplets`'s filter, and `delete_triplet`; `jena_store.py`'s `get_triplets`'s filter (the only vulnerable site there — `add_triplets`/`delete_triplet` already use rdflib's native `Graph.add`/`.remove` with `URIRef` rather than building query strings)
- **Fixed along the way** (caught in review, by @ZohaibHassan16): `_format_object_for_sparql`'s URI branch — used when a triplet's *object* is itself a URI rather than a literal — only checked for spaces and `>` inline instead of running the same `validate_uri` check applied to subject/predicate, leaving the object position as a narrower but real gap in both Blazegraph and RDF4J. Also fixed test flakiness in `RDF4JStore`'s test fixtures, which weren't mocking `_connect()` and so were making real network calls
- New `tests/triplet_store/test_sparql_injection.py` (12+ tests) reproducing the advisory's own injection payload (`http://example.com/a> ... ; CLEAR ALL ; INSERT DATA { ...`) against all three backends' write and read paths, asserting the malicious query is never built or sent. Full triplet_store suite: 330+ tests passing
- Side note, not part of this fix: found that `jena_store.py`'s `get_triplets()` builds syntactically invalid SPARQL for its WHERE-clause filters (missing a `FILTER()`/separator before the equality conditions) — a pre-existing correctness bug, unrelated to the injection fix, left alone here and worth a separate follow-up
- **Cypher injection via unvalidated node labels, relationship types, and property keys** (#910, GHSA-482h-hw99-h62p) by @KaifAhmad1
- Node labels and property keys passed to `create_node`/`create_relationship` were interpolated directly into Cypher strings in the Neptune, Neo4j, and FalkorDB graph stores. Property *values* are parameterized, but labels and keys can't be bound as query parameters, and nothing validated them — so a document-derived entity type or property name (the normal ingest path) could close the current Cypher token early and append arbitrary statements (e.g. `DETACH DELETE`), running with the application's database credentials
- New shared `semantica/graph_store/query_sanitize.py`: `sanitize_identifier()` generalizes `age_store.py`'s existing `_sanitize_label`/`_sanitize_rel_type` (the only backend that already validated this) into a helper the other backends import without an import cycle with `graph_store.py`/`methods.py`
- Applied at every label/relationship-type/property-key interpolation site in `amazon_neptune.py`, `neo4j_store.py`, `falkordb_store.py`, `graph_store.py` (`degree_centrality`'s own query builder), and `methods.py` (`update_relationship`'s own query builder) — covers `create_node`, `create_nodes`, `create_relationship`, `get_nodes`, `get_relationships`, `get_neighbors`, `shortest_path`, `update_node`, `create_index`, and all relationship-type filters across the three backends
- **Fixed along the way** (caught in review, by @Sameer6305): `depth`/`max_depth` path-length parameters are meant to be integers, but `Neo4jStore.get_neighbors()`/`shortest_path()` interpolated them into the Cypher variable-length-path syntax (`*1..{depth}`) without coercion — unlike the Neptune/FalkorDB equivalents, which already cast to `int()`. A string `depth` (e.g. `"1]->(x) DETACH DELETE x //"`) reached the query verbatim. Added the same `int()` coercion Neptune/FalkorDB already had, plus `GraphStore.get_neighbors()`'s `hops`/`depth` alias resolution
- New `tests/graph_store/test_cypher_injection.py` (unit tests on `sanitize_identifier` plus the labels/keys/rel-types injection payload run against Neptune/Neo4j/FalkorDB `create_node`/`create_relationship`, asserting the malicious query is never built or sent) and the depth-coercion regression above; plus additions to `tests/test_graph_store.py` (`degree_centrality`) and `tests/test_graph_store_methods.py` (`update_relationship`). Full graph_store suite: 224+ tests passing
- **4 critical/high vulnerabilities in the Explorer API and vector store: RCE, SSRF, XXE, and DoS, plus Cypher/SPARQL injection hardening found along the way** (#898) by @Sunil56224972
- **[CWE-502] Arbitrary code execution via `pickle.load()`**: `VectorStore.save()`/`load()` used `pickle` for the on-disk `store_data.pkl`; a crafted `.pkl` file placed in the store directory (file upload, shared filesystem, or supply-chain compromise) could execute arbitrary code on deserialization. Replaced with JSON — vectors and metadata are fully JSON-serializable, so nothing is lost — and `load()` now refuses any legacy `.pkl` file it finds with a migration error rather than deserializing it
- **[CWE-918] SSRF via redirect bypass in `ontology.py`'s URL fetcher**: `_validate_fetch_url()` correctly blocked private/loopback/reserved addresses on the caller-supplied URL, but `_fetch_url_sync()` fetched with `allow_redirects=True`, so a validated *public* first hop could 302 to `http://169.254.169.254/...` (cloud instance metadata) or an internal service, and `requests` followed it with no re-check. Redirects are now followed manually, capped at 5 hops, with `_validate_fetch_url()` re-run against every hop's target — including relative `Location` headers, resolved via `urljoin()` before validation — and every response (redirect or final) is explicitly closed to avoid leaking connections back to the pool
- **[CWE-611] XXE injection in the RDF/XML parser**: `_safe_parse_rdf()` depended on `defusedxml` for XXE protection, but `defusedxml` wasn't declared in `pyproject.toml`'s `explorer` extra, so it was silently absent in normal installs and the code fell back to a bare warning plus unsafe parsing — a crafted RDF/XML ontology with an external entity could read arbitrary server files. Added `defusedxml>=0.7.1` to the extra, and `_safe_parse_rdf()` now fails closed: it raises rather than parsing untrusted RDF/XML if `defusedxml` isn't importable, replacing an earlier regex-based DOCTYPE-stripping fallback that was reviewed and rejected as bypassable
- **[CWE-770] DoS via unbounded SPARQL graph materialization**: `_build_rdflib_graph()` loaded up to 999,999 nodes and 999,999 edges into memory per query, and with up to 4 concurrent SPARQL requests permitted, an attacker could exhaust server memory. Added a 50,000 node/edge cap (`_SPARQL_MAX_GRAPH_NODES`); oversized graphs now return a clean error instead of attempting materialization
- **Cypher injection via Apache AGE's `graph_name` and `$$`-delimiter breakout**: `graph_name` was interpolated unvalidated into `cypher('{graph_name}', $$ ... $$)`, and raw Cypher query text containing `$$` could close AGE's dollar-quoted string delimiter early and append arbitrary SQL. `graph_name` is now validated against the same identifier allowlist `age_store.py` already used for labels/relationship types, and any query containing `$$` is rejected outright
- **SPARQL Explorer route (`/api/sparql`) hardened against comment/PREFIX-hiding bypass**: `_is_read_only_query()` now strips comments and PREFIX/BASE declarations before checking the leading keyword, and additionally scans the full query body for SPARQL Update keywords (INSERT/DELETE/DROP/LOAD/CLEAR/CREATE/COPY/MOVE/ADD) — so `SELECT ... ; DROP ALL` is now rejected by the keyword scan itself rather than relying solely on rdflib's parser
- **Fixed along the way** (maintainer follow-up, addressing automated review findings and a regression introduced across several rounds of iteration on the original fix):
- `VectorStore.save()`'s numpy handling used `list(v)` for the JSON fallback path, which produces `numpy.float32` elements that `json.dump()` can't serialize — changed to `v.tolist()`
- the SPARQL graph-size `ValueError` was raised outside `execute_sparql()`'s exception handling and surfaced as an unhandled 500 instead of a clean API error — moved inside
- every streamed `requests` response in the ontology redirect loop, including the one actually read and returned, is now closed in a `finally` block — a connection-pool leak that a rework of the redirect logic had briefly reintroduced after an earlier fix
- a later commit meant to add opt-in API-key auth (`explorer/auth.py`, gated on `EXPLORER_API_KEY`) instead **replaced and silently disabled** the `Depends(require_auth)` enforcement already merged into `main` for GHSA-j4mq-hprp-987v (Critical — unauthenticated Explorer API), removed the `/ws/graph-updates` handshake check, and — unlike `require_auth` — failed *open* (allowed all requests) whenever its key was unset. Merging that version would have silently reverted an already-fixed Critical CVE the moment this branch landed. Removed `explorer/auth.py`; restored the per-router `Depends(require_auth)` wiring and the WebSocket auth check; kept the one genuine improvement in that commit (adding `X-API-Key` to the CORS `allow_headers` list) by folding it into the existing CORS config
- the new SPARQL keyword-scan's comment-stripping regex (`#[^\n]*`) also matched the `#` inside standard RDF namespace IRIs (e.g. `.../1999/02/22-rdf-syntax-ns#`), corrupting any query with a normal `rdf:`/`rdfs:`-style `PREFIX` declaration — caught because the hardening's own bundled tests failed against two of its own cases. Fixed by only treating `#` as a comment-start at line-start or after whitespace; the companion `PREFIX`/`BASE` regex was also fixed to accept bare `BASE <...>` declarations, which have no prefix-name token between the keyword and the IRI
- New/updated regression tests: `tests/explorer/test_ontology_ssrf.py` (redirect re-validation, relative-redirect resolution, response closing, redirect-cap enforcement), `tests/test_security_regression.py` (Cypher/SPARQL injection, XXE, numpy serialization, SSRF redirect handling), plus additions to `tests/explorer/test_sparql_route.py`, `tests/vector_store/test_vector_store.py`, and `tests/explorer/test_explorer_auth.py`
- Note: the Cypher-injection hardening here is scoped to `age_store.py`'s `graph_name`/`$$` breakout, found while reviewing this PR. The broader label/property-key/relationship-type injection across the Neptune, Neo4j, and FalkorDB backends (GHSA-482h-hw99-h62p, #910) and the triplet-store SPARQL injection across Blazegraph/RDF4J/Jena (GHSA-8vgg-8mr4-r236, #911) are covered by separate, still-open PRs, as is the unauthenticated-Explorer-API fix referenced above (GHSA-j4mq-hprp-987v, #909, already merged)
- **CI/CD supply-chain hardening against mutable-tag Action compromise (LiteLLM/Trivy-class attack)** (#824) by @KaifAhmad1 - **CI/CD supply-chain hardening against mutable-tag Action compromise (LiteLLM/Trivy-class attack)** (#824) by @KaifAhmad1
- Every third-party GitHub Action across all 8 workflows is now pinned to a full commit SHA instead of a mutable tag (`@v7``@3d3c42e... # v7`), closing the exact vector used against LiteLLM in March 2026 (a compromised Trivy Action tag stole a long-lived publishing token) - Every third-party GitHub Action across all 8 workflows is now pinned to a full commit SHA instead of a mutable tag (`@v7``@3d3c42e... # v7`), closing the exact vector used against LiteLLM in March 2026 (a compromised Trivy Action tag stole a long-lived publishing token)
- Added `verify-action-pins.yml` + `.github/scripts/verify-action-pins.sh`: a CI check that fails closed on any `uses:` reference that isn't a full SHA (catching a newly introduced mutable tag, not just auditing existing pins) and re-verifies every pin against the GitHub API on each workflow change, on push to `main`, and weekly; an unresolvable API lookup is treated as a failure rather than a silent skip - Added `verify-action-pins.yml` + `.github/scripts/verify-action-pins.sh`: a CI check that fails closed on any `uses:` reference that isn't a full SHA (catching a newly introduced mutable tag, not just auditing existing pins) and re-verifies every pin against the GitHub API on each workflow change, on push to `main`, and weekly; an unresolvable API lookup is treated as a failure rather than a silent skip
+12 -6
View File
@@ -132,7 +132,7 @@ compliant = graph.check_decision_rules({"category": "vendor_selection"}) # poli
```bash ```bash
semantica doctor semantica doctor
# Python 3.11.9 pass # Python 3.11.9 pass
# semantica 0.6.0 pass # semantica 0.6.5 pass
# faiss vector store pass # faiss vector store pass
# Config file pass ~/.semantica/config.yaml # Config file pass ~/.semantica/config.yaml
``` ```
@@ -1474,12 +1474,18 @@ For contributor / dev-server setup: **[explorer/README.md: Local Setup Guide](ex
--- ---
## What's New in v0.6.0 ## What's New in v0.6.5
- **Named-Graph Support for `JenaStore`:** Migrated onto `rdflib.Dataset(default_union=False)`, completing cross-backend named-graph parity across Blazegraph, RDF4J, and Jena; `add_triplets()` gains a `graph=` option **Security release — upgrading is strongly recommended.** Fixes for 5 externally-reported vulnerabilities in the Explorer API and graph/triplet store backends, plus a CodeQL-flagged ReDoS:
- **SPARQL CONSTRUCT Query Templates:** Parameterized, injection-safe `CONSTRUCT` templates extended from Blazegraph-only to RDF4J and Jena, plus pipeline integration via the `construct_template` step type
- **Databricks Connector:** `DatabricksIngestor` for Unity Catalog + Delta Lake ingestion, with PAT/OAuth M2M auth, table/query ingestion, and catalog/schema/table/lineage introspection. Install with `pip install "semantica[db-databricks]"` - **Missing authentication on all Explorer API routes** (GHSA-j4mq-hprp-987v, Critical): every route now requires `SEMANTICA_API_KEY`, fails closed (503) rather than open when unconfigured
- **SQLite Vector Store Backend:** `SQLiteVecStore`, a disk-backed local vector store on `sqlite-vec`'s `vec0` virtual tables, with Cosine/L2 metrics, metadata filtering, and WAL mode. Install with `pip install semantica[vectorstore-sqlite]` - **SSRF via redirect bypass in ontology URL fetching** (GHSA-8c7v-62gr-hj6g, High): redirect targets are now re-validated at every hop and the connection is pinned to the validated address, closing a DNS check-then-use race
- **Cypher injection via unvalidated node labels and property keys** (GHSA-482h-hw99-h62p, Critical): Neptune, Neo4j, and FalkorDB now sanitize every label/relationship-type/property-key interpolation site
- **SPARQL injection via unvalidated triplet IRIs** (GHSA-8vgg-8mr4-r236, Critical): Blazegraph, RDF4J, and Jena now validate subject/predicate/object IRIs before interpolation
- **Missing Origin validation on the WebSocket handshake** (GHSA-4643-wpgq-w329, Moderate, anonymous-mode only): `/ws/graph-updates` now checks `Origin` against the same allowlist `CORSMiddleware` enforces for HTTP
- **Polynomial ReDoS in SPARQL query validation** (CodeQL `py/polynomial-redos`): fixed a backtracking regex in the Explorer's SPARQL route
Also includes: embedded Oxigraph backend for `TripletStore`, PROV-O trust/spec completeness for `ProvenanceManager`, and the Altair Anzo triplet store backend.
→ [Full release notes](RELEASE_NOTES.md) · [Changelog](CHANGELOG.md) → [Full release notes](RELEASE_NOTES.md) · [Changelog](CHANGELOG.md)
+3
View File
@@ -9,7 +9,10 @@ flyctl launch --copy-config --config deploy/fly/fly.toml --no-deploy
# Fly.io private networking uses .internal hostnames — do not use localhost # Fly.io private networking uses .internal hostnames — do not use localhost
# unless FalkorDB is a co-located process inside the same Machine. # unless FalkorDB is a co-located process inside the same Machine.
flyctl secrets set FALKORDB_HOST=<falkordb-app-name>.internal FALKORDB_PORT=6379 flyctl secrets set FALKORDB_HOST=<falkordb-app-name>.internal FALKORDB_PORT=6379
flyctl secrets set SEMANTICA_API_KEY=$(openssl rand -hex 32)
flyctl deploy --config deploy/fly/fly.toml flyctl deploy --config deploy/fly/fly.toml
``` ```
Change `app` in `fly.toml` before launch if the default app name is already taken. Change `app` in `fly.toml` before launch if the default app name is already taken.
Fly apps get a public `*.fly.dev` URL by default, so `SEMANTICA_API_KEY` is required — without it the Explorer refuses every protected route (503) rather than serving anonymously. Pass the same value as the `X-API-Key` header from any client that talks to the deployed API.
+3
View File
@@ -9,7 +9,10 @@ railway add --database redis
railway variable --set "FALKORDB_HOST=${{Redis.REDISHOST}}" railway variable --set "FALKORDB_HOST=${{Redis.REDISHOST}}"
railway variable --set "FALKORDB_PORT=${{Redis.REDISPORT}}" railway variable --set "FALKORDB_PORT=${{Redis.REDISPORT}}"
railway variable --set "ALLOWED_ORIGINS=https://${{RAILWAY_PUBLIC_DOMAIN}}" railway variable --set "ALLOWED_ORIGINS=https://${{RAILWAY_PUBLIC_DOMAIN}}"
railway variable --set "SEMANTICA_API_KEY=$(openssl rand -hex 32)"
railway up railway up
``` ```
The Redis plugin variables are wired to the requested FalkorDB env names for deployment compatibility. The Explorer currently reads these settings but does not persist graph state to FalkorDB. The Redis plugin variables are wired to the requested FalkorDB env names for deployment compatibility. The Explorer currently reads these settings but does not persist graph state to FalkorDB.
Railway exposes this service on a public domain, so `SEMANTICA_API_KEY` is required — without it the Explorer refuses every protected route (503) rather than serving anonymously. Pass the same value as the `X-API-Key` header from any client that talks to the deployed API.
+2
View File
@@ -9,3 +9,5 @@ render blueprint apply deploy/render/render.yaml
``` ```
After creation, update `ALLOWED_ORIGINS` in the Render dashboard if you attach a custom domain. After creation, update `ALLOWED_ORIGINS` in the Render dashboard if you attach a custom domain.
`SEMANTICA_API_KEY` is auto-generated by the blueprint (`generateValue: true`) since this service gets a public `onrender.com` URL — without it the Explorer refuses every protected route (503) rather than serving anonymously. Find the generated value in the Render dashboard's environment tab and pass it as the `X-API-Key` header from any client that talks to the deployed API.
+2
View File
@@ -20,6 +20,8 @@ services:
type: keyvalue type: keyvalue
name: semantica-explorer-redis name: semantica-explorer-redis
property: port property: port
- key: SEMANTICA_API_KEY
generateValue: true
- type: keyvalue - type: keyvalue
name: semantica-explorer-redis name: semantica-explorer-redis
+2
View File
@@ -16,6 +16,8 @@ services:
ALLOWED_ORIGINS: http://localhost:5173,http://127.0.0.1:5173,http://localhost:8000,http://127.0.0.1:8000 ALLOWED_ORIGINS: http://localhost:5173,http://127.0.0.1:5173,http://localhost:8000,http://127.0.0.1:8000
FALKORDB_HOST: falkordb FALKORDB_HOST: falkordb
FALKORDB_PORT: "6379" FALKORDB_PORT: "6379"
# Local dev only: this compose file is not for public exposure.
SEMANTICA_ALLOW_ANONYMOUS: "true"
volumes: volumes:
- ./semantica:/app/semantica - ./semantica:/app/semantica
- ./pyproject.toml:/app/pyproject.toml:ro - ./pyproject.toml:/app/pyproject.toml:ro
+5
View File
@@ -8,6 +8,11 @@ services:
FALKORDB_HOST: falkordb FALKORDB_HOST: falkordb
FALKORDB_PORT: "6379" FALKORDB_PORT: "6379"
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-http://localhost:8000,http://127.0.0.1:8000} ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-http://localhost:8000,http://127.0.0.1:8000}
# Required for API access - the Explorer refuses all protected routes
# (503) until this is set. Generate one with `openssl rand -hex 32`.
SEMANTICA_API_KEY: ${SEMANTICA_API_KEY:-}
# Trusted local-only setups only: bypasses the API key entirely.
SEMANTICA_ALLOW_ANONYMOUS: ${SEMANTICA_ALLOW_ANONYMOUS:-false}
depends_on: depends_on:
falkordb: falkordb:
condition: service_started condition: service_started
+8 -8
View File
@@ -13,33 +13,33 @@ icon: "quote-left"
<Tab title="BibTeX"> <Tab title="BibTeX">
```bibtex ```bibtex
@software{semantica2026, @software{semantica2026,
title = {Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering}, title = {Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems},
author = {Hawksight AI}, author = {Semantica},
year = {2026}, year = {2026},
url = {https://github.com/semantica-agi/semantica}, url = {https://github.com/semantica-agi/semantica},
version = {0.6.0}, version = {0.6.5},
doi = {10.5281/zenodo.XXXXXXX} doi = {10.5281/zenodo.XXXXXXX}
} }
``` ```
</Tab> </Tab>
<Tab title="APA"> <Tab title="APA">
Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.6.0) \[Computer software\]. https://github.com/semantica-agi/semantica Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* (Version 0.6.5) \[Computer software\]. https://github.com/semantica-agi/semantica
</Tab> </Tab>
<Tab title="MLA"> <Tab title="MLA">
Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.6.0, GitHub, 2026, https://github.com/semantica-agi/semantica. Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.5, GitHub, 2026, https://github.com/semantica-agi/semantica.
</Tab> </Tab>
<Tab title="Chicago"> <Tab title="Chicago">
Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.6.0. GitHub, 2026. https://github.com/semantica-agi/semantica. Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.5. GitHub, 2026. https://github.com/semantica-agi/semantica.
</Tab> </Tab>
<Tab title="IEEE"> <Tab title="IEEE">
Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.6.0, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," Version 0.6.5, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
</Tab> </Tab>
</Tabs> </Tabs>
## Acknowledgment Text ## Acknowledgment Text
> "This work uses Semantica (Hawksight AI, 2026), an open-source framework for semantic layer construction and knowledge engineering." > "This work uses Semantica (2026), an open-source graph-native infrastructure framework for context and accountable AI systems, providing Context Graphs, knowledge graphs, and full decision provenance."
## Share Your Research ## Share Your Research
+1 -1
View File
@@ -17,7 +17,7 @@ icon: "circle-question"
| API key required? | Optional: pattern extraction works with no keys | | API key required? | Optional: pattern extraction works with no keys |
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement | | Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes | | Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
| Latest version? | **v0.6.0** (July 2026) | | Latest version? | **v0.6.5** (August 2026) |
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped | | Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
+1 -1
View File
@@ -42,7 +42,7 @@ icon: "rocket"
Verify installation: Verify installation:
```python ```python
import semantica import semantica
print(semantica.__version__) # 0.6.0 print(semantica.__version__) # 0.6.5
``` ```
</Check> </Check>
</Step> </Step>
+3 -2
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "semantica" name = "semantica"
version = "0.6.0" version = "0.6.5"
description = "Accountability and context layer for AI agents. Context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable." description = "Accountability and context layer for AI agents. Context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
readme = "README.md" readme = "README.md"
license = { text = "MIT" } license = { text = "MIT" }
@@ -233,7 +233,8 @@ explorer = [
"fastapi>=0.100.0", "fastapi>=0.100.0",
"uvicorn[standard]>=0.22.0", "uvicorn[standard]>=0.22.0",
"websockets>=15.0.1", "websockets>=15.0.1",
"python-multipart>=0.0.6" "python-multipart>=0.0.6",
"defusedxml>=0.7.1"
] ]
explorer-lite = [ explorer-lite = [
"streamlit>=1.25.0", "streamlit>=1.25.0",
+1 -1
View File
@@ -10,7 +10,7 @@ Main exports:
- Config: Configuration management - Config: Configuration management
""" """
__version__ = "0.6.0" __version__ = "0.6.5"
__author__ = "Semantica Contributors" __author__ = "Semantica Contributors"
__license__ = "MIT" __license__ = "MIT"
+11 -1
View File
@@ -4065,7 +4065,7 @@ def server(ctx: click.Context) -> None:
@click.option("--port", default=8000, type=int, show_default=True) @click.option("--port", default=8000, type=int, show_default=True)
@click.option("--workers", default=1, type=int, show_default=True) @click.option("--workers", default=1, type=int, show_default=True)
@click.option("--reload", is_flag=True, default=False, help="Enable hot reload.") @click.option("--reload", is_flag=True, default=False, help="Enable hot reload.")
@click.option("--host", default="0.0.0.0", show_default=True) @click.option("--host", default="127.0.0.1", show_default=True)
@click.pass_obj @click.pass_obj
def server_start(cli_ctx: CLIContext, port: int, workers: int, reload: bool, host: str) -> None: def server_start(cli_ctx: CLIContext, port: int, workers: int, reload: bool, host: str) -> None:
"""Start the REST API server. """Start the REST API server.
@@ -4076,6 +4076,16 @@ def server_start(cli_ctx: CLIContext, port: int, workers: int, reload: bool, hos
""" """
cli_ctx = _require_ctx(cli_ctx) cli_ctx = _require_ctx(cli_ctx)
_LOOPBACK_HOSTS = {"127.0.0.1", "::1", "localhost"}
if host not in _LOOPBACK_HOSTS:
console.print(
f"[{_WARN_STY}] ⚠[/{_WARN_STY}] Binding to [cyan]{host}[/cyan] exposes "
"the server to the network. Set SEMANTICA_API_KEY before doing this "
"in any reachable environment — without it, protected routes refuse "
"all requests (503), and with SEMANTICA_ALLOW_ANONYMOUS=true they are "
"wide open."
)
def _action() -> None: def _action() -> None:
import subprocess as sp import subprocess as sp
cmd = [ cmd = [
+16 -6
View File
@@ -83,12 +83,22 @@ def main(argv=None):
_LOOPBACK_HOSTS = {"127.0.0.1", "::1", "localhost"} _LOOPBACK_HOSTS = {"127.0.0.1", "::1", "localhost"}
if args.host not in _LOOPBACK_HOSTS: if args.host not in _LOOPBACK_HOSTS:
_err.print( import os as _os
f"[bold yellow]Warning:[/bold yellow] Binding to " if _os.environ.get("SEMANTICA_ALLOW_ANONYMOUS", "").strip().lower() == "true":
f"[cyan]{args.host}[/cyan] exposes the Explorer to the network. " _err.print(
"The API has no authentication — all graph data will be readable " f"[bold yellow]Warning:[/bold yellow] Binding to "
"and writable by any host that can reach this port." f"[cyan]{args.host}[/cyan] with SEMANTICA_ALLOW_ANONYMOUS=true "
) "exposes the Explorer to the network with no authentication — "
"all graph data will be readable and writable by any host that "
"can reach this port."
)
elif not _os.environ.get("SEMANTICA_API_KEY"):
_err.print(
f"[bold yellow]Warning:[/bold yellow] Binding to "
f"[cyan]{args.host}[/cyan] but SEMANTICA_API_KEY is not set — "
"protected routes will refuse all requests (503) until it is "
"configured."
)
if not args.no_browser: if not args.no_browser:
import threading import threading
+60 -17
View File
@@ -8,13 +8,14 @@ from contextlib import asynccontextmanager
from pathlib import Path from pathlib import Path
from typing import Optional from typing import Optional
from fastapi import FastAPI, HTTPException, Request, WebSocket, WebSocketDisconnect from fastapi import Depends, FastAPI, HTTPException, Request, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, HTMLResponse, JSONResponse from fastapi.responses import FileResponse, HTMLResponse, JSONResponse
from fastapi.staticfiles import StaticFiles from fastapi.staticfiles import StaticFiles
from .. import __version__ from .. import __version__
from ..context.context_graph import ContextGraph from ..context.context_graph import ContextGraph
from .dependencies import anonymous_access_allowed, get_expected_api_key, is_valid_api_key, require_auth
from .session import GraphSession from .session import GraphSession
from .ws import ConnectionManager from .ws import ConnectionManager
@@ -97,6 +98,22 @@ def create_app(
@asynccontextmanager @asynccontextmanager
async def lifespan(app: FastAPI): async def lifespan(app: FastAPI):
import logging as _lifespan_logging
_lifespan_logger = _lifespan_logging.getLogger(__name__)
if anonymous_access_allowed():
_lifespan_logger.warning(
"Explorer is running with SEMANTICA_ALLOW_ANONYMOUS=true — "
"all API routes are unauthenticated. Do not expose this "
"process beyond localhost."
)
elif get_expected_api_key():
_lifespan_logger.info("Explorer API authentication: enabled (SEMANTICA_API_KEY set).")
else:
_lifespan_logger.warning(
"Explorer API authentication: NOT CONFIGURED. All protected "
"routes will return 503 until SEMANTICA_API_KEY is set."
)
app.state.event_loop = asyncio.get_running_loop() app.state.event_loop = asyncio.get_running_loop()
app.state.ws_manager = ConnectionManager() app.state.ws_manager = ConnectionManager()
app.state.session = active_session app.state.session = active_session
@@ -113,17 +130,17 @@ def create_app(
app.state.explorer_settings = settings app.state.explorer_settings = settings
# allow_credentials lets browsers send cookies/auth headers cross-origin. # allow_credentials lets browsers send cookies/auth headers cross-origin.
# The Explorer has no authentication, so credentials serve no purpose and # Credentials aren't needed for the X-API-Key auth scheme below, and
# enabling them when origins are broadened creates cross-site request risk. # enabling them when origins are broadened creates cross-site request
# Set EXPLORER_CORS_CREDENTIALS=true explicitly to opt in (e.g. for a # risk. Set EXPLORER_CORS_CREDENTIALS=true explicitly to opt in (e.g.
# reverse-proxy setup that injects its own auth layer). # for a reverse-proxy setup that injects its own cookie-based auth).
_allow_credentials = os.environ.get("EXPLORER_CORS_CREDENTIALS", "false").lower() == "true" _allow_credentials = os.environ.get("EXPLORER_CORS_CREDENTIALS", "false").lower() == "true"
app.add_middleware( app.add_middleware(
CORSMiddleware, CORSMiddleware,
allow_origins=settings["allowed_origins"], allow_origins=settings["allowed_origins"],
allow_credentials=_allow_credentials, allow_credentials=_allow_credentials,
allow_methods=["GET", "POST", "DELETE", "OPTIONS"], allow_methods=["GET", "POST", "DELETE", "OPTIONS"],
allow_headers=["Content-Type", "Authorization"], allow_headers=["Content-Type", "Authorization", "X-API-Key"],
max_age=600, max_age=600,
) )
@@ -159,22 +176,48 @@ def create_app(
from .routes.temporal import router as temporal_router from .routes.temporal import router as temporal_router
from .routes.vocabulary import router as vocabulary_router from .routes.vocabulary import router as vocabulary_router
app.include_router(graph_router) _auth = [Depends(require_auth)]
app.include_router(analytics_router) app.include_router(graph_router, dependencies=_auth)
app.include_router(decisions_router) app.include_router(analytics_router, dependencies=_auth)
app.include_router(temporal_router) app.include_router(decisions_router, dependencies=_auth)
app.include_router(enrich_router) app.include_router(temporal_router, dependencies=_auth)
app.include_router(export_import_router) app.include_router(enrich_router, dependencies=_auth)
app.include_router(annotations_router) app.include_router(export_import_router, dependencies=_auth)
app.include_router(sparql_router) app.include_router(annotations_router, dependencies=_auth)
app.include_router(provenance_router) app.include_router(sparql_router, dependencies=_auth)
app.include_router(vocabulary_router) app.include_router(provenance_router, dependencies=_auth)
app.include_router(ontology_router) app.include_router(vocabulary_router, dependencies=_auth)
app.include_router(ontology_router, dependencies=_auth)
_WS_MAX_MESSAGE_BYTES = 64 * 1024 # 64 KB — control messages only _WS_MAX_MESSAGE_BYTES = 64 * 1024 # 64 KB — control messages only
@app.websocket("/ws/graph-updates") @app.websocket("/ws/graph-updates")
async def websocket_endpoint(websocket: WebSocket): async def websocket_endpoint(websocket: WebSocket):
# CORSMiddleware doesn't cover WebSocket handshakes (Starlette's
# CORS support only wraps HTTP), so under SEMANTICA_ALLOW_ANONYMOUS
# the key check below accepts any origin — loopback binding isn't a
# boundary against a browser, since any page the operator has open
# can still reach ws://localhost:.../ws/graph-updates directly.
# Reject a foreign Origin explicitly here, against the same
# allowlist CORSMiddleware already enforces for HTTP
# (GHSA-4643-wpgq-w329). Browsers always send Origin on a
# cross-origin WebSocket handshake; native/CLI clients omit it
# entirely, so a missing Origin is allowed through — the browser is
# the only threat this check is closing.
origin = websocket.headers.get("origin")
allowed_origins = app.state.explorer_settings["allowed_origins"]
if origin is not None and origin not in allowed_origins:
await websocket.close(code=4403) # forbidden
return
# Browsers can't set custom headers on a WebSocket handshake, so
# accept the key via header (non-browser clients) or query param
# (browser clients), same SEMANTICA_API_KEY the REST routes check.
candidate = websocket.headers.get("x-api-key") or websocket.query_params.get("api_key")
if not is_valid_api_key(candidate):
await websocket.close(code=4401) # unauthorized
return
manager: ConnectionManager = app.state.ws_manager manager: ConnectionManager = app.state.ws_manager
await manager.connect(websocket) await manager.connect(websocket)
await manager.send_personal(websocket, "connection_ack", {"connected": True}) await manager.send_personal(websocket, "connection_ack", {"connected": True})
+63 -2
View File
@@ -2,14 +2,75 @@
Semantica Explorer : FastAPI Dependencies Semantica Explorer : FastAPI Dependencies
Provides ``Depends()``-compatible callables for injecting the Provides ``Depends()``-compatible callables for injecting the
current ``GraphSession`` and ``ConnectionManager`` into route handlers. current ``GraphSession`` and ``ConnectionManager`` into route handlers,
and for enforcing API-key authentication on protected routes.
""" """
from fastapi import Request, HTTPException, status import hmac
import os
from typing import Optional
from fastapi import Request, HTTPException, Security, status
from fastapi.security.api_key import APIKeyHeader
from .session import GraphSession from .session import GraphSession
from .ws import ConnectionManager from .ws import ConnectionManager
_api_key_header = APIKeyHeader(name="X-API-Key", auto_error=False)
def get_expected_api_key() -> Optional[str]:
"""Read the configured API key from the environment on every call.
Read fresh (not cached) so tests and ops tooling can rotate the key
without restarting the process.
"""
return os.environ.get("SEMANTICA_API_KEY") or None
def anonymous_access_allowed() -> bool:
return os.environ.get("SEMANTICA_ALLOW_ANONYMOUS", "").strip().lower() == "true"
def is_valid_api_key(candidate: Optional[str]) -> bool:
"""Return True if *candidate* matches the configured key, or if the
server has explicitly opted into anonymous access."""
if anonymous_access_allowed():
return True
expected = get_expected_api_key()
if not expected:
return False
return bool(candidate) and hmac.compare_digest(candidate, expected)
def require_auth(api_key: Optional[str] = Security(_api_key_header)) -> None:
"""Dependency enforcing the ``X-API-Key`` header on protected routes.
Every Explorer/API router (except health/info/static assets) should be
mounted with ``dependencies=[Depends(require_auth)]``. If
SEMANTICA_API_KEY is unset, requests are refused with 503 rather than
silently served unauthenticated SEMANTICA_ALLOW_ANONYMOUS=true opts
into that explicitly for local development.
"""
if anonymous_access_allowed():
return
expected = get_expected_api_key()
if not expected:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail=(
"Server is not configured for authentication. Set the "
"SEMANTICA_API_KEY environment variable, or explicitly opt "
"into unauthenticated access (development only) with "
"SEMANTICA_ALLOW_ANONYMOUS=true."
),
)
if not api_key or not hmac.compare_digest(api_key, expected):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or missing API key. Send it as the X-API-Key header.",
)
def get_session(request: Request) -> GraphSession: def get_session(request: Request) -> GraphSession:
"""Retrieve the GraphSession stored on ``app.state``.""" """Retrieve the GraphSession stored on ``app.state``."""
+165 -21
View File
@@ -11,7 +11,7 @@ import uuid
from datetime import datetime, UTC from datetime import datetime, UTC
from difflib import SequenceMatcher from difflib import SequenceMatcher
from typing import Any, Dict, List, Optional, Tuple from typing import Any, Dict, List, Optional, Tuple
from urllib.parse import urlparse from urllib.parse import urljoin, urlparse
from typing_extensions import Literal from typing_extensions import Literal
from fastapi import APIRouter, Depends, HTTPException, Query, Request from fastapi import APIRouter, Depends, HTTPException, Query, Request
@@ -978,8 +978,18 @@ def _normalize_format(fmt: Optional[str]) -> str:
return _FORMAT_ALIASES.get(lower, lower) return _FORMAT_ALIASES.get(lower, lower)
def _validate_fetch_url(url: str) -> None: def _validate_fetch_url(url: str) -> List[str]:
"""Reject non-HTTP(S) schemes and private/loopback/link-local targets.""" """Reject non-HTTP(S) schemes and private/loopback/link-local targets.
Returns every resolved, validated IP address (deduplicated, in
resolution order) so the caller can pin the actual connection to them
(see _make_pinned_session) with fallback across all of them not just
the first since a hostname can have multiple A/AAAA records and the
first one isn't guaranteed reachable. Resolving the hostname again at
connect time would open a DNS check-then-use window (a low-TTL or
rebinding DNS answer could differ between this check and the client's
own lookup), which is what pinning to these specific addresses avoids.
"""
parsed = urlparse(url) parsed = urlparse(url)
if parsed.scheme not in ("http", "https"): if parsed.scheme not in ("http", "https"):
raise HTTPException(status_code=422, detail="Only http and https URLs are allowed.") raise HTTPException(status_code=422, detail="Only http and https URLs are allowed.")
@@ -990,6 +1000,7 @@ def _validate_fetch_url(url: str) -> None:
addrinfos = socket.getaddrinfo(hostname, None) addrinfos = socket.getaddrinfo(hostname, None)
except socket.gaierror as exc: except socket.gaierror as exc:
raise HTTPException(status_code=422, detail=f"Cannot resolve hostname '{hostname}': {exc}") from exc raise HTTPException(status_code=422, detail=f"Cannot resolve hostname '{hostname}': {exc}") from exc
validated_ips: List[str] = []
for _family, _type, _proto, _canonname, sockaddr in addrinfos: for _family, _type, _proto, _canonname, sockaddr in addrinfos:
try: try:
ip = ipaddress.ip_address(sockaddr[0]) ip = ipaddress.ip_address(sockaddr[0])
@@ -1000,28 +1011,161 @@ def _validate_fetch_url(url: str) -> None:
status_code=422, status_code=422,
detail="Fetching from private, loopback, or reserved network addresses is not allowed.", detail="Fetching from private, loopback, or reserved network addresses is not allowed.",
) )
if sockaddr[0] not in validated_ips:
validated_ips.append(sockaddr[0])
if not validated_ips:
raise HTTPException(status_code=422, detail=f"Cannot resolve hostname '{hostname}' to a usable address.")
return validated_ips
def _make_pinned_session(pinned_ips: List[str], url: str):
"""Build a requests.Session whose connection is pinned to pinned_ips
(tried in order, falling back on connection failure), regardless of
what url's hostname resolves to at connect time.
_validate_fetch_url() resolves and validates the hostname once; letting
the HTTP client resolve it again independently at connect time reopens
the exact gap that validation exists to close a low-TTL or rebinding
DNS answer can differ between the two lookups. This pins the pool's
connect target to the already-validated addresses directly (bypassing
DNS resolution for the connection entirely), while keeping the original
hostname as the outgoing HTTP Host header and, for HTTPS, the TLS SNI
server_hostname / assert_hostname otherwise the connection would
reach the right IP but present the wrong identity, breaking name-based
virtual hosting and (for HTTPS) certificate hostname verification.
Falls back across every validated address (not just the first) so a
hostname with multiple A/AAAA records doesn't fail outright just
because the first-returned address happens to be unreachable.
Note: urllib3's Connection.host is a property that reads/writes the
same underlying value as `_dns_host` in this version it is NOT the
separate "presented identity" field it is in some older releases, so
overriding just `_dns_host` post-construction (as an earlier version of
this fix did) actually changes the Host header too. Pinning the pool's
`host` directly and restoring the real hostname via an explicit Host
header (+ SNI params for HTTPS) is the correct mechanism here.
"""
import requests as _req
import urllib3.util.connection as _u3_connection
from urllib3.exceptions import NewConnectionError
parsed = urlparse(url)
hostname = parsed.hostname
port = parsed.port
default_port = 443 if parsed.scheme == "https" else 80
host_header = hostname if port in (None, default_port) else f"{hostname}:{port}"
class _MultiIPConnectionMixin:
"""Overrides _new_conn to fall back across every pinned IP in
order, instead of urllib3's default single-host connect."""
def _new_conn(self):
last_exc: Optional[BaseException] = None
for ip in pinned_ips:
try:
return _u3_connection.create_connection(
(ip, self.port),
self.timeout,
source_address=self.source_address,
socket_options=self.socket_options,
)
except OSError as exc:
last_exc = exc
continue
raise NewConnectionError(
self, f"Failed to establish a connection to any of {pinned_ips}: {last_exc}"
)
class _PinnedIPHTTPAdapter(_req.adapters.HTTPAdapter):
def get_connection_with_tls_context(self, request, verify, proxies=None, cert=None):
# A proxy would perform its own DNS resolution of the target
# host on this process's behalf — a resolution outside this
# process's visibility or control, so there is no client-side
# pin that closes that race. Proxies are disabled outright for
# this SSRF-sensitive fetcher (session.trust_env=False below),
# so this should be unreachable via environment proxies; fail
# closed rather than silently skip pinning if a proxy is
# somehow still configured (e.g. passed explicitly in the
# future). _validate_fetch_url's destination classification is
# a separate, always-enforced check — this only guards the
# secondary DNS-pinning hardening.
if _req.utils.select_proxy(request.url, proxies):
raise HTTPException(
status_code=502,
detail="Proxied requests are not supported for ontology URL fetching.",
)
host_params, pool_kwargs = self.build_connection_pool_key_attributes(request, verify, cert)
if host_params.get("scheme") == "https":
pool_kwargs.setdefault("assert_hostname", hostname)
pool_kwargs.setdefault("server_hostname", hostname)
host_params["host"] = pinned_ips[0]
pool = self.poolmanager.connection_from_host(**host_params, pool_kwargs=pool_kwargs)
base_connection_cls = pool.ConnectionCls
if not issubclass(base_connection_cls, _MultiIPConnectionMixin):
pool.ConnectionCls = type(
"_PinnedConnection", (_MultiIPConnectionMixin, base_connection_cls), {}
)
return pool
session = _req.Session()
# Never honor HTTP_PROXY/HTTPS_PROXY/NO_PROXY env vars for this
# SSRF-sensitive fetcher: a configured proxy would perform its own DNS
# resolution of the target host outside this process's control,
# silently reopening the DNS check-then-use race pinning exists to
# close. See _PinnedIPHTTPAdapter.get_connection_with_tls_context for
# the fail-closed backstop if a proxy is somehow still configured.
session.trust_env = False
session.headers["Host"] = host_header
adapter = _PinnedIPHTTPAdapter()
session.mount("http://", adapter)
session.mount("https://", adapter)
return session
def _fetch_url_sync(url: str) -> bytes: def _fetch_url_sync(url: str) -> bytes:
_validate_fetch_url(url) pinned_ips = _validate_fetch_url(url)
import requests as _req _MAX_REDIRECTS = 5
current_url = url
try: try:
resp = _req.get( for _ in range(_MAX_REDIRECTS + 1):
url, session = _make_pinned_session(pinned_ips, current_url)
headers={"Accept": "text/turtle, application/rdf+xml, application/ld+json, */*;q=0.1"}, try:
timeout=30, resp = session.get(
stream=True, current_url,
allow_redirects=True, headers={"Accept": "text/turtle, application/rdf+xml, application/ld+json, */*;q=0.1"},
) timeout=30,
resp.raise_for_status() stream=True,
chunks: List[bytes] = [] allow_redirects=False, # SECURITY: follow redirects manually
total = 0 )
for chunk in resp.iter_content(65536): if resp.is_redirect or resp.is_permanent_redirect:
total += len(chunk) redirect_url = resp.headers.get("Location")
if total > _MAX_FETCH_BYTES: resp.close() # Release the streamed connection before following the redirect
raise HTTPException(status_code=413, detail="Remote resource exceeds 20 MB limit.") if not redirect_url:
chunks.append(chunk) raise HTTPException(status_code=502, detail="Redirect without Location header.")
return b"".join(chunks) # Resolve relative redirects (e.g. /ontology.ttl) against the current URL
redirect_url = urljoin(current_url, redirect_url)
# Re-validate the redirect target to prevent SSRF via
# open-redirect to internal/cloud-metadata endpoints, and
# get fresh pins for the new host.
pinned_ips = _validate_fetch_url(redirect_url)
current_url = redirect_url
continue
try:
resp.raise_for_status()
chunks: List[bytes] = []
total = 0
for chunk in resp.iter_content(65536):
total += len(chunk)
if total > _MAX_FETCH_BYTES:
raise HTTPException(status_code=413, detail="Remote resource exceeds 20 MB limit.")
chunks.append(chunk)
return b"".join(chunks)
finally:
resp.close() # Release the streamed connection once fully read (or on error)
finally:
session.close()
raise HTTPException(status_code=502, detail=f"Too many redirects (max {_MAX_REDIRECTS}).")
except HTTPException: except HTTPException:
raise raise
except Exception as exc: except Exception as exc:
+129 -11
View File
@@ -3,11 +3,16 @@ SPARQL routes backed by an in-memory rdflib projection of the current graph.
Security contract Security contract
----------------- -----------------
* Only SELECT, ASK, CONSTRUCT, and DESCRIBE are accepted (allowlist enforced * Only SELECT, ASK, CONSTRUCT, and DESCRIBE are accepted, and the query
before graph construction so rejected queries never touch the session). body is scanned for SPARQL Update keywords (INSERT/DELETE/DROP/LOAD/
* Multi-statement injections that start with an allowed keyword (e.g. CLEAR/CREATE/COPY/MOVE/ADD) after stripping comments and PREFIX/BASE
``SELECT ... ; DROP ALL``) pass the prefix check and reach rdflib, which declarations both enforced before graph construction, so rejected
rejects non-SELECT/ASK/CONSTRUCT/DESCRIBE update syntax in the parser. queries never touch the session. A multi-statement injection appended
after an allowed keyword (e.g. ``SELECT ... ; DROP ALL``) is caught by
the keyword scan itself, not left to rdflib's parser.
* rdflib's parser remains a second line of defense for malformed multi-
statement syntax that doesn't contain any forbidden keyword (e.g.
``SELECT ... ; ASK ...``), which SPARQL 1.1 Query doesn't permit.
* The in-memory rdflib graph is a read-only projection the live * The in-memory rdflib graph is a read-only projection the live
``GraphSession`` is never mutated by this route. ``GraphSession`` is never mutated by this route.
""" """
@@ -26,14 +31,76 @@ from ..session import GraphSession
router = APIRouter(prefix="/api/sparql", tags=["Power User Tools"]) router = APIRouter(prefix="/api/sparql", tags=["Power User Tools"])
_ALLOWED_QUERY_TYPES = re.compile( _ALLOWED_QUERY_TYPES = re.compile(
r"^\s*(SELECT|ASK|CONSTRUCT|DESCRIBE)\b", r"^(SELECT|ASK|CONSTRUCT|DESCRIBE)\b",
re.IGNORECASE, re.IGNORECASE,
) )
# SPARQL Update keywords that must never appear in read-only queries.
# These are checked AFTER comment/prefix stripping to prevent bypass via
# comments like: # INSERT DATA { ... }\nSELECT ...
_FORBIDDEN_KEYWORDS = re.compile(
r"\b(INSERT|DELETE|DROP|LOAD|CLEAR|CREATE|COPY|MOVE|ADD)\b",
re.IGNORECASE,
)
# Matches SPARQL single-line comments (# ...) and PREFIX/BASE declarations.
# The comment regex only treats '#' as a comment-starter at line-start or
# after whitespace — not mid-token — since RDF namespace IRIs commonly
# contain a literal '#' (e.g. ".../1999/02/22-rdf-syntax-ns#"), and a naive
# `#[^\n]*` would truncate every such PREFIX declaration's IRI, corrupting
# the query. BASE declarations have no prefix name between the keyword and
# the IRI (`BASE <...>`, vs. `PREFIX ex: <...>`), so the prefix-name token
# is optional.
#
# ReDoS fix (CodeQL py/polynomial-redos, issue #1897):
#
# The original pattern `<[^>]*>\s*` was vulnerable because `\s*` (which
# matches newlines) could overlap with `[^>]*` on inputs that contain no
# closing `>` (e.g. `base<!!<!<...`), forcing the engine to explore every
# possible split between the two quantifiers — O(n²) backtracking.
#
# The fix uses `<[^>\r\n]*>` for the IRI body: excluding CR and LF from
# the character class means the IRI match can never span a line boundary,
# and the disjoint trailing `[ \t]*` (horizontal whitespace only) has zero
# character-class overlap with `[^>\r\n]*`, so the engine has exactly one
# way to match. No end-of-line anchor is needed or used, which correctly
# handles both inline prologues (`PREFIX ex: <...> SELECT ...` on one line)
# and CRLF line endings (`\r\n`) without any special casing.
_COMMENT_LINE = re.compile(r"(?:^|(?<=\s))#[^\n]*", re.MULTILINE)
_PREFIX_DECL = re.compile(
r"^[ \t]*(?:PREFIX[ \t]+\S+|BASE)[ \t]*<[^>\r\n]*>[ \t]*",
re.IGNORECASE | re.MULTILINE,
)
def _is_read_only_query(query: str) -> bool: def _is_read_only_query(query: str) -> bool:
"""Return True only for SELECT / ASK / CONSTRUCT / DESCRIBE queries.""" """Return True only for genuine read-only SPARQL queries.
return bool(_ALLOWED_QUERY_TYPES.match(query))
Strips comments, PREFIX/BASE declarations, and leading whitespace before
checking the first keyword. Also rejects queries containing SPARQL Update
keywords anywhere in the body, preventing injection via embedded strings
or multi-statement tricks.
Note: callers are responsible for enforcing any input-length limit *before*
calling this function so that an oversized-query rejection can be surfaced
as a distinct, actionable error rather than the generic read-only message.
"""
# 1. Remove single-line comments that could hide the real query type
cleaned = _COMMENT_LINE.sub("", query)
# 2. Remove PREFIX/BASE declarations
cleaned = _PREFIX_DECL.sub("", cleaned)
# 3. Strip remaining whitespace
cleaned = cleaned.strip()
# 4. Check that the first keyword is a read-only query type
if not _ALLOWED_QUERY_TYPES.match(cleaned):
return False
# 5. Block any forbidden (mutating) keywords anywhere in the query
if _FORBIDDEN_KEYWORDS.search(cleaned):
return False
return True
class SparqlRequest(BaseModel): class SparqlRequest(BaseModel):
@@ -59,8 +126,27 @@ def _build_rdflib_graph(session: GraphSession) -> rdflib.Graph:
graph.bind("ent", NS) graph.bind("ent", NS)
graph.bind("prop", PROP) graph.bind("prop", PROP)
nodes, _ = session.get_nodes(skip=0, limit=999_999) # SECURITY: Cap the number of entities materialized into memory to
edges, _ = session.get_edges(skip=0, limit=999_999) # prevent denial-of-service via memory exhaustion. Without this guard
# an attacker can send concurrent SPARQL queries that each load ~1M
# nodes/edges into rdflib Graph objects, consuming gigabytes of RAM.
nodes, total_nodes = session.get_nodes(skip=0, limit=_SPARQL_MAX_GRAPH_NODES + 1)
if len(nodes) > _SPARQL_MAX_GRAPH_NODES:
raise ValueError(
f"Graph has more than {_SPARQL_MAX_GRAPH_NODES:,} nodes. "
f"SPARQL queries are limited to graphs with at most "
f"{_SPARQL_MAX_GRAPH_NODES:,} nodes to prevent excessive "
f"memory usage. Use the REST API for large graph operations."
)
edges, _ = session.get_edges(skip=0, limit=_SPARQL_MAX_GRAPH_NODES + 1)
if len(edges) > _SPARQL_MAX_GRAPH_NODES:
raise ValueError(
f"Graph has more than {_SPARQL_MAX_GRAPH_NODES:,} edges. "
f"SPARQL queries are limited to graphs with at most "
f"{_SPARQL_MAX_GRAPH_NODES:,} edges to prevent excessive "
f"memory usage. Use the REST API for large graph operations."
)
for node in nodes: for node in nodes:
subject = NS[str(node.get("id", ""))] subject = NS[str(node.get("id", ""))]
@@ -91,6 +177,12 @@ def _build_rdflib_graph(session: GraphSession) -> rdflib.Graph:
_SPARQL_MAX_ROWS = 5_000 # hard cap on returned rows _SPARQL_MAX_ROWS = 5_000 # hard cap on returned rows
_SPARQL_TIMEOUT_S = 30 # seconds before abandoning the await _SPARQL_TIMEOUT_S = 30 # seconds before abandoning the await
_SPARQL_MAX_CONCURRENT = 4 # semaphore: max simultaneous executions _SPARQL_MAX_CONCURRENT = 4 # semaphore: max simultaneous executions
_SPARQL_MAX_GRAPH_NODES = 50_000 # cap on graph nodes/edges to prevent OOM
# Defense-in-depth against ReDoS: reject inputs longer than this before any
# regex work so that even a future regex regression is bounded. Checked in
# execute_sparql() (not inside _is_read_only_query) so the route can return
# a distinct, actionable error message rather than the generic read-only one.
_SPARQL_MAX_QUERY_LEN = 10_000 # chars
# Semaphore caps how many graph.query calls run concurrently so that # Semaphore caps how many graph.query calls run concurrently so that
# timed-out threads (which keep running in the pool) cannot crowd out # timed-out threads (which keep running in the pool) cannot crowd out
@@ -119,6 +211,24 @@ async def execute_sparql(
req: SparqlRequest, req: SparqlRequest,
session: GraphSession = Depends(get_session), session: GraphSession = Depends(get_session),
): ):
# Resource-limit check: reject oversized queries before any regex work.
# This is intentionally a separate, earlier check from _is_read_only_query
# so clients receive a specific, actionable message rather than the generic
# read-only rejection, and operators can tune _SPARQL_MAX_QUERY_LEN without
# touching query-semantics code.
if len(req.query) > _SPARQL_MAX_QUERY_LEN:
return SparqlResponse(
columns=[],
rows=[],
total=0,
error=(
f"Query exceeds the maximum allowed length of "
f"{_SPARQL_MAX_QUERY_LEN:,} characters "
f"({len(req.query):,} received). "
f"Please shorten your query."
),
)
if not _is_read_only_query(req.query): if not _is_read_only_query(req.query):
return SparqlResponse( return SparqlResponse(
columns=[], columns=[],
@@ -127,7 +237,15 @@ async def execute_sparql(
error="Only SELECT, ASK, CONSTRUCT, and DESCRIBE queries are permitted.", error="Only SELECT, ASK, CONSTRUCT, and DESCRIBE queries are permitted.",
) )
graph = await asyncio.to_thread(_build_rdflib_graph, session) try:
graph = await asyncio.to_thread(_build_rdflib_graph, session)
except ValueError as exc:
return SparqlResponse(
columns=[],
rows=[],
total=0,
error=str(exc),
)
async with _sparql_semaphore: async with _sparql_semaphore:
try: try:
+14 -12
View File
@@ -14,21 +14,23 @@ _HAS_DEFUSEDXML = importlib.util.find_spec("defusedxml") is not None
def _safe_parse_rdf(g: rdflib.Graph, data: bytes, rdf_format: str) -> None: def _safe_parse_rdf(g: rdflib.Graph, data: bytes, rdf_format: str) -> None:
"""Parse RDF bytes into *g*, guarding against XXE for XML-based formats.""" """Parse RDF bytes into *g*, guarding against XXE for XML-based formats.
Raises:
ImportError: If ``defusedxml`` is not installed and the format is XML-based.
"""
xml_formats = {"xml", "rdf", "rdf/xml", "application/rdf+xml"} xml_formats = {"xml", "rdf", "rdf/xml", "application/rdf+xml"}
if rdf_format.lower() in xml_formats: if rdf_format.lower() in xml_formats:
if _HAS_DEFUSEDXML: if not _HAS_DEFUSEDXML:
# defusedxml patches xml.etree so rdflib's XML parser inherits the fix # Fail closed: refuse to parse untrusted XML without XXE protection.
import defusedxml raise ImportError(
defusedxml.defuse_stdlib() "defusedxml is required to safely parse RDF/XML content but is "
else: "not installed. Install it with: pip install defusedxml "
# Warn once; best-effort protection via rdflib's own parser "(or install semantica with the explorer extra: "
import warnings "pip install 'semantica[explorer]')"
warnings.warn(
"defusedxml is not installed. Install it (`pip install defusedxml`) "
"to protect RDF/XML parsing against XXE attacks.",
stacklevel=4,
) )
import defusedxml
defusedxml.defuse_stdlib()
g.parse(data=data, format=rdf_format) g.parse(data=data, format=rdf_format)
def _get_best_label(graph: rdflib.Graph, subject: rdflib.URIRef, predicate: rdflib.URIRef) -> str: def _get_best_label(graph: rdflib.Graph, subject: rdflib.URIRef, predicate: rdflib.URIRef) -> str:
+20
View File
@@ -338,6 +338,13 @@ class ApacheAgeStore:
"host=localhost dbname=agedb user=postgres password=postgres", "host=localhost dbname=agedb user=postgres password=postgres",
) )
self.graph_name = graph_name or config.get("graph_name", "semantica") self.graph_name = graph_name or config.get("graph_name", "semantica")
# SECURITY: Sanitize graph_name to prevent SQL injection in cypher() calls.
# The graph_name is interpolated into SQL: cypher('{graph_name}', $$ ... $$)
if not re.match(r"^[A-Za-z_][A-Za-z0-9_]*$", self.graph_name):
raise ValidationError(
f"Invalid graph_name '{self.graph_name}': must contain only "
"alphanumeric characters and underscores."
)
self._conn = None self._conn = None
@@ -445,7 +452,20 @@ class ApacheAgeStore:
Returns: Returns:
List of raw row tuples from the cursor. List of raw row tuples from the cursor.
Raises:
ValidationError: If the query contains ``$$`` which could break
out of the AGE dollar-quoted string delimiter.
""" """
# SECURITY: Reject queries containing $$ to prevent breakout from
# AGE's dollar-quoted string delimiter. An attacker who injects $$
# into the Cypher query can terminate the cypher() argument and
# append arbitrary SQL.
if "$$" in cypher:
raise ValidationError(
"Query contains forbidden '$$' sequence. "
"Dollar-quoted delimiters are not allowed in Cypher queries."
)
self._ensure_connection() self._ensure_connection()
sql = ( sql = (
f"SELECT * FROM cypher('{self.graph_name}', $$ {cypher} $$) " f"SELECT * FROM cypher('{self.graph_name}', $$ {cypher} $$) "
+16 -15
View File
@@ -47,6 +47,7 @@ from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker from ..utils.progress_tracker import get_progress_tracker
from .query_sanitize import sanitize_identifier
# Optional boto3 for AWS credentials and SigV4 signing # Optional boto3 for AWS credentials and SigV4 signing
try: try:
@@ -981,7 +982,8 @@ class AmazonNeptuneStore:
node_id = props_copy.pop("id", None) or self._generate_id() node_id = props_copy.pop("id", None) or self._generate_id()
use_merge = options.get("merge", True) use_merge = options.get("merge", True)
label_str = ":".join(labels) if labels else "Node" label_str = ":".join(sanitize_identifier(l, "label") for l in labels) if labels else "Node"
safe_keys = [sanitize_identifier(k, "property key") for k in props_copy.keys()]
# Build parameters # Build parameters
params = {"node_id": str(node_id)} params = {"node_id": str(node_id)}
@@ -990,9 +992,7 @@ class AmazonNeptuneStore:
if use_merge: if use_merge:
# MERGE: Return existing node if ID matches, or create new # MERGE: Return existing node if ID matches, or create new
set_parts = [] set_parts = [f"n.{key} = ${key}" for key in safe_keys]
for key in props_copy.keys():
set_parts.append(f"n.{key} = ${key}")
if set_parts: if set_parts:
set_clause = ", ".join(set_parts) set_clause = ", ".join(set_parts)
@@ -1005,9 +1005,7 @@ class AmazonNeptuneStore:
query = f"MERGE (n:{label_str} {{`~id`: $node_id}}) RETURN n" query = f"MERGE (n:{label_str} {{`~id`: $node_id}}) RETURN n"
else: else:
# CREATE: Will fail if node with same ID exists # CREATE: Will fail if node with same ID exists
prop_parts = ["`~id`: $node_id"] prop_parts = ["`~id`: $node_id"] + [f"{key}: ${key}" for key in safe_keys]
for key in props_copy.keys():
prop_parts.append(f"{key}: ${key}")
prop_assignments = ", ".join(prop_parts) prop_assignments = ", ".join(prop_parts)
query = f"CREATE (n:{label_str} {{{prop_assignments}}}) RETURN n" query = f"CREATE (n:{label_str} {{{prop_assignments}}}) RETURN n"
@@ -1162,7 +1160,7 @@ class AmazonNeptuneStore:
# Build query # Build query
if labels: if labels:
label_str = ":".join(labels) label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
query = f"MATCH (n:{label_str})" query = f"MATCH (n:{label_str})"
else: else:
query = "MATCH (n)" query = "MATCH (n)"
@@ -1172,8 +1170,9 @@ class AmazonNeptuneStore:
if properties: if properties:
conditions = [] conditions = []
for key, value in properties.items(): for key, value in properties.items():
param_key = f"prop_{key}" safe_key = sanitize_identifier(key, "property key")
conditions.append(f"n.{key} = ${param_key}") param_key = f"prop_{safe_key}"
conditions.append(f"n.{safe_key} = ${param_key}")
params[param_key] = value params[param_key] = value
query += " WHERE " + " AND ".join(conditions) query += " WHERE " + " AND ".join(conditions)
@@ -1341,15 +1340,16 @@ class AmazonNeptuneStore:
} }
# Build property assignments including ~id # Build property assignments including ~id
safe_rel_type = sanitize_identifier(rel_type, "relationship type")
prop_parts = ["`~id`: $rel_id"] prop_parts = ["`~id`: $rel_id"]
for key, value in props_copy.items(): for key, value in props_copy.items():
prop_parts.append(f"{key}: ${key}") prop_parts.append(f"{sanitize_identifier(key, 'property key')}: ${key}")
params[key] = value params[key] = value
prop_assignments = ", ".join(prop_parts) prop_assignments = ", ".join(prop_parts)
query = ( query = (
f"MATCH (a), (b) WHERE id(a) = $start_id AND id(b) = $end_id " f"MATCH (a), (b) WHERE id(a) = $start_id AND id(b) = $end_id "
f"CREATE (a)-[r:{rel_type} {{{prop_assignments}}}]->(b) RETURN r" f"CREATE (a)-[r:{safe_rel_type} {{{prop_assignments}}}]->(b) RETURN r"
) )
records = self._run_query(query, params) records = self._run_query(query, params)
@@ -1405,7 +1405,7 @@ class AmazonNeptuneStore:
try: try:
self._ensure_connected() self._ensure_connected()
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
params = {} params = {}
if node_id is not None: if node_id is not None:
@@ -1564,7 +1564,8 @@ class AmazonNeptuneStore:
try: try:
self._ensure_connected() self._ensure_connected()
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
depth = int(depth)
if direction == "out": if direction == "out":
pattern = f"-[r{type_filter}*1..{depth}]->" pattern = f"-[r{type_filter}*1..{depth}]->"
@@ -1634,7 +1635,7 @@ class AmazonNeptuneStore:
try: try:
self._ensure_connected() self._ensure_connected()
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
# Neptune doesn't support named path patterns in shortestPath # Neptune doesn't support named path patterns in shortestPath
# Use iterative depth search instead # Use iterative depth search instead
+25 -14
View File
@@ -41,6 +41,7 @@ from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker from ..utils.progress_tracker import get_progress_tracker
from .query_sanitize import sanitize_identifier
# Optional FalkorDB import # Optional FalkorDB import
try: try:
@@ -330,11 +331,11 @@ class FalkorDBStore:
try: try:
graph = self._ensure_graph() graph = self._ensure_graph()
label_str = ":".join(labels) label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
# Build property string for Cypher # Build property string for Cypher
props_str = ", ".join( props_str = ", ".join(
f"{k}: ${k}" for k in properties.keys() f"{sanitize_identifier(k, 'property key')}: ${k}" for k in properties.keys()
) )
query = f"CREATE (n:{label_str} {{{props_str}}}) RETURN id(n) as id, n" query = f"CREATE (n:{label_str} {{{props_str}}}) RETURN id(n) as id, n"
@@ -392,9 +393,9 @@ class FalkorDBStore:
labels = node.get("labels", []) labels = node.get("labels", [])
properties = node.get("properties", {}) properties = node.get("properties", {})
label_str = ":".join(labels) if labels else "Node" label_str = ":".join(sanitize_identifier(l, "label") for l in labels) if labels else "Node"
props_str = ", ".join( props_str = ", ".join(
f"{k}: ${k}" for k in properties.keys() f"{sanitize_identifier(k, 'property key')}: ${k}" for k in properties.keys()
) )
query = f"CREATE (n:{label_str} {{{props_str}}}) RETURN id(n) as id" query = f"CREATE (n:{label_str} {{{props_str}}}) RETURN id(n) as id"
@@ -447,7 +448,7 @@ class FalkorDBStore:
# Build query # Build query
if labels: if labels:
label_str = ":".join(labels) label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
query = f"MATCH (n:{label_str})" query = f"MATCH (n:{label_str})"
else: else:
query = "MATCH (n)" query = "MATCH (n)"
@@ -456,7 +457,8 @@ class FalkorDBStore:
if properties: if properties:
conditions = [] conditions = []
for key in properties.keys(): for key in properties.keys():
conditions.append(f"n.{key} = ${key}") safe_key = sanitize_identifier(key, "property key")
conditions.append(f"n.{safe_key} = ${safe_key}")
query += " WHERE " + " AND ".join(conditions) query += " WHERE " + " AND ".join(conditions)
query += f" RETURN id(n) as id, n, labels(n) as labels LIMIT {limit}" query += f" RETURN id(n) as id, n, labels(n) as labels LIMIT {limit}"
@@ -504,7 +506,8 @@ class FalkorDBStore:
# Build SET clause # Build SET clause
set_parts = [] set_parts = []
for key in properties.keys(): for key in properties.keys():
set_parts.append(f"n.{key} = ${key}") safe_key = sanitize_identifier(key, "property key")
set_parts.append(f"n.{safe_key} = ${safe_key}")
if merge: if merge:
query = f"MATCH (n) WHERE id(n) = $node_id SET {', '.join(set_parts)} RETURN id(n) as id, n, labels(n) as labels" query = f"MATCH (n) WHERE id(n) = $node_id SET {', '.join(set_parts)} RETURN id(n) as id, n, labels(n) as labels"
@@ -592,9 +595,13 @@ class FalkorDBStore:
graph = self._ensure_graph() graph = self._ensure_graph()
properties = properties or {} properties = properties or {}
safe_rel_type = sanitize_identifier(rel_type, "relationship type")
# Build property string # Build property string
if properties: if properties:
props_str = ", ".join(f"{k}: ${k}" for k in properties.keys()) props_str = ", ".join(
f"{sanitize_identifier(k, 'property key')}: ${k}" for k in properties.keys()
)
props_str = f" {{{props_str}}}" props_str = f" {{{props_str}}}"
else: else:
props_str = "" props_str = ""
@@ -602,7 +609,7 @@ class FalkorDBStore:
query = f""" query = f"""
MATCH (a), (b) MATCH (a), (b)
WHERE id(a) = $start_id AND id(b) = $end_id WHERE id(a) = $start_id AND id(b) = $end_id
CREATE (a)-[r:{rel_type}{props_str}]->(b) CREATE (a)-[r:{safe_rel_type}{props_str}]->(b)
RETURN id(r) as id, type(r) as type RETURN id(r) as id, type(r) as type
""" """
@@ -656,7 +663,7 @@ class FalkorDBStore:
""" """
try: try:
graph = self._ensure_graph() graph = self._ensure_graph()
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
if node_id is not None: if node_id is not None:
if direction == "out": if direction == "out":
@@ -806,7 +813,8 @@ class FalkorDBStore:
""" """
try: try:
graph = self._ensure_graph() graph = self._ensure_graph()
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
depth = int(depth)
if direction == "out": if direction == "out":
pattern = f"-[r{type_filter}*1..{depth}]->" pattern = f"-[r{type_filter}*1..{depth}]->"
@@ -860,7 +868,8 @@ class FalkorDBStore:
""" """
try: try:
graph = self._ensure_graph() graph = self._ensure_graph()
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
max_depth = int(max_depth)
query = f""" query = f"""
MATCH path = shortestPath((start)-[r{type_filter}*..{max_depth}]-(end)) MATCH path = shortestPath((start)-[r{type_filter}*..{max_depth}]-(end))
@@ -929,11 +938,13 @@ class FalkorDBStore:
""" """
try: try:
graph = self._ensure_graph() graph = self._ensure_graph()
safe_label = sanitize_identifier(label, "label")
safe_property = sanitize_identifier(property_name, "property key")
if index_type == "fulltext": if index_type == "fulltext":
query = f"CALL db.idx.fulltext.createNodeIndex('{label}', '{property_name}')" query = f"CALL db.idx.fulltext.createNodeIndex('{safe_label}', '{safe_property}')"
else: else:
query = f"CREATE INDEX FOR (n:{label}) ON (n.{property_name})" query = f"CREATE INDEX FOR (n:{safe_label}) ON (n.{safe_property})"
graph.query(query) graph.query(query)
self.logger.info(f"Created {index_type} index on {label}.{property_name}") self.logger.info(f"Created {index_type} index on {label}.{property_name}")
+4 -3
View File
@@ -38,6 +38,7 @@ from ..utils.exceptions import ValidationError
from ..utils.logging import get_logger from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker from ..utils.progress_tracker import get_progress_tracker
from .config import graph_store_config from .config import graph_store_config
from .query_sanitize import sanitize_identifier
class NodeManager: class NodeManager:
@@ -393,12 +394,12 @@ class GraphAnalytics:
""" """
# Build query based on direction # Build query based on direction
if labels: if labels:
label_str = ":".join(labels) label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
match = f"MATCH (n:{label_str})" match = f"MATCH (n:{label_str})"
else: else:
match = "MATCH (n)" match = "MATCH (n)"
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
if direction == "out": if direction == "out":
query = f""" query = f"""
@@ -756,7 +757,7 @@ class GraphStore:
**options: Additional options **options: Additional options
""" """
# Support 'hops' as alias for 'depth' for ContextRetriever compatibility # Support 'hops' as alias for 'depth' for ContextRetriever compatibility
actual_depth = options.get("hops", depth) actual_depth = int(options.get("hops", depth))
return self._manager.analytics.get_neighbors( return self._manager.analytics.get_neighbors(
node_id, rel_type, direction, actual_depth, **options node_id, rel_type, direction, actual_depth, **options
) )
+3 -1
View File
@@ -62,6 +62,7 @@ from typing import Any, Dict, List, Optional, Union
from .config import graph_store_config from .config import graph_store_config
from .graph_store import GraphAnalytics, GraphStore, NodeManager, QueryEngine, RelationshipManager from .graph_store import GraphAnalytics, GraphStore, NodeManager, QueryEngine, RelationshipManager
from .query_sanitize import sanitize_identifier
from .registry import method_registry from .registry import method_registry
# Global store instance # Global store instance
@@ -357,7 +358,8 @@ def update_relationship(
# Default implementation - execute update query # Default implementation - execute update query
store = _get_store() store = _get_store()
set_parts = ", ".join(f"r.{k} = ${k}" for k in properties.keys()) safe_keys = [sanitize_identifier(k, "property key") for k in properties.keys()]
set_parts = ", ".join(f"r.{k} = ${k}" for k in safe_keys)
query = f"MATCH ()-[r]->() WHERE id(r) = $rel_id SET {set_parts} RETURN id(r) as id, type(r) as type, r" query = f"MATCH ()-[r]->() WHERE id(r) = $rel_id SET {set_parts} RETURN id(r) as id, type(r) as type, r"
params = {"rel_id": rel_id, **properties} params = {"rel_id": rel_id, **properties}
result = store.execute_query(query, params) result = store.execute_query(query, params)
+21 -11
View File
@@ -38,6 +38,7 @@ from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker from ..utils.progress_tracker import get_progress_tracker
from .query_sanitize import sanitize_identifier
# Optional Neo4j import # Optional Neo4j import
try: try:
@@ -366,7 +367,7 @@ class Neo4jStore:
) )
try: try:
label_str = ":".join(labels) label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
query = f"CREATE (n:{label_str} $props) RETURN id(n) as id, n" query = f"CREATE (n:{label_str} $props) RETURN id(n) as id, n"
with self.get_session() as session: with self.get_session() as session:
@@ -424,7 +425,7 @@ class Neo4jStore:
labels = node.get("labels", []) labels = node.get("labels", [])
properties = node.get("properties", {}) properties = node.get("properties", {})
label_str = ":".join(labels) if labels else "Node" label_str = ":".join(sanitize_identifier(l, "label") for l in labels) if labels else "Node"
query = f"CREATE (n:{label_str} $props) RETURN id(n) as id, n" query = f"CREATE (n:{label_str} $props) RETURN id(n) as id, n"
result = session.run(query, {"props": properties}) result = session.run(query, {"props": properties})
@@ -505,7 +506,7 @@ class Neo4jStore:
try: try:
# Build query # Build query
if labels: if labels:
label_str = ":".join(labels) label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
query = f"MATCH (n:{label_str})" query = f"MATCH (n:{label_str})"
else: else:
query = "MATCH (n)" query = "MATCH (n)"
@@ -514,7 +515,8 @@ class Neo4jStore:
if properties: if properties:
conditions = [] conditions = []
for key, value in properties.items(): for key, value in properties.items():
conditions.append(f"n.{key} = ${key}") safe_key = sanitize_identifier(key, "property key")
conditions.append(f"n.{safe_key} = ${safe_key}")
query += " WHERE " + " AND ".join(conditions) query += " WHERE " + " AND ".join(conditions)
query += f" RETURN id(n) as id, n, labels(n) as labels LIMIT {limit}" query += f" RETURN id(n) as id, n, labels(n) as labels LIMIT {limit}"
@@ -635,10 +637,11 @@ class Neo4jStore:
try: try:
properties = properties or {} properties = properties or {}
safe_rel_type = sanitize_identifier(rel_type, "relationship type")
query = f""" query = f"""
MATCH (a), (b) MATCH (a), (b)
WHERE id(a) = $start_id AND id(b) = $end_id WHERE id(a) = $start_id AND id(b) = $end_id
CREATE (a)-[r:{rel_type} $props]->(b) CREATE (a)-[r:{safe_rel_type} $props]->(b)
RETURN id(r) as id, type(r) as type, r RETURN id(r) as id, type(r) as type, r
""" """
@@ -696,7 +699,7 @@ class Neo4jStore:
List of matching relationships List of matching relationships
""" """
try: try:
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
if node_id is not None: if node_id is not None:
if direction == "out": if direction == "out":
@@ -857,7 +860,8 @@ class Neo4jStore:
List of neighboring nodes with path information List of neighboring nodes with path information
""" """
try: try:
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
depth = int(depth)
if direction == "out": if direction == "out":
pattern = f"-[r{type_filter}*1..{depth}]->" pattern = f"-[r{type_filter}*1..{depth}]->"
@@ -910,7 +914,8 @@ class Neo4jStore:
Shortest path information or None if not found Shortest path information or None if not found
""" """
try: try:
type_filter = f":{rel_type}" if rel_type else "" type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
max_depth = int(max_depth)
query = f""" query = f"""
MATCH path = shortestPath((start)-[r{type_filter}*..{max_depth}]-(end)) MATCH path = shortestPath((start)-[r{type_filter}*..{max_depth}]-(end))
@@ -975,17 +980,22 @@ class Neo4jStore:
True if index created successfully True if index created successfully
""" """
try: try:
index_name = options.get("index_name", f"idx_{label}_{property_name}") safe_label = sanitize_identifier(label, "label")
safe_property = sanitize_identifier(property_name, "property key")
index_name = sanitize_identifier(
options.get("index_name", f"idx_{safe_label}_{safe_property}"),
"index name",
)
if index_type == "fulltext": if index_type == "fulltext":
query = f""" query = f"""
CREATE FULLTEXT INDEX {index_name} IF NOT EXISTS CREATE FULLTEXT INDEX {index_name} IF NOT EXISTS
FOR (n:{label}) ON EACH [n.{property_name}] FOR (n:{safe_label}) ON EACH [n.{safe_property}]
""" """
else: else:
query = f""" query = f"""
CREATE INDEX {index_name} IF NOT EXISTS CREATE INDEX {index_name} IF NOT EXISTS
FOR (n:{label}) ON (n.{property_name}) FOR (n:{safe_label}) ON (n.{safe_property})
""" """
with self.get_session() as session: with self.get_session() as session:
+32
View File
@@ -0,0 +1,32 @@
"""
Shared identifier validation for Cypher/SPARQL query builders.
Node labels, relationship types, and property keys can't be bound as query
parameters the way values can, so any such identifier that reaches a query
string unvalidated is a direct injection point (GHSA-482h-hw99-h62p).
`age_store.py` already validates its labels/relationship types this way;
this module generalizes that pattern for reuse across the other graph
store backends without introducing an import cycle with `graph_store.py`
or `methods.py`.
"""
import re
from ..utils.exceptions import ValidationError
_IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
def sanitize_identifier(name: str, kind: str = "identifier") -> str:
"""Validate a Cypher/SPARQL label, relationship type, or property key.
Only alphanumeric/underscore identifiers starting with a letter or
underscore are allowed.
"""
if not isinstance(name, str) or not _IDENTIFIER_RE.match(name):
raise ValidationError(
f"Invalid {kind}: {name!r}. Must start with a letter or "
"underscore and contain only alphanumeric characters and "
"underscores."
)
return name
+23 -7
View File
@@ -38,6 +38,7 @@ except (ImportError, OSError):
from ..utils.exceptions import ProcessingError, ValidationError from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker from ..utils.progress_tracker import get_progress_tracker
from .ssrf import parse_bool, request_with_ssrf_guard
@dataclass @dataclass
@@ -78,11 +79,16 @@ class RESTIngestor:
Args: Args:
config: Optional REST API ingestion configuration dictionary config: Optional REST API ingestion configuration dictionary
**kwargs: Additional configuration parameters (merged into config) **kwargs: Additional configuration parameters (merged into config).
Recognized keys include ``allow_private_ips`` (default False) to
opt into fetching private/loopback/link-local endpoints.
""" """
self.logger = get_logger("api_ingestor") self.logger = get_logger("api_ingestor")
self.config = config or {} self.config = config or {}
self.config.update(kwargs) self.config.update(kwargs)
self.allow_private_ips = parse_bool(
self.config.get("allow_private_ips"), default=False
)
# Initialize session with retry strategy # Initialize session with retry strategy
self.session = requests.Session() self.session = requests.Session()
@@ -103,7 +109,10 @@ class RESTIngestor:
# Initialize progress tracker # Initialize progress tracker
self.progress_tracker = get_progress_tracker() self.progress_tracker = get_progress_tracker()
self.logger.debug("REST API ingestor initialized") self.logger.debug(
"REST API ingestor initialized (allow_private_ips=%s)",
self.allow_private_ips,
)
def ingest_endpoint( def ingest_endpoint(
self, self,
@@ -137,7 +146,7 @@ class RESTIngestor:
- metadata: Additional metadata - metadata: Additional metadata
Raises: Raises:
ValidationError: If endpoint is invalid ValidationError: If endpoint is invalid or fails SSRF checks
ProcessingError: If request fails ProcessingError: If request fails
""" """
tracking_id = self.progress_tracker.start_tracking( tracking_id = self.progress_tracker.start_tracking(
@@ -153,10 +162,12 @@ class RESTIngestor:
if headers: if headers:
request_headers.update(headers) request_headers.update(headers)
# Make request # Make request (SSRF-safe; validates URL and each redirect hop)
response = self.session.request( response = request_with_ssrf_guard(
method=method, method,
url=endpoint, endpoint,
session=self.session,
allow_private_ips=self.allow_private_ips,
headers=request_headers, headers=request_headers,
params=params, params=params,
data=data, data=data,
@@ -196,6 +207,11 @@ class RESTIngestor:
}, },
) )
except ValidationError:
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message=f"Invalid endpoint URL: {endpoint}"
)
raise
except requests.exceptions.RequestException as e: except requests.exceptions.RequestException as e:
self.progress_tracker.stop_tracking( self.progress_tracker.stop_tracking(
tracking_id, status="failed", message=str(e) tracking_id, status="failed", message=str(e)
+6
View File
@@ -553,6 +553,12 @@ def ingest_web(
# Get config # Get config
config = ingest_config.get_method_config("web") config = ingest_config.get_method_config("web")
config.update(kwargs) config.update(kwargs)
if "allow_private_ips" in config:
from .ssrf import parse_bool
config["allow_private_ips"] = parse_bool(
config["allow_private_ips"], default=False
)
ingestor = WebIngestor(**config) ingestor = WebIngestor(**config)
+293
View File
@@ -0,0 +1,293 @@
"""SSRF safeguards for ingest HTTP clients.
Validates outbound request URLs before they reach ``requests`` / urllib3 so
user-supplied targets cannot reach private, loopback, or link-local
addresses (including cloud metadata endpoints).
"""
from __future__ import annotations
import concurrent.futures
import ipaddress
import socket
import threading
from typing import Any, Iterable, Optional
from urllib.parse import urljoin, urlparse
import requests
from ..utils.exceptions import ValidationError
ALLOWED_URL_SCHEMES = frozenset({"http", "https"})
_TRUE_STRINGS = frozenset({"1", "true", "yes", "on"})
_FALSE_STRINGS = frozenset({"0", "false", "no", "off"})
# Keep DNS resolution bounded so fake/unreachable hosts in tests and offline
# environments cannot hang request validation indefinitely.
_DNS_RESOLVE_TIMEOUT_SECONDS = 2.0
_DNS_EXECUTOR_WORKERS = 4
# Bound manual redirect following so open redirect chains cannot hang fetches.
_DEFAULT_MAX_REDIRECTS = 10
_REDIRECT_STATUS_CODES = frozenset({301, 302, 303, 307, 308})
_STRIP_BODY_ON_REDIRECT = frozenset({301, 302, 303})
_dns_executor: Optional[concurrent.futures.ThreadPoolExecutor] = None
_dns_executor_lock = threading.Lock()
def parse_bool(value: Any, default: bool = False) -> bool:
"""Parse a config value into a bool without truthy-string pitfalls.
``bool('false')`` is ``True`` in Python, which would silently disable SSRF
protections when string-typed config reaches ingestors. This helper accepts
only explicit bools and a small allowlist of string/int forms.
Args:
value: Config value to interpret. ``None`` yields *default*.
default: Value returned when *value* is ``None``.
Raises:
ValidationError: If *value* is not a recognized boolean form.
"""
if value is None:
return default
if isinstance(value, bool):
return value
if isinstance(value, (int, float)):
if value == 1:
return True
if value == 0:
return False
raise ValidationError(f"Invalid boolean value: {value!r}")
if isinstance(value, str):
normalized = value.strip().lower()
if normalized in _TRUE_STRINGS:
return True
if normalized in _FALSE_STRINGS:
return False
raise ValidationError(f"Invalid boolean value: {value!r}")
raise ValidationError(
f"Invalid boolean type: {type(value).__name__} ({value!r})"
)
def _shutdown_executor(executor: concurrent.futures.ThreadPoolExecutor) -> None:
"""Shut down *executor* without waiting for in-flight DNS lookups."""
try:
executor.shutdown(wait=False, cancel_futures=True)
except TypeError:
# cancel_futures was added in Python 3.9.
executor.shutdown(wait=False)
def _get_dns_executor() -> concurrent.futures.ThreadPoolExecutor:
"""Return a process-wide executor for bounded DNS lookups."""
global _dns_executor
with _dns_executor_lock:
if _dns_executor is None:
_dns_executor = concurrent.futures.ThreadPoolExecutor(
max_workers=_DNS_EXECUTOR_WORKERS,
thread_name_prefix="semantica-ssrf-dns",
)
return _dns_executor
# Explicit blocked networks from issue #867, plus common non-routable ranges.
BLOCKED_NETWORKS = (
ipaddress.ip_network("0.0.0.0/8"),
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("127.0.0.0/8"),
ipaddress.ip_network("169.254.0.0/16"), # link-local / cloud metadata
ipaddress.ip_network("172.16.0.0/12"),
ipaddress.ip_network("192.168.0.0/16"),
ipaddress.ip_network("::1/128"),
ipaddress.ip_network("fc00::/7"),
ipaddress.ip_network("fe80::/10"),
)
def _ip_is_blocked(addr: ipaddress._BaseAddress) -> bool:
if (
addr.is_private
or addr.is_loopback
or addr.is_link_local
or addr.is_reserved
or addr.is_multicast
or addr.is_unspecified
):
return True
return any(addr in network for network in BLOCKED_NETWORKS)
def _hostname_resolves_to_blocked(hostname: str) -> bool:
"""Return True if any resolved address for *hostname* is blocked.
DNS lookups run on a shared thread pool with ``Future.result(timeout=...)``.
Do not use ``with ThreadPoolExecutor(...)`` here: on timeout, leaving the
context waits for the hung ``getaddrinfo`` worker and defeats the bound.
Raises:
ValidationError: If DNS resolution fails or times out. Fail closed so
outbound requests never proceed without confirmed safe IPs.
"""
executor = _get_dns_executor()
owned_executor = False
try:
try:
future = executor.submit(socket.getaddrinfo, hostname, None)
except RuntimeError:
# Shared executor was shut down; use a throwaway pool that never
# blocks the caller on shutdown.
executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
owned_executor = True
future = executor.submit(socket.getaddrinfo, hostname, None)
resolved: Iterable = future.result(timeout=_DNS_RESOLVE_TIMEOUT_SECONDS)
except (socket.gaierror, concurrent.futures.TimeoutError, OSError) as exc:
raise ValidationError(
f"URL host '{hostname}' could not be resolved safely "
"(DNS error or timeout); request blocked"
) from exc
finally:
if owned_executor:
_shutdown_executor(executor)
for info in resolved:
sockaddr = info[4]
addr = ipaddress.ip_address(sockaddr[0])
if _ip_is_blocked(addr):
return True
return False
def validate_url_for_request(
url: str, *, allow_private_ips: bool = False
) -> None:
"""Validate that *url* is safe to fetch over HTTP(S).
Args:
url: Absolute URL to validate.
allow_private_ips: When True, skip private/loopback/link-local checks
(for trusted internal deployments).
Raises:
ValidationError: If the scheme is not http/https, the URL is malformed,
or the host targets a blocked address space.
"""
if not isinstance(url, str) or not url.strip():
raise ValidationError("URL must be a non-empty string")
parsed = urlparse(url.strip())
scheme = (parsed.scheme or "").lower()
if scheme not in ALLOWED_URL_SCHEMES:
raise ValidationError(
f"URL scheme '{parsed.scheme}' is not permitted. "
"Only http and https are allowed."
)
if not parsed.netloc:
raise ValidationError(
f"Invalid URL format: {url}. "
"URL must include scheme (http/https) and netloc (domain)."
)
host = parsed.hostname
if not host:
raise ValidationError(
f"Invalid URL format: {url}. "
"URL must include a hostname."
)
if allow_private_ips:
return
lowered = host.lower().rstrip(".")
if lowered == "localhost" or lowered.endswith(".localhost"):
raise ValidationError(f"URL host is not allowed: {host}")
try:
literal_ip = ipaddress.ip_address(host)
except ValueError:
literal_ip = None
if literal_ip is not None:
if _ip_is_blocked(literal_ip):
raise ValidationError(f"URL points to a blocked address: {host}")
return
if _hostname_resolves_to_blocked(host):
raise ValidationError(
f"URL host '{host}' resolves to a blocked (private/loopback/"
"link-local) address"
)
def request_with_ssrf_guard(
method: str,
url: str,
*,
session: Optional[requests.Session] = None,
allow_private_ips: bool = False,
max_redirects: int = _DEFAULT_MAX_REDIRECTS,
**kwargs: Any,
) -> requests.Response:
"""Perform an HTTP request with SSRF checks on *url* and every redirect.
``requests`` follows redirects by default, which would allow a validated
public URL to bounce into private/loopback/link-local space. This helper
disables automatic redirects and re-validates each ``Location`` target
before issuing the next hop.
"""
kwargs = dict(kwargs)
kwargs.pop("allow_redirects", None)
validate_url_for_request(url, allow_private_ips=allow_private_ips)
requester = session.request if session is not None else requests.request
current_url = url
current_method = method.upper()
redirects_followed = 0
while True:
response = requester(
current_method,
current_url,
allow_redirects=False,
**kwargs,
)
if response.status_code not in _REDIRECT_STATUS_CODES:
return response
if redirects_followed >= max_redirects:
response.close()
raise ValidationError(
f"Exceeded maximum redirects ({max_redirects}) while "
f"fetching '{url}'"
)
location = response.headers.get("Location")
if not location or not str(location).strip():
response.close()
raise ValidationError(
f"Redirect from '{current_url}' is missing a Location header"
)
next_url = urljoin(current_url, str(location).strip())
validate_url_for_request(next_url, allow_private_ips=allow_private_ips)
# Match requests' historical method rewriting for 301/302/303.
if (
response.status_code in _STRIP_BODY_ON_REDIRECT
and current_method not in {"GET", "HEAD"}
):
current_method = "GET"
for key in ("data", "json", "files"):
kwargs.pop(key, None)
# Params apply to the original request URL only; Location is authoritative.
kwargs.pop("params", None)
response.close()
current_url = next_url
redirects_followed += 1
+49 -18
View File
@@ -48,6 +48,7 @@ from urllib3.util.retry import Retry
from ..utils.exceptions import ProcessingError, ValidationError from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker from ..utils.progress_tracker import get_progress_tracker
from .ssrf import parse_bool, request_with_ssrf_guard, validate_url_for_request
@dataclass @dataclass
@@ -338,10 +339,14 @@ class SitemapCrawler:
Sets up the crawler with configuration options. Sets up the crawler with configuration options.
Args: Args:
**config: Crawler configuration options (currently unused) **config: Crawler configuration options. Recognized keys:
- allow_private_ips: Allow private/loopback sitemap hosts
""" """
self.logger = get_logger("sitemap_crawler") self.logger = get_logger("sitemap_crawler")
self.config = config self.config = config
self.allow_private_ips = parse_bool(
config.get("allow_private_ips"), default=False
)
def parse_sitemap(self, sitemap_url: str) -> List[str]: def parse_sitemap(self, sitemap_url: str) -> List[str]:
""" """
@@ -359,10 +364,16 @@ class SitemapCrawler:
Raises: Raises:
ProcessingError: If sitemap cannot be fetched or parsed ProcessingError: If sitemap cannot be fetched or parsed
ValidationError: If sitemap_url fails SSRF checks
""" """
try: try:
# Fetch sitemap # Fetch sitemap (SSRF-safe; validates URL and each redirect hop)
response = requests.get(sitemap_url, timeout=30) response = request_with_ssrf_guard(
"GET",
sitemap_url,
allow_private_ips=self.allow_private_ips,
timeout=30,
)
response.raise_for_status() response.raise_for_status()
# Parse XML # Parse XML
@@ -391,6 +402,8 @@ class SitemapCrawler:
) )
return urls return urls
except ValidationError:
raise
except Exception as e: except Exception as e:
self.logger.error(f"Failed to parse sitemap {sitemap_url}: {e}") self.logger.error(f"Failed to parse sitemap {sitemap_url}: {e}")
raise ProcessingError(f"Failed to parse sitemap: {e}") from e raise ProcessingError(f"Failed to parse sitemap: {e}") from e
@@ -411,10 +424,16 @@ class SitemapCrawler:
Raises: Raises:
ProcessingError: If sitemap index cannot be fetched or parsed ProcessingError: If sitemap index cannot be fetched or parsed
ValidationError: If index_url fails SSRF checks
""" """
try: try:
# Fetch sitemap index # Fetch sitemap index (SSRF-safe; validates URL and each redirect hop)
response = requests.get(index_url, timeout=30) response = request_with_ssrf_guard(
"GET",
index_url,
allow_private_ips=self.allow_private_ips,
timeout=30,
)
response.raise_for_status() response.raise_for_status()
# Parse XML # Parse XML
@@ -448,6 +467,8 @@ class SitemapCrawler:
) )
return all_urls return all_urls
except ValidationError:
raise
except Exception as e: except Exception as e:
self.logger.error(f"Failed to crawl sitemap index {index_url}: {e}") self.logger.error(f"Failed to crawl sitemap index {index_url}: {e}")
raise ProcessingError(f"Failed to crawl sitemap index: {e}") from e raise ProcessingError(f"Failed to crawl sitemap index: {e}") from e
@@ -487,6 +508,7 @@ class WebIngestor:
max_retries: int = 3, max_retries: int = 3,
backoff_factor: float = 1.0, backoff_factor: float = 1.0,
timeout: int = 30, timeout: int = 30,
allow_private_ips: bool = False,
config: Optional[Dict[str, Any]] = None, config: Optional[Dict[str, Any]] = None,
**kwargs, **kwargs,
): ):
@@ -503,12 +525,19 @@ class WebIngestor:
max_retries: Maximum number of retry attempts (default: 3) max_retries: Maximum number of retry attempts (default: 3)
backoff_factor: Backoff factor for retries (default: 1.0) backoff_factor: Backoff factor for retries (default: 1.0)
timeout: Request timeout in seconds (default: 30) timeout: Request timeout in seconds (default: 30)
allow_private_ips: Allow fetching private/loopback/link-local hosts
(default: False). Opt in only for trusted internal deployments.
config: Optional configuration dictionary (merged with kwargs) config: Optional configuration dictionary (merged with kwargs)
**kwargs: Additional configuration parameters **kwargs: Additional configuration parameters
""" """
self.logger = get_logger("web_ingestor") self.logger = get_logger("web_ingestor")
self.config = config or {} self.config = config or {}
self.config.update(kwargs) self.config.update(kwargs)
self.allow_private_ips = parse_bool(
self.config.get("allow_private_ips", allow_private_ips),
default=False,
)
self.config["allow_private_ips"] = self.allow_private_ips
# Initialize HTTP session with retry strategy # Initialize HTTP session with retry strategy
self.session = requests.Session() self.session = requests.Session()
@@ -544,7 +573,8 @@ class WebIngestor:
self.logger.debug( self.logger.debug(
f"Web ingestor initialized: user_agent={user_agent}, " f"Web ingestor initialized: user_agent={user_agent}, "
f"delay={delay}, respect_robots={respect_robots}" f"delay={delay}, respect_robots={respect_robots}, "
f"allow_private_ips={self.allow_private_ips}"
) )
def ingest_url( def ingest_url(
@@ -574,16 +604,9 @@ class WebIngestor:
) )
try: try:
# Validate URL format # Validate before robots check: RobotsChecker.read() makes an unguarded
try: # HTTP request to <host>/robots.txt and must not reach blocked addresses.
parsed = urlparse(url) validate_url_for_request(url, allow_private_ips=self.allow_private_ips)
if not parsed.scheme or not parsed.netloc:
raise ValidationError(
f"Invalid URL format: {url}. "
"URL must include scheme (http/https) and netloc (domain)."
)
except Exception as e:
raise ValidationError(f"Invalid URL: {url}") from e
# Check robots.txt compliance # Check robots.txt compliance
if self.robots_checker and not self.robots_checker.can_fetch(url): if self.robots_checker and not self.robots_checker.can_fetch(url):
@@ -593,11 +616,19 @@ class WebIngestor:
# Apply rate limiting (wait if necessary) # Apply rate limiting (wait if necessary)
self.rate_limiter.wait_if_needed() self.rate_limiter.wait_if_needed()
# Fetch content with retry logic # Fetch content with retry logic (SSRF-safe redirects)
try: try:
request_timeout = timeout or self.config.get("timeout", 30) request_timeout = timeout or self.config.get("timeout", 30)
response = self.session.get(url, timeout=request_timeout) response = request_with_ssrf_guard(
"GET",
url,
session=self.session,
allow_private_ips=self.allow_private_ips,
timeout=request_timeout,
)
response.raise_for_status() response.raise_for_status()
except ValidationError:
raise
except requests.RequestException as e: except requests.RequestException as e:
self.progress_tracker.stop_tracking( self.progress_tracker.stop_tracking(
tracking_id, status="failed", message=str(e) tracking_id, status="failed", message=str(e)
+39 -14
View File
@@ -6,9 +6,14 @@ capturing all steps, inputs, outputs, and transformations.
Usage: Usage:
from semantica.pipeline.pipeline_provenance import PipelineWithProvenance from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
from semantica.pipeline import PipelineBuilder
pipeline = PipelineWithProvenance(provenance=True) builder = PipelineBuilder()
result = pipeline.run(data) builder.add_step("ingest", "file_ingest")
pipeline = builder.build("my_pipeline")
runner = PipelineWithProvenance(pipeline, provenance=True)
result = runner.run(data)
# Tracks all pipeline steps with complete lineage # Tracks all pipeline steps with complete lineage
Author: Semantica Contributors Author: Semantica Contributors
@@ -16,26 +21,37 @@ License: MIT
""" """
from typing import Optional, Any, Dict, List from typing import Optional, Any, Dict, List
from datetime import datetime from datetime import datetime, timezone
import uuid import uuid
import time import time
from .pipeline_builder import Pipeline
from .execution_engine import ExecutionEngine
class PipelineWithProvenance: class PipelineWithProvenance:
"""Pipeline executor with complete provenance tracking.""" """Pipeline executor with complete provenance tracking."""
def __init__( def __init__(
self, self,
pipeline: Pipeline,
provenance: bool = False, provenance: bool = False,
agent_id: Optional[str] = None, agent_id: Optional[str] = None,
is_automated: bool = True, is_automated: bool = True,
**config, **engine_config,
): ):
"""Initialize pipeline with optional provenance.""" """Initialize provenance-tracked pipeline runner.
from .pipeline import Pipeline
Args:
pipeline: A built Pipeline instance (from PipelineBuilder.build()).
provenance: Whether to record provenance metadata.
agent_id: Identifier for the agent running the pipeline.
is_automated: Whether the execution is automated (vs. human-triggered).
**engine_config: Extra keyword arguments forwarded to ExecutionEngine.
"""
self._pipeline = pipeline
self._engine = ExecutionEngine(**engine_config)
self.provenance = provenance self.provenance = provenance
self._pipeline = Pipeline(**config)
self._prov_manager = None self._prov_manager = None
self._agent_id = agent_id or self.__class__.__name__ self._agent_id = agent_id or self.__class__.__name__
self._is_automated = is_automated self._is_automated = is_automated
@@ -47,15 +63,24 @@ class PipelineWithProvenance:
except ImportError: except ImportError:
self.provenance = False self.provenance = False
def run(self, data: Any, source: Optional[str] = None, **kwargs): def run(self, data: Any = None, source: Optional[str] = None, **kwargs):
"""Run pipeline with provenance tracking.""" """Run pipeline with provenance tracking.
Args:
data: Input data to feed into the pipeline.
source: Provenance source label (defaults to "pipeline_execution").
**kwargs: Extra options forwarded to ExecutionEngine.execute_pipeline().
Returns:
ExecutionResult from the engine.
"""
pipeline_id = f"pipeline_{uuid.uuid4().hex[:8]}" pipeline_id = f"pipeline_{uuid.uuid4().hex[:8]}"
start_time = time.time() start_time = time.time()
activity_started_at_time = datetime.utcnow().isoformat() activity_started_at_time = datetime.now(timezone.utc).isoformat()
result = self._pipeline.run(data, **kwargs) result = self._engine.execute_pipeline(self._pipeline, data=data, **kwargs)
elapsed = time.time() - start_time elapsed = time.time() - start_time
activity_ended_at_time = datetime.utcnow().isoformat() activity_ended_at_time = datetime.now(timezone.utc).isoformat()
if self.provenance and self._prov_manager: if self.provenance and self._prov_manager:
self._prov_manager.track_entity( self._prov_manager.track_entity(
@@ -69,9 +94,9 @@ class PipelineWithProvenance:
activity_started_at_time=activity_started_at_time, activity_started_at_time=activity_started_at_time,
activity_ended_at_time=activity_ended_at_time, activity_ended_at_time=activity_ended_at_time,
metadata={ metadata={
"steps": len(self._pipeline.steps) if hasattr(self._pipeline, 'steps') else 0, "steps": len(self._pipeline.steps),
"duration_seconds": elapsed, "duration_seconds": elapsed,
"status": "completed" "status": "completed" if result.success else "failed",
} }
) )
+7 -3
View File
@@ -399,12 +399,16 @@ response = llm.generate("What is artificial intelligence?")
```python ```python
from semantica.pipeline.pipeline_provenance import PipelineWithProvenance from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
from semantica.pipeline import PipelineBuilder
# Create pipeline with provenance builder = PipelineBuilder()
pipeline = PipelineWithProvenance(provenance=True) builder.add_step("ingest", "file_ingest")
pipeline = builder.build("my_pipeline")
runner = PipelineWithProvenance(pipeline, provenance=True)
# Run pipeline - all steps tracked # Run pipeline - all steps tracked
result = pipeline.run( result = runner.run(
data=input_data, data=input_data,
source="input_file.json" source="input_file.json"
) )
+34 -15
View File
@@ -10,7 +10,7 @@ import os
import uvicorn import uvicorn
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
from pathlib import Path from pathlib import Path
from fastapi import FastAPI, HTTPException, Request from fastapi import Depends, FastAPI, HTTPException, Request
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, JSONResponse from fastapi.responses import FileResponse, JSONResponse
from starlette.middleware.base import BaseHTTPMiddleware from starlette.middleware.base import BaseHTTPMiddleware
@@ -25,6 +25,7 @@ try:
from .context.context_graph import ContextGraph from .context.context_graph import ContextGraph
from .explorer.session import GraphSession from .explorer.session import GraphSession
from .explorer.ws import ConnectionManager from .explorer.ws import ConnectionManager
from .explorer.dependencies import anonymous_access_allowed, get_expected_api_key, require_auth
EXPLORER_AVAILABLE = True EXPLORER_AVAILABLE = True
except ImportError: except ImportError:
EXPLORER_AVAILABLE = False EXPLORER_AVAILABLE = False
@@ -39,6 +40,18 @@ async def lifespan(app: FastAPI):
logging.info("Starting up Semantica API...") logging.info("Starting up Semantica API...")
if EXPLORER_AVAILABLE: if EXPLORER_AVAILABLE:
if anonymous_access_allowed():
logging.warning(
"SEMANTICA_ALLOW_ANONYMOUS=true — all Explorer API routes are "
"unauthenticated. Do not expose this process beyond localhost."
)
elif get_expected_api_key():
logging.info("Explorer API authentication: enabled (SEMANTICA_API_KEY set).")
else:
logging.warning(
"Explorer API authentication: NOT CONFIGURED. Protected routes "
"will return 503 until SEMANTICA_API_KEY is set."
)
try: try:
logging.info("Initializing Graph engine and Database connection...") logging.info("Initializing Graph engine and Database connection...")
graph = ContextGraph() graph = ContextGraph()
@@ -79,7 +92,7 @@ app.add_middleware(
allow_origins=_cors_origins, allow_origins=_cors_origins,
allow_credentials=True, allow_credentials=True,
allow_methods=["GET", "POST", "DELETE", "OPTIONS"], allow_methods=["GET", "POST", "DELETE", "OPTIONS"],
allow_headers=["Content-Type", "Authorization"], allow_headers=["Content-Type", "Authorization", "X-API-Key"],
max_age=600, max_age=600,
) )
@@ -159,17 +172,18 @@ if EXPLORER_AVAILABLE:
sparql sparql
) )
app.include_router(analytics.router) _auth = [Depends(require_auth)]
app.include_router(annotations.router) app.include_router(analytics.router, dependencies=_auth)
app.include_router(decisions.router) app.include_router(annotations.router, dependencies=_auth)
app.include_router(enrich.router) app.include_router(decisions.router, dependencies=_auth)
app.include_router(export_import.router) app.include_router(enrich.router, dependencies=_auth)
app.include_router(graph.router) app.include_router(export_import.router, dependencies=_auth)
app.include_router(ontology.router) app.include_router(graph.router, dependencies=_auth)
app.include_router(temporal.router) app.include_router(ontology.router, dependencies=_auth)
app.include_router(vocabulary.router) app.include_router(temporal.router, dependencies=_auth)
app.include_router(provenance.router) app.include_router(vocabulary.router, dependencies=_auth)
app.include_router(sparql.router) app.include_router(provenance.router, dependencies=_auth)
app.include_router(sparql.router, dependencies=_auth)
logging.info("Explorer, Vocabulary, SPARQL, Provenance, and Ontology API routes successfully mounted.") logging.info("Explorer, Vocabulary, SPARQL, Provenance, and Ontology API routes successfully mounted.")
except Exception as exc: except Exception as exc:
@@ -237,8 +251,13 @@ async def serve_spa(full_path: str):
) )
def main(): def main():
"""Server entry point.""" """Server entry point.
uvicorn.run(app, host="0.0.0.0", port=8000)
Binds to loopback by default; set SEMANTICA_HOST to expose beyond
localhost (e.g. behind a reverse proxy that terminates auth/TLS).
"""
host = os.environ.get("SEMANTICA_HOST", "127.0.0.1")
uvicorn.run(app, host=host, port=8000)
if __name__ == "__main__": if __name__ == "__main__":
main() main()
+5 -2
View File
@@ -1180,7 +1180,9 @@ def split_graph_based(
) )
entities = ner_extractor.extract(text) entities = ner_extractor.extract(text)
relations = relation_extractor.extract(text) # Relation extraction requires the extracted entities; failing to pass
# them raises and can trigger the broad fallback to recursive splitting.
relations = relation_extractor.extract(text, entities)
# Build graph # Build graph
G = nx.Graph() G = nx.Graph()
@@ -1623,8 +1625,9 @@ def split_sliding_window(
) )
try: try:
effective_stride = stride if stride is not None else (chunk_size - overlap)
chunker = SlidingWindowChunker( chunker = SlidingWindowChunker(
chunk_size=chunk_size, overlap=overlap, stride=stride, **kwargs chunk_size=chunk_size, overlap=overlap, stride=effective_stride, **kwargs
) )
return chunker.chunk(text, preserve_boundaries=preserve_boundaries, **kwargs) return chunker.chunk(text, preserve_boundaries=preserve_boundaries, **kwargs)
except Exception as e: except Exception as e:
+28 -12
View File
@@ -312,7 +312,7 @@ class BlazegraphStore:
try: try:
# Use SPARQL INSERT for bulk loading # Use SPARQL INSERT for bulk loading
graph = options.get("graph", "") graph = options.get("graph", "")
graph_clause = f"GRAPH <{graph}>" if graph else "" graph_clause = f"GRAPH <{sparql_escaping.validate_uri(graph)}>" if graph else ""
# Build INSERT query # Build INSERT query
insert_data = self._build_insert_data(triplets) insert_data = self._build_insert_data(triplets)
@@ -344,8 +344,10 @@ class BlazegraphStore:
if format == "turtle": if format == "turtle":
lines = [] lines = []
for triplet in triplets: for triplet in triplets:
subject = sparql_escaping.validate_uri(triplet.subject)
predicate = sparql_escaping.validate_uri(triplet.predicate)
lines.append( lines.append(
f"<{triplet.subject}> <{triplet.predicate}> {self._format_object_for_sparql(triplet)} ." f"<{subject}> <{predicate}> {self._format_object_for_sparql(triplet)} ."
) )
return "\n".join(lines) return "\n".join(lines)
else: else:
@@ -353,11 +355,20 @@ class BlazegraphStore:
return self._triplets_to_rdf(triplets, "turtle") return self._triplets_to_rdf(triplets, "turtle")
def _build_insert_data(self, triplets: List[Triplet]) -> str: def _build_insert_data(self, triplets: List[Triplet]) -> str:
"""Build SPARQL INSERT DATA clause.""" """Build SPARQL INSERT DATA clause.
Validates subject/predicate as safe IRIs via
sparql_escaping.validate_uri before interpolation unlike the
object (handled by _format_object_for_sparql), subject/predicate
can't be parameterized in a raw HTTP SPARQL Update POST, so an
unvalidated value is a direct injection point (GHSA-8vgg-8mr4-r236).
"""
lines = [] lines = []
for triplet in triplets: for triplet in triplets:
subject = sparql_escaping.validate_uri(triplet.subject)
predicate = sparql_escaping.validate_uri(triplet.predicate)
lines.append( lines.append(
f"<{triplet.subject}> <{triplet.predicate}> {self._format_object_for_sparql(triplet)} ." f"<{subject}> <{predicate}> {self._format_object_for_sparql(triplet)} ."
) )
return " ".join(lines) return " ".join(lines)
@@ -380,11 +391,14 @@ class BlazegraphStore:
if self._is_uri_value(obj): if self._is_uri_value(obj):
if obj.startswith("<") and obj.endswith(">"): if obj.startswith("<") and obj.endswith(">"):
inner = obj[1:-1] # Validate the inner IRI with the same disallowed-character
if " " in inner or ">" in inner: # set as the unwrapped branch below — a narrower ad-hoc
raise ValueError(f"IRI contains invalid characters: {obj!r}") # check here previously let a pre-wrapped object bypass
return obj # validate_uri() entirely (GHSA-8vgg-8mr4-r236 follow-up).
return f"<{obj}>" inner = sparql_escaping.validate_uri(obj[1:-1])
return f"<{inner}>"
validated_obj = sparql_escaping.validate_uri(obj)
return f"<{validated_obj}>"
escaped = self._escape_literal(obj) escaped = self._escape_literal(obj)
datatype = metadata.get("datatype") or metadata.get("literal_datatype") datatype = metadata.get("datatype") or metadata.get("literal_datatype")
@@ -460,9 +474,9 @@ class BlazegraphStore:
# Build SPARQL query # Build SPARQL query
where_clauses = [] where_clauses = []
if subject: if subject:
where_clauses.append(f"?s = <{subject}>") where_clauses.append(f"?s = <{sparql_escaping.validate_uri(subject)}>")
if predicate: if predicate:
where_clauses.append(f"?p = <{predicate}>") where_clauses.append(f"?p = <{sparql_escaping.validate_uri(predicate)}>")
if object: if object:
where_clauses.append( where_clauses.append(
f"?o = {self._format_object_for_sparql(Triplet(subject='', predicate='', object=object))}" f"?o = {self._format_object_for_sparql(Triplet(subject='', predicate='', object=object))}"
@@ -494,8 +508,10 @@ class BlazegraphStore:
update_endpoint = self._get_update_endpoint() update_endpoint = self._get_update_endpoint()
subject = sparql_escaping.validate_uri(triplet.subject)
predicate = sparql_escaping.validate_uri(triplet.predicate)
query = ( query = (
f"DELETE DATA {{ <{triplet.subject}> <{triplet.predicate}> " f"DELETE DATA {{ <{subject}> <{predicate}> "
f"{self._format_object_for_sparql(triplet)} }}" f"{self._format_object_for_sparql(triplet)} }}"
) )
+3 -3
View File
@@ -319,11 +319,11 @@ class JenaStore:
# Build SPARQL query # Build SPARQL query
query_parts = [] query_parts = []
if subject: if subject:
query_parts.append(f"?s = <{subject}>") query_parts.append(f"?s = <{sparql_escaping.validate_uri(subject)}>")
if predicate: if predicate:
query_parts.append(f"?p = <{predicate}>") query_parts.append(f"?p = <{sparql_escaping.validate_uri(predicate)}>")
if object: if object:
query_parts.append(f"?o = <{object}>") query_parts.append(f"?o = <{sparql_escaping.validate_uri(object)}>")
where_clause = " ".join(query_parts) if query_parts else "" where_clause = " ".join(query_parts) if query_parts else ""
query = f"SELECT ?s ?p ?o WHERE {{ ?s ?p ?o {where_clause} }}" query = f"SELECT ?s ?p ?o WHERE {{ ?s ?p ?o {where_clause} }}"
+33 -12
View File
@@ -452,11 +452,11 @@ class RDF4JStore:
# Build SPARQL query # Build SPARQL query
where_clauses = [] where_clauses = []
if subject: if subject:
where_clauses.append(f"?s = <{subject}>") where_clauses.append(f"?s = <{sparql_escaping.validate_uri(subject)}>")
if predicate: if predicate:
where_clauses.append(f"?p = <{predicate}>") where_clauses.append(f"?p = <{sparql_escaping.validate_uri(predicate)}>")
if object: if object:
where_clauses.append(f"?o = <{object}>") where_clauses.append(f"?o = <{sparql_escaping.validate_uri(object)}>")
where_clause = " ".join(where_clauses) if where_clauses else "" where_clause = " ".join(where_clauses) if where_clauses else ""
query = f"SELECT ?s ?p ?o WHERE {{ ?s ?p ?o {where_clause} }}" query = f"SELECT ?s ?p ?o WHERE {{ ?s ?p ?o {where_clause} }}"
@@ -484,8 +484,18 @@ class RDF4JStore:
update_endpoint = self._get_update_endpoint() update_endpoint = self._get_update_endpoint()
# Use SPARQL DELETE # Use SPARQL DELETE.
query = f"DELETE DATA {{ <{triplet.subject}> <{triplet.predicate}> <{triplet.object}> }}" # subject/predicate must be IRIs — validate_uri enforces that and
# blocks injection through '>' or other SPARQL metacharacters.
# object can be an IRI *or* a literal, so it is routed through
# _format_object_for_ntriples (which internally calls validate_uri
# for URI-shaped values and escape_literal for strings), matching
# the same object-handling semantics used by the add path and by
# BlazegraphStore.delete_triplet (GHSA-8vgg-8mr4-r236 regression fix).
subject = sparql_escaping.validate_uri(triplet.subject)
predicate = sparql_escaping.validate_uri(triplet.predicate)
obj_str = self._format_object_for_ntriples(triplet)
query = f"DELETE DATA {{ <{subject}> <{predicate}> {obj_str} }}"
try: try:
response = requests.post( response = requests.post(
@@ -529,11 +539,14 @@ class RDF4JStore:
if self._is_uri_value(obj): if self._is_uri_value(obj):
if obj.startswith("<") and obj.endswith(">"): if obj.startswith("<") and obj.endswith(">"):
inner = obj[1:-1] # Validate the inner IRI with the same disallowed-character
if " " in inner or ">" in inner: # set as the unwrapped branch below — a narrower ad-hoc
raise ValueError(f"IRI contains invalid characters: {obj!r}") # check here previously let a pre-wrapped object bypass
return obj # validate_uri() entirely (GHSA-8vgg-8mr4-r236 follow-up).
return f"<{obj}>" inner = sparql_escaping.validate_uri(obj[1:-1])
return f"<{inner}>"
validated_obj = sparql_escaping.validate_uri(obj)
return f"<{validated_obj}>"
escaped = sparql_escaping.escape_literal(obj) escaped = sparql_escaping.escape_literal(obj)
datatype = metadata.get("datatype") or metadata.get("literal_datatype") datatype = metadata.get("datatype") or metadata.get("literal_datatype")
@@ -550,9 +563,17 @@ class RDF4JStore:
return f'"{escaped}"' return f'"{escaped}"'
def _triplets_to_ntriples(self, triplets: List[Triplet]) -> str: def _triplets_to_ntriples(self, triplets: List[Triplet]) -> str:
"""Convert triplets to N-Triples format.""" """Convert triplets to N-Triples format.
Validates subject/predicate via sparql_escaping.validate_uri: an
unvalidated value containing '>' or a newline could terminate the
current triple line early and splice extra triples into the
upload stream (GHSA-8vgg-8mr4-r236).
"""
lines = [] lines = []
for triplet in triplets: for triplet in triplets:
subject = sparql_escaping.validate_uri(triplet.subject)
predicate = sparql_escaping.validate_uri(triplet.predicate)
obj_str = self._format_object_for_ntriples(triplet) obj_str = self._format_object_for_ntriples(triplet)
lines.append(f"<{triplet.subject}> <{triplet.predicate}> {obj_str} .") lines.append(f"<{subject}> <{predicate}> {obj_str} .")
return "\n".join(lines) return "\n".join(lines)
+32 -13
View File
@@ -573,8 +573,8 @@ class VectorStore:
Args: Args:
path: Directory path to save to path: Directory path to save to
""" """
import json
import os import os
import pickle
os.makedirs(path, exist_ok=True) os.makedirs(path, exist_ok=True)
@@ -586,17 +586,20 @@ class VectorStore:
elif self._backend_store is not None and hasattr(self._backend_store, "save_index"): elif self._backend_store is not None and hasattr(self._backend_store, "save_index"):
self._backend_store.save_index(os.path.join(path, "index.bin")) self._backend_store.save_index(os.path.join(path, "index.bin"))
# Save Python-level data # Save Python-level data using JSON (safe serialization).
# pickle is intentionally avoided to prevent arbitrary code execution
# if a malicious .pkl file is placed in the store directory.
data = { data = {
"vectors": getattr(self, "vectors", {}), "vectors": {k: v.tolist() if hasattr(v, "tolist") else list(v)
for k, v in getattr(self, "vectors", {}).items()},
"metadata": getattr(self, "metadata", {}), "metadata": getattr(self, "metadata", {}),
"config": self.config, "config": self.config,
"backend": self.backend, "backend": self.backend,
"dimension": self.dimension "dimension": self.dimension
} }
with open(os.path.join(path, "store_data.pkl"), "wb") as f: with open(os.path.join(path, "store_data.json"), "w", encoding="utf-8") as f:
pickle.dump(data, f) json.dump(data, f)
self.logger.info(f"Saved vector store to {path}") self.logger.info(f"Saved vector store to {path}")
@@ -606,17 +609,33 @@ class VectorStore:
Args: Args:
path: Directory path to load from path: Directory path to load from
"""
import os
import pickle
data_path = os.path.join(path, "store_data.pkl") Raises:
if not os.path.exists(data_path): RuntimeError: If only a legacy pickle file is found (security risk).
self.logger.warning(f"Store data not found: {data_path}") """
import json
import os
json_path = os.path.join(path, "store_data.json")
legacy_pkl_path = os.path.join(path, "store_data.pkl")
if os.path.exists(json_path):
data_path = json_path
elif os.path.exists(legacy_pkl_path):
# Refuse to load pickle files to prevent arbitrary code execution.
# A crafted .pkl file can execute arbitrary Python when deserialized.
raise RuntimeError(
f"Legacy pickle file found at {legacy_pkl_path}. "
"Pickle deserialization is disabled for security (arbitrary code "
"execution risk). Please re-save the vector store to migrate "
"to the safe JSON format: vs.save(path)"
)
else:
self.logger.warning(f"Store data not found in: {path}")
return return
with open(data_path, "rb") as f: with open(data_path, "r", encoding="utf-8") as f:
data = pickle.load(f) data = json.load(f)
self.vectors = data.get("vectors", {}) self.vectors = data.get("vectors", {})
self.metadata = data.get("metadata", {}) self.metadata = data.get("metadata", {})
@@ -0,0 +1,631 @@
"""
Coverage for ClusterBuilder, MergeStrategyManager, PropertyMergeRule,
EntityMerger.merge_duplicates, embedding similarity, and incremental detection.
Addresses issue #866.
"""
import unittest
from unittest.mock import patch
from semantica.deduplication.cluster_builder import Cluster, ClusterBuilder
from semantica.deduplication.duplicate_detector import (
DuplicateDetector,
DuplicateGroup,
)
from semantica.deduplication.entity_merger import EntityMerger
from semantica.deduplication.merge_strategy import (
MergeStrategy,
MergeStrategyManager,
PropertyMergeRule,
)
from semantica.deduplication.methods import calculate_similarity
from semantica.deduplication.similarity_calculator import SimilarityCalculator
from semantica.utils.exceptions import ValidationError
class TestClusterBuilderCoverage(unittest.TestCase):
"""ClusterBuilder: union-find clustering and size filtering."""
def setUp(self):
self.entities = [
{
"id": "a1",
"name": "Apple Inc.",
"type": "Company",
"properties": {"industry": "Technology"},
},
{
"id": "a2",
"name": "Apple",
"type": "Company",
"properties": {"industry": "Tech"},
},
{
"id": "m1",
"name": "Microsoft Corporation",
"type": "Company",
"properties": {"industry": "Software"},
},
{
"id": "g1",
"name": "Google LLC",
"type": "Company",
"properties": {"industry": "Internet"},
},
]
def test_known_duplicates_share_cluster_id(self):
"""Entities linked by high-similarity pairs land in the same cluster."""
builder = ClusterBuilder(similarity_threshold=0.4, min_cluster_size=2)
result = builder.build_clusters(self.entities)
id_to_cluster = {}
for cluster in result.clusters:
for entity in cluster.entities:
id_to_cluster[entity["id"]] = cluster.cluster_id
self.assertIn("a1", id_to_cluster)
self.assertIn("a2", id_to_cluster)
self.assertEqual(id_to_cluster["a1"], id_to_cluster["a2"])
def test_unrelated_entities_get_different_cluster_ids(self):
"""Unrelated brands must not share a cluster ID when they form clusters."""
builder = ClusterBuilder(similarity_threshold=0.8, min_cluster_size=2)
a1 = {"id": "a1", "name": "Apple Inc.", "type": "Company"}
a2 = {"id": "a2", "name": "Apple", "type": "Company"}
m1 = {"id": "m1", "name": "Microsoft Corporation", "type": "Company"}
m2 = {"id": "m2", "name": "Microsoft Corp", "type": "Company"}
# Deterministic pairs: intra-brand duplicates only — no Apple↔Microsoft edge
pairs = [(a1, a2, 0.95), (m1, m2, 0.94)]
with patch.object(
builder.similarity_calculator,
"batch_calculate_similarity",
return_value=pairs,
):
clusters = builder._graph_based_clustering(
[a1, a2, m1, m2], threshold=0.8
)
id_to_cluster = {
e["id"]: c.cluster_id for c in clusters for e in c.entities
}
self.assertEqual(set(id_to_cluster), {"a1", "a2", "m1", "m2"})
self.assertEqual(id_to_cluster["a1"], id_to_cluster["a2"])
self.assertEqual(id_to_cluster["m1"], id_to_cluster["m2"])
self.assertNotEqual(id_to_cluster["a1"], id_to_cluster["m1"])
def test_singleton_entities_are_unclustered(self):
"""Entities with no similar peers remain unclustered (min_cluster_size=2)."""
builder = ClusterBuilder(similarity_threshold=0.9, min_cluster_size=2)
result = builder.build_clusters(self.entities)
unclustered_ids = {e["id"] for e in result.unclustered}
# High threshold: Google and Microsoft should not form a pair cluster
self.assertTrue(
"g1" in unclustered_ids or "m1" in unclustered_ids,
"Dissimilar entities should appear in unclustered",
)
def test_graph_clustering_from_similarity_pairs(self):
"""Union-find path merges transitively via mocked similarity pairs."""
builder = ClusterBuilder(similarity_threshold=0.8, min_cluster_size=2)
e1 = {"id": "1", "name": "A"}
e2 = {"id": "2", "name": "B"}
e3 = {"id": "3", "name": "C"}
# 1~2 and 2~3 => all three in one cluster
pairs = [(e1, e2, 0.95), (e2, e3, 0.92)]
with patch.object(
builder.similarity_calculator,
"batch_calculate_similarity",
return_value=pairs,
):
clusters = builder._graph_based_clustering([e1, e2, e3], threshold=0.8)
self.assertEqual(len(clusters), 1)
self.assertEqual({e["id"] for e in clusters[0].entities}, {"1", "2", "3"})
def test_unrelated_pairs_form_separate_clusters(self):
"""Two disjoint similarity pairs produce two cluster IDs."""
builder = ClusterBuilder(similarity_threshold=0.8, min_cluster_size=2)
a1, a2 = {"id": "a1", "name": "Apple"}, {"id": "a2", "name": "Apple Inc"}
m1, m2 = {"id": "m1", "name": "MSFT"}, {"id": "m2", "name": "Microsoft"}
pairs = [(a1, a2, 0.95), (m1, m2, 0.94)]
with patch.object(
builder.similarity_calculator,
"batch_calculate_similarity",
return_value=pairs,
):
clusters = builder._graph_based_clustering([a1, a2, m1, m2], threshold=0.8)
self.assertEqual(len(clusters), 2)
cluster_ids = {c.cluster_id for c in clusters}
self.assertEqual(len(cluster_ids), 2)
def test_quality_metrics_populated(self):
builder = ClusterBuilder(similarity_threshold=0.4, min_cluster_size=2)
result = builder.build_clusters(self.entities[:2])
self.assertIn("total_clusters", result.quality_metrics)
self.assertIn("average_quality", result.quality_metrics)
def test_update_clusters_adds_matching_entity(self):
"""Incremental update attaches a near-duplicate into an existing cluster."""
builder = ClusterBuilder(similarity_threshold=0.8, min_cluster_size=2)
a1 = {
"id": "a1",
"name": "Apple Inc.",
"type": "Company",
"properties": {"industry": "Technology"},
}
a2 = {
"id": "a2",
"name": "Apple",
"type": "Company",
"properties": {"industry": "Tech"},
}
existing = [Cluster(cluster_id="cluster_0", entities=[a1])]
with patch.object(
builder, "_entity_cluster_similarity", return_value=0.95
), patch.object(
builder.similarity_calculator,
"batch_calculate_similarity",
return_value=[(a1, a2, 0.95)],
):
result = builder.update_clusters(existing, [a2])
self.assertEqual(len(result.clusters), 1)
clustered_ids = {e["id"] for e in result.clusters[0].entities}
self.assertEqual(clustered_ids, {"a1", "a2"})
self.assertEqual(result.unclustered, [])
class TestMergeStrategyManagerCoverage(unittest.TestCase):
"""MergeStrategyManager and PropertyMergeRule conflict resolution."""
def setUp(self):
self.entities = [
{
"id": "e1",
"name": "Apple Inc.",
"type": "Company",
"properties": {
"industry": "Technology",
"description": "Short",
"hq": "Cupertino",
},
"relationships": [
{"subject": "e1", "predicate": "competitor", "object": "Microsoft"}
],
"confidence": 0.7,
},
{
"id": "e2",
"name": "Apple",
"type": "Company",
"properties": {
"industry": "Tech",
"description": "A much longer company description",
"founded": "1976",
},
"relationships": [
{"subject": "e2", "predicate": "competitor", "object": "Google"}
],
"confidence": 0.9,
},
]
def test_keep_first_strategy_selects_first_entity(self):
manager = MergeStrategyManager(default_strategy="keep_first")
result = manager.merge_entities(self.entities, strategy="keep_first")
self.assertEqual(result.merged_entity["id"], "e1")
self.assertEqual(result.metadata["strategy"], "keep_first")
def test_keep_last_strategy_selects_last_entity(self):
manager = MergeStrategyManager()
result = manager.merge_entities(self.entities, strategy="keep_last")
self.assertEqual(result.merged_entity["id"], "e2")
def test_keep_most_complete_prefers_richer_entity(self):
# e1 has 3 props + 1 rel; e2 has 3 props + 1 rel — tie goes to max() first max
richer = [
{
"id": "sparse",
"name": "Sparse",
"type": "Company",
"properties": {"a": 1},
"relationships": [],
},
{
"id": "rich",
"name": "Rich Co",
"type": "Company",
"properties": {"a": 1, "b": 2, "c": 3},
"relationships": [{"subject": "rich", "predicate": "owns", "object": "x"}],
},
]
manager = MergeStrategyManager(default_strategy="keep_most_complete")
result = manager.merge_entities(richer)
self.assertEqual(result.merged_entity["id"], "rich")
def test_keep_highest_confidence_selects_confident_entity(self):
manager = MergeStrategyManager()
result = manager.merge_entities(
self.entities, strategy=MergeStrategy.KEEP_HIGHEST_CONFIDENCE
)
self.assertEqual(result.merged_entity["id"], "e2")
def test_conflicting_property_keep_first(self):
manager = MergeStrategyManager(default_strategy="keep_first")
result = manager.merge_entities(self.entities, strategy="keep_first")
# Base is e1; conflicting industry keeps first value under keep_first
self.assertEqual(result.merged_entity["properties"]["industry"], "Technology")
# Non-conflicting property from e2 is still absorbed
self.assertEqual(result.merged_entity["properties"]["founded"], "1976")
def test_conflicting_property_keep_last(self):
manager = MergeStrategyManager()
manager.add_property_rule("industry", "keep_last")
result = manager.merge_entities(self.entities, strategy="keep_first")
self.assertEqual(result.merged_entity["properties"]["industry"], "Tech")
def test_merge_all_combines_conflicting_values(self):
manager = MergeStrategyManager()
manager.add_property_rule("industry", "merge_all")
result = manager.merge_entities(self.entities, strategy="keep_first")
industry = result.merged_entity["properties"]["industry"]
self.assertIsInstance(industry, list)
self.assertIn("Technology", industry)
self.assertIn("Tech", industry)
def test_relationships_are_unioned(self):
manager = MergeStrategyManager(default_strategy="keep_first")
result = manager.merge_entities(self.entities)
objects = {r.get("object") for r in result.merged_entity["relationships"]}
self.assertIn("Microsoft", objects)
self.assertIn("Google", objects)
def test_empty_entities_raises_validation_error(self):
manager = MergeStrategyManager()
with self.assertRaises(ValidationError):
manager.merge_entities([])
def test_single_entity_returns_unchanged(self):
manager = MergeStrategyManager()
result = manager.merge_entities([self.entities[0]])
self.assertEqual(result.merged_entity["id"], "e1")
self.assertEqual(result.merged_entities, [self.entities[0]])
def test_invalid_default_strategy_falls_back(self):
manager = MergeStrategyManager(default_strategy="not_a_real_strategy")
self.assertEqual(manager.default_strategy, MergeStrategy.KEEP_MOST_COMPLETE)
def test_validate_merge_reports_missing_name(self):
manager = MergeStrategyManager()
result = manager.merge_entities(self.entities)
result.merged_entity["name"] = None
validation = manager.validate_merge(result)
self.assertFalse(validation["valid"])
self.assertTrue(any("name" in issue.lower() for issue in validation["issues"]))
class TestPropertyMergeRuleCoverage(unittest.TestCase):
"""PropertyMergeRule: custom per-property conflict resolution."""
def test_custom_rule_takes_longer_description(self):
manager = MergeStrategyManager(default_strategy="keep_first")
def longer_string(v1, v2):
return v1 if len(str(v1)) >= len(str(v2)) else v2
manager.add_property_rule(
"description",
"custom",
conflict_resolution=longer_string,
priority=10,
)
entities = [
{
"id": "e1",
"name": "Alpha",
"type": "Org",
"properties": {"description": "Short"},
},
{
"id": "e2",
"name": "Alpha Inc",
"type": "Org",
"properties": {
"description": "This is a much longer description of the entity"
},
},
]
result = manager.merge_entities(entities, strategy="keep_first")
self.assertEqual(
result.merged_entity["properties"]["description"],
"This is a much longer description of the entity",
)
def test_property_merge_rule_dataclass_fields(self):
rule = PropertyMergeRule(
property_name="description",
strategy=MergeStrategy.CUSTOM,
conflict_resolution=lambda a, b: a,
priority=5,
)
self.assertEqual(rule.property_name, "description")
self.assertEqual(rule.strategy, MergeStrategy.CUSTOM)
self.assertEqual(rule.priority, 5)
self.assertIsNotNone(rule.conflict_resolution)
def test_top_level_name_rule_overrides_base(self):
manager = MergeStrategyManager(default_strategy="keep_first")
manager.add_property_rule("name", "keep_last")
entities = [
{"id": "e1", "name": "First Name", "type": "Org", "properties": {}},
{"id": "e2", "name": "Second Name", "type": "Org", "properties": {}},
]
result = manager.merge_entities(entities)
self.assertEqual(result.merged_entity["name"], "Second Name")
def test_invalid_property_strategy_defaults(self):
manager = MergeStrategyManager()
manager.add_property_rule("industry", "bogus_strategy")
self.assertEqual(
manager.property_rules["industry"].strategy,
MergeStrategy.KEEP_MOST_COMPLETE,
)
class TestEntityMergerMergeDuplicates(unittest.TestCase):
"""EntityMerger.merge_duplicates end-to-end with DuplicateGroup path."""
def setUp(self):
self.entities = [
{
"id": "a1",
"name": "Apple Inc.",
"type": "Company",
"properties": {"industry": "Technology", "hq": "Cupertino"},
"relationships": [],
},
{
"id": "a2",
"name": "Apple",
"type": "Company",
"properties": {"industry": "Tech"},
"relationships": [],
},
{
"id": "m1",
"name": "Microsoft Corp",
"type": "Company",
"properties": {"industry": "Software"},
"relationships": [],
},
]
def test_merge_duplicates_returns_operations_for_groups(self):
merger = EntityMerger(
preserve_provenance=True,
detector={
"similarity_threshold": 0.4,
"confidence_threshold": 0.4,
},
)
operations = merger.merge_duplicates(self.entities, strategy="keep_first")
self.assertGreater(len(operations), 0)
op = operations[0]
self.assertGreaterEqual(len(op.source_entities), 2)
self.assertIn("id", op.merged_entity)
self.assertIn("group_confidence", op.metadata)
# Provenance preserved (written by EntityMerger._add_provenance()).
provenance = op.merged_entity.get("metadata", {}).get("provenance", {})
self.assertIn("merged_from", provenance)
self.assertIn("merge_count", provenance)
self.assertEqual(provenance["merge_count"], len(op.source_entities))
def test_merge_duplicates_provenance_absent_when_disabled(self):
merger = EntityMerger(
preserve_provenance=False,
detector={
"similarity_threshold": 0.4,
"confidence_threshold": 0.4,
},
)
operations = merger.merge_duplicates(self.entities, strategy="keep_first")
self.assertGreater(len(operations), 0)
op = operations[0]
provenance = op.merged_entity.get("metadata", {}).get("provenance")
self.assertTrue(
provenance is None or provenance == {},
"metadata.provenance should be absent when preserve_provenance=False",
)
def test_merge_duplicates_with_explicit_duplicate_group(self):
"""merge_entity_group path used when a DuplicateGroup is already known."""
group = DuplicateGroup(
entities=[self.entities[0], self.entities[1]],
similarity_scores={("a1", "a2"): 0.85},
confidence=0.9,
)
merger = EntityMerger(preserve_provenance=True)
op = merger.merge_entity_group(group.entities, strategy="keep_most_complete")
self.assertEqual(len(op.source_entities), 2)
self.assertEqual(
{e["id"] for e in op.source_entities},
{"a1", "a2"},
)
self.assertIn(op.merged_entity["id"], {"a1", "a2"})
def test_merge_duplicates_recorded_in_history(self):
merger = EntityMerger(
detector={"similarity_threshold": 0.4, "confidence_threshold": 0.4}
)
ops = merger.merge_duplicates(self.entities)
self.assertEqual(len(merger.get_merge_history()), len(ops))
class TestEmbeddingSimilarityCoverage(unittest.TestCase):
"""SimilarityCalculator / methods embedding path (vectors, no external model)."""
def test_calculate_similarity_method_embedding(self):
e1 = {
"id": "1",
"name": "Alpha",
"embedding": [1.0, 0.0, 0.0],
}
e2 = {
"id": "2",
"name": "Beta",
"embedding": [1.0, 0.0, 0.0],
}
result = calculate_similarity(e1, e2, method="embedding")
self.assertEqual(result.method, "embedding")
self.assertAlmostEqual(result.score, 1.0)
def test_calculate_similarity_method_embedding_missing_vectors(self):
result = calculate_similarity(
{"id": "1", "name": "A"},
{"id": "2", "name": "B"},
method="embedding",
)
self.assertEqual(result.score, 0.0)
self.assertEqual(result.method, "embedding")
def test_calculate_similarity_method_embedding_orthogonal(self):
e1 = {"embedding": [1.0, 0.0]}
e2 = {"embedding": [0.0, 1.0]}
result = calculate_similarity(e1, e2, method="embedding")
# Cosine of orthogonal vectors is 0; normalized to (0+1)/2 = 0.5
self.assertAlmostEqual(result.score, 0.5)
def test_similarity_calculator_includes_embedding_component(self):
calculator = SimilarityCalculator(
string_weight=0.2,
property_weight=0.2,
relationship_weight=0.0,
embedding_weight=0.6,
prefilter_enabled=False,
)
e1 = {
"id": "1",
"name": "Apple Inc.",
"properties": {},
"embedding": [0.9, 0.1, 0.0],
}
e2 = {
"id": "2",
"name": "Apple",
"properties": {},
"embedding": [0.85, 0.15, 0.0],
}
result = calculator.calculate_similarity(e1, e2, track=False)
self.assertIn("embedding", result.components)
self.assertGreater(result.components["embedding"], 0.5)
self.assertGreater(result.score, 0.0)
def test_embedding_similarity_mismatched_dimensions(self):
calculator = SimilarityCalculator()
score = calculator.calculate_embedding_similarity([1.0, 0.0], [1.0, 0.0, 0.0])
self.assertEqual(score, 0.0)
class TestIncrementalDetectionCoverage(unittest.TestCase):
"""Incremental O(n×m) detection: new entities vs existing set."""
def setUp(self):
self.existing = [
{
"id": "a1",
"name": "Apple Inc.",
"type": "Company",
"properties": {"industry": "Technology"},
},
{
"id": "m1",
"name": "Microsoft Corp",
"type": "Company",
"properties": {"industry": "Software"},
},
]
self.new = [
{
"id": "a2",
"name": "Apple",
"type": "Company",
"properties": {"industry": "Tech"},
},
{
"id": "g1",
"name": "Google LLC",
"type": "Company",
"properties": {"industry": "Internet"},
},
]
def test_incremental_detect_finds_new_vs_existing_duplicate(self):
detector = DuplicateDetector(
similarity_threshold=0.4,
confidence_threshold=0.4,
)
candidates = detector.incremental_detect(self.new, self.existing)
names = {(c.entity1["name"], c.entity2["name"]) for c in candidates}
found = any(
("Apple" in pair and "Apple Inc." in pair) for pair in names
)
self.assertTrue(found, f"Expected Apple/Apple Inc. match, got {names}")
def test_incremental_detect_does_not_compare_within_new_set(self):
"""Incremental path only compares new×existing, not new×new."""
detector = DuplicateDetector(
similarity_threshold=0.3,
confidence_threshold=0.3,
)
# Two near-identical new entities; existing is unrelated
new = [
{"id": "n1", "name": "Acme Corp", "type": "Company", "properties": {}},
{"id": "n2", "name": "Acme Corporation", "type": "Company", "properties": {}},
]
existing = [
{"id": "z1", "name": "Zebra Industries", "type": "Company", "properties": {}}
]
candidates = detector.incremental_detect(new, existing)
pair_ids = {
frozenset((c.entity1["id"], c.entity2["id"])) for c in candidates
}
self.assertNotIn(frozenset({"n1", "n2"}), pair_ids)
def test_methods_incremental_detect_duplicates(self):
from semantica.deduplication.duplicate_detector import DuplicateCandidate
from semantica.deduplication.methods import detect_duplicates
results = detect_duplicates(
self.new + self.existing,
method="incremental",
similarity_threshold=0.4,
confidence_threshold=0.4,
new_entities=self.new,
existing_entities=self.existing,
)
self.assertIsInstance(results, list)
self.assertGreater(len(results), 0)
for candidate in results:
self.assertIsInstance(candidate, DuplicateCandidate)
self.assertGreaterEqual(candidate.similarity_score, 0.4)
# Wrapper must route new×existing only: one id from each set
ids = {candidate.entity1["id"], candidate.entity2["id"]}
self.assertTrue(ids & {"a2", "g1"})
self.assertTrue(ids & {"a1", "m1"})
names = {
frozenset((c.entity1["name"], c.entity2["name"])) for c in results
}
self.assertIn(frozenset({"Apple", "Apple Inc."}), names)
if __name__ == "__main__":
unittest.main()
+17
View File
@@ -0,0 +1,17 @@
"""Shared fixtures for explorer API tests.
Most of this suite predates the API-key auth layer added for
GHSA-j4mq-hprp-987v and exercises route logic, not authentication. Default
every test under tests/explorer/ to SEMANTICA_ALLOW_ANONYMOUS=true so those
tests keep talking to the Explorer without needing an X-API-Key header.
Auth-specific tests (test_explorer_auth.py) override this per-test via the
same `monkeypatch` fixture.
"""
import pytest
@pytest.fixture(autouse=True)
def _default_to_anonymous_explorer_access(monkeypatch):
monkeypatch.setenv("SEMANTICA_ALLOW_ANONYMOUS", "true")
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
+211
View File
@@ -0,0 +1,211 @@
"""Tests for the API-key auth dependency added for GHSA-j4mq-hprp-987v
(missing authentication on all Explorer API routes).
Covers: protected routes refuse requests when no key is configured (fail
closed, not fail open), reject wrong/missing keys once a key is
configured, accept the correct key, remain reachable when
SEMANTICA_ALLOW_ANONYMOUS=true is set explicitly, and that health/info/
static routes stay public regardless. Also covers the /ws/graph-updates
handshake, which can't use the same FastAPI Depends() plumbing since
browsers can't set custom headers on a WebSocket handshake.
"""
import pytest
from semantica.context.context_graph import ContextGraph
from semantica.explorer.app import create_app
from semantica.explorer.session import GraphSession
try:
from starlette.testclient import TestClient
except ImportError:
pytest.skip(
"starlette TestClient is required for explorer tests. Install semantica[explorer].",
allow_module_level=True,
)
def _build_sample_graph() -> ContextGraph:
graph = ContextGraph(advanced_analytics=False)
graph.add_node("python", node_type="language", content="Python")
return graph
@pytest.fixture
def client():
session = GraphSession(_build_sample_graph())
app = create_app(session=session)
with TestClient(app) as test_client:
yield test_client
# ---------------------------------------------------------------------------
# Fail-closed: no SEMANTICA_API_KEY and no explicit anonymous opt-in.
# ---------------------------------------------------------------------------
def test_protected_route_returns_503_when_auth_not_configured(client, monkeypatch):
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
resp = client.get("/api/graph/nodes")
assert resp.status_code == 503
def test_write_route_also_refuses_when_auth_not_configured(client, monkeypatch):
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
resp = client.post("/api/export", json={"format": "json"})
assert resp.status_code == 503
# ---------------------------------------------------------------------------
# Configured key: wrong/missing key rejected, correct key accepted.
# ---------------------------------------------------------------------------
def test_protected_route_rejects_missing_key(client, monkeypatch):
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.setenv("SEMANTICA_API_KEY", "correct-key")
resp = client.get("/api/graph/nodes")
assert resp.status_code == 401
def test_protected_route_rejects_wrong_key(client, monkeypatch):
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.setenv("SEMANTICA_API_KEY", "correct-key")
resp = client.get("/api/graph/nodes", headers={"X-API-Key": "wrong-key"})
assert resp.status_code == 401
def test_protected_route_accepts_correct_key(client, monkeypatch):
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.setenv("SEMANTICA_API_KEY", "correct-key")
resp = client.get("/api/graph/nodes", headers={"X-API-Key": "correct-key"})
assert resp.status_code == 200
# ---------------------------------------------------------------------------
# Explicit opt-in: SEMANTICA_ALLOW_ANONYMOUS=true.
# ---------------------------------------------------------------------------
def test_anonymous_opt_in_allows_requests_without_a_key(client, monkeypatch):
monkeypatch.setenv("SEMANTICA_ALLOW_ANONYMOUS", "true")
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
resp = client.get("/api/graph/nodes")
assert resp.status_code == 200
# ---------------------------------------------------------------------------
# Public routes stay public regardless of auth configuration.
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("path", ["/api/health", "/api/info"])
def test_public_routes_stay_public_when_auth_not_configured(client, monkeypatch, path):
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
resp = client.get(path)
assert resp.status_code == 200
# ---------------------------------------------------------------------------
# WebSocket handshake: header or query-param key, same policy as REST.
# ---------------------------------------------------------------------------
def test_websocket_rejects_connection_without_key_when_configured(client, monkeypatch):
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.setenv("SEMANTICA_API_KEY", "correct-key")
with pytest.raises(Exception):
with client.websocket_connect("/ws/graph-updates"):
pass
def test_websocket_accepts_connection_with_correct_query_param_key(client, monkeypatch):
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.setenv("SEMANTICA_API_KEY", "correct-key")
with client.websocket_connect("/ws/graph-updates?api_key=correct-key") as websocket:
ack = websocket.receive_json()
assert ack["event"] == "connection_ack"
def test_websocket_accepts_connection_with_header_key(client, monkeypatch):
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.setenv("SEMANTICA_API_KEY", "correct-key")
with client.websocket_connect(
"/ws/graph-updates", headers={"X-API-Key": "correct-key"}
) as websocket:
ack = websocket.receive_json()
assert ack["event"] == "connection_ack"
# ---------------------------------------------------------------------------
# WebSocket Origin validation (GHSA-4643-wpgq-w329): CORSMiddleware doesn't
# cover WebSocket handshakes at all, so under SEMANTICA_ALLOW_ANONYMOUS the
# key check alone accepted a handshake from any origin — loopback binding is
# not a boundary against a browser, since any page the operator has open can
# still reach ws://localhost:.../ws/graph-updates. These pin the fix: a
# hostile Origin is refused even in anonymous mode (and even with a correct
# key), a same-origin/allowlisted Origin still works, and a missing Origin
# (native/CLI clients, which never set the header) is still allowed through.
# ---------------------------------------------------------------------------
def test_websocket_rejects_hostile_origin_under_anonymous_mode(client, monkeypatch):
monkeypatch.setenv("SEMANTICA_ALLOW_ANONYMOUS", "true")
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
with pytest.raises(Exception):
with client.websocket_connect(
"/ws/graph-updates", headers={"Origin": "https://evil.example"}
):
pass
def test_websocket_rejects_hostile_origin_even_with_correct_key(client, monkeypatch):
"""Defense in depth: Origin is checked before the API key, so a hostile
page that somehow obtained a valid key still can't hijack the socket."""
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
monkeypatch.setenv("SEMANTICA_API_KEY", "correct-key")
with pytest.raises(Exception):
with client.websocket_connect(
"/ws/graph-updates",
headers={"Origin": "https://evil.example", "X-API-Key": "correct-key"},
):
pass
def test_websocket_accepts_allowlisted_origin_under_anonymous_mode(client, monkeypatch):
monkeypatch.setenv("SEMANTICA_ALLOW_ANONYMOUS", "true")
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
with client.websocket_connect(
"/ws/graph-updates", headers={"Origin": "http://localhost:5173"}
) as websocket:
ack = websocket.receive_json()
assert ack["event"] == "connection_ack"
def test_websocket_accepts_missing_origin_under_anonymous_mode(client, monkeypatch):
"""Native/CLI clients never send an Origin header — only browsers do —
so a missing Origin must still be allowed through; the browser is the
only threat this check closes."""
monkeypatch.setenv("SEMANTICA_ALLOW_ANONYMOUS", "true")
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
with client.websocket_connect("/ws/graph-updates") as websocket:
ack = websocket.receive_json()
assert ack["event"] == "connection_ack"
+392
View File
@@ -0,0 +1,392 @@
"""Regression tests for DNS check-then-use (TOCTOU) hardening in the
ontology URL fetcher (GHSA-8c7v-62gr-hj6g's secondary "smaller" gap).
`_validate_fetch_url` resolves and validates a hostname once; if the actual
HTTP client resolved it again independently at connect time, a low-TTL or
rebinding DNS answer could differ between the two lookups, reopening the
SSRF window the validation exists to close. `_make_pinned_session` closes
this by pinning the connection pool's `host` directly to the already-
validated IP (bypassing DNS resolution for the connection entirely), while
explicitly restoring the real hostname as the outgoing HTTP `Host` header
and, for HTTPS, the TLS SNI `server_hostname` / `assert_hostname` so the
connection reaches the pinned IP but still presents (and verifies against)
the original hostname's identity.
test_ontology_ssrf.py covers the redirect-handling logic around this with
mocks; this file proves the pinning mechanism itself works end-to-end
against real local servers, with no DNS mocking at all the test hostname
is never resolved, which is exactly the property being verified. It also
includes a negative control (mismatched cert hostname) proving TLS
verification is genuinely enforced against the real hostname, not silently
bypassed or checked against the pinned IP instead.
"""
import http.server
import socket
import threading
import pytest
from semantica.explorer.routes import ontology as ontology_mod
def _start_local_server():
captured = {}
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
captured["host_header"] = self.headers.get("Host")
body = b"pinned response"
self.send_response(200)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *_args):
pass
server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
return server, thread, captured
def test_pinned_session_connects_to_pinned_ip_without_resolving_hostname():
"""A session built by _make_pinned_session must reach the pinned IP
directly. The request URL uses a hostname that cannot be resolved via
real DNS ('.invalid' is reserved by RFC 2606) if pinning weren't
working, this request would fail with a name-resolution error instead
of reaching the local server, since nothing else could route it there.
"""
server, thread, captured = _start_local_server()
port = server.server_address[1]
url = f"http://pinned-test.invalid:{port}/resource"
try:
session = ontology_mod._make_pinned_session(["127.0.0.1"], url)
try:
resp = session.get(url, timeout=5)
assert resp.status_code == 200
assert resp.content == b"pinned response"
finally:
session.close()
finally:
server.shutdown()
thread.join(timeout=2)
# Host header must still be the original hostname, not the pinned IP —
# proving connection target and presented identity are decoupled
# correctly (this is what keeps virtual hosting / TLS SNI correct).
assert captured["host_header"] == f"pinned-test.invalid:{port}"
def test_pinned_session_ignores_a_different_real_resolution():
"""Even if the hostname *does* resolve to something else via real DNS,
the pinned session must still go to the pinned IP this is the actual
TOCTOU property: the connection uses what was validated, not whatever
a fresh lookup returns. 'localhost' reliably resolves to a loopback
address, which is deliberately NOT where our test server listens on
(127.0.0.1 specifically) but since Windows/most stacks map
'localhost' to 127.0.0.1 too, use a distinct high loopback address
(127.0.0.2) for the server so a real 'localhost' resolution (127.0.0.1)
provably would NOT reach it, isolating the assertion to pinning alone.
"""
captured = {}
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
captured["host_header"] = self.headers.get("Host")
body = b"pinned via explicit ip"
self.send_response(200)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *_args):
pass
try:
server = http.server.HTTPServer(("127.0.0.2", 0), Handler)
except OSError:
# 127.0.0.2 isn't bindable in this environment (uncommon, but
# possible in some sandboxes) — skip rather than false-fail.
import pytest
pytest.skip("127.0.0.2 is not bindable in this environment")
port = server.server_address[1]
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
url = f"http://localhost:{port}/resource"
try:
session = ontology_mod._make_pinned_session(["127.0.0.2"], url)
try:
resp = session.get(url, timeout=5)
assert resp.status_code == 200
assert resp.content == b"pinned via explicit ip"
finally:
session.close()
finally:
server.shutdown()
thread.join(timeout=2)
assert captured["host_header"] == f"localhost:{port}"
def test_pinned_session_falls_back_across_multiple_pinned_ips():
"""A hostname can have multiple A/AAAA records; pinning to only the
first-returned address means a fetch fails outright if that specific
address happens to be unreachable even though a later one would work.
_make_pinned_session must fall back through every pinned IP in order.
"""
server, thread, captured = _start_local_server()
port = server.server_address[1]
url = f"http://pinned-test.invalid:{port}/resource"
# 127.0.0.3 has nothing listening on this port — connection refused,
# forcing a fallback to the second (real) address.
unreachable_ip = "127.0.0.3"
try:
session = ontology_mod._make_pinned_session([unreachable_ip, "127.0.0.1"], url)
try:
resp = session.get(url, timeout=5)
assert resp.status_code == 200
assert resp.content == b"pinned response"
finally:
session.close()
finally:
server.shutdown()
thread.join(timeout=2)
def test_pinned_session_raises_when_every_pinned_ip_is_unreachable():
"""If none of the pinned IPs are reachable, the session must raise
rather than silently falling back to resolving the hostname itself
(which would reopen the exact TOCTOU window pinning exists to close)."""
import requests
url = "http://pinned-test.invalid:9/resource" # port 9 (discard) — nothing listens
session = ontology_mod._make_pinned_session(["127.0.0.3", "127.0.0.4"], url)
try:
with pytest.raises(requests.exceptions.ConnectionError):
session.get(url, timeout=5)
finally:
session.close()
def test_pinned_session_disables_environment_proxy_trust():
"""A pinned session must never honor HTTP_PROXY/HTTPS_PROXY env vars —
a proxy would perform its own DNS resolution of the target host outside
this process's control, reopening the exact TOCTOU window pinning
exists to close."""
session = ontology_mod._make_pinned_session(["127.0.0.1"], "http://example.org/")
try:
assert session.trust_env is False
finally:
session.close()
def test_pinned_session_ignores_env_proxy_and_connects_directly(monkeypatch):
"""End-to-end: even with HTTP_PROXY pointed at an address that would
fail if contacted, a pinned session must reach the real local server
directly proving the env var is genuinely not consulted, not just
that the trust_env flag is set."""
monkeypatch.setenv("HTTP_PROXY", "http://127.0.0.5:1/") # would fail if ever used
server, thread, _captured = _start_local_server()
port = server.server_address[1]
url = f"http://pinned-test.invalid:{port}/resource"
try:
session = ontology_mod._make_pinned_session(["127.0.0.1"], url)
try:
resp = session.get(url, timeout=5)
assert resp.status_code == 200
assert resp.content == b"pinned response"
finally:
session.close()
finally:
server.shutdown()
thread.join(timeout=2)
def test_pinned_session_fails_closed_if_a_proxy_is_explicitly_forced():
"""Backstop: if a proxy is somehow still configured on the session
despite trust_env=False (e.g. set explicitly, as a future code path
might), the adapter must fail closed with a clear error rather than
silently connecting through the proxy unpinned."""
from fastapi import HTTPException
session = ontology_mod._make_pinned_session(["127.0.0.1"], "http://example.org/")
session.proxies = {"http": "http://127.0.0.5:1"}
try:
with pytest.raises(HTTPException) as exc_info:
session.get("http://example.org/", timeout=5)
assert exc_info.value.status_code == 502
finally:
session.close()
def test_validate_fetch_url_returns_the_resolved_ip():
"""_validate_fetch_url must return every IP it validated, so callers can
pin the connection to them (with fallback across all of them)."""
def fake_getaddrinfo(host, *_a, **_k):
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0))]
import unittest.mock as mock
with mock.patch.object(ontology_mod.socket, "getaddrinfo", side_effect=fake_getaddrinfo):
resolved_ips = ontology_mod._validate_fetch_url("http://example.org/ontology.ttl")
assert resolved_ips == ["93.184.216.34"]
def test_validate_fetch_url_returns_all_validated_ips_deduplicated():
"""A hostname with multiple A/AAAA records must return every distinct
validated address, in resolution order, so the caller can fall back
across all of them rather than failing if only the first is
unreachable."""
def fake_getaddrinfo(host, *_a, **_k):
return [
(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0)),
(socket.AF_INET, socket.SOCK_DGRAM, 17, "", ("93.184.216.34", 0)), # duplicate, different socktype
(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.35", 0)),
]
import unittest.mock as mock
with mock.patch.object(ontology_mod.socket, "getaddrinfo", side_effect=fake_getaddrinfo):
resolved_ips = ontology_mod._validate_fetch_url("http://example.org/ontology.ttl")
assert resolved_ips == ["93.184.216.34", "93.184.216.35"]
def test_validate_fetch_url_still_rejects_private_ip():
"""Confirm the pinning refactor didn't loosen the original address
classification a hostname resolving to a private/internal address
must still be rejected before any IP is returned."""
import ipaddress
import unittest.mock as mock
import pytest
from fastapi import HTTPException
def fake_getaddrinfo(host, *_a, **_k):
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("169.254.169.254", 0))]
with mock.patch.object(ontology_mod.socket, "getaddrinfo", side_effect=fake_getaddrinfo):
with pytest.raises(HTTPException) as exc_info:
ontology_mod._validate_fetch_url("http://attacker.example/ontology.ttl")
assert exc_info.value.status_code == 422
# ---------------------------------------------------------------------------
# HTTPS: SNI + certificate hostname verification must use the real hostname,
# not the pinned IP — this is the highest-risk part of pinning to get wrong,
# since a mistake here could silently weaken TLS verification rather than
# just breaking connectivity. Requires the optional `cryptography` package
# to mint a throwaway self-signed cert; skipped gracefully without it.
# ---------------------------------------------------------------------------
def _make_self_signed_cert(hostname: str, tmp_path):
import datetime
pytest.importorskip("cryptography")
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.x509.oid import NameOID
key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, hostname)])
now = datetime.datetime.now(datetime.timezone.utc)
cert = (
x509.CertificateBuilder()
.subject_name(name)
.issuer_name(name)
.public_key(key.public_key())
.serial_number(x509.random_serial_number())
.not_valid_before(now - datetime.timedelta(days=1))
.not_valid_after(now + datetime.timedelta(days=1))
.add_extension(x509.SubjectAlternativeName([x509.DNSName(hostname)]), critical=False)
.sign(key, hashes.SHA256())
)
cert_path = tmp_path / "cert.pem"
key_path = tmp_path / "key.pem"
cert_path.write_bytes(cert.public_bytes(serialization.Encoding.PEM))
key_path.write_bytes(
key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.NoEncryption(),
)
)
return str(cert_path), str(key_path)
def _start_local_https_server(cert_path, key_path):
import ssl
captured = {}
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
captured["host_header"] = self.headers.get("Host")
body = b"tls pinned response"
self.send_response(200)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *_args):
pass
server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
ssl_ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_ctx.minimum_version = ssl.TLSVersion.TLSv1_2
ssl_ctx.load_cert_chain(cert_path, key_path)
server.socket = ssl_ctx.wrap_socket(server.socket, server_side=True)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
return server, thread, captured
def test_pinned_https_session_verifies_against_real_hostname_not_pinned_ip(tmp_path):
"""A pinned HTTPS connection must present + verify SNI/cert against the
real hostname, even though the socket connects to the pinned IP. The
cert's SAN is the hostname, never '127.0.0.1' — if pinning verified
against the IP instead (or against nothing), this would either fail
for the wrong reason or silently succeed with no real verification."""
cert_path, key_path = _make_self_signed_cert("pinned-tls-test.invalid", tmp_path)
server, thread, captured = _start_local_https_server(cert_path, key_path)
port = server.server_address[1]
url = f"https://pinned-tls-test.invalid:{port}/resource"
try:
session = ontology_mod._make_pinned_session(["127.0.0.1"], url)
try:
resp = session.get(url, timeout=5, verify=cert_path)
finally:
session.close()
finally:
server.shutdown()
thread.join(timeout=2)
assert resp.status_code == 200
assert resp.content == b"tls pinned response"
assert captured["host_header"] == f"pinned-tls-test.invalid:{port}"
def test_pinned_https_session_rejects_hostname_mismatch(tmp_path):
"""Negative control: requesting a hostname that does NOT match the
cert's SAN must still fail verification — proving pinning doesn't
silently bypass or misdirect certificate hostname checking."""
cert_path, key_path = _make_self_signed_cert("pinned-tls-test.invalid", tmp_path)
server, thread, _captured = _start_local_https_server(cert_path, key_path)
port = server.server_address[1]
url = f"https://wrong-name.invalid:{port}/resource"
try:
session = ontology_mod._make_pinned_session(["127.0.0.1"], url)
try:
import requests
with pytest.raises(requests.exceptions.SSLError):
session.get(url, timeout=5, verify=cert_path)
finally:
session.close()
finally:
server.shutdown()
thread.join(timeout=2)
+113
View File
@@ -0,0 +1,113 @@
"""Regression tests for outbound URL fetching in ontology.py (SSRF hardening).
`_fetch_url_sync` disables `requests`' automatic redirect following and
re-validates every hop with `_validate_fetch_url` (see GHSA-8c7v-62gr-hj6g:
unvalidated redirect targets previously let a public first hop 302 the
server into fetching cloud metadata / loopback services). It also pins each
hop's connection to the IP `_validate_fetch_url` already resolved and
validated, via `_make_pinned_session`, closing the DNS check-then-use gap
between that validation and the client's own (potentially different) lookup.
These tests cover the redirect-handling logic itself: relative `Location`
headers must resolve correctly instead of being rejected outright, redirect
targets that resolve to private/loopback addresses must still be blocked,
and every response must be closed (no leaked connections across hops).
`test_ontology_dns_pinning.py` covers the pinning mechanism
(`_make_pinned_session`, `_validate_fetch_url`'s returned IP) directly.
"""
import socket
from unittest.mock import MagicMock, patch
import pytest
from semantica.explorer.routes import ontology as ontology_mod
def _fake_getaddrinfo(host, *args, **kwargs):
# These tests are about the redirect-handling logic, not the address
# classifier itself, so every host resolves to a public IP unless a
# test overrides the side_effect to simulate an internal target.
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0))]
def _make_response(is_redirect=False, is_permanent=False, location=None, body=b"ok"):
resp = MagicMock()
resp.is_redirect = is_redirect
resp.is_permanent_redirect = is_permanent
resp.headers = {"Location": location} if location else {}
resp.raise_for_status = MagicMock()
resp.iter_content = MagicMock(return_value=iter([body]))
resp.close = MagicMock()
return resp
def _patch_session(responses):
"""Patch _make_pinned_session so _fetch_url_sync's session.get(...)
calls return the given responses in order, without touching the real
requests.Session/pinning machinery (that's covered by test_pinning.py).
"""
fake_session = MagicMock()
fake_session.get = MagicMock(side_effect=responses)
fake_session.close = MagicMock()
return patch.object(ontology_mod, "_make_pinned_session", return_value=fake_session), fake_session
@patch.object(ontology_mod.socket, "getaddrinfo", side_effect=_fake_getaddrinfo)
def test_relative_redirect_location_is_resolved(mock_getaddrinfo):
"""A relative Location header (e.g. '/ontology.ttl') must resolve against
the current URL via urljoin, not be rejected as a malformed URL."""
redirect_resp = _make_response(is_redirect=True, location="/ontology.ttl")
final_resp = _make_response(body=b"final content")
patcher, fake_session = _patch_session([redirect_resp, final_resp])
with patcher:
result = ontology_mod._fetch_url_sync("http://example.org/start")
assert result == b"final content"
second_call_url = fake_session.get.call_args_list[1].args[0]
assert second_call_url == "http://example.org/ontology.ttl"
redirect_resp.close.assert_called_once()
final_resp.close.assert_called_once()
@patch.object(ontology_mod.socket, "getaddrinfo", side_effect=_fake_getaddrinfo)
def test_redirect_to_private_ip_is_rejected(mock_getaddrinfo):
"""Re-validation must reject a redirect target resolving to a private
address even though the first hop was a validated public URL this is
the exact GHSA-8c7v scenario: public first hop, malicious redirect."""
def getaddrinfo_side_effect(host, *a, **k):
if host == "internal.example":
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("169.254.169.254", 0))]
return _fake_getaddrinfo(host, *a, **k)
mock_getaddrinfo.side_effect = getaddrinfo_side_effect
redirect_resp = _make_response(is_redirect=True, location="http://internal.example/latest/meta-data/")
patcher, fake_session = _patch_session([redirect_resp])
with patcher:
with pytest.raises(ontology_mod.HTTPException) as exc_info:
ontology_mod._fetch_url_sync("http://example.org/start")
assert exc_info.value.status_code == 422
redirect_resp.close.assert_called_once()
@patch.object(ontology_mod.socket, "getaddrinfo", side_effect=_fake_getaddrinfo)
def test_final_response_is_closed(mock_getaddrinfo):
final_resp = _make_response(body=b"content")
patcher, _fake_session = _patch_session([final_resp])
with patcher:
ontology_mod._fetch_url_sync("http://example.org/start")
final_resp.close.assert_called_once()
@patch.object(ontology_mod.socket, "getaddrinfo", side_effect=_fake_getaddrinfo)
def test_redirect_chain_exceeding_cap_is_rejected(mock_getaddrinfo):
responses = [_make_response(is_redirect=True, location=f"/hop{i}") for i in range(10)]
patcher, _fake_session = _patch_session(responses)
with patcher:
with pytest.raises(ontology_mod.HTTPException) as exc_info:
ontology_mod._fetch_url_sync("http://example.org/start")
assert exc_info.value.status_code == 502
assert all(r.close.called for r in responses[:6])
+73 -5
View File
@@ -199,14 +199,27 @@ def test_allowlist_rejected_query_never_touches_the_graph(client, query):
mock_build.assert_not_called() mock_build.assert_not_called()
def test_multi_statement_injection_reaches_graph_but_fails_in_parser(client): def test_multi_statement_injection_is_rejected_by_forbidden_keyword_check(client):
"""Confirms the distinction between allowlist rejection and parser rejection: """A string starting with an allowed keyword (SELECT) but containing a
a string starting with SELECT passes _is_read_only_query and builds a graph, forbidden Update keyword later in the body ('; DROP ALL') is now
but rdflib.Graph.query() rejects the trailing '; DROP ALL' syntax.""" rejected by _is_read_only_query's keyword scan itself, before a graph
is ever built a stronger, earlier rejection than relying solely on
rdflib's parser to reject the syntax."""
with patch.object(sparql_mod, "_build_rdflib_graph") as mock_build:
resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o } ; DROP ALL")
assert resp.status_code == 200
assert resp.json()["error"] is not None
mock_build.assert_not_called()
def test_malformed_syntax_without_forbidden_keywords_still_fails_in_parser(client):
"""The parser remains a real second line of defense for malformed
queries that don't contain any forbidden keyword — these pass
_is_read_only_query and reach rdflib, which rejects the syntax."""
with patch.object( with patch.object(
sparql_mod, "_build_rdflib_graph", wraps=sparql_mod._build_rdflib_graph sparql_mod, "_build_rdflib_graph", wraps=sparql_mod._build_rdflib_graph
) as spy_build: ) as spy_build:
resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o } ; DROP ALL") resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o } ; ASK { ?x ?y ?z }")
assert resp.status_code == 200 assert resp.status_code == 200
assert resp.json()["error"] is not None assert resp.json()["error"] is not None
spy_build.assert_called_once() spy_build.assert_called_once()
@@ -290,6 +303,61 @@ def test_query_timeout_returns_clean_error_not_a_crash(client):
assert payload["rows"] == [] assert payload["rows"] == []
def test_oversized_graph_returns_clean_error_not_a_crash(client):
"""The DoS-prevention node cap (GHSA-8c7v-adjacent hardening) must return
a normal SparqlResponse error, not an unhandled 500. Regression test for
a bug where _build_rdflib_graph's ValueError was raised outside of
execute_sparql's try/except, before the semaphore block."""
with patch.object(sparql_mod, "_SPARQL_MAX_GRAPH_NODES", 1):
resp = _post(client, "SELECT ?s WHERE { ?s a <http://semantica.local/entity/language> }")
assert resp.status_code == 200
payload = resp.json()
assert payload["error"] is not None
assert "more than" in payload["error"].lower()
assert payload["rows"] == []
def test_oversized_query_returns_distinct_length_error(client):
"""A query exceeding _SPARQL_MAX_QUERY_LEN must be rejected with a
specific, actionable error message not the generic read-only message.
Clients need to distinguish a size-limit rejection from an actual
non-read-only query rejection to react correctly (e.g. split the query
vs. rewrite it)."""
with patch.object(sparql_mod, "_SPARQL_MAX_QUERY_LEN", 10):
resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o }") # 30 chars > 10
assert resp.status_code == 200
payload = resp.json()
assert payload["error"] is not None
# Must mention the limit, not the generic read-only message
assert "length" in payload["error"].lower() or "characters" in payload["error"].lower()
assert "Only SELECT" not in payload["error"]
assert payload["rows"] == []
assert payload["columns"] == []
assert payload["total"] == 0
def test_oversized_query_never_touches_the_graph(client):
"""An oversized query must be rejected before _build_rdflib_graph is
called the length guard must short-circuit the entire pipeline."""
with patch.object(sparql_mod, "_SPARQL_MAX_QUERY_LEN", 10):
with patch.object(sparql_mod, "_build_rdflib_graph") as mock_build:
resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o }")
assert resp.status_code == 200
assert resp.json()["error"] is not None
mock_build.assert_not_called()
def test_query_exactly_at_length_limit_is_accepted(client):
"""A query whose length equals the limit exactly must not be rejected —
the guard is strictly greater-than, not greater-than-or-equal."""
short_query = "ASK {}"
with patch.object(sparql_mod, "_SPARQL_MAX_QUERY_LEN", len(short_query)):
resp = _post(client, short_query)
assert resp.status_code == 200
payload = resp.json()
assert payload["error"] is None
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Data-mapping fidelity: does the graph->RDF projection reflect session state? # Data-mapping fidelity: does the graph->RDF projection reflect session state?
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+396
View File
@@ -0,0 +1,396 @@
"""Regression tests for GHSA-482h-hw99-h62p: unvalidated node labels and
property keys allowed arbitrary Cypher injection in the Neptune, Neo4j, and
FalkorDB graph stores (labels/keys can't be bound as query parameters, so
an unvalidated value reaching the query string is a direct injection
point).
Mirrors the advisory's own PoC shape: a label/key crafted to close the
current Cypher token early and append a destructive statement
(`DETACH DELETE victim`). Before the fix, these reached `_run_query` /
`session.run` / `graph.query` verbatim. After the fix, `sanitize_identifier`
(graph_store/query_sanitize.py) rejects them with ValidationError before
any query is built, matching the existing age_store.py `_sanitize_label`
behavior used as the reference implementation.
"""
import pytest
import unittest
from unittest.mock import MagicMock
from semantica.graph_store.query_sanitize import sanitize_identifier
from semantica.utils.exceptions import ProcessingError, ValidationError
# AmazonNeptuneStore/Neo4jStore/FalkorDBStore.create_node() wrap their whole
# body in `except Exception: raise ProcessingError(...)` (pre-existing,
# unrelated to this fix), so the ValidationError sanitize_identifier raises
# surfaces to callers as ProcessingError. Either way the malicious query is
# never built or sent — these tests assert exactly that via the query-capture
# stubs, and check the wrapped message to confirm it's the sanitizer firing.
EVIL_LABEL = "N}) MATCH (victim) DETACH DELETE victim //"
EVIL_KEY = "k1`: 1}) MATCH (victim) DETACH DELETE victim //"
def _wire(store):
store.logger = MagicMock()
store.progress_tracker = MagicMock()
store.progress_tracker.start_tracking.return_value = "tid"
store.config = {}
return store
class TestSanitizeIdentifier(unittest.TestCase):
def test_valid_identifiers_pass_through_unchanged(self):
self.assertEqual(sanitize_identifier("Person"), "Person")
self.assertEqual(sanitize_identifier("_hidden"), "_hidden")
self.assertEqual(sanitize_identifier("Rel_Type2"), "Rel_Type2")
def test_injection_payload_is_rejected(self):
with self.assertRaises(ValidationError):
sanitize_identifier(EVIL_LABEL)
def test_property_key_injection_payload_is_rejected(self):
with self.assertRaises(ValidationError):
sanitize_identifier(EVIL_KEY)
def test_rejects_non_string(self):
with self.assertRaises(ValidationError):
sanitize_identifier(123) # type: ignore[arg-type]
def test_rejects_spaces_and_dashes(self):
with self.assertRaises(ValidationError):
sanitize_identifier("no spaces")
with self.assertRaises(ValidationError):
sanitize_identifier("no-dashes")
class TestAmazonNeptuneCypherInjection(unittest.TestCase):
def _make_store(self):
from semantica.graph_store.amazon_neptune import AmazonNeptuneStore
store = _wire(AmazonNeptuneStore.__new__(AmazonNeptuneStore))
store._connected = True
store._ensure_connected = lambda: None
store._generate_id = lambda: "generated-id"
store._run_query = MagicMock(return_value=[])
store._parse_results = lambda r: []
return store
def test_create_node_rejects_malicious_label_before_querying(self):
store = self._make_store()
with self.assertRaises(ProcessingError) as ctx:
store.create_node(labels=[EVIL_LABEL], properties={"name": "x"})
self.assertIn("Invalid label", str(ctx.exception))
store._run_query.assert_not_called()
def test_create_node_rejects_malicious_property_key_before_querying(self):
store = self._make_store()
with self.assertRaises(ProcessingError) as ctx:
store.create_node(labels=["Person"], properties={"name": "x", EVIL_KEY: 1})
self.assertIn("Invalid property key", str(ctx.exception))
store._run_query.assert_not_called()
def test_create_node_with_legitimate_labels_still_works(self):
store = self._make_store()
store.create_node(labels=["Person", "Employee"], properties={"name": "Alice"})
query = store._run_query.call_args[0][0]
self.assertIn("Person:Employee", query)
self.assertNotIn("DETACH DELETE", query)
class TestNeo4jCypherInjection(unittest.TestCase):
def _make_store(self):
from semantica.graph_store import neo4j_store as m
store = _wire(m.Neo4jStore.__new__(m.Neo4jStore))
captured = {}
class Session:
def run(self, q, params=None):
captured["query"] = q
rec = {"n": {"name": "x"}, "id": 1}
return type("R", (), {"single": lambda self: rec})()
def __enter__(self):
return self
def __exit__(self, *a):
return False
store.get_session = lambda: Session()
store._captured = captured
return store
def test_create_node_rejects_malicious_label_before_querying(self):
store = self._make_store()
with self.assertRaises(ProcessingError) as ctx:
store.create_node(labels=["Person", EVIL_LABEL], properties={"name": "x"})
self.assertIn("Invalid label", str(ctx.exception))
self.assertNotIn("query", store._captured)
def test_create_relationship_rejects_malicious_rel_type(self):
store = self._make_store()
with self.assertRaises(ProcessingError) as ctx:
store.create_relationship(start_node_id=1, end_node_id=2, rel_type=EVIL_LABEL)
self.assertIn("Invalid relationship type", str(ctx.exception))
self.assertNotIn("query", store._captured)
class TestFalkorDBCypherInjection(unittest.TestCase):
def _make_store(self):
from semantica.graph_store import falkordb_store as m
store = _wire(m.FalkorDBStore.__new__(m.FalkorDBStore))
captured = {}
class Graph:
def query(self, q, params=None):
captured["query"] = q
return type("R", (), {"result_set": []})()
store._ensure_graph = lambda: Graph()
store._captured = captured
return store
def test_create_node_rejects_malicious_label_and_key(self):
store = self._make_store()
with self.assertRaises(ProcessingError) as ctx:
store.create_node(labels=[EVIL_LABEL], properties={"name": "x", EVIL_KEY: 1})
self.assertIn("Invalid label", str(ctx.exception))
self.assertNotIn("query", store._captured)
def test_create_node_with_legitimate_input_still_works(self):
store = self._make_store()
store.create_node(labels=["Person"], properties={"name": "Alice"})
query = store._captured["query"]
self.assertIn("Person", query)
self.assertNotIn("DETACH DELETE", query)
# ---------------------------------------------------------------------------
# BLOCKER 1 regression: Neo4jStore.get_neighbors / shortest_path depth coercion
# ---------------------------------------------------------------------------
# Payload representative of the confirmed injection (review BLOCKER 1):
# supplying a string as `depth` / `max_depth` previously reached the Cypher
# f-string verbatim because Neo4jStore did not call int() like Neptune/FalkorDB.
#
# After the fix `depth = int(depth)` / `max_depth = int(max_depth)` are added
# at the top of each method's try-block. A malicious string raises ValueError
# (wrapped in ProcessingError), and the session.run / _run_query mock must
# never be called.
EVIL_DEPTH = "1]->(x) DETACH DELETE x //"
class TestNeo4jDepthInjection(unittest.TestCase):
"""Regression tests for BLOCKER 1: Neo4jStore depth/max_depth coercion."""
def _make_store(self):
"""Build a Neo4jStore with a session that records every query string sent to it."""
from semantica.graph_store import neo4j_store as m
store = _wire(m.Neo4jStore.__new__(m.Neo4jStore))
captured = {}
class IterSession:
"""Returns an empty iterator so get_neighbors' for-loop completes cleanly."""
def run(self, q, params=None):
captured["query"] = q
return iter([])
def __enter__(self):
return self
def __exit__(self, *a):
return False
class SingleSession:
"""Returns a FakeResult whose .single() yields None (shortest_path)."""
def run(self, q, params=None):
captured["query"] = q
return type("R", (), {"single": lambda self: None})()
def __enter__(self):
return self
def __exit__(self, *a):
return False
store._captured = captured
store._iter_session = IterSession
store._single_session = SingleSession
return store
# -- get_neighbors --------------------------------------------------------
def test_get_neighbors_malicious_depth_raises_before_query(self):
"""Malicious string depth must never reach session.run."""
store = self._make_store()
store.get_session = lambda: store._iter_session()
with self.assertRaises(ProcessingError):
store.get_neighbors(node_id=1, depth=EVIL_DEPTH)
self.assertNotIn("query", store._captured,
"session.run was called — injected query reached the database layer")
def test_get_neighbors_malicious_depth_does_not_contain_payload(self):
"""Double-check: if somehow a query were built, it must not contain the payload."""
store = self._make_store()
store.get_session = lambda: store._iter_session()
with pytest.raises(ProcessingError):
store.get_neighbors(node_id=1, depth=EVIL_DEPTH)
query = store._captured.get("query", "")
self.assertNotIn("DETACH DELETE", query,
f"Injection payload found in query: {query!r}")
def test_get_neighbors_legitimate_depth_works(self):
"""Valid integer depth must still produce a correct traversal pattern."""
store = self._make_store()
store.get_session = lambda: store._iter_session()
result = store.get_neighbors(node_id=1, depth=2)
self.assertIsInstance(result, list)
self.assertIn("query", store._captured)
self.assertIn("*1..2", store._captured["query"])
self.assertNotIn("DETACH DELETE", store._captured["query"])
def test_get_neighbors_depth_string_int_is_coerced(self):
"""A string representation of a valid integer must be coerced and work."""
store = self._make_store()
store.get_session = lambda: store._iter_session()
result = store.get_neighbors(node_id=1, depth="3")
self.assertIsInstance(result, list)
self.assertIn("*1..3", store._captured["query"])
# -- shortest_path --------------------------------------------------------
def test_shortest_path_malicious_max_depth_raises_before_query(self):
"""Malicious string max_depth must never reach session.run."""
store = self._make_store()
store.get_session = lambda: store._single_session()
with self.assertRaises(ProcessingError):
store.shortest_path(start_node_id=1, end_node_id=2, max_depth=EVIL_DEPTH)
self.assertNotIn("query", store._captured,
"session.run was called — injected query reached the database layer")
def test_shortest_path_malicious_max_depth_does_not_contain_payload(self):
store = self._make_store()
store.get_session = lambda: store._single_session()
with pytest.raises(ProcessingError):
store.shortest_path(start_node_id=1, end_node_id=2, max_depth=EVIL_DEPTH)
query = store._captured.get("query", "")
self.assertNotIn("DETACH DELETE", query,
f"Injection payload found in query: {query!r}")
def test_shortest_path_legitimate_max_depth_works(self):
"""Valid integer max_depth must produce a correct shortestPath pattern."""
store = self._make_store()
store.get_session = lambda: store._single_session()
result = store.shortest_path(start_node_id=1, end_node_id=2, max_depth=5)
self.assertIsNone(result) # single() returns None → correct
self.assertIn("query", store._captured)
self.assertIn("*..5", store._captured["query"])
self.assertNotIn("DETACH DELETE", store._captured["query"])
def test_shortest_path_max_depth_string_int_is_coerced(self):
store = self._make_store()
store.get_session = lambda: store._single_session()
store.shortest_path(start_node_id=1, end_node_id=2, max_depth="7")
self.assertIn("*..7", store._captured["query"])
# ---------------------------------------------------------------------------
# BLOCKER 2 regression: GraphStore.get_neighbors hops forwarding
# ---------------------------------------------------------------------------
# Before the fix, GraphStore.get_neighbors() set:
# actual_depth = options.get("hops", depth)
# and forwarded the raw value to Neo4jStore.get_neighbors(depth=actual_depth).
# Because Neo4jStore did not coerce depth, an attacker-controlled hops string
# reached Cypher verbatim.
#
# The fix adds int() at the GraphStore facade:
# actual_depth = int(options.get("hops", depth))
# This closes the path regardless of which backend is wired up.
class TestGraphStoreHopsForwarding(unittest.TestCase):
"""Regression tests for BLOCKER 2: GraphStore hops→depth forwarding."""
def _make_graph_store_with_neo4j(self):
"""
Wire a GraphStore whose backend is a Neo4j store stub that records every
query passed to session.run. Returns (graph_store, captured_dict).
"""
from semantica.graph_store import neo4j_store as m
from semantica.graph_store.graph_store import (
GraphAnalytics,
GraphManager,
GraphStore,
)
captured = {}
class IterSession:
def run(self, q, params=None):
captured["query"] = q
return iter([])
def __enter__(self):
return self
def __exit__(self, *a):
return False
neo4j_stub = _wire(m.Neo4jStore.__new__(m.Neo4jStore))
neo4j_stub.get_session = lambda: IterSession()
gs = GraphStore.__new__(GraphStore)
gs.logger = MagicMock()
gs.progress_tracker = MagicMock()
gs._store_backend = neo4j_stub
gs._manager = GraphManager(neo4j_stub)
return gs, captured
# -- hops injection -------------------------------------------------------
def test_hops_malicious_string_raises_before_query(self):
"""Malicious hops string must be rejected before session.run is reached."""
gs, captured = self._make_graph_store_with_neo4j()
with self.assertRaises(Exception):
gs.get_neighbors(node_id=1, hops=EVIL_DEPTH)
self.assertNotIn("query", captured,
"session.run was called — injected hops reached the database layer")
def test_hops_malicious_string_payload_not_in_any_query(self):
"""Belt-and-suspenders: payload text must not appear in any built query."""
gs, captured = self._make_graph_store_with_neo4j()
with pytest.raises(ValueError):
gs.get_neighbors(node_id=1, hops=EVIL_DEPTH)
query = captured.get("query", "")
self.assertNotIn("DETACH DELETE", query,
f"Injection payload found in forwarded query: {query!r}")
def test_hops_legitimate_integer_works(self):
"""Valid integer hops value must produce a correct query."""
gs, captured = self._make_graph_store_with_neo4j()
result = gs.get_neighbors(node_id=1, hops=2)
self.assertIsInstance(result, list)
self.assertIn("query", captured)
self.assertIn("*1..2", captured["query"])
self.assertNotIn("DETACH DELETE", captured["query"])
def test_hops_string_int_is_coerced_and_works(self):
"""String '3' forwarded as hops must be coerced to int and produce *1..3."""
gs, captured = self._make_graph_store_with_neo4j()
result = gs.get_neighbors(node_id=1, hops="3")
self.assertIsInstance(result, list)
self.assertIn("*1..3", captured["query"])
def test_depth_param_still_works_without_hops(self):
"""depth positional arg (no hops kwarg) must still be coerced and forwarded."""
gs, captured = self._make_graph_store_with_neo4j()
result = gs.get_neighbors(node_id=1, depth=4)
self.assertIsInstance(result, list)
self.assertIn("*1..4", captured["query"])
if __name__ == "__main__":
unittest.main()
+407
View File
@@ -0,0 +1,407 @@
"""SSRF protection tests for web and API ingestors (issue #867)."""
from unittest.mock import MagicMock, patch
import pytest
import urllib3.connectionpool as pool
from semantica.ingest.api_ingestor import RESTIngestor
from semantica.ingest.ssrf import (
parse_bool,
request_with_ssrf_guard,
validate_url_for_request,
)
from semantica.ingest.web_ingestor import SitemapCrawler, WebIngestor
from semantica.utils.exceptions import ValidationError
class TestParseBool:
def test_bool_passthrough(self):
assert parse_bool(True) is True
assert parse_bool(False) is False
def test_none_uses_default(self):
assert parse_bool(None) is False
assert parse_bool(None, default=True) is True
def test_string_true_values(self):
for value in ("true", "TRUE", "1", "yes", "on", " Yes "):
assert parse_bool(value) is True
def test_string_false_values(self):
for value in ("false", "FALSE", "0", "no", "off", " No "):
assert parse_bool(value) is False
def test_int_zero_one(self):
assert parse_bool(0) is False
assert parse_bool(1) is True
def test_rejects_unknown_string(self):
with pytest.raises(ValidationError, match="Invalid boolean"):
parse_bool("maybe")
class TestValidateUrlForRequest:
def test_accepts_https(self):
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
validate_url_for_request("https://example.com/path")
def test_rejects_file_scheme(self):
with pytest.raises(ValidationError, match="not permitted"):
validate_url_for_request("file://localhost/etc/passwd")
def test_rejects_gopher_scheme(self):
with pytest.raises(ValidationError, match="not permitted"):
validate_url_for_request("gopher://example.com/1")
def test_rejects_literal_private_ips(self):
for url in (
"http://10.0.0.1/",
"http://192.168.1.1/",
"http://172.16.5.5/internal",
"http://127.0.0.1:9999/internal",
"http://169.254.169.254/latest/meta-data/",
):
with pytest.raises(ValidationError, match="blocked"):
validate_url_for_request(url)
def test_rejects_localhost_hostname(self):
with pytest.raises(ValidationError, match="not allowed"):
validate_url_for_request("http://localhost/admin")
def test_rejects_hostname_resolving_to_private_ip(self):
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("10.0.0.5", 0))],
):
with pytest.raises(ValidationError, match="blocked"):
validate_url_for_request("http://internal.corp/secret")
def test_allow_private_ips_opt_in(self):
validate_url_for_request(
"http://127.0.0.1:8080/health", allow_private_ips=True
)
# Scheme allowlist still applies when private IPs are permitted
with pytest.raises(ValidationError, match="not permitted"):
validate_url_for_request(
"file://localhost/x", allow_private_ips=True
)
def test_dns_failure_raises(self):
import socket
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
side_effect=socket.gaierror("name or service not known"),
):
with pytest.raises(ValidationError, match="could not be resolved safely"):
validate_url_for_request("http://does-not-resolve.invalid/path")
def test_dns_timeout_raises(self):
import concurrent.futures
with patch(
"semantica.ingest.ssrf.concurrent.futures.Future.result",
side_effect=concurrent.futures.TimeoutError(),
):
with pytest.raises(ValidationError, match="could not be resolved safely"):
validate_url_for_request("http://slow-dns.example/path")
def test_hung_getaddrinfo_returns_within_timeout_bound(self):
"""Timeout must not wait on executor shutdown for a blocking resolver."""
import time
import semantica.ingest.ssrf as ssrf
hang_seconds = 1.5
bound = 0.1
def hanging_getaddrinfo(*_args, **_kwargs):
time.sleep(hang_seconds)
return [(None, None, None, None, ("93.184.216.34", 0))]
with patch.object(ssrf, "_DNS_RESOLVE_TIMEOUT_SECONDS", bound), patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
side_effect=hanging_getaddrinfo,
):
start = time.monotonic()
with pytest.raises(ValidationError, match="could not be resolved safely"):
ssrf._hostname_resolves_to_blocked("hanging.example")
elapsed = time.monotonic() - start
# Must fail closed near the configured bound, not after hang_seconds
# (which is what ``with ThreadPoolExecutor`` shutdown waiting causes).
assert elapsed < hang_seconds / 2
assert elapsed < bound + 0.75
class TestRequestWithSsrfGuardRedirects:
def test_blocks_redirect_to_loopback(self):
redirect = MagicMock()
redirect.status_code = 302
redirect.headers = {"Location": "http://127.0.0.1/secret"}
redirect.close = MagicMock()
session = MagicMock()
session.request.return_value = redirect
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
with pytest.raises(ValidationError, match="blocked"):
request_with_ssrf_guard(
"GET",
"https://example.com/start",
session=session,
)
session.request.assert_called_once()
assert session.request.call_args.kwargs.get("allow_redirects") is False
def test_blocks_redirect_to_metadata_ip(self):
redirect = MagicMock()
redirect.status_code = 301
redirect.headers = {"Location": "http://169.254.169.254/latest/meta-data/"}
redirect.close = MagicMock()
session = MagicMock()
session.request.return_value = redirect
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
with pytest.raises(ValidationError, match="blocked"):
request_with_ssrf_guard(
"GET",
"https://example.com/start",
session=session,
)
def test_follows_safe_redirect(self):
redirect = MagicMock()
redirect.status_code = 302
redirect.headers = {"Location": "https://example.com/final"}
redirect.close = MagicMock()
final = MagicMock()
final.status_code = 200
final.headers = {}
session = MagicMock()
session.request.side_effect = [redirect, final]
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
response = request_with_ssrf_guard(
"GET",
"https://example.com/start",
session=session,
)
assert response is final
assert session.request.call_count == 2
assert all(
call.kwargs.get("allow_redirects") is False
for call in session.request.call_args_list
)
class TestWebIngestorSSRF:
def test_private_ip_never_reaches_urllib3(self):
ingestor = WebIngestor(respect_robots=False, delay=0)
attempts = []
def intercepting_urlopen(self, method, url, **kw):
attempts.append((self.host, self.port, url))
raise Exception("intercepted at urllib3.urlopen")
urls = [
"http://10.0.0.1/",
"http://192.168.1.1/",
"http://127.0.0.1:9999/internal",
"http://169.254.169.254/latest/meta-data/",
]
with patch.object(pool.HTTPConnectionPool, "urlopen", intercepting_urlopen):
for url in urls:
attempts.clear()
with pytest.raises(ValidationError):
ingestor.ingest_url(url)
assert attempts == [], f"SSRF target reached urllib3: {url}"
def test_file_scheme_rejected_by_semantica(self):
ingestor = WebIngestor(respect_robots=False, delay=0)
with pytest.raises(ValidationError, match="not permitted"):
ingestor.ingest_url("file://localhost/etc/passwd")
def test_allow_private_ips_permits_loopback_fetch(self):
ingestor = WebIngestor(
respect_robots=False, delay=0, allow_private_ips=True
)
with patch.object(ingestor.session, "request") as mock_request, patch.object(
ingestor, "extract_content"
) as mock_extract:
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.text = "ok"
mock_resp.raise_for_status = MagicMock()
mock_request.return_value = mock_resp
mock_extract.return_value = MagicMock(status_code=None)
ingestor.ingest_url("http://127.0.0.1:9/probe")
mock_request.assert_called_once()
assert mock_request.call_args.kwargs.get("allow_redirects") is False
mock_extract.assert_called_once()
def test_allow_private_ips_string_false_keeps_ssrf_on(self):
ingestor = WebIngestor(
respect_robots=False, delay=0, allow_private_ips="false"
)
assert ingestor.allow_private_ips is False
with pytest.raises(ValidationError, match="blocked"):
ingestor.ingest_url("http://127.0.0.1:9/probe")
def test_allow_private_ips_string_true_opts_in(self):
ingestor = WebIngestor(
respect_robots=False, delay=0, allow_private_ips="true"
)
assert ingestor.allow_private_ips is True
with patch.object(ingestor.session, "request") as mock_request, patch.object(
ingestor, "extract_content"
) as mock_extract:
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.text = "ok"
mock_resp.raise_for_status = MagicMock()
mock_request.return_value = mock_resp
mock_extract.return_value = MagicMock(status_code=None)
ingestor.ingest_url("http://127.0.0.1:9/probe")
mock_request.assert_called_once()
def test_redirect_to_private_ip_blocked(self):
ingestor = WebIngestor(respect_robots=False, delay=0)
redirect = MagicMock()
redirect.status_code = 302
redirect.headers = {"Location": "http://127.0.0.1/admin"}
redirect.close = MagicMock()
with patch.object(ingestor.session, "request", return_value=redirect), patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
with pytest.raises(ValidationError, match="blocked"):
ingestor.ingest_url("https://example.com/public")
class TestSitemapCrawlerSSRF:
def test_private_sitemap_url_rejected(self):
crawler = SitemapCrawler()
with pytest.raises(ValidationError):
crawler.parse_sitemap("http://10.0.0.1/sitemap.xml")
def test_allow_private_ips_string_false_keeps_ssrf_on(self):
crawler = SitemapCrawler(allow_private_ips="false")
assert crawler.allow_private_ips is False
with pytest.raises(ValidationError):
crawler.parse_sitemap("http://10.0.0.1/sitemap.xml")
def test_allow_private_ips_string_true_opts_in(self):
crawler = SitemapCrawler(allow_private_ips="true")
assert crawler.allow_private_ips is True
def test_redirect_to_private_ip_blocked(self):
crawler = SitemapCrawler()
redirect = MagicMock()
redirect.status_code = 302
redirect.headers = {"Location": "http://169.254.169.254/latest/meta-data/"}
redirect.close = MagicMock()
with patch(
"semantica.ingest.ssrf.requests.request", return_value=redirect
), patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
with pytest.raises(ValidationError, match="blocked"):
crawler.parse_sitemap("https://example.com/sitemap.xml")
class TestRESTIngestorSSRF:
def test_private_endpoint_never_reaches_session(self):
with patch("requests.Session") as MockSession:
mock_session = MockSession.return_value
ingestor = RESTIngestor()
with pytest.raises(ValidationError):
ingestor.ingest_endpoint("http://169.254.169.254/latest/meta-data/")
mock_session.request.assert_not_called()
def test_file_scheme_rejected(self):
ingestor = RESTIngestor()
with pytest.raises(ValidationError, match="not permitted"):
ingestor.ingest_endpoint("file://localhost/secret")
def test_allow_private_ips_opt_in(self):
with patch("requests.Session") as MockSession:
mock_session = MockSession.return_value
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"ok": True}
mock_response.headers = {"Content-Type": "application/json"}
mock_session.request.return_value = mock_response
ingestor = RESTIngestor(allow_private_ips=True)
data = ingestor.ingest_endpoint("http://127.0.0.1:8080/health")
assert data.data == {"ok": True}
mock_session.request.assert_called_once()
assert mock_session.request.call_args.kwargs.get("allow_redirects") is False
def test_allow_private_ips_string_false_keeps_ssrf_on(self):
with patch("requests.Session") as MockSession:
mock_session = MockSession.return_value
ingestor = RESTIngestor(allow_private_ips="false")
assert ingestor.allow_private_ips is False
with pytest.raises(ValidationError, match="blocked"):
ingestor.ingest_endpoint("http://127.0.0.1:8080/health")
mock_session.request.assert_not_called()
def test_allow_private_ips_string_true_opts_in(self):
with patch("requests.Session") as MockSession:
mock_session = MockSession.return_value
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {"ok": True}
mock_response.headers = {"Content-Type": "application/json"}
mock_session.request.return_value = mock_response
ingestor = RESTIngestor(allow_private_ips="true")
assert ingestor.allow_private_ips is True
data = ingestor.ingest_endpoint("http://127.0.0.1:8080/health")
assert data.data == {"ok": True}
mock_session.request.assert_called_once()
def test_redirect_to_private_ip_blocked(self):
with patch("requests.Session") as MockSession:
mock_session = MockSession.return_value
mock_session.headers = {}
redirect = MagicMock()
redirect.status_code = 302
redirect.headers = {"Location": "http://127.0.0.1/secret"}
redirect.close = MagicMock()
mock_session.request.return_value = redirect
ingestor = RESTIngestor()
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
with pytest.raises(ValidationError, match="blocked"):
ingestor.ingest_endpoint("https://example.com/api")
mock_session.request.assert_called_once()
@@ -0,0 +1,64 @@
"""Tests for PipelineWithProvenance.
Verifies that:
1. The import path is correct (no ModuleNotFoundError).
2. PipelineWithProvenance accepts a built Pipeline and runs it via ExecutionEngine.
3. Provenance tracking gracefully degrades when the provenance package is absent.
"""
import sys
from unittest.mock import patch
import pytest
from semantica.pipeline import PipelineBuilder
from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
from semantica.pipeline.execution_engine import ExecutionResult
class TestPipelineWithProvenance:
"""Tests for PipelineWithProvenance."""
@pytest.fixture
def simple_pipeline(self):
"""Build a minimal two-step pipeline for testing."""
builder = PipelineBuilder()
builder.add_step("ingest", "file_ingest")
builder.add_step("parse", "document_parse")
return builder.build("test_provenance_pipeline")
def test_instantiation_with_pipeline(self, simple_pipeline):
"""Should accept a built Pipeline instance."""
runner = PipelineWithProvenance(simple_pipeline, provenance=False)
assert runner._pipeline is simple_pipeline
def test_run_returns_execution_result(self, simple_pipeline):
"""run() should delegate to ExecutionEngine and return an ExecutionResult."""
runner = PipelineWithProvenance(simple_pipeline, provenance=False)
result = runner.run()
assert isinstance(result, ExecutionResult)
assert result.success is True
def test_getattr_delegates_to_pipeline(self, simple_pipeline):
"""Attribute access should fall through to the wrapped Pipeline."""
runner = PipelineWithProvenance(simple_pipeline, provenance=False)
assert runner.name == "test_provenance_pipeline"
assert len(runner.steps) == 2
def test_provenance_disabled_when_import_fails(self, simple_pipeline):
"""When semantica.provenance is unavailable, provenance should be disabled."""
# Force the provenance import to raise ImportError
with patch.dict(sys.modules, {"semantica.provenance": None}):
runner = PipelineWithProvenance(simple_pipeline, provenance=True)
assert runner.provenance is False
assert runner._prov_manager is None
# Should still execute successfully without provenance
result = runner.run()
assert isinstance(result, ExecutionResult)
assert result.success is True
def test_run_with_data(self, simple_pipeline):
"""run() should accept data and kwargs without error."""
runner = PipelineWithProvenance(simple_pipeline, provenance=False)
result = runner.run(data={"key": "value"})
assert isinstance(result, ExecutionResult)
@@ -167,7 +167,6 @@ class TestProvenanceEnabledDisabled:
"""Test all modules work with provenance=False.""" """Test all modules work with provenance=False."""
modules_to_test = [ modules_to_test = [
('semantica.context.context_provenance', 'ContextManagerWithProvenance'), ('semantica.context.context_provenance', 'ContextManagerWithProvenance'),
('semantica.pipeline.pipeline_provenance', 'PipelineWithProvenance'),
] ]
for module_path, class_name in modules_to_test: for module_path, class_name in modules_to_test:
@@ -183,7 +182,6 @@ class TestProvenanceEnabledDisabled:
"""Test all modules work with provenance=True.""" """Test all modules work with provenance=True."""
modules_to_test = [ modules_to_test = [
('semantica.context.context_provenance', 'ContextManagerWithProvenance'), ('semantica.context.context_provenance', 'ContextManagerWithProvenance'),
('semantica.pipeline.pipeline_provenance', 'PipelineWithProvenance'),
] ]
for module_path, class_name in modules_to_test: for module_path, class_name in modules_to_test:
@@ -195,6 +193,23 @@ class TestProvenanceEnabledDisabled:
except ImportError: except ImportError:
pytest.skip(f"{module_path} not available") pytest.skip(f"{module_path} not available")
def test_pipeline_with_provenance_supports_provenance_flag(self):
"""PipelineWithProvenance accepts provenance=True/False.
PipelineWithProvenance requires a built Pipeline instance (unlike
other *WithProvenance wrappers that own their internal state), so it
cannot participate in the generic no-argument constructor loop above.
"""
try:
from semantica.pipeline.pipeline_builder import Pipeline
from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
pipeline = Pipeline(name="compat_test")
assert PipelineWithProvenance(pipeline, provenance=False).provenance is False
assert PipelineWithProvenance(pipeline, provenance=True).provenance is True
except ImportError:
pytest.skip("pipeline_provenance not available")
class TestAllModulesEdgeCases: class TestAllModulesEdgeCases:
"""Test edge cases across all provenance modules.""" """Test edge cases across all provenance modules."""
@@ -219,10 +234,14 @@ class TestAllModulesEdgeCases:
"""Test each module has independent provenance manager.""" """Test each module has independent provenance manager."""
try: try:
from semantica.context.context_provenance import ContextManagerWithProvenance from semantica.context.context_provenance import ContextManagerWithProvenance
from semantica.pipeline.pipeline_builder import Pipeline
from semantica.pipeline.pipeline_provenance import PipelineWithProvenance from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
ctx = ContextManagerWithProvenance(provenance=True) ctx = ContextManagerWithProvenance(provenance=True)
pipe = PipelineWithProvenance(provenance=True) pipe = PipelineWithProvenance(
Pipeline(name="independence_test"),
provenance=True,
)
# Each should have its own manager # Each should have its own manager
assert ctx._prov_manager is not None assert ctx._prov_manager is not None
@@ -154,10 +154,13 @@ class TestModuleSpecificEdgeCases:
def test_pipeline_with_empty_data(self): def test_pipeline_with_empty_data(self):
"""Test pipeline with empty data.""" """Test pipeline with empty data."""
try: try:
from semantica.pipeline.pipeline_builder import Pipeline
from semantica.pipeline.pipeline_provenance import PipelineWithProvenance from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
pipeline = PipelineWithProvenance(provenance=True)
pipeline = Pipeline(name="edge_case_test")
runner = PipelineWithProvenance(pipeline, provenance=True)
# Should handle empty data # Should handle empty data
assert pipeline is not None assert runner is not None
except ImportError: except ImportError:
pytest.skip("Pipeline not available") pytest.skip("Pipeline not available")
@@ -29,13 +29,16 @@ class TestRealModuleIntegration:
def test_pipeline_real_execution_tracking(self): def test_pipeline_real_execution_tracking(self):
"""Test pipeline tracks execution with provenance.""" """Test pipeline tracks execution with provenance."""
try: try:
from semantica.pipeline.pipeline_builder import Pipeline
from semantica.pipeline.pipeline_provenance import PipelineWithProvenance from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
pipeline = PipelineWithProvenance(provenance=True) pipeline = Pipeline(name="provenance_tracking_test")
runner = PipelineWithProvenance(pipeline, provenance=True)
# Verify provenance setup # Verify provenance setup
assert pipeline.provenance is True assert runner.provenance is True
assert pipeline._prov_manager is not None assert runner._prov_manager is not None
assert isinstance(runner._prov_manager, ProvenanceManager)
except ImportError: except ImportError:
pytest.skip("Pipeline not available") pytest.skip("Pipeline not available")
+592
View File
@@ -0,0 +1,592 @@
"""Tests for previously untested split chunker classes (issue #864)."""
import pytest
from semantica.split.kg_chunkers import (
EntityAwareChunker,
GraphBasedChunker,
HierarchicalChunker,
OntologyAwareChunker,
RelationAwareChunker,
)
from semantica.split.methods import (
SEMANTIC_EXTRACT_AVAILABLE,
NETWORKX_AVAILABLE,
split_by_characters,
split_by_paragraphs,
split_by_sentences,
split_by_words,
split_entity_aware,
split_graph_based,
split_hierarchical,
split_ontology_aware,
split_recursive,
split_relation_aware,
split_sliding_window,
split_structural,
)
from semantica.split.semantic_chunker import Chunk
from semantica.split.sliding_window_chunker import SlidingWindowChunker
from semantica.split.structural_chunker import StructuralChunker, StructuralElement
from semantica.split.table_chunker import TableChunk, TableChunker
from semantica.utils.exceptions import ValidationError
requires_semantic_extract = pytest.mark.skipif(
not SEMANTIC_EXTRACT_AVAILABLE,
reason="semantica.semantic_extract is not available",
)
requires_networkx = pytest.mark.skipif(
not NETWORKX_AVAILABLE,
reason="networkx is not available",
)
# ---------------------------------------------------------------------------
# SlidingWindowChunker
# ---------------------------------------------------------------------------
class TestSlidingWindowChunker:
def test_init_defaults_and_validation(self):
chunker = SlidingWindowChunker(chunk_size=100, overlap=20)
assert chunker.chunk_size == 100
assert chunker.overlap == 20
assert chunker.stride == 80
with pytest.raises(ValidationError):
SlidingWindowChunker(chunk_size=0)
with pytest.raises(ValidationError):
SlidingWindowChunker(chunk_size=100, overlap=-1)
with pytest.raises(ValidationError):
SlidingWindowChunker(chunk_size=100, overlap=100)
def test_empty_text_returns_empty(self):
chunker = SlidingWindowChunker(chunk_size=50, overlap=10)
assert chunker.chunk("") == []
def test_fixed_size_overlap_invariant(self):
"""Last `overlap` chars of chunk N appear at the start of chunk N+1."""
text = "abcdefghijklmnopqrstuvwxyz0123456789" * 3 # 108 chars
overlap = 10
chunk_size = 30
chunker = SlidingWindowChunker(
chunk_size=chunk_size, overlap=overlap, stride=chunk_size - overlap
)
chunks = chunker.chunk(text, preserve_boundaries=False)
assert len(chunks) >= 2
for i in range(len(chunks) - 1):
# Final chunk may be shorter than overlap; compare shared window only
shared = min(overlap, len(chunks[i].text), len(chunks[i + 1].text))
expected_overlap = chunks[i].text[-shared:]
actual_prefix = chunks[i + 1].text[:shared]
assert actual_prefix == expected_overlap, (
f"Overlap mismatch between chunk {i} and {i + 1}: "
f"{expected_overlap!r} != {actual_prefix!r}"
)
for i in range(len(chunks) - 1):
assert (
chunks[i + 1].start_index - chunks[i].start_index
== chunk_size - overlap
)
def test_chunk_with_overlap_helper(self):
text = "word " * 40
chunker = SlidingWindowChunker(chunk_size=50, overlap=0)
chunks = chunker.chunk_with_overlap(text, overlap_size=15)
assert len(chunks) >= 2
assert chunker.overlap == 0
def test_boundary_preservation_avoids_mid_word_when_possible(self):
text = (
"Alice went to the market. Bob bought apples. "
"Carol cooked dinner. Dave drove home."
)
chunker = SlidingWindowChunker(chunk_size=40, overlap=10)
chunks = chunker.chunk(text, preserve_boundaries=True)
assert len(chunks) >= 1
for chunk in chunks:
assert isinstance(chunk, Chunk)
assert chunk.text
assert chunk.metadata.get("chunk_index") is not None
# ---------------------------------------------------------------------------
# StructuralChunker
# ---------------------------------------------------------------------------
class TestStructuralChunker:
MARKDOWN_DOC = """# Introduction
This is the intro paragraph about the project.
## Details
Here are more details about how it works.
- item one
- item two
- item three
## Conclusion
Final thoughts on the subject.
"""
def test_empty_text_returns_empty(self):
chunker = StructuralChunker(max_chunk_size=500)
assert chunker.chunk("") == []
def test_heading_based_splits(self):
chunker = StructuralChunker(respect_headers=True, max_chunk_size=200)
chunks = chunker.chunk(self.MARKDOWN_DOC)
assert len(chunks) >= 1
for chunk in chunks:
assert isinstance(chunk, Chunk)
assert chunk.metadata.get("structure_preserved") is True
assert "element_types" in chunk.metadata
all_types = []
for chunk in chunks:
all_types.extend(chunk.metadata["element_types"])
assert "heading" in all_types
assert "paragraph" in all_types
def test_heading_boundaries_separate_sections(self):
"""Distinct top-level headings must not be merged into one chunk."""
doc = """# Alpha
Content exclusively about alpha topic here.
# Beta
Content exclusively about beta topic here.
"""
chunker = StructuralChunker(respect_headers=True, max_chunk_size=50)
chunks = chunker.chunk(doc)
assert len(chunks) >= 2
alpha_chunks = [c for c in chunks if "exclusively about alpha" in c.text]
beta_chunks = [c for c in chunks if "exclusively about beta" in c.text]
assert alpha_chunks, "Alpha section body missing from chunks"
assert beta_chunks, "Beta section body missing from chunks"
# Heading-boundary invariant: alpha and beta bodies stay in separate chunks
for chunk in chunks:
has_alpha = "exclusively about alpha" in chunk.text
has_beta = "exclusively about beta" in chunk.text
assert not (has_alpha and has_beta), (
f"Sections merged across heading boundary: {chunk.text!r}"
)
def test_extract_structure_detects_headings_and_lists(self):
chunker = StructuralChunker()
elements = chunker._extract_structure(self.MARKDOWN_DOC)
types = [e.type for e in elements]
assert "heading" in types
assert "list" in types
assert "paragraph" in types
assert all(isinstance(e, StructuralElement) for e in elements)
def test_code_block_preserved(self):
text = """# Code
```python
def hello():
return "world"
```
After the code.
"""
chunker = StructuralChunker(max_chunk_size=2000)
elements = chunker._extract_structure(text)
types = [e.type for e in elements]
assert "code_block" in types
code = next(e for e in elements if e.type == "code_block")
assert "def hello" in code.text
# ---------------------------------------------------------------------------
# TableChunker
# ---------------------------------------------------------------------------
class TestTableChunker:
def _sample_table(self, n_rows: int = 10):
headers = ["Name", "Age", "City"]
rows = [[f"Person{i}", str(20 + i), f"City{i}"] for i in range(n_rows)]
return {"headers": headers, "rows": rows}
def test_rows_are_not_split_mid_row(self):
"""Each chunk contains complete rows only — never a partial row."""
table = self._sample_table(10)
chunker = TableChunker(max_rows=3, preserve_headers=True)
chunks = chunker.chunk_table(table)
assert len(chunks) == 4 # 3+3+3+1
for chunk in chunks:
assert isinstance(chunk, TableChunk)
assert chunk.headers == ["Name", "Age", "City"]
for row in chunk.rows:
assert len(row) == 3
assert chunk.metadata["row_count"] == len(chunk.rows)
flattened = [row for c in chunks for row in c.rows]
assert flattened == table["rows"]
def test_markdown_table_chunk_does_not_split_rows(self):
md = """| Name | Age | City |
| --- | --- | --- |
| Alice | 30 | NYC |
| Bob | 25 | LA |
| Carol | 40 | SF |
| Dave | 35 | CHI |
"""
chunker = TableChunker(max_rows=2, preserve_headers=True)
chunks = chunker.chunk(md)
assert len(chunks) == 2
for chunk in chunks:
assert chunk.metadata["chunk_type"] == "table"
data_lines = [
line
for line in chunk.text.split("\n")
if line and "---" not in line and not line.startswith("Name")
]
for line in data_lines:
cells = [c.strip() for c in line.split("|")]
assert len(cells) == 3
def test_non_table_text_returns_single_chunk(self):
chunker = TableChunker()
chunks = chunker.chunk("Just plain text without a table.")
assert len(chunks) == 1
assert chunks[0].metadata.get("error") == "No table found"
def test_extract_table_schema(self):
table = {
"headers": ["id", "active", "label"],
"rows": [
["1", "true", "alpha"],
["2", "false", "beta"],
],
}
schema = TableChunker().extract_table_schema(table)
assert schema["column_count"] == 3
assert schema["row_count"] == 2
assert schema["column_types"]["id"] == "numeric"
assert schema["column_types"]["active"] == "boolean"
assert schema["column_types"]["label"] == "text"
def test_chunk_by_columns(self):
table = self._sample_table(3)
chunker = TableChunker(chunk_by_columns=True, preserve_headers=True)
chunks = chunker.chunk_table(table, max_columns=2)
assert len(chunks) == 2
assert chunks[0].headers == ["Name", "Age"]
assert chunks[1].headers == ["City"]
for chunk in chunks:
for row in chunk.rows:
assert len(row) == len(chunk.headers)
# ---------------------------------------------------------------------------
# EntityAwareChunker (real optional deps via importorskip / skipif)
# ---------------------------------------------------------------------------
class TestEntityAwareChunker:
def test_init(self):
chunker = EntityAwareChunker(
chunk_size=500, chunk_overlap=50, ner_method="pattern"
)
assert chunker.chunk_size == 500
assert chunker.ner_method == "pattern"
assert chunker.preserve_entities is True
def test_empty_text(self):
chunker = EntityAwareChunker(chunk_size=100, ner_method="pattern")
chunks = chunker.chunk("")
assert isinstance(chunks, list)
@requires_semantic_extract
def test_entity_boundaries_preserved_with_pattern_ner(self):
"""Entity spans stay intact when using real pattern NER."""
pytest.importorskip("semantica.semantic_extract")
entity_text = "AppleInc"
# Use a contiguous token the pattern NER can latch onto
text = (
"Intro sentence one goes here. Intro sentence two goes here. "
f"{entity_text} was founded in Cupertino California recently. "
"More filler sentences keep the document long enough to chunk. "
"Yet another sentence about products and services worldwide. "
"Final sentence for padding the overall document length out."
)
chunks = split_entity_aware(
text,
chunk_size=90,
ner_method="pattern",
preserve_entities=True,
)
assert len(chunks) >= 1
containing = [c for c in chunks if entity_text in c.text]
assert containing, "Expected entity text to appear in at least one chunk"
for chunk in containing:
idx = chunk.text.index(entity_text)
assert chunk.text[idx : idx + len(entity_text)] == entity_text
@requires_semantic_extract
def test_entity_aware_chunker_with_pattern_ner(self):
pytest.importorskip("semantica.semantic_extract")
text = (
"Alice Johnson founded Acme Corporation in New York. "
"Bob Smith joined the company later. "
"They expanded operations across Europe and Asia. "
) * 5
chunker = EntityAwareChunker(
chunk_size=120, ner_method="pattern", preserve_entities=True
)
chunks = chunker.chunk(text)
assert len(chunks) >= 1
assert all(isinstance(c, Chunk) for c in chunks)
# ---------------------------------------------------------------------------
# RelationAware / GraphBased / OntologyAware / Hierarchical
# ---------------------------------------------------------------------------
class TestRelationAwareChunker:
def test_init(self):
chunker = RelationAwareChunker(chunk_size=100, relation_method="pattern")
assert chunker.chunk_size == 100
assert chunker.relation_method == "pattern"
@requires_semantic_extract
def test_chunk_with_pattern_extractors(self):
pytest.importorskip("semantica.semantic_extract")
text = (
"Alice works at Acme. Bob reports to Alice. "
"Carol founded Acme in 2010. More padding text follows here. "
) * 4
chunker = RelationAwareChunker(
chunk_size=100, relation_method="pattern", ner_method="pattern"
)
chunks = chunker.chunk(text)
assert isinstance(chunks, list)
assert len(chunks) >= 1
assert all(isinstance(c, Chunk) for c in chunks)
class TestGraphBasedChunker:
def test_init(self):
chunker = GraphBasedChunker(
chunk_size=500, strategy="community", algorithm="louvain"
)
assert chunker.strategy == "community"
assert chunker.algorithm == "louvain"
@requires_semantic_extract
@requires_networkx
def test_chunk_with_real_optional_deps(self):
pytest.importorskip("networkx")
pytest.importorskip("semantica.semantic_extract")
text = (
"Alice met Bob at Acme Corporation yesterday afternoon. "
"Bob introduced Carol to the Acme engineering team. "
"Carol and Alice later discussed graph-based retrieval methods. "
) * 3
chunker = GraphBasedChunker(
chunk_size=200,
strategy="community",
algorithm="louvain",
ner_method="pattern",
relation_method="pattern",
)
chunks = chunker.chunk(text)
assert len(chunks) >= 1
assert all(isinstance(c, Chunk) for c in chunks)
# Ensure the graph-based path actually ran (not fallback-to-recursive).
assert any(
c.metadata.get("method") == "graph_based" for c in chunks
), "Expected at least one graph_based chunk"
assert any(
c.metadata.get("strategy") == "community"
and c.metadata.get("algorithm") == "louvain"
for c in chunks
), "Expected graph_based chunk metadata to include strategy/algorithm"
assert not any(
c.metadata.get("method") == "recursive" for c in chunks
), "Graph-based fallback to recursive was triggered"
class TestOntologyAwareChunker:
def test_init(self):
chunker = OntologyAwareChunker(chunk_size=200, preserve_concepts=True)
assert chunker.chunk_size == 200
assert chunker.preserve_concepts is True
@requires_semantic_extract
def test_chunk_uses_entity_aware_path(self):
pytest.importorskip("semantica.semantic_extract")
text = "Concept Alpha relates to Concept Beta in the taxonomy. " * 8
chunker = OntologyAwareChunker(
chunk_size=120, preserve_concepts=True, ner_method="pattern"
)
chunks = chunker.chunk(text)
assert len(chunks) >= 1
assert all(isinstance(c, Chunk) for c in chunks)
class TestHierarchicalChunker:
def test_hierarchical_markdown_sections(self):
text = """# Section One
Paragraph under section one with enough content to matter.
# Section Two
Paragraph under section two also with sufficient content.
"""
chunker = HierarchicalChunker(
levels=["section", "paragraph"], chunk_sizes=[2000, 500]
)
chunks = chunker.chunk(text)
assert len(chunks) >= 1
for chunk in chunks:
assert chunk.metadata.get("hierarchical") is True
assert chunk.metadata.get("levels") == ["section", "paragraph"]
def test_split_hierarchical_function(self):
text = "Para one.\n\nPara two.\n\nPara three."
chunks = split_hierarchical(text, levels=["paragraph"], chunk_sizes=[1000])
assert len(chunks) >= 1
# ---------------------------------------------------------------------------
# Exported method functions (public API smoke coverage)
# ---------------------------------------------------------------------------
class TestSplitMethodFunctions:
SAMPLE = (
"First sentence about knowledge graphs. "
"Second sentence covers entity extraction. "
"Third sentence discusses relation awareness. "
"Fourth sentence wraps up the example."
)
MARKDOWN = """# Intro
Intro paragraph with enough text to matter for structural splitting.
# Body
Body paragraph under a distinct heading for separation checks.
"""
def test_split_recursive(self):
chunks = split_recursive(self.SAMPLE, chunk_size=60)
assert len(chunks) >= 1
assert all(isinstance(c, Chunk) for c in chunks)
def test_split_by_sentences(self):
chunks = split_by_sentences(self.SAMPLE, chunk_size=80)
assert len(chunks) >= 1
def test_split_by_paragraphs(self):
text = "Para A content here.\n\nPara B content here.\n\nPara C content here."
chunks = split_by_paragraphs(text, chunk_size=50)
assert len(chunks) >= 1
def test_split_by_characters(self):
chunks = split_by_characters(self.SAMPLE, chunk_size=40)
assert len(chunks) >= 2
def test_split_by_words(self):
chunks = split_by_words(self.SAMPLE, chunk_size=10)
assert len(chunks) >= 1
def test_split_structural(self):
chunks = split_structural(
self.MARKDOWN, max_chunk_size=80, respect_headers=True
)
assert len(chunks) >= 2
assert all(isinstance(c, Chunk) for c in chunks)
def test_split_sliding_window(self):
chunks = split_sliding_window(
self.SAMPLE * 3,
chunk_size=40,
overlap=10,
preserve_boundaries=False,
)
assert len(chunks) >= 2
assert all(isinstance(c, Chunk) for c in chunks)
# Verify the sliding-window path was taken, not the recursive fallback.
# chunks[1].metadata["has_overlap"] is set only by SlidingWindowChunker.
assert chunks[1].metadata.get("has_overlap") is True, (
"Expected sliding-window chunks to carry has_overlap=True; "
"fallback to recursive may have occurred"
)
assert chunks[1].metadata.get("method") != "recursive", (
"Sliding-window fallback to recursive was triggered unexpectedly"
)
@requires_semantic_extract
def test_split_entity_aware(self):
pytest.importorskip("semantica.semantic_extract")
chunks = split_entity_aware(
self.SAMPLE * 3, chunk_size=80, ner_method="pattern"
)
assert len(chunks) >= 1
@requires_semantic_extract
def test_split_relation_aware(self):
pytest.importorskip("semantica.semantic_extract")
chunks = split_relation_aware(
self.SAMPLE * 3,
chunk_size=80,
relation_method="pattern",
ner_method="pattern",
)
assert len(chunks) >= 1
@requires_semantic_extract
@requires_networkx
def test_split_graph_based(self):
pytest.importorskip("networkx")
pytest.importorskip("semantica.semantic_extract")
chunks = split_graph_based(
self.SAMPLE * 3,
chunk_size=120,
strategy="community",
algorithm="louvain",
ner_method="pattern",
relation_method="pattern",
)
assert len(chunks) >= 1
assert all(isinstance(c, Chunk) for c in chunks)
# Ensure we didn't satisfy the test via the broad recursive fallback.
assert any(
c.metadata.get("method") == "graph_based" for c in chunks
), "Expected at least one graph_based chunk"
assert any(
c.metadata.get("strategy") == "community"
and c.metadata.get("algorithm") == "louvain"
for c in chunks
), "Expected graph_based chunk metadata to include strategy/algorithm"
assert not any(
c.metadata.get("method") == "recursive" for c in chunks
), "Graph-based fallback to recursive was triggered"
@requires_semantic_extract
def test_split_ontology_aware(self):
pytest.importorskip("semantica.semantic_extract")
chunks = split_ontology_aware(
self.SAMPLE * 3, chunk_size=80, ner_method="pattern"
)
assert len(chunks) >= 1
+20
View File
@@ -215,6 +215,26 @@ class TestGraphStore(unittest.TestCase):
result = self.store.execute_query("MATCH (n) RETURN n") result = self.store.execute_query("MATCH (n) RETURN n")
self.assertEqual(result["summary"], "Mock query executed") self.assertEqual(result["summary"], "Mock query executed")
def test_degree_centrality_rejects_malicious_label(self):
"""Regression test for GHSA-482h-hw99-h62p: degree_centrality()
interpolates labels/rel_type directly into a Cypher MATCH clause
(graph_store.py's own query builder, not delegated to the backend),
so an unvalidated label was a direct injection point."""
from semantica.utils.exceptions import ValidationError
evil_label = "N}) MATCH (victim) DETACH DELETE victim //"
with self.assertRaises(ValidationError):
self.store._manager.analytics.degree_centrality(labels=[evil_label])
def test_degree_centrality_rejects_malicious_rel_type(self):
from semantica.utils.exceptions import ValidationError
evil_rel_type = "R]-() DETACH DELETE n //"
with self.assertRaises(ValidationError):
self.store._manager.analytics.degree_centrality(rel_type=evil_rel_type)
def test_degree_centrality_with_legitimate_input_still_works(self):
result = self.store._manager.analytics.degree_centrality(labels=["Person"])
self.assertEqual(result, []) # MockGraphStore.execute_query returns no records
class TestGraphStoreInitialization(unittest.TestCase): class TestGraphStoreInitialization(unittest.TestCase):
def test_falkordb_initialization(self): def test_falkordb_initialization(self):
with patch('semantica.graph_store.falkordb_store.FalkorDBStore', side_effect=MockGraphStore) as mock_falkor: with patch('semantica.graph_store.falkordb_store.FalkorDBStore', side_effect=MockGraphStore) as mock_falkor:
+21
View File
@@ -57,5 +57,26 @@ class TestGraphStoreMethods(unittest.TestCase):
# Verify # Verify
self.mock_store.execute_query.assert_called_once_with(query, None) self.mock_store.execute_query.assert_called_once_with(query, None)
def test_update_relationship_rejects_malicious_property_key(self):
"""Regression test for GHSA-482h-hw99-h62p: update_relationship()
interpolates property keys directly into a Cypher SET clause
(methods.py's own query builder, not delegated to the backend
store), so an unvalidated key was a direct injection point."""
from semantica.utils.exceptions import ValidationError
evil_key = "x} MATCH (victim) DETACH DELETE victim //"
with self.assertRaises(ValidationError):
methods.update_relationship(1, {evil_key: "value"})
self.mock_store.execute_query.assert_not_called()
def test_update_relationship_with_legitimate_keys_still_works(self):
self.mock_store.execute_query.return_value = {
"records": [{"id": 1, "type": "KNOWS"}]
}
result = methods.update_relationship(1, {"weight": 0.5})
query = self.mock_store.execute_query.call_args[0][0]
self.assertIn("r.weight = $weight", query)
self.assertNotIn("DETACH DELETE", query)
if __name__ == '__main__': if __name__ == '__main__':
unittest.main() unittest.main()
+312
View File
@@ -0,0 +1,312 @@
"""
Regression tests for security fixes in PR #898.
Covers:
1. Cypher Injection Prevention (AGE Store)
2. SPARQL Injection Prevention (read-only query validation)
3. XXE Protection (rdf_parser fail-closed)
4. Vector save numpy serialization
5. SPARQL graph cap error handling
6. SSRF redirect handling (relative URLs, resp.close)
Explorer API-key authentication (GHSA-j4mq-hprp-987v) has its own, more
thorough test suite at tests/explorer/test_explorer_auth.py it isn't
duplicated here.
"""
import pytest
# Import the real implementation rather than re-declaring the regexes here:
# an earlier version of this file inlined a copy that silently drifted from
# semantica/explorer/routes/sparql.py's actual behavior (the inlined
# _COMMENT_LINE regex stripped '#' mid-token, corrupting any PREFIX
# declaration using a namespace IRI with a literal '#', e.g. the standard
# rdf:/rdfs: namespaces) and neither the code nor this test caught it,
# since both had the same bug. Importing the real function makes that class
# of drift impossible.
from semantica.explorer.routes.sparql import _is_read_only_query
class TestSparqlReadOnlyValidation:
"""Regression tests for SPARQL injection prevention."""
def test_select_allowed(self):
assert _is_read_only_query("SELECT ?s ?p ?o WHERE { ?s ?p ?o }")
def test_ask_allowed(self):
assert _is_read_only_query("ASK { ?s ?p ?o }")
def test_construct_allowed(self):
assert _is_read_only_query("CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }")
def test_describe_allowed(self):
assert _is_read_only_query("DESCRIBE <http://example.org>")
def test_insert_blocked(self):
assert not _is_read_only_query("INSERT DATA { <s> <p> <o> }")
def test_delete_blocked(self):
assert not _is_read_only_query("DELETE WHERE { ?s ?p ?o }")
def test_drop_blocked(self):
assert not _is_read_only_query("DROP GRAPH <http://example.org>")
def test_comment_bypass_blocked(self):
"""Attacker hides INSERT behind a comment, SELECT follows."""
query = "# innocent comment\nINSERT DATA { <s> <p> <o> }"
assert not _is_read_only_query(query)
def test_comment_hiding_real_query(self):
"""Comment at top with SELECT visible, but INSERT in body."""
query = "# SELECT everything\nINSERT DATA { <s> <p> <o> }"
assert not _is_read_only_query(query)
def test_prefix_before_select_allowed(self):
"""PREFIX declarations before SELECT should still be allowed."""
query = "PREFIX ex: <http://example.org/>\nSELECT ?s WHERE { ?s ex:p ?o }"
assert _is_read_only_query(query)
def test_prefix_before_insert_blocked(self):
"""PREFIX declarations can't disguise an INSERT."""
query = "PREFIX ex: <http://example.org/>\nINSERT DATA { ex:s ex:p ex:o }"
assert not _is_read_only_query(query)
def test_multiple_prefixes_then_select(self):
query = (
"PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n"
"PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n"
"SELECT ?s ?label WHERE { ?s rdfs:label ?label }"
)
assert _is_read_only_query(query)
def test_select_with_insert_keyword_blocked(self):
"""Even if SELECT is first, INSERT in body should be blocked."""
query = "SELECT ?s WHERE { ?s ?p ?o } ; INSERT DATA { <a> <b> <c> }"
assert not _is_read_only_query(query)
def test_case_insensitive_insert(self):
assert not _is_read_only_query("insert data { <s> <p> <o> }")
def test_load_blocked(self):
assert not _is_read_only_query("LOAD <http://evil.com/data.ttl>")
def test_clear_blocked(self):
assert not _is_read_only_query("CLEAR ALL")
def test_empty_query_rejected(self):
assert not _is_read_only_query("")
def test_whitespace_only_rejected(self):
assert not _is_read_only_query(" \n\t ")
def test_base_before_select(self):
query = "BASE <http://example.org/>\nSELECT ?s WHERE { ?s ?p ?o }"
assert _is_read_only_query(query)
def test_namespace_iri_with_hash_fragment_not_treated_as_comment(self):
"""A '#' inside a PREFIX declaration's IRI (standard for RDF/RDFS/OWL
namespaces) must not be mistaken for a comment-start a naive
`#[^\\n]*` strip corrupts the IRI and truncates the rest of the
query with it."""
query = (
"PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n"
"SELECT ?s WHERE { ?s rdf:type ?o }"
)
assert _is_read_only_query(query)
def test_real_comment_after_namespace_iri_still_stripped(self):
"""A genuine trailing comment must still be recognized even on a
line that also contains a '#'-bearing IRI earlier in the query."""
query = (
"PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n"
"SELECT ?s WHERE { ?s rdf:type ?o } # trailing comment INSERT DATA"
)
assert _is_read_only_query(query)
def test_inline_prefix_before_select_allowed(self):
"""PREFIX declaration on the same line as the query keyword (inline
prologue) must be stripped correctly so SELECT is seen first.
Regression for the [ \\t]*(?:\\n|$) anchor that rejected this form."""
query = "PREFIX ex: <http://example.org/> SELECT ?s WHERE { ?s ex:p ?o }"
assert _is_read_only_query(query)
def test_crlf_line_endings_with_prefix(self):
"""Windows CRLF line endings (\\r\\n) between PREFIX and SELECT must
be handled correctly. The previous (?:\\n|$) anchor did not allow
the \\r before \\n, causing stripping to fail."""
query = "PREFIX ex: <http://example.org/>\r\nSELECT ?s WHERE { ?s ?p ?o }"
assert _is_read_only_query(query)
def test_crlf_multiple_prefixes_then_select(self):
"""Multiple PREFIX lines with CRLF endings should all be stripped."""
query = (
"PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\r\n"
"PREFIX ex: <http://example.org/>\r\n"
"SELECT ?s WHERE { ?s rdf:type ex:Thing }"
)
assert _is_read_only_query(query)
def test_inline_prefix_before_insert_still_blocked(self):
"""An inline PREFIX followed by INSERT must not be allowed — the
inline stripping fix must not open a bypass for write operations."""
query = "PREFIX ex: <http://example.org/> INSERT DATA { ex:s ex:p ex:o }"
assert not _is_read_only_query(query)
def test_long_valid_query_not_rejected_by_is_read_only(self):
"""_is_read_only_query must not enforce the length limit itself —
that responsibility belongs to execute_sparql() so the route can
return a distinct, actionable error. A syntactically valid but long
SELECT query must still return True from this function."""
long_query = "SELECT ?s WHERE { ?s ?p ?o } # " + ("x" * 20_000)
assert _is_read_only_query(long_query)
# ===================================================================
# 2. Cypher injection prevention
# ===================================================================
class TestCypherInjection:
"""Regression tests for Cypher/SQL injection prevention."""
def test_sanitize_label_valid(self):
from semantica.graph_store.age_store import _sanitize_label
assert _sanitize_label("Entity") == "Entity"
assert _sanitize_label("my_label_123") == "my_label_123"
def test_sanitize_label_injection(self):
from semantica.graph_store.age_store import _sanitize_label
with pytest.raises(Exception): # ValidationError
_sanitize_label("Entity') OR 1=1--")
def test_sanitize_label_special_chars(self):
from semantica.graph_store.age_store import _sanitize_label
with pytest.raises(Exception):
_sanitize_label("Entity;DROP TABLE")
def test_value_to_cypher_literal_string_escaping(self):
from semantica.graph_store.age_store import _value_to_cypher_literal
result = _value_to_cypher_literal("O'Brien")
assert "\\'" in result # Single quote should be escaped
def test_value_to_cypher_literal_backslash(self):
from semantica.graph_store.age_store import _value_to_cypher_literal
result = _value_to_cypher_literal("path\\to\\file")
assert "\\\\" in result
def test_dollar_dollar_breakout_blocked(self):
"""$$ in a Cypher query would break out of AGE's delimiter."""
from semantica.graph_store.age_store import ApacheAgeStore
store = ApacheAgeStore.__new__(ApacheAgeStore)
store.graph_name = "test_graph"
store._conn = None
with pytest.raises(Exception): # ValidationError
store._execute_cypher("MATCH (n) RETURN n $$ ) AS (x agtype); DROP TABLE users; --")
def test_graph_name_sanitization(self):
"""Graph name with SQL injection should be rejected."""
from semantica.graph_store.age_store import ApacheAgeStore
with pytest.raises(Exception): # ValidationError
ApacheAgeStore(
connection_string="host=localhost",
graph_name="test'); DROP TABLE--"
)
def test_graph_name_valid(self):
from semantica.graph_store.age_store import ApacheAgeStore
store = ApacheAgeStore(
connection_string="host=localhost",
graph_name="my_graph_123"
)
assert store.graph_name == "my_graph_123"
def test_property_key_injection(self):
from semantica.graph_store.age_store import _props_to_cypher_literal
with pytest.raises(Exception):
_props_to_cypher_literal({"key; DROP": "value"})
# ===================================================================
# 3. XXE Protection (fail-closed)
# ===================================================================
class TestXXEProtection:
"""Regression tests for XXE prevention in rdf_parser."""
def test_defusedxml_check_exists(self):
"""The _HAS_DEFUSEDXML flag must exist."""
from semantica.explorer.utils.rdf_parser import _HAS_DEFUSEDXML
assert isinstance(_HAS_DEFUSEDXML, bool)
def test_safe_parse_rdf_function_exists(self):
"""_safe_parse_rdf must be importable."""
from semantica.explorer.utils.rdf_parser import _safe_parse_rdf
assert callable(_safe_parse_rdf)
# ===================================================================
# 4. Numpy vector serialization
# ===================================================================
class TestVectorSerialization:
"""Regression test for numpy array serialization in vector_store."""
def test_tolist_on_numpy_like(self):
"""Objects with tolist() should use it instead of list()."""
class FakeNumpyArray:
def __init__(self, data):
self._data = data
def tolist(self):
return self._data
def __iter__(self):
# list() would call this and fail for multi-dim arrays
raise TypeError("Use tolist() for numpy arrays")
arr = FakeNumpyArray([1.0, 2.0, 3.0])
# Simulate the fixed logic
result = arr.tolist() if hasattr(arr, "tolist") else list(arr)
assert result == [1.0, 2.0, 3.0]
def test_regular_list_still_works(self):
"""Regular lists (no tolist) should use list()."""
data = [1.0, 2.0, 3.0]
result = data.tolist() if hasattr(data, "tolist") else list(data)
assert result == [1.0, 2.0, 3.0]
# ===================================================================
# 5. SSRF redirect handling
# ===================================================================
class TestSSRFRedirectHandling:
"""Regression tests for SSRF redirect fixes."""
def test_urljoin_resolves_relative(self):
"""Relative Location headers must be resolved against current URL."""
from urllib.parse import urljoin
base = "https://example.com/api/ontology"
relative = "/ontology.ttl"
result = urljoin(base, relative)
assert result == "https://example.com/ontology.ttl"
def test_urljoin_absolute_passthrough(self):
"""Absolute Location headers should pass through unchanged."""
from urllib.parse import urljoin
base = "https://example.com/api/ontology"
absolute = "https://other.com/data.ttl"
result = urljoin(base, absolute)
assert result == "https://other.com/data.ttl"
def test_urljoin_relative_path(self):
"""Relative path without leading slash."""
from urllib.parse import urljoin
base = "https://example.com/api/v1/resource"
relative = "../data.ttl"
result = urljoin(base, relative)
assert result == "https://example.com/api/data.ttl"
if __name__ == "__main__":
pytest.main([__file__, "-v"])
@@ -0,0 +1,275 @@
"""Regression tests for GHSA-8vgg-8mr4-r236: unvalidated triplet IRIs
allowed arbitrary SPARQL update injection in the Blazegraph and RDF4J
stores, and query-filter injection on the Jena read path.
Triplet.subject/predicate (and, in some builders, .object) are document
text in the normal ingest pipeline entity names extracted from ingested
content. A subject containing '>' closes the '<...>' IRI token early, so
the rest of the value is parsed as more SPARQL, letting an attacker
append operations like CLEAR ALL that run with the application's store
credentials.
Mirrors the advisory's own PoC payload: a subject/predicate crafted to
close the current triple pattern and append a destructive `; CLEAR ALL ;`
statement. After the fix, sparql_escaping.validate_uri (already used by
anzo_store.py, the one backend that was already hardened) rejects it
before any query/update text is built.
"""
import unittest
from unittest.mock import patch
from semantica.semantic_extract.triplet_extractor import Triplet
from semantica.triplet_store.blazegraph_store import BlazegraphStore
from semantica.triplet_store.rdf4j_store import RDF4JStore
from semantica.triplet_store.jena_store import JenaStore
from semantica.utils.exceptions import ProcessingError, ValidationError
# The advisory's own injection payload: closes the <...> token, then the
# triple pattern, then appends a store-wide wipe and a rogue insert.
EVIL_SUBJECT = (
"http://example.com/a> <http://example.com/b> <http://example.com/c> . } "
"; CLEAR ALL ; INSERT DATA { <http://evil.example/owned"
)
class TestBlazegraphSparqlInjection(unittest.TestCase):
@patch.object(BlazegraphStore, "_connect", autospec=True)
def _make_store(self, _mock_connect):
return BlazegraphStore(endpoint="http://localhost:9999/blazegraph")
def test_build_insert_data_rejects_malicious_subject(self):
store = self._make_store()
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="x")
with self.assertRaises(ValidationError):
store._build_insert_data([triplet])
def test_triplets_to_rdf_rejects_malicious_subject(self):
store = self._make_store()
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="x")
with self.assertRaises(ValidationError):
store._triplets_to_rdf([triplet])
def test_delete_triplet_rejects_malicious_subject(self):
store = self._make_store()
store.connected = True
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="x")
with self.assertRaises(ValidationError):
store.delete_triplet(triplet)
def test_get_triplets_rejects_malicious_subject_filter(self):
store = self._make_store()
with self.assertRaises(ValidationError):
store.get_triplets(subject=EVIL_SUBJECT)
def test_bulk_load_rejects_malicious_graph_option(self):
# bulk_load() wraps its whole body in except Exception: raise
# ProcessingError(...) (pre-existing, unrelated to this fix), so the
# ValidationError sanitize_uri raises surfaces as ProcessingError.
# The security property that matters — the malicious query is never
# sent — holds either way.
store = self._make_store()
store.connected = True
triplet = Triplet(subject="http://s", predicate="http://p", object="x")
with self.assertRaises(ProcessingError) as ctx:
store.bulk_load([triplet], graph=EVIL_SUBJECT)
self.assertIn("Invalid URI", str(ctx.exception))
def test_legitimate_triplet_still_builds_correct_query(self):
store = self._make_store()
triplet = Triplet(subject="http://s", predicate="http://p", object="x")
insert_data = store._build_insert_data([triplet])
self.assertIn("<http://s> <http://p>", insert_data)
self.assertNotIn("CLEAR ALL", insert_data)
def test_format_object_rejects_malicious_pre_wrapped_iri(self):
"""A caller-supplied object already wrapped in '<...>' must still be
fully validated, not just checked for a literal space/'>' a
narrower ad-hoc check here previously let this branch bypass
validate_uri() entirely (Codex-flagged follow-up to GHSA-8vgg)."""
store = self._make_store()
evil_object = f"<{EVIL_SUBJECT}>"
triplet = Triplet(subject="http://s", predicate="http://p", object=evil_object)
with self.assertRaises(ValidationError):
store._format_object_for_sparql(triplet)
def test_format_object_accepts_legitimate_pre_wrapped_iri(self):
store = self._make_store()
triplet = Triplet(subject="http://s", predicate="http://p", object="<http://o>")
self.assertEqual(store._format_object_for_sparql(triplet), "<http://o>")
class TestRDF4JSparqlInjection(unittest.TestCase):
@patch.object(RDF4JStore, "_connect", autospec=True)
def _make_store(self, _mock_connect):
return RDF4JStore(endpoint="http://localhost:9999/rdf4j", repository_id="mem")
def test_triplets_to_ntriples_rejects_malicious_subject(self):
store = self._make_store()
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="x")
with self.assertRaises(ValidationError):
store._triplets_to_ntriples([triplet])
def test_delete_triplet_rejects_malicious_subject(self):
store = self._make_store()
store.connected = True
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="http://o")
with self.assertRaises(ValidationError):
store.delete_triplet(triplet)
# ------------------------------------------------------------------
# Regression tests for the literal-object bug fixed after the
# adversarial review of PR #911: delete_triplet() previously called
# validate_uri(triplet.object) unconditionally, which rejected every
# non-URI object with ValidationError even though literal objects are
# perfectly legal in RDF. The fix routes the object through
# _format_object_for_ntriples so URI-valued objects are still validated
# while literal objects go through escape_literal unchanged.
# ------------------------------------------------------------------
def _make_connected_store_with_captured_query(self):
"""Return (store, captured_dict) where captured['update'] is the
SPARQL update string passed to requests.post once delete_triplet
succeeds."""
import requests as req_mod
from unittest.mock import MagicMock
store = self._make_store()
store.connected = True
captured = {}
mock_resp = MagicMock()
mock_resp.raise_for_status = MagicMock()
def fake_post(url, **kwargs):
captured["update"] = kwargs.get("data", {}).get("update", "")
return mock_resp
store._post = fake_post # not used directly; patch requests.post below
store._captured = captured
return store, captured
def test_delete_triplet_literal_object_succeeds(self):
"""A triplet with a plain-string literal object must delete without
raising ValidationError the regression that prompted this fix."""
store, captured = self._make_connected_store_with_captured_query()
with patch("requests.post") as mock_post:
mock_post.return_value.__enter__ = lambda s: s
mock_post.return_value.raise_for_status = lambda: None
result = store.delete_triplet(
Triplet(subject="http://s", predicate="http://p", object="Paris")
)
self.assertEqual(result, {"success": True})
# Confirm query shape: object must be a quoted literal, not <Paris>
query_sent = mock_post.call_args[1]["data"]["update"]
self.assertIn("<http://s> <http://p>", query_sent)
self.assertIn('"Paris"', query_sent)
self.assertNotIn("<Paris>", query_sent)
self.assertNotIn("CLEAR ALL", query_sent)
def test_delete_triplet_uri_object_still_works(self):
"""A triplet whose object is a URI must still delete correctly."""
store, _ = self._make_connected_store_with_captured_query()
with patch("requests.post") as mock_post:
mock_post.return_value.raise_for_status = lambda: None
result = store.delete_triplet(
Triplet(subject="http://s", predicate="http://p", object="http://o")
)
self.assertEqual(result, {"success": True})
query_sent = mock_post.call_args[1]["data"]["update"]
self.assertIn("<http://s> <http://p> <http://o>", query_sent)
self.assertNotIn("CLEAR ALL", query_sent)
def test_delete_triplet_malicious_uri_object_rejected_before_post(self):
"""A URI-shaped object containing '>' must be rejected by
_format_object_for_ntriples validate_uri before requests.post
is ever called."""
evil_obj = "http://evil.com/a>;CLEARALL"
store, _ = self._make_connected_store_with_captured_query()
with patch("requests.post") as mock_post:
with self.assertRaises(ValidationError):
store.delete_triplet(
Triplet(subject="http://s", predicate="http://p", object=evil_obj)
)
mock_post.assert_not_called()
def test_delete_triplet_malicious_subject_still_rejected(self):
"""Subject injection protection must remain intact after the fix."""
store, _ = self._make_connected_store_with_captured_query()
with patch("requests.post") as mock_post:
with self.assertRaises(ValidationError):
store.delete_triplet(
Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="http://o")
)
mock_post.assert_not_called()
def test_get_triplets_rejects_malicious_subject_filter(self):
store = self._make_store()
with self.assertRaises(ValidationError):
store.get_triplets(subject=EVIL_SUBJECT)
def test_legitimate_triplet_still_builds_correct_ntriples(self):
store = self._make_store()
triplet = Triplet(subject="http://s", predicate="http://p", object="x")
ntriples = store._triplets_to_ntriples([triplet])
self.assertIn("<http://s> <http://p>", ntriples)
self.assertNotIn("CLEAR ALL", ntriples)
def test_format_object_rejects_malicious_pre_wrapped_iri(self):
"""Same pre-wrapped-object bypass as Blazegraph, fixed in
_format_object_for_ntriples."""
store = self._make_store()
evil_object = f"<{EVIL_SUBJECT}>"
triplet = Triplet(subject="http://s", predicate="http://p", object=evil_object)
with self.assertRaises(ValidationError):
store._format_object_for_ntriples(triplet)
def test_format_object_accepts_legitimate_pre_wrapped_iri(self):
store = self._make_store()
triplet = Triplet(subject="http://s", predicate="http://p", object="<http://o>")
self.assertEqual(store._format_object_for_ntriples(triplet), "<http://o>")
class TestJenaSparqlInjection(unittest.TestCase):
def setUp(self):
from rdflib import Graph
self.store = JenaStore()
self.store.graph = Graph()
def test_get_triplets_never_queries_with_malicious_subject_filter(self):
"""get_triplets() catches all exceptions and returns [] (pre-existing,
broad error-handling behavior unrelated to this fix), so the
observable contract is: the malicious filter must never reach
graph.query() at all."""
with patch.object(self.store.graph, "query", wraps=self.store.graph.query) as spy:
result = self.store.get_triplets(subject=EVIL_SUBJECT)
self.assertEqual(result, [])
spy.assert_not_called()
def test_get_triplets_with_legitimate_filter_still_reaches_query(self):
"""A validated identifier must not be rejected by the sanitizer —
it should reach graph.query(). (Whether the WHERE-clause filter
syntax jena_store.py builds is itself correct SPARQL is a separate,
pre-existing question this test doesn't assert on: the query here
is `{ ?s ?p ?o ?s = <http://s> }`, missing a FILTER()/separator,
and unrelated to sanitize_uri.)"""
self.store.graph.parse(
data='<http://s> <http://p> "x" .', format="ntriples"
)
with patch.object(self.store.graph, "query", wraps=self.store.graph.query) as spy:
self.store.get_triplets(subject="http://s")
spy.assert_called_once()
self.assertIn("http://s", spy.call_args[0][0])
if __name__ == "__main__":
unittest.main()
+46
View File
@@ -1,3 +1,6 @@
import json
import shutil
import tempfile
import unittest import unittest
from unittest.mock import MagicMock, patch from unittest.mock import MagicMock, patch
import numpy as np import numpy as np
@@ -134,5 +137,48 @@ class TestVectorStore(unittest.TestCase):
self.assertTrue(mock_backend.called) self.assertTrue(mock_backend.called)
def test_save_load_roundtrip_numpy_vectors(self):
"""save()/load() must handle numpy float32 vectors without raising.
Regression test: json.dump() rejects numpy scalar types, so a naive
`list(v)` conversion (which yields np.float32 elements, not native
floats) raises TypeError. `v.tolist()` converts recursively to
native Python floats and must be used instead.
"""
store = VectorStore(backend="inmemory", dimension=3)
store.vectors = {"v1": np.array([0.1, 0.2, 0.3], dtype=np.float32)}
store.metadata = {"v1": {"id": "1"}}
tmpdir = tempfile.mkdtemp()
try:
store.save(tmpdir) # must not raise TypeError
# The JSON file itself must be valid and free of numpy types.
with open(f"{tmpdir}/store_data.json", "r", encoding="utf-8") as f:
data = json.load(f)
self.assertTrue(all(isinstance(x, float) for x in data["vectors"]["v1"]))
loaded = VectorStore(backend="inmemory", dimension=3)
loaded.load(tmpdir)
np.testing.assert_allclose(
loaded.vectors["v1"], [0.1, 0.2, 0.3], rtol=1e-6
)
self.assertEqual(loaded.metadata["v1"], {"id": "1"})
finally:
shutil.rmtree(tmpdir, ignore_errors=True)
def test_load_rejects_legacy_pickle(self):
"""load() must refuse legacy .pkl stores rather than deserializing them."""
store = VectorStore(backend="inmemory", dimension=3)
tmpdir = tempfile.mkdtemp()
try:
with open(f"{tmpdir}/store_data.pkl", "wb") as f:
f.write(b"not a real pickle, just needs to exist")
with self.assertRaises(RuntimeError):
store.load(tmpdir)
finally:
shutil.rmtree(tmpdir, ignore_errors=True)
if __name__ == '__main__': if __name__ == '__main__':
unittest.main() unittest.main()