mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-30 04:40:16 +00:00
Compare commits
23
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5b319560fb | ||
|
|
f29c4310a1 | ||
|
|
a2886a4e41 | ||
|
|
a0aa415fc4 | ||
|
|
ae4f1d4030 | ||
|
|
ea3416ed32 | ||
|
|
154a7347cd | ||
|
|
f2f1d6787d | ||
|
|
646c70ce63 | ||
|
|
c5981aa306 | ||
|
|
d507fda1b0 | ||
|
|
7bf7474ac1 | ||
|
|
546e27cec5 | ||
|
|
a8330874d3 | ||
|
|
1c3ac66fd9 | ||
|
|
b846ff88d4 | ||
|
|
69b79e3d67 | ||
|
|
9012492c97 | ||
|
|
6ec546b551 | ||
|
|
6002965c55 | ||
|
|
abc10bc8e0 | ||
|
|
1f053e005c | ||
|
|
9ecae47a8a |
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.6.5] - 2026-08-11
|
||||
|
||||
### Added
|
||||
|
||||
- **Embedded Oxigraph backend for `TripletStore`** (#838, closes #834) by @Linxiushen
|
||||
@@ -244,6 +246,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Security
|
||||
|
||||
- **DNS check-then-use hardening for the ontology URL fetcher, and a remaining object-IRI validation gap** (#916, follow-up to GHSA-8c7v-62gr-hj6g and GHSA-8vgg-8mr4-r236) by @KaifAhmad1
|
||||
- **DNS check-then-use (TOCTOU) window**: GHSA-8c7v-62gr-hj6g's own fix description flagged this as a secondary gap — `_validate_fetch_url()` resolved and validated a hostname once, but `_fetch_url_sync()` then let `requests` resolve the same hostname again independently at connect time. A low-TTL or rebinding DNS answer could differ between the two lookups, reopening the SSRF window the validation exists to close
|
||||
- `_validate_fetch_url()` now returns the validated IP, and a new `_make_pinned_session()` builds a per-hop `requests.Session` whose connection pool is pinned directly to that IP — bypassing DNS resolution for the connection entirely — while explicitly restoring the real hostname as the outgoing HTTP `Host` header and, for HTTPS, the TLS SNI `server_hostname`/`assert_hostname`, so the connection reaches the validated IP but still presents (and is verified against) the real hostname's identity, keeping virtual hosting and certificate validation correct
|
||||
- Caught during implementation: an earlier draft set urllib3's `_dns_host` post-construction, assuming (as in some urllib3 releases) that it was decoupled from `host`. In the version this project installs (2.7.0), `host` is a property that reads/writes `_dns_host` directly, so that approach would have silently changed the Host header too — caught by an end-to-end test against a real local server before landing, rather than shipping. Verified with real (non-mocked) local HTTP and HTTPS servers, the latter using a generated self-signed certificate to prove SNI/cert-hostname verification checks the real hostname rather than the pinned IP, plus a negative control confirming a hostname/cert mismatch is still correctly rejected, not silently bypassed
|
||||
- **Object-IRI validation gap** (GHSA-8vgg-8mr4-r236 follow-up, distinct from the object-branch fix already shipped in #911): a triplet object already wrapped in `<...>` skipped `sparql_escaping.validate_uri()` in both `blazegraph_store.py` and `rdf4j_store.py`'s `_format_object_for_sparql`/`_format_object_for_ntriples`, only checking the inner content for a literal space or `>` — the pre-wrapped and unwrapped branches now validate identically
|
||||
- **Fixed along the way** (caught in automated review across two follow-up rounds): `_validate_fetch_url()` originally pinned to only the first resolved IP, so a hostname with multiple A/AAAA records would fail outright if that specific address was unreachable — it now returns every validated IP and `_make_pinned_session()` falls back through all of them, verified by pinning to a genuinely unreachable address followed by a working one and confirming the fetch still succeeds; the test HTTPS server allowed TLSv1/TLSv1.1 by not setting a minimum version, now pinned to TLSv1.2; and when an HTTP(S) proxy applied, pinning was silently skipped in favor of the unpinned path — proxies are now disabled outright for this fetcher (`session.trust_env = False`, so `HTTP_PROXY`/`HTTPS_PROXY` env vars are never consulted) with a fail-closed 502 backstop if a proxy is ever forced onto the session some other way, verified by pointing `HTTP_PROXY` at an address that would fail if actually used and confirming the fetch still succeeds directly
|
||||
- New `tests/explorer/test_ontology_dns_pinning.py` (12 tests: real local HTTP/HTTPS servers including 2 real-TLS checks, multi-IP fallback success/failure, and no-proxy-trust verification — gracefully skipped without the optional `cryptography` package where applicable); updated `tests/explorer/test_ontology_ssrf.py` for the new per-hop session construction; 4 new tests in `tests/triplet_store/test_sparql_injection.py` for the object-IRI fix. Full `explorer` + `triplet_store` suite: 572 passed
|
||||
|
||||
- **Missing Origin validation on the `/ws/graph-updates` WebSocket handshake** (#917, GHSA-4643-wpgq-w329) by @KaifAhmad1
|
||||
- `CORSMiddleware` doesn't cover WebSocket handshakes at all (Starlette's CORS support only wraps HTTP), so under `SEMANTICA_ALLOW_ANONYMOUS=true` — the mode `docker-compose.dev.yml` ships — the anonymous-mode key bypass accepted a `/ws/graph-updates` connection from any origin. Loopback binding isn't a boundary against a browser: any page the operator has open can still reach `ws://localhost:8000/ws/graph-updates` directly, and `ConnectionManager.broadcast` sends every `graph_mutation` to every connected socket with no per-connection scoping. Combined with `/api/import` accepting `multipart/form-data` (a CORS-safelisted content type that skips preflight), a hostile page could write to the graph over REST and read the result back over the unauthenticated WebSocket
|
||||
- Not affected: any deployment with `SEMANTICA_API_KEY` configured — the handshake already rejects without a valid key in that mode. This was an anonymous-mode-only, development-configuration exposure
|
||||
- Fix: check the handshake's `Origin` header against `app.state.explorer_settings['allowed_origins']` — the same list `CORSMiddleware` already enforces for HTTP — before the key check. A missing `Origin` (native/CLI clients, which never set the header) is still allowed through, since the browser is the only threat this closes
|
||||
- 4 new tests in `tests/explorer/test_explorer_auth.py`: hostile Origin rejected under anonymous mode; hostile Origin rejected even with a correct key (Origin is checked first, so a leaked key alone can't hijack the socket); an allowlisted Origin still connects; a missing Origin still connects. Full `explorer` suite: 226 passed
|
||||
|
||||
- **Polynomial-time ReDoS in the SPARQL route's `_PREFIX_DECL` regex** (#915, CodeQL `py/polynomial-redos`) by @Sameer6305
|
||||
- The prior pattern's trailing `\s*` overlapped with the preceding `<[^>]*>` IRI-body match on inputs containing no closing `>` (e.g. `base<` followed by thousands of `!<` repetitions), forcing the regex engine to explore every possible split between the two quantifiers — O(n²) backtracking reachable from `req.query` via `_is_read_only_query()`
|
||||
- Fixed by making the two quantifiers character-disjoint: horizontal whitespace only (`[ \t]`, never overlapping the IRI body) instead of `\s*`, and excluding CR/LF from the IRI body (`[^>\r\n]*`) so it can never span a line boundary. Independently verified: the exact pathological payload (`base<` + `!<` × 5,000/20,000) scales linearly (0.238ms → 0.841ms for 4x input, not the ~16x a surviving quadratic blowup would show)
|
||||
- Added `_SPARQL_MAX_QUERY_LEN = 10_000` as defense-in-depth, checked in `execute_sparql()` before any regex work so a future pattern regression stays bounded regardless
|
||||
- Two correctness regressions raised in review were checked and did not reproduce: comment-then-prefix stripping order means an inline comment after a `PREFIX` line (`PREFIX ex: <...> # comment`) is already gone by the time `_PREFIX_DECL` runs, verified directly against the pipeline; and the allowlist's `.sub()`-based cleaning only ever affects the yes/no decision, never the query actually sent to `graph.query()` — so even the narrow case of a multi-line string literal that happens to start a line with the literal text `PREFIX` or `BASE` can only cause a legitimate query to be wrongly rejected, never let something malicious through, since rdflib's parser still gates whatever actually executes
|
||||
- 20 new/updated tests in `tests/explorer/test_sparql_route.py` and `tests/test_security_regression.py` (inline prologues, CRLF line endings, multi-line CRLF prefix chains, oversized-query rejection). 225 `explorer` + 82 SPARQL-specific tests passing
|
||||
|
||||
- **SPARQL injection via unvalidated triplet IRIs** (#911, GHSA-8vgg-8mr4-r236) by @KaifAhmad1
|
||||
- `Triplet.subject`/`.predicate` (and, in some builders, `.object`) were interpolated directly into SPARQL update/query strings in the Blazegraph and RDF4J stores, and into a SELECT filter in the Jena store. A subject containing `>` closes the `<...>` IRI token early, so the rest of the value is parsed as more SPARQL. Entity names are document text in the normal ingest pipeline, so anyone whose content gets processed could append operations like `CLEAR ALL`, running with the application's store credentials
|
||||
- Applied the existing `sparql_escaping.validate_uri` (already used by `anzo_store.py`, the one backend that was already hardened — this generalizes its approach rather than inventing a new one) at every subject/predicate/object interpolation site: `blazegraph_store.py`'s `_build_insert_data`, `_triplets_to_rdf`, `bulk_load`'s `graph` option, `get_triplets`'s filter, and `delete_triplet`; `rdf4j_store.py`'s `_triplets_to_ntriples`, `get_triplets`'s filter, and `delete_triplet`; `jena_store.py`'s `get_triplets`'s filter (the only vulnerable site there — `add_triplets`/`delete_triplet` already use rdflib's native `Graph.add`/`.remove` with `URIRef` rather than building query strings)
|
||||
- **Fixed along the way** (caught in review, by @ZohaibHassan16): `_format_object_for_sparql`'s URI branch — used when a triplet's *object* is itself a URI rather than a literal — only checked for spaces and `>` inline instead of running the same `validate_uri` check applied to subject/predicate, leaving the object position as a narrower but real gap in both Blazegraph and RDF4J. Also fixed test flakiness in `RDF4JStore`'s test fixtures, which weren't mocking `_connect()` and so were making real network calls
|
||||
- New `tests/triplet_store/test_sparql_injection.py` (12+ tests) reproducing the advisory's own injection payload (`http://example.com/a> ... ; CLEAR ALL ; INSERT DATA { ...`) against all three backends' write and read paths, asserting the malicious query is never built or sent. Full triplet_store suite: 330+ tests passing
|
||||
- Side note, not part of this fix: found that `jena_store.py`'s `get_triplets()` builds syntactically invalid SPARQL for its WHERE-clause filters (missing a `FILTER()`/separator before the equality conditions) — a pre-existing correctness bug, unrelated to the injection fix, left alone here and worth a separate follow-up
|
||||
|
||||
- **Cypher injection via unvalidated node labels, relationship types, and property keys** (#910, GHSA-482h-hw99-h62p) by @KaifAhmad1
|
||||
- Node labels and property keys passed to `create_node`/`create_relationship` were interpolated directly into Cypher strings in the Neptune, Neo4j, and FalkorDB graph stores. Property *values* are parameterized, but labels and keys can't be bound as query parameters, and nothing validated them — so a document-derived entity type or property name (the normal ingest path) could close the current Cypher token early and append arbitrary statements (e.g. `DETACH DELETE`), running with the application's database credentials
|
||||
- New shared `semantica/graph_store/query_sanitize.py`: `sanitize_identifier()` generalizes `age_store.py`'s existing `_sanitize_label`/`_sanitize_rel_type` (the only backend that already validated this) into a helper the other backends import without an import cycle with `graph_store.py`/`methods.py`
|
||||
- Applied at every label/relationship-type/property-key interpolation site in `amazon_neptune.py`, `neo4j_store.py`, `falkordb_store.py`, `graph_store.py` (`degree_centrality`'s own query builder), and `methods.py` (`update_relationship`'s own query builder) — covers `create_node`, `create_nodes`, `create_relationship`, `get_nodes`, `get_relationships`, `get_neighbors`, `shortest_path`, `update_node`, `create_index`, and all relationship-type filters across the three backends
|
||||
- **Fixed along the way** (caught in review, by @Sameer6305): `depth`/`max_depth` path-length parameters are meant to be integers, but `Neo4jStore.get_neighbors()`/`shortest_path()` interpolated them into the Cypher variable-length-path syntax (`*1..{depth}`) without coercion — unlike the Neptune/FalkorDB equivalents, which already cast to `int()`. A string `depth` (e.g. `"1]->(x) DETACH DELETE x //"`) reached the query verbatim. Added the same `int()` coercion Neptune/FalkorDB already had, plus `GraphStore.get_neighbors()`'s `hops`/`depth` alias resolution
|
||||
- New `tests/graph_store/test_cypher_injection.py` (unit tests on `sanitize_identifier` plus the labels/keys/rel-types injection payload run against Neptune/Neo4j/FalkorDB `create_node`/`create_relationship`, asserting the malicious query is never built or sent) and the depth-coercion regression above; plus additions to `tests/test_graph_store.py` (`degree_centrality`) and `tests/test_graph_store_methods.py` (`update_relationship`). Full graph_store suite: 224+ tests passing
|
||||
|
||||
- **4 critical/high vulnerabilities in the Explorer API and vector store: RCE, SSRF, XXE, and DoS, plus Cypher/SPARQL injection hardening found along the way** (#898) by @Sunil56224972
|
||||
- **[CWE-502] Arbitrary code execution via `pickle.load()`**: `VectorStore.save()`/`load()` used `pickle` for the on-disk `store_data.pkl`; a crafted `.pkl` file placed in the store directory (file upload, shared filesystem, or supply-chain compromise) could execute arbitrary code on deserialization. Replaced with JSON — vectors and metadata are fully JSON-serializable, so nothing is lost — and `load()` now refuses any legacy `.pkl` file it finds with a migration error rather than deserializing it
|
||||
- **[CWE-918] SSRF via redirect bypass in `ontology.py`'s URL fetcher**: `_validate_fetch_url()` correctly blocked private/loopback/reserved addresses on the caller-supplied URL, but `_fetch_url_sync()` fetched with `allow_redirects=True`, so a validated *public* first hop could 302 to `http://169.254.169.254/...` (cloud instance metadata) or an internal service, and `requests` followed it with no re-check. Redirects are now followed manually, capped at 5 hops, with `_validate_fetch_url()` re-run against every hop's target — including relative `Location` headers, resolved via `urljoin()` before validation — and every response (redirect or final) is explicitly closed to avoid leaking connections back to the pool
|
||||
- **[CWE-611] XXE injection in the RDF/XML parser**: `_safe_parse_rdf()` depended on `defusedxml` for XXE protection, but `defusedxml` wasn't declared in `pyproject.toml`'s `explorer` extra, so it was silently absent in normal installs and the code fell back to a bare warning plus unsafe parsing — a crafted RDF/XML ontology with an external entity could read arbitrary server files. Added `defusedxml>=0.7.1` to the extra, and `_safe_parse_rdf()` now fails closed: it raises rather than parsing untrusted RDF/XML if `defusedxml` isn't importable, replacing an earlier regex-based DOCTYPE-stripping fallback that was reviewed and rejected as bypassable
|
||||
- **[CWE-770] DoS via unbounded SPARQL graph materialization**: `_build_rdflib_graph()` loaded up to 999,999 nodes and 999,999 edges into memory per query, and with up to 4 concurrent SPARQL requests permitted, an attacker could exhaust server memory. Added a 50,000 node/edge cap (`_SPARQL_MAX_GRAPH_NODES`); oversized graphs now return a clean error instead of attempting materialization
|
||||
- **Cypher injection via Apache AGE's `graph_name` and `$$`-delimiter breakout**: `graph_name` was interpolated unvalidated into `cypher('{graph_name}', $$ ... $$)`, and raw Cypher query text containing `$$` could close AGE's dollar-quoted string delimiter early and append arbitrary SQL. `graph_name` is now validated against the same identifier allowlist `age_store.py` already used for labels/relationship types, and any query containing `$$` is rejected outright
|
||||
- **SPARQL Explorer route (`/api/sparql`) hardened against comment/PREFIX-hiding bypass**: `_is_read_only_query()` now strips comments and PREFIX/BASE declarations before checking the leading keyword, and additionally scans the full query body for SPARQL Update keywords (INSERT/DELETE/DROP/LOAD/CLEAR/CREATE/COPY/MOVE/ADD) — so `SELECT ... ; DROP ALL` is now rejected by the keyword scan itself rather than relying solely on rdflib's parser
|
||||
- **Fixed along the way** (maintainer follow-up, addressing automated review findings and a regression introduced across several rounds of iteration on the original fix):
|
||||
- `VectorStore.save()`'s numpy handling used `list(v)` for the JSON fallback path, which produces `numpy.float32` elements that `json.dump()` can't serialize — changed to `v.tolist()`
|
||||
- the SPARQL graph-size `ValueError` was raised outside `execute_sparql()`'s exception handling and surfaced as an unhandled 500 instead of a clean API error — moved inside
|
||||
- every streamed `requests` response in the ontology redirect loop, including the one actually read and returned, is now closed in a `finally` block — a connection-pool leak that a rework of the redirect logic had briefly reintroduced after an earlier fix
|
||||
- a later commit meant to add opt-in API-key auth (`explorer/auth.py`, gated on `EXPLORER_API_KEY`) instead **replaced and silently disabled** the `Depends(require_auth)` enforcement already merged into `main` for GHSA-j4mq-hprp-987v (Critical — unauthenticated Explorer API), removed the `/ws/graph-updates` handshake check, and — unlike `require_auth` — failed *open* (allowed all requests) whenever its key was unset. Merging that version would have silently reverted an already-fixed Critical CVE the moment this branch landed. Removed `explorer/auth.py`; restored the per-router `Depends(require_auth)` wiring and the WebSocket auth check; kept the one genuine improvement in that commit (adding `X-API-Key` to the CORS `allow_headers` list) by folding it into the existing CORS config
|
||||
- the new SPARQL keyword-scan's comment-stripping regex (`#[^\n]*`) also matched the `#` inside standard RDF namespace IRIs (e.g. `.../1999/02/22-rdf-syntax-ns#`), corrupting any query with a normal `rdf:`/`rdfs:`-style `PREFIX` declaration — caught because the hardening's own bundled tests failed against two of its own cases. Fixed by only treating `#` as a comment-start at line-start or after whitespace; the companion `PREFIX`/`BASE` regex was also fixed to accept bare `BASE <...>` declarations, which have no prefix-name token between the keyword and the IRI
|
||||
- New/updated regression tests: `tests/explorer/test_ontology_ssrf.py` (redirect re-validation, relative-redirect resolution, response closing, redirect-cap enforcement), `tests/test_security_regression.py` (Cypher/SPARQL injection, XXE, numpy serialization, SSRF redirect handling), plus additions to `tests/explorer/test_sparql_route.py`, `tests/vector_store/test_vector_store.py`, and `tests/explorer/test_explorer_auth.py`
|
||||
- Note: the Cypher-injection hardening here is scoped to `age_store.py`'s `graph_name`/`$$` breakout, found while reviewing this PR. The broader label/property-key/relationship-type injection across the Neptune, Neo4j, and FalkorDB backends (GHSA-482h-hw99-h62p, #910) and the triplet-store SPARQL injection across Blazegraph/RDF4J/Jena (GHSA-8vgg-8mr4-r236, #911) are covered by separate, still-open PRs, as is the unauthenticated-Explorer-API fix referenced above (GHSA-j4mq-hprp-987v, #909, already merged)
|
||||
|
||||
- **CI/CD supply-chain hardening against mutable-tag Action compromise (LiteLLM/Trivy-class attack)** (#824) by @KaifAhmad1
|
||||
- Every third-party GitHub Action across all 8 workflows is now pinned to a full commit SHA instead of a mutable tag (`@v7` → `@3d3c42e... # v7`), closing the exact vector used against LiteLLM in March 2026 (a compromised Trivy Action tag stole a long-lived publishing token)
|
||||
- Added `verify-action-pins.yml` + `.github/scripts/verify-action-pins.sh`: a CI check that fails closed on any `uses:` reference that isn't a full SHA (catching a newly introduced mutable tag, not just auditing existing pins) and re-verifies every pin against the GitHub API on each workflow change, on push to `main`, and weekly; an unresolvable API lookup is treated as a failure rather than a silent skip
|
||||
|
||||
@@ -132,7 +132,7 @@ compliant = graph.check_decision_rules({"category": "vendor_selection"}) # poli
|
||||
```bash
|
||||
semantica doctor
|
||||
# Python 3.11.9 pass
|
||||
# semantica 0.6.0 pass
|
||||
# semantica 0.6.5 pass
|
||||
# faiss vector store pass
|
||||
# Config file pass ~/.semantica/config.yaml
|
||||
```
|
||||
@@ -1474,12 +1474,18 @@ For contributor / dev-server setup: **[explorer/README.md: Local Setup Guide](ex
|
||||
|
||||
---
|
||||
|
||||
## What's New in v0.6.0
|
||||
## What's New in v0.6.5
|
||||
|
||||
- **Named-Graph Support for `JenaStore`:** Migrated onto `rdflib.Dataset(default_union=False)`, completing cross-backend named-graph parity across Blazegraph, RDF4J, and Jena; `add_triplets()` gains a `graph=` option
|
||||
- **SPARQL CONSTRUCT Query Templates:** Parameterized, injection-safe `CONSTRUCT` templates extended from Blazegraph-only to RDF4J and Jena, plus pipeline integration via the `construct_template` step type
|
||||
- **Databricks Connector:** `DatabricksIngestor` for Unity Catalog + Delta Lake ingestion, with PAT/OAuth M2M auth, table/query ingestion, and catalog/schema/table/lineage introspection. Install with `pip install "semantica[db-databricks]"`
|
||||
- **SQLite Vector Store Backend:** `SQLiteVecStore`, a disk-backed local vector store on `sqlite-vec`'s `vec0` virtual tables, with Cosine/L2 metrics, metadata filtering, and WAL mode. Install with `pip install semantica[vectorstore-sqlite]`
|
||||
**Security release — upgrading is strongly recommended.** Fixes for 5 externally-reported vulnerabilities in the Explorer API and graph/triplet store backends, plus a CodeQL-flagged ReDoS:
|
||||
|
||||
- **Missing authentication on all Explorer API routes** (GHSA-j4mq-hprp-987v, Critical): every route now requires `SEMANTICA_API_KEY`, fails closed (503) rather than open when unconfigured
|
||||
- **SSRF via redirect bypass in ontology URL fetching** (GHSA-8c7v-62gr-hj6g, High): redirect targets are now re-validated at every hop and the connection is pinned to the validated address, closing a DNS check-then-use race
|
||||
- **Cypher injection via unvalidated node labels and property keys** (GHSA-482h-hw99-h62p, Critical): Neptune, Neo4j, and FalkorDB now sanitize every label/relationship-type/property-key interpolation site
|
||||
- **SPARQL injection via unvalidated triplet IRIs** (GHSA-8vgg-8mr4-r236, Critical): Blazegraph, RDF4J, and Jena now validate subject/predicate/object IRIs before interpolation
|
||||
- **Missing Origin validation on the WebSocket handshake** (GHSA-4643-wpgq-w329, Moderate, anonymous-mode only): `/ws/graph-updates` now checks `Origin` against the same allowlist `CORSMiddleware` enforces for HTTP
|
||||
- **Polynomial ReDoS in SPARQL query validation** (CodeQL `py/polynomial-redos`): fixed a backtracking regex in the Explorer's SPARQL route
|
||||
|
||||
Also includes: embedded Oxigraph backend for `TripletStore`, PROV-O trust/spec completeness for `ProvenanceManager`, and the Altair Anzo triplet store backend.
|
||||
|
||||
→ [Full release notes](RELEASE_NOTES.md) · [Changelog](CHANGELOG.md)
|
||||
|
||||
|
||||
+8
-8
@@ -13,33 +13,33 @@ icon: "quote-left"
|
||||
<Tab title="BibTeX">
|
||||
```bibtex
|
||||
@software{semantica2026,
|
||||
title = {Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering},
|
||||
author = {Hawksight AI},
|
||||
title = {Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems},
|
||||
author = {Semantica},
|
||||
year = {2026},
|
||||
url = {https://github.com/semantica-agi/semantica},
|
||||
version = {0.6.0},
|
||||
version = {0.6.5},
|
||||
doi = {10.5281/zenodo.XXXXXXX}
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="APA">
|
||||
Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.6.0) \[Computer software\]. https://github.com/semantica-agi/semantica
|
||||
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* (Version 0.6.5) \[Computer software\]. https://github.com/semantica-agi/semantica
|
||||
</Tab>
|
||||
<Tab title="MLA">
|
||||
Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.6.0, GitHub, 2026, https://github.com/semantica-agi/semantica.
|
||||
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.5, GitHub, 2026, https://github.com/semantica-agi/semantica.
|
||||
</Tab>
|
||||
<Tab title="Chicago">
|
||||
Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.6.0. GitHub, 2026. https://github.com/semantica-agi/semantica.
|
||||
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.5. GitHub, 2026. https://github.com/semantica-agi/semantica.
|
||||
</Tab>
|
||||
<Tab title="IEEE">
|
||||
Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.6.0, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
|
||||
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," Version 0.6.5, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Acknowledgment Text
|
||||
|
||||
> "This work uses Semantica (Hawksight AI, 2026), an open-source framework for semantic layer construction and knowledge engineering."
|
||||
> "This work uses Semantica (2026), an open-source graph-native infrastructure framework for context and accountable AI systems, providing Context Graphs, knowledge graphs, and full decision provenance."
|
||||
|
||||
|
||||
## Share Your Research
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ icon: "circle-question"
|
||||
| API key required? | Optional: pattern extraction works with no keys |
|
||||
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
|
||||
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
|
||||
| Latest version? | **v0.6.0** (July 2026) |
|
||||
| Latest version? | **v0.6.5** (August 2026) |
|
||||
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
|
||||
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ icon: "rocket"
|
||||
Verify installation:
|
||||
```python
|
||||
import semantica
|
||||
print(semantica.__version__) # 0.6.0
|
||||
print(semantica.__version__) # 0.6.5
|
||||
```
|
||||
</Check>
|
||||
</Step>
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "semantica"
|
||||
version = "0.6.0"
|
||||
version = "0.6.5"
|
||||
description = "Accountability and context layer for AI agents. Context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
|
||||
readme = "README.md"
|
||||
license = { text = "MIT" }
|
||||
|
||||
@@ -10,7 +10,7 @@ Main exports:
|
||||
- Config: Configuration management
|
||||
"""
|
||||
|
||||
__version__ = "0.6.0"
|
||||
__version__ = "0.6.5"
|
||||
__author__ = "Semantica Contributors"
|
||||
__license__ = "MIT"
|
||||
|
||||
|
||||
+59
-21
@@ -8,16 +8,16 @@ from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request, WebSocket, WebSocketDisconnect
|
||||
from fastapi import Depends, FastAPI, HTTPException, Request, WebSocket, WebSocketDisconnect
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import FileResponse, HTMLResponse, JSONResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
|
||||
from .. import __version__
|
||||
from ..context.context_graph import ContextGraph
|
||||
from .dependencies import anonymous_access_allowed, get_expected_api_key, is_valid_api_key, require_auth
|
||||
from .session import GraphSession
|
||||
from .ws import ConnectionManager
|
||||
from .auth import APIKeyAuthMiddleware, warn_if_unauthenticated
|
||||
|
||||
|
||||
def _read_int_env(name: str, default: int) -> int:
|
||||
@@ -98,6 +98,22 @@ def create_app(
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
import logging as _lifespan_logging
|
||||
_lifespan_logger = _lifespan_logging.getLogger(__name__)
|
||||
if anonymous_access_allowed():
|
||||
_lifespan_logger.warning(
|
||||
"Explorer is running with SEMANTICA_ALLOW_ANONYMOUS=true — "
|
||||
"all API routes are unauthenticated. Do not expose this "
|
||||
"process beyond localhost."
|
||||
)
|
||||
elif get_expected_api_key():
|
||||
_lifespan_logger.info("Explorer API authentication: enabled (SEMANTICA_API_KEY set).")
|
||||
else:
|
||||
_lifespan_logger.warning(
|
||||
"Explorer API authentication: NOT CONFIGURED. All protected "
|
||||
"routes will return 503 until SEMANTICA_API_KEY is set."
|
||||
)
|
||||
|
||||
app.state.event_loop = asyncio.get_running_loop()
|
||||
app.state.ws_manager = ConnectionManager()
|
||||
app.state.session = active_session
|
||||
@@ -114,10 +130,10 @@ def create_app(
|
||||
app.state.explorer_settings = settings
|
||||
|
||||
# allow_credentials lets browsers send cookies/auth headers cross-origin.
|
||||
# The Explorer has no authentication, so credentials serve no purpose and
|
||||
# enabling them when origins are broadened creates cross-site request risk.
|
||||
# Set EXPLORER_CORS_CREDENTIALS=true explicitly to opt in (e.g. for a
|
||||
# reverse-proxy setup that injects its own auth layer).
|
||||
# Credentials aren't needed for the X-API-Key auth scheme below, and
|
||||
# enabling them when origins are broadened creates cross-site request
|
||||
# risk. Set EXPLORER_CORS_CREDENTIALS=true explicitly to opt in (e.g.
|
||||
# for a reverse-proxy setup that injects its own cookie-based auth).
|
||||
_allow_credentials = os.environ.get("EXPLORER_CORS_CREDENTIALS", "false").lower() == "true"
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
@@ -128,10 +144,6 @@ def create_app(
|
||||
max_age=600,
|
||||
)
|
||||
|
||||
# API key authentication (opt-in via EXPLORER_API_KEY env var)
|
||||
app.add_middleware(APIKeyAuthMiddleware)
|
||||
warn_if_unauthenticated()
|
||||
|
||||
import logging as _logging
|
||||
_logger = _logging.getLogger(__name__)
|
||||
|
||||
@@ -164,22 +176,48 @@ def create_app(
|
||||
from .routes.temporal import router as temporal_router
|
||||
from .routes.vocabulary import router as vocabulary_router
|
||||
|
||||
app.include_router(graph_router)
|
||||
app.include_router(analytics_router)
|
||||
app.include_router(decisions_router)
|
||||
app.include_router(temporal_router)
|
||||
app.include_router(enrich_router)
|
||||
app.include_router(export_import_router)
|
||||
app.include_router(annotations_router)
|
||||
app.include_router(sparql_router)
|
||||
app.include_router(provenance_router)
|
||||
app.include_router(vocabulary_router)
|
||||
app.include_router(ontology_router)
|
||||
_auth = [Depends(require_auth)]
|
||||
app.include_router(graph_router, dependencies=_auth)
|
||||
app.include_router(analytics_router, dependencies=_auth)
|
||||
app.include_router(decisions_router, dependencies=_auth)
|
||||
app.include_router(temporal_router, dependencies=_auth)
|
||||
app.include_router(enrich_router, dependencies=_auth)
|
||||
app.include_router(export_import_router, dependencies=_auth)
|
||||
app.include_router(annotations_router, dependencies=_auth)
|
||||
app.include_router(sparql_router, dependencies=_auth)
|
||||
app.include_router(provenance_router, dependencies=_auth)
|
||||
app.include_router(vocabulary_router, dependencies=_auth)
|
||||
app.include_router(ontology_router, dependencies=_auth)
|
||||
|
||||
_WS_MAX_MESSAGE_BYTES = 64 * 1024 # 64 KB — control messages only
|
||||
|
||||
@app.websocket("/ws/graph-updates")
|
||||
async def websocket_endpoint(websocket: WebSocket):
|
||||
# CORSMiddleware doesn't cover WebSocket handshakes (Starlette's
|
||||
# CORS support only wraps HTTP), so under SEMANTICA_ALLOW_ANONYMOUS
|
||||
# the key check below accepts any origin — loopback binding isn't a
|
||||
# boundary against a browser, since any page the operator has open
|
||||
# can still reach ws://localhost:.../ws/graph-updates directly.
|
||||
# Reject a foreign Origin explicitly here, against the same
|
||||
# allowlist CORSMiddleware already enforces for HTTP
|
||||
# (GHSA-4643-wpgq-w329). Browsers always send Origin on a
|
||||
# cross-origin WebSocket handshake; native/CLI clients omit it
|
||||
# entirely, so a missing Origin is allowed through — the browser is
|
||||
# the only threat this check is closing.
|
||||
origin = websocket.headers.get("origin")
|
||||
allowed_origins = app.state.explorer_settings["allowed_origins"]
|
||||
if origin is not None and origin not in allowed_origins:
|
||||
await websocket.close(code=4403) # forbidden
|
||||
return
|
||||
|
||||
# Browsers can't set custom headers on a WebSocket handshake, so
|
||||
# accept the key via header (non-browser clients) or query param
|
||||
# (browser clients), same SEMANTICA_API_KEY the REST routes check.
|
||||
candidate = websocket.headers.get("x-api-key") or websocket.query_params.get("api_key")
|
||||
if not is_valid_api_key(candidate):
|
||||
await websocket.close(code=4401) # unauthorized
|
||||
return
|
||||
|
||||
manager: ConnectionManager = app.state.ws_manager
|
||||
await manager.connect(websocket)
|
||||
await manager.send_personal(websocket, "connection_ack", {"connected": True})
|
||||
|
||||
@@ -1,102 +0,0 @@
|
||||
"""
|
||||
Semantica Explorer : Authentication Middleware
|
||||
|
||||
Provides opt-in API key authentication for all Explorer API routes.
|
||||
|
||||
Enable by setting the ``EXPLORER_API_KEY`` environment variable. When set,
|
||||
every request to ``/api/*`` must include either:
|
||||
|
||||
- An ``Authorization: Bearer <key>`` header, or
|
||||
- An ``X-API-Key: <key>`` header.
|
||||
|
||||
When ``EXPLORER_API_KEY`` is not set, authentication is disabled and the
|
||||
Explorer operates in open/development mode (with a startup warning).
|
||||
"""
|
||||
|
||||
import hmac
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import HTTPException, Request, status
|
||||
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
|
||||
from starlette.responses import Response
|
||||
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _get_api_key() -> Optional[str]:
|
||||
"""Read the configured API key from the environment."""
|
||||
return os.environ.get("EXPLORER_API_KEY")
|
||||
|
||||
|
||||
def _extract_token(request: Request) -> Optional[str]:
|
||||
"""Extract the API key from the request headers."""
|
||||
# Check Authorization: Bearer <key>
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
if auth_header.startswith("Bearer "):
|
||||
return auth_header[7:].strip()
|
||||
|
||||
# Check X-API-Key: <key>
|
||||
api_key_header = request.headers.get("X-API-Key", "")
|
||||
if api_key_header:
|
||||
return api_key_header.strip()
|
||||
|
||||
return None
|
||||
|
||||
|
||||
class APIKeyAuthMiddleware(BaseHTTPMiddleware):
|
||||
"""
|
||||
Middleware that enforces API key authentication on ``/api/*`` routes.
|
||||
|
||||
Skips authentication for:
|
||||
- Non-API routes (static files, health checks, WebSocket, docs)
|
||||
- OPTIONS requests (CORS preflight)
|
||||
- When ``EXPLORER_API_KEY`` is not configured (open mode)
|
||||
"""
|
||||
|
||||
async def dispatch(
|
||||
self, request: Request, call_next: RequestResponseEndpoint
|
||||
) -> Response:
|
||||
api_key = _get_api_key()
|
||||
|
||||
# If no API key is configured, allow all requests (open mode)
|
||||
if not api_key:
|
||||
return await call_next(request)
|
||||
|
||||
# Skip authentication for non-API paths
|
||||
path = request.url.path
|
||||
if not path.startswith("/api/"):
|
||||
return await call_next(request)
|
||||
|
||||
# Skip CORS preflight
|
||||
if request.method == "OPTIONS":
|
||||
return await call_next(request)
|
||||
|
||||
# Validate the token
|
||||
token = _extract_token(request)
|
||||
if not token:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Missing API key. Provide via 'Authorization: Bearer <key>' or 'X-API-Key: <key>' header.",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
# Constant-time comparison to prevent timing attacks
|
||||
if not hmac.compare_digest(token, api_key):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Invalid API key.",
|
||||
)
|
||||
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
def warn_if_unauthenticated() -> None:
|
||||
"""Log a warning at startup if no API key is configured."""
|
||||
if not _get_api_key():
|
||||
_logger.warning(
|
||||
"EXPLORER_API_KEY is not set. The Explorer API is running WITHOUT "
|
||||
"authentication. Set EXPLORER_API_KEY to enable API key protection "
|
||||
"for all /api/* endpoints."
|
||||
)
|
||||
@@ -978,28 +978,18 @@ def _normalize_format(fmt: Optional[str]) -> str:
|
||||
return _FORMAT_ALIASES.get(lower, lower)
|
||||
|
||||
|
||||
import threading
|
||||
import urllib3.util.connection
|
||||
def _validate_fetch_url(url: str) -> List[str]:
|
||||
"""Reject non-HTTP(S) schemes and private/loopback/link-local targets.
|
||||
|
||||
if not hasattr(urllib3.util.connection, "_orig_create_connection"):
|
||||
urllib3.util.connection._orig_create_connection = urllib3.util.connection.create_connection
|
||||
|
||||
_dns_pin_tls = threading.local()
|
||||
|
||||
def _patched_create_connection(address, *args, **kwargs):
|
||||
host, port = address
|
||||
pinned_host = getattr(_dns_pin_tls, 'pinned_host', None)
|
||||
pinned_ip = getattr(_dns_pin_tls, 'pinned_ip', None)
|
||||
|
||||
if pinned_host and pinned_ip and host == pinned_host:
|
||||
return urllib3.util.connection._orig_create_connection((pinned_ip, port), *args, **kwargs)
|
||||
return urllib3.util.connection._orig_create_connection(address, *args, **kwargs)
|
||||
|
||||
urllib3.util.connection.create_connection = _patched_create_connection
|
||||
|
||||
|
||||
def _validate_fetch_url(url: str) -> str:
|
||||
"""Reject non-HTTP(S) schemes and private/loopback/link-local targets."""
|
||||
Returns every resolved, validated IP address (deduplicated, in
|
||||
resolution order) so the caller can pin the actual connection to them
|
||||
(see _make_pinned_session) with fallback across all of them — not just
|
||||
the first — since a hostname can have multiple A/AAAA records and the
|
||||
first one isn't guaranteed reachable. Resolving the hostname again at
|
||||
connect time would open a DNS check-then-use window (a low-TTL or
|
||||
rebinding DNS answer could differ between this check and the client's
|
||||
own lookup), which is what pinning to these specific addresses avoids.
|
||||
"""
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in ("http", "https"):
|
||||
raise HTTPException(status_code=422, detail="Only http and https URLs are allowed.")
|
||||
@@ -1010,7 +1000,7 @@ def _validate_fetch_url(url: str) -> str:
|
||||
addrinfos = socket.getaddrinfo(hostname, None)
|
||||
except socket.gaierror as exc:
|
||||
raise HTTPException(status_code=422, detail=f"Cannot resolve hostname '{hostname}': {exc}") from exc
|
||||
safe_ips = []
|
||||
validated_ips: List[str] = []
|
||||
for _family, _type, _proto, _canonname, sockaddr in addrinfos:
|
||||
try:
|
||||
ip = ipaddress.ip_address(sockaddr[0])
|
||||
@@ -1021,56 +1011,160 @@ def _validate_fetch_url(url: str) -> str:
|
||||
status_code=422,
|
||||
detail="Fetching from private, loopback, or reserved network addresses is not allowed.",
|
||||
)
|
||||
safe_ips.append(str(ip))
|
||||
if not safe_ips:
|
||||
raise HTTPException(status_code=422, detail="Could not resolve to a valid IP address.")
|
||||
return safe_ips[0]
|
||||
if sockaddr[0] not in validated_ips:
|
||||
validated_ips.append(sockaddr[0])
|
||||
if not validated_ips:
|
||||
raise HTTPException(status_code=422, detail=f"Cannot resolve hostname '{hostname}' to a usable address.")
|
||||
return validated_ips
|
||||
|
||||
|
||||
def _make_pinned_session(pinned_ips: List[str], url: str):
|
||||
"""Build a requests.Session whose connection is pinned to pinned_ips
|
||||
(tried in order, falling back on connection failure), regardless of
|
||||
what url's hostname resolves to at connect time.
|
||||
|
||||
_validate_fetch_url() resolves and validates the hostname once; letting
|
||||
the HTTP client resolve it again independently at connect time reopens
|
||||
the exact gap that validation exists to close — a low-TTL or rebinding
|
||||
DNS answer can differ between the two lookups. This pins the pool's
|
||||
connect target to the already-validated addresses directly (bypassing
|
||||
DNS resolution for the connection entirely), while keeping the original
|
||||
hostname as the outgoing HTTP Host header and, for HTTPS, the TLS SNI
|
||||
server_hostname / assert_hostname — otherwise the connection would
|
||||
reach the right IP but present the wrong identity, breaking name-based
|
||||
virtual hosting and (for HTTPS) certificate hostname verification.
|
||||
|
||||
Falls back across every validated address (not just the first) so a
|
||||
hostname with multiple A/AAAA records doesn't fail outright just
|
||||
because the first-returned address happens to be unreachable.
|
||||
|
||||
Note: urllib3's Connection.host is a property that reads/writes the
|
||||
same underlying value as `_dns_host` in this version — it is NOT the
|
||||
separate "presented identity" field it is in some older releases, so
|
||||
overriding just `_dns_host` post-construction (as an earlier version of
|
||||
this fix did) actually changes the Host header too. Pinning the pool's
|
||||
`host` directly and restoring the real hostname via an explicit Host
|
||||
header (+ SNI params for HTTPS) is the correct mechanism here.
|
||||
"""
|
||||
import requests as _req
|
||||
import urllib3.util.connection as _u3_connection
|
||||
from urllib3.exceptions import NewConnectionError
|
||||
|
||||
parsed = urlparse(url)
|
||||
hostname = parsed.hostname
|
||||
port = parsed.port
|
||||
default_port = 443 if parsed.scheme == "https" else 80
|
||||
host_header = hostname if port in (None, default_port) else f"{hostname}:{port}"
|
||||
|
||||
class _MultiIPConnectionMixin:
|
||||
"""Overrides _new_conn to fall back across every pinned IP in
|
||||
order, instead of urllib3's default single-host connect."""
|
||||
|
||||
def _new_conn(self):
|
||||
last_exc: Optional[BaseException] = None
|
||||
for ip in pinned_ips:
|
||||
try:
|
||||
return _u3_connection.create_connection(
|
||||
(ip, self.port),
|
||||
self.timeout,
|
||||
source_address=self.source_address,
|
||||
socket_options=self.socket_options,
|
||||
)
|
||||
except OSError as exc:
|
||||
last_exc = exc
|
||||
continue
|
||||
raise NewConnectionError(
|
||||
self, f"Failed to establish a connection to any of {pinned_ips}: {last_exc}"
|
||||
)
|
||||
|
||||
class _PinnedIPHTTPAdapter(_req.adapters.HTTPAdapter):
|
||||
def get_connection_with_tls_context(self, request, verify, proxies=None, cert=None):
|
||||
# A proxy would perform its own DNS resolution of the target
|
||||
# host on this process's behalf — a resolution outside this
|
||||
# process's visibility or control, so there is no client-side
|
||||
# pin that closes that race. Proxies are disabled outright for
|
||||
# this SSRF-sensitive fetcher (session.trust_env=False below),
|
||||
# so this should be unreachable via environment proxies; fail
|
||||
# closed rather than silently skip pinning if a proxy is
|
||||
# somehow still configured (e.g. passed explicitly in the
|
||||
# future). _validate_fetch_url's destination classification is
|
||||
# a separate, always-enforced check — this only guards the
|
||||
# secondary DNS-pinning hardening.
|
||||
if _req.utils.select_proxy(request.url, proxies):
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail="Proxied requests are not supported for ontology URL fetching.",
|
||||
)
|
||||
host_params, pool_kwargs = self.build_connection_pool_key_attributes(request, verify, cert)
|
||||
if host_params.get("scheme") == "https":
|
||||
pool_kwargs.setdefault("assert_hostname", hostname)
|
||||
pool_kwargs.setdefault("server_hostname", hostname)
|
||||
host_params["host"] = pinned_ips[0]
|
||||
pool = self.poolmanager.connection_from_host(**host_params, pool_kwargs=pool_kwargs)
|
||||
base_connection_cls = pool.ConnectionCls
|
||||
if not issubclass(base_connection_cls, _MultiIPConnectionMixin):
|
||||
pool.ConnectionCls = type(
|
||||
"_PinnedConnection", (_MultiIPConnectionMixin, base_connection_cls), {}
|
||||
)
|
||||
return pool
|
||||
|
||||
session = _req.Session()
|
||||
# Never honor HTTP_PROXY/HTTPS_PROXY/NO_PROXY env vars for this
|
||||
# SSRF-sensitive fetcher: a configured proxy would perform its own DNS
|
||||
# resolution of the target host outside this process's control,
|
||||
# silently reopening the DNS check-then-use race pinning exists to
|
||||
# close. See _PinnedIPHTTPAdapter.get_connection_with_tls_context for
|
||||
# the fail-closed backstop if a proxy is somehow still configured.
|
||||
session.trust_env = False
|
||||
session.headers["Host"] = host_header
|
||||
adapter = _PinnedIPHTTPAdapter()
|
||||
session.mount("http://", adapter)
|
||||
session.mount("https://", adapter)
|
||||
return session
|
||||
|
||||
|
||||
def _fetch_url_sync(url: str) -> bytes:
|
||||
import requests as _req
|
||||
pinned_ips = _validate_fetch_url(url)
|
||||
_MAX_REDIRECTS = 5
|
||||
current_url = url
|
||||
try:
|
||||
for _ in range(_MAX_REDIRECTS + 1):
|
||||
safe_ip = _validate_fetch_url(current_url)
|
||||
_dns_pin_tls.pinned_host = urlparse(current_url).hostname
|
||||
_dns_pin_tls.pinned_ip = safe_ip
|
||||
session = _make_pinned_session(pinned_ips, current_url)
|
||||
try:
|
||||
resp = _req.get(
|
||||
resp = session.get(
|
||||
current_url,
|
||||
headers={"Accept": "text/turtle, application/rdf+xml, application/ld+json, */*;q=0.1"},
|
||||
timeout=30,
|
||||
stream=True,
|
||||
allow_redirects=False, # SECURITY: follow redirects manually
|
||||
)
|
||||
if resp.is_redirect or resp.is_permanent_redirect:
|
||||
redirect_url = resp.headers.get("Location")
|
||||
resp.close() # Release the streamed connection before following the redirect
|
||||
if not redirect_url:
|
||||
raise HTTPException(status_code=502, detail="Redirect without Location header.")
|
||||
# Resolve relative redirects (e.g. /ontology.ttl) against the current URL
|
||||
redirect_url = urljoin(current_url, redirect_url)
|
||||
# Re-validate the redirect target to prevent SSRF via
|
||||
# open-redirect to internal/cloud-metadata endpoints, and
|
||||
# get fresh pins for the new host.
|
||||
pinned_ips = _validate_fetch_url(redirect_url)
|
||||
current_url = redirect_url
|
||||
continue
|
||||
try:
|
||||
resp.raise_for_status()
|
||||
chunks: List[bytes] = []
|
||||
total = 0
|
||||
for chunk in resp.iter_content(65536):
|
||||
total += len(chunk)
|
||||
if total > _MAX_FETCH_BYTES:
|
||||
raise HTTPException(status_code=413, detail="Remote resource exceeds 20 MB limit.")
|
||||
chunks.append(chunk)
|
||||
return b"".join(chunks)
|
||||
finally:
|
||||
resp.close() # Release the streamed connection once fully read (or on error)
|
||||
finally:
|
||||
_dns_pin_tls.pinned_host = None
|
||||
_dns_pin_tls.pinned_ip = None
|
||||
if resp.is_redirect or resp.is_permanent_redirect:
|
||||
redirect_url = resp.headers.get("Location")
|
||||
resp.close() # Release the streamed connection before following the redirect
|
||||
if not redirect_url:
|
||||
raise HTTPException(status_code=502, detail="Redirect without Location header.")
|
||||
# Resolve relative redirects (e.g. /ontology.ttl) against the current URL
|
||||
redirect_url = urljoin(current_url, redirect_url)
|
||||
# Re-validate the redirect target to prevent SSRF via
|
||||
# open-redirect to internal/cloud-metadata endpoints.
|
||||
_validate_fetch_url(redirect_url)
|
||||
current_url = redirect_url
|
||||
continue
|
||||
try:
|
||||
resp.raise_for_status()
|
||||
chunks: List[bytes] = []
|
||||
total = 0
|
||||
for chunk in resp.iter_content(65536):
|
||||
total += len(chunk)
|
||||
if total > _MAX_FETCH_BYTES:
|
||||
raise HTTPException(status_code=413, detail="Remote resource exceeds 20 MB limit.")
|
||||
chunks.append(chunk)
|
||||
return b"".join(chunks)
|
||||
finally:
|
||||
resp.close() # Release the streamed connection once fully read (or on error)
|
||||
session.close()
|
||||
raise HTTPException(status_code=502, detail=f"Too many redirects (max {_MAX_REDIRECTS}).")
|
||||
except HTTPException:
|
||||
raise
|
||||
|
||||
@@ -3,11 +3,16 @@ SPARQL routes backed by an in-memory rdflib projection of the current graph.
|
||||
|
||||
Security contract
|
||||
-----------------
|
||||
* Only SELECT, ASK, CONSTRUCT, and DESCRIBE are accepted (allowlist enforced
|
||||
before graph construction so rejected queries never touch the session).
|
||||
* Multi-statement injections that start with an allowed keyword (e.g.
|
||||
``SELECT ... ; DROP ALL``) pass the prefix check and reach rdflib, which
|
||||
rejects non-SELECT/ASK/CONSTRUCT/DESCRIBE update syntax in the parser.
|
||||
* Only SELECT, ASK, CONSTRUCT, and DESCRIBE are accepted, and the query
|
||||
body is scanned for SPARQL Update keywords (INSERT/DELETE/DROP/LOAD/
|
||||
CLEAR/CREATE/COPY/MOVE/ADD) after stripping comments and PREFIX/BASE
|
||||
declarations — both enforced before graph construction, so rejected
|
||||
queries never touch the session. A multi-statement injection appended
|
||||
after an allowed keyword (e.g. ``SELECT ... ; DROP ALL``) is caught by
|
||||
the keyword scan itself, not left to rdflib's parser.
|
||||
* rdflib's parser remains a second line of defense for malformed multi-
|
||||
statement syntax that doesn't contain any forbidden keyword (e.g.
|
||||
``SELECT ... ; ASK ...``), which SPARQL 1.1 Query doesn't permit.
|
||||
* The in-memory rdflib graph is a read-only projection — the live
|
||||
``GraphSession`` is never mutated by this route.
|
||||
"""
|
||||
@@ -38,9 +43,34 @@ _FORBIDDEN_KEYWORDS = re.compile(
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Matches SPARQL single-line comments (# ...) and PREFIX declarations
|
||||
_COMMENT_LINE = re.compile(r"#[^\n]*", re.MULTILINE)
|
||||
_PREFIX_DECL = re.compile(r"^\s*(?:PREFIX|BASE)\s+(?:\S+\s+)?<[^>]*>\s*", re.IGNORECASE | re.MULTILINE)
|
||||
# Matches SPARQL single-line comments (# ...) and PREFIX/BASE declarations.
|
||||
# The comment regex only treats '#' as a comment-starter at line-start or
|
||||
# after whitespace — not mid-token — since RDF namespace IRIs commonly
|
||||
# contain a literal '#' (e.g. ".../1999/02/22-rdf-syntax-ns#"), and a naive
|
||||
# `#[^\n]*` would truncate every such PREFIX declaration's IRI, corrupting
|
||||
# the query. BASE declarations have no prefix name between the keyword and
|
||||
# the IRI (`BASE <...>`, vs. `PREFIX ex: <...>`), so the prefix-name token
|
||||
# is optional.
|
||||
#
|
||||
# ReDoS fix (CodeQL py/polynomial-redos, issue #1897):
|
||||
#
|
||||
# The original pattern `<[^>]*>\s*` was vulnerable because `\s*` (which
|
||||
# matches newlines) could overlap with `[^>]*` on inputs that contain no
|
||||
# closing `>` (e.g. `base<!!<!<...`), forcing the engine to explore every
|
||||
# possible split between the two quantifiers — O(n²) backtracking.
|
||||
#
|
||||
# The fix uses `<[^>\r\n]*>` for the IRI body: excluding CR and LF from
|
||||
# the character class means the IRI match can never span a line boundary,
|
||||
# and the disjoint trailing `[ \t]*` (horizontal whitespace only) has zero
|
||||
# character-class overlap with `[^>\r\n]*`, so the engine has exactly one
|
||||
# way to match. No end-of-line anchor is needed or used, which correctly
|
||||
# handles both inline prologues (`PREFIX ex: <...> SELECT ...` on one line)
|
||||
# and CRLF line endings (`\r\n`) without any special casing.
|
||||
_COMMENT_LINE = re.compile(r"(?:^|(?<=\s))#[^\n]*", re.MULTILINE)
|
||||
_PREFIX_DECL = re.compile(
|
||||
r"^[ \t]*(?:PREFIX[ \t]+\S+|BASE)[ \t]*<[^>\r\n]*>[ \t]*",
|
||||
re.IGNORECASE | re.MULTILINE,
|
||||
)
|
||||
|
||||
|
||||
def _is_read_only_query(query: str) -> bool:
|
||||
@@ -50,11 +80,15 @@ def _is_read_only_query(query: str) -> bool:
|
||||
checking the first keyword. Also rejects queries containing SPARQL Update
|
||||
keywords anywhere in the body, preventing injection via embedded strings
|
||||
or multi-statement tricks.
|
||||
|
||||
Note: callers are responsible for enforcing any input-length limit *before*
|
||||
calling this function so that an oversized-query rejection can be surfaced
|
||||
as a distinct, actionable error rather than the generic read-only message.
|
||||
"""
|
||||
# 1. Remove PREFIX/BASE declarations (do this first so # in URIs aren't mangled by comment stripping)
|
||||
cleaned = _PREFIX_DECL.sub("", query)
|
||||
# 2. Remove single-line comments that could hide the real query type
|
||||
cleaned = _COMMENT_LINE.sub("", cleaned)
|
||||
# 1. Remove single-line comments that could hide the real query type
|
||||
cleaned = _COMMENT_LINE.sub("", query)
|
||||
# 2. Remove PREFIX/BASE declarations
|
||||
cleaned = _PREFIX_DECL.sub("", cleaned)
|
||||
# 3. Strip remaining whitespace
|
||||
cleaned = cleaned.strip()
|
||||
|
||||
@@ -144,6 +178,11 @@ _SPARQL_MAX_ROWS = 5_000 # hard cap on returned rows
|
||||
_SPARQL_TIMEOUT_S = 30 # seconds before abandoning the await
|
||||
_SPARQL_MAX_CONCURRENT = 4 # semaphore: max simultaneous executions
|
||||
_SPARQL_MAX_GRAPH_NODES = 50_000 # cap on graph nodes/edges to prevent OOM
|
||||
# Defense-in-depth against ReDoS: reject inputs longer than this before any
|
||||
# regex work so that even a future regex regression is bounded. Checked in
|
||||
# execute_sparql() (not inside _is_read_only_query) so the route can return
|
||||
# a distinct, actionable error message rather than the generic read-only one.
|
||||
_SPARQL_MAX_QUERY_LEN = 10_000 # chars
|
||||
|
||||
# Semaphore caps how many graph.query calls run concurrently so that
|
||||
# timed-out threads (which keep running in the pool) cannot crowd out
|
||||
@@ -172,6 +211,24 @@ async def execute_sparql(
|
||||
req: SparqlRequest,
|
||||
session: GraphSession = Depends(get_session),
|
||||
):
|
||||
# Resource-limit check: reject oversized queries before any regex work.
|
||||
# This is intentionally a separate, earlier check from _is_read_only_query
|
||||
# so clients receive a specific, actionable message rather than the generic
|
||||
# read-only rejection, and operators can tune _SPARQL_MAX_QUERY_LEN without
|
||||
# touching query-semantics code.
|
||||
if len(req.query) > _SPARQL_MAX_QUERY_LEN:
|
||||
return SparqlResponse(
|
||||
columns=[],
|
||||
rows=[],
|
||||
total=0,
|
||||
error=(
|
||||
f"Query exceeds the maximum allowed length of "
|
||||
f"{_SPARQL_MAX_QUERY_LEN:,} characters "
|
||||
f"({len(req.query):,} received). "
|
||||
f"Please shorten your query."
|
||||
),
|
||||
)
|
||||
|
||||
if not _is_read_only_query(req.query):
|
||||
return SparqlResponse(
|
||||
columns=[],
|
||||
|
||||
@@ -47,6 +47,7 @@ from typing import Any, Dict, List, Optional, Union
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .query_sanitize import sanitize_identifier
|
||||
|
||||
# Optional boto3 for AWS credentials and SigV4 signing
|
||||
try:
|
||||
@@ -981,7 +982,8 @@ class AmazonNeptuneStore:
|
||||
node_id = props_copy.pop("id", None) or self._generate_id()
|
||||
use_merge = options.get("merge", True)
|
||||
|
||||
label_str = ":".join(labels) if labels else "Node"
|
||||
label_str = ":".join(sanitize_identifier(l, "label") for l in labels) if labels else "Node"
|
||||
safe_keys = [sanitize_identifier(k, "property key") for k in props_copy.keys()]
|
||||
|
||||
# Build parameters
|
||||
params = {"node_id": str(node_id)}
|
||||
@@ -990,9 +992,7 @@ class AmazonNeptuneStore:
|
||||
|
||||
if use_merge:
|
||||
# MERGE: Return existing node if ID matches, or create new
|
||||
set_parts = []
|
||||
for key in props_copy.keys():
|
||||
set_parts.append(f"n.{key} = ${key}")
|
||||
set_parts = [f"n.{key} = ${key}" for key in safe_keys]
|
||||
|
||||
if set_parts:
|
||||
set_clause = ", ".join(set_parts)
|
||||
@@ -1005,9 +1005,7 @@ class AmazonNeptuneStore:
|
||||
query = f"MERGE (n:{label_str} {{`~id`: $node_id}}) RETURN n"
|
||||
else:
|
||||
# CREATE: Will fail if node with same ID exists
|
||||
prop_parts = ["`~id`: $node_id"]
|
||||
for key in props_copy.keys():
|
||||
prop_parts.append(f"{key}: ${key}")
|
||||
prop_parts = ["`~id`: $node_id"] + [f"{key}: ${key}" for key in safe_keys]
|
||||
prop_assignments = ", ".join(prop_parts)
|
||||
query = f"CREATE (n:{label_str} {{{prop_assignments}}}) RETURN n"
|
||||
|
||||
@@ -1162,7 +1160,7 @@ class AmazonNeptuneStore:
|
||||
|
||||
# Build query
|
||||
if labels:
|
||||
label_str = ":".join(labels)
|
||||
label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
|
||||
query = f"MATCH (n:{label_str})"
|
||||
else:
|
||||
query = "MATCH (n)"
|
||||
@@ -1172,8 +1170,9 @@ class AmazonNeptuneStore:
|
||||
if properties:
|
||||
conditions = []
|
||||
for key, value in properties.items():
|
||||
param_key = f"prop_{key}"
|
||||
conditions.append(f"n.{key} = ${param_key}")
|
||||
safe_key = sanitize_identifier(key, "property key")
|
||||
param_key = f"prop_{safe_key}"
|
||||
conditions.append(f"n.{safe_key} = ${param_key}")
|
||||
params[param_key] = value
|
||||
query += " WHERE " + " AND ".join(conditions)
|
||||
|
||||
@@ -1341,15 +1340,16 @@ class AmazonNeptuneStore:
|
||||
}
|
||||
|
||||
# Build property assignments including ~id
|
||||
safe_rel_type = sanitize_identifier(rel_type, "relationship type")
|
||||
prop_parts = ["`~id`: $rel_id"]
|
||||
for key, value in props_copy.items():
|
||||
prop_parts.append(f"{key}: ${key}")
|
||||
prop_parts.append(f"{sanitize_identifier(key, 'property key')}: ${key}")
|
||||
params[key] = value
|
||||
|
||||
prop_assignments = ", ".join(prop_parts)
|
||||
query = (
|
||||
f"MATCH (a), (b) WHERE id(a) = $start_id AND id(b) = $end_id "
|
||||
f"CREATE (a)-[r:{rel_type} {{{prop_assignments}}}]->(b) RETURN r"
|
||||
f"CREATE (a)-[r:{safe_rel_type} {{{prop_assignments}}}]->(b) RETURN r"
|
||||
)
|
||||
|
||||
records = self._run_query(query, params)
|
||||
@@ -1405,7 +1405,7 @@ class AmazonNeptuneStore:
|
||||
try:
|
||||
self._ensure_connected()
|
||||
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
params = {}
|
||||
|
||||
if node_id is not None:
|
||||
@@ -1564,7 +1564,8 @@ class AmazonNeptuneStore:
|
||||
try:
|
||||
self._ensure_connected()
|
||||
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
depth = int(depth)
|
||||
|
||||
if direction == "out":
|
||||
pattern = f"-[r{type_filter}*1..{depth}]->"
|
||||
@@ -1634,7 +1635,7 @@ class AmazonNeptuneStore:
|
||||
try:
|
||||
self._ensure_connected()
|
||||
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
|
||||
# Neptune doesn't support named path patterns in shortestPath
|
||||
# Use iterative depth search instead
|
||||
|
||||
@@ -41,6 +41,7 @@ from typing import Any, Dict, List, Optional, Union
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .query_sanitize import sanitize_identifier
|
||||
|
||||
# Optional FalkorDB import
|
||||
try:
|
||||
@@ -330,11 +331,11 @@ class FalkorDBStore:
|
||||
try:
|
||||
graph = self._ensure_graph()
|
||||
|
||||
label_str = ":".join(labels)
|
||||
label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
|
||||
|
||||
# Build property string for Cypher
|
||||
props_str = ", ".join(
|
||||
f"{k}: ${k}" for k in properties.keys()
|
||||
f"{sanitize_identifier(k, 'property key')}: ${k}" for k in properties.keys()
|
||||
)
|
||||
|
||||
query = f"CREATE (n:{label_str} {{{props_str}}}) RETURN id(n) as id, n"
|
||||
@@ -392,9 +393,9 @@ class FalkorDBStore:
|
||||
labels = node.get("labels", [])
|
||||
properties = node.get("properties", {})
|
||||
|
||||
label_str = ":".join(labels) if labels else "Node"
|
||||
label_str = ":".join(sanitize_identifier(l, "label") for l in labels) if labels else "Node"
|
||||
props_str = ", ".join(
|
||||
f"{k}: ${k}" for k in properties.keys()
|
||||
f"{sanitize_identifier(k, 'property key')}: ${k}" for k in properties.keys()
|
||||
)
|
||||
|
||||
query = f"CREATE (n:{label_str} {{{props_str}}}) RETURN id(n) as id"
|
||||
@@ -447,7 +448,7 @@ class FalkorDBStore:
|
||||
|
||||
# Build query
|
||||
if labels:
|
||||
label_str = ":".join(labels)
|
||||
label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
|
||||
query = f"MATCH (n:{label_str})"
|
||||
else:
|
||||
query = "MATCH (n)"
|
||||
@@ -456,7 +457,8 @@ class FalkorDBStore:
|
||||
if properties:
|
||||
conditions = []
|
||||
for key in properties.keys():
|
||||
conditions.append(f"n.{key} = ${key}")
|
||||
safe_key = sanitize_identifier(key, "property key")
|
||||
conditions.append(f"n.{safe_key} = ${safe_key}")
|
||||
query += " WHERE " + " AND ".join(conditions)
|
||||
|
||||
query += f" RETURN id(n) as id, n, labels(n) as labels LIMIT {limit}"
|
||||
@@ -504,7 +506,8 @@ class FalkorDBStore:
|
||||
# Build SET clause
|
||||
set_parts = []
|
||||
for key in properties.keys():
|
||||
set_parts.append(f"n.{key} = ${key}")
|
||||
safe_key = sanitize_identifier(key, "property key")
|
||||
set_parts.append(f"n.{safe_key} = ${safe_key}")
|
||||
|
||||
if merge:
|
||||
query = f"MATCH (n) WHERE id(n) = $node_id SET {', '.join(set_parts)} RETURN id(n) as id, n, labels(n) as labels"
|
||||
@@ -592,9 +595,13 @@ class FalkorDBStore:
|
||||
graph = self._ensure_graph()
|
||||
properties = properties or {}
|
||||
|
||||
safe_rel_type = sanitize_identifier(rel_type, "relationship type")
|
||||
|
||||
# Build property string
|
||||
if properties:
|
||||
props_str = ", ".join(f"{k}: ${k}" for k in properties.keys())
|
||||
props_str = ", ".join(
|
||||
f"{sanitize_identifier(k, 'property key')}: ${k}" for k in properties.keys()
|
||||
)
|
||||
props_str = f" {{{props_str}}}"
|
||||
else:
|
||||
props_str = ""
|
||||
@@ -602,7 +609,7 @@ class FalkorDBStore:
|
||||
query = f"""
|
||||
MATCH (a), (b)
|
||||
WHERE id(a) = $start_id AND id(b) = $end_id
|
||||
CREATE (a)-[r:{rel_type}{props_str}]->(b)
|
||||
CREATE (a)-[r:{safe_rel_type}{props_str}]->(b)
|
||||
RETURN id(r) as id, type(r) as type
|
||||
"""
|
||||
|
||||
@@ -656,7 +663,7 @@ class FalkorDBStore:
|
||||
"""
|
||||
try:
|
||||
graph = self._ensure_graph()
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
|
||||
if node_id is not None:
|
||||
if direction == "out":
|
||||
@@ -806,7 +813,8 @@ class FalkorDBStore:
|
||||
"""
|
||||
try:
|
||||
graph = self._ensure_graph()
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
depth = int(depth)
|
||||
|
||||
if direction == "out":
|
||||
pattern = f"-[r{type_filter}*1..{depth}]->"
|
||||
@@ -860,7 +868,8 @@ class FalkorDBStore:
|
||||
"""
|
||||
try:
|
||||
graph = self._ensure_graph()
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
max_depth = int(max_depth)
|
||||
|
||||
query = f"""
|
||||
MATCH path = shortestPath((start)-[r{type_filter}*..{max_depth}]-(end))
|
||||
@@ -929,11 +938,13 @@ class FalkorDBStore:
|
||||
"""
|
||||
try:
|
||||
graph = self._ensure_graph()
|
||||
safe_label = sanitize_identifier(label, "label")
|
||||
safe_property = sanitize_identifier(property_name, "property key")
|
||||
|
||||
if index_type == "fulltext":
|
||||
query = f"CALL db.idx.fulltext.createNodeIndex('{label}', '{property_name}')"
|
||||
query = f"CALL db.idx.fulltext.createNodeIndex('{safe_label}', '{safe_property}')"
|
||||
else:
|
||||
query = f"CREATE INDEX FOR (n:{label}) ON (n.{property_name})"
|
||||
query = f"CREATE INDEX FOR (n:{safe_label}) ON (n.{safe_property})"
|
||||
|
||||
graph.query(query)
|
||||
self.logger.info(f"Created {index_type} index on {label}.{property_name}")
|
||||
|
||||
@@ -38,6 +38,7 @@ from ..utils.exceptions import ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .config import graph_store_config
|
||||
from .query_sanitize import sanitize_identifier
|
||||
|
||||
|
||||
class NodeManager:
|
||||
@@ -393,12 +394,12 @@ class GraphAnalytics:
|
||||
"""
|
||||
# Build query based on direction
|
||||
if labels:
|
||||
label_str = ":".join(labels)
|
||||
label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
|
||||
match = f"MATCH (n:{label_str})"
|
||||
else:
|
||||
match = "MATCH (n)"
|
||||
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
|
||||
if direction == "out":
|
||||
query = f"""
|
||||
@@ -756,7 +757,7 @@ class GraphStore:
|
||||
**options: Additional options
|
||||
"""
|
||||
# Support 'hops' as alias for 'depth' for ContextRetriever compatibility
|
||||
actual_depth = options.get("hops", depth)
|
||||
actual_depth = int(options.get("hops", depth))
|
||||
return self._manager.analytics.get_neighbors(
|
||||
node_id, rel_type, direction, actual_depth, **options
|
||||
)
|
||||
|
||||
@@ -62,6 +62,7 @@ from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
from .config import graph_store_config
|
||||
from .graph_store import GraphAnalytics, GraphStore, NodeManager, QueryEngine, RelationshipManager
|
||||
from .query_sanitize import sanitize_identifier
|
||||
from .registry import method_registry
|
||||
|
||||
# Global store instance
|
||||
@@ -357,7 +358,8 @@ def update_relationship(
|
||||
|
||||
# Default implementation - execute update query
|
||||
store = _get_store()
|
||||
set_parts = ", ".join(f"r.{k} = ${k}" for k in properties.keys())
|
||||
safe_keys = [sanitize_identifier(k, "property key") for k in properties.keys()]
|
||||
set_parts = ", ".join(f"r.{k} = ${k}" for k in safe_keys)
|
||||
query = f"MATCH ()-[r]->() WHERE id(r) = $rel_id SET {set_parts} RETURN id(r) as id, type(r) as type, r"
|
||||
params = {"rel_id": rel_id, **properties}
|
||||
result = store.execute_query(query, params)
|
||||
|
||||
@@ -38,6 +38,7 @@ from typing import Any, Dict, List, Optional, Union
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .query_sanitize import sanitize_identifier
|
||||
|
||||
# Optional Neo4j import
|
||||
try:
|
||||
@@ -366,7 +367,7 @@ class Neo4jStore:
|
||||
)
|
||||
|
||||
try:
|
||||
label_str = ":".join(labels)
|
||||
label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
|
||||
query = f"CREATE (n:{label_str} $props) RETURN id(n) as id, n"
|
||||
|
||||
with self.get_session() as session:
|
||||
@@ -424,7 +425,7 @@ class Neo4jStore:
|
||||
labels = node.get("labels", [])
|
||||
properties = node.get("properties", {})
|
||||
|
||||
label_str = ":".join(labels) if labels else "Node"
|
||||
label_str = ":".join(sanitize_identifier(l, "label") for l in labels) if labels else "Node"
|
||||
query = f"CREATE (n:{label_str} $props) RETURN id(n) as id, n"
|
||||
|
||||
result = session.run(query, {"props": properties})
|
||||
@@ -505,7 +506,7 @@ class Neo4jStore:
|
||||
try:
|
||||
# Build query
|
||||
if labels:
|
||||
label_str = ":".join(labels)
|
||||
label_str = ":".join(sanitize_identifier(l, "label") for l in labels)
|
||||
query = f"MATCH (n:{label_str})"
|
||||
else:
|
||||
query = "MATCH (n)"
|
||||
@@ -514,7 +515,8 @@ class Neo4jStore:
|
||||
if properties:
|
||||
conditions = []
|
||||
for key, value in properties.items():
|
||||
conditions.append(f"n.{key} = ${key}")
|
||||
safe_key = sanitize_identifier(key, "property key")
|
||||
conditions.append(f"n.{safe_key} = ${safe_key}")
|
||||
query += " WHERE " + " AND ".join(conditions)
|
||||
|
||||
query += f" RETURN id(n) as id, n, labels(n) as labels LIMIT {limit}"
|
||||
@@ -635,10 +637,11 @@ class Neo4jStore:
|
||||
|
||||
try:
|
||||
properties = properties or {}
|
||||
safe_rel_type = sanitize_identifier(rel_type, "relationship type")
|
||||
query = f"""
|
||||
MATCH (a), (b)
|
||||
WHERE id(a) = $start_id AND id(b) = $end_id
|
||||
CREATE (a)-[r:{rel_type} $props]->(b)
|
||||
CREATE (a)-[r:{safe_rel_type} $props]->(b)
|
||||
RETURN id(r) as id, type(r) as type, r
|
||||
"""
|
||||
|
||||
@@ -696,7 +699,7 @@ class Neo4jStore:
|
||||
List of matching relationships
|
||||
"""
|
||||
try:
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
|
||||
if node_id is not None:
|
||||
if direction == "out":
|
||||
@@ -857,7 +860,8 @@ class Neo4jStore:
|
||||
List of neighboring nodes with path information
|
||||
"""
|
||||
try:
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
depth = int(depth)
|
||||
|
||||
if direction == "out":
|
||||
pattern = f"-[r{type_filter}*1..{depth}]->"
|
||||
@@ -910,7 +914,8 @@ class Neo4jStore:
|
||||
Shortest path information or None if not found
|
||||
"""
|
||||
try:
|
||||
type_filter = f":{rel_type}" if rel_type else ""
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
max_depth = int(max_depth)
|
||||
|
||||
query = f"""
|
||||
MATCH path = shortestPath((start)-[r{type_filter}*..{max_depth}]-(end))
|
||||
@@ -975,17 +980,22 @@ class Neo4jStore:
|
||||
True if index created successfully
|
||||
"""
|
||||
try:
|
||||
index_name = options.get("index_name", f"idx_{label}_{property_name}")
|
||||
safe_label = sanitize_identifier(label, "label")
|
||||
safe_property = sanitize_identifier(property_name, "property key")
|
||||
index_name = sanitize_identifier(
|
||||
options.get("index_name", f"idx_{safe_label}_{safe_property}"),
|
||||
"index name",
|
||||
)
|
||||
|
||||
if index_type == "fulltext":
|
||||
query = f"""
|
||||
CREATE FULLTEXT INDEX {index_name} IF NOT EXISTS
|
||||
FOR (n:{label}) ON EACH [n.{property_name}]
|
||||
FOR (n:{safe_label}) ON EACH [n.{safe_property}]
|
||||
"""
|
||||
else:
|
||||
query = f"""
|
||||
CREATE INDEX {index_name} IF NOT EXISTS
|
||||
FOR (n:{label}) ON (n.{property_name})
|
||||
FOR (n:{safe_label}) ON (n.{safe_property})
|
||||
"""
|
||||
|
||||
with self.get_session() as session:
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
"""
|
||||
Shared identifier validation for Cypher/SPARQL query builders.
|
||||
|
||||
Node labels, relationship types, and property keys can't be bound as query
|
||||
parameters the way values can, so any such identifier that reaches a query
|
||||
string unvalidated is a direct injection point (GHSA-482h-hw99-h62p).
|
||||
`age_store.py` already validates its labels/relationship types this way;
|
||||
this module generalizes that pattern for reuse across the other graph
|
||||
store backends without introducing an import cycle with `graph_store.py`
|
||||
or `methods.py`.
|
||||
"""
|
||||
|
||||
import re
|
||||
|
||||
from ..utils.exceptions import ValidationError
|
||||
|
||||
_IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
|
||||
|
||||
def sanitize_identifier(name: str, kind: str = "identifier") -> str:
|
||||
"""Validate a Cypher/SPARQL label, relationship type, or property key.
|
||||
|
||||
Only alphanumeric/underscore identifiers starting with a letter or
|
||||
underscore are allowed.
|
||||
"""
|
||||
if not isinstance(name, str) or not _IDENTIFIER_RE.match(name):
|
||||
raise ValidationError(
|
||||
f"Invalid {kind}: {name!r}. Must start with a letter or "
|
||||
"underscore and contain only alphanumeric characters and "
|
||||
"underscores."
|
||||
)
|
||||
return name
|
||||
@@ -312,7 +312,7 @@ class BlazegraphStore:
|
||||
try:
|
||||
# Use SPARQL INSERT for bulk loading
|
||||
graph = options.get("graph", "")
|
||||
graph_clause = f"GRAPH <{graph}>" if graph else ""
|
||||
graph_clause = f"GRAPH <{sparql_escaping.validate_uri(graph)}>" if graph else ""
|
||||
|
||||
# Build INSERT query
|
||||
insert_data = self._build_insert_data(triplets)
|
||||
@@ -344,8 +344,10 @@ class BlazegraphStore:
|
||||
if format == "turtle":
|
||||
lines = []
|
||||
for triplet in triplets:
|
||||
subject = sparql_escaping.validate_uri(triplet.subject)
|
||||
predicate = sparql_escaping.validate_uri(triplet.predicate)
|
||||
lines.append(
|
||||
f"<{triplet.subject}> <{triplet.predicate}> {self._format_object_for_sparql(triplet)} ."
|
||||
f"<{subject}> <{predicate}> {self._format_object_for_sparql(triplet)} ."
|
||||
)
|
||||
return "\n".join(lines)
|
||||
else:
|
||||
@@ -353,11 +355,20 @@ class BlazegraphStore:
|
||||
return self._triplets_to_rdf(triplets, "turtle")
|
||||
|
||||
def _build_insert_data(self, triplets: List[Triplet]) -> str:
|
||||
"""Build SPARQL INSERT DATA clause."""
|
||||
"""Build SPARQL INSERT DATA clause.
|
||||
|
||||
Validates subject/predicate as safe IRIs via
|
||||
sparql_escaping.validate_uri before interpolation — unlike the
|
||||
object (handled by _format_object_for_sparql), subject/predicate
|
||||
can't be parameterized in a raw HTTP SPARQL Update POST, so an
|
||||
unvalidated value is a direct injection point (GHSA-8vgg-8mr4-r236).
|
||||
"""
|
||||
lines = []
|
||||
for triplet in triplets:
|
||||
subject = sparql_escaping.validate_uri(triplet.subject)
|
||||
predicate = sparql_escaping.validate_uri(triplet.predicate)
|
||||
lines.append(
|
||||
f"<{triplet.subject}> <{triplet.predicate}> {self._format_object_for_sparql(triplet)} ."
|
||||
f"<{subject}> <{predicate}> {self._format_object_for_sparql(triplet)} ."
|
||||
)
|
||||
return " ".join(lines)
|
||||
|
||||
@@ -380,11 +391,14 @@ class BlazegraphStore:
|
||||
|
||||
if self._is_uri_value(obj):
|
||||
if obj.startswith("<") and obj.endswith(">"):
|
||||
inner = obj[1:-1]
|
||||
if " " in inner or ">" in inner:
|
||||
raise ValueError(f"IRI contains invalid characters: {obj!r}")
|
||||
return obj
|
||||
return f"<{obj}>"
|
||||
# Validate the inner IRI with the same disallowed-character
|
||||
# set as the unwrapped branch below — a narrower ad-hoc
|
||||
# check here previously let a pre-wrapped object bypass
|
||||
# validate_uri() entirely (GHSA-8vgg-8mr4-r236 follow-up).
|
||||
inner = sparql_escaping.validate_uri(obj[1:-1])
|
||||
return f"<{inner}>"
|
||||
validated_obj = sparql_escaping.validate_uri(obj)
|
||||
return f"<{validated_obj}>"
|
||||
|
||||
escaped = self._escape_literal(obj)
|
||||
datatype = metadata.get("datatype") or metadata.get("literal_datatype")
|
||||
@@ -460,9 +474,9 @@ class BlazegraphStore:
|
||||
# Build SPARQL query
|
||||
where_clauses = []
|
||||
if subject:
|
||||
where_clauses.append(f"?s = <{subject}>")
|
||||
where_clauses.append(f"?s = <{sparql_escaping.validate_uri(subject)}>")
|
||||
if predicate:
|
||||
where_clauses.append(f"?p = <{predicate}>")
|
||||
where_clauses.append(f"?p = <{sparql_escaping.validate_uri(predicate)}>")
|
||||
if object:
|
||||
where_clauses.append(
|
||||
f"?o = {self._format_object_for_sparql(Triplet(subject='', predicate='', object=object))}"
|
||||
@@ -494,8 +508,10 @@ class BlazegraphStore:
|
||||
|
||||
update_endpoint = self._get_update_endpoint()
|
||||
|
||||
subject = sparql_escaping.validate_uri(triplet.subject)
|
||||
predicate = sparql_escaping.validate_uri(triplet.predicate)
|
||||
query = (
|
||||
f"DELETE DATA {{ <{triplet.subject}> <{triplet.predicate}> "
|
||||
f"DELETE DATA {{ <{subject}> <{predicate}> "
|
||||
f"{self._format_object_for_sparql(triplet)} }}"
|
||||
)
|
||||
|
||||
|
||||
@@ -319,11 +319,11 @@ class JenaStore:
|
||||
# Build SPARQL query
|
||||
query_parts = []
|
||||
if subject:
|
||||
query_parts.append(f"?s = <{subject}>")
|
||||
query_parts.append(f"?s = <{sparql_escaping.validate_uri(subject)}>")
|
||||
if predicate:
|
||||
query_parts.append(f"?p = <{predicate}>")
|
||||
query_parts.append(f"?p = <{sparql_escaping.validate_uri(predicate)}>")
|
||||
if object:
|
||||
query_parts.append(f"?o = <{object}>")
|
||||
query_parts.append(f"?o = <{sparql_escaping.validate_uri(object)}>")
|
||||
|
||||
where_clause = " ".join(query_parts) if query_parts else ""
|
||||
query = f"SELECT ?s ?p ?o WHERE {{ ?s ?p ?o {where_clause} }}"
|
||||
|
||||
@@ -452,11 +452,11 @@ class RDF4JStore:
|
||||
# Build SPARQL query
|
||||
where_clauses = []
|
||||
if subject:
|
||||
where_clauses.append(f"?s = <{subject}>")
|
||||
where_clauses.append(f"?s = <{sparql_escaping.validate_uri(subject)}>")
|
||||
if predicate:
|
||||
where_clauses.append(f"?p = <{predicate}>")
|
||||
where_clauses.append(f"?p = <{sparql_escaping.validate_uri(predicate)}>")
|
||||
if object:
|
||||
where_clauses.append(f"?o = <{object}>")
|
||||
where_clauses.append(f"?o = <{sparql_escaping.validate_uri(object)}>")
|
||||
|
||||
where_clause = " ".join(where_clauses) if where_clauses else ""
|
||||
query = f"SELECT ?s ?p ?o WHERE {{ ?s ?p ?o {where_clause} }}"
|
||||
@@ -484,8 +484,18 @@ class RDF4JStore:
|
||||
|
||||
update_endpoint = self._get_update_endpoint()
|
||||
|
||||
# Use SPARQL DELETE
|
||||
query = f"DELETE DATA {{ <{triplet.subject}> <{triplet.predicate}> <{triplet.object}> }}"
|
||||
# Use SPARQL DELETE.
|
||||
# subject/predicate must be IRIs — validate_uri enforces that and
|
||||
# blocks injection through '>' or other SPARQL metacharacters.
|
||||
# object can be an IRI *or* a literal, so it is routed through
|
||||
# _format_object_for_ntriples (which internally calls validate_uri
|
||||
# for URI-shaped values and escape_literal for strings), matching
|
||||
# the same object-handling semantics used by the add path and by
|
||||
# BlazegraphStore.delete_triplet (GHSA-8vgg-8mr4-r236 regression fix).
|
||||
subject = sparql_escaping.validate_uri(triplet.subject)
|
||||
predicate = sparql_escaping.validate_uri(triplet.predicate)
|
||||
obj_str = self._format_object_for_ntriples(triplet)
|
||||
query = f"DELETE DATA {{ <{subject}> <{predicate}> {obj_str} }}"
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
@@ -529,11 +539,14 @@ class RDF4JStore:
|
||||
|
||||
if self._is_uri_value(obj):
|
||||
if obj.startswith("<") and obj.endswith(">"):
|
||||
inner = obj[1:-1]
|
||||
if " " in inner or ">" in inner:
|
||||
raise ValueError(f"IRI contains invalid characters: {obj!r}")
|
||||
return obj
|
||||
return f"<{obj}>"
|
||||
# Validate the inner IRI with the same disallowed-character
|
||||
# set as the unwrapped branch below — a narrower ad-hoc
|
||||
# check here previously let a pre-wrapped object bypass
|
||||
# validate_uri() entirely (GHSA-8vgg-8mr4-r236 follow-up).
|
||||
inner = sparql_escaping.validate_uri(obj[1:-1])
|
||||
return f"<{inner}>"
|
||||
validated_obj = sparql_escaping.validate_uri(obj)
|
||||
return f"<{validated_obj}>"
|
||||
|
||||
escaped = sparql_escaping.escape_literal(obj)
|
||||
datatype = metadata.get("datatype") or metadata.get("literal_datatype")
|
||||
@@ -550,9 +563,17 @@ class RDF4JStore:
|
||||
return f'"{escaped}"'
|
||||
|
||||
def _triplets_to_ntriples(self, triplets: List[Triplet]) -> str:
|
||||
"""Convert triplets to N-Triples format."""
|
||||
"""Convert triplets to N-Triples format.
|
||||
|
||||
Validates subject/predicate via sparql_escaping.validate_uri: an
|
||||
unvalidated value containing '>' or a newline could terminate the
|
||||
current triple line early and splice extra triples into the
|
||||
upload stream (GHSA-8vgg-8mr4-r236).
|
||||
"""
|
||||
lines = []
|
||||
for triplet in triplets:
|
||||
subject = sparql_escaping.validate_uri(triplet.subject)
|
||||
predicate = sparql_escaping.validate_uri(triplet.predicate)
|
||||
obj_str = self._format_object_for_ntriples(triplet)
|
||||
lines.append(f"<{triplet.subject}> <{triplet.predicate}> {obj_str} .")
|
||||
lines.append(f"<{subject}> <{predicate}> {obj_str} .")
|
||||
return "\n".join(lines)
|
||||
|
||||
@@ -150,3 +150,62 @@ def test_websocket_accepts_connection_with_header_key(client, monkeypatch):
|
||||
) as websocket:
|
||||
ack = websocket.receive_json()
|
||||
assert ack["event"] == "connection_ack"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# WebSocket Origin validation (GHSA-4643-wpgq-w329): CORSMiddleware doesn't
|
||||
# cover WebSocket handshakes at all, so under SEMANTICA_ALLOW_ANONYMOUS the
|
||||
# key check alone accepted a handshake from any origin — loopback binding is
|
||||
# not a boundary against a browser, since any page the operator has open can
|
||||
# still reach ws://localhost:.../ws/graph-updates. These pin the fix: a
|
||||
# hostile Origin is refused even in anonymous mode (and even with a correct
|
||||
# key), a same-origin/allowlisted Origin still works, and a missing Origin
|
||||
# (native/CLI clients, which never set the header) is still allowed through.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_websocket_rejects_hostile_origin_under_anonymous_mode(client, monkeypatch):
|
||||
monkeypatch.setenv("SEMANTICA_ALLOW_ANONYMOUS", "true")
|
||||
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
|
||||
|
||||
with pytest.raises(Exception):
|
||||
with client.websocket_connect(
|
||||
"/ws/graph-updates", headers={"Origin": "https://evil.example"}
|
||||
):
|
||||
pass
|
||||
|
||||
|
||||
def test_websocket_rejects_hostile_origin_even_with_correct_key(client, monkeypatch):
|
||||
"""Defense in depth: Origin is checked before the API key, so a hostile
|
||||
page that somehow obtained a valid key still can't hijack the socket."""
|
||||
monkeypatch.delenv("SEMANTICA_ALLOW_ANONYMOUS", raising=False)
|
||||
monkeypatch.setenv("SEMANTICA_API_KEY", "correct-key")
|
||||
|
||||
with pytest.raises(Exception):
|
||||
with client.websocket_connect(
|
||||
"/ws/graph-updates",
|
||||
headers={"Origin": "https://evil.example", "X-API-Key": "correct-key"},
|
||||
):
|
||||
pass
|
||||
|
||||
|
||||
def test_websocket_accepts_allowlisted_origin_under_anonymous_mode(client, monkeypatch):
|
||||
monkeypatch.setenv("SEMANTICA_ALLOW_ANONYMOUS", "true")
|
||||
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
|
||||
|
||||
with client.websocket_connect(
|
||||
"/ws/graph-updates", headers={"Origin": "http://localhost:5173"}
|
||||
) as websocket:
|
||||
ack = websocket.receive_json()
|
||||
assert ack["event"] == "connection_ack"
|
||||
|
||||
|
||||
def test_websocket_accepts_missing_origin_under_anonymous_mode(client, monkeypatch):
|
||||
"""Native/CLI clients never send an Origin header — only browsers do —
|
||||
so a missing Origin must still be allowed through; the browser is the
|
||||
only threat this check closes."""
|
||||
monkeypatch.setenv("SEMANTICA_ALLOW_ANONYMOUS", "true")
|
||||
monkeypatch.delenv("SEMANTICA_API_KEY", raising=False)
|
||||
|
||||
with client.websocket_connect("/ws/graph-updates") as websocket:
|
||||
ack = websocket.receive_json()
|
||||
assert ack["event"] == "connection_ack"
|
||||
|
||||
@@ -0,0 +1,392 @@
|
||||
"""Regression tests for DNS check-then-use (TOCTOU) hardening in the
|
||||
ontology URL fetcher (GHSA-8c7v-62gr-hj6g's secondary "smaller" gap).
|
||||
|
||||
`_validate_fetch_url` resolves and validates a hostname once; if the actual
|
||||
HTTP client resolved it again independently at connect time, a low-TTL or
|
||||
rebinding DNS answer could differ between the two lookups, reopening the
|
||||
SSRF window the validation exists to close. `_make_pinned_session` closes
|
||||
this by pinning the connection pool's `host` directly to the already-
|
||||
validated IP (bypassing DNS resolution for the connection entirely), while
|
||||
explicitly restoring the real hostname as the outgoing HTTP `Host` header
|
||||
and, for HTTPS, the TLS SNI `server_hostname` / `assert_hostname` — so the
|
||||
connection reaches the pinned IP but still presents (and verifies against)
|
||||
the original hostname's identity.
|
||||
|
||||
test_ontology_ssrf.py covers the redirect-handling logic around this with
|
||||
mocks; this file proves the pinning mechanism itself works end-to-end
|
||||
against real local servers, with no DNS mocking at all — the test hostname
|
||||
is never resolved, which is exactly the property being verified. It also
|
||||
includes a negative control (mismatched cert hostname) proving TLS
|
||||
verification is genuinely enforced against the real hostname, not silently
|
||||
bypassed or checked against the pinned IP instead.
|
||||
"""
|
||||
|
||||
import http.server
|
||||
import socket
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
from semantica.explorer.routes import ontology as ontology_mod
|
||||
|
||||
|
||||
def _start_local_server():
|
||||
captured = {}
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
captured["host_header"] = self.headers.get("Host")
|
||||
body = b"pinned response"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *_args):
|
||||
pass
|
||||
|
||||
server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
return server, thread, captured
|
||||
|
||||
|
||||
def test_pinned_session_connects_to_pinned_ip_without_resolving_hostname():
|
||||
"""A session built by _make_pinned_session must reach the pinned IP
|
||||
directly. The request URL uses a hostname that cannot be resolved via
|
||||
real DNS ('.invalid' is reserved by RFC 2606) — if pinning weren't
|
||||
working, this request would fail with a name-resolution error instead
|
||||
of reaching the local server, since nothing else could route it there.
|
||||
"""
|
||||
server, thread, captured = _start_local_server()
|
||||
port = server.server_address[1]
|
||||
url = f"http://pinned-test.invalid:{port}/resource"
|
||||
try:
|
||||
session = ontology_mod._make_pinned_session(["127.0.0.1"], url)
|
||||
try:
|
||||
resp = session.get(url, timeout=5)
|
||||
assert resp.status_code == 200
|
||||
assert resp.content == b"pinned response"
|
||||
finally:
|
||||
session.close()
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join(timeout=2)
|
||||
|
||||
# Host header must still be the original hostname, not the pinned IP —
|
||||
# proving connection target and presented identity are decoupled
|
||||
# correctly (this is what keeps virtual hosting / TLS SNI correct).
|
||||
assert captured["host_header"] == f"pinned-test.invalid:{port}"
|
||||
|
||||
|
||||
def test_pinned_session_ignores_a_different_real_resolution():
|
||||
"""Even if the hostname *does* resolve to something else via real DNS,
|
||||
the pinned session must still go to the pinned IP — this is the actual
|
||||
TOCTOU property: the connection uses what was validated, not whatever
|
||||
a fresh lookup returns. 'localhost' reliably resolves to a loopback
|
||||
address, which is deliberately NOT where our test server listens on
|
||||
(127.0.0.1 specifically) — but since Windows/most stacks map
|
||||
'localhost' to 127.0.0.1 too, use a distinct high loopback address
|
||||
(127.0.0.2) for the server so a real 'localhost' resolution (127.0.0.1)
|
||||
provably would NOT reach it, isolating the assertion to pinning alone.
|
||||
"""
|
||||
captured = {}
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
captured["host_header"] = self.headers.get("Host")
|
||||
body = b"pinned via explicit ip"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *_args):
|
||||
pass
|
||||
|
||||
try:
|
||||
server = http.server.HTTPServer(("127.0.0.2", 0), Handler)
|
||||
except OSError:
|
||||
# 127.0.0.2 isn't bindable in this environment (uncommon, but
|
||||
# possible in some sandboxes) — skip rather than false-fail.
|
||||
import pytest
|
||||
pytest.skip("127.0.0.2 is not bindable in this environment")
|
||||
|
||||
port = server.server_address[1]
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
url = f"http://localhost:{port}/resource"
|
||||
try:
|
||||
session = ontology_mod._make_pinned_session(["127.0.0.2"], url)
|
||||
try:
|
||||
resp = session.get(url, timeout=5)
|
||||
assert resp.status_code == 200
|
||||
assert resp.content == b"pinned via explicit ip"
|
||||
finally:
|
||||
session.close()
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join(timeout=2)
|
||||
|
||||
assert captured["host_header"] == f"localhost:{port}"
|
||||
|
||||
|
||||
def test_pinned_session_falls_back_across_multiple_pinned_ips():
|
||||
"""A hostname can have multiple A/AAAA records; pinning to only the
|
||||
first-returned address means a fetch fails outright if that specific
|
||||
address happens to be unreachable even though a later one would work.
|
||||
_make_pinned_session must fall back through every pinned IP in order.
|
||||
"""
|
||||
server, thread, captured = _start_local_server()
|
||||
port = server.server_address[1]
|
||||
url = f"http://pinned-test.invalid:{port}/resource"
|
||||
# 127.0.0.3 has nothing listening on this port — connection refused,
|
||||
# forcing a fallback to the second (real) address.
|
||||
unreachable_ip = "127.0.0.3"
|
||||
try:
|
||||
session = ontology_mod._make_pinned_session([unreachable_ip, "127.0.0.1"], url)
|
||||
try:
|
||||
resp = session.get(url, timeout=5)
|
||||
assert resp.status_code == 200
|
||||
assert resp.content == b"pinned response"
|
||||
finally:
|
||||
session.close()
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join(timeout=2)
|
||||
|
||||
|
||||
def test_pinned_session_raises_when_every_pinned_ip_is_unreachable():
|
||||
"""If none of the pinned IPs are reachable, the session must raise
|
||||
rather than silently falling back to resolving the hostname itself
|
||||
(which would reopen the exact TOCTOU window pinning exists to close)."""
|
||||
import requests
|
||||
|
||||
url = "http://pinned-test.invalid:9/resource" # port 9 (discard) — nothing listens
|
||||
session = ontology_mod._make_pinned_session(["127.0.0.3", "127.0.0.4"], url)
|
||||
try:
|
||||
with pytest.raises(requests.exceptions.ConnectionError):
|
||||
session.get(url, timeout=5)
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
def test_pinned_session_disables_environment_proxy_trust():
|
||||
"""A pinned session must never honor HTTP_PROXY/HTTPS_PROXY env vars —
|
||||
a proxy would perform its own DNS resolution of the target host outside
|
||||
this process's control, reopening the exact TOCTOU window pinning
|
||||
exists to close."""
|
||||
session = ontology_mod._make_pinned_session(["127.0.0.1"], "http://example.org/")
|
||||
try:
|
||||
assert session.trust_env is False
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
def test_pinned_session_ignores_env_proxy_and_connects_directly(monkeypatch):
|
||||
"""End-to-end: even with HTTP_PROXY pointed at an address that would
|
||||
fail if contacted, a pinned session must reach the real local server
|
||||
directly — proving the env var is genuinely not consulted, not just
|
||||
that the trust_env flag is set."""
|
||||
monkeypatch.setenv("HTTP_PROXY", "http://127.0.0.5:1/") # would fail if ever used
|
||||
server, thread, _captured = _start_local_server()
|
||||
port = server.server_address[1]
|
||||
url = f"http://pinned-test.invalid:{port}/resource"
|
||||
try:
|
||||
session = ontology_mod._make_pinned_session(["127.0.0.1"], url)
|
||||
try:
|
||||
resp = session.get(url, timeout=5)
|
||||
assert resp.status_code == 200
|
||||
assert resp.content == b"pinned response"
|
||||
finally:
|
||||
session.close()
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join(timeout=2)
|
||||
|
||||
|
||||
def test_pinned_session_fails_closed_if_a_proxy_is_explicitly_forced():
|
||||
"""Backstop: if a proxy is somehow still configured on the session
|
||||
despite trust_env=False (e.g. set explicitly, as a future code path
|
||||
might), the adapter must fail closed with a clear error rather than
|
||||
silently connecting through the proxy unpinned."""
|
||||
from fastapi import HTTPException
|
||||
|
||||
session = ontology_mod._make_pinned_session(["127.0.0.1"], "http://example.org/")
|
||||
session.proxies = {"http": "http://127.0.0.5:1"}
|
||||
try:
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
session.get("http://example.org/", timeout=5)
|
||||
assert exc_info.value.status_code == 502
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
def test_validate_fetch_url_returns_the_resolved_ip():
|
||||
"""_validate_fetch_url must return every IP it validated, so callers can
|
||||
pin the connection to them (with fallback across all of them)."""
|
||||
def fake_getaddrinfo(host, *_a, **_k):
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0))]
|
||||
|
||||
import unittest.mock as mock
|
||||
with mock.patch.object(ontology_mod.socket, "getaddrinfo", side_effect=fake_getaddrinfo):
|
||||
resolved_ips = ontology_mod._validate_fetch_url("http://example.org/ontology.ttl")
|
||||
|
||||
assert resolved_ips == ["93.184.216.34"]
|
||||
|
||||
|
||||
def test_validate_fetch_url_returns_all_validated_ips_deduplicated():
|
||||
"""A hostname with multiple A/AAAA records must return every distinct
|
||||
validated address, in resolution order, so the caller can fall back
|
||||
across all of them rather than failing if only the first is
|
||||
unreachable."""
|
||||
def fake_getaddrinfo(host, *_a, **_k):
|
||||
return [
|
||||
(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 0)),
|
||||
(socket.AF_INET, socket.SOCK_DGRAM, 17, "", ("93.184.216.34", 0)), # duplicate, different socktype
|
||||
(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.35", 0)),
|
||||
]
|
||||
|
||||
import unittest.mock as mock
|
||||
with mock.patch.object(ontology_mod.socket, "getaddrinfo", side_effect=fake_getaddrinfo):
|
||||
resolved_ips = ontology_mod._validate_fetch_url("http://example.org/ontology.ttl")
|
||||
|
||||
assert resolved_ips == ["93.184.216.34", "93.184.216.35"]
|
||||
|
||||
|
||||
def test_validate_fetch_url_still_rejects_private_ip():
|
||||
"""Confirm the pinning refactor didn't loosen the original address
|
||||
classification — a hostname resolving to a private/internal address
|
||||
must still be rejected before any IP is returned."""
|
||||
import ipaddress
|
||||
import unittest.mock as mock
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
|
||||
def fake_getaddrinfo(host, *_a, **_k):
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("169.254.169.254", 0))]
|
||||
|
||||
with mock.patch.object(ontology_mod.socket, "getaddrinfo", side_effect=fake_getaddrinfo):
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
ontology_mod._validate_fetch_url("http://attacker.example/ontology.ttl")
|
||||
|
||||
assert exc_info.value.status_code == 422
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HTTPS: SNI + certificate hostname verification must use the real hostname,
|
||||
# not the pinned IP — this is the highest-risk part of pinning to get wrong,
|
||||
# since a mistake here could silently weaken TLS verification rather than
|
||||
# just breaking connectivity. Requires the optional `cryptography` package
|
||||
# to mint a throwaway self-signed cert; skipped gracefully without it.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _make_self_signed_cert(hostname: str, tmp_path):
|
||||
import datetime
|
||||
|
||||
pytest.importorskip("cryptography")
|
||||
from cryptography import x509
|
||||
from cryptography.hazmat.primitives import hashes, serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import rsa
|
||||
from cryptography.x509.oid import NameOID
|
||||
|
||||
key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
|
||||
name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, hostname)])
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
cert = (
|
||||
x509.CertificateBuilder()
|
||||
.subject_name(name)
|
||||
.issuer_name(name)
|
||||
.public_key(key.public_key())
|
||||
.serial_number(x509.random_serial_number())
|
||||
.not_valid_before(now - datetime.timedelta(days=1))
|
||||
.not_valid_after(now + datetime.timedelta(days=1))
|
||||
.add_extension(x509.SubjectAlternativeName([x509.DNSName(hostname)]), critical=False)
|
||||
.sign(key, hashes.SHA256())
|
||||
)
|
||||
|
||||
cert_path = tmp_path / "cert.pem"
|
||||
key_path = tmp_path / "key.pem"
|
||||
cert_path.write_bytes(cert.public_bytes(serialization.Encoding.PEM))
|
||||
key_path.write_bytes(
|
||||
key.private_bytes(
|
||||
encoding=serialization.Encoding.PEM,
|
||||
format=serialization.PrivateFormat.TraditionalOpenSSL,
|
||||
encryption_algorithm=serialization.NoEncryption(),
|
||||
)
|
||||
)
|
||||
return str(cert_path), str(key_path)
|
||||
|
||||
|
||||
def _start_local_https_server(cert_path, key_path):
|
||||
import ssl
|
||||
|
||||
captured = {}
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
captured["host_header"] = self.headers.get("Host")
|
||||
body = b"tls pinned response"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, *_args):
|
||||
pass
|
||||
|
||||
server = http.server.HTTPServer(("127.0.0.1", 0), Handler)
|
||||
ssl_ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
ssl_ctx.minimum_version = ssl.TLSVersion.TLSv1_2
|
||||
ssl_ctx.load_cert_chain(cert_path, key_path)
|
||||
server.socket = ssl_ctx.wrap_socket(server.socket, server_side=True)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
return server, thread, captured
|
||||
|
||||
|
||||
def test_pinned_https_session_verifies_against_real_hostname_not_pinned_ip(tmp_path):
|
||||
"""A pinned HTTPS connection must present + verify SNI/cert against the
|
||||
real hostname, even though the socket connects to the pinned IP. The
|
||||
cert's SAN is the hostname, never '127.0.0.1' — if pinning verified
|
||||
against the IP instead (or against nothing), this would either fail
|
||||
for the wrong reason or silently succeed with no real verification."""
|
||||
cert_path, key_path = _make_self_signed_cert("pinned-tls-test.invalid", tmp_path)
|
||||
server, thread, captured = _start_local_https_server(cert_path, key_path)
|
||||
port = server.server_address[1]
|
||||
url = f"https://pinned-tls-test.invalid:{port}/resource"
|
||||
try:
|
||||
session = ontology_mod._make_pinned_session(["127.0.0.1"], url)
|
||||
try:
|
||||
resp = session.get(url, timeout=5, verify=cert_path)
|
||||
finally:
|
||||
session.close()
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join(timeout=2)
|
||||
|
||||
assert resp.status_code == 200
|
||||
assert resp.content == b"tls pinned response"
|
||||
assert captured["host_header"] == f"pinned-tls-test.invalid:{port}"
|
||||
|
||||
|
||||
def test_pinned_https_session_rejects_hostname_mismatch(tmp_path):
|
||||
"""Negative control: requesting a hostname that does NOT match the
|
||||
cert's SAN must still fail verification — proving pinning doesn't
|
||||
silently bypass or misdirect certificate hostname checking."""
|
||||
cert_path, key_path = _make_self_signed_cert("pinned-tls-test.invalid", tmp_path)
|
||||
server, thread, _captured = _start_local_https_server(cert_path, key_path)
|
||||
port = server.server_address[1]
|
||||
url = f"https://wrong-name.invalid:{port}/resource"
|
||||
try:
|
||||
session = ontology_mod._make_pinned_session(["127.0.0.1"], url)
|
||||
try:
|
||||
import requests
|
||||
with pytest.raises(requests.exceptions.SSLError):
|
||||
session.get(url, timeout=5, verify=cert_path)
|
||||
finally:
|
||||
session.close()
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join(timeout=2)
|
||||
@@ -3,12 +3,17 @@
|
||||
`_fetch_url_sync` disables `requests`' automatic redirect following and
|
||||
re-validates every hop with `_validate_fetch_url` (see GHSA-8c7v-62gr-hj6g:
|
||||
unvalidated redirect targets previously let a public first hop 302 the
|
||||
server into fetching cloud metadata / loopback services).
|
||||
server into fetching cloud metadata / loopback services). It also pins each
|
||||
hop's connection to the IP `_validate_fetch_url` already resolved and
|
||||
validated, via `_make_pinned_session`, closing the DNS check-then-use gap
|
||||
between that validation and the client's own (potentially different) lookup.
|
||||
|
||||
These tests cover the redirect-handling logic itself: relative `Location`
|
||||
headers must resolve correctly instead of being rejected outright, redirect
|
||||
targets that resolve to private/loopback addresses must still be blocked,
|
||||
and every response must be closed (no leaked connections across hops).
|
||||
`test_ontology_dns_pinning.py` covers the pinning mechanism
|
||||
(`_make_pinned_session`, `_validate_fetch_url`'s returned IP) directly.
|
||||
"""
|
||||
|
||||
import socket
|
||||
@@ -37,6 +42,17 @@ def _make_response(is_redirect=False, is_permanent=False, location=None, body=b"
|
||||
return resp
|
||||
|
||||
|
||||
def _patch_session(responses):
|
||||
"""Patch _make_pinned_session so _fetch_url_sync's session.get(...)
|
||||
calls return the given responses in order, without touching the real
|
||||
requests.Session/pinning machinery (that's covered by test_pinning.py).
|
||||
"""
|
||||
fake_session = MagicMock()
|
||||
fake_session.get = MagicMock(side_effect=responses)
|
||||
fake_session.close = MagicMock()
|
||||
return patch.object(ontology_mod, "_make_pinned_session", return_value=fake_session), fake_session
|
||||
|
||||
|
||||
@patch.object(ontology_mod.socket, "getaddrinfo", side_effect=_fake_getaddrinfo)
|
||||
def test_relative_redirect_location_is_resolved(mock_getaddrinfo):
|
||||
"""A relative Location header (e.g. '/ontology.ttl') must resolve against
|
||||
@@ -44,11 +60,12 @@ def test_relative_redirect_location_is_resolved(mock_getaddrinfo):
|
||||
redirect_resp = _make_response(is_redirect=True, location="/ontology.ttl")
|
||||
final_resp = _make_response(body=b"final content")
|
||||
|
||||
with patch("requests.get", side_effect=[redirect_resp, final_resp]) as mock_get:
|
||||
patcher, fake_session = _patch_session([redirect_resp, final_resp])
|
||||
with patcher:
|
||||
result = ontology_mod._fetch_url_sync("http://example.org/start")
|
||||
|
||||
assert result == b"final content"
|
||||
second_call_url = mock_get.call_args_list[1].args[0]
|
||||
second_call_url = fake_session.get.call_args_list[1].args[0]
|
||||
assert second_call_url == "http://example.org/ontology.ttl"
|
||||
redirect_resp.close.assert_called_once()
|
||||
final_resp.close.assert_called_once()
|
||||
@@ -67,7 +84,8 @@ def test_redirect_to_private_ip_is_rejected(mock_getaddrinfo):
|
||||
mock_getaddrinfo.side_effect = getaddrinfo_side_effect
|
||||
redirect_resp = _make_response(is_redirect=True, location="http://internal.example/latest/meta-data/")
|
||||
|
||||
with patch("requests.get", side_effect=[redirect_resp]):
|
||||
patcher, fake_session = _patch_session([redirect_resp])
|
||||
with patcher:
|
||||
with pytest.raises(ontology_mod.HTTPException) as exc_info:
|
||||
ontology_mod._fetch_url_sync("http://example.org/start")
|
||||
|
||||
@@ -78,7 +96,8 @@ def test_redirect_to_private_ip_is_rejected(mock_getaddrinfo):
|
||||
@patch.object(ontology_mod.socket, "getaddrinfo", side_effect=_fake_getaddrinfo)
|
||||
def test_final_response_is_closed(mock_getaddrinfo):
|
||||
final_resp = _make_response(body=b"content")
|
||||
with patch("requests.get", side_effect=[final_resp]):
|
||||
patcher, _fake_session = _patch_session([final_resp])
|
||||
with patcher:
|
||||
ontology_mod._fetch_url_sync("http://example.org/start")
|
||||
final_resp.close.assert_called_once()
|
||||
|
||||
@@ -86,7 +105,8 @@ def test_final_response_is_closed(mock_getaddrinfo):
|
||||
@patch.object(ontology_mod.socket, "getaddrinfo", side_effect=_fake_getaddrinfo)
|
||||
def test_redirect_chain_exceeding_cap_is_rejected(mock_getaddrinfo):
|
||||
responses = [_make_response(is_redirect=True, location=f"/hop{i}") for i in range(10)]
|
||||
with patch("requests.get", side_effect=responses):
|
||||
patcher, _fake_session = _patch_session(responses)
|
||||
with patcher:
|
||||
with pytest.raises(ontology_mod.HTTPException) as exc_info:
|
||||
ontology_mod._fetch_url_sync("http://example.org/start")
|
||||
assert exc_info.value.status_code == 502
|
||||
|
||||
@@ -199,14 +199,27 @@ def test_allowlist_rejected_query_never_touches_the_graph(client, query):
|
||||
mock_build.assert_not_called()
|
||||
|
||||
|
||||
def test_multi_statement_injection_reaches_graph_but_fails_in_parser(client):
|
||||
"""Confirms the distinction between allowlist rejection and parser rejection:
|
||||
a string starting with SELECT passes _is_read_only_query and builds a graph,
|
||||
but rdflib.Graph.query() rejects the trailing '; DROP ALL' syntax."""
|
||||
def test_multi_statement_injection_is_rejected_by_forbidden_keyword_check(client):
|
||||
"""A string starting with an allowed keyword (SELECT) but containing a
|
||||
forbidden Update keyword later in the body ('; DROP ALL') is now
|
||||
rejected by _is_read_only_query's keyword scan itself, before a graph
|
||||
is ever built — a stronger, earlier rejection than relying solely on
|
||||
rdflib's parser to reject the syntax."""
|
||||
with patch.object(sparql_mod, "_build_rdflib_graph") as mock_build:
|
||||
resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o } ; DROP ALL")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["error"] is not None
|
||||
mock_build.assert_not_called()
|
||||
|
||||
|
||||
def test_malformed_syntax_without_forbidden_keywords_still_fails_in_parser(client):
|
||||
"""The parser remains a real second line of defense for malformed
|
||||
queries that don't contain any forbidden keyword — these pass
|
||||
_is_read_only_query and reach rdflib, which rejects the syntax."""
|
||||
with patch.object(
|
||||
sparql_mod, "_build_rdflib_graph", wraps=sparql_mod._build_rdflib_graph
|
||||
) as spy_build:
|
||||
resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o } ; DROP ALL")
|
||||
resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o } ; ASK { ?x ?y ?z }")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["error"] is not None
|
||||
spy_build.assert_called_once()
|
||||
@@ -304,6 +317,47 @@ def test_oversized_graph_returns_clean_error_not_a_crash(client):
|
||||
assert payload["rows"] == []
|
||||
|
||||
|
||||
def test_oversized_query_returns_distinct_length_error(client):
|
||||
"""A query exceeding _SPARQL_MAX_QUERY_LEN must be rejected with a
|
||||
specific, actionable error message — not the generic read-only message.
|
||||
Clients need to distinguish a size-limit rejection from an actual
|
||||
non-read-only query rejection to react correctly (e.g. split the query
|
||||
vs. rewrite it)."""
|
||||
with patch.object(sparql_mod, "_SPARQL_MAX_QUERY_LEN", 10):
|
||||
resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o }") # 30 chars > 10
|
||||
assert resp.status_code == 200
|
||||
payload = resp.json()
|
||||
assert payload["error"] is not None
|
||||
# Must mention the limit, not the generic read-only message
|
||||
assert "length" in payload["error"].lower() or "characters" in payload["error"].lower()
|
||||
assert "Only SELECT" not in payload["error"]
|
||||
assert payload["rows"] == []
|
||||
assert payload["columns"] == []
|
||||
assert payload["total"] == 0
|
||||
|
||||
|
||||
def test_oversized_query_never_touches_the_graph(client):
|
||||
"""An oversized query must be rejected before _build_rdflib_graph is
|
||||
called — the length guard must short-circuit the entire pipeline."""
|
||||
with patch.object(sparql_mod, "_SPARQL_MAX_QUERY_LEN", 10):
|
||||
with patch.object(sparql_mod, "_build_rdflib_graph") as mock_build:
|
||||
resp = _post(client, "SELECT ?s WHERE { ?s ?p ?o }")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["error"] is not None
|
||||
mock_build.assert_not_called()
|
||||
|
||||
|
||||
def test_query_exactly_at_length_limit_is_accepted(client):
|
||||
"""A query whose length equals the limit exactly must not be rejected —
|
||||
the guard is strictly greater-than, not greater-than-or-equal."""
|
||||
short_query = "ASK {}"
|
||||
with patch.object(sparql_mod, "_SPARQL_MAX_QUERY_LEN", len(short_query)):
|
||||
resp = _post(client, short_query)
|
||||
assert resp.status_code == 200
|
||||
payload = resp.json()
|
||||
assert payload["error"] is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data-mapping fidelity: does the graph->RDF projection reflect session state?
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,396 @@
|
||||
"""Regression tests for GHSA-482h-hw99-h62p: unvalidated node labels and
|
||||
property keys allowed arbitrary Cypher injection in the Neptune, Neo4j, and
|
||||
FalkorDB graph stores (labels/keys can't be bound as query parameters, so
|
||||
an unvalidated value reaching the query string is a direct injection
|
||||
point).
|
||||
|
||||
Mirrors the advisory's own PoC shape: a label/key crafted to close the
|
||||
current Cypher token early and append a destructive statement
|
||||
(`DETACH DELETE victim`). Before the fix, these reached `_run_query` /
|
||||
`session.run` / `graph.query` verbatim. After the fix, `sanitize_identifier`
|
||||
(graph_store/query_sanitize.py) rejects them with ValidationError before
|
||||
any query is built, matching the existing age_store.py `_sanitize_label`
|
||||
behavior used as the reference implementation.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import unittest
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from semantica.graph_store.query_sanitize import sanitize_identifier
|
||||
from semantica.utils.exceptions import ProcessingError, ValidationError
|
||||
|
||||
# AmazonNeptuneStore/Neo4jStore/FalkorDBStore.create_node() wrap their whole
|
||||
# body in `except Exception: raise ProcessingError(...)` (pre-existing,
|
||||
# unrelated to this fix), so the ValidationError sanitize_identifier raises
|
||||
# surfaces to callers as ProcessingError. Either way the malicious query is
|
||||
# never built or sent — these tests assert exactly that via the query-capture
|
||||
# stubs, and check the wrapped message to confirm it's the sanitizer firing.
|
||||
|
||||
EVIL_LABEL = "N}) MATCH (victim) DETACH DELETE victim //"
|
||||
EVIL_KEY = "k1`: 1}) MATCH (victim) DETACH DELETE victim //"
|
||||
|
||||
|
||||
def _wire(store):
|
||||
store.logger = MagicMock()
|
||||
store.progress_tracker = MagicMock()
|
||||
store.progress_tracker.start_tracking.return_value = "tid"
|
||||
store.config = {}
|
||||
return store
|
||||
|
||||
|
||||
class TestSanitizeIdentifier(unittest.TestCase):
|
||||
def test_valid_identifiers_pass_through_unchanged(self):
|
||||
self.assertEqual(sanitize_identifier("Person"), "Person")
|
||||
self.assertEqual(sanitize_identifier("_hidden"), "_hidden")
|
||||
self.assertEqual(sanitize_identifier("Rel_Type2"), "Rel_Type2")
|
||||
|
||||
def test_injection_payload_is_rejected(self):
|
||||
with self.assertRaises(ValidationError):
|
||||
sanitize_identifier(EVIL_LABEL)
|
||||
|
||||
def test_property_key_injection_payload_is_rejected(self):
|
||||
with self.assertRaises(ValidationError):
|
||||
sanitize_identifier(EVIL_KEY)
|
||||
|
||||
def test_rejects_non_string(self):
|
||||
with self.assertRaises(ValidationError):
|
||||
sanitize_identifier(123) # type: ignore[arg-type]
|
||||
|
||||
def test_rejects_spaces_and_dashes(self):
|
||||
with self.assertRaises(ValidationError):
|
||||
sanitize_identifier("no spaces")
|
||||
with self.assertRaises(ValidationError):
|
||||
sanitize_identifier("no-dashes")
|
||||
|
||||
|
||||
class TestAmazonNeptuneCypherInjection(unittest.TestCase):
|
||||
def _make_store(self):
|
||||
from semantica.graph_store.amazon_neptune import AmazonNeptuneStore
|
||||
|
||||
store = _wire(AmazonNeptuneStore.__new__(AmazonNeptuneStore))
|
||||
store._connected = True
|
||||
store._ensure_connected = lambda: None
|
||||
store._generate_id = lambda: "generated-id"
|
||||
store._run_query = MagicMock(return_value=[])
|
||||
store._parse_results = lambda r: []
|
||||
return store
|
||||
|
||||
def test_create_node_rejects_malicious_label_before_querying(self):
|
||||
store = self._make_store()
|
||||
with self.assertRaises(ProcessingError) as ctx:
|
||||
store.create_node(labels=[EVIL_LABEL], properties={"name": "x"})
|
||||
self.assertIn("Invalid label", str(ctx.exception))
|
||||
store._run_query.assert_not_called()
|
||||
|
||||
def test_create_node_rejects_malicious_property_key_before_querying(self):
|
||||
store = self._make_store()
|
||||
with self.assertRaises(ProcessingError) as ctx:
|
||||
store.create_node(labels=["Person"], properties={"name": "x", EVIL_KEY: 1})
|
||||
self.assertIn("Invalid property key", str(ctx.exception))
|
||||
store._run_query.assert_not_called()
|
||||
|
||||
def test_create_node_with_legitimate_labels_still_works(self):
|
||||
store = self._make_store()
|
||||
store.create_node(labels=["Person", "Employee"], properties={"name": "Alice"})
|
||||
query = store._run_query.call_args[0][0]
|
||||
self.assertIn("Person:Employee", query)
|
||||
self.assertNotIn("DETACH DELETE", query)
|
||||
|
||||
|
||||
class TestNeo4jCypherInjection(unittest.TestCase):
|
||||
def _make_store(self):
|
||||
from semantica.graph_store import neo4j_store as m
|
||||
|
||||
store = _wire(m.Neo4jStore.__new__(m.Neo4jStore))
|
||||
captured = {}
|
||||
|
||||
class Session:
|
||||
def run(self, q, params=None):
|
||||
captured["query"] = q
|
||||
rec = {"n": {"name": "x"}, "id": 1}
|
||||
return type("R", (), {"single": lambda self: rec})()
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
store.get_session = lambda: Session()
|
||||
store._captured = captured
|
||||
return store
|
||||
|
||||
def test_create_node_rejects_malicious_label_before_querying(self):
|
||||
store = self._make_store()
|
||||
with self.assertRaises(ProcessingError) as ctx:
|
||||
store.create_node(labels=["Person", EVIL_LABEL], properties={"name": "x"})
|
||||
self.assertIn("Invalid label", str(ctx.exception))
|
||||
self.assertNotIn("query", store._captured)
|
||||
|
||||
def test_create_relationship_rejects_malicious_rel_type(self):
|
||||
store = self._make_store()
|
||||
with self.assertRaises(ProcessingError) as ctx:
|
||||
store.create_relationship(start_node_id=1, end_node_id=2, rel_type=EVIL_LABEL)
|
||||
self.assertIn("Invalid relationship type", str(ctx.exception))
|
||||
self.assertNotIn("query", store._captured)
|
||||
|
||||
|
||||
class TestFalkorDBCypherInjection(unittest.TestCase):
|
||||
def _make_store(self):
|
||||
from semantica.graph_store import falkordb_store as m
|
||||
|
||||
store = _wire(m.FalkorDBStore.__new__(m.FalkorDBStore))
|
||||
captured = {}
|
||||
|
||||
class Graph:
|
||||
def query(self, q, params=None):
|
||||
captured["query"] = q
|
||||
return type("R", (), {"result_set": []})()
|
||||
|
||||
store._ensure_graph = lambda: Graph()
|
||||
store._captured = captured
|
||||
return store
|
||||
|
||||
def test_create_node_rejects_malicious_label_and_key(self):
|
||||
store = self._make_store()
|
||||
with self.assertRaises(ProcessingError) as ctx:
|
||||
store.create_node(labels=[EVIL_LABEL], properties={"name": "x", EVIL_KEY: 1})
|
||||
self.assertIn("Invalid label", str(ctx.exception))
|
||||
self.assertNotIn("query", store._captured)
|
||||
|
||||
def test_create_node_with_legitimate_input_still_works(self):
|
||||
store = self._make_store()
|
||||
store.create_node(labels=["Person"], properties={"name": "Alice"})
|
||||
query = store._captured["query"]
|
||||
self.assertIn("Person", query)
|
||||
self.assertNotIn("DETACH DELETE", query)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# BLOCKER 1 regression: Neo4jStore.get_neighbors / shortest_path depth coercion
|
||||
# ---------------------------------------------------------------------------
|
||||
# Payload representative of the confirmed injection (review BLOCKER 1):
|
||||
# supplying a string as `depth` / `max_depth` previously reached the Cypher
|
||||
# f-string verbatim because Neo4jStore did not call int() like Neptune/FalkorDB.
|
||||
#
|
||||
# After the fix `depth = int(depth)` / `max_depth = int(max_depth)` are added
|
||||
# at the top of each method's try-block. A malicious string raises ValueError
|
||||
# (wrapped in ProcessingError), and the session.run / _run_query mock must
|
||||
# never be called.
|
||||
|
||||
EVIL_DEPTH = "1]->(x) DETACH DELETE x //"
|
||||
|
||||
|
||||
class TestNeo4jDepthInjection(unittest.TestCase):
|
||||
"""Regression tests for BLOCKER 1: Neo4jStore depth/max_depth coercion."""
|
||||
|
||||
def _make_store(self):
|
||||
"""Build a Neo4jStore with a session that records every query string sent to it."""
|
||||
from semantica.graph_store import neo4j_store as m
|
||||
|
||||
store = _wire(m.Neo4jStore.__new__(m.Neo4jStore))
|
||||
captured = {}
|
||||
|
||||
class IterSession:
|
||||
"""Returns an empty iterator so get_neighbors' for-loop completes cleanly."""
|
||||
def run(self, q, params=None):
|
||||
captured["query"] = q
|
||||
return iter([])
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
class SingleSession:
|
||||
"""Returns a FakeResult whose .single() yields None (shortest_path)."""
|
||||
def run(self, q, params=None):
|
||||
captured["query"] = q
|
||||
return type("R", (), {"single": lambda self: None})()
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
store._captured = captured
|
||||
store._iter_session = IterSession
|
||||
store._single_session = SingleSession
|
||||
return store
|
||||
|
||||
# -- get_neighbors --------------------------------------------------------
|
||||
|
||||
def test_get_neighbors_malicious_depth_raises_before_query(self):
|
||||
"""Malicious string depth must never reach session.run."""
|
||||
store = self._make_store()
|
||||
store.get_session = lambda: store._iter_session()
|
||||
with self.assertRaises(ProcessingError):
|
||||
store.get_neighbors(node_id=1, depth=EVIL_DEPTH)
|
||||
self.assertNotIn("query", store._captured,
|
||||
"session.run was called — injected query reached the database layer")
|
||||
|
||||
def test_get_neighbors_malicious_depth_does_not_contain_payload(self):
|
||||
"""Double-check: if somehow a query were built, it must not contain the payload."""
|
||||
store = self._make_store()
|
||||
store.get_session = lambda: store._iter_session()
|
||||
with pytest.raises(ProcessingError):
|
||||
store.get_neighbors(node_id=1, depth=EVIL_DEPTH)
|
||||
query = store._captured.get("query", "")
|
||||
self.assertNotIn("DETACH DELETE", query,
|
||||
f"Injection payload found in query: {query!r}")
|
||||
|
||||
def test_get_neighbors_legitimate_depth_works(self):
|
||||
"""Valid integer depth must still produce a correct traversal pattern."""
|
||||
store = self._make_store()
|
||||
store.get_session = lambda: store._iter_session()
|
||||
result = store.get_neighbors(node_id=1, depth=2)
|
||||
self.assertIsInstance(result, list)
|
||||
self.assertIn("query", store._captured)
|
||||
self.assertIn("*1..2", store._captured["query"])
|
||||
self.assertNotIn("DETACH DELETE", store._captured["query"])
|
||||
|
||||
def test_get_neighbors_depth_string_int_is_coerced(self):
|
||||
"""A string representation of a valid integer must be coerced and work."""
|
||||
store = self._make_store()
|
||||
store.get_session = lambda: store._iter_session()
|
||||
result = store.get_neighbors(node_id=1, depth="3")
|
||||
self.assertIsInstance(result, list)
|
||||
self.assertIn("*1..3", store._captured["query"])
|
||||
|
||||
# -- shortest_path --------------------------------------------------------
|
||||
|
||||
def test_shortest_path_malicious_max_depth_raises_before_query(self):
|
||||
"""Malicious string max_depth must never reach session.run."""
|
||||
store = self._make_store()
|
||||
store.get_session = lambda: store._single_session()
|
||||
with self.assertRaises(ProcessingError):
|
||||
store.shortest_path(start_node_id=1, end_node_id=2, max_depth=EVIL_DEPTH)
|
||||
self.assertNotIn("query", store._captured,
|
||||
"session.run was called — injected query reached the database layer")
|
||||
|
||||
def test_shortest_path_malicious_max_depth_does_not_contain_payload(self):
|
||||
store = self._make_store()
|
||||
store.get_session = lambda: store._single_session()
|
||||
with pytest.raises(ProcessingError):
|
||||
store.shortest_path(start_node_id=1, end_node_id=2, max_depth=EVIL_DEPTH)
|
||||
query = store._captured.get("query", "")
|
||||
self.assertNotIn("DETACH DELETE", query,
|
||||
f"Injection payload found in query: {query!r}")
|
||||
|
||||
def test_shortest_path_legitimate_max_depth_works(self):
|
||||
"""Valid integer max_depth must produce a correct shortestPath pattern."""
|
||||
store = self._make_store()
|
||||
store.get_session = lambda: store._single_session()
|
||||
result = store.shortest_path(start_node_id=1, end_node_id=2, max_depth=5)
|
||||
self.assertIsNone(result) # single() returns None → correct
|
||||
self.assertIn("query", store._captured)
|
||||
self.assertIn("*..5", store._captured["query"])
|
||||
self.assertNotIn("DETACH DELETE", store._captured["query"])
|
||||
|
||||
def test_shortest_path_max_depth_string_int_is_coerced(self):
|
||||
store = self._make_store()
|
||||
store.get_session = lambda: store._single_session()
|
||||
store.shortest_path(start_node_id=1, end_node_id=2, max_depth="7")
|
||||
self.assertIn("*..7", store._captured["query"])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# BLOCKER 2 regression: GraphStore.get_neighbors hops forwarding
|
||||
# ---------------------------------------------------------------------------
|
||||
# Before the fix, GraphStore.get_neighbors() set:
|
||||
# actual_depth = options.get("hops", depth)
|
||||
# and forwarded the raw value to Neo4jStore.get_neighbors(depth=actual_depth).
|
||||
# Because Neo4jStore did not coerce depth, an attacker-controlled hops string
|
||||
# reached Cypher verbatim.
|
||||
#
|
||||
# The fix adds int() at the GraphStore facade:
|
||||
# actual_depth = int(options.get("hops", depth))
|
||||
# This closes the path regardless of which backend is wired up.
|
||||
|
||||
class TestGraphStoreHopsForwarding(unittest.TestCase):
|
||||
"""Regression tests for BLOCKER 2: GraphStore hops→depth forwarding."""
|
||||
|
||||
def _make_graph_store_with_neo4j(self):
|
||||
"""
|
||||
Wire a GraphStore whose backend is a Neo4j store stub that records every
|
||||
query passed to session.run. Returns (graph_store, captured_dict).
|
||||
"""
|
||||
from semantica.graph_store import neo4j_store as m
|
||||
from semantica.graph_store.graph_store import (
|
||||
GraphAnalytics,
|
||||
GraphManager,
|
||||
GraphStore,
|
||||
)
|
||||
|
||||
captured = {}
|
||||
|
||||
class IterSession:
|
||||
def run(self, q, params=None):
|
||||
captured["query"] = q
|
||||
return iter([])
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
neo4j_stub = _wire(m.Neo4jStore.__new__(m.Neo4jStore))
|
||||
neo4j_stub.get_session = lambda: IterSession()
|
||||
|
||||
gs = GraphStore.__new__(GraphStore)
|
||||
gs.logger = MagicMock()
|
||||
gs.progress_tracker = MagicMock()
|
||||
gs._store_backend = neo4j_stub
|
||||
gs._manager = GraphManager(neo4j_stub)
|
||||
|
||||
return gs, captured
|
||||
|
||||
# -- hops injection -------------------------------------------------------
|
||||
|
||||
def test_hops_malicious_string_raises_before_query(self):
|
||||
"""Malicious hops string must be rejected before session.run is reached."""
|
||||
gs, captured = self._make_graph_store_with_neo4j()
|
||||
with self.assertRaises(Exception):
|
||||
gs.get_neighbors(node_id=1, hops=EVIL_DEPTH)
|
||||
self.assertNotIn("query", captured,
|
||||
"session.run was called — injected hops reached the database layer")
|
||||
|
||||
def test_hops_malicious_string_payload_not_in_any_query(self):
|
||||
"""Belt-and-suspenders: payload text must not appear in any built query."""
|
||||
gs, captured = self._make_graph_store_with_neo4j()
|
||||
with pytest.raises(ValueError):
|
||||
gs.get_neighbors(node_id=1, hops=EVIL_DEPTH)
|
||||
query = captured.get("query", "")
|
||||
self.assertNotIn("DETACH DELETE", query,
|
||||
f"Injection payload found in forwarded query: {query!r}")
|
||||
|
||||
def test_hops_legitimate_integer_works(self):
|
||||
"""Valid integer hops value must produce a correct query."""
|
||||
gs, captured = self._make_graph_store_with_neo4j()
|
||||
result = gs.get_neighbors(node_id=1, hops=2)
|
||||
self.assertIsInstance(result, list)
|
||||
self.assertIn("query", captured)
|
||||
self.assertIn("*1..2", captured["query"])
|
||||
self.assertNotIn("DETACH DELETE", captured["query"])
|
||||
|
||||
def test_hops_string_int_is_coerced_and_works(self):
|
||||
"""String '3' forwarded as hops must be coerced to int and produce *1..3."""
|
||||
gs, captured = self._make_graph_store_with_neo4j()
|
||||
result = gs.get_neighbors(node_id=1, hops="3")
|
||||
self.assertIsInstance(result, list)
|
||||
self.assertIn("*1..3", captured["query"])
|
||||
|
||||
def test_depth_param_still_works_without_hops(self):
|
||||
"""depth positional arg (no hops kwarg) must still be coerced and forwarded."""
|
||||
gs, captured = self._make_graph_store_with_neo4j()
|
||||
result = gs.get_neighbors(node_id=1, depth=4)
|
||||
self.assertIsInstance(result, list)
|
||||
self.assertIn("*1..4", captured["query"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -215,6 +215,26 @@ class TestGraphStore(unittest.TestCase):
|
||||
result = self.store.execute_query("MATCH (n) RETURN n")
|
||||
self.assertEqual(result["summary"], "Mock query executed")
|
||||
|
||||
def test_degree_centrality_rejects_malicious_label(self):
|
||||
"""Regression test for GHSA-482h-hw99-h62p: degree_centrality()
|
||||
interpolates labels/rel_type directly into a Cypher MATCH clause
|
||||
(graph_store.py's own query builder, not delegated to the backend),
|
||||
so an unvalidated label was a direct injection point."""
|
||||
from semantica.utils.exceptions import ValidationError
|
||||
evil_label = "N}) MATCH (victim) DETACH DELETE victim //"
|
||||
with self.assertRaises(ValidationError):
|
||||
self.store._manager.analytics.degree_centrality(labels=[evil_label])
|
||||
|
||||
def test_degree_centrality_rejects_malicious_rel_type(self):
|
||||
from semantica.utils.exceptions import ValidationError
|
||||
evil_rel_type = "R]-() DETACH DELETE n //"
|
||||
with self.assertRaises(ValidationError):
|
||||
self.store._manager.analytics.degree_centrality(rel_type=evil_rel_type)
|
||||
|
||||
def test_degree_centrality_with_legitimate_input_still_works(self):
|
||||
result = self.store._manager.analytics.degree_centrality(labels=["Person"])
|
||||
self.assertEqual(result, []) # MockGraphStore.execute_query returns no records
|
||||
|
||||
class TestGraphStoreInitialization(unittest.TestCase):
|
||||
def test_falkordb_initialization(self):
|
||||
with patch('semantica.graph_store.falkordb_store.FalkorDBStore', side_effect=MockGraphStore) as mock_falkor:
|
||||
|
||||
@@ -57,5 +57,26 @@ class TestGraphStoreMethods(unittest.TestCase):
|
||||
# Verify
|
||||
self.mock_store.execute_query.assert_called_once_with(query, None)
|
||||
|
||||
def test_update_relationship_rejects_malicious_property_key(self):
|
||||
"""Regression test for GHSA-482h-hw99-h62p: update_relationship()
|
||||
interpolates property keys directly into a Cypher SET clause
|
||||
(methods.py's own query builder, not delegated to the backend
|
||||
store), so an unvalidated key was a direct injection point."""
|
||||
from semantica.utils.exceptions import ValidationError
|
||||
|
||||
evil_key = "x} MATCH (victim) DETACH DELETE victim //"
|
||||
with self.assertRaises(ValidationError):
|
||||
methods.update_relationship(1, {evil_key: "value"})
|
||||
self.mock_store.execute_query.assert_not_called()
|
||||
|
||||
def test_update_relationship_with_legitimate_keys_still_works(self):
|
||||
self.mock_store.execute_query.return_value = {
|
||||
"records": [{"id": 1, "type": "KNOWS"}]
|
||||
}
|
||||
result = methods.update_relationship(1, {"weight": 0.5})
|
||||
query = self.mock_store.execute_query.call_args[0][0]
|
||||
self.assertIn("r.weight = $weight", query)
|
||||
self.assertNotIn("DETACH DELETE", query)
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -2,48 +2,29 @@
|
||||
Regression tests for security fixes in PR #898.
|
||||
|
||||
Covers:
|
||||
1. API Key Authentication (Explorer)
|
||||
2. Cypher Injection Prevention (AGE Store)
|
||||
3. SPARQL Injection Prevention (read-only query validation)
|
||||
4. XXE Protection (rdf_parser fail-closed)
|
||||
5. Vector save numpy serialization
|
||||
6. SPARQL graph cap error handling
|
||||
7. SSRF redirect handling (relative URLs, resp.close)
|
||||
1. Cypher Injection Prevention (AGE Store)
|
||||
2. SPARQL Injection Prevention (read-only query validation)
|
||||
3. XXE Protection (rdf_parser fail-closed)
|
||||
4. Vector save numpy serialization
|
||||
5. SPARQL graph cap error handling
|
||||
6. SSRF redirect handling (relative URLs, resp.close)
|
||||
|
||||
Explorer API-key authentication (GHSA-j4mq-hprp-987v) has its own, more
|
||||
thorough test suite at tests/explorer/test_explorer_auth.py — it isn't
|
||||
duplicated here.
|
||||
"""
|
||||
|
||||
import re
|
||||
import pytest
|
||||
|
||||
|
||||
# ===================================================================
|
||||
# 1. SPARQL read-only query validation (injection prevention)
|
||||
# ===================================================================
|
||||
|
||||
# Inline the validation logic so tests don't require full app context
|
||||
_ALLOWED_QUERY_TYPES = re.compile(
|
||||
r"^(SELECT|ASK|CONSTRUCT|DESCRIBE)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_FORBIDDEN_KEYWORDS = re.compile(
|
||||
r"\b(INSERT|DELETE|DROP|LOAD|CLEAR|CREATE|COPY|MOVE|ADD)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_COMMENT_LINE = re.compile(r"#[^\n]*", re.MULTILINE)
|
||||
_PREFIX_DECL = re.compile(
|
||||
r"^\s*(?:PREFIX|BASE)\s+(?:\S+\s+)?<[^>]*>\s*",
|
||||
re.IGNORECASE | re.MULTILINE,
|
||||
)
|
||||
|
||||
|
||||
def _is_read_only_query(query: str) -> bool:
|
||||
cleaned = _PREFIX_DECL.sub("", query)
|
||||
cleaned = _COMMENT_LINE.sub("", cleaned)
|
||||
cleaned = cleaned.strip()
|
||||
if not _ALLOWED_QUERY_TYPES.match(cleaned):
|
||||
return False
|
||||
if _FORBIDDEN_KEYWORDS.search(cleaned):
|
||||
return False
|
||||
return True
|
||||
# Import the real implementation rather than re-declaring the regexes here:
|
||||
# an earlier version of this file inlined a copy that silently drifted from
|
||||
# semantica/explorer/routes/sparql.py's actual behavior (the inlined
|
||||
# _COMMENT_LINE regex stripped '#' mid-token, corrupting any PREFIX
|
||||
# declaration using a namespace IRI with a literal '#', e.g. the standard
|
||||
# rdf:/rdfs: namespaces) and neither the code nor this test caught it,
|
||||
# since both had the same bug. Importing the real function makes that class
|
||||
# of drift impossible.
|
||||
from semantica.explorer.routes.sparql import _is_read_only_query
|
||||
|
||||
|
||||
class TestSparqlReadOnlyValidation:
|
||||
@@ -122,6 +103,63 @@ class TestSparqlReadOnlyValidation:
|
||||
query = "BASE <http://example.org/>\nSELECT ?s WHERE { ?s ?p ?o }"
|
||||
assert _is_read_only_query(query)
|
||||
|
||||
def test_namespace_iri_with_hash_fragment_not_treated_as_comment(self):
|
||||
"""A '#' inside a PREFIX declaration's IRI (standard for RDF/RDFS/OWL
|
||||
namespaces) must not be mistaken for a comment-start — a naive
|
||||
`#[^\\n]*` strip corrupts the IRI and truncates the rest of the
|
||||
query with it."""
|
||||
query = (
|
||||
"PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n"
|
||||
"SELECT ?s WHERE { ?s rdf:type ?o }"
|
||||
)
|
||||
assert _is_read_only_query(query)
|
||||
|
||||
def test_real_comment_after_namespace_iri_still_stripped(self):
|
||||
"""A genuine trailing comment must still be recognized even on a
|
||||
line that also contains a '#'-bearing IRI earlier in the query."""
|
||||
query = (
|
||||
"PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n"
|
||||
"SELECT ?s WHERE { ?s rdf:type ?o } # trailing comment INSERT DATA"
|
||||
)
|
||||
assert _is_read_only_query(query)
|
||||
|
||||
def test_inline_prefix_before_select_allowed(self):
|
||||
"""PREFIX declaration on the same line as the query keyword (inline
|
||||
prologue) must be stripped correctly so SELECT is seen first.
|
||||
Regression for the [ \\t]*(?:\\n|$) anchor that rejected this form."""
|
||||
query = "PREFIX ex: <http://example.org/> SELECT ?s WHERE { ?s ex:p ?o }"
|
||||
assert _is_read_only_query(query)
|
||||
|
||||
def test_crlf_line_endings_with_prefix(self):
|
||||
"""Windows CRLF line endings (\\r\\n) between PREFIX and SELECT must
|
||||
be handled correctly. The previous (?:\\n|$) anchor did not allow
|
||||
the \\r before \\n, causing stripping to fail."""
|
||||
query = "PREFIX ex: <http://example.org/>\r\nSELECT ?s WHERE { ?s ?p ?o }"
|
||||
assert _is_read_only_query(query)
|
||||
|
||||
def test_crlf_multiple_prefixes_then_select(self):
|
||||
"""Multiple PREFIX lines with CRLF endings should all be stripped."""
|
||||
query = (
|
||||
"PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\r\n"
|
||||
"PREFIX ex: <http://example.org/>\r\n"
|
||||
"SELECT ?s WHERE { ?s rdf:type ex:Thing }"
|
||||
)
|
||||
assert _is_read_only_query(query)
|
||||
|
||||
def test_inline_prefix_before_insert_still_blocked(self):
|
||||
"""An inline PREFIX followed by INSERT must not be allowed — the
|
||||
inline stripping fix must not open a bypass for write operations."""
|
||||
query = "PREFIX ex: <http://example.org/> INSERT DATA { ex:s ex:p ex:o }"
|
||||
assert not _is_read_only_query(query)
|
||||
|
||||
def test_long_valid_query_not_rejected_by_is_read_only(self):
|
||||
"""_is_read_only_query must not enforce the length limit itself —
|
||||
that responsibility belongs to execute_sparql() so the route can
|
||||
return a distinct, actionable error. A syntactically valid but long
|
||||
SELECT query must still return True from this function."""
|
||||
long_query = "SELECT ?s WHERE { ?s ?p ?o } # " + ("x" * 20_000)
|
||||
assert _is_read_only_query(long_query)
|
||||
|
||||
|
||||
# ===================================================================
|
||||
# 2. Cypher injection prevention
|
||||
@@ -270,38 +308,5 @@ class TestSSRFRedirectHandling:
|
||||
assert result == "https://example.com/api/data.ttl"
|
||||
|
||||
|
||||
# ===================================================================
|
||||
# 6. API Key Auth
|
||||
# ===================================================================
|
||||
|
||||
class TestAPIKeyAuth:
|
||||
"""Regression tests for API key authentication."""
|
||||
|
||||
def test_auth_module_importable(self):
|
||||
from semantica.explorer.auth import APIKeyAuthMiddleware
|
||||
assert APIKeyAuthMiddleware is not None
|
||||
|
||||
def test_extract_bearer_token(self):
|
||||
from semantica.explorer.auth import _extract_token
|
||||
from unittest.mock import MagicMock
|
||||
req = MagicMock()
|
||||
req.headers = {"Authorization": "Bearer test-key-123"}
|
||||
assert _extract_token(req) == "test-key-123"
|
||||
|
||||
def test_extract_api_key_header(self):
|
||||
from semantica.explorer.auth import _extract_token
|
||||
from unittest.mock import MagicMock
|
||||
req = MagicMock()
|
||||
req.headers = {"X-API-Key": "my-secret-key", "Authorization": ""}
|
||||
assert _extract_token(req) == "my-secret-key"
|
||||
|
||||
def test_extract_no_token(self):
|
||||
from semantica.explorer.auth import _extract_token
|
||||
from unittest.mock import MagicMock
|
||||
req = MagicMock()
|
||||
req.headers = {}
|
||||
assert _extract_token(req) is None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
||||
@@ -0,0 +1,275 @@
|
||||
"""Regression tests for GHSA-8vgg-8mr4-r236: unvalidated triplet IRIs
|
||||
allowed arbitrary SPARQL update injection in the Blazegraph and RDF4J
|
||||
stores, and query-filter injection on the Jena read path.
|
||||
|
||||
Triplet.subject/predicate (and, in some builders, .object) are document
|
||||
text in the normal ingest pipeline — entity names extracted from ingested
|
||||
content. A subject containing '>' closes the '<...>' IRI token early, so
|
||||
the rest of the value is parsed as more SPARQL, letting an attacker
|
||||
append operations like CLEAR ALL that run with the application's store
|
||||
credentials.
|
||||
|
||||
Mirrors the advisory's own PoC payload: a subject/predicate crafted to
|
||||
close the current triple pattern and append a destructive `; CLEAR ALL ;`
|
||||
statement. After the fix, sparql_escaping.validate_uri (already used by
|
||||
anzo_store.py, the one backend that was already hardened) rejects it
|
||||
before any query/update text is built.
|
||||
"""
|
||||
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from semantica.semantic_extract.triplet_extractor import Triplet
|
||||
from semantica.triplet_store.blazegraph_store import BlazegraphStore
|
||||
from semantica.triplet_store.rdf4j_store import RDF4JStore
|
||||
from semantica.triplet_store.jena_store import JenaStore
|
||||
from semantica.utils.exceptions import ProcessingError, ValidationError
|
||||
|
||||
# The advisory's own injection payload: closes the <...> token, then the
|
||||
# triple pattern, then appends a store-wide wipe and a rogue insert.
|
||||
EVIL_SUBJECT = (
|
||||
"http://example.com/a> <http://example.com/b> <http://example.com/c> . } "
|
||||
"; CLEAR ALL ; INSERT DATA { <http://evil.example/owned"
|
||||
)
|
||||
|
||||
|
||||
class TestBlazegraphSparqlInjection(unittest.TestCase):
|
||||
@patch.object(BlazegraphStore, "_connect", autospec=True)
|
||||
def _make_store(self, _mock_connect):
|
||||
return BlazegraphStore(endpoint="http://localhost:9999/blazegraph")
|
||||
|
||||
def test_build_insert_data_rejects_malicious_subject(self):
|
||||
store = self._make_store()
|
||||
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="x")
|
||||
with self.assertRaises(ValidationError):
|
||||
store._build_insert_data([triplet])
|
||||
|
||||
def test_triplets_to_rdf_rejects_malicious_subject(self):
|
||||
store = self._make_store()
|
||||
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="x")
|
||||
with self.assertRaises(ValidationError):
|
||||
store._triplets_to_rdf([triplet])
|
||||
|
||||
def test_delete_triplet_rejects_malicious_subject(self):
|
||||
store = self._make_store()
|
||||
store.connected = True
|
||||
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="x")
|
||||
with self.assertRaises(ValidationError):
|
||||
store.delete_triplet(triplet)
|
||||
|
||||
def test_get_triplets_rejects_malicious_subject_filter(self):
|
||||
store = self._make_store()
|
||||
with self.assertRaises(ValidationError):
|
||||
store.get_triplets(subject=EVIL_SUBJECT)
|
||||
|
||||
def test_bulk_load_rejects_malicious_graph_option(self):
|
||||
# bulk_load() wraps its whole body in except Exception: raise
|
||||
# ProcessingError(...) (pre-existing, unrelated to this fix), so the
|
||||
# ValidationError sanitize_uri raises surfaces as ProcessingError.
|
||||
# The security property that matters — the malicious query is never
|
||||
# sent — holds either way.
|
||||
store = self._make_store()
|
||||
store.connected = True
|
||||
triplet = Triplet(subject="http://s", predicate="http://p", object="x")
|
||||
with self.assertRaises(ProcessingError) as ctx:
|
||||
store.bulk_load([triplet], graph=EVIL_SUBJECT)
|
||||
self.assertIn("Invalid URI", str(ctx.exception))
|
||||
|
||||
def test_legitimate_triplet_still_builds_correct_query(self):
|
||||
store = self._make_store()
|
||||
triplet = Triplet(subject="http://s", predicate="http://p", object="x")
|
||||
insert_data = store._build_insert_data([triplet])
|
||||
self.assertIn("<http://s> <http://p>", insert_data)
|
||||
self.assertNotIn("CLEAR ALL", insert_data)
|
||||
|
||||
def test_format_object_rejects_malicious_pre_wrapped_iri(self):
|
||||
"""A caller-supplied object already wrapped in '<...>' must still be
|
||||
fully validated, not just checked for a literal space/'>' — a
|
||||
narrower ad-hoc check here previously let this branch bypass
|
||||
validate_uri() entirely (Codex-flagged follow-up to GHSA-8vgg)."""
|
||||
store = self._make_store()
|
||||
evil_object = f"<{EVIL_SUBJECT}>"
|
||||
triplet = Triplet(subject="http://s", predicate="http://p", object=evil_object)
|
||||
with self.assertRaises(ValidationError):
|
||||
store._format_object_for_sparql(triplet)
|
||||
|
||||
def test_format_object_accepts_legitimate_pre_wrapped_iri(self):
|
||||
store = self._make_store()
|
||||
triplet = Triplet(subject="http://s", predicate="http://p", object="<http://o>")
|
||||
self.assertEqual(store._format_object_for_sparql(triplet), "<http://o>")
|
||||
|
||||
|
||||
class TestRDF4JSparqlInjection(unittest.TestCase):
|
||||
@patch.object(RDF4JStore, "_connect", autospec=True)
|
||||
def _make_store(self, _mock_connect):
|
||||
return RDF4JStore(endpoint="http://localhost:9999/rdf4j", repository_id="mem")
|
||||
|
||||
def test_triplets_to_ntriples_rejects_malicious_subject(self):
|
||||
store = self._make_store()
|
||||
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="x")
|
||||
with self.assertRaises(ValidationError):
|
||||
store._triplets_to_ntriples([triplet])
|
||||
|
||||
def test_delete_triplet_rejects_malicious_subject(self):
|
||||
store = self._make_store()
|
||||
store.connected = True
|
||||
triplet = Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="http://o")
|
||||
with self.assertRaises(ValidationError):
|
||||
store.delete_triplet(triplet)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Regression tests for the literal-object bug fixed after the
|
||||
# adversarial review of PR #911: delete_triplet() previously called
|
||||
# validate_uri(triplet.object) unconditionally, which rejected every
|
||||
# non-URI object with ValidationError even though literal objects are
|
||||
# perfectly legal in RDF. The fix routes the object through
|
||||
# _format_object_for_ntriples so URI-valued objects are still validated
|
||||
# while literal objects go through escape_literal unchanged.
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _make_connected_store_with_captured_query(self):
|
||||
"""Return (store, captured_dict) where captured['update'] is the
|
||||
SPARQL update string passed to requests.post once delete_triplet
|
||||
succeeds."""
|
||||
import requests as req_mod
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
store = self._make_store()
|
||||
store.connected = True
|
||||
captured = {}
|
||||
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.raise_for_status = MagicMock()
|
||||
|
||||
def fake_post(url, **kwargs):
|
||||
captured["update"] = kwargs.get("data", {}).get("update", "")
|
||||
return mock_resp
|
||||
|
||||
store._post = fake_post # not used directly; patch requests.post below
|
||||
store._captured = captured
|
||||
return store, captured
|
||||
|
||||
def test_delete_triplet_literal_object_succeeds(self):
|
||||
"""A triplet with a plain-string literal object must delete without
|
||||
raising ValidationError — the regression that prompted this fix."""
|
||||
store, captured = self._make_connected_store_with_captured_query()
|
||||
|
||||
with patch("requests.post") as mock_post:
|
||||
mock_post.return_value.__enter__ = lambda s: s
|
||||
mock_post.return_value.raise_for_status = lambda: None
|
||||
|
||||
result = store.delete_triplet(
|
||||
Triplet(subject="http://s", predicate="http://p", object="Paris")
|
||||
)
|
||||
|
||||
self.assertEqual(result, {"success": True})
|
||||
# Confirm query shape: object must be a quoted literal, not <Paris>
|
||||
query_sent = mock_post.call_args[1]["data"]["update"]
|
||||
self.assertIn("<http://s> <http://p>", query_sent)
|
||||
self.assertIn('"Paris"', query_sent)
|
||||
self.assertNotIn("<Paris>", query_sent)
|
||||
self.assertNotIn("CLEAR ALL", query_sent)
|
||||
|
||||
def test_delete_triplet_uri_object_still_works(self):
|
||||
"""A triplet whose object is a URI must still delete correctly."""
|
||||
store, _ = self._make_connected_store_with_captured_query()
|
||||
|
||||
with patch("requests.post") as mock_post:
|
||||
mock_post.return_value.raise_for_status = lambda: None
|
||||
|
||||
result = store.delete_triplet(
|
||||
Triplet(subject="http://s", predicate="http://p", object="http://o")
|
||||
)
|
||||
|
||||
self.assertEqual(result, {"success": True})
|
||||
query_sent = mock_post.call_args[1]["data"]["update"]
|
||||
self.assertIn("<http://s> <http://p> <http://o>", query_sent)
|
||||
self.assertNotIn("CLEAR ALL", query_sent)
|
||||
|
||||
def test_delete_triplet_malicious_uri_object_rejected_before_post(self):
|
||||
"""A URI-shaped object containing '>' must be rejected by
|
||||
_format_object_for_ntriples → validate_uri before requests.post
|
||||
is ever called."""
|
||||
evil_obj = "http://evil.com/a>;CLEARALL"
|
||||
store, _ = self._make_connected_store_with_captured_query()
|
||||
|
||||
with patch("requests.post") as mock_post:
|
||||
with self.assertRaises(ValidationError):
|
||||
store.delete_triplet(
|
||||
Triplet(subject="http://s", predicate="http://p", object=evil_obj)
|
||||
)
|
||||
mock_post.assert_not_called()
|
||||
|
||||
def test_delete_triplet_malicious_subject_still_rejected(self):
|
||||
"""Subject injection protection must remain intact after the fix."""
|
||||
store, _ = self._make_connected_store_with_captured_query()
|
||||
|
||||
with patch("requests.post") as mock_post:
|
||||
with self.assertRaises(ValidationError):
|
||||
store.delete_triplet(
|
||||
Triplet(subject=EVIL_SUBJECT, predicate="http://p", object="http://o")
|
||||
)
|
||||
mock_post.assert_not_called()
|
||||
|
||||
def test_get_triplets_rejects_malicious_subject_filter(self):
|
||||
store = self._make_store()
|
||||
with self.assertRaises(ValidationError):
|
||||
store.get_triplets(subject=EVIL_SUBJECT)
|
||||
|
||||
def test_legitimate_triplet_still_builds_correct_ntriples(self):
|
||||
store = self._make_store()
|
||||
triplet = Triplet(subject="http://s", predicate="http://p", object="x")
|
||||
ntriples = store._triplets_to_ntriples([triplet])
|
||||
self.assertIn("<http://s> <http://p>", ntriples)
|
||||
self.assertNotIn("CLEAR ALL", ntriples)
|
||||
|
||||
def test_format_object_rejects_malicious_pre_wrapped_iri(self):
|
||||
"""Same pre-wrapped-object bypass as Blazegraph, fixed in
|
||||
_format_object_for_ntriples."""
|
||||
store = self._make_store()
|
||||
evil_object = f"<{EVIL_SUBJECT}>"
|
||||
triplet = Triplet(subject="http://s", predicate="http://p", object=evil_object)
|
||||
with self.assertRaises(ValidationError):
|
||||
store._format_object_for_ntriples(triplet)
|
||||
|
||||
def test_format_object_accepts_legitimate_pre_wrapped_iri(self):
|
||||
store = self._make_store()
|
||||
triplet = Triplet(subject="http://s", predicate="http://p", object="<http://o>")
|
||||
self.assertEqual(store._format_object_for_ntriples(triplet), "<http://o>")
|
||||
|
||||
|
||||
class TestJenaSparqlInjection(unittest.TestCase):
|
||||
def setUp(self):
|
||||
from rdflib import Graph
|
||||
|
||||
self.store = JenaStore()
|
||||
self.store.graph = Graph()
|
||||
|
||||
def test_get_triplets_never_queries_with_malicious_subject_filter(self):
|
||||
"""get_triplets() catches all exceptions and returns [] (pre-existing,
|
||||
broad error-handling behavior unrelated to this fix), so the
|
||||
observable contract is: the malicious filter must never reach
|
||||
graph.query() at all."""
|
||||
with patch.object(self.store.graph, "query", wraps=self.store.graph.query) as spy:
|
||||
result = self.store.get_triplets(subject=EVIL_SUBJECT)
|
||||
self.assertEqual(result, [])
|
||||
spy.assert_not_called()
|
||||
|
||||
def test_get_triplets_with_legitimate_filter_still_reaches_query(self):
|
||||
"""A validated identifier must not be rejected by the sanitizer —
|
||||
it should reach graph.query(). (Whether the WHERE-clause filter
|
||||
syntax jena_store.py builds is itself correct SPARQL is a separate,
|
||||
pre-existing question this test doesn't assert on: the query here
|
||||
is `{ ?s ?p ?o ?s = <http://s> }`, missing a FILTER()/separator,
|
||||
and unrelated to sanitize_uri.)"""
|
||||
self.store.graph.parse(
|
||||
data='<http://s> <http://p> "x" .', format="ntriples"
|
||||
)
|
||||
with patch.object(self.store.graph, "query", wraps=self.store.graph.query) as spy:
|
||||
self.store.get_triplets(subject="http://s")
|
||||
spy.assert_called_once()
|
||||
self.assertIn("http://s", spy.call_args[0][0])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user