Compare commits

..
Author SHA1 Message Date
KaifAhmad1 50927f99b5 docs: surface explainability scope note near the top of the README
Moves a concise version of the system-level vs. foundation-model
explainability clarification up next to the opening pitch, so it's
visible before readers scroll to the high-stakes-domains section.
2026-08-16 17:46:12 +05:30
KaifAhmad1 476237952d docs: clarify explainability is system-level, not foundation-model internal
Adds a consistent scope note to README and docs (concepts, FAQ, index)
stating Semantica does not expose or reconstruct an LLM's internal
reasoning/chain-of-thought. It explains and audits the AI system
around the model: context, provenance, policies, decisions, and
execution history.
2026-08-16 17:39:23 +05:30
60 changed files with 376 additions and 4127 deletions
-42
View File
@@ -11,15 +11,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **Semantica RDF vocabulary, and deterministic entity/relationship IRIs** (#1109, closes #1107, closes #1101) by @fabio-rovai, reviewed by @KaifAhmad1
- Every RDF/JSON-LD export mints terms in `https://semantica.dev/ns#`, and until now nothing declared what those terms meant — the namespace 404s and no vocabulary shipped with the package, so a consumer receiving an export had no way to tell `sem:text` from a typo of it, and no closed-world checker could validate an export at all
- `semantica/ontology/vocabulary/semantica-ns.ttl` declares the terms the exporters actually emit — drawn from the emitting call sites in `export/rdf_exporter.py`, `export/json_exporter.py` and `provenance/manager.py`, not from what a vocabulary "ought" to contain. Ships inside the package (`from semantica.ontology.vocabulary import vocabulary_turtle`) so it loads without a network round trip, and is the same document intended to be served at the namespace IRI once hosting/content-negotiation is sorted
- `tests/ontology/test_vocabulary.py` ties the document to the code: every term a serializer can write must be declared, so adding a term to an exporter without declaring it fails the build
- The missing-id fallback minted entity/relationship IRIs from Python's builtin `hash()`, randomised per process (`PYTHONHASHSEED`), so the same entity got a different IRI on every run and exports couldn't be diffed, deduplicated, or joined to an earlier provenance record. It also wrote `<semantica:entity_N>`, an IRI in the scheme `semantica` rather than the expansion of the declared prefix, so those nodes never joined with anything written through it. Minting now uses SHA-256 and writes a full IRI in the declared namespace; the same fix applies to the default entity/relationship types in the Turtle path
- **Fixed during review** (Qodo): the temporal fallback minted from `source_id` only, while the main serializer accepts `source_id` or `source` — relationships using the second form hashed two empty strings, which the previous randomised `hash()` masked by making the IRI unstable anyway; once deterministic, unrelated relationships at the same list index collided on one IRI across exports. Endpoints are now resolved the same way `serialize_to_turtle` resolves them, before minting. `sem:confidence` also lost its declared `xsd:decimal` range: the N-Triples serializer types the same value `xsd:float`, and the two are disjoint, so declaring either contradicted one of the exporters (tracked in #1100) — a new `test_declared_ranges_do_not_contradict_what_the_exporters_emit` guards the whole class of that mistake
- **Fixed in follow-up**: `serialize_to_rdfxml`'s default entity type still wrote the bare string `"semantica:Entity"` into an `rdf:resource` attribute, which (unlike a Turtle angle-bracket or an XML element name) is not namespace-expanded — the exact #1101 failure mode, just on the untested RDF/XML path. `json_exporter.py`'s `semantica:format` and `@type: "semantica:KnowledgeGraph"` were emitted but absent from both the vocabulary and the test's `EMITTED_TERMS` guard set, so the "undeclared terms fail the build" claim didn't actually cover them — both are now declared and guarded. `MANIFEST.in` didn't mirror the `pyproject.toml` package-data addition, so a source-distribution install could omit the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only `PATH`, breaking it on Windows; now overrides only `PYTHONHASHSEED` on top of the inherited environment
- 229 export and ontology tests pass
- **First-class CrewAI integration** (#962)
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
- `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()`
@@ -82,14 +73,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **`split`/chunking paths bypassed the centralized spaCy model cache, reloading the model on every call** (#1042, closes #998) by @Accute9, reviewed by @Sameer6305
- `semantica/split/methods.py`'s `split_by_sentences()` and `semantica/split/semantic_chunker.py`'s `SemanticChunker.__init__` each called `spacy.load()` directly instead of reusing the process-level cache added in #889/`semantic_extract/methods.py`'s `load_spacy_model()` — every call/construction re-paid the ~120ms model-load cost independently of `NERExtractor`, which already used the cache
- Both now route through `load_spacy_model()`, sharing one cached `Language` instance per model name across `split_by_sentences()`, `SemanticChunker`, and `NERExtractor`; a missing model still falls back to regex/paragraph chunking without poisoning the cache for a later successful load
- **Fixed during review** (@Sameer6305): `NERExtractor.__init__()` still had a direct `spacy.load()` call site with the same cache-bypass issue, outside the two files named in #998 but sharing the same root cause; routed through the cache alongside stale test patch targets and a strengthened cache-configuration assertion
- **Fixed during review** (@KaifAhmad1): `SemanticChunker.__init__` only caught `OSError` around `load_spacy_model()`, while the sibling fix to `NERExtractor` in this same PR added a broader `except Exception` for a model that is installed but fails at runtime (e.g. a config incompatible with the installed spaCy version). A broken-but-present model crashed `SemanticChunker()` outright instead of degrading to fallback chunking like every other path in this PR. Added the matching `except Exception` branch, leaving `self.nlp` as `None`; new `test_semantic_chunker_falls_back_when_spacy_runtime_is_broken` mirrors the existing `NERExtractor` regression test for the same scenario
- New `tests/split/test_spacy_model_cache.py`: cache reuse across repeated calls/instances, shared cache between `split_by_sentences()`/`SemanticChunker`/`NERExtractor`, distinct model names loading separately, missing-model fallback without poisoning the cache, and the broken-runtime fallback added above
- `pytest tests/split/test_spacy_model_cache.py tests/split/test_splitter.py tests/split/test_chunkers.py`: all passing (3 pre-existing, unrelated `tests/test_ner_configurations.py` failures confirmed present on `main` before this PR)
- **`export_yaml` raised a raw `AttributeError` on list input, silently wrote empty exports for unrecognized dict keys, and graph payloads were reconciled differently by every exporter** (#958, closes #956, #952, #953) by @pravit-amp, reviewed by @Sameer6305
- Graph payloads circulate under two vocabularies, `entities`/`relationships` and `nodes`/`edges`, and each exporter reconciled them locally with a different idiom — `LPGExporter` in particular dropped every entity whenever `nodes` was present but empty, the exact shape `JSONExporter` emits. A new `normalize_graph_payload()` in `utils/helpers.py` centralizes that decision once, adopted by `LPGExporter`, `ArangoAQLExporter`, `Neo4jCSVExporter`, and both YAML exporters; `ContextGraph.to_dict()` now round-trips through YAML correctly as a result
- `export_yaml(records, path)` on a bare list previously failed with `AttributeError` from inside the exporter; it and the other YAML methods now reject non-mapping input with an actionable `ProcessingError` naming the expected keys, since these formats distinguish entities/relationships/triplets and guessing which one a list represents would mislabel the records
@@ -185,31 +168,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Security
- **Tarball restore path traversal, latent SQL injection, DNS-rebinding TOCTOU in the shared SSRF guard, stored XSS in report generation, and unvalidated SPARQL object IRIs in AnzoStore** (#1079) by @KaifAhmad1
- `semantica backup restore`'s tar extraction (`cli.py`) stripped only the literal `semantica-backup/` prefix and called `tar.extract()` with no path-containment check, no symlink/hardlink validation, and (on Python <3.12) no extraction filter — a crafted archive member (`../../<file>`, or a symlink pointing outside the restore root) could write arbitrary files above the restore directory. Every member is now validated for resolved-path containment before extraction, symlink/hardlink targets are rejected both lexically (absolute path, `..` segments) and by resolution, and `filter="data"` is applied on Python ≥3.12
- `DataExporter.export_table_data()` (`db_ingestor.py`) was missing the `text` import from `sqlalchemy` — a `NameError` that made the method non-functional, but latently: the query it built from raw f-string interpolation of `table_name`/`schema`/`where`/`order_by` was already injectable, so fixing the import alone (without also fixing the injection) would have silently armed it. Both are fixed together: the import is restored, `table_name`/`schema` are now validated against a strict identifier allowlist, and `where`/`order_by` are checked against a blocklist (statement separators, comments, UNION, DDL/DML keywords, time-based blind-injection primitives, schema-enumeration terms). This is a blocklist, not a grammar — it closes the concrete UNION-exfiltration path and common injection primitives, but a boolean-blind subquery using none of the blocked keywords could still get through; `where`/`order_by` must be treated as trusted/operator input, not exposed to untrusted end users, and the docstrings now say so explicitly
- `request_with_ssrf_guard()` (`ssrf.py`) validated a hostname's resolved IPs, then let the underlying HTTP client re-resolve the same hostname independently at connect time — a low-TTL or DNS-rebinding answer could differ between the two lookups, so a hostname that validated as public could still connect to a private/internal address. Ported the IP-pinning pattern already used by `explorer/routes/ontology.py`'s `_make_pinned_session` into the shared ingest guard: the one resolution that decides accept/reject is now also the one the connection is pinned to, via a custom `HTTPAdapter` that presents the real hostname over TLS SNI / Host header while connecting only to the validated IPs. Also closes the RFC 6598 Carrier-Grade NAT gap noted as a known limitation in #905/#868: `100.64.0.0/10` is now in `BLOCKED_NETWORKS`
- `ReportGenerator._generate_html()` (`export/report_generator.py`) f-string-interpolated report title/summary/metrics into HTML with no escaping — an ingested entity or document whose content flowed into a report (e.g. `<img src=x onerror=...>`) executed as stored XSS when the report was opened. All interpolated values are now `html.escape()`d
- `AnzoStore._format_object_for_sparql()` (`triplet_store/anzo_store.py`) validated the subject/predicate of a triplet via `sparql_escaping.validate_uri()` before interpolating them into a SPARQL `INSERT DATA` clause, but delegated the **object** position to a separate formatter that wrapped it as `<{obj}>` without the same validation — an object value containing `>`/`}`/`{`/`"` could close the intended `<...>` token early and inject additional SPARQL Update operations. The Blazegraph/RDF4J backends were hardened for the equivalent gap previously; Anzo's object position now goes through the same `validate_uri()` check
- Also hardened in the same pass: Apache AGE's `create_index()` `index_type` parameter is now allowlisted (was interpolated raw into a `USING` clause); Neo4j's `limit` is now explicitly validated (raises `ValidationError` for non-integer input instead of falling through to a generic `ProcessingError`); the `ffprobe` metadata-extraction subprocess call is guarded against a filename starting with `-` being parsed as an option; the MCP server no longer echoes raw exception text to JSON-RPC clients, logging full details server-side and returning a generic message plus the exception class name instead
- **Fixed during review** (@KaifAhmad1): the SSRF IP-pinning change introduced a connection-pool leak of its own — `requests.Session.mount()` silently drops whatever adapter it replaces without closing it, so a multi-hop redirect chain on a reused session leaked one pooled connection per hop. Pinned adapters are now tagged and explicitly closed before being replaced, both per-hop and on final restore
- **Fixed during review** (@KaifAhmad1): mounting a pinned adapter and setting a Host header on a caller-supplied `Session` is not inherently thread-safe — two guarded calls sharing the same session from different threads could interleave their mount/restore cycles. Added a per-session lock (`_get_session_lock`) so concurrent guarded calls on the same session now serialize instead of racing; verified with a two-thread test showing correct serialization and zero cross-contamination of per-request Host headers
- **Fixed during automated PR review** (Qodo): `export_table_data()`'s new identifier/fragment validation raised `ValidationError` from inside a `try` whose blanket `except Exception` re-wrapped it as `ProcessingError`, masking the distinction between "bad input" and "the export itself failed" that callers rely on elsewhere in this module. Added the `except ValidationError: raise` guard already used by its sibling methods
- **Fixed during automated PR review** (Qodo): on a hop where IP pinning doesn't apply (`allow_private_ips=True`), `_apply_connection_pin()` unconditionally popped the session's `Host` header instead of restoring whatever it was before pinning touched it — a caller-supplied session carrying its own legitimate `Host` override (e.g. fronting a private endpoint under a different name) had that override silently dropped for the in-flight request, only reappearing afterward via the outer `finally` restore. It now restores the session's own pre-call header state (set back if present, popped only if it was truly absent) instead of always popping
- **Fixed during automated PR review** (Qodo): the `where`/`order_by` blocklist matched keywords/punctuation inside properly quoted string literals and identifiers too, so legitimate data like `status = 'union'` or `name = 'a--b'` was rejected as if it were SQL syntax. The blocklist now runs against a copy with quoted-literal contents masked out (`_mask_sql_literals`) — a malformed/unterminated quote sequence doesn't match the masking pattern and is left fully exposed to the blocklist, so this closes false positives without opening a masking-based bypass; the fragment actually used in the query is unchanged
- Re-ran each finding's proof-of-concept (or an equivalent adversarial test) against the fix and confirmed it is blocked: tar path/symlink traversal (both lexical and resolved-path forms), SQL UNION exfiltration and identifier breakout, DNS-rebinding TOCTOU (including under a configured `HTTP_PROXY`, which the pinning adapter also rejects outright since a proxy would resolve DNS itself), stored XSS, and the AnzoStore SPARQL injection
- `pytest tests/ingest/`: 266 passed, 2 skipped (10 pre-existing failures unrelated to this change — identical failure set confirmed on unmodified `main`); full regression sweep across `graph_store`, `export`, `triplet_store`, `parse`, and backup/restore: 313 passed
- **`Authorization`/`Proxy-Authorization` credentials could leak to a different origin across HTTP redirects, and several ingest paths bypassed the shared SSRF/redirect guard entirely** (#1067, closes #947) by @Sameer6305, reviewed by @KaifAhmad1
- `request_with_ssrf_guard()` previously only stripped sensitive headers from per-request `kwargs["headers"]` on a cross-origin redirect; session-level `Authorization`/`Proxy-Authorization` headers, `session.auth`, and `session.trust_env` (`.netrc` lookup) could all still resurrect credentials on the hop to a foreign origin. All five credential sources are now stripped case-insensitively, kept stripped for the remainder of a multi-hop redirect chain (no resurrection even if a later hop returns to the original host), and unconditionally restored via `finally` — including on exceptions and redirect-limit errors
- `MCPClient._send_request_http()` and `PublicAPIIngestor.detect_public_api()`/`ingest_public_api()` called `httpx.post()`/`requests.post()`/`session.request()` directly, bypassing `request_with_ssrf_guard()` entirely. Both now route through the shared guard, including when `validate_no_auth=False`
- `SeedDataManager.load_from_api()` mutated the caller-supplied `headers` dict in place when adding an API-key `Authorization` header, silently leaking the key back into a dict the caller might reuse elsewhere. Now copies before modifying
- **Fixed during review** (@KaifAhmad1): `allow_private_ips=True` (used to let MCP servers run on localhost/internal networks) was applied to every redirect hop, not just the operator-configured host — a compromised or malicious MCP server could 302-redirect to an internal address (e.g. `169.254.169.254` cloud metadata) and the guard would follow it unchecked, defeating the SSRF protection this PR otherwise adds. Added `allow_private_ips_on_redirect` to `request_with_ssrf_guard()`: a redirect target inherits the original host's private-IP trust only when it matches that host; any other host falls back to strict validation. `MCPClient` now pins `allow_private_ips_on_redirect=False`, so only same-host redirects on a trusted MCP server keep working — a cross-host hop into private address space is blocked
- **Fixed during review** (@KaifAhmad1): `detect_public_api()` only caught `requests.exceptions.RequestException`, but `request_with_ssrf_guard()` raises `ValidationError` (a disjoint hierarchy) for SSRF-blocked hosts, blocked redirect targets, missing `Location`, or exceeded redirect limits — unlike its sibling `ingest_public_api()`, which already caught it. Callers (including `is_public_api()`) got an undocumented raw `ValidationError` instead of `ProcessingError`, and the error-logging call was skipped. Now catches `(ValidationError, ProcessingError)` and re-raises, matching the sibling method
- **Fixed during review** (@KaifAhmad1): `detect_public_api()`/`ingest_public_api()` forwarded `**options` into `request_with_ssrf_guard(..., session=self.session, allow_private_ips=self.allow_private_ips, **request_options)` without stripping `session`/`allow_private_ips` from `request_options` first — a caller passing either through the per-call `**options` (a plausible mistake, since `allow_private_ips` is also a documented constructor-level knob) got a raw `TypeError: got multiple values for keyword argument`. Both are now popped from `request_options` before the call
- New regression coverage added during review: `TestAllowPrivateIpsOnRedirect` (cross-host redirect into private space blocked, same-host redirect trust preserved, default behavior unchanged for existing callers that don't pass the new kwarg) and `TestMCPClientAuthRedirect::test_redirect_to_private_ip_is_blocked`/`test_same_host_redirect_on_private_mcp_server_is_not_blocked` in `tests/ingest/test_auth_header_redirect_security.py`; `test_detect_public_api_propagates_ssrf_validation_error` and duplicate-kwarg regression tests for both methods in `tests/ingest/test_public_api_ingestor.py`
- `pytest tests/ingest/test_auth_header_redirect_security.py tests/ingest/test_public_api_ingestor.py tests/test_seed_manager.py tests/ingest/test_submodules.py tests/ingest/test_cookbook_integration.py`: 111 passed
- **`FeedIngestor`/`FeedMonitor` (RSS/Atom feed ingestion) had no SSRF protection, allowing requests to internal/private network targets** (#928, closes #927) by @ZohaibHassan16
- `FeedIngestor.ingest_feed()`, `discover_feeds()` (link-tag fetch, common-path HEAD probe, and feed-validation GET), and `FeedMonitor.check_updates()` all called `requests.get()`/`requests.head()` directly with default redirect-following and no scheme allowlist or private/loopback/link-local IP validation — despite `semantica/ingest/ssrf.py`'s `request_with_ssrf_guard()` already existing and being used by `web_ingestor.py`/`api_ingestor.py`. `ingest_feed()`'s own URL check only verified `urlparse(url).scheme`/`.netloc` were non-empty, never that the scheme was http/https or that the resolved target IP was safe. Reachable via the public `ingest_feed()`/`ingest()` entry points with any caller-supplied feed URL
- All 5 call sites now route through `request_with_ssrf_guard()`, which validates scheme (http/https only) and resolved IP before the request, and re-validates every redirect `Location` before following it — closing both the direct-IP and redirect-chain SSRF paths. Added an `allow_private_ips` config option to both `FeedIngestor` and `FeedMonitor`, consistent with the other ingestors
-1
View File
@@ -1,2 +1 @@
recursive-include semantica/static *
recursive-include semantica/ontology/vocabulary *.ttl
+23 -18
View File
@@ -2,15 +2,7 @@
<img src="Semantica Logo.png" alt="Semantica" width="420"/>
<div style="display:flex; gap:10px; align-items:center; flex-wrap:wrap;">
<a href="https://trendshift.io/repositories/18986?utm_source=repository-badge&amp;utm_medium=badge&amp;utm_campaign=badge-repository-18986" target="_blank" rel="noopener noreferrer">
<img src="https://trendshift.io/api/badge/repositories/18986" alt="semantica-agi/semantica | Trendshift" width="250" height="55"/>
</a>
<a href="https://trendshift.io/repositories/18986?utm_source=trendshift-badge&amp;utm_medium=badge&amp;utm_campaign=badge-trendshift-18986" target="_blank" rel="noopener noreferrer">
<img src="https://trendshift.io/api/badge/trendshift/repositories/18986/weekly?language=Python" alt="semantica-agi/semantica | Trendshift" width="250" height="55"/>
</a>
</div>
<a href="https://trendshift.io/repositories/18986?utm_source=repository-badge&amp;utm_medium=badge&amp;utm_campaign=badge-repository-18986" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/18986" alt="semantica-agi%2Fsemantica | Trendshift" width="250" height="55"/></a>
### Graph-Native Infrastructure for Context and Accountable AI Systems
@@ -303,10 +295,17 @@ graph.add_causal_relationship(d1, d2, relationship_type="CAUSED")
prov.track_entity("patient_P4821", source="ehr/medication_orders_2024.json",
metadata={"extractor": "NamedEntityRecognizer"})
# Export W3C PROV-O for regulator submission - to_kg_dict() is the official
# adapter that emits the {"entities": [...], "relationships": [...]} /
# source_id shape RDFExporter expects, so no manual field mapping is needed
kg = graph.to_kg_dict()
# Export W3C PROV-O for regulator submission - RDFExporter expects
# {"entities": [...], "relationships": [...]}, so map ContextGraph.to_dict()'s
# {"nodes": [...], "edges": [...]} shape onto it first
graph_dict = graph.to_dict()
kg = {
"entities": [{"id": n["id"], "type": n["type"], "text": n["content"]} for n in graph_dict["nodes"]],
"relationships": [
{"source_id": e["source"], "target_id": e["target"], "type": e["type"]}
for e in graph_dict["edges"]
],
}
RDFExporter().export(kg, "audit_trail.ttl", format="turtle")
```
@@ -880,14 +879,20 @@ fact = BiTemporalFact(
recorded_at=datetime(2024, 3, 5),
)
# Query facts valid within a time window - to_kg_dict() is the official
# adapter that emits {"entities", "relationships"} with source_id/target_id
# keys, the shape query_time_range() expects (no manual mapping required)
kg = graph.to_kg_dict()
# Query facts valid within a time window - query_time_range() expects
# {"relationships": [...]} with source_id/target_id keys, which differs from
# ContextGraph.to_dict()'s {"nodes", "edges"} shape, so map it first
graph_dict = graph.to_dict()
kg_relationships = {
"relationships": [
{**e, "source_id": e["source"], "target_id": e["target"]}
for e in graph_dict["edges"]
]
}
tq = TemporalGraphQuery()
facts_in_window = tq.query_time_range(
kg, query="valid_facts", start_time="2024-01-01", end_time="2024-12-31"
kg_relationships, query="valid_facts", start_time="2024-01-01", end_time="2024-12-31"
)
# Normalize natural language temporal expressions - returns a (start, end) range
+1 -1
View File
@@ -162,7 +162,7 @@ semantica-explorer --graph my_graph.json --no-browser
```
<Warning>
`--host 0.0.0.0` makes Explorer reachable on every network interface. Since v0.6.5 the Explorer API requires `SEMANTICA_API_KEY` (sent as the `X-API-Key` header) and fails closed with `503` when unconfigured; unauthenticated access is only possible when `SEMANTICA_ALLOW_ANONYMOUS=true` is set explicitly. Only use this on a trusted private network.
`--host 0.0.0.0` makes Explorer reachable on every network interface. The server has no built-in authentication. Only use this on a trusted private network.
</Warning>
+1 -10
View File
@@ -63,9 +63,7 @@ semantica-explorer --graph my_graph.json --no-browser
python -m semantica.explorer --graph my_graph.json
```
> **Security note:** Since v0.6.5 the Explorer API requires an API key on protected routes. Set the `SEMANTICA_API_KEY` environment variable and send it as the `X-API-Key` header; without a configured key, protected routes fail closed with `503` rather than serving anonymously. To opt into unauthenticated access for local development only, set `SEMANTICA_ALLOW_ANONYMOUS=true` explicitly. (`/api/health` and `/api/info` are intentionally unauthenticated.)
>
> The default `--host 127.0.0.1` binds to localhost only, so it is not reachable from other machines on your network. If you bind to `0.0.0.0`, all graph data is readable and writable by any host that can reach the port (subject to API-key auth). The CLI prints a warning when binding to a non-loopback host in anonymous mode or when `SEMANTICA_API_KEY` is unset.
> **Security note:** The Explorer API has no built-in authentication. The default `--host 127.0.0.1` binds to localhost only, so it is not reachable from other machines on your network. If you bind to `0.0.0.0`, all graph data is readable and writable by any host that can reach the port. The CLI will print a warning in that case.
---
@@ -150,8 +148,6 @@ This writes the compiled assets to `../semantica/static/`. The Python server the
| --- | --- | --- |
| `EXPLORER_CORS_ORIGINS` | `http://localhost:5173,http://127.0.0.1:5173` | Comma-separated list of allowed CORS origins |
| `EXPLORER_CORS_CREDENTIALS` | `false` | Set to `true` to allow credentialed cross-origin requests (only needed behind an authenticating reverse proxy) |
| `SEMANTICA_API_KEY` | *(unset)* | API key required on protected routes since v0.6.5; send it as the `X-API-Key` header. When unset, protected routes fail closed with `503`. |
| `SEMANTICA_ALLOW_ANONYMOUS` | `false` | Set to `true` to opt into unauthenticated access (local development only). |
---
@@ -255,11 +251,6 @@ Vite automatically tries the next available port and prints the actual URL in th
- Confirm the backend exposes the `/ws/graph-updates` WebSocket endpoint.
- Check DevTools → Network → WS tab for the connection status and error code.
- Ensure the backend version matches the frontend — mixing major versions can cause protocol mismatches.
- **Authentication:** `/ws/graph-updates` enforces the same API key as the REST routes. Browsers cannot set custom headers on a WebSocket handshake, so pass the key as a query parameter instead:
```
ws://127.0.0.1:8000/ws/graph-updates?api_key=<your-key>
```
Non-browser clients (native apps, scripts) may send it as the `X-API-Key` header. A missing or incorrect key results in close code `4401`; if `SEMANTICA_API_KEY` is unset and `SEMANTICA_ALLOW_ANONYMOUS` is not `true`, the connection is also rejected. Note that API keys in URLs appear in server logs — prefer the header for non-browser clients.
---
@@ -162,11 +162,7 @@ const SIGMA_SETTINGS = {
hideLabelsOnMove: true,
hideEdgesOnMove: true,
enableEdgeEvents: true,
// #1009: edge labels (the edge `type` — "works_for", "leads", ...) were
// hardcoded off, so edge text never rendered regardless of data. The
// labelDensity / labelGridCellSize / labelRenderedSizeThreshold settings
// below already throttle label density for both nodes and edges.
renderEdgeLabels: true,
renderEdgeLabels: false,
labelDensity: 0.7,
labelGridCellSize: 140,
zIndex: true,
@@ -745,12 +741,6 @@ function buildEffectAvailability(
? { enabled: true, available: true, reason: "Panel enabled" }
: { enabled: false, available: false, reason: "Disabled by toggle" };
// #1009: edge labels are immediately available once the graph is loaded —
// they have no async analytics or zoom-tier dependency.
const edgeLabels = effectsState.edgeLabelsEnabled
? { enabled: true, available: true, reason: "Ready" }
: { enabled: false, available: false, reason: "Disabled by toggle" };
const diagnostics = !GRAPH_THEME.effects.diagnostics.enabledInDev
? { enabled: false, available: false, reason: "Disabled in production" }
: effectsState.diagnosticsEnabled
@@ -768,7 +758,6 @@ function buildEffectAvailability(
communities,
centrality,
legend,
edgeLabels,
diagnostics,
};
}
@@ -1222,12 +1211,6 @@ function applySceneState(
size: resolvedStyle.size,
zIndex: resolvedStyle.zIndex,
curvature: resolvedStyle.curvature,
// #1009: Sigma's edge label renderer draws data.label — the graph
// stores the relationship type in edgeType, which the renderer never
// saw, so enabling renderEdgeLabels alone left edges blank.
// Use || rather than ?? so that an empty-string edgeType (possible
// when the API returns type: "") does not produce a blank label.
label: resolvedStyle.hidden ? undefined : String(attrs.edgeType || data.label || ""),
};
});
@@ -1312,9 +1295,6 @@ export const GraphCanvas = forwardRef<GraphCanvasHandle, GraphCanvasProps>(
const onEdgeClickRef = useRef(onEdgeClick);
const onSceneRuntimeChangeRef = useRef(onSceneRuntimeChange);
const onCameraStateChangeRef = useRef(onCameraStateChange);
// #1009: tracked as a ref so the Sigma creation effect always reads the
// current value without needing effectsState in its dependency array.
const effectsStateRef = useRef(effectsState);
const [hoveredNodeId, setHoveredNodeId] = useState<string | null>(null);
const [zoomTier, setZoomTier] = useState<GraphZoomTier>("overview");
const [analyticsSnapshot, setAnalyticsSnapshot] = useState<GraphAnalyticsSnapshot | null>(null);
@@ -1343,7 +1323,6 @@ export const GraphCanvas = forwardRef<GraphCanvasHandle, GraphCanvasProps>(
onEdgeClickRef.current = onEdgeClick;
onSceneRuntimeChangeRef.current = onSceneRuntimeChange;
onCameraStateChangeRef.current = onCameraStateChange;
effectsStateRef.current = effectsState;
const behaviors = useMemo<GraphBehavior[]>(
() => [
@@ -1856,13 +1835,7 @@ export const GraphCanvas = forwardRef<GraphCanvasHandle, GraphCanvasProps>(
return;
}
const sigma = new Sigma(displayGraphRef.current, containerRef.current, {
...SIGMA_SETTINGS,
// #1009: initialize with the current toggle value rather than the
// static default so that a user who disabled Edge Labels before
// graph/Sigma initialization sees the correct state after mount.
renderEdgeLabels: effectsStateRef.current.edgeLabelsEnabled,
});
const sigma = new Sigma(displayGraphRef.current, containerRef.current, SIGMA_SETTINGS);
sigmaRef.current = sigma;
appliedGraphVersionRef.current = graphVersionRef.current;
@@ -1964,17 +1937,6 @@ export const GraphCanvas = forwardRef<GraphCanvasHandle, GraphCanvasProps>(
});
}, [behaviors, dispatchToBehaviors, getBehaviorContext, graphReady, syncCameraState]);
// #1009: renderEdgeLabels follows the Effects-panel toggle instead of
// staying hardcoded — dense graphs get their label-free edges back.
useEffect(() => {
const sigma = sigmaRef.current;
if (!sigma) {
return;
}
sigma.setSetting("renderEdgeLabels", effectsState.edgeLabelsEnabled);
sigma.scheduleRefresh();
}, [effectsState.edgeLabelsEnabled]);
useEffect(() => {
return () => {
const sigma = sigmaRef.current;
@@ -148,7 +148,6 @@ const DEFAULT_EFFECTS_STATE: GraphEffectsState = {
communitiesEnabled: false,
centralityEnabled: false,
legendEnabled: false,
edgeLabelsEnabled: true,
diagnosticsEnabled: false,
lensMode: "neighborhood",
effectQuality: "bounded",
@@ -2099,15 +2099,6 @@ function createCollapsedNeighborhoodGraph(
return collapsedGraph;
}
// Normalize an edge relationship type: empty string, null, and undefined all
// fall back to the project-wide default used consistently across every
// aggregation path. Keep this local — it exists only to guarantee that the
// three code paths (single-entry, multi-entry, community-grouped) produce the
// same semantics and do not diverge again.
function normalizeEdgeType(value: string | null | undefined): string {
return value || "related_to";
}
function aggregateDisplayGraph(graphRef: GraphRef): Graph<NodeAttributes, EdgeAttributes> {
const aggregated = new Graph<NodeAttributes, EdgeAttributes>({
type: "directed",
@@ -2133,13 +2124,10 @@ function aggregateDisplayGraph(graphRef: GraphRef): Graph<NodeAttributes, EdgeAt
const [{ edgeId, attrs }] = entries;
aggregated.mergeDirectedEdgeWithKey(edgeId, sourceId, targetId, {
...attrs,
// #1009: normalize empty/null/undefined edgeType so Sigma's label
// renderer never receives a blank string on the single-entry path.
edgeType: normalizeEdgeType(attrs.edgeType),
dominantEdgeType: normalizeEdgeType(attrs.dominantEdgeType ?? attrs.edgeType),
rawEdgeIds: collectRawEdgeIds(attrs, edgeId),
isAggregated: isAggregatedEdgeAttributes(attrs),
aggregateCount: attrs.aggregateCount ?? collectRawEdgeIds(attrs, edgeId).length,
dominantEdgeType: attrs.dominantEdgeType ?? attrs.edgeType,
representativeWeight: attrs.representativeWeight ?? Number(attrs.weight ?? 1),
});
return;
@@ -2162,11 +2150,10 @@ function aggregateDisplayGraph(graphRef: GraphRef): Graph<NodeAttributes, EdgeAt
const rawEdgeIds = entries.flatMap(({ edgeId, attrs }) => collectRawEdgeIds(attrs, edgeId));
const typeCounts = new Map<string, number>();
entries.forEach(({ attrs }) => {
const edgeType = normalizeEdgeType(attrs.edgeType);
const edgeType = String(attrs.edgeType ?? "related_to");
typeCounts.set(edgeType, (typeCounts.get(edgeType) ?? 0) + 1);
});
const dominantEdgeType = [...typeCounts.entries()].sort((left, right) => right[1] - left[1])[0]?.[0]
?? normalizeEdgeType(representative.attrs.edgeType);
const dominantEdgeType = [...typeCounts.entries()].sort((left, right) => right[1] - left[1])[0]?.[0] ?? representative.attrs.edgeType ?? "related_to";
const reverseKey = `${targetId}${sourceId}`;
const isBidirectionalBundle = groupedEdges.has(reverseKey);
const syntheticEdgeId = `${AGGREGATED_EDGE_PREFIX}${sourceId}::${targetId}`;
@@ -2180,10 +2167,10 @@ function aggregateDisplayGraph(graphRef: GraphRef): Graph<NodeAttributes, EdgeAt
rawEdgeIds,
isAggregated: true,
aggregateCount: rawEdgeIds.length,
dominantEdgeType: dominantEdgeType,
dominantEdgeType: String(dominantEdgeType),
representativeWeight: Number(representative.attrs.weight ?? 1),
weight: Number(representative.attrs.weight ?? 1),
edgeType: representative.attrs.edgeType || dominantEdgeType,
edgeType: String(representative.attrs.edgeType ?? dominantEdgeType ?? "related_to"),
parallelCount: rawEdgeIds.length,
familySize: rawEdgeIds.length,
bundleKind: isBidirectionalBundle ? "bidirectional" : "parallel",
@@ -2293,7 +2280,7 @@ function buildCommunityGroupedGraph(): GraphDisplayResult {
};
bucket.rawEdgeIds.push(String(edgeId));
bucket.weight = Math.max(bucket.weight, Number((attrs as EdgeAttributes).weight ?? 1));
const edgeType = normalizeEdgeType((attrs as EdgeAttributes).edgeType);
const edgeType = String((attrs as EdgeAttributes).edgeType ?? "related_to");
bucket.typeCounts.set(edgeType, (bucket.typeCounts.get(edgeType) ?? 0) + 1);
groupedEdges.set(key, bucket);
});
@@ -2409,8 +2396,7 @@ function buildCommunityGroupedGraph(): GraphDisplayResult {
if (!visibleGroupedEdgeKeys.has(key)) {
return;
}
const dominantEdgeType = [...bundle.typeCounts.entries()].sort((left, right) => right[1] - left[1])[0]?.[0]
?? "related_to";
const dominantEdgeType = [...bundle.typeCounts.entries()].sort((left, right) => right[1] - left[1])[0]?.[0] ?? "related_to";
const reverseKey = `${bundle.targetId}${bundle.sourceId}`;
const syntheticEdgeId = `${AGGREGATED_EDGE_PREFIX}${key}`;
const aggregateCount = bundle.rawEdgeIds.length;
@@ -1,7 +1,6 @@
import type { CSSProperties } from "react";
import type {
GraphDiagnosticsSnapshot,
GraphEffectAvailability,
GraphEffectToggle,
} from "../types";
@@ -31,11 +30,6 @@ const EFFECT_ROWS: EffectRowConfig[] = [
label: "Neighborhood Lens",
description: "Local emphasis around the hovered or selected node.",
},
{
key: "edgeLabelsEnabled",
label: "Edge Labels",
description: "Draw the relationship type on graph edges. Off restores label-free edges on dense graphs.",
},
{
key: "legendEnabled",
label: "Semantic Legend",
@@ -43,17 +37,6 @@ const EFFECT_ROWS: EffectRowConfig[] = [
},
];
// Maps the effect toggle keys rendered by this plugin to their corresponding
// availability keys in GraphDiagnosticsSnapshot["effectAvailability"]. Kept
// local because this plugin only renders a subset of all effects.
const EFFECT_AVAILABILITY_KEYS: Partial<Record<GraphEffectToggle, keyof GraphDiagnosticsSnapshot["effectAvailability"]>> = {
pathPulseEnabled: "pathPulse",
pathFlowEnabled: "pathFlow",
lensEnabled: "lens",
edgeLabelsEnabled: "edgeLabels",
legendEnabled: "legend",
};
function renderAvailabilityText(availability: GraphEffectAvailability) {
if (availability.available) {
if (typeof availability.visibleSegments === "number" && typeof availability.segmentCap === "number") {
@@ -156,9 +139,15 @@ export const explorationEffectsPlugin: GraphPlugin = {
description={row.description}
checked={effectsState[row.key]}
availability={
(EFFECT_AVAILABILITY_KEYS[row.key] !== undefined
? availability?.[EFFECT_AVAILABILITY_KEYS[row.key]!]
: undefined) ?? {
availability?.[
row.key === "pathPulseEnabled"
? "pathPulse"
: row.key === "pathFlowEnabled"
? "pathFlow"
: row.key === "lensEnabled"
? "lens"
: "legend"
] ?? {
enabled: effectsState[row.key],
available: false,
reason: "Waiting for graph runtime",
@@ -47,11 +47,6 @@ const SCENE_EFFECT_ROWS: EffectRowConfig[] = [
label: "Contours",
description: "Low-contrast density halos around the strongest visible anchors.",
},
{
key: "edgeLabelsEnabled",
label: "Edge Labels",
description: "Draw the relationship type on graph edges. Off restores label-free edges on dense graphs.",
},
{
key: "legendEnabled",
label: "Regions Summary",
@@ -88,7 +83,6 @@ const AVAILABILITY_KEYS: Record<GraphEffectToggle, keyof GraphDiagnosticsSnapsho
communitiesEnabled: "communities",
centralityEnabled: "centrality",
legendEnabled: "legend",
edgeLabelsEnabled: "edgeLabels",
diagnosticsEnabled: "diagnostics",
};
@@ -103,7 +103,6 @@ export type GraphEffectToggle =
| "communitiesEnabled"
| "centralityEnabled"
| "legendEnabled"
| "edgeLabelsEnabled"
| "diagnosticsEnabled";
export interface GraphEffectsState {
@@ -114,7 +113,6 @@ export interface GraphEffectsState {
semanticRegionsEnabled: boolean;
contoursEnabled: boolean;
pathfindingEnabled: boolean;
edgeLabelsEnabled: boolean;
communitiesEnabled: boolean;
centralityEnabled: boolean;
legendEnabled: boolean;
@@ -188,7 +186,6 @@ export interface GraphDiagnosticsSnapshot {
communities: GraphEffectAvailability;
centrality: GraphEffectAvailability;
legend: GraphEffectAvailability;
edgeLabels: GraphEffectAvailability;
diagnostics: GraphEffectAvailability;
};
}
@@ -1061,198 +1061,3 @@ test("checkGroupedViewAvailability returns available when communities exist", ()
assert.equal(result.reason, null);
});
// ── #1009: edge label data-path regression tests ─────────────────────────────
test("resolveDisplayGraph parallel-bundle preserves edgeType on aggregated edge", () => {
addNode("a");
addNode("b");
batchMergeEdges([
{ id: "e1", source: "a", target: "b", attributes: { edgeType: "causes", weight: 1, properties: {} } },
{ id: "e2", source: "a", target: "b", attributes: { edgeType: "causes", weight: 2, properties: {} } },
]);
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
assert.equal(displayGraph.size, 1);
const edgeId = displayGraph.edges()[0];
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string; isAggregated?: boolean };
assert.equal(attrs.isAggregated, true);
// The aggregated representative must carry the relationship text through to
// the edgeReducer's label assignment.
assert.equal(typeof attrs.edgeType, "string");
assert.ok((attrs.edgeType ?? "").length > 0, "aggregated edge must have a non-empty edgeType");
});
test("resolveDisplayGraph parallel-bundle picks dominant edgeType across mixed types", () => {
addNode("a");
addNode("b");
batchMergeEdges([
{ id: "e1", source: "a", target: "b", attributes: { edgeType: "inhibits", weight: 1, properties: {} } },
{ id: "e2", source: "a", target: "b", attributes: { edgeType: "inhibits", weight: 1, properties: {} } },
{ id: "e3", source: "a", target: "b", attributes: { edgeType: "activates", weight: 1, properties: {} } },
]);
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
const edgeId = displayGraph.edges()[0];
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string; dominantEdgeType?: string };
// "inhibits" appears twice so it must be the dominant type.
assert.equal(attrs.edgeType, "inhibits");
assert.equal(attrs.dominantEdgeType, "inhibits");
});
test("resolveDisplayGraph grouped view community edges carry non-empty edgeType", () => {
const left = ["g1", "g2", "g3", "g4"];
const right = ["h1", "h2", "h3", "h4"];
[...left, ...right].forEach((nodeId, index) => addNode(nodeId, index < left.length ? "left" : "right"));
let edgeIndex = 0;
for (let i = 0; i < left.length; i += 1) {
for (let j = 0; j < left.length; j += 1) {
if (i !== j) {
batchMergeEdges([{
id: `lg-${edgeIndex++}`,
source: left[i],
target: left[j],
attributes: { edgeType: "co_occurs", weight: 3, properties: {} },
}]);
}
}
}
for (let i = 0; i < right.length; i += 1) {
for (let j = 0; j < right.length; j += 1) {
if (i !== j) {
batchMergeEdges([{
id: `rg-${edgeIndex++}`,
source: right[i],
target: right[j],
attributes: { edgeType: "co_occurs", weight: 3, properties: {} },
}]);
}
}
}
batchMergeEdges([{ id: "bridge-g", source: "g1", target: "h1", attributes: { edgeType: "interacts_with", weight: 0.1, properties: {} } }]);
const { graph: displayGraph, state } = resolveDisplayGraph("", [], [], "grouped", { aggregationEnabled: true });
assert.equal(state.groupedViewAvailable, true);
const communityEdges = displayGraph.edges().filter((edgeId) => {
const attrs = displayGraph.getEdgeAttributes(edgeId) as { bundleKind?: string };
return attrs.bundleKind === "community";
});
assert.ok(communityEdges.length > 0, "expected at least one community bundle edge");
for (const edgeId of communityEdges) {
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string };
assert.equal(typeof attrs.edgeType, "string");
assert.ok((attrs.edgeType ?? "").length > 0, `community edge ${edgeId} must have a non-empty edgeType`);
}
});
test("resolveDisplayGraph raw edge preserves exact edgeType string for label rendering", () => {
addNode("src");
addNode("tgt");
batchMergeEdges([{
id: "raw-1",
source: "src",
target: "tgt",
attributes: { edgeType: "works_for", weight: 1, properties: {} },
}]);
// In full view without aggregation the edge passes through unchanged.
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: false });
assert.equal(displayGraph.size, 1);
const edgeId = displayGraph.edges()[0];
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string };
assert.equal(attrs.edgeType, "works_for");
});
test("resolveDisplayGraph does not produce empty-string edgeType on aggregated edges when source has empty type", () => {
addNode("a");
addNode("b");
// Simulate an API response where type is empty string — the aggregation
// path must not propagate a blank label.
batchMergeEdges([
{ id: "e-empty-1", source: "a", target: "b", attributes: { edgeType: "", weight: 1, properties: {} } },
{ id: "e-empty-2", source: "a", target: "b", attributes: { edgeType: "", weight: 1, properties: {} } },
]);
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
const edgeId = displayGraph.edges()[0];
const attrs = displayGraph.getEdgeAttributes(edgeId) as {
edgeType?: string;
isAggregated?: boolean;
};
assert.equal(attrs.isAggregated, true);
// The aggregation falls back to "related_to" when all source edgeTypes are
// empty, so the rendered label should never be an empty string.
assert.equal(attrs.edgeType, "related_to");
});
test("resolveEdgeElementStyle hidden class produces hidden:true for suppressed edges", () => {
// Verify the data condition the edgeReducer relies on: hidden-classified
// edges must have hidden:true so that the label assignment sets undefined.
const style = resolveEdgeElementStyle(
GRAPH_THEME,
"overview",
"inactive",
{
edgeType: "causes",
weight: 1,
properties: {},
edgeVariant: "line",
visualPriority: 0.05,
baseSize: 0.3,
},
"source",
"target",
"full",
"inactive-edge",
"hidden",
);
assert.equal(style.hidden, true);
});
// ── #1009 maintainer-blocking regression: single-edge empty edgeType ─────────
test("resolveDisplayGraph single-edge normalizes empty-string edgeType to related_to", () => {
addNode("a");
addNode("b");
// One edge only — exercises the entries.length === 1 path in aggregateDisplayGraph.
batchMergeEdges([{
id: "e-single-empty",
source: "a",
target: "b",
attributes: { edgeType: "", weight: 1, properties: {} },
}]);
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
assert.equal(displayGraph.size, 1);
const edgeId = displayGraph.edges()[0];
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string; dominantEdgeType?: string };
assert.equal(attrs.edgeType, "related_to",
"single-edge path must normalize empty edgeType to the canonical fallback");
assert.equal(attrs.dominantEdgeType, "related_to",
"single-edge dominantEdgeType must also be normalized");
});
test("resolveDisplayGraph single-edge preserves a valid non-empty edgeType unchanged", () => {
addNode("a");
addNode("b");
batchMergeEdges([{
id: "e-single-valid",
source: "a",
target: "b",
attributes: { edgeType: "works_for", weight: 1, properties: {} },
}]);
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
assert.equal(displayGraph.size, 1);
const edgeId = displayGraph.edges()[0];
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string };
assert.equal(attrs.edgeType, "works_for",
"single-edge path must not alter a valid relationship type");
});
+2 -12
View File
@@ -93,14 +93,7 @@ def _handle_tools_call(req_id: Any, params: dict) -> dict:
result = tool["_handler"](args)
except Exception as exc:
log.exception("Tool %s raised an exception", name)
# The exception's class name (e.g. "ValidationError", "TimeoutError")
# is safe to surface — unlike str(exc), it never carries paths,
# connection strings, or other internal detail — and lets the
# client distinguish failure kinds without a full message.
return _err(
req_id, _INTERNAL_ERROR,
f"Tool '{name}' failed ({type(exc).__name__}). See server logs for details.",
)
return _err(req_id, _INTERNAL_ERROR, str(exc))
# MCP spec: content must be a list of content items
return _ok(req_id, {
@@ -178,10 +171,7 @@ class SemanticaMCPServer:
log.exception("Unhandled error in method %s", method)
if req_id is None:
return None
return _err(
req_id, _INTERNAL_ERROR,
f"Method '{method}' failed ({type(exc).__name__}). See server logs for details.",
)
return _err(req_id, _INTERNAL_ERROR, str(exc))
# ------------------------------------------------------------------
def run(self) -> None:
+1 -1
View File
@@ -271,7 +271,7 @@ include = ["semantica*", "integrations*"]
[tool.setuptools.package-data]
# Explicit patterns are more reliable than **/* across setuptools versions.
# static/* covers index.html / favicon; static/assets/* covers all JS/CSS chunks.
"semantica" = ["static/*", "static/assets/*", "ontology/vocabulary/*.ttl"]
"semantica" = ["static/*", "static/assets/*"]
[tool.black]
line-length = 88
+5 -58
View File
@@ -20,7 +20,7 @@ if sys.platform == "win32":
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
from dataclasses import asdict, dataclass, field, is_dataclass
from pathlib import Path, PurePosixPath, PureWindowsPath
from pathlib import Path
from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Sequence, Tuple
import yaml
@@ -3933,68 +3933,15 @@ def backup_restore(cli_ctx: CLIContext, source: str, local_dry: bool) -> None:
try:
if _tf.is_tarfile(str(work_path)):
restore_root = Path.cwd().resolve()
restore_root = Path.cwd()
with _tf.open(str(work_path), "r:*") as tar:
# Dry-run listing was already handled above; extract now
for member in tar.getmembers():
# Strip the leading "semantica-backup/" prefix
member.name = member.name.replace("semantica-backup/", "", 1)
if not member.name:
continue
# Reject members whose resolved path escapes the
# restore root (path traversal / absolute paths),
# regardless of the "semantica-backup/" prefix.
member_path = (restore_root / member.name).resolve()
try:
member_path.relative_to(restore_root)
except ValueError:
raise click.ClickException(
f"Refusing to restore '{member.name}': "
"path escapes the restore directory."
)
# Reject symlink/hardlink members whose target
# escapes the restore root. Checked two ways:
# lexically (linkname itself, so an absolute path or
# a literal ".." segment is rejected outright, with
# no dependence on what else does or doesn't already
# exist on disk) and by resolution (catches any
# remaining traversal the lexical check misses).
if member.issym() or member.islnk():
linkname = member.linkname or ""
linkname_parts = PurePosixPath(
linkname.replace("\\", "/")
).parts
if (
not linkname
or os.path.isabs(linkname)
or PureWindowsPath(linkname).is_absolute()
or ".." in linkname_parts
):
raise click.ClickException(
f"Refusing to restore '{member.name}': "
"link target is absolute or traverses "
"out of the archive."
)
link_target = (
member_path.parent / linkname
).resolve()
try:
link_target.relative_to(restore_root)
except ValueError:
raise click.ClickException(
f"Refusing to restore '{member.name}': "
"link target escapes the restore directory."
)
extract_kwargs: Dict[str, Any] = {"path": str(restore_root)}
if hasattr(_tf, "data_filter"):
# Python >=3.12: also reject device files, and
# further harden the traversal/ownership checks.
extract_kwargs["filter"] = "data"
tar.extract(member, **extract_kwargs)
console.print(f" restored: {member.name}")
if member.name:
tar.extract(member, path=str(restore_root))
console.print(f" restored: {member.name}")
elif src.is_dir():
restore_root = Path.cwd()
for f in src.rglob("*"):
+4 -95
View File
@@ -72,9 +72,9 @@ Example Usage:
... node_embeddings=True)
>>>
>>> # Basic graph operations
>>> graph.add_node("Python", "language", popularity="high")
>>> graph.add_node("Programming", "concept")
>>> graph.add_edge("Python", "Programming", "related_to")
>>> graph.add_node("Python", type="language", properties={"popularity": "high"})
>>> graph.add_node("Programming", type="concept")
>>> graph.add_edge("Python", "Programming", type="related_to")
>>> centrality = graph.get_node_centrality("Python")
>>> similar = graph.find_similar_nodes("Python", similarity_type="content")
>>> analysis = graph.analyze_graph_with_kg()
@@ -88,7 +88,7 @@ Example Usage:
... confidence=0.95,
... entities=["customer_123", "property_456"]
... )
>>> precedents = graph.find_precedents(decision_id, limit=5)
>>> precedents = graph.find_precedents("loan_approval", limit=5)
>>> influence = graph.analyze_decision_influence(decision_id)
>>> insights = graph.get_decision_insights()
>>> causality = graph.trace_decision_causality(decision_id)
@@ -2449,97 +2449,6 @@ class ContextGraph:
},
}
def to_kg_dict(self, entities_only: bool = False) -> Dict[str, Any]:
"""Export graph in the canonical knowledge-graph shape.
This is the official adapter that converts the ContextGraph's internal
``{"nodes", "edges"}`` / ``source`` representation into the
``{"entities", "relationships"}`` / ``source_id`` shape expected by
downstream consumers such as
:class:`~semantica.export.rdf_exporter.RDFExporter` and
:meth:`~semantica.kg.temporal_query.TemporalGraphQuery.query_time_range`.
Users no longer need to hand-map field names between APIs.
Args:
entities_only: If True, only nodes whose ``node_type`` is
``"entity"`` are exported as entities. When False (default),
every node is exported. Relationships whose endpoints are not
in the exported entity set are dropped to avoid dangling
references in downstream consumers.
Returns:
dict: A knowledge-graph dictionary with:
- ``entities``: list of ``{"id", "text", "type", "properties",
"metadata"}`` (plus ``valid_from`` / ``valid_until`` when set)
- ``relationships``: list of ``{"source_id", "target_id",
"type", "weight", "id", "familyId"}`` (plus ``metadata`` and
``valid_from`` / ``valid_until`` when set)
- ``statistics``: ``{"entity_count", "relationship_count"}``
"""
with self._lock:
entities_out = []
for n in self.nodes.values():
if entities_only and n.node_type != "entity":
continue
# Normalize the entity id to ``str`` so it matches ContextEdge,
# which coerces its endpoints to ``str`` in ``__post_init__``.
# Without this, non-string node ids (e.g. numeric ids loaded via
# ``from_dict``) would fail the ``valid_ids`` membership check
# below and silently drop otherwise-valid relationships.
entity_id = str(n.node_id)
entity: Dict[str, Any] = {
"id": entity_id,
"text": n.content,
"type": n.node_type,
# ``properties`` / ``metadata`` may be ``None`` when a node
# was loaded from JSON containing an explicit ``null``;
# guard with ``or {}`` so ``dict(...)`` never raises.
"properties": dict(n.properties or {}),
"metadata": dict(n.metadata or {}),
}
if n.valid_from is not None:
entity["valid_from"] = n.valid_from
if n.valid_until is not None:
entity["valid_until"] = n.valid_until
entities_out.append(entity)
# When only entity nodes are exported, drop relationships whose
# endpoints were filtered out so downstream consumers never see a
# source_id/target_id that is absent from ``entities``.
valid_ids = {e["id"] for e in entities_out} if entities_only else None
relationships_out = []
for e in self.edges:
if valid_ids is not None and (
e.source_id not in valid_ids or e.target_id not in valid_ids
):
continue
rel: Dict[str, Any] = {
"id": e.edge_id,
"familyId": e.family_id or e.edge_id,
"source_id": e.source_id,
"target_id": e.target_id,
"type": e.edge_type,
"weight": e.weight,
}
if e.metadata:
rel["metadata"] = dict(e.metadata)
if e.valid_from is not None:
rel["valid_from"] = e.valid_from
if e.valid_until is not None:
rel["valid_until"] = e.valid_until
relationships_out.append(rel)
return {
"entities": entities_out,
"relationships": relationships_out,
"statistics": {
"entity_count": len(entities_out),
"relationship_count": len(relationships_out),
},
}
def from_dict(self, graph_dict: Dict[str, Any]) -> None:
"""Load graph from dictionary format."""
# Clear existing graph
+1 -2
View File
@@ -45,7 +45,6 @@ License: MIT
from dataclasses import dataclass
from typing import Any, Dict, List, Optional, Union
from ..utils.entity_ids import get_entity_id
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -505,7 +504,7 @@ class EntityMerger:
# Record source entities
provenance["merged_from"] = [
{
"id": get_entity_id(e),
"id": self._get_entity_value(e, "id"),
"name": self._get_entity_value(e, "name"),
"source": self._get_entity_value(e, "metadata", {}).get("source") if hasattr(e, "metadata") or isinstance(e, dict) else None,
}
+2 -9
View File
@@ -45,7 +45,6 @@ from dataclasses import dataclass, field
from enum import Enum
from typing import Any, Callable, Dict, List, Optional, Tuple, Union
from ..utils.entity_ids import get_entity_id
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -318,20 +317,14 @@ class MergeStrategyManager:
message=f"Building merged entity... ({current_step}/{total_steps}, remaining: {remaining_steps} steps)"
)
# Build merged entity
merged_from = []
for entity in entities:
entity_id = get_entity_id(entity)
if entity_id is not None:
merged_from.append(entity_id)
merged_entity = {
"id": get_entity_id(base_entity),
"id": base_entity.get("id"),
"name": self._merge_top_level_field("name", entities, base_entity),
"type": self._merge_top_level_field("type", entities, base_entity),
"properties": merged_properties,
"relationships": merged_relationships,
"metadata": self._merge_metadata(entities, base_entity),
"merged_from": merged_from,
"merged_from": [e.get("id") for e in entities if e.get("id")],
"merge_strategy": strategy.value,
}
+11 -46
View File
@@ -33,42 +33,11 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Set, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory, hash_data
from ..utils.helpers import ensure_directory
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
SEMANTICA_NS = "https://semantica.dev/ns#"
#: Written when an entity carries no type of its own. A full IRI rather than the
#: prefixed form, because the Turtle serializer writes it inside angle brackets,
#: where `semantica:Entity` would be read as an IRI in the scheme `semantica`
#: rather than as the prefix expansion (issue #1101).
DEFAULT_ENTITY_TYPE = f"{SEMANTICA_NS}Entity"
#: Written when a relationship carries no type of its own. Same reasoning.
DEFAULT_RELATION_TYPE = f"{SEMANTICA_NS}related_to"
def mint_entity_iri(text: str) -> str:
"""Mint a stable IRI for an entity that arrived without an id.
Python's builtin ``hash()`` is randomised per process (PYTHONHASHSEED), so
minting from it gave the same entity a different IRI on every run: exports
could not be diffed, deduplicated against an earlier load, or joined to a
provenance record written by an earlier process. SHA-256 is stable across
runs and machines, which is what an identifier has to be.
"""
digest = hash_data(str(text))[:16]
return f"{SEMANTICA_NS}entity_{digest}"
def mint_relationship_iri(index: int, source: Any, target: Any) -> str:
"""Mint a stable IRI for a relationship that arrived without an id."""
digest = hash_data(f"{source}\x00{target}")[:16]
return f"{SEMANTICA_NS}rel_{index}_{digest}"
class NamespaceManager:
"""
RDF namespace management engine.
@@ -391,9 +360,9 @@ class RDFSerializer:
entity_id = entity.get("id")
if not entity_id:
entity_text = entity.get("text", "")
entity_id = mint_entity_iri(entity_text)
entity_id = f"semantica:entity_{hash(entity_text)}"
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
entity_type = entity.get("type", "semantica:Entity")
text = entity.get("text") or entity.get("label", "")
confidence = entity.get("confidence", 1.0)
@@ -407,7 +376,7 @@ class RDFSerializer:
for idx, rel in enumerate(relationships):
source_id = rel.get("source_id") or rel.get("source")
target_id = rel.get("target_id") or rel.get("target")
rel_type = rel.get("type", DEFAULT_RELATION_TYPE)
rel_type = rel.get("type", "semantica:related_to")
lines.append(f"<{source_id}> <{rel_type}> <{target_id}> .")
@@ -444,14 +413,10 @@ class RDFSerializer:
if time_axis in ("transaction", "both"):
axes.append(("tx", rel.get("recorded_at"), rel.get("superseded_at")))
# Resolve endpoints the same way serialize_to_turtle does: both
# representations are accepted upstream, and minting from source_id
# alone hashes empty strings for every relationship that uses source,
# so unrelated relationships at the same index would collide on a
# deterministic IRI.
source_id = rel.get("source_id") or rel.get("source") or ""
target_id = rel.get("target_id") or rel.get("target") or ""
rel_base_id = rel.get("id") or mint_relationship_iri(idx, source_id, target_id)
rel_base_id = (
rel.get("id")
or f"semantica:rel_{idx}_{hash(str(rel.get('source_id', '')) + str(rel.get('target_id', '')))}"
)
lines = [""] # blank separator
for axis_name, from_val, until_val in axes:
@@ -523,9 +488,9 @@ class RDFSerializer:
entity_id = entity.get("id")
if not entity_id:
entity_text = entity.get("text", "")
entity_id = mint_entity_iri(entity_text)
entity_id = f"semantica:entity_{hash(entity_text)}"
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
entity_type = entity.get("type", "semantica:Entity")
text = entity.get("text") or entity.get("label", "")
confidence = entity.get("confidence", 1.0)
@@ -679,7 +644,7 @@ class RDFSerializer:
entity_id = entity.get("id")
if not entity_id:
entity_text = entity.get("text", "")
entity_id = mint_entity_iri(entity_text)
entity_id = f"semantica:entity_{hash(entity_text)}"
subject = expand_uri(entity_id)
+6 -16
View File
@@ -23,7 +23,6 @@ Author: Semantica Contributors
License: MIT
"""
import html
import json
from datetime import datetime
from pathlib import Path
@@ -363,7 +362,7 @@ class ReportGenerator:
' <meta name="viewport" content="width=device-width, initial-scale=1.0">'
)
title = data.get("title", "Report")
lines.append(f" <title>{html.escape(str(title))}</title>")
lines.append(f" <title>{title}</title>")
lines.append(" <style>")
lines.append(" body { font-family: Arial, sans-serif; margin: 20px; }")
lines.append(" h1 { color: #333; }")
@@ -381,14 +380,11 @@ class ReportGenerator:
# Title
title = data.get("title", "Report")
lines.append(f" <h1>{html.escape(str(title))}</h1>")
lines.append(f" <h1>{title}</h1>")
# Generated at
if "generated_at" in data:
lines.append(
f' <p><strong>Generated:</strong> '
f'{html.escape(str(data["generated_at"]))}</p>'
)
lines.append(f' <p><strong>Generated:</strong> {data["generated_at"]}</p>')
# Summary
if "summary" in data:
@@ -397,13 +393,10 @@ class ReportGenerator:
if isinstance(summary, dict):
lines.append(" <ul>")
for key, value in summary.items():
lines.append(
f" <li><strong>{html.escape(str(key))}:</strong> "
f"{html.escape(str(value))}</li>"
)
lines.append(f" <li><strong>{key}:</strong> {value}</li>")
lines.append(" </ul>")
else:
lines.append(f" <p>{html.escape(str(summary))}</p>")
lines.append(f" <p>{summary}</p>")
# Metrics
if "metrics" in data:
@@ -490,10 +483,7 @@ class ReportGenerator:
else:
value_str = str(value)
lines.append(
f" <tr><td>{html.escape(str(key))}</td>"
f"<td>{html.escape(value_str)}</td></tr>"
)
lines.append(f" <tr><td>{key}</td><td>{value_str}</td></tr>")
lines.append(" </table>")
-11
View File
@@ -66,12 +66,6 @@ except (ImportError, OSError):
# Helpers
# ---------------------------------------------------------------------------
# create_index's index_type reaches a raw SQL keyword position (`USING
# {index_type}`) that can't be bound as a query parameter; only the
# documented, PostgreSQL-recognized types are allowed through.
_ALLOWED_INDEX_TYPES = frozenset({"btree", "gin", "hash", "gist", "brin"})
def _sanitize_label(label: str) -> str:
"""
Sanitize a Cypher label to prevent injection.
@@ -1220,11 +1214,6 @@ class ApacheAgeStore:
safe_label = _sanitize_label(label)
if not re.match(r"^[A-Za-z_][A-Za-z0-9_]*$", property_name):
raise ValidationError(f"Invalid property name: '{property_name}'")
if index_type not in _ALLOWED_INDEX_TYPES:
raise ValidationError(
f"Invalid index_type: {index_type!r}. "
f"Allowed: {sorted(_ALLOWED_INDEX_TYPES)}"
)
index_name = options.get(
"index_name", f"idx_{self.graph_name}_{safe_label}_{property_name}"
-19
View File
@@ -503,16 +503,6 @@ class Neo4jStore:
Returns:
List of matching nodes
"""
# LIMIT can't be bound as a query parameter in a way Neo4j accepts
# here, so it's interpolated directly; validate explicitly rather
# than trust the `limit: int` type hint, which Python doesn't
# enforce at runtime. Done outside the try/except below so a bad
# limit raises ValidationError, not a generic ProcessingError.
try:
limit = int(limit)
except (TypeError, ValueError) as exc:
raise ValidationError(f"Invalid limit: {limit!r}") from exc
try:
# Build query
if labels:
@@ -708,15 +698,6 @@ class Neo4jStore:
Returns:
List of matching relationships
"""
# See get_nodes: LIMIT is interpolated directly, so validate
# explicitly rather than trust the unenforced `limit: int` hint,
# outside the try/except below so a bad limit raises
# ValidationError, not a generic ProcessingError.
try:
limit = int(limit)
except (TypeError, ValueError) as exc:
raise ValidationError(f"Invalid limit: {limit!r}") from exc
try:
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
+5 -115
View File
@@ -29,7 +29,6 @@ License: MIT
"""
import json
import re
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any, Dict, List, Optional
@@ -39,96 +38,6 @@ from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
_IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
# Fragments that turn a filter/order clause into a second statement, a
# data-exfiltration UNION, a time-based blind-injection oracle, or schema
# enumeration, rather than a boolean/ordering expression.
_SQL_FRAGMENT_BLOCKLIST_RE = re.compile(
r";|--|/\*|\*/|\bunion\b|\binsert\b|\bupdate\b|\bdelete\b|\bdrop\b|"
r"\balter\b|\bcreate\b|\bexec\b|\bexecute\b|\bgrant\b|\brevoke\b|"
r"\battach\b|\bpragma\b|\bxp_\w+|\bsp_\w+|\binto\s+outfile\b|\bload_file\b|"
r"\bsleep\s*\(|\bbenchmark\s*\(|\bpg_sleep\s*\(|\bwaitfor\b|"
r"\bdbms_\w+|\butl_\w+|\binformation_schema\b|\bpg_catalog\b",
re.IGNORECASE,
)
# SQL single-quoted string literals ('' is the standard escaped-quote) and
# double-quoted identifiers ("" likewise) — matched only when properly
# closed, so a malformed/unterminated quote sequence is left alone and
# still hits the blocklist above rather than being treated as "inside a
# literal" and skipped.
_SQL_STRING_LITERAL_RE = re.compile(r"'(?:[^']|'')*'")
_SQL_QUOTED_IDENTIFIER_RE = re.compile(r'"(?:[^"]|"")*"')
def _mask_sql_literals(fragment: str) -> str:
"""Blank the contents of quoted literals so they can't trip the blocklist.
A legitimate value or quoted identifier that happens to contain a
blocked word or character as *data* e.g. ``status = 'union'`` or
``"my--column" = 1`` is not SQL syntax and shouldn't be rejected as
if it were. Only the quoted span's interior is replaced (with `?`,
keeping the surrounding quotes and the fragment's length/positions
intact for the error message); text outside any properly closed quote
is passed through unchanged and still fully scrutinized.
"""
fragment = _SQL_STRING_LITERAL_RE.sub(
lambda m: "'" + "?" * (len(m.group(0)) - 2) + "'", fragment
)
fragment = _SQL_QUOTED_IDENTIFIER_RE.sub(
lambda m: '"' + "?" * (len(m.group(0)) - 2) + '"', fragment
)
return fragment
def _validate_sql_identifier(name: str, kind: str) -> str:
"""Validate a table/schema name used as a raw SQL identifier.
``export_table_data`` interpolates *name* directly into the query text
(SQLAlchemy has no bind-parameter syntax for identifiers), so anything
outside a plain alphanumeric/underscore identifier is a potential
breakout of the surrounding ``"..."`` quoting.
"""
if not isinstance(name, str) or not _IDENTIFIER_RE.match(name):
raise ValidationError(
f"Invalid {kind}: {name!r}. Must start with a letter or "
"underscore and contain only alphanumeric characters and "
"underscores."
)
return name
def _validate_sql_fragment(fragment: str, kind: str) -> str:
"""Reject WHERE/ORDER BY fragments that smuggle a second statement.
These clauses can't be bound as query parameters (they're arbitrary
boolean/ordering expressions, not values), so this blocks the concrete
injection primitives (statement separators, comments, UNION, DML/DDL
keywords, time-based blind oracles, schema enumeration) rather than
parameterizing.
This is a blocklist, not a grammar: it cannot exhaustively prove
*fragment* is safe, only reject known-dangerous constructs, so a
boolean-blind subquery expressed with none of the blocked keywords
(e.g. ``id = (SELECT 1 FROM t WHERE ...)``) still passes. ``where``/
``order_by`` are a raw-SQL-fragment API by design (see
``export_table_data``'s docstring); treat them as trusted/operator
input, not something to expose directly to untrusted end users.
"""
if not isinstance(fragment, str):
raise ValidationError(f"Invalid {kind}: must be a string")
# Check the blocklist against literal-masked text so a blocked word
# appearing only as quoted data (not as SQL syntax) doesn't false-
# positive; the original, unmodified fragment is still what's returned
# and used in the query.
if _SQL_FRAGMENT_BLOCKLIST_RE.search(_mask_sql_literals(fragment)):
raise ValidationError(
f"Invalid {kind}: {fragment!r} contains disallowed SQL "
"keywords or statement-boundary characters"
)
return fragment
@dataclass
class TableData:
@@ -344,13 +253,8 @@ class DataExporter:
schema: Schema name (for databases with schema support, optional)
limit: Maximum number of rows to export (optional)
offset: Row offset for pagination (optional)
where: WHERE clause for filtering (optional, e.g., "age > 18").
Raw SQL, checked against a keyword/character blocklist (see
``_validate_sql_fragment``) but not fully sanitized treat
as trusted/operator input, never pass untrusted end-user
text here directly.
order_by: ORDER BY clause for sorting (optional, e.g., "name ASC").
Same trust requirement as ``where``.
where: WHERE clause for filtering (optional, e.g., "age > 18")
order_by: ORDER BY clause for sorting (optional, e.g., "name ASC")
**options: Additional export options (unused)
Returns:
@@ -365,16 +269,7 @@ class DataExporter:
ProcessingError: If table export fails
"""
try:
from sqlalchemy import inspect, text
_validate_sql_identifier(table_name, "table_name")
if schema:
_validate_sql_identifier(schema, "schema")
if where:
_validate_sql_fragment(where, "where")
if order_by:
_validate_sql_fragment(order_by, "order_by")
from sqlalchemy import inspect
inspector = inspect(connection)
# Get column information
@@ -449,8 +344,6 @@ class DataExporter:
schema=schema,
)
except ValidationError:
raise
except Exception as e:
self.logger.error(f"Failed to export table {table_name}: {e}")
raise ProcessingError(f"Failed to export table: {e}") from e
@@ -867,11 +760,8 @@ class DBIngestor:
schema: Schema name (for databases with schema support, optional)
limit: Maximum number of rows to export (optional)
offset: Row offset for pagination (optional)
where: WHERE clause for filtering (optional, e.g., "status = 'active'").
Raw SQL passed through to ``export_table_data`` same trust
requirement documented there: not for untrusted end-user text.
order_by: ORDER BY clause for sorting (optional, e.g., "created_at DESC").
Same trust requirement as ``where``.
where: WHERE clause for filtering (optional, e.g., "status = 'active'")
order_by: ORDER BY clause for sorting (optional, e.g., "created_at DESC")
transform: Whether to apply data transformations (default: False)
**filters: Additional filtering options (merged with above parameters)
+23 -26
View File
@@ -41,7 +41,6 @@ from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from .ssrf import request_with_ssrf_guard
@dataclass
@@ -342,38 +341,36 @@ class MCPClient:
raise
def _send_request_http(self, request: Dict[str, Any]) -> Optional[Dict[str, Any]]:
"""Send request via HTTP, with redirect-safe credential handling.
Uses ``request_with_ssrf_guard`` so that:
* ``Authorization`` / ``Proxy-Authorization`` headers are **not**
forwarded to a different origin if the MCP server issues a redirect
(issue #947).
* The redirect chain is bounded (default 10 hops).
``allow_private_ips=True`` is set because MCP servers are explicitly
configured by the operator and frequently run on localhost or an
internal network the same trust model as ``allow_private_ips`` opt-in
in the other ingestors. That trust covers only ``self.url`` itself:
``allow_private_ips_on_redirect=False`` keeps redirect targets held to
the normal public-address check, so a compromised or malicious MCP
server cannot use a redirect to route the client into private/
internal address space (e.g. cloud metadata) that the operator never
configured. Scheme validation (http/https only) and the
auth-stripping logic remain active regardless of these flags.
"""
"""Send request via HTTP."""
try:
response = request_with_ssrf_guard(
"POST",
import httpx
response = httpx.post(
self.url,
headers=self.headers,
json=request,
headers=self.headers,
timeout=self.config.get("timeout", 30.0),
allow_private_ips=True,
allow_private_ips_on_redirect=False,
)
response.raise_for_status()
return response.json()
except (ImportError, OSError):
# Fallback to requests if httpx not available
try:
import requests
response = requests.post(
self.url,
json=request,
headers=self.headers,
timeout=self.config.get("timeout", 30.0),
)
response.raise_for_status()
return response.json()
except (ImportError, OSError):
raise ProcessingError(
"HTTP transport requires 'httpx' or 'requests' package. "
"Install with: pip install httpx or pip install requests"
)
except Exception as e:
self.logger.error(f"Failed to send HTTP request: {e}")
raise
+6 -32
View File
@@ -45,7 +45,6 @@ except ModuleNotFoundError: # pragma: no cover - fallback for minimal installs
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from .api_ingestor import APIData, RESTIngestor
from .ssrf import request_with_ssrf_guard
AUTH_HEADER_NAMES = {
"authorization",
@@ -360,31 +359,18 @@ class PublicAPIIngestor(RESTIngestor):
request_options = options.copy()
timeout = request_options.pop("timeout", self.config.get("timeout", 30))
rate_limit_delay = request_options.pop("rate_limit_delay", None)
# session and allow_private_ips are always supplied explicitly below;
# drop any caller-provided copies so request_with_ssrf_guard() does
# not receive duplicate keyword arguments.
request_options.pop("session", None)
request_options.pop("allow_private_ips", None)
request_headers = self._merged_headers(headers)
try:
self._wait_if_needed(rate_limit_delay=rate_limit_delay)
# Route through the SSRF guard so that:
# * redirects to private/loopback IPs are blocked, and
# * Authorization / Proxy-Authorization are stripped on
# cross-origin redirects (issue #947).
response = request_with_ssrf_guard(
method,
endpoint,
session=self.session,
response = self.session.request(
method=method,
url=endpoint,
headers=request_headers,
params=params,
timeout=timeout,
allow_private_ips=self.allow_private_ips,
**request_options,
)
except (ValidationError, ProcessingError):
raise
except requests.exceptions.RequestException as exc:
self.logger.error(f"Failed to detect public API {endpoint}: {exc}")
raise ProcessingError(f"Failed to detect public API: {exc}") from exc
@@ -454,30 +440,18 @@ class PublicAPIIngestor(RESTIngestor):
request_options = options.copy()
timeout = request_options.pop("timeout", self.config.get("timeout", 30))
# session and allow_private_ips are always supplied explicitly below;
# drop any caller-provided copies so request_with_ssrf_guard() does
# not receive duplicate keyword arguments.
request_options.pop("session", None)
request_options.pop("allow_private_ips", None)
request_headers = self._merged_headers(headers)
try:
self._wait_if_needed(rate_limit_delay=rate_limit_delay)
# Route through the SSRF guard so that:
# * redirects to private/loopback IPs are blocked, and
# * Authorization / Proxy-Authorization are stripped on
# cross-origin redirects even when validate_no_auth=False
# (issue #947).
response = request_with_ssrf_guard(
method,
endpoint,
session=self.session,
response = self.session.request(
method=method,
url=endpoint,
headers=request_headers,
params=params,
data=data,
json=json_data,
timeout=timeout,
allow_private_ips=self.allow_private_ips,
**request_options,
)
+52 -462
View File
@@ -11,7 +11,7 @@ import concurrent.futures
import ipaddress
import socket
import threading
from typing import Any, Iterable, List, Optional
from typing import Any, Iterable, Optional
from urllib.parse import urljoin, urlparse
import requests
@@ -138,7 +138,6 @@ def _get_dns_executor() -> concurrent.futures.ThreadPoolExecutor:
BLOCKED_NETWORKS = (
ipaddress.ip_network("0.0.0.0/8"),
ipaddress.ip_network("10.0.0.0/8"),
ipaddress.ip_network("100.64.0.0/10"), # CGNAT (RFC 6598) — routable inside carrier/cloud NAT
ipaddress.ip_network("127.0.0.0/8"),
ipaddress.ip_network("169.254.0.0/16"), # link-local / cloud metadata
ipaddress.ip_network("172.16.0.0/12"),
@@ -263,225 +262,12 @@ def validate_url_for_request(
)
def _resolve_pinned_ips(
url: str, *, allow_private_ips: bool
) -> Optional[List[str]]:
"""Validate *url* and return every resolved IP for connection pinning.
``validate_url_for_request`` and the subsequent connection used to
resolve the same hostname independently, which reopens a DNS-rebinding
TOCTOU window: a low-TTL or rebinding DNS answer can differ between the
validation lookup and the connect-time lookup, so a hostname that
validated as public can still connect to a private/internal address.
This performs the one resolution that is actually used for both the
accept/reject decision *and* the connection (see
``_make_pinned_adapter``), closing that window the same way
``explorer/routes/ontology.py``'s ``_validate_fetch_url`` /
``_make_pinned_session`` pair already does.
Returns ``None`` when ``allow_private_ips`` is True (the caller
explicitly trusts this host, e.g. an operator-configured internal
endpoint that may rely on live DNS/service discovery pinning is
skipped so it keeps resolving normally) or when the URL has no host.
Otherwise returns the deduplicated, resolution-ordered list of
validated IP addresses.
"""
validate_url_for_request(url, allow_private_ips=allow_private_ips)
if allow_private_ips:
return None
host = urlparse(url).hostname
if not host:
return None
try:
literal_ip = ipaddress.ip_address(host)
except ValueError:
literal_ip = None
if literal_ip is not None:
return [str(literal_ip)]
executor = _get_dns_executor()
owned_executor = False
try:
try:
future = executor.submit(socket.getaddrinfo, host, None)
except RuntimeError:
executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
owned_executor = True
future = executor.submit(socket.getaddrinfo, host, None)
resolved: Iterable = future.result(timeout=_DNS_RESOLVE_TIMEOUT_SECONDS)
except (socket.gaierror, concurrent.futures.TimeoutError, OSError) as exc:
raise ValidationError(
f"URL host '{host}' could not be resolved safely "
"(DNS error or timeout); request blocked"
) from exc
finally:
if owned_executor:
_shutdown_executor(executor)
pinned_ips: List[str] = []
for info in resolved:
addr = ipaddress.ip_address(info[4][0])
if _ip_is_blocked(addr):
raise ValidationError(
f"URL host '{host}' resolves to a blocked (private/loopback/"
"link-local) address"
)
addr_str = str(addr)
if addr_str not in pinned_ips:
pinned_ips.append(addr_str)
if not pinned_ips:
raise ValidationError(
f"URL host '{host}' could not be resolved to a usable address"
)
return pinned_ips
def _make_pinned_adapter(pinned_ips: List[str], hostname: str) -> "requests.adapters.HTTPAdapter":
"""Build an HTTPAdapter that connects only to *pinned_ips*.
Falls back across every pinned address in order (a hostname can have
multiple A/AAAA records) while presenting *hostname* as the TLS SNI /
certificate identity and outgoing Host header, so DNS resolution is
bypassed entirely for the actual connection mirroring
``explorer/routes/ontology.py``'s ``_make_pinned_session``.
"""
import urllib3.util.connection as _u3_connection
from urllib3.exceptions import NewConnectionError
class _MultiIPConnectionMixin:
def _new_conn(self):
last_exc: Optional[BaseException] = None
for ip in pinned_ips:
try:
return _u3_connection.create_connection(
(ip, self.port),
self.timeout,
source_address=self.source_address,
socket_options=self.socket_options,
)
except OSError as exc:
last_exc = exc
continue
raise NewConnectionError(
self,
f"Failed to establish a connection to any of {pinned_ips}: {last_exc}",
)
class _PinnedIPHTTPAdapter(requests.adapters.HTTPAdapter):
def get_connection_with_tls_context(self, request, verify, proxies=None, cert=None):
# A proxy performs its own DNS resolution outside this
# process's control, which would silently reopen the exact
# rebinding race pinning exists to close. Fail closed instead.
if requests.utils.select_proxy(request.url, proxies):
raise ValidationError(
"Proxied requests are not supported through the "
"SSRF-guarded request path (a proxy would resolve the "
"host itself and bypass IP pinning)."
)
host_params, pool_kwargs = self.build_connection_pool_key_attributes(
request, verify, cert
)
if host_params.get("scheme") == "https":
pool_kwargs.setdefault("assert_hostname", hostname)
pool_kwargs.setdefault("server_hostname", hostname)
host_params["host"] = pinned_ips[0]
pool = self.poolmanager.connection_from_host(
**host_params, pool_kwargs=pool_kwargs
)
base_connection_cls = pool.ConnectionCls
if not issubclass(base_connection_cls, _MultiIPConnectionMixin):
pool.ConnectionCls = type(
"_PinnedConnection",
(_MultiIPConnectionMixin, base_connection_cls),
{},
)
return pool
return _PinnedIPHTTPAdapter()
def _apply_connection_pin(
active_session: "requests.Session",
url: str,
pinned_ips: Optional[List[str]],
orig_http_adapter: "requests.adapters.HTTPAdapter",
orig_https_adapter: "requests.adapters.HTTPAdapter",
had_host_header: bool,
orig_host_header: Optional[str],
) -> None:
"""Mount (or remove) IP pinning on *active_session* for the next hop."""
# Session.mount() silently drops whatever adapter it replaces without
# closing it. Across a multi-hop redirect chain, each hop gets its own
# fresh pinned adapter (a new pool), so failing to close the one from
# the previous hop would leak its pooled connection.
_current = active_session.adapters.get("http://")
if getattr(_current, "_semantica_pinned", False):
_current.close()
parsed = urlparse(url)
if pinned_ips:
port = parsed.port
default_port = _DEFAULT_PORTS.get(parsed.scheme, 80)
host_header = (
parsed.hostname
if port in (None, default_port)
else f"{parsed.hostname}:{port}"
)
adapter = _make_pinned_adapter(pinned_ips, parsed.hostname or "")
adapter._semantica_pinned = True
active_session.mount("http://", adapter)
active_session.mount("https://", adapter)
active_session.headers["Host"] = host_header
else:
active_session.mount("http://", orig_http_adapter)
active_session.mount("https://", orig_https_adapter)
# Restore the session's own pre-call Host header state rather than
# unconditionally clearing it — a caller-supplied session may carry
# a legitimate Host override (e.g. a private/internal endpoint
# fronted by a name that differs from the connection host), which
# a hop that happens not to need pinning must not silently drop.
if had_host_header:
active_session.headers["Host"] = orig_host_header
else:
active_session.headers.pop("Host", None)
_SESSION_LOCK_ATTR = "_semantica_ssrf_lock"
_session_lock_registry_lock = threading.Lock()
def _get_session_lock(session: "requests.Session") -> threading.Lock:
"""Return a lock private to *session*, creating one on first use.
request_with_ssrf_guard mutates a caller-supplied session's adapters
and Host header for the duration of one guarded call (including every
redirect hop). Without serializing on the session itself, two guarded
calls sharing the same session from different threads could interleave
their mount()/restore cycles one call's request could go out pinned
to (or carrying the Host header for) a completely different call's
target host. Double-checked locking so concurrent first-use doesn't
attach two different locks to the same session.
"""
lock = getattr(session, _SESSION_LOCK_ATTR, None)
if lock is not None:
return lock
with _session_lock_registry_lock:
lock = getattr(session, _SESSION_LOCK_ATTR, None)
if lock is None:
lock = threading.Lock()
setattr(session, _SESSION_LOCK_ATTR, lock)
return lock
def request_with_ssrf_guard(
method: str,
url: str,
*,
session: Optional[requests.Session] = None,
allow_private_ips: bool = False,
allow_private_ips_on_redirect: Optional[bool] = None,
max_redirects: int = _DEFAULT_MAX_REDIRECTS,
**kwargs: Any,
) -> requests.Response:
@@ -491,263 +277,67 @@ def request_with_ssrf_guard(
public URL to bounce into private/loopback/link-local space. This helper
disables automatic redirects and re-validates each ``Location`` target
before issuing the next hop.
``allow_private_ips`` trusts the caller's own *url* (e.g. an
operator-configured internal endpoint). That trust follows a redirect
only when the redirect target's host matches the original host (e.g. a
same-host path redirect on a private/localhost server); a redirect to a
*different* host is validated with ``allow_private_ips_on_redirect``
instead, which defaults to ``allow_private_ips`` for backward
compatibility but can be pinned to ``False`` by callers that want to
trust only the original host and never extend private-IP eligibility to
any other host a redirect chain might reach otherwise a private-IP-
eligible endpoint could be tricked into redirecting into arbitrary
internal address space (e.g. cloud metadata) the caller never
configured.
Authorization / credential-header handling (issue #947)
--------------------------------------------------------
Credentials are stripped from **all** sources that ``requests`` can use to
attach an ``Authorization`` header whenever a redirect changes origin:
1. ``kwargs["headers"]`` per-request header dict (already handled).
2. ``session.headers`` session-level headers that ``requests`` merges
automatically; cleared for the hop and restored via ``finally``.
3. ``kwargs["auth"]`` per-request auth tuple/callable; removed from the
local ``kwargs`` copy when stripping is required. This copy never
escapes to the caller, so there is nothing to restore.
4. ``session.auth`` session-level auth handler that ``requests`` merges
via ``merge_setting(auth, self.auth)`` inside ``prepare_request``;
cleared for the hop and restored via ``finally``.
5. ``session.trust_env`` when ``True``, ``requests`` reads ``~/.netrc``
for the *redirect target* host and calls ``prepare_auth()`` with those
credentials even after sources 3 and 4 are cleared; disabled for
cross-origin hops and restored via ``finally``.
Leaving any one of these intact allows ``requests`` to re-attach
credentials on the hop to the foreign origin, defeating the header-level
strip.
Session state that was removed is unconditionally restored in a ``finally``
block so the session is left in its original state after this call returns,
regardless of how it exits (normal return, exception, redirect cap). A
caller-supplied session is also serialized on internally (see
``_get_session_lock``): two guarded calls sharing the same session from
different threads block on each other for the call's duration rather than
interleaving their mutations, so concurrent use of a shared session is
safe, if not concurrent.
Once credentials have been stripped for a cross-origin hop they are NOT
re-added for subsequent hops in the same chain, even if a later hop
happens to point back to the original host. This prevents credential
resurrection via crafted multi-hop redirect chains.
"""
kwargs = dict(kwargs)
kwargs.pop("allow_redirects", None)
redirect_allow_private_ips = (
allow_private_ips
if allow_private_ips_on_redirect is None
else allow_private_ips_on_redirect
)
_original_host = (urlparse(url).hostname or "").lower()
validate_url_for_request(url, allow_private_ips=allow_private_ips)
current_pinned_ips = _resolve_pinned_ips(url, allow_private_ips=allow_private_ips)
_owns_session = session is None
active_session = session if session is not None else requests.Session()
requester = active_session.request
requester = session.request if session is not None else requests.request
current_url = url
current_method = method.upper()
redirects_followed = 0
# A caller-supplied session is mutated (adapters + Host header, and
# potentially auth/trust_env below) for the duration of this call,
# including every redirect hop; serialize on the session itself so a
# second guarded call sharing it from another thread can't interleave
# its own mount()/restore cycle into the middle of this one. An owned
# session is private to this call, so no lock is needed. Released in
# the outermost `finally` below, alongside the state it protects.
_session_lock = None if _owns_session else _get_session_lock(active_session)
if _session_lock is not None:
_session_lock.acquire()
while True:
response = requester(
current_method,
current_url,
allow_redirects=False,
**kwargs,
)
# Snapshot the session's pre-existing adapters/Host header so pinning
# (mounted per-hop below) can be fully undone when this call returns —
# required for a caller-supplied session, which outlives this call.
_orig_http_adapter = (
active_session.adapters.get("http://") or requests.adapters.HTTPAdapter()
)
_orig_https_adapter = (
active_session.adapters.get("https://") or requests.adapters.HTTPAdapter()
)
_had_host_header = "Host" in active_session.headers
_orig_host_header = active_session.headers.get("Host")
# -- issue #947: snapshot every session-level credential source so we can
# restore them unconditionally when this call exits.
_SENSITIVE = ("Authorization", "Proxy-Authorization")
_session_auth_backup: dict = {}
_session_auth_handler_backup: Any = None # session.auth backup
_session_trust_env_backup: bool = True # session.trust_env backup
if session is not None:
for _h in _SENSITIVE:
# requests stores session headers in a case-insensitive dict;
# .get() matches regardless of the casing used at insertion time.
_val = session.headers.get(_h)
if _val is not None:
_session_auth_backup[_h] = _val
# Snapshot session.auth (HTTPBasicAuth, tuple, callable, or None).
_session_auth_handler_backup = session.auth
# Snapshot session.trust_env (controls .netrc / env proxy lookup).
_session_trust_env_backup = session.trust_env
# Track whether credentials have been stripped for this redirect chain.
# Once stripped they must not reappear on any subsequent hop.
_auth_stripped = False
try:
while True:
_apply_connection_pin(
active_session,
current_url,
current_pinned_ips,
_orig_http_adapter,
_orig_https_adapter,
_had_host_header,
_orig_host_header,
)
response = requester(
current_method,
current_url,
allow_redirects=False,
**kwargs,
)
if response.status_code not in _REDIRECT_STATUS_CODES:
return response
if redirects_followed >= max_redirects:
response.close()
raise ValidationError(
f"Exceeded maximum redirects ({max_redirects}) while "
f"fetching '{url}'"
)
location = response.headers.get("Location")
if not location or not str(location).strip():
response.close()
raise ValidationError(
f"Redirect from '{current_url}' is missing a Location header"
)
next_url = urljoin(current_url, str(location).strip())
next_host = (urlparse(next_url).hostname or "").lower()
# A redirect back to the original host inherits the caller's
# trust in that host (e.g. a same-host path redirect on a
# private/localhost MCP server). A redirect to a *different*
# host must not inherit that trust, even if the original host
# was private/internal — otherwise a compromised or malicious
# endpoint could redirect into arbitrary private address space
# (e.g. cloud metadata) the caller never configured.
hop_allow_private_ips = (
allow_private_ips
if next_host and next_host == _original_host
else redirect_allow_private_ips
)
current_pinned_ips = _resolve_pinned_ips(
next_url, allow_private_ips=hop_allow_private_ips
)
# Do not leak sensitive headers or auth handlers to a different
# origin on redirects. All four credential sources are cleared:
# • kwargs["headers"] — per-request header dict
# • session.headers — session-level header dict
# • kwargs["auth"] — per-request auth tuple/callable
# • session.auth — session-level auth handler
#
# Once stripped (_auth_stripped=True), credentials stay absent for
# the remainder of the chain — even if a later hop targets the
# original host — to prevent credential resurrection.
if _auth_stripped or _should_strip_auth(current_url, next_url):
_auth_stripped = True
# 1. Strip from per-request kwargs headers.
kwargs = dict(kwargs)
headers = dict(kwargs.get("headers") or {})
for sensitive in _SENSITIVE:
headers.pop(sensitive, None)
# Also remove any case variant the caller may have used
# (e.g. "authorization" or "AUTHORIZATION").
for key in list(headers):
if key.lower() == sensitive.lower():
del headers[key]
kwargs["headers"] = headers
# 2. Strip per-request auth kwarg so requests cannot call
# prepare_auth() with the caller's credential on this hop.
kwargs.pop("auth", None)
# 3. Strip session-level headers so requests cannot re-inject
# them when merging session + per-request headers for this hop.
if session is not None:
for sensitive in _SENSITIVE:
# CaseInsensitiveDict.pop(key, None) handles any casing.
session.headers.pop(sensitive, None)
# 4. Clear session.auth so prepare_request's merge_setting()
# cannot fall back to the session-level auth handler and
# reattach credentials on the foreign-origin hop.
session.auth = None
# 5. Disable .netrc / environment-proxy credential lookup so
# requests cannot inject credentials from ~/.netrc for the
# redirect target host on this hop.
session.trust_env = False
# Match requests' historical method rewriting for 301/302/303.
if (
response.status_code in _STRIP_BODY_ON_REDIRECT
and current_method not in {"GET", "HEAD"}
):
current_method = "GET"
for key in ("data", "json", "files"):
kwargs.pop(key, None)
# Params apply to the original request URL only; Location is authoritative.
kwargs.pop("params", None)
if response.status_code not in _REDIRECT_STATUS_CODES:
return response
if redirects_followed >= max_redirects:
response.close()
current_url = next_url
redirects_followed += 1
raise ValidationError(
f"Exceeded maximum redirects ({max_redirects}) while "
f"fetching '{url}'"
)
finally:
if _owns_session:
# No caller holds a reference to this session; just release it.
active_session.close()
else:
# Unconditionally restore every session credential source and
# pinning artifact we touched, so the session is in its
# original state after this call returns or raises.
if _session_auth_backup:
for _h, _v in _session_auth_backup.items():
active_session.headers[_h] = _v
# Restore session.auth to whatever it was before this call.
active_session.auth = _session_auth_handler_backup
# Restore session.trust_env (.netrc / env-proxy lookup flag).
active_session.trust_env = _session_trust_env_backup
# Undo any IP-pinning adapter/Host header mounted for a hop,
# closing the last pinned adapter so its pooled connection
# isn't leaked (see _apply_connection_pin).
_current = active_session.adapters.get("http://")
if getattr(_current, "_semantica_pinned", False):
_current.close()
active_session.mount("http://", _orig_http_adapter)
active_session.mount("https://", _orig_https_adapter)
if _had_host_header:
active_session.headers["Host"] = _orig_host_header
else:
active_session.headers.pop("Host", None)
if _session_lock is not None:
_session_lock.release()
location = response.headers.get("Location")
if not location or not str(location).strip():
response.close()
raise ValidationError(
f"Redirect from '{current_url}' is missing a Location header"
)
next_url = urljoin(current_url, str(location).strip())
validate_url_for_request(next_url, allow_private_ips=allow_private_ips)
# Do not leak sensitive headers to a different origin on redirects:
# reuse the caller's headers only while host, port, and scheme keep
# the credential safe, mirroring requests' should_strip_auth.
if _should_strip_auth(current_url, next_url):
kwargs = dict(kwargs)
headers = dict(kwargs.get("headers") or {})
for sensitive in ("Authorization", "Proxy-Authorization"):
headers.pop(sensitive, None)
kwargs["headers"] = headers
# Match requests' historical method rewriting for 301/302/303.
if (
response.status_code in _STRIP_BODY_ON_REDIRECT
and current_method not in {"GET", "HEAD"}
):
current_method = "GET"
for key in ("data", "json", "files"):
kwargs.pop(key, None)
# Params apply to the original request URL only; Location is authoritative.
kwargs.pop("params", None)
response.close()
current_url = next_url
redirects_followed += 1
+31 -76
View File
@@ -22,9 +22,8 @@ License: MIT
from typing import Any, Dict, List, Optional
from ..deduplication.duplicate_detector import DuplicateDetector, DuplicateGroup
from ..deduplication.duplicate_detector import DuplicateDetector
from ..deduplication.entity_merger import EntityMerger
from ..utils.entity_ids import get_entity_id
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -139,7 +138,9 @@ class EntityResolver:
self.logger.debug(
f"Detecting duplicate groups with threshold {self.similarity_threshold}"
)
duplicate_groups = self._detect_duplicate_groups(entities)
duplicate_groups = self.duplicate_detector.detect_duplicate_groups(
entities, threshold=self.similarity_threshold
)
self.logger.debug(f"Found {len(duplicate_groups)} duplicate group(s)")
@@ -149,7 +150,6 @@ class EntityResolver:
# Step 2: Merge duplicates in each group
merged_entities = []
processed_entity_ids = set() # Track which entities have been merged
processed_entity_objects = set()
for group in duplicate_groups:
# Skip groups with less than 2 entities (not duplicates)
@@ -157,16 +157,9 @@ class EntityResolver:
continue
# Merge the duplicate group into a single canonical entity
if self.resolution_strategy == "exact":
merge_operations = [
self.entity_merger.merge_entity_group(
group.entities, **self.config
)
]
else:
merge_operations = self.entity_merger.merge_duplicates(
group.entities, **self.config
)
merge_operations = self.entity_merger.merge_duplicates(
group.entities, **self.config
)
# Process each merge operation
for operation in merge_operations:
@@ -175,26 +168,30 @@ class EntityResolver:
# Mark all source entities as processed
for source_entity in operation.source_entities:
entity_id = self._get_entity_id(source_entity)
if entity_id is None:
processed_entity_objects.add(id(source_entity))
continue
try:
entity_id = (
source_entity.get("id")
if isinstance(source_entity, dict)
else getattr(source_entity, "id", None)
) or (
source_entity.get("entity_id")
if isinstance(source_entity, dict)
else getattr(source_entity, "entity_id", None)
)
if entity_id:
processed_entity_ids.add(entity_id)
except TypeError:
processed_entity_objects.add(id(source_entity))
# Step 3: Add non-duplicate entities (entities not in any duplicate group)
for entity in entities:
entity_id = self._get_entity_id(entity)
if entity_id is None:
is_unprocessed = id(entity) not in processed_entity_objects
else:
try:
is_unprocessed = entity_id not in processed_entity_ids
except TypeError:
is_unprocessed = id(entity) not in processed_entity_objects
if is_unprocessed:
entity_id = (
entity.get("id")
if isinstance(entity, dict)
else getattr(entity, "id", None)
) or (
entity.get("entity_id")
if isinstance(entity, dict)
else getattr(entity, "entity_id", None)
)
if entity_id and entity_id not in processed_entity_ids:
# This entity was not merged, add it as-is
merged_entities.append(entity)
@@ -216,48 +213,6 @@ class EntityResolver:
)
raise
def _detect_duplicate_groups(
self, entities: List[Dict[str, Any]]
) -> List[DuplicateGroup]:
"""Detect duplicate groups according to the configured strategy."""
if self.resolution_strategy != "exact":
return self.duplicate_detector.detect_duplicate_groups(
entities, threshold=self.similarity_threshold
)
groups = {}
for entity in entities:
name = self._get_entity_name(entity)
normalized = str(name).strip() if name is not None else ""
if normalized:
groups.setdefault(normalized.casefold(), []).append(entity)
return [
DuplicateGroup(entities=group, confidence=1.0)
for group in groups.values()
if len(group) > 1
]
@staticmethod
def _get_entity_id(entity: Any) -> Any:
"""Return an entity ID while supporting dictionary and object inputs."""
return get_entity_id(entity)
@staticmethod
def _get_entity_name(entity: Any) -> Optional[str]:
"""Return an entity name, falling back to text-based entity input."""
if isinstance(entity, dict):
name = entity.get("name")
return (
name if name is not None and str(name).strip() else entity.get("text")
)
name = getattr(entity, "name", None)
return (
name
if name is not None and str(name).strip()
else getattr(entity, "text", None)
)
def merge_duplicates(self, entities: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""
Merge duplicate entities.
@@ -278,13 +233,13 @@ class EntityResolver:
processed_ids = set()
for op in merge_operations:
for source_entity in op.source_entities:
entity_id = get_entity_id(source_entity)
if entity_id is not None:
entity_id = source_entity.get("id") or source_entity.get("entity_id")
if entity_id:
processed_ids.add(entity_id)
for entity in entities:
entity_id = get_entity_id(entity)
if entity_id is not None and entity_id not in processed_ids:
entity_id = entity.get("id") or entity.get("entity_id")
if entity_id and entity_id not in processed_ids:
merged_entities.append(entity)
self.logger.info(f"Merged to {len(merged_entities)} entities")
+5 -19
View File
@@ -185,25 +185,8 @@ class GraphValidator:
# 3. Relationship Validation
for i, rel in enumerate(relationships):
# Endpoints may use either the legacy ``source``/``target`` keys or
# the canonical ``source_id``/``target_id`` keys emitted by
# ``ContextGraph.to_kg_dict()``. Accept either variant so both
# representations validate consistently.
src = rel.get("source")
if src is None:
src = rel.get("source_id")
tgt = rel.get("target")
if tgt is None:
tgt = rel.get("target_id")
# Check required fields: ``type`` plus a resolvable source/target.
missing = set()
if "type" not in rel:
missing.add("type")
if src is None:
missing.add("source")
if tgt is None:
missing.add("target")
# Check required fields
missing = self.required_rel_fields - set(rel.keys())
if missing:
issues.append(ValidationIssue(
code="MISSING_FIELD",
@@ -213,6 +196,9 @@ class GraphValidator:
details={"index": i}
))
continue
src = rel.get("source")
tgt = rel.get("target")
# Check Dangling Edges
def is_valid_id(node_id):
+1 -8
View File
@@ -535,8 +535,7 @@ class TemporalGraphQuery:
relationships = [
rel
for rel in relationships
if (rel.get("source") or rel.get("source_id")) == entity
or (rel.get("target") or rel.get("target_id")) == entity
if rel.get("source") == entity or rel.get("target") == entity
]
if relationship:
@@ -643,14 +642,8 @@ class TemporalGraphQuery:
parsed_end_time = self._parse_time(end_time) if end_time else None
for rel in relationships:
# Accept both the legacy ``source``/``target`` keys and the
# canonical ``source_id``/``target_id`` keys from ``to_kg_dict()``.
s = rel.get("source")
if s is None:
s = rel.get("source_id")
t = rel.get("target")
if t is None:
t = rel.get("target_id")
# Check temporal validity
if start_time or end_time:
+4 -44
View File
@@ -33,12 +33,6 @@ class SHACLViolation:
value: Optional[str] = None
shape: Optional[str] = None
explanation: Optional[str] = None
# Real constraint parameters extracted from the source shape (sh:sourceShape),
# used to render accurate plain-English explanations.
min_count: Optional[int] = None
max_count: Optional[int] = None
datatype: Optional[str] = None
class_: Optional[str] = None
def to_dict(self) -> Dict[str, Any]:
return {
@@ -50,10 +44,6 @@ class SHACLViolation:
"value": self.value,
"shape": self.shape,
"explanation": self.explanation,
"min_count": self.min_count,
"max_count": self.max_count,
"datatype": self.datatype,
"class_": self.class_,
}
@@ -128,10 +118,10 @@ class SHACLValidationReport:
focus_node=v.focus_node,
path=v.result_path or "",
value=v.value or "",
min_count=v.min_count if v.min_count is not None else "?",
max_count=v.max_count if v.max_count is not None else "?",
datatype=v.datatype or "the expected datatype",
class_=v.class_ or "the required class",
min_count=1,
max_count=1,
datatype=v.message or "",
class_=v.message or "",
)
def to_dict(self) -> Dict[str, Any]:
@@ -218,32 +208,6 @@ def _run_pyshacl(
shape_node = results_graph.value(result, SH.sourceShape)
shape = str(shape_node) if shape_node is not None else None
# Look up the real constraint parameters from the source shape so that
# explain_violations can render accurate values instead of placeholders.
# Note: sh:qualifiedMinCount / sh:qualifiedMaxCount are not handled here;
# such violations fall back to the "?" placeholder in explain_violations.
min_count: Optional[int] = None
max_count: Optional[int] = None
datatype: Optional[str] = None
class_: Optional[str] = None
if shape_node is not None:
min_node = shacl_g.value(shape_node, SH.minCount)
if min_node is not None:
try:
min_count = int(str(min_node))
except (TypeError, ValueError):
min_count = None
max_node = shacl_g.value(shape_node, SH.maxCount)
if max_node is not None:
try:
max_count = int(str(max_node))
except (TypeError, ValueError):
max_count = None
dt_node = shacl_g.value(shape_node, SH.datatype)
datatype = str(dt_node) if dt_node is not None else None
cls_node = shacl_g.value(shape_node, SH["class"])
class_ = str(cls_node) if cls_node is not None else None
v = SHACLViolation(
focus_node=focus,
result_path=path,
@@ -252,10 +216,6 @@ def _run_pyshacl(
message=msg,
value=val,
shape=shape,
min_count=min_count,
max_count=max_count,
datatype=datatype,
class_=class_,
)
if sev_str == "Violation":
violations.append(v)
-35
View File
@@ -1,35 +0,0 @@
"""The vocabulary Semantica's exporters emit terms from.
Every RDF export mints terms in ``https://semantica.dev/ns#``: ``sem:text``,
``sem:confidence``, the default ``sem:Entity`` type, and the rest. Until this
file existed, nothing declared what those terms meant, so a consumer receiving
an export could not tell ``sem:text`` from a typo of it, and no closed-world
check could be run against them at all (issue #1107).
The document ships inside the package so it can be loaded without a network
round trip, and is the same file intended to be served at the namespace IRI.
>>> from semantica.ontology.vocabulary import vocabulary_turtle
>>> ttl = vocabulary_turtle()
"""
from __future__ import annotations
from pathlib import Path
VOCABULARY_FILENAME = "semantica-ns.ttl"
#: The namespace the vocabulary declares terms in.
NAMESPACE = "https://semantica.dev/ns#"
__all__ = ["NAMESPACE", "VOCABULARY_FILENAME", "vocabulary_path", "vocabulary_turtle"]
def vocabulary_path() -> Path:
"""Filesystem path to the vocabulary document."""
return Path(__file__).parent / VOCABULARY_FILENAME
def vocabulary_turtle() -> str:
"""The vocabulary document as Turtle."""
return vocabulary_path().read_text(encoding="utf-8")
@@ -1,155 +0,0 @@
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix dct: <http://purl.org/dc/terms/> .
@prefix prov: <http://www.w3.org/ns/prov#> .
@prefix time: <http://www.w3.org/2006/time#> .
@prefix sem: <https://semantica.dev/ns#> .
<https://semantica.dev/ns> a owl:Ontology ;
rdfs:label "Semantica vocabulary" ;
rdfs:comment """Declares the terms the Semantica exporters emit in
https://semantica.dev/ns#. Drafted from the emitting call sites in
semantica 0.6.5: export/rdf_exporter.py, export/json_exporter.py and
provenance/manager.py. Every term below appears in output the package
produces today; no term has been invented for completeness.""" ;
owl:versionInfo "0.1.0-draft" ;
dct:created "2026-08-19"^^xsd:date .
# ── Classes ──────────────────────────────────────────────────────────────────
sem:Entity a owl:Class ;
rdfs:label "Entity" ;
rdfs:comment """The default type given to an extracted entity when the
source carries no type of its own. Emitted by serialize_to_turtle as the
fallback for entity.get("type").""" ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:Relationship a owl:Class ;
rdfs:label "Relationship" ;
rdfs:comment """A reified relationship, as emitted in the JSON-LD export
where a relationship carries sem:type, sem:source and sem:target rather than
being written as a single triple.""" ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:KnowledgeGraph a owl:Class ;
rdfs:label "Knowledge Graph" ;
rdfs:comment """The document-level type of a JSON-LD export: the @type of
the top-level node carrying sem:entities, sem:relationships and
sem:exportedAt. Emitted by _convert_kg_to_jsonld in export/json_exporter.py.""" ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
# ── Properties on an entity ──────────────────────────────────────────────────
sem:text a owl:DatatypeProperty ;
rdfs:label "text" ;
rdfs:comment """The surface text of an extracted entity. Carries the same
intent as rdfs:label; declared separately because the exporters emit it under
this IRI.""" ;
rdfs:domain sem:Entity ;
rdfs:range xsd:string ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:confidence a owl:DatatypeProperty ;
rdfs:label "confidence" ;
rdfs:comment """Extractor confidence in the assertion, on the unit interval.
Emitted for both entities and relationships, so the domain is left open rather
than tied to sem:Entity.
No rdfs:range is declared, deliberately. The Turtle serializer writes the value
bare, which the Turtle grammar reads as xsd:decimal, while the N-Triples
serializer types it xsd:float explicitly, and those two datatypes are disjoint.
Declaring either one would make the vocabulary contradict one of the exporters.
Issue #1100 tracks the disagreement; a range belongs here once the serializers
agree on one.""" ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:metadata a owl:AnnotationProperty ;
rdfs:label "metadata" ;
rdfs:comment """Free-form metadata carried through from extraction. An
annotation property because its value is an arbitrary structure rather than a
modelled one.""" ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
# ── Relationship terms (JSON-LD export) ──────────────────────────────────────
sem:related_to a owl:ObjectProperty ;
rdfs:label "related to" ;
rdfs:comment """The default predicate for a relationship whose type the
extractor did not determine. Deliberately unspecific: it asserts that two
entities are connected and nothing about how.""" ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:source a owl:ObjectProperty ;
rdfs:label "source" ;
rdfs:comment "The subject entity of a reified relationship." ;
rdfs:domain sem:Relationship ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:target a owl:ObjectProperty ;
rdfs:label "target" ;
rdfs:comment "The object entity of a reified relationship." ;
rdfs:domain sem:Relationship ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:type a owl:DatatypeProperty ;
rdfs:label "type" ;
rdfs:comment """The relationship type as a label, as emitted in the JSON-LD
export. Distinct from rdf:type, which relates a node to a class rather than to
a string.""" ;
rdfs:domain sem:Relationship ;
rdfs:range xsd:string ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
# ── Document-level terms (JSON-LD export) ────────────────────────────────────
sem:entities a owl:ObjectProperty ;
rdfs:label "entities" ;
rdfs:comment "Ordered list of entities in an exported graph document." ;
rdfs:range sem:Entity ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:relationships a owl:ObjectProperty ;
rdfs:label "relationships" ;
rdfs:comment "Ordered list of relationships in an exported graph document." ;
rdfs:range sem:Relationship ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:exportedAt a owl:DatatypeProperty ;
rdfs:label "exported at" ;
rdfs:comment """When the export was written. Emitted as an ISO 8601 local
timestamp, so the range is xsd:dateTime rather than xsd:dateTimeStamp: the
values carry no timezone offset.""" ;
rdfs:range xsd:dateTime ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:format a owl:DatatypeProperty ;
rdfs:label "format" ;
rdfs:comment """The serialization format label written on a JSON-LD
document (currently always the literal "json-ld"). Emitted by
JSONExporter.export_to_jsonld in export/json_exporter.py.""" ;
rdfs:range xsd:string ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
# ── Temporal term (OWL-Time export) ──────────────────────────────────────────
sem:openEndedInterval a owl:DatatypeProperty ;
rdfs:label "open ended interval" ;
rdfs:comment """True when an interval has no known end. OWL-Time has no
standard predicate for this, which is the reason the exporter mints one: an
interval with no time:hasEnd is ambiguous between "ongoing" and "end not
recorded", and this term resolves that in favour of the first.""" ;
rdfs:domain time:Interval ;
rdfs:range xsd:boolean ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
# ── Provenance roles ─────────────────────────────────────────────────────────
sem:role_generator a prov:Role ;
rdfs:label "generator" ;
rdfs:comment """The default role in a prov:qualifiedAssociation, used when
an agent generated an entity rather than approving or reviewing it. Typed as
prov:Role so that prov:hadRole has a declared value rather than an undeclared
IRI.""" ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
+1 -9
View File
@@ -243,14 +243,6 @@ class MediaParser:
try:
import subprocess
# ffprobe's own argument parser doesn't reliably honor a bare
# "--" end-of-options marker, so a filename starting with "-"
# could otherwise be parsed as an option; neutralize that by
# forcing a relative-path prefix ffprobe can't mistake for a flag.
ffprobe_path = str(file_path)
if ffprobe_path.startswith("-"):
ffprobe_path = f"./{ffprobe_path}"
result = subprocess.run(
[
"ffprobe",
@@ -260,7 +252,7 @@ class MediaParser:
"json",
"-show_format",
"-show_streams",
ffprobe_path,
str(file_path),
],
capture_output=True,
text=True,
+2 -5
View File
@@ -501,11 +501,8 @@ class SeedDataManager:
else:
full_url = api_url
# Prepare headers — copy the caller's dict so we never mutate it in-place.
# Without the copy, adding "Authorization" here would silently modify the
# caller's original dict and potentially leak the key to subsequent calls
# that reuse the same dict without expecting it to contain credentials.
request_headers = dict(headers) if headers else {}
# Prepare headers
request_headers = headers or {}
if api_key:
request_headers["Authorization"] = f"Bearer {api_key}"
+1 -6
View File
@@ -144,12 +144,7 @@ class NERExtractor:
self._ml_runtime_usable = True
if "ml" in self.method and SPACY_AVAILABLE:
try:
# Deferred import: keeps semantic_extract.methods out of the
# module-level import graph and routes loading through the
# process-level cache so repeated NERExtractor constructions
# never pay the ~120 ms spacy.load() cost more than once.
from .methods import load_spacy_model
self.nlp = load_spacy_model(self.model_name)
self.nlp = spacy.load(self.model_name)
except OSError:
self.logger.warning(
f"spaCy model {self.model_name} not found. ML method will fallback."
+2 -3
View File
@@ -97,7 +97,7 @@ from .semantic_chunker import Chunk
logger = get_logger("split_methods")
# Try to import optional dependencies
_, SPACY_AVAILABLE = safe_import("spacy")
spacy, SPACY_AVAILABLE = safe_import("spacy")
nltk, NLTK_AVAILABLE = safe_import("nltk")
tiktoken, TIKTOKEN_AVAILABLE = safe_import("tiktoken")
@@ -336,8 +336,7 @@ def split_by_sentences(
# Try spaCy first
if SPACY_AVAILABLE and kwargs.get("use_spacy", True):
try:
from ..semantic_extract.methods import load_spacy_model
nlp = load_spacy_model("en_core_web_sm")
nlp = spacy.load("en_core_web_sm")
doc = nlp(text)
sentences = [sent.text for sent in doc.sents]
except Exception:
+2 -11
View File
@@ -36,8 +36,7 @@ from ..utils.helpers import safe_import
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
_, SPACY_AVAILABLE = safe_import("spacy")
spacy, SPACY_AVAILABLE = safe_import("spacy")
@dataclass
@@ -80,19 +79,11 @@ class SemanticChunker:
if SPACY_AVAILABLE:
model_name = config.get("model", "en_core_web_sm")
try:
from ..semantic_extract.methods import load_spacy_model
self.nlp = load_spacy_model(model_name)
self.nlp = spacy.load(model_name)
except OSError:
self.logger.warning(
f"spaCy model {model_name} not found. Using fallback chunking."
)
except Exception:
self.logger.warning(
"spaCy model %s failed to initialize and will be disabled "
"for this chunker instance. Using fallback chunking.",
model_name,
exc_info=True,
)
def chunk(self, text: str, **options) -> List[Chunk]:
"""
+2 -2
View File
@@ -383,9 +383,9 @@ class AnzoStore:
if self._is_uri_value(obj):
if obj.startswith("<") and obj.endswith(">"):
inner = obj[1:-1]
sparql_escaping.validate_uri(inner)
if " " in inner or ">" in inner:
raise ValueError(f"IRI contains invalid characters: {obj!r}")
return obj
sparql_escaping.validate_uri(obj)
return f"<{obj}>"
escaped = sparql_escaping.escape_literal(obj)
-16
View File
@@ -1,16 +0,0 @@
"""Helpers for reading entity identifiers consistently across the KG pipeline."""
from typing import Any
def get_entity_id(entity: Any) -> Any:
"""Return a truthy identifier from either supported entity ID field.
The KG pipeline treats empty and otherwise falsy identifiers as missing.
Prefer the canonical ``id`` field when it is populated, then fall back to
the compatible ``entity_id`` alias.
"""
if isinstance(entity, dict):
return entity.get("id") or entity.get("entity_id") or None
return getattr(entity, "id", None) or getattr(entity, "entity_id", None) or None
+7 -19
View File
@@ -398,7 +398,9 @@ def chunk_list(items: List[Any], chunk_size: int) -> List[List[Any]]:
Returns:
List of chunks
"""
return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]
return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]
def flatten_dict(
d: Dict[str, Any], parent_key: str = "", sep: str = "."
) -> Dict[str, Any]:
@@ -412,32 +414,18 @@ def flatten_dict(
Returns:
Flattened dictionary
Raises:
ValueError: If two input paths produce the same flattened key.
"""
result = {}
items = []
for k, v in d.items():
new_key = f"{parent_key}{sep}{k}" if parent_key else k
if isinstance(v, dict):
nested = flatten_dict(v, new_key, sep=sep)
for key, value in nested.items():
if key in result:
raise ValueError(
f"Key collision while flattening dictionary: {key}"
)
result[key] = value
items.extend(flatten_dict(v, new_key, sep=sep).items())
else:
if new_key in result:
raise ValueError(
f"Key collision while flattening dictionary: {new_key}"
)
result[new_key] = v
items.append((new_key, v))
return result
return dict(items)
def get_nested_value(
@@ -1,142 +0,0 @@
#!/usr/bin/env python3
"""Regression tests for the ContextGraph module docstring example.
The "Example Usage" block in ``semantica/context/context_graph.py`` previously
called ``add_node``/``add_edge`` with keyword arguments those methods do not
accept (``type=`` and ``properties=``), so the documented example raised
``TypeError`` -- and the near-miss variants silently nested the properties dict
instead of failing.
These tests keep the documented example executable and pin the two behaviours
that made the original mistake easy to miss.
"""
import doctest
import re
from typing import Dict, List
import pytest
import semantica.context.context_graph as context_graph_module
from semantica.context.context_graph import ContextGraph
# The example block runs to the next top-level section header (a line starting
# in column 0, e.g. "Production Use Cases:") or the end of the docstring.
# Terminating on the next header rather than on a blank line keeps the capture
# intact when the example gains blank lines or extra paragraphs.
_EXAMPLE_BLOCK_RE = re.compile(r"^Example Usage:\n(.*?)(?=^\S|\Z)", re.DOTALL | re.MULTILINE)
# ``type=`` as its own keyword, but not the legitimate ``node_type=``/``edge_type=``.
_BARE_TYPE_KWARG_RE = re.compile(r"(?<![\w])type\s*=")
def _example_block() -> str:
"""Return the 'Example Usage' block from the module docstring."""
doc = context_graph_module.__doc__ or ""
match = _EXAMPLE_BLOCK_RE.search(doc)
assert match, "module docstring no longer contains an 'Example Usage:' block"
block = match.group(1).strip()
assert block, "the 'Example Usage:' block in the module docstring is empty"
return block
def _example_statements() -> List[str]:
"""Return the documented ``>>>`` statements, continuation lines included."""
statements = [example.source for example in doctest.DocTestParser().get_examples(_example_block())]
assert statements, "the 'Example Usage:' block no longer contains any '>>>' statements"
return statements
def _statements_calling(method: str) -> List[str]:
"""Return the documented statements that call ``graph.<method>(``."""
return [stmt for stmt in _example_statements() if "graph.{}(".format(method) in stmt]
def _run_example() -> Dict[str, object]:
"""Execute the documented example verbatim and return its namespace."""
source = "".join(_example_statements())
namespace: Dict[str, object] = {}
exec(compile(source, "<context_graph module docstring>", "exec"), namespace)
return namespace
class TestDocstringExampleIsRunnable:
"""The documented example must execute exactly as written."""
def test_documented_calls_execute(self):
# Run the docstring text itself so this test cannot drift from the docs.
ns = _run_example()
graph = ns["graph"]
assert "Python" in graph.nodes
assert "Programming" in graph.nodes
assert graph.nodes["Python"].node_type == "language"
assert graph.nodes["Programming"].node_type == "concept"
neighbors = graph.get_neighbors("Python", hops=1)
assert any(n["id"] == "Programming" for n in neighbors)
# record_decision must return a non-empty string ID.
assert isinstance(ns["decision_id"], str) and ns["decision_id"]
# find_precedents must be called with that ID and return a list.
assert isinstance(ns["precedents"], list)
def test_node_properties_are_stored_flat(self):
"""``popularity`` must land as a top-level property, not nested.
Passing the previously documented ``properties={...}`` does not raise --
it stores a dict *inside* the properties dict, which is why the original
docs bug could reach a user's graph unnoticed.
"""
graph = ContextGraph(advanced_analytics=False)
graph.add_node("Python", "language", popularity="high")
assert graph.nodes["Python"].properties == {"popularity": "high"}
assert graph.find_node("Python")["metadata"]["popularity"] == "high"
assert "properties" not in graph.nodes["Python"].properties
def test_edge_type_is_positional_not_a_property(self):
"""``related_to`` must be the edge type, not a stray metadata key."""
graph = ContextGraph(advanced_analytics=False)
graph.add_node("Python", "language")
graph.add_node("Programming", "concept")
graph.add_edge("Python", "Programming", "related_to")
edge = graph.edges[0]
assert edge.edge_type == "related_to"
assert "type" not in edge.metadata
class TestDocstringExampleDoesNotRegress:
"""Guard the docstring text itself, not just equivalent code."""
def test_add_node_example_supplies_node_type_positionally(self):
calls = _statements_calling("add_node")
assert calls, "the 'Example Usage:' block no longer calls graph.add_node()"
for call in calls:
assert not _BARE_TYPE_KWARG_RE.search(call), (
f"add_node example passes type= as a keyword: {call!r}. "
"node_type is positional-required; type= falls through to "
"**properties and the call raises TypeError."
)
assert "properties=" not in call, (
f"add_node example passes properties=: {call!r}. "
"add_node has no properties parameter; extra properties are "
"passed as **kwargs."
)
def test_add_edge_example_supplies_edge_type_positionally(self):
calls = _statements_calling("add_edge")
assert calls, "the 'Example Usage:' block no longer calls graph.add_edge()"
for call in calls:
assert not _BARE_TYPE_KWARG_RE.search(call), (
f"add_edge example passes type= as a keyword: {call!r}. "
"The parameter is edge_type; type= is silently absorbed into "
"**properties and pollutes edge metadata."
)
def test_broken_form_still_raises(self):
"""Pin the signature contract the example has to respect."""
graph = ContextGraph(advanced_analytics=False)
with pytest.raises(TypeError, match="node_type"):
graph.add_node("Python", type="language", properties={"popularity": "high"})
-150
View File
@@ -1,150 +0,0 @@
"""Tests for ContextGraph.to_kg_dict() — the official KG-shape adapter.
These tests lock in the contract that to_kg_dict() emits the
``{"entities", "relationships"}`` / ``source_id`` shape expected by
downstream consumers (RDFExporter, TemporalGraphQuery.query_time_range),
so users never need to hand-map field names.
"""
from semantica.context.context_graph import ContextEdge, ContextGraph, ContextNode
def _build_graph():
g = ContextGraph()
g._add_internal_node(ContextNode(node_id="e1", node_type="entity", content="Alice"))
g._add_internal_node(ContextNode(node_id="e2", node_type="entity", content="Bob"))
g._add_internal_node(
ContextNode(node_id="c1", node_type="conversation", content="chat log")
)
g._add_internal_edge(
ContextEdge(
source_id="e1",
target_id="e2",
edge_type="knows",
valid_from="2024-01-01",
valid_until="2024-12-31",
)
)
# Edge touching a non-entity node — used to test entities_only filtering.
g._add_internal_edge(
ContextEdge(source_id="c1", target_id="e1", edge_type="mentions")
)
return g
def test_basic_shape():
kg = _build_graph().to_kg_dict()
assert set(kg.keys()) == {"entities", "relationships", "statistics"}
# Entity shape uses id/text/type (not id/content).
entity = next(e for e in kg["entities"] if e["id"] == "e1")
assert entity["text"] == "Alice"
assert entity["type"] == "entity"
def test_relationship_uses_source_id_target_id():
kg = _build_graph().to_kg_dict()
rel = next(r for r in kg["relationships"] if r["type"] == "knows")
assert rel["source_id"] == "e1"
assert rel["target_id"] == "e2"
# "source"/"target" (the internal names) must NOT leak through.
assert "source" not in rel
assert "target" not in rel
def test_temporal_fields_passthrough():
kg = _build_graph().to_kg_dict()
rel = next(r for r in kg["relationships"] if r["type"] == "knows")
assert rel["valid_from"] == "2024-01-01"
assert rel["valid_until"] == "2024-12-31"
def test_statistics_counts():
kg = _build_graph().to_kg_dict()
assert kg["statistics"]["entity_count"] == len(kg["entities"])
assert kg["statistics"]["relationship_count"] == len(kg["relationships"])
def test_entities_only_filters_nodes():
kg = _build_graph().to_kg_dict(entities_only=True)
types = {e["type"] for e in kg["entities"]}
assert types == {"entity"}
assert len(kg["entities"]) == 2
def test_entities_only_drops_dangling_relationships():
# The "mentions" edge points from a conversation node (filtered out under
# entities_only) and must not appear as a dangling relationship.
kg = _build_graph().to_kg_dict(entities_only=True)
rel_types = {r["type"] for r in kg["relationships"]}
assert "mentions" not in rel_types
assert rel_types == {"knows"}
def test_returned_dicts_are_isolated_from_internal_state():
g = _build_graph()
kg = g.to_kg_dict()
entity = next(e for e in kg["entities"] if e["id"] == "e1")
# Mutating the returned dict must not corrupt internal node properties.
entity["properties"]["injected"] = True
assert "injected" not in g.nodes["e1"].properties
def test_null_properties_and_metadata_do_not_crash():
"""Nodes loaded from JSON ``null`` keep None props/metadata; to_kg_dict
must normalize them instead of raising TypeError (Qodo bug 1)."""
g = ContextGraph()
n = ContextNode(node_id="e1", node_type="entity", content="Alice")
n.properties = None
n.metadata = None
g._add_internal_node(n)
kg = g.to_kg_dict()
entity = kg["entities"][0]
assert entity["properties"] == {}
assert entity["metadata"] == {}
def test_non_string_node_id_is_normalized_and_keeps_edges():
"""ContextEdge coerces endpoints to str; entity ids must be coerced too
so entities_only filtering does not drop valid edges (Qodo bug 3)."""
g = ContextGraph()
g._add_internal_node(ContextNode(node_id=1, node_type="entity", content="one"))
g._add_internal_node(ContextNode(node_id=2, node_type="entity", content="two"))
g._add_internal_edge(ContextEdge(source_id=1, target_id=2, edge_type="links"))
kg = g.to_kg_dict(entities_only=True)
ids = {e["id"] for e in kg["entities"]}
assert ids == {"1", "2"}
assert all(isinstance(e["id"], str) for e in kg["entities"])
# The edge must survive filtering despite the int-vs-str origin.
assert {r["type"] for r in kg["relationships"]} == {"links"}
def test_output_is_consumable_by_kg_utilities():
"""to_kg_dict output must validate and be traversable by KG utilities that
historically read ``source``/``target`` (Qodo bug 2, consumer side)."""
from semantica.kg.graph_validator import GraphValidator, ValidationSeverity
from semantica.kg.temporal_query import TemporalGraphQuery
g = ContextGraph()
g._add_internal_node(ContextNode(node_id="e1", node_type="entity", content="Alice"))
g._add_internal_node(ContextNode(node_id="e2", node_type="entity", content="Bob"))
g._add_internal_edge(ContextEdge(source_id="e1", target_id="e2", edge_type="knows"))
kg = g.to_kg_dict()
# Validator requires entity ``name``; add it so only endpoint compat is tested.
for e in kg["entities"]:
e["name"] = e["text"]
result = GraphValidator().validate(kg)
endpoint_errors = [
i for i in result.issues
if i.code in {"MISSING_FIELD", "DANGLING_EDGE"}
and i.element_type == "relationship"
]
assert endpoint_errors == [], endpoint_errors
# TemporalGraphQuery.analyze_evolution must see the relationship for "e1".
tq = TemporalGraphQuery()
filtered = tq.analyze_evolution(kg, entity="e1")
assert filtered is not None
@@ -1,141 +0,0 @@
"""Minted IRIs must be stable and must sit in the declared namespace (issue #1101).
An entity that arrives without an id gets one minted for it. That identifier was
built from Python's builtin ``hash()``, which is randomised per process, so the
same entity received a different IRI on every run and exports could not be
diffed, deduplicated against an earlier load, or joined to a provenance record
written by an earlier process.
It was also written as ``semantica:entity_N`` inside angle brackets, which is an
IRI in the scheme ``semantica`` rather than the expansion of the declared
``semantica:`` prefix, so it never joined with anything written through it.
"""
import os
import subprocess
import sys
from semantica.export.rdf_exporter import (
DEFAULT_ENTITY_TYPE,
DEFAULT_RELATION_TYPE,
RDFExporter,
SEMANTICA_NS,
mint_entity_iri,
mint_relationship_iri,
)
UNIDENTIFIED = {
"entities": [{"text": "Acme Corp", "type": "https://example.org/Org"}],
"relationships": [],
}
def test_minted_entity_iri_is_stable_within_a_process():
assert mint_entity_iri("Acme Corp") == mint_entity_iri("Acme Corp")
def test_minted_entity_iri_is_stable_across_processes():
"""The regression that matters: identity must survive a restart."""
script = (
"from semantica.export.rdf_exporter import mint_entity_iri;"
"print(mint_entity_iri('Acme Corp'))"
)
runs = {
subprocess.run(
[sys.executable, "-c", script],
capture_output=True,
text=True,
check=True,
env={**os.environ, "PYTHONHASHSEED": seed},
).stdout.strip()
for seed in ("0", "1", "random")
}
assert len(runs) == 1, f"minted IRI differs between processes: {runs}"
def test_minted_iris_are_in_the_declared_namespace():
assert mint_entity_iri("Acme Corp").startswith(SEMANTICA_NS)
assert mint_relationship_iri(0, "a", "b").startswith(SEMANTICA_NS)
def test_distinct_entities_get_distinct_iris():
assert mint_entity_iri("Acme Corp") != mint_entity_iri("Acme Corporation")
def test_turtle_export_writes_a_resolvable_minted_iri():
turtle = RDFExporter().export_to_rdf(UNIDENTIFIED, format="turtle")
assert f"<{SEMANTICA_NS}entity_" in turtle
assert "<semantica:entity_" not in turtle, "scheme 'semantica' is not the prefix"
def test_ntriples_export_agrees_with_turtle_on_the_minted_iri():
exporter = RDFExporter()
minted = mint_entity_iri("Acme Corp")
assert minted in exporter.export_to_rdf(UNIDENTIFIED, format="turtle")
assert minted in exporter.export_to_rdf(UNIDENTIFIED, format="ntriples")
def test_default_types_are_written_as_full_iris_in_turtle():
untyped = {"entities": [{"id": "https://example.org/e1", "text": "A"}],
"relationships": [{"source": "https://example.org/e1",
"target": "https://example.org/e2"}]}
turtle = RDFExporter().export_to_rdf(untyped, format="turtle")
assert f"<{DEFAULT_ENTITY_TYPE}>" in turtle
assert f"<{DEFAULT_RELATION_TYPE}>" in turtle
assert "<semantica:Entity>" not in turtle
assert "<semantica:related_to>" not in turtle
def test_default_entity_type_is_a_full_iri_in_rdfxml():
"""RDF/XML's rdf:resource is an attribute value, not a QName context, so a
prefixed default there (``semantica:Entity``) resolves to the scheme
``semantica`` rather than the declared namespace the same failure mode
fixed for Turtle in #1101, missed here because the original tests only
checked Turtle output.
"""
untyped = {"entities": [{"id": "https://example.org/e1", "text": "A"}],
"relationships": []}
rdfxml = RDFExporter().export_to_rdf(untyped, format="rdfxml")
assert f'rdf:resource="{DEFAULT_ENTITY_TYPE}"' in rdfxml
assert 'rdf:resource="semantica:Entity"' not in rdfxml
def test_temporal_minting_uses_either_endpoint_representation():
"""Relationships may carry source/target or source_id/target_id (#1109 review).
Minting from source_id alone hashed empty strings for every relationship
that used the other representation, so once the IRI became deterministic,
unrelated relationships at the same index collided on it and their temporal
data aliased when the exports were loaded together.
"""
def temporal(rel):
return RDFExporter().export_to_rdf(
{"entities": [], "relationships": [rel]},
format="turtle",
include_temporal=True,
)
a = temporal({"source": "https://example.org/a", "target": "https://example.org/b",
"type": "https://example.org/worksFor", "valid_from": "2020-01-01T00:00:00Z"})
b = temporal({"source": "https://example.org/c", "target": "https://example.org/d",
"type": "https://example.org/worksFor", "valid_from": "2020-01-01T00:00:00Z"})
assert f"<{SEMANTICA_NS}rel_" in a
assert a != b, "different endpoints must not mint the same temporal IRI"
def test_temporal_minting_agrees_across_the_two_representations():
"""The same relationship written either way is the same relationship."""
def mint(rel):
return mint_relationship_iri(
0,
rel.get("source_id") or rel.get("source") or "",
rel.get("target_id") or rel.get("target") or "",
)
assert mint({"source": "a", "target": "b"}) == mint({"source_id": "a", "target_id": "b"})
-35
View File
@@ -1,35 +0,0 @@
"""
Shared pytest fixtures for the ingest test suite.
The ``mock_dns`` fixture is applied to *every* test in this directory
(``autouse=True``). It stubs out ``socket.getaddrinfo`` inside the SSRF
guard module so that unit tests that mock ``requests.Session.request`` do not
accidentally hit the network for DNS resolution which would fail in offline
CI environments and cause intermittent timeouts.
Tests that explicitly need to exercise DNS-related behaviour (e.g. checking
that a hostname resolving to a private IP is blocked) override this fixture
by patching ``semantica.ingest.ssrf.socket.getaddrinfo`` with their own
``side_effect`` *inside* the test body; that inner patch wins because
``unittest.mock.patch`` applies patches in innermost-last order.
"""
from __future__ import annotations
import socket
from unittest.mock import patch
import pytest
_PUBLIC_IP = "93.184.216.34" # example.com — a safe, routable public address
@pytest.fixture(autouse=True)
def mock_dns():
"""Map every hostname to a safe public IP for the duration of each test."""
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[
(socket.AF_INET, socket.SOCK_STREAM, 6, "", (_PUBLIC_IP, 0))
],
):
yield
File diff suppressed because it is too large Load Diff
+20 -20
View File
@@ -10,20 +10,19 @@ class TestCookbookIntegration:
@pytest.fixture
def mock_mcp_server(self):
# MCPClient._send_request_http now routes through request_with_ssrf_guard,
# which calls requests.request (not httpx.post / requests.post directly).
# Patch at the point where the guard issues the actual HTTP call.
with patch("requests.Session.request") as mock_request:
def side_effect(method, url, json=None, **kwargs):
# We need to patch both httpx and requests because MCPClient tries httpx first
with patch("httpx.post") as mock_httpx_post, \
patch("requests.post") as mock_requests_post:
def side_effect(url, json=None, **kwargs):
if not json:
return MagicMock()
rpc_method = json.get("method")
method = json.get("method")
response_mock = MagicMock()
response_mock.status_code = 200
if rpc_method == "initialize":
if method == "initialize":
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -33,7 +32,7 @@ class TestCookbookIntegration:
"serverInfo": {"name": "test_server", "version": "1.0"}
}
}
elif rpc_method == "resources/list":
elif method == "resources/list":
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -45,7 +44,7 @@ class TestCookbookIntegration:
]
}
}
elif rpc_method == "tools/list":
elif method == "tools/list":
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -57,7 +56,7 @@ class TestCookbookIntegration:
]
}
}
elif rpc_method == "resources/read":
elif method == "resources/read":
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -67,13 +66,13 @@ class TestCookbookIntegration:
]
}
}
elif rpc_method == "tools/call":
elif method == "tools/call":
tool_name = json.get("params", {}).get("name")
content = [{"type": "text", "text": "Tool Output"}]
if tool_name == "query_inventory":
content = [{"type": "text", "text": '{"warehouse_id": "WH001", "level": 100}'}]
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -87,11 +86,12 @@ class TestCookbookIntegration:
"id": json.get("id"),
"result": {}
}
return response_mock
mock_request.side_effect = side_effect
yield mock_request
mock_httpx_post.side_effect = side_effect
mock_requests_post.side_effect = side_effect
yield mock_httpx_post
def test_financial_data_integration(self, mock_mcp_server):
"""
+2 -2
View File
@@ -222,7 +222,7 @@ def test_discover_feeds_empty() -> None:
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("93.184.216.34", 0))],
):
with patch("requests.Session.request", side_effect=fake_request):
with patch("requests.request", side_effect=fake_request):
feeds = ingestor.discover_feeds("http://site.com")
assert len(feeds) == 0
@@ -254,7 +254,7 @@ def test_discover_feeds_found() -> None:
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("93.184.216.34", 0))],
):
with patch("requests.Session.request", side_effect=fake_request):
with patch("requests.request", side_effect=fake_request):
feeds = ingestor.discover_feeds("http://site.com")
assert "http://site.com/rss.xml" in feeds
+7 -78
View File
@@ -198,82 +198,15 @@ def test_public_api_detection_reports_auth_required() -> None:
headers={"WWW-Authenticate": "Bearer"},
)
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
"https://api.example.com/private"
)
detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
"https://api.example.com/private"
)
assert detection.is_public is False
assert detection.requires_auth is True
assert detection.response_status == 401
def test_detect_public_api_propagates_ssrf_validation_error() -> None:
"""detect_public_api() must surface ValidationError, not swallow it.
request_with_ssrf_guard() raises ValidationError (not
requests.exceptions.RequestException) for SSRF-blocked hosts, so
detect_public_api()'s error handling must catch it explicitly like its
sibling ingest_public_api() already does.
"""
with patch("requests.Session") as mock_session_class:
mock_session = mock_session_class.return_value
mock_session.headers = {}
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("127.0.0.1", 0))],
):
with pytest.raises(ValidationError):
PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
"https://blocked.example.com/data"
)
def test_detect_public_api_rejects_duplicate_session_and_allow_private_ips_kwargs() -> None:
"""Passing session/allow_private_ips through **options must not crash.
Both are always supplied explicitly to request_with_ssrf_guard(); caller
copies must be dropped from **options rather than causing a
'got multiple values for keyword argument' TypeError.
"""
with patch("requests.Session") as mock_session_class:
mock_session = mock_session_class.return_value
mock_session.headers = {}
mock_session.request.return_value = _mock_response(
headers={"Content-Type": "application/json"}
)
detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
"https://jsonplaceholder.typicode.com/posts",
allow_private_ips=True,
session=object(),
)
assert detection.is_public is True
def test_ingest_public_api_rejects_duplicate_session_and_allow_private_ips_kwargs() -> None:
with patch("requests.Session") as mock_session_class:
mock_session = mock_session_class.return_value
mock_session.headers = {}
mock_session.request.return_value = _mock_response(
json_payload=[{"id": 1}],
headers={"Content-Type": "application/json"},
)
result = PublicAPIIngestor(rate_limit_delay=0).ingest_public_api(
"https://jsonplaceholder.typicode.com/posts",
allow_private_ips=True,
session=object(),
)
assert result.response_status == 200
def test_public_api_ingestor_rejects_authentication_inputs() -> None:
with patch("requests.Session") as mock_session_class:
mock_session = mock_session_class.return_value
@@ -305,14 +238,10 @@ def test_public_api_ingestor_parses_string_boolean_config() -> None:
config={"validate_no_auth": "false"},
rate_limit_delay=0,
)
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
result = ingestor.ingest_public_api(
"https://api.example.com/data",
headers={"Authorization": "Bearer token"},
)
result = ingestor.ingest_public_api(
"https://api.example.com/data",
headers={"Authorization": "Bearer token"},
)
assert ingestor.validate_no_auth is False
assert result.data == payload
+1 -1
View File
@@ -447,7 +447,7 @@ class TestSitemapCrawlerSSRF:
redirect.close = MagicMock()
with patch(
"requests.Session.request", return_value=redirect
"semantica.ingest.ssrf.requests.request", return_value=redirect
), patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
+81 -54
View File
@@ -195,62 +195,89 @@ class TestMCPIngestor:
class TestMCPClient:
def test_call_tool(self):
# MCPClient._send_request_http now routes through request_with_ssrf_guard,
# which calls requests.request (not requests.post) with allow_redirects=False.
# Patch the requests.request call inside ssrf.py.
with patch("requests.Session.request") as mock_request:
mock_response = MagicMock()
mock_response.status_code = 200
# Response for initialize
init_response = {
"jsonrpc": "2.0",
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
"id": 1,
}
# Response for tool call
tool_response = {
"jsonrpc": "2.0",
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
"id": 2,
}
mock_response.json.side_effect = [init_response, tool_response]
mock_request.return_value = mock_response
client = MCPClient(url="http://localhost:8000")
client.connect()
client.call_tool("my_tool", {"arg": "val"})
# Patch requests.post globally if requests is used, or httpx.post if httpx is used.
# The code tries importing httpx, then requests.
# We should patch both or ensure we catch the right one.
# Simpler to patch sys.modules to simulate httpx missing, then patch requests.
with patch.dict(sys.modules, {'httpx': None}):
with patch("requests.post") as mock_post:
mock_response = MagicMock()
mock_response.status_code = 200
# Sequence of calls:
# 1. connect() calls _connect_http() -> calls _initialize() -> calls _send_request()
# _send_request() calls requests.post with method="initialize"
# 2. call_tool() calls _send_request() with method="tools/call"
# Response for initialize
init_response = {
"jsonrpc": "2.0",
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
"id": 1
}
# Response for tool call
tool_response = {
"jsonrpc": "2.0",
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
"id": 2
}
mock_response.json.side_effect = [init_response, tool_response]
mock_post.return_value = mock_response
client = MCPClient(url="http://localhost:8000")
client.connect()
result = client.call_tool("my_tool", {"arg": "val"})
# result is the dict returned by tool call?
# call_tool returns dict?
# Check MCPClient.call_tool implementation
# It calls _send_request, which returns response.json().
# But wait, call_tool might process the result.
# Let's check call_tool implementation in mcp_client.py (not read yet, but assumed).
# Wait, I read mcp_client.py but didn't check call_tool specifically.
# Assuming call_tool returns result part or whole response.
# Actually, let's verify call_tool in mcp_client.py
pass
def test_call_tool_mock_check(self):
# Redo with the corrected patch target.
with patch("requests.Session.request") as mock_request:
mock_response = MagicMock()
mock_response.status_code = 200
init_response = {
"jsonrpc": "2.0",
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
"id": 1,
}
tool_response = {
"jsonrpc": "2.0",
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
"id": 2,
}
mock_response.json.side_effect = [init_response, tool_response]
mock_request.return_value = mock_response
client = MCPClient(url="http://localhost:8000")
client.connect()
result = client.call_tool("my_tool", {"arg": "val"})
assert result["content"] == [{"type": "text", "text": "Tool Result"}]
# Redoing the test with more specific mocking logic
with patch.dict(sys.modules, {'httpx': None}):
with patch("requests.post") as mock_post:
mock_response = MagicMock()
mock_response.status_code = 200
# initialize response
init_response = {
"jsonrpc": "2.0",
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
"id": 1
}
# tool call response - Assuming call_tool returns the 'result' part of JSON-RPC response
# If call_tool implementation wraps it, we need to know.
# Let's assume standard behavior for now.
tool_response = {
"jsonrpc": "2.0",
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
"id": 2
}
mock_response.json.side_effect = [init_response, tool_response]
mock_post.return_value = mock_response
client = MCPClient(url="http://localhost:8000")
client.connect()
result = client.call_tool("my_tool", {"arg": "val"})
# Verify result.
# If call_tool returns the 'result' dict from JSON-RPC:
assert result["content"] == [{"type": "text", "text": "Tool Result"}]
class TestGDriveIngestor:
def test_init_raises_if_no_google_libs(self):
+1 -56
View File
@@ -1,10 +1,9 @@
import pytest
from semantica.utils.types import Entity
from semantica.kg.graph_builder import GraphBuilder
from semantica.kg.entity_resolver import EntityResolver
from semantica.kg.graph_analyzer import GraphAnalyzer
from semantica.utils.entity_ids import get_entity_id
from semantica.utils.types import Entity
def test_full_entity_pipeline():
"""
@@ -108,60 +107,6 @@ def test_direct_entity_objects_in_analyzer():
print("Direct Entity objects test passed successfully!")
def test_entity_id_only_merge_remaps_relationship_endpoints():
"""Entity aliases must survive merging and relationship remapping."""
builder = GraphBuilder(
merge_entities=True,
entity_resolution_strategy="exact",
resolve_conflicts=False,
)
graph = builder.build(
{
"entities": [
{"entity_id": "alice:1", "name": "Alice", "type": "Person"},
{"entity_id": "alice:2", "name": " Alice ", "type": "Person"},
{"entity_id": "org:1", "name": "Acme", "type": "Organization"},
],
"relationships": [
{
"source_id": "alice:2",
"target_id": "org:1",
"type": "WORKS_FOR",
}
],
}
)
merged_alice = next(
entity for entity in graph["entities"] if entity["name"] == "Alice"
)
relationship = graph["relationships"][0]
entity_ids = {
entity.get("id") or entity.get("entity_id") for entity in graph["entities"]
}
assert merged_alice["id"] == "alice:1"
assert set(merged_alice["merged_from"]) == {"alice:1", "alice:2"}
assert {
item["id"] for item in merged_alice["metadata"]["provenance"]["merged_from"]
} == {"alice:1", "alice:2"}
assert relationship["source"] == "alice:1"
assert relationship["target"] == "org:1"
assert {relationship["source"], relationship["target"]} <= entity_ids
def test_entity_id_helper_ignores_falsy_identifiers():
"""ID extraction must match the KG pipeline's falsy-ID contract."""
assert get_entity_id({"id": "", "entity_id": "alias:1"}) == "alias:1"
assert get_entity_id({"id": 0, "entity_id": "alias:2"}) == "alias:2"
assert (
get_entity_id({"id": "primary:1", "entity_id": "alias:3"})
== "primary:1"
)
assert get_entity_id({"id": "", "entity_id": 0}) is None
if __name__ == "__main__":
test_full_entity_pipeline()
test_direct_entity_objects_in_analyzer()
-56
View File
@@ -1,56 +0,0 @@
from semantica.kg.entity_resolver import EntityResolver
def test_exact_resolution_does_not_merge_similar_names():
entities = [
{"id": "1", "name": "Alice"},
{"id": "2", "name": "Alicia"},
]
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
assert {entity["id"] for entity in resolved} == {"1", "2"}
def test_exact_resolution_merges_case_and_whitespace_variants():
entities = [
{"id": "1", "name": " Alice "},
{"id": "2", "name": "alice"},
]
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
assert len(resolved) == 1
def test_resolution_preserves_non_duplicate_entities_without_ids():
entities = [
{"name": "Alice"},
{"name": "Bob"},
]
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
assert resolved == entities
def test_exact_resolution_does_not_merge_whitespace_only_names():
entities = [
{"id": "1", "name": " "},
{"id": "2", "name": "\t"},
]
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
assert {entity["id"] for entity in resolved} == {"1", "2"}
def test_exact_resolution_falls_back_to_text_when_name_is_blank():
entities = [
{"id": "1", "name": " ", "text": "Alice"},
{"id": "2", "name": "Alice"},
]
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
assert len(resolved) == 1
-97
View File
@@ -414,103 +414,6 @@ class TestSHACLHierarchicalAndValidation(unittest.TestCase):
self.assertIsNotNone(v.explanation)
self.assertIn("https://example.com/john", v.explanation)
# 32b
def test_explain_violations_uses_real_constraint_values(self):
"""explain_violations must render the real min/max/datatype/class values,
not hardcoded placeholders (regression for PR #318)."""
from semantica.ontology.ontology_validator import (
SHACLValidationReport,
SHACLViolation,
)
max_v = SHACLViolation(
focus_node="https://example.com/john",
result_path="ex:age",
constraint="MaxCountConstraintComponent",
max_count=3,
)
dt_v = SHACLViolation(
focus_node="https://example.com/john",
result_path="ex:age",
constraint="DatatypeConstraintComponent",
value="abc",
datatype="http://www.w3.org/2001/XMLSchema#integer",
)
cls_v = SHACLViolation(
focus_node="https://example.com/john",
result_path="ex:knows",
constraint="ClassConstraintComponent",
value="https://example.com/thing",
class_="https://example.com/Person",
)
report = SHACLValidationReport(
conforms=False, violations=[max_v, dt_v, cls_v]
)
report.explain_violations()
# MaxCount must show the real limit (3), not the hardcoded 1.
self.assertIn("3", max_v.explanation)
self.assertNotIn("At most 1 value", max_v.explanation)
# Datatype must show the real datatype IRI, not the message.
self.assertIn(
"http://www.w3.org/2001/XMLSchema#integer", dt_v.explanation
)
# Class must show the real class IRI.
self.assertIn("https://example.com/Person", cls_v.explanation)
# 32c
def test_run_pyshacl_extracts_constraint_values_from_shape(self):
"""_run_pyshacl must back-reference sh:sourceShape to populate the real
constraint parameters on each violation."""
try:
import pyshacl # noqa: F401
import rdflib # noqa: F401
except ImportError:
self.skipTest("pyshacl/rdflib not installed")
from semantica.ontology.ontology_validator import _run_pyshacl
shacl = """
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix ex: <http://example.com/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
ex:PersonShape a sh:NodeShape ;
sh:targetClass ex:Person ;
sh:property [
sh:path ex:age ;
sh:datatype xsd:integer ;
sh:maxCount 2 ;
] .
"""
data = """
@prefix ex: <http://example.com/> .
ex:john a ex:Person ;
ex:age "not-a-number" ;
ex:age 1 ;
ex:age 2 ;
ex:age 3 .
"""
report = _run_pyshacl(data, shacl)
self.assertFalse(report.conforms)
# Datatype violation should carry the real xsd:integer datatype.
dt = [
v
for v in report.violations
if v.constraint == "DatatypeConstraintComponent"
]
self.assertTrue(dt)
self.assertTrue(
dt[0].datatype.endswith("integer"),
f"expected integer datatype, got {dt[0].datatype}",
)
# MaxCount violation should carry the real max_count == 2.
mc = [
v
for v in report.violations
if v.constraint == "MaxCountConstraintComponent"
]
if mc:
self.assertEqual(mc[0].max_count, 2)
# 33
def test_shacl_violation_to_dict(self):
from semantica.ontology.ontology_validator import SHACLViolation
-116
View File
@@ -1,116 +0,0 @@
"""The vocabulary must stay true to what the exporters emit (issue #1107).
A vocabulary document that drifts from the code is worse than none, because it
states that terms mean something while the exporters emit different ones. These
tests tie the two together: every term the serializers can write must be
declared here, so adding a term to an exporter without declaring it fails the
build rather than shipping an undeclared IRI.
"""
import pytest
rdflib = pytest.importorskip("rdflib")
from semantica.export.rdf_exporter import ( # noqa: E402
DEFAULT_ENTITY_TYPE,
DEFAULT_RELATION_TYPE,
SEMANTICA_NS,
)
from semantica.ontology.vocabulary import ( # noqa: E402
NAMESPACE,
vocabulary_path,
vocabulary_turtle,
)
#: Every term the exporters emit in the Semantica namespace, by local name.
#: RDF and OWL-Time paths in export/rdf_exporter.py, document and relationship
#: terms in export/json_exporter.py, roles in provenance/manager.py.
EMITTED_TERMS = {
"Entity",
"Relationship",
"KnowledgeGraph",
"text",
"confidence",
"metadata",
"related_to",
"source",
"target",
"type",
"entities",
"relationships",
"exportedAt",
"format",
"openEndedInterval",
"role_generator",
}
@pytest.fixture(scope="module")
def graph():
g = rdflib.Graph()
g.parse(data=vocabulary_turtle(), format="turtle")
return g
def test_vocabulary_ships_with_the_package():
assert vocabulary_path().is_file()
def test_vocabulary_parses(graph):
assert len(graph) > 0
def test_namespace_matches_the_one_the_exporters_use():
assert NAMESPACE == SEMANTICA_NS
def test_every_emitted_term_is_declared(graph):
declared = {
str(s)[len(NAMESPACE) :]
for s in set(graph.subjects())
if isinstance(s, rdflib.URIRef) and str(s).startswith(NAMESPACE)
}
missing = EMITTED_TERMS - declared
assert not missing, f"emitted but not declared in the vocabulary: {sorted(missing)}"
def test_the_defaults_the_exporters_fall_back_to_are_declared(graph):
for iri in (DEFAULT_ENTITY_TYPE, DEFAULT_RELATION_TYPE):
assert (rdflib.URIRef(iri), None, None) in graph, f"{iri} is not declared"
def test_every_declared_term_carries_a_label_and_a_comment(graph):
for subject in set(graph.subjects()):
if not (isinstance(subject, rdflib.URIRef) and str(subject).startswith(NAMESPACE)):
continue
assert graph.value(subject, rdflib.RDFS.label), f"{subject} has no rdfs:label"
assert graph.value(subject, rdflib.RDFS.comment), f"{subject} has no rdfs:comment"
def test_declared_ranges_do_not_contradict_what_the_exporters_emit(graph):
"""A declared range must match the datatype the serializers actually write.
Caught by review on #1109: sem:confidence was declared xsd:decimal while the
N-Triples serializer types the same value xsd:float. A vocabulary that
contradicts the code is worse than no vocabulary, so any range declared here
has to be one the exporters really emit.
"""
import re
from semantica.export.rdf_exporter import RDFExporter
sample = {
"entities": [{"id": "https://example.org/e1", "text": "A",
"type": "https://example.org/T", "confidence": 0.5}],
"relationships": [],
}
emitted = RDFExporter().export_to_rdf(sample, format="ntriples")
for subject, _, range_ in graph.triples((None, rdflib.RDFS.range, None)):
if not str(subject).startswith(NAMESPACE):
continue
local = str(subject)[len(NAMESPACE):]
for match in re.finditer(rf'<{NAMESPACE}{local}> "[^"]*"\^\^<([^>]+)>', emitted):
assert match.group(1) == str(range_), (
f"{local}: vocabulary declares {range_}, N-Triples emits {match.group(1)}"
)
-321
View File
@@ -1,321 +0,0 @@
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
from semantica.semantic_extract import methods as se_methods
from semantica.split import methods as split_methods
from semantica.split import semantic_chunker
from semantica.semantic_extract import ner_extractor as ner_extractor_module
from semantica.semantic_extract.ner_extractor import NERExtractor
@pytest.fixture(autouse=True)
def clear_cache():
se_methods.clear_spacy_model_cache()
yield
se_methods.clear_spacy_model_cache()
@pytest.fixture(autouse=True)
def force_spacy_available(monkeypatch):
# split.methods, split.semantic_chunker, and ner_extractor each compute
# their own SPACY_AVAILABLE flag from the real environment at import time;
# force all true so these tests exercise the spaCy branch regardless of
# whether spaCy is actually installed where they run.
monkeypatch.setattr(split_methods, "SPACY_AVAILABLE", True)
monkeypatch.setattr(semantic_chunker, "SPACY_AVAILABLE", True)
monkeypatch.setattr(ner_extractor_module, "SPACY_AVAILABLE", True)
def _fake_spacy(load):
return SimpleNamespace(
load=load,
util=SimpleNamespace(is_package=lambda _name: True),
)
def _nlp_mock(sentences=("Hello world.",)):
"""A stand-in spaCy Language object: callable, returns a doc with .sents."""
nlp = MagicMock()
nlp.return_value = SimpleNamespace(
sents=[SimpleNamespace(text=s) for s in sentences]
)
return nlp
class TestSpacyModelCache:
"""split.methods and split.semantic_chunker must share the cached model
defined in semantic_extract.methods instead of each calling spacy.load()
independently.
"""
def test_split_by_sentences_reuses_cached_model(self, monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append((name, kwargs))
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
split_methods.split_by_sentences("Hello world. Bye world.")
split_methods.split_by_sentences("Another sentence here.")
split_methods.split_by_sentences("A third call.")
assert len(calls) == 1, "spacy.load should run once, not once per call"
assert calls[0][0] == "en_core_web_sm"
def test_semantic_chunker_reuses_cached_model_across_instances(self, monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append((name, kwargs))
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
chunker1 = semantic_chunker.SemanticChunker()
chunker2 = semantic_chunker.SemanticChunker()
assert len(calls) == 1, "each new SemanticChunker should not reload the model"
assert chunker1.nlp is chunker2.nlp
def test_split_methods_and_semantic_chunker_share_the_cache(self, monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append((name, kwargs))
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
split_methods.split_by_sentences("Test sentence for split.methods.")
semantic_chunker.SemanticChunker()
assert len(calls) == 1, (
"split.methods and split.semantic_chunker must share one cached "
"model instead of each loading their own"
)
def test_distinct_model_names_load_separately(self, monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append((name, kwargs))
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
sm_chunker = semantic_chunker.SemanticChunker(model="en_core_web_sm")
lg_chunker = semantic_chunker.SemanticChunker(model="en_core_web_lg")
sm_chunker_again = semantic_chunker.SemanticChunker(model="en_core_web_sm")
assert [name for name, _ in calls] == ["en_core_web_sm", "en_core_web_lg"]
assert sm_chunker.nlp is sm_chunker_again.nlp
assert sm_chunker.nlp is not lg_chunker.nlp
def test_no_disable_kwarg_requested(self, monkeypatch):
"""split.methods and split.semantic_chunker both want the full
pipeline (they need .sents, which requires the parser/senter). If
either one later starts requesting a trimmed pipeline (e.g.
disable=["ner"]), the name-only cache key in load_spacy_model would
silently hand back a cached model built for a different config --
this test should catch that the moment it happens.
"""
calls = []
def fake_load(_name, **kwargs):
calls.append(kwargs)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
split_methods.split_by_sentences("Hello world.")
se_methods.clear_spacy_model_cache()
semantic_chunker.SemanticChunker()
assert len(calls) == 2
assert all(kwargs == {} for kwargs in calls), (
"neither caller should pass any pipeline-configuration kwargs; "
"the name-only cache key in load_spacy_model cannot distinguish "
"models loaded with different component configs"
)
def test_missing_model_falls_back_without_poisoning_cache(self, monkeypatch):
attempts = []
def failing_load(name, **_kwargs):
attempts.append(name)
raise OSError(f"Can't find model '{name}'")
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load))
# split_by_sentences should fall back to regex splitting, not raise
chunks = split_methods.split_by_sentences("Hello world. Bye world.")
assert chunks, "fallback splitting should still produce chunks"
# SemanticChunker should leave .nlp as None rather than propagate
chunker = semantic_chunker.SemanticChunker()
assert chunker.nlp is None
assert len(attempts) == 2, "a failed load must not be cached"
# Once the model is available, both callers should now get it, and
# share a single successful load.
def working_load(name, **_kwargs):
attempts.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load))
chunker2 = semantic_chunker.SemanticChunker()
split_methods.split_by_sentences("One more sentence.")
assert len(attempts) == 3, (
"the model should load once after it becomes available"
)
assert chunker2.nlp is not None
def test_semantic_chunker_falls_back_when_spacy_runtime_is_broken(
self, monkeypatch
):
"""A spaCy model that is installed but unusable at runtime (e.g. a
config incompatible with the installed spaCy version) must degrade
SemanticChunker to fallback chunking, not crash __init__ -- mirrors
TestNERExtractorSpacyModelCache's equivalent broken-runtime test.
"""
def broken_load(name, **_kwargs):
raise RuntimeError("ConfigSchemaNlp is not fully defined")
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(broken_load))
chunker = semantic_chunker.SemanticChunker()
assert chunker.nlp is None
class TestNERExtractorSpacyModelCache:
"""NERExtractor(method="ml") must reuse the centralized cache in
semantic_extract.methods, not call spacy.load() on every construction.
These tests mirror TestSpacyModelCache but focus on the NERExtractor path,
confirming that all three callers (split_by_sentences, SemanticChunker, and
NERExtractor) draw from the same process-level cache.
"""
def test_ner_extractor_reuses_cached_model_across_instances(self, monkeypatch):
"""Two NERExtractor(method='ml') constructions with the same model name
must cause exactly one underlying spacy.load() call."""
calls = []
def fake_load(name, **kwargs):
calls.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
e1 = NERExtractor(method="ml")
e2 = NERExtractor(method="ml")
e3 = NERExtractor(method="ml", model="en_core_web_sm")
assert len(calls) == 1, (
"repeated NERExtractor constructions should not reload the model"
)
assert e1.nlp is e2.nlp is e3.nlp
def test_ner_extractor_and_split_callers_share_one_cached_model(self, monkeypatch):
"""NERExtractor, SemanticChunker, and split_by_sentences must all use
the same cached Language object for the same model name."""
calls = []
def fake_load(name, **kwargs):
calls.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
split_methods.split_by_sentences("First sentence.")
semantic_chunker.SemanticChunker()
NERExtractor(method="ml")
assert len(calls) == 1, (
"split_by_sentences, SemanticChunker, and NERExtractor must share "
"one cached model instead of each loading their own"
)
def test_ner_extractor_distinct_model_names_load_separately(self, monkeypatch):
"""Different model names must produce separate cache entries."""
calls = []
def fake_load(name, **kwargs):
calls.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
sm = NERExtractor(method="ml", model="en_core_web_sm")
lg = NERExtractor(method="ml", model="en_core_web_lg")
sm_again = NERExtractor(method="ml", model="en_core_web_sm")
assert calls == ["en_core_web_sm", "en_core_web_lg"]
assert sm.nlp is sm_again.nlp
assert sm.nlp is not lg.nlp
def test_ner_extractor_failed_load_not_cached_and_retried(self, monkeypatch):
"""A missing model must not poison the cache. A subsequent construction
after the model becomes available must succeed and share the loaded model."""
attempts = []
def failing_load(name, **_kwargs):
attempts.append(name)
raise OSError(f"Can't find model '{name}'")
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load))
# Construction with missing model: nlp must remain None, no crash
extractor1 = NERExtractor(method="ml")
assert extractor1.nlp is None
assert len(attempts) == 1, "one load attempt expected for the missing model"
# Second construction: must retry (cache must not hold the failure)
extractor2 = NERExtractor(method="ml")
assert extractor2.nlp is None
assert len(attempts) == 2, "a failed load must not be cached"
# Now install a working model and verify recovery
def working_load(name, **_kwargs):
attempts.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load))
extractor3 = NERExtractor(method="ml")
extractor4 = NERExtractor(method="ml")
assert extractor3.nlp is not None
assert extractor3.nlp is extractor4.nlp
assert len(attempts) == 3, (
"exactly one successful load expected after the model becomes available"
)
def test_ner_extractor_non_ml_method_does_not_load_model(self, monkeypatch):
"""NERExtractor with a non-ml method must not touch the spaCy cache."""
calls = []
def fake_load(name, **kwargs):
calls.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
NERExtractor(method="pattern")
NERExtractor(method="llm")
NERExtractor(method="regex")
assert calls == [], "non-ml methods must not trigger any spacy.load()"
if __name__ == "__main__":
pytest.main([__file__])
+9 -7
View File
@@ -30,16 +30,18 @@ class TestSplitter(unittest.TestCase):
splitter = TextSplitter(method=["recursive", "token"])
self.assertEqual(splitter.methods, ["recursive", "token"])
@patch('semantica.semantic_extract.methods.spacy')
@patch('semantica.split.semantic_chunker.spacy')
def test_semantic_chunker_initialization(self, mock_spacy):
# SemanticChunker now loads spaCy through the centralized
# load_spacy_model() in semantic_extract.methods, so we patch
# methods.spacy rather than the removed semantic_chunker.spacy binding.
# Mock spacy.load to return a mock nlp object
mock_nlp = MagicMock()
mock_spacy.load.return_value = mock_nlp
with patch('semantica.split.semantic_chunker.SPACY_AVAILABLE', True):
chunker = SemanticChunker(chunk_size=100)
# We need to ensure SPACY_AVAILABLE is True for this test context if possible,
# but it is imported at module level.
# If spacy is not installed, it sets SPACY_AVAILABLE = False.
# We might need to patch the module attribute or just test fallback if spacy missing.
chunker = SemanticChunker(chunk_size=100)
self.assertEqual(chunker.chunk_size, 100)
def test_chunk_dataclass(self):
@@ -54,11 +54,6 @@ from semantica.context.decision_models import (
validate_decision,
)
# ── Export module ──────────────────────────────────────────────────────────────
# Set by the exporter's own `import pyarrow` attempt; False when pyarrow is
# missing or unimportable.
from semantica.export.parquet_exporter import PARQUET_AVAILABLE
# ── KG module ──────────────────────────────────────────────────────────────────
from semantica.kg import (
CentralityCalculator,
@@ -986,17 +981,6 @@ class TestParquetExportRealData:
Requires: pyarrow (optional dep tests skip if not installed).
"""
# ParquetExporter imports fine without pyarrow and only raises ImportError
# when an export actually runs, so guarding on that import never skips
# anything. Guard on the exporter's own availability flag instead: it is set
# by the same `import pyarrow` / `import pyarrow.parquet` the exporter gates
# on, so the skip condition cannot drift from the runtime check — including
# when pyarrow is present on the path but fails to import.
pytestmark = pytest.mark.skipif(
not PARQUET_AVAILABLE,
reason="pyarrow not installed",
)
@pytest.fixture
def kg_data(self):
return {
@@ -1016,12 +1000,16 @@ class TestParquetExportRealData:
}
def test_parquet_exporter_importable(self):
from semantica.export import ParquetExporter
assert ParquetExporter is not None
try:
from semantica.export import ParquetExporter
except ImportError as e:
pytest.skip(f"ParquetExporter not available: {e}")
def test_parquet_export_entities_to_file(self, kg_data, tmp_path):
from semantica.export import ParquetExporter
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
exporter = ParquetExporter(compression="snappy")
out_path = tmp_path / "github_entities.parquet"
@@ -1030,7 +1018,10 @@ class TestParquetExportRealData:
assert out_path.stat().st_size > 0
def test_parquet_export_relationships_to_file(self, kg_data, tmp_path):
from semantica.export import ParquetExporter
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
exporter = ParquetExporter(compression="gzip")
out_path = tmp_path / "github_relationships.parquet"
@@ -1039,7 +1030,10 @@ class TestParquetExportRealData:
assert out_path.stat().st_size > 0
def test_parquet_export_knowledge_graph(self, kg_data, tmp_path):
from semantica.export import ParquetExporter
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
exporter = ParquetExporter(compression="snappy")
base_path = tmp_path / "github_kg"
@@ -1049,24 +1043,30 @@ class TestParquetExportRealData:
assert len(files) >= 1
def test_parquet_export_snappy_compression(self, kg_data, tmp_path):
from semantica.export import ParquetExporter
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
exporter = ParquetExporter(compression="snappy")
out_path = tmp_path / "snappy_test.parquet"
exporter.export_entities(kg_data["entities"], str(out_path))
assert out_path.exists()
def test_parquet_export_none_compression(self, kg_data, tmp_path):
from semantica.export import ParquetExporter
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
exporter = ParquetExporter(compression="none")
out_path = tmp_path / "uncompressed_test.parquet"
exporter.export_entities(kg_data["entities"], str(out_path))
assert out_path.exists()
def test_parquet_convenience_function(self, kg_data, tmp_path):
from semantica.export.methods import export_parquet
try:
from semantica.export.methods import export_parquet
except ImportError:
pytest.skip("pyarrow not installed")
out_path = tmp_path / "convenience_test.parquet"
export_parquet(kg_data["entities"], str(out_path))
assert out_path.exists()
+8 -21
View File
@@ -101,15 +101,9 @@ class TestNERConfigurations(unittest.TestCase):
self.assertEqual(entities[0].metadata["extraction_method"], "ml")
self.assertEqual(entities[0].metadata["model"], "en_core_web_trf")
@patch('semantica.semantic_extract.methods.spacy')
@patch('semantica.semantic_extract.ner_extractor.spacy')
def test_ner_ml_init_falls_back_when_spacy_runtime_is_broken(self, mock_spacy):
"""Test NER init does not crash when spaCy is installed but unusable at runtime.
The model load now goes through load_spacy_model() in semantic_extract.methods,
so we patch methods.spacy (not ner_extractor.spacy) to inject the failure.
"""
from semantica.semantic_extract.methods import clear_spacy_model_cache
clear_spacy_model_cache()
"""Test NER init does not crash when spaCy is installed but unusable at runtime."""
mock_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined")
with patch('semantica.semantic_extract.ner_extractor.SPACY_AVAILABLE', True):
@@ -118,23 +112,17 @@ class TestNERConfigurations(unittest.TestCase):
self.assertIsNone(extractor.nlp)
self.assertFalse(extractor._ml_runtime_usable)
@patch('semantica.semantic_extract.ner_extractor.spacy')
@patch('semantica.semantic_extract.methods.get_entity_method')
@patch('semantica.semantic_extract.methods.spacy')
def test_ner_ml_runtime_failure_disables_repeated_ml_load_attempts(
self,
mock_methods_spacy,
mock_get_method,
mock_init_spacy,
):
"""Test degraded ML mode skips repeated spaCy load attempts after init failure.
The model load at construction time now goes through load_spacy_model() in
semantic_extract.methods, so methods.spacy is the single mock target for the
init-time failure. After the RuntimeError is raised, _ml_runtime_usable is
False and no further spacy.load (or extract_entities_ml) calls are made.
"""
from semantica.semantic_extract.methods import clear_spacy_model_cache
clear_spacy_model_cache()
mock_methods_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined")
"""Test degraded ML mode skips repeated spaCy load attempts after init failure."""
mock_init_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined")
mock_ml_method = MagicMock(return_value=[])
mock_get_method.side_effect = lambda name: mock_ml_method if name == "ml" else (lambda *_args, **_kwargs: [])
@@ -144,9 +132,8 @@ class TestNERConfigurations(unittest.TestCase):
entities = extractor.extract_entities(self.text)
self.assertFalse(extractor._ml_runtime_usable)
# methods.spacy.load called once during __init__ (the RuntimeError); not again
# during extract_entities because _filter_unusable_methods removes "ml".
self.assertEqual(mock_methods_spacy.load.call_count, 1)
self.assertEqual(mock_init_spacy.load.call_count, 1)
self.assertEqual(mock_methods_spacy.load.call_count, 0)
self.assertEqual(mock_ml_method.call_count, 0)
self.assertIsInstance(entities, list)
-54
View File
@@ -209,60 +209,6 @@ def test_load_from_api_allows_private_when_configured(mock_guard, seed_manager):
call_kwargs = mock_guard.call_args[1]
assert call_kwargs["allow_private_ips"] is True
@patch("semantica.seed.seed_manager.request_with_ssrf_guard")
def test_load_from_api_does_not_mutate_caller_headers_dict(mock_guard, seed_manager):
"""Regression test for issue #947 audit: load_from_api must not mutate the
caller's headers dict in-place when api_key is provided.
Before the fix, ``request_headers = headers or {}`` aliased the caller's dict.
Writing ``request_headers["Authorization"] = ...`` then silently modified the
caller's original dict, potentially leaking credentials to subsequent calls
that reused the same headers dict without expecting it to carry Authorization.
"""
mock_response = MagicMock()
mock_response.json.return_value = {"results": []}
mock_guard.return_value = mock_response
# Caller owns this dict and expects it to be unchanged after the call.
original_headers = {"X-Custom-Header": "value"}
headers_before = dict(original_headers) # snapshot
seed_manager.load_from_api(
api_url="http://api.example.com",
api_key="secret-key",
headers=original_headers,
)
# The caller's dict must be unchanged — Authorization must NOT have been added.
assert original_headers == headers_before, (
"load_from_api must not mutate the caller's headers dict; "
f"expected {headers_before!r}, got {original_headers!r}"
)
# The guard must still have received Authorization (in its own copy).
call_kwargs = mock_guard.call_args[1]
guard_headers = call_kwargs.get("headers", {})
assert guard_headers.get("Authorization") == "Bearer secret-key"
@patch("semantica.seed.seed_manager.request_with_ssrf_guard")
def test_load_from_api_does_not_mutate_empty_headers_dict(mock_guard, seed_manager):
"""When headers=None, a fresh dict is created — no aliasing to a shared mutable default."""
mock_response = MagicMock()
mock_response.json.return_value = {"results": []}
mock_guard.return_value = mock_response
seed_manager.load_from_api(
api_url="http://api.example.com",
api_key="key",
headers=None,
)
call_kwargs = mock_guard.call_args[1]
guard_headers = call_kwargs.get("headers", {})
assert guard_headers.get("Authorization") == "Bearer key"
def test_load_source(seed_manager, temp_data_dir):
json_file = temp_data_dir / "source.json"
with open(json_file, "w") as f:
-15
View File
@@ -31,21 +31,6 @@ class TestHelpers(unittest.TestCase):
dict2 = {"b": {"d": 3}, "e": 4}
merged = helpers.merge_dicts(dict1, dict2, deep=True)
self.assertEqual(merged, {"a": 1, "b": {"c": 2, "d": 3}, "e": 4})
def test_flatten_dict(self):
data = {"a": {"b": 1, "c": 2}}
result = helpers.flatten_dict(data)
self.assertEqual(result, {"a.b": 1, "a.c": 2})
def test_flatten_dict_key_collision(self):
data = {
"a.b": 1,
"a": {
"b": 2
}
}
with self.assertRaises(ValueError):
helpers.flatten_dict(data)
def test_safe_import_returns_module_and_flag(self):
module, available = helpers.safe_import("json")