mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
50927f99b5 | ||
|
|
476237952d |
@@ -11,15 +11,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
|
||||
- **Semantica RDF vocabulary, and deterministic entity/relationship IRIs** (#1109, closes #1107, closes #1101) by @fabio-rovai, reviewed by @KaifAhmad1
|
||||
- Every RDF/JSON-LD export mints terms in `https://semantica.dev/ns#`, and until now nothing declared what those terms meant — the namespace 404s and no vocabulary shipped with the package, so a consumer receiving an export had no way to tell `sem:text` from a typo of it, and no closed-world checker could validate an export at all
|
||||
- `semantica/ontology/vocabulary/semantica-ns.ttl` declares the terms the exporters actually emit — drawn from the emitting call sites in `export/rdf_exporter.py`, `export/json_exporter.py` and `provenance/manager.py`, not from what a vocabulary "ought" to contain. Ships inside the package (`from semantica.ontology.vocabulary import vocabulary_turtle`) so it loads without a network round trip, and is the same document intended to be served at the namespace IRI once hosting/content-negotiation is sorted
|
||||
- `tests/ontology/test_vocabulary.py` ties the document to the code: every term a serializer can write must be declared, so adding a term to an exporter without declaring it fails the build
|
||||
- The missing-id fallback minted entity/relationship IRIs from Python's builtin `hash()`, randomised per process (`PYTHONHASHSEED`), so the same entity got a different IRI on every run and exports couldn't be diffed, deduplicated, or joined to an earlier provenance record. It also wrote `<semantica:entity_N>`, an IRI in the scheme `semantica` rather than the expansion of the declared prefix, so those nodes never joined with anything written through it. Minting now uses SHA-256 and writes a full IRI in the declared namespace; the same fix applies to the default entity/relationship types in the Turtle path
|
||||
- **Fixed during review** (Qodo): the temporal fallback minted from `source_id` only, while the main serializer accepts `source_id` or `source` — relationships using the second form hashed two empty strings, which the previous randomised `hash()` masked by making the IRI unstable anyway; once deterministic, unrelated relationships at the same list index collided on one IRI across exports. Endpoints are now resolved the same way `serialize_to_turtle` resolves them, before minting. `sem:confidence` also lost its declared `xsd:decimal` range: the N-Triples serializer types the same value `xsd:float`, and the two are disjoint, so declaring either contradicted one of the exporters (tracked in #1100) — a new `test_declared_ranges_do_not_contradict_what_the_exporters_emit` guards the whole class of that mistake
|
||||
- **Fixed in follow-up**: `serialize_to_rdfxml`'s default entity type still wrote the bare string `"semantica:Entity"` into an `rdf:resource` attribute, which (unlike a Turtle angle-bracket or an XML element name) is not namespace-expanded — the exact #1101 failure mode, just on the untested RDF/XML path. `json_exporter.py`'s `semantica:format` and `@type: "semantica:KnowledgeGraph"` were emitted but absent from both the vocabulary and the test's `EMITTED_TERMS` guard set, so the "undeclared terms fail the build" claim didn't actually cover them — both are now declared and guarded. `MANIFEST.in` didn't mirror the `pyproject.toml` package-data addition, so a source-distribution install could omit the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only `PATH`, breaking it on Windows; now overrides only `PYTHONHASHSEED` on top of the inherited environment
|
||||
- 229 export and ontology tests pass
|
||||
|
||||
- **First-class CrewAI integration** (#962)
|
||||
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
|
||||
- `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()`
|
||||
@@ -82,14 +73,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`split`/chunking paths bypassed the centralized spaCy model cache, reloading the model on every call** (#1042, closes #998) by @Accute9, reviewed by @Sameer6305
|
||||
- `semantica/split/methods.py`'s `split_by_sentences()` and `semantica/split/semantic_chunker.py`'s `SemanticChunker.__init__` each called `spacy.load()` directly instead of reusing the process-level cache added in #889/`semantic_extract/methods.py`'s `load_spacy_model()` — every call/construction re-paid the ~120ms model-load cost independently of `NERExtractor`, which already used the cache
|
||||
- Both now route through `load_spacy_model()`, sharing one cached `Language` instance per model name across `split_by_sentences()`, `SemanticChunker`, and `NERExtractor`; a missing model still falls back to regex/paragraph chunking without poisoning the cache for a later successful load
|
||||
- **Fixed during review** (@Sameer6305): `NERExtractor.__init__()` still had a direct `spacy.load()` call site with the same cache-bypass issue, outside the two files named in #998 but sharing the same root cause; routed through the cache alongside stale test patch targets and a strengthened cache-configuration assertion
|
||||
- **Fixed during review** (@KaifAhmad1): `SemanticChunker.__init__` only caught `OSError` around `load_spacy_model()`, while the sibling fix to `NERExtractor` in this same PR added a broader `except Exception` for a model that is installed but fails at runtime (e.g. a config incompatible with the installed spaCy version). A broken-but-present model crashed `SemanticChunker()` outright instead of degrading to fallback chunking like every other path in this PR. Added the matching `except Exception` branch, leaving `self.nlp` as `None`; new `test_semantic_chunker_falls_back_when_spacy_runtime_is_broken` mirrors the existing `NERExtractor` regression test for the same scenario
|
||||
- New `tests/split/test_spacy_model_cache.py`: cache reuse across repeated calls/instances, shared cache between `split_by_sentences()`/`SemanticChunker`/`NERExtractor`, distinct model names loading separately, missing-model fallback without poisoning the cache, and the broken-runtime fallback added above
|
||||
- `pytest tests/split/test_spacy_model_cache.py tests/split/test_splitter.py tests/split/test_chunkers.py`: all passing (3 pre-existing, unrelated `tests/test_ner_configurations.py` failures confirmed present on `main` before this PR)
|
||||
|
||||
- **`export_yaml` raised a raw `AttributeError` on list input, silently wrote empty exports for unrecognized dict keys, and graph payloads were reconciled differently by every exporter** (#958, closes #956, #952, #953) by @pravit-amp, reviewed by @Sameer6305
|
||||
- Graph payloads circulate under two vocabularies, `entities`/`relationships` and `nodes`/`edges`, and each exporter reconciled them locally with a different idiom — `LPGExporter` in particular dropped every entity whenever `nodes` was present but empty, the exact shape `JSONExporter` emits. A new `normalize_graph_payload()` in `utils/helpers.py` centralizes that decision once, adopted by `LPGExporter`, `ArangoAQLExporter`, `Neo4jCSVExporter`, and both YAML exporters; `ContextGraph.to_dict()` now round-trips through YAML correctly as a result
|
||||
- `export_yaml(records, path)` on a bare list previously failed with `AttributeError` from inside the exporter; it and the other YAML methods now reject non-mapping input with an actionable `ProcessingError` naming the expected keys, since these formats distinguish entities/relationships/triplets and guessing which one a list represents would mislabel the records
|
||||
@@ -185,31 +168,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Security
|
||||
|
||||
- **Tarball restore path traversal, latent SQL injection, DNS-rebinding TOCTOU in the shared SSRF guard, stored XSS in report generation, and unvalidated SPARQL object IRIs in AnzoStore** (#1079) by @KaifAhmad1
|
||||
- `semantica backup restore`'s tar extraction (`cli.py`) stripped only the literal `semantica-backup/` prefix and called `tar.extract()` with no path-containment check, no symlink/hardlink validation, and (on Python <3.12) no extraction filter — a crafted archive member (`../../<file>`, or a symlink pointing outside the restore root) could write arbitrary files above the restore directory. Every member is now validated for resolved-path containment before extraction, symlink/hardlink targets are rejected both lexically (absolute path, `..` segments) and by resolution, and `filter="data"` is applied on Python ≥3.12
|
||||
- `DataExporter.export_table_data()` (`db_ingestor.py`) was missing the `text` import from `sqlalchemy` — a `NameError` that made the method non-functional, but latently: the query it built from raw f-string interpolation of `table_name`/`schema`/`where`/`order_by` was already injectable, so fixing the import alone (without also fixing the injection) would have silently armed it. Both are fixed together: the import is restored, `table_name`/`schema` are now validated against a strict identifier allowlist, and `where`/`order_by` are checked against a blocklist (statement separators, comments, UNION, DDL/DML keywords, time-based blind-injection primitives, schema-enumeration terms). This is a blocklist, not a grammar — it closes the concrete UNION-exfiltration path and common injection primitives, but a boolean-blind subquery using none of the blocked keywords could still get through; `where`/`order_by` must be treated as trusted/operator input, not exposed to untrusted end users, and the docstrings now say so explicitly
|
||||
- `request_with_ssrf_guard()` (`ssrf.py`) validated a hostname's resolved IPs, then let the underlying HTTP client re-resolve the same hostname independently at connect time — a low-TTL or DNS-rebinding answer could differ between the two lookups, so a hostname that validated as public could still connect to a private/internal address. Ported the IP-pinning pattern already used by `explorer/routes/ontology.py`'s `_make_pinned_session` into the shared ingest guard: the one resolution that decides accept/reject is now also the one the connection is pinned to, via a custom `HTTPAdapter` that presents the real hostname over TLS SNI / Host header while connecting only to the validated IPs. Also closes the RFC 6598 Carrier-Grade NAT gap noted as a known limitation in #905/#868: `100.64.0.0/10` is now in `BLOCKED_NETWORKS`
|
||||
- `ReportGenerator._generate_html()` (`export/report_generator.py`) f-string-interpolated report title/summary/metrics into HTML with no escaping — an ingested entity or document whose content flowed into a report (e.g. `<img src=x onerror=...>`) executed as stored XSS when the report was opened. All interpolated values are now `html.escape()`d
|
||||
- `AnzoStore._format_object_for_sparql()` (`triplet_store/anzo_store.py`) validated the subject/predicate of a triplet via `sparql_escaping.validate_uri()` before interpolating them into a SPARQL `INSERT DATA` clause, but delegated the **object** position to a separate formatter that wrapped it as `<{obj}>` without the same validation — an object value containing `>`/`}`/`{`/`"` could close the intended `<...>` token early and inject additional SPARQL Update operations. The Blazegraph/RDF4J backends were hardened for the equivalent gap previously; Anzo's object position now goes through the same `validate_uri()` check
|
||||
- Also hardened in the same pass: Apache AGE's `create_index()` `index_type` parameter is now allowlisted (was interpolated raw into a `USING` clause); Neo4j's `limit` is now explicitly validated (raises `ValidationError` for non-integer input instead of falling through to a generic `ProcessingError`); the `ffprobe` metadata-extraction subprocess call is guarded against a filename starting with `-` being parsed as an option; the MCP server no longer echoes raw exception text to JSON-RPC clients, logging full details server-side and returning a generic message plus the exception class name instead
|
||||
- **Fixed during review** (@KaifAhmad1): the SSRF IP-pinning change introduced a connection-pool leak of its own — `requests.Session.mount()` silently drops whatever adapter it replaces without closing it, so a multi-hop redirect chain on a reused session leaked one pooled connection per hop. Pinned adapters are now tagged and explicitly closed before being replaced, both per-hop and on final restore
|
||||
- **Fixed during review** (@KaifAhmad1): mounting a pinned adapter and setting a Host header on a caller-supplied `Session` is not inherently thread-safe — two guarded calls sharing the same session from different threads could interleave their mount/restore cycles. Added a per-session lock (`_get_session_lock`) so concurrent guarded calls on the same session now serialize instead of racing; verified with a two-thread test showing correct serialization and zero cross-contamination of per-request Host headers
|
||||
- **Fixed during automated PR review** (Qodo): `export_table_data()`'s new identifier/fragment validation raised `ValidationError` from inside a `try` whose blanket `except Exception` re-wrapped it as `ProcessingError`, masking the distinction between "bad input" and "the export itself failed" that callers rely on elsewhere in this module. Added the `except ValidationError: raise` guard already used by its sibling methods
|
||||
- **Fixed during automated PR review** (Qodo): on a hop where IP pinning doesn't apply (`allow_private_ips=True`), `_apply_connection_pin()` unconditionally popped the session's `Host` header instead of restoring whatever it was before pinning touched it — a caller-supplied session carrying its own legitimate `Host` override (e.g. fronting a private endpoint under a different name) had that override silently dropped for the in-flight request, only reappearing afterward via the outer `finally` restore. It now restores the session's own pre-call header state (set back if present, popped only if it was truly absent) instead of always popping
|
||||
- **Fixed during automated PR review** (Qodo): the `where`/`order_by` blocklist matched keywords/punctuation inside properly quoted string literals and identifiers too, so legitimate data like `status = 'union'` or `name = 'a--b'` was rejected as if it were SQL syntax. The blocklist now runs against a copy with quoted-literal contents masked out (`_mask_sql_literals`) — a malformed/unterminated quote sequence doesn't match the masking pattern and is left fully exposed to the blocklist, so this closes false positives without opening a masking-based bypass; the fragment actually used in the query is unchanged
|
||||
- Re-ran each finding's proof-of-concept (or an equivalent adversarial test) against the fix and confirmed it is blocked: tar path/symlink traversal (both lexical and resolved-path forms), SQL UNION exfiltration and identifier breakout, DNS-rebinding TOCTOU (including under a configured `HTTP_PROXY`, which the pinning adapter also rejects outright since a proxy would resolve DNS itself), stored XSS, and the AnzoStore SPARQL injection
|
||||
- `pytest tests/ingest/`: 266 passed, 2 skipped (10 pre-existing failures unrelated to this change — identical failure set confirmed on unmodified `main`); full regression sweep across `graph_store`, `export`, `triplet_store`, `parse`, and backup/restore: 313 passed
|
||||
|
||||
- **`Authorization`/`Proxy-Authorization` credentials could leak to a different origin across HTTP redirects, and several ingest paths bypassed the shared SSRF/redirect guard entirely** (#1067, closes #947) by @Sameer6305, reviewed by @KaifAhmad1
|
||||
- `request_with_ssrf_guard()` previously only stripped sensitive headers from per-request `kwargs["headers"]` on a cross-origin redirect; session-level `Authorization`/`Proxy-Authorization` headers, `session.auth`, and `session.trust_env` (`.netrc` lookup) could all still resurrect credentials on the hop to a foreign origin. All five credential sources are now stripped case-insensitively, kept stripped for the remainder of a multi-hop redirect chain (no resurrection even if a later hop returns to the original host), and unconditionally restored via `finally` — including on exceptions and redirect-limit errors
|
||||
- `MCPClient._send_request_http()` and `PublicAPIIngestor.detect_public_api()`/`ingest_public_api()` called `httpx.post()`/`requests.post()`/`session.request()` directly, bypassing `request_with_ssrf_guard()` entirely. Both now route through the shared guard, including when `validate_no_auth=False`
|
||||
- `SeedDataManager.load_from_api()` mutated the caller-supplied `headers` dict in place when adding an API-key `Authorization` header, silently leaking the key back into a dict the caller might reuse elsewhere. Now copies before modifying
|
||||
- **Fixed during review** (@KaifAhmad1): `allow_private_ips=True` (used to let MCP servers run on localhost/internal networks) was applied to every redirect hop, not just the operator-configured host — a compromised or malicious MCP server could 302-redirect to an internal address (e.g. `169.254.169.254` cloud metadata) and the guard would follow it unchecked, defeating the SSRF protection this PR otherwise adds. Added `allow_private_ips_on_redirect` to `request_with_ssrf_guard()`: a redirect target inherits the original host's private-IP trust only when it matches that host; any other host falls back to strict validation. `MCPClient` now pins `allow_private_ips_on_redirect=False`, so only same-host redirects on a trusted MCP server keep working — a cross-host hop into private address space is blocked
|
||||
- **Fixed during review** (@KaifAhmad1): `detect_public_api()` only caught `requests.exceptions.RequestException`, but `request_with_ssrf_guard()` raises `ValidationError` (a disjoint hierarchy) for SSRF-blocked hosts, blocked redirect targets, missing `Location`, or exceeded redirect limits — unlike its sibling `ingest_public_api()`, which already caught it. Callers (including `is_public_api()`) got an undocumented raw `ValidationError` instead of `ProcessingError`, and the error-logging call was skipped. Now catches `(ValidationError, ProcessingError)` and re-raises, matching the sibling method
|
||||
- **Fixed during review** (@KaifAhmad1): `detect_public_api()`/`ingest_public_api()` forwarded `**options` into `request_with_ssrf_guard(..., session=self.session, allow_private_ips=self.allow_private_ips, **request_options)` without stripping `session`/`allow_private_ips` from `request_options` first — a caller passing either through the per-call `**options` (a plausible mistake, since `allow_private_ips` is also a documented constructor-level knob) got a raw `TypeError: got multiple values for keyword argument`. Both are now popped from `request_options` before the call
|
||||
- New regression coverage added during review: `TestAllowPrivateIpsOnRedirect` (cross-host redirect into private space blocked, same-host redirect trust preserved, default behavior unchanged for existing callers that don't pass the new kwarg) and `TestMCPClientAuthRedirect::test_redirect_to_private_ip_is_blocked`/`test_same_host_redirect_on_private_mcp_server_is_not_blocked` in `tests/ingest/test_auth_header_redirect_security.py`; `test_detect_public_api_propagates_ssrf_validation_error` and duplicate-kwarg regression tests for both methods in `tests/ingest/test_public_api_ingestor.py`
|
||||
- `pytest tests/ingest/test_auth_header_redirect_security.py tests/ingest/test_public_api_ingestor.py tests/test_seed_manager.py tests/ingest/test_submodules.py tests/ingest/test_cookbook_integration.py`: 111 passed
|
||||
|
||||
- **`FeedIngestor`/`FeedMonitor` (RSS/Atom feed ingestion) had no SSRF protection, allowing requests to internal/private network targets** (#928, closes #927) by @ZohaibHassan16
|
||||
- `FeedIngestor.ingest_feed()`, `discover_feeds()` (link-tag fetch, common-path HEAD probe, and feed-validation GET), and `FeedMonitor.check_updates()` all called `requests.get()`/`requests.head()` directly with default redirect-following and no scheme allowlist or private/loopback/link-local IP validation — despite `semantica/ingest/ssrf.py`'s `request_with_ssrf_guard()` already existing and being used by `web_ingestor.py`/`api_ingestor.py`. `ingest_feed()`'s own URL check only verified `urlparse(url).scheme`/`.netloc` were non-empty, never that the scheme was http/https or that the resolved target IP was safe. Reachable via the public `ingest_feed()`/`ingest()` entry points with any caller-supplied feed URL
|
||||
- All 5 call sites now route through `request_with_ssrf_guard()`, which validates scheme (http/https only) and resolved IP before the request, and re-validates every redirect `Location` before following it — closing both the direct-IP and redirect-chain SSRF paths. Added an `allow_private_ips` config option to both `FeedIngestor` and `FeedMonitor`, consistent with the other ingestors
|
||||
|
||||
@@ -1,2 +1 @@
|
||||
recursive-include semantica/static *
|
||||
recursive-include semantica/ontology/vocabulary *.ttl
|
||||
|
||||
@@ -2,15 +2,7 @@
|
||||
|
||||
<img src="Semantica Logo.png" alt="Semantica" width="420"/>
|
||||
|
||||
<div style="display:flex; gap:10px; align-items:center; flex-wrap:wrap;">
|
||||
<a href="https://trendshift.io/repositories/18986?utm_source=repository-badge&utm_medium=badge&utm_campaign=badge-repository-18986" target="_blank" rel="noopener noreferrer">
|
||||
<img src="https://trendshift.io/api/badge/repositories/18986" alt="semantica-agi/semantica | Trendshift" width="250" height="55"/>
|
||||
</a>
|
||||
|
||||
<a href="https://trendshift.io/repositories/18986?utm_source=trendshift-badge&utm_medium=badge&utm_campaign=badge-trendshift-18986" target="_blank" rel="noopener noreferrer">
|
||||
<img src="https://trendshift.io/api/badge/trendshift/repositories/18986/weekly?language=Python" alt="semantica-agi/semantica | Trendshift" width="250" height="55"/>
|
||||
</a>
|
||||
</div>
|
||||
<a href="https://trendshift.io/repositories/18986?utm_source=repository-badge&utm_medium=badge&utm_campaign=badge-repository-18986" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/18986" alt="semantica-agi%2Fsemantica | Trendshift" width="250" height="55"/></a>
|
||||
|
||||
### Graph-Native Infrastructure for Context and Accountable AI Systems
|
||||
|
||||
@@ -303,10 +295,17 @@ graph.add_causal_relationship(d1, d2, relationship_type="CAUSED")
|
||||
prov.track_entity("patient_P4821", source="ehr/medication_orders_2024.json",
|
||||
metadata={"extractor": "NamedEntityRecognizer"})
|
||||
|
||||
# Export W3C PROV-O for regulator submission - to_kg_dict() is the official
|
||||
# adapter that emits the {"entities": [...], "relationships": [...]} /
|
||||
# source_id shape RDFExporter expects, so no manual field mapping is needed
|
||||
kg = graph.to_kg_dict()
|
||||
# Export W3C PROV-O for regulator submission - RDFExporter expects
|
||||
# {"entities": [...], "relationships": [...]}, so map ContextGraph.to_dict()'s
|
||||
# {"nodes": [...], "edges": [...]} shape onto it first
|
||||
graph_dict = graph.to_dict()
|
||||
kg = {
|
||||
"entities": [{"id": n["id"], "type": n["type"], "text": n["content"]} for n in graph_dict["nodes"]],
|
||||
"relationships": [
|
||||
{"source_id": e["source"], "target_id": e["target"], "type": e["type"]}
|
||||
for e in graph_dict["edges"]
|
||||
],
|
||||
}
|
||||
RDFExporter().export(kg, "audit_trail.ttl", format="turtle")
|
||||
```
|
||||
|
||||
@@ -880,14 +879,20 @@ fact = BiTemporalFact(
|
||||
recorded_at=datetime(2024, 3, 5),
|
||||
)
|
||||
|
||||
# Query facts valid within a time window - to_kg_dict() is the official
|
||||
# adapter that emits {"entities", "relationships"} with source_id/target_id
|
||||
# keys, the shape query_time_range() expects (no manual mapping required)
|
||||
kg = graph.to_kg_dict()
|
||||
# Query facts valid within a time window - query_time_range() expects
|
||||
# {"relationships": [...]} with source_id/target_id keys, which differs from
|
||||
# ContextGraph.to_dict()'s {"nodes", "edges"} shape, so map it first
|
||||
graph_dict = graph.to_dict()
|
||||
kg_relationships = {
|
||||
"relationships": [
|
||||
{**e, "source_id": e["source"], "target_id": e["target"]}
|
||||
for e in graph_dict["edges"]
|
||||
]
|
||||
}
|
||||
|
||||
tq = TemporalGraphQuery()
|
||||
facts_in_window = tq.query_time_range(
|
||||
kg, query="valid_facts", start_time="2024-01-01", end_time="2024-12-31"
|
||||
kg_relationships, query="valid_facts", start_time="2024-01-01", end_time="2024-12-31"
|
||||
)
|
||||
|
||||
# Normalize natural language temporal expressions - returns a (start, end) range
|
||||
|
||||
@@ -162,7 +162,7 @@ semantica-explorer --graph my_graph.json --no-browser
|
||||
```
|
||||
|
||||
<Warning>
|
||||
`--host 0.0.0.0` makes Explorer reachable on every network interface. Since v0.6.5 the Explorer API requires `SEMANTICA_API_KEY` (sent as the `X-API-Key` header) and fails closed with `503` when unconfigured; unauthenticated access is only possible when `SEMANTICA_ALLOW_ANONYMOUS=true` is set explicitly. Only use this on a trusted private network.
|
||||
`--host 0.0.0.0` makes Explorer reachable on every network interface. The server has no built-in authentication. Only use this on a trusted private network.
|
||||
</Warning>
|
||||
|
||||
|
||||
|
||||
+1
-10
@@ -63,9 +63,7 @@ semantica-explorer --graph my_graph.json --no-browser
|
||||
python -m semantica.explorer --graph my_graph.json
|
||||
```
|
||||
|
||||
> **Security note:** Since v0.6.5 the Explorer API requires an API key on protected routes. Set the `SEMANTICA_API_KEY` environment variable and send it as the `X-API-Key` header; without a configured key, protected routes fail closed with `503` rather than serving anonymously. To opt into unauthenticated access for local development only, set `SEMANTICA_ALLOW_ANONYMOUS=true` explicitly. (`/api/health` and `/api/info` are intentionally unauthenticated.)
|
||||
>
|
||||
> The default `--host 127.0.0.1` binds to localhost only, so it is not reachable from other machines on your network. If you bind to `0.0.0.0`, all graph data is readable and writable by any host that can reach the port (subject to API-key auth). The CLI prints a warning when binding to a non-loopback host in anonymous mode or when `SEMANTICA_API_KEY` is unset.
|
||||
> **Security note:** The Explorer API has no built-in authentication. The default `--host 127.0.0.1` binds to localhost only, so it is not reachable from other machines on your network. If you bind to `0.0.0.0`, all graph data is readable and writable by any host that can reach the port. The CLI will print a warning in that case.
|
||||
|
||||
---
|
||||
|
||||
@@ -150,8 +148,6 @@ This writes the compiled assets to `../semantica/static/`. The Python server the
|
||||
| --- | --- | --- |
|
||||
| `EXPLORER_CORS_ORIGINS` | `http://localhost:5173,http://127.0.0.1:5173` | Comma-separated list of allowed CORS origins |
|
||||
| `EXPLORER_CORS_CREDENTIALS` | `false` | Set to `true` to allow credentialed cross-origin requests (only needed behind an authenticating reverse proxy) |
|
||||
| `SEMANTICA_API_KEY` | *(unset)* | API key required on protected routes since v0.6.5; send it as the `X-API-Key` header. When unset, protected routes fail closed with `503`. |
|
||||
| `SEMANTICA_ALLOW_ANONYMOUS` | `false` | Set to `true` to opt into unauthenticated access (local development only). |
|
||||
|
||||
---
|
||||
|
||||
@@ -255,11 +251,6 @@ Vite automatically tries the next available port and prints the actual URL in th
|
||||
- Confirm the backend exposes the `/ws/graph-updates` WebSocket endpoint.
|
||||
- Check DevTools → Network → WS tab for the connection status and error code.
|
||||
- Ensure the backend version matches the frontend — mixing major versions can cause protocol mismatches.
|
||||
- **Authentication:** `/ws/graph-updates` enforces the same API key as the REST routes. Browsers cannot set custom headers on a WebSocket handshake, so pass the key as a query parameter instead:
|
||||
```
|
||||
ws://127.0.0.1:8000/ws/graph-updates?api_key=<your-key>
|
||||
```
|
||||
Non-browser clients (native apps, scripts) may send it as the `X-API-Key` header. A missing or incorrect key results in close code `4401`; if `SEMANTICA_API_KEY` is unset and `SEMANTICA_ALLOW_ANONYMOUS` is not `true`, the connection is also rejected. Note that API keys in URLs appear in server logs — prefer the header for non-browser clients.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -162,11 +162,7 @@ const SIGMA_SETTINGS = {
|
||||
hideLabelsOnMove: true,
|
||||
hideEdgesOnMove: true,
|
||||
enableEdgeEvents: true,
|
||||
// #1009: edge labels (the edge `type` — "works_for", "leads", ...) were
|
||||
// hardcoded off, so edge text never rendered regardless of data. The
|
||||
// labelDensity / labelGridCellSize / labelRenderedSizeThreshold settings
|
||||
// below already throttle label density for both nodes and edges.
|
||||
renderEdgeLabels: true,
|
||||
renderEdgeLabels: false,
|
||||
labelDensity: 0.7,
|
||||
labelGridCellSize: 140,
|
||||
zIndex: true,
|
||||
@@ -745,12 +741,6 @@ function buildEffectAvailability(
|
||||
? { enabled: true, available: true, reason: "Panel enabled" }
|
||||
: { enabled: false, available: false, reason: "Disabled by toggle" };
|
||||
|
||||
// #1009: edge labels are immediately available once the graph is loaded —
|
||||
// they have no async analytics or zoom-tier dependency.
|
||||
const edgeLabels = effectsState.edgeLabelsEnabled
|
||||
? { enabled: true, available: true, reason: "Ready" }
|
||||
: { enabled: false, available: false, reason: "Disabled by toggle" };
|
||||
|
||||
const diagnostics = !GRAPH_THEME.effects.diagnostics.enabledInDev
|
||||
? { enabled: false, available: false, reason: "Disabled in production" }
|
||||
: effectsState.diagnosticsEnabled
|
||||
@@ -768,7 +758,6 @@ function buildEffectAvailability(
|
||||
communities,
|
||||
centrality,
|
||||
legend,
|
||||
edgeLabels,
|
||||
diagnostics,
|
||||
};
|
||||
}
|
||||
@@ -1222,12 +1211,6 @@ function applySceneState(
|
||||
size: resolvedStyle.size,
|
||||
zIndex: resolvedStyle.zIndex,
|
||||
curvature: resolvedStyle.curvature,
|
||||
// #1009: Sigma's edge label renderer draws data.label — the graph
|
||||
// stores the relationship type in edgeType, which the renderer never
|
||||
// saw, so enabling renderEdgeLabels alone left edges blank.
|
||||
// Use || rather than ?? so that an empty-string edgeType (possible
|
||||
// when the API returns type: "") does not produce a blank label.
|
||||
label: resolvedStyle.hidden ? undefined : String(attrs.edgeType || data.label || ""),
|
||||
};
|
||||
});
|
||||
|
||||
@@ -1312,9 +1295,6 @@ export const GraphCanvas = forwardRef<GraphCanvasHandle, GraphCanvasProps>(
|
||||
const onEdgeClickRef = useRef(onEdgeClick);
|
||||
const onSceneRuntimeChangeRef = useRef(onSceneRuntimeChange);
|
||||
const onCameraStateChangeRef = useRef(onCameraStateChange);
|
||||
// #1009: tracked as a ref so the Sigma creation effect always reads the
|
||||
// current value without needing effectsState in its dependency array.
|
||||
const effectsStateRef = useRef(effectsState);
|
||||
const [hoveredNodeId, setHoveredNodeId] = useState<string | null>(null);
|
||||
const [zoomTier, setZoomTier] = useState<GraphZoomTier>("overview");
|
||||
const [analyticsSnapshot, setAnalyticsSnapshot] = useState<GraphAnalyticsSnapshot | null>(null);
|
||||
@@ -1343,7 +1323,6 @@ export const GraphCanvas = forwardRef<GraphCanvasHandle, GraphCanvasProps>(
|
||||
onEdgeClickRef.current = onEdgeClick;
|
||||
onSceneRuntimeChangeRef.current = onSceneRuntimeChange;
|
||||
onCameraStateChangeRef.current = onCameraStateChange;
|
||||
effectsStateRef.current = effectsState;
|
||||
|
||||
const behaviors = useMemo<GraphBehavior[]>(
|
||||
() => [
|
||||
@@ -1856,13 +1835,7 @@ export const GraphCanvas = forwardRef<GraphCanvasHandle, GraphCanvasProps>(
|
||||
return;
|
||||
}
|
||||
|
||||
const sigma = new Sigma(displayGraphRef.current, containerRef.current, {
|
||||
...SIGMA_SETTINGS,
|
||||
// #1009: initialize with the current toggle value rather than the
|
||||
// static default so that a user who disabled Edge Labels before
|
||||
// graph/Sigma initialization sees the correct state after mount.
|
||||
renderEdgeLabels: effectsStateRef.current.edgeLabelsEnabled,
|
||||
});
|
||||
const sigma = new Sigma(displayGraphRef.current, containerRef.current, SIGMA_SETTINGS);
|
||||
sigmaRef.current = sigma;
|
||||
appliedGraphVersionRef.current = graphVersionRef.current;
|
||||
|
||||
@@ -1964,17 +1937,6 @@ export const GraphCanvas = forwardRef<GraphCanvasHandle, GraphCanvasProps>(
|
||||
});
|
||||
}, [behaviors, dispatchToBehaviors, getBehaviorContext, graphReady, syncCameraState]);
|
||||
|
||||
// #1009: renderEdgeLabels follows the Effects-panel toggle instead of
|
||||
// staying hardcoded — dense graphs get their label-free edges back.
|
||||
useEffect(() => {
|
||||
const sigma = sigmaRef.current;
|
||||
if (!sigma) {
|
||||
return;
|
||||
}
|
||||
sigma.setSetting("renderEdgeLabels", effectsState.edgeLabelsEnabled);
|
||||
sigma.scheduleRefresh();
|
||||
}, [effectsState.edgeLabelsEnabled]);
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
const sigma = sigmaRef.current;
|
||||
|
||||
@@ -148,7 +148,6 @@ const DEFAULT_EFFECTS_STATE: GraphEffectsState = {
|
||||
communitiesEnabled: false,
|
||||
centralityEnabled: false,
|
||||
legendEnabled: false,
|
||||
edgeLabelsEnabled: true,
|
||||
diagnosticsEnabled: false,
|
||||
lensMode: "neighborhood",
|
||||
effectQuality: "bounded",
|
||||
|
||||
@@ -2099,15 +2099,6 @@ function createCollapsedNeighborhoodGraph(
|
||||
return collapsedGraph;
|
||||
}
|
||||
|
||||
// Normalize an edge relationship type: empty string, null, and undefined all
|
||||
// fall back to the project-wide default used consistently across every
|
||||
// aggregation path. Keep this local — it exists only to guarantee that the
|
||||
// three code paths (single-entry, multi-entry, community-grouped) produce the
|
||||
// same semantics and do not diverge again.
|
||||
function normalizeEdgeType(value: string | null | undefined): string {
|
||||
return value || "related_to";
|
||||
}
|
||||
|
||||
function aggregateDisplayGraph(graphRef: GraphRef): Graph<NodeAttributes, EdgeAttributes> {
|
||||
const aggregated = new Graph<NodeAttributes, EdgeAttributes>({
|
||||
type: "directed",
|
||||
@@ -2133,13 +2124,10 @@ function aggregateDisplayGraph(graphRef: GraphRef): Graph<NodeAttributes, EdgeAt
|
||||
const [{ edgeId, attrs }] = entries;
|
||||
aggregated.mergeDirectedEdgeWithKey(edgeId, sourceId, targetId, {
|
||||
...attrs,
|
||||
// #1009: normalize empty/null/undefined edgeType so Sigma's label
|
||||
// renderer never receives a blank string on the single-entry path.
|
||||
edgeType: normalizeEdgeType(attrs.edgeType),
|
||||
dominantEdgeType: normalizeEdgeType(attrs.dominantEdgeType ?? attrs.edgeType),
|
||||
rawEdgeIds: collectRawEdgeIds(attrs, edgeId),
|
||||
isAggregated: isAggregatedEdgeAttributes(attrs),
|
||||
aggregateCount: attrs.aggregateCount ?? collectRawEdgeIds(attrs, edgeId).length,
|
||||
dominantEdgeType: attrs.dominantEdgeType ?? attrs.edgeType,
|
||||
representativeWeight: attrs.representativeWeight ?? Number(attrs.weight ?? 1),
|
||||
});
|
||||
return;
|
||||
@@ -2162,11 +2150,10 @@ function aggregateDisplayGraph(graphRef: GraphRef): Graph<NodeAttributes, EdgeAt
|
||||
const rawEdgeIds = entries.flatMap(({ edgeId, attrs }) => collectRawEdgeIds(attrs, edgeId));
|
||||
const typeCounts = new Map<string, number>();
|
||||
entries.forEach(({ attrs }) => {
|
||||
const edgeType = normalizeEdgeType(attrs.edgeType);
|
||||
const edgeType = String(attrs.edgeType ?? "related_to");
|
||||
typeCounts.set(edgeType, (typeCounts.get(edgeType) ?? 0) + 1);
|
||||
});
|
||||
const dominantEdgeType = [...typeCounts.entries()].sort((left, right) => right[1] - left[1])[0]?.[0]
|
||||
?? normalizeEdgeType(representative.attrs.edgeType);
|
||||
const dominantEdgeType = [...typeCounts.entries()].sort((left, right) => right[1] - left[1])[0]?.[0] ?? representative.attrs.edgeType ?? "related_to";
|
||||
const reverseKey = `${targetId}→${sourceId}`;
|
||||
const isBidirectionalBundle = groupedEdges.has(reverseKey);
|
||||
const syntheticEdgeId = `${AGGREGATED_EDGE_PREFIX}${sourceId}::${targetId}`;
|
||||
@@ -2180,10 +2167,10 @@ function aggregateDisplayGraph(graphRef: GraphRef): Graph<NodeAttributes, EdgeAt
|
||||
rawEdgeIds,
|
||||
isAggregated: true,
|
||||
aggregateCount: rawEdgeIds.length,
|
||||
dominantEdgeType: dominantEdgeType,
|
||||
dominantEdgeType: String(dominantEdgeType),
|
||||
representativeWeight: Number(representative.attrs.weight ?? 1),
|
||||
weight: Number(representative.attrs.weight ?? 1),
|
||||
edgeType: representative.attrs.edgeType || dominantEdgeType,
|
||||
edgeType: String(representative.attrs.edgeType ?? dominantEdgeType ?? "related_to"),
|
||||
parallelCount: rawEdgeIds.length,
|
||||
familySize: rawEdgeIds.length,
|
||||
bundleKind: isBidirectionalBundle ? "bidirectional" : "parallel",
|
||||
@@ -2293,7 +2280,7 @@ function buildCommunityGroupedGraph(): GraphDisplayResult {
|
||||
};
|
||||
bucket.rawEdgeIds.push(String(edgeId));
|
||||
bucket.weight = Math.max(bucket.weight, Number((attrs as EdgeAttributes).weight ?? 1));
|
||||
const edgeType = normalizeEdgeType((attrs as EdgeAttributes).edgeType);
|
||||
const edgeType = String((attrs as EdgeAttributes).edgeType ?? "related_to");
|
||||
bucket.typeCounts.set(edgeType, (bucket.typeCounts.get(edgeType) ?? 0) + 1);
|
||||
groupedEdges.set(key, bucket);
|
||||
});
|
||||
@@ -2409,8 +2396,7 @@ function buildCommunityGroupedGraph(): GraphDisplayResult {
|
||||
if (!visibleGroupedEdgeKeys.has(key)) {
|
||||
return;
|
||||
}
|
||||
const dominantEdgeType = [...bundle.typeCounts.entries()].sort((left, right) => right[1] - left[1])[0]?.[0]
|
||||
?? "related_to";
|
||||
const dominantEdgeType = [...bundle.typeCounts.entries()].sort((left, right) => right[1] - left[1])[0]?.[0] ?? "related_to";
|
||||
const reverseKey = `${bundle.targetId}→${bundle.sourceId}`;
|
||||
const syntheticEdgeId = `${AGGREGATED_EDGE_PREFIX}${key}`;
|
||||
const aggregateCount = bundle.rawEdgeIds.length;
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
import type { CSSProperties } from "react";
|
||||
|
||||
import type {
|
||||
GraphDiagnosticsSnapshot,
|
||||
GraphEffectAvailability,
|
||||
GraphEffectToggle,
|
||||
} from "../types";
|
||||
@@ -31,11 +30,6 @@ const EFFECT_ROWS: EffectRowConfig[] = [
|
||||
label: "Neighborhood Lens",
|
||||
description: "Local emphasis around the hovered or selected node.",
|
||||
},
|
||||
{
|
||||
key: "edgeLabelsEnabled",
|
||||
label: "Edge Labels",
|
||||
description: "Draw the relationship type on graph edges. Off restores label-free edges on dense graphs.",
|
||||
},
|
||||
{
|
||||
key: "legendEnabled",
|
||||
label: "Semantic Legend",
|
||||
@@ -43,17 +37,6 @@ const EFFECT_ROWS: EffectRowConfig[] = [
|
||||
},
|
||||
];
|
||||
|
||||
// Maps the effect toggle keys rendered by this plugin to their corresponding
|
||||
// availability keys in GraphDiagnosticsSnapshot["effectAvailability"]. Kept
|
||||
// local because this plugin only renders a subset of all effects.
|
||||
const EFFECT_AVAILABILITY_KEYS: Partial<Record<GraphEffectToggle, keyof GraphDiagnosticsSnapshot["effectAvailability"]>> = {
|
||||
pathPulseEnabled: "pathPulse",
|
||||
pathFlowEnabled: "pathFlow",
|
||||
lensEnabled: "lens",
|
||||
edgeLabelsEnabled: "edgeLabels",
|
||||
legendEnabled: "legend",
|
||||
};
|
||||
|
||||
function renderAvailabilityText(availability: GraphEffectAvailability) {
|
||||
if (availability.available) {
|
||||
if (typeof availability.visibleSegments === "number" && typeof availability.segmentCap === "number") {
|
||||
@@ -156,9 +139,15 @@ export const explorationEffectsPlugin: GraphPlugin = {
|
||||
description={row.description}
|
||||
checked={effectsState[row.key]}
|
||||
availability={
|
||||
(EFFECT_AVAILABILITY_KEYS[row.key] !== undefined
|
||||
? availability?.[EFFECT_AVAILABILITY_KEYS[row.key]!]
|
||||
: undefined) ?? {
|
||||
availability?.[
|
||||
row.key === "pathPulseEnabled"
|
||||
? "pathPulse"
|
||||
: row.key === "pathFlowEnabled"
|
||||
? "pathFlow"
|
||||
: row.key === "lensEnabled"
|
||||
? "lens"
|
||||
: "legend"
|
||||
] ?? {
|
||||
enabled: effectsState[row.key],
|
||||
available: false,
|
||||
reason: "Waiting for graph runtime",
|
||||
|
||||
@@ -47,11 +47,6 @@ const SCENE_EFFECT_ROWS: EffectRowConfig[] = [
|
||||
label: "Contours",
|
||||
description: "Low-contrast density halos around the strongest visible anchors.",
|
||||
},
|
||||
{
|
||||
key: "edgeLabelsEnabled",
|
||||
label: "Edge Labels",
|
||||
description: "Draw the relationship type on graph edges. Off restores label-free edges on dense graphs.",
|
||||
},
|
||||
{
|
||||
key: "legendEnabled",
|
||||
label: "Regions Summary",
|
||||
@@ -88,7 +83,6 @@ const AVAILABILITY_KEYS: Record<GraphEffectToggle, keyof GraphDiagnosticsSnapsho
|
||||
communitiesEnabled: "communities",
|
||||
centralityEnabled: "centrality",
|
||||
legendEnabled: "legend",
|
||||
edgeLabelsEnabled: "edgeLabels",
|
||||
diagnosticsEnabled: "diagnostics",
|
||||
};
|
||||
|
||||
|
||||
@@ -103,7 +103,6 @@ export type GraphEffectToggle =
|
||||
| "communitiesEnabled"
|
||||
| "centralityEnabled"
|
||||
| "legendEnabled"
|
||||
| "edgeLabelsEnabled"
|
||||
| "diagnosticsEnabled";
|
||||
|
||||
export interface GraphEffectsState {
|
||||
@@ -114,7 +113,6 @@ export interface GraphEffectsState {
|
||||
semanticRegionsEnabled: boolean;
|
||||
contoursEnabled: boolean;
|
||||
pathfindingEnabled: boolean;
|
||||
edgeLabelsEnabled: boolean;
|
||||
communitiesEnabled: boolean;
|
||||
centralityEnabled: boolean;
|
||||
legendEnabled: boolean;
|
||||
@@ -188,7 +186,6 @@ export interface GraphDiagnosticsSnapshot {
|
||||
communities: GraphEffectAvailability;
|
||||
centrality: GraphEffectAvailability;
|
||||
legend: GraphEffectAvailability;
|
||||
edgeLabels: GraphEffectAvailability;
|
||||
diagnostics: GraphEffectAvailability;
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1061,198 +1061,3 @@ test("checkGroupedViewAvailability returns available when communities exist", ()
|
||||
assert.equal(result.reason, null);
|
||||
});
|
||||
|
||||
|
||||
// ── #1009: edge label data-path regression tests ─────────────────────────────
|
||||
|
||||
test("resolveDisplayGraph parallel-bundle preserves edgeType on aggregated edge", () => {
|
||||
addNode("a");
|
||||
addNode("b");
|
||||
batchMergeEdges([
|
||||
{ id: "e1", source: "a", target: "b", attributes: { edgeType: "causes", weight: 1, properties: {} } },
|
||||
{ id: "e2", source: "a", target: "b", attributes: { edgeType: "causes", weight: 2, properties: {} } },
|
||||
]);
|
||||
|
||||
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
|
||||
assert.equal(displayGraph.size, 1);
|
||||
|
||||
const edgeId = displayGraph.edges()[0];
|
||||
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string; isAggregated?: boolean };
|
||||
assert.equal(attrs.isAggregated, true);
|
||||
// The aggregated representative must carry the relationship text through to
|
||||
// the edgeReducer's label assignment.
|
||||
assert.equal(typeof attrs.edgeType, "string");
|
||||
assert.ok((attrs.edgeType ?? "").length > 0, "aggregated edge must have a non-empty edgeType");
|
||||
});
|
||||
|
||||
test("resolveDisplayGraph parallel-bundle picks dominant edgeType across mixed types", () => {
|
||||
addNode("a");
|
||||
addNode("b");
|
||||
batchMergeEdges([
|
||||
{ id: "e1", source: "a", target: "b", attributes: { edgeType: "inhibits", weight: 1, properties: {} } },
|
||||
{ id: "e2", source: "a", target: "b", attributes: { edgeType: "inhibits", weight: 1, properties: {} } },
|
||||
{ id: "e3", source: "a", target: "b", attributes: { edgeType: "activates", weight: 1, properties: {} } },
|
||||
]);
|
||||
|
||||
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
|
||||
const edgeId = displayGraph.edges()[0];
|
||||
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string; dominantEdgeType?: string };
|
||||
// "inhibits" appears twice so it must be the dominant type.
|
||||
assert.equal(attrs.edgeType, "inhibits");
|
||||
assert.equal(attrs.dominantEdgeType, "inhibits");
|
||||
});
|
||||
|
||||
test("resolveDisplayGraph grouped view community edges carry non-empty edgeType", () => {
|
||||
const left = ["g1", "g2", "g3", "g4"];
|
||||
const right = ["h1", "h2", "h3", "h4"];
|
||||
[...left, ...right].forEach((nodeId, index) => addNode(nodeId, index < left.length ? "left" : "right"));
|
||||
|
||||
let edgeIndex = 0;
|
||||
for (let i = 0; i < left.length; i += 1) {
|
||||
for (let j = 0; j < left.length; j += 1) {
|
||||
if (i !== j) {
|
||||
batchMergeEdges([{
|
||||
id: `lg-${edgeIndex++}`,
|
||||
source: left[i],
|
||||
target: left[j],
|
||||
attributes: { edgeType: "co_occurs", weight: 3, properties: {} },
|
||||
}]);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (let i = 0; i < right.length; i += 1) {
|
||||
for (let j = 0; j < right.length; j += 1) {
|
||||
if (i !== j) {
|
||||
batchMergeEdges([{
|
||||
id: `rg-${edgeIndex++}`,
|
||||
source: right[i],
|
||||
target: right[j],
|
||||
attributes: { edgeType: "co_occurs", weight: 3, properties: {} },
|
||||
}]);
|
||||
}
|
||||
}
|
||||
}
|
||||
batchMergeEdges([{ id: "bridge-g", source: "g1", target: "h1", attributes: { edgeType: "interacts_with", weight: 0.1, properties: {} } }]);
|
||||
|
||||
const { graph: displayGraph, state } = resolveDisplayGraph("", [], [], "grouped", { aggregationEnabled: true });
|
||||
assert.equal(state.groupedViewAvailable, true);
|
||||
|
||||
const communityEdges = displayGraph.edges().filter((edgeId) => {
|
||||
const attrs = displayGraph.getEdgeAttributes(edgeId) as { bundleKind?: string };
|
||||
return attrs.bundleKind === "community";
|
||||
});
|
||||
assert.ok(communityEdges.length > 0, "expected at least one community bundle edge");
|
||||
|
||||
for (const edgeId of communityEdges) {
|
||||
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string };
|
||||
assert.equal(typeof attrs.edgeType, "string");
|
||||
assert.ok((attrs.edgeType ?? "").length > 0, `community edge ${edgeId} must have a non-empty edgeType`);
|
||||
}
|
||||
});
|
||||
|
||||
test("resolveDisplayGraph raw edge preserves exact edgeType string for label rendering", () => {
|
||||
addNode("src");
|
||||
addNode("tgt");
|
||||
batchMergeEdges([{
|
||||
id: "raw-1",
|
||||
source: "src",
|
||||
target: "tgt",
|
||||
attributes: { edgeType: "works_for", weight: 1, properties: {} },
|
||||
}]);
|
||||
|
||||
// In full view without aggregation the edge passes through unchanged.
|
||||
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: false });
|
||||
assert.equal(displayGraph.size, 1);
|
||||
|
||||
const edgeId = displayGraph.edges()[0];
|
||||
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string };
|
||||
assert.equal(attrs.edgeType, "works_for");
|
||||
});
|
||||
|
||||
test("resolveDisplayGraph does not produce empty-string edgeType on aggregated edges when source has empty type", () => {
|
||||
addNode("a");
|
||||
addNode("b");
|
||||
// Simulate an API response where type is empty string — the aggregation
|
||||
// path must not propagate a blank label.
|
||||
batchMergeEdges([
|
||||
{ id: "e-empty-1", source: "a", target: "b", attributes: { edgeType: "", weight: 1, properties: {} } },
|
||||
{ id: "e-empty-2", source: "a", target: "b", attributes: { edgeType: "", weight: 1, properties: {} } },
|
||||
]);
|
||||
|
||||
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
|
||||
const edgeId = displayGraph.edges()[0];
|
||||
const attrs = displayGraph.getEdgeAttributes(edgeId) as {
|
||||
edgeType?: string;
|
||||
isAggregated?: boolean;
|
||||
};
|
||||
assert.equal(attrs.isAggregated, true);
|
||||
// The aggregation falls back to "related_to" when all source edgeTypes are
|
||||
// empty, so the rendered label should never be an empty string.
|
||||
assert.equal(attrs.edgeType, "related_to");
|
||||
});
|
||||
|
||||
test("resolveEdgeElementStyle hidden class produces hidden:true for suppressed edges", () => {
|
||||
// Verify the data condition the edgeReducer relies on: hidden-classified
|
||||
// edges must have hidden:true so that the label assignment sets undefined.
|
||||
const style = resolveEdgeElementStyle(
|
||||
GRAPH_THEME,
|
||||
"overview",
|
||||
"inactive",
|
||||
{
|
||||
edgeType: "causes",
|
||||
weight: 1,
|
||||
properties: {},
|
||||
edgeVariant: "line",
|
||||
visualPriority: 0.05,
|
||||
baseSize: 0.3,
|
||||
},
|
||||
"source",
|
||||
"target",
|
||||
"full",
|
||||
"inactive-edge",
|
||||
"hidden",
|
||||
);
|
||||
assert.equal(style.hidden, true);
|
||||
});
|
||||
|
||||
// ── #1009 maintainer-blocking regression: single-edge empty edgeType ─────────
|
||||
|
||||
test("resolveDisplayGraph single-edge normalizes empty-string edgeType to related_to", () => {
|
||||
addNode("a");
|
||||
addNode("b");
|
||||
// One edge only — exercises the entries.length === 1 path in aggregateDisplayGraph.
|
||||
batchMergeEdges([{
|
||||
id: "e-single-empty",
|
||||
source: "a",
|
||||
target: "b",
|
||||
attributes: { edgeType: "", weight: 1, properties: {} },
|
||||
}]);
|
||||
|
||||
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
|
||||
assert.equal(displayGraph.size, 1);
|
||||
|
||||
const edgeId = displayGraph.edges()[0];
|
||||
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string; dominantEdgeType?: string };
|
||||
assert.equal(attrs.edgeType, "related_to",
|
||||
"single-edge path must normalize empty edgeType to the canonical fallback");
|
||||
assert.equal(attrs.dominantEdgeType, "related_to",
|
||||
"single-edge dominantEdgeType must also be normalized");
|
||||
});
|
||||
|
||||
test("resolveDisplayGraph single-edge preserves a valid non-empty edgeType unchanged", () => {
|
||||
addNode("a");
|
||||
addNode("b");
|
||||
batchMergeEdges([{
|
||||
id: "e-single-valid",
|
||||
source: "a",
|
||||
target: "b",
|
||||
attributes: { edgeType: "works_for", weight: 1, properties: {} },
|
||||
}]);
|
||||
|
||||
const { graph: displayGraph } = resolveDisplayGraph("", [], [], "full", { aggregationEnabled: true });
|
||||
assert.equal(displayGraph.size, 1);
|
||||
|
||||
const edgeId = displayGraph.edges()[0];
|
||||
const attrs = displayGraph.getEdgeAttributes(edgeId) as { edgeType?: string };
|
||||
assert.equal(attrs.edgeType, "works_for",
|
||||
"single-edge path must not alter a valid relationship type");
|
||||
});
|
||||
|
||||
+2
-12
@@ -93,14 +93,7 @@ def _handle_tools_call(req_id: Any, params: dict) -> dict:
|
||||
result = tool["_handler"](args)
|
||||
except Exception as exc:
|
||||
log.exception("Tool %s raised an exception", name)
|
||||
# The exception's class name (e.g. "ValidationError", "TimeoutError")
|
||||
# is safe to surface — unlike str(exc), it never carries paths,
|
||||
# connection strings, or other internal detail — and lets the
|
||||
# client distinguish failure kinds without a full message.
|
||||
return _err(
|
||||
req_id, _INTERNAL_ERROR,
|
||||
f"Tool '{name}' failed ({type(exc).__name__}). See server logs for details.",
|
||||
)
|
||||
return _err(req_id, _INTERNAL_ERROR, str(exc))
|
||||
|
||||
# MCP spec: content must be a list of content items
|
||||
return _ok(req_id, {
|
||||
@@ -178,10 +171,7 @@ class SemanticaMCPServer:
|
||||
log.exception("Unhandled error in method %s", method)
|
||||
if req_id is None:
|
||||
return None
|
||||
return _err(
|
||||
req_id, _INTERNAL_ERROR,
|
||||
f"Method '{method}' failed ({type(exc).__name__}). See server logs for details.",
|
||||
)
|
||||
return _err(req_id, _INTERNAL_ERROR, str(exc))
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
def run(self) -> None:
|
||||
|
||||
+1
-1
@@ -271,7 +271,7 @@ include = ["semantica*", "integrations*"]
|
||||
[tool.setuptools.package-data]
|
||||
# Explicit patterns are more reliable than **/* across setuptools versions.
|
||||
# static/* covers index.html / favicon; static/assets/* covers all JS/CSS chunks.
|
||||
"semantica" = ["static/*", "static/assets/*", "ontology/vocabulary/*.ttl"]
|
||||
"semantica" = ["static/*", "static/assets/*"]
|
||||
|
||||
[tool.black]
|
||||
line-length = 88
|
||||
|
||||
+5
-58
@@ -20,7 +20,7 @@ if sys.platform == "win32":
|
||||
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
from dataclasses import asdict, dataclass, field, is_dataclass
|
||||
from pathlib import Path, PurePosixPath, PureWindowsPath
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Sequence, Tuple
|
||||
|
||||
import yaml
|
||||
@@ -3933,68 +3933,15 @@ def backup_restore(cli_ctx: CLIContext, source: str, local_dry: bool) -> None:
|
||||
|
||||
try:
|
||||
if _tf.is_tarfile(str(work_path)):
|
||||
restore_root = Path.cwd().resolve()
|
||||
restore_root = Path.cwd()
|
||||
with _tf.open(str(work_path), "r:*") as tar:
|
||||
# Dry-run listing was already handled above; extract now
|
||||
for member in tar.getmembers():
|
||||
# Strip the leading "semantica-backup/" prefix
|
||||
member.name = member.name.replace("semantica-backup/", "", 1)
|
||||
if not member.name:
|
||||
continue
|
||||
|
||||
# Reject members whose resolved path escapes the
|
||||
# restore root (path traversal / absolute paths),
|
||||
# regardless of the "semantica-backup/" prefix.
|
||||
member_path = (restore_root / member.name).resolve()
|
||||
try:
|
||||
member_path.relative_to(restore_root)
|
||||
except ValueError:
|
||||
raise click.ClickException(
|
||||
f"Refusing to restore '{member.name}': "
|
||||
"path escapes the restore directory."
|
||||
)
|
||||
|
||||
# Reject symlink/hardlink members whose target
|
||||
# escapes the restore root. Checked two ways:
|
||||
# lexically (linkname itself, so an absolute path or
|
||||
# a literal ".." segment is rejected outright, with
|
||||
# no dependence on what else does or doesn't already
|
||||
# exist on disk) and by resolution (catches any
|
||||
# remaining traversal the lexical check misses).
|
||||
if member.issym() or member.islnk():
|
||||
linkname = member.linkname or ""
|
||||
linkname_parts = PurePosixPath(
|
||||
linkname.replace("\\", "/")
|
||||
).parts
|
||||
if (
|
||||
not linkname
|
||||
or os.path.isabs(linkname)
|
||||
or PureWindowsPath(linkname).is_absolute()
|
||||
or ".." in linkname_parts
|
||||
):
|
||||
raise click.ClickException(
|
||||
f"Refusing to restore '{member.name}': "
|
||||
"link target is absolute or traverses "
|
||||
"out of the archive."
|
||||
)
|
||||
link_target = (
|
||||
member_path.parent / linkname
|
||||
).resolve()
|
||||
try:
|
||||
link_target.relative_to(restore_root)
|
||||
except ValueError:
|
||||
raise click.ClickException(
|
||||
f"Refusing to restore '{member.name}': "
|
||||
"link target escapes the restore directory."
|
||||
)
|
||||
|
||||
extract_kwargs: Dict[str, Any] = {"path": str(restore_root)}
|
||||
if hasattr(_tf, "data_filter"):
|
||||
# Python >=3.12: also reject device files, and
|
||||
# further harden the traversal/ownership checks.
|
||||
extract_kwargs["filter"] = "data"
|
||||
tar.extract(member, **extract_kwargs)
|
||||
console.print(f" restored: {member.name}")
|
||||
if member.name:
|
||||
tar.extract(member, path=str(restore_root))
|
||||
console.print(f" restored: {member.name}")
|
||||
elif src.is_dir():
|
||||
restore_root = Path.cwd()
|
||||
for f in src.rglob("*"):
|
||||
|
||||
@@ -72,9 +72,9 @@ Example Usage:
|
||||
... node_embeddings=True)
|
||||
>>>
|
||||
>>> # Basic graph operations
|
||||
>>> graph.add_node("Python", "language", popularity="high")
|
||||
>>> graph.add_node("Programming", "concept")
|
||||
>>> graph.add_edge("Python", "Programming", "related_to")
|
||||
>>> graph.add_node("Python", type="language", properties={"popularity": "high"})
|
||||
>>> graph.add_node("Programming", type="concept")
|
||||
>>> graph.add_edge("Python", "Programming", type="related_to")
|
||||
>>> centrality = graph.get_node_centrality("Python")
|
||||
>>> similar = graph.find_similar_nodes("Python", similarity_type="content")
|
||||
>>> analysis = graph.analyze_graph_with_kg()
|
||||
@@ -88,7 +88,7 @@ Example Usage:
|
||||
... confidence=0.95,
|
||||
... entities=["customer_123", "property_456"]
|
||||
... )
|
||||
>>> precedents = graph.find_precedents(decision_id, limit=5)
|
||||
>>> precedents = graph.find_precedents("loan_approval", limit=5)
|
||||
>>> influence = graph.analyze_decision_influence(decision_id)
|
||||
>>> insights = graph.get_decision_insights()
|
||||
>>> causality = graph.trace_decision_causality(decision_id)
|
||||
@@ -2449,97 +2449,6 @@ class ContextGraph:
|
||||
},
|
||||
}
|
||||
|
||||
def to_kg_dict(self, entities_only: bool = False) -> Dict[str, Any]:
|
||||
"""Export graph in the canonical knowledge-graph shape.
|
||||
|
||||
This is the official adapter that converts the ContextGraph's internal
|
||||
``{"nodes", "edges"}`` / ``source`` representation into the
|
||||
``{"entities", "relationships"}`` / ``source_id`` shape expected by
|
||||
downstream consumers such as
|
||||
:class:`~semantica.export.rdf_exporter.RDFExporter` and
|
||||
:meth:`~semantica.kg.temporal_query.TemporalGraphQuery.query_time_range`.
|
||||
|
||||
Users no longer need to hand-map field names between APIs.
|
||||
|
||||
Args:
|
||||
entities_only: If True, only nodes whose ``node_type`` is
|
||||
``"entity"`` are exported as entities. When False (default),
|
||||
every node is exported. Relationships whose endpoints are not
|
||||
in the exported entity set are dropped to avoid dangling
|
||||
references in downstream consumers.
|
||||
|
||||
Returns:
|
||||
dict: A knowledge-graph dictionary with:
|
||||
- ``entities``: list of ``{"id", "text", "type", "properties",
|
||||
"metadata"}`` (plus ``valid_from`` / ``valid_until`` when set)
|
||||
- ``relationships``: list of ``{"source_id", "target_id",
|
||||
"type", "weight", "id", "familyId"}`` (plus ``metadata`` and
|
||||
``valid_from`` / ``valid_until`` when set)
|
||||
- ``statistics``: ``{"entity_count", "relationship_count"}``
|
||||
"""
|
||||
with self._lock:
|
||||
entities_out = []
|
||||
for n in self.nodes.values():
|
||||
if entities_only and n.node_type != "entity":
|
||||
continue
|
||||
# Normalize the entity id to ``str`` so it matches ContextEdge,
|
||||
# which coerces its endpoints to ``str`` in ``__post_init__``.
|
||||
# Without this, non-string node ids (e.g. numeric ids loaded via
|
||||
# ``from_dict``) would fail the ``valid_ids`` membership check
|
||||
# below and silently drop otherwise-valid relationships.
|
||||
entity_id = str(n.node_id)
|
||||
entity: Dict[str, Any] = {
|
||||
"id": entity_id,
|
||||
"text": n.content,
|
||||
"type": n.node_type,
|
||||
# ``properties`` / ``metadata`` may be ``None`` when a node
|
||||
# was loaded from JSON containing an explicit ``null``;
|
||||
# guard with ``or {}`` so ``dict(...)`` never raises.
|
||||
"properties": dict(n.properties or {}),
|
||||
"metadata": dict(n.metadata or {}),
|
||||
}
|
||||
if n.valid_from is not None:
|
||||
entity["valid_from"] = n.valid_from
|
||||
if n.valid_until is not None:
|
||||
entity["valid_until"] = n.valid_until
|
||||
entities_out.append(entity)
|
||||
|
||||
# When only entity nodes are exported, drop relationships whose
|
||||
# endpoints were filtered out so downstream consumers never see a
|
||||
# source_id/target_id that is absent from ``entities``.
|
||||
valid_ids = {e["id"] for e in entities_out} if entities_only else None
|
||||
|
||||
relationships_out = []
|
||||
for e in self.edges:
|
||||
if valid_ids is not None and (
|
||||
e.source_id not in valid_ids or e.target_id not in valid_ids
|
||||
):
|
||||
continue
|
||||
rel: Dict[str, Any] = {
|
||||
"id": e.edge_id,
|
||||
"familyId": e.family_id or e.edge_id,
|
||||
"source_id": e.source_id,
|
||||
"target_id": e.target_id,
|
||||
"type": e.edge_type,
|
||||
"weight": e.weight,
|
||||
}
|
||||
if e.metadata:
|
||||
rel["metadata"] = dict(e.metadata)
|
||||
if e.valid_from is not None:
|
||||
rel["valid_from"] = e.valid_from
|
||||
if e.valid_until is not None:
|
||||
rel["valid_until"] = e.valid_until
|
||||
relationships_out.append(rel)
|
||||
|
||||
return {
|
||||
"entities": entities_out,
|
||||
"relationships": relationships_out,
|
||||
"statistics": {
|
||||
"entity_count": len(entities_out),
|
||||
"relationship_count": len(relationships_out),
|
||||
},
|
||||
}
|
||||
|
||||
def from_dict(self, graph_dict: Dict[str, Any]) -> None:
|
||||
"""Load graph from dictionary format."""
|
||||
# Clear existing graph
|
||||
|
||||
@@ -45,7 +45,6 @@ License: MIT
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
from ..utils.entity_ids import get_entity_id
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
@@ -505,7 +504,7 @@ class EntityMerger:
|
||||
# Record source entities
|
||||
provenance["merged_from"] = [
|
||||
{
|
||||
"id": get_entity_id(e),
|
||||
"id": self._get_entity_value(e, "id"),
|
||||
"name": self._get_entity_value(e, "name"),
|
||||
"source": self._get_entity_value(e, "metadata", {}).get("source") if hasattr(e, "metadata") or isinstance(e, dict) else None,
|
||||
}
|
||||
|
||||
@@ -45,7 +45,6 @@ from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Any, Callable, Dict, List, Optional, Tuple, Union
|
||||
|
||||
from ..utils.entity_ids import get_entity_id
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
@@ -318,20 +317,14 @@ class MergeStrategyManager:
|
||||
message=f"Building merged entity... ({current_step}/{total_steps}, remaining: {remaining_steps} steps)"
|
||||
)
|
||||
# Build merged entity
|
||||
merged_from = []
|
||||
for entity in entities:
|
||||
entity_id = get_entity_id(entity)
|
||||
if entity_id is not None:
|
||||
merged_from.append(entity_id)
|
||||
|
||||
merged_entity = {
|
||||
"id": get_entity_id(base_entity),
|
||||
"id": base_entity.get("id"),
|
||||
"name": self._merge_top_level_field("name", entities, base_entity),
|
||||
"type": self._merge_top_level_field("type", entities, base_entity),
|
||||
"properties": merged_properties,
|
||||
"relationships": merged_relationships,
|
||||
"metadata": self._merge_metadata(entities, base_entity),
|
||||
"merged_from": merged_from,
|
||||
"merged_from": [e.get("id") for e in entities if e.get("id")],
|
||||
"merge_strategy": strategy.value,
|
||||
}
|
||||
|
||||
|
||||
@@ -33,42 +33,11 @@ from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Set, Union
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.helpers import ensure_directory, hash_data
|
||||
from ..utils.helpers import ensure_directory
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
|
||||
|
||||
SEMANTICA_NS = "https://semantica.dev/ns#"
|
||||
|
||||
#: Written when an entity carries no type of its own. A full IRI rather than the
|
||||
#: prefixed form, because the Turtle serializer writes it inside angle brackets,
|
||||
#: where `semantica:Entity` would be read as an IRI in the scheme `semantica`
|
||||
#: rather than as the prefix expansion (issue #1101).
|
||||
DEFAULT_ENTITY_TYPE = f"{SEMANTICA_NS}Entity"
|
||||
|
||||
#: Written when a relationship carries no type of its own. Same reasoning.
|
||||
DEFAULT_RELATION_TYPE = f"{SEMANTICA_NS}related_to"
|
||||
|
||||
|
||||
def mint_entity_iri(text: str) -> str:
|
||||
"""Mint a stable IRI for an entity that arrived without an id.
|
||||
|
||||
Python's builtin ``hash()`` is randomised per process (PYTHONHASHSEED), so
|
||||
minting from it gave the same entity a different IRI on every run: exports
|
||||
could not be diffed, deduplicated against an earlier load, or joined to a
|
||||
provenance record written by an earlier process. SHA-256 is stable across
|
||||
runs and machines, which is what an identifier has to be.
|
||||
"""
|
||||
digest = hash_data(str(text))[:16]
|
||||
return f"{SEMANTICA_NS}entity_{digest}"
|
||||
|
||||
|
||||
def mint_relationship_iri(index: int, source: Any, target: Any) -> str:
|
||||
"""Mint a stable IRI for a relationship that arrived without an id."""
|
||||
digest = hash_data(f"{source}\x00{target}")[:16]
|
||||
return f"{SEMANTICA_NS}rel_{index}_{digest}"
|
||||
|
||||
|
||||
class NamespaceManager:
|
||||
"""
|
||||
RDF namespace management engine.
|
||||
@@ -391,9 +360,9 @@ class RDFSerializer:
|
||||
entity_id = entity.get("id")
|
||||
if not entity_id:
|
||||
entity_text = entity.get("text", "")
|
||||
entity_id = mint_entity_iri(entity_text)
|
||||
entity_id = f"semantica:entity_{hash(entity_text)}"
|
||||
|
||||
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
|
||||
entity_type = entity.get("type", "semantica:Entity")
|
||||
text = entity.get("text") or entity.get("label", "")
|
||||
confidence = entity.get("confidence", 1.0)
|
||||
|
||||
@@ -407,7 +376,7 @@ class RDFSerializer:
|
||||
for idx, rel in enumerate(relationships):
|
||||
source_id = rel.get("source_id") or rel.get("source")
|
||||
target_id = rel.get("target_id") or rel.get("target")
|
||||
rel_type = rel.get("type", DEFAULT_RELATION_TYPE)
|
||||
rel_type = rel.get("type", "semantica:related_to")
|
||||
|
||||
lines.append(f"<{source_id}> <{rel_type}> <{target_id}> .")
|
||||
|
||||
@@ -444,14 +413,10 @@ class RDFSerializer:
|
||||
if time_axis in ("transaction", "both"):
|
||||
axes.append(("tx", rel.get("recorded_at"), rel.get("superseded_at")))
|
||||
|
||||
# Resolve endpoints the same way serialize_to_turtle does: both
|
||||
# representations are accepted upstream, and minting from source_id
|
||||
# alone hashes empty strings for every relationship that uses source,
|
||||
# so unrelated relationships at the same index would collide on a
|
||||
# deterministic IRI.
|
||||
source_id = rel.get("source_id") or rel.get("source") or ""
|
||||
target_id = rel.get("target_id") or rel.get("target") or ""
|
||||
rel_base_id = rel.get("id") or mint_relationship_iri(idx, source_id, target_id)
|
||||
rel_base_id = (
|
||||
rel.get("id")
|
||||
or f"semantica:rel_{idx}_{hash(str(rel.get('source_id', '')) + str(rel.get('target_id', '')))}"
|
||||
)
|
||||
|
||||
lines = [""] # blank separator
|
||||
for axis_name, from_val, until_val in axes:
|
||||
@@ -523,9 +488,9 @@ class RDFSerializer:
|
||||
entity_id = entity.get("id")
|
||||
if not entity_id:
|
||||
entity_text = entity.get("text", "")
|
||||
entity_id = mint_entity_iri(entity_text)
|
||||
entity_id = f"semantica:entity_{hash(entity_text)}"
|
||||
|
||||
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
|
||||
entity_type = entity.get("type", "semantica:Entity")
|
||||
text = entity.get("text") or entity.get("label", "")
|
||||
confidence = entity.get("confidence", 1.0)
|
||||
|
||||
@@ -679,7 +644,7 @@ class RDFSerializer:
|
||||
entity_id = entity.get("id")
|
||||
if not entity_id:
|
||||
entity_text = entity.get("text", "")
|
||||
entity_id = mint_entity_iri(entity_text)
|
||||
entity_id = f"semantica:entity_{hash(entity_text)}"
|
||||
|
||||
subject = expand_uri(entity_id)
|
||||
|
||||
|
||||
@@ -23,7 +23,6 @@ Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
import html
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
@@ -363,7 +362,7 @@ class ReportGenerator:
|
||||
' <meta name="viewport" content="width=device-width, initial-scale=1.0">'
|
||||
)
|
||||
title = data.get("title", "Report")
|
||||
lines.append(f" <title>{html.escape(str(title))}</title>")
|
||||
lines.append(f" <title>{title}</title>")
|
||||
lines.append(" <style>")
|
||||
lines.append(" body { font-family: Arial, sans-serif; margin: 20px; }")
|
||||
lines.append(" h1 { color: #333; }")
|
||||
@@ -381,14 +380,11 @@ class ReportGenerator:
|
||||
|
||||
# Title
|
||||
title = data.get("title", "Report")
|
||||
lines.append(f" <h1>{html.escape(str(title))}</h1>")
|
||||
lines.append(f" <h1>{title}</h1>")
|
||||
|
||||
# Generated at
|
||||
if "generated_at" in data:
|
||||
lines.append(
|
||||
f' <p><strong>Generated:</strong> '
|
||||
f'{html.escape(str(data["generated_at"]))}</p>'
|
||||
)
|
||||
lines.append(f' <p><strong>Generated:</strong> {data["generated_at"]}</p>')
|
||||
|
||||
# Summary
|
||||
if "summary" in data:
|
||||
@@ -397,13 +393,10 @@ class ReportGenerator:
|
||||
if isinstance(summary, dict):
|
||||
lines.append(" <ul>")
|
||||
for key, value in summary.items():
|
||||
lines.append(
|
||||
f" <li><strong>{html.escape(str(key))}:</strong> "
|
||||
f"{html.escape(str(value))}</li>"
|
||||
)
|
||||
lines.append(f" <li><strong>{key}:</strong> {value}</li>")
|
||||
lines.append(" </ul>")
|
||||
else:
|
||||
lines.append(f" <p>{html.escape(str(summary))}</p>")
|
||||
lines.append(f" <p>{summary}</p>")
|
||||
|
||||
# Metrics
|
||||
if "metrics" in data:
|
||||
@@ -490,10 +483,7 @@ class ReportGenerator:
|
||||
else:
|
||||
value_str = str(value)
|
||||
|
||||
lines.append(
|
||||
f" <tr><td>{html.escape(str(key))}</td>"
|
||||
f"<td>{html.escape(value_str)}</td></tr>"
|
||||
)
|
||||
lines.append(f" <tr><td>{key}</td><td>{value_str}</td></tr>")
|
||||
|
||||
lines.append(" </table>")
|
||||
|
||||
|
||||
@@ -66,12 +66,6 @@ except (ImportError, OSError):
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# create_index's index_type reaches a raw SQL keyword position (`USING
|
||||
# {index_type}`) that can't be bound as a query parameter; only the
|
||||
# documented, PostgreSQL-recognized types are allowed through.
|
||||
_ALLOWED_INDEX_TYPES = frozenset({"btree", "gin", "hash", "gist", "brin"})
|
||||
|
||||
|
||||
def _sanitize_label(label: str) -> str:
|
||||
"""
|
||||
Sanitize a Cypher label to prevent injection.
|
||||
@@ -1220,11 +1214,6 @@ class ApacheAgeStore:
|
||||
safe_label = _sanitize_label(label)
|
||||
if not re.match(r"^[A-Za-z_][A-Za-z0-9_]*$", property_name):
|
||||
raise ValidationError(f"Invalid property name: '{property_name}'")
|
||||
if index_type not in _ALLOWED_INDEX_TYPES:
|
||||
raise ValidationError(
|
||||
f"Invalid index_type: {index_type!r}. "
|
||||
f"Allowed: {sorted(_ALLOWED_INDEX_TYPES)}"
|
||||
)
|
||||
|
||||
index_name = options.get(
|
||||
"index_name", f"idx_{self.graph_name}_{safe_label}_{property_name}"
|
||||
|
||||
@@ -503,16 +503,6 @@ class Neo4jStore:
|
||||
Returns:
|
||||
List of matching nodes
|
||||
"""
|
||||
# LIMIT can't be bound as a query parameter in a way Neo4j accepts
|
||||
# here, so it's interpolated directly; validate explicitly rather
|
||||
# than trust the `limit: int` type hint, which Python doesn't
|
||||
# enforce at runtime. Done outside the try/except below so a bad
|
||||
# limit raises ValidationError, not a generic ProcessingError.
|
||||
try:
|
||||
limit = int(limit)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValidationError(f"Invalid limit: {limit!r}") from exc
|
||||
|
||||
try:
|
||||
# Build query
|
||||
if labels:
|
||||
@@ -708,15 +698,6 @@ class Neo4jStore:
|
||||
Returns:
|
||||
List of matching relationships
|
||||
"""
|
||||
# See get_nodes: LIMIT is interpolated directly, so validate
|
||||
# explicitly rather than trust the unenforced `limit: int` hint,
|
||||
# outside the try/except below so a bad limit raises
|
||||
# ValidationError, not a generic ProcessingError.
|
||||
try:
|
||||
limit = int(limit)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValidationError(f"Invalid limit: {limit!r}") from exc
|
||||
|
||||
try:
|
||||
type_filter = f":{sanitize_identifier(rel_type, 'relationship type')}" if rel_type else ""
|
||||
|
||||
|
||||
@@ -29,7 +29,6 @@ License: MIT
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
@@ -39,96 +38,6 @@ from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
|
||||
_IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
|
||||
# Fragments that turn a filter/order clause into a second statement, a
|
||||
# data-exfiltration UNION, a time-based blind-injection oracle, or schema
|
||||
# enumeration, rather than a boolean/ordering expression.
|
||||
_SQL_FRAGMENT_BLOCKLIST_RE = re.compile(
|
||||
r";|--|/\*|\*/|\bunion\b|\binsert\b|\bupdate\b|\bdelete\b|\bdrop\b|"
|
||||
r"\balter\b|\bcreate\b|\bexec\b|\bexecute\b|\bgrant\b|\brevoke\b|"
|
||||
r"\battach\b|\bpragma\b|\bxp_\w+|\bsp_\w+|\binto\s+outfile\b|\bload_file\b|"
|
||||
r"\bsleep\s*\(|\bbenchmark\s*\(|\bpg_sleep\s*\(|\bwaitfor\b|"
|
||||
r"\bdbms_\w+|\butl_\w+|\binformation_schema\b|\bpg_catalog\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# SQL single-quoted string literals ('' is the standard escaped-quote) and
|
||||
# double-quoted identifiers ("" likewise) — matched only when properly
|
||||
# closed, so a malformed/unterminated quote sequence is left alone and
|
||||
# still hits the blocklist above rather than being treated as "inside a
|
||||
# literal" and skipped.
|
||||
_SQL_STRING_LITERAL_RE = re.compile(r"'(?:[^']|'')*'")
|
||||
_SQL_QUOTED_IDENTIFIER_RE = re.compile(r'"(?:[^"]|"")*"')
|
||||
|
||||
|
||||
def _mask_sql_literals(fragment: str) -> str:
|
||||
"""Blank the contents of quoted literals so they can't trip the blocklist.
|
||||
|
||||
A legitimate value or quoted identifier that happens to contain a
|
||||
blocked word or character as *data* — e.g. ``status = 'union'`` or
|
||||
``"my--column" = 1`` — is not SQL syntax and shouldn't be rejected as
|
||||
if it were. Only the quoted span's interior is replaced (with `?`,
|
||||
keeping the surrounding quotes and the fragment's length/positions
|
||||
intact for the error message); text outside any properly closed quote
|
||||
is passed through unchanged and still fully scrutinized.
|
||||
"""
|
||||
fragment = _SQL_STRING_LITERAL_RE.sub(
|
||||
lambda m: "'" + "?" * (len(m.group(0)) - 2) + "'", fragment
|
||||
)
|
||||
fragment = _SQL_QUOTED_IDENTIFIER_RE.sub(
|
||||
lambda m: '"' + "?" * (len(m.group(0)) - 2) + '"', fragment
|
||||
)
|
||||
return fragment
|
||||
|
||||
|
||||
def _validate_sql_identifier(name: str, kind: str) -> str:
|
||||
"""Validate a table/schema name used as a raw SQL identifier.
|
||||
|
||||
``export_table_data`` interpolates *name* directly into the query text
|
||||
(SQLAlchemy has no bind-parameter syntax for identifiers), so anything
|
||||
outside a plain alphanumeric/underscore identifier is a potential
|
||||
breakout of the surrounding ``"..."`` quoting.
|
||||
"""
|
||||
if not isinstance(name, str) or not _IDENTIFIER_RE.match(name):
|
||||
raise ValidationError(
|
||||
f"Invalid {kind}: {name!r}. Must start with a letter or "
|
||||
"underscore and contain only alphanumeric characters and "
|
||||
"underscores."
|
||||
)
|
||||
return name
|
||||
|
||||
|
||||
def _validate_sql_fragment(fragment: str, kind: str) -> str:
|
||||
"""Reject WHERE/ORDER BY fragments that smuggle a second statement.
|
||||
|
||||
These clauses can't be bound as query parameters (they're arbitrary
|
||||
boolean/ordering expressions, not values), so this blocks the concrete
|
||||
injection primitives (statement separators, comments, UNION, DML/DDL
|
||||
keywords, time-based blind oracles, schema enumeration) rather than
|
||||
parameterizing.
|
||||
|
||||
This is a blocklist, not a grammar: it cannot exhaustively prove
|
||||
*fragment* is safe, only reject known-dangerous constructs, so a
|
||||
boolean-blind subquery expressed with none of the blocked keywords
|
||||
(e.g. ``id = (SELECT 1 FROM t WHERE ...)``) still passes. ``where``/
|
||||
``order_by`` are a raw-SQL-fragment API by design (see
|
||||
``export_table_data``'s docstring); treat them as trusted/operator
|
||||
input, not something to expose directly to untrusted end users.
|
||||
"""
|
||||
if not isinstance(fragment, str):
|
||||
raise ValidationError(f"Invalid {kind}: must be a string")
|
||||
# Check the blocklist against literal-masked text so a blocked word
|
||||
# appearing only as quoted data (not as SQL syntax) doesn't false-
|
||||
# positive; the original, unmodified fragment is still what's returned
|
||||
# and used in the query.
|
||||
if _SQL_FRAGMENT_BLOCKLIST_RE.search(_mask_sql_literals(fragment)):
|
||||
raise ValidationError(
|
||||
f"Invalid {kind}: {fragment!r} contains disallowed SQL "
|
||||
"keywords or statement-boundary characters"
|
||||
)
|
||||
return fragment
|
||||
|
||||
|
||||
@dataclass
|
||||
class TableData:
|
||||
@@ -344,13 +253,8 @@ class DataExporter:
|
||||
schema: Schema name (for databases with schema support, optional)
|
||||
limit: Maximum number of rows to export (optional)
|
||||
offset: Row offset for pagination (optional)
|
||||
where: WHERE clause for filtering (optional, e.g., "age > 18").
|
||||
Raw SQL, checked against a keyword/character blocklist (see
|
||||
``_validate_sql_fragment``) but not fully sanitized — treat
|
||||
as trusted/operator input, never pass untrusted end-user
|
||||
text here directly.
|
||||
order_by: ORDER BY clause for sorting (optional, e.g., "name ASC").
|
||||
Same trust requirement as ``where``.
|
||||
where: WHERE clause for filtering (optional, e.g., "age > 18")
|
||||
order_by: ORDER BY clause for sorting (optional, e.g., "name ASC")
|
||||
**options: Additional export options (unused)
|
||||
|
||||
Returns:
|
||||
@@ -365,16 +269,7 @@ class DataExporter:
|
||||
ProcessingError: If table export fails
|
||||
"""
|
||||
try:
|
||||
from sqlalchemy import inspect, text
|
||||
|
||||
_validate_sql_identifier(table_name, "table_name")
|
||||
if schema:
|
||||
_validate_sql_identifier(schema, "schema")
|
||||
if where:
|
||||
_validate_sql_fragment(where, "where")
|
||||
if order_by:
|
||||
_validate_sql_fragment(order_by, "order_by")
|
||||
|
||||
from sqlalchemy import inspect
|
||||
inspector = inspect(connection)
|
||||
|
||||
# Get column information
|
||||
@@ -449,8 +344,6 @@ class DataExporter:
|
||||
schema=schema,
|
||||
)
|
||||
|
||||
except ValidationError:
|
||||
raise
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to export table {table_name}: {e}")
|
||||
raise ProcessingError(f"Failed to export table: {e}") from e
|
||||
@@ -867,11 +760,8 @@ class DBIngestor:
|
||||
schema: Schema name (for databases with schema support, optional)
|
||||
limit: Maximum number of rows to export (optional)
|
||||
offset: Row offset for pagination (optional)
|
||||
where: WHERE clause for filtering (optional, e.g., "status = 'active'").
|
||||
Raw SQL passed through to ``export_table_data`` — same trust
|
||||
requirement documented there: not for untrusted end-user text.
|
||||
order_by: ORDER BY clause for sorting (optional, e.g., "created_at DESC").
|
||||
Same trust requirement as ``where``.
|
||||
where: WHERE clause for filtering (optional, e.g., "status = 'active'")
|
||||
order_by: ORDER BY clause for sorting (optional, e.g., "created_at DESC")
|
||||
transform: Whether to apply data transformations (default: False)
|
||||
**filters: Additional filtering options (merged with above parameters)
|
||||
|
||||
|
||||
@@ -41,7 +41,6 @@ from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from .ssrf import request_with_ssrf_guard
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -342,38 +341,36 @@ class MCPClient:
|
||||
raise
|
||||
|
||||
def _send_request_http(self, request: Dict[str, Any]) -> Optional[Dict[str, Any]]:
|
||||
"""Send request via HTTP, with redirect-safe credential handling.
|
||||
|
||||
Uses ``request_with_ssrf_guard`` so that:
|
||||
|
||||
* ``Authorization`` / ``Proxy-Authorization`` headers are **not**
|
||||
forwarded to a different origin if the MCP server issues a redirect
|
||||
(issue #947).
|
||||
* The redirect chain is bounded (default 10 hops).
|
||||
|
||||
``allow_private_ips=True`` is set because MCP servers are explicitly
|
||||
configured by the operator and frequently run on localhost or an
|
||||
internal network — the same trust model as ``allow_private_ips`` opt-in
|
||||
in the other ingestors. That trust covers only ``self.url`` itself:
|
||||
``allow_private_ips_on_redirect=False`` keeps redirect targets held to
|
||||
the normal public-address check, so a compromised or malicious MCP
|
||||
server cannot use a redirect to route the client into private/
|
||||
internal address space (e.g. cloud metadata) that the operator never
|
||||
configured. Scheme validation (http/https only) and the
|
||||
auth-stripping logic remain active regardless of these flags.
|
||||
"""
|
||||
"""Send request via HTTP."""
|
||||
try:
|
||||
response = request_with_ssrf_guard(
|
||||
"POST",
|
||||
import httpx
|
||||
|
||||
response = httpx.post(
|
||||
self.url,
|
||||
headers=self.headers,
|
||||
json=request,
|
||||
headers=self.headers,
|
||||
timeout=self.config.get("timeout", 30.0),
|
||||
allow_private_ips=True,
|
||||
allow_private_ips_on_redirect=False,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except (ImportError, OSError):
|
||||
# Fallback to requests if httpx not available
|
||||
try:
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
self.url,
|
||||
json=request,
|
||||
headers=self.headers,
|
||||
timeout=self.config.get("timeout", 30.0),
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except (ImportError, OSError):
|
||||
raise ProcessingError(
|
||||
"HTTP transport requires 'httpx' or 'requests' package. "
|
||||
"Install with: pip install httpx or pip install requests"
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to send HTTP request: {e}")
|
||||
raise
|
||||
|
||||
@@ -45,7 +45,6 @@ except ModuleNotFoundError: # pragma: no cover - fallback for minimal installs
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from .api_ingestor import APIData, RESTIngestor
|
||||
from .ssrf import request_with_ssrf_guard
|
||||
|
||||
AUTH_HEADER_NAMES = {
|
||||
"authorization",
|
||||
@@ -360,31 +359,18 @@ class PublicAPIIngestor(RESTIngestor):
|
||||
request_options = options.copy()
|
||||
timeout = request_options.pop("timeout", self.config.get("timeout", 30))
|
||||
rate_limit_delay = request_options.pop("rate_limit_delay", None)
|
||||
# session and allow_private_ips are always supplied explicitly below;
|
||||
# drop any caller-provided copies so request_with_ssrf_guard() does
|
||||
# not receive duplicate keyword arguments.
|
||||
request_options.pop("session", None)
|
||||
request_options.pop("allow_private_ips", None)
|
||||
request_headers = self._merged_headers(headers)
|
||||
|
||||
try:
|
||||
self._wait_if_needed(rate_limit_delay=rate_limit_delay)
|
||||
# Route through the SSRF guard so that:
|
||||
# * redirects to private/loopback IPs are blocked, and
|
||||
# * Authorization / Proxy-Authorization are stripped on
|
||||
# cross-origin redirects (issue #947).
|
||||
response = request_with_ssrf_guard(
|
||||
method,
|
||||
endpoint,
|
||||
session=self.session,
|
||||
response = self.session.request(
|
||||
method=method,
|
||||
url=endpoint,
|
||||
headers=request_headers,
|
||||
params=params,
|
||||
timeout=timeout,
|
||||
allow_private_ips=self.allow_private_ips,
|
||||
**request_options,
|
||||
)
|
||||
except (ValidationError, ProcessingError):
|
||||
raise
|
||||
except requests.exceptions.RequestException as exc:
|
||||
self.logger.error(f"Failed to detect public API {endpoint}: {exc}")
|
||||
raise ProcessingError(f"Failed to detect public API: {exc}") from exc
|
||||
@@ -454,30 +440,18 @@ class PublicAPIIngestor(RESTIngestor):
|
||||
|
||||
request_options = options.copy()
|
||||
timeout = request_options.pop("timeout", self.config.get("timeout", 30))
|
||||
# session and allow_private_ips are always supplied explicitly below;
|
||||
# drop any caller-provided copies so request_with_ssrf_guard() does
|
||||
# not receive duplicate keyword arguments.
|
||||
request_options.pop("session", None)
|
||||
request_options.pop("allow_private_ips", None)
|
||||
request_headers = self._merged_headers(headers)
|
||||
|
||||
try:
|
||||
self._wait_if_needed(rate_limit_delay=rate_limit_delay)
|
||||
# Route through the SSRF guard so that:
|
||||
# * redirects to private/loopback IPs are blocked, and
|
||||
# * Authorization / Proxy-Authorization are stripped on
|
||||
# cross-origin redirects even when validate_no_auth=False
|
||||
# (issue #947).
|
||||
response = request_with_ssrf_guard(
|
||||
method,
|
||||
endpoint,
|
||||
session=self.session,
|
||||
response = self.session.request(
|
||||
method=method,
|
||||
url=endpoint,
|
||||
headers=request_headers,
|
||||
params=params,
|
||||
data=data,
|
||||
json=json_data,
|
||||
timeout=timeout,
|
||||
allow_private_ips=self.allow_private_ips,
|
||||
**request_options,
|
||||
)
|
||||
|
||||
|
||||
+52
-462
@@ -11,7 +11,7 @@ import concurrent.futures
|
||||
import ipaddress
|
||||
import socket
|
||||
import threading
|
||||
from typing import Any, Iterable, List, Optional
|
||||
from typing import Any, Iterable, Optional
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
import requests
|
||||
@@ -138,7 +138,6 @@ def _get_dns_executor() -> concurrent.futures.ThreadPoolExecutor:
|
||||
BLOCKED_NETWORKS = (
|
||||
ipaddress.ip_network("0.0.0.0/8"),
|
||||
ipaddress.ip_network("10.0.0.0/8"),
|
||||
ipaddress.ip_network("100.64.0.0/10"), # CGNAT (RFC 6598) — routable inside carrier/cloud NAT
|
||||
ipaddress.ip_network("127.0.0.0/8"),
|
||||
ipaddress.ip_network("169.254.0.0/16"), # link-local / cloud metadata
|
||||
ipaddress.ip_network("172.16.0.0/12"),
|
||||
@@ -263,225 +262,12 @@ def validate_url_for_request(
|
||||
)
|
||||
|
||||
|
||||
def _resolve_pinned_ips(
|
||||
url: str, *, allow_private_ips: bool
|
||||
) -> Optional[List[str]]:
|
||||
"""Validate *url* and return every resolved IP for connection pinning.
|
||||
|
||||
``validate_url_for_request`` and the subsequent connection used to
|
||||
resolve the same hostname independently, which reopens a DNS-rebinding
|
||||
TOCTOU window: a low-TTL or rebinding DNS answer can differ between the
|
||||
validation lookup and the connect-time lookup, so a hostname that
|
||||
validated as public can still connect to a private/internal address.
|
||||
This performs the one resolution that is actually used for both the
|
||||
accept/reject decision *and* the connection (see
|
||||
``_make_pinned_adapter``), closing that window the same way
|
||||
``explorer/routes/ontology.py``'s ``_validate_fetch_url`` /
|
||||
``_make_pinned_session`` pair already does.
|
||||
|
||||
Returns ``None`` when ``allow_private_ips`` is True (the caller
|
||||
explicitly trusts this host, e.g. an operator-configured internal
|
||||
endpoint that may rely on live DNS/service discovery — pinning is
|
||||
skipped so it keeps resolving normally) or when the URL has no host.
|
||||
Otherwise returns the deduplicated, resolution-ordered list of
|
||||
validated IP addresses.
|
||||
"""
|
||||
validate_url_for_request(url, allow_private_ips=allow_private_ips)
|
||||
if allow_private_ips:
|
||||
return None
|
||||
|
||||
host = urlparse(url).hostname
|
||||
if not host:
|
||||
return None
|
||||
|
||||
try:
|
||||
literal_ip = ipaddress.ip_address(host)
|
||||
except ValueError:
|
||||
literal_ip = None
|
||||
if literal_ip is not None:
|
||||
return [str(literal_ip)]
|
||||
|
||||
executor = _get_dns_executor()
|
||||
owned_executor = False
|
||||
try:
|
||||
try:
|
||||
future = executor.submit(socket.getaddrinfo, host, None)
|
||||
except RuntimeError:
|
||||
executor = concurrent.futures.ThreadPoolExecutor(max_workers=1)
|
||||
owned_executor = True
|
||||
future = executor.submit(socket.getaddrinfo, host, None)
|
||||
resolved: Iterable = future.result(timeout=_DNS_RESOLVE_TIMEOUT_SECONDS)
|
||||
except (socket.gaierror, concurrent.futures.TimeoutError, OSError) as exc:
|
||||
raise ValidationError(
|
||||
f"URL host '{host}' could not be resolved safely "
|
||||
"(DNS error or timeout); request blocked"
|
||||
) from exc
|
||||
finally:
|
||||
if owned_executor:
|
||||
_shutdown_executor(executor)
|
||||
|
||||
pinned_ips: List[str] = []
|
||||
for info in resolved:
|
||||
addr = ipaddress.ip_address(info[4][0])
|
||||
if _ip_is_blocked(addr):
|
||||
raise ValidationError(
|
||||
f"URL host '{host}' resolves to a blocked (private/loopback/"
|
||||
"link-local) address"
|
||||
)
|
||||
addr_str = str(addr)
|
||||
if addr_str not in pinned_ips:
|
||||
pinned_ips.append(addr_str)
|
||||
if not pinned_ips:
|
||||
raise ValidationError(
|
||||
f"URL host '{host}' could not be resolved to a usable address"
|
||||
)
|
||||
return pinned_ips
|
||||
|
||||
|
||||
def _make_pinned_adapter(pinned_ips: List[str], hostname: str) -> "requests.adapters.HTTPAdapter":
|
||||
"""Build an HTTPAdapter that connects only to *pinned_ips*.
|
||||
|
||||
Falls back across every pinned address in order (a hostname can have
|
||||
multiple A/AAAA records) while presenting *hostname* as the TLS SNI /
|
||||
certificate identity and outgoing Host header, so DNS resolution is
|
||||
bypassed entirely for the actual connection — mirroring
|
||||
``explorer/routes/ontology.py``'s ``_make_pinned_session``.
|
||||
"""
|
||||
import urllib3.util.connection as _u3_connection
|
||||
from urllib3.exceptions import NewConnectionError
|
||||
|
||||
class _MultiIPConnectionMixin:
|
||||
def _new_conn(self):
|
||||
last_exc: Optional[BaseException] = None
|
||||
for ip in pinned_ips:
|
||||
try:
|
||||
return _u3_connection.create_connection(
|
||||
(ip, self.port),
|
||||
self.timeout,
|
||||
source_address=self.source_address,
|
||||
socket_options=self.socket_options,
|
||||
)
|
||||
except OSError as exc:
|
||||
last_exc = exc
|
||||
continue
|
||||
raise NewConnectionError(
|
||||
self,
|
||||
f"Failed to establish a connection to any of {pinned_ips}: {last_exc}",
|
||||
)
|
||||
|
||||
class _PinnedIPHTTPAdapter(requests.adapters.HTTPAdapter):
|
||||
def get_connection_with_tls_context(self, request, verify, proxies=None, cert=None):
|
||||
# A proxy performs its own DNS resolution outside this
|
||||
# process's control, which would silently reopen the exact
|
||||
# rebinding race pinning exists to close. Fail closed instead.
|
||||
if requests.utils.select_proxy(request.url, proxies):
|
||||
raise ValidationError(
|
||||
"Proxied requests are not supported through the "
|
||||
"SSRF-guarded request path (a proxy would resolve the "
|
||||
"host itself and bypass IP pinning)."
|
||||
)
|
||||
host_params, pool_kwargs = self.build_connection_pool_key_attributes(
|
||||
request, verify, cert
|
||||
)
|
||||
if host_params.get("scheme") == "https":
|
||||
pool_kwargs.setdefault("assert_hostname", hostname)
|
||||
pool_kwargs.setdefault("server_hostname", hostname)
|
||||
host_params["host"] = pinned_ips[0]
|
||||
pool = self.poolmanager.connection_from_host(
|
||||
**host_params, pool_kwargs=pool_kwargs
|
||||
)
|
||||
base_connection_cls = pool.ConnectionCls
|
||||
if not issubclass(base_connection_cls, _MultiIPConnectionMixin):
|
||||
pool.ConnectionCls = type(
|
||||
"_PinnedConnection",
|
||||
(_MultiIPConnectionMixin, base_connection_cls),
|
||||
{},
|
||||
)
|
||||
return pool
|
||||
|
||||
return _PinnedIPHTTPAdapter()
|
||||
|
||||
|
||||
def _apply_connection_pin(
|
||||
active_session: "requests.Session",
|
||||
url: str,
|
||||
pinned_ips: Optional[List[str]],
|
||||
orig_http_adapter: "requests.adapters.HTTPAdapter",
|
||||
orig_https_adapter: "requests.adapters.HTTPAdapter",
|
||||
had_host_header: bool,
|
||||
orig_host_header: Optional[str],
|
||||
) -> None:
|
||||
"""Mount (or remove) IP pinning on *active_session* for the next hop."""
|
||||
# Session.mount() silently drops whatever adapter it replaces without
|
||||
# closing it. Across a multi-hop redirect chain, each hop gets its own
|
||||
# fresh pinned adapter (a new pool), so failing to close the one from
|
||||
# the previous hop would leak its pooled connection.
|
||||
_current = active_session.adapters.get("http://")
|
||||
if getattr(_current, "_semantica_pinned", False):
|
||||
_current.close()
|
||||
|
||||
parsed = urlparse(url)
|
||||
if pinned_ips:
|
||||
port = parsed.port
|
||||
default_port = _DEFAULT_PORTS.get(parsed.scheme, 80)
|
||||
host_header = (
|
||||
parsed.hostname
|
||||
if port in (None, default_port)
|
||||
else f"{parsed.hostname}:{port}"
|
||||
)
|
||||
adapter = _make_pinned_adapter(pinned_ips, parsed.hostname or "")
|
||||
adapter._semantica_pinned = True
|
||||
active_session.mount("http://", adapter)
|
||||
active_session.mount("https://", adapter)
|
||||
active_session.headers["Host"] = host_header
|
||||
else:
|
||||
active_session.mount("http://", orig_http_adapter)
|
||||
active_session.mount("https://", orig_https_adapter)
|
||||
# Restore the session's own pre-call Host header state rather than
|
||||
# unconditionally clearing it — a caller-supplied session may carry
|
||||
# a legitimate Host override (e.g. a private/internal endpoint
|
||||
# fronted by a name that differs from the connection host), which
|
||||
# a hop that happens not to need pinning must not silently drop.
|
||||
if had_host_header:
|
||||
active_session.headers["Host"] = orig_host_header
|
||||
else:
|
||||
active_session.headers.pop("Host", None)
|
||||
|
||||
|
||||
_SESSION_LOCK_ATTR = "_semantica_ssrf_lock"
|
||||
_session_lock_registry_lock = threading.Lock()
|
||||
|
||||
|
||||
def _get_session_lock(session: "requests.Session") -> threading.Lock:
|
||||
"""Return a lock private to *session*, creating one on first use.
|
||||
|
||||
request_with_ssrf_guard mutates a caller-supplied session's adapters
|
||||
and Host header for the duration of one guarded call (including every
|
||||
redirect hop). Without serializing on the session itself, two guarded
|
||||
calls sharing the same session from different threads could interleave
|
||||
their mount()/restore cycles — one call's request could go out pinned
|
||||
to (or carrying the Host header for) a completely different call's
|
||||
target host. Double-checked locking so concurrent first-use doesn't
|
||||
attach two different locks to the same session.
|
||||
"""
|
||||
lock = getattr(session, _SESSION_LOCK_ATTR, None)
|
||||
if lock is not None:
|
||||
return lock
|
||||
with _session_lock_registry_lock:
|
||||
lock = getattr(session, _SESSION_LOCK_ATTR, None)
|
||||
if lock is None:
|
||||
lock = threading.Lock()
|
||||
setattr(session, _SESSION_LOCK_ATTR, lock)
|
||||
return lock
|
||||
|
||||
|
||||
def request_with_ssrf_guard(
|
||||
method: str,
|
||||
url: str,
|
||||
*,
|
||||
session: Optional[requests.Session] = None,
|
||||
allow_private_ips: bool = False,
|
||||
allow_private_ips_on_redirect: Optional[bool] = None,
|
||||
max_redirects: int = _DEFAULT_MAX_REDIRECTS,
|
||||
**kwargs: Any,
|
||||
) -> requests.Response:
|
||||
@@ -491,263 +277,67 @@ def request_with_ssrf_guard(
|
||||
public URL to bounce into private/loopback/link-local space. This helper
|
||||
disables automatic redirects and re-validates each ``Location`` target
|
||||
before issuing the next hop.
|
||||
|
||||
``allow_private_ips`` trusts the caller's own *url* (e.g. an
|
||||
operator-configured internal endpoint). That trust follows a redirect
|
||||
only when the redirect target's host matches the original host (e.g. a
|
||||
same-host path redirect on a private/localhost server); a redirect to a
|
||||
*different* host is validated with ``allow_private_ips_on_redirect``
|
||||
instead, which defaults to ``allow_private_ips`` for backward
|
||||
compatibility but can be pinned to ``False`` by callers that want to
|
||||
trust only the original host and never extend private-IP eligibility to
|
||||
any other host a redirect chain might reach — otherwise a private-IP-
|
||||
eligible endpoint could be tricked into redirecting into arbitrary
|
||||
internal address space (e.g. cloud metadata) the caller never
|
||||
configured.
|
||||
|
||||
Authorization / credential-header handling (issue #947)
|
||||
--------------------------------------------------------
|
||||
Credentials are stripped from **all** sources that ``requests`` can use to
|
||||
attach an ``Authorization`` header whenever a redirect changes origin:
|
||||
|
||||
1. ``kwargs["headers"]`` — per-request header dict (already handled).
|
||||
2. ``session.headers`` — session-level headers that ``requests`` merges
|
||||
automatically; cleared for the hop and restored via ``finally``.
|
||||
3. ``kwargs["auth"]`` — per-request auth tuple/callable; removed from the
|
||||
local ``kwargs`` copy when stripping is required. This copy never
|
||||
escapes to the caller, so there is nothing to restore.
|
||||
4. ``session.auth`` — session-level auth handler that ``requests`` merges
|
||||
via ``merge_setting(auth, self.auth)`` inside ``prepare_request``;
|
||||
cleared for the hop and restored via ``finally``.
|
||||
5. ``session.trust_env`` — when ``True``, ``requests`` reads ``~/.netrc``
|
||||
for the *redirect target* host and calls ``prepare_auth()`` with those
|
||||
credentials even after sources 3 and 4 are cleared; disabled for
|
||||
cross-origin hops and restored via ``finally``.
|
||||
|
||||
Leaving any one of these intact allows ``requests`` to re-attach
|
||||
credentials on the hop to the foreign origin, defeating the header-level
|
||||
strip.
|
||||
|
||||
Session state that was removed is unconditionally restored in a ``finally``
|
||||
block so the session is left in its original state after this call returns,
|
||||
regardless of how it exits (normal return, exception, redirect cap). A
|
||||
caller-supplied session is also serialized on internally (see
|
||||
``_get_session_lock``): two guarded calls sharing the same session from
|
||||
different threads block on each other for the call's duration rather than
|
||||
interleaving their mutations, so concurrent use of a shared session is
|
||||
safe, if not concurrent.
|
||||
|
||||
Once credentials have been stripped for a cross-origin hop they are NOT
|
||||
re-added for subsequent hops in the same chain, even if a later hop
|
||||
happens to point back to the original host. This prevents credential
|
||||
resurrection via crafted multi-hop redirect chains.
|
||||
"""
|
||||
kwargs = dict(kwargs)
|
||||
kwargs.pop("allow_redirects", None)
|
||||
|
||||
redirect_allow_private_ips = (
|
||||
allow_private_ips
|
||||
if allow_private_ips_on_redirect is None
|
||||
else allow_private_ips_on_redirect
|
||||
)
|
||||
_original_host = (urlparse(url).hostname or "").lower()
|
||||
validate_url_for_request(url, allow_private_ips=allow_private_ips)
|
||||
|
||||
current_pinned_ips = _resolve_pinned_ips(url, allow_private_ips=allow_private_ips)
|
||||
|
||||
_owns_session = session is None
|
||||
active_session = session if session is not None else requests.Session()
|
||||
requester = active_session.request
|
||||
requester = session.request if session is not None else requests.request
|
||||
current_url = url
|
||||
current_method = method.upper()
|
||||
redirects_followed = 0
|
||||
|
||||
# A caller-supplied session is mutated (adapters + Host header, and
|
||||
# potentially auth/trust_env below) for the duration of this call,
|
||||
# including every redirect hop; serialize on the session itself so a
|
||||
# second guarded call sharing it from another thread can't interleave
|
||||
# its own mount()/restore cycle into the middle of this one. An owned
|
||||
# session is private to this call, so no lock is needed. Released in
|
||||
# the outermost `finally` below, alongside the state it protects.
|
||||
_session_lock = None if _owns_session else _get_session_lock(active_session)
|
||||
if _session_lock is not None:
|
||||
_session_lock.acquire()
|
||||
while True:
|
||||
response = requester(
|
||||
current_method,
|
||||
current_url,
|
||||
allow_redirects=False,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
# Snapshot the session's pre-existing adapters/Host header so pinning
|
||||
# (mounted per-hop below) can be fully undone when this call returns —
|
||||
# required for a caller-supplied session, which outlives this call.
|
||||
_orig_http_adapter = (
|
||||
active_session.adapters.get("http://") or requests.adapters.HTTPAdapter()
|
||||
)
|
||||
_orig_https_adapter = (
|
||||
active_session.adapters.get("https://") or requests.adapters.HTTPAdapter()
|
||||
)
|
||||
_had_host_header = "Host" in active_session.headers
|
||||
_orig_host_header = active_session.headers.get("Host")
|
||||
|
||||
# -- issue #947: snapshot every session-level credential source so we can
|
||||
# restore them unconditionally when this call exits.
|
||||
_SENSITIVE = ("Authorization", "Proxy-Authorization")
|
||||
_session_auth_backup: dict = {}
|
||||
_session_auth_handler_backup: Any = None # session.auth backup
|
||||
_session_trust_env_backup: bool = True # session.trust_env backup
|
||||
|
||||
if session is not None:
|
||||
for _h in _SENSITIVE:
|
||||
# requests stores session headers in a case-insensitive dict;
|
||||
# .get() matches regardless of the casing used at insertion time.
|
||||
_val = session.headers.get(_h)
|
||||
if _val is not None:
|
||||
_session_auth_backup[_h] = _val
|
||||
# Snapshot session.auth (HTTPBasicAuth, tuple, callable, or None).
|
||||
_session_auth_handler_backup = session.auth
|
||||
# Snapshot session.trust_env (controls .netrc / env proxy lookup).
|
||||
_session_trust_env_backup = session.trust_env
|
||||
|
||||
# Track whether credentials have been stripped for this redirect chain.
|
||||
# Once stripped they must not reappear on any subsequent hop.
|
||||
_auth_stripped = False
|
||||
|
||||
try:
|
||||
while True:
|
||||
_apply_connection_pin(
|
||||
active_session,
|
||||
current_url,
|
||||
current_pinned_ips,
|
||||
_orig_http_adapter,
|
||||
_orig_https_adapter,
|
||||
_had_host_header,
|
||||
_orig_host_header,
|
||||
)
|
||||
response = requester(
|
||||
current_method,
|
||||
current_url,
|
||||
allow_redirects=False,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
if response.status_code not in _REDIRECT_STATUS_CODES:
|
||||
return response
|
||||
|
||||
if redirects_followed >= max_redirects:
|
||||
response.close()
|
||||
raise ValidationError(
|
||||
f"Exceeded maximum redirects ({max_redirects}) while "
|
||||
f"fetching '{url}'"
|
||||
)
|
||||
|
||||
location = response.headers.get("Location")
|
||||
if not location or not str(location).strip():
|
||||
response.close()
|
||||
raise ValidationError(
|
||||
f"Redirect from '{current_url}' is missing a Location header"
|
||||
)
|
||||
|
||||
next_url = urljoin(current_url, str(location).strip())
|
||||
next_host = (urlparse(next_url).hostname or "").lower()
|
||||
# A redirect back to the original host inherits the caller's
|
||||
# trust in that host (e.g. a same-host path redirect on a
|
||||
# private/localhost MCP server). A redirect to a *different*
|
||||
# host must not inherit that trust, even if the original host
|
||||
# was private/internal — otherwise a compromised or malicious
|
||||
# endpoint could redirect into arbitrary private address space
|
||||
# (e.g. cloud metadata) the caller never configured.
|
||||
hop_allow_private_ips = (
|
||||
allow_private_ips
|
||||
if next_host and next_host == _original_host
|
||||
else redirect_allow_private_ips
|
||||
)
|
||||
current_pinned_ips = _resolve_pinned_ips(
|
||||
next_url, allow_private_ips=hop_allow_private_ips
|
||||
)
|
||||
|
||||
# Do not leak sensitive headers or auth handlers to a different
|
||||
# origin on redirects. All four credential sources are cleared:
|
||||
# • kwargs["headers"] — per-request header dict
|
||||
# • session.headers — session-level header dict
|
||||
# • kwargs["auth"] — per-request auth tuple/callable
|
||||
# • session.auth — session-level auth handler
|
||||
#
|
||||
# Once stripped (_auth_stripped=True), credentials stay absent for
|
||||
# the remainder of the chain — even if a later hop targets the
|
||||
# original host — to prevent credential resurrection.
|
||||
if _auth_stripped or _should_strip_auth(current_url, next_url):
|
||||
_auth_stripped = True
|
||||
|
||||
# 1. Strip from per-request kwargs headers.
|
||||
kwargs = dict(kwargs)
|
||||
headers = dict(kwargs.get("headers") or {})
|
||||
for sensitive in _SENSITIVE:
|
||||
headers.pop(sensitive, None)
|
||||
# Also remove any case variant the caller may have used
|
||||
# (e.g. "authorization" or "AUTHORIZATION").
|
||||
for key in list(headers):
|
||||
if key.lower() == sensitive.lower():
|
||||
del headers[key]
|
||||
kwargs["headers"] = headers
|
||||
|
||||
# 2. Strip per-request auth kwarg so requests cannot call
|
||||
# prepare_auth() with the caller's credential on this hop.
|
||||
kwargs.pop("auth", None)
|
||||
|
||||
# 3. Strip session-level headers so requests cannot re-inject
|
||||
# them when merging session + per-request headers for this hop.
|
||||
if session is not None:
|
||||
for sensitive in _SENSITIVE:
|
||||
# CaseInsensitiveDict.pop(key, None) handles any casing.
|
||||
session.headers.pop(sensitive, None)
|
||||
|
||||
# 4. Clear session.auth so prepare_request's merge_setting()
|
||||
# cannot fall back to the session-level auth handler and
|
||||
# reattach credentials on the foreign-origin hop.
|
||||
session.auth = None
|
||||
|
||||
# 5. Disable .netrc / environment-proxy credential lookup so
|
||||
# requests cannot inject credentials from ~/.netrc for the
|
||||
# redirect target host on this hop.
|
||||
session.trust_env = False
|
||||
|
||||
# Match requests' historical method rewriting for 301/302/303.
|
||||
if (
|
||||
response.status_code in _STRIP_BODY_ON_REDIRECT
|
||||
and current_method not in {"GET", "HEAD"}
|
||||
):
|
||||
current_method = "GET"
|
||||
for key in ("data", "json", "files"):
|
||||
kwargs.pop(key, None)
|
||||
|
||||
# Params apply to the original request URL only; Location is authoritative.
|
||||
kwargs.pop("params", None)
|
||||
if response.status_code not in _REDIRECT_STATUS_CODES:
|
||||
return response
|
||||
|
||||
if redirects_followed >= max_redirects:
|
||||
response.close()
|
||||
current_url = next_url
|
||||
redirects_followed += 1
|
||||
raise ValidationError(
|
||||
f"Exceeded maximum redirects ({max_redirects}) while "
|
||||
f"fetching '{url}'"
|
||||
)
|
||||
|
||||
finally:
|
||||
if _owns_session:
|
||||
# No caller holds a reference to this session; just release it.
|
||||
active_session.close()
|
||||
else:
|
||||
# Unconditionally restore every session credential source and
|
||||
# pinning artifact we touched, so the session is in its
|
||||
# original state after this call returns or raises.
|
||||
if _session_auth_backup:
|
||||
for _h, _v in _session_auth_backup.items():
|
||||
active_session.headers[_h] = _v
|
||||
# Restore session.auth to whatever it was before this call.
|
||||
active_session.auth = _session_auth_handler_backup
|
||||
# Restore session.trust_env (.netrc / env-proxy lookup flag).
|
||||
active_session.trust_env = _session_trust_env_backup
|
||||
# Undo any IP-pinning adapter/Host header mounted for a hop,
|
||||
# closing the last pinned adapter so its pooled connection
|
||||
# isn't leaked (see _apply_connection_pin).
|
||||
_current = active_session.adapters.get("http://")
|
||||
if getattr(_current, "_semantica_pinned", False):
|
||||
_current.close()
|
||||
active_session.mount("http://", _orig_http_adapter)
|
||||
active_session.mount("https://", _orig_https_adapter)
|
||||
if _had_host_header:
|
||||
active_session.headers["Host"] = _orig_host_header
|
||||
else:
|
||||
active_session.headers.pop("Host", None)
|
||||
if _session_lock is not None:
|
||||
_session_lock.release()
|
||||
location = response.headers.get("Location")
|
||||
if not location or not str(location).strip():
|
||||
response.close()
|
||||
raise ValidationError(
|
||||
f"Redirect from '{current_url}' is missing a Location header"
|
||||
)
|
||||
|
||||
next_url = urljoin(current_url, str(location).strip())
|
||||
validate_url_for_request(next_url, allow_private_ips=allow_private_ips)
|
||||
|
||||
# Do not leak sensitive headers to a different origin on redirects:
|
||||
# reuse the caller's headers only while host, port, and scheme keep
|
||||
# the credential safe, mirroring requests' should_strip_auth.
|
||||
if _should_strip_auth(current_url, next_url):
|
||||
kwargs = dict(kwargs)
|
||||
headers = dict(kwargs.get("headers") or {})
|
||||
for sensitive in ("Authorization", "Proxy-Authorization"):
|
||||
headers.pop(sensitive, None)
|
||||
kwargs["headers"] = headers
|
||||
|
||||
# Match requests' historical method rewriting for 301/302/303.
|
||||
if (
|
||||
response.status_code in _STRIP_BODY_ON_REDIRECT
|
||||
and current_method not in {"GET", "HEAD"}
|
||||
):
|
||||
current_method = "GET"
|
||||
for key in ("data", "json", "files"):
|
||||
kwargs.pop(key, None)
|
||||
|
||||
# Params apply to the original request URL only; Location is authoritative.
|
||||
kwargs.pop("params", None)
|
||||
|
||||
response.close()
|
||||
current_url = next_url
|
||||
redirects_followed += 1
|
||||
|
||||
@@ -22,9 +22,8 @@ License: MIT
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from ..deduplication.duplicate_detector import DuplicateDetector, DuplicateGroup
|
||||
from ..deduplication.duplicate_detector import DuplicateDetector
|
||||
from ..deduplication.entity_merger import EntityMerger
|
||||
from ..utils.entity_ids import get_entity_id
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
|
||||
@@ -139,7 +138,9 @@ class EntityResolver:
|
||||
self.logger.debug(
|
||||
f"Detecting duplicate groups with threshold {self.similarity_threshold}"
|
||||
)
|
||||
duplicate_groups = self._detect_duplicate_groups(entities)
|
||||
duplicate_groups = self.duplicate_detector.detect_duplicate_groups(
|
||||
entities, threshold=self.similarity_threshold
|
||||
)
|
||||
|
||||
self.logger.debug(f"Found {len(duplicate_groups)} duplicate group(s)")
|
||||
|
||||
@@ -149,7 +150,6 @@ class EntityResolver:
|
||||
# Step 2: Merge duplicates in each group
|
||||
merged_entities = []
|
||||
processed_entity_ids = set() # Track which entities have been merged
|
||||
processed_entity_objects = set()
|
||||
|
||||
for group in duplicate_groups:
|
||||
# Skip groups with less than 2 entities (not duplicates)
|
||||
@@ -157,16 +157,9 @@ class EntityResolver:
|
||||
continue
|
||||
|
||||
# Merge the duplicate group into a single canonical entity
|
||||
if self.resolution_strategy == "exact":
|
||||
merge_operations = [
|
||||
self.entity_merger.merge_entity_group(
|
||||
group.entities, **self.config
|
||||
)
|
||||
]
|
||||
else:
|
||||
merge_operations = self.entity_merger.merge_duplicates(
|
||||
group.entities, **self.config
|
||||
)
|
||||
merge_operations = self.entity_merger.merge_duplicates(
|
||||
group.entities, **self.config
|
||||
)
|
||||
|
||||
# Process each merge operation
|
||||
for operation in merge_operations:
|
||||
@@ -175,26 +168,30 @@ class EntityResolver:
|
||||
|
||||
# Mark all source entities as processed
|
||||
for source_entity in operation.source_entities:
|
||||
entity_id = self._get_entity_id(source_entity)
|
||||
if entity_id is None:
|
||||
processed_entity_objects.add(id(source_entity))
|
||||
continue
|
||||
try:
|
||||
entity_id = (
|
||||
source_entity.get("id")
|
||||
if isinstance(source_entity, dict)
|
||||
else getattr(source_entity, "id", None)
|
||||
) or (
|
||||
source_entity.get("entity_id")
|
||||
if isinstance(source_entity, dict)
|
||||
else getattr(source_entity, "entity_id", None)
|
||||
)
|
||||
if entity_id:
|
||||
processed_entity_ids.add(entity_id)
|
||||
except TypeError:
|
||||
processed_entity_objects.add(id(source_entity))
|
||||
|
||||
# Step 3: Add non-duplicate entities (entities not in any duplicate group)
|
||||
for entity in entities:
|
||||
entity_id = self._get_entity_id(entity)
|
||||
if entity_id is None:
|
||||
is_unprocessed = id(entity) not in processed_entity_objects
|
||||
else:
|
||||
try:
|
||||
is_unprocessed = entity_id not in processed_entity_ids
|
||||
except TypeError:
|
||||
is_unprocessed = id(entity) not in processed_entity_objects
|
||||
if is_unprocessed:
|
||||
entity_id = (
|
||||
entity.get("id")
|
||||
if isinstance(entity, dict)
|
||||
else getattr(entity, "id", None)
|
||||
) or (
|
||||
entity.get("entity_id")
|
||||
if isinstance(entity, dict)
|
||||
else getattr(entity, "entity_id", None)
|
||||
)
|
||||
if entity_id and entity_id not in processed_entity_ids:
|
||||
# This entity was not merged, add it as-is
|
||||
merged_entities.append(entity)
|
||||
|
||||
@@ -216,48 +213,6 @@ class EntityResolver:
|
||||
)
|
||||
raise
|
||||
|
||||
def _detect_duplicate_groups(
|
||||
self, entities: List[Dict[str, Any]]
|
||||
) -> List[DuplicateGroup]:
|
||||
"""Detect duplicate groups according to the configured strategy."""
|
||||
if self.resolution_strategy != "exact":
|
||||
return self.duplicate_detector.detect_duplicate_groups(
|
||||
entities, threshold=self.similarity_threshold
|
||||
)
|
||||
|
||||
groups = {}
|
||||
for entity in entities:
|
||||
name = self._get_entity_name(entity)
|
||||
normalized = str(name).strip() if name is not None else ""
|
||||
if normalized:
|
||||
groups.setdefault(normalized.casefold(), []).append(entity)
|
||||
|
||||
return [
|
||||
DuplicateGroup(entities=group, confidence=1.0)
|
||||
for group in groups.values()
|
||||
if len(group) > 1
|
||||
]
|
||||
|
||||
@staticmethod
|
||||
def _get_entity_id(entity: Any) -> Any:
|
||||
"""Return an entity ID while supporting dictionary and object inputs."""
|
||||
return get_entity_id(entity)
|
||||
|
||||
@staticmethod
|
||||
def _get_entity_name(entity: Any) -> Optional[str]:
|
||||
"""Return an entity name, falling back to text-based entity input."""
|
||||
if isinstance(entity, dict):
|
||||
name = entity.get("name")
|
||||
return (
|
||||
name if name is not None and str(name).strip() else entity.get("text")
|
||||
)
|
||||
name = getattr(entity, "name", None)
|
||||
return (
|
||||
name
|
||||
if name is not None and str(name).strip()
|
||||
else getattr(entity, "text", None)
|
||||
)
|
||||
|
||||
def merge_duplicates(self, entities: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Merge duplicate entities.
|
||||
@@ -278,13 +233,13 @@ class EntityResolver:
|
||||
processed_ids = set()
|
||||
for op in merge_operations:
|
||||
for source_entity in op.source_entities:
|
||||
entity_id = get_entity_id(source_entity)
|
||||
if entity_id is not None:
|
||||
entity_id = source_entity.get("id") or source_entity.get("entity_id")
|
||||
if entity_id:
|
||||
processed_ids.add(entity_id)
|
||||
|
||||
for entity in entities:
|
||||
entity_id = get_entity_id(entity)
|
||||
if entity_id is not None and entity_id not in processed_ids:
|
||||
entity_id = entity.get("id") or entity.get("entity_id")
|
||||
if entity_id and entity_id not in processed_ids:
|
||||
merged_entities.append(entity)
|
||||
|
||||
self.logger.info(f"Merged to {len(merged_entities)} entities")
|
||||
|
||||
@@ -185,25 +185,8 @@ class GraphValidator:
|
||||
|
||||
# 3. Relationship Validation
|
||||
for i, rel in enumerate(relationships):
|
||||
# Endpoints may use either the legacy ``source``/``target`` keys or
|
||||
# the canonical ``source_id``/``target_id`` keys emitted by
|
||||
# ``ContextGraph.to_kg_dict()``. Accept either variant so both
|
||||
# representations validate consistently.
|
||||
src = rel.get("source")
|
||||
if src is None:
|
||||
src = rel.get("source_id")
|
||||
tgt = rel.get("target")
|
||||
if tgt is None:
|
||||
tgt = rel.get("target_id")
|
||||
|
||||
# Check required fields: ``type`` plus a resolvable source/target.
|
||||
missing = set()
|
||||
if "type" not in rel:
|
||||
missing.add("type")
|
||||
if src is None:
|
||||
missing.add("source")
|
||||
if tgt is None:
|
||||
missing.add("target")
|
||||
# Check required fields
|
||||
missing = self.required_rel_fields - set(rel.keys())
|
||||
if missing:
|
||||
issues.append(ValidationIssue(
|
||||
code="MISSING_FIELD",
|
||||
@@ -213,6 +196,9 @@ class GraphValidator:
|
||||
details={"index": i}
|
||||
))
|
||||
continue
|
||||
|
||||
src = rel.get("source")
|
||||
tgt = rel.get("target")
|
||||
|
||||
# Check Dangling Edges
|
||||
def is_valid_id(node_id):
|
||||
|
||||
@@ -535,8 +535,7 @@ class TemporalGraphQuery:
|
||||
relationships = [
|
||||
rel
|
||||
for rel in relationships
|
||||
if (rel.get("source") or rel.get("source_id")) == entity
|
||||
or (rel.get("target") or rel.get("target_id")) == entity
|
||||
if rel.get("source") == entity or rel.get("target") == entity
|
||||
]
|
||||
|
||||
if relationship:
|
||||
@@ -643,14 +642,8 @@ class TemporalGraphQuery:
|
||||
parsed_end_time = self._parse_time(end_time) if end_time else None
|
||||
|
||||
for rel in relationships:
|
||||
# Accept both the legacy ``source``/``target`` keys and the
|
||||
# canonical ``source_id``/``target_id`` keys from ``to_kg_dict()``.
|
||||
s = rel.get("source")
|
||||
if s is None:
|
||||
s = rel.get("source_id")
|
||||
t = rel.get("target")
|
||||
if t is None:
|
||||
t = rel.get("target_id")
|
||||
|
||||
# Check temporal validity
|
||||
if start_time or end_time:
|
||||
|
||||
@@ -33,12 +33,6 @@ class SHACLViolation:
|
||||
value: Optional[str] = None
|
||||
shape: Optional[str] = None
|
||||
explanation: Optional[str] = None
|
||||
# Real constraint parameters extracted from the source shape (sh:sourceShape),
|
||||
# used to render accurate plain-English explanations.
|
||||
min_count: Optional[int] = None
|
||||
max_count: Optional[int] = None
|
||||
datatype: Optional[str] = None
|
||||
class_: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
@@ -50,10 +44,6 @@ class SHACLViolation:
|
||||
"value": self.value,
|
||||
"shape": self.shape,
|
||||
"explanation": self.explanation,
|
||||
"min_count": self.min_count,
|
||||
"max_count": self.max_count,
|
||||
"datatype": self.datatype,
|
||||
"class_": self.class_,
|
||||
}
|
||||
|
||||
|
||||
@@ -128,10 +118,10 @@ class SHACLValidationReport:
|
||||
focus_node=v.focus_node,
|
||||
path=v.result_path or "",
|
||||
value=v.value or "",
|
||||
min_count=v.min_count if v.min_count is not None else "?",
|
||||
max_count=v.max_count if v.max_count is not None else "?",
|
||||
datatype=v.datatype or "the expected datatype",
|
||||
class_=v.class_ or "the required class",
|
||||
min_count=1,
|
||||
max_count=1,
|
||||
datatype=v.message or "",
|
||||
class_=v.message or "",
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
@@ -218,32 +208,6 @@ def _run_pyshacl(
|
||||
shape_node = results_graph.value(result, SH.sourceShape)
|
||||
shape = str(shape_node) if shape_node is not None else None
|
||||
|
||||
# Look up the real constraint parameters from the source shape so that
|
||||
# explain_violations can render accurate values instead of placeholders.
|
||||
# Note: sh:qualifiedMinCount / sh:qualifiedMaxCount are not handled here;
|
||||
# such violations fall back to the "?" placeholder in explain_violations.
|
||||
min_count: Optional[int] = None
|
||||
max_count: Optional[int] = None
|
||||
datatype: Optional[str] = None
|
||||
class_: Optional[str] = None
|
||||
if shape_node is not None:
|
||||
min_node = shacl_g.value(shape_node, SH.minCount)
|
||||
if min_node is not None:
|
||||
try:
|
||||
min_count = int(str(min_node))
|
||||
except (TypeError, ValueError):
|
||||
min_count = None
|
||||
max_node = shacl_g.value(shape_node, SH.maxCount)
|
||||
if max_node is not None:
|
||||
try:
|
||||
max_count = int(str(max_node))
|
||||
except (TypeError, ValueError):
|
||||
max_count = None
|
||||
dt_node = shacl_g.value(shape_node, SH.datatype)
|
||||
datatype = str(dt_node) if dt_node is not None else None
|
||||
cls_node = shacl_g.value(shape_node, SH["class"])
|
||||
class_ = str(cls_node) if cls_node is not None else None
|
||||
|
||||
v = SHACLViolation(
|
||||
focus_node=focus,
|
||||
result_path=path,
|
||||
@@ -252,10 +216,6 @@ def _run_pyshacl(
|
||||
message=msg,
|
||||
value=val,
|
||||
shape=shape,
|
||||
min_count=min_count,
|
||||
max_count=max_count,
|
||||
datatype=datatype,
|
||||
class_=class_,
|
||||
)
|
||||
if sev_str == "Violation":
|
||||
violations.append(v)
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
"""The vocabulary Semantica's exporters emit terms from.
|
||||
|
||||
Every RDF export mints terms in ``https://semantica.dev/ns#``: ``sem:text``,
|
||||
``sem:confidence``, the default ``sem:Entity`` type, and the rest. Until this
|
||||
file existed, nothing declared what those terms meant, so a consumer receiving
|
||||
an export could not tell ``sem:text`` from a typo of it, and no closed-world
|
||||
check could be run against them at all (issue #1107).
|
||||
|
||||
The document ships inside the package so it can be loaded without a network
|
||||
round trip, and is the same file intended to be served at the namespace IRI.
|
||||
|
||||
>>> from semantica.ontology.vocabulary import vocabulary_turtle
|
||||
>>> ttl = vocabulary_turtle()
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
VOCABULARY_FILENAME = "semantica-ns.ttl"
|
||||
|
||||
#: The namespace the vocabulary declares terms in.
|
||||
NAMESPACE = "https://semantica.dev/ns#"
|
||||
|
||||
__all__ = ["NAMESPACE", "VOCABULARY_FILENAME", "vocabulary_path", "vocabulary_turtle"]
|
||||
|
||||
|
||||
def vocabulary_path() -> Path:
|
||||
"""Filesystem path to the vocabulary document."""
|
||||
return Path(__file__).parent / VOCABULARY_FILENAME
|
||||
|
||||
|
||||
def vocabulary_turtle() -> str:
|
||||
"""The vocabulary document as Turtle."""
|
||||
return vocabulary_path().read_text(encoding="utf-8")
|
||||
@@ -1,155 +0,0 @@
|
||||
@prefix owl: <http://www.w3.org/2002/07/owl#> .
|
||||
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
|
||||
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
||||
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
|
||||
@prefix dct: <http://purl.org/dc/terms/> .
|
||||
@prefix prov: <http://www.w3.org/ns/prov#> .
|
||||
@prefix time: <http://www.w3.org/2006/time#> .
|
||||
@prefix sem: <https://semantica.dev/ns#> .
|
||||
|
||||
<https://semantica.dev/ns> a owl:Ontology ;
|
||||
rdfs:label "Semantica vocabulary" ;
|
||||
rdfs:comment """Declares the terms the Semantica exporters emit in
|
||||
https://semantica.dev/ns#. Drafted from the emitting call sites in
|
||||
semantica 0.6.5: export/rdf_exporter.py, export/json_exporter.py and
|
||||
provenance/manager.py. Every term below appears in output the package
|
||||
produces today; no term has been invented for completeness.""" ;
|
||||
owl:versionInfo "0.1.0-draft" ;
|
||||
dct:created "2026-08-19"^^xsd:date .
|
||||
|
||||
# ── Classes ──────────────────────────────────────────────────────────────────
|
||||
|
||||
sem:Entity a owl:Class ;
|
||||
rdfs:label "Entity" ;
|
||||
rdfs:comment """The default type given to an extracted entity when the
|
||||
source carries no type of its own. Emitted by serialize_to_turtle as the
|
||||
fallback for entity.get("type").""" ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:Relationship a owl:Class ;
|
||||
rdfs:label "Relationship" ;
|
||||
rdfs:comment """A reified relationship, as emitted in the JSON-LD export
|
||||
where a relationship carries sem:type, sem:source and sem:target rather than
|
||||
being written as a single triple.""" ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:KnowledgeGraph a owl:Class ;
|
||||
rdfs:label "Knowledge Graph" ;
|
||||
rdfs:comment """The document-level type of a JSON-LD export: the @type of
|
||||
the top-level node carrying sem:entities, sem:relationships and
|
||||
sem:exportedAt. Emitted by _convert_kg_to_jsonld in export/json_exporter.py.""" ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
# ── Properties on an entity ──────────────────────────────────────────────────
|
||||
|
||||
sem:text a owl:DatatypeProperty ;
|
||||
rdfs:label "text" ;
|
||||
rdfs:comment """The surface text of an extracted entity. Carries the same
|
||||
intent as rdfs:label; declared separately because the exporters emit it under
|
||||
this IRI.""" ;
|
||||
rdfs:domain sem:Entity ;
|
||||
rdfs:range xsd:string ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:confidence a owl:DatatypeProperty ;
|
||||
rdfs:label "confidence" ;
|
||||
rdfs:comment """Extractor confidence in the assertion, on the unit interval.
|
||||
Emitted for both entities and relationships, so the domain is left open rather
|
||||
than tied to sem:Entity.
|
||||
|
||||
No rdfs:range is declared, deliberately. The Turtle serializer writes the value
|
||||
bare, which the Turtle grammar reads as xsd:decimal, while the N-Triples
|
||||
serializer types it xsd:float explicitly, and those two datatypes are disjoint.
|
||||
Declaring either one would make the vocabulary contradict one of the exporters.
|
||||
Issue #1100 tracks the disagreement; a range belongs here once the serializers
|
||||
agree on one.""" ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:metadata a owl:AnnotationProperty ;
|
||||
rdfs:label "metadata" ;
|
||||
rdfs:comment """Free-form metadata carried through from extraction. An
|
||||
annotation property because its value is an arbitrary structure rather than a
|
||||
modelled one.""" ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
# ── Relationship terms (JSON-LD export) ──────────────────────────────────────
|
||||
|
||||
sem:related_to a owl:ObjectProperty ;
|
||||
rdfs:label "related to" ;
|
||||
rdfs:comment """The default predicate for a relationship whose type the
|
||||
extractor did not determine. Deliberately unspecific: it asserts that two
|
||||
entities are connected and nothing about how.""" ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:source a owl:ObjectProperty ;
|
||||
rdfs:label "source" ;
|
||||
rdfs:comment "The subject entity of a reified relationship." ;
|
||||
rdfs:domain sem:Relationship ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:target a owl:ObjectProperty ;
|
||||
rdfs:label "target" ;
|
||||
rdfs:comment "The object entity of a reified relationship." ;
|
||||
rdfs:domain sem:Relationship ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:type a owl:DatatypeProperty ;
|
||||
rdfs:label "type" ;
|
||||
rdfs:comment """The relationship type as a label, as emitted in the JSON-LD
|
||||
export. Distinct from rdf:type, which relates a node to a class rather than to
|
||||
a string.""" ;
|
||||
rdfs:domain sem:Relationship ;
|
||||
rdfs:range xsd:string ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
# ── Document-level terms (JSON-LD export) ────────────────────────────────────
|
||||
|
||||
sem:entities a owl:ObjectProperty ;
|
||||
rdfs:label "entities" ;
|
||||
rdfs:comment "Ordered list of entities in an exported graph document." ;
|
||||
rdfs:range sem:Entity ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:relationships a owl:ObjectProperty ;
|
||||
rdfs:label "relationships" ;
|
||||
rdfs:comment "Ordered list of relationships in an exported graph document." ;
|
||||
rdfs:range sem:Relationship ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:exportedAt a owl:DatatypeProperty ;
|
||||
rdfs:label "exported at" ;
|
||||
rdfs:comment """When the export was written. Emitted as an ISO 8601 local
|
||||
timestamp, so the range is xsd:dateTime rather than xsd:dateTimeStamp: the
|
||||
values carry no timezone offset.""" ;
|
||||
rdfs:range xsd:dateTime ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:format a owl:DatatypeProperty ;
|
||||
rdfs:label "format" ;
|
||||
rdfs:comment """The serialization format label written on a JSON-LD
|
||||
document (currently always the literal "json-ld"). Emitted by
|
||||
JSONExporter.export_to_jsonld in export/json_exporter.py.""" ;
|
||||
rdfs:range xsd:string ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
# ── Temporal term (OWL-Time export) ──────────────────────────────────────────
|
||||
|
||||
sem:openEndedInterval a owl:DatatypeProperty ;
|
||||
rdfs:label "open ended interval" ;
|
||||
rdfs:comment """True when an interval has no known end. OWL-Time has no
|
||||
standard predicate for this, which is the reason the exporter mints one: an
|
||||
interval with no time:hasEnd is ambiguous between "ongoing" and "end not
|
||||
recorded", and this term resolves that in favour of the first.""" ;
|
||||
rdfs:domain time:Interval ;
|
||||
rdfs:range xsd:boolean ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
# ── Provenance roles ─────────────────────────────────────────────────────────
|
||||
|
||||
sem:role_generator a prov:Role ;
|
||||
rdfs:label "generator" ;
|
||||
rdfs:comment """The default role in a prov:qualifiedAssociation, used when
|
||||
an agent generated an entity rather than approving or reviewing it. Typed as
|
||||
prov:Role so that prov:hadRole has a declared value rather than an undeclared
|
||||
IRI.""" ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
@@ -243,14 +243,6 @@ class MediaParser:
|
||||
try:
|
||||
import subprocess
|
||||
|
||||
# ffprobe's own argument parser doesn't reliably honor a bare
|
||||
# "--" end-of-options marker, so a filename starting with "-"
|
||||
# could otherwise be parsed as an option; neutralize that by
|
||||
# forcing a relative-path prefix ffprobe can't mistake for a flag.
|
||||
ffprobe_path = str(file_path)
|
||||
if ffprobe_path.startswith("-"):
|
||||
ffprobe_path = f"./{ffprobe_path}"
|
||||
|
||||
result = subprocess.run(
|
||||
[
|
||||
"ffprobe",
|
||||
@@ -260,7 +252,7 @@ class MediaParser:
|
||||
"json",
|
||||
"-show_format",
|
||||
"-show_streams",
|
||||
ffprobe_path,
|
||||
str(file_path),
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
|
||||
@@ -501,11 +501,8 @@ class SeedDataManager:
|
||||
else:
|
||||
full_url = api_url
|
||||
|
||||
# Prepare headers — copy the caller's dict so we never mutate it in-place.
|
||||
# Without the copy, adding "Authorization" here would silently modify the
|
||||
# caller's original dict and potentially leak the key to subsequent calls
|
||||
# that reuse the same dict without expecting it to contain credentials.
|
||||
request_headers = dict(headers) if headers else {}
|
||||
# Prepare headers
|
||||
request_headers = headers or {}
|
||||
if api_key:
|
||||
request_headers["Authorization"] = f"Bearer {api_key}"
|
||||
|
||||
|
||||
@@ -144,12 +144,7 @@ class NERExtractor:
|
||||
self._ml_runtime_usable = True
|
||||
if "ml" in self.method and SPACY_AVAILABLE:
|
||||
try:
|
||||
# Deferred import: keeps semantic_extract.methods out of the
|
||||
# module-level import graph and routes loading through the
|
||||
# process-level cache so repeated NERExtractor constructions
|
||||
# never pay the ~120 ms spacy.load() cost more than once.
|
||||
from .methods import load_spacy_model
|
||||
self.nlp = load_spacy_model(self.model_name)
|
||||
self.nlp = spacy.load(self.model_name)
|
||||
except OSError:
|
||||
self.logger.warning(
|
||||
f"spaCy model {self.model_name} not found. ML method will fallback."
|
||||
|
||||
@@ -97,7 +97,7 @@ from .semantic_chunker import Chunk
|
||||
logger = get_logger("split_methods")
|
||||
|
||||
# Try to import optional dependencies
|
||||
_, SPACY_AVAILABLE = safe_import("spacy")
|
||||
spacy, SPACY_AVAILABLE = safe_import("spacy")
|
||||
|
||||
nltk, NLTK_AVAILABLE = safe_import("nltk")
|
||||
tiktoken, TIKTOKEN_AVAILABLE = safe_import("tiktoken")
|
||||
@@ -336,8 +336,7 @@ def split_by_sentences(
|
||||
# Try spaCy first
|
||||
if SPACY_AVAILABLE and kwargs.get("use_spacy", True):
|
||||
try:
|
||||
from ..semantic_extract.methods import load_spacy_model
|
||||
nlp = load_spacy_model("en_core_web_sm")
|
||||
nlp = spacy.load("en_core_web_sm")
|
||||
doc = nlp(text)
|
||||
sentences = [sent.text for sent in doc.sents]
|
||||
except Exception:
|
||||
|
||||
@@ -36,8 +36,7 @@ from ..utils.helpers import safe_import
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
|
||||
|
||||
_, SPACY_AVAILABLE = safe_import("spacy")
|
||||
spacy, SPACY_AVAILABLE = safe_import("spacy")
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -80,19 +79,11 @@ class SemanticChunker:
|
||||
if SPACY_AVAILABLE:
|
||||
model_name = config.get("model", "en_core_web_sm")
|
||||
try:
|
||||
from ..semantic_extract.methods import load_spacy_model
|
||||
self.nlp = load_spacy_model(model_name)
|
||||
self.nlp = spacy.load(model_name)
|
||||
except OSError:
|
||||
self.logger.warning(
|
||||
f"spaCy model {model_name} not found. Using fallback chunking."
|
||||
)
|
||||
except Exception:
|
||||
self.logger.warning(
|
||||
"spaCy model %s failed to initialize and will be disabled "
|
||||
"for this chunker instance. Using fallback chunking.",
|
||||
model_name,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
def chunk(self, text: str, **options) -> List[Chunk]:
|
||||
"""
|
||||
|
||||
@@ -383,9 +383,9 @@ class AnzoStore:
|
||||
if self._is_uri_value(obj):
|
||||
if obj.startswith("<") and obj.endswith(">"):
|
||||
inner = obj[1:-1]
|
||||
sparql_escaping.validate_uri(inner)
|
||||
if " " in inner or ">" in inner:
|
||||
raise ValueError(f"IRI contains invalid characters: {obj!r}")
|
||||
return obj
|
||||
sparql_escaping.validate_uri(obj)
|
||||
return f"<{obj}>"
|
||||
|
||||
escaped = sparql_escaping.escape_literal(obj)
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
"""Helpers for reading entity identifiers consistently across the KG pipeline."""
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
def get_entity_id(entity: Any) -> Any:
|
||||
"""Return a truthy identifier from either supported entity ID field.
|
||||
|
||||
The KG pipeline treats empty and otherwise falsy identifiers as missing.
|
||||
Prefer the canonical ``id`` field when it is populated, then fall back to
|
||||
the compatible ``entity_id`` alias.
|
||||
"""
|
||||
if isinstance(entity, dict):
|
||||
return entity.get("id") or entity.get("entity_id") or None
|
||||
|
||||
return getattr(entity, "id", None) or getattr(entity, "entity_id", None) or None
|
||||
@@ -398,7 +398,9 @@ def chunk_list(items: List[Any], chunk_size: int) -> List[List[Any]]:
|
||||
Returns:
|
||||
List of chunks
|
||||
"""
|
||||
return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]
|
||||
return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]
|
||||
|
||||
|
||||
def flatten_dict(
|
||||
d: Dict[str, Any], parent_key: str = "", sep: str = "."
|
||||
) -> Dict[str, Any]:
|
||||
@@ -412,32 +414,18 @@ def flatten_dict(
|
||||
|
||||
Returns:
|
||||
Flattened dictionary
|
||||
|
||||
Raises:
|
||||
ValueError: If two input paths produce the same flattened key.
|
||||
"""
|
||||
result = {}
|
||||
items = []
|
||||
|
||||
for k, v in d.items():
|
||||
new_key = f"{parent_key}{sep}{k}" if parent_key else k
|
||||
|
||||
if isinstance(v, dict):
|
||||
nested = flatten_dict(v, new_key, sep=sep)
|
||||
|
||||
for key, value in nested.items():
|
||||
if key in result:
|
||||
raise ValueError(
|
||||
f"Key collision while flattening dictionary: {key}"
|
||||
)
|
||||
result[key] = value
|
||||
items.extend(flatten_dict(v, new_key, sep=sep).items())
|
||||
else:
|
||||
if new_key in result:
|
||||
raise ValueError(
|
||||
f"Key collision while flattening dictionary: {new_key}"
|
||||
)
|
||||
result[new_key] = v
|
||||
items.append((new_key, v))
|
||||
|
||||
return result
|
||||
return dict(items)
|
||||
|
||||
|
||||
def get_nested_value(
|
||||
|
||||
@@ -1,142 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regression tests for the ContextGraph module docstring example.
|
||||
|
||||
The "Example Usage" block in ``semantica/context/context_graph.py`` previously
|
||||
called ``add_node``/``add_edge`` with keyword arguments those methods do not
|
||||
accept (``type=`` and ``properties=``), so the documented example raised
|
||||
``TypeError`` -- and the near-miss variants silently nested the properties dict
|
||||
instead of failing.
|
||||
|
||||
These tests keep the documented example executable and pin the two behaviours
|
||||
that made the original mistake easy to miss.
|
||||
"""
|
||||
|
||||
import doctest
|
||||
import re
|
||||
from typing import Dict, List
|
||||
|
||||
import pytest
|
||||
|
||||
import semantica.context.context_graph as context_graph_module
|
||||
from semantica.context.context_graph import ContextGraph
|
||||
|
||||
# The example block runs to the next top-level section header (a line starting
|
||||
# in column 0, e.g. "Production Use Cases:") or the end of the docstring.
|
||||
# Terminating on the next header rather than on a blank line keeps the capture
|
||||
# intact when the example gains blank lines or extra paragraphs.
|
||||
_EXAMPLE_BLOCK_RE = re.compile(r"^Example Usage:\n(.*?)(?=^\S|\Z)", re.DOTALL | re.MULTILINE)
|
||||
|
||||
# ``type=`` as its own keyword, but not the legitimate ``node_type=``/``edge_type=``.
|
||||
_BARE_TYPE_KWARG_RE = re.compile(r"(?<![\w])type\s*=")
|
||||
|
||||
|
||||
def _example_block() -> str:
|
||||
"""Return the 'Example Usage' block from the module docstring."""
|
||||
doc = context_graph_module.__doc__ or ""
|
||||
match = _EXAMPLE_BLOCK_RE.search(doc)
|
||||
assert match, "module docstring no longer contains an 'Example Usage:' block"
|
||||
block = match.group(1).strip()
|
||||
assert block, "the 'Example Usage:' block in the module docstring is empty"
|
||||
return block
|
||||
|
||||
|
||||
def _example_statements() -> List[str]:
|
||||
"""Return the documented ``>>>`` statements, continuation lines included."""
|
||||
statements = [example.source for example in doctest.DocTestParser().get_examples(_example_block())]
|
||||
assert statements, "the 'Example Usage:' block no longer contains any '>>>' statements"
|
||||
return statements
|
||||
|
||||
|
||||
def _statements_calling(method: str) -> List[str]:
|
||||
"""Return the documented statements that call ``graph.<method>(``."""
|
||||
return [stmt for stmt in _example_statements() if "graph.{}(".format(method) in stmt]
|
||||
|
||||
|
||||
def _run_example() -> Dict[str, object]:
|
||||
"""Execute the documented example verbatim and return its namespace."""
|
||||
source = "".join(_example_statements())
|
||||
namespace: Dict[str, object] = {}
|
||||
exec(compile(source, "<context_graph module docstring>", "exec"), namespace)
|
||||
return namespace
|
||||
|
||||
|
||||
class TestDocstringExampleIsRunnable:
|
||||
"""The documented example must execute exactly as written."""
|
||||
|
||||
def test_documented_calls_execute(self):
|
||||
# Run the docstring text itself so this test cannot drift from the docs.
|
||||
ns = _run_example()
|
||||
graph = ns["graph"]
|
||||
|
||||
assert "Python" in graph.nodes
|
||||
assert "Programming" in graph.nodes
|
||||
assert graph.nodes["Python"].node_type == "language"
|
||||
assert graph.nodes["Programming"].node_type == "concept"
|
||||
|
||||
neighbors = graph.get_neighbors("Python", hops=1)
|
||||
assert any(n["id"] == "Programming" for n in neighbors)
|
||||
|
||||
# record_decision must return a non-empty string ID.
|
||||
assert isinstance(ns["decision_id"], str) and ns["decision_id"]
|
||||
# find_precedents must be called with that ID and return a list.
|
||||
assert isinstance(ns["precedents"], list)
|
||||
|
||||
def test_node_properties_are_stored_flat(self):
|
||||
"""``popularity`` must land as a top-level property, not nested.
|
||||
|
||||
Passing the previously documented ``properties={...}`` does not raise --
|
||||
it stores a dict *inside* the properties dict, which is why the original
|
||||
docs bug could reach a user's graph unnoticed.
|
||||
"""
|
||||
graph = ContextGraph(advanced_analytics=False)
|
||||
graph.add_node("Python", "language", popularity="high")
|
||||
|
||||
assert graph.nodes["Python"].properties == {"popularity": "high"}
|
||||
assert graph.find_node("Python")["metadata"]["popularity"] == "high"
|
||||
assert "properties" not in graph.nodes["Python"].properties
|
||||
|
||||
def test_edge_type_is_positional_not_a_property(self):
|
||||
"""``related_to`` must be the edge type, not a stray metadata key."""
|
||||
graph = ContextGraph(advanced_analytics=False)
|
||||
graph.add_node("Python", "language")
|
||||
graph.add_node("Programming", "concept")
|
||||
graph.add_edge("Python", "Programming", "related_to")
|
||||
|
||||
edge = graph.edges[0]
|
||||
assert edge.edge_type == "related_to"
|
||||
assert "type" not in edge.metadata
|
||||
|
||||
|
||||
class TestDocstringExampleDoesNotRegress:
|
||||
"""Guard the docstring text itself, not just equivalent code."""
|
||||
|
||||
def test_add_node_example_supplies_node_type_positionally(self):
|
||||
calls = _statements_calling("add_node")
|
||||
assert calls, "the 'Example Usage:' block no longer calls graph.add_node()"
|
||||
for call in calls:
|
||||
assert not _BARE_TYPE_KWARG_RE.search(call), (
|
||||
f"add_node example passes type= as a keyword: {call!r}. "
|
||||
"node_type is positional-required; type= falls through to "
|
||||
"**properties and the call raises TypeError."
|
||||
)
|
||||
assert "properties=" not in call, (
|
||||
f"add_node example passes properties=: {call!r}. "
|
||||
"add_node has no properties parameter; extra properties are "
|
||||
"passed as **kwargs."
|
||||
)
|
||||
|
||||
def test_add_edge_example_supplies_edge_type_positionally(self):
|
||||
calls = _statements_calling("add_edge")
|
||||
assert calls, "the 'Example Usage:' block no longer calls graph.add_edge()"
|
||||
for call in calls:
|
||||
assert not _BARE_TYPE_KWARG_RE.search(call), (
|
||||
f"add_edge example passes type= as a keyword: {call!r}. "
|
||||
"The parameter is edge_type; type= is silently absorbed into "
|
||||
"**properties and pollutes edge metadata."
|
||||
)
|
||||
|
||||
def test_broken_form_still_raises(self):
|
||||
"""Pin the signature contract the example has to respect."""
|
||||
graph = ContextGraph(advanced_analytics=False)
|
||||
with pytest.raises(TypeError, match="node_type"):
|
||||
graph.add_node("Python", type="language", properties={"popularity": "high"})
|
||||
@@ -1,150 +0,0 @@
|
||||
"""Tests for ContextGraph.to_kg_dict() — the official KG-shape adapter.
|
||||
|
||||
These tests lock in the contract that to_kg_dict() emits the
|
||||
``{"entities", "relationships"}`` / ``source_id`` shape expected by
|
||||
downstream consumers (RDFExporter, TemporalGraphQuery.query_time_range),
|
||||
so users never need to hand-map field names.
|
||||
"""
|
||||
|
||||
from semantica.context.context_graph import ContextEdge, ContextGraph, ContextNode
|
||||
|
||||
|
||||
def _build_graph():
|
||||
g = ContextGraph()
|
||||
g._add_internal_node(ContextNode(node_id="e1", node_type="entity", content="Alice"))
|
||||
g._add_internal_node(ContextNode(node_id="e2", node_type="entity", content="Bob"))
|
||||
g._add_internal_node(
|
||||
ContextNode(node_id="c1", node_type="conversation", content="chat log")
|
||||
)
|
||||
g._add_internal_edge(
|
||||
ContextEdge(
|
||||
source_id="e1",
|
||||
target_id="e2",
|
||||
edge_type="knows",
|
||||
valid_from="2024-01-01",
|
||||
valid_until="2024-12-31",
|
||||
)
|
||||
)
|
||||
# Edge touching a non-entity node — used to test entities_only filtering.
|
||||
g._add_internal_edge(
|
||||
ContextEdge(source_id="c1", target_id="e1", edge_type="mentions")
|
||||
)
|
||||
return g
|
||||
|
||||
|
||||
def test_basic_shape():
|
||||
kg = _build_graph().to_kg_dict()
|
||||
assert set(kg.keys()) == {"entities", "relationships", "statistics"}
|
||||
# Entity shape uses id/text/type (not id/content).
|
||||
entity = next(e for e in kg["entities"] if e["id"] == "e1")
|
||||
assert entity["text"] == "Alice"
|
||||
assert entity["type"] == "entity"
|
||||
|
||||
|
||||
def test_relationship_uses_source_id_target_id():
|
||||
kg = _build_graph().to_kg_dict()
|
||||
rel = next(r for r in kg["relationships"] if r["type"] == "knows")
|
||||
assert rel["source_id"] == "e1"
|
||||
assert rel["target_id"] == "e2"
|
||||
# "source"/"target" (the internal names) must NOT leak through.
|
||||
assert "source" not in rel
|
||||
assert "target" not in rel
|
||||
|
||||
|
||||
def test_temporal_fields_passthrough():
|
||||
kg = _build_graph().to_kg_dict()
|
||||
rel = next(r for r in kg["relationships"] if r["type"] == "knows")
|
||||
assert rel["valid_from"] == "2024-01-01"
|
||||
assert rel["valid_until"] == "2024-12-31"
|
||||
|
||||
|
||||
def test_statistics_counts():
|
||||
kg = _build_graph().to_kg_dict()
|
||||
assert kg["statistics"]["entity_count"] == len(kg["entities"])
|
||||
assert kg["statistics"]["relationship_count"] == len(kg["relationships"])
|
||||
|
||||
|
||||
def test_entities_only_filters_nodes():
|
||||
kg = _build_graph().to_kg_dict(entities_only=True)
|
||||
types = {e["type"] for e in kg["entities"]}
|
||||
assert types == {"entity"}
|
||||
assert len(kg["entities"]) == 2
|
||||
|
||||
|
||||
def test_entities_only_drops_dangling_relationships():
|
||||
# The "mentions" edge points from a conversation node (filtered out under
|
||||
# entities_only) and must not appear as a dangling relationship.
|
||||
kg = _build_graph().to_kg_dict(entities_only=True)
|
||||
rel_types = {r["type"] for r in kg["relationships"]}
|
||||
assert "mentions" not in rel_types
|
||||
assert rel_types == {"knows"}
|
||||
|
||||
|
||||
def test_returned_dicts_are_isolated_from_internal_state():
|
||||
g = _build_graph()
|
||||
kg = g.to_kg_dict()
|
||||
entity = next(e for e in kg["entities"] if e["id"] == "e1")
|
||||
# Mutating the returned dict must not corrupt internal node properties.
|
||||
entity["properties"]["injected"] = True
|
||||
assert "injected" not in g.nodes["e1"].properties
|
||||
|
||||
|
||||
def test_null_properties_and_metadata_do_not_crash():
|
||||
"""Nodes loaded from JSON ``null`` keep None props/metadata; to_kg_dict
|
||||
must normalize them instead of raising TypeError (Qodo bug 1)."""
|
||||
g = ContextGraph()
|
||||
n = ContextNode(node_id="e1", node_type="entity", content="Alice")
|
||||
n.properties = None
|
||||
n.metadata = None
|
||||
g._add_internal_node(n)
|
||||
|
||||
kg = g.to_kg_dict()
|
||||
entity = kg["entities"][0]
|
||||
assert entity["properties"] == {}
|
||||
assert entity["metadata"] == {}
|
||||
|
||||
|
||||
def test_non_string_node_id_is_normalized_and_keeps_edges():
|
||||
"""ContextEdge coerces endpoints to str; entity ids must be coerced too
|
||||
so entities_only filtering does not drop valid edges (Qodo bug 3)."""
|
||||
g = ContextGraph()
|
||||
g._add_internal_node(ContextNode(node_id=1, node_type="entity", content="one"))
|
||||
g._add_internal_node(ContextNode(node_id=2, node_type="entity", content="two"))
|
||||
g._add_internal_edge(ContextEdge(source_id=1, target_id=2, edge_type="links"))
|
||||
|
||||
kg = g.to_kg_dict(entities_only=True)
|
||||
ids = {e["id"] for e in kg["entities"]}
|
||||
assert ids == {"1", "2"}
|
||||
assert all(isinstance(e["id"], str) for e in kg["entities"])
|
||||
# The edge must survive filtering despite the int-vs-str origin.
|
||||
assert {r["type"] for r in kg["relationships"]} == {"links"}
|
||||
|
||||
|
||||
def test_output_is_consumable_by_kg_utilities():
|
||||
"""to_kg_dict output must validate and be traversable by KG utilities that
|
||||
historically read ``source``/``target`` (Qodo bug 2, consumer side)."""
|
||||
from semantica.kg.graph_validator import GraphValidator, ValidationSeverity
|
||||
from semantica.kg.temporal_query import TemporalGraphQuery
|
||||
|
||||
g = ContextGraph()
|
||||
g._add_internal_node(ContextNode(node_id="e1", node_type="entity", content="Alice"))
|
||||
g._add_internal_node(ContextNode(node_id="e2", node_type="entity", content="Bob"))
|
||||
g._add_internal_edge(ContextEdge(source_id="e1", target_id="e2", edge_type="knows"))
|
||||
kg = g.to_kg_dict()
|
||||
|
||||
# Validator requires entity ``name``; add it so only endpoint compat is tested.
|
||||
for e in kg["entities"]:
|
||||
e["name"] = e["text"]
|
||||
|
||||
result = GraphValidator().validate(kg)
|
||||
endpoint_errors = [
|
||||
i for i in result.issues
|
||||
if i.code in {"MISSING_FIELD", "DANGLING_EDGE"}
|
||||
and i.element_type == "relationship"
|
||||
]
|
||||
assert endpoint_errors == [], endpoint_errors
|
||||
|
||||
# TemporalGraphQuery.analyze_evolution must see the relationship for "e1".
|
||||
tq = TemporalGraphQuery()
|
||||
filtered = tq.analyze_evolution(kg, entity="e1")
|
||||
assert filtered is not None
|
||||
@@ -1,141 +0,0 @@
|
||||
"""Minted IRIs must be stable and must sit in the declared namespace (issue #1101).
|
||||
|
||||
An entity that arrives without an id gets one minted for it. That identifier was
|
||||
built from Python's builtin ``hash()``, which is randomised per process, so the
|
||||
same entity received a different IRI on every run and exports could not be
|
||||
diffed, deduplicated against an earlier load, or joined to a provenance record
|
||||
written by an earlier process.
|
||||
|
||||
It was also written as ``semantica:entity_N`` inside angle brackets, which is an
|
||||
IRI in the scheme ``semantica`` rather than the expansion of the declared
|
||||
``semantica:`` prefix, so it never joined with anything written through it.
|
||||
"""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from semantica.export.rdf_exporter import (
|
||||
DEFAULT_ENTITY_TYPE,
|
||||
DEFAULT_RELATION_TYPE,
|
||||
RDFExporter,
|
||||
SEMANTICA_NS,
|
||||
mint_entity_iri,
|
||||
mint_relationship_iri,
|
||||
)
|
||||
|
||||
UNIDENTIFIED = {
|
||||
"entities": [{"text": "Acme Corp", "type": "https://example.org/Org"}],
|
||||
"relationships": [],
|
||||
}
|
||||
|
||||
|
||||
def test_minted_entity_iri_is_stable_within_a_process():
|
||||
assert mint_entity_iri("Acme Corp") == mint_entity_iri("Acme Corp")
|
||||
|
||||
|
||||
def test_minted_entity_iri_is_stable_across_processes():
|
||||
"""The regression that matters: identity must survive a restart."""
|
||||
script = (
|
||||
"from semantica.export.rdf_exporter import mint_entity_iri;"
|
||||
"print(mint_entity_iri('Acme Corp'))"
|
||||
)
|
||||
runs = {
|
||||
subprocess.run(
|
||||
[sys.executable, "-c", script],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
env={**os.environ, "PYTHONHASHSEED": seed},
|
||||
).stdout.strip()
|
||||
for seed in ("0", "1", "random")
|
||||
}
|
||||
assert len(runs) == 1, f"minted IRI differs between processes: {runs}"
|
||||
|
||||
|
||||
def test_minted_iris_are_in_the_declared_namespace():
|
||||
assert mint_entity_iri("Acme Corp").startswith(SEMANTICA_NS)
|
||||
assert mint_relationship_iri(0, "a", "b").startswith(SEMANTICA_NS)
|
||||
|
||||
|
||||
def test_distinct_entities_get_distinct_iris():
|
||||
assert mint_entity_iri("Acme Corp") != mint_entity_iri("Acme Corporation")
|
||||
|
||||
|
||||
def test_turtle_export_writes_a_resolvable_minted_iri():
|
||||
turtle = RDFExporter().export_to_rdf(UNIDENTIFIED, format="turtle")
|
||||
|
||||
assert f"<{SEMANTICA_NS}entity_" in turtle
|
||||
assert "<semantica:entity_" not in turtle, "scheme 'semantica' is not the prefix"
|
||||
|
||||
|
||||
def test_ntriples_export_agrees_with_turtle_on_the_minted_iri():
|
||||
exporter = RDFExporter()
|
||||
minted = mint_entity_iri("Acme Corp")
|
||||
|
||||
assert minted in exporter.export_to_rdf(UNIDENTIFIED, format="turtle")
|
||||
assert minted in exporter.export_to_rdf(UNIDENTIFIED, format="ntriples")
|
||||
|
||||
|
||||
def test_default_types_are_written_as_full_iris_in_turtle():
|
||||
untyped = {"entities": [{"id": "https://example.org/e1", "text": "A"}],
|
||||
"relationships": [{"source": "https://example.org/e1",
|
||||
"target": "https://example.org/e2"}]}
|
||||
turtle = RDFExporter().export_to_rdf(untyped, format="turtle")
|
||||
|
||||
assert f"<{DEFAULT_ENTITY_TYPE}>" in turtle
|
||||
assert f"<{DEFAULT_RELATION_TYPE}>" in turtle
|
||||
assert "<semantica:Entity>" not in turtle
|
||||
assert "<semantica:related_to>" not in turtle
|
||||
|
||||
|
||||
def test_default_entity_type_is_a_full_iri_in_rdfxml():
|
||||
"""RDF/XML's rdf:resource is an attribute value, not a QName context, so a
|
||||
|
||||
prefixed default there (``semantica:Entity``) resolves to the scheme
|
||||
``semantica`` rather than the declared namespace — the same failure mode
|
||||
fixed for Turtle in #1101, missed here because the original tests only
|
||||
checked Turtle output.
|
||||
"""
|
||||
untyped = {"entities": [{"id": "https://example.org/e1", "text": "A"}],
|
||||
"relationships": []}
|
||||
rdfxml = RDFExporter().export_to_rdf(untyped, format="rdfxml")
|
||||
|
||||
assert f'rdf:resource="{DEFAULT_ENTITY_TYPE}"' in rdfxml
|
||||
assert 'rdf:resource="semantica:Entity"' not in rdfxml
|
||||
|
||||
|
||||
def test_temporal_minting_uses_either_endpoint_representation():
|
||||
"""Relationships may carry source/target or source_id/target_id (#1109 review).
|
||||
|
||||
Minting from source_id alone hashed empty strings for every relationship
|
||||
that used the other representation, so once the IRI became deterministic,
|
||||
unrelated relationships at the same index collided on it and their temporal
|
||||
data aliased when the exports were loaded together.
|
||||
"""
|
||||
def temporal(rel):
|
||||
return RDFExporter().export_to_rdf(
|
||||
{"entities": [], "relationships": [rel]},
|
||||
format="turtle",
|
||||
include_temporal=True,
|
||||
)
|
||||
|
||||
a = temporal({"source": "https://example.org/a", "target": "https://example.org/b",
|
||||
"type": "https://example.org/worksFor", "valid_from": "2020-01-01T00:00:00Z"})
|
||||
b = temporal({"source": "https://example.org/c", "target": "https://example.org/d",
|
||||
"type": "https://example.org/worksFor", "valid_from": "2020-01-01T00:00:00Z"})
|
||||
|
||||
assert f"<{SEMANTICA_NS}rel_" in a
|
||||
assert a != b, "different endpoints must not mint the same temporal IRI"
|
||||
|
||||
|
||||
def test_temporal_minting_agrees_across_the_two_representations():
|
||||
"""The same relationship written either way is the same relationship."""
|
||||
def mint(rel):
|
||||
return mint_relationship_iri(
|
||||
0,
|
||||
rel.get("source_id") or rel.get("source") or "",
|
||||
rel.get("target_id") or rel.get("target") or "",
|
||||
)
|
||||
|
||||
assert mint({"source": "a", "target": "b"}) == mint({"source_id": "a", "target_id": "b"})
|
||||
@@ -1,35 +0,0 @@
|
||||
"""
|
||||
Shared pytest fixtures for the ingest test suite.
|
||||
|
||||
The ``mock_dns`` fixture is applied to *every* test in this directory
|
||||
(``autouse=True``). It stubs out ``socket.getaddrinfo`` inside the SSRF
|
||||
guard module so that unit tests that mock ``requests.Session.request`` do not
|
||||
accidentally hit the network for DNS resolution — which would fail in offline
|
||||
CI environments and cause intermittent timeouts.
|
||||
|
||||
Tests that explicitly need to exercise DNS-related behaviour (e.g. checking
|
||||
that a hostname resolving to a private IP is blocked) override this fixture
|
||||
by patching ``semantica.ingest.ssrf.socket.getaddrinfo`` with their own
|
||||
``side_effect`` *inside* the test body; that inner patch wins because
|
||||
``unittest.mock.patch`` applies patches in innermost-last order.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
_PUBLIC_IP = "93.184.216.34" # example.com — a safe, routable public address
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def mock_dns():
|
||||
"""Map every hostname to a safe public IP for the duration of each test."""
|
||||
with patch(
|
||||
"semantica.ingest.ssrf.socket.getaddrinfo",
|
||||
return_value=[
|
||||
(socket.AF_INET, socket.SOCK_STREAM, 6, "", (_PUBLIC_IP, 0))
|
||||
],
|
||||
):
|
||||
yield
|
||||
File diff suppressed because it is too large
Load Diff
@@ -10,20 +10,19 @@ class TestCookbookIntegration:
|
||||
|
||||
@pytest.fixture
|
||||
def mock_mcp_server(self):
|
||||
# MCPClient._send_request_http now routes through request_with_ssrf_guard,
|
||||
# which calls requests.request (not httpx.post / requests.post directly).
|
||||
# Patch at the point where the guard issues the actual HTTP call.
|
||||
with patch("requests.Session.request") as mock_request:
|
||||
|
||||
def side_effect(method, url, json=None, **kwargs):
|
||||
# We need to patch both httpx and requests because MCPClient tries httpx first
|
||||
with patch("httpx.post") as mock_httpx_post, \
|
||||
patch("requests.post") as mock_requests_post:
|
||||
|
||||
def side_effect(url, json=None, **kwargs):
|
||||
if not json:
|
||||
return MagicMock()
|
||||
|
||||
rpc_method = json.get("method")
|
||||
|
||||
method = json.get("method")
|
||||
response_mock = MagicMock()
|
||||
response_mock.status_code = 200
|
||||
|
||||
if rpc_method == "initialize":
|
||||
|
||||
if method == "initialize":
|
||||
response_mock.json.return_value = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": json.get("id"),
|
||||
@@ -33,7 +32,7 @@ class TestCookbookIntegration:
|
||||
"serverInfo": {"name": "test_server", "version": "1.0"}
|
||||
}
|
||||
}
|
||||
elif rpc_method == "resources/list":
|
||||
elif method == "resources/list":
|
||||
response_mock.json.return_value = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": json.get("id"),
|
||||
@@ -45,7 +44,7 @@ class TestCookbookIntegration:
|
||||
]
|
||||
}
|
||||
}
|
||||
elif rpc_method == "tools/list":
|
||||
elif method == "tools/list":
|
||||
response_mock.json.return_value = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": json.get("id"),
|
||||
@@ -57,7 +56,7 @@ class TestCookbookIntegration:
|
||||
]
|
||||
}
|
||||
}
|
||||
elif rpc_method == "resources/read":
|
||||
elif method == "resources/read":
|
||||
response_mock.json.return_value = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": json.get("id"),
|
||||
@@ -67,13 +66,13 @@ class TestCookbookIntegration:
|
||||
]
|
||||
}
|
||||
}
|
||||
elif rpc_method == "tools/call":
|
||||
elif method == "tools/call":
|
||||
tool_name = json.get("params", {}).get("name")
|
||||
content = [{"type": "text", "text": "Tool Output"}]
|
||||
|
||||
|
||||
if tool_name == "query_inventory":
|
||||
content = [{"type": "text", "text": '{"warehouse_id": "WH001", "level": 100}'}]
|
||||
|
||||
|
||||
response_mock.json.return_value = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": json.get("id"),
|
||||
@@ -87,11 +86,12 @@ class TestCookbookIntegration:
|
||||
"id": json.get("id"),
|
||||
"result": {}
|
||||
}
|
||||
|
||||
|
||||
return response_mock
|
||||
|
||||
mock_request.side_effect = side_effect
|
||||
yield mock_request
|
||||
|
||||
mock_httpx_post.side_effect = side_effect
|
||||
mock_requests_post.side_effect = side_effect
|
||||
yield mock_httpx_post
|
||||
|
||||
def test_financial_data_integration(self, mock_mcp_server):
|
||||
"""
|
||||
|
||||
@@ -222,7 +222,7 @@ def test_discover_feeds_empty() -> None:
|
||||
"semantica.ingest.ssrf.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", ("93.184.216.34", 0))],
|
||||
):
|
||||
with patch("requests.Session.request", side_effect=fake_request):
|
||||
with patch("requests.request", side_effect=fake_request):
|
||||
feeds = ingestor.discover_feeds("http://site.com")
|
||||
|
||||
assert len(feeds) == 0
|
||||
@@ -254,7 +254,7 @@ def test_discover_feeds_found() -> None:
|
||||
"semantica.ingest.ssrf.socket.getaddrinfo",
|
||||
return_value=[(2, 1, 6, "", ("93.184.216.34", 0))],
|
||||
):
|
||||
with patch("requests.Session.request", side_effect=fake_request):
|
||||
with patch("requests.request", side_effect=fake_request):
|
||||
feeds = ingestor.discover_feeds("http://site.com")
|
||||
|
||||
assert "http://site.com/rss.xml" in feeds
|
||||
|
||||
@@ -198,82 +198,15 @@ def test_public_api_detection_reports_auth_required() -> None:
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.ssrf.socket.getaddrinfo",
|
||||
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
|
||||
):
|
||||
detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
|
||||
"https://api.example.com/private"
|
||||
)
|
||||
detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
|
||||
"https://api.example.com/private"
|
||||
)
|
||||
|
||||
assert detection.is_public is False
|
||||
assert detection.requires_auth is True
|
||||
assert detection.response_status == 401
|
||||
|
||||
|
||||
def test_detect_public_api_propagates_ssrf_validation_error() -> None:
|
||||
"""detect_public_api() must surface ValidationError, not swallow it.
|
||||
|
||||
request_with_ssrf_guard() raises ValidationError (not
|
||||
requests.exceptions.RequestException) for SSRF-blocked hosts, so
|
||||
detect_public_api()'s error handling must catch it explicitly like its
|
||||
sibling ingest_public_api() already does.
|
||||
"""
|
||||
with patch("requests.Session") as mock_session_class:
|
||||
mock_session = mock_session_class.return_value
|
||||
mock_session.headers = {}
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.ssrf.socket.getaddrinfo",
|
||||
return_value=[(None, None, None, None, ("127.0.0.1", 0))],
|
||||
):
|
||||
with pytest.raises(ValidationError):
|
||||
PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
|
||||
"https://blocked.example.com/data"
|
||||
)
|
||||
|
||||
|
||||
def test_detect_public_api_rejects_duplicate_session_and_allow_private_ips_kwargs() -> None:
|
||||
"""Passing session/allow_private_ips through **options must not crash.
|
||||
|
||||
Both are always supplied explicitly to request_with_ssrf_guard(); caller
|
||||
copies must be dropped from **options rather than causing a
|
||||
'got multiple values for keyword argument' TypeError.
|
||||
"""
|
||||
with patch("requests.Session") as mock_session_class:
|
||||
mock_session = mock_session_class.return_value
|
||||
mock_session.headers = {}
|
||||
mock_session.request.return_value = _mock_response(
|
||||
headers={"Content-Type": "application/json"}
|
||||
)
|
||||
|
||||
detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
|
||||
"https://jsonplaceholder.typicode.com/posts",
|
||||
allow_private_ips=True,
|
||||
session=object(),
|
||||
)
|
||||
|
||||
assert detection.is_public is True
|
||||
|
||||
|
||||
def test_ingest_public_api_rejects_duplicate_session_and_allow_private_ips_kwargs() -> None:
|
||||
with patch("requests.Session") as mock_session_class:
|
||||
mock_session = mock_session_class.return_value
|
||||
mock_session.headers = {}
|
||||
mock_session.request.return_value = _mock_response(
|
||||
json_payload=[{"id": 1}],
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
|
||||
result = PublicAPIIngestor(rate_limit_delay=0).ingest_public_api(
|
||||
"https://jsonplaceholder.typicode.com/posts",
|
||||
allow_private_ips=True,
|
||||
session=object(),
|
||||
)
|
||||
|
||||
assert result.response_status == 200
|
||||
|
||||
|
||||
def test_public_api_ingestor_rejects_authentication_inputs() -> None:
|
||||
with patch("requests.Session") as mock_session_class:
|
||||
mock_session = mock_session_class.return_value
|
||||
@@ -305,14 +238,10 @@ def test_public_api_ingestor_parses_string_boolean_config() -> None:
|
||||
config={"validate_no_auth": "false"},
|
||||
rate_limit_delay=0,
|
||||
)
|
||||
with patch(
|
||||
"semantica.ingest.ssrf.socket.getaddrinfo",
|
||||
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
|
||||
):
|
||||
result = ingestor.ingest_public_api(
|
||||
"https://api.example.com/data",
|
||||
headers={"Authorization": "Bearer token"},
|
||||
)
|
||||
result = ingestor.ingest_public_api(
|
||||
"https://api.example.com/data",
|
||||
headers={"Authorization": "Bearer token"},
|
||||
)
|
||||
|
||||
assert ingestor.validate_no_auth is False
|
||||
assert result.data == payload
|
||||
|
||||
@@ -447,7 +447,7 @@ class TestSitemapCrawlerSSRF:
|
||||
redirect.close = MagicMock()
|
||||
|
||||
with patch(
|
||||
"requests.Session.request", return_value=redirect
|
||||
"semantica.ingest.ssrf.requests.request", return_value=redirect
|
||||
), patch(
|
||||
"semantica.ingest.ssrf.socket.getaddrinfo",
|
||||
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
|
||||
|
||||
@@ -195,62 +195,89 @@ class TestMCPIngestor:
|
||||
|
||||
class TestMCPClient:
|
||||
def test_call_tool(self):
|
||||
# MCPClient._send_request_http now routes through request_with_ssrf_guard,
|
||||
# which calls requests.request (not requests.post) with allow_redirects=False.
|
||||
# Patch the requests.request call inside ssrf.py.
|
||||
with patch("requests.Session.request") as mock_request:
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
|
||||
# Response for initialize
|
||||
init_response = {
|
||||
"jsonrpc": "2.0",
|
||||
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
|
||||
"id": 1,
|
||||
}
|
||||
|
||||
# Response for tool call
|
||||
tool_response = {
|
||||
"jsonrpc": "2.0",
|
||||
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
|
||||
"id": 2,
|
||||
}
|
||||
|
||||
mock_response.json.side_effect = [init_response, tool_response]
|
||||
mock_request.return_value = mock_response
|
||||
|
||||
client = MCPClient(url="http://localhost:8000")
|
||||
client.connect()
|
||||
|
||||
client.call_tool("my_tool", {"arg": "val"})
|
||||
# Patch requests.post globally if requests is used, or httpx.post if httpx is used.
|
||||
# The code tries importing httpx, then requests.
|
||||
# We should patch both or ensure we catch the right one.
|
||||
# Simpler to patch sys.modules to simulate httpx missing, then patch requests.
|
||||
|
||||
with patch.dict(sys.modules, {'httpx': None}):
|
||||
with patch("requests.post") as mock_post:
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
|
||||
# Sequence of calls:
|
||||
# 1. connect() calls _connect_http() -> calls _initialize() -> calls _send_request()
|
||||
# _send_request() calls requests.post with method="initialize"
|
||||
# 2. call_tool() calls _send_request() with method="tools/call"
|
||||
|
||||
# Response for initialize
|
||||
init_response = {
|
||||
"jsonrpc": "2.0",
|
||||
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
|
||||
"id": 1
|
||||
}
|
||||
|
||||
# Response for tool call
|
||||
tool_response = {
|
||||
"jsonrpc": "2.0",
|
||||
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
|
||||
"id": 2
|
||||
}
|
||||
|
||||
mock_response.json.side_effect = [init_response, tool_response]
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
client = MCPClient(url="http://localhost:8000")
|
||||
client.connect()
|
||||
|
||||
result = client.call_tool("my_tool", {"arg": "val"})
|
||||
|
||||
# result is the dict returned by tool call?
|
||||
# call_tool returns dict?
|
||||
# Check MCPClient.call_tool implementation
|
||||
# It calls _send_request, which returns response.json().
|
||||
# But wait, call_tool might process the result.
|
||||
# Let's check call_tool implementation in mcp_client.py (not read yet, but assumed).
|
||||
# Wait, I read mcp_client.py but didn't check call_tool specifically.
|
||||
# Assuming call_tool returns result part or whole response.
|
||||
|
||||
# Actually, let's verify call_tool in mcp_client.py
|
||||
pass
|
||||
|
||||
def test_call_tool_mock_check(self):
|
||||
# Redo with the corrected patch target.
|
||||
with patch("requests.Session.request") as mock_request:
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
|
||||
init_response = {
|
||||
"jsonrpc": "2.0",
|
||||
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
|
||||
"id": 1,
|
||||
}
|
||||
|
||||
tool_response = {
|
||||
"jsonrpc": "2.0",
|
||||
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
|
||||
"id": 2,
|
||||
}
|
||||
|
||||
mock_response.json.side_effect = [init_response, tool_response]
|
||||
mock_request.return_value = mock_response
|
||||
|
||||
client = MCPClient(url="http://localhost:8000")
|
||||
client.connect()
|
||||
|
||||
result = client.call_tool("my_tool", {"arg": "val"})
|
||||
|
||||
assert result["content"] == [{"type": "text", "text": "Tool Result"}]
|
||||
# Redoing the test with more specific mocking logic
|
||||
with patch.dict(sys.modules, {'httpx': None}):
|
||||
with patch("requests.post") as mock_post:
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 200
|
||||
|
||||
# initialize response
|
||||
init_response = {
|
||||
"jsonrpc": "2.0",
|
||||
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
|
||||
"id": 1
|
||||
}
|
||||
|
||||
# tool call response - Assuming call_tool returns the 'result' part of JSON-RPC response
|
||||
# If call_tool implementation wraps it, we need to know.
|
||||
# Let's assume standard behavior for now.
|
||||
tool_response = {
|
||||
"jsonrpc": "2.0",
|
||||
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
|
||||
"id": 2
|
||||
}
|
||||
|
||||
mock_response.json.side_effect = [init_response, tool_response]
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
client = MCPClient(url="http://localhost:8000")
|
||||
client.connect()
|
||||
|
||||
result = client.call_tool("my_tool", {"arg": "val"})
|
||||
|
||||
# Verify result.
|
||||
# If call_tool returns the 'result' dict from JSON-RPC:
|
||||
assert result["content"] == [{"type": "text", "text": "Tool Result"}]
|
||||
|
||||
class TestGDriveIngestor:
|
||||
def test_init_raises_if_no_google_libs(self):
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
|
||||
import pytest
|
||||
from semantica.utils.types import Entity
|
||||
from semantica.kg.graph_builder import GraphBuilder
|
||||
from semantica.kg.entity_resolver import EntityResolver
|
||||
from semantica.kg.graph_analyzer import GraphAnalyzer
|
||||
from semantica.utils.entity_ids import get_entity_id
|
||||
from semantica.utils.types import Entity
|
||||
|
||||
def test_full_entity_pipeline():
|
||||
"""
|
||||
@@ -108,60 +107,6 @@ def test_direct_entity_objects_in_analyzer():
|
||||
|
||||
print("Direct Entity objects test passed successfully!")
|
||||
|
||||
|
||||
def test_entity_id_only_merge_remaps_relationship_endpoints():
|
||||
"""Entity aliases must survive merging and relationship remapping."""
|
||||
builder = GraphBuilder(
|
||||
merge_entities=True,
|
||||
entity_resolution_strategy="exact",
|
||||
resolve_conflicts=False,
|
||||
)
|
||||
|
||||
graph = builder.build(
|
||||
{
|
||||
"entities": [
|
||||
{"entity_id": "alice:1", "name": "Alice", "type": "Person"},
|
||||
{"entity_id": "alice:2", "name": " Alice ", "type": "Person"},
|
||||
{"entity_id": "org:1", "name": "Acme", "type": "Organization"},
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"source_id": "alice:2",
|
||||
"target_id": "org:1",
|
||||
"type": "WORKS_FOR",
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
merged_alice = next(
|
||||
entity for entity in graph["entities"] if entity["name"] == "Alice"
|
||||
)
|
||||
relationship = graph["relationships"][0]
|
||||
entity_ids = {
|
||||
entity.get("id") or entity.get("entity_id") for entity in graph["entities"]
|
||||
}
|
||||
|
||||
assert merged_alice["id"] == "alice:1"
|
||||
assert set(merged_alice["merged_from"]) == {"alice:1", "alice:2"}
|
||||
assert {
|
||||
item["id"] for item in merged_alice["metadata"]["provenance"]["merged_from"]
|
||||
} == {"alice:1", "alice:2"}
|
||||
assert relationship["source"] == "alice:1"
|
||||
assert relationship["target"] == "org:1"
|
||||
assert {relationship["source"], relationship["target"]} <= entity_ids
|
||||
|
||||
|
||||
def test_entity_id_helper_ignores_falsy_identifiers():
|
||||
"""ID extraction must match the KG pipeline's falsy-ID contract."""
|
||||
assert get_entity_id({"id": "", "entity_id": "alias:1"}) == "alias:1"
|
||||
assert get_entity_id({"id": 0, "entity_id": "alias:2"}) == "alias:2"
|
||||
assert (
|
||||
get_entity_id({"id": "primary:1", "entity_id": "alias:3"})
|
||||
== "primary:1"
|
||||
)
|
||||
assert get_entity_id({"id": "", "entity_id": 0}) is None
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_full_entity_pipeline()
|
||||
test_direct_entity_objects_in_analyzer()
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
from semantica.kg.entity_resolver import EntityResolver
|
||||
|
||||
|
||||
def test_exact_resolution_does_not_merge_similar_names():
|
||||
entities = [
|
||||
{"id": "1", "name": "Alice"},
|
||||
{"id": "2", "name": "Alicia"},
|
||||
]
|
||||
|
||||
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
|
||||
|
||||
assert {entity["id"] for entity in resolved} == {"1", "2"}
|
||||
|
||||
|
||||
def test_exact_resolution_merges_case_and_whitespace_variants():
|
||||
entities = [
|
||||
{"id": "1", "name": " Alice "},
|
||||
{"id": "2", "name": "alice"},
|
||||
]
|
||||
|
||||
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
|
||||
|
||||
assert len(resolved) == 1
|
||||
|
||||
|
||||
def test_resolution_preserves_non_duplicate_entities_without_ids():
|
||||
entities = [
|
||||
{"name": "Alice"},
|
||||
{"name": "Bob"},
|
||||
]
|
||||
|
||||
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
|
||||
|
||||
assert resolved == entities
|
||||
|
||||
|
||||
def test_exact_resolution_does_not_merge_whitespace_only_names():
|
||||
entities = [
|
||||
{"id": "1", "name": " "},
|
||||
{"id": "2", "name": "\t"},
|
||||
]
|
||||
|
||||
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
|
||||
|
||||
assert {entity["id"] for entity in resolved} == {"1", "2"}
|
||||
|
||||
|
||||
def test_exact_resolution_falls_back_to_text_when_name_is_blank():
|
||||
entities = [
|
||||
{"id": "1", "name": " ", "text": "Alice"},
|
||||
{"id": "2", "name": "Alice"},
|
||||
]
|
||||
|
||||
resolved = EntityResolver(strategy="exact").resolve_entities(entities)
|
||||
|
||||
assert len(resolved) == 1
|
||||
@@ -414,103 +414,6 @@ class TestSHACLHierarchicalAndValidation(unittest.TestCase):
|
||||
self.assertIsNotNone(v.explanation)
|
||||
self.assertIn("https://example.com/john", v.explanation)
|
||||
|
||||
# 32b
|
||||
def test_explain_violations_uses_real_constraint_values(self):
|
||||
"""explain_violations must render the real min/max/datatype/class values,
|
||||
not hardcoded placeholders (regression for PR #318)."""
|
||||
from semantica.ontology.ontology_validator import (
|
||||
SHACLValidationReport,
|
||||
SHACLViolation,
|
||||
)
|
||||
|
||||
max_v = SHACLViolation(
|
||||
focus_node="https://example.com/john",
|
||||
result_path="ex:age",
|
||||
constraint="MaxCountConstraintComponent",
|
||||
max_count=3,
|
||||
)
|
||||
dt_v = SHACLViolation(
|
||||
focus_node="https://example.com/john",
|
||||
result_path="ex:age",
|
||||
constraint="DatatypeConstraintComponent",
|
||||
value="abc",
|
||||
datatype="http://www.w3.org/2001/XMLSchema#integer",
|
||||
)
|
||||
cls_v = SHACLViolation(
|
||||
focus_node="https://example.com/john",
|
||||
result_path="ex:knows",
|
||||
constraint="ClassConstraintComponent",
|
||||
value="https://example.com/thing",
|
||||
class_="https://example.com/Person",
|
||||
)
|
||||
report = SHACLValidationReport(
|
||||
conforms=False, violations=[max_v, dt_v, cls_v]
|
||||
)
|
||||
report.explain_violations()
|
||||
# MaxCount must show the real limit (3), not the hardcoded 1.
|
||||
self.assertIn("3", max_v.explanation)
|
||||
self.assertNotIn("At most 1 value", max_v.explanation)
|
||||
# Datatype must show the real datatype IRI, not the message.
|
||||
self.assertIn(
|
||||
"http://www.w3.org/2001/XMLSchema#integer", dt_v.explanation
|
||||
)
|
||||
# Class must show the real class IRI.
|
||||
self.assertIn("https://example.com/Person", cls_v.explanation)
|
||||
|
||||
# 32c
|
||||
def test_run_pyshacl_extracts_constraint_values_from_shape(self):
|
||||
"""_run_pyshacl must back-reference sh:sourceShape to populate the real
|
||||
constraint parameters on each violation."""
|
||||
try:
|
||||
import pyshacl # noqa: F401
|
||||
import rdflib # noqa: F401
|
||||
except ImportError:
|
||||
self.skipTest("pyshacl/rdflib not installed")
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
|
||||
shacl = """
|
||||
@prefix sh: <http://www.w3.org/ns/shacl#> .
|
||||
@prefix ex: <http://example.com/> .
|
||||
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
|
||||
|
||||
ex:PersonShape a sh:NodeShape ;
|
||||
sh:targetClass ex:Person ;
|
||||
sh:property [
|
||||
sh:path ex:age ;
|
||||
sh:datatype xsd:integer ;
|
||||
sh:maxCount 2 ;
|
||||
] .
|
||||
"""
|
||||
data = """
|
||||
@prefix ex: <http://example.com/> .
|
||||
ex:john a ex:Person ;
|
||||
ex:age "not-a-number" ;
|
||||
ex:age 1 ;
|
||||
ex:age 2 ;
|
||||
ex:age 3 .
|
||||
"""
|
||||
report = _run_pyshacl(data, shacl)
|
||||
self.assertFalse(report.conforms)
|
||||
# Datatype violation should carry the real xsd:integer datatype.
|
||||
dt = [
|
||||
v
|
||||
for v in report.violations
|
||||
if v.constraint == "DatatypeConstraintComponent"
|
||||
]
|
||||
self.assertTrue(dt)
|
||||
self.assertTrue(
|
||||
dt[0].datatype.endswith("integer"),
|
||||
f"expected integer datatype, got {dt[0].datatype}",
|
||||
)
|
||||
# MaxCount violation should carry the real max_count == 2.
|
||||
mc = [
|
||||
v
|
||||
for v in report.violations
|
||||
if v.constraint == "MaxCountConstraintComponent"
|
||||
]
|
||||
if mc:
|
||||
self.assertEqual(mc[0].max_count, 2)
|
||||
|
||||
# 33
|
||||
def test_shacl_violation_to_dict(self):
|
||||
from semantica.ontology.ontology_validator import SHACLViolation
|
||||
|
||||
@@ -1,116 +0,0 @@
|
||||
"""The vocabulary must stay true to what the exporters emit (issue #1107).
|
||||
|
||||
A vocabulary document that drifts from the code is worse than none, because it
|
||||
states that terms mean something while the exporters emit different ones. These
|
||||
tests tie the two together: every term the serializers can write must be
|
||||
declared here, so adding a term to an exporter without declaring it fails the
|
||||
build rather than shipping an undeclared IRI.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
rdflib = pytest.importorskip("rdflib")
|
||||
|
||||
from semantica.export.rdf_exporter import ( # noqa: E402
|
||||
DEFAULT_ENTITY_TYPE,
|
||||
DEFAULT_RELATION_TYPE,
|
||||
SEMANTICA_NS,
|
||||
)
|
||||
from semantica.ontology.vocabulary import ( # noqa: E402
|
||||
NAMESPACE,
|
||||
vocabulary_path,
|
||||
vocabulary_turtle,
|
||||
)
|
||||
|
||||
#: Every term the exporters emit in the Semantica namespace, by local name.
|
||||
#: RDF and OWL-Time paths in export/rdf_exporter.py, document and relationship
|
||||
#: terms in export/json_exporter.py, roles in provenance/manager.py.
|
||||
EMITTED_TERMS = {
|
||||
"Entity",
|
||||
"Relationship",
|
||||
"KnowledgeGraph",
|
||||
"text",
|
||||
"confidence",
|
||||
"metadata",
|
||||
"related_to",
|
||||
"source",
|
||||
"target",
|
||||
"type",
|
||||
"entities",
|
||||
"relationships",
|
||||
"exportedAt",
|
||||
"format",
|
||||
"openEndedInterval",
|
||||
"role_generator",
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def graph():
|
||||
g = rdflib.Graph()
|
||||
g.parse(data=vocabulary_turtle(), format="turtle")
|
||||
return g
|
||||
|
||||
|
||||
def test_vocabulary_ships_with_the_package():
|
||||
assert vocabulary_path().is_file()
|
||||
|
||||
|
||||
def test_vocabulary_parses(graph):
|
||||
assert len(graph) > 0
|
||||
|
||||
|
||||
def test_namespace_matches_the_one_the_exporters_use():
|
||||
assert NAMESPACE == SEMANTICA_NS
|
||||
|
||||
|
||||
def test_every_emitted_term_is_declared(graph):
|
||||
declared = {
|
||||
str(s)[len(NAMESPACE) :]
|
||||
for s in set(graph.subjects())
|
||||
if isinstance(s, rdflib.URIRef) and str(s).startswith(NAMESPACE)
|
||||
}
|
||||
missing = EMITTED_TERMS - declared
|
||||
assert not missing, f"emitted but not declared in the vocabulary: {sorted(missing)}"
|
||||
|
||||
|
||||
def test_the_defaults_the_exporters_fall_back_to_are_declared(graph):
|
||||
for iri in (DEFAULT_ENTITY_TYPE, DEFAULT_RELATION_TYPE):
|
||||
assert (rdflib.URIRef(iri), None, None) in graph, f"{iri} is not declared"
|
||||
|
||||
|
||||
def test_every_declared_term_carries_a_label_and_a_comment(graph):
|
||||
for subject in set(graph.subjects()):
|
||||
if not (isinstance(subject, rdflib.URIRef) and str(subject).startswith(NAMESPACE)):
|
||||
continue
|
||||
assert graph.value(subject, rdflib.RDFS.label), f"{subject} has no rdfs:label"
|
||||
assert graph.value(subject, rdflib.RDFS.comment), f"{subject} has no rdfs:comment"
|
||||
|
||||
|
||||
def test_declared_ranges_do_not_contradict_what_the_exporters_emit(graph):
|
||||
"""A declared range must match the datatype the serializers actually write.
|
||||
|
||||
Caught by review on #1109: sem:confidence was declared xsd:decimal while the
|
||||
N-Triples serializer types the same value xsd:float. A vocabulary that
|
||||
contradicts the code is worse than no vocabulary, so any range declared here
|
||||
has to be one the exporters really emit.
|
||||
"""
|
||||
import re
|
||||
|
||||
from semantica.export.rdf_exporter import RDFExporter
|
||||
|
||||
sample = {
|
||||
"entities": [{"id": "https://example.org/e1", "text": "A",
|
||||
"type": "https://example.org/T", "confidence": 0.5}],
|
||||
"relationships": [],
|
||||
}
|
||||
emitted = RDFExporter().export_to_rdf(sample, format="ntriples")
|
||||
|
||||
for subject, _, range_ in graph.triples((None, rdflib.RDFS.range, None)):
|
||||
if not str(subject).startswith(NAMESPACE):
|
||||
continue
|
||||
local = str(subject)[len(NAMESPACE):]
|
||||
for match in re.finditer(rf'<{NAMESPACE}{local}> "[^"]*"\^\^<([^>]+)>', emitted):
|
||||
assert match.group(1) == str(range_), (
|
||||
f"{local}: vocabulary declares {range_}, N-Triples emits {match.group(1)}"
|
||||
)
|
||||
@@ -1,321 +0,0 @@
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from semantica.semantic_extract import methods as se_methods
|
||||
from semantica.split import methods as split_methods
|
||||
from semantica.split import semantic_chunker
|
||||
from semantica.semantic_extract import ner_extractor as ner_extractor_module
|
||||
from semantica.semantic_extract.ner_extractor import NERExtractor
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clear_cache():
|
||||
se_methods.clear_spacy_model_cache()
|
||||
yield
|
||||
se_methods.clear_spacy_model_cache()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def force_spacy_available(monkeypatch):
|
||||
# split.methods, split.semantic_chunker, and ner_extractor each compute
|
||||
# their own SPACY_AVAILABLE flag from the real environment at import time;
|
||||
# force all true so these tests exercise the spaCy branch regardless of
|
||||
# whether spaCy is actually installed where they run.
|
||||
monkeypatch.setattr(split_methods, "SPACY_AVAILABLE", True)
|
||||
monkeypatch.setattr(semantic_chunker, "SPACY_AVAILABLE", True)
|
||||
monkeypatch.setattr(ner_extractor_module, "SPACY_AVAILABLE", True)
|
||||
|
||||
|
||||
def _fake_spacy(load):
|
||||
return SimpleNamespace(
|
||||
load=load,
|
||||
util=SimpleNamespace(is_package=lambda _name: True),
|
||||
)
|
||||
|
||||
|
||||
def _nlp_mock(sentences=("Hello world.",)):
|
||||
"""A stand-in spaCy Language object: callable, returns a doc with .sents."""
|
||||
nlp = MagicMock()
|
||||
nlp.return_value = SimpleNamespace(
|
||||
sents=[SimpleNamespace(text=s) for s in sentences]
|
||||
)
|
||||
return nlp
|
||||
|
||||
|
||||
class TestSpacyModelCache:
|
||||
"""split.methods and split.semantic_chunker must share the cached model
|
||||
defined in semantic_extract.methods instead of each calling spacy.load()
|
||||
independently.
|
||||
"""
|
||||
|
||||
def test_split_by_sentences_reuses_cached_model(self, monkeypatch):
|
||||
calls = []
|
||||
|
||||
def fake_load(name, **kwargs):
|
||||
calls.append((name, kwargs))
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
split_methods.split_by_sentences("Hello world. Bye world.")
|
||||
split_methods.split_by_sentences("Another sentence here.")
|
||||
split_methods.split_by_sentences("A third call.")
|
||||
|
||||
assert len(calls) == 1, "spacy.load should run once, not once per call"
|
||||
assert calls[0][0] == "en_core_web_sm"
|
||||
|
||||
def test_semantic_chunker_reuses_cached_model_across_instances(self, monkeypatch):
|
||||
calls = []
|
||||
|
||||
def fake_load(name, **kwargs):
|
||||
calls.append((name, kwargs))
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
chunker1 = semantic_chunker.SemanticChunker()
|
||||
chunker2 = semantic_chunker.SemanticChunker()
|
||||
|
||||
assert len(calls) == 1, "each new SemanticChunker should not reload the model"
|
||||
assert chunker1.nlp is chunker2.nlp
|
||||
|
||||
def test_split_methods_and_semantic_chunker_share_the_cache(self, monkeypatch):
|
||||
calls = []
|
||||
|
||||
def fake_load(name, **kwargs):
|
||||
calls.append((name, kwargs))
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
split_methods.split_by_sentences("Test sentence for split.methods.")
|
||||
semantic_chunker.SemanticChunker()
|
||||
|
||||
assert len(calls) == 1, (
|
||||
"split.methods and split.semantic_chunker must share one cached "
|
||||
"model instead of each loading their own"
|
||||
)
|
||||
|
||||
def test_distinct_model_names_load_separately(self, monkeypatch):
|
||||
calls = []
|
||||
|
||||
def fake_load(name, **kwargs):
|
||||
calls.append((name, kwargs))
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
sm_chunker = semantic_chunker.SemanticChunker(model="en_core_web_sm")
|
||||
lg_chunker = semantic_chunker.SemanticChunker(model="en_core_web_lg")
|
||||
sm_chunker_again = semantic_chunker.SemanticChunker(model="en_core_web_sm")
|
||||
|
||||
assert [name for name, _ in calls] == ["en_core_web_sm", "en_core_web_lg"]
|
||||
assert sm_chunker.nlp is sm_chunker_again.nlp
|
||||
assert sm_chunker.nlp is not lg_chunker.nlp
|
||||
|
||||
def test_no_disable_kwarg_requested(self, monkeypatch):
|
||||
"""split.methods and split.semantic_chunker both want the full
|
||||
pipeline (they need .sents, which requires the parser/senter). If
|
||||
either one later starts requesting a trimmed pipeline (e.g.
|
||||
disable=["ner"]), the name-only cache key in load_spacy_model would
|
||||
silently hand back a cached model built for a different config --
|
||||
this test should catch that the moment it happens.
|
||||
"""
|
||||
calls = []
|
||||
|
||||
def fake_load(_name, **kwargs):
|
||||
calls.append(kwargs)
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
split_methods.split_by_sentences("Hello world.")
|
||||
se_methods.clear_spacy_model_cache()
|
||||
semantic_chunker.SemanticChunker()
|
||||
|
||||
assert len(calls) == 2
|
||||
assert all(kwargs == {} for kwargs in calls), (
|
||||
"neither caller should pass any pipeline-configuration kwargs; "
|
||||
"the name-only cache key in load_spacy_model cannot distinguish "
|
||||
"models loaded with different component configs"
|
||||
)
|
||||
|
||||
def test_missing_model_falls_back_without_poisoning_cache(self, monkeypatch):
|
||||
attempts = []
|
||||
|
||||
def failing_load(name, **_kwargs):
|
||||
attempts.append(name)
|
||||
raise OSError(f"Can't find model '{name}'")
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load))
|
||||
|
||||
# split_by_sentences should fall back to regex splitting, not raise
|
||||
chunks = split_methods.split_by_sentences("Hello world. Bye world.")
|
||||
assert chunks, "fallback splitting should still produce chunks"
|
||||
|
||||
# SemanticChunker should leave .nlp as None rather than propagate
|
||||
chunker = semantic_chunker.SemanticChunker()
|
||||
assert chunker.nlp is None
|
||||
|
||||
assert len(attempts) == 2, "a failed load must not be cached"
|
||||
|
||||
# Once the model is available, both callers should now get it, and
|
||||
# share a single successful load.
|
||||
def working_load(name, **_kwargs):
|
||||
attempts.append(name)
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load))
|
||||
|
||||
chunker2 = semantic_chunker.SemanticChunker()
|
||||
split_methods.split_by_sentences("One more sentence.")
|
||||
|
||||
assert len(attempts) == 3, (
|
||||
"the model should load once after it becomes available"
|
||||
)
|
||||
assert chunker2.nlp is not None
|
||||
|
||||
def test_semantic_chunker_falls_back_when_spacy_runtime_is_broken(
|
||||
self, monkeypatch
|
||||
):
|
||||
"""A spaCy model that is installed but unusable at runtime (e.g. a
|
||||
config incompatible with the installed spaCy version) must degrade
|
||||
SemanticChunker to fallback chunking, not crash __init__ -- mirrors
|
||||
TestNERExtractorSpacyModelCache's equivalent broken-runtime test.
|
||||
"""
|
||||
|
||||
def broken_load(name, **_kwargs):
|
||||
raise RuntimeError("ConfigSchemaNlp is not fully defined")
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(broken_load))
|
||||
|
||||
chunker = semantic_chunker.SemanticChunker()
|
||||
|
||||
assert chunker.nlp is None
|
||||
|
||||
|
||||
class TestNERExtractorSpacyModelCache:
|
||||
"""NERExtractor(method="ml") must reuse the centralized cache in
|
||||
semantic_extract.methods, not call spacy.load() on every construction.
|
||||
|
||||
These tests mirror TestSpacyModelCache but focus on the NERExtractor path,
|
||||
confirming that all three callers (split_by_sentences, SemanticChunker, and
|
||||
NERExtractor) draw from the same process-level cache.
|
||||
"""
|
||||
|
||||
def test_ner_extractor_reuses_cached_model_across_instances(self, monkeypatch):
|
||||
"""Two NERExtractor(method='ml') constructions with the same model name
|
||||
must cause exactly one underlying spacy.load() call."""
|
||||
calls = []
|
||||
|
||||
def fake_load(name, **kwargs):
|
||||
calls.append(name)
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
e1 = NERExtractor(method="ml")
|
||||
e2 = NERExtractor(method="ml")
|
||||
e3 = NERExtractor(method="ml", model="en_core_web_sm")
|
||||
|
||||
assert len(calls) == 1, (
|
||||
"repeated NERExtractor constructions should not reload the model"
|
||||
)
|
||||
assert e1.nlp is e2.nlp is e3.nlp
|
||||
|
||||
def test_ner_extractor_and_split_callers_share_one_cached_model(self, monkeypatch):
|
||||
"""NERExtractor, SemanticChunker, and split_by_sentences must all use
|
||||
the same cached Language object for the same model name."""
|
||||
calls = []
|
||||
|
||||
def fake_load(name, **kwargs):
|
||||
calls.append(name)
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
split_methods.split_by_sentences("First sentence.")
|
||||
semantic_chunker.SemanticChunker()
|
||||
NERExtractor(method="ml")
|
||||
|
||||
assert len(calls) == 1, (
|
||||
"split_by_sentences, SemanticChunker, and NERExtractor must share "
|
||||
"one cached model instead of each loading their own"
|
||||
)
|
||||
|
||||
def test_ner_extractor_distinct_model_names_load_separately(self, monkeypatch):
|
||||
"""Different model names must produce separate cache entries."""
|
||||
calls = []
|
||||
|
||||
def fake_load(name, **kwargs):
|
||||
calls.append(name)
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
sm = NERExtractor(method="ml", model="en_core_web_sm")
|
||||
lg = NERExtractor(method="ml", model="en_core_web_lg")
|
||||
sm_again = NERExtractor(method="ml", model="en_core_web_sm")
|
||||
|
||||
assert calls == ["en_core_web_sm", "en_core_web_lg"]
|
||||
assert sm.nlp is sm_again.nlp
|
||||
assert sm.nlp is not lg.nlp
|
||||
|
||||
def test_ner_extractor_failed_load_not_cached_and_retried(self, monkeypatch):
|
||||
"""A missing model must not poison the cache. A subsequent construction
|
||||
after the model becomes available must succeed and share the loaded model."""
|
||||
attempts = []
|
||||
|
||||
def failing_load(name, **_kwargs):
|
||||
attempts.append(name)
|
||||
raise OSError(f"Can't find model '{name}'")
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load))
|
||||
|
||||
# Construction with missing model: nlp must remain None, no crash
|
||||
extractor1 = NERExtractor(method="ml")
|
||||
assert extractor1.nlp is None
|
||||
assert len(attempts) == 1, "one load attempt expected for the missing model"
|
||||
|
||||
# Second construction: must retry (cache must not hold the failure)
|
||||
extractor2 = NERExtractor(method="ml")
|
||||
assert extractor2.nlp is None
|
||||
assert len(attempts) == 2, "a failed load must not be cached"
|
||||
|
||||
# Now install a working model and verify recovery
|
||||
def working_load(name, **_kwargs):
|
||||
attempts.append(name)
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load))
|
||||
|
||||
extractor3 = NERExtractor(method="ml")
|
||||
extractor4 = NERExtractor(method="ml")
|
||||
|
||||
assert extractor3.nlp is not None
|
||||
assert extractor3.nlp is extractor4.nlp
|
||||
assert len(attempts) == 3, (
|
||||
"exactly one successful load expected after the model becomes available"
|
||||
)
|
||||
|
||||
def test_ner_extractor_non_ml_method_does_not_load_model(self, monkeypatch):
|
||||
"""NERExtractor with a non-ml method must not touch the spaCy cache."""
|
||||
calls = []
|
||||
|
||||
def fake_load(name, **kwargs):
|
||||
calls.append(name)
|
||||
return _nlp_mock()
|
||||
|
||||
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
|
||||
|
||||
NERExtractor(method="pattern")
|
||||
NERExtractor(method="llm")
|
||||
NERExtractor(method="regex")
|
||||
|
||||
assert calls == [], "non-ml methods must not trigger any spacy.load()"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__])
|
||||
@@ -30,16 +30,18 @@ class TestSplitter(unittest.TestCase):
|
||||
splitter = TextSplitter(method=["recursive", "token"])
|
||||
self.assertEqual(splitter.methods, ["recursive", "token"])
|
||||
|
||||
@patch('semantica.semantic_extract.methods.spacy')
|
||||
@patch('semantica.split.semantic_chunker.spacy')
|
||||
def test_semantic_chunker_initialization(self, mock_spacy):
|
||||
# SemanticChunker now loads spaCy through the centralized
|
||||
# load_spacy_model() in semantic_extract.methods, so we patch
|
||||
# methods.spacy rather than the removed semantic_chunker.spacy binding.
|
||||
# Mock spacy.load to return a mock nlp object
|
||||
mock_nlp = MagicMock()
|
||||
mock_spacy.load.return_value = mock_nlp
|
||||
|
||||
with patch('semantica.split.semantic_chunker.SPACY_AVAILABLE', True):
|
||||
chunker = SemanticChunker(chunk_size=100)
|
||||
|
||||
# We need to ensure SPACY_AVAILABLE is True for this test context if possible,
|
||||
# but it is imported at module level.
|
||||
# If spacy is not installed, it sets SPACY_AVAILABLE = False.
|
||||
# We might need to patch the module attribute or just test fallback if spacy missing.
|
||||
|
||||
chunker = SemanticChunker(chunk_size=100)
|
||||
self.assertEqual(chunker.chunk_size, 100)
|
||||
|
||||
def test_chunk_dataclass(self):
|
||||
|
||||
@@ -54,11 +54,6 @@ from semantica.context.decision_models import (
|
||||
validate_decision,
|
||||
)
|
||||
|
||||
# ── Export module ──────────────────────────────────────────────────────────────
|
||||
# Set by the exporter's own `import pyarrow` attempt; False when pyarrow is
|
||||
# missing or unimportable.
|
||||
from semantica.export.parquet_exporter import PARQUET_AVAILABLE
|
||||
|
||||
# ── KG module ──────────────────────────────────────────────────────────────────
|
||||
from semantica.kg import (
|
||||
CentralityCalculator,
|
||||
@@ -986,17 +981,6 @@ class TestParquetExportRealData:
|
||||
Requires: pyarrow (optional dep — tests skip if not installed).
|
||||
"""
|
||||
|
||||
# ParquetExporter imports fine without pyarrow and only raises ImportError
|
||||
# when an export actually runs, so guarding on that import never skips
|
||||
# anything. Guard on the exporter's own availability flag instead: it is set
|
||||
# by the same `import pyarrow` / `import pyarrow.parquet` the exporter gates
|
||||
# on, so the skip condition cannot drift from the runtime check — including
|
||||
# when pyarrow is present on the path but fails to import.
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not PARQUET_AVAILABLE,
|
||||
reason="pyarrow not installed",
|
||||
)
|
||||
|
||||
@pytest.fixture
|
||||
def kg_data(self):
|
||||
return {
|
||||
@@ -1016,12 +1000,16 @@ class TestParquetExportRealData:
|
||||
}
|
||||
|
||||
def test_parquet_exporter_importable(self):
|
||||
from semantica.export import ParquetExporter
|
||||
|
||||
assert ParquetExporter is not None
|
||||
try:
|
||||
from semantica.export import ParquetExporter
|
||||
except ImportError as e:
|
||||
pytest.skip(f"ParquetExporter not available: {e}")
|
||||
|
||||
def test_parquet_export_entities_to_file(self, kg_data, tmp_path):
|
||||
from semantica.export import ParquetExporter
|
||||
try:
|
||||
from semantica.export import ParquetExporter
|
||||
except ImportError:
|
||||
pytest.skip("pyarrow not installed")
|
||||
|
||||
exporter = ParquetExporter(compression="snappy")
|
||||
out_path = tmp_path / "github_entities.parquet"
|
||||
@@ -1030,7 +1018,10 @@ class TestParquetExportRealData:
|
||||
assert out_path.stat().st_size > 0
|
||||
|
||||
def test_parquet_export_relationships_to_file(self, kg_data, tmp_path):
|
||||
from semantica.export import ParquetExporter
|
||||
try:
|
||||
from semantica.export import ParquetExporter
|
||||
except ImportError:
|
||||
pytest.skip("pyarrow not installed")
|
||||
|
||||
exporter = ParquetExporter(compression="gzip")
|
||||
out_path = tmp_path / "github_relationships.parquet"
|
||||
@@ -1039,7 +1030,10 @@ class TestParquetExportRealData:
|
||||
assert out_path.stat().st_size > 0
|
||||
|
||||
def test_parquet_export_knowledge_graph(self, kg_data, tmp_path):
|
||||
from semantica.export import ParquetExporter
|
||||
try:
|
||||
from semantica.export import ParquetExporter
|
||||
except ImportError:
|
||||
pytest.skip("pyarrow not installed")
|
||||
|
||||
exporter = ParquetExporter(compression="snappy")
|
||||
base_path = tmp_path / "github_kg"
|
||||
@@ -1049,24 +1043,30 @@ class TestParquetExportRealData:
|
||||
assert len(files) >= 1
|
||||
|
||||
def test_parquet_export_snappy_compression(self, kg_data, tmp_path):
|
||||
from semantica.export import ParquetExporter
|
||||
|
||||
try:
|
||||
from semantica.export import ParquetExporter
|
||||
except ImportError:
|
||||
pytest.skip("pyarrow not installed")
|
||||
exporter = ParquetExporter(compression="snappy")
|
||||
out_path = tmp_path / "snappy_test.parquet"
|
||||
exporter.export_entities(kg_data["entities"], str(out_path))
|
||||
assert out_path.exists()
|
||||
|
||||
def test_parquet_export_none_compression(self, kg_data, tmp_path):
|
||||
from semantica.export import ParquetExporter
|
||||
|
||||
try:
|
||||
from semantica.export import ParquetExporter
|
||||
except ImportError:
|
||||
pytest.skip("pyarrow not installed")
|
||||
exporter = ParquetExporter(compression="none")
|
||||
out_path = tmp_path / "uncompressed_test.parquet"
|
||||
exporter.export_entities(kg_data["entities"], str(out_path))
|
||||
assert out_path.exists()
|
||||
|
||||
def test_parquet_convenience_function(self, kg_data, tmp_path):
|
||||
from semantica.export.methods import export_parquet
|
||||
|
||||
try:
|
||||
from semantica.export.methods import export_parquet
|
||||
except ImportError:
|
||||
pytest.skip("pyarrow not installed")
|
||||
out_path = tmp_path / "convenience_test.parquet"
|
||||
export_parquet(kg_data["entities"], str(out_path))
|
||||
assert out_path.exists()
|
||||
|
||||
@@ -101,15 +101,9 @@ class TestNERConfigurations(unittest.TestCase):
|
||||
self.assertEqual(entities[0].metadata["extraction_method"], "ml")
|
||||
self.assertEqual(entities[0].metadata["model"], "en_core_web_trf")
|
||||
|
||||
@patch('semantica.semantic_extract.methods.spacy')
|
||||
@patch('semantica.semantic_extract.ner_extractor.spacy')
|
||||
def test_ner_ml_init_falls_back_when_spacy_runtime_is_broken(self, mock_spacy):
|
||||
"""Test NER init does not crash when spaCy is installed but unusable at runtime.
|
||||
|
||||
The model load now goes through load_spacy_model() in semantic_extract.methods,
|
||||
so we patch methods.spacy (not ner_extractor.spacy) to inject the failure.
|
||||
"""
|
||||
from semantica.semantic_extract.methods import clear_spacy_model_cache
|
||||
clear_spacy_model_cache()
|
||||
"""Test NER init does not crash when spaCy is installed but unusable at runtime."""
|
||||
mock_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined")
|
||||
|
||||
with patch('semantica.semantic_extract.ner_extractor.SPACY_AVAILABLE', True):
|
||||
@@ -118,23 +112,17 @@ class TestNERConfigurations(unittest.TestCase):
|
||||
self.assertIsNone(extractor.nlp)
|
||||
self.assertFalse(extractor._ml_runtime_usable)
|
||||
|
||||
@patch('semantica.semantic_extract.ner_extractor.spacy')
|
||||
@patch('semantica.semantic_extract.methods.get_entity_method')
|
||||
@patch('semantica.semantic_extract.methods.spacy')
|
||||
def test_ner_ml_runtime_failure_disables_repeated_ml_load_attempts(
|
||||
self,
|
||||
mock_methods_spacy,
|
||||
mock_get_method,
|
||||
mock_init_spacy,
|
||||
):
|
||||
"""Test degraded ML mode skips repeated spaCy load attempts after init failure.
|
||||
|
||||
The model load at construction time now goes through load_spacy_model() in
|
||||
semantic_extract.methods, so methods.spacy is the single mock target for the
|
||||
init-time failure. After the RuntimeError is raised, _ml_runtime_usable is
|
||||
False and no further spacy.load (or extract_entities_ml) calls are made.
|
||||
"""
|
||||
from semantica.semantic_extract.methods import clear_spacy_model_cache
|
||||
clear_spacy_model_cache()
|
||||
mock_methods_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined")
|
||||
"""Test degraded ML mode skips repeated spaCy load attempts after init failure."""
|
||||
mock_init_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined")
|
||||
mock_ml_method = MagicMock(return_value=[])
|
||||
mock_get_method.side_effect = lambda name: mock_ml_method if name == "ml" else (lambda *_args, **_kwargs: [])
|
||||
|
||||
@@ -144,9 +132,8 @@ class TestNERConfigurations(unittest.TestCase):
|
||||
entities = extractor.extract_entities(self.text)
|
||||
|
||||
self.assertFalse(extractor._ml_runtime_usable)
|
||||
# methods.spacy.load called once during __init__ (the RuntimeError); not again
|
||||
# during extract_entities because _filter_unusable_methods removes "ml".
|
||||
self.assertEqual(mock_methods_spacy.load.call_count, 1)
|
||||
self.assertEqual(mock_init_spacy.load.call_count, 1)
|
||||
self.assertEqual(mock_methods_spacy.load.call_count, 0)
|
||||
self.assertEqual(mock_ml_method.call_count, 0)
|
||||
self.assertIsInstance(entities, list)
|
||||
|
||||
|
||||
@@ -209,60 +209,6 @@ def test_load_from_api_allows_private_when_configured(mock_guard, seed_manager):
|
||||
call_kwargs = mock_guard.call_args[1]
|
||||
assert call_kwargs["allow_private_ips"] is True
|
||||
|
||||
|
||||
@patch("semantica.seed.seed_manager.request_with_ssrf_guard")
|
||||
def test_load_from_api_does_not_mutate_caller_headers_dict(mock_guard, seed_manager):
|
||||
"""Regression test for issue #947 audit: load_from_api must not mutate the
|
||||
caller's headers dict in-place when api_key is provided.
|
||||
|
||||
Before the fix, ``request_headers = headers or {}`` aliased the caller's dict.
|
||||
Writing ``request_headers["Authorization"] = ...`` then silently modified the
|
||||
caller's original dict, potentially leaking credentials to subsequent calls
|
||||
that reused the same headers dict without expecting it to carry Authorization.
|
||||
"""
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = {"results": []}
|
||||
mock_guard.return_value = mock_response
|
||||
|
||||
# Caller owns this dict and expects it to be unchanged after the call.
|
||||
original_headers = {"X-Custom-Header": "value"}
|
||||
headers_before = dict(original_headers) # snapshot
|
||||
|
||||
seed_manager.load_from_api(
|
||||
api_url="http://api.example.com",
|
||||
api_key="secret-key",
|
||||
headers=original_headers,
|
||||
)
|
||||
|
||||
# The caller's dict must be unchanged — Authorization must NOT have been added.
|
||||
assert original_headers == headers_before, (
|
||||
"load_from_api must not mutate the caller's headers dict; "
|
||||
f"expected {headers_before!r}, got {original_headers!r}"
|
||||
)
|
||||
|
||||
# The guard must still have received Authorization (in its own copy).
|
||||
call_kwargs = mock_guard.call_args[1]
|
||||
guard_headers = call_kwargs.get("headers", {})
|
||||
assert guard_headers.get("Authorization") == "Bearer secret-key"
|
||||
|
||||
|
||||
@patch("semantica.seed.seed_manager.request_with_ssrf_guard")
|
||||
def test_load_from_api_does_not_mutate_empty_headers_dict(mock_guard, seed_manager):
|
||||
"""When headers=None, a fresh dict is created — no aliasing to a shared mutable default."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = {"results": []}
|
||||
mock_guard.return_value = mock_response
|
||||
|
||||
seed_manager.load_from_api(
|
||||
api_url="http://api.example.com",
|
||||
api_key="key",
|
||||
headers=None,
|
||||
)
|
||||
|
||||
call_kwargs = mock_guard.call_args[1]
|
||||
guard_headers = call_kwargs.get("headers", {})
|
||||
assert guard_headers.get("Authorization") == "Bearer key"
|
||||
|
||||
def test_load_source(seed_manager, temp_data_dir):
|
||||
json_file = temp_data_dir / "source.json"
|
||||
with open(json_file, "w") as f:
|
||||
|
||||
@@ -31,21 +31,6 @@ class TestHelpers(unittest.TestCase):
|
||||
dict2 = {"b": {"d": 3}, "e": 4}
|
||||
merged = helpers.merge_dicts(dict1, dict2, deep=True)
|
||||
self.assertEqual(merged, {"a": 1, "b": {"c": 2, "d": 3}, "e": 4})
|
||||
def test_flatten_dict(self):
|
||||
data = {"a": {"b": 1, "c": 2}}
|
||||
result = helpers.flatten_dict(data)
|
||||
self.assertEqual(result, {"a.b": 1, "a.c": 2})
|
||||
|
||||
def test_flatten_dict_key_collision(self):
|
||||
data = {
|
||||
"a.b": 1,
|
||||
"a": {
|
||||
"b": 2
|
||||
}
|
||||
}
|
||||
|
||||
with self.assertRaises(ValueError):
|
||||
helpers.flatten_dict(data)
|
||||
|
||||
def test_safe_import_returns_module_and_flag(self):
|
||||
module, available = helpers.safe_import("json")
|
||||
|
||||
Reference in New Issue
Block a user