mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into owl-time-reachable-interval

Browse Source
This commit is contained in:
Mohd Kaif
2026-08-21 18:16:52 +05:30
committed by GitHub
parent d7ee22cf1f 0dc26350f9
commit 6653cbe879
44 changed files with 4773 additions and 527 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+95 -3
View File
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.6.6] - 2026-08-20
### Added
- **Semantica RDF vocabulary, and deterministic entity/relationship IRIs** (#1109, closes #1107, closes #1101) by @fabio-rovai, reviewed by @KaifAhmad1
@@ -18,9 +20,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- The missing-id fallback minted entity/relationship IRIs from Python's builtin `hash()`, randomised per process (`PYTHONHASHSEED`), so the same entity got a different IRI on every run and exports couldn't be diffed, deduplicated, or joined to an earlier provenance record. It also wrote `<semantica:entity_N>`, an IRI in the scheme `semantica` rather than the expansion of the declared prefix, so those nodes never joined with anything written through it. Minting now uses SHA-256 and writes a full IRI in the declared namespace; the same fix applies to the default entity/relationship types in the Turtle path
- **Fixed during review** (Qodo): the temporal fallback minted from `source_id` only, while the main serializer accepts `source_id` or `source` — relationships using the second form hashed two empty strings, which the previous randomised `hash()` masked by making the IRI unstable anyway; once deterministic, unrelated relationships at the same list index collided on one IRI across exports. Endpoints are now resolved the same way `serialize_to_turtle` resolves them, before minting. `sem:confidence` also lost its declared `xsd:decimal` range: the N-Triples serializer types the same value `xsd:float`, and the two are disjoint, so declaring either contradicted one of the exporters (tracked in #1100) — a new `test_declared_ranges_do_not_contradict_what_the_exporters_emit` guards the whole class of that mistake
- **Fixed in follow-up**: `serialize_to_rdfxml`'s default entity type still wrote the bare string `"semantica:Entity"` into an `rdf:resource` attribute, which (unlike a Turtle angle-bracket or an XML element name) is not namespace-expanded — the exact #1101 failure mode, just on the untested RDF/XML path. `json_exporter.py`'s `semantica:format` and `@type: "semantica:KnowledgeGraph"` were emitted but absent from both the vocabulary and the test's `EMITTED_TERMS` guard set, so the "undeclared terms fail the build" claim didn't actually cover them — both are now declared and guarded. `MANIFEST.in` didn't mirror the `pyproject.toml` package-data addition, so a source-distribution install could omit the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only `PATH`, breaking it on Windows; now overrides only `PYTHONHASHSEED` on top of the inherited environment
- 229 export and ontology tests pass
- **Also fixed, on the JSON-LD paths**: the first fix covered the Turtle, N-Triples and RDF/XML serializers, and left both JSON-LD writers interpolating the entity's own text into `f"semantica:entity/{text}"` and the endpoints into `f"semantica:rel/{source}_{target}"`. Three consequences, all live in 0.6.5: an entity whose text contained a space produced an invalid IRI, and a JSON-LD parser dropped that node in full rather than reporting it, so the entity disappeared from the export; every relationship carrying `source`/`target` rather than `source_id`/`target_id` minted the identical `semantica:rel/_`, collapsing all of them onto one node whose types and endpoints merged; and the JSON-LD `@id` disagreed with the Turtle IRI for the same entity, so the two serializations of one knowledge graph were two different graphs. Both JSON-LD writers now use `mint_entity_iri`/`mint_relationship_iri`, and `JSONExporter.export_entities`/`export_relationships` declare the `semantica` prefix their `@context` was already writing `semantica:entities` against — without it a processor reads that as an IRI in the scheme `semantica`, which is the original #1101 defect on a third path
- `tests/export/test_jsonld_iri_minting.py` parses each export with a real JSON-LD processor and asserts the entity survives, the relationships stay distinct, no term expands into the `semantica` scheme, and the JSON-LD `@id` equals the Turtle IRI
- 236 export and ontology tests pass
- **First-class CrewAI integration** (#962)
- **First-class CrewAI integration** (#988, closes #962) by @Shindevrp
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
- `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()`
- `integrations/crewai/SemanticaDecisionTool` — a CrewAI `BaseTool` wrapping `AgentContext` with 5 decision-intelligence actions (`record_decision`, `find_precedents`, `trace_causal_chain`, `analyze_impact`, `check_policy`)
@@ -50,6 +54,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- New `tests/export/test_distance_exporter_metric_errors.py`: 6 tests covering success, single/multiple failures, opt-out, the no-path-vs-error distinction, and default-schema stability; existing `tests/export/test_distance_exporter.py` updated for the new tuple return type
- Full `tests/export/` suite: 77 passed
- **`ContextGraph.to_kg_dict()`: an adapter converting a `ContextGraph`'s internal `nodes`/`edges`/`source` shape into the canonical `entities`/`relationships`/`source_id` shape `RDFExporter` and `TemporalGraphQuery` consume** (#1081) by @cxzg007
- Previously there was no supported way to feed a `ContextGraph` into those consumers without hand-rolling the field remapping; `to_kg_dict()` does it once, with an `entities_only` option that drops relationships left dangling by the filter
- **Fixed during review** (Qodo): null `properties`/`metadata` on a node loaded from JSON raised `TypeError` when copied — both are now guarded with `or {}`; entity ids are coerced to `str(node_id)` to match `ContextEdge`'s already-str-coerced endpoints, so valid relationships were no longer dropped by `entities_only` filtering
- `RDFExporter`'s validator and `TemporalGraphQuery` now also accept `source_id`/`target_id` endpoints, the shape `to_kg_dict()` emits
### Changed
- **`GraphBuilder`'s 6 public methods now have Google-style docstrings** (#878, closes #876) by @cakeni
@@ -59,7 +68,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Corrected during review**: `add_temporal_edge`/`create_temporal_snapshot` docstrings overclaimed numeric-timestamp support; `_parse_time()` only special-cases `str` and `datetime`, falling back to a bare `str()` cast for anything else (not true numeric parsing). Narrowed to "datetime or ISO-formatted string"
- **Fixed along the way**: `build()`'s `**options` documented a default only for `extract`; `extract_relations`, `extract_triplets`, `ner_method`, `relation_method`, and `triplet_method` all have concrete defaults in `_extract_from_text()` (`True`, `True`, `"llm"`, `"llm"`, `"llm"`) that were left unstated, inconsistent with CONTRIBUTING.md's own docstring example of noting defaults inline
- `python -m pytest tests/kg/test_kg.py tests/kg/test_graph_builder_external.py -q`: 45 passed
- **`GraphBuilder` raw-text extraction now defaults to local extractors instead of LLM extraction** (closes #930) by @dex0shubham
- **`GraphBuilder` raw-text extraction now defaults to local extractors instead of LLM extraction** (#941, closes #930) by @dex0shubham
- `GraphBuilder._extract_from_text()` defaulted `ner_method`, `relation_method`, and `triplet_method` to `"llm"`, and ran relation extraction unconditionally (`extract_relations` defaulted to `True`) — all four contradicting the defaults documented in the `build()` docstring at the time (`"ml"` / `"pattern"` / `False`), and diverging from the standalone extractors (`NERExtractor` defaults to `method="ml"`, `RelationExtractor` and `TripletExtractor` to `method="pattern"`). The practical effect was that any raw-text `build()` call silently required a configured provider, an API key, and network access
- Defaults are now `ner_method="ml"`, `relation_method="pattern"`, `triplet_method="pattern"`, and `extract_relations=False`, matching the docstring. LLM extraction remains fully available and is now opt-in
- **To restore the previous behaviour**, pass the methods explicitly:
@@ -80,8 +89,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- New regression coverage in `tests/kg/test_graph_builder_extraction_defaults.py` pinning all four defaults, verifying that no default resolves to `"llm"`, confirming explicit LLM opt-in still routes correctly, asserting extractors are constructed once across repeated texts, covering fallback method lists (e.g. `ner_method=["pattern", "ml"]`) for all three extractors, asserting relations are forwarded to triplet extraction (and that `None` is forwarded when relation extraction is disabled or fails), and running the real default path end to end with no provider mocked. Verified to fail against the pre-fix code
- Full `kg` suite: 473 passed
- **Explorer graph canvas now renders edge labels** (#1013, closes #1009) by @yzxcj797
- `GraphCanvas.tsx` had no edge-label rendering path at all; Sigma's edge-label renderer draws `data.label`, but the graph state stored the relationship type under `edgeType`, so simply enabling the renderer would have left every edge blank. `graphSceneState`'s edge reducer now maps `edgeType` onto `label` (suppressed for hidden edges)
- Rendering is gated behind a new `edgeLabelsEnabled` entry in the Effects panel (default on), wired through the existing `GraphEffectToggle`/`GraphEffectsState` plumbing, so dense graphs can still turn labels off
- **Fixed during review** (Qodo): two follow-up passes closed gaps the first cut left — label rendering wasn't wired through `explorationEffectsPluginPhaseC.tsx`'s Phase C variant, and toggling the effect off mid-session didn't clear already-rendered labels
- New coverage in `explorer/tests/graphSceneState.display.test.ts`
- **Removed `GraphWorkspaceShell.tsx`, `GraphRuntimeStage.tsx`, and `useGraphData.ts` — a second, unused implementation of the graph-loading/error-handling logic already fixed in `GraphWorkspace.tsx`** (#984, resolves the cleanup tracked in #981 by #980's review note) by @lakshayxi
- 1,564 lines removed; the surviving `GraphWorkspace` path is now the only implementation, so the "two copies that drifted apart" root cause #980 fixed can't recur in the copy nobody was maintaining
- **Explorer README and `docs/explorer-setup.md` corrected to describe the authentication 0.6.5 actually shipped**, plus a documented `/ws/graph-updates` auth note (#1040, fixes #1028) by @Kyou12138
- Both docs still claimed the Explorer API had no built-in authentication after v0.6.5 added mandatory `SEMANTICA_API_KEY` enforcement with a `503` fail-closed default; corrected to describe the actual behavior, including that only protected routes require the key (`/api/health`/`/api/info` stay open), the non-loopback-bind CLI warning only fires in anonymous mode or when the key is unset, and `SEMANTICA_API_KEY`/`SEMANTICA_ALLOW_ANONYMOUS` are documented in the environment-variable table
- **CI: pinned `github/codeql-action` to current v4** (#986) by @ZohaibHassan16, and **pinned Python dependencies in `requirements-ci.txt` for reproducible CI runs** (#945) by @yunaremaia, closing the gap where an unpinned CI dependency could silently change behavior between runs
- **README now states up front that Semantica's explainability is system-level, not foundation-model-internal** (#1033, #1034) by @KaifAhmad1
- Nothing in the README previously scoped what "explainable" meant, leaving readers to assume Semantica could expose or reconstruct an LLM's internal reasoning. A callout now states explicitly that Semantica explains and audits what the AI *system* did — context fed in, decisions produced, provenance, relationships, policies applied — not the model's private internal reasoning, and moved the note near the top of the README rather than leaving it implicit
### Fixed
- **Every timestamp an export or a provenance record wrote was timezone-naive** (closes #1114) by @fabio-rovai
- `semantica/export/` stamped with `datetime.now().isoformat()`, which reads the machine's **local** clock; `semantica/provenance/` stamped with `datetime.utcnow().isoformat()`, which reads **UTC**. Both produce a naive value and both serialize identically, so nothing downstream can tell which zone a given timestamp belongs to — the same string means two different instants depending on which module wrote it
- In RDF the consequence is silent rather than loud. Under XSD 1.1 a value with no timezone compared against one with a timezone is indeterminate whenever the two fall inside the ±14 hour window; SPARQL turns an indeterminate comparison into an error, and `FILTER` discards errors as non-matches. A timezone-qualified query over an Oxigraph store returns an answer with every Semantica-written record quietly absent from it, which is a poor property for `prov:generatedAtTime`, `prov:startedAtTime`, `prov:endedAtTime` and `prov:atTime` to have
- New `utc_now()`/`utc_now_iso()` in `semantica/utils/helpers.py`, exported from `semantica.utils`, and used at all 29 call sites in `export/` (`json_exporter`, `yaml_exporter`, `report_generator`, `export_provenance`) and `provenance/` (`manager`, `schemas`, `bridge_axiom`). Values now read `2026-08-19T14:19:04.229937+00:00`: one unambiguous instant, comparable against any correctly stamped value, and valid `xsd:dateTimeStamp`. `sem:exportedAt`'s range in `semantica/ontology/vocabulary/semantica-ns.ttl` is tightened from `xsd:dateTime` accordingly, and its comment no longer has to explain why the weaker range was necessary
- `datetime.utcnow()` is deprecated as of Python 3.12 and scheduled for removal; constructing a `ProvenanceEntry` under `-W error::DeprecationWarning` on 3.13 raised, and no longer does
- New `tests/export/test_timestamp_timezones.py` and `tests/provenance/test_timestamp_timezones.py`: offset presence on every export and provenance path, PROV-O literals valid as `xsd:dateTimeStamp`, comparison against a timezone-aware instant without `TypeError`, the Oxigraph filter that dropped the naive value (with a bound inside the indeterminate window, so the test cannot pass by accident), and the document `@id` remaining a valid IRI with `+00:00` in it. 11 of the 13 fail on the parent commit
- **Fixed during review** (Qodo): once new entries carry `+00:00` and stored ones do not, `ProvenanceManager.query_recorded_between` and `audit_log` compared ISO timestamps as raw strings, so they ordered by spelling rather than by instant — an inclusive naive bound naming a stored offset-bearing timestamp sorted *below* it and dropped the record, and a bound written in another offset landed wherever its digits fell (`19:45+05:30` is 14:15Z, but sorted after 14:19Z). Both now compare instants through a new `to_utc_datetime()` helper that reads a missing offset as UTC, which is what the values written before this change actually were; a bound that cannot be read as a timestamp keeps the historical string comparison rather than raising on a call that used to work
- The remaining 147 naive call sites are in `context/`, `vector_store/`, `seed/` and elsewhere, where timestamps are compared against values parsed from previously stored naive strings. Converting those without a read-side migration would raise `TypeError: can't compare offset-naive and offset-aware datetimes` on existing data, so they are deliberately left for a separate change
- **`split`/chunking paths bypassed the centralized spaCy model cache, reloading the model on every call** (#1042, closes #998) by @Accute9, reviewed by @Sameer6305
- `semantica/split/methods.py`'s `split_by_sentences()` and `semantica/split/semantic_chunker.py`'s `SemanticChunker.__init__` each called `spacy.load()` directly instead of reusing the process-level cache added in #889/`semantic_extract/methods.py`'s `load_spacy_model()` — every call/construction re-paid the ~120ms model-load cost independently of `NERExtractor`, which already used the cache
- Both now route through `load_spacy_model()`, sharing one cached `Language` instance per model name across `split_by_sentences()`, `SemanticChunker`, and `NERExtractor`; a missing model still falls back to regex/paragraph chunking without poisoning the cache for a later successful load
@@ -183,6 +218,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- New regression coverage in `tests/export/test_distance_exporter.py`: warnings fire on exception for all four helpers, exported sentinel values/shape stay unchanged, and the legitimate "no KG backend" `None` path still logs nothing
- Full `tests/export/` suite: 71 passed
- **`explain_violations` rendered hardcoded placeholders (`min_count=1`, `max_count=1`) instead of the SHACL shape's real constraint values, and misused the violation message text as the datatype/class value** (#1094) by @cxzg007
- `_run_pyshacl` never read `sh:minCount`/`sh:maxCount`/`sh:datatype`/`sh:class` back from the violation's `sh:sourceShape`, so every plain-English explanation was wrong regardless of what the shape actually declared. `SHACLViolation` now carries those four fields (also exposed via `to_dict()`), populated by back-referencing `sh:sourceShape`; `explain_violations` renders the real values, falling back to `"?"` when a value is genuinely absent
- **Known limitation**: `sh:qualifiedMinCount`/`sh:qualifiedMaxCount` are not handled yet and still fall back to the `"?"` placeholder
- New regression tests cover both the rendering path and the `sh:sourceShape` back-reference (skipped when `pyshacl`/`rdflib` are absent)
- **Entity merging silently dropped `entity_id` aliases, and exact-match entity resolution had three correctness gaps** (#1086, #1026) by @T1mn
- `entity_merger.py`/`merge_strategy.py`/`entity_resolver.py` used inconsistent logic for extracting an entity's id across the merge path, so a merged entity could lose the `entity_id` aliases that let later lookups find it under its old identity. A new `semantica/utils/entity_ids.py` unifies id extraction across all three call sites
- `EntityResolver`'s exact-match path is now honored rather than silently falling through to fuzzy matching in some cases; entities with no identifier are preserved instead of being dropped, and blank exact-match names are ignored rather than matching every other blank name
- New/expanded coverage in `tests/kg/test_entity_pipeline.py` and `tests/kg/test_entity_resolver_exact.py`
- **`flatten_dict()` silently collided keys when a flattened path from one branch matched a literal key already present at the target depth** (#1062) by @shahzaib-ahmadcs
- Two differently-shaped inputs could flatten to the same output key, with the second write silently overwriting the first — no error, no warning, just a dropped value. Collisions are now detected and handled explicitly instead of overwriting
- **`ExcelParser.__init__` raised `NameError` on every instantiation — `get_progress_tracker()` was called but never imported** (#1016, closes #1014) by @pravit-amp
- Same defect as the one fixed for `SimilarityCalculator` in #530, this time in `semantica/parse/excel_parser.py`; the existing test imported the class but never constructed it, so nothing caught the missing import. Added construction coverage for every parser exported from `semantica.parse`, driven off `__all__` so future additions are covered automatically, living outside `test_parse_comprehensive.py` (whose `setUp` mocks `get_progress_tracker` into each module and would mock away the exact interaction under test)
- **Graph analytics (`centrality_calculator.py`, `community_detector.py`, `connectivity_analyzer.py`) dropped isolated nodes and diverged on how each computed its working view of the graph** (#1011) by @T1mn
- Each analyzer had its own ad hoc logic for building the node/edge set it operated over, and none of them included nodes with no edges — a node with zero connections simply vanished from centrality scores, community assignments, and connectivity reports instead of appearing with a zero/singleton value. A new shared `semantica/kg/_graph_view.py` centralizes graph-view construction (including node fallbacks and community payload shaping) for all three analyzers, which are now ~250 lines lighter combined
- New `tests/kg/test_analytics_node_scope.py` covering isolated-node presence across all three analyzers
- **Explorer fired temporal-bounds and snapshot requests before the graph itself had loaded, tripling failed requests when the backend was down and leaving the timeline scrubber with nothing to scrub** (#1003) by @lakshayxi
- Two new predicate functions gate the temporal effects on the graph having actually loaded (an empty graph still counts as loaded); confirmed against a downed backend that this cuts three failing requests per page load down to one
- **`SeedDataManager.load_from_database()` never actually reached the database, and connection failures were mislabeled as a missing optional dependency** (#995, closes #973) by @yzxcj797
- `DBIngestor.execute_query`/`export_table` need the connection string as their first positional argument; `load_from_database()` only passed it into the constructor's config dict, which those methods never read, so every call raised `TypeError` before connecting. Also split the combined `except (ImportError, OSError)` handling apart — a genuine connection failure was reported as `"module not available"`, sending debugging in the wrong direction; `OSError` now propagates as an actual failure, chained via `from e`
- **SPARQL `CONSTRUCT` detection matched inside a leading `#`-comment, misclassifying `SELECT`/`ASK` queries as `CONSTRUCT` across all four SPARQL backends** (#951) by @pravit-amp
- `CONSTRUCT_QUERY_RE` skipped comments with a bare `\#[^\n]*`, whose backtracking `*` let a `# CONSTRUCT ...` comment line "swallow" the real query-form keyword on the next line for a query like `# CONSTRUCT ...\nSELECT ...`. The mistaken `CONSTRUCT` classification sent `Accept: text/turtle` and tried to parse a SELECT/ASK response body as Turtle, failing with a misleading parse error. The regex now requires a comment to reach a line terminator (LF or CR, per the SPARQL grammar) before matching
- **`k_shortest_paths` mutated caller-visible graph state during traversal and ignored direction when excluding already-used edges** (#1000) by @T1mn
- `semantica/kg/path_finder.py`'s search left side effects behind after returning, and edge exclusion during Yen's-algorithm-style path removal didn't respect the traversal direction of directed graphs, letting a later search see edges that should have been available. Both fixed; new coverage in `tests/kg/test_path_finder.py`
- **`trace_decision_causality()` ignored explicitly recorded causal edges, inferring causes only from shared NER entities plus timestamp ordering** (#983) by @hsd2514
- A `CAUSED`/`INFLUENCED`/`PRECEDENT_FOR` edge added via `add_causal_relationship()` had no effect on the trace — when entity extraction found nothing in common between two decisions, `trace_decision_chain()` came back empty even with an explicit edge stored in the graph. Explicit causal edges are now traversed first as ground truth, with entity/timestamp inference kept as an additive fallback for pairs with no explicit link; edges whose source has no decision record (e.g. a graph restored via `from_dict`) are skipped so a stale edge can't abort the trace
- **`RepoIngestor`'s module-level DNS resolve cache had no lock, raising `RuntimeError: OrderedDict mutated during iteration` under concurrent `ingest_repository()` calls** (#979) by @manjunathbhaskar
- `_REPO_HOST_RESOLVE_CACHE` is a shared `OrderedDict` read, written, and pruned by every thread with no synchronization — reliably reproduced with 32 threads hammering resolution under a low TTL and small cache cap. Now guarded by a lock
- **`GraphBuilder` didn't remap relationship endpoints after entity resolution merged nodes, leaving relationships pointing at ids that no longer existed in the resolved graph** (#978) by @T1mn
- New coverage in `tests/kg/test_graph_builder_external.py`; a follow-up commit hardens the remapping against edge cases found during review
- **Explorer's dev server esbuild target didn't match the browser targets the production build declares**, occasionally producing dev-only syntax errors on older browsers (#966) by @le-czs
- `explorer/vite.config.ts` now sets the dev esbuild target explicitly to match
- **`normalize`'s number normalizer accepted currency symbols without validating them against the surrounding text, and an earlier fix's currency-code matching wasn't token-bounded** (#940) by @Mr-Neutr0n, reviewed by @ZohaibHassan16
- Symbol currencies are now validated before being accepted; currency codes are matched on token boundaries so a code embedded inside a longer token no longer false-positives
- **`ContextGraph.to_dict()` was the one reader on the class that didn't hold `self._lock`, raising `RuntimeError: dictionary changed size during iteration` under a concurrent writer and risking a torn snapshot otherwise** (#929) by @pravit-amp
- Every other reader (`stats()`, `density()`, `find_nodes()`, `find_edges()`, `get_neighbors()`, `get_nodes_by_label()`, `state_at()`, `save_to_file()`) already took the lock after it was introduced; `to_dict()` predated that change and was missed. `save_to_file()` was safe only incidentally, since it builds its payload inline under its own lock rather than delegating to `to_dict()`
- **`PipelineWithProvenance` had a broken import and no working `run()` method** (#862) by @Karunasagar12
- `from .pipeline import Pipeline` failed because `Pipeline` lives in `pipeline_builder.py`, not a nonexistent `pipeline.py` — fixed to `from .pipeline_builder import Pipeline`. The class also had no `run()`; it now delegates to `ExecutionEngine.execute_pipeline()`, the intended execution path for a built `Pipeline`. The constructor now accepts a built `Pipeline` instance directly
### Security
- **Tarball restore path traversal, latent SQL injection, DNS-rebinding TOCTOU in the shared SSRF guard, stored XSS in report generation, and unvalidated SPARQL object IRIs in AnzoStore** (#1079) by @KaifAhmad1
@@ -243,6 +331,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Caught by the new gate on its first run**: `python -m pip install -e ".[all]"` pulled in `setuptools==79.0.1`, vulnerable to CVE-2026-59890/GHSA-h35f-9h28-mq5c/PYSEC-2026-3447 (Unicode-normalization bypass of `MANIFEST.in` exclude/prune patterns on macOS APFS/HFS+, letting excluded files leak into a built sdist), fixed in `83.0.0`. `[build-system] requires` had the exact same too-permissive-floor pattern this whole entry is about (`setuptools>=61.0`), and `actions/setup-python`'s baked-in `setuptools` isn't governed by that pin at all since it's outside any isolated build. Bumped `[build-system] requires` to `setuptools>=83.0.0`, and the `Security` workflow now runs `pip install --upgrade pip setuptools` before auditing so the scanned environment can't have a stale ambient copy regardless of what governs it
- Full `explorer` suite: 241 passed
- **`SeedDataManager.load_from_api()` made unguarded HTTP requests, with no SSRF protection at all** (#942) by @ZohaibHassan16
- `load_from_api()` called `requests.get()` directly instead of going through `semantica/ingest/ssrf.py`'s `request_with_ssrf_guard()`, unlike every other ingestor in this module — a caller-supplied `api_url` could target internal/private network addresses with no validation. Now routes through the shared guard, gaining redirect validation and bounded DNS resolution for free
- **Follow-up** (#959, closes #943) by @yunaremaia: added an `allow_private_ips` opt-in (parsed via the shared `parse_bool` helper) for trusted internal deployments that legitimately need to load from a private-network API, while keeping the guard's block-by-default behavior for everyone else
## [0.6.5] - 2026-08-11
### Added
README.md
+10 -10
View File
@@ -142,7 +142,7 @@ compliant = graph.check_decision_rules({"category": "vendor_selection"}) # poli
```bash
semantica doctor
# Python 3.11.9 pass
# semantica 0.6.5 pass
# semantica 0.6.6 pass
# faiss vector store pass
# Config file pass ~/.semantica/config.yaml
```
@@ -1466,18 +1466,18 @@ For contributor / dev-server setup: **[explorer/README.md: Local Setup Guide](ex
---
## What's New in v0.6.5
## What's New in v0.6.6
**Security release — upgrading is strongly recommended.** Fixes for 5 externally-reported vulnerabilities in the Explorer API and graph/triplet store backends, plus a CodeQL-flagged ReDoS:
**Security release — upgrading is strongly recommended.** Fixes for a privately disclosed batch of vulnerabilities spanning backup/restore, database export, outbound requests, and triplet-store backends, plus SSRF hardening across ingestion:
- **Missing authentication on all Explorer API routes** (GHSA-j4mq-hprp-987v, Critical): every route now requires `SEMANTICA_API_KEY`, fails closed (503) rather than open when unconfigured
- **SSRF via redirect bypass in ontology URL fetching** (GHSA-8c7v-62gr-hj6g, High): redirect targets are now re-validated at every hop and the connection is pinned to the validated address, closing a DNS check-then-use race
- **Cypher injection via unvalidated node labels and property keys** (GHSA-482h-hw99-h62p, Critical): Neptune, Neo4j, and FalkorDB now sanitize every label/relationship-type/property-key interpolation site
- **SPARQL injection via unvalidated triplet IRIs** (GHSA-8vgg-8mr4-r236, Critical): Blazegraph, RDF4J, and Jena now validate subject/predicate/object IRIs before interpolation
- **Missing Origin validation on the WebSocket handshake** (GHSA-4643-wpgq-w329, Moderate, anonymous-mode only): `/ws/graph-updates` now checks `Origin` against the same allowlist `CORSMiddleware` enforces for HTTP
- **Polynomial ReDoS in SPARQL query validation** (CodeQL `py/polynomial-redos`): fixed a backtracking regex in the Explorer's SPARQL route
- **Tarball restore path traversal**: `semantica backup restore` now validates every archive member for path containment and rejects symlink/hardlink escapes before extraction
- **Latent SQL injection in `DataExporter.export_table_data()`**: table/schema names are now identifier-allowlisted and `where`/`order_by` fragments are blocklist-checked
- **DNS-rebinding TOCTOU in the shared SSRF guard**: the resolved IP that passes validation is now the one the connection is pinned to, closing the check-then-use race (also closes the `100.64.0.0/10` CGNAT gap)
- **Stored XSS in HTML report generation** and **unvalidated SPARQL object IRIs in AnzoStore** (SPARQL injection): both now escape/validate before interpolation
- **`Authorization`/`Proxy-Authorization` credential leakage across redirects**, plus **SSRF gaps in `FeedIngestor`/`FeedMonitor`, `RepoIngestor`, and the MCP/public-API ingest paths**: all now route through the shared, redirect-safe SSRF guard
- **HTTP response header injection and an unbounded-memory DoS** in the Explorer API, and a **`fastapi`/`python-multipart` ReDoS** (PYSEC-2024-38): floors raised, inputs sanitized, candidate pools capped
Also includes: embedded Oxigraph backend for `TripletStore`, PROV-O trust/spec completeness for `ProvenanceManager`, and the Altair Anzo triplet store backend.
Also ships: **first-class CrewAI integration** (`semantica[crewai]`, extraction/decision tools + a knowledge source), **`ContextGraph` retraction and purge** (GDPR-style erasure without a full `clear()`), a declared **Semantica RDF vocabulary with deterministic entity/relationship IRIs** (stable, diffable exports), and **timezone-aware timestamps** across `export/` and `provenance/`.
→ [Full release notes](RELEASE_NOTES.md) · [Changelog](CHANGELOG.md)
docs/citation.md
+5 -5
View File
@@ -17,22 +17,22 @@ icon: "quote-left"
author = {Semantica},
year = {2026},
url = {https://github.com/semantica-agi/semantica},
version = {0.6.5},
version = {0.6.6},
doi = {10.5281/zenodo.XXXXXXX}
}
```
</Tab>
<Tab title="APA">
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* (Version 0.6.5) \[Computer software\]. https://github.com/semantica-agi/semantica
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* (Version 0.6.6) \[Computer software\]. https://github.com/semantica-agi/semantica
</Tab>
<Tab title="MLA">
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.5, GitHub, 2026, https://github.com/semantica-agi/semantica.
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.6, GitHub, 2026, https://github.com/semantica-agi/semantica.
</Tab>
<Tab title="Chicago">
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.5. GitHub, 2026. https://github.com/semantica-agi/semantica.
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.6. GitHub, 2026. https://github.com/semantica-agi/semantica.
</Tab>
<Tab title="IEEE">
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," Version 0.6.5, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," Version 0.6.6, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
</Tab>
</Tabs>
docs/faq.md
+1 -1
View File
@@ -17,7 +17,7 @@ icon: "circle-question"
| API key required? | Optional: pattern extraction works with no keys |
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
| Latest version? | **v0.6.5** (August 2026) |
| Latest version? | **v0.6.6** (August 2026) |
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
docs/getting-started.md
+1 -1
View File
@@ -42,7 +42,7 @@ icon: "rocket"
Verify installation:
```python
import semantica
print(semantica.__version__) # 0.6.5
print(semantica.__version__) # 0.6.6
```
</Check>
</Step>
explorer/package-lock.json
Generated
+1489 -14
View File
File diff suppressed because it is too large Load Diff
explorer/package.json
+3 -1
View File
@@ -9,7 +9,7 @@
"lint": "eslint .",
"preview": "vite preview",
"test:graph-store": "node --test tests/graphStore.multi-edge.test.mjs",
"test:graph-workspace": "node --import tsx --test tests/graphSceneState.display.test.ts tests/temporalLifecycle.test.ts",
"test:graph-workspace": "node --import tsx --test tests/markdownContentViewer.test.ts tests/graphSceneState.display.test.ts tests/temporalLifecycle.test.ts",
"test:plugin-registry": "node --import tsx --test tests/pluginRegistry.temporal.test.mjs"
},
"dependencies": {
@@ -29,6 +29,8 @@
"react-arborist": "^3.4.3",
"react-dom": "^19.2.4",
"react-dropzone": "^15.0.0",
"react-markdown": "^10.1.0",
"remark-gfm": "^4.0.1",
"sigma": "^3.0.2",
"vis-data": "^8.0.3",
"vis-timeline": "^8.5.0"
explorer/src/workspaces/GraphWorkspace/GraphInspectorPanel.tsx
+20
View File
@@ -3,6 +3,7 @@ import { Loader2 } from "lucide-react";
import { graph } from "../../store/graphStore";
import { GRAPH_THEME, withAlpha } from "./graphTheme";
import type { GraphSelectedNodeKind } from "./types";
import { MarkdownContentViewer } from "./MarkdownContentViewer";
export type LinkPrediction = {
target: string;
@@ -364,6 +365,11 @@ export function GraphInspectorPanel({
([key]) =>
!["x","y","valid_from","valid_until","content","source","source_url","pmid","pmids","evidence","provenance","confidence"].includes(key),
);
const nodeContent = (typeof attributes?.content === "string" && attributes.content)
? attributes.content
: (typeof properties.content === "string" && properties.content)
? properties.content
: "";
return (
<aside style={{ padding: 24, display: "flex", flexDirection: "column", gap: 18 }}>
@@ -408,6 +414,20 @@ export function GraphInspectorPanel({
</div>
) : null}
{/* Content Section — only rendered when the node carries actual content.
This matches the existing inspector convention: sections that have no
data for the current node are either hidden (temporal bounds) or closed
by default (Source Attribution, Properties). Always showing an open
empty panel would add noise for every relationship/predicate node. */}
{nodeContent && (
<details className="node-panel-collapse" open>
<summary className="node-panel-summary">Content</summary>
<div className="node-panel-body" style={{ marginTop: 8 }}>
<MarkdownContentViewer content={nodeContent} />
</div>
</details>
)}
{/* Actions */}
<section style={sectionStyle}>
<div style={sectionTitleStyle}>Actions</div>
explorer/src/workspaces/GraphWorkspace/MarkdownContentViewer.tsx
+403
View File
@@ -0,0 +1,403 @@
import { useState, useRef, useEffect, type CSSProperties } from "react";
import ReactMarkdown from "react-markdown";
import remarkGfm from "remark-gfm";
import { Check, Copy, Code2, Eye, ExternalLink, Image as ImageIcon } from "lucide-react";
import { GRAPH_THEME } from "./graphTheme";
export interface MarkdownContentViewerProps {
content?: string | null;
className?: string;
defaultMode?: "preview" | "source";
}
export function isSafeUrl(url?: string): boolean {
if (!url) return false;
const trimmed = url.trim();
// Reject whitespace-only strings — new URL("", base) would resolve to the base
// protocol and produce a false positive. This guards direct callers of the exported
// function; markdown parsers normalise whitespace-only destinations to "" which
// already fails the !url check above.
if (!trimmed) return false;
if (trimmed.startsWith("//")) return false;
if (trimmed.startsWith("#")) return true;
if (trimmed.startsWith("/")) return true;
try {
const parsed = new URL(trimmed, "http://localhost");
return ["http:", "https:", "mailto:"].includes(parsed.protocol);
} catch {
return false;
}
}
export function MarkdownContentViewer({
content,
className,
defaultMode = "preview",
}: MarkdownContentViewerProps) {
const [activeMode, setActiveMode] = useState<"preview" | "source">(defaultMode);
const [copied, setCopied] = useState(false);
// Track the content value for which the copied indicator is valid.
// When content changes (i.e. the user selects a different node), reset the
// copied indicator inline during render rather than in a useEffect — this
// avoids a cascading-render lint error and is the React-recommended pattern
// for resetting derived visual state on prop changes.
const [copiedForContent, setCopiedForContent] = useState<string | null | undefined>(content);
if (copiedForContent !== content) {
setCopiedForContent(content);
if (copied) {
// Clear the stale indicator synchronously so the new node's copy button
// never shows "Copied" from the previous selection.
setCopied(false);
}
}
const copyTimeoutRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// Clean up any outstanding timeout on unmount.
useEffect(() => {
return () => {
if (copyTimeoutRef.current) {
clearTimeout(copyTimeoutRef.current);
}
};
}, []);
const rawContent = typeof content === "string" ? content : "";
const hasContent = rawContent.trim().length > 0;
const handleCopy = async () => {
if (!hasContent) return;
try {
await navigator.clipboard.writeText(rawContent);
if (copyTimeoutRef.current) {
clearTimeout(copyTimeoutRef.current);
}
setCopied(true);
copyTimeoutRef.current = setTimeout(() => setCopied(false), 1500);
} catch {
// Clipboard write unavailable
}
};
return (
<div className={className} style={viewerContainerStyle}>
<div style={viewerHeaderStyle}>
<div style={{ display: "flex", gap: 4 }} role="tablist">
<button
type="button"
role="tab"
aria-selected={activeMode === "preview"}
onClick={() => setActiveMode("preview")}
style={{ ...tabBtnStyle, ...(activeMode === "preview" ? activeTabBtnStyle : {}) }}
>
<Eye size={12} style={{ marginRight: 5 }} />
Preview
</button>
<button
type="button"
role="tab"
aria-selected={activeMode === "source"}
onClick={() => setActiveMode("source")}
style={{ ...tabBtnStyle, ...(activeMode === "source" ? activeTabBtnStyle : {}) }}
>
<Code2 size={12} style={{ marginRight: 5 }} />
Source
</button>
</div>
{hasContent && (
<button type="button" onClick={() => void handleCopy()} style={copyBtnStyle} title="Copy raw content">
{copied ? (
<>
<Check size={12} color="#3fb950" style={{ marginRight: 4 }} />
<span style={{ color: "#3fb950", fontSize: 11 }}>Copied</span>
</>
) : (
<>
<Copy size={12} style={{ marginRight: 4 }} />
<span style={{ fontSize: 11 }}>Copy</span>
</>
)}
</button>
)}
</div>
<div style={viewerBodyStyle}>
{!hasContent ? (
<div style={emptyTextStyle}>No content available for this node.</div>
) : activeMode === "source" ? (
<pre style={sourcePreStyle}>
<code style={sourceCodeStyle}>{rawContent}</code>
</pre>
) : (
<div style={previewStyle}>
<ReactMarkdown
remarkPlugins={[remarkGfm]}
components={{
// C-1: react-markdown passes a HAST `node` prop (the raw AST
// Element) to every custom component override via passNode:true.
// In React 19 any unknown prop spreads onto a native element are
// serialised as HTML attributes, producing node="[object Object]"
// on every rendered link. Fix: destructure `node` by name so it
// is explicitly discarded, then spread `...rest` to preserve all
// other legitimate HAST/remark-gfm attributes — e.g. the `id`,
// `aria-describedby`, `aria-label`, `data-footnote-ref`,
// `data-footnote-backref`, and `class` attrs that GFM footnotes
// require for correct in-page navigation and accessibility.
//
// C-2: fragment links (#anchor, GFM footnote backlinks) must
// navigate within the current document. External links continue
// to use target="_blank" with noopener noreferrer.
//
// eslint-disable-next-line @typescript-eslint/no-unused-vars
a: ({ href, children, title, node: _node, ...rest }) => {
if (!isSafeUrl(href)) {
return <span style={{ color: GRAPH_THEME.ui.text.muted, textDecoration: "line-through" }}>{children}</span>;
}
// isSafeUrl returning true guarantees href is a non-empty string.
const safeHref = href ?? "";
// Fragment links (#section, footnote backlinks like
// #user-content-fnref-1) are in-document anchors. Opening them
// in a new tab would break GFM footnote back-navigation.
const isFragment = safeHref.startsWith("#");
if (isFragment) {
return (
<a href={safeHref} title={title} style={linkStyle} {...rest}>
{children}
</a>
);
}
return (
<a href={safeHref} title={title} target="_blank" rel="noopener noreferrer" style={linkStyle} {...rest}>
{children}
<ExternalLink size={10} style={{ marginLeft: 3, verticalAlign: "middle", display: "inline" }} />
</a>
);
},
img: ({ src, alt }) => (
<span style={imageBadgeStyle} title={src || "Image"}>
<ImageIcon size={12} style={{ marginRight: 5 }} />
<span>Image: {alt || src || "unlabeled"}</span>
</span>
),
h1: ({ children }) => <h1 style={h1Style}>{children}</h1>,
h2: ({ children }) => <h2 style={h2Style}>{children}</h2>,
h3: ({ children }) => <h3 style={h3Style}>{children}</h3>,
h4: ({ children }) => <h4 style={h4Style}>{children}</h4>,
p: ({ children }) => <p style={{ margin: "0 0 8px 0" }}>{children}</p>,
ul: ({ children }) => <ul style={{ margin: "0 0 8px 0", paddingLeft: 18 }}>{children}</ul>,
ol: ({ children }) => <ol style={{ margin: "0 0 8px 0", paddingLeft: 18 }}>{children}</ol>,
li: ({ children }) => <li style={{ marginBottom: 3 }}>{children}</li>,
blockquote: ({ children }) => <blockquote style={blockquoteStyle}>{children}</blockquote>,
hr: () => <hr style={{ border: "none", borderTop: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`, margin: "10px 0" }} />,
table: ({ children }) => (
<div style={{ width: "100%", overflowX: "auto", margin: "8px 0", borderRadius: 6, border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>
<table style={{ width: "100%", borderCollapse: "collapse", fontSize: 12 }}>{children}</table>
</div>
),
thead: ({ children }) => <thead style={{ background: "rgba(255, 255, 255, 0.04)" }}>{children}</thead>,
tbody: ({ children }) => <tbody>{children}</tbody>,
tr: ({ children }) => <tr style={{ borderBottom: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</tr>,
th: ({ children }) => <th style={{ padding: "6px 8px", textAlign: "left", fontWeight: 700, color: GRAPH_THEME.ui.text.strong, borderRight: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</th>,
td: ({ children }) => <td style={{ padding: "6px 8px", color: GRAPH_THEME.ui.text.body, borderRight: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}` }}>{children}</td>,
pre: ({ children }) => <pre style={preBlockStyle}>{children}</pre>,
// C-1: discard `node` here too — code elements are custom components
// and would otherwise receive node="[object Object]" in the DOM.
code: ({ className: codeClass, children }) => {
const isInline = !codeClass && typeof children === "string" && !children.includes("\n");
return (
<code style={isInline ? inlineCodeStyle : blockCodeStyle}>
{children}
</code>
);
},
}}
>
{rawContent}
</ReactMarkdown>
</div>
)}
</div>
</div>
);
}
/* ─── Styles ──────────────────────────────────────────────────────── */
const viewerContainerStyle: CSSProperties = {
display: "flex",
flexDirection: "column",
background: "rgba(255, 255, 255, 0.025)",
border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
borderRadius: 12,
overflow: "hidden",
};
const viewerHeaderStyle: CSSProperties = {
display: "flex",
alignItems: "center",
justifyContent: "space-between",
padding: "6px 10px",
background: "rgba(0, 0, 0, 0.2)",
borderBottom: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
};
const tabBtnStyle: CSSProperties = {
display: "inline-flex",
alignItems: "center",
padding: "4px 9px",
borderRadius: 6,
border: "1px solid transparent",
background: "transparent",
color: GRAPH_THEME.ui.text.muted,
fontSize: 12,
fontWeight: 600,
cursor: "pointer",
transition: "all 150ms ease",
};
const activeTabBtnStyle: CSSProperties = {
background: GRAPH_THEME.ui.timeline.playheadSoft,
border: `1px solid ${GRAPH_THEME.ui.control.activeBorder}`,
color: GRAPH_THEME.ui.timeline.playhead,
};
const copyBtnStyle: CSSProperties = {
display: "inline-flex",
alignItems: "center",
padding: "3px 8px",
borderRadius: 6,
border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
background: "rgba(255, 255, 255, 0.04)",
color: GRAPH_THEME.ui.text.subtle,
fontSize: 11,
cursor: "pointer",
};
const viewerBodyStyle: CSSProperties = {
padding: 12,
maxHeight: 380,
overflowY: "auto",
};
const emptyTextStyle: CSSProperties = {
color: GRAPH_THEME.ui.text.muted,
fontSize: 12,
lineHeight: 1.5,
fontStyle: "italic",
};
const sourcePreStyle: CSSProperties = {
margin: 0,
padding: 10,
borderRadius: 8,
background: "rgba(0, 0, 0, 0.3)",
border: "1px solid rgba(255, 255, 255, 0.05)",
overflowX: "auto",
};
const sourceCodeStyle: CSSProperties = {
fontFamily: "'JetBrains Mono', 'Fira Code', monospace",
fontSize: 12,
lineHeight: 1.6,
color: GRAPH_THEME.ui.text.strong,
whiteSpace: "pre-wrap",
wordBreak: "break-word",
userSelect: "text",
};
const previewStyle: CSSProperties = {
color: GRAPH_THEME.ui.text.body,
fontSize: 13,
lineHeight: 1.6,
wordBreak: "break-word",
};
const h1Style: CSSProperties = {
fontSize: 16,
fontWeight: 700,
color: GRAPH_THEME.ui.text.strong,
marginTop: 8,
marginBottom: 6,
paddingBottom: 3,
borderBottom: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
};
const h2Style: CSSProperties = {
fontSize: 14,
fontWeight: 700,
color: GRAPH_THEME.ui.text.strong,
marginTop: 8,
marginBottom: 4,
};
const h3Style: CSSProperties = {
fontSize: 13,
fontWeight: 600,
color: GRAPH_THEME.ui.text.strong,
marginTop: 6,
marginBottom: 4,
};
const h4Style: CSSProperties = {
fontSize: 12,
fontWeight: 600,
color: GRAPH_THEME.ui.text.strong,
marginTop: 4,
marginBottom: 2,
};
const blockquoteStyle: CSSProperties = {
margin: "8px 0",
padding: "6px 12px",
borderLeft: `3px solid ${GRAPH_THEME.ui.timeline.playhead}`,
background: "rgba(98, 226, 205, 0.05)",
borderRadius: "0 6px 6px 0",
color: GRAPH_THEME.ui.text.body,
fontStyle: "italic",
};
const linkStyle: CSSProperties = {
color: "#79c0ff",
textDecoration: "underline",
textUnderlineOffset: "3px",
wordBreak: "break-all",
};
const imageBadgeStyle: CSSProperties = {
display: "inline-flex",
alignItems: "center",
padding: "3px 7px",
background: "rgba(255, 255, 255, 0.04)",
border: `1px solid ${GRAPH_THEME.ui.surface.panelBorder}`,
borderRadius: 6,
color: GRAPH_THEME.ui.text.muted,
fontSize: 11,
margin: "3px 0",
};
const inlineCodeStyle: CSSProperties = {
fontFamily: "'JetBrains Mono', monospace",
fontSize: 12,
padding: "2px 5px",
borderRadius: 4,
background: "rgba(255, 255, 255, 0.07)",
color: "#e6edf3",
border: "1px solid rgba(255, 255, 255, 0.08)",
};
const preBlockStyle: CSSProperties = {
margin: "8px 0",
padding: 10,
borderRadius: 8,
background: "rgba(0, 0, 0, 0.35)",
border: "1px solid rgba(255, 255, 255, 0.08)",
overflowX: "auto",
};
const blockCodeStyle: CSSProperties = {
fontFamily: "'JetBrains Mono', monospace",
fontSize: 12,
lineHeight: 1.5,
color: "#e6edf3",
};
explorer/tests/markdownContentViewer.test.ts
+264
View File
@@ -0,0 +1,264 @@
import test from "node:test";
import assert from "node:assert/strict";
import React from "react";
import { renderToString } from "react-dom/server";
(globalThis as any).React = React;
import { isSafeUrl, MarkdownContentViewer } from "../src/workspaces/GraphWorkspace/MarkdownContentViewer.tsx";
test("isSafeUrl permits safe http, https, and mailto URLs and relative paths", () => {
assert.equal(isSafeUrl("https://example.com"), true);
assert.equal(isSafeUrl("http://localhost:8000"), true);
assert.equal(isSafeUrl("mailto:user@example.com"), true);
assert.equal(isSafeUrl("#section-1"), true);
assert.equal(isSafeUrl("/relative/path"), true);
});
test("isSafeUrl rejects protocol-relative URLs and dangerous schemes", () => {
// Protocol-relative URLs (must be blocked)
assert.equal(isSafeUrl("//evil.com"), false);
assert.equal(isSafeUrl("//localhost:8000"), false);
assert.equal(isSafeUrl("//"), false);
// Dangerous schemes
assert.equal(isSafeUrl("javascript:alert('xss')"), false);
assert.equal(isSafeUrl("JAVASCRIPT:alert(1)"), false);
assert.equal(isSafeUrl("data:text/html;base64,PHNjcmlwdD4="), false);
assert.equal(isSafeUrl("vbscript:MsgBox(1)"), false);
assert.equal(isSafeUrl(""), false);
assert.equal(isSafeUrl(undefined), false);
});
// ─── C URL contract: whitespace-only strings ────────────────────────────────
// The CommonMark parser normalises whitespace-only link destinations to "" so
// these values are unreachable through normal markdown rendering. However, the
// function is exported and its direct-call contract must be correct.
test("isSafeUrl rejects whitespace-only strings (contract correctness)", () => {
assert.equal(isSafeUrl(" "), false, "single space must be rejected");
assert.equal(isSafeUrl("\t"), false, "tab must be rejected");
assert.equal(isSafeUrl("\n"), false, "newline must be rejected");
assert.equal(isSafeUrl(" "), false, "multiple spaces must be rejected");
assert.equal(isSafeUrl(" \t\n "), false, "mixed whitespace must be rejected");
});
test("renders Preview mode with formatted Markdown elements and tabs", () => {
const markdown = `# Main Title\n\n**Bold Statement**\n\n* Item A\n* Item B`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content: markdown, defaultMode: "preview" }));
// Tab buttons are present
assert.equal(html.includes("Preview"), true);
assert.equal(html.includes("Source"), true);
assert.equal(html.includes("Copy"), true);
// Formatted preview elements
assert.equal(html.includes("Main Title"), true);
assert.equal(html.includes("Bold Statement"), true);
assert.equal(html.includes("<strong>Bold Statement</strong>"), true);
assert.equal(html.includes("Item A"), true);
assert.equal(html.includes("Item B"), true);
});
test("renders Source mode with exact unmodified text inside pre/code", () => {
const markdown = `# Title 🚀\n\n * Indented item\n\n\`\`\`python\ndef test():\n return "α + β"\n\`\`\``;
const html = renderToString(React.createElement(MarkdownContentViewer, { content: markdown, defaultMode: "source" }));
assert.equal(html.includes("<pre"), true);
assert.equal(html.includes("<code"), true);
assert.equal(html.includes("# Title 🚀"), true);
assert.equal(html.includes(" * Indented item"), true);
assert.equal(html.includes('return &quot;α + β&quot;'), true);
});
test("renders raw HTML safely as escaped text without executing elements", () => {
const dangerousHtml = `<script>alert("XSS")</script><iframe src="https://evil.com"></iframe>`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content: dangerousHtml, defaultMode: "preview" }));
// Script and iframe tags must NOT be rendered as active DOM tags
assert.equal(html.includes("<script>"), false);
assert.equal(html.includes("<iframe"), false);
// Content is escaped as text
assert.equal(html.includes("&lt;script&gt;"), true);
});
// ─── C-1: HAST node prop must not reach the DOM ─────────────────────────────
// react-markdown passes a HAST `node` (Element) object to custom component
// overrides. Before this fix, ...props spread caused React 19 to serialise it
// as node="[object Object]" on every <a> and <code> element.
test("rendered links do not expose the HAST node object as a DOM attribute", () => {
const content = `[Example](https://example.com)\n\nInline \`code\` here.`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// The rendered HTML must not contain the serialised HAST object
assert.equal(html.includes("node="), false, "node= attribute must not appear in rendered HTML");
assert.equal(html.includes("[object Object]"), false, "serialised HAST object must not appear in rendered HTML");
// The link must still render correctly with the right href
assert.equal(html.includes('href="https://example.com"'), true, "href must be present");
});
// ─── C-2: Fragment links must not open in a new tab ─────────────────────────
// Links to in-document anchors such as #section or GFM footnote backlinks like
// #user-content-fn-1 must stay in the current document. Only external links
// use target="_blank".
test("fragment links render in the current document without target blank", () => {
const content = `[Jump to section](#introduction)\n\n[External](https://example.com)`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// Fragment link must have the href
assert.equal(html.includes('href="#introduction"'), true, "fragment href must be present");
// Confirm no target=_blank attribute appears anywhere near the fragment link.
// We check that the output contains a fragment href WITHOUT target="_blank"
// by verifying the two strings are not both present (the external link has
// target blank; the fragment link must not).
const fragmentLinkIdx = html.indexOf('href="#introduction"');
assert.notEqual(fragmentLinkIdx, -1, "fragment link must be rendered");
// Inspect the 80 chars around the fragment href — should not contain target
const fragmentContext = html.slice(Math.max(0, fragmentLinkIdx - 10), fragmentLinkIdx + 90);
assert.equal(fragmentContext.includes('target="_blank"'), false, "fragment link must not have target=_blank");
// External link must still have target blank
assert.equal(html.includes('href="https://example.com"'), true, "external href must be present");
assert.equal(html.includes('target="_blank"'), true, "external link must have target=_blank");
assert.equal(html.includes('rel="noopener noreferrer"'), true, "external link must have rel");
});
test("GFM footnote backlinks render without target blank", () => {
// GFM footnote syntax: footnote ref in text + definition below
const content = `See the note[^1] for more.\n\n[^1]: This is the footnote text.`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// The footnote reference link (#user-content-fn-1) and backlink
// (#user-content-fnref-1) are fragment links and must not open in a new tab.
// We verify no fragment href is paired with target=_blank.
// Extract all href="#..." occurrences and confirm none is adjacent to target=_blank.
const anchorMatches = [...html.matchAll(/href="#[^"]*"/g)];
assert.ok(anchorMatches.length > 0, "GFM footnotes must produce fragment links");
for (const match of anchorMatches) {
const start = match.index ?? 0;
const context = html.slice(Math.max(0, start - 10), start + 120);
assert.equal(
context.includes('target="_blank"'),
false,
`fragment link ${match[0]} must not have target=_blank`,
);
}
});
// ─── C-1-R: GFM footnote attributes must be preserved (regression test) ─────
// The C-1 fix (removing the HAST `node` prop) must NOT silently drop other
// legitimate HAST attributes. remark-gfm generates the following on footnote
// links that are required for correct in-page navigation and accessibility:
//
// Footnote reference anchor:
// id="user-content-fnref-1" ← backlink target
// data-footnote-ref="true"
// aria-describedby="footnote-label"
//
// Footnote back-link anchor:
// data-footnote-backref=""
// aria-label="Back to reference 1" ← screen-reader label
// class="data-footnote-backref"
//
// If these are absent, clicking the ↩ back-link cannot scroll back to the
// in-text reference, and screen readers cannot announce the backlink purpose.
test("GFM footnote links preserve generated id, aria, and class attributes", () => {
const content = `See the note[^1] for more.\n\n[^1]: This is the footnote text.`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// The HAST `node` object must not appear serialised as a DOM attribute.
assert.equal(html.includes("node="), false, "node= attribute must not appear in HTML");
assert.equal(html.includes("[object Object]"), false, "serialised HAST object must not appear in HTML");
// Footnote reference anchor must retain its id so the backlink can navigate to it.
assert.equal(
html.includes('id="user-content-fnref-1"'),
true,
"footnote reference anchor must retain id for back-navigation",
);
// Footnote backlink must retain its aria-label for screen-reader accessibility.
assert.equal(
html.includes('aria-label="Back to reference 1"'),
true,
"footnote backlink must retain aria-label for accessibility",
);
// Footnote backlink must retain its class attribute.
assert.equal(
html.includes('class="data-footnote-backref"'),
true,
"footnote backlink must retain class attribute",
);
});
test("renders safe links as <a> with target blank and unclickable span for unsafe links", () => {
const content = `[Safe Link](https://getsemantica.ai)\n\n[Unsafe Scheme](javascript:alert(1))\n\n[Protocol Relative](//evil.com)`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// Safe link renders as <a> with security attributes
assert.equal(html.includes('href="https://getsemantica.ai"'), true);
assert.equal(html.includes('target="_blank"'), true);
assert.equal(html.includes('rel="noopener noreferrer"'), true);
// Unsafe links do NOT render as <a> tags
assert.equal(html.includes('href="javascript:alert(1)"'), false);
assert.equal(html.includes('href="//evil.com"'), false);
assert.equal(html.includes("Unsafe Scheme"), true);
assert.equal(html.includes("Protocol Relative"), true);
});
test("renders remote images as safe placeholder badges instead of <img> tags", () => {
const content = `![System Diagram](https://example.com/diagram.png)`;
const html = renderToString(React.createElement(MarkdownContentViewer, { content, defaultMode: "preview" }));
// No <img> tag rendered
assert.equal(html.includes("<img"), false);
// Image placeholder badge rendered
assert.equal(html.includes("Image:"), true);
assert.equal(html.includes("System Diagram"), true);
});
test("renders clear empty-state message when content is empty or null", () => {
const emptyHtml = renderToString(React.createElement(MarkdownContentViewer, { content: "" }));
assert.equal(emptyHtml.includes("No content available for this node."), true);
const nullHtml = renderToString(React.createElement(MarkdownContentViewer, { content: null }));
assert.equal(nullHtml.includes("No content available for this node."), true);
});
test("renders plain text cleanly without requiring Markdown formatting", () => {
const plainText = "Plain entity summary text without markdown formatting.";
const html = renderToString(React.createElement(MarkdownContentViewer, { content: plainText, defaultMode: "preview" }));
assert.equal(html.includes(plainText), true);
});
test("handles very large Markdown content without failure", () => {
const largeContent = `# Large Knowledge Node\n\n` + "Structured observation paragraph. ".repeat(400);
assert.equal(largeContent.length > 10000, true);
const html = renderToString(React.createElement(MarkdownContentViewer, { content: largeContent, defaultMode: "preview" }));
assert.equal(html.includes("Large Knowledge Node"), true);
});
// ─── H-2: Stale copied state lifecycle (SSR-compatible portion) ─────────────
// Full state-transition testing (Node A → copy → Node B) requires an interactive
// framework. The lifecycle correctness is guaranteed by the render-phase
// previous-prop synchronisation pattern: a `copiedForContent` state value tracks
// the content for which the copied indicator was set; when `content` changes, the
// mismatch is detected during render and `copied` is reset to false in the same
// React batch, before the new node's UI is painted. What we CAN verify in SSR
// is that the initial render for any content value shows the Copy button (not the
// Copied indicator), which confirms the initial state is always clean.
test("copy button always starts in un-copied state on initial render", () => {
const html = renderToString(React.createElement(MarkdownContentViewer, {
content: "# Some Node\n\nDescription text.",
defaultMode: "preview",
}));
// Initial render must show 'Copy', never 'Copied'
assert.equal(html.includes("Copy"), true, "Copy button must be present on initial render");
assert.equal(html.includes("Copied"), false, "Copied indicator must NOT be present on initial render");
});
pyproject.toml
+3 -3
View File
@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
[project]
name = "semantica"
version = "0.6.5"
description = "Accountability and context layer for AI agents. Context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
version = "0.6.6"
description = "Graph-Native Infrastructure for Context and Accountable AI Systems: context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
readme = "README.md"
license = { text = "MIT" }
@@ -103,7 +103,7 @@ Discord = "https://discord.gg/sV34vps5hH"
llm-openai = ["openai>=1.0.0"]
llm-groq = ["groq>=0.4.0"]
llm-gemini = ["google-genai>=0.1.0"]
llm-anthropic = ["anthropic>=0.18.0"]
llm-anthropic = ["anthropic>=0.122.0"]
llm-ollama = ["ollama>=0.1.0"]
llm-deepseek = ["openai>=1.0.0"]
llm-litellm = ["litellm>=1.83.9"]
requirements-ci.txt
+3 -3
View File
@@ -159,9 +159,9 @@ annotated-types==0.8.0 \
--hash=sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7 \
--hash=sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0
# via pydantic
anthropic==0.121.0 \
--hash=sha256:6048713fa441e59e1cba8363171cd2a86273b25bd213e9c7ac70a523af88b011 \
--hash=sha256:e79d6e08ab3376602fc9a70d4d5ea3540817c76cf7e16658bed790834e1833d6
anthropic==0.122.0 \
--hash=sha256:45ec906452ffae6b5f7f0c53d01f50bfb7e4ce878d7ae8e4309d13171e557e67 \
--hash=sha256:ffec56ae96657c8d19fa575ec96f140f380c353a07ab7d61b92eb18ee6536601
# via semantica (pyproject.toml)
antlr4-python3-runtime==4.9.3 \
--hash=sha256:f224469b4168294902bb1efa80a8bf7855f24c99aef99cbefc1bcd3cce77881b
semantica/__init__.py
+1 -1
View File
@@ -10,7 +10,7 @@ Main exports:
- Config: Configuration management
"""
__version__ = "0.6.5"
__version__ = "0.6.6"
__author__ = "Semantica Contributors"
__license__ = "MIT"
semantica/export/export_provenance.py
+4 -3
View File
@@ -14,9 +14,10 @@ License: MIT
"""
from typing import Any, Optional
from datetime import datetime
import uuid
from ..utils.helpers import utc_now_iso
class ExporterWithProvenance:
"""Base exporter with provenance tracking."""
@@ -45,9 +46,9 @@ class ExporterWithProvenance:
def export(self, data: Any, destination: str, **kwargs):
"""Export data with provenance tracking."""
activity_started_at_time = datetime.utcnow().isoformat()
activity_started_at_time = utc_now_iso()
result = self._exporter.export(data, destination, **kwargs)
activity_ended_at_time = datetime.utcnow().isoformat()
activity_ended_at_time = utc_now_iso()
if self.provenance and self._prov_manager:
self._prov_manager.track_entity(
semantica/export/json_exporter.py
+34 -25
View File
@@ -24,14 +24,14 @@ License: MIT
"""
import json
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory, write_json_file
from ..utils.helpers import ensure_directory, utc_now_iso, write_json_file
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from .rdf_exporter import SEMANTICA_NS, mint_entity_iri, mint_relationship_iri
class JSONExporter:
@@ -265,11 +265,12 @@ class JSONExporter:
json_data = {
"@context": {
"@vocab": "https://semantica.dev/vocab/",
"semantica": SEMANTICA_NS,
"entities": {"@id": "semantica:entities", "@container": "@list"},
},
"entities": entities,
"metadata": {
"exported_at": datetime.now().isoformat(),
"exported_at": utc_now_iso(),
"entity_count": len(entities),
**options.get("metadata", {}),
},
@@ -294,6 +295,7 @@ class JSONExporter:
json_data = {
"@context": {
"@vocab": "https://semantica.dev/vocab/",
"semantica": SEMANTICA_NS,
"relationships": {
"@id": "semantica:relationships",
"@container": "@list",
@@ -301,7 +303,7 @@ class JSONExporter:
},
"relationships": relationships,
"metadata": {
"exported_at": datetime.now().isoformat(),
"exported_at": utc_now_iso(),
"relationship_count": len(relationships),
**options.get("metadata", {}),
},
@@ -339,7 +341,7 @@ class JSONExporter:
if include_metadata:
if "metadata" not in result:
result["metadata"] = {}
result["metadata"]["exported_at"] = datetime.now().isoformat()
result["metadata"]["exported_at"] = utc_now_iso()
if include_provenance:
result["metadata"]["format"] = "json"
@@ -349,7 +351,7 @@ class JSONExporter:
"data": data,
"count": len(data),
"metadata": {
"exported_at": datetime.now().isoformat(),
"exported_at": utc_now_iso(),
"format": "json" if include_provenance else None,
**options.get("metadata", {}),
},
@@ -358,7 +360,7 @@ class JSONExporter:
# Single value
return {
"value": data,
"metadata": {"exported_at": datetime.now().isoformat()}
"metadata": {"exported_at": utc_now_iso()}
if include_metadata
else {},
}
@@ -410,9 +412,9 @@ class JSONExporter:
# Add metadata and provenance if requested
if include_metadata:
jsonld["@id"] = f"https://semantica.dev/data/{datetime.now().isoformat()}"
jsonld["@id"] = f"https://semantica.dev/data/{utc_now_iso()}"
if include_provenance:
jsonld["semantica:exportedAt"] = datetime.now().isoformat()
jsonld["semantica:exportedAt"] = utc_now_iso()
jsonld["semantica:format"] = "json-ld"
return jsonld
@@ -444,7 +446,7 @@ class JSONExporter:
"nodes": kg.get("nodes", []),
"edges": kg.get("edges", []),
"metadata": {
"exported_at": datetime.now().isoformat(),
"exported_at": utc_now_iso(),
**kg.get("metadata", {}),
**options.get("metadata", {}),
},
@@ -481,7 +483,7 @@ class JSONExporter:
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
},
"@id": f"https://semantica.dev/graph/{datetime.now().isoformat()}",
"@id": f"https://semantica.dev/graph/{utc_now_iso()}",
"@type": "semantica:KnowledgeGraph",
}
@@ -495,14 +497,15 @@ class JSONExporter:
relationships = kg.get("relationships", [])
if relationships:
jsonld["semantica:relationships"] = [
self._relationship_to_jsonld(r) for r in relationships
self._relationship_to_jsonld(r, index)
for index, r in enumerate(relationships)
]
self.logger.debug(
f"Converted {len(relationships)} relationship(s) to JSON-LD"
)
# Add metadata
jsonld["semantica:exportedAt"] = datetime.now().isoformat()
jsonld["semantica:exportedAt"] = utc_now_iso()
if "metadata" in kg:
jsonld["semantica:metadata"] = kg["metadata"]
@@ -526,11 +529,13 @@ class JSONExporter:
Returns:
Dictionary in JSON-LD format representing the entity
"""
# Generate @id if not provided
entity_id = entity.get("id")
if not entity_id:
entity_text = entity.get("text") or entity.get("label", "unknown")
entity_id = f"semantica:entity/{entity_text}"
# Generate @id if not provided. Minted exactly as the RDF serializers
# mint it (#1101), so the JSON-LD and Turtle exports of one knowledge
# graph name the same entity with the same IRI. Interpolating the raw
# text into f"semantica:entity/{text}" produced an invalid IRI for any
# text containing a space, and a JSON-LD parser dropped the whole node.
entity_text = entity.get("text") or entity.get("label", "unknown")
entity_id = entity.get("id") or mint_entity_iri(entity_text)
jsonld = {
"@id": entity_id,
@@ -545,7 +550,9 @@ class JSONExporter:
return jsonld
def _relationship_to_jsonld(self, rel: Dict[str, Any]) -> Dict[str, Any]:
def _relationship_to_jsonld(
self, rel: Dict[str, Any], index: int = 0
) -> Dict[str, Any]:
"""
Convert relationship to JSON-LD format.
@@ -560,16 +567,18 @@ class JSONExporter:
- type: Relationship type (optional)
- confidence: Confidence score (optional)
- metadata: Metadata dictionary (optional)
index: Position of the relationship in the exported list, used when
minting an IRI for a relationship that arrived without an id
Returns:
Dictionary in JSON-LD format representing the relationship
"""
# Generate @id if not provided
rel_id = rel.get("id")
if not rel_id:
source_id = rel.get("source_id") or rel.get("source", "")
target_id = rel.get("target_id") or rel.get("target", "")
rel_id = f"semantica:rel/{source_id}_{target_id}"
# Generate @id if not provided, from the same mint the RDF serializers
# use, including the list index that separates two relationships
# sharing a pair of endpoints (#1101).
source_id = rel.get("source_id") or rel.get("source", "")
target_id = rel.get("target_id") or rel.get("target", "")
rel_id = rel.get("id") or mint_relationship_iri(index, source_id, target_id)
jsonld = {
"@id": rel_id,
semantica/export/owl_exporter.py
+362 -90
View File
@@ -22,8 +22,10 @@ Author: Semantica Contributors
License: MIT
"""
import re
from datetime import datetime
from pathlib import Path
from urllib.parse import quote
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
@@ -37,6 +39,9 @@ from ..utils.progress_tracker import get_progress_tracker
# PROV-exported URIs co-resolve under one shared namespace by default.
from ..provenance.manager import DEFAULT_BASE_URI
#: Module-level logger, for the classmethod helpers that have no instance.
logger = get_logger("owl_exporter")
class OWLExporter:
"""
@@ -225,6 +230,7 @@ class OWLExporter:
Returns:
String containing OWL-XML serialization
"""
esc_xml = self._escape_xml
ontology_uri = ontology.get("uri") or self.ontology_uri
ontology_name = ontology.get("name", "SemanticaOntology")
version = ontology.get("version") or self.version
@@ -237,97 +243,125 @@ class OWLExporter:
lines.append("")
# Ontology declaration
lines.append(f' <owl:Ontology rdf:about="{ontology_uri}">')
lines.append(f" <rdfs:label>{ontology_name}</rdfs:label>")
lines.append(f" <owl:versionInfo>{version}</owl:versionInfo>")
lines.append(f' <owl:Ontology rdf:about="{esc_xml(ontology_uri)}">')
lines.append(f" <rdfs:label>{esc_xml(ontology_name)}</rdfs:label>")
lines.append(f" <owl:versionInfo>{esc_xml(version)}</owl:versionInfo>")
if ontology.get("description"):
lines.append(
f' <rdfs:comment>{ontology.get("description")}</rdfs:comment>'
f' <rdfs:comment>{esc_xml(ontology.get("description"))}</rdfs:comment>'
)
lines.append(" </owl:Ontology>")
lines.append("")
class_index = self._class_iri_index(ontology, ontology_uri)
object_properties, data_properties = self._split_properties(ontology)
def _as_list(value):
if value is None:
return []
return value if isinstance(value, list) else [value]
# Classes
classes = ontology.get("classes", [])
for cls in classes:
class_uri = cls.get("uri") or cls.get("id", "")
for cls in ontology.get("classes", []) or []:
if not isinstance(cls, dict):
continue
class_uri = self._term_iri(cls, ontology_uri)
if not class_uri:
self.logger.warning(
"Skipping a class with no name, uri or id: it would serialise "
"as an empty rdf:about"
)
continue
class_name = cls.get("name") or cls.get("label", "")
lines.append(f' <owl:Class rdf:about="{class_uri}">')
lines.append(f" <rdfs:label>{class_name}</rdfs:label>")
lines.append(f' <owl:Class rdf:about="{esc_xml(class_uri)}">')
lines.append(f" <rdfs:label>{esc_xml(class_name)}</rdfs:label>")
if cls.get("comment"):
lines.append(f' <rdfs:comment>{cls.get("comment")}</rdfs:comment>')
comment = cls.get("comment") or cls.get("description")
if comment:
lines.append(f" <rdfs:comment>{esc_xml(comment)}</rdfs:comment>")
# Subclass relationships
if cls.get("subClassOf"):
parent = cls.get("subClassOf")
lines.append(f' <rdfs:subClassOf rdf:resource="{parent}"/>')
for parent in _as_list(cls.get("subClassOf") or cls.get("parent")):
parent_iri = self._resolve_class_ref(parent, ontology_uri, class_index)
if parent_iri:
lines.append(
f' <rdfs:subClassOf rdf:resource="{esc_xml(parent_iri)}"/>'
)
# Equivalent classes
if cls.get("equivalentClass"):
equiv = cls.get("equivalentClass")
lines.append(f' <owl:equivalentClass rdf:resource="{equiv}"/>')
for equiv in _as_list(cls.get("equivalentClass")):
equiv_iri = self._resolve_class_ref(equiv, ontology_uri, class_index)
if equiv_iri:
lines.append(
f' <owl:equivalentClass rdf:resource="{esc_xml(equiv_iri)}"/>'
)
lines.append(" </owl:Class>")
lines.append("")
# Object properties
object_properties = ontology.get("object_properties", [])
for prop in object_properties:
prop_uri = prop.get("uri") or prop.get("id", "")
prop_uri = self._term_iri(prop, ontology_uri)
if not prop_uri:
self.logger.warning(
"Skipping an object property with no name, uri or id"
)
continue
prop_name = prop.get("name") or prop.get("label", "")
lines.append(f' <owl:ObjectProperty rdf:about="{prop_uri}">')
lines.append(f" <rdfs:label>{prop_name}</rdfs:label>")
lines.append(f' <owl:ObjectProperty rdf:about="{esc_xml(prop_uri)}">')
lines.append(f" <rdfs:label>{esc_xml(prop_name)}</rdfs:label>")
if prop.get("comment"):
lines.append(f' <rdfs:comment>{prop.get("comment")}</rdfs:comment>')
comment = prop.get("comment") or prop.get("description")
if comment:
lines.append(f" <rdfs:comment>{esc_xml(comment)}</rdfs:comment>")
# Domain
if prop.get("domain"):
domain = prop.get("domain")
if isinstance(domain, list):
for d in domain:
lines.append(f' <rdfs:domain rdf:resource="{d}"/>')
else:
lines.append(f' <rdfs:domain rdf:resource="{domain}"/>')
for domain in _as_list(prop.get("domain")):
domain_iri = self._resolve_class_ref(domain, ontology_uri, class_index)
if domain_iri:
lines.append(
f' <rdfs:domain rdf:resource="{esc_xml(domain_iri)}"/>'
)
# Range
if prop.get("range"):
range_val = prop.get("range")
if isinstance(range_val, list):
for r in range_val:
lines.append(f' <rdfs:range rdf:resource="{r}"/>')
else:
lines.append(f' <rdfs:range rdf:resource="{range_val}"/>')
for range_val in _as_list(prop.get("range")):
range_iri = self._resolve_class_ref(range_val, ontology_uri, class_index)
if range_iri:
lines.append(
f' <rdfs:range rdf:resource="{esc_xml(range_iri)}"/>'
)
lines.append(" </owl:ObjectProperty>")
lines.append("")
# Data properties
data_properties = ontology.get("data_properties", [])
for prop in data_properties:
prop_uri = prop.get("uri") or prop.get("id", "")
prop_uri = self._term_iri(prop, ontology_uri)
if not prop_uri:
self.logger.warning("Skipping a data property with no name, uri or id")
continue
prop_name = prop.get("name") or prop.get("label", "")
lines.append(f' <owl:DatatypeProperty rdf:about="{prop_uri}">')
lines.append(f" <rdfs:label>{prop_name}</rdfs:label>")
lines.append(f' <owl:DatatypeProperty rdf:about="{esc_xml(prop_uri)}">')
lines.append(f" <rdfs:label>{esc_xml(prop_name)}</rdfs:label>")
if prop.get("comment"):
lines.append(f' <rdfs:comment>{prop.get("comment")}</rdfs:comment>')
comment = prop.get("comment") or prop.get("description")
if comment:
lines.append(f" <rdfs:comment>{esc_xml(comment)}</rdfs:comment>")
# Domain
if prop.get("domain"):
domain = prop.get("domain")
lines.append(f' <rdfs:domain rdf:resource="{domain}"/>')
for domain in _as_list(prop.get("domain")):
domain_iri = self._resolve_class_ref(domain, ontology_uri, class_index)
if domain_iri:
lines.append(
f' <rdfs:domain rdf:resource="{esc_xml(domain_iri)}"/>'
)
# Range
if prop.get("range"):
range_type = prop.get("range", "xsd:string")
lines.append(
f' <rdfs:range rdf:resource="http://www.w3.org/2001/XMLSchema#{range_type}"/>'
)
for range_val in _as_list(prop.get("range")):
range_iri = self._resolve_datatype_iri(range_val)
if range_iri:
lines.append(
f' <rdfs:range rdf:resource="{esc_xml(range_iri)}"/>'
)
lines.append(" </owl:DatatypeProperty>")
lines.append("")
@@ -335,6 +369,223 @@ class OWLExporter:
lines.append("</rdf:RDF>")
return "\n".join(lines)
# ── Ontology-dict normalisation ───────────────────────────────────────────
#
# OntologyGenerator emits a single `properties` list tagged with
# type/@type, while hand-authored ontologies use `object_properties` and
# `data_properties`. Both shapes are accepted; everything below works from
# the normalised view so the two cannot drift apart again (#1103).
_XSD_NS = "http://www.w3.org/2001/XMLSchema#"
#: Prefixes the generator and hand-authored ontologies actually use. A
#: prefixed name is not an absolute IRI: `owl:Thing` matches the generic
#: scheme grammar, so treating it as one produced <owl:Thing> as a domain,
#: which is a different term from http://www.w3.org/2002/07/owl#Thing.
_KNOWN_PREFIXES = {
"owl": "http://www.w3.org/2002/07/owl#",
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
"xsd": _XSD_NS,
"skos": "http://www.w3.org/2004/02/skos/core#",
"dc": "http://purl.org/dc/elements/1.1/",
"dcterms": "http://purl.org/dc/terms/",
"foaf": "http://xmlns.com/foaf/0.1/",
"sem": "https://semantica.dev/ns#",
"semantica": "https://semantica.dev/ns#",
}
#: Schemes that really do introduce an absolute IRI without `//`.
_ABSOLUTE_SCHEMES = ("urn:", "doi:", "mailto:", "tag:", "uuid:")
@classmethod
def _is_absolute_iri(cls, value: str) -> bool:
if not isinstance(value, str):
return False
value = value.strip()
if "://" in value:
return bool(re.match(r"^[A-Za-z][A-Za-z0-9+.\-]*://", value))
return value.lower().startswith(cls._ABSOLUTE_SCHEMES)
@classmethod
def _expand_prefixed_name(cls, value: str) -> str:
"""Expand a known prefixed name, or return "" when it cannot be expanded."""
prefix, _, local = value.partition(":")
namespace = cls._KNOWN_PREFIXES.get(prefix)
return f"{namespace}{local}" if namespace and local else ""
@staticmethod
def _iri_safe(local: str) -> str:
"""
Percent-encode a local name so it can sit inside <>.
A name is free text. "Customer Account" pasted onto a base gives an IRI
with a space in it, which rdflib only warns about and Oxigraph rejects
with "Invalid IRI code point".
"""
return quote(local.strip(), safe="~._-!$&'()*+,;=:@/?")
@classmethod
def _join_iri(cls, base: str, local: str) -> str:
"""Append a local name to a base IRI, respecting hash and slash bases."""
if not base:
return ""
local = cls._iri_safe(local)
if not local:
return ""
separator = "" if base.endswith(("#", "/", ":")) else "#"
return f"{base}{separator}{local}"
@classmethod
def _term_iri(cls, term: Dict[str, Any], base: str) -> str:
"""
Resolve the IRI of a class or property.
Returns "" when the term carries nothing usable, so the caller can skip
it. Interpolating an empty string into <> silently resolves against the
parser's base — under rdflib that is the current working directory — and
collapses every such term onto one subject.
"""
for key in ("uri", "iri", "id"):
value = term.get(key)
if isinstance(value, str) and value.strip():
value = value.strip()
return value if cls._is_absolute_iri(value) else cls._join_iri(base, value)
name = term.get("name") or term.get("label")
if isinstance(name, str) and name.strip():
return cls._join_iri(base, name.strip())
return ""
@classmethod
def _class_iri_index(cls, ontology: Dict[str, Any], base: str) -> Dict[str, str]:
"""Map class name and label to the IRI that class is actually exported under."""
index: Dict[str, str] = {}
for class_def in ontology.get("classes", []) or []:
if not isinstance(class_def, dict):
continue
iri = cls._term_iri(class_def, base)
if not iri:
continue
for key in (class_def.get("name"), class_def.get("label")):
if isinstance(key, str) and key.strip():
index.setdefault(key.strip(), iri)
return index
@classmethod
def _resolve_class_ref(cls, value: Any, base: str, index: Dict[str, str]) -> str:
"""
Resolve a domain/range reference to an absolute IRI.
The generator writes bare class names here. Looking the name up in the
class index first means a reference always lands on the IRI that class
was exported under, rather than on a re-derived guess.
"""
if not isinstance(value, str) or not value.strip():
return ""
value = value.strip()
if cls._is_absolute_iri(value):
return value
if value in index:
return index[value]
if ":" in value:
return cls._expand_prefixed_name(value)
return cls._join_iri(base, value)
@classmethod
def _resolve_datatype_iri(cls, value: Any) -> str:
"""
Resolve a data property range to an absolute datatype IRI.
Accepts "string", "xsd:string" and a full IRI alike. The previous
`xsd:{range}` interpolation doubled the prefix whenever the generator
had already written "xsd:string".
"""
if not isinstance(value, str) or not value.strip():
return ""
value = value.strip()
if value.startswith(("xsd:", "XSD:")):
return cls._XSD_NS + value.split(":", 1)[1]
if cls._is_absolute_iri(value):
return value
return cls._XSD_NS + value
@classmethod
def _ttl_datatype_ref(cls, value: Any) -> str:
"""
Render a data property range for Turtle.
XSD datatypes are written with the xsd: prefix the header already
declares; anything else is written as a full IRI. Both are the same
term, this only keeps the compact style the module was written in.
"""
iri = cls._resolve_datatype_iri(value)
if not iri:
return ""
if iri.startswith(cls._XSD_NS):
return f"xsd:{iri[len(cls._XSD_NS):]}"
return f"<{iri}>"
@classmethod
def _split_properties(
cls, ontology: Dict[str, Any]
) -> "tuple[List[Dict[str, Any]], List[Dict[str, Any]]]":
"""
Return (object_properties, data_properties) across both dict shapes.
A property listed under an explicit key keeps that key's kind. A
property from the generator's combined `properties` list is classified
by its own type/@type, defaulting to a data property.
"""
object_props: List[Dict[str, Any]] = []
data_props: List[Dict[str, Any]] = []
for prop in ontology.get("object_properties", []) or []:
if isinstance(prop, dict):
object_props.append(prop)
for prop in ontology.get("data_properties", []) or []:
if isinstance(prop, dict):
data_props.append(prop)
skipped = 0
untyped = []
for prop in ontology.get("properties", []) or []:
if not isinstance(prop, dict):
skipped += 1
continue
kind = str(prop.get("type") or "").strip().lower()
owl_type = str(prop.get("@type") or "").strip().lower()
if kind in ("object", "objectproperty") or owl_type.endswith("objectproperty"):
object_props.append(prop)
else:
if not kind and not owl_type:
untyped.append(prop.get("name") or prop.get("uri") or "<unnamed>")
data_props.append(prop)
if skipped:
logger.warning(
f"Skipped {skipped} entr(y/ies) in 'properties' that are not "
"dictionaries and cannot be exported"
)
if untyped:
logger.warning(
"Exported as data properties because they declare no type or "
f"@type: {', '.join(str(name) for name in untyped)}"
)
return object_props, data_props
@staticmethod
def _escape_xml(value: Any) -> str:
"""Escape a value for safe embedding in XML text or an attribute value."""
return (
str(value)
.replace("&", "&amp;")
.replace("<", "&lt;")
.replace(">", "&gt;")
.replace('"', "&quot;")
)
@staticmethod
def _escape_ttl_str(value: str) -> str:
"""Escape a string value for safe embedding in a Turtle string literal."""
@@ -387,62 +638,83 @@ class OWLExporter:
lines.append(self._ttl_block(ontology_uri, "owl:Ontology", onto_predicates))
lines.append("")
class_index = self._class_iri_index(ontology, ontology_uri)
object_properties, data_properties = self._split_properties(ontology)
def _as_list(value):
if value is None:
return []
return value if isinstance(value, list) else [value]
# Classes
for cls in ontology.get("classes", []):
class_uri = cls.get("uri") or cls.get("id", "")
for cls in ontology.get("classes", []) or []:
if not isinstance(cls, dict):
continue
class_uri = self._term_iri(cls, ontology_uri)
if not class_uri:
self.logger.warning(
"Skipping a class with no name, uri or id: it would serialise as <>"
)
continue
class_name = cls.get("name") or cls.get("label", "")
predicates = [f'rdfs:label "{esc(class_name)}"']
comment = cls.get("comment")
comment = cls.get("comment") or cls.get("description")
if comment:
predicates.append(f'rdfs:comment "{esc(comment)}"')
sub_class = cls.get("subClassOf")
if sub_class:
predicates.append(f"rdfs:subClassOf <{sub_class}>")
equiv = cls.get("equivalentClass")
if equiv:
predicates.append(f"owl:equivalentClass <{equiv}>")
for parent in _as_list(cls.get("subClassOf") or cls.get("parent")):
parent_iri = self._resolve_class_ref(parent, ontology_uri, class_index)
if parent_iri:
predicates.append(f"rdfs:subClassOf <{parent_iri}>")
for equiv in _as_list(cls.get("equivalentClass")):
equiv_iri = self._resolve_class_ref(equiv, ontology_uri, class_index)
if equiv_iri:
predicates.append(f"owl:equivalentClass <{equiv_iri}>")
lines.append(self._ttl_block(class_uri, "owl:Class", predicates))
lines.append("")
# Object properties
for prop in ontology.get("object_properties", []):
prop_uri = prop.get("uri") or prop.get("id", "")
for prop in object_properties:
prop_uri = self._term_iri(prop, ontology_uri)
if not prop_uri:
self.logger.warning(
"Skipping an object property with no name, uri or id"
)
continue
prop_name = prop.get("name") or prop.get("label", "")
predicates = [f'rdfs:label "{esc(prop_name)}"']
comment = prop.get("comment")
comment = prop.get("comment") or prop.get("description")
if comment:
predicates.append(f'rdfs:comment "{esc(comment)}"')
domain = prop.get("domain")
if domain:
if isinstance(domain, list):
for d in domain:
predicates.append(f"rdfs:domain <{d}>")
else:
predicates.append(f"rdfs:domain <{domain}>")
range_val = prop.get("range")
if range_val:
if isinstance(range_val, list):
for r in range_val:
predicates.append(f"rdfs:range <{r}>")
else:
predicates.append(f"rdfs:range <{range_val}>")
for domain in _as_list(prop.get("domain")):
domain_iri = self._resolve_class_ref(domain, ontology_uri, class_index)
if domain_iri:
predicates.append(f"rdfs:domain <{domain_iri}>")
for range_val in _as_list(prop.get("range")):
range_iri = self._resolve_class_ref(range_val, ontology_uri, class_index)
if range_iri:
predicates.append(f"rdfs:range <{range_iri}>")
lines.append(self._ttl_block(prop_uri, "owl:ObjectProperty", predicates))
lines.append("")
# Data properties
for prop in ontology.get("data_properties", []):
prop_uri = prop.get("uri") or prop.get("id", "")
for prop in data_properties:
prop_uri = self._term_iri(prop, ontology_uri)
if not prop_uri:
self.logger.warning("Skipping a data property with no name, uri or id")
continue
prop_name = prop.get("name") or prop.get("label", "")
predicates = [f'rdfs:label "{esc(prop_name)}"']
comment = prop.get("comment")
comment = prop.get("comment") or prop.get("description")
if comment:
predicates.append(f'rdfs:comment "{esc(comment)}"')
domain = prop.get("domain")
if domain:
predicates.append(f"rdfs:domain <{domain}>")
range_type = prop.get("range")
if range_type:
predicates.append(f"rdfs:range xsd:{range_type}")
for domain in _as_list(prop.get("domain")):
domain_iri = self._resolve_class_ref(domain, ontology_uri, class_index)
if domain_iri:
predicates.append(f"rdfs:domain <{domain_iri}>")
for range_val in _as_list(prop.get("range")):
range_ref = self._ttl_datatype_ref(range_val)
if range_ref:
predicates.append(f"rdfs:range {range_ref}")
lines.append(self._ttl_block(prop_uri, "owl:DatatypeProperty", predicates))
lines.append("")
semantica/export/rdf_exporter.py
+142 -36
View File
@@ -30,6 +30,7 @@ License: MIT
"""
from pathlib import Path
from decimal import Decimal, InvalidOperation
from typing import Any, Dict, List, Optional, Set, Union
from ..utils.exceptions import ProcessingError, ValidationError
@@ -49,6 +50,75 @@ DEFAULT_ENTITY_TYPE = f"{SEMANTICA_NS}Entity"
#: Written when a relationship carries no type of its own. Same reasoning.
DEFAULT_RELATION_TYPE = f"{SEMANTICA_NS}related_to"
#: The one datatype every serializer writes confidence in.
#:
#: The four paths used to disagree: Turtle wrote the value bare, which the
#: Turtle grammar reads as xsd:decimal, N-Triples typed it xsd:float, RDF/XML
#: emitted a plain literal with no datatype, and JSON-LD emitted a native
#: number, which becomes xsd:double. Those are four distinct RDF terms for one
#: value (issue #1100).
#:
#: xsd:decimal is the choice because it is what the Turtle path already
#: produced, so the most used output is unchanged, and because it is exact:
#: xsd:float is 32 bit binary, and cannot represent 0.9 or 0.95 at all.
CONFIDENCE_DATATYPE = "http://www.w3.org/2001/XMLSchema#decimal"
#: Largest power of ten a confidence may carry. xsd:decimal has no exponent
#: notation, so a value has to be written out in full, and a compact literal
#: such as "1e100000000" would expand to a hundred million digits.
MAX_CONFIDENCE_EXPONENT = 100
def normalize_confidence(value: Any) -> Optional[str]:
"""
Return the canonical xsd:decimal lexical form of a confidence value.
Returns None when the value cannot be a decimal, so callers omit the triple
rather than writing something the vocabulary contradicts. The Turtle path
used to interpolate the raw value, so a confidence of "high" produced
`semantica:confidence high .` and made the whole document unparseable
(issue #1102).
Booleans are rejected. `bool` is a subclass of `int` in Python, so True
would otherwise silently become a confidence of 1.
"""
if value is None or isinstance(value, bool):
return None
if isinstance(value, str):
value = value.strip()
if not value:
return None
if not isinstance(value, (int, float, str, Decimal)):
return None
try:
decimal_value = Decimal(str(value))
except (InvalidOperation, ValueError, TypeError):
return None
# NaN and the infinities are Decimal values with no xsd:decimal form.
if not decimal_value.is_finite():
return None
# xsd:decimal has no exponent notation, so writing one means expanding it.
# "1e100000000" is eleven characters that expand to a hundred million, and
# the export path continues past validation errors, so a single malformed
# field could exhaust memory. Nothing near this magnitude is a confidence.
if not -MAX_CONFIDENCE_EXPONENT <= decimal_value.adjusted() <= MAX_CONFIDENCE_EXPONENT:
return None
# `str(Decimal("0.00001"))` gives "0.00001", but a float that has already
# been through repr can arrive as "1e-05", which xsd:decimal does not allow.
formatted = format(decimal_value, "f")
if "." in formatted:
formatted = formatted.rstrip("0").rstrip(".") or "0"
# Decimal keeps the sign of zero, so 0.0 and -0.0 would serialise as two
# distinct RDF terms and defeat the point of a canonical form.
if formatted.lstrip("-").strip("0.") == "":
formatted = "0"
return formatted
def mint_entity_iri(text: str) -> str:
"""Mint a stable IRI for an entity that arrived without an id.
@@ -395,11 +465,20 @@ class RDFSerializer:
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
text = entity.get("text") or entity.get("label", "")
confidence = entity.get("confidence", 1.0)
confidence = normalize_confidence(entity.get("confidence", 1.0))
lines.append(f"<{entity_id}> a <{entity_type}> ;")
lines.append(f' semantica:text "{text}" ;')
lines.append(f" semantica:confidence {confidence} .")
if confidence is None:
self.logger.warning(
f"Entity {entity_id} has a confidence that is not a number "
f"({entity.get('confidence')!r}), so no confidence is written"
)
lines.append(f' semantica:text "{text}" .')
else:
lines.append(f' semantica:text "{text}" ;')
lines.append(
f' semantica:confidence "{confidence}"^^<{CONFIDENCE_DATATYPE}> .'
)
lines.append("")
# Convert relationships to RDF triplets
@@ -576,15 +655,22 @@ class RDFSerializer:
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
text = entity.get("text") or entity.get("label", "")
confidence = entity.get("confidence", 1.0)
confidence = normalize_confidence(entity.get("confidence", 1.0))
# RDF/XML syntax: rdf:Description with rdf:about
lines.append(f' <rdf:Description rdf:about="{entity_id}">')
lines.append(f' <rdf:type rdf:resource="{entity_type}"/>')
lines.append(f" <semantica:text>{text}</semantica:text>")
lines.append(
f" <semantica:confidence>{confidence}</semantica:confidence>"
)
if confidence is None:
self.logger.warning(
f"Entity {entity_id} has a confidence that is not a number "
f"({entity.get('confidence')!r}), so no confidence is written"
)
else:
lines.append(
f' <semantica:confidence rdf:datatype="{CONFIDENCE_DATATYPE}">'
f"{confidence}</semantica:confidence>"
)
lines.append(" </rdf:Description>")
lines.append("")
@@ -649,41 +735,52 @@ class RDFSerializer:
# Convert entities to JSON-LD
entities = rdf_data.get("entities", [])
for entity in entities:
# Generate @id if not provided
entity_id = entity.get("id")
if not entity_id:
entity_text = entity.get("text", "")
entity_id = f"semantica:entity/{entity_text}"
# Generate @id if not provided. Minted the same way the Turtle and
# N-Triples paths mint it (#1101), so one knowledge graph carries
# the same node identity whichever serializer wrote it. The former
# f"semantica:entity/{text}" interpolated the raw text into an IRI:
# any entity whose text contained a space produced an invalid IRI
# and was dropped in full by a JSON-LD parser, silently.
entity_id = entity.get("id") or mint_entity_iri(entity.get("text", ""))
jsonld["@graph"].append(
{
"@id": entity_id,
"@type": entity.get("type", "semantica:Entity"),
"semantica:text": entity.get("text") or entity.get("label", ""),
"semantica:confidence": entity.get("confidence", 1.0),
node = {
"@id": entity_id,
"@type": entity.get("type", "semantica:Entity"),
"semantica:text": entity.get("text") or entity.get("label", ""),
}
confidence = normalize_confidence(entity.get("confidence", 1.0))
if confidence is None:
self.logger.warning(
f"Entity {entity_id} has a confidence that is not a number "
f"({entity.get('confidence')!r}), so no confidence is written"
)
else:
# A native JSON number becomes xsd:double once expanded, so the
# value is written as a typed literal instead.
node["semantica:confidence"] = {
"@value": confidence,
"@type": CONFIDENCE_DATATYPE,
}
)
jsonld["@graph"].append(node)
# Convert relationships to JSON-LD
relationships = rdf_data.get("relationships", [])
for rel in relationships:
# Generate @id if not provided
rel_id = rel.get("id")
if not rel_id:
source_id = rel.get("source_id", "")
target_id = rel.get("target_id", "")
rel_id = f"semantica:rel/{source_id}_{target_id}"
for index, rel in enumerate(relationships):
# Endpoints are resolved both ways, as serialize_to_turtle resolves
# them: a relationship carrying source/target rather than
# source_id/target_id used to hash into f"semantica:rel/_", so every
# such relationship in an export collapsed onto one node and their
# types and endpoints merged.
source = rel.get("source_id") or rel.get("source", "")
target = rel.get("target_id") or rel.get("target", "")
rel_id = rel.get("id") or mint_relationship_iri(index, source, target)
jsonld["@graph"].append(
{
"@id": rel_id,
"@type": "semantica:Relationship",
"semantica:source": {
"@id": rel.get("source_id") or rel.get("source")
},
"semantica:target": {
"@id": rel.get("target_id") or rel.get("target")
},
"semantica:source": {"@id": source},
"semantica:target": {"@id": target},
"semantica:type": rel.get("type", "related_to"),
}
)
@@ -746,11 +843,20 @@ class RDFSerializer:
f'{subject} {expand_uri("semantica:text")} "{safe_text}" .'
)
# Confidence property
confidence = entity.get("confidence")
if confidence is not None:
# Confidence property. The default matches the other serializers,
# which have always written one; omitting it here was half of why
# Turtle and N-Triples of one KG were different graphs (#1100).
raw_confidence = entity.get("confidence", 1.0)
confidence = normalize_confidence(raw_confidence)
if confidence is None:
self.logger.warning(
f"Entity {entity.get('id')} has a confidence that is not a "
f"number ({raw_confidence!r}), so no confidence is written"
)
else:
lines.append(
f'{subject} {expand_uri("semantica:confidence")} "{confidence}"^^<http://www.w3.org/2001/XMLSchema#float> .'
f'{subject} {expand_uri("semantica:confidence")} '
f'"{confidence}"^^<{CONFIDENCE_DATATYPE}> .'
)
# Convert relationships
semantica/export/report_generator.py
+4 -5
View File
@@ -25,12 +25,11 @@ License: MIT
import html
import json
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory
from ..utils.helpers import ensure_directory, utc_now_iso
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -252,7 +251,7 @@ class ReportGenerator:
# Build report data with summary
report_data = {
"title": "Quality Assurance Report",
"generated_at": datetime.now().isoformat(),
"generated_at": utc_now_iso(),
"metrics": quality_metrics,
"summary": self._generate_quality_summary(quality_metrics),
}
@@ -278,7 +277,7 @@ class ReportGenerator:
"""
report_data = {
"title": "Analysis Report",
"generated_at": datetime.now().isoformat(),
"generated_at": utc_now_iso(),
"analysis": analysis_results,
"summary": self._generate_analysis_summary(analysis_results),
}
@@ -304,7 +303,7 @@ class ReportGenerator:
"""
report_data = {
"title": "Framework Metrics Report",
"generated_at": datetime.now().isoformat(),
"generated_at": utc_now_iso(),
"metrics": metrics,
"summary": self._generate_metrics_summary(metrics),
}
semantica/export/yaml_exporter.py
+6 -6
View File
@@ -22,7 +22,6 @@ License: MIT
"""
from collections.abc import Mapping
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
@@ -33,6 +32,7 @@ from ..utils.helpers import (
_require_recognized_keys,
ensure_directory,
normalize_graph_payload,
utc_now_iso,
)
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -216,7 +216,7 @@ class SemanticNetworkYAMLExporter:
records = normalize_graph_payload(semantic_network)
yaml_data = {
"metadata": {
"exported_at": datetime.now().isoformat(),
"exported_at": utc_now_iso(),
"version": "1.0",
**semantic_network.get("metadata", {}),
},
@@ -309,7 +309,7 @@ class SemanticNetworkYAMLExporter:
if include_metadata:
yaml_data["metadata"] = {
"exported_at": datetime.now().isoformat(),
"exported_at": utc_now_iso(),
"entity_count": len(entities),
}
@@ -333,7 +333,7 @@ class SemanticNetworkYAMLExporter:
if include_properties:
yaml_data["metadata"] = {
"exported_at": datetime.now().isoformat(),
"exported_at": utc_now_iso(),
"relationship_count": len(relationships),
}
@@ -370,7 +370,7 @@ class SemanticNetworkYAMLExporter:
}
yaml_data["metadata"] = {
"exported_at": datetime.now().isoformat(),
"exported_at": utc_now_iso(),
"triplet_count": len(triplets),
}
@@ -410,7 +410,7 @@ class SemanticNetworkYAMLExporter:
yaml_data = {
"pipeline_stage": pipeline_stage,
"metadata": {
"extracted_at": datetime.now().isoformat(),
"extracted_at": utc_now_iso(),
**extracted_data.get("metadata", {}),
},
"semantic_network": semantic_network,
semantica/kg/graph_builder.py
+18 -2
View File
@@ -313,7 +313,10 @@ class GraphBuilder:
if not isinstance(relationship, dict):
continue
for endpoint in ("source", "target"):
for endpoint, endpoint_alias in (
("source", "source_id"),
("target", "target_id"),
):
endpoint_id = relationship.get(endpoint)
try:
canonical_id = endpoint_map.get(endpoint_id)
@@ -322,8 +325,21 @@ class GraphBuilder:
# to report rather than making graph construction fail here.
continue
if canonical_id is not None and canonical_id != endpoint_id:
if canonical_id is None:
continue
remapped = canonical_id != endpoint_id
if remapped:
relationship[endpoint] = canonical_id
if (
endpoint_alias in relationship
and relationship[endpoint_alias] != canonical_id
):
relationship[endpoint_alias] = canonical_id
remapped = True
if remapped:
remapped_count += 1
if remapped_count:
semantica/ontology/engine.py
+12
View File
@@ -203,6 +203,8 @@ class OntologyEngine:
include_inherited: bool = True,
severity: str = "Violation",
quality_tier: str = "standard",
target_namespace: Optional[str] = None,
attach_domainless_properties: bool = False,
validate_output: bool = False,
**options,
) -> str:
@@ -217,6 +219,12 @@ class OntologyEngine:
include_inherited: Propagate parent class property shapes to child shapes.
severity: Default severity — "Violation", "Warning", or "Info".
quality_tier: Constraint completeness — "basic", "standard" (default), "strict".
target_namespace: Namespace sh:targetClass and sh:path are expanded in,
when the ontology supplies no absolute IRIs. Separate from base_uri,
which says where the shape resources themselves live.
attach_domainless_properties: Attach properties with no declared domain
to every node shape. Off by default: it states a constraint the
ontology does not.
validate_output: Syntax-check output via rdflib before returning.
Returns:
@@ -236,12 +244,16 @@ class OntologyEngine:
or (ns.get("base_uri") if isinstance(ns, dict) else None)
or "https://semantica.dev/shapes/"
)
# These two are constructor-only on SHACLGenerator, so forwarding
# them through generate(**options) silently dropped them.
generator = SHACLGenerator(
base_uri=resolved_base,
shapes_uri=shapes_uri,
include_inherited=include_inherited,
severity=severity,
quality_tier=quality_tier,
target_namespace=target_namespace,
attach_domainless_properties=attach_domainless_properties,
)
graph = generator.generate(ontology, **options)
self.progress.update_tracking(tracking_id, message="Serializing SHACL graph")
semantica/ontology/ontology_generator.py
+194 -23
View File
@@ -30,9 +30,11 @@ Author: Semantica Contributors
License: MIT
"""
import re
from dataclasses import dataclass, field, replace as dataclass_replace
from datetime import datetime
from typing import Any, Dict, List, Optional
from urllib.parse import quote
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
@@ -438,10 +440,13 @@ class OntologyGenerator:
entities=entities, relationships=relationships, classes=classes, **prop_options
)
# Add types to classes
# Add types to classes.
# ClassInferrer sets "uri": None when it was given no namespace manager,
# so the key is present and a `not in` guard never fires: every class
# then reached the exporters with no IRI at all (#1103).
for cls in classes:
cls["@type"] = "owl:Class"
if "uri" not in cls:
if not cls.get("uri"):
cls["uri"] = self.namespace_manager.generate_class_iri(cls["name"])
# Add types to properties
@@ -451,7 +456,7 @@ class OntologyGenerator:
else:
prop["@type"] = "owl:DatatypeProperty"
if "uri" not in prop:
if not prop.get("uri"):
prop["uri"] = self.namespace_manager.generate_property_iri(prop["name"])
return {
@@ -692,12 +697,19 @@ class OntologyOptimizer:
Returns:
Improved ontology
"""
# Ensure all classes have required fields
# Ensure all classes have required fields. Both guards used `not in`,
# which misses a key that is present and None, and the URI fallback
# assigned a bare class name where an absolute IRI is required (#1103).
#
# The base comes from the ontology being optimized. OntologyOptimizer
# holds no namespace manager, so reaching for one here would raise
# AttributeError on every ontology carrying a class with no URI.
base_uri = ontology.get("uri") or DEFAULT_ONTOLOGY_BASE_URI
classes = ontology.get("classes", [])
for cls in classes:
if "uri" not in cls:
cls["uri"] = cls.get("name", "Entity")
if "label" not in cls:
if not cls.get("uri"):
cls["uri"] = _mint_term_iri(base_uri, cls.get("name", "Entity"))
if not cls.get("label"):
cls["label"] = cls.get("name", "Entity")
# Ensure all properties have domains and ranges
@@ -744,6 +756,25 @@ class NodeShape:
severity: str = "Violation"
#: Used when an ontology carries no URI of its own.
DEFAULT_ONTOLOGY_BASE_URI = "https://semantica.dev/ontology/"
def _mint_term_iri(base_uri: str, name: str) -> str:
"""
Mint an absolute IRI for a term from a base and a name.
The name is percent-encoded: names are free text, and "Customer Account"
pasted onto a base gives an IRI with a space in it, which strict parsers
reject outright.
"""
local = quote(str(name).strip(), safe="~._-!$&'()*+,;=:@")
if not local:
local = "Entity"
separator = "" if base_uri.endswith(("#", "/", ":")) else "#"
return f"{base_uri}{separator}{local}"
@dataclass
class SHACLGraph:
"""Internal model representing the complete SHACL shapes graph."""
@@ -751,6 +782,13 @@ class SHACLGraph:
shapes_uri: str
node_shapes: List[NodeShape] = field(default_factory=list)
prefixes: Dict[str, str] = field(default_factory=dict)
# Bare names mapped to the absolute IRI the data uses for them. Shapes are
# indexed internally by name; this is what those names expand to at
# serialisation time (#1104). Classes and properties are kept apart because
# a property may legitimately share a class's name, and a single map would
# silently give it the class's IRI.
class_iris: Dict[str, str] = field(default_factory=dict)
property_iris: Dict[str, str] = field(default_factory=dict)
class SHACLGenerator:
@@ -783,8 +821,23 @@ class SHACLGenerator:
include_inherited: bool = True,
severity: str = "Violation",
quality_tier: str = "standard",
target_namespace: Optional[str] = None,
attach_domainless_properties: bool = False,
config: Optional[Dict[str, Any]] = None,
):
"""
Args:
base_uri: Namespace the shape resources themselves live in.
target_namespace: Namespace the terms being validated live in, used
to expand sh:targetClass and sh:path when the ontology does not
supply absolute IRIs. This is deliberately separate from
base_uri: shapes that target their own namespace match nothing,
and pySHACL reports that as conforming (#1104).
attach_domainless_properties: When True, a property with no declared
domain is attached to every node shape, which is the pre-0.6.6
behaviour. It invents a constraint the ontology never stated, so
it is off by default (#1105).
"""
self.logger = get_logger("ontology_shacl")
self.progress_tracker = get_progress_tracker()
self.base_uri = base_uri.rstrip("/") + "/"
@@ -792,6 +845,8 @@ class SHACLGenerator:
self.include_inherited = include_inherited
self.severity = severity
self.quality_tier = quality_tier
self.target_namespace = target_namespace
self.attach_domainless_properties = attach_domainless_properties
self.config = config or {}
# ── Public API ────────────────────────────────────────────────────────────
@@ -829,10 +884,15 @@ class SHACLGenerator:
"ex": base_uri,
}
target_ns = self._resolve_target_namespace(ontology, base_uri)
prefixes["ex"] = target_ns
graph = SHACLGraph(
base_uri=base_uri,
shapes_uri=self.shapes_uri,
prefixes=prefixes,
class_iris=self._build_term_index(classes, target_ns),
property_iris=self._build_term_index(properties, target_ns),
)
self.progress_tracker.update_tracking(tracking_id, message="Generating node shapes")
@@ -901,6 +961,80 @@ class SHACLGenerator:
# ── Internal pipeline stages ──────────────────────────────────────────────
_DEFAULT_TARGET_NAMESPACE = "https://semantica.dev/ns#"
@staticmethod
def _is_absolute_iri(value: Any) -> bool:
return isinstance(value, str) and bool(
re.match(r"^[A-Za-z][A-Za-z0-9+.\-]*:", value.strip())
)
@staticmethod
def _join_iri(base: str, local: str) -> str:
separator = "" if base.endswith(("#", "/", ":")) else "#"
return f"{base}{separator}{local}"
def _resolve_target_namespace(self, ontology: Dict[str, Any], base_uri: str) -> str:
"""
Decide which namespace sh:targetClass and sh:path are expanded in.
base_uri says where the shapes live. It is only the right answer here
when the ontology declared it, meaning shapes and terms deliberately
share a namespace. Falling back to the shapes namespace produces shapes
that target terms no data graph uses.
"""
if self.target_namespace:
return self.target_namespace
namespace = ontology.get("namespace")
if isinstance(namespace, dict) and namespace.get("base_uri"):
return str(namespace["base_uri"])
# An IRI already carried by a term is the most reliable evidence of
# where the data lives, so prefer it over any configured default.
for terms in (ontology.get("classes"), ontology.get("properties")):
for term in terms or []:
if not isinstance(term, dict):
continue
for key in ("uri", "iri", "id"):
value = term.get(key)
if self._is_absolute_iri(value):
value = value.strip()
cut = max(value.rfind("#"), value.rfind("/"))
if cut != -1:
return value[: cut + 1]
if ontology.get("uri") and self._is_absolute_iri(ontology["uri"]):
return str(ontology["uri"])
if base_uri != self.base_uri:
return base_uri
return self._DEFAULT_TARGET_NAMESPACE
def _build_term_index(
self, terms: List[Dict[str, Any]], target_ns: str
) -> Dict[str, str]:
"""Map each term's name to the absolute IRI it expands to."""
index: Dict[str, str] = {}
for term in terms or []:
if not isinstance(term, dict):
continue
name = term.get("name")
if not isinstance(name, str) or not name.strip():
continue
name = name.strip()
iri = ""
for key in ("uri", "iri", "id"):
value = term.get(key)
if isinstance(value, str) and value.strip():
value = value.strip()
iri = value if self._is_absolute_iri(value) else self._join_iri(target_ns, value)
break
index.setdefault(name, iri or self._join_iri(target_ns, name))
return index
def _build_class_index(
self, classes: List[Dict[str, Any]]
) -> Dict[str, Dict[str, Any]]:
@@ -949,13 +1083,23 @@ class SHACLGenerator:
self.logger.debug(
f"Property '{pname}' domain '{d}' has no matching node shape — skipped"
)
else:
# No domain declared → attach to all shapes
self.logger.debug(
f"Property '{pname}' has no domain — attaching to all node shapes"
elif self.attach_domainless_properties:
self.logger.warning(
f"Property '{pname}' declares no domain and is being attached "
"to every node shape because attach_domainless_properties is "
"set. This states a constraint the ontology does not."
)
for node_shape in graph.node_shapes:
node_shape.property_shapes.append(self._build_property_shape(prop))
else:
# Attaching here would state a constraint the ontology does not.
# With minCount 1 that invalidates every instance of every
# class, so the property is left unattached (#1105).
self.logger.warning(
f"Property '{pname}' declares no domain, so it is not attached "
"to any node shape. Declare a domain, or pass "
"attach_domainless_properties=True to restore the old behaviour."
)
def _build_property_shape(self, prop: Dict[str, Any]) -> PropertyShape:
ptype = prop.get("type", "")
@@ -1035,13 +1179,40 @@ class SHACLGenerator:
def _prefix_decls(self, graph: SHACLGraph) -> str:
return "\n".join(f"@prefix {p}: <{u}> ." for p, u in sorted(graph.prefixes.items()))
def _uri(self, graph: SHACLGraph, local: str) -> str:
"""Return a compact URI reference; fall back to ex:local for bare names."""
def _term_iri(self, graph: SHACLGraph, local: str, kind: str = "class") -> str:
"""
Resolve a class or property name to the absolute IRI the data uses.
Every serializer goes through here. Turtle alone was corrected at first,
which left JSON-LD and N-Triples still pasting names onto the shapes
namespace, so the shapes they produced went on matching nothing (#1104).
`kind` selects the index: a property may share a class's name, and the
two can carry different IRIs.
"""
if local.startswith("http://") or local.startswith("https://"):
return f"<{local}>"
return local
index = graph.property_iris if kind == "property" else graph.class_iris
resolved = index.get(local)
if resolved:
return resolved
# Fall back to the other index before giving up: sh:class names a class,
# but a caller may pass a term only registered on the other side.
other = graph.class_iris if kind == "property" else graph.property_iris
resolved = other.get(local)
if resolved:
return resolved
if ":" in local:
return local
return f"ex:{local}"
separator = "" if graph.base_uri.endswith(("#", "/", ":")) else "#"
return f"{graph.base_uri}{separator}{local}"
def _uri(self, graph: SHACLGraph, local: str, kind: str = "class") -> str:
"""Turtle-facing wrapper: the resolved IRI, in angle brackets."""
resolved = self._term_iri(graph, local, kind)
if resolved.startswith(("http://", "https://", "urn:")):
return f"<{resolved}>"
return resolved
def _serialize_turtle(self, graph: SHACLGraph) -> str:
lines = [self._prefix_decls(graph), ""]
@@ -1068,11 +1239,11 @@ class SHACLGenerator:
is_last = i == len(node_shape.property_shapes) - 1
terminator = " ." if is_last else " ;"
parts = [" sh:property ["]
parts.append(f" sh:path {self._uri(graph, ps.path)} ;")
parts.append(f' sh:path {self._uri(graph, ps.path, "property")} ;')
if ps.datatype:
parts.append(f" sh:datatype {ps.datatype} ;")
if ps.class_:
parts.append(f" sh:class {self._uri(graph, ps.class_)} ;")
parts.append(f' sh:class {self._uri(graph, ps.class_, "class")} ;')
if ps.min_count is not None:
parts.append(f" sh:minCount {ps.min_count} ;")
if ps.max_count is not None:
@@ -1113,7 +1284,7 @@ class SHACLGenerator:
node: Dict[str, Any] = {
"@id": shape_id,
"@type": "sh:NodeShape",
"sh:targetClass": {"@id": f"{graph.base_uri}{node_shape.target_class}"},
"sh:targetClass": {"@id": self._term_iri(graph, node_shape.target_class, "class")},
}
if node_shape.name:
node["sh:name"] = node_shape.name
@@ -1126,7 +1297,7 @@ class SHACLGenerator:
props = []
for ps in node_shape.property_shapes:
p: Dict[str, Any] = {
"sh:path": {"@id": f"{graph.base_uri}{ps.path}"}
"sh:path": {"@id": self._term_iri(graph, ps.path, "property")}
}
if ps.datatype:
dt = ps.datatype.replace(
@@ -1134,7 +1305,7 @@ class SHACLGenerator:
)
p["sh:datatype"] = {"@id": dt}
if ps.class_:
p["sh:class"] = {"@id": f"{graph.base_uri}{ps.class_}"}
p["sh:class"] = {"@id": self._term_iri(graph, ps.class_, "class")}
if ps.min_count is not None:
p["sh:minCount"] = ps.min_count
if ps.max_count is not None:
@@ -1167,7 +1338,7 @@ class SHACLGenerator:
for i, node_shape in enumerate(graph.node_shapes):
shape_uri = f"<{graph.base_uri}{node_shape.target_class}Shape>"
class_uri = f"<{graph.base_uri}{node_shape.target_class}>"
class_uri = f'<{self._term_iri(graph, node_shape.target_class, "class")}>'
t(shape_uri, f"<{RDF}type>", f"<{SHACL}NodeShape>")
t(shape_uri, f"<{SHACL}targetClass>", class_uri)
if node_shape.name:
@@ -1182,13 +1353,13 @@ class SHACLGenerator:
for j, ps in enumerate(node_shape.property_shapes):
bnode = f"_:ps{i}_{j}"
t(shape_uri, f"<{SHACL}property>", bnode)
prop_uri = f"<{graph.base_uri}{ps.path}>"
prop_uri = f'<{self._term_iri(graph, ps.path, "property")}>'
t(bnode, f"<{SHACL}path>", prop_uri)
if ps.datatype:
dt_uri = ps.datatype.replace("xsd:", XSD)
t(bnode, f"<{SHACL}datatype>", f"<{dt_uri}>")
if ps.class_:
t(bnode, f"<{SHACL}class>", f"<{graph.base_uri}{ps.class_}>")
t(bnode, f"<{SHACL}class>", f'<{self._term_iri(graph, ps.class_, "class")}>')
if ps.min_count is not None:
t(bnode, f"<{SHACL}minCount>", f'"{ps.min_count}"^^<{XSD}integer>')
if ps.max_count is not None:
semantica/ontology/vocabulary/semantica-ns.ttl
+13 -10
View File
@@ -53,16 +53,17 @@ this IRI.""" ;
sem:confidence a owl:DatatypeProperty ;
rdfs:label "confidence" ;
rdfs:range xsd:decimal ;
rdfs:comment """Extractor confidence in the assertion, on the unit interval.
Emitted for both entities and relationships, so the domain is left open rather
than tied to sem:Entity.
No rdfs:range is declared, deliberately. The Turtle serializer writes the value
bare, which the Turtle grammar reads as xsd:decimal, while the N-Triples
serializer types it xsd:float explicitly, and those two datatypes are disjoint.
Declaring either one would make the vocabulary contradict one of the exporters.
Issue #1100 tracks the disagreement; a range belongs here once the serializers
agree on one.""" ;
The range was left undeclared when this vocabulary first shipped, because the
four serializers disagreed: Turtle wrote the value bare, which the Turtle
grammar reads as xsd:decimal, N-Triples typed it xsd:float, RDF/XML wrote a
plain literal, and JSON-LD wrote a native number, which expands to xsd:double.
Declaring any one of them would have contradicted three exporters. Issue #1100
settled that on xsd:decimal, which every serializer now writes.""" ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:metadata a owl:AnnotationProperty ;
@@ -118,10 +119,12 @@ sem:relationships a owl:ObjectProperty ;
sem:exportedAt a owl:DatatypeProperty ;
rdfs:label "exported at" ;
rdfs:comment """When the export was written. Emitted as an ISO 8601 local
timestamp, so the range is xsd:dateTime rather than xsd:dateTimeStamp: the
values carry no timezone offset.""" ;
rdfs:range xsd:dateTime ;
rdfs:comment """When the export was written, as an ISO 8601 timestamp with
an explicit UTC offset. The range was xsd:dateTime while the exporters stamped
with a naive datetime.now(); with the offset present (#1114) the value is a
determinate instant, comparable against a timestamp written anywhere else, so
the range is the stricter xsd:dateTimeStamp, which requires the offset.""" ;
rdfs:range xsd:dateTimeStamp ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
sem:format a owl:DatatypeProperty ;
semantica/provenance/bridge_axiom.py
+3 -2
View File
@@ -61,9 +61,10 @@ License: MIT
from dataclasses import dataclass, field
from typing import Optional, Dict, Any, List
from datetime import datetime
import uuid
from ..utils.helpers import utc_now_iso
@dataclass
class BridgeAxiom:
@@ -280,7 +281,7 @@ class TranslationChain:
"type": layer_type,
"value": value,
"source": source,
"timestamp": datetime.utcnow().isoformat(),
"timestamp": utc_now_iso(),
**kwargs
}
self.layers.append(layer)
semantica/provenance/manager.py
+50 -15
View File
@@ -26,7 +26,7 @@ License: MIT
from typing import Optional, List, Dict, Any, Union
from collections.abc import Mapping
from datetime import datetime
from datetime import datetime, timezone
from contextlib import contextmanager
import copy
import inspect
@@ -36,8 +36,13 @@ import threading
from .schemas import ProvenanceEntry, SourceReference, AgentRecord, ActivityRecord
from .storage import ProvenanceStorage, InMemoryStorage, SQLiteStorage
from .integrity import compute_checksum, verify_checksum
from ..utils.helpers import to_utc_datetime, utc_now_iso
from ..utils.logging import get_logger
#: Sort key for an entry whose timestamp cannot be read as one, so an
#: unreadable value orders first instead of raising during a sort.
_EPOCH = datetime(1, 1, 1, tzinfo=timezone.utc)
# Issue #825, Part B Tier 3 — configurable base URI for export_prov(), shared
# with RDFExporter's NamespaceManager "semantica" entry (semantica/export/
# rdf_exporter.py) so KG-exported and PROV-exported URIs for the same
@@ -363,8 +368,8 @@ class ProvenanceManager:
source_quote=kwargs.get("source_quote"),
confidence=kwargs.get("confidence", 1.0),
metadata=metadata or {},
first_seen=existing.first_seen if existing else datetime.utcnow().isoformat(),
last_updated=datetime.utcnow().isoformat(),
first_seen=existing.first_seen if existing else utc_now_iso(),
last_updated=utc_now_iso(),
parent_entity_id=parent_id,
used_entities=list(kwargs.get("used_entities", [])),
activity_started_at_time=activity_info["activity_started_at_time"],
@@ -455,8 +460,8 @@ class ProvenanceManager:
source_location=kwargs.get("source_location"),
confidence=kwargs.get("confidence", 1.0),
metadata=metadata or {},
first_seen=datetime.utcnow().isoformat(),
last_updated=datetime.utcnow().isoformat(),
first_seen=utc_now_iso(),
last_updated=utc_now_iso(),
activity_started_at_time=activity_info["activity_started_at_time"],
activity_ended_at_time=activity_info["activity_ended_at_time"],
acted_on_behalf_of=kwargs.get("acted_on_behalf_of"),
@@ -534,7 +539,7 @@ class ProvenanceManager:
# split (issue #825, Part A item 4).
derived_from_id=parent_chunk_id,
metadata=metadata,
timestamp=datetime.utcnow().isoformat(),
timestamp=utc_now_iso(),
activity_started_at_time=activity_info["activity_started_at_time"],
activity_ended_at_time=activity_info["activity_ended_at_time"],
)
@@ -604,7 +609,7 @@ class ProvenanceManager:
**metadata,
**source.metadata
},
timestamp=datetime.utcnow().isoformat(),
timestamp=utc_now_iso(),
activity_started_at_time=activity_info["activity_started_at_time"],
activity_ended_at_time=activity_info["activity_ended_at_time"],
)
@@ -959,11 +964,27 @@ class ProvenanceManager:
Returns:
List of matching entries as dicts, sorted by timestamp ascending.
"""
matches = [
e for e in self.storage.retrieve_all()
if e.timestamp and start <= e.timestamp <= end
]
matches.sort(key=lambda e: e.timestamp)
# Compare instants, not spellings. Since #1114 new entries carry a
# +00:00 offset while entries written earlier do not, and a raw string
# comparison orders those two by length: an inclusive naive bound equal
# to a stored offset-bearing timestamp would sort below it and drop the
# record. A bound in another offset was mis-ordered the same way.
start_at = to_utc_datetime(start)
end_at = to_utc_datetime(end)
entries = [e for e in self.storage.retrieve_all() if e.timestamp]
if start_at is None or end_at is None:
# A bound this module cannot read as a timestamp keeps the historical
# string comparison rather than raising on a call that used to work.
matches = [e for e in entries if start <= e.timestamp <= end]
else:
matches = [
e for e in entries
if (at := to_utc_datetime(e.timestamp)) is not None
and start_at <= at <= end_at
]
matches.sort(key=lambda e: (to_utc_datetime(e.timestamp) or _EPOCH, e.timestamp))
return [e.to_dict() for e in matches]
def get_all_sources(self, entity_id: str) -> List[Dict[str, Any]]:
@@ -1071,7 +1092,7 @@ class ProvenanceManager:
entry = copy.deepcopy(existing)
entry.invalidated = True
entry.invalidated_at_time = datetime.utcnow().isoformat()
entry.invalidated_at_time = utc_now_iso()
entry.invalidated_by = agent_id
entry.invalidation_reason = reason
entry.previous_version_id = history_id
@@ -1163,8 +1184,22 @@ class ProvenanceManager:
"""
entries = self.storage.retrieve_all()
if since:
entries = [e for e in entries if getattr(e, "timestamp", "") >= since]
entries.sort(key=lambda e: getattr(e, "timestamp", ""))
since_at = to_utc_datetime(since)
if since_at is None:
entries = [e for e in entries
if getattr(e, "timestamp", "") >= since]
else:
entries = [
e for e in entries
if (at := to_utc_datetime(getattr(e, "timestamp", None)))
is not None and at >= since_at
]
entries.sort(
key=lambda e: (
to_utc_datetime(getattr(e, "timestamp", None)) or _EPOCH,
getattr(e, "timestamp", ""),
)
)
if format == "json":
return [
semantica/provenance/schemas.py
+3 -1
View File
@@ -30,6 +30,8 @@ from dataclasses import dataclass, field
from typing import Optional, List, Dict, Any
from datetime import datetime
from ..utils.helpers import utc_now_iso
@dataclass
class ProvenanceEntry:
@@ -91,7 +93,7 @@ class ProvenanceEntry:
source_quote: Optional[str] = None
# Temporal tracking (from kg.ProvenanceTracker)
timestamp: str = field(default_factory=lambda: datetime.utcnow().isoformat())
timestamp: str = field(default_factory=lambda: utc_now_iso())
first_seen: Optional[str] = None
last_updated: Optional[str] = None
semantica/utils/__init__.py
+6
View File
@@ -82,6 +82,9 @@ from .helpers import (
normalize_entities,
normalize_graph_payload,
parse_timestamp,
to_utc_datetime,
utc_now,
utc_now_iso,
read_json_file,
retry_on_error,
safe_filename,
@@ -193,6 +196,9 @@ __all__ = [
"get_file_size",
"format_timestamp",
"parse_timestamp",
"to_utc_datetime",
"utc_now",
"utc_now_iso",
"merge_dicts",
"chunk_list",
"flatten_dict",
semantica/utils/helpers.py
+59
View File
@@ -320,6 +320,65 @@ def format_timestamp(
return dt.strftime(format_str)
def utc_now() -> datetime:
"""
Current instant as a timezone-aware UTC datetime.
``datetime.now()`` reads the local clock and ``datetime.utcnow()`` reads UTC,
but both return a naive datetime, and the two are indistinguishable once
serialized: a consumer cannot tell which zone the value belongs to, and an
RDF timestamp without an offset is not comparable against one that has an
offset (a SPARQL FILTER drops it rather than reporting an error). Use this
for any timestamp that leaves the process.
Returns:
Current UTC time, timezone-aware
"""
return datetime.now(timezone.utc)
def utc_now_iso() -> str:
"""
Current instant as an ISO 8601 string carrying an explicit UTC offset.
Returns:
Timestamp string such as ``2026-08-19T14:19:04.229937+00:00``, which is
a valid ``xsd:dateTimeStamp`` and orders correctly against timestamps
written in any other timezone
"""
return utc_now().isoformat()
def to_utc_datetime(value: Union[str, datetime, None]) -> Optional[datetime]:
"""
Read an ISO 8601 timestamp as a timezone-aware UTC instant.
Timestamps written before #1114 carry no offset. They were produced by
``datetime.utcnow()``, so a missing offset is read as UTC: that keeps a
stored naive value and the same instant written with an offset comparing
equal, instead of ordering by how the timestamp happens to be spelled.
Args:
value: ISO 8601 string or datetime. ``Z`` is accepted as the offset.
Returns:
Timezone-aware UTC datetime, or None if the value cannot be read as a
timestamp, so callers can fall back rather than raise on stored data
"""
if value is None:
return None
if isinstance(value, datetime):
parsed = value
else:
try:
parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00"))
except ValueError:
return None
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=timezone.utc)
return parsed.astimezone(timezone.utc)
def parse_timestamp(timestamp_str: str, format_str: Optional[str] = None) -> datetime:
"""
Parse timestamp string to datetime.
tests/export/test_confidence_literal_typing.py
+212
View File
@@ -0,0 +1,212 @@
"""
Regression tests for #1100 and #1102.
#1100: the four RDF serializers rendered the same confidence value four
different ways. Turtle wrote it bare, which the Turtle grammar reads as
xsd:decimal. N-Triples typed it xsd:float. RDF/XML wrote a plain literal with
no datatype at all. JSON-LD emitted a native JSON number, which becomes
xsd:double. Those are four distinct RDF terms, so a FILTER matches at most one
of them and merging two exports of one graph yields two confidence values for
the same entity.
#1102: the Turtle path interpolated the value with no type check, so a
non-numeric confidence produced `semantica:confidence high .`, which is not
parseable Turtle. One bad field made the whole export unreadable.
"""
import json
import pytest
rdflib = pytest.importorskip("rdflib")
from rdflib import Graph, URIRef # noqa: E402
from rdflib.compare import isomorphic # noqa: E402
from semantica.export.rdf_exporter import RDFSerializer # noqa: E402
NS = "https://semantica.dev/ns#"
CONFIDENCE = URIRef(NS + "confidence")
XSD_DECIMAL = URIRef("http://www.w3.org/2001/XMLSchema#decimal")
def _kg(confidence):
entity = {"id": NS + "e1", "text": "Acme", "type": NS + "ORG"}
if confidence is not _ABSENT:
entity["confidence"] = confidence
return {"entities": [entity], "relationships": []}
_ABSENT = object()
def _graphs(kg):
"""Parse every serialization of one KG into a graph, keyed by format."""
serializer = RDFSerializer()
out = {}
out["turtle"] = Graph()
out["turtle"].parse(data=serializer.serialize_to_turtle(json.loads(json.dumps(kg))),
format="turtle")
out["ntriples"] = Graph()
out["ntriples"].parse(data=serializer.serialize_to_ntriples(json.loads(json.dumps(kg))),
format="nt")
out["rdfxml"] = Graph()
out["rdfxml"].parse(data=serializer.serialize_to_rdfxml(json.loads(json.dumps(kg))),
format="xml")
jsonld = serializer.serialize_to_jsonld(json.loads(json.dumps(kg)))
out["jsonld"] = Graph()
out["jsonld"].parse(
data=jsonld if isinstance(jsonld, str) else json.dumps(jsonld), format="json-ld"
)
return out
def _confidence_terms(graph):
return [o for _, p, o in graph if p == CONFIDENCE]
def test_every_serializer_agrees_on_the_confidence_term():
"""The heart of #1100. One value, one RDF term, whatever the format."""
terms = {}
for name, graph in _graphs(_kg(0.9)).items():
values = _confidence_terms(graph)
assert len(values) == 1, f"{name} emitted {len(values)} confidence triples"
terms[name] = values[0]
distinct = set(terms.values())
assert len(distinct) == 1, (
"the same confidence serialised as different RDF terms: "
+ ", ".join(f"{k}={v!r} ({v.datatype})" for k, v in terms.items())
)
def test_the_agreed_term_is_a_typed_decimal():
for name, graph in _graphs(_kg(0.9)).items():
term = _confidence_terms(graph)[0]
assert term.datatype == XSD_DECIMAL, f"{name} typed it {term.datatype}"
assert str(term) == "0.9", f"{name} wrote the lexical form {str(term)!r}"
def test_turtle_and_ntriples_are_the_same_graph():
"""#1100 as filed: two serializations of one KG must not be two graphs."""
graphs = _graphs(_kg(0.9))
assert isomorphic(graphs["turtle"], graphs["ntriples"]), (
"Turtle only:\n"
+ "\n".join(str(t) for t in set(graphs["turtle"]) - set(graphs["ntriples"]))
+ "\nN-Triples only:\n"
+ "\n".join(str(t) for t in set(graphs["ntriples"]) - set(graphs["turtle"]))
)
def test_a_numeric_string_is_accepted():
for name, graph in _graphs(_kg("0.85")).items():
terms = _confidence_terms(graph)
assert terms, f"{name} dropped a usable numeric string"
assert terms[0].datatype == XSD_DECIMAL
assert str(terms[0]) == "0.85"
def test_an_integer_confidence_is_accepted():
for name, graph in _graphs(_kg(1)).items():
terms = _confidence_terms(graph)
assert terms, f"{name} dropped an integer confidence"
assert terms[0].datatype == XSD_DECIMAL
def test_a_small_value_is_not_written_in_exponent_notation():
"""1e-05 is a valid Python repr and an invalid xsd:decimal lexical form."""
for name, graph in _graphs(_kg(0.00001)).items():
term = _confidence_terms(graph)[0]
assert "e" not in str(term).lower(), f"{name} wrote {str(term)!r}"
assert term.value is not None, f"{name} produced an ill-typed literal"
# ── #1102 ────────────────────────────────────────────────────────────────────
@pytest.mark.parametrize(
"bad", ["high", "", "not a number", None, True, False, [0.9], {"v": 0.9},
float("nan"), float("inf")],
)
def test_an_unusable_confidence_never_breaks_the_export(bad):
"""`semantica:confidence high .` made the whole Turtle document unparseable."""
serializer = RDFSerializer()
kg = _kg(bad)
turtle = serializer.serialize_to_turtle(json.loads(json.dumps(kg, default=str)))
graph = Graph()
graph.parse(data=turtle, format="turtle") # must not raise
assert _confidence_terms(graph) == [], (
f"{bad!r} was emitted as a confidence value: {_confidence_terms(graph)}"
)
# The rest of the entity must survive.
assert (URIRef(NS + "e1"), URIRef(NS + "text"), rdflib.Literal("Acme")) in graph
def test_an_unusable_confidence_is_dropped_consistently_everywhere():
for name, graph in _graphs(_kg("high")).items():
assert _confidence_terms(graph) == [], f"{name} still emitted it"
def test_all_serializers_stay_parseable_with_an_unusable_confidence():
graphs = _graphs(_kg("high")) # each parse would raise on malformed output
assert isomorphic(graphs["turtle"], graphs["ntriples"])
def test_the_emitted_datatype_matches_the_shipped_vocabulary():
"""
Drift guard. The vocabulary shipped with no rdfs:range on sem:confidence
precisely because the serializers disagreed. Now that they agree, the range
is declared, and the two must not drift apart again.
"""
from rdflib import RDFS
from semantica.export.rdf_exporter import CONFIDENCE_DATATYPE
from semantica.ontology.vocabulary import vocabulary_turtle
vocabulary = Graph()
vocabulary.parse(data=vocabulary_turtle(), format="turtle")
declared = list(vocabulary.objects(URIRef(NS + "confidence"), RDFS.range))
assert declared, "sem:confidence declares no rdfs:range"
assert str(declared[0]) == CONFIDENCE_DATATYPE, (
f"vocabulary says {declared[0]}, serializers write {CONFIDENCE_DATATYPE}"
)
# ── Review findings on the first revision of this fix ────────────────────────
@pytest.mark.parametrize("huge", ["1e100000000", "1E1000000", "-1e999999", 10**400])
def test_an_absurd_magnitude_is_rejected_not_expanded(huge):
"""
xsd:decimal has no exponent notation, so the value has to be written out.
"1e100000000" is eleven characters that expand to a hundred million digits,
and the export path continues past validation errors.
"""
from semantica.export.rdf_exporter import normalize_confidence
assert normalize_confidence(huge) is None
def test_a_legitimately_small_confidence_is_still_accepted():
from semantica.export.rdf_exporter import normalize_confidence
assert normalize_confidence(1e-9) == "0.000000001"
def test_signed_zero_is_normalized():
"""0.0 and -0.0 would otherwise be two distinct RDF terms."""
from semantica.export.rdf_exporter import normalize_confidence
assert normalize_confidence(0.0) == normalize_confidence(-0.0) == "0"
def test_signed_zero_gives_one_term_across_serializers():
positive = {n: _confidence_terms(g)[0] for n, g in _graphs(_kg(0.0)).items()}
negative = {n: _confidence_terms(g)[0] for n, g in _graphs(_kg(-0.0)).items()}
assert set(positive.values()) | set(negative.values()) == set(positive.values())
assert len(set(positive.values())) == 1
tests/export/test_jsonld_iri_minting.py
+139
View File
@@ -0,0 +1,139 @@
"""The JSON-LD paths must mint the same IRIs as the RDF paths (issue #1101).
#1101 was fixed for the Turtle, N-Triples and RDF/XML serializers: an entity
arriving without an id gets a deterministic IRI in the declared namespace. The
JSON-LD paths were left interpolating the entity's own text into
``f"semantica:entity/{text}"`` and the relationship's endpoints into
``f"semantica:rel/{source}_{target}"``, which fails three ways:
* a text containing a space produces an invalid IRI, and a JSON-LD parser drops
the whole node rather than complaining, so the entity vanishes from the export;
* relationships carrying ``source``/``target`` rather than ``source_id``/
``target_id`` all minted ``semantica:rel/_``, so every one of them collapsed
onto a single node whose types and endpoints merged;
* the JSON-LD @id and the Turtle IRI for one entity disagreed, so the two
serializations of one knowledge graph were two different graphs.
``JSONExporter.export_entities`` and ``export_relationships`` also wrote
``semantica:entities`` into a context that never declared the ``semantica``
prefix, which a JSON-LD processor reads as an IRI in the scheme ``semantica`` —
the original #1101 failure mode, on a path the first fix did not cover.
"""
import json
import pytest
from semantica.export.json_exporter import JSONExporter
from semantica.export.rdf_exporter import (
RDFExporter,
SEMANTICA_NS,
mint_entity_iri,
mint_relationship_iri,
)
KG = {
"entities": [
{"text": "Acme Corp", "type": "https://example.org/Org"},
{"id": "https://example.org/e2", "text": "Bob"},
],
"relationships": [
{"source": "https://example.org/a", "target": "https://example.org/b",
"type": "https://example.org/employs"},
{"source": "https://example.org/b", "target": "https://example.org/a",
"type": "https://example.org/works_for"},
],
}
def _graph(document: str):
"""Parse a JSON-LD document the way a consumer would."""
rdflib = pytest.importorskip("rdflib")
graph = rdflib.Graph()
graph.parse(data=document, format="json-ld")
return graph
def test_jsonld_entity_id_is_the_minted_iri_not_the_interpolated_text():
graph = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))["@graph"]
assert graph[0]["@id"] == mint_entity_iri("Acme Corp")
assert graph[0]["@id"].startswith(SEMANTICA_NS)
assert "semantica:entity/" not in json.dumps(graph)
def test_json_exporter_mints_the_same_entity_iri_as_the_rdf_exporter():
"""One knowledge graph, two exporters, one node identity."""
from_rdf = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))["@graph"]
from_json = JSONExporter()._convert_kg_to_jsonld(KG)
assert from_json["semantica:entities"][0]["@id"] == from_rdf[0]["@id"]
assert from_json["semantica:relationships"][0]["@id"] == from_rdf[2]["@id"]
def test_minted_jsonld_iri_agrees_with_the_turtle_serialization():
"""The two serializations of one graph must name the same entity alike."""
turtle = RDFExporter().export_to_rdf(KG, format="turtle")
jsonld = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))
minted = mint_entity_iri("Acme Corp")
assert f"<{minted}>" in turtle
assert jsonld["@graph"][0]["@id"] == minted
def test_entity_whose_text_contains_a_space_survives_a_jsonld_parse():
"""The regression that lost data: an invalid IRI is dropped, not reported."""
kg = {"entities": [{"text": "Acme Corp"}], "relationships": []}
graph = _graph(json.dumps(JSONExporter()._convert_kg_to_jsonld(kg)))
subjects = {str(s) for s in graph.subjects()}
assert mint_entity_iri("Acme Corp") in subjects
def test_relationships_carrying_source_and_target_do_not_collide():
"""Two relationships, two nodes: ``semantica:rel/_`` merged them into one."""
jsonld = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))
relationships = [n for n in jsonld["@graph"]
if n["@type"] == "semantica:Relationship"]
ids = {node["@id"] for node in relationships}
assert len(ids) == len(relationships) == 2
assert ids == {
mint_relationship_iri(0, "https://example.org/a", "https://example.org/b"),
mint_relationship_iri(1, "https://example.org/b", "https://example.org/a"),
}
graph = _graph(json.dumps(jsonld))
assert len({str(s) for s in graph.subjects()} & ids) == 2
def test_no_export_path_writes_an_iri_in_the_semantica_scheme(tmp_path):
"""Nothing may expand to the scheme ``semantica`` rather than the namespace."""
documents = [
RDFExporter().export_to_rdf(KG, format="jsonld"),
json.dumps(JSONExporter()._convert_kg_to_jsonld(KG)),
]
exporter = JSONExporter()
exporter.export_entities(KG["entities"], tmp_path / "entities.json")
exporter.export_relationships(KG["relationships"], tmp_path / "relationships.json")
documents.append((tmp_path / "entities.json").read_text())
documents.append((tmp_path / "relationships.json").read_text())
for document in documents:
for term in _graph(document).all_nodes():
assert not str(term).startswith("semantica:"), document
for predicate in _graph(document).predicates():
assert not str(predicate).startswith("semantica:"), document
def test_entity_and_relationship_lists_expand_into_the_declared_namespace(tmp_path):
exporter = JSONExporter()
exporter.export_entities(KG["entities"], tmp_path / "entities.json")
exporter.export_relationships(KG["relationships"], tmp_path / "relationships.json")
predicates = set()
for name in ("entities.json", "relationships.json"):
predicates |= {str(p) for p in _graph((tmp_path / name).read_text()).predicates()}
assert f"{SEMANTICA_NS}entities" in predicates
assert f"{SEMANTICA_NS}relationships" in predicates
tests/export/test_owl_exporter_generator_schema.py
+291
View File
@@ -0,0 +1,291 @@
"""
Regression tests for #1103.
OWLExporter read `object_properties` / `data_properties` while OntologyGenerator
emits a single `properties` list, so every generated property was dropped. Class
IRIs arrived as None and were interpolated into `<>`, collapsing every class onto
the empty relative IRI, so an ontology of N classes serialised as one node
carrying N labels.
These tests drive the exporter with what the generator actually produces, and
assert on the parsed graph rather than on the serialised text.
"""
import pytest
rdflib = pytest.importorskip("rdflib")
from rdflib import Graph, RDF, RDFS, OWL, URIRef, Literal # noqa: E402
from semantica.export.owl_exporter import OWLExporter # noqa: E402
from semantica.ontology.ontology_generator import OntologyGenerator # noqa: E402
XSD = "http://www.w3.org/2001/XMLSchema#"
@pytest.fixture(scope="module")
def generated_ontology():
"""A real OntologyGenerator run, not a hand-written stand-in."""
data = {
"entities": [
{"type": "Person", "name": "John", "age": 30},
{"type": "Person", "name": "Jane", "age": 25},
{"type": "Organization", "name": "Acme"},
{"type": "Organization", "name": "Globex"},
],
"relationships": [
{"source": "John", "target": "Acme", "type": "works_at"},
{"source": "Jane", "target": "Globex", "type": "works_at"},
],
}
return OntologyGenerator().generate_ontology(data)
@pytest.fixture(scope="module")
def turtle_graph(generated_ontology):
ttl = OWLExporter()._export_owl_turtle(generated_ontology)
graph = Graph()
graph.parse(data=ttl, format="turtle")
return graph
@pytest.fixture(scope="module")
def xml_graph(generated_ontology):
xml = OWLExporter()._export_owl_xml(generated_ontology)
graph = Graph()
graph.parse(data=xml, format="xml")
return graph
def test_generator_mints_class_iris(generated_ontology):
"""The `uri` key is present with a None value, so a `not in` guard misses it."""
classes = generated_ontology["classes"]
assert classes, "fixture produced no classes"
for cls in classes:
assert cls.get("uri"), f"class {cls.get('name')!r} has no URI: {cls.get('uri')!r}"
assert str(cls["uri"]).startswith("http"), cls["uri"]
def test_every_class_is_a_distinct_absolute_iri(turtle_graph, generated_ontology):
subjects = set(turtle_graph.subjects(RDF.type, OWL.Class))
assert len(subjects) == len(generated_ontology["classes"])
for subject in subjects:
assert isinstance(subject, URIRef)
assert str(subject) != "", "class collapsed onto the empty relative IRI"
assert str(subject).startswith("http"), subject
def test_class_labels_are_not_stacked_on_one_node(turtle_graph):
"""Two classes must not share a subject and pile up two rdfs:label values."""
for subject in set(turtle_graph.subjects(RDF.type, OWL.Class)):
labels = list(turtle_graph.objects(subject, RDFS.label))
assert len(labels) == 1, f"{subject} carries {len(labels)} labels: {labels}"
def test_no_generated_property_is_dropped(turtle_graph, generated_ontology):
declared = set(turtle_graph.subjects(RDF.type, OWL.ObjectProperty)) | set(
turtle_graph.subjects(RDF.type, OWL.DatatypeProperty)
)
expected = {URIRef(p["uri"]) for p in generated_ontology["properties"]}
assert expected, "fixture produced no properties"
assert expected <= declared, f"dropped: {expected - declared}"
def test_properties_keep_their_owl_type(turtle_graph, generated_ontology):
by_uri = {p["uri"]: p for p in generated_ontology["properties"]}
for uri, prop in by_uri.items():
expected = OWL.ObjectProperty if prop["type"] == "object" else OWL.DatatypeProperty
assert (URIRef(uri), RDF.type, expected) in turtle_graph, (
f"{prop['name']} ({prop['type']}) is not typed {expected}"
)
def test_object_property_domain_and_range_are_class_iris(turtle_graph, generated_ontology):
"""The generator emits bare class names; they must resolve, not stay relative."""
class_iris = {URIRef(c["uri"]) for c in generated_ontology["classes"]}
obj_props = [p for p in generated_ontology["properties"] if p["type"] == "object"]
assert obj_props, "fixture produced no object properties"
for prop in obj_props:
subject = URIRef(prop["uri"])
for predicate in (RDFS.domain, RDFS.range):
values = list(turtle_graph.objects(subject, predicate))
assert values, f"{prop['name']} has no {predicate}"
for value in values:
assert value in class_iris, f"{prop['name']} {predicate} = {value!r}"
def test_data_property_range_is_a_single_well_formed_xsd_iri(turtle_graph, generated_ontology):
"""`rdfs:range xsd:{range}` doubled the prefix when range was already 'xsd:string'."""
data_props = [p for p in generated_ontology["properties"] if p["type"] != "object"]
assert data_props, "fixture produced no data properties"
for prop in data_props:
ranges = list(turtle_graph.objects(URIRef(prop["uri"]), RDFS.range))
assert ranges, f"{prop['name']} has no range"
for value in ranges:
assert str(value).startswith(XSD), f"{prop['name']} range = {value!r}"
assert "xsd:" not in str(value), f"doubled prefix: {value!r}"
def test_xml_and_turtle_describe_the_same_ontology(turtle_graph, xml_graph):
"""The two serialisations of one ontology must not be different graphs."""
def summary(graph):
return {
"classes": set(graph.subjects(RDF.type, OWL.Class)),
"object_properties": set(graph.subjects(RDF.type, OWL.ObjectProperty)),
"data_properties": set(graph.subjects(RDF.type, OWL.DatatypeProperty)),
}
assert summary(turtle_graph) == summary(xml_graph)
def test_explicit_object_and_data_property_keys_still_work():
"""The pre-existing hand-authored shape must keep working."""
ontology = {
"uri": "https://example.org/onto/",
"name": "Hand",
"classes": [{"uri": "https://example.org/onto/Person", "name": "Person"}],
"object_properties": [
{
"uri": "https://example.org/onto/knows",
"name": "knows",
"domain": "https://example.org/onto/Person",
"range": "https://example.org/onto/Person",
}
],
"data_properties": [
{"uri": "https://example.org/onto/age", "name": "age", "range": "integer"}
],
}
graph = Graph()
graph.parse(data=OWLExporter()._export_owl_turtle(ontology), format="turtle")
assert (URIRef("https://example.org/onto/knows"), RDF.type, OWL.ObjectProperty) in graph
assert (URIRef("https://example.org/onto/age"), RDF.type, OWL.DatatypeProperty) in graph
assert (
URIRef("https://example.org/onto/age"),
RDFS.range,
URIRef(XSD + "integer"),
) in graph
def test_a_class_without_any_identifier_is_skipped_not_emitted_as_empty():
"""An unusable class must not become `<>` and swallow the document IRI."""
ontology = {
"uri": "https://example.org/onto/",
"name": "Partial",
"classes": [{"comment": "no name, no uri, no id"}],
}
graph = Graph()
graph.parse(data=OWLExporter()._export_owl_turtle(ontology), format="turtle")
assert set(graph.subjects(RDF.type, OWL.Class)) == set()
assert (URIRef("https://example.org/onto/"), RDF.type, OWL.Ontology) in graph
# ── Review findings on the first revision of this fix ────────────────────────
def test_a_name_with_a_space_still_mints_a_valid_iri():
"""The name fallback pasted free text onto a base, producing `<... ...>`."""
import pyoxigraph
ontology = {
"uri": "https://example.org/onto/",
"name": "Spaces",
"classes": [{"name": "Customer Account"}],
}
turtle = OWLExporter()._export_owl_turtle(ontology)
graph = Graph()
graph.parse(data=turtle, format="turtle")
subjects = [str(s) for s in graph.subjects(RDF.type, OWL.Class)]
assert subjects, "the class was dropped entirely"
assert " " not in subjects[0], subjects[0]
# rdflib only warns about a space in an IRI; a strict parser refuses it.
pyoxigraph.Store().load(
turtle.encode(), format=pyoxigraph.RdfFormat.TURTLE, base_iri=None
)
def test_owl_thing_expands_instead_of_becoming_its_own_scheme():
"""`owl:Thing` matches the generic scheme grammar but is a prefixed name."""
ontology = {
"uri": "https://example.org/onto/",
"name": "Thing",
"classes": [{"name": "Person", "uri": "https://example.org/onto/Person"}],
"object_properties": [
{
"name": "relatedTo",
"uri": "https://example.org/onto/relatedTo",
"domain": ["owl:Thing"],
"range": ["owl:Thing"],
}
],
}
graph = Graph()
graph.parse(data=OWLExporter()._export_owl_turtle(ontology), format="turtle")
subject = URIRef("https://example.org/onto/relatedTo")
for predicate in (RDFS.domain, RDFS.range):
values = [str(v) for v in graph.objects(subject, predicate)]
assert values == ["http://www.w3.org/2002/07/owl#Thing"], values
def test_the_generators_owl_thing_fallback_round_trips():
"""stage 4 assigns domain/range of ["owl:Thing"], so this is the live path."""
ontology = {
"uri": "https://example.org/onto/",
"name": "Generated",
"classes": [{"name": "Person", "uri": "https://example.org/onto/Person"}],
"properties": [
{"name": "linkedTo", "type": "object", "uri": "https://example.org/onto/linkedTo",
"domain": ["owl:Thing"], "range": ["owl:Thing"], "@type": "owl:ObjectProperty"}
],
}
graph = Graph()
graph.parse(data=OWLExporter()._export_owl_turtle(ontology), format="turtle")
assert (
URIRef("https://example.org/onto/linkedTo"),
RDFS.domain,
URIRef("http://www.w3.org/2002/07/owl#Thing"),
) in graph
def test_optimizing_a_class_without_a_uri_does_not_raise():
"""improve_coherence lives on OntologyOptimizer, which owns no namespace manager."""
from semantica.ontology.ontology_generator import OntologyOptimizer
result = OntologyOptimizer().improve_coherence(
{"uri": "https://example.org/onto/", "classes": [{"name": "Person"}], "properties": []}
)
minted = result["classes"][0]["uri"]
assert minted.startswith("https://example.org/onto/"), minted
assert " " not in minted
def test_optimizing_falls_back_to_a_base_when_the_ontology_has_no_uri():
from semantica.ontology.ontology_generator import OntologyOptimizer
result = OntologyOptimizer().improve_coherence(
{"classes": [{"name": "Customer Account"}], "properties": []}
)
minted = result["classes"][0]["uri"]
assert minted.startswith("http"), minted
assert " " not in minted, minted
def test_unusable_property_entries_are_reported_not_silently_dropped(caplog):
import logging
ontology = {
"uri": "https://example.org/onto/",
"name": "Malformed",
"classes": [],
"properties": ["not a dict", {"name": "untyped", "uri": "https://example.org/onto/untyped"}],
}
with caplog.at_level(logging.WARNING):
OWLExporter()._export_owl_turtle(ontology)
messages = " ".join(record.getMessage() for record in caplog.records)
assert "not dictionaries" in messages or "not\ndictionaries" in messages or "dictionaries" in messages
assert "untyped" in messages
tests/export/test_timestamp_timezones.py
+141
View File
@@ -0,0 +1,141 @@
"""Timestamps that leave the process must carry a timezone (issue #1114).
Every timestamp an exporter wrote was naive: ``datetime.now().isoformat()``
reads the local clock, ``datetime.utcnow().isoformat()`` reads UTC, and the two
serialize identically, so nothing downstream can tell which zone a value belongs
to. In RDF the consequence is not a parse error but a silent one: under XSD 1.1
a value with no timezone compared against one with a timezone is indeterminate
whenever they fall inside the +/-14 hour window, SPARQL turns that into an error,
and FILTER discards errors as non-matches. A timezone-qualified query therefore
returns an answer with every Semantica-written record quietly missing from it.
"""
from datetime import datetime, timedelta, timezone
import pytest
from semantica.export.json_exporter import JSONExporter
from semantica.export.report_generator import ReportGenerator
from semantica.export.yaml_exporter import SemanticNetworkYAMLExporter
from semantica.utils.helpers import utc_now, utc_now_iso
KG = {
"entities": [{"id": "https://example.org/e1", "text": "Bob"}],
"relationships": [],
}
def assert_offset_aware(value):
"""An ISO 8601 string is only an instant if it says which zone it is in."""
assert isinstance(value, str), value
parsed = datetime.fromisoformat(value)
assert parsed.tzinfo is not None, f"timezone-naive timestamp: {value!r}"
assert parsed.utcoffset() is not None
def test_utc_now_iso_is_offset_aware():
assert_offset_aware(utc_now_iso())
assert utc_now().tzinfo is not None
def test_jsonld_export_timestamp_is_offset_aware():
document = JSONExporter()._convert_kg_to_jsonld(KG)
assert_offset_aware(document["semantica:exportedAt"])
def test_json_export_metadata_timestamp_is_offset_aware(tmp_path):
import json
exporter = JSONExporter()
exporter.export_entities(KG["entities"], tmp_path / "entities.json")
exporter.export_relationships([], tmp_path / "relationships.json")
for name in ("entities.json", "relationships.json"):
payload = json.loads((tmp_path / name).read_text())
assert_offset_aware(payload["metadata"]["exported_at"])
def test_yaml_export_timestamp_is_offset_aware():
yaml = pytest.importorskip("yaml")
document = SemanticNetworkYAMLExporter().export_entities(KG["entities"])
payload = yaml.safe_load(document)
assert_offset_aware(payload["metadata"]["exported_at"])
def test_report_timestamp_is_offset_aware():
import json
report = json.loads(
ReportGenerator().generate_quality_report({"score": 0.9}, format="json")
)
assert_offset_aware(report["generated_at"])
def test_exported_timestamp_compares_against_a_timezone_aware_instant():
"""The naive form raised TypeError here, or compared as if it were UTC."""
exported = datetime.fromisoformat(
JSONExporter()._convert_kg_to_jsonld(KG)["semantica:exportedAt"]
)
assert exported <= utc_now()
assert exported > datetime(2020, 1, 1, tzinfo=timezone.utc)
def test_exported_timestamp_survives_a_timezone_qualified_sparql_filter():
"""The regression in #1114: a strict engine dropped the naive value."""
pyoxigraph = pytest.importorskip("pyoxigraph")
exported = JSONExporter()._convert_kg_to_jsonld(KG)["semantica:exportedAt"]
store = pyoxigraph.Store()
store.load(
(
'<https://example.org/export> '
'<https://semantica.dev/ns#exportedAt> '
f'"{exported}"^^<http://www.w3.org/2001/XMLSchema#dateTime> .'
).encode(),
format=pyoxigraph.RdfFormat.N_TRIPLES,
)
# The bound has to sit inside the +/-14 hour window that makes an
# untimezoned comparison indeterminate. A bound years away is determinate
# even for a naive value, and the test would pass without the fix.
bound = (utc_now() + timedelta(hours=1)).isoformat().replace("+00:00", "Z")
rows = list(store.query(
"PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> "
"SELECT ?e WHERE { ?e <https://semantica.dev/ns#exportedAt> ?t . "
f'FILTER (?t < "{bound}"^^xsd:dateTime) }}'
))
assert len(rows) == 1, "the export was dropped by a timezone-qualified filter"
def test_document_iri_carrying_an_offset_is_a_valid_iri():
"""The offset puts '+' and ':' in the @id; both are legal in a path."""
rdflib = pytest.importorskip("rdflib")
document_iri = JSONExporter()._convert_kg_to_jsonld(KG)["@id"]
assert "+00:00" in document_iri
assert rdflib.term._is_valid_uri(document_iri)
graph = rdflib.Graph()
graph.add((
rdflib.URIRef(document_iri),
rdflib.RDF.type,
rdflib.URIRef("https://semantica.dev/ns#KnowledgeGraph"),
))
reparsed = rdflib.Graph().parse(data=graph.serialize(format="nt"), format="nt")
assert document_iri in {str(s) for s in reparsed.subjects()}
def test_vocabulary_range_matches_what_the_exporter_writes():
"""The declared range says the offset is required; the export must carry it."""
rdflib = pytest.importorskip("rdflib")
from rdflib.namespace import RDFS, XSD
from semantica.ontology.vocabulary import NAMESPACE, vocabulary_turtle
graph = rdflib.Graph()
graph.parse(data=vocabulary_turtle(), format="turtle")
declared = graph.value(rdflib.URIRef(f"{NAMESPACE}exportedAt"), RDFS.range)
assert declared == XSD.dateTimeStamp
exported = JSONExporter()._convert_kg_to_jsonld(KG)["semantica:exportedAt"]
assert datetime.fromisoformat(exported).utcoffset() is not None
tests/ingest/test_notebook_06.py
+23 -5
View File
@@ -5,6 +5,7 @@ from unittest.mock import MagicMock, patch
from semantica.ingest import FileIngestor, WebIngestor, DBIngestor, StreamIngestor, FeedIngestor
from semantica.kg import GraphBuilder, EntityResolver, ProvenanceTracker
from semantica.provenance import InMemoryStorage, ProvenanceManager
from semantica.conflicts import ConflictDetector
pytestmark = pytest.mark.integration
@@ -75,13 +76,30 @@ class TestNotebook06MultiSourceIntegration:
for entity in all_entities:
provenance_tracker.track_entity(entity.get("id"), entity.get("source"), entity)
# Endpoints stay on "source"/"target", which is what GraphBuilder's
# dict normalization expects; the originating document moves to
# "document". The literal previously set "source" twice, so the
# endpoint id was silently overwritten by the document name.
relationships = [
{"source": "e2", "target": "e1", "type": "CEO_of", "source": "file1"}
{"id": "r1", "source": "e2", "target": "e1",
"type": "CEO_of", "document": "file1"}
]
with patch.object(provenance_tracker, 'track_relationship'):
for rel in relationships:
provenance_tracker.track_relationship(rel.get("source"), rel.get("target"), rel.get("source"), rel)
# kg.ProvenanceTracker has no track_relationship and never did; that
# lives on ProvenanceManager, which is where ProvenanceTracker's own
# DeprecationWarning points callers. Called for real rather than
# patched, so this step actually exercises something.
#
# Storage is pinned to in-memory: with no argument, ProvenanceManager
# falls back to the mutable class-level _default_storage_path, so an
# earlier test setting it would make this write SQLite to disk and
# turn the result order-dependent.
provenance_manager = ProvenanceManager(storage=InMemoryStorage())
for rel in relationships:
entry = provenance_manager.track_relationship(
rel["id"], rel["document"], metadata=rel
)
assert entry is not None
# --- Step 5: Build Unified KG ---
builder = GraphBuilder()
tests/kg/test_entity_pipeline.py
+66 -7
View File
@@ -34,13 +34,6 @@ def test_full_entity_pipeline():
}
graph_data = builder.build(sources=sources)
# DEBUG: Print graph_data keys and entities count
print(f"DEBUG: graph_data keys: {list(graph_data.keys())}")
print(f"DEBUG: Entities count: {len(graph_data.get('entities', []))}")
print(f"DEBUG: Relationships count: {len(graph_data.get('relationships', []))}")
if graph_data.get('entities'):
print(f"DEBUG: First entity: {graph_data['entities'][0]}")
# Verify graph data contains the entities and normalized relationships
assert len(graph_data["entities"]) >= 3
assert len(graph_data["relationships"]) == 3
@@ -149,6 +142,8 @@ def test_entity_id_only_merge_remaps_relationship_endpoints():
} == {"alice:1", "alice:2"}
assert relationship["source"] == "alice:1"
assert relationship["target"] == "org:1"
assert relationship["source_id"] == "alice:1"
assert relationship["target_id"] == "org:1"
assert {relationship["source"], relationship["target"]} <= entity_ids
@@ -162,6 +157,70 @@ def test_entity_id_helper_ignores_falsy_identifiers():
)
assert get_entity_id({"id": "", "entity_id": 0}) is None
def test_entity_merge_repairs_stale_alias_for_canonical_endpoint():
"""A canonical endpoint must not retain a merged-away endpoint alias."""
graph = GraphBuilder(
merge_entities=True,
entity_resolution_strategy="exact",
resolve_conflicts=False,
).build(
{
"entities": [
{"entity_id": "alice:1", "name": "Alice", "type": "Person"},
{"entity_id": "alice:2", "name": " Alice ", "type": "Person"},
{"entity_id": "org:1", "name": "Acme", "type": "Organization"},
],
"relationships": [
{
"source": "alice:1",
"source_id": "alice:2",
"target": "org:1",
"target_id": "org:1",
"type": "WORKS_FOR",
}
],
}
)
relationship = graph["relationships"][0]
assert relationship["source"] == "alice:1"
assert relationship["source_id"] == "alice:1"
assert relationship["target"] == "org:1"
assert relationship["target_id"] == "org:1"
def test_entity_merge_remaps_both_stale_endpoint_aliases():
"""Both endpoint fields must be remapped when they start with an old ID."""
graph = GraphBuilder(
merge_entities=True,
entity_resolution_strategy="exact",
resolve_conflicts=False,
).build(
{
"entities": [
{"entity_id": "alice:1", "name": "Alice", "type": "Person"},
{"entity_id": "alice:2", "name": " Alice ", "type": "Person"},
{"entity_id": "org:1", "name": "Acme", "type": "Organization"},
],
"relationships": [
{
"source": "alice:2",
"source_id": "alice:2",
"target": "org:1",
"type": "WORKS_FOR",
}
],
}
)
relationship = graph["relationships"][0]
assert relationship["source"] == "alice:1"
assert relationship["source_id"] == "alice:1"
assert relationship["target"] == "org:1"
if __name__ == "__main__":
test_full_entity_pipeline()
test_direct_entity_objects_in_analyzer()
tests/ontology/test_ontology_advanced.py
+23 -10
View File
@@ -293,20 +293,33 @@ class TestSHACLHierarchicalAndValidation(unittest.TestCase):
f"Duplicate paths in {node_shape.target_class}: {paths}")
# 21
def test_no_domain_property_attaches_to_all_shapes(self):
onto = {
"classes": [{"name": "A"}, {"name": "B"}],
"properties": [
{"name": "globalProp", "type": "datatype", "range": "string"}
# no domain
],
}
gen = self._make_gen()
graph = gen.generate(onto)
_NO_DOMAIN_ONTOLOGY = {
"classes": [{"name": "A"}, {"name": "B"}],
"properties": [
{"name": "globalProp", "type": "datatype", "range": "string"}
# no domain
],
}
def test_no_domain_property_attaches_to_all_shapes_when_opted_in(self):
"""attach_domainless_properties=True keeps the pre-0.6.6 behaviour."""
gen = self._make_gen(attach_domainless_properties=True)
graph = gen.generate(self._NO_DOMAIN_ONTOLOGY)
for node_shape in graph.node_shapes:
paths = {ps.path for ps in node_shape.property_shapes}
self.assertIn("globalProp", paths)
def test_no_domain_property_is_not_attached_by_default(self):
"""
Attaching states a constraint the ontology does not (#1105). This test
previously asserted the opposite, which pinned the defect in place.
"""
gen = self._make_gen()
graph = gen.generate(self._NO_DOMAIN_ONTOLOGY)
for node_shape in graph.node_shapes:
paths = {ps.path for ps in node_shape.property_shapes}
self.assertNotIn("globalProp", paths)
# 22
def test_empty_classes_produces_no_shapes(self):
gen = self._make_gen()
tests/ontology/test_shacl_target_namespace.py
+330
View File
@@ -0,0 +1,330 @@
"""
Regression tests for #1104 and #1105.
#1104: SHACLGenerator used one namespace for two different jobs. `base_uri`
names where the shape resources live, and it was also used to expand every
sh:targetClass and sh:path. With the default `https://semantica.dev/shapes/`
that made shapes target `.../shapes/Person`, while data carries
`.../ns#Person` or the ontology's own class IRI. The shapes matched nothing.
pySHACL then reported conforms=True, because a shape with no focus nodes is
vacuously satisfied, so the mismatch was invisible to the validator the
package ships with.
#1105: a property with no declared domain was attached to every node shape,
which invents a constraint the ontology never stated. With minCount 1 that
makes every instance of every class invalid.
These tests validate real data through pySHACL rather than reading the shapes
text, so a shape that targets nothing cannot pass by being ignored.
"""
import pytest
rdflib = pytest.importorskip("rdflib")
pyshacl = pytest.importorskip("pyshacl")
from rdflib import Graph, RDF, Namespace # noqa: E402
from semantica.ontology.ontology_generator import SHACLGenerator # noqa: E402
SH = Namespace("http://www.w3.org/ns/shacl#")
ONTOLOGY_NS = "https://example.org/onto#"
SHAPES_NS = "https://semantica.dev/shapes/"
def _ontology(*, declare_namespace: bool, carry_class_uris: bool):
"""
Build the same ontology in the shapes the generator can hand over.
A generated ontology carries class URIs; a hand written one often carries
only names, and only sometimes declares a namespace. All of them have to
produce shapes that match the data.
"""
def class_def(name):
entry = {"name": name, "label": name}
if carry_class_uris:
entry["uri"] = ONTOLOGY_NS + name
return entry
def prop_def(name, **extra):
entry = {"name": name, "type": "datatype", "range": "string", **extra}
if carry_class_uris:
entry["uri"] = ONTOLOGY_NS + name
return entry
ontology = {
"classes": [class_def("Person"), class_def("Organization")],
"properties": [
prop_def("fullName", domain="Person", required=True),
# No domain. The ontology never says which class this belongs to.
prop_def("sourceDocument", required=True),
],
}
if declare_namespace:
ontology["namespace"] = {"base_uri": ONTOLOGY_NS}
return ontology
SHAPES = [
pytest.param(True, True, id="namespace+uris"),
pytest.param(True, False, id="namespace-only"),
pytest.param(False, True, id="uris-only"),
]
# A Person with no fullName. This violates the shape the ontology does state.
VIOLATING_DATA = f"""
@prefix ex: <{ONTOLOGY_NS}> .
ex:alice a ex:Person .
"""
# A Person that satisfies every constraint the ontology actually declares.
CONFORMING_DATA = f"""
@prefix ex: <{ONTOLOGY_NS}> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
ex:bob a ex:Person ;
ex:fullName "Bob Smith"^^xsd:string .
"""
def _shapes_graph(ontology, **kwargs):
generator = SHACLGenerator(**kwargs)
graph = generator.generate(ontology)
shapes = Graph()
shapes.parse(data=generator.serialize(graph, "turtle"), format="turtle")
return shapes
def _validate(data_ttl, shapes_graph):
data = Graph()
data.parse(data=data_ttl, format="turtle")
conforms, _, text = pyshacl.validate(
data, shacl_graph=shapes_graph, inference="none", advanced=True
)
return conforms, text
@pytest.mark.parametrize("declare_namespace,carry_class_uris", SHAPES)
def test_target_class_names_a_class_the_data_can_instantiate(declare_namespace, carry_class_uris):
shapes = _shapes_graph(_ontology(
declare_namespace=declare_namespace, carry_class_uris=carry_class_uris
))
targets = {str(o) for o in shapes.objects(None, SH.targetClass)}
assert targets == {ONTOLOGY_NS + "Person", ONTOLOGY_NS + "Organization"}, targets
assert not any(t.startswith(SHAPES_NS) for t in targets), (
f"shapes still target their own shapes namespace: {targets}"
)
@pytest.mark.parametrize("declare_namespace,carry_class_uris", SHAPES)
def test_property_path_names_a_predicate_the_data_uses(declare_namespace, carry_class_uris):
shapes = _shapes_graph(_ontology(
declare_namespace=declare_namespace, carry_class_uris=carry_class_uris
))
paths = {str(o) for o in shapes.objects(None, SH.path)}
assert paths, "no sh:path emitted at all"
for path in paths:
assert path.startswith(ONTOLOGY_NS), path
@pytest.mark.parametrize("declare_namespace,carry_class_uris", SHAPES)
def test_a_real_violation_is_actually_reported(declare_namespace, carry_class_uris):
"""The killer case. Shapes that match nothing make pySHACL return conforms=True."""
shapes = _shapes_graph(_ontology(
declare_namespace=declare_namespace, carry_class_uris=carry_class_uris
))
conforms, text = _validate(VIOLATING_DATA, shapes)
assert not conforms, (
"a Person with no fullName was reported as conforming, which means the "
f"shapes matched no focus nodes:\n{text}"
)
assert "fullName" in text
@pytest.mark.parametrize("declare_namespace,carry_class_uris", SHAPES)
def test_conforming_data_still_conforms(declare_namespace, carry_class_uris):
shapes = _shapes_graph(_ontology(
declare_namespace=declare_namespace, carry_class_uris=carry_class_uris
))
conforms, text = _validate(CONFORMING_DATA, shapes)
assert conforms, text
def test_default_target_namespace_is_the_vocabulary_not_the_shapes_namespace():
"""With nothing declared anywhere, targets must not land in the shapes namespace."""
ontology = {
"classes": [{"name": "Person", "label": "Person"}],
"properties": [{"name": "fullName", "type": "datatype", "range": "string",
"domain": "Person", "required": True}],
}
shapes = _shapes_graph(ontology)
targets = {str(o) for o in shapes.objects(None, SH.targetClass)}
assert targets, "no sh:targetClass emitted at all"
for target in targets:
assert not target.startswith(SHAPES_NS), target
def test_shape_resources_are_distinct_from_the_classes_they_target():
shapes = _shapes_graph(_ontology(declare_namespace=True, carry_class_uris=True))
node_shapes = {str(s) for s in shapes.subjects(RDF.type, SH.NodeShape)}
targets = {str(o) for o in shapes.objects(None, SH.targetClass)}
assert node_shapes, "no node shapes emitted"
assert node_shapes.isdisjoint(targets), (
f"a shape and the class it targets are the same resource: {node_shapes & targets}"
)
# ── #1105 ────────────────────────────────────────────────────────────────────
def test_domainless_property_is_not_asserted_on_every_class():
"""minCount 1 on a domain-less property invalidates every instance of every class."""
generator = SHACLGenerator()
graph = generator.generate(_ontology(declare_namespace=True, carry_class_uris=True))
carriers = [
shape.target_class
for shape in graph.node_shapes
for prop in shape.property_shapes
if prop.path.endswith("sourceDocument")
]
assert carriers == [], f"a property with no declared domain was attached to {carriers}"
def test_domainless_property_does_not_invalidate_conforming_data():
shapes = _shapes_graph(_ontology(declare_namespace=True, carry_class_uris=True))
conforms, text = _validate(CONFORMING_DATA, shapes)
assert conforms, f"invented a constraint the ontology never declared:\n{text}"
assert "sourceDocument" not in text
def test_domainless_attachment_is_available_as_an_explicit_opt_in():
"""The old behaviour stays reachable for anyone who relied on it."""
generator = SHACLGenerator(attach_domainless_properties=True)
graph = generator.generate(_ontology(declare_namespace=True, carry_class_uris=True))
carriers = {
shape.target_class
for shape in graph.node_shapes
for prop in shape.property_shapes
if prop.path.endswith("sourceDocument")
}
assert len(carriers) == len(graph.node_shapes)
# ── Review findings on the first revision of this fix ────────────────────────
@pytest.mark.parametrize("fmt,parse_as", [
("turtle", "turtle"), ("n-triples", "nt"), ("json-ld", "json-ld"),
])
def test_every_format_targets_the_ontology_namespace(fmt, parse_as):
"""
The first revision fixed Turtle alone. JSON-LD and N-Triples went on
pasting names onto the shapes namespace, so two of the three formats still
produced shapes that matched nothing.
"""
generator = SHACLGenerator()
graph = generator.generate(_ontology(declare_namespace=False, carry_class_uris=True))
shapes = Graph()
shapes.parse(data=generator.serialize(graph, fmt), format=parse_as)
targets = {str(o) for o in shapes.objects(None, SH.targetClass)}
assert targets == {ONTOLOGY_NS + "Person", ONTOLOGY_NS + "Organization"}, targets
assert not any(t.startswith(SHAPES_NS) for t in targets), targets
@pytest.mark.parametrize("fmt,parse_as", [
("turtle", "turtle"), ("n-triples", "nt"), ("json-ld", "json-ld"),
])
def test_every_format_reports_a_real_violation(fmt, parse_as):
generator = SHACLGenerator()
graph = generator.generate(_ontology(declare_namespace=False, carry_class_uris=True))
shapes = Graph()
shapes.parse(data=generator.serialize(graph, fmt), format=parse_as)
conforms, text = _validate(VIOLATING_DATA, shapes)
assert not conforms, f"{fmt} shapes matched no focus nodes:\n{text}"
def test_a_property_sharing_a_class_name_keeps_its_own_iri():
"""One name-keyed map gave the property the class's IRI, so sh:path validated
the wrong predicate."""
ontology = {
"classes": [{"name": "Account", "uri": ONTOLOGY_NS + "Account"}],
"properties": [
{
"name": "Account",
"uri": ONTOLOGY_NS + "accountNumber",
"type": "datatype",
"range": "string",
"domain": "Account",
"required": True,
}
],
}
shapes = _shapes_graph(ontology)
paths = {str(o) for o in shapes.objects(None, SH.path)}
targets = {str(o) for o in shapes.objects(None, SH.targetClass)}
assert paths == {ONTOLOGY_NS + "accountNumber"}, paths
assert targets == {ONTOLOGY_NS + "Account"}, targets
def test_sh_class_resolves_to_the_class_namespace():
ontology = {
"classes": [
{"name": "Person", "uri": ONTOLOGY_NS + "Person"},
{"name": "Organization", "uri": ONTOLOGY_NS + "Organization"},
],
"properties": [
{
"name": "worksAt",
"uri": ONTOLOGY_NS + "worksAt",
"type": "object",
"range": "Organization",
"domain": "Person",
}
],
}
shapes = _shapes_graph(ontology)
classes = {str(o) for o in shapes.objects(None, SH["class"])}
assert classes == {ONTOLOGY_NS + "Organization"}, classes
def test_the_engine_forwards_the_new_options():
"""to_shacl passed them through generate(**options), which never reads them."""
from semantica.ontology.engine import OntologyEngine
ontology = _ontology(declare_namespace=False, carry_class_uris=False)
turtle = OntologyEngine().to_shacl(ontology, target_namespace="https://forwarded.example/ns#")
shapes = Graph()
shapes.parse(data=turtle, format="turtle")
targets = {str(o) for o in shapes.objects(None, SH.targetClass)}
assert all(t.startswith("https://forwarded.example/ns#") for t in targets), targets
attached = OntologyEngine().to_shacl(ontology, attach_domainless_properties=True)
assert "sourceDocument" in attached
default = OntologyEngine().to_shacl(ontology)
assert "sourceDocument" not in default
def test_the_opt_in_warns_rather_than_whispering(caplog):
import logging
with caplog.at_level(logging.WARNING):
SHACLGenerator(attach_domainless_properties=True).generate(
_ontology(declare_namespace=True, carry_class_uris=True)
)
messages = " ".join(record.getMessage() for record in caplog.records)
assert "sourceDocument" in messages
tests/provenance/test_timestamp_timezones.py
+153
View File
@@ -0,0 +1,153 @@
"""Provenance timestamps must carry a timezone (issue #1114).
The provenance package stamped every record with ``datetime.utcnow()``, which
returns a naive datetime that happens to hold UTC. The exporters stamped theirs
with ``datetime.now()``, which returns a naive datetime holding local time. Both
serialize identically, so a graph mixing the two cannot be ordered, and the
values reach RDF as ``prov:generatedAtTime``/``startedAtTime``/``endedAtTime``
typed ``xsd:dateTime``, where a timezone-qualified SPARQL comparison discards
them. ``datetime.utcnow()`` is also deprecated as of Python 3.12.
"""
import warnings
from datetime import datetime
import pytest
from semantica.provenance.manager import ProvenanceManager
from semantica.provenance.schemas import ProvenanceEntry
from semantica.utils.helpers import utc_now
def assert_offset_aware(value):
parsed = datetime.fromisoformat(value)
assert parsed.tzinfo is not None, f"timezone-naive timestamp: {value!r}"
def test_provenance_entry_default_timestamp_is_offset_aware():
entry = ProvenanceEntry(entity_id="e1", entity_type="Doc", activity_id="act1")
assert_offset_aware(entry.timestamp)
assert datetime.fromisoformat(entry.timestamp) <= utc_now()
def test_creating_an_entry_raises_no_deprecation_warning():
"""datetime.utcnow() is deprecated and scheduled for removal."""
with warnings.catch_warnings():
warnings.simplefilter("error", DeprecationWarning)
ProvenanceEntry(entity_id="e1", entity_type="Doc", activity_id="act1")
def test_tracked_entity_timestamps_are_offset_aware():
manager = ProvenanceManager()
manager.track_entity("e1", source="doc.pdf")
entry = manager.storage.retrieve_all()[0]
assert_offset_aware(entry.timestamp)
for field in ("first_seen", "last_updated"):
value = getattr(entry, field, None)
if value:
assert_offset_aware(value)
def test_prov_o_export_timestamps_are_offset_aware():
"""The values land in RDF typed xsd:dateTime, so the offset is the contract."""
rdflib = pytest.importorskip("rdflib")
from rdflib.namespace import XSD
manager = ProvenanceManager()
manager.track_entity("e_parent", source="doc.pdf")
manager.track_entity(
"e_child", source="doc.pdf", parent_entity_id="e_parent",
used_entities=["e_parent"], activity_id="act_transform",
)
graph = rdflib.Graph()
graph.parse(data=manager.export_prov(format="turtle"), format="turtle")
stamps = [o for o in graph.objects()
if isinstance(o, rdflib.Literal) and o.datatype == XSD.dateTime]
assert stamps, "no xsd:dateTime literals in the PROV-O export"
for stamp in stamps:
assert_offset_aware(str(stamp))
def test_prov_o_timestamps_are_valid_datetimestamp():
"""xsd:dateTimeStamp requires an explicit timezone; these now qualify."""
rdflib = pytest.importorskip("rdflib")
from rdflib.namespace import XSD
manager = ProvenanceManager()
manager.track_entity("e1", source="doc.pdf")
graph = rdflib.Graph()
graph.parse(data=manager.export_prov(format="turtle"), format="turtle")
for stamp in [o for o in graph.objects()
if isinstance(o, rdflib.Literal) and o.datatype == XSD.dateTime]:
assert rdflib.Literal(str(stamp), datatype=XSD.dateTime).ill_typed is False
assert datetime.fromisoformat(str(stamp)).utcoffset() is not None
class TestRangeQueriesCompareInstants:
"""Range APIs compared ISO strings, so they ordered by spelling (#1121 review).
Once new entries carry ``+00:00`` and stored ones do not, a raw string
comparison puts an inclusive naive bound *below* the offset-bearing
timestamp it names, dropping the record, and a bound written in another
offset lands wherever its digits fall rather than at its instant.
"""
@staticmethod
def _manager_with(timestamps):
manager = ProvenanceManager()
for index, stamp in enumerate(timestamps):
manager.storage.store(ProvenanceEntry(
entity_id=f"e{index}", entity_type="Doc",
activity_id="act", timestamp=stamp,
))
return manager
def test_inclusive_bound_written_without_an_offset_still_matches(self):
manager = self._manager_with(["2026-08-19T14:19:04.229937+00:00"])
found = manager.query_recorded_between(
"2026-08-19T00:00:00", "2026-08-19T14:19:04.229937"
)
assert [e["entity_id"] for e in found] == ["e0"]
def test_bound_in_another_offset_selects_by_instant(self):
"""19:45+05:30 is 14:15Z: before the entry, though its digits are after."""
manager = self._manager_with(["2026-08-19T14:19:04+00:00"])
assert manager.query_recorded_between(
"2026-08-19T00:00:00Z", "2026-08-19T19:45:00+05:30"
) == []
assert len(manager.query_recorded_between(
"2026-08-19T00:00:00Z", "2026-08-19T19:50:00+05:30"
)) == 1
def test_legacy_and_offset_bearing_entries_are_both_found_and_ordered(self):
manager = self._manager_with([
"2026-08-19T14:19:05+00:00", # written after #1114
"2026-08-19T14:19:04", # written before it, meaning UTC
])
found = manager.query_recorded_between(
"2026-08-19T14:00:00Z", "2026-08-19T15:00:00Z"
)
assert [e["entity_id"] for e in found] == ["e1", "e0"]
def test_audit_log_since_reads_a_naive_bound_as_utc(self):
manager = self._manager_with([
"2026-08-19T14:19:05+00:00",
"2026-08-19T09:00:00",
])
recent = manager.audit_log(since="2026-08-19T14:19:05", format="json")
assert [e["entity_id"] for e in recent] == ["e0"]
def test_an_unreadable_bound_falls_back_to_the_previous_behaviour(self):
"""A call that used to work with a non-timestamp bound must not raise."""
manager = self._manager_with(["2026-08-19T14:19:04+00:00"])
assert manager.query_recorded_between("not-a-date", "also-not") == []
assert manager.audit_log(since="not-a-date", format="json") == []
tests/visualization/__init__.py
View File
tests/visualization/_plotly_doubles.py
+29
View File
@@ -0,0 +1,29 @@
"""Shared helper for visualization tests.
The visualization modules treat Plotly as optional: they bind ``px``, ``go`` and
``make_subplots`` to ``None`` when the import fails, and raise ``ProcessingError``
from ``_check_dependencies()``. Tests that exercise a Plotly-backed path need
those names to be usable, otherwise ``patch("...go.Figure")`` fails on ``None``
and the visualizers refuse to run.
``plotly_doubles`` fills in a double for each alias that is ``None``, so the
tests describe their own requirements instead of depending on whether Plotly
happens to be installed. When Plotly is installed the aliases are left alone and
the patches keep asserting against the real attribute names.
"""
from contextlib import ExitStack, contextmanager
from unittest.mock import MagicMock, patch
PLOTLY_ALIASES = ("px", "go", "make_subplots")
@contextmanager
def plotly_doubles(*modules):
"""Stand in for the module-level Plotly aliases that are unavailable."""
with ExitStack() as stack:
for module in modules:
for alias in PLOTLY_ALIASES:
if getattr(module, alias, "unused") is None:
stack.enter_context(patch.object(module, alias, MagicMock()))
yield
tests/visualization/test_kg_visualizer_normalize_graph.py
+43 -42
View File
@@ -9,27 +9,16 @@ and must raise a clear ProcessingError for anything else.
"""
import contextlib
import sys
import unittest
from dataclasses import dataclass, field
from typing import List
from unittest.mock import MagicMock, patch
# ---------------------------------------------------------------------------
# Stub out heavy optional deps before importing the module under test
# ---------------------------------------------------------------------------
sys.modules.setdefault("matplotlib", MagicMock())
sys.modules.setdefault("matplotlib.pyplot", MagicMock())
sys.modules.setdefault("matplotlib.patches", MagicMock())
sys.modules.setdefault("plotly", MagicMock())
sys.modules.setdefault("plotly.express", MagicMock())
sys.modules.setdefault("plotly.graph_objects", MagicMock())
sys.modules.setdefault("plotly.subplots", MagicMock())
sys.modules.setdefault("graphviz", MagicMock())
sys.modules.setdefault("seaborn", MagicMock())
from semantica.utils.exceptions import ProcessingError # noqa: E402
from semantica.visualization import kg_visualizer # noqa: E402
from semantica.visualization.kg_visualizer import KGVisualizer # noqa: E402
from tests.visualization._plotly_doubles import plotly_doubles # noqa: E402
# ---------------------------------------------------------------------------
# Minimal fixtures
@@ -191,10 +180,6 @@ class TestVisualizeNetworkAcceptsKGObject(unittest.TestCase):
def _run_visualize_network(self, graph_arg):
"""Run visualize_network with all Plotly internals mocked."""
mock_fig = MagicMock()
mock_go = sys.modules["plotly.graph_objects"]
mock_go.Figure.return_value = mock_fig
mock_go.Scatter.return_value = MagicMock()
mock_go.Layout.return_value = MagicMock()
viz = _make_viz()
@@ -207,6 +192,10 @@ class TestVisualizeNetworkAcceptsKGObject(unittest.TestCase):
# ColorPalette helpers
with (
plotly_doubles(kg_visualizer),
patch("semantica.visualization.kg_visualizer.go.Figure", return_value=mock_fig),
patch("semantica.visualization.kg_visualizer.go.Scatter"),
patch("semantica.visualization.kg_visualizer.go.Layout"),
patch(
"semantica.visualization.kg_visualizer.ColorPalette.get_entity_type_colors",
return_value={"Person": "#ff0000"},
@@ -254,9 +243,12 @@ class TestAllVisualizeMethodsAcceptKGObject(unittest.TestCase):
self.viz._normalize_graph = MagicMock(return_value=GRAPH_DICT)
self.viz._visualize_network_plotly = MagicMock(return_value=MagicMock())
communities = {"node_assignments": {"e1": 0, "e2": 1}, "num_communities": 2}
with patch(
"semantica.visualization.kg_visualizer.ColorPalette.get_community_colors",
return_value=["#ff0000", "#00ff00"],
with (
plotly_doubles(kg_visualizer),
patch(
"semantica.visualization.kg_visualizer.ColorPalette.get_community_colors",
return_value=["#ff0000", "#00ff00"],
),
):
self.viz.visualize_communities(self.kg, communities=communities)
self.viz._normalize_graph.assert_called_once_with(self.kg)
@@ -264,22 +256,24 @@ class TestAllVisualizeMethodsAcceptKGObject(unittest.TestCase):
def test_visualize_centrality_accepts_kg_object(self):
self.viz._normalize_graph = MagicMock(return_value=GRAPH_DICT)
self.viz._visualize_network_plotly = MagicMock(return_value=MagicMock())
self.viz.visualize_centrality(self.kg, centrality={"centrality": {}})
with plotly_doubles(kg_visualizer):
self.viz.visualize_centrality(self.kg, centrality={"centrality": {}})
self.viz._normalize_graph.assert_called_once_with(self.kg)
def test_visualize_entity_types_accepts_kg_object(self):
self.viz._normalize_graph = MagicMock(return_value=GRAPH_DICT)
mock_px = sys.modules["plotly.express"]
mock_px.bar.return_value = MagicMock()
self.viz.visualize_entity_types(self.kg)
with plotly_doubles(kg_visualizer), patch("semantica.visualization.kg_visualizer.px.bar"):
self.viz.visualize_entity_types(self.kg)
self.viz._normalize_graph.assert_called_once_with(self.kg)
def test_visualize_relationship_matrix_accepts_kg_object(self):
self.viz._normalize_graph = MagicMock(return_value=GRAPH_DICT)
mock_go = sys.modules["plotly.graph_objects"]
mock_go.Figure.return_value = MagicMock()
mock_go.Heatmap.return_value = MagicMock()
self.viz.visualize_relationship_matrix(self.kg)
with (
plotly_doubles(kg_visualizer),
patch("semantica.visualization.kg_visualizer.go.Figure"),
patch("semantica.visualization.kg_visualizer.go.Heatmap"),
):
self.viz.visualize_relationship_matrix(self.kg)
self.viz._normalize_graph.assert_called_once_with(self.kg)
@@ -361,10 +355,6 @@ class TestFormalKnowledgeGraphType(unittest.TestCase):
def _run_visualize_network(self, graph_arg):
mock_fig = MagicMock()
mock_go = sys.modules["plotly.graph_objects"]
mock_go.Figure.return_value = mock_fig
mock_go.Scatter.return_value = MagicMock()
mock_go.Layout.return_value = MagicMock()
viz = _make_viz()
fake_pos = {"e1": (0.0, 0.0), "e2": (1.0, 1.0)}
viz.force_layout = MagicMock()
@@ -372,6 +362,10 @@ class TestFormalKnowledgeGraphType(unittest.TestCase):
viz.hierarchical_layout = MagicMock()
viz.circular_layout = MagicMock()
with (
plotly_doubles(kg_visualizer),
patch("semantica.visualization.kg_visualizer.go.Figure", return_value=mock_fig),
patch("semantica.visualization.kg_visualizer.go.Scatter"),
patch("semantica.visualization.kg_visualizer.go.Layout"),
patch(
"semantica.visualization.kg_visualizer.ColorPalette.get_entity_type_colors",
return_value={"Person": "#ff0000"},
@@ -392,9 +386,12 @@ class TestFormalKnowledgeGraphType(unittest.TestCase):
viz._normalize_graph = MagicMock(return_value=GRAPH_DICT)
viz._visualize_network_plotly = MagicMock(return_value=MagicMock())
communities = {"node_assignments": {"e1": 0, "e2": 1}, "num_communities": 2}
with patch(
"semantica.visualization.kg_visualizer.ColorPalette.get_community_colors",
return_value=["#ff0000", "#00ff00"],
with (
plotly_doubles(kg_visualizer),
patch(
"semantica.visualization.kg_visualizer.ColorPalette.get_community_colors",
return_value=["#ff0000", "#00ff00"],
),
):
viz.visualize_communities(kg, communities=communities)
viz._normalize_graph.assert_called_once_with(kg)
@@ -404,24 +401,28 @@ class TestFormalKnowledgeGraphType(unittest.TestCase):
viz = _make_viz()
viz._normalize_graph = MagicMock(return_value=GRAPH_DICT)
viz._visualize_network_plotly = MagicMock(return_value=MagicMock())
viz.visualize_centrality(kg, centrality={"centrality": {}})
with plotly_doubles(kg_visualizer):
viz.visualize_centrality(kg, centrality={"centrality": {}})
viz._normalize_graph.assert_called_once_with(kg)
def test_visualize_entity_types_accepts_knowledge_graph(self):
kg = self._make_kg()
viz = _make_viz()
viz._normalize_graph = MagicMock(return_value=GRAPH_DICT)
sys.modules["plotly.express"].bar.return_value = MagicMock()
viz.visualize_entity_types(kg)
with plotly_doubles(kg_visualizer), patch("semantica.visualization.kg_visualizer.px.bar"):
viz.visualize_entity_types(kg)
viz._normalize_graph.assert_called_once_with(kg)
def test_visualize_relationship_matrix_accepts_knowledge_graph(self):
kg = self._make_kg()
viz = _make_viz()
viz._normalize_graph = MagicMock(return_value=GRAPH_DICT)
sys.modules["plotly.graph_objects"].Figure.return_value = MagicMock()
sys.modules["plotly.graph_objects"].Heatmap.return_value = MagicMock()
viz.visualize_relationship_matrix(kg)
with (
plotly_doubles(kg_visualizer),
patch("semantica.visualization.kg_visualizer.go.Figure"),
patch("semantica.visualization.kg_visualizer.go.Heatmap"),
):
viz.visualize_relationship_matrix(kg)
viz._normalize_graph.assert_called_once_with(kg)
def test_knowledge_graph_importable_from_kg_module(self):
tests/visualization/test_optional_dependencies.py
+77 -111
View File
@@ -1,149 +1,115 @@
import unittest
from unittest.mock import MagicMock, patch
import importlib
import sys
import unittest
from contextlib import contextmanager
from unittest.mock import patch
import numpy as np
# Helper to mock modules
def mock_module(name):
m = MagicMock()
sys.modules[name] = m
return m
from tests.visualization._plotly_doubles import plotly_doubles
@contextmanager
def import_without(module_name, *dependencies):
"""Import a module with selected optional dependencies unavailable."""
package_name, attribute = module_name.rsplit(".", 1)
package = importlib.import_module(package_name)
missing = object()
original_module = sys.modules.pop(module_name, missing)
original_attribute = getattr(package, attribute, missing)
try:
with patch.dict(sys.modules, {name: None for name in dependencies}):
yield importlib.import_module(module_name)
finally:
sys.modules.pop(module_name, None)
if original_module is not missing:
sys.modules[module_name] = original_module
if original_attribute is missing:
package.__dict__.pop(attribute, None)
else:
setattr(package, attribute, original_attribute)
class TestOptionalDependencies(unittest.TestCase):
@classmethod
def setUpClass(cls):
# Mock heavy/problematic dependencies globally to prevent environment crashes
# We use a dict to save original modules if they exist, but for this test file
# we generally want to run in a controlled "clean" environment.
cls.modules_to_patch = [
'sklearn', 'sklearn.decomposition', 'sklearn.manifold',
'scipy', 'scipy.optimize',
'matplotlib', 'matplotlib.pyplot', 'matplotlib.patches',
'plotly', 'plotly.express', 'plotly.graph_objects', 'plotly.subplots',
'networkx', 'seaborn'
]
cls.original_modules = {}
for mod in cls.modules_to_patch:
if mod in sys.modules:
cls.original_modules[mod] = sys.modules[mod]
sys.modules[mod] = MagicMock()
@classmethod
def tearDownClass(cls):
# Restore original modules
for mod in cls.modules_to_patch:
if mod in cls.original_modules:
sys.modules[mod] = cls.original_modules[mod]
else:
del sys.modules[mod]
def setUp(self):
# Clear cached visualization modules to ensure fresh imports
self.viz_modules = [
'semantica.visualization.embedding_visualizer',
'semantica.visualization.ontology_visualizer',
'semantica.visualization.kg_visualizer',
'semantica.visualization.utils.export_formats'
]
for mod in self.viz_modules:
if mod in sys.modules:
del sys.modules[mod]
def test_embedding_visualizer_without_umap(self):
"""Test EmbeddingVisualizer behavior when umap is missing."""
# Ensure umap is missing
with patch.dict(sys.modules, {'umap': None}):
from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
# Setup PCA mock to verify fallback
mock_pca_class = sys.modules['sklearn.decomposition'].PCA
mock_pca_instance = mock_pca_class.return_value
# Configure fit_transform to return correct shape (n_samples, 2)
mock_pca_instance.fit_transform.return_value = np.zeros((4, 2))
viz = EmbeddingVisualizer()
# Use numpy array!
embeddings = np.array([[0, 1, 2], [1, 0, 3], [0, 0, 0], [1, 1, 1]])
# Should fallback to PCA when method="umap" is used but umap is None
# The code logs a warning and uses PCA
viz.visualize_2d_projection(embeddings, method="umap")
# Verify PCA was called
with import_without(
"semantica.visualization.embedding_visualizer", "umap"
) as module:
with plotly_doubles(module), patch.object(module, "PCA") as mock_pca_class:
mock_pca_class.return_value.fit_transform.return_value = np.zeros((4, 2))
viz = module.EmbeddingVisualizer()
embeddings = np.array([[0, 1, 2], [1, 0, 3], [0, 0, 0], [1, 1, 1]])
viz.visualize_2d_projection(embeddings, method="umap")
mock_pca_class.assert_called()
def test_ontology_visualizer_without_graphviz(self):
"""Test OntologyVisualizer behavior when graphviz is missing."""
# Ensure graphviz is missing
with patch.dict(sys.modules, {'graphviz': None}):
from semantica.visualization.ontology_visualizer import OntologyVisualizer, ProcessingError
viz = OntologyVisualizer()
with import_without(
"semantica.visualization.ontology_visualizer", "graphviz"
) as module:
viz = module.OntologyVisualizer()
ontology = {
"classes": [
{"name": "A", "label": "A"},
{"name": "B", "label": "B", "parent": "A"}
{"name": "B", "label": "B", "parent": "A"},
]
}
with self.assertRaises(ProcessingError) as cm:
with self.assertRaises(module.ProcessingError) as cm:
viz.visualize_hierarchy(ontology, output="dot", file_path="test.dot")
self.assertIn("Graphviz is required for DOT export", str(cm.exception))
def test_analytics_visualizer_without_plotly(self):
"""Test AnalyticsVisualizer behavior when plotly is missing."""
with patch.dict(sys.modules, {'plotly': None, 'plotly.express': None, 'plotly.graph_objects': None}):
from semantica.visualization.analytics_visualizer import AnalyticsVisualizer, ProcessingError
# Need to ensure numpy is available for init (it's imported at top level)
# But we are testing plotly missing.
viz = AnalyticsVisualizer()
with self.assertRaises(ProcessingError) as cm:
viz.visualize_centrality_rankings({"node1": 1.0})
self.assertIn("Plotly is required", str(cm.exception))
with import_without(
"semantica.visualization.analytics_visualizer",
"plotly",
"plotly.express",
"plotly.graph_objects",
) as module:
viz = module.AnalyticsVisualizer()
def test_analytics_visualizer_without_plotly(self):
"""Test AnalyticsVisualizer behavior when plotly is missing."""
with patch.dict(sys.modules, {'plotly': None, 'plotly.express': None, 'plotly.graph_objects': None}):
from semantica.visualization.analytics_visualizer import AnalyticsVisualizer, ProcessingError
viz = AnalyticsVisualizer()
with self.assertRaises(ProcessingError) as cm:
with self.assertRaises(module.ProcessingError) as cm:
viz.visualize_centrality_rankings({})
self.assertIn("Plotly is required", str(cm.exception))
def test_semantic_network_visualizer_without_plotly(self):
"""Test SemanticNetworkVisualizer behavior when plotly is missing."""
with patch.dict(sys.modules, {'plotly': None, 'plotly.express': None, 'plotly.graph_objects': None}):
from semantica.visualization.semantic_network_visualizer import SemanticNetworkVisualizer, ProcessingError
viz = SemanticNetworkVisualizer()
with self.assertRaises(ProcessingError) as cm:
with import_without(
"semantica.visualization.semantic_network_visualizer",
"plotly",
"plotly.express",
"plotly.graph_objects",
) as module:
viz = module.SemanticNetworkVisualizer()
with self.assertRaises(module.ProcessingError) as cm:
viz.visualize_network({})
self.assertIn("Plotly is required", str(cm.exception))
def test_temporal_visualizer_without_plotly(self):
"""Test TemporalVisualizer behavior when plotly is missing."""
with patch.dict(sys.modules, {'plotly': None, 'plotly.express': None, 'plotly.graph_objects': None}):
from semantica.visualization.temporal_visualizer import TemporalVisualizer, ProcessingError
viz = TemporalVisualizer()
with self.assertRaises(ProcessingError) as cm:
with import_without(
"semantica.visualization.temporal_visualizer",
"plotly",
"plotly.express",
"plotly.graph_objects",
) as module:
viz = module.TemporalVisualizer()
with self.assertRaises(module.ProcessingError) as cm:
viz.visualize_timeline({"events": []})
self.assertIn("Plotly is required", str(cm.exception))
if __name__ == '__main__':
if __name__ == "__main__":
unittest.main()
tests/visualization/test_visualization.py
+1 -23
View File
@@ -1,27 +1,5 @@
import unittest
from unittest.mock import MagicMock, patch, ANY
import sys
import types
# Helper to create a mock package
def mock_package(name):
m = MagicMock()
m.__path__ = []
sys.modules[name] = m
return m
# Mock libraries before importing module under test
# We need to ensure matplotlib behaves like a package for seaborn
sys.modules['matplotlib'] = MagicMock()
sys.modules['matplotlib.colors'] = MagicMock()
sys.modules['matplotlib.pyplot'] = MagicMock()
sys.modules['matplotlib.patches'] = MagicMock()
sys.modules['plotly'] = MagicMock()
sys.modules['plotly.express'] = MagicMock()
sys.modules['plotly.graph_objects'] = MagicMock()
sys.modules['plotly.subplots'] = MagicMock()
sys.modules['graphviz'] = MagicMock()
sys.modules['seaborn'] = MagicMock()
from unittest.mock import MagicMock, patch
from semantica.visualization.kg_visualizer import KGVisualizer
from semantica.visualization.ontology_visualizer import OntologyVisualizer
tests/visualization/test_visualization_advanced.py
+20 -35
View File
@@ -1,34 +1,25 @@
import unittest
from contextlib import ExitStack
from unittest.mock import MagicMock, patch
import sys
import numpy as np
# Mock heavy libraries before importing visualization modules
sys.modules['matplotlib'] = MagicMock()
sys.modules['matplotlib.pyplot'] = MagicMock()
sys.modules['matplotlib.colors'] = MagicMock()
sys.modules['matplotlib.patches'] = MagicMock()
sys.modules['plotly'] = MagicMock()
sys.modules['plotly.express'] = MagicMock()
sys.modules['plotly.graph_objects'] = MagicMock()
sys.modules['plotly.subplots'] = MagicMock()
sys.modules['seaborn'] = MagicMock()
sys.modules['umap'] = MagicMock()
sys.modules['sklearn'] = MagicMock()
sys.modules['sklearn.decomposition'] = MagicMock()
sys.modules['sklearn.manifold'] = MagicMock()
from semantica.visualization import analytics_visualizer, embedding_visualizer
from semantica.visualization.analytics_visualizer import AnalyticsVisualizer
from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
from semantica.visualization.utils.color_schemes import ColorScheme
from tests.visualization._plotly_doubles import plotly_doubles
class TestVisualizationAdvanced(unittest.TestCase):
def setUp(self):
self.mock_logger = MagicMock()
self.mock_tracker = MagicMock()
stack = ExitStack()
self.addCleanup(stack.close)
stack.enter_context(plotly_doubles(analytics_visualizer, embedding_visualizer))
self.patchers = [
patch('semantica.visualization.analytics_visualizer.get_logger', return_value=self.mock_logger),
patch('semantica.visualization.analytics_visualizer.get_progress_tracker', return_value=self.mock_tracker),
@@ -53,22 +44,17 @@ class TestVisualizationAdvanced(unittest.TestCase):
viz = AnalyticsVisualizer()
centrality = {"n1": 0.5, "n2": 0.3}
# Access the mock that was injected
import plotly.graph_objects as go
# Reset mock to ensure clean state
go.Bar.reset_mock()
viz.visualize_centrality_rankings(centrality, output="interactive")
go.Bar.assert_called()
with (
patch('semantica.visualization.analytics_visualizer.go.Bar') as mock_bar,
patch('semantica.visualization.analytics_visualizer.go.Figure'),
):
viz.visualize_centrality_rankings(centrality, output="interactive")
mock_bar.assert_called()
def test_visualize_community_structure(self):
viz = AnalyticsVisualizer()
if hasattr(viz, 'visualize_community_structure'):
import plotly.graph_objects as go
# Reset mocks
go.Figure.reset_mock()
graph = MagicMock()
communities = {"c1": ["n1", "n2"]}
@@ -89,8 +75,6 @@ class TestVisualizationAdvanced(unittest.TestCase):
viz = EmbeddingVisualizer()
embeddings = np.random.rand(10, 128)
import plotly.graph_objects as go
# Mock UMAP/TSNE/PCA
with patch('semantica.visualization.embedding_visualizer.umap') as mock_umap, \
patch('semantica.visualization.embedding_visualizer.TSNE') as mock_tsne, \
@@ -116,12 +100,13 @@ class TestVisualizationAdvanced(unittest.TestCase):
viz = EmbeddingVisualizer()
embeddings = np.random.rand(5, 5)
import plotly.graph_objects as go
go.Heatmap.reset_mock()
if hasattr(viz, 'visualize_similarity_heatmap'):
viz.visualize_similarity_heatmap(embeddings)
go.Heatmap.assert_called()
with (
patch('semantica.visualization.embedding_visualizer.go.Heatmap') as mock_heatmap,
patch('semantica.visualization.embedding_visualizer.go.Figure'),
):
viz.visualize_similarity_heatmap(embeddings)
mock_heatmap.assert_called()
if __name__ == '__main__':
unittest.main()
tests/visualization/test_visualization_comprehensive.py
+17 -34
View File
@@ -1,26 +1,9 @@
import unittest
from unittest.mock import MagicMock, patch
import sys
import numpy as np
from pathlib import Path
import pytest
# Mock heavy libraries before importing visualization modules
sys.modules['matplotlib'] = MagicMock()
sys.modules['matplotlib.pyplot'] = MagicMock()
sys.modules['matplotlib.colors'] = MagicMock()
sys.modules['matplotlib.patches'] = MagicMock()
sys.modules['plotly'] = MagicMock()
sys.modules['plotly.express'] = MagicMock()
sys.modules['plotly.graph_objects'] = MagicMock()
sys.modules['plotly.subplots'] = MagicMock()
sys.modules['seaborn'] = MagicMock()
sys.modules['umap'] = MagicMock()
sys.modules['sklearn'] = MagicMock()
sys.modules['sklearn.decomposition'] = MagicMock()
sys.modules['sklearn.manifold'] = MagicMock()
sys.modules['networkx'] = MagicMock()
sys.modules['graphviz'] = MagicMock()
# Import visualizers
from semantica.visualization.kg_visualizer import KGVisualizer
@@ -52,24 +35,24 @@ class TestVisualizationComprehensive(unittest.TestCase):
patch('semantica.visualization.analytics_visualizer.get_progress_tracker', return_value=self.mock_tracker),
patch('semantica.visualization.temporal_visualizer.get_logger', return_value=self.mock_logger),
patch('semantica.visualization.temporal_visualizer.get_progress_tracker', return_value=self.mock_tracker),
# Mock Layouts
patch('semantica.visualization.kg_visualizer.ForceDirectedLayout', MagicMock()),
patch('semantica.visualization.kg_visualizer.HierarchicalLayout', MagicMock()),
patch('semantica.visualization.kg_visualizer.CircularLayout', MagicMock()),
patch('semantica.visualization.ontology_visualizer.HierarchicalLayout', MagicMock()),
patch('semantica.visualization.semantic_network_visualizer.ForceDirectedLayout', MagicMock()),
patch('semantica.visualization.kg_visualizer.go', MagicMock()),
patch('semantica.visualization.kg_visualizer.px', MagicMock()),
patch('semantica.visualization.ontology_visualizer.go', MagicMock()),
patch('semantica.visualization.ontology_visualizer.make_subplots', MagicMock()),
patch('semantica.visualization.embedding_visualizer.go', MagicMock()),
patch('semantica.visualization.embedding_visualizer.px', MagicMock()),
patch('semantica.visualization.semantic_network_visualizer.go', MagicMock()),
patch('semantica.visualization.semantic_network_visualizer.px', MagicMock()),
patch('semantica.visualization.analytics_visualizer.go', MagicMock()),
patch('semantica.visualization.analytics_visualizer.px', MagicMock()),
patch('semantica.visualization.analytics_visualizer.make_subplots', MagicMock()),
patch('semantica.visualization.temporal_visualizer.go', MagicMock()),
patch('semantica.visualization.temporal_visualizer.px', MagicMock()),
]
for p in self.patchers:
p.start()
# Reset plotly mocks
import plotly.graph_objects as go
import plotly.express as px
go.Figure.reset_mock()
px.bar.reset_mock()
px.scatter.reset_mock()
def tearDown(self):
for p in self.patchers:
p.stop()
@@ -210,8 +193,8 @@ class TestVisualizationComprehensive(unittest.TestCase):
embeddings = np.random.rand(10, 10)
# Test visualize_2d_projection (mock UMAP/PCA)
with patch('semantica.visualization.embedding_visualizer.umap.UMAP') as MockUMAP:
MockUMAP.return_value.fit_transform.return_value = np.random.rand(10, 2)
with patch('semantica.visualization.embedding_visualizer.umap', MagicMock()) as mock_umap:
mock_umap.UMAP.return_value.fit_transform.return_value = np.random.rand(10, 2)
viz.visualize_2d_projection(embeddings)
# Test visualize_similarity_heatmap
@@ -219,8 +202,8 @@ class TestVisualizationComprehensive(unittest.TestCase):
# Test visualize_clustering
clusters = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
with patch('semantica.visualization.embedding_visualizer.umap.UMAP') as MockUMAP:
MockUMAP.return_value.fit_transform.return_value = np.random.rand(10, 2)
with patch('semantica.visualization.embedding_visualizer.umap', MagicMock()) as mock_umap:
mock_umap.UMAP.return_value.fit_transform.return_value = np.random.rand(10, 2)
viz.visualize_clustering(embeddings, clusters)
if __name__ == '__main__':