mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Merge branch 'main' into fix/kg-validator-entity-id
This commit is contained in:
Guofang.Tang
+95
-3
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.6.6] - 2026-08-20
|
||||
|
||||
### Added
|
||||
|
||||
- **Semantica RDF vocabulary, and deterministic entity/relationship IRIs** (#1109, closes #1107, closes #1101) by @fabio-rovai, reviewed by @KaifAhmad1
|
||||
@@ -18,9 +20,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- The missing-id fallback minted entity/relationship IRIs from Python's builtin `hash()`, randomised per process (`PYTHONHASHSEED`), so the same entity got a different IRI on every run and exports couldn't be diffed, deduplicated, or joined to an earlier provenance record. It also wrote `<semantica:entity_N>`, an IRI in the scheme `semantica` rather than the expansion of the declared prefix, so those nodes never joined with anything written through it. Minting now uses SHA-256 and writes a full IRI in the declared namespace; the same fix applies to the default entity/relationship types in the Turtle path
|
||||
- **Fixed during review** (Qodo): the temporal fallback minted from `source_id` only, while the main serializer accepts `source_id` or `source` — relationships using the second form hashed two empty strings, which the previous randomised `hash()` masked by making the IRI unstable anyway; once deterministic, unrelated relationships at the same list index collided on one IRI across exports. Endpoints are now resolved the same way `serialize_to_turtle` resolves them, before minting. `sem:confidence` also lost its declared `xsd:decimal` range: the N-Triples serializer types the same value `xsd:float`, and the two are disjoint, so declaring either contradicted one of the exporters (tracked in #1100) — a new `test_declared_ranges_do_not_contradict_what_the_exporters_emit` guards the whole class of that mistake
|
||||
- **Fixed in follow-up**: `serialize_to_rdfxml`'s default entity type still wrote the bare string `"semantica:Entity"` into an `rdf:resource` attribute, which (unlike a Turtle angle-bracket or an XML element name) is not namespace-expanded — the exact #1101 failure mode, just on the untested RDF/XML path. `json_exporter.py`'s `semantica:format` and `@type: "semantica:KnowledgeGraph"` were emitted but absent from both the vocabulary and the test's `EMITTED_TERMS` guard set, so the "undeclared terms fail the build" claim didn't actually cover them — both are now declared and guarded. `MANIFEST.in` didn't mirror the `pyproject.toml` package-data addition, so a source-distribution install could omit the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only `PATH`, breaking it on Windows; now overrides only `PYTHONHASHSEED` on top of the inherited environment
|
||||
- 229 export and ontology tests pass
|
||||
- **Also fixed, on the JSON-LD paths**: the first fix covered the Turtle, N-Triples and RDF/XML serializers, and left both JSON-LD writers interpolating the entity's own text into `f"semantica:entity/{text}"` and the endpoints into `f"semantica:rel/{source}_{target}"`. Three consequences, all live in 0.6.5: an entity whose text contained a space produced an invalid IRI, and a JSON-LD parser dropped that node in full rather than reporting it, so the entity disappeared from the export; every relationship carrying `source`/`target` rather than `source_id`/`target_id` minted the identical `semantica:rel/_`, collapsing all of them onto one node whose types and endpoints merged; and the JSON-LD `@id` disagreed with the Turtle IRI for the same entity, so the two serializations of one knowledge graph were two different graphs. Both JSON-LD writers now use `mint_entity_iri`/`mint_relationship_iri`, and `JSONExporter.export_entities`/`export_relationships` declare the `semantica` prefix their `@context` was already writing `semantica:entities` against — without it a processor reads that as an IRI in the scheme `semantica`, which is the original #1101 defect on a third path
|
||||
- `tests/export/test_jsonld_iri_minting.py` parses each export with a real JSON-LD processor and asserts the entity survives, the relationships stay distinct, no term expands into the `semantica` scheme, and the JSON-LD `@id` equals the Turtle IRI
|
||||
- 236 export and ontology tests pass
|
||||
|
||||
- **First-class CrewAI integration** (#962)
|
||||
- **First-class CrewAI integration** (#988, closes #962) by @Shindevrp
|
||||
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
|
||||
- `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()`
|
||||
- `integrations/crewai/SemanticaDecisionTool` — a CrewAI `BaseTool` wrapping `AgentContext` with 5 decision-intelligence actions (`record_decision`, `find_precedents`, `trace_causal_chain`, `analyze_impact`, `check_policy`)
|
||||
@@ -50,6 +54,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- New `tests/export/test_distance_exporter_metric_errors.py`: 6 tests covering success, single/multiple failures, opt-out, the no-path-vs-error distinction, and default-schema stability; existing `tests/export/test_distance_exporter.py` updated for the new tuple return type
|
||||
- Full `tests/export/` suite: 77 passed
|
||||
|
||||
- **`ContextGraph.to_kg_dict()`: an adapter converting a `ContextGraph`'s internal `nodes`/`edges`/`source` shape into the canonical `entities`/`relationships`/`source_id` shape `RDFExporter` and `TemporalGraphQuery` consume** (#1081) by @cxzg007
|
||||
- Previously there was no supported way to feed a `ContextGraph` into those consumers without hand-rolling the field remapping; `to_kg_dict()` does it once, with an `entities_only` option that drops relationships left dangling by the filter
|
||||
- **Fixed during review** (Qodo): null `properties`/`metadata` on a node loaded from JSON raised `TypeError` when copied — both are now guarded with `or {}`; entity ids are coerced to `str(node_id)` to match `ContextEdge`'s already-str-coerced endpoints, so valid relationships were no longer dropped by `entities_only` filtering
|
||||
- `RDFExporter`'s validator and `TemporalGraphQuery` now also accept `source_id`/`target_id` endpoints, the shape `to_kg_dict()` emits
|
||||
|
||||
### Changed
|
||||
|
||||
- **`GraphBuilder`'s 6 public methods now have Google-style docstrings** (#878, closes #876) by @cakeni
|
||||
@@ -59,7 +68,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- **Corrected during review**: `add_temporal_edge`/`create_temporal_snapshot` docstrings overclaimed numeric-timestamp support; `_parse_time()` only special-cases `str` and `datetime`, falling back to a bare `str()` cast for anything else (not true numeric parsing). Narrowed to "datetime or ISO-formatted string"
|
||||
- **Fixed along the way**: `build()`'s `**options` documented a default only for `extract`; `extract_relations`, `extract_triplets`, `ner_method`, `relation_method`, and `triplet_method` all have concrete defaults in `_extract_from_text()` (`True`, `True`, `"llm"`, `"llm"`, `"llm"`) that were left unstated, inconsistent with CONTRIBUTING.md's own docstring example of noting defaults inline
|
||||
- `python -m pytest tests/kg/test_kg.py tests/kg/test_graph_builder_external.py -q`: 45 passed
|
||||
- **`GraphBuilder` raw-text extraction now defaults to local extractors instead of LLM extraction** (closes #930) by @dex0shubham
|
||||
- **`GraphBuilder` raw-text extraction now defaults to local extractors instead of LLM extraction** (#941, closes #930) by @dex0shubham
|
||||
- `GraphBuilder._extract_from_text()` defaulted `ner_method`, `relation_method`, and `triplet_method` to `"llm"`, and ran relation extraction unconditionally (`extract_relations` defaulted to `True`) — all four contradicting the defaults documented in the `build()` docstring at the time (`"ml"` / `"pattern"` / `False`), and diverging from the standalone extractors (`NERExtractor` defaults to `method="ml"`, `RelationExtractor` and `TripletExtractor` to `method="pattern"`). The practical effect was that any raw-text `build()` call silently required a configured provider, an API key, and network access
|
||||
- Defaults are now `ner_method="ml"`, `relation_method="pattern"`, `triplet_method="pattern"`, and `extract_relations=False`, matching the docstring. LLM extraction remains fully available and is now opt-in
|
||||
- **To restore the previous behaviour**, pass the methods explicitly:
|
||||
@@ -80,8 +89,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- New regression coverage in `tests/kg/test_graph_builder_extraction_defaults.py` pinning all four defaults, verifying that no default resolves to `"llm"`, confirming explicit LLM opt-in still routes correctly, asserting extractors are constructed once across repeated texts, covering fallback method lists (e.g. `ner_method=["pattern", "ml"]`) for all three extractors, asserting relations are forwarded to triplet extraction (and that `None` is forwarded when relation extraction is disabled or fails), and running the real default path end to end with no provider mocked. Verified to fail against the pre-fix code
|
||||
- Full `kg` suite: 473 passed
|
||||
|
||||
- **Explorer graph canvas now renders edge labels** (#1013, closes #1009) by @yzxcj797
|
||||
- `GraphCanvas.tsx` had no edge-label rendering path at all; Sigma's edge-label renderer draws `data.label`, but the graph state stored the relationship type under `edgeType`, so simply enabling the renderer would have left every edge blank. `graphSceneState`'s edge reducer now maps `edgeType` onto `label` (suppressed for hidden edges)
|
||||
- Rendering is gated behind a new `edgeLabelsEnabled` entry in the Effects panel (default on), wired through the existing `GraphEffectToggle`/`GraphEffectsState` plumbing, so dense graphs can still turn labels off
|
||||
- **Fixed during review** (Qodo): two follow-up passes closed gaps the first cut left — label rendering wasn't wired through `explorationEffectsPluginPhaseC.tsx`'s Phase C variant, and toggling the effect off mid-session didn't clear already-rendered labels
|
||||
- New coverage in `explorer/tests/graphSceneState.display.test.ts`
|
||||
|
||||
- **Removed `GraphWorkspaceShell.tsx`, `GraphRuntimeStage.tsx`, and `useGraphData.ts` — a second, unused implementation of the graph-loading/error-handling logic already fixed in `GraphWorkspace.tsx`** (#984, resolves the cleanup tracked in #981 by #980's review note) by @lakshayxi
|
||||
- 1,564 lines removed; the surviving `GraphWorkspace` path is now the only implementation, so the "two copies that drifted apart" root cause #980 fixed can't recur in the copy nobody was maintaining
|
||||
|
||||
- **Explorer README and `docs/explorer-setup.md` corrected to describe the authentication 0.6.5 actually shipped**, plus a documented `/ws/graph-updates` auth note (#1040, fixes #1028) by @Kyou12138
|
||||
- Both docs still claimed the Explorer API had no built-in authentication after v0.6.5 added mandatory `SEMANTICA_API_KEY` enforcement with a `503` fail-closed default; corrected to describe the actual behavior, including that only protected routes require the key (`/api/health`/`/api/info` stay open), the non-loopback-bind CLI warning only fires in anonymous mode or when the key is unset, and `SEMANTICA_API_KEY`/`SEMANTICA_ALLOW_ANONYMOUS` are documented in the environment-variable table
|
||||
|
||||
- **CI: pinned `github/codeql-action` to current v4** (#986) by @ZohaibHassan16, and **pinned Python dependencies in `requirements-ci.txt` for reproducible CI runs** (#945) by @yunaremaia, closing the gap where an unpinned CI dependency could silently change behavior between runs
|
||||
|
||||
- **README now states up front that Semantica's explainability is system-level, not foundation-model-internal** (#1033, #1034) by @KaifAhmad1
|
||||
- Nothing in the README previously scoped what "explainable" meant, leaving readers to assume Semantica could expose or reconstruct an LLM's internal reasoning. A callout now states explicitly that Semantica explains and audits what the AI *system* did — context fed in, decisions produced, provenance, relationships, policies applied — not the model's private internal reasoning, and moved the note near the top of the README rather than leaving it implicit
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Every timestamp an export or a provenance record wrote was timezone-naive** (closes #1114) by @fabio-rovai
|
||||
- `semantica/export/` stamped with `datetime.now().isoformat()`, which reads the machine's **local** clock; `semantica/provenance/` stamped with `datetime.utcnow().isoformat()`, which reads **UTC**. Both produce a naive value and both serialize identically, so nothing downstream can tell which zone a given timestamp belongs to — the same string means two different instants depending on which module wrote it
|
||||
- In RDF the consequence is silent rather than loud. Under XSD 1.1 a value with no timezone compared against one with a timezone is indeterminate whenever the two fall inside the ±14 hour window; SPARQL turns an indeterminate comparison into an error, and `FILTER` discards errors as non-matches. A timezone-qualified query over an Oxigraph store returns an answer with every Semantica-written record quietly absent from it, which is a poor property for `prov:generatedAtTime`, `prov:startedAtTime`, `prov:endedAtTime` and `prov:atTime` to have
|
||||
- New `utc_now()`/`utc_now_iso()` in `semantica/utils/helpers.py`, exported from `semantica.utils`, and used at all 29 call sites in `export/` (`json_exporter`, `yaml_exporter`, `report_generator`, `export_provenance`) and `provenance/` (`manager`, `schemas`, `bridge_axiom`). Values now read `2026-08-19T14:19:04.229937+00:00`: one unambiguous instant, comparable against any correctly stamped value, and valid `xsd:dateTimeStamp`. `sem:exportedAt`'s range in `semantica/ontology/vocabulary/semantica-ns.ttl` is tightened from `xsd:dateTime` accordingly, and its comment no longer has to explain why the weaker range was necessary
|
||||
- `datetime.utcnow()` is deprecated as of Python 3.12 and scheduled for removal; constructing a `ProvenanceEntry` under `-W error::DeprecationWarning` on 3.13 raised, and no longer does
|
||||
- New `tests/export/test_timestamp_timezones.py` and `tests/provenance/test_timestamp_timezones.py`: offset presence on every export and provenance path, PROV-O literals valid as `xsd:dateTimeStamp`, comparison against a timezone-aware instant without `TypeError`, the Oxigraph filter that dropped the naive value (with a bound inside the indeterminate window, so the test cannot pass by accident), and the document `@id` remaining a valid IRI with `+00:00` in it. 11 of the 13 fail on the parent commit
|
||||
- **Fixed during review** (Qodo): once new entries carry `+00:00` and stored ones do not, `ProvenanceManager.query_recorded_between` and `audit_log` compared ISO timestamps as raw strings, so they ordered by spelling rather than by instant — an inclusive naive bound naming a stored offset-bearing timestamp sorted *below* it and dropped the record, and a bound written in another offset landed wherever its digits fell (`19:45+05:30` is 14:15Z, but sorted after 14:19Z). Both now compare instants through a new `to_utc_datetime()` helper that reads a missing offset as UTC, which is what the values written before this change actually were; a bound that cannot be read as a timestamp keeps the historical string comparison rather than raising on a call that used to work
|
||||
- The remaining 147 naive call sites are in `context/`, `vector_store/`, `seed/` and elsewhere, where timestamps are compared against values parsed from previously stored naive strings. Converting those without a read-side migration would raise `TypeError: can't compare offset-naive and offset-aware datetimes` on existing data, so they are deliberately left for a separate change
|
||||
|
||||
- **`split`/chunking paths bypassed the centralized spaCy model cache, reloading the model on every call** (#1042, closes #998) by @Accute9, reviewed by @Sameer6305
|
||||
- `semantica/split/methods.py`'s `split_by_sentences()` and `semantica/split/semantic_chunker.py`'s `SemanticChunker.__init__` each called `spacy.load()` directly instead of reusing the process-level cache added in #889/`semantic_extract/methods.py`'s `load_spacy_model()` — every call/construction re-paid the ~120ms model-load cost independently of `NERExtractor`, which already used the cache
|
||||
- Both now route through `load_spacy_model()`, sharing one cached `Language` instance per model name across `split_by_sentences()`, `SemanticChunker`, and `NERExtractor`; a missing model still falls back to regex/paragraph chunking without poisoning the cache for a later successful load
|
||||
@@ -183,6 +218,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- New regression coverage in `tests/export/test_distance_exporter.py`: warnings fire on exception for all four helpers, exported sentinel values/shape stay unchanged, and the legitimate "no KG backend" `None` path still logs nothing
|
||||
- Full `tests/export/` suite: 71 passed
|
||||
|
||||
- **`explain_violations` rendered hardcoded placeholders (`min_count=1`, `max_count=1`) instead of the SHACL shape's real constraint values, and misused the violation message text as the datatype/class value** (#1094) by @cxzg007
|
||||
- `_run_pyshacl` never read `sh:minCount`/`sh:maxCount`/`sh:datatype`/`sh:class` back from the violation's `sh:sourceShape`, so every plain-English explanation was wrong regardless of what the shape actually declared. `SHACLViolation` now carries those four fields (also exposed via `to_dict()`), populated by back-referencing `sh:sourceShape`; `explain_violations` renders the real values, falling back to `"?"` when a value is genuinely absent
|
||||
- **Known limitation**: `sh:qualifiedMinCount`/`sh:qualifiedMaxCount` are not handled yet and still fall back to the `"?"` placeholder
|
||||
- New regression tests cover both the rendering path and the `sh:sourceShape` back-reference (skipped when `pyshacl`/`rdflib` are absent)
|
||||
|
||||
- **Entity merging silently dropped `entity_id` aliases, and exact-match entity resolution had three correctness gaps** (#1086, #1026) by @T1mn
|
||||
- `entity_merger.py`/`merge_strategy.py`/`entity_resolver.py` used inconsistent logic for extracting an entity's id across the merge path, so a merged entity could lose the `entity_id` aliases that let later lookups find it under its old identity. A new `semantica/utils/entity_ids.py` unifies id extraction across all three call sites
|
||||
- `EntityResolver`'s exact-match path is now honored rather than silently falling through to fuzzy matching in some cases; entities with no identifier are preserved instead of being dropped, and blank exact-match names are ignored rather than matching every other blank name
|
||||
- New/expanded coverage in `tests/kg/test_entity_pipeline.py` and `tests/kg/test_entity_resolver_exact.py`
|
||||
|
||||
- **`flatten_dict()` silently collided keys when a flattened path from one branch matched a literal key already present at the target depth** (#1062) by @shahzaib-ahmadcs
|
||||
- Two differently-shaped inputs could flatten to the same output key, with the second write silently overwriting the first — no error, no warning, just a dropped value. Collisions are now detected and handled explicitly instead of overwriting
|
||||
|
||||
- **`ExcelParser.__init__` raised `NameError` on every instantiation — `get_progress_tracker()` was called but never imported** (#1016, closes #1014) by @pravit-amp
|
||||
- Same defect as the one fixed for `SimilarityCalculator` in #530, this time in `semantica/parse/excel_parser.py`; the existing test imported the class but never constructed it, so nothing caught the missing import. Added construction coverage for every parser exported from `semantica.parse`, driven off `__all__` so future additions are covered automatically, living outside `test_parse_comprehensive.py` (whose `setUp` mocks `get_progress_tracker` into each module and would mock away the exact interaction under test)
|
||||
|
||||
- **Graph analytics (`centrality_calculator.py`, `community_detector.py`, `connectivity_analyzer.py`) dropped isolated nodes and diverged on how each computed its working view of the graph** (#1011) by @T1mn
|
||||
- Each analyzer had its own ad hoc logic for building the node/edge set it operated over, and none of them included nodes with no edges — a node with zero connections simply vanished from centrality scores, community assignments, and connectivity reports instead of appearing with a zero/singleton value. A new shared `semantica/kg/_graph_view.py` centralizes graph-view construction (including node fallbacks and community payload shaping) for all three analyzers, which are now ~250 lines lighter combined
|
||||
- New `tests/kg/test_analytics_node_scope.py` covering isolated-node presence across all three analyzers
|
||||
|
||||
- **Explorer fired temporal-bounds and snapshot requests before the graph itself had loaded, tripling failed requests when the backend was down and leaving the timeline scrubber with nothing to scrub** (#1003) by @lakshayxi
|
||||
- Two new predicate functions gate the temporal effects on the graph having actually loaded (an empty graph still counts as loaded); confirmed against a downed backend that this cuts three failing requests per page load down to one
|
||||
|
||||
- **`SeedDataManager.load_from_database()` never actually reached the database, and connection failures were mislabeled as a missing optional dependency** (#995, closes #973) by @yzxcj797
|
||||
- `DBIngestor.execute_query`/`export_table` need the connection string as their first positional argument; `load_from_database()` only passed it into the constructor's config dict, which those methods never read, so every call raised `TypeError` before connecting. Also split the combined `except (ImportError, OSError)` handling apart — a genuine connection failure was reported as `"module not available"`, sending debugging in the wrong direction; `OSError` now propagates as an actual failure, chained via `from e`
|
||||
|
||||
- **SPARQL `CONSTRUCT` detection matched inside a leading `#`-comment, misclassifying `SELECT`/`ASK` queries as `CONSTRUCT` across all four SPARQL backends** (#951) by @pravit-amp
|
||||
- `CONSTRUCT_QUERY_RE` skipped comments with a bare `\#[^\n]*`, whose backtracking `*` let a `# CONSTRUCT ...` comment line "swallow" the real query-form keyword on the next line for a query like `# CONSTRUCT ...\nSELECT ...`. The mistaken `CONSTRUCT` classification sent `Accept: text/turtle` and tried to parse a SELECT/ASK response body as Turtle, failing with a misleading parse error. The regex now requires a comment to reach a line terminator (LF or CR, per the SPARQL grammar) before matching
|
||||
|
||||
- **`k_shortest_paths` mutated caller-visible graph state during traversal and ignored direction when excluding already-used edges** (#1000) by @T1mn
|
||||
- `semantica/kg/path_finder.py`'s search left side effects behind after returning, and edge exclusion during Yen's-algorithm-style path removal didn't respect the traversal direction of directed graphs, letting a later search see edges that should have been available. Both fixed; new coverage in `tests/kg/test_path_finder.py`
|
||||
|
||||
- **`trace_decision_causality()` ignored explicitly recorded causal edges, inferring causes only from shared NER entities plus timestamp ordering** (#983) by @hsd2514
|
||||
- A `CAUSED`/`INFLUENCED`/`PRECEDENT_FOR` edge added via `add_causal_relationship()` had no effect on the trace — when entity extraction found nothing in common between two decisions, `trace_decision_chain()` came back empty even with an explicit edge stored in the graph. Explicit causal edges are now traversed first as ground truth, with entity/timestamp inference kept as an additive fallback for pairs with no explicit link; edges whose source has no decision record (e.g. a graph restored via `from_dict`) are skipped so a stale edge can't abort the trace
|
||||
|
||||
- **`RepoIngestor`'s module-level DNS resolve cache had no lock, raising `RuntimeError: OrderedDict mutated during iteration` under concurrent `ingest_repository()` calls** (#979) by @manjunathbhaskar
|
||||
- `_REPO_HOST_RESOLVE_CACHE` is a shared `OrderedDict` read, written, and pruned by every thread with no synchronization — reliably reproduced with 32 threads hammering resolution under a low TTL and small cache cap. Now guarded by a lock
|
||||
|
||||
- **`GraphBuilder` didn't remap relationship endpoints after entity resolution merged nodes, leaving relationships pointing at ids that no longer existed in the resolved graph** (#978) by @T1mn
|
||||
- New coverage in `tests/kg/test_graph_builder_external.py`; a follow-up commit hardens the remapping against edge cases found during review
|
||||
|
||||
- **Explorer's dev server esbuild target didn't match the browser targets the production build declares**, occasionally producing dev-only syntax errors on older browsers (#966) by @le-czs
|
||||
- `explorer/vite.config.ts` now sets the dev esbuild target explicitly to match
|
||||
|
||||
- **`normalize`'s number normalizer accepted currency symbols without validating them against the surrounding text, and an earlier fix's currency-code matching wasn't token-bounded** (#940) by @Mr-Neutr0n, reviewed by @ZohaibHassan16
|
||||
- Symbol currencies are now validated before being accepted; currency codes are matched on token boundaries so a code embedded inside a longer token no longer false-positives
|
||||
|
||||
- **`ContextGraph.to_dict()` was the one reader on the class that didn't hold `self._lock`, raising `RuntimeError: dictionary changed size during iteration` under a concurrent writer and risking a torn snapshot otherwise** (#929) by @pravit-amp
|
||||
- Every other reader (`stats()`, `density()`, `find_nodes()`, `find_edges()`, `get_neighbors()`, `get_nodes_by_label()`, `state_at()`, `save_to_file()`) already took the lock after it was introduced; `to_dict()` predated that change and was missed. `save_to_file()` was safe only incidentally, since it builds its payload inline under its own lock rather than delegating to `to_dict()`
|
||||
|
||||
- **`PipelineWithProvenance` had a broken import and no working `run()` method** (#862) by @Karunasagar12
|
||||
- `from .pipeline import Pipeline` failed because `Pipeline` lives in `pipeline_builder.py`, not a nonexistent `pipeline.py` — fixed to `from .pipeline_builder import Pipeline`. The class also had no `run()`; it now delegates to `ExecutionEngine.execute_pipeline()`, the intended execution path for a built `Pipeline`. The constructor now accepts a built `Pipeline` instance directly
|
||||
|
||||
### Security
|
||||
|
||||
- **Tarball restore path traversal, latent SQL injection, DNS-rebinding TOCTOU in the shared SSRF guard, stored XSS in report generation, and unvalidated SPARQL object IRIs in AnzoStore** (#1079) by @KaifAhmad1
|
||||
@@ -243,6 +331,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- **Caught by the new gate on its first run**: `python -m pip install -e ".[all]"` pulled in `setuptools==79.0.1`, vulnerable to CVE-2026-59890/GHSA-h35f-9h28-mq5c/PYSEC-2026-3447 (Unicode-normalization bypass of `MANIFEST.in` exclude/prune patterns on macOS APFS/HFS+, letting excluded files leak into a built sdist), fixed in `83.0.0`. `[build-system] requires` had the exact same too-permissive-floor pattern this whole entry is about (`setuptools>=61.0`), and `actions/setup-python`'s baked-in `setuptools` isn't governed by that pin at all since it's outside any isolated build. Bumped `[build-system] requires` to `setuptools>=83.0.0`, and the `Security` workflow now runs `pip install --upgrade pip setuptools` before auditing so the scanned environment can't have a stale ambient copy regardless of what governs it
|
||||
- Full `explorer` suite: 241 passed
|
||||
|
||||
- **`SeedDataManager.load_from_api()` made unguarded HTTP requests, with no SSRF protection at all** (#942) by @ZohaibHassan16
|
||||
- `load_from_api()` called `requests.get()` directly instead of going through `semantica/ingest/ssrf.py`'s `request_with_ssrf_guard()`, unlike every other ingestor in this module — a caller-supplied `api_url` could target internal/private network addresses with no validation. Now routes through the shared guard, gaining redirect validation and bounded DNS resolution for free
|
||||
- **Follow-up** (#959, closes #943) by @yunaremaia: added an `allow_private_ips` opt-in (parsed via the shared `parse_bool` helper) for trusted internal deployments that legitimately need to load from a private-network API, while keeping the guard's block-by-default behavior for everyone else
|
||||
|
||||
## [0.6.5] - 2026-08-11
|
||||
|
||||
### Added
|
||||
|
||||
@@ -142,7 +142,7 @@ compliant = graph.check_decision_rules({"category": "vendor_selection"}) # poli
|
||||
```bash
|
||||
semantica doctor
|
||||
# Python 3.11.9 pass
|
||||
# semantica 0.6.5 pass
|
||||
# semantica 0.6.6 pass
|
||||
# faiss vector store pass
|
||||
# Config file pass ~/.semantica/config.yaml
|
||||
```
|
||||
@@ -1466,18 +1466,18 @@ For contributor / dev-server setup: **[explorer/README.md: Local Setup Guide](ex
|
||||
|
||||
---
|
||||
|
||||
## What's New in v0.6.5
|
||||
## What's New in v0.6.6
|
||||
|
||||
**Security release — upgrading is strongly recommended.** Fixes for 5 externally-reported vulnerabilities in the Explorer API and graph/triplet store backends, plus a CodeQL-flagged ReDoS:
|
||||
**Security release — upgrading is strongly recommended.** Fixes for a privately disclosed batch of vulnerabilities spanning backup/restore, database export, outbound requests, and triplet-store backends, plus SSRF hardening across ingestion:
|
||||
|
||||
- **Missing authentication on all Explorer API routes** (GHSA-j4mq-hprp-987v, Critical): every route now requires `SEMANTICA_API_KEY`, fails closed (503) rather than open when unconfigured
|
||||
- **SSRF via redirect bypass in ontology URL fetching** (GHSA-8c7v-62gr-hj6g, High): redirect targets are now re-validated at every hop and the connection is pinned to the validated address, closing a DNS check-then-use race
|
||||
- **Cypher injection via unvalidated node labels and property keys** (GHSA-482h-hw99-h62p, Critical): Neptune, Neo4j, and FalkorDB now sanitize every label/relationship-type/property-key interpolation site
|
||||
- **SPARQL injection via unvalidated triplet IRIs** (GHSA-8vgg-8mr4-r236, Critical): Blazegraph, RDF4J, and Jena now validate subject/predicate/object IRIs before interpolation
|
||||
- **Missing Origin validation on the WebSocket handshake** (GHSA-4643-wpgq-w329, Moderate, anonymous-mode only): `/ws/graph-updates` now checks `Origin` against the same allowlist `CORSMiddleware` enforces for HTTP
|
||||
- **Polynomial ReDoS in SPARQL query validation** (CodeQL `py/polynomial-redos`): fixed a backtracking regex in the Explorer's SPARQL route
|
||||
- **Tarball restore path traversal**: `semantica backup restore` now validates every archive member for path containment and rejects symlink/hardlink escapes before extraction
|
||||
- **Latent SQL injection in `DataExporter.export_table_data()`**: table/schema names are now identifier-allowlisted and `where`/`order_by` fragments are blocklist-checked
|
||||
- **DNS-rebinding TOCTOU in the shared SSRF guard**: the resolved IP that passes validation is now the one the connection is pinned to, closing the check-then-use race (also closes the `100.64.0.0/10` CGNAT gap)
|
||||
- **Stored XSS in HTML report generation** and **unvalidated SPARQL object IRIs in AnzoStore** (SPARQL injection): both now escape/validate before interpolation
|
||||
- **`Authorization`/`Proxy-Authorization` credential leakage across redirects**, plus **SSRF gaps in `FeedIngestor`/`FeedMonitor`, `RepoIngestor`, and the MCP/public-API ingest paths**: all now route through the shared, redirect-safe SSRF guard
|
||||
- **HTTP response header injection and an unbounded-memory DoS** in the Explorer API, and a **`fastapi`/`python-multipart` ReDoS** (PYSEC-2024-38): floors raised, inputs sanitized, candidate pools capped
|
||||
|
||||
Also includes: embedded Oxigraph backend for `TripletStore`, PROV-O trust/spec completeness for `ProvenanceManager`, and the Altair Anzo triplet store backend.
|
||||
Also ships: **first-class CrewAI integration** (`semantica[crewai]`, extraction/decision tools + a knowledge source), **`ContextGraph` retraction and purge** (GDPR-style erasure without a full `clear()`), a declared **Semantica RDF vocabulary with deterministic entity/relationship IRIs** (stable, diffable exports), and **timezone-aware timestamps** across `export/` and `provenance/`.
|
||||
|
||||
→ [Full release notes](RELEASE_NOTES.md) · [Changelog](CHANGELOG.md)
|
||||
|
||||
|
||||
+5
-5
@@ -17,22 +17,22 @@ icon: "quote-left"
|
||||
author = {Semantica},
|
||||
year = {2026},
|
||||
url = {https://github.com/semantica-agi/semantica},
|
||||
version = {0.6.5},
|
||||
version = {0.6.6},
|
||||
doi = {10.5281/zenodo.XXXXXXX}
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="APA">
|
||||
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* (Version 0.6.5) \[Computer software\]. https://github.com/semantica-agi/semantica
|
||||
Semantica. (2026). *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems* (Version 0.6.6) \[Computer software\]. https://github.com/semantica-agi/semantica
|
||||
</Tab>
|
||||
<Tab title="MLA">
|
||||
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.5, GitHub, 2026, https://github.com/semantica-agi/semantica.
|
||||
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.6, GitHub, 2026, https://github.com/semantica-agi/semantica.
|
||||
</Tab>
|
||||
<Tab title="Chicago">
|
||||
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.5. GitHub, 2026. https://github.com/semantica-agi/semantica.
|
||||
Semantica. *Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems*. Version 0.6.6. GitHub, 2026. https://github.com/semantica-agi/semantica.
|
||||
</Tab>
|
||||
<Tab title="IEEE">
|
||||
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," Version 0.6.5, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
|
||||
Semantica, "Semantica: Graph-Native Infrastructure for Context and Accountable AI Systems," Version 0.6.6, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ icon: "circle-question"
|
||||
| API key required? | Optional: pattern extraction works with no keys |
|
||||
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
|
||||
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
|
||||
| Latest version? | **v0.6.5** (August 2026) |
|
||||
| Latest version? | **v0.6.6** (August 2026) |
|
||||
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
|
||||
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ icon: "rocket"
|
||||
Verify installation:
|
||||
```python
|
||||
import semantica
|
||||
print(semantica.__version__) # 0.6.5
|
||||
print(semantica.__version__) # 0.6.6
|
||||
```
|
||||
</Check>
|
||||
</Step>
|
||||
|
||||
+2
-2
@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "semantica"
|
||||
version = "0.6.5"
|
||||
description = "Accountability and context layer for AI agents. Context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
|
||||
version = "0.6.6"
|
||||
description = "Graph-Native Infrastructure for Context and Accountable AI Systems: context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
|
||||
readme = "README.md"
|
||||
license = { text = "MIT" }
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ Main exports:
|
||||
- Config: Configuration management
|
||||
"""
|
||||
|
||||
__version__ = "0.6.5"
|
||||
__version__ = "0.6.6"
|
||||
__author__ = "Semantica Contributors"
|
||||
__license__ = "MIT"
|
||||
|
||||
|
||||
@@ -14,9 +14,10 @@ License: MIT
|
||||
"""
|
||||
|
||||
from typing import Any, Optional
|
||||
from datetime import datetime
|
||||
import uuid
|
||||
|
||||
from ..utils.helpers import utc_now_iso
|
||||
|
||||
|
||||
class ExporterWithProvenance:
|
||||
"""Base exporter with provenance tracking."""
|
||||
@@ -45,9 +46,9 @@ class ExporterWithProvenance:
|
||||
|
||||
def export(self, data: Any, destination: str, **kwargs):
|
||||
"""Export data with provenance tracking."""
|
||||
activity_started_at_time = datetime.utcnow().isoformat()
|
||||
activity_started_at_time = utc_now_iso()
|
||||
result = self._exporter.export(data, destination, **kwargs)
|
||||
activity_ended_at_time = datetime.utcnow().isoformat()
|
||||
activity_ended_at_time = utc_now_iso()
|
||||
|
||||
if self.provenance and self._prov_manager:
|
||||
self._prov_manager.track_entity(
|
||||
|
||||
@@ -24,14 +24,14 @@ License: MIT
|
||||
"""
|
||||
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.helpers import ensure_directory, write_json_file
|
||||
from ..utils.helpers import ensure_directory, utc_now_iso, write_json_file
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .rdf_exporter import SEMANTICA_NS, mint_entity_iri, mint_relationship_iri
|
||||
|
||||
|
||||
class JSONExporter:
|
||||
@@ -265,11 +265,12 @@ class JSONExporter:
|
||||
json_data = {
|
||||
"@context": {
|
||||
"@vocab": "https://semantica.dev/vocab/",
|
||||
"semantica": SEMANTICA_NS,
|
||||
"entities": {"@id": "semantica:entities", "@container": "@list"},
|
||||
},
|
||||
"entities": entities,
|
||||
"metadata": {
|
||||
"exported_at": datetime.now().isoformat(),
|
||||
"exported_at": utc_now_iso(),
|
||||
"entity_count": len(entities),
|
||||
**options.get("metadata", {}),
|
||||
},
|
||||
@@ -294,6 +295,7 @@ class JSONExporter:
|
||||
json_data = {
|
||||
"@context": {
|
||||
"@vocab": "https://semantica.dev/vocab/",
|
||||
"semantica": SEMANTICA_NS,
|
||||
"relationships": {
|
||||
"@id": "semantica:relationships",
|
||||
"@container": "@list",
|
||||
@@ -301,7 +303,7 @@ class JSONExporter:
|
||||
},
|
||||
"relationships": relationships,
|
||||
"metadata": {
|
||||
"exported_at": datetime.now().isoformat(),
|
||||
"exported_at": utc_now_iso(),
|
||||
"relationship_count": len(relationships),
|
||||
**options.get("metadata", {}),
|
||||
},
|
||||
@@ -339,7 +341,7 @@ class JSONExporter:
|
||||
if include_metadata:
|
||||
if "metadata" not in result:
|
||||
result["metadata"] = {}
|
||||
result["metadata"]["exported_at"] = datetime.now().isoformat()
|
||||
result["metadata"]["exported_at"] = utc_now_iso()
|
||||
if include_provenance:
|
||||
result["metadata"]["format"] = "json"
|
||||
|
||||
@@ -349,7 +351,7 @@ class JSONExporter:
|
||||
"data": data,
|
||||
"count": len(data),
|
||||
"metadata": {
|
||||
"exported_at": datetime.now().isoformat(),
|
||||
"exported_at": utc_now_iso(),
|
||||
"format": "json" if include_provenance else None,
|
||||
**options.get("metadata", {}),
|
||||
},
|
||||
@@ -358,7 +360,7 @@ class JSONExporter:
|
||||
# Single value
|
||||
return {
|
||||
"value": data,
|
||||
"metadata": {"exported_at": datetime.now().isoformat()}
|
||||
"metadata": {"exported_at": utc_now_iso()}
|
||||
if include_metadata
|
||||
else {},
|
||||
}
|
||||
@@ -410,9 +412,9 @@ class JSONExporter:
|
||||
|
||||
# Add metadata and provenance if requested
|
||||
if include_metadata:
|
||||
jsonld["@id"] = f"https://semantica.dev/data/{datetime.now().isoformat()}"
|
||||
jsonld["@id"] = f"https://semantica.dev/data/{utc_now_iso()}"
|
||||
if include_provenance:
|
||||
jsonld["semantica:exportedAt"] = datetime.now().isoformat()
|
||||
jsonld["semantica:exportedAt"] = utc_now_iso()
|
||||
jsonld["semantica:format"] = "json-ld"
|
||||
|
||||
return jsonld
|
||||
@@ -444,7 +446,7 @@ class JSONExporter:
|
||||
"nodes": kg.get("nodes", []),
|
||||
"edges": kg.get("edges", []),
|
||||
"metadata": {
|
||||
"exported_at": datetime.now().isoformat(),
|
||||
"exported_at": utc_now_iso(),
|
||||
**kg.get("metadata", {}),
|
||||
**options.get("metadata", {}),
|
||||
},
|
||||
@@ -481,7 +483,7 @@ class JSONExporter:
|
||||
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
|
||||
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
|
||||
},
|
||||
"@id": f"https://semantica.dev/graph/{datetime.now().isoformat()}",
|
||||
"@id": f"https://semantica.dev/graph/{utc_now_iso()}",
|
||||
"@type": "semantica:KnowledgeGraph",
|
||||
}
|
||||
|
||||
@@ -495,14 +497,15 @@ class JSONExporter:
|
||||
relationships = kg.get("relationships", [])
|
||||
if relationships:
|
||||
jsonld["semantica:relationships"] = [
|
||||
self._relationship_to_jsonld(r) for r in relationships
|
||||
self._relationship_to_jsonld(r, index)
|
||||
for index, r in enumerate(relationships)
|
||||
]
|
||||
self.logger.debug(
|
||||
f"Converted {len(relationships)} relationship(s) to JSON-LD"
|
||||
)
|
||||
|
||||
# Add metadata
|
||||
jsonld["semantica:exportedAt"] = datetime.now().isoformat()
|
||||
jsonld["semantica:exportedAt"] = utc_now_iso()
|
||||
if "metadata" in kg:
|
||||
jsonld["semantica:metadata"] = kg["metadata"]
|
||||
|
||||
@@ -526,11 +529,13 @@ class JSONExporter:
|
||||
Returns:
|
||||
Dictionary in JSON-LD format representing the entity
|
||||
"""
|
||||
# Generate @id if not provided
|
||||
entity_id = entity.get("id")
|
||||
if not entity_id:
|
||||
entity_text = entity.get("text") or entity.get("label", "unknown")
|
||||
entity_id = f"semantica:entity/{entity_text}"
|
||||
# Generate @id if not provided. Minted exactly as the RDF serializers
|
||||
# mint it (#1101), so the JSON-LD and Turtle exports of one knowledge
|
||||
# graph name the same entity with the same IRI. Interpolating the raw
|
||||
# text into f"semantica:entity/{text}" produced an invalid IRI for any
|
||||
# text containing a space, and a JSON-LD parser dropped the whole node.
|
||||
entity_text = entity.get("text") or entity.get("label", "unknown")
|
||||
entity_id = entity.get("id") or mint_entity_iri(entity_text)
|
||||
|
||||
jsonld = {
|
||||
"@id": entity_id,
|
||||
@@ -545,7 +550,9 @@ class JSONExporter:
|
||||
|
||||
return jsonld
|
||||
|
||||
def _relationship_to_jsonld(self, rel: Dict[str, Any]) -> Dict[str, Any]:
|
||||
def _relationship_to_jsonld(
|
||||
self, rel: Dict[str, Any], index: int = 0
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert relationship to JSON-LD format.
|
||||
|
||||
@@ -560,16 +567,18 @@ class JSONExporter:
|
||||
- type: Relationship type (optional)
|
||||
- confidence: Confidence score (optional)
|
||||
- metadata: Metadata dictionary (optional)
|
||||
index: Position of the relationship in the exported list, used when
|
||||
minting an IRI for a relationship that arrived without an id
|
||||
|
||||
Returns:
|
||||
Dictionary in JSON-LD format representing the relationship
|
||||
"""
|
||||
# Generate @id if not provided
|
||||
rel_id = rel.get("id")
|
||||
if not rel_id:
|
||||
source_id = rel.get("source_id") or rel.get("source", "")
|
||||
target_id = rel.get("target_id") or rel.get("target", "")
|
||||
rel_id = f"semantica:rel/{source_id}_{target_id}"
|
||||
# Generate @id if not provided, from the same mint the RDF serializers
|
||||
# use, including the list index that separates two relationships
|
||||
# sharing a pair of endpoints (#1101).
|
||||
source_id = rel.get("source_id") or rel.get("source", "")
|
||||
target_id = rel.get("target_id") or rel.get("target", "")
|
||||
rel_id = rel.get("id") or mint_relationship_iri(index, source_id, target_id)
|
||||
|
||||
jsonld = {
|
||||
"@id": rel_id,
|
||||
|
||||
@@ -600,11 +600,13 @@ class RDFSerializer:
|
||||
# Convert entities to JSON-LD
|
||||
entities = rdf_data.get("entities", [])
|
||||
for entity in entities:
|
||||
# Generate @id if not provided
|
||||
entity_id = entity.get("id")
|
||||
if not entity_id:
|
||||
entity_text = entity.get("text", "")
|
||||
entity_id = f"semantica:entity/{entity_text}"
|
||||
# Generate @id if not provided. Minted the same way the Turtle and
|
||||
# N-Triples paths mint it (#1101), so one knowledge graph carries
|
||||
# the same node identity whichever serializer wrote it. The former
|
||||
# f"semantica:entity/{text}" interpolated the raw text into an IRI:
|
||||
# any entity whose text contained a space produced an invalid IRI
|
||||
# and was dropped in full by a JSON-LD parser, silently.
|
||||
entity_id = entity.get("id") or mint_entity_iri(entity.get("text", ""))
|
||||
|
||||
jsonld["@graph"].append(
|
||||
{
|
||||
@@ -617,24 +619,22 @@ class RDFSerializer:
|
||||
|
||||
# Convert relationships to JSON-LD
|
||||
relationships = rdf_data.get("relationships", [])
|
||||
for rel in relationships:
|
||||
# Generate @id if not provided
|
||||
rel_id = rel.get("id")
|
||||
if not rel_id:
|
||||
source_id = rel.get("source_id", "")
|
||||
target_id = rel.get("target_id", "")
|
||||
rel_id = f"semantica:rel/{source_id}_{target_id}"
|
||||
for index, rel in enumerate(relationships):
|
||||
# Endpoints are resolved both ways, as serialize_to_turtle resolves
|
||||
# them: a relationship carrying source/target rather than
|
||||
# source_id/target_id used to hash into f"semantica:rel/_", so every
|
||||
# such relationship in an export collapsed onto one node and their
|
||||
# types and endpoints merged.
|
||||
source = rel.get("source_id") or rel.get("source", "")
|
||||
target = rel.get("target_id") or rel.get("target", "")
|
||||
rel_id = rel.get("id") or mint_relationship_iri(index, source, target)
|
||||
|
||||
jsonld["@graph"].append(
|
||||
{
|
||||
"@id": rel_id,
|
||||
"@type": "semantica:Relationship",
|
||||
"semantica:source": {
|
||||
"@id": rel.get("source_id") or rel.get("source")
|
||||
},
|
||||
"semantica:target": {
|
||||
"@id": rel.get("target_id") or rel.get("target")
|
||||
},
|
||||
"semantica:source": {"@id": source},
|
||||
"semantica:target": {"@id": target},
|
||||
"semantica:type": rel.get("type", "related_to"),
|
||||
}
|
||||
)
|
||||
|
||||
@@ -25,12 +25,11 @@ License: MIT
|
||||
|
||||
import html
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.helpers import ensure_directory
|
||||
from ..utils.helpers import ensure_directory, utc_now_iso
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
|
||||
@@ -252,7 +251,7 @@ class ReportGenerator:
|
||||
# Build report data with summary
|
||||
report_data = {
|
||||
"title": "Quality Assurance Report",
|
||||
"generated_at": datetime.now().isoformat(),
|
||||
"generated_at": utc_now_iso(),
|
||||
"metrics": quality_metrics,
|
||||
"summary": self._generate_quality_summary(quality_metrics),
|
||||
}
|
||||
@@ -278,7 +277,7 @@ class ReportGenerator:
|
||||
"""
|
||||
report_data = {
|
||||
"title": "Analysis Report",
|
||||
"generated_at": datetime.now().isoformat(),
|
||||
"generated_at": utc_now_iso(),
|
||||
"analysis": analysis_results,
|
||||
"summary": self._generate_analysis_summary(analysis_results),
|
||||
}
|
||||
@@ -304,7 +303,7 @@ class ReportGenerator:
|
||||
"""
|
||||
report_data = {
|
||||
"title": "Framework Metrics Report",
|
||||
"generated_at": datetime.now().isoformat(),
|
||||
"generated_at": utc_now_iso(),
|
||||
"metrics": metrics,
|
||||
"summary": self._generate_metrics_summary(metrics),
|
||||
}
|
||||
|
||||
@@ -22,7 +22,6 @@ License: MIT
|
||||
"""
|
||||
|
||||
from collections.abc import Mapping
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
@@ -33,6 +32,7 @@ from ..utils.helpers import (
|
||||
_require_recognized_keys,
|
||||
ensure_directory,
|
||||
normalize_graph_payload,
|
||||
utc_now_iso,
|
||||
)
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
@@ -216,7 +216,7 @@ class SemanticNetworkYAMLExporter:
|
||||
records = normalize_graph_payload(semantic_network)
|
||||
yaml_data = {
|
||||
"metadata": {
|
||||
"exported_at": datetime.now().isoformat(),
|
||||
"exported_at": utc_now_iso(),
|
||||
"version": "1.0",
|
||||
**semantic_network.get("metadata", {}),
|
||||
},
|
||||
@@ -309,7 +309,7 @@ class SemanticNetworkYAMLExporter:
|
||||
|
||||
if include_metadata:
|
||||
yaml_data["metadata"] = {
|
||||
"exported_at": datetime.now().isoformat(),
|
||||
"exported_at": utc_now_iso(),
|
||||
"entity_count": len(entities),
|
||||
}
|
||||
|
||||
@@ -333,7 +333,7 @@ class SemanticNetworkYAMLExporter:
|
||||
|
||||
if include_properties:
|
||||
yaml_data["metadata"] = {
|
||||
"exported_at": datetime.now().isoformat(),
|
||||
"exported_at": utc_now_iso(),
|
||||
"relationship_count": len(relationships),
|
||||
}
|
||||
|
||||
@@ -370,7 +370,7 @@ class SemanticNetworkYAMLExporter:
|
||||
}
|
||||
|
||||
yaml_data["metadata"] = {
|
||||
"exported_at": datetime.now().isoformat(),
|
||||
"exported_at": utc_now_iso(),
|
||||
"triplet_count": len(triplets),
|
||||
}
|
||||
|
||||
@@ -410,7 +410,7 @@ class SemanticNetworkYAMLExporter:
|
||||
yaml_data = {
|
||||
"pipeline_stage": pipeline_stage,
|
||||
"metadata": {
|
||||
"extracted_at": datetime.now().isoformat(),
|
||||
"extracted_at": utc_now_iso(),
|
||||
**extracted_data.get("metadata", {}),
|
||||
},
|
||||
"semantic_network": semantic_network,
|
||||
|
||||
@@ -118,10 +118,12 @@ sem:relationships a owl:ObjectProperty ;
|
||||
|
||||
sem:exportedAt a owl:DatatypeProperty ;
|
||||
rdfs:label "exported at" ;
|
||||
rdfs:comment """When the export was written. Emitted as an ISO 8601 local
|
||||
timestamp, so the range is xsd:dateTime rather than xsd:dateTimeStamp: the
|
||||
values carry no timezone offset.""" ;
|
||||
rdfs:range xsd:dateTime ;
|
||||
rdfs:comment """When the export was written, as an ISO 8601 timestamp with
|
||||
an explicit UTC offset. The range was xsd:dateTime while the exporters stamped
|
||||
with a naive datetime.now(); with the offset present (#1114) the value is a
|
||||
determinate instant, comparable against a timestamp written anywhere else, so
|
||||
the range is the stricter xsd:dateTimeStamp, which requires the offset.""" ;
|
||||
rdfs:range xsd:dateTimeStamp ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
sem:format a owl:DatatypeProperty ;
|
||||
|
||||
@@ -61,9 +61,10 @@ License: MIT
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
import uuid
|
||||
|
||||
from ..utils.helpers import utc_now_iso
|
||||
|
||||
|
||||
@dataclass
|
||||
class BridgeAxiom:
|
||||
@@ -280,7 +281,7 @@ class TranslationChain:
|
||||
"type": layer_type,
|
||||
"value": value,
|
||||
"source": source,
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
"timestamp": utc_now_iso(),
|
||||
**kwargs
|
||||
}
|
||||
self.layers.append(layer)
|
||||
|
||||
@@ -26,7 +26,7 @@ License: MIT
|
||||
|
||||
from typing import Optional, List, Dict, Any, Union
|
||||
from collections.abc import Mapping
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from contextlib import contextmanager
|
||||
import copy
|
||||
import inspect
|
||||
@@ -36,8 +36,13 @@ import threading
|
||||
from .schemas import ProvenanceEntry, SourceReference, AgentRecord, ActivityRecord
|
||||
from .storage import ProvenanceStorage, InMemoryStorage, SQLiteStorage
|
||||
from .integrity import compute_checksum, verify_checksum
|
||||
from ..utils.helpers import to_utc_datetime, utc_now_iso
|
||||
from ..utils.logging import get_logger
|
||||
|
||||
#: Sort key for an entry whose timestamp cannot be read as one, so an
|
||||
#: unreadable value orders first instead of raising during a sort.
|
||||
_EPOCH = datetime(1, 1, 1, tzinfo=timezone.utc)
|
||||
|
||||
# Issue #825, Part B Tier 3 — configurable base URI for export_prov(), shared
|
||||
# with RDFExporter's NamespaceManager "semantica" entry (semantica/export/
|
||||
# rdf_exporter.py) so KG-exported and PROV-exported URIs for the same
|
||||
@@ -363,8 +368,8 @@ class ProvenanceManager:
|
||||
source_quote=kwargs.get("source_quote"),
|
||||
confidence=kwargs.get("confidence", 1.0),
|
||||
metadata=metadata or {},
|
||||
first_seen=existing.first_seen if existing else datetime.utcnow().isoformat(),
|
||||
last_updated=datetime.utcnow().isoformat(),
|
||||
first_seen=existing.first_seen if existing else utc_now_iso(),
|
||||
last_updated=utc_now_iso(),
|
||||
parent_entity_id=parent_id,
|
||||
used_entities=list(kwargs.get("used_entities", [])),
|
||||
activity_started_at_time=activity_info["activity_started_at_time"],
|
||||
@@ -455,8 +460,8 @@ class ProvenanceManager:
|
||||
source_location=kwargs.get("source_location"),
|
||||
confidence=kwargs.get("confidence", 1.0),
|
||||
metadata=metadata or {},
|
||||
first_seen=datetime.utcnow().isoformat(),
|
||||
last_updated=datetime.utcnow().isoformat(),
|
||||
first_seen=utc_now_iso(),
|
||||
last_updated=utc_now_iso(),
|
||||
activity_started_at_time=activity_info["activity_started_at_time"],
|
||||
activity_ended_at_time=activity_info["activity_ended_at_time"],
|
||||
acted_on_behalf_of=kwargs.get("acted_on_behalf_of"),
|
||||
@@ -534,7 +539,7 @@ class ProvenanceManager:
|
||||
# split (issue #825, Part A item 4).
|
||||
derived_from_id=parent_chunk_id,
|
||||
metadata=metadata,
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
timestamp=utc_now_iso(),
|
||||
activity_started_at_time=activity_info["activity_started_at_time"],
|
||||
activity_ended_at_time=activity_info["activity_ended_at_time"],
|
||||
)
|
||||
@@ -604,7 +609,7 @@ class ProvenanceManager:
|
||||
**metadata,
|
||||
**source.metadata
|
||||
},
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
timestamp=utc_now_iso(),
|
||||
activity_started_at_time=activity_info["activity_started_at_time"],
|
||||
activity_ended_at_time=activity_info["activity_ended_at_time"],
|
||||
)
|
||||
@@ -959,11 +964,27 @@ class ProvenanceManager:
|
||||
Returns:
|
||||
List of matching entries as dicts, sorted by timestamp ascending.
|
||||
"""
|
||||
matches = [
|
||||
e for e in self.storage.retrieve_all()
|
||||
if e.timestamp and start <= e.timestamp <= end
|
||||
]
|
||||
matches.sort(key=lambda e: e.timestamp)
|
||||
# Compare instants, not spellings. Since #1114 new entries carry a
|
||||
# +00:00 offset while entries written earlier do not, and a raw string
|
||||
# comparison orders those two by length: an inclusive naive bound equal
|
||||
# to a stored offset-bearing timestamp would sort below it and drop the
|
||||
# record. A bound in another offset was mis-ordered the same way.
|
||||
start_at = to_utc_datetime(start)
|
||||
end_at = to_utc_datetime(end)
|
||||
entries = [e for e in self.storage.retrieve_all() if e.timestamp]
|
||||
|
||||
if start_at is None or end_at is None:
|
||||
# A bound this module cannot read as a timestamp keeps the historical
|
||||
# string comparison rather than raising on a call that used to work.
|
||||
matches = [e for e in entries if start <= e.timestamp <= end]
|
||||
else:
|
||||
matches = [
|
||||
e for e in entries
|
||||
if (at := to_utc_datetime(e.timestamp)) is not None
|
||||
and start_at <= at <= end_at
|
||||
]
|
||||
|
||||
matches.sort(key=lambda e: (to_utc_datetime(e.timestamp) or _EPOCH, e.timestamp))
|
||||
return [e.to_dict() for e in matches]
|
||||
|
||||
def get_all_sources(self, entity_id: str) -> List[Dict[str, Any]]:
|
||||
@@ -1071,7 +1092,7 @@ class ProvenanceManager:
|
||||
|
||||
entry = copy.deepcopy(existing)
|
||||
entry.invalidated = True
|
||||
entry.invalidated_at_time = datetime.utcnow().isoformat()
|
||||
entry.invalidated_at_time = utc_now_iso()
|
||||
entry.invalidated_by = agent_id
|
||||
entry.invalidation_reason = reason
|
||||
entry.previous_version_id = history_id
|
||||
@@ -1163,8 +1184,22 @@ class ProvenanceManager:
|
||||
"""
|
||||
entries = self.storage.retrieve_all()
|
||||
if since:
|
||||
entries = [e for e in entries if getattr(e, "timestamp", "") >= since]
|
||||
entries.sort(key=lambda e: getattr(e, "timestamp", ""))
|
||||
since_at = to_utc_datetime(since)
|
||||
if since_at is None:
|
||||
entries = [e for e in entries
|
||||
if getattr(e, "timestamp", "") >= since]
|
||||
else:
|
||||
entries = [
|
||||
e for e in entries
|
||||
if (at := to_utc_datetime(getattr(e, "timestamp", None)))
|
||||
is not None and at >= since_at
|
||||
]
|
||||
entries.sort(
|
||||
key=lambda e: (
|
||||
to_utc_datetime(getattr(e, "timestamp", None)) or _EPOCH,
|
||||
getattr(e, "timestamp", ""),
|
||||
)
|
||||
)
|
||||
|
||||
if format == "json":
|
||||
return [
|
||||
|
||||
@@ -30,6 +30,8 @@ from dataclasses import dataclass, field
|
||||
from typing import Optional, List, Dict, Any
|
||||
from datetime import datetime
|
||||
|
||||
from ..utils.helpers import utc_now_iso
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProvenanceEntry:
|
||||
@@ -91,7 +93,7 @@ class ProvenanceEntry:
|
||||
source_quote: Optional[str] = None
|
||||
|
||||
# Temporal tracking (from kg.ProvenanceTracker)
|
||||
timestamp: str = field(default_factory=lambda: datetime.utcnow().isoformat())
|
||||
timestamp: str = field(default_factory=lambda: utc_now_iso())
|
||||
first_seen: Optional[str] = None
|
||||
last_updated: Optional[str] = None
|
||||
|
||||
|
||||
@@ -82,6 +82,9 @@ from .helpers import (
|
||||
normalize_entities,
|
||||
normalize_graph_payload,
|
||||
parse_timestamp,
|
||||
to_utc_datetime,
|
||||
utc_now,
|
||||
utc_now_iso,
|
||||
read_json_file,
|
||||
retry_on_error,
|
||||
safe_filename,
|
||||
@@ -193,6 +196,9 @@ __all__ = [
|
||||
"get_file_size",
|
||||
"format_timestamp",
|
||||
"parse_timestamp",
|
||||
"to_utc_datetime",
|
||||
"utc_now",
|
||||
"utc_now_iso",
|
||||
"merge_dicts",
|
||||
"chunk_list",
|
||||
"flatten_dict",
|
||||
|
||||
@@ -320,6 +320,65 @@ def format_timestamp(
|
||||
return dt.strftime(format_str)
|
||||
|
||||
|
||||
def utc_now() -> datetime:
|
||||
"""
|
||||
Current instant as a timezone-aware UTC datetime.
|
||||
|
||||
``datetime.now()`` reads the local clock and ``datetime.utcnow()`` reads UTC,
|
||||
but both return a naive datetime, and the two are indistinguishable once
|
||||
serialized: a consumer cannot tell which zone the value belongs to, and an
|
||||
RDF timestamp without an offset is not comparable against one that has an
|
||||
offset (a SPARQL FILTER drops it rather than reporting an error). Use this
|
||||
for any timestamp that leaves the process.
|
||||
|
||||
Returns:
|
||||
Current UTC time, timezone-aware
|
||||
"""
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def utc_now_iso() -> str:
|
||||
"""
|
||||
Current instant as an ISO 8601 string carrying an explicit UTC offset.
|
||||
|
||||
Returns:
|
||||
Timestamp string such as ``2026-08-19T14:19:04.229937+00:00``, which is
|
||||
a valid ``xsd:dateTimeStamp`` and orders correctly against timestamps
|
||||
written in any other timezone
|
||||
"""
|
||||
return utc_now().isoformat()
|
||||
|
||||
|
||||
def to_utc_datetime(value: Union[str, datetime, None]) -> Optional[datetime]:
|
||||
"""
|
||||
Read an ISO 8601 timestamp as a timezone-aware UTC instant.
|
||||
|
||||
Timestamps written before #1114 carry no offset. They were produced by
|
||||
``datetime.utcnow()``, so a missing offset is read as UTC: that keeps a
|
||||
stored naive value and the same instant written with an offset comparing
|
||||
equal, instead of ordering by how the timestamp happens to be spelled.
|
||||
|
||||
Args:
|
||||
value: ISO 8601 string or datetime. ``Z`` is accepted as the offset.
|
||||
|
||||
Returns:
|
||||
Timezone-aware UTC datetime, or None if the value cannot be read as a
|
||||
timestamp, so callers can fall back rather than raise on stored data
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, datetime):
|
||||
parsed = value
|
||||
else:
|
||||
try:
|
||||
parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return parsed.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def parse_timestamp(timestamp_str: str, format_str: Optional[str] = None) -> datetime:
|
||||
"""
|
||||
Parse timestamp string to datetime.
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""The JSON-LD paths must mint the same IRIs as the RDF paths (issue #1101).
|
||||
|
||||
#1101 was fixed for the Turtle, N-Triples and RDF/XML serializers: an entity
|
||||
arriving without an id gets a deterministic IRI in the declared namespace. The
|
||||
JSON-LD paths were left interpolating the entity's own text into
|
||||
``f"semantica:entity/{text}"`` and the relationship's endpoints into
|
||||
``f"semantica:rel/{source}_{target}"``, which fails three ways:
|
||||
|
||||
* a text containing a space produces an invalid IRI, and a JSON-LD parser drops
|
||||
the whole node rather than complaining, so the entity vanishes from the export;
|
||||
* relationships carrying ``source``/``target`` rather than ``source_id``/
|
||||
``target_id`` all minted ``semantica:rel/_``, so every one of them collapsed
|
||||
onto a single node whose types and endpoints merged;
|
||||
* the JSON-LD @id and the Turtle IRI for one entity disagreed, so the two
|
||||
serializations of one knowledge graph were two different graphs.
|
||||
|
||||
``JSONExporter.export_entities`` and ``export_relationships`` also wrote
|
||||
``semantica:entities`` into a context that never declared the ``semantica``
|
||||
prefix, which a JSON-LD processor reads as an IRI in the scheme ``semantica`` —
|
||||
the original #1101 failure mode, on a path the first fix did not cover.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from semantica.export.json_exporter import JSONExporter
|
||||
from semantica.export.rdf_exporter import (
|
||||
RDFExporter,
|
||||
SEMANTICA_NS,
|
||||
mint_entity_iri,
|
||||
mint_relationship_iri,
|
||||
)
|
||||
|
||||
KG = {
|
||||
"entities": [
|
||||
{"text": "Acme Corp", "type": "https://example.org/Org"},
|
||||
{"id": "https://example.org/e2", "text": "Bob"},
|
||||
],
|
||||
"relationships": [
|
||||
{"source": "https://example.org/a", "target": "https://example.org/b",
|
||||
"type": "https://example.org/employs"},
|
||||
{"source": "https://example.org/b", "target": "https://example.org/a",
|
||||
"type": "https://example.org/works_for"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _graph(document: str):
|
||||
"""Parse a JSON-LD document the way a consumer would."""
|
||||
rdflib = pytest.importorskip("rdflib")
|
||||
graph = rdflib.Graph()
|
||||
graph.parse(data=document, format="json-ld")
|
||||
return graph
|
||||
|
||||
|
||||
def test_jsonld_entity_id_is_the_minted_iri_not_the_interpolated_text():
|
||||
graph = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))["@graph"]
|
||||
|
||||
assert graph[0]["@id"] == mint_entity_iri("Acme Corp")
|
||||
assert graph[0]["@id"].startswith(SEMANTICA_NS)
|
||||
assert "semantica:entity/" not in json.dumps(graph)
|
||||
|
||||
|
||||
def test_json_exporter_mints_the_same_entity_iri_as_the_rdf_exporter():
|
||||
"""One knowledge graph, two exporters, one node identity."""
|
||||
from_rdf = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))["@graph"]
|
||||
from_json = JSONExporter()._convert_kg_to_jsonld(KG)
|
||||
|
||||
assert from_json["semantica:entities"][0]["@id"] == from_rdf[0]["@id"]
|
||||
assert from_json["semantica:relationships"][0]["@id"] == from_rdf[2]["@id"]
|
||||
|
||||
|
||||
def test_minted_jsonld_iri_agrees_with_the_turtle_serialization():
|
||||
"""The two serializations of one graph must name the same entity alike."""
|
||||
turtle = RDFExporter().export_to_rdf(KG, format="turtle")
|
||||
jsonld = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))
|
||||
|
||||
minted = mint_entity_iri("Acme Corp")
|
||||
assert f"<{minted}>" in turtle
|
||||
assert jsonld["@graph"][0]["@id"] == minted
|
||||
|
||||
|
||||
def test_entity_whose_text_contains_a_space_survives_a_jsonld_parse():
|
||||
"""The regression that lost data: an invalid IRI is dropped, not reported."""
|
||||
kg = {"entities": [{"text": "Acme Corp"}], "relationships": []}
|
||||
graph = _graph(json.dumps(JSONExporter()._convert_kg_to_jsonld(kg)))
|
||||
|
||||
subjects = {str(s) for s in graph.subjects()}
|
||||
assert mint_entity_iri("Acme Corp") in subjects
|
||||
|
||||
|
||||
def test_relationships_carrying_source_and_target_do_not_collide():
|
||||
"""Two relationships, two nodes: ``semantica:rel/_`` merged them into one."""
|
||||
jsonld = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))
|
||||
relationships = [n for n in jsonld["@graph"]
|
||||
if n["@type"] == "semantica:Relationship"]
|
||||
|
||||
ids = {node["@id"] for node in relationships}
|
||||
assert len(ids) == len(relationships) == 2
|
||||
assert ids == {
|
||||
mint_relationship_iri(0, "https://example.org/a", "https://example.org/b"),
|
||||
mint_relationship_iri(1, "https://example.org/b", "https://example.org/a"),
|
||||
}
|
||||
|
||||
graph = _graph(json.dumps(jsonld))
|
||||
assert len({str(s) for s in graph.subjects()} & ids) == 2
|
||||
|
||||
|
||||
def test_no_export_path_writes_an_iri_in_the_semantica_scheme(tmp_path):
|
||||
"""Nothing may expand to the scheme ``semantica`` rather than the namespace."""
|
||||
documents = [
|
||||
RDFExporter().export_to_rdf(KG, format="jsonld"),
|
||||
json.dumps(JSONExporter()._convert_kg_to_jsonld(KG)),
|
||||
]
|
||||
exporter = JSONExporter()
|
||||
exporter.export_entities(KG["entities"], tmp_path / "entities.json")
|
||||
exporter.export_relationships(KG["relationships"], tmp_path / "relationships.json")
|
||||
documents.append((tmp_path / "entities.json").read_text())
|
||||
documents.append((tmp_path / "relationships.json").read_text())
|
||||
|
||||
for document in documents:
|
||||
for term in _graph(document).all_nodes():
|
||||
assert not str(term).startswith("semantica:"), document
|
||||
for predicate in _graph(document).predicates():
|
||||
assert not str(predicate).startswith("semantica:"), document
|
||||
|
||||
|
||||
def test_entity_and_relationship_lists_expand_into_the_declared_namespace(tmp_path):
|
||||
exporter = JSONExporter()
|
||||
exporter.export_entities(KG["entities"], tmp_path / "entities.json")
|
||||
exporter.export_relationships(KG["relationships"], tmp_path / "relationships.json")
|
||||
|
||||
predicates = set()
|
||||
for name in ("entities.json", "relationships.json"):
|
||||
predicates |= {str(p) for p in _graph((tmp_path / name).read_text()).predicates()}
|
||||
|
||||
assert f"{SEMANTICA_NS}entities" in predicates
|
||||
assert f"{SEMANTICA_NS}relationships" in predicates
|
||||
@@ -0,0 +1,141 @@
|
||||
"""Timestamps that leave the process must carry a timezone (issue #1114).
|
||||
|
||||
Every timestamp an exporter wrote was naive: ``datetime.now().isoformat()``
|
||||
reads the local clock, ``datetime.utcnow().isoformat()`` reads UTC, and the two
|
||||
serialize identically, so nothing downstream can tell which zone a value belongs
|
||||
to. In RDF the consequence is not a parse error but a silent one: under XSD 1.1
|
||||
a value with no timezone compared against one with a timezone is indeterminate
|
||||
whenever they fall inside the +/-14 hour window, SPARQL turns that into an error,
|
||||
and FILTER discards errors as non-matches. A timezone-qualified query therefore
|
||||
returns an answer with every Semantica-written record quietly missing from it.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from semantica.export.json_exporter import JSONExporter
|
||||
from semantica.export.report_generator import ReportGenerator
|
||||
from semantica.export.yaml_exporter import SemanticNetworkYAMLExporter
|
||||
from semantica.utils.helpers import utc_now, utc_now_iso
|
||||
|
||||
KG = {
|
||||
"entities": [{"id": "https://example.org/e1", "text": "Bob"}],
|
||||
"relationships": [],
|
||||
}
|
||||
|
||||
|
||||
def assert_offset_aware(value):
|
||||
"""An ISO 8601 string is only an instant if it says which zone it is in."""
|
||||
assert isinstance(value, str), value
|
||||
parsed = datetime.fromisoformat(value)
|
||||
assert parsed.tzinfo is not None, f"timezone-naive timestamp: {value!r}"
|
||||
assert parsed.utcoffset() is not None
|
||||
|
||||
|
||||
def test_utc_now_iso_is_offset_aware():
|
||||
assert_offset_aware(utc_now_iso())
|
||||
assert utc_now().tzinfo is not None
|
||||
|
||||
|
||||
def test_jsonld_export_timestamp_is_offset_aware():
|
||||
document = JSONExporter()._convert_kg_to_jsonld(KG)
|
||||
assert_offset_aware(document["semantica:exportedAt"])
|
||||
|
||||
|
||||
def test_json_export_metadata_timestamp_is_offset_aware(tmp_path):
|
||||
import json
|
||||
|
||||
exporter = JSONExporter()
|
||||
exporter.export_entities(KG["entities"], tmp_path / "entities.json")
|
||||
exporter.export_relationships([], tmp_path / "relationships.json")
|
||||
|
||||
for name in ("entities.json", "relationships.json"):
|
||||
payload = json.loads((tmp_path / name).read_text())
|
||||
assert_offset_aware(payload["metadata"]["exported_at"])
|
||||
|
||||
|
||||
def test_yaml_export_timestamp_is_offset_aware():
|
||||
yaml = pytest.importorskip("yaml")
|
||||
|
||||
document = SemanticNetworkYAMLExporter().export_entities(KG["entities"])
|
||||
payload = yaml.safe_load(document)
|
||||
assert_offset_aware(payload["metadata"]["exported_at"])
|
||||
|
||||
|
||||
def test_report_timestamp_is_offset_aware():
|
||||
import json
|
||||
|
||||
report = json.loads(
|
||||
ReportGenerator().generate_quality_report({"score": 0.9}, format="json")
|
||||
)
|
||||
assert_offset_aware(report["generated_at"])
|
||||
|
||||
|
||||
def test_exported_timestamp_compares_against_a_timezone_aware_instant():
|
||||
"""The naive form raised TypeError here, or compared as if it were UTC."""
|
||||
exported = datetime.fromisoformat(
|
||||
JSONExporter()._convert_kg_to_jsonld(KG)["semantica:exportedAt"]
|
||||
)
|
||||
assert exported <= utc_now()
|
||||
assert exported > datetime(2020, 1, 1, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def test_exported_timestamp_survives_a_timezone_qualified_sparql_filter():
|
||||
"""The regression in #1114: a strict engine dropped the naive value."""
|
||||
pyoxigraph = pytest.importorskip("pyoxigraph")
|
||||
|
||||
exported = JSONExporter()._convert_kg_to_jsonld(KG)["semantica:exportedAt"]
|
||||
store = pyoxigraph.Store()
|
||||
store.load(
|
||||
(
|
||||
'<https://example.org/export> '
|
||||
'<https://semantica.dev/ns#exportedAt> '
|
||||
f'"{exported}"^^<http://www.w3.org/2001/XMLSchema#dateTime> .'
|
||||
).encode(),
|
||||
format=pyoxigraph.RdfFormat.N_TRIPLES,
|
||||
)
|
||||
# The bound has to sit inside the +/-14 hour window that makes an
|
||||
# untimezoned comparison indeterminate. A bound years away is determinate
|
||||
# even for a naive value, and the test would pass without the fix.
|
||||
bound = (utc_now() + timedelta(hours=1)).isoformat().replace("+00:00", "Z")
|
||||
rows = list(store.query(
|
||||
"PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> "
|
||||
"SELECT ?e WHERE { ?e <https://semantica.dev/ns#exportedAt> ?t . "
|
||||
f'FILTER (?t < "{bound}"^^xsd:dateTime) }}'
|
||||
))
|
||||
assert len(rows) == 1, "the export was dropped by a timezone-qualified filter"
|
||||
|
||||
|
||||
def test_document_iri_carrying_an_offset_is_a_valid_iri():
|
||||
"""The offset puts '+' and ':' in the @id; both are legal in a path."""
|
||||
rdflib = pytest.importorskip("rdflib")
|
||||
|
||||
document_iri = JSONExporter()._convert_kg_to_jsonld(KG)["@id"]
|
||||
assert "+00:00" in document_iri
|
||||
assert rdflib.term._is_valid_uri(document_iri)
|
||||
|
||||
graph = rdflib.Graph()
|
||||
graph.add((
|
||||
rdflib.URIRef(document_iri),
|
||||
rdflib.RDF.type,
|
||||
rdflib.URIRef("https://semantica.dev/ns#KnowledgeGraph"),
|
||||
))
|
||||
reparsed = rdflib.Graph().parse(data=graph.serialize(format="nt"), format="nt")
|
||||
assert document_iri in {str(s) for s in reparsed.subjects()}
|
||||
|
||||
|
||||
def test_vocabulary_range_matches_what_the_exporter_writes():
|
||||
"""The declared range says the offset is required; the export must carry it."""
|
||||
rdflib = pytest.importorskip("rdflib")
|
||||
from rdflib.namespace import RDFS, XSD
|
||||
|
||||
from semantica.ontology.vocabulary import NAMESPACE, vocabulary_turtle
|
||||
|
||||
graph = rdflib.Graph()
|
||||
graph.parse(data=vocabulary_turtle(), format="turtle")
|
||||
declared = graph.value(rdflib.URIRef(f"{NAMESPACE}exportedAt"), RDFS.range)
|
||||
assert declared == XSD.dateTimeStamp
|
||||
|
||||
exported = JSONExporter()._convert_kg_to_jsonld(KG)["semantica:exportedAt"]
|
||||
assert datetime.fromisoformat(exported).utcoffset() is not None
|
||||
@@ -0,0 +1,153 @@
|
||||
"""Provenance timestamps must carry a timezone (issue #1114).
|
||||
|
||||
The provenance package stamped every record with ``datetime.utcnow()``, which
|
||||
returns a naive datetime that happens to hold UTC. The exporters stamped theirs
|
||||
with ``datetime.now()``, which returns a naive datetime holding local time. Both
|
||||
serialize identically, so a graph mixing the two cannot be ordered, and the
|
||||
values reach RDF as ``prov:generatedAtTime``/``startedAtTime``/``endedAtTime``
|
||||
typed ``xsd:dateTime``, where a timezone-qualified SPARQL comparison discards
|
||||
them. ``datetime.utcnow()`` is also deprecated as of Python 3.12.
|
||||
"""
|
||||
|
||||
import warnings
|
||||
from datetime import datetime
|
||||
|
||||
import pytest
|
||||
|
||||
from semantica.provenance.manager import ProvenanceManager
|
||||
from semantica.provenance.schemas import ProvenanceEntry
|
||||
from semantica.utils.helpers import utc_now
|
||||
|
||||
|
||||
def assert_offset_aware(value):
|
||||
parsed = datetime.fromisoformat(value)
|
||||
assert parsed.tzinfo is not None, f"timezone-naive timestamp: {value!r}"
|
||||
|
||||
|
||||
def test_provenance_entry_default_timestamp_is_offset_aware():
|
||||
entry = ProvenanceEntry(entity_id="e1", entity_type="Doc", activity_id="act1")
|
||||
assert_offset_aware(entry.timestamp)
|
||||
assert datetime.fromisoformat(entry.timestamp) <= utc_now()
|
||||
|
||||
|
||||
def test_creating_an_entry_raises_no_deprecation_warning():
|
||||
"""datetime.utcnow() is deprecated and scheduled for removal."""
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error", DeprecationWarning)
|
||||
ProvenanceEntry(entity_id="e1", entity_type="Doc", activity_id="act1")
|
||||
|
||||
|
||||
def test_tracked_entity_timestamps_are_offset_aware():
|
||||
manager = ProvenanceManager()
|
||||
manager.track_entity("e1", source="doc.pdf")
|
||||
|
||||
entry = manager.storage.retrieve_all()[0]
|
||||
assert_offset_aware(entry.timestamp)
|
||||
for field in ("first_seen", "last_updated"):
|
||||
value = getattr(entry, field, None)
|
||||
if value:
|
||||
assert_offset_aware(value)
|
||||
|
||||
|
||||
def test_prov_o_export_timestamps_are_offset_aware():
|
||||
"""The values land in RDF typed xsd:dateTime, so the offset is the contract."""
|
||||
rdflib = pytest.importorskip("rdflib")
|
||||
from rdflib.namespace import XSD
|
||||
|
||||
manager = ProvenanceManager()
|
||||
manager.track_entity("e_parent", source="doc.pdf")
|
||||
manager.track_entity(
|
||||
"e_child", source="doc.pdf", parent_entity_id="e_parent",
|
||||
used_entities=["e_parent"], activity_id="act_transform",
|
||||
)
|
||||
|
||||
graph = rdflib.Graph()
|
||||
graph.parse(data=manager.export_prov(format="turtle"), format="turtle")
|
||||
|
||||
stamps = [o for o in graph.objects()
|
||||
if isinstance(o, rdflib.Literal) and o.datatype == XSD.dateTime]
|
||||
assert stamps, "no xsd:dateTime literals in the PROV-O export"
|
||||
for stamp in stamps:
|
||||
assert_offset_aware(str(stamp))
|
||||
|
||||
|
||||
def test_prov_o_timestamps_are_valid_datetimestamp():
|
||||
"""xsd:dateTimeStamp requires an explicit timezone; these now qualify."""
|
||||
rdflib = pytest.importorskip("rdflib")
|
||||
from rdflib.namespace import XSD
|
||||
|
||||
manager = ProvenanceManager()
|
||||
manager.track_entity("e1", source="doc.pdf")
|
||||
graph = rdflib.Graph()
|
||||
graph.parse(data=manager.export_prov(format="turtle"), format="turtle")
|
||||
|
||||
for stamp in [o for o in graph.objects()
|
||||
if isinstance(o, rdflib.Literal) and o.datatype == XSD.dateTime]:
|
||||
assert rdflib.Literal(str(stamp), datatype=XSD.dateTime).ill_typed is False
|
||||
assert datetime.fromisoformat(str(stamp)).utcoffset() is not None
|
||||
|
||||
|
||||
class TestRangeQueriesCompareInstants:
|
||||
"""Range APIs compared ISO strings, so they ordered by spelling (#1121 review).
|
||||
|
||||
Once new entries carry ``+00:00`` and stored ones do not, a raw string
|
||||
comparison puts an inclusive naive bound *below* the offset-bearing
|
||||
timestamp it names, dropping the record, and a bound written in another
|
||||
offset lands wherever its digits fall rather than at its instant.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _manager_with(timestamps):
|
||||
manager = ProvenanceManager()
|
||||
for index, stamp in enumerate(timestamps):
|
||||
manager.storage.store(ProvenanceEntry(
|
||||
entity_id=f"e{index}", entity_type="Doc",
|
||||
activity_id="act", timestamp=stamp,
|
||||
))
|
||||
return manager
|
||||
|
||||
def test_inclusive_bound_written_without_an_offset_still_matches(self):
|
||||
manager = self._manager_with(["2026-08-19T14:19:04.229937+00:00"])
|
||||
|
||||
found = manager.query_recorded_between(
|
||||
"2026-08-19T00:00:00", "2026-08-19T14:19:04.229937"
|
||||
)
|
||||
assert [e["entity_id"] for e in found] == ["e0"]
|
||||
|
||||
def test_bound_in_another_offset_selects_by_instant(self):
|
||||
"""19:45+05:30 is 14:15Z: before the entry, though its digits are after."""
|
||||
manager = self._manager_with(["2026-08-19T14:19:04+00:00"])
|
||||
|
||||
assert manager.query_recorded_between(
|
||||
"2026-08-19T00:00:00Z", "2026-08-19T19:45:00+05:30"
|
||||
) == []
|
||||
assert len(manager.query_recorded_between(
|
||||
"2026-08-19T00:00:00Z", "2026-08-19T19:50:00+05:30"
|
||||
)) == 1
|
||||
|
||||
def test_legacy_and_offset_bearing_entries_are_both_found_and_ordered(self):
|
||||
manager = self._manager_with([
|
||||
"2026-08-19T14:19:05+00:00", # written after #1114
|
||||
"2026-08-19T14:19:04", # written before it, meaning UTC
|
||||
])
|
||||
|
||||
found = manager.query_recorded_between(
|
||||
"2026-08-19T14:00:00Z", "2026-08-19T15:00:00Z"
|
||||
)
|
||||
assert [e["entity_id"] for e in found] == ["e1", "e0"]
|
||||
|
||||
def test_audit_log_since_reads_a_naive_bound_as_utc(self):
|
||||
manager = self._manager_with([
|
||||
"2026-08-19T14:19:05+00:00",
|
||||
"2026-08-19T09:00:00",
|
||||
])
|
||||
|
||||
recent = manager.audit_log(since="2026-08-19T14:19:05", format="json")
|
||||
assert [e["entity_id"] for e in recent] == ["e0"]
|
||||
|
||||
def test_an_unreadable_bound_falls_back_to_the_previous_behaviour(self):
|
||||
"""A call that used to work with a non-timestamp bound must not raise."""
|
||||
manager = self._manager_with(["2026-08-19T14:19:04+00:00"])
|
||||
|
||||
assert manager.query_recorded_between("not-a-date", "also-not") == []
|
||||
assert manager.audit_log(since="not-a-date", format="json") == []
|
||||
Reference in New Issue
Block a user