From 75b026c6ddb3dc8da98a17db0a86f1723625eafd Mon Sep 17 00:00:00 2001 From: FABIOTESS Date: Wed, 19 Aug 2026 15:09:41 +0100 Subject: [PATCH] fix(export): mint JSON-LD @ids the same way the RDF serializers do (#1101) The #1101 fix covered serialize_to_turtle, serialize_to_ntriples and serialize_to_rdfxml. Both JSON-LD writers were left interpolating the entity's own text into f"semantica:entity/{text}" and the endpoints into f"semantica:rel/{source}_{target}". Three consequences, all reproducible on 0.6.5 through the public API: - An entity whose text contains a space, which is most organisation and person names an extractor produces, mints an invalid IRI. A JSON-LD parser drops that node in full and says nothing, so the entity is simply missing from the export: rdflib reads 6 triples for {"text": "AcmeCorp"} and 2 for {"text": "Acme Corp"}. - serialize_to_jsonld resolved endpoints from source_id/target_id only, while the rest of the module accepts source/target too. Every relationship carrying the second form minted the identical "semantica:rel/_", so all of them collapsed onto one node and their types and endpoints merged into a graph nobody wrote. - The JSON-LD @id and the Turtle IRI for one entity disagreed (ns#entity/Acme Corp vs ns#entity_a73cb4563ee2e72c), so the two serializations of one knowledge graph were two different graphs. Both writers now use mint_entity_iri/mint_relationship_iri, resolving endpoints both ways and passing the list index the RDF paths pass, so one knowledge graph carries one node identity whichever serializer wrote it. JSONExporter.export_entities and export_relationships also declare the semantica prefix their @context was already writing "semantica:entities" against. Without the declaration a processor reads that as an IRI in the scheme semantica rather than the namespace expansion, which is the original #1101 defect on a third path: rdflib returns the predicate literally as semantica:entities. tests/export/test_jsonld_iri_minting.py parses each export with a real JSON-LD processor rather than asserting on the JSON text, and covers all seven claims above. Each test fails on the parent commit. 236 export and ontology tests pass. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 4 +- semantica/export/json_exporter.py | 36 +++--- semantica/export/rdf_exporter.py | 36 +++--- tests/export/test_jsonld_iri_minting.py | 139 ++++++++++++++++++++++++ 4 files changed, 183 insertions(+), 32 deletions(-) create mode 100644 tests/export/test_jsonld_iri_minting.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c81bc4a..5cb0fdd7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,7 +18,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The missing-id fallback minted entity/relationship IRIs from Python's builtin `hash()`, randomised per process (`PYTHONHASHSEED`), so the same entity got a different IRI on every run and exports couldn't be diffed, deduplicated, or joined to an earlier provenance record. It also wrote ``, an IRI in the scheme `semantica` rather than the expansion of the declared prefix, so those nodes never joined with anything written through it. Minting now uses SHA-256 and writes a full IRI in the declared namespace; the same fix applies to the default entity/relationship types in the Turtle path - **Fixed during review** (Qodo): the temporal fallback minted from `source_id` only, while the main serializer accepts `source_id` or `source` — relationships using the second form hashed two empty strings, which the previous randomised `hash()` masked by making the IRI unstable anyway; once deterministic, unrelated relationships at the same list index collided on one IRI across exports. Endpoints are now resolved the same way `serialize_to_turtle` resolves them, before minting. `sem:confidence` also lost its declared `xsd:decimal` range: the N-Triples serializer types the same value `xsd:float`, and the two are disjoint, so declaring either contradicted one of the exporters (tracked in #1100) — a new `test_declared_ranges_do_not_contradict_what_the_exporters_emit` guards the whole class of that mistake - **Fixed in follow-up**: `serialize_to_rdfxml`'s default entity type still wrote the bare string `"semantica:Entity"` into an `rdf:resource` attribute, which (unlike a Turtle angle-bracket or an XML element name) is not namespace-expanded — the exact #1101 failure mode, just on the untested RDF/XML path. `json_exporter.py`'s `semantica:format` and `@type: "semantica:KnowledgeGraph"` were emitted but absent from both the vocabulary and the test's `EMITTED_TERMS` guard set, so the "undeclared terms fail the build" claim didn't actually cover them — both are now declared and guarded. `MANIFEST.in` didn't mirror the `pyproject.toml` package-data addition, so a source-distribution install could omit the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only `PATH`, breaking it on Windows; now overrides only `PYTHONHASHSEED` on top of the inherited environment - - 229 export and ontology tests pass + - **Also fixed, on the JSON-LD paths**: the first fix covered the Turtle, N-Triples and RDF/XML serializers, and left both JSON-LD writers interpolating the entity's own text into `f"semantica:entity/{text}"` and the endpoints into `f"semantica:rel/{source}_{target}"`. Three consequences, all live in 0.6.5: an entity whose text contained a space produced an invalid IRI, and a JSON-LD parser dropped that node in full rather than reporting it, so the entity disappeared from the export; every relationship carrying `source`/`target` rather than `source_id`/`target_id` minted the identical `semantica:rel/_`, collapsing all of them onto one node whose types and endpoints merged; and the JSON-LD `@id` disagreed with the Turtle IRI for the same entity, so the two serializations of one knowledge graph were two different graphs. Both JSON-LD writers now use `mint_entity_iri`/`mint_relationship_iri`, and `JSONExporter.export_entities`/`export_relationships` declare the `semantica` prefix their `@context` was already writing `semantica:entities` against — without it a processor reads that as an IRI in the scheme `semantica`, which is the original #1101 defect on a third path + - `tests/export/test_jsonld_iri_minting.py` parses each export with a real JSON-LD processor and asserts the entity survives, the relationships stay distinct, no term expands into the `semantica` scheme, and the JSON-LD `@id` equals the Turtle IRI + - 236 export and ontology tests pass - **First-class CrewAI integration** (#962) - New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`) diff --git a/semantica/export/json_exporter.py b/semantica/export/json_exporter.py index 2bca8cbf..aa1641ae 100644 --- a/semantica/export/json_exporter.py +++ b/semantica/export/json_exporter.py @@ -32,6 +32,7 @@ from ..utils.exceptions import ProcessingError, ValidationError from ..utils.helpers import ensure_directory, write_json_file from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker +from .rdf_exporter import SEMANTICA_NS, mint_entity_iri, mint_relationship_iri class JSONExporter: @@ -265,6 +266,7 @@ class JSONExporter: json_data = { "@context": { "@vocab": "https://semantica.dev/vocab/", + "semantica": SEMANTICA_NS, "entities": {"@id": "semantica:entities", "@container": "@list"}, }, "entities": entities, @@ -294,6 +296,7 @@ class JSONExporter: json_data = { "@context": { "@vocab": "https://semantica.dev/vocab/", + "semantica": SEMANTICA_NS, "relationships": { "@id": "semantica:relationships", "@container": "@list", @@ -495,7 +498,8 @@ class JSONExporter: relationships = kg.get("relationships", []) if relationships: jsonld["semantica:relationships"] = [ - self._relationship_to_jsonld(r) for r in relationships + self._relationship_to_jsonld(r, index) + for index, r in enumerate(relationships) ] self.logger.debug( f"Converted {len(relationships)} relationship(s) to JSON-LD" @@ -526,11 +530,13 @@ class JSONExporter: Returns: Dictionary in JSON-LD format representing the entity """ - # Generate @id if not provided - entity_id = entity.get("id") - if not entity_id: - entity_text = entity.get("text") or entity.get("label", "unknown") - entity_id = f"semantica:entity/{entity_text}" + # Generate @id if not provided. Minted exactly as the RDF serializers + # mint it (#1101), so the JSON-LD and Turtle exports of one knowledge + # graph name the same entity with the same IRI. Interpolating the raw + # text into f"semantica:entity/{text}" produced an invalid IRI for any + # text containing a space, and a JSON-LD parser dropped the whole node. + entity_text = entity.get("text") or entity.get("label", "unknown") + entity_id = entity.get("id") or mint_entity_iri(entity_text) jsonld = { "@id": entity_id, @@ -545,7 +551,9 @@ class JSONExporter: return jsonld - def _relationship_to_jsonld(self, rel: Dict[str, Any]) -> Dict[str, Any]: + def _relationship_to_jsonld( + self, rel: Dict[str, Any], index: int = 0 + ) -> Dict[str, Any]: """ Convert relationship to JSON-LD format. @@ -560,16 +568,18 @@ class JSONExporter: - type: Relationship type (optional) - confidence: Confidence score (optional) - metadata: Metadata dictionary (optional) + index: Position of the relationship in the exported list, used when + minting an IRI for a relationship that arrived without an id Returns: Dictionary in JSON-LD format representing the relationship """ - # Generate @id if not provided - rel_id = rel.get("id") - if not rel_id: - source_id = rel.get("source_id") or rel.get("source", "") - target_id = rel.get("target_id") or rel.get("target", "") - rel_id = f"semantica:rel/{source_id}_{target_id}" + # Generate @id if not provided, from the same mint the RDF serializers + # use, including the list index that separates two relationships + # sharing a pair of endpoints (#1101). + source_id = rel.get("source_id") or rel.get("source", "") + target_id = rel.get("target_id") or rel.get("target", "") + rel_id = rel.get("id") or mint_relationship_iri(index, source_id, target_id) jsonld = { "@id": rel_id, diff --git a/semantica/export/rdf_exporter.py b/semantica/export/rdf_exporter.py index 63eac4fe..de3d0dd0 100644 --- a/semantica/export/rdf_exporter.py +++ b/semantica/export/rdf_exporter.py @@ -600,11 +600,13 @@ class RDFSerializer: # Convert entities to JSON-LD entities = rdf_data.get("entities", []) for entity in entities: - # Generate @id if not provided - entity_id = entity.get("id") - if not entity_id: - entity_text = entity.get("text", "") - entity_id = f"semantica:entity/{entity_text}" + # Generate @id if not provided. Minted the same way the Turtle and + # N-Triples paths mint it (#1101), so one knowledge graph carries + # the same node identity whichever serializer wrote it. The former + # f"semantica:entity/{text}" interpolated the raw text into an IRI: + # any entity whose text contained a space produced an invalid IRI + # and was dropped in full by a JSON-LD parser, silently. + entity_id = entity.get("id") or mint_entity_iri(entity.get("text", "")) jsonld["@graph"].append( { @@ -617,24 +619,22 @@ class RDFSerializer: # Convert relationships to JSON-LD relationships = rdf_data.get("relationships", []) - for rel in relationships: - # Generate @id if not provided - rel_id = rel.get("id") - if not rel_id: - source_id = rel.get("source_id", "") - target_id = rel.get("target_id", "") - rel_id = f"semantica:rel/{source_id}_{target_id}" + for index, rel in enumerate(relationships): + # Endpoints are resolved both ways, as serialize_to_turtle resolves + # them: a relationship carrying source/target rather than + # source_id/target_id used to hash into f"semantica:rel/_", so every + # such relationship in an export collapsed onto one node and their + # types and endpoints merged. + source = rel.get("source_id") or rel.get("source", "") + target = rel.get("target_id") or rel.get("target", "") + rel_id = rel.get("id") or mint_relationship_iri(index, source, target) jsonld["@graph"].append( { "@id": rel_id, "@type": "semantica:Relationship", - "semantica:source": { - "@id": rel.get("source_id") or rel.get("source") - }, - "semantica:target": { - "@id": rel.get("target_id") or rel.get("target") - }, + "semantica:source": {"@id": source}, + "semantica:target": {"@id": target}, "semantica:type": rel.get("type", "related_to"), } ) diff --git a/tests/export/test_jsonld_iri_minting.py b/tests/export/test_jsonld_iri_minting.py new file mode 100644 index 00000000..26b24bf7 --- /dev/null +++ b/tests/export/test_jsonld_iri_minting.py @@ -0,0 +1,139 @@ +"""The JSON-LD paths must mint the same IRIs as the RDF paths (issue #1101). + +#1101 was fixed for the Turtle, N-Triples and RDF/XML serializers: an entity +arriving without an id gets a deterministic IRI in the declared namespace. The +JSON-LD paths were left interpolating the entity's own text into +``f"semantica:entity/{text}"`` and the relationship's endpoints into +``f"semantica:rel/{source}_{target}"``, which fails three ways: + +* a text containing a space produces an invalid IRI, and a JSON-LD parser drops + the whole node rather than complaining, so the entity vanishes from the export; +* relationships carrying ``source``/``target`` rather than ``source_id``/ + ``target_id`` all minted ``semantica:rel/_``, so every one of them collapsed + onto a single node whose types and endpoints merged; +* the JSON-LD @id and the Turtle IRI for one entity disagreed, so the two + serializations of one knowledge graph were two different graphs. + +``JSONExporter.export_entities`` and ``export_relationships`` also wrote +``semantica:entities`` into a context that never declared the ``semantica`` +prefix, which a JSON-LD processor reads as an IRI in the scheme ``semantica`` — +the original #1101 failure mode, on a path the first fix did not cover. +""" + +import json + +import pytest + +from semantica.export.json_exporter import JSONExporter +from semantica.export.rdf_exporter import ( + RDFExporter, + SEMANTICA_NS, + mint_entity_iri, + mint_relationship_iri, +) + +KG = { + "entities": [ + {"text": "Acme Corp", "type": "https://example.org/Org"}, + {"id": "https://example.org/e2", "text": "Bob"}, + ], + "relationships": [ + {"source": "https://example.org/a", "target": "https://example.org/b", + "type": "https://example.org/employs"}, + {"source": "https://example.org/b", "target": "https://example.org/a", + "type": "https://example.org/works_for"}, + ], +} + + +def _graph(document: str): + """Parse a JSON-LD document the way a consumer would.""" + rdflib = pytest.importorskip("rdflib") + graph = rdflib.Graph() + graph.parse(data=document, format="json-ld") + return graph + + +def test_jsonld_entity_id_is_the_minted_iri_not_the_interpolated_text(): + graph = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))["@graph"] + + assert graph[0]["@id"] == mint_entity_iri("Acme Corp") + assert graph[0]["@id"].startswith(SEMANTICA_NS) + assert "semantica:entity/" not in json.dumps(graph) + + +def test_json_exporter_mints_the_same_entity_iri_as_the_rdf_exporter(): + """One knowledge graph, two exporters, one node identity.""" + from_rdf = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))["@graph"] + from_json = JSONExporter()._convert_kg_to_jsonld(KG) + + assert from_json["semantica:entities"][0]["@id"] == from_rdf[0]["@id"] + assert from_json["semantica:relationships"][0]["@id"] == from_rdf[2]["@id"] + + +def test_minted_jsonld_iri_agrees_with_the_turtle_serialization(): + """The two serializations of one graph must name the same entity alike.""" + turtle = RDFExporter().export_to_rdf(KG, format="turtle") + jsonld = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld")) + + minted = mint_entity_iri("Acme Corp") + assert f"<{minted}>" in turtle + assert jsonld["@graph"][0]["@id"] == minted + + +def test_entity_whose_text_contains_a_space_survives_a_jsonld_parse(): + """The regression that lost data: an invalid IRI is dropped, not reported.""" + kg = {"entities": [{"text": "Acme Corp"}], "relationships": []} + graph = _graph(json.dumps(JSONExporter()._convert_kg_to_jsonld(kg))) + + subjects = {str(s) for s in graph.subjects()} + assert mint_entity_iri("Acme Corp") in subjects + + +def test_relationships_carrying_source_and_target_do_not_collide(): + """Two relationships, two nodes: ``semantica:rel/_`` merged them into one.""" + jsonld = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld")) + relationships = [n for n in jsonld["@graph"] + if n["@type"] == "semantica:Relationship"] + + ids = {node["@id"] for node in relationships} + assert len(ids) == len(relationships) == 2 + assert ids == { + mint_relationship_iri(0, "https://example.org/a", "https://example.org/b"), + mint_relationship_iri(1, "https://example.org/b", "https://example.org/a"), + } + + graph = _graph(json.dumps(jsonld)) + assert len({str(s) for s in graph.subjects()} & ids) == 2 + + +def test_no_export_path_writes_an_iri_in_the_semantica_scheme(tmp_path): + """Nothing may expand to the scheme ``semantica`` rather than the namespace.""" + documents = [ + RDFExporter().export_to_rdf(KG, format="jsonld"), + json.dumps(JSONExporter()._convert_kg_to_jsonld(KG)), + ] + exporter = JSONExporter() + exporter.export_entities(KG["entities"], tmp_path / "entities.json") + exporter.export_relationships(KG["relationships"], tmp_path / "relationships.json") + documents.append((tmp_path / "entities.json").read_text()) + documents.append((tmp_path / "relationships.json").read_text()) + + for document in documents: + for term in _graph(document).all_nodes(): + assert not str(term).startswith("semantica:"), document + for predicate in _graph(document).predicates(): + assert not str(predicate).startswith("semantica:"), document + + +def test_entity_and_relationship_lists_expand_into_the_declared_namespace(tmp_path): + exporter = JSONExporter() + exporter.export_entities(KG["entities"], tmp_path / "entities.json") + exporter.export_relationships(KG["relationships"], tmp_path / "relationships.json") + + predicates = set() + for name in ("entities.json", "relationships.json"): + predicates |= {str(p) for p in _graph((tmp_path / name).read_text()).predicates()} + + assert f"{SEMANTICA_NS}entities" in predicates + assert f"{SEMANTICA_NS}relationships" in predicates