mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
fix(export): mint JSON-LD @ids the same way the RDF serializers do (#1101)
The #1101 fix covered serialize_to_turtle, serialize_to_ntriples and serialize_to_rdfxml. Both JSON-LD writers were left interpolating the entity's own text into f"semantica:entity/{text}" and the endpoints into f"semantica:rel/{source}_{target}". Three consequences, all reproducible on 0.6.5 through the public API: - An entity whose text contains a space, which is most organisation and person names an extractor produces, mints an invalid IRI. A JSON-LD parser drops that node in full and says nothing, so the entity is simply missing from the export: rdflib reads 6 triples for {"text": "AcmeCorp"} and 2 for {"text": "Acme Corp"}. - serialize_to_jsonld resolved endpoints from source_id/target_id only, while the rest of the module accepts source/target too. Every relationship carrying the second form minted the identical "semantica:rel/_", so all of them collapsed onto one node and their types and endpoints merged into a graph nobody wrote. - The JSON-LD @id and the Turtle IRI for one entity disagreed (ns#entity/Acme Corp vs ns#entity_a73cb4563ee2e72c), so the two serializations of one knowledge graph were two different graphs. Both writers now use mint_entity_iri/mint_relationship_iri, resolving endpoints both ways and passing the list index the RDF paths pass, so one knowledge graph carries one node identity whichever serializer wrote it. JSONExporter.export_entities and export_relationships also declare the semantica prefix their @context was already writing "semantica:entities" against. Without the declaration a processor reads that as an IRI in the scheme semantica rather than the namespace expansion, which is the original #1101 defect on a third path: rdflib returns the predicate literally as semantica:entities. tests/export/test_jsonld_iri_minting.py parses each export with a real JSON-LD processor rather than asserting on the JSON text, and covers all seven claims above. Each test fails on the parent commit. 236 export and ontology tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
FABIOTESS
co-authored by
Claude Opus 5
Claude Opus 5
parent
2a303cf4da
commit
75b026c6dd
+3
-1
@@ -18,7 +18,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- The missing-id fallback minted entity/relationship IRIs from Python's builtin `hash()`, randomised per process (`PYTHONHASHSEED`), so the same entity got a different IRI on every run and exports couldn't be diffed, deduplicated, or joined to an earlier provenance record. It also wrote `<semantica:entity_N>`, an IRI in the scheme `semantica` rather than the expansion of the declared prefix, so those nodes never joined with anything written through it. Minting now uses SHA-256 and writes a full IRI in the declared namespace; the same fix applies to the default entity/relationship types in the Turtle path
|
||||
- **Fixed during review** (Qodo): the temporal fallback minted from `source_id` only, while the main serializer accepts `source_id` or `source` — relationships using the second form hashed two empty strings, which the previous randomised `hash()` masked by making the IRI unstable anyway; once deterministic, unrelated relationships at the same list index collided on one IRI across exports. Endpoints are now resolved the same way `serialize_to_turtle` resolves them, before minting. `sem:confidence` also lost its declared `xsd:decimal` range: the N-Triples serializer types the same value `xsd:float`, and the two are disjoint, so declaring either contradicted one of the exporters (tracked in #1100) — a new `test_declared_ranges_do_not_contradict_what_the_exporters_emit` guards the whole class of that mistake
|
||||
- **Fixed in follow-up**: `serialize_to_rdfxml`'s default entity type still wrote the bare string `"semantica:Entity"` into an `rdf:resource` attribute, which (unlike a Turtle angle-bracket or an XML element name) is not namespace-expanded — the exact #1101 failure mode, just on the untested RDF/XML path. `json_exporter.py`'s `semantica:format` and `@type: "semantica:KnowledgeGraph"` were emitted but absent from both the vocabulary and the test's `EMITTED_TERMS` guard set, so the "undeclared terms fail the build" claim didn't actually cover them — both are now declared and guarded. `MANIFEST.in` didn't mirror the `pyproject.toml` package-data addition, so a source-distribution install could omit the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only `PATH`, breaking it on Windows; now overrides only `PYTHONHASHSEED` on top of the inherited environment
|
||||
- 229 export and ontology tests pass
|
||||
- **Also fixed, on the JSON-LD paths**: the first fix covered the Turtle, N-Triples and RDF/XML serializers, and left both JSON-LD writers interpolating the entity's own text into `f"semantica:entity/{text}"` and the endpoints into `f"semantica:rel/{source}_{target}"`. Three consequences, all live in 0.6.5: an entity whose text contained a space produced an invalid IRI, and a JSON-LD parser dropped that node in full rather than reporting it, so the entity disappeared from the export; every relationship carrying `source`/`target` rather than `source_id`/`target_id` minted the identical `semantica:rel/_`, collapsing all of them onto one node whose types and endpoints merged; and the JSON-LD `@id` disagreed with the Turtle IRI for the same entity, so the two serializations of one knowledge graph were two different graphs. Both JSON-LD writers now use `mint_entity_iri`/`mint_relationship_iri`, and `JSONExporter.export_entities`/`export_relationships` declare the `semantica` prefix their `@context` was already writing `semantica:entities` against — without it a processor reads that as an IRI in the scheme `semantica`, which is the original #1101 defect on a third path
|
||||
- `tests/export/test_jsonld_iri_minting.py` parses each export with a real JSON-LD processor and asserts the entity survives, the relationships stay distinct, no term expands into the `semantica` scheme, and the JSON-LD `@id` equals the Turtle IRI
|
||||
- 236 export and ontology tests pass
|
||||
|
||||
- **First-class CrewAI integration** (#962)
|
||||
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
|
||||
|
||||
@@ -32,6 +32,7 @@ from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.helpers import ensure_directory, write_json_file
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .rdf_exporter import SEMANTICA_NS, mint_entity_iri, mint_relationship_iri
|
||||
|
||||
|
||||
class JSONExporter:
|
||||
@@ -265,6 +266,7 @@ class JSONExporter:
|
||||
json_data = {
|
||||
"@context": {
|
||||
"@vocab": "https://semantica.dev/vocab/",
|
||||
"semantica": SEMANTICA_NS,
|
||||
"entities": {"@id": "semantica:entities", "@container": "@list"},
|
||||
},
|
||||
"entities": entities,
|
||||
@@ -294,6 +296,7 @@ class JSONExporter:
|
||||
json_data = {
|
||||
"@context": {
|
||||
"@vocab": "https://semantica.dev/vocab/",
|
||||
"semantica": SEMANTICA_NS,
|
||||
"relationships": {
|
||||
"@id": "semantica:relationships",
|
||||
"@container": "@list",
|
||||
@@ -495,7 +498,8 @@ class JSONExporter:
|
||||
relationships = kg.get("relationships", [])
|
||||
if relationships:
|
||||
jsonld["semantica:relationships"] = [
|
||||
self._relationship_to_jsonld(r) for r in relationships
|
||||
self._relationship_to_jsonld(r, index)
|
||||
for index, r in enumerate(relationships)
|
||||
]
|
||||
self.logger.debug(
|
||||
f"Converted {len(relationships)} relationship(s) to JSON-LD"
|
||||
@@ -526,11 +530,13 @@ class JSONExporter:
|
||||
Returns:
|
||||
Dictionary in JSON-LD format representing the entity
|
||||
"""
|
||||
# Generate @id if not provided
|
||||
entity_id = entity.get("id")
|
||||
if not entity_id:
|
||||
entity_text = entity.get("text") or entity.get("label", "unknown")
|
||||
entity_id = f"semantica:entity/{entity_text}"
|
||||
# Generate @id if not provided. Minted exactly as the RDF serializers
|
||||
# mint it (#1101), so the JSON-LD and Turtle exports of one knowledge
|
||||
# graph name the same entity with the same IRI. Interpolating the raw
|
||||
# text into f"semantica:entity/{text}" produced an invalid IRI for any
|
||||
# text containing a space, and a JSON-LD parser dropped the whole node.
|
||||
entity_text = entity.get("text") or entity.get("label", "unknown")
|
||||
entity_id = entity.get("id") or mint_entity_iri(entity_text)
|
||||
|
||||
jsonld = {
|
||||
"@id": entity_id,
|
||||
@@ -545,7 +551,9 @@ class JSONExporter:
|
||||
|
||||
return jsonld
|
||||
|
||||
def _relationship_to_jsonld(self, rel: Dict[str, Any]) -> Dict[str, Any]:
|
||||
def _relationship_to_jsonld(
|
||||
self, rel: Dict[str, Any], index: int = 0
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert relationship to JSON-LD format.
|
||||
|
||||
@@ -560,16 +568,18 @@ class JSONExporter:
|
||||
- type: Relationship type (optional)
|
||||
- confidence: Confidence score (optional)
|
||||
- metadata: Metadata dictionary (optional)
|
||||
index: Position of the relationship in the exported list, used when
|
||||
minting an IRI for a relationship that arrived without an id
|
||||
|
||||
Returns:
|
||||
Dictionary in JSON-LD format representing the relationship
|
||||
"""
|
||||
# Generate @id if not provided
|
||||
rel_id = rel.get("id")
|
||||
if not rel_id:
|
||||
source_id = rel.get("source_id") or rel.get("source", "")
|
||||
target_id = rel.get("target_id") or rel.get("target", "")
|
||||
rel_id = f"semantica:rel/{source_id}_{target_id}"
|
||||
# Generate @id if not provided, from the same mint the RDF serializers
|
||||
# use, including the list index that separates two relationships
|
||||
# sharing a pair of endpoints (#1101).
|
||||
source_id = rel.get("source_id") or rel.get("source", "")
|
||||
target_id = rel.get("target_id") or rel.get("target", "")
|
||||
rel_id = rel.get("id") or mint_relationship_iri(index, source_id, target_id)
|
||||
|
||||
jsonld = {
|
||||
"@id": rel_id,
|
||||
|
||||
@@ -600,11 +600,13 @@ class RDFSerializer:
|
||||
# Convert entities to JSON-LD
|
||||
entities = rdf_data.get("entities", [])
|
||||
for entity in entities:
|
||||
# Generate @id if not provided
|
||||
entity_id = entity.get("id")
|
||||
if not entity_id:
|
||||
entity_text = entity.get("text", "")
|
||||
entity_id = f"semantica:entity/{entity_text}"
|
||||
# Generate @id if not provided. Minted the same way the Turtle and
|
||||
# N-Triples paths mint it (#1101), so one knowledge graph carries
|
||||
# the same node identity whichever serializer wrote it. The former
|
||||
# f"semantica:entity/{text}" interpolated the raw text into an IRI:
|
||||
# any entity whose text contained a space produced an invalid IRI
|
||||
# and was dropped in full by a JSON-LD parser, silently.
|
||||
entity_id = entity.get("id") or mint_entity_iri(entity.get("text", ""))
|
||||
|
||||
jsonld["@graph"].append(
|
||||
{
|
||||
@@ -617,24 +619,22 @@ class RDFSerializer:
|
||||
|
||||
# Convert relationships to JSON-LD
|
||||
relationships = rdf_data.get("relationships", [])
|
||||
for rel in relationships:
|
||||
# Generate @id if not provided
|
||||
rel_id = rel.get("id")
|
||||
if not rel_id:
|
||||
source_id = rel.get("source_id", "")
|
||||
target_id = rel.get("target_id", "")
|
||||
rel_id = f"semantica:rel/{source_id}_{target_id}"
|
||||
for index, rel in enumerate(relationships):
|
||||
# Endpoints are resolved both ways, as serialize_to_turtle resolves
|
||||
# them: a relationship carrying source/target rather than
|
||||
# source_id/target_id used to hash into f"semantica:rel/_", so every
|
||||
# such relationship in an export collapsed onto one node and their
|
||||
# types and endpoints merged.
|
||||
source = rel.get("source_id") or rel.get("source", "")
|
||||
target = rel.get("target_id") or rel.get("target", "")
|
||||
rel_id = rel.get("id") or mint_relationship_iri(index, source, target)
|
||||
|
||||
jsonld["@graph"].append(
|
||||
{
|
||||
"@id": rel_id,
|
||||
"@type": "semantica:Relationship",
|
||||
"semantica:source": {
|
||||
"@id": rel.get("source_id") or rel.get("source")
|
||||
},
|
||||
"semantica:target": {
|
||||
"@id": rel.get("target_id") or rel.get("target")
|
||||
},
|
||||
"semantica:source": {"@id": source},
|
||||
"semantica:target": {"@id": target},
|
||||
"semantica:type": rel.get("type", "related_to"),
|
||||
}
|
||||
)
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""The JSON-LD paths must mint the same IRIs as the RDF paths (issue #1101).
|
||||
|
||||
#1101 was fixed for the Turtle, N-Triples and RDF/XML serializers: an entity
|
||||
arriving without an id gets a deterministic IRI in the declared namespace. The
|
||||
JSON-LD paths were left interpolating the entity's own text into
|
||||
``f"semantica:entity/{text}"`` and the relationship's endpoints into
|
||||
``f"semantica:rel/{source}_{target}"``, which fails three ways:
|
||||
|
||||
* a text containing a space produces an invalid IRI, and a JSON-LD parser drops
|
||||
the whole node rather than complaining, so the entity vanishes from the export;
|
||||
* relationships carrying ``source``/``target`` rather than ``source_id``/
|
||||
``target_id`` all minted ``semantica:rel/_``, so every one of them collapsed
|
||||
onto a single node whose types and endpoints merged;
|
||||
* the JSON-LD @id and the Turtle IRI for one entity disagreed, so the two
|
||||
serializations of one knowledge graph were two different graphs.
|
||||
|
||||
``JSONExporter.export_entities`` and ``export_relationships`` also wrote
|
||||
``semantica:entities`` into a context that never declared the ``semantica``
|
||||
prefix, which a JSON-LD processor reads as an IRI in the scheme ``semantica`` —
|
||||
the original #1101 failure mode, on a path the first fix did not cover.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from semantica.export.json_exporter import JSONExporter
|
||||
from semantica.export.rdf_exporter import (
|
||||
RDFExporter,
|
||||
SEMANTICA_NS,
|
||||
mint_entity_iri,
|
||||
mint_relationship_iri,
|
||||
)
|
||||
|
||||
KG = {
|
||||
"entities": [
|
||||
{"text": "Acme Corp", "type": "https://example.org/Org"},
|
||||
{"id": "https://example.org/e2", "text": "Bob"},
|
||||
],
|
||||
"relationships": [
|
||||
{"source": "https://example.org/a", "target": "https://example.org/b",
|
||||
"type": "https://example.org/employs"},
|
||||
{"source": "https://example.org/b", "target": "https://example.org/a",
|
||||
"type": "https://example.org/works_for"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _graph(document: str):
|
||||
"""Parse a JSON-LD document the way a consumer would."""
|
||||
rdflib = pytest.importorskip("rdflib")
|
||||
graph = rdflib.Graph()
|
||||
graph.parse(data=document, format="json-ld")
|
||||
return graph
|
||||
|
||||
|
||||
def test_jsonld_entity_id_is_the_minted_iri_not_the_interpolated_text():
|
||||
graph = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))["@graph"]
|
||||
|
||||
assert graph[0]["@id"] == mint_entity_iri("Acme Corp")
|
||||
assert graph[0]["@id"].startswith(SEMANTICA_NS)
|
||||
assert "semantica:entity/" not in json.dumps(graph)
|
||||
|
||||
|
||||
def test_json_exporter_mints_the_same_entity_iri_as_the_rdf_exporter():
|
||||
"""One knowledge graph, two exporters, one node identity."""
|
||||
from_rdf = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))["@graph"]
|
||||
from_json = JSONExporter()._convert_kg_to_jsonld(KG)
|
||||
|
||||
assert from_json["semantica:entities"][0]["@id"] == from_rdf[0]["@id"]
|
||||
assert from_json["semantica:relationships"][0]["@id"] == from_rdf[2]["@id"]
|
||||
|
||||
|
||||
def test_minted_jsonld_iri_agrees_with_the_turtle_serialization():
|
||||
"""The two serializations of one graph must name the same entity alike."""
|
||||
turtle = RDFExporter().export_to_rdf(KG, format="turtle")
|
||||
jsonld = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))
|
||||
|
||||
minted = mint_entity_iri("Acme Corp")
|
||||
assert f"<{minted}>" in turtle
|
||||
assert jsonld["@graph"][0]["@id"] == minted
|
||||
|
||||
|
||||
def test_entity_whose_text_contains_a_space_survives_a_jsonld_parse():
|
||||
"""The regression that lost data: an invalid IRI is dropped, not reported."""
|
||||
kg = {"entities": [{"text": "Acme Corp"}], "relationships": []}
|
||||
graph = _graph(json.dumps(JSONExporter()._convert_kg_to_jsonld(kg)))
|
||||
|
||||
subjects = {str(s) for s in graph.subjects()}
|
||||
assert mint_entity_iri("Acme Corp") in subjects
|
||||
|
||||
|
||||
def test_relationships_carrying_source_and_target_do_not_collide():
|
||||
"""Two relationships, two nodes: ``semantica:rel/_`` merged them into one."""
|
||||
jsonld = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))
|
||||
relationships = [n for n in jsonld["@graph"]
|
||||
if n["@type"] == "semantica:Relationship"]
|
||||
|
||||
ids = {node["@id"] for node in relationships}
|
||||
assert len(ids) == len(relationships) == 2
|
||||
assert ids == {
|
||||
mint_relationship_iri(0, "https://example.org/a", "https://example.org/b"),
|
||||
mint_relationship_iri(1, "https://example.org/b", "https://example.org/a"),
|
||||
}
|
||||
|
||||
graph = _graph(json.dumps(jsonld))
|
||||
assert len({str(s) for s in graph.subjects()} & ids) == 2
|
||||
|
||||
|
||||
def test_no_export_path_writes_an_iri_in_the_semantica_scheme(tmp_path):
|
||||
"""Nothing may expand to the scheme ``semantica`` rather than the namespace."""
|
||||
documents = [
|
||||
RDFExporter().export_to_rdf(KG, format="jsonld"),
|
||||
json.dumps(JSONExporter()._convert_kg_to_jsonld(KG)),
|
||||
]
|
||||
exporter = JSONExporter()
|
||||
exporter.export_entities(KG["entities"], tmp_path / "entities.json")
|
||||
exporter.export_relationships(KG["relationships"], tmp_path / "relationships.json")
|
||||
documents.append((tmp_path / "entities.json").read_text())
|
||||
documents.append((tmp_path / "relationships.json").read_text())
|
||||
|
||||
for document in documents:
|
||||
for term in _graph(document).all_nodes():
|
||||
assert not str(term).startswith("semantica:"), document
|
||||
for predicate in _graph(document).predicates():
|
||||
assert not str(predicate).startswith("semantica:"), document
|
||||
|
||||
|
||||
def test_entity_and_relationship_lists_expand_into_the_declared_namespace(tmp_path):
|
||||
exporter = JSONExporter()
|
||||
exporter.export_entities(KG["entities"], tmp_path / "entities.json")
|
||||
exporter.export_relationships(KG["relationships"], tmp_path / "relationships.json")
|
||||
|
||||
predicates = set()
|
||||
for name in ("entities.json", "relationships.json"):
|
||||
predicates |= {str(p) for p in _graph((tmp_path / name).read_text()).predicates()}
|
||||
|
||||
assert f"{SEMANTICA_NS}entities" in predicates
|
||||
assert f"{SEMANTICA_NS}relationships" in predicates
|
||||
Reference in New Issue
Block a user