mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
fix: close remaining review gaps in vocabulary/deterministic-IRI PR
serialize_to_rdfxml still defaulted entity_type to the bare string "semantica:Entity" written into an rdf:resource attribute, which isn't namespace-expanded the way a Turtle angle-bracket or XML element name is - the same #1101 failure mode, just on the path the original tests didn't cover. Now uses the full-IRI DEFAULT_ENTITY_TYPE like the Turtle path. json_exporter.py emits semantica:format and @type: "semantica:KnowledgeGraph", neither of which was declared in the vocabulary or included in EMITTED_TERMS, so the "undeclared terms fail the build" guarantee didn't actually cover them. Both are now declared with rdfs:label/comment and added to the guard set. MANIFEST.in didn't mirror the pyproject.toml package-data addition, so a source-distribution install could ship without the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only PATH, breaking it on Windows and any host needing other inherited env vars; now overrides only PYTHONHASHSEED on top of the inherited environment. Also folds mint_entity_iri/mint_relationship_iri's hand-rolled hashlib.sha256(...).hexdigest() into the existing hash_data() helper this file already imports alongside. 229 export and ontology tests pass, including a new regression test for the RDF/XML default-type fix. Co-Authored-By: fabio-rovai <fabio@thetesseractacademy.com>
This commit is contained in:
co-authored by
fabio-rovai
parent
e55c03bd39
commit
2d75952476
@@ -11,6 +11,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
|
- **Semantica RDF vocabulary, and deterministic entity/relationship IRIs** (#1109, closes #1107, closes #1101) by @fabio-rovai, reviewed by @KaifAhmad1
|
||||||
|
- Every RDF/JSON-LD export mints terms in `https://semantica.dev/ns#`, and until now nothing declared what those terms meant — the namespace 404s and no vocabulary shipped with the package, so a consumer receiving an export had no way to tell `sem:text` from a typo of it, and no closed-world checker could validate an export at all
|
||||||
|
- `semantica/ontology/vocabulary/semantica-ns.ttl` declares the terms the exporters actually emit — drawn from the emitting call sites in `export/rdf_exporter.py`, `export/json_exporter.py` and `provenance/manager.py`, not from what a vocabulary "ought" to contain. Ships inside the package (`from semantica.ontology.vocabulary import vocabulary_turtle`) so it loads without a network round trip, and is the same document intended to be served at the namespace IRI once hosting/content-negotiation is sorted
|
||||||
|
- `tests/ontology/test_vocabulary.py` ties the document to the code: every term a serializer can write must be declared, so adding a term to an exporter without declaring it fails the build
|
||||||
|
- The missing-id fallback minted entity/relationship IRIs from Python's builtin `hash()`, randomised per process (`PYTHONHASHSEED`), so the same entity got a different IRI on every run and exports couldn't be diffed, deduplicated, or joined to an earlier provenance record. It also wrote `<semantica:entity_N>`, an IRI in the scheme `semantica` rather than the expansion of the declared prefix, so those nodes never joined with anything written through it. Minting now uses SHA-256 and writes a full IRI in the declared namespace; the same fix applies to the default entity/relationship types in the Turtle path
|
||||||
|
- **Fixed during review** (Qodo): the temporal fallback minted from `source_id` only, while the main serializer accepts `source_id` or `source` — relationships using the second form hashed two empty strings, which the previous randomised `hash()` masked by making the IRI unstable anyway; once deterministic, unrelated relationships at the same list index collided on one IRI across exports. Endpoints are now resolved the same way `serialize_to_turtle` resolves them, before minting. `sem:confidence` also lost its declared `xsd:decimal` range: the N-Triples serializer types the same value `xsd:float`, and the two are disjoint, so declaring either contradicted one of the exporters (tracked in #1100) — a new `test_declared_ranges_do_not_contradict_what_the_exporters_emit` guards the whole class of that mistake
|
||||||
|
- **Fixed in follow-up**: `serialize_to_rdfxml`'s default entity type still wrote the bare string `"semantica:Entity"` into an `rdf:resource` attribute, which (unlike a Turtle angle-bracket or an XML element name) is not namespace-expanded — the exact #1101 failure mode, just on the untested RDF/XML path. `json_exporter.py`'s `semantica:format` and `@type: "semantica:KnowledgeGraph"` were emitted but absent from both the vocabulary and the test's `EMITTED_TERMS` guard set, so the "undeclared terms fail the build" claim didn't actually cover them — both are now declared and guarded. `MANIFEST.in` didn't mirror the `pyproject.toml` package-data addition, so a source-distribution install could omit the vocabulary file. The cross-process minting-stability test replaced the subprocess's entire environment with a POSIX-only `PATH`, breaking it on Windows; now overrides only `PYTHONHASHSEED` on top of the inherited environment
|
||||||
|
- 229 export and ontology tests pass
|
||||||
|
|
||||||
- **First-class CrewAI integration** (#962)
|
- **First-class CrewAI integration** (#962)
|
||||||
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
|
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
|
||||||
- `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()`
|
- `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()`
|
||||||
|
|||||||
@@ -1 +1,2 @@
|
|||||||
recursive-include semantica/static *
|
recursive-include semantica/static *
|
||||||
|
recursive-include semantica/ontology/vocabulary *.ttl
|
||||||
|
|||||||
@@ -30,11 +30,10 @@ License: MIT
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import hashlib
|
|
||||||
from typing import Any, Dict, List, Optional, Set, Union
|
from typing import Any, Dict, List, Optional, Set, Union
|
||||||
|
|
||||||
from ..utils.exceptions import ProcessingError, ValidationError
|
from ..utils.exceptions import ProcessingError, ValidationError
|
||||||
from ..utils.helpers import ensure_directory
|
from ..utils.helpers import ensure_directory, hash_data
|
||||||
from ..utils.logging import get_logger
|
from ..utils.logging import get_logger
|
||||||
from ..utils.progress_tracker import get_progress_tracker
|
from ..utils.progress_tracker import get_progress_tracker
|
||||||
|
|
||||||
@@ -60,13 +59,13 @@ def mint_entity_iri(text: str) -> str:
|
|||||||
provenance record written by an earlier process. SHA-256 is stable across
|
provenance record written by an earlier process. SHA-256 is stable across
|
||||||
runs and machines, which is what an identifier has to be.
|
runs and machines, which is what an identifier has to be.
|
||||||
"""
|
"""
|
||||||
digest = hashlib.sha256(str(text).encode("utf-8")).hexdigest()[:16]
|
digest = hash_data(str(text))[:16]
|
||||||
return f"{SEMANTICA_NS}entity_{digest}"
|
return f"{SEMANTICA_NS}entity_{digest}"
|
||||||
|
|
||||||
|
|
||||||
def mint_relationship_iri(index: int, source: Any, target: Any) -> str:
|
def mint_relationship_iri(index: int, source: Any, target: Any) -> str:
|
||||||
"""Mint a stable IRI for a relationship that arrived without an id."""
|
"""Mint a stable IRI for a relationship that arrived without an id."""
|
||||||
digest = hashlib.sha256(f"{source}\x00{target}".encode("utf-8")).hexdigest()[:16]
|
digest = hash_data(f"{source}\x00{target}")[:16]
|
||||||
return f"{SEMANTICA_NS}rel_{index}_{digest}"
|
return f"{SEMANTICA_NS}rel_{index}_{digest}"
|
||||||
|
|
||||||
|
|
||||||
@@ -526,7 +525,7 @@ class RDFSerializer:
|
|||||||
entity_text = entity.get("text", "")
|
entity_text = entity.get("text", "")
|
||||||
entity_id = mint_entity_iri(entity_text)
|
entity_id = mint_entity_iri(entity_text)
|
||||||
|
|
||||||
entity_type = entity.get("type", "semantica:Entity")
|
entity_type = entity.get("type", DEFAULT_ENTITY_TYPE)
|
||||||
text = entity.get("text") or entity.get("label", "")
|
text = entity.get("text") or entity.get("label", "")
|
||||||
confidence = entity.get("confidence", 1.0)
|
confidence = entity.get("confidence", 1.0)
|
||||||
|
|
||||||
|
|||||||
@@ -33,6 +33,13 @@ where a relationship carries sem:type, sem:source and sem:target rather than
|
|||||||
being written as a single triple.""" ;
|
being written as a single triple.""" ;
|
||||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||||
|
|
||||||
|
sem:KnowledgeGraph a owl:Class ;
|
||||||
|
rdfs:label "Knowledge Graph" ;
|
||||||
|
rdfs:comment """The document-level type of a JSON-LD export: the @type of
|
||||||
|
the top-level node carrying sem:entities, sem:relationships and
|
||||||
|
sem:exportedAt. Emitted by _convert_kg_to_jsonld in export/json_exporter.py.""" ;
|
||||||
|
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||||
|
|
||||||
# ── Properties on an entity ──────────────────────────────────────────────────
|
# ── Properties on an entity ──────────────────────────────────────────────────
|
||||||
|
|
||||||
sem:text a owl:DatatypeProperty ;
|
sem:text a owl:DatatypeProperty ;
|
||||||
@@ -117,6 +124,14 @@ values carry no timezone offset.""" ;
|
|||||||
rdfs:range xsd:dateTime ;
|
rdfs:range xsd:dateTime ;
|
||||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||||
|
|
||||||
|
sem:format a owl:DatatypeProperty ;
|
||||||
|
rdfs:label "format" ;
|
||||||
|
rdfs:comment """The serialization format label written on a JSON-LD
|
||||||
|
document (currently always the literal "json-ld"). Emitted by
|
||||||
|
JSONExporter.export_to_jsonld in export/json_exporter.py.""" ;
|
||||||
|
rdfs:range xsd:string ;
|
||||||
|
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||||
|
|
||||||
# ── Temporal term (OWL-Time export) ──────────────────────────────────────────
|
# ── Temporal term (OWL-Time export) ──────────────────────────────────────────
|
||||||
|
|
||||||
sem:openEndedInterval a owl:DatatypeProperty ;
|
sem:openEndedInterval a owl:DatatypeProperty ;
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ IRI in the scheme ``semantica`` rather than the expansion of the declared
|
|||||||
``semantica:`` prefix, so it never joined with anything written through it.
|
``semantica:`` prefix, so it never joined with anything written through it.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
@@ -45,7 +46,7 @@ def test_minted_entity_iri_is_stable_across_processes():
|
|||||||
capture_output=True,
|
capture_output=True,
|
||||||
text=True,
|
text=True,
|
||||||
check=True,
|
check=True,
|
||||||
env={"PYTHONHASHSEED": seed, "PATH": "/usr/bin:/bin"},
|
env={**os.environ, "PYTHONHASHSEED": seed},
|
||||||
).stdout.strip()
|
).stdout.strip()
|
||||||
for seed in ("0", "1", "random")
|
for seed in ("0", "1", "random")
|
||||||
}
|
}
|
||||||
@@ -88,6 +89,22 @@ def test_default_types_are_written_as_full_iris_in_turtle():
|
|||||||
assert "<semantica:related_to>" not in turtle
|
assert "<semantica:related_to>" not in turtle
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_entity_type_is_a_full_iri_in_rdfxml():
|
||||||
|
"""RDF/XML's rdf:resource is an attribute value, not a QName context, so a
|
||||||
|
|
||||||
|
prefixed default there (``semantica:Entity``) resolves to the scheme
|
||||||
|
``semantica`` rather than the declared namespace — the same failure mode
|
||||||
|
fixed for Turtle in #1101, missed here because the original tests only
|
||||||
|
checked Turtle output.
|
||||||
|
"""
|
||||||
|
untyped = {"entities": [{"id": "https://example.org/e1", "text": "A"}],
|
||||||
|
"relationships": []}
|
||||||
|
rdfxml = RDFExporter().export_to_rdf(untyped, format="rdfxml")
|
||||||
|
|
||||||
|
assert f'rdf:resource="{DEFAULT_ENTITY_TYPE}"' in rdfxml
|
||||||
|
assert 'rdf:resource="semantica:Entity"' not in rdfxml
|
||||||
|
|
||||||
|
|
||||||
def test_temporal_minting_uses_either_endpoint_representation():
|
def test_temporal_minting_uses_either_endpoint_representation():
|
||||||
"""Relationships may carry source/target or source_id/target_id (#1109 review).
|
"""Relationships may carry source/target or source_id/target_id (#1109 review).
|
||||||
|
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ from semantica.ontology.vocabulary import ( # noqa: E402
|
|||||||
EMITTED_TERMS = {
|
EMITTED_TERMS = {
|
||||||
"Entity",
|
"Entity",
|
||||||
"Relationship",
|
"Relationship",
|
||||||
|
"KnowledgeGraph",
|
||||||
"text",
|
"text",
|
||||||
"confidence",
|
"confidence",
|
||||||
"metadata",
|
"metadata",
|
||||||
@@ -38,6 +39,7 @@ EMITTED_TERMS = {
|
|||||||
"entities",
|
"entities",
|
||||||
"relationships",
|
"relationships",
|
||||||
"exportedAt",
|
"exportedAt",
|
||||||
|
"format",
|
||||||
"openEndedInterval",
|
"openEndedInterval",
|
||||||
"role_generator",
|
"role_generator",
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user