mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Merge branch 'main' into fix/1184-causal-edge-vocabulary
This commit is contained in:
Mohd Kaif
@@ -8,7 +8,7 @@ icon: "shield-check"
|
||||
|
||||
SHACL (Shapes Constraint Language) is a standard for validating graph-based data. While an ontology defines the conceptual *schema* (the "what" exists in your domain), SHACL defines the structural *rules and constraints* (the "how" it should be structured).
|
||||
|
||||
In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and `_run_pyshacl` evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated.
|
||||
In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and the public `run_shacl_validation` function evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated. The historical `_run_pyshacl` name remains available as a compatibility alias.
|
||||
|
||||
## Why Use SHACL Validation?
|
||||
|
||||
@@ -55,7 +55,7 @@ Let's look at a simple, universally understood example: ensuring every `Employee
|
||||
```python
|
||||
from semantica.context import ContextGraph
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
# 1. Prepare your data graph
|
||||
graph = ContextGraph()
|
||||
@@ -95,7 +95,7 @@ data_ttl = """
|
||||
"""
|
||||
|
||||
# 5. Run Validation
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
|
||||
# 6. Analyze the Report
|
||||
print(f"Graph conforms: {report.conforms}")
|
||||
@@ -265,10 +265,10 @@ cve_id_shape = NodeShape(
|
||||
|
||||
## Step 4 — Run validation and read the report
|
||||
|
||||
Serialize the graph to RDF, then run `_run_pyshacl` against the shapes.
|
||||
Serialize the graph to RDF, then run `run_shacl_validation` against the shapes.
|
||||
|
||||
```python
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
# Prepare your RDF data string (since export_rdf primarily exports structural metadata,
|
||||
# you typically serialize your custom data graph to Turtle using rdflib or similar).
|
||||
@@ -281,7 +281,7 @@ data_ttl = """
|
||||
"""
|
||||
|
||||
# Run SHACL validation
|
||||
report = _run_pyshacl(
|
||||
report = run_shacl_validation(
|
||||
data_ttl,
|
||||
shacl_ttl,
|
||||
data_graph_format="turtle",
|
||||
@@ -366,8 +366,8 @@ print(f"Malware nodes missing 'family': {len(missing_family)}")
|
||||
# e.g. graph.update_node(node_id, {"family": "UNKNOWN — requires triage"})
|
||||
|
||||
# After remediation, re-run validation to confirm the fix
|
||||
# (re-export the patched graph to Turtle first, then call _run_pyshacl again)
|
||||
report2 = _run_pyshacl(patched_data_ttl, shacl_ttl)
|
||||
# (re-export the patched graph to Turtle first, then call run_shacl_validation again)
|
||||
report2 = run_shacl_validation(patched_data_ttl, shacl_ttl)
|
||||
print(f"Violations after remediation: {report2.violation_count}")
|
||||
# Violations after remediation: 0
|
||||
```
|
||||
@@ -377,7 +377,7 @@ print(f"Violations after remediation: {report2.violation_count}")
|
||||
## Common Pitfalls
|
||||
|
||||
- **Assuming the ontology automatically enforces data quality**: `SHACLGenerator` generates shapes based on what it observes in the data. If your data is missing a field, the generator won't know it was mandatory unless you explicitly inject the constraint (as shown in Step 3).
|
||||
- **Passing `ContextGraph` directly to SHACL validators**: The `_run_pyshacl` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object.
|
||||
- **Passing `ContextGraph` directly to SHACL validators**: The `run_shacl_validation` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object.
|
||||
- **Forgetting RDF serialization**: You must serialize your graph (often via a temporary file using `export_rdf`) before validating it.
|
||||
- **Treating validation as a one-time step**: Validation should be integrated as an automated step in your CI/CD pipeline or data ingestion flow, acting as a recurring gatekeeper rather than a one-off script.
|
||||
- **Ignoring validation reports**: A graph that does not conform must be remediated. Failing to review the `violation_count` and address the issues negates the purpose of SHACL validation.
|
||||
@@ -418,7 +418,7 @@ print(f"Violations after remediation: {report2.violation_count}")
|
||||
# rdfs True <- the entailment manufactured the type
|
||||
```
|
||||
|
||||
Mitigations: prefer not to declare `rdfs:range` on properties you intend to constrain with `sh:class`; when class membership is the thing under test, run validation without RDFS entailment (`inference="none"`); or express the check as a constraint the entailment cannot satisfy (for example a literal property constraint). Note the trade-off: with entailment off, `sh:targetClass` no longer reaches subclasses, so subclass hierarchies need explicit typing or inference-aware target selection. Semantica's own `_run_pyshacl` wrapper already calls pyshacl with `inference="none"`, so this pitfall only bites when calling `pyshacl.validate` directly with entailment enabled.
|
||||
Mitigations: prefer not to declare `rdfs:range` on properties you intend to constrain with `sh:class`; when class membership is the thing under test, run validation without RDFS entailment (`inference="none"`); or express the check as a constraint the entailment cannot satisfy (for example a literal property constraint). Note the trade-off: with entailment off, `sh:targetClass` no longer reaches subclasses, so subclass hierarchies need explicit typing or inference-aware target selection. Semantica's own `run_shacl_validation` wrapper already calls pyshacl with `inference="none"`, so this pitfall only bites when calling `pyshacl.validate` directly with entailment enabled.
|
||||
- **Trusting `conforms: True` without checking the inference mode**: an inference-enabled run can hide the exact violations the shapes were written to catch (see above). Record which inference mode validation ran under alongside the result, and re-run shape sets that contain `sh:class`/`sh:node` with entailment off before treating a pass as authoritative.
|
||||
|
||||
---
|
||||
@@ -435,7 +435,7 @@ A DoD CTI team enforces STIX-compatible constraints on a threat graph before sha
|
||||
from semantica.context import AgentContext, ContextGraph
|
||||
from semantica.vector_store import VectorStore
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
graph = ContextGraph()
|
||||
ctx = AgentContext(
|
||||
@@ -487,7 +487,7 @@ data_ttl = """
|
||||
<http://example.org/hammertoss> a ex:Malware .
|
||||
"""
|
||||
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
print(f"CTI graph conforms : {report.conforms}")
|
||||
print(f"Violations : {report.violation_count}")
|
||||
print(f"Warnings : {report.warning_count}")
|
||||
@@ -508,7 +508,7 @@ A SOC team validates zero-trust policy nodes before publishing them to the polic
|
||||
```python
|
||||
from semantica.context import ContextGraph
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
graph = ContextGraph()
|
||||
graph.add_node("policy-001", "Policy", "MFA Required for Tier-1 Resources",
|
||||
@@ -555,7 +555,7 @@ data_ttl = """
|
||||
<http://example.org/policy-002> a ex:Policy .
|
||||
"""
|
||||
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
print(f"Policy graph conforms: {report.conforms}")
|
||||
# Policy graph conforms: False
|
||||
|
||||
@@ -573,7 +573,7 @@ A clinical informatics team validates trial ontology nodes before loading them i
|
||||
|
||||
```python
|
||||
from semantica.ontology import LLMOntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
from semantica.export import export_rdf
|
||||
import tempfile, os
|
||||
|
||||
@@ -625,7 +625,7 @@ with open(tmp.name) as f:
|
||||
data_ttl = f.read()
|
||||
os.unlink(tmp.name)
|
||||
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
print(f"Trial data conforms: {report.conforms}")
|
||||
print(f"Warnings : {report.warning_count}")
|
||||
```
|
||||
@@ -639,7 +639,7 @@ A credit risk team validates every `LoanApplication` node against Basel III CRE2
|
||||
```python
|
||||
from semantica.context import ContextGraph
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
graph = ContextGraph()
|
||||
graph.add_node("loan-001", "LoanApplication", "Prime mortgage APP-2025-88421",
|
||||
@@ -684,7 +684,7 @@ data_ttl = """
|
||||
ex:ltv "0.65" .
|
||||
"""
|
||||
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
print(f"Loan portfolio conforms: {report.conforms}")
|
||||
# Loan portfolio conforms: False
|
||||
|
||||
@@ -714,14 +714,14 @@ Call this function as a pre-publish gate; exit code 1 blocks the pipeline.
|
||||
```python
|
||||
import sys
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
|
||||
shacl_gen = SHACLGenerator(base_uri="https://example.org/shapes/")
|
||||
shacl_graph = shacl_gen.generate(ontology)
|
||||
shacl_ttl = shacl_gen.serialize(shacl_graph, format="turtle")
|
||||
|
||||
report = _run_pyshacl(data_graph_str, shacl_ttl)
|
||||
report = run_shacl_validation(data_graph_str, shacl_ttl)
|
||||
|
||||
if not report.conforms:
|
||||
print(f"Graph validation FAILED — {report.violation_count} violation(s)")
|
||||
@@ -739,7 +739,6 @@ def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
|
||||
|
||||
- [Ontology Management](ontology) — generate the OWL ontology that SHACL shapes are derived from
|
||||
- [Reasoning & Rules](reasoning) — complement SHACL structural constraints with logical inference rules
|
||||
- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `_run_pyshacl` input
|
||||
- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `run_shacl_validation` input
|
||||
- [Conflict Resolution](conflict-resolution) — detect and resolve data conflicts before SHACL validation
|
||||
- [Change Management](change-management) — version-gate SHACL shapes alongside ontology versions
|
||||
|
||||
|
||||
+9
-9
@@ -6,9 +6,9 @@ accelerate==1.14.0 \
|
||||
# via
|
||||
# docling-ibm-models
|
||||
# docling-slim
|
||||
agno==2.8.7 \
|
||||
--hash=sha256:6a2763eb469163f7b79ab1da6ca2f22d8619f6b9d614574f975d9c12bb4323ea \
|
||||
--hash=sha256:d49396a2062ee6994ca82695b9bd1e1b95667fec432c544afa38133e564bf090
|
||||
agno==2.9.0 \
|
||||
--hash=sha256:7777674b3931b341fad4fcf02a61b185a08588c509101348facf87feb2144c0c \
|
||||
--hash=sha256:7d9c134703e3c2798023cd57dcb9caa8e1174f6914813f9b130becfc3521a46f
|
||||
# via semantica (pyproject.toml)
|
||||
agnoctl==0.1.3 \
|
||||
--hash=sha256:6fce1d2482b1f2e0a3d14b0a7c12fbd49d8df4f0bf0a4fd9fd91753cbff5efdc \
|
||||
@@ -403,9 +403,9 @@ boto3==1.43.69 \
|
||||
--hash=sha256:4eb494d05b2bd08a7eee61b8ac4c34745c99e9bbce435c91f8d15d372dd8c2db \
|
||||
--hash=sha256:76297a0b415849c63575ae08a4f1661b2dc8ee0100f104b86f98aa69b47fa2c7
|
||||
# via semantica (pyproject.toml)
|
||||
botocore==1.43.69 \
|
||||
--hash=sha256:5caa46b740d9a886137146ffbb69edb691f702bfe74c64e85621947ae00181fd \
|
||||
--hash=sha256:b1f0e01c53d6b84ee9c184ebf3636c3b3aef85e0ae8498c74afb8734ff224f87
|
||||
botocore==1.43.73 \
|
||||
--hash=sha256:068433028e011ccbeab1dd7c46b1090c24e378397693c66e67ca571176498daa \
|
||||
--hash=sha256:0fa1e63c24b3531be3e1bc1687a88b3be9e63a430153f24edd93efc162bb1c51
|
||||
# via
|
||||
# boto3
|
||||
# s3transfer
|
||||
@@ -1731,9 +1731,9 @@ google-crc32c==1.8.0 \
|
||||
# via
|
||||
# google-cloud-storage
|
||||
# google-resumable-media
|
||||
google-genai==2.17.0 \
|
||||
--hash=sha256:6b640a2390c82b4a240873eddb9f518c6d2c33244b2de16ed3d14526a6da7f57 \
|
||||
--hash=sha256:a4835563c60aee646c9c4b261c507aa4a624710d25017012d20dc65abf3d9a54
|
||||
google-genai==2.18.1 \
|
||||
--hash=sha256:36a5949233e64a60f6cc4521bff7a76b7c569d0aa227bbe9fa642213b8a3a3b2 \
|
||||
--hash=sha256:a1e2be75c16234adc6641afd1ad4dd44218c9eec005d938bdc428585a048918a
|
||||
# via semantica (pyproject.toml)
|
||||
google-resumable-media==2.10.1 \
|
||||
--hash=sha256:224975032ddb73f7ed9e2f0f4cc08ed1b06874c52d48cc8533e3eb72980b21a0 \
|
||||
|
||||
@@ -28,12 +28,35 @@ from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.helpers import ensure_directory, utc_now_iso, write_json_file
|
||||
from ..utils.helpers import ensure_directory, hash_data, utc_now_iso, write_json_file
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .rdf_exporter import SEMANTICA_NS, mint_entity_iri, mint_relationship_iri
|
||||
|
||||
|
||||
def _content_iri(prefix: str, payload: Any) -> str:
|
||||
"""Mint a document IRI from what was exported, not when.
|
||||
|
||||
Minting from ``utc_now_iso()`` gave every export of the same graph a new
|
||||
identity a few microseconds apart, so re-exporting an unchanged graph was
|
||||
never idempotent and merging exports duplicated every node (#1147). This
|
||||
mirrors ``mint_entity_iri`` (#1109): identical content hashes to the same
|
||||
IRI, and any change to the content changes it too. ``default=str`` keeps
|
||||
the hash defined for values ``json.dumps`` would otherwise reject, such as
|
||||
``datetime`` objects a caller may have left in the graph.
|
||||
|
||||
Args:
|
||||
prefix: IRI prefix the digest is appended to
|
||||
payload: JSON-serializable value whose content determines the digest
|
||||
|
||||
Returns:
|
||||
A stable IRI of the form ``{prefix}{16-hex-char digest}``
|
||||
"""
|
||||
canonical = json.dumps(payload, sort_keys=True, default=str)
|
||||
digest = hash_data(canonical)[:16]
|
||||
return f"{prefix}{digest}"
|
||||
|
||||
|
||||
def _is_jsonld_document(data: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Report whether a dictionary is already a JSON-LD document.
|
||||
@@ -230,7 +253,10 @@ class JSONExporter:
|
||||
- statistics: Statistics dictionary (optional)
|
||||
file_path: Output JSON file path
|
||||
format: Export format - 'json' or 'json-ld' (default: self.format)
|
||||
**options: Additional options passed to conversion methods
|
||||
**options: Additional options passed to conversion methods:
|
||||
- graph_uri: Caller-supplied IRI for the graph node when
|
||||
format='json-ld', overriding the default content-derived
|
||||
IRI (see #1147)
|
||||
|
||||
Example:
|
||||
>>> kg = {
|
||||
@@ -401,7 +427,9 @@ class JSONExporter:
|
||||
data: Data to convert (dict, list, or any value)
|
||||
include_metadata: Whether to include metadata (default: True)
|
||||
include_provenance: Whether to include provenance (default: True)
|
||||
**options: Additional options passed to knowledge graph conversion
|
||||
**options: Additional options passed to knowledge graph conversion:
|
||||
- document_uri: Caller-supplied IRI for the document node,
|
||||
overriding the default content-derived IRI (see #1147)
|
||||
|
||||
Returns:
|
||||
Dictionary in JSON-LD format with @context, @graph/@value, and metadata
|
||||
@@ -451,13 +479,17 @@ class JSONExporter:
|
||||
|
||||
# Add metadata and provenance if requested
|
||||
if include_metadata:
|
||||
self._attach_document_metadata(jsonld, include_provenance)
|
||||
self._attach_document_metadata(
|
||||
jsonld, include_provenance, options.get("document_uri")
|
||||
)
|
||||
|
||||
return jsonld
|
||||
|
||||
@staticmethod
|
||||
def _attach_document_metadata(
|
||||
jsonld: Dict[str, Any], include_provenance: bool
|
||||
jsonld: Dict[str, Any],
|
||||
include_provenance: bool,
|
||||
document_uri: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Attach the export's own metadata without naming the graph.
|
||||
@@ -473,6 +505,9 @@ class JSONExporter:
|
||||
Args:
|
||||
jsonld: Document being built, modified in place
|
||||
include_provenance: Whether to record how and when it was exported
|
||||
document_uri: Caller-supplied IRI for the document node. Falls back
|
||||
to a content-derived IRI (#1147) so re-exporting unchanged data
|
||||
is idempotent instead of minting a new identity every time.
|
||||
"""
|
||||
# A caller may hand us a document that is deliberately a named graph.
|
||||
# That name is theirs to keep, but our own statements must not end up
|
||||
@@ -483,7 +518,10 @@ class JSONExporter:
|
||||
# Do not overwrite an identifier the payload already carries: the
|
||||
# knowledge-graph conversion names its own document node.
|
||||
if "@id" not in jsonld or payload_is_named_graph:
|
||||
document["@id"] = f"https://semantica.dev/data/{utc_now_iso()}"
|
||||
content = {key: value for key, value in jsonld.items() if key != "@context"}
|
||||
document["@id"] = document_uri or _content_iri(
|
||||
"https://semantica.dev/data/", content
|
||||
)
|
||||
if include_provenance:
|
||||
document["semantica:exportedAt"] = utc_now_iso()
|
||||
document["semantica:format"] = "json-ld"
|
||||
@@ -553,7 +591,9 @@ class JSONExporter:
|
||||
- entities: List of entity dictionaries
|
||||
- relationships: List of relationship dictionaries
|
||||
- metadata: Metadata dictionary (optional)
|
||||
**options: Additional options (unused)
|
||||
**options: Additional options:
|
||||
- graph_uri: Caller-supplied IRI for the graph node,
|
||||
overriding the default content-derived IRI (see #1147)
|
||||
|
||||
Returns:
|
||||
Dictionary in JSON-LD format with @context, @id, @type, and graph data
|
||||
@@ -566,7 +606,11 @@ class JSONExporter:
|
||||
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
|
||||
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
|
||||
},
|
||||
"@id": f"https://semantica.dev/graph/{utc_now_iso()}",
|
||||
# Minted from the graph's own content rather than the wall clock
|
||||
# (#1147): re-exporting an unchanged graph must produce the same
|
||||
# subject, or merging repeated exports duplicates every node.
|
||||
"@id": options.get("graph_uri")
|
||||
or _content_iri("https://semantica.dev/graph/", kg),
|
||||
"@type": "semantica:KnowledgeGraph",
|
||||
}
|
||||
|
||||
|
||||
@@ -159,6 +159,7 @@ from .ontology_validator import (
|
||||
SHACLValidationReport,
|
||||
SHACLViolation,
|
||||
ValidationResult,
|
||||
run_shacl_validation,
|
||||
validate_ontology,
|
||||
)
|
||||
from .owl_generator import OWLGenerator
|
||||
@@ -192,6 +193,7 @@ __all__ = [
|
||||
"PropertyShape",
|
||||
"SHACLValidationReport",
|
||||
"SHACLViolation",
|
||||
"run_shacl_validation",
|
||||
# OWL/RDF generation
|
||||
"OWLGenerator",
|
||||
# Requirements and competency questions
|
||||
|
||||
@@ -145,14 +145,14 @@ class SHACLValidationReport:
|
||||
}
|
||||
|
||||
|
||||
def _run_pyshacl(
|
||||
def run_shacl_validation(
|
||||
data_graph_str: str,
|
||||
shacl_str: str,
|
||||
data_graph_format: str = "turtle",
|
||||
shacl_format: str = "turtle",
|
||||
) -> SHACLValidationReport:
|
||||
"""
|
||||
Run pyshacl validation and return a structured SHACLValidationReport.
|
||||
Run pySHACL validation and return a structured SHACLValidationReport.
|
||||
|
||||
Args:
|
||||
data_graph_str: Serialized data graph string.
|
||||
@@ -272,6 +272,21 @@ def _run_pyshacl(
|
||||
raw_report=results_text,
|
||||
)
|
||||
|
||||
|
||||
def _run_pyshacl(
|
||||
data_graph_str: str,
|
||||
shacl_str: str,
|
||||
data_graph_format: str = "turtle",
|
||||
shacl_format: str = "turtle",
|
||||
) -> SHACLValidationReport:
|
||||
"""Backward-compatible alias for :func:`run_shacl_validation`."""
|
||||
return run_shacl_validation(
|
||||
data_graph_str,
|
||||
shacl_str,
|
||||
data_graph_format=data_graph_format,
|
||||
shacl_format=shacl_format,
|
||||
)
|
||||
|
||||
@dataclass
|
||||
class ValidationResult:
|
||||
"""Result of an ontology validation operation."""
|
||||
|
||||
@@ -779,9 +779,11 @@ def extract_entities_huggingface(
|
||||
"""
|
||||
loader = HuggingFaceModelLoader(device=device)
|
||||
# Pass kwargs (like aggregation_strategy) to load_ner_model
|
||||
model_obj = loader.load_ner_model(model, **kwargs)
|
||||
loader_kwargs = {
|
||||
key: value for key, value in kwargs.items() if key != "huggingface_model"
|
||||
}
|
||||
model_obj = loader.load_ner_model(model, **loader_kwargs)
|
||||
results = loader.extract_entities(model_obj, text)
|
||||
|
||||
entities = []
|
||||
|
||||
# Check if manual aggregation is needed (raw IOB tags detected)
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
"""The document IRI of a JSON-LD export must depend on content, not the clock
|
||||
(issue #1147).
|
||||
|
||||
``_convert_kg_to_jsonld`` minted the graph's ``@id`` from ``utc_now_iso()``,
|
||||
and the generic ``_attach_document_metadata`` path did the same for a plain
|
||||
document ``@id``. Exporting an unchanged graph therefore produced a new
|
||||
subject every time: three exports of one one-entity graph merged into 3
|
||||
``semantica:KnowledgeGraph`` nodes and 15 triples for what should have been a
|
||||
single graph. Neither identifier resolves and the timestamp is already
|
||||
recorded correctly in ``semantica:exportedAt``, so the fix mints the IRI from
|
||||
the exported content instead (mirroring ``mint_entity_iri``, #1109), with an
|
||||
optional caller-supplied override for callers who already name their graphs.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from rdflib import RDF, Graph, URIRef
|
||||
|
||||
from semantica.export.json_exporter import JSONExporter
|
||||
|
||||
KG = {
|
||||
"entities": [{"id": "https://example.org/e1", "text": "Acme Corp", "type": "ORG"}],
|
||||
"relationships": [],
|
||||
}
|
||||
|
||||
OTHER_KG = {
|
||||
"entities": [
|
||||
{"id": "https://example.org/e1", "text": "Acme Corp Renamed", "type": "ORG"}
|
||||
],
|
||||
"relationships": [],
|
||||
}
|
||||
|
||||
|
||||
def _export(kg, tmp_path, name="out.jsonld", **options):
|
||||
path = tmp_path / name
|
||||
JSONExporter().export_knowledge_graph(kg, path, format="json-ld", **options)
|
||||
return path
|
||||
|
||||
|
||||
def test_reexporting_an_unchanged_graph_is_idempotent(tmp_path):
|
||||
"""The whole point of an identifier: same content, same @id."""
|
||||
first = json.loads(_export(KG, tmp_path, "a.jsonld").read_text())
|
||||
second = json.loads(_export(KG, tmp_path, "b.jsonld").read_text())
|
||||
|
||||
assert first["@id"] == second["@id"]
|
||||
|
||||
|
||||
def test_a_changed_graph_gets_a_different_id(tmp_path):
|
||||
unchanged = json.loads(_export(KG, tmp_path, "a.jsonld").read_text())
|
||||
changed = json.loads(_export(OTHER_KG, tmp_path, "b.jsonld").read_text())
|
||||
|
||||
assert unchanged["@id"] != changed["@id"]
|
||||
|
||||
|
||||
def test_merging_repeated_exports_yields_one_graph_node(tmp_path):
|
||||
"""Regression for the exact repro in #1147: churn no longer multiplies nodes."""
|
||||
merged = Graph()
|
||||
for i in range(3):
|
||||
path = _export(KG, tmp_path, f"churn{i}.jsonld")
|
||||
merged.parse(str(path), format="json-ld")
|
||||
|
||||
# Exactly one subject typed as a KnowledgeGraph, regardless of how many
|
||||
# times the unchanged graph was exported and merged.
|
||||
kg_nodes = set(
|
||||
merged.subjects(RDF.type, URIRef("https://semantica.dev/ns#KnowledgeGraph"))
|
||||
)
|
||||
assert len(kg_nodes) == 1
|
||||
|
||||
entity_nodes = set(
|
||||
merged.subjects(RDF.type, URIRef("https://semantica.dev/vocab/ORG"))
|
||||
)
|
||||
assert len(entity_nodes) == 1
|
||||
|
||||
|
||||
def test_exported_at_still_varies_between_exports(tmp_path):
|
||||
"""Identity is now content-derived, but provenance still records each run."""
|
||||
first = json.loads(_export(KG, tmp_path, "a.jsonld").read_text())
|
||||
second = json.loads(_export(KG, tmp_path, "b.jsonld").read_text())
|
||||
|
||||
assert first["@id"] == second["@id"]
|
||||
assert first["semantica:exportedAt"] != second["semantica:exportedAt"]
|
||||
|
||||
|
||||
def test_caller_supplied_graph_uri_is_honored(tmp_path):
|
||||
path = _export(KG, tmp_path, graph_uri="https://example.org/my-graph")
|
||||
document = json.loads(path.read_text())
|
||||
|
||||
assert document["@id"] == "https://example.org/my-graph"
|
||||
|
||||
|
||||
def _document_node_id(document):
|
||||
"""The generic (non-knowledge-graph) path hangs its own @id off a member
|
||||
of @graph rather than the top level, to avoid re-creating the named-graph
|
||||
bug fixed by #1145. Find that member and return its @id."""
|
||||
for node in document["@graph"]:
|
||||
if "semantica:exportedAt" in node:
|
||||
return node["@id"]
|
||||
raise AssertionError(f"no document metadata node in @graph: {document}")
|
||||
|
||||
|
||||
def test_caller_supplied_document_uri_is_honored_for_a_generic_export(tmp_path):
|
||||
payload = {"note": "no entities or relationships here"}
|
||||
path = tmp_path / "generic.jsonld"
|
||||
JSONExporter().export(
|
||||
payload, path, format="json-ld", document_uri="https://example.org/my-doc"
|
||||
)
|
||||
document = json.loads(path.read_text())
|
||||
|
||||
assert _document_node_id(document) == "https://example.org/my-doc"
|
||||
|
||||
|
||||
def test_generic_document_id_is_also_content_derived(tmp_path):
|
||||
"""The non-knowledge-graph path (_attach_document_metadata) gets the same fix."""
|
||||
payload = {"note": "plain data, no @id of its own"}
|
||||
|
||||
first = tmp_path / "a.jsonld"
|
||||
second = tmp_path / "b.jsonld"
|
||||
JSONExporter().export(payload, first, format="json-ld")
|
||||
JSONExporter().export(dict(payload), second, format="json-ld")
|
||||
|
||||
first_id = _document_node_id(json.loads(first.read_text()))
|
||||
second_id = _document_node_id(json.loads(second.read_text()))
|
||||
assert first_id == second_id
|
||||
|
||||
|
||||
def test_document_id_still_differs_for_different_generic_payloads(tmp_path):
|
||||
a = tmp_path / "a.jsonld"
|
||||
b = tmp_path / "b.jsonld"
|
||||
JSONExporter().export({"note": "one"}, a, format="json-ld")
|
||||
JSONExporter().export({"note": "two"}, b, format="json-ld")
|
||||
|
||||
a_id = _document_node_id(json.loads(a.read_text()))
|
||||
b_id = _document_node_id(json.loads(b.read_text()))
|
||||
assert a_id != b_id
|
||||
@@ -107,20 +107,31 @@ def test_exported_timestamp_survives_a_timezone_qualified_sparql_filter():
|
||||
assert len(rows) == 1, "the export was dropped by a timezone-qualified filter"
|
||||
|
||||
|
||||
def test_document_iri_carrying_an_offset_is_a_valid_iri():
|
||||
"""The offset puts '+' and ':' in the @id; both are legal in a path."""
|
||||
def test_document_iri_is_a_valid_iri():
|
||||
"""The graph @id must be a valid IRI regardless of how it is minted.
|
||||
|
||||
Before #1147, this @id was minted from the offset-carrying timestamp
|
||||
itself (``+00:00`` interpolated straight into the path), so this test
|
||||
asserted the offset survived without breaking IRI validity. #1147 mints
|
||||
the @id from the graph's content instead, so the timestamp no longer
|
||||
appears here at all — it stays in ``semantica:exportedAt`` (still
|
||||
offset-aware, per ``test_jsonld_export_timestamp_is_offset_aware`` above).
|
||||
What's left worth guarding is the general case: whatever the @id is
|
||||
minted from, it has to be a valid IRI that round-trips through RDF.
|
||||
"""
|
||||
rdflib = pytest.importorskip("rdflib")
|
||||
|
||||
document_iri = JSONExporter()._convert_kg_to_jsonld(KG)["@id"]
|
||||
assert "+00:00" in document_iri
|
||||
assert rdflib.term._is_valid_uri(document_iri)
|
||||
|
||||
graph = rdflib.Graph()
|
||||
graph.add((
|
||||
rdflib.URIRef(document_iri),
|
||||
rdflib.RDF.type,
|
||||
rdflib.URIRef("https://semantica.dev/ns#KnowledgeGraph"),
|
||||
))
|
||||
graph.add(
|
||||
(
|
||||
rdflib.URIRef(document_iri),
|
||||
rdflib.RDF.type,
|
||||
rdflib.URIRef("https://semantica.dev/ns#KnowledgeGraph"),
|
||||
)
|
||||
)
|
||||
reparsed = rdflib.Graph().parse(data=graph.serialize(format="nt"), format="nt")
|
||||
assert document_iri in {str(s) for s in reparsed.subjects()}
|
||||
|
||||
|
||||
@@ -525,6 +525,65 @@ class TestSHACLHierarchicalAndValidation(unittest.TestCase):
|
||||
self.assertEqual(mc[0].max_count, 2)
|
||||
|
||||
# 33
|
||||
def test_public_run_shacl_validation_api(self):
|
||||
"""The public API validates data and retains the legacy alias."""
|
||||
try:
|
||||
import pyshacl # noqa: F401
|
||||
import rdflib # noqa: F401
|
||||
except ImportError:
|
||||
self.skipTest("pyshacl/rdflib not installed")
|
||||
from semantica.ontology import run_shacl_validation
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
|
||||
data = "@prefix ex: <http://example.org/> . ex:alice a ex:Person ."
|
||||
shacl = """
|
||||
@prefix ex: <http://example.org/> .
|
||||
@prefix sh: <http://www.w3.org/ns/shacl#> .
|
||||
ex:PersonShape a sh:NodeShape ; sh:targetClass ex:Person ;
|
||||
sh:property [ sh:path ex:name ; sh:minCount 1 ] .
|
||||
"""
|
||||
public_report = run_shacl_validation(data, shacl)
|
||||
legacy_report = _run_pyshacl(data, shacl)
|
||||
self.assertFalse(public_report.conforms)
|
||||
self.assertEqual(public_report.violation_count, 1)
|
||||
self.assertEqual(legacy_report.conforms, public_report.conforms)
|
||||
self.assertEqual(legacy_report.violation_count, public_report.violation_count)
|
||||
self.assertEqual(
|
||||
[
|
||||
(v.focus_node, v.result_path, v.constraint, v.severity, v.message)
|
||||
for v in legacy_report.violations
|
||||
],
|
||||
[
|
||||
(v.focus_node, v.result_path, v.constraint, v.severity, v.message)
|
||||
for v in public_report.violations
|
||||
],
|
||||
)
|
||||
|
||||
# 34
|
||||
def test_public_run_shacl_validation_conforming_graph(self):
|
||||
"""The public API reports a valid graph without violations."""
|
||||
try:
|
||||
import pyshacl # noqa: F401
|
||||
import rdflib # noqa: F401
|
||||
except ImportError:
|
||||
self.skipTest("pyshacl/rdflib not installed")
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
data = """
|
||||
@prefix ex: <http://example.org/> .
|
||||
ex:alice a ex:Person ; ex:name "Alice" .
|
||||
"""
|
||||
shacl = """
|
||||
@prefix ex: <http://example.org/> .
|
||||
@prefix sh: <http://www.w3.org/ns/shacl#> .
|
||||
ex:PersonShape a sh:NodeShape ; sh:targetClass ex:Person ;
|
||||
sh:property [ sh:path ex:name ; sh:minCount 1 ] .
|
||||
"""
|
||||
|
||||
report = run_shacl_validation(data, shacl)
|
||||
|
||||
self.assertTrue(report.conforms)
|
||||
self.assertEqual(report.violation_count, 0)
|
||||
def test_shacl_violation_to_dict(self):
|
||||
from semantica.ontology.ontology_validator import SHACLViolation
|
||||
v = SHACLViolation(
|
||||
|
||||
Reference in New Issue
Block a user