mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Merge pull request #1189 from mikemikimike/feat/public-shacl-validation
feat(ontology): expose public SHACL validation API
This commit is contained in:
@@ -8,7 +8,7 @@ icon: "shield-check"
|
||||
|
||||
SHACL (Shapes Constraint Language) is a standard for validating graph-based data. While an ontology defines the conceptual *schema* (the "what" exists in your domain), SHACL defines the structural *rules and constraints* (the "how" it should be structured).
|
||||
|
||||
In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and `_run_pyshacl` evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated.
|
||||
In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and the public `run_shacl_validation` function evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated. The historical `_run_pyshacl` name remains available as a compatibility alias.
|
||||
|
||||
## Why Use SHACL Validation?
|
||||
|
||||
@@ -55,7 +55,7 @@ Let's look at a simple, universally understood example: ensuring every `Employee
|
||||
```python
|
||||
from semantica.context import ContextGraph
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
# 1. Prepare your data graph
|
||||
graph = ContextGraph()
|
||||
@@ -95,7 +95,7 @@ data_ttl = """
|
||||
"""
|
||||
|
||||
# 5. Run Validation
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
|
||||
# 6. Analyze the Report
|
||||
print(f"Graph conforms: {report.conforms}")
|
||||
@@ -265,10 +265,10 @@ cve_id_shape = NodeShape(
|
||||
|
||||
## Step 4 — Run validation and read the report
|
||||
|
||||
Serialize the graph to RDF, then run `_run_pyshacl` against the shapes.
|
||||
Serialize the graph to RDF, then run `run_shacl_validation` against the shapes.
|
||||
|
||||
```python
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
# Prepare your RDF data string (since export_rdf primarily exports structural metadata,
|
||||
# you typically serialize your custom data graph to Turtle using rdflib or similar).
|
||||
@@ -281,7 +281,7 @@ data_ttl = """
|
||||
"""
|
||||
|
||||
# Run SHACL validation
|
||||
report = _run_pyshacl(
|
||||
report = run_shacl_validation(
|
||||
data_ttl,
|
||||
shacl_ttl,
|
||||
data_graph_format="turtle",
|
||||
@@ -366,8 +366,8 @@ print(f"Malware nodes missing 'family': {len(missing_family)}")
|
||||
# e.g. graph.update_node(node_id, {"family": "UNKNOWN — requires triage"})
|
||||
|
||||
# After remediation, re-run validation to confirm the fix
|
||||
# (re-export the patched graph to Turtle first, then call _run_pyshacl again)
|
||||
report2 = _run_pyshacl(patched_data_ttl, shacl_ttl)
|
||||
# (re-export the patched graph to Turtle first, then call run_shacl_validation again)
|
||||
report2 = run_shacl_validation(patched_data_ttl, shacl_ttl)
|
||||
print(f"Violations after remediation: {report2.violation_count}")
|
||||
# Violations after remediation: 0
|
||||
```
|
||||
@@ -377,7 +377,7 @@ print(f"Violations after remediation: {report2.violation_count}")
|
||||
## Common Pitfalls
|
||||
|
||||
- **Assuming the ontology automatically enforces data quality**: `SHACLGenerator` generates shapes based on what it observes in the data. If your data is missing a field, the generator won't know it was mandatory unless you explicitly inject the constraint (as shown in Step 3).
|
||||
- **Passing `ContextGraph` directly to SHACL validators**: The `_run_pyshacl` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object.
|
||||
- **Passing `ContextGraph` directly to SHACL validators**: The `run_shacl_validation` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object.
|
||||
- **Forgetting RDF serialization**: You must serialize your graph (often via a temporary file using `export_rdf`) before validating it.
|
||||
- **Treating validation as a one-time step**: Validation should be integrated as an automated step in your CI/CD pipeline or data ingestion flow, acting as a recurring gatekeeper rather than a one-off script.
|
||||
- **Ignoring validation reports**: A graph that does not conform must be remediated. Failing to review the `violation_count` and address the issues negates the purpose of SHACL validation.
|
||||
@@ -418,7 +418,7 @@ print(f"Violations after remediation: {report2.violation_count}")
|
||||
# rdfs True <- the entailment manufactured the type
|
||||
```
|
||||
|
||||
Mitigations: prefer not to declare `rdfs:range` on properties you intend to constrain with `sh:class`; when class membership is the thing under test, run validation without RDFS entailment (`inference="none"`); or express the check as a constraint the entailment cannot satisfy (for example a literal property constraint). Note the trade-off: with entailment off, `sh:targetClass` no longer reaches subclasses, so subclass hierarchies need explicit typing or inference-aware target selection. Semantica's own `_run_pyshacl` wrapper already calls pyshacl with `inference="none"`, so this pitfall only bites when calling `pyshacl.validate` directly with entailment enabled.
|
||||
Mitigations: prefer not to declare `rdfs:range` on properties you intend to constrain with `sh:class`; when class membership is the thing under test, run validation without RDFS entailment (`inference="none"`); or express the check as a constraint the entailment cannot satisfy (for example a literal property constraint). Note the trade-off: with entailment off, `sh:targetClass` no longer reaches subclasses, so subclass hierarchies need explicit typing or inference-aware target selection. Semantica's own `run_shacl_validation` wrapper already calls pyshacl with `inference="none"`, so this pitfall only bites when calling `pyshacl.validate` directly with entailment enabled.
|
||||
- **Trusting `conforms: True` without checking the inference mode**: an inference-enabled run can hide the exact violations the shapes were written to catch (see above). Record which inference mode validation ran under alongside the result, and re-run shape sets that contain `sh:class`/`sh:node` with entailment off before treating a pass as authoritative.
|
||||
|
||||
---
|
||||
@@ -435,7 +435,7 @@ A DoD CTI team enforces STIX-compatible constraints on a threat graph before sha
|
||||
from semantica.context import AgentContext, ContextGraph
|
||||
from semantica.vector_store import VectorStore
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
graph = ContextGraph()
|
||||
ctx = AgentContext(
|
||||
@@ -487,7 +487,7 @@ data_ttl = """
|
||||
<http://example.org/hammertoss> a ex:Malware .
|
||||
"""
|
||||
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
print(f"CTI graph conforms : {report.conforms}")
|
||||
print(f"Violations : {report.violation_count}")
|
||||
print(f"Warnings : {report.warning_count}")
|
||||
@@ -508,7 +508,7 @@ A SOC team validates zero-trust policy nodes before publishing them to the polic
|
||||
```python
|
||||
from semantica.context import ContextGraph
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
graph = ContextGraph()
|
||||
graph.add_node("policy-001", "Policy", "MFA Required for Tier-1 Resources",
|
||||
@@ -555,7 +555,7 @@ data_ttl = """
|
||||
<http://example.org/policy-002> a ex:Policy .
|
||||
"""
|
||||
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
print(f"Policy graph conforms: {report.conforms}")
|
||||
# Policy graph conforms: False
|
||||
|
||||
@@ -573,7 +573,7 @@ A clinical informatics team validates trial ontology nodes before loading them i
|
||||
|
||||
```python
|
||||
from semantica.ontology import LLMOntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
from semantica.export import export_rdf
|
||||
import tempfile, os
|
||||
|
||||
@@ -625,7 +625,7 @@ with open(tmp.name) as f:
|
||||
data_ttl = f.read()
|
||||
os.unlink(tmp.name)
|
||||
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
print(f"Trial data conforms: {report.conforms}")
|
||||
print(f"Warnings : {report.warning_count}")
|
||||
```
|
||||
@@ -639,7 +639,7 @@ A credit risk team validates every `LoanApplication` node against Basel III CRE2
|
||||
```python
|
||||
from semantica.context import ContextGraph
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
graph = ContextGraph()
|
||||
graph.add_node("loan-001", "LoanApplication", "Prime mortgage APP-2025-88421",
|
||||
@@ -684,7 +684,7 @@ data_ttl = """
|
||||
ex:ltv "0.65" .
|
||||
"""
|
||||
|
||||
report = _run_pyshacl(data_ttl, shacl_ttl)
|
||||
report = run_shacl_validation(data_ttl, shacl_ttl)
|
||||
print(f"Loan portfolio conforms: {report.conforms}")
|
||||
# Loan portfolio conforms: False
|
||||
|
||||
@@ -714,14 +714,14 @@ Call this function as a pre-publish gate; exit code 1 blocks the pipeline.
|
||||
```python
|
||||
import sys
|
||||
from semantica.ontology import OntologyGenerator, SHACLGenerator
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
|
||||
shacl_gen = SHACLGenerator(base_uri="https://example.org/shapes/")
|
||||
shacl_graph = shacl_gen.generate(ontology)
|
||||
shacl_ttl = shacl_gen.serialize(shacl_graph, format="turtle")
|
||||
|
||||
report = _run_pyshacl(data_graph_str, shacl_ttl)
|
||||
report = run_shacl_validation(data_graph_str, shacl_ttl)
|
||||
|
||||
if not report.conforms:
|
||||
print(f"Graph validation FAILED — {report.violation_count} violation(s)")
|
||||
@@ -739,7 +739,6 @@ def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
|
||||
|
||||
- [Ontology Management](ontology) — generate the OWL ontology that SHACL shapes are derived from
|
||||
- [Reasoning & Rules](reasoning) — complement SHACL structural constraints with logical inference rules
|
||||
- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `_run_pyshacl` input
|
||||
- [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `run_shacl_validation` input
|
||||
- [Conflict Resolution](conflict-resolution) — detect and resolve data conflicts before SHACL validation
|
||||
- [Change Management](change-management) — version-gate SHACL shapes alongside ontology versions
|
||||
|
||||
|
||||
@@ -159,6 +159,7 @@ from .ontology_validator import (
|
||||
SHACLValidationReport,
|
||||
SHACLViolation,
|
||||
ValidationResult,
|
||||
run_shacl_validation,
|
||||
validate_ontology,
|
||||
)
|
||||
from .owl_generator import OWLGenerator
|
||||
@@ -192,6 +193,7 @@ __all__ = [
|
||||
"PropertyShape",
|
||||
"SHACLValidationReport",
|
||||
"SHACLViolation",
|
||||
"run_shacl_validation",
|
||||
# OWL/RDF generation
|
||||
"OWLGenerator",
|
||||
# Requirements and competency questions
|
||||
|
||||
@@ -145,14 +145,14 @@ class SHACLValidationReport:
|
||||
}
|
||||
|
||||
|
||||
def _run_pyshacl(
|
||||
def run_shacl_validation(
|
||||
data_graph_str: str,
|
||||
shacl_str: str,
|
||||
data_graph_format: str = "turtle",
|
||||
shacl_format: str = "turtle",
|
||||
) -> SHACLValidationReport:
|
||||
"""
|
||||
Run pyshacl validation and return a structured SHACLValidationReport.
|
||||
Run pySHACL validation and return a structured SHACLValidationReport.
|
||||
|
||||
Args:
|
||||
data_graph_str: Serialized data graph string.
|
||||
@@ -272,6 +272,21 @@ def _run_pyshacl(
|
||||
raw_report=results_text,
|
||||
)
|
||||
|
||||
|
||||
def _run_pyshacl(
|
||||
data_graph_str: str,
|
||||
shacl_str: str,
|
||||
data_graph_format: str = "turtle",
|
||||
shacl_format: str = "turtle",
|
||||
) -> SHACLValidationReport:
|
||||
"""Backward-compatible alias for :func:`run_shacl_validation`."""
|
||||
return run_shacl_validation(
|
||||
data_graph_str,
|
||||
shacl_str,
|
||||
data_graph_format=data_graph_format,
|
||||
shacl_format=shacl_format,
|
||||
)
|
||||
|
||||
@dataclass
|
||||
class ValidationResult:
|
||||
"""Result of an ontology validation operation."""
|
||||
|
||||
@@ -525,6 +525,65 @@ class TestSHACLHierarchicalAndValidation(unittest.TestCase):
|
||||
self.assertEqual(mc[0].max_count, 2)
|
||||
|
||||
# 33
|
||||
def test_public_run_shacl_validation_api(self):
|
||||
"""The public API validates data and retains the legacy alias."""
|
||||
try:
|
||||
import pyshacl # noqa: F401
|
||||
import rdflib # noqa: F401
|
||||
except ImportError:
|
||||
self.skipTest("pyshacl/rdflib not installed")
|
||||
from semantica.ontology import run_shacl_validation
|
||||
from semantica.ontology.ontology_validator import _run_pyshacl
|
||||
|
||||
data = "@prefix ex: <http://example.org/> . ex:alice a ex:Person ."
|
||||
shacl = """
|
||||
@prefix ex: <http://example.org/> .
|
||||
@prefix sh: <http://www.w3.org/ns/shacl#> .
|
||||
ex:PersonShape a sh:NodeShape ; sh:targetClass ex:Person ;
|
||||
sh:property [ sh:path ex:name ; sh:minCount 1 ] .
|
||||
"""
|
||||
public_report = run_shacl_validation(data, shacl)
|
||||
legacy_report = _run_pyshacl(data, shacl)
|
||||
self.assertFalse(public_report.conforms)
|
||||
self.assertEqual(public_report.violation_count, 1)
|
||||
self.assertEqual(legacy_report.conforms, public_report.conforms)
|
||||
self.assertEqual(legacy_report.violation_count, public_report.violation_count)
|
||||
self.assertEqual(
|
||||
[
|
||||
(v.focus_node, v.result_path, v.constraint, v.severity, v.message)
|
||||
for v in legacy_report.violations
|
||||
],
|
||||
[
|
||||
(v.focus_node, v.result_path, v.constraint, v.severity, v.message)
|
||||
for v in public_report.violations
|
||||
],
|
||||
)
|
||||
|
||||
# 34
|
||||
def test_public_run_shacl_validation_conforming_graph(self):
|
||||
"""The public API reports a valid graph without violations."""
|
||||
try:
|
||||
import pyshacl # noqa: F401
|
||||
import rdflib # noqa: F401
|
||||
except ImportError:
|
||||
self.skipTest("pyshacl/rdflib not installed")
|
||||
from semantica.ontology import run_shacl_validation
|
||||
|
||||
data = """
|
||||
@prefix ex: <http://example.org/> .
|
||||
ex:alice a ex:Person ; ex:name "Alice" .
|
||||
"""
|
||||
shacl = """
|
||||
@prefix ex: <http://example.org/> .
|
||||
@prefix sh: <http://www.w3.org/ns/shacl#> .
|
||||
ex:PersonShape a sh:NodeShape ; sh:targetClass ex:Person ;
|
||||
sh:property [ sh:path ex:name ; sh:minCount 1 ] .
|
||||
"""
|
||||
|
||||
report = run_shacl_validation(data, shacl)
|
||||
|
||||
self.assertTrue(report.conforms)
|
||||
self.assertEqual(report.violation_count, 0)
|
||||
def test_shacl_violation_to_dict(self):
|
||||
from semantica.ontology.ontology_validator import SHACLViolation
|
||||
v = SHACLViolation(
|
||||
|
||||
Reference in New Issue
Block a user