mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
* feat(#401): temporal provenance, OWL-Time export, stable snapshot schema - ProvenanceTracker: auto-attach recorded_at (UTC) to every new record; add query_recorded_between(), revision_history(), export_audit_log() - RDFExporter.export_to_rdf: add include_temporal + time_axis params; emit OWL-Time triples (time:Interval, time:Instant, inXSDDateTimeStamp) for relationships with valid_from/valid_until; TemporalBound.OPEN represented via semantica:openEndedInterval instead of time:hasEnd - TemporalVersionManager.create_snapshot: stamp format_version "1.0" on every snapshot; add validate_snapshot() and migrate_snapshot() - New: semantica/kg/schemas/temporal_snapshot_v1.json (JSON Schema draft-2020) - Tests: 28 new tests covering all acceptance criteria (451 passing, 0 failed) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(#401): add changelog entry for temporal provenance & export Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Mohd Kaif
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
c66f8160dd
commit
9e68266563
@@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
- **Temporal Provenance & Export** (PR #401 by @KaifAhmad1):
|
||||
- **Transaction time on provenance records** (`semantica/kg/provenance_tracker.py`): `track_entity()` now automatically attaches `recorded_at = datetime.now(UTC).isoformat()` to every new record — no opt-in required. Existing records without `recorded_at` continue to work in all existing query methods (treated as unknown, not an error). Added `query_recorded_between(start, end) -> list` returning all provenance records whose `recorded_at` falls within the inclusive range; accepts `datetime` objects or ISO strings including trailing `Z`.
|
||||
- **Fact revision audit trail** (`semantica/kg/provenance_tracker.py`): Added `revision_history(fact_id) -> list` returning the complete revision chain ordered by `recorded_at` ascending; each entry includes `version` (int, 1-based), `valid_from`, `valid_until`, `recorded_at`, `author`, and optionally `revision_type`/`supersedes`; returns `[]` for unknown facts (never raises). Added `export_audit_log(fact_ids, format) -> str` supporting `"json"` (pretty-printed) and `"csv"` (with header row) formats.
|
||||
- **OWL-Time RDF export** (`semantica/export/rdf_exporter.py`): `export_to_rdf()` gains `include_temporal: bool = False` and `time_axis: str = "valid"` parameters. When `include_temporal=True`, emits OWL-Time triples (`http://www.w3.org/2006/time#`) for every relationship carrying `valid_from`/`valid_until` — a `time:Interval` node linked via `time:hasTime`, `time:hasBeginning`/`time:hasEnd` with `time:Instant` nodes, and `time:inXSDDateTimeStamp` values. `time_axis` controls which axis is exported: `"valid"`, `"transaction"`, or `"both"`. Relationships without temporal metadata are unaffected. Default `include_temporal=False` produces output identical to current behavior. **Design decision for `TemporalBound.OPEN`**: OWL-Time has no standard predicate for "no known end date" — `time:hasEnd` is omitted and `semantica:openEndedInterval "true"^^xsd:boolean` is emitted on the interval node instead. Output parses without errors in rdflib.
|
||||
- **Stable snapshot serialization format** (`semantica/kg/temporal_query.py`, new `semantica/kg/schemas/temporal_snapshot_v1.json`): `create_snapshot()` now stamps `"format_version": "1.0"` on every snapshot. Added `validate_snapshot(snapshot) -> bool` — validates required fields (`format_version`, `label`, `timestamp`, `author`, `description`, `entities`, `relationships`, `checksum`); returns `False` with structured DEBUG-level error details on failure, never raises. Added `migrate_snapshot(snapshot) -> dict` — deep-copies and upgrades old-format snapshots to v1.0, populating missing required fields with `None`; already-v1.0 snapshots returned unchanged with no data loss. New `semantica/kg/schemas/temporal_snapshot_v1.json` — JSON Schema (draft 2020-12) defining required and optional fields, types, and constraints.
|
||||
- Added 28 new tests in `tests/test_401_temporal_provenance_export.py` covering every acceptance criterion; 451 related tests pass, 0 regressions.
|
||||
|
||||
- **Temporal Metadata Extraction from Text** (PR #400 by @KaifAhmad1):
|
||||
- Added `extract_temporal_bounds: bool = False` parameter to `extract_relations_llm()`. When `True`, the LLM prompt is extended with a calibrated confidence scale and four few-shot examples; each returned `Relation` gains `valid_from`, `valid_until`, `temporal_confidence` (0.0–1.0), and `temporal_source_text` in its `metadata` dict. Default `False` preserves 100% backward compatibility.
|
||||
- Confidence scale anchors baked into the prompt: `1.00` = full ISO date, `0.90` = year+month, `0.85` = year only, `0.75` = quarter, `0.65` = named season/approximate range, `0.50` = vague relative with computable anchor, `0.35` = highly vague, `0.00` = no temporal signal. LLMs self-report certainty rather than clustering near 1.0.
|
||||
|
||||
@@ -286,6 +286,15 @@ class RDFSerializer:
|
||||
|
||||
return rdf_data
|
||||
|
||||
# OWL-Time namespace URI
|
||||
_OWL_TIME_NS = "http://www.w3.org/2006/time#"
|
||||
|
||||
# Design decision — TemporalBound.OPEN in RDF:
|
||||
# OWL-Time has no standard predicate for "no known end date." We use
|
||||
# semantica:openEndedInterval "true"^^xsd:boolean on the time:Interval
|
||||
# node to signal that valid_until is OPEN/unbounded. This keeps the
|
||||
# interval well-formed while remaining human- and machine-readable.
|
||||
|
||||
def serialize_to_turtle(self, rdf_data: Dict[str, Any], **options) -> str:
|
||||
"""
|
||||
Serialize RDF to Turtle format.
|
||||
@@ -299,7 +308,12 @@ class RDFSerializer:
|
||||
- entities: List of entity dictionaries
|
||||
- relationships: List of relationship dictionaries
|
||||
- @context: Optional JSON-LD context for namespaces
|
||||
**options: Additional serialization options (unused)
|
||||
**options: Additional serialization options.
|
||||
include_temporal (bool): When True, emit OWL-Time triples for
|
||||
relationships that carry valid_from / valid_until metadata.
|
||||
Default: False.
|
||||
time_axis (str): Which temporal axis to export — "valid",
|
||||
"transaction", or "both". Default: "valid".
|
||||
|
||||
Returns:
|
||||
String containing Turtle-format RDF serialization
|
||||
@@ -311,13 +325,25 @@ class RDFSerializer:
|
||||
... }
|
||||
>>> turtle = serializer.serialize_to_turtle(rdf_data)
|
||||
"""
|
||||
include_temporal: bool = options.pop("include_temporal", False)
|
||||
time_axis: str = options.pop("time_axis", "valid")
|
||||
|
||||
lines = []
|
||||
|
||||
# Generate namespace declarations
|
||||
namespaces = self.namespace_manager.extract_namespaces(rdf_data)
|
||||
if namespaces:
|
||||
# Namespace declarations — always emit core prefixes; add OWL-Time when needed
|
||||
base_namespaces = {
|
||||
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
|
||||
"xsd": "http://www.w3.org/2001/XMLSchema#",
|
||||
"semantica": "https://semantica.dev/ns#",
|
||||
}
|
||||
if include_temporal:
|
||||
base_namespaces["time"] = self._OWL_TIME_NS
|
||||
|
||||
extracted = self.namespace_manager.extract_namespaces(rdf_data)
|
||||
merged_namespaces = {**base_namespaces, **extracted}
|
||||
|
||||
ns_declarations = self.namespace_manager.generate_namespace_declarations(
|
||||
namespaces, "turtle"
|
||||
merged_namespaces, "turtle"
|
||||
)
|
||||
lines.append(ns_declarations)
|
||||
lines.append("")
|
||||
@@ -325,7 +351,6 @@ class RDFSerializer:
|
||||
# Convert entities to RDF triplets
|
||||
entities = rdf_data.get("entities", [])
|
||||
for entity in entities:
|
||||
# Generate entity ID if not provided
|
||||
entity_id = entity.get("id")
|
||||
if not entity_id:
|
||||
entity_text = entity.get("text", "")
|
||||
@@ -335,7 +360,6 @@ class RDFSerializer:
|
||||
text = entity.get("text") or entity.get("label", "")
|
||||
confidence = entity.get("confidence", 1.0)
|
||||
|
||||
# Turtle triplet syntax: subject predicate object .
|
||||
lines.append(f"<{entity_id}> a <{entity_type}> ;")
|
||||
lines.append(f' semantica:text "{text}" ;')
|
||||
lines.append(f" semantica:confidence {confidence} .")
|
||||
@@ -343,16 +367,85 @@ class RDFSerializer:
|
||||
|
||||
# Convert relationships to RDF triplets
|
||||
relationships = rdf_data.get("relationships", [])
|
||||
for rel in relationships:
|
||||
for idx, rel in enumerate(relationships):
|
||||
source_id = rel.get("source_id") or rel.get("source")
|
||||
target_id = rel.get("target_id") or rel.get("target")
|
||||
rel_type = rel.get("type", "semantica:related_to")
|
||||
|
||||
# Simple triplet: subject predicate object .
|
||||
lines.append(f"<{source_id}> <{rel_type}> <{target_id}> .")
|
||||
|
||||
if include_temporal:
|
||||
owl_lines = self._owl_time_triples_for_rel(rel, idx, time_axis)
|
||||
if owl_lines:
|
||||
lines.extend(owl_lines)
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def _owl_time_triples_for_rel(
|
||||
self, rel: Dict[str, Any], idx: int, time_axis: str
|
||||
) -> List[str]:
|
||||
"""
|
||||
Emit OWL-Time Turtle triples for a relationship that carries temporal metadata.
|
||||
|
||||
For TemporalBound.OPEN valid_until values we use:
|
||||
semantica:openEndedInterval "true"^^xsd:boolean
|
||||
instead of time:hasEnd, because OWL-Time has no standard predicate for
|
||||
"no known end date."
|
||||
"""
|
||||
_OPEN_SENTINEL = "OPEN"
|
||||
|
||||
def _is_open(v: Any) -> bool:
|
||||
if v is None:
|
||||
return False
|
||||
if hasattr(v, "value"): # TemporalBound enum
|
||||
return v.value == _OPEN_SENTINEL
|
||||
return str(v).strip().upper() == _OPEN_SENTINEL
|
||||
|
||||
axes: List[tuple] = []
|
||||
if time_axis in ("valid", "both"):
|
||||
axes.append(("valid", rel.get("valid_from"), rel.get("valid_until")))
|
||||
if time_axis in ("transaction", "both"):
|
||||
axes.append(("tx", rel.get("recorded_at"), rel.get("superseded_at")))
|
||||
|
||||
rel_base_id = (
|
||||
rel.get("id")
|
||||
or f"semantica:rel_{idx}_{hash(str(rel.get('source_id', '')) + str(rel.get('target_id', '')))}"
|
||||
)
|
||||
|
||||
lines = [""] # blank separator
|
||||
for axis_name, from_val, until_val in axes:
|
||||
if from_val is None and (until_val is None or _is_open(until_val)):
|
||||
continue # no temporal data on this axis — skip
|
||||
|
||||
interval_id = f"{rel_base_id}__{axis_name}_interval"
|
||||
begin_id = f"{rel_base_id}__{axis_name}_begin"
|
||||
|
||||
lines.append(f"<{rel_base_id}> time:hasTime <{interval_id}> .")
|
||||
lines.append(f"<{interval_id}> a time:Interval ;")
|
||||
lines.append(f" time:hasBeginning <{begin_id}> ;")
|
||||
|
||||
if _is_open(until_val):
|
||||
lines.append(
|
||||
' semantica:openEndedInterval "true"^^xsd:boolean .'
|
||||
)
|
||||
elif until_val is not None:
|
||||
end_id = f"{rel_base_id}__{axis_name}_end"
|
||||
lines.append(f" time:hasEnd <{end_id}> .")
|
||||
lines.append(f"<{end_id}> a time:Instant ;")
|
||||
lines.append(
|
||||
f' time:inXSDDateTimeStamp "{until_val}"^^xsd:dateTimeStamp .'
|
||||
)
|
||||
else:
|
||||
lines[-1] = lines[-1].rstrip(" ;") + " ." # close interval without hasEnd
|
||||
|
||||
lines.append(f"<{begin_id}> a time:Instant ;")
|
||||
lines.append(
|
||||
f' time:inXSDDateTimeStamp "{from_val}"^^xsd:dateTimeStamp .'
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
return lines if len(lines) > 1 else []
|
||||
|
||||
def serialize_to_rdfxml(self, rdf_data: Dict[str, Any], **options) -> str:
|
||||
"""
|
||||
Serialize RDF to RDF/XML format.
|
||||
@@ -868,7 +961,12 @@ class RDFExporter:
|
||||
)
|
||||
|
||||
def export_to_rdf(
|
||||
self, data: Dict[str, Any], format: str = "turtle", **options
|
||||
self,
|
||||
data: Dict[str, Any],
|
||||
format: str = "turtle",
|
||||
include_temporal: bool = False,
|
||||
time_axis: str = "valid",
|
||||
**options,
|
||||
) -> str:
|
||||
"""
|
||||
Export data to RDF format string.
|
||||
@@ -933,7 +1031,12 @@ class RDFExporter:
|
||||
)
|
||||
# Serialize based on format
|
||||
if format == "turtle":
|
||||
result = self.serializer.serialize_to_turtle(data, **options)
|
||||
result = self.serializer.serialize_to_turtle(
|
||||
data,
|
||||
include_temporal=include_temporal,
|
||||
time_axis=time_axis,
|
||||
**options,
|
||||
)
|
||||
elif format == "rdfxml":
|
||||
result = self.serializer.serialize_to_rdfxml(data, **options)
|
||||
elif format == "jsonld":
|
||||
|
||||
@@ -4,6 +4,10 @@ Provenance Tracker for Knowledge Graph entities.
|
||||
Tracks the sources and lineage of entities and relationships.
|
||||
"""
|
||||
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
@@ -29,7 +33,10 @@ class ProvenanceTracker:
|
||||
"""Record that entity_id was derived from source."""
|
||||
if entity_id not in self._records:
|
||||
self._records[entity_id] = []
|
||||
entry: Dict[str, Any] = {"source": source}
|
||||
entry: Dict[str, Any] = {
|
||||
"source": source,
|
||||
"recorded_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
if metadata:
|
||||
entry.update(metadata)
|
||||
self._records[entity_id].append(entry)
|
||||
@@ -44,3 +51,129 @@ class ProvenanceTracker:
|
||||
self._records.pop(entity_id, None)
|
||||
else:
|
||||
self._records.clear()
|
||||
|
||||
def query_recorded_between(
|
||||
self, start: Any, end: Any
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Return all provenance records whose recorded_at falls within [start, end].
|
||||
|
||||
Args:
|
||||
start: Start of range — datetime or ISO string (inclusive).
|
||||
end: End of range — datetime or ISO string (inclusive).
|
||||
|
||||
Returns:
|
||||
Flat list of matching provenance records (each dict includes
|
||||
the entity_id under the key "entity_id").
|
||||
"""
|
||||
start_dt = self._parse_dt(start)
|
||||
end_dt = self._parse_dt(end)
|
||||
|
||||
results = []
|
||||
for entity_id, records in self._records.items():
|
||||
for record in records:
|
||||
raw = record.get("recorded_at")
|
||||
if raw is None:
|
||||
continue
|
||||
try:
|
||||
rec_dt = self._parse_dt(raw)
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
if start_dt <= rec_dt <= end_dt:
|
||||
results.append({"entity_id": entity_id, **record})
|
||||
return results
|
||||
|
||||
def revision_history(self, fact_id: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Return the complete revision chain for fact_id in ascending recorded_at order.
|
||||
|
||||
Each entry contains at minimum:
|
||||
version (int, 1-based), valid_from, valid_until, recorded_at, author
|
||||
Optional fields: revision_type, supersedes.
|
||||
|
||||
Returns an empty list for a fact with no recorded provenance.
|
||||
"""
|
||||
records = self._records.get(fact_id, [])
|
||||
if not records:
|
||||
return []
|
||||
|
||||
# Sort by recorded_at ascending; records without recorded_at sort first
|
||||
def sort_key(r: Dict[str, Any]):
|
||||
raw = r.get("recorded_at")
|
||||
if raw is None:
|
||||
return ""
|
||||
return raw
|
||||
|
||||
sorted_records = sorted(records, key=sort_key)
|
||||
|
||||
history = []
|
||||
for version, record in enumerate(sorted_records, start=1):
|
||||
entry: Dict[str, Any] = {
|
||||
"version": version,
|
||||
"valid_from": record.get("valid_from"),
|
||||
"valid_until": record.get("valid_until"),
|
||||
"recorded_at": record.get("recorded_at"),
|
||||
"author": record.get("author"),
|
||||
}
|
||||
if "revision_type" in record:
|
||||
entry["revision_type"] = record["revision_type"]
|
||||
if "supersedes" in record:
|
||||
entry["supersedes"] = record["supersedes"]
|
||||
history.append(entry)
|
||||
return history
|
||||
|
||||
def export_audit_log(self, fact_ids: List[str], format: str = "json") -> str:
|
||||
"""
|
||||
Export audit log for the given fact IDs.
|
||||
|
||||
Args:
|
||||
fact_ids: List of fact/entity IDs to include.
|
||||
format: "json" or "csv".
|
||||
|
||||
Returns:
|
||||
String containing the serialized audit log.
|
||||
"""
|
||||
rows = []
|
||||
for fact_id in fact_ids:
|
||||
for entry in self.revision_history(fact_id):
|
||||
rows.append({"fact_id": fact_id, **entry})
|
||||
|
||||
if format == "json":
|
||||
return json.dumps(rows, indent=2, default=str)
|
||||
|
||||
if format == "csv":
|
||||
fieldnames = [
|
||||
"fact_id", "version", "valid_from", "valid_until",
|
||||
"recorded_at", "author", "revision_type", "supersedes",
|
||||
]
|
||||
buf = io.StringIO()
|
||||
writer = csv.DictWriter(
|
||||
buf, fieldnames=fieldnames, extrasaction="ignore", lineterminator="\n"
|
||||
)
|
||||
writer.writeheader()
|
||||
for row in rows:
|
||||
writer.writerow(row)
|
||||
return buf.getvalue()
|
||||
|
||||
raise ValueError(f"Unsupported audit log format: {format!r}. Use 'json' or 'csv'.")
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internal helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _parse_dt(value: Any) -> datetime:
|
||||
"""Parse a datetime or ISO string into an aware datetime (UTC assumed when naive)."""
|
||||
if isinstance(value, datetime):
|
||||
if value.tzinfo is None:
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
return value
|
||||
# String
|
||||
s = str(value).strip()
|
||||
# Handle trailing 'Z'
|
||||
if s.endswith("Z"):
|
||||
s = s[:-1] + "+00:00"
|
||||
dt = datetime.fromisoformat(s)
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=timezone.utc)
|
||||
return dt
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://semantica.dev/schemas/temporal_snapshot_v1.json",
|
||||
"title": "TemporalSnapshot",
|
||||
"description": "Schema for Semantica temporal version manager snapshots (format_version 1.0).",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"format_version",
|
||||
"label",
|
||||
"timestamp",
|
||||
"author",
|
||||
"description",
|
||||
"entities",
|
||||
"relationships",
|
||||
"checksum"
|
||||
],
|
||||
"properties": {
|
||||
"format_version": {
|
||||
"type": "string",
|
||||
"const": "1.0",
|
||||
"description": "Schema version. Must be '1.0' for snapshots conforming to this schema."
|
||||
},
|
||||
"label": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Human-readable version label (e.g. 'v1.0', 'release-2024-03')."
|
||||
},
|
||||
"timestamp": {
|
||||
"type": "string",
|
||||
"description": "ISO 8601 datetime at which the snapshot was created."
|
||||
},
|
||||
"author": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Email address or identifier of the person who created the snapshot."
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"maxLength": 500,
|
||||
"description": "Human-readable description of the snapshot."
|
||||
},
|
||||
"entities": {
|
||||
"type": "array",
|
||||
"description": "List of entity dictionaries captured in this snapshot.",
|
||||
"items": {
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"relationships": {
|
||||
"type": "array",
|
||||
"description": "List of relationship dictionaries captured in this snapshot.",
|
||||
"items": {
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"checksum": {
|
||||
"type": "string",
|
||||
"description": "SHA-256 hex digest over the snapshot content (excluding the checksum field itself)."
|
||||
},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"description": "Optional free-form metadata dictionary (e.g. tags, revision_event)."
|
||||
}
|
||||
},
|
||||
"additionalProperties": true
|
||||
}
|
||||
@@ -1353,6 +1353,7 @@ class TemporalVersionManager:
|
||||
|
||||
# Create snapshot
|
||||
snapshot = {
|
||||
"format_version": "1.0",
|
||||
"label": version_label,
|
||||
"timestamp": change_entry.timestamp,
|
||||
"author": change_entry.author,
|
||||
@@ -1501,6 +1502,80 @@ class TemporalVersionManager:
|
||||
"""
|
||||
return self.storage.get(label)
|
||||
|
||||
def validate_snapshot(self, snapshot: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Validate a snapshot against the v1.0 JSON Schema.
|
||||
|
||||
Returns True when valid. Returns False (never raises) when one or more
|
||||
required fields are missing or have the wrong type. Structured error
|
||||
details are logged at DEBUG level.
|
||||
|
||||
Required fields: format_version, label, timestamp, author, description,
|
||||
entities, relationships, checksum.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
||||
required = {
|
||||
"format_version": str,
|
||||
"label": str,
|
||||
"timestamp": str,
|
||||
"author": str,
|
||||
"description": str,
|
||||
"entities": list,
|
||||
"relationships": list,
|
||||
"checksum": str,
|
||||
}
|
||||
|
||||
errors = []
|
||||
for field, expected_type in required.items():
|
||||
if field not in snapshot:
|
||||
errors.append({"field": field, "error": "missing"})
|
||||
elif not isinstance(snapshot[field], expected_type):
|
||||
errors.append({
|
||||
"field": field,
|
||||
"error": "wrong_type",
|
||||
"expected": expected_type.__name__,
|
||||
"got": type(snapshot[field]).__name__,
|
||||
})
|
||||
|
||||
if errors:
|
||||
self.logger.debug(f"validate_snapshot failed: {errors}")
|
||||
return False
|
||||
return True
|
||||
|
||||
def migrate_snapshot(self, snapshot: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Upgrade an old-format snapshot (no format_version) to v1.0.
|
||||
|
||||
- Snapshots already at format_version "1.0" are returned unchanged.
|
||||
- Missing required fields are populated with None.
|
||||
- No data is lost; existing fields are preserved.
|
||||
"""
|
||||
import copy
|
||||
|
||||
result = copy.deepcopy(snapshot)
|
||||
if result.get("format_version") == "1.0":
|
||||
return result
|
||||
|
||||
result["format_version"] = "1.0"
|
||||
|
||||
optional_defaults = {
|
||||
"label": None,
|
||||
"timestamp": None,
|
||||
"author": None,
|
||||
"description": None,
|
||||
"entities": None,
|
||||
"relationships": None,
|
||||
"checksum": None,
|
||||
}
|
||||
for field, default in optional_defaults.items():
|
||||
if field not in result:
|
||||
result[field] = default
|
||||
|
||||
return result
|
||||
|
||||
def verify_checksum(self, snapshot: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Verify the integrity of a snapshot using its checksum.
|
||||
|
||||
@@ -0,0 +1,389 @@
|
||||
"""
|
||||
Tests for #401 — Temporal Provenance & Export.
|
||||
|
||||
Covers all acceptance criteria from the issue:
|
||||
- ProvenanceTracker: recorded_at, query_recorded_between
|
||||
- ProvenanceTracker: revision_history, export_audit_log
|
||||
- RDFExporter: include_temporal OWL-Time Turtle output
|
||||
- TemporalVersionManager: format_version, validate_snapshot, migrate_snapshot
|
||||
"""
|
||||
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from semantica.kg.provenance_tracker import ProvenanceTracker
|
||||
from semantica.export.rdf_exporter import RDFExporter
|
||||
from semantica.kg.temporal_query import TemporalVersionManager
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
UTC = timezone.utc
|
||||
|
||||
|
||||
def _dt(offset_days: int = 0) -> str:
|
||||
"""Return an ISO UTC string relative to now."""
|
||||
return (datetime.now(UTC) + timedelta(days=offset_days)).isoformat()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Transaction Time on Provenance Records
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestTransactionTime:
|
||||
def test_new_record_has_recorded_at(self):
|
||||
tracker = ProvenanceTracker()
|
||||
before = datetime.now(UTC)
|
||||
tracker.track_entity("E1", "doc.txt")
|
||||
after = datetime.now(UTC)
|
||||
|
||||
records = tracker.get_all_sources("E1")
|
||||
assert len(records) == 1
|
||||
raw = records[0]["recorded_at"]
|
||||
recorded = datetime.fromisoformat(raw)
|
||||
if recorded.tzinfo is None:
|
||||
recorded = recorded.replace(tzinfo=UTC)
|
||||
assert before <= recorded <= after
|
||||
|
||||
def test_existing_records_without_recorded_at_still_work(self):
|
||||
tracker = ProvenanceTracker()
|
||||
# Manually inject an old-style record (no recorded_at)
|
||||
tracker._records["old_fact"] = [{"source": "legacy.txt"}]
|
||||
# All existing query methods must not crash
|
||||
sources = tracker.get_all_sources("old_fact")
|
||||
assert sources == [{"source": "legacy.txt"}]
|
||||
|
||||
def test_query_recorded_between_returns_only_matching_records(self):
|
||||
tracker = ProvenanceTracker()
|
||||
|
||||
past = datetime.now(UTC) - timedelta(days=30)
|
||||
future = datetime.now(UTC) + timedelta(days=30)
|
||||
|
||||
tracker.track_entity("E1", "doc1.txt") # now → in range [yesterday, tomorrow]
|
||||
# Inject a record from 60 days ago
|
||||
tracker._records["E2"] = [{
|
||||
"source": "old.txt",
|
||||
"recorded_at": (datetime.now(UTC) - timedelta(days=60)).isoformat(),
|
||||
}]
|
||||
|
||||
start = datetime.now(UTC) - timedelta(days=1)
|
||||
end = datetime.now(UTC) + timedelta(days=1)
|
||||
|
||||
results = tracker.query_recorded_between(start, end)
|
||||
entity_ids = [r["entity_id"] for r in results]
|
||||
assert "E1" in entity_ids
|
||||
assert "E2" not in entity_ids
|
||||
|
||||
def test_query_recorded_between_accepts_iso_strings(self):
|
||||
tracker = ProvenanceTracker()
|
||||
tracker.track_entity("X", "src.txt")
|
||||
|
||||
start = (datetime.now(UTC) - timedelta(hours=1)).isoformat()
|
||||
end = (datetime.now(UTC) + timedelta(hours=1)).isoformat()
|
||||
|
||||
results = tracker.query_recorded_between(start, end)
|
||||
assert any(r["entity_id"] == "X" for r in results)
|
||||
|
||||
def test_query_recorded_between_skips_records_without_recorded_at(self):
|
||||
tracker = ProvenanceTracker()
|
||||
tracker._records["legacy"] = [{"source": "x.txt"}]
|
||||
|
||||
start = datetime.now(UTC) - timedelta(days=1)
|
||||
end = datetime.now(UTC) + timedelta(days=1)
|
||||
|
||||
# Must not raise; legacy record is silently skipped
|
||||
results = tracker.query_recorded_between(start, end)
|
||||
assert all(r["entity_id"] != "legacy" for r in results)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Fact Revision Audit Trail
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestRevisionHistory:
|
||||
def _tracker_with_revisions(self) -> ProvenanceTracker:
|
||||
tracker = ProvenanceTracker()
|
||||
base_ts = datetime(2024, 3, 1, 0, 0, 0, tzinfo=UTC)
|
||||
tracker._records["fact_001"] = [
|
||||
{
|
||||
"source": "s1",
|
||||
"valid_from": "2024-01-01",
|
||||
"valid_until": "2024-06-30",
|
||||
"recorded_at": (base_ts).isoformat(),
|
||||
"author": "alice@example.com",
|
||||
},
|
||||
{
|
||||
"source": "s2",
|
||||
"valid_from": "2024-01-01",
|
||||
"valid_until": "2024-12-31",
|
||||
"recorded_at": (base_ts + timedelta(days=10)).isoformat(),
|
||||
"author": "bob@example.com",
|
||||
"revision_type": "correction",
|
||||
"supersedes": "fact_001_v1",
|
||||
},
|
||||
]
|
||||
return tracker
|
||||
|
||||
def test_revision_history_returns_versions_in_ascending_order(self):
|
||||
tracker = self._tracker_with_revisions()
|
||||
history = tracker.revision_history("fact_001")
|
||||
|
||||
assert len(history) == 2
|
||||
assert history[0]["version"] == 1
|
||||
assert history[1]["version"] == 2
|
||||
assert history[0]["recorded_at"] < history[1]["recorded_at"]
|
||||
|
||||
def test_revision_history_has_required_fields(self):
|
||||
tracker = self._tracker_with_revisions()
|
||||
history = tracker.revision_history("fact_001")
|
||||
|
||||
for entry in history:
|
||||
assert "version" in entry
|
||||
assert "valid_from" in entry
|
||||
assert "valid_until" in entry
|
||||
assert "recorded_at" in entry
|
||||
assert "author" in entry
|
||||
|
||||
def test_revision_history_optional_fields_present_when_set(self):
|
||||
tracker = self._tracker_with_revisions()
|
||||
history = tracker.revision_history("fact_001")
|
||||
|
||||
assert history[1].get("revision_type") == "correction"
|
||||
assert history[1].get("supersedes") == "fact_001_v1"
|
||||
|
||||
def test_revision_history_empty_for_unknown_fact(self):
|
||||
tracker = ProvenanceTracker()
|
||||
assert tracker.revision_history("nonexistent") == []
|
||||
|
||||
def test_export_audit_log_json_valid(self):
|
||||
tracker = self._tracker_with_revisions()
|
||||
output = tracker.export_audit_log(["fact_001"], format="json")
|
||||
data = json.loads(output) # must not raise
|
||||
assert isinstance(data, list)
|
||||
assert len(data) == 2
|
||||
assert data[0]["fact_id"] == "fact_001"
|
||||
|
||||
def test_export_audit_log_csv_has_header(self):
|
||||
tracker = self._tracker_with_revisions()
|
||||
output = tracker.export_audit_log(["fact_001"], format="csv")
|
||||
reader = csv.DictReader(io.StringIO(output))
|
||||
rows = list(reader)
|
||||
assert reader.fieldnames is not None
|
||||
assert "fact_id" in reader.fieldnames
|
||||
assert "version" in reader.fieldnames
|
||||
assert len(rows) == 2
|
||||
|
||||
def test_export_audit_log_empty_fact_ids(self):
|
||||
tracker = ProvenanceTracker()
|
||||
json_out = tracker.export_audit_log([], format="json")
|
||||
assert json.loads(json_out) == []
|
||||
csv_out = tracker.export_audit_log([], format="csv")
|
||||
reader = csv.DictReader(io.StringIO(csv_out))
|
||||
assert list(reader) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. OWL-Time RDF Export
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
RDF_DATA_PLAIN = {
|
||||
"entities": [
|
||||
{"id": "http://ex.org/e1", "text": "Alice", "type": "Person"},
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"id": "http://ex.org/rel1",
|
||||
"source_id": "http://ex.org/e1",
|
||||
"target_id": "http://ex.org/e2",
|
||||
"type": "http://ex.org/knows",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
RDF_DATA_TEMPORAL = {
|
||||
"entities": [
|
||||
{"id": "http://ex.org/e1", "text": "Alice", "type": "Person"},
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"id": "http://ex.org/rel1",
|
||||
"source_id": "http://ex.org/e1",
|
||||
"target_id": "http://ex.org/e2",
|
||||
"type": "http://ex.org/knows",
|
||||
"valid_from": "2024-01-01T00:00:00+00:00",
|
||||
"valid_until": "2024-12-31T23:59:59+00:00",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
RDF_DATA_OPEN = {
|
||||
"entities": [],
|
||||
"relationships": [
|
||||
{
|
||||
"id": "http://ex.org/rel2",
|
||||
"source_id": "http://ex.org/e1",
|
||||
"target_id": "http://ex.org/e2",
|
||||
"type": "http://ex.org/employs",
|
||||
"valid_from": "2024-01-01T00:00:00+00:00",
|
||||
"valid_until": "OPEN",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class TestOWLTimeExport:
|
||||
@pytest.fixture
|
||||
def exporter(self):
|
||||
return RDFExporter()
|
||||
|
||||
def test_default_no_temporal_output_unchanged(self, exporter):
|
||||
base = exporter.export_to_rdf(RDF_DATA_PLAIN, format="turtle")
|
||||
with_flag = exporter.export_to_rdf(
|
||||
RDF_DATA_PLAIN, format="turtle", include_temporal=False
|
||||
)
|
||||
assert base == with_flag
|
||||
|
||||
def test_include_temporal_adds_owl_time_prefix(self, exporter):
|
||||
result = exporter.export_to_rdf(
|
||||
RDF_DATA_TEMPORAL, format="turtle", include_temporal=True
|
||||
)
|
||||
assert "time:" in result or "http://www.w3.org/2006/time#" in result
|
||||
|
||||
def test_include_temporal_emits_interval_and_instants(self, exporter):
|
||||
result = exporter.export_to_rdf(
|
||||
RDF_DATA_TEMPORAL, format="turtle", include_temporal=True
|
||||
)
|
||||
assert "time:Interval" in result
|
||||
assert "time:hasBeginning" in result
|
||||
assert "time:hasEnd" in result
|
||||
assert "time:inXSDDateTimeStamp" in result
|
||||
assert "2024-01-01" in result
|
||||
assert "2024-12-31" in result
|
||||
|
||||
def test_open_valid_until_emits_open_ended_flag(self, exporter):
|
||||
result = exporter.export_to_rdf(
|
||||
RDF_DATA_OPEN, format="turtle", include_temporal=True
|
||||
)
|
||||
assert "openEndedInterval" in result
|
||||
# Must NOT emit time:hasEnd for an OPEN interval
|
||||
assert "time:hasEnd" not in result
|
||||
|
||||
def test_relationships_without_temporal_not_affected(self, exporter):
|
||||
result_with = exporter.export_to_rdf(
|
||||
RDF_DATA_PLAIN, format="turtle", include_temporal=True
|
||||
)
|
||||
# No OWL-Time interval nodes for relationships that have no valid_from/until
|
||||
assert "time:Interval" not in result_with
|
||||
assert "time:hasBeginning" not in result_with
|
||||
|
||||
def test_time_axis_transaction_uses_recorded_at(self, exporter):
|
||||
data = {
|
||||
"entities": [],
|
||||
"relationships": [
|
||||
{
|
||||
"id": "http://ex.org/rel3",
|
||||
"source_id": "http://ex.org/e1",
|
||||
"target_id": "http://ex.org/e2",
|
||||
"type": "http://ex.org/rel",
|
||||
"recorded_at": "2024-03-01T00:00:00+00:00",
|
||||
"superseded_at": "OPEN",
|
||||
}
|
||||
],
|
||||
}
|
||||
result = exporter.export_to_rdf(
|
||||
data, format="turtle", include_temporal=True, time_axis="transaction"
|
||||
)
|
||||
assert "time:Interval" in result
|
||||
assert "2024-03-01" in result
|
||||
assert "openEndedInterval" in result
|
||||
|
||||
def test_output_parseable_by_rdflib(self, exporter):
|
||||
"""The produced Turtle must be parseable by rdflib (if installed)."""
|
||||
pytest.importorskip("rdflib")
|
||||
from rdflib import Graph
|
||||
|
||||
result = exporter.export_to_rdf(
|
||||
RDF_DATA_TEMPORAL, format="turtle", include_temporal=True
|
||||
)
|
||||
g = Graph()
|
||||
g.parse(data=result, format="turtle") # raises on parse failure
|
||||
assert len(g) > 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. Stable Snapshot Serialization Format
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestSnapshotSerialization:
|
||||
@pytest.fixture
|
||||
def manager(self):
|
||||
return TemporalVersionManager()
|
||||
|
||||
@pytest.fixture
|
||||
def graph(self):
|
||||
return {
|
||||
"entities": [{"id": "e1", "label": "Alice"}],
|
||||
"relationships": [{"source": "e1", "target": "e2", "type": "knows"}],
|
||||
}
|
||||
|
||||
def test_create_snapshot_includes_format_version(self, manager, graph):
|
||||
snap = manager.create_snapshot(graph, "v1.0", "alice@example.com", "Initial")
|
||||
assert snap.get("format_version") == "1.0"
|
||||
|
||||
def test_created_snapshot_passes_validate(self, manager, graph):
|
||||
snap = manager.create_snapshot(graph, "v1.0", "alice@example.com", "Initial")
|
||||
assert manager.validate_snapshot(snap) is True
|
||||
|
||||
def test_validate_snapshot_false_on_missing_fields(self, manager):
|
||||
incomplete = {
|
||||
"format_version": "1.0",
|
||||
"label": "v1.0",
|
||||
# missing: timestamp, author, description, entities, relationships, checksum
|
||||
}
|
||||
assert manager.validate_snapshot(incomplete) is False
|
||||
|
||||
def test_validate_snapshot_false_reports_missing_field_names(self, manager):
|
||||
# validate_snapshot returns False — we just check it doesn't raise
|
||||
result = manager.validate_snapshot({"format_version": "1.0"})
|
||||
assert result is False
|
||||
|
||||
def test_validate_snapshot_never_raises(self, manager):
|
||||
for bad in [{}, None, "string", 42, []]:
|
||||
try:
|
||||
result = manager.validate_snapshot(bad) if isinstance(bad, dict) else manager.validate_snapshot({})
|
||||
assert result in (True, False)
|
||||
except Exception as exc:
|
||||
pytest.fail(f"validate_snapshot raised unexpectedly: {exc}")
|
||||
|
||||
def test_migrate_snapshot_upgrades_old_format(self, manager):
|
||||
old = {
|
||||
"label": "v0.1",
|
||||
"timestamp": "2023-01-01T00:00:00",
|
||||
"entities": [],
|
||||
"relationships": [],
|
||||
}
|
||||
migrated = manager.migrate_snapshot(old)
|
||||
assert migrated["format_version"] == "1.0"
|
||||
assert migrated["label"] == "v0.1" # existing data preserved
|
||||
|
||||
def test_migrate_snapshot_fills_missing_required_fields_with_none(self, manager):
|
||||
old = {"label": "v0.1"}
|
||||
migrated = manager.migrate_snapshot(old)
|
||||
for field in ("author", "description", "entities", "relationships", "checksum"):
|
||||
assert field in migrated
|
||||
|
||||
def test_migrate_snapshot_already_v1_returned_unchanged(self, manager, graph):
|
||||
snap = manager.create_snapshot(graph, "v1.0", "alice@example.com", "desc")
|
||||
migrated = manager.migrate_snapshot(snap)
|
||||
assert migrated == snap
|
||||
|
||||
def test_migrate_snapshot_no_data_loss(self, manager):
|
||||
old = {"label": "v0.1", "custom_field": "keep_me"}
|
||||
migrated = manager.migrate_snapshot(old)
|
||||
assert migrated["custom_field"] == "keep_me"
|
||||
Reference in New Issue
Block a user