semantica/cookbook/introduction/22_Provenance_Tracking.ipynb

Provenance Tracking (W3C PROV-O)

Overview

In high-stakes domains — healthcare, legal, finance, research — a Knowledge Graph is only as trustworthy as its ability to answer "where did this fact come from?". Semantica's provenance module provides audit-grade, W3C PROV-O-aligned tracking for every entity, relationship and chunk that flows through your pipeline.

In this cookbook you will learn how to:

• Track entities and relationships with source details (DOI, page, verbatim quote, confidence)
• Walk the full lineage of a fact (document → chunk → entity → KG)
• Audit revision history and all sources behind an entity
• Invalidate a fact without deleting it (prov:Invalidation) — corrections stay provable
• Verify tamper-evidence with chained SHA-256 checksums

The Scenario: a research team ingests findings from two scientific papers (with DOIs) into a Knowledge Graph. A regulator later asks: "Which paper, which figure, and which exact sentence supports the claim that fish biomass increased by 463%? And was that fact ever corrected?"

In [ ]:

!pip install -q semantica

In [ ]:

import json
from semantica.provenance import (
    ProvenanceManager,
    compute_checksum,
    verify_checksum,
)

# In-memory storage for this demo; pass storage_path="provenance.db"
# (or a config with provenance.storage_path) for a persistent SQLite backend.
prov = ProvenanceManager()
print("ProvenanceManager ready (in-memory storage)")

Step 1: Track Entities with Audit-Grade Source Details

Every fact we ingest carries its evidence with it: the source identifier (a DOI here), the location inside the source (a figure), the verbatim quote, and the extractor's confidence.

In [ ]:

# Finding from paper #1
entry_biomass = prov.track_entity(
    entity_id="claim_biomass_increase",
    source="DOI:10.1371/journal.pone.0023601",
    confidence=0.92,
    source_location="Figure 2",
    source_quote="Total fish biomass increased by 463% ...",
)

# Supporting entity from paper #2
entry_reserve = prov.track_entity(
    entity_id="marine_reserve_1",
    source="DOI:10.1126/science.1088121",
    confidence=0.88,
    source_location="Table 1",
    source_quote="... no-take marine reserve at Cabo Pulmo ...",
)

print("Tracked:", entry_biomass.entity_id, "|", entry_reserve.entity_id)

Step 2: Track the Relationship Between Facts

Facts rarely stand alone. The claim about biomass increase is about the marine reserve — that relationship is a first-class provenance-tracked object too.

track_relationship() has no dedicated subject/object fields, so by convention we record which two entities it connects inside metadata.

In [ ]:

rel = prov.track_relationship(
    relationship_id="rel_biomass_about_reserve",
    source="DOI:10.1371/journal.pone.0023601",
    metadata={
        "type": "measured_at",
        # No dedicated endpoint fields on track_relationship() yet -- record
        # which entities this relationship connects here by convention.
        "subject_entity_id": "claim_biomass_increase",
        "object_entity_id": "marine_reserve_1",
    },
)

print("Relationship tracked:", rel.entity_id, "|", rel.metadata["subject_entity_id"], "->", rel.metadata["object_entity_id"])

Step 3: Walk the Lineage

get_lineage reconstructs everything known about a fact; trace_lineage returns the ordered chain of ProvenanceEntry records — every version, every activity, every agent that touched it.

In [ ]:

lineage = prov.get_lineage("claim_biomass_increase")
print(json.dumps(lineage, indent=2, default=str)[:800])

print("\n--- ordered chain ---")
for e in prov.trace_lineage("claim_biomass_increase"):
    print(f"{e.entity_id} | seq#{e.sequence_id} | {e.activity_id}")

Step 4: Audit Sources and Revision History

When the regulator asks "has this fact ever been corrected?", revision_history answers with the full version chain, and get_all_sources lists every source document that ever supported the entity.

In [ ]:

revisions = prov.revision_history("claim_biomass_increase")
print(f"{len(revisions)} revision(s) on record")

for s in prov.get_all_sources("claim_biomass_increase"):
    print("source:", s)

Step 5: Invalidate — Correct Without Deleting

Suppose paper #1 is retracted in part. An audit trail must not silently delete the fact: invalidate archives the pre-invalidation state and appends a fresh prov:Invalidation entry naming who retracted it and why.

In [ ]:

invalidated = prov.invalidate(
    entity_id="claim_biomass_increase",
    agent_id="reviewer_dr_chen",
    reason="Partial retraction: Figure 2 statistics corrected by publisher (see erratum).",
)
print("Invalidated:", invalidated.entity_id, "| invalidated flag:", getattr(invalidated, "invalidated", True))

stats = prov.get_statistics()
print("\nStorage statistics:", json.dumps(stats, indent=2, default=str))

Step 6: Verify Tamper-Evidence

Each entry carries a deterministic SHA-256 checksum chained to the previous entry. Recompute and compare to detect any after-the-fact corruption of the provenance record.

In [ ]:

# entry_biomass was returned by track_entity in Step 1
ok = verify_checksum(entry_biomass)
print("Checksum verified:", ok)

print("Computed:", compute_checksum(entry_biomass)[:16], "...")
print("Stored:  ", entry_biomass.checksum[:16] if getattr(entry_biomass, 'checksum', None) else "(see entry fields)")
chain = prov.verify_chain()
print("Chain verification:", json.dumps(chain, default=str)[:200])

Summary

Need	Call
Record a fact's evidence	prov.track_entity(entity_id, source, confidence=..., source_location=..., source_quote=...)
Record a relationship	prov.track_relationship(relationship_id, source, metadata=...)
Full lineage of a fact	prov.get_lineage(entity_id) / prov.trace_lineage(entity_id)
"Was it ever corrected?"	prov.revision_history(entity_id)
"Which sources support it?"	prov.get_all_sources(entity_id)
Retract without deleting	prov.invalidate(entity_id, agent_id, reason=...)
Tamper check	verify_checksum(entry)

Where to go next

• Conflict Detection and Resolution (notebook 17) — what happens when two sources disagree.
• Your First Knowledge Graph (notebook 08) — plug provenance=True into extractors so tracking happens automatically during ingestion.
• The module docstring (help(semantica.provenance)) documents opt-in integration with kg, split and conflicts trackers.