diff --git a/cookbook/introduction/22_Provenance_Tracking.ipynb b/cookbook/introduction/22_Provenance_Tracking.ipynb new file mode 100644 index 00000000..ce285f3d --- /dev/null +++ b/cookbook/introduction/22_Provenance_Tracking.ipynb @@ -0,0 +1,243 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Provenance Tracking (W3C PROV-O)\n", + "\n", + "## Overview\n", + "\n", + "In high-stakes domains — healthcare, legal, finance, research — a Knowledge Graph is only as trustworthy as its ability to answer **\"where did this fact come from?\"**. Semantica's `provenance` module provides audit-grade, W3C PROV-O-aligned tracking for every entity, relationship and chunk that flows through your pipeline.\n", + "\n", + "In this cookbook you will learn how to:\n", + "\n", + "- Track entities and relationships with **source details** (DOI, page, verbatim quote, confidence)\n", + "- Walk the full **lineage** of a fact (document → chunk → entity → KG)\n", + "- Audit **revision history** and **all sources** behind an entity\n", + "- **Invalidate** a fact without deleting it (prov:Invalidation) — corrections stay provable\n", + "- Verify **tamper-evidence** with chained SHA-256 checksums\n", + "\n", + "**The Scenario:** a research team ingests findings from two scientific papers (with DOIs) into a Knowledge Graph. A regulator later asks: *\"Which paper, which figure, and which exact sentence supports the claim that fish biomass increased by 463%? And was that fact ever corrected?\"*" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install -q semantica" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "from semantica.provenance import (\n", + " ProvenanceManager,\n", + " compute_checksum,\n", + " verify_checksum,\n", + ")\n", + "\n", + "# In-memory storage for this demo; pass storage_path=\"provenance.db\"\n", + "# (or a config with provenance.storage_path) for a persistent SQLite backend.\n", + "prov = ProvenanceManager()\n", + "print(\"ProvenanceManager ready (in-memory storage)\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 1: Track Entities with Audit-Grade Source Details\n", + "\n", + "Every fact we ingest carries its evidence with it: the **source identifier** (a DOI here), the **location** inside the source (a figure), the **verbatim quote**, and the extractor's **confidence**." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Finding from paper #1\n", + "entry_biomass = prov.track_entity(\n", + " entity_id=\"claim_biomass_increase\",\n", + " source=\"DOI:10.1371/journal.pone.0023601\",\n", + " confidence=0.92,\n", + " source_location=\"Figure 2\",\n", + " source_quote=\"Total fish biomass increased by 463% ...\",\n", + ")\n", + "\n", + "# Supporting entity from paper #2\n", + "entry_reserve = prov.track_entity(\n", + " entity_id=\"marine_reserve_1\",\n", + " source=\"DOI:10.1126/science.1088121\",\n", + " confidence=0.88,\n", + " source_location=\"Table 1\",\n", + " source_quote=\"... no-take marine reserve at Cabo Pulmo ...\",\n", + ")\n", + "\n", + "print(\"Tracked:\", entry_biomass.entity_id, \"|\", entry_reserve.entity_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 2: Track the Relationship Between Facts\n", + "\n", + "Facts rarely stand alone. The claim about biomass increase is *about* the marine reserve — that relationship is a first-class provenance-tracked object too." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "rel = prov.track_relationship(\n", + " relationship_id=\"rel_biomass_about_reserve\",\n", + " source=\"DOI:10.1371/journal.pone.0023601\",\n", + " metadata={\"type\": \"measured_at\"},\n", + ")\n", + "\n", + "print(\"Relationship tracked:\", rel.entity_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 3: Walk the Lineage\n", + "\n", + "`get_lineage` reconstructs everything known about a fact; `trace_lineage` returns the ordered chain of `ProvenanceEntry` records — every version, every activity, every agent that touched it." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "lineage = prov.get_lineage(\"claim_biomass_increase\")\n", + "print(json.dumps(lineage, indent=2, default=str)[:800])\n", + "\n", + "print(\"\\n--- ordered chain ---\")\n", + "for e in prov.trace_lineage(\"claim_biomass_increase\"):\n", + " print(f\"{e.entity_id} | v{getattr(e, 'version', '?')} | {e.activity_id}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 4: Audit Sources and Revision History\n", + "\n", + "When the regulator asks *\"has this fact ever been corrected?\"*, `revision_history` answers with the full version chain, and `get_all_sources` lists every source document that ever supported the entity." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "revisions = prov.revision_history(\"claim_biomass_increase\")\n", + "print(f\"{len(revisions)} revision(s) on record\")\n", + "\n", + "for s in prov.get_all_sources(\"claim_biomass_increase\"):\n", + " print(\"source:\", s)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 5: Invalidate — Correct Without Deleting\n", + "\n", + "Suppose paper #1 is retracted in part. An audit trail must **not** silently delete the fact: `invalidate` archives the pre-invalidation state and appends a fresh `prov:Invalidation` entry naming **who** retracted it and **why**." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "invalidated = prov.invalidate(\n", + " entity_id=\"claim_biomass_increase\",\n", + " agent_id=\"reviewer_dr_chen\",\n", + " reason=\"Partial retraction: Figure 2 statistics corrected by publisher (see erratum).\",\n", + ")\n", + "print(\"Invalidated:\", invalidated.entity_id, \"| invalidated flag:\", getattr(invalidated, \"invalidated\", True))\n", + "\n", + "stats = prov.get_statistics()\n", + "print(\"\\nStorage statistics:\", json.dumps(stats, indent=2, default=str))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 6: Verify Tamper-Evidence\n", + "\n", + "Each entry carries a deterministic SHA-256 checksum chained to the previous entry. Recompute and compare to detect any after-the-fact corruption of the provenance record." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# entry_biomass was returned by track_entity in Step 1\n", + "ok = verify_checksum(entry_biomass)\n", + "print(\"Checksum verified:\", ok)\n", + "\n", + "print(\"Computed:\", compute_checksum(entry_biomass)[:16], \"...\")\n", + "print(\"Stored: \", entry_biomass.checksum[:16] if getattr(entry_biomass, 'checksum', None) else \"(see entry fields)\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "| Need | Call |\n", + "|---|---|\n", + "| Record a fact's evidence | `prov.track_entity(entity_id, source, confidence=..., source_location=..., source_quote=...)` |\n", + "| Record a relationship | `prov.track_relationship(relationship_id, source, metadata=...)` |\n", + "| Full lineage of a fact | `prov.get_lineage(entity_id)` / `prov.trace_lineage(entity_id)` |\n", + "| \"Was it ever corrected?\" | `prov.revision_history(entity_id)` |\n", + "| \"Which sources support it?\" | `prov.get_all_sources(entity_id)` |\n", + "| Retract without deleting | `prov.invalidate(entity_id, agent_id, reason=...)` |\n", + "| Tamper check | `verify_checksum(entry)` |\n", + "\n", + "### Where to go next\n", + "\n", + "- **Conflict Detection and Resolution** (notebook 17) — what happens when two sources disagree.\n", + "- **Your First Knowledge Graph** (notebook 08) — plug `provenance=True` into extractors so tracking happens automatically during ingestion.\n", + "- The module docstring (`help(semantica.provenance)`) documents opt-in integration with `kg`, `split` and `conflicts` trackers." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}