KaifAhmad1
bdd99f7924
Adds an end-to-end cookbook use case that turns 9 real US federal AI-governance and cybersecurity-regulation documents into an explainable, ontology-driven knowledge graph: ingestion, chunking, entity/relation/triplet extraction, ontology import/generation/ evaluation (6 vendored real W3C ontologies plus SKOS taxonomy), entity resolution, SHACL validation, deterministic reasoning, PROV-O provenance, an Oxigraph-backed persistent RDF store, conflict detection, temporal reasoning, SPARQL, JSON-LD, GraphRAG retrieval, and a five-agent Decision Intelligence workflow. Real library rough edges hit along the way (noisy extraction over dense prose, EntityResolver's batch merge not firing, the stub OntologyValidator, find_precedents_advanced()'s vector-store bug, and two VectorStore/HybridSearch bugs that drop metadata or crash for non-inmemory backends) are reported honestly in the notebook output and README rather than hidden.
5.3 KiB
Vendored
Data
Real, official U.S. federal AI-governance and cybersecurity-regulation documents. No synthetic or LLM-generated content. Run python download_data.py to fetch everything into raw/. The script fails loudly if a source has moved rather than silently substituting placeholder text.
raw/source_manifest.json is generated by the download script and records the exact URL, retrieval timestamp, and byte size for every file. This is what the notebook's PROV-O step cites as each requirement clause's source.
Documents
| File | Document | Source | Sector | Parsed with |
|---|---|---|---|---|
nist_ai_rmf_1.0.pdf |
NIST AI Risk Management Framework (AI RMF 1.0), NIST AI 100-1 | nvlpubs.nist.gov | Cross-sector AI governance | PDFParser |
nist_csf_1.1.pdf |
NIST Cybersecurity Framework v1.1 (Apr 2018) | nvlpubs.nist.gov | Cross-sector cybersecurity | PDFParser |
nist_csf_2.0.pdf |
NIST Cybersecurity Framework 2.0, CSWP 29 (Feb 2024) | nvlpubs.nist.gov | Cross-sector cybersecurity | PDFParser |
nist_sp800-66r2_hipaa_security.pdf |
NIST SP 800-66 Rev. 2: Implementing the HIPAA Security Rule | nvlpubs.nist.gov | Healthcare | PDFParser |
hipaa_security_rule_45cfr164_subpart_c.xml |
HIPAA Security Rule, 45 CFR Part 164 Subpart C | eCFR versioner API | Healthcare | ingest_xml |
eo_14110_safe_secure_trustworthy_ai.pdf |
Executive Order 14110: Safe, Secure, and Trustworthy AI | Federal Register | Cross-sector AI policy | PDFParser |
omb_m24-10_ai_governance.pdf |
OMB Memorandum M-24-10 (Mar 2024) | whitehouse.gov | Cross-sector AI governance | PDFParser |
nist_ai_600-1_genai_profile.pdf |
NIST AI 600-1: Generative AI Profile (2024) | nvlpubs.nist.gov | Cross-sector AI governance | PDFParser |
fed_compliance_plan_omb_m24-10.pdf |
Federal Reserve: Compliance Plan for OMB M-24-10 (Sep 2024) | federalreserve.gov | Financial services | DoclingParser (optional, falls back to PDFParser) |
The Federal Reserve document is parsed with DoclingParser rather than PDFParser: a layout-aware, ML-based converter that preserves document structure (headings, tables) as Markdown instead of flattening to plain text. In a real run it recovered 16 real headings (for example ## Overview) from this document in about 33 seconds on CPU. It's used for one document, not all nine, because that per-page cost adds up fast. See the notebook's Step 1 for the accuracy and speed tradeoff this represents. If docling isn't installed, ingestion falls back to PDFParser automatically.
requirement_clauses.json
20 requirement clauses, hand-curated from the real ingested text above. Each text field is a verified real substring; the notebook asserts this before trusting any of them, and loads the file via Semantica's own JSONParser rather than as an inline Python literal. Each entry carries doc (which document it's from), sector, topic (a real SKOS concept, see ../ontology/skos/regulatory_taxonomy.ttl), citation (for example "45 CFR 164.308"), and text (the real matched substring).
Notes on sourcing
- HIPAA Security Rule is fetched via eCFR's public versioner API (
/api/versioner/v1/full/{date}/title-45.xml?part=164&subpart=C) rather than eCFR's regular web pages, which sit behind a bot-detection challenge that blocks plain HTTP clients. The API is eCFR's officially documented programmatic access path and returns the same authoritative text. The script resolves the current date dynamically via/api/versioner/v1/titles.json, so it keeps working as time passes. - Financial-services document: the original candidate, U.S. Treasury's "Managing Artificial Intelligence-Specific Cybersecurity Risks in the Financial Services Sector," is also blocked by bot-detection at
home.treasury.govwith no working API alternative found. It was substituted with the Federal Reserve's real, public compliance plan for OMB M-24-10, still a genuine financial-sector AI-governance document, and one that creates an actualimplementsrelationship back to the OMB M-24-10 document already in this dataset. - Every other URL returns the document directly with a plain
requests.get()and a descriptive User-Agent. No bypass techniques were used or needed.
Data dictionary (what the notebook extracts)
Each document is ingested as one reg:Regulation, which is also a dcat:Dataset. The notebook's Step 6 loads reg:RequirementClause instances from requirement_clauses.json, individual obligations, controls, and definitions, each carrying a reg:sourceCitation (for example "45 CFR 164.308") pointing back to the exact real-document location it came from.