{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/06_Regulatory_Compliance.ipynb)\n",
    "\n",
    "# Regulatory Compliance Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete regulatory compliance pipeline: ingest regulatory documents from multiple sources (SEC, FINRA, regulatory databases), extract compliance rules, build compliance ontology, validate compliance, and generate compliance reports.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n",
    "- **Parsing**: DocumentParser, PDFParser, HTMLParser, StructuredDataParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, TripleExtractor, SemanticAnalyzer\n",
    "- **KG**: GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer\n",
    "- **Ontology**: OntologyGenerator, ClassInferrer, PropertyGenerator, OntologyValidator\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor, ValidationEngine, ConflictDetector\n",
    "- **Export**: JSONExporter, CSVExporter, RDFExporter, OWLExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, OntologyVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Regulatory Documents → Parse → Extract Compliance Rules → Build Compliance Ontology → Validate Compliance → Generate Reports → Visualize**\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Ingest Regulatory Documents from Multiple Sources\n",
    "\n",
    "Ingest regulatory documents from SEC, FINRA, and regulatory databases.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, WebIngestor, DBIngestor, FeedIngestor\n",
    "from semantica.parse import DocumentParser, PDFParser, HTMLParser, StructuredDataParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, TripleExtractor, SemanticAnalyzer\n",
    "from semantica.kg import GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer\n",
    "from semantica.ontology import OntologyGenerator, ClassInferrer, PropertyGenerator, OntologyValidator\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    
    "from semantica.conflicts import ConflictDetector\n",
    "from semantica.export import JSONExporter, CSVExporter, RDFExporter, OWLExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, OntologyVisualizer, AnalyticsVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "\n",
    "document_parser = DocumentParser()\n",
    "pdf_parser = PDFParser()\n",
    "html_parser = HTMLParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "\n",
    "# Real regulatory data sources\n",
    "regulatory_sources = [\n",
    "    \"https://www.sec.gov/rules/final.shtml\",  # SEC Final Rules\n",
    "    \"https://www.finra.org/rules-guidance\",  # FINRA Rules\n",
    "    \"https://www.federalreserve.gov/newsevents/pressreleases.htm\"  # Federal Reserve\n",
    "]\n",
    "\n",
    "regulatory_feeds = [\n",
    "    \"https://feeds.reuters.com/reuters/businessNews\",\n",
    "    \"https://rss.cnn.com/rss/money_latest.rss\"\n",
    "]\n",
    "\n",
    "# Real database connection for regulatory documents\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/regulatory_db\"\n",
    "db_query = \"SELECT regulation_id, title, effective_date, compliance_requirements FROM regulations WHERE effective_date > CURRENT_DATE - INTERVAL '1 year' ORDER BY effective_date DESC\"\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample regulatory document data\n",
    "regulatory_file = os.path.join(temp_dir, \"regulatory_document.json\")\n",
    "regulatory_data = {\n",
    "    \"regulation_id\": \"REG-2024-001\",\n",
    "    \"title\": \"Data Privacy and Security Requirements\",\n",
    "    \"effective_date\": (datetime.now() - timedelta(days=60)).isoformat(),\n",
    "    \"compliance_requirements\": [\n",
    "        \"Encrypt sensitive customer data\",\n",
    "        \"Maintain audit logs for 7 years\",\n",
    "        \"Report breaches within 72 hours\",\n",
    "        \"Conduct annual security assessments\"\n",
    "    ],\n",
    "    \"applicable_entities\": [\"Financial Institutions\", \"Broker-Dealers\", \"Investment Advisors\"],\n",
    "    \"penalties\": {\n",
    "        \"non_compliance\": \"Fines up to $1M per violation\",\n",
    "        \"willful_violation\": \"Criminal penalties\"\n",
    "    }\n",
    "}\n",
    "\n",
    "with open(regulatory_file, 'w') as f:\n",
    "    json.dump(regulatory_data, f, indent=2)\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(regulatory_file, read_content=True)\n",
    "parsed_data = structured_parser.parse_json(regulatory_file)\n",
    "\n",
    "# Ingest from regulatory sources\n",
    "regulatory_web_list = []\n",
    "for source_url in regulatory_sources[:1]:\n",
    "    web_content = web_ingestor.ingest_url(source_url)\n",
    "    if web_content:\n",
    "        regulatory_web_list.append(web_content)\n",
    "        print(f\"  Ingested regulatory source: {source_url}\")\n",
    "\n",
    "print(f\"\\n📊 Ingestion Summary:\")\n",
    "print(f\"  Regulatory documents: {len([file_objects]) if file_objects else 0}\")\n",
    "print(f\"  Regulatory web sources: {len(regulatory_web_list)}\")\n",
    "print(f\"  Database sources: 1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract Compliance Rules\n",
    "\n",
    "Extract compliance rules and requirements from regulatory documents.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "triple_extractor = TripleExtractor()\n",
    "semantic_analyzer = SemanticAnalyzer()\n",
    "\n",
    "compliance_entities = []\n",
    "compliance_relationships = []\n",
    "\n",
    "# Extract from regulatory data\n",
    "if parsed_data and parsed_data.data:\n",
    "    regulation = parsed_data.data if isinstance(parsed_data.data, dict) else parsed_data.data[0] if isinstance(parsed_data.data, list) else {}\n",
    "    \n",
    "    if isinstance(regulation, dict):\n",
    "        # Regulation entity\n",
    "        compliance_entities.append({\n",
    "            \"id\": regulation.get(\"regulation_id\", \"\"),\n",
    "            \"type\": \"Regulation\",\n",
    "            \"name\": regulation.get(\"title\", \"\"),\n",
    "            \"properties\": {\n",
    "                \"effective_date\": regulation.get(\"effective_date\", \"\"),\n",
    "                \"regulation_id\": regulation.get(\"regulation_id\", \"\")\n",
    "            }\n",
    "        })\n",
    "        \n",
    "        # Compliance requirements\n",
    "        for i, requirement in enumerate(regulation.get(\"compliance_requirements\", [])):\n",
    "            compliance_entities.append({\n",
    "                \"id\": f\"{regulation.get('regulation_id', '')}_req_{i}\",\n",
    "                \"type\": \"Compliance_Requirement\",\n",
    "                \"name\": requirement,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            compliance_relationships.append({\n",
    "                \"source\": regulation.get(\"regulation_id\", \"\"),\n",
    "                \"target\": f\"{regulation.get('regulation_id', '')}_req_{i}\",\n",
    "                \"type\": \"has_requirement\",\n",
    "                \"properties\": {}\n",
    "            })\n",
    "        \n",
    "        # Applicable entities\n",
    "        for entity_type in regulation.get(\"applicable_entities\", []):\n",
    "            compliance_entities.append({\n",
    "                \"id\": entity_type,\n",
    "                \"type\": \"Regulated_Entity\",\n",
    "                \"name\": entity_type,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            compliance_relationships.append({\n",
    "                \"source\": regulation.get(\"regulation_id\", \"\"),\n",
    "                \"target\": entity_type,\n",
    "                \"type\": \"applies_to\",\n",
    "                \"properties\": {}\n",
    "            })\n",
    "\n",
    "print(f\"Extracted {len(compliance_entities)} compliance entities\")\n",
    "print(f\"Extracted {len(compliance_relationships)} compliance relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build Compliance Ontology\n",
    "\n",
    "Build compliance ontology from extracted rules and requirements.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "ontology_generator = OntologyGenerator()\n",
    "class_inferrer = ClassInferrer()\n",
    "property_generator = PropertyGenerator()\n",
    "ontology_validator = OntologyValidator()\n",
    "\n",
        "compliance_kg = builder.build(compliance_entities, compliance_relationships)\n",
        "\n",
        "compliance_ontology = ontology_generator.generate_ontology({\"entities\": compliance_entities, \"relationships\": compliance_relationships}, entities=compliance_entities, relationships=compliance_relationships)\n",
        "\n",
        "classes = class_inferrer.infer_classes(compliance_entities)\n",
        "properties = property_generator.infer_properties(compliance_entities, compliance_relationships, classes)\n",
        "\n",
        "validation_result = ontology_validator.validate_ontology(compliance_ontology)\n",
    "\n",
    "print(f\"Built compliance knowledge graph\")\n",
    "print(f\"  Entities: {len(compliance_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(compliance_kg.get('relationships', []))}\")\n",
    "print(f\"Generated compliance ontology\")\n",
    "print(f\"  Classes: {len(compliance_ontology.get('classes', []))}\")\n",
    "print(f\"  Properties: {len(compliance_ontology.get('properties', []))}\")\n",
    "print(f\"  Ontology valid: {validation_result.valid}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Validate Compliance\n",
    "\n",
    "Validate data against compliance rules using inference engine.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "validation_engine = ValidationEngine()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "\n",
    "# Define compliance validation rules\n",
    "inference_engine.add_rule(\"IF data_encrypted is true AND audit_logs_maintained is true THEN data_security_compliant\")\n",
    "inference_engine.add_rule(\"IF breach_reported_within_72h is true THEN breach_reporting_compliant\")\n",
    "inference_engine.add_rule(\"IF annual_assessment_conducted is true THEN assessment_compliant\")\n",
    "\n",
    "# Sample data to validate\n",
    "sample_data = {\n",
    "    \"data_encrypted\": True,\n",
    "    \"audit_logs_maintained\": True,\n",
    "    \"breach_reported_within_72h\": True,\n",
    "    \"annual_assessment_conducted\": True\n",
    "}\n",
    "\n",
    "# Add facts for validation\n",
    "for key, value in sample_data.items():\n",
    "    inference_engine.add_fact({key: value})\n",
    "\n",
    "compliance_status = inference_engine.forward_chain()\n",
    "\n",
    "# Analyze compliance graph\n",
    "metrics = graph_analyzer.compute_metrics(compliance_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(compliance_kg)\n",
    "\n",
    "print(f\"Compliance validation complete\")\n",
    "print(f\"  Compliance status: {len(compliance_status)} rules satisfied\")\n",
    "print(f\"  Graph metrics: density {metrics.get('density', 0):.3f}\")\n",
    "print(f\"  Connected components: {len(connectivity.get('components', []))}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Generate Compliance Reports\n",
    "\n",
    "Generate comprehensive compliance reports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "csv_exporter = CSVExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "owl_exporter = OWLExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(compliance_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(compliance_kg, os.path.join(temp_dir, \"compliance_kg.json\"))\n",
    "csv_exporter.export_entities(compliance_entities, os.path.join(temp_dir, \"compliance_entities.csv\"))\n",
    "rdf_exporter.export_knowledge_graph(compliance_kg, os.path.join(temp_dir, \"compliance_kg.rdf\"))\n",
    "owl_exporter.export(compliance_ontology, os.path.join(temp_dir, \"compliance_ontology.owl\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Compliance validation identified {len(compliance_status)} satisfied rules\",\n",
    "    \"regulations_analyzed\": len([e for e in compliance_entities if e.get(\"type\") == \"Regulation\"]),\n",
    "    \"requirements\": len([e for e in compliance_entities if e.get(\"type\") == \"Compliance_Requirement\"]),\n",
    "    \"compliance_status\": len(compliance_status),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "print(f\"Report length: {len(report)} characters\")\n",
    "print(f\"Graph quality score: {quality_score.get('overall_score', 0):.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Visualize Compliance\n",
    "\n",
    "Visualize compliance knowledge graph and ontology.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kg_visualizer = KGVisualizer()\n",
    "ontology_visualizer = OntologyVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(compliance_kg, output=\"interactive\")\n",
    "ontology_viz = ontology_visualizer.visualize_hierarchy(compliance_ontology, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(compliance_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Regulatory Documents → Parse → Extract Rules → Build Ontology → Validate Compliance → Generate Reports → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
