{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/03_Threat_Correlation.ipynb)\n",
    "\n",
    "# Threat Correlation Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete threat correlation pipeline for cybersecurity: ingest threat feeds from multiple sources, extract IOCs, build temporal knowledge graph, correlate threats, detect campaigns, and generate reports.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, RepoIngestor, EmailIngestor, MCPIngestor\n",
    "- **Parsing**: XMLParser, StructuredDataParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector\n",
    "- **KG**: GraphBuilder, TemporalGraphQuery, TemporalPatternDetector, GraphAnalyzer, ConnectivityAnalyzer\n",
    "- **Reasoning**: InferenceEngine, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor, ProvenanceTracker, ConflictDetector\n",
    "- **Export**: RDFExporter, ReportGenerator\n",
    "- **Visualization**: AnalyticsVisualizer, TemporalVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Multiple Threat Feeds → Parse → Extract IOCs → Build Temporal KG → Correlate Threats → Detect Campaigns → Generate Reports → Visualize**\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Ingest Threat Feeds\n",
    "\n",
    "Ingest threat intelligence from multiple sources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, FeedIngestor, DBIngestor, WebIngestor\n",
    "from semantica.parse import XMLParser, StructuredDataParser, JSONParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector\n",
    "from semantica.kg import GraphBuilder, TemporalGraphQuery, TemporalPatternDetector, GraphAnalyzer, ConnectivityAnalyzer\n",
    "from semantica.reasoning import InferenceEngine, ExplanationGenerator\n",
    
    "from semantica.kg import ProvenanceTracker\n",
    "from semantica.conflicts import ConflictDetector\n",
    "from semantica.export import RDFExporter, ReportGenerator\n",
    "from semantica.visualization import AnalyticsVisualizer, TemporalVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "xml_parser = XMLParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "json_parser = JSONParser()\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Real threat intelligence feed URLs\n",
    "threat_feeds = [\n",
    "    \"https://www.cisa.gov/news.xml\",  # CISA Security Advisories\n",
    "    \"https://www.us-cert.gov/ncas/alerts.xml\",  # US-CERT Alerts\n",
    "    \"https://feeds.feedburner.com/SecurityWeek\",  # Security Week\n",
    "    \"https://www.darkreading.com/rss.xml\"  # Dark Reading\n",
    "]\n",
    "\n",
    "# Real database connection pattern (PostgreSQL example)\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/threat_intel_db\"\n",
    "db_query = \"SELECT ioc, ioc_type, timestamp, severity, source FROM threat_indicators WHERE timestamp > NOW() - INTERVAL '7 days'\"\n",
    "\n",
    "# Real web API endpoints for threat intelligence\n",
    "threat_apis = [\n",
    "    \"https://api.github.com/repos/mitre/cti/contents/enterprise-attack/attack-pattern\",  # MITRE ATT&CK\n",
    "    \"https://www.virustotal.com/vtapi/v2/domain/report\",  # VirusTotal API (requires API key)\n",
    "    \"https://api.shodan.io/shodan/host/search\"  # Shodan API (requires API key)\n",
    "]\n",
    "\n",
    "# Ingest from real RSS feeds\n",
    "feed_data_list = []\n",
    "for feed_url in threat_feeds:\n",
    "    feed_data = feed_ingestor.ingest_feed(feed_url)\n",
    "    if feed_data:\n",
    "        feed_data_list.append(feed_data)\n",
    "        print(f\"  Ingested feed: {feed_url}\")\n",
    "        print(f\"  Items: {len(feed_data.items) if hasattr(feed_data, 'items') else 0}\")\n",
    "\n",
    "# Ingest from web APIs (example with public API)\n",
    "web_content = web_ingestor.ingest_url(\"https://api.github.com/repos/mitre/cti\")\n",
    "if web_content:\n",
    "    print(f\"  Ingested web content\")\n",
    "\n",
    "# Database ingestion pattern (would connect to real database)\n",
    "# Example: Export from threat intelligence database\n",
    "db_data = db_ingestor.export_table(\n",
    "    connection_string=db_connection_string,\n",
    "    table_name=\"threat_indicators\",\n",
    "    limit=1000\n",
    ")\n",
    "print(f\"  Query pattern: {db_query}\")\n",
    "# Simulate database structure for demonstration if needed\n",
    "if not db_data:\n",
    "    db_data = {\n",
    "        \"data\": [\n",
    "            {\"ioc\": \"192.168.1.100\", \"ioc_type\": \"IP\", \"timestamp\": datetime.now().isoformat(), \"severity\": \"high\", \"source\": \"threat_feed\"},\n",
    "            {\"ioc\": \"malicious-domain.com\", \"ioc_type\": \"Domain\", \"timestamp\": datetime.now().isoformat(), \"severity\": \"medium\", \"source\": \"threat_feed\"}\n",
    "        ]\n",
    "    }\n",
    "\n",
    "# Parse feed data\n",
    "parsed_feeds = []\n",
    "for feed_data in feed_data_list:\n",
    "    if hasattr(feed_data, 'items'):\n",
    "        for item in feed_data.items[:10]:  # Process first 10 items\n",
    "            parsed_feeds.append({\n",
    "                \"title\": item.title if hasattr(item, 'title') else \"\",\n",
    "                \"description\": item.description if hasattr(item, 'description') else \"\",\n",
    "                \"published\": item.published if hasattr(item, 'published') else \"\",\n",
    "                \"link\": item.link if hasattr(item, 'link') else \"\"\n",
    "            })\n",
    "\n",
    "parsed_db = structured_parser.parse_json(json.dumps(db_data)) if db_data else None\n",
    "\n",
    "print(f\"\\n📊 Ingestion Summary:\")\n",
    "print(f\"  Feeds ingested: {len(feed_data_list)}\")\n",
    "print(f\"  Feed items processed: {len(parsed_feeds)}\")\n",
    "print(f\"  Database records: {len(db_data.get('data', [])) if db_data else 0}\")\n",
    "print(f\"  Web sources: 1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract IOCs\n",
    "\n",
    "Extract Indicators of Compromise (IOCs) from threat feeds.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "event_detector = EventDetector()\n",
    "\n",
    "all_threat_texts = []\n",
    "if parsed_xml and parsed_xml.elements:\n",
    "    for elem in parsed_xml.elements:\n",
    "        if hasattr(elem, 'text') and elem.text:\n",
    "            all_threat_texts.append(elem.text)\n",
    "\n",
    "for db_record in parsed_db.get(\"data\", threat_db_data):\n",
    "    threat_text = f\"IOC: {db_record.get('ioc', '')} Type: {db_record.get('type', '')} Severity: {db_record.get('severity', '')}\"\n",
    "    all_threat_texts.append(threat_text)\n",
    "\n",
    "all_entities = []\n",
    "all_relationships = []\n",
    "all_events = []\n",
    "\n",
    "for text in all_threat_texts:\n",
    "    entities = ner_extractor.extract(text)\n",
    "    all_entities.extend(entities)\n",
    "    \n",
    "    relationships = relation_extractor.extract(text, entities)\n",
    "    all_relationships.extend(relationships)\n",
    "    \n",
    "    events = event_detector.detect_events(text)\n",
    "    all_events.extend(events)\n",
    "\n",
    "print(f\"Extracted {len(all_entities)} IOCs\")\n",
    "print(f\"Extracted {len(all_relationships)} relationships\")\n",
    "print(f\"Detected {len(all_events)} events\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build Temporal Knowledge Graph\n",
    "\n",
    "Build a temporal knowledge graph from extracted IOCs and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "\n",
    "threat_entities = []\n",
    "for i, entity in enumerate(all_entities[:10], 1):\n",
    "    threat_entities.append({\n",
    "        \"id\": f\"ioc_{i}\",\n",
    "        \"type\": entity.get(\"type\", \"IOC\"),\n",
    "        \"name\": entity.get(\"text\", entity.get(\"entity\", \"\")),\n",
    "        \"properties\": {\"timestamp\": datetime.now().isoformat()}\n",
    "    })\n",
    "\n",
    "threat_relationships = []\n",
    "for i, rel in enumerate(all_relationships[:5], 1):\n",
    "    threat_relationships.append({\n",
    "        \"source\": f\"ioc_{i}\",\n",
    "        \"target\": f\"ioc_{i+1}\",\n",
    "        \"type\": rel.get(\"type\", \"related_to\"),\n",
    "        \"properties\": {\"timestamp\": datetime.now().isoformat()}\n",
    "    })\n",
    "\n",
    "threat_kg = builder.build(threat_entities, threat_relationships)\n",
    "\n",
    "print(f\"Built temporal knowledge graph\")\n",
    "print(f\"  Entities: {len(threat_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(threat_kg.get('relationships', []))}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Correlate Threats\n",
    "\n",
    "Correlate threats using temporal queries and inference.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temporal_query = TemporalGraphQuery()\n",
    "pattern_detector = TemporalPatternDetector()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "inference_engine = InferenceEngine()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "start_time = (datetime.now() - timedelta(days=7)).isoformat()\n",
    "end_time = datetime.now().isoformat()\n",
    "\n",
    "temporal_results = temporal_query.query_time_range(\n",
    "    graph=threat_kg,\n",
    "    query=\"Find threats in the last 7 days\",\n",
    "    start_time=start_time,\n",
    "    end_time=end_time\n",
    ")\n",
    "\n",
    "patterns = pattern_detector.detect_temporal_patterns(\n",
    "    threat_kg,\n",
    "    pattern_type=\"sequence\",\n",
    "    min_frequency=1\n",
    ")\n",
    "\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(threat_kg)\n",
    "\n",
    "inference_engine.add_rule(\"IF IOC has high severity AND IOC is related to another IOC THEN potential_campaign\")\n",
    "for entity in threat_entities[:3]:\n",
    "    if entity.get(\"properties\", {}).get(\"severity\") == \"high\":\n",
    "        inference_engine.add_fact({\"ioc\": entity.get(\"id\"), \"severity\": \"high\"})\n",
    "\n",
    "correlations = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"Temporal query returned {len(temporal_results.get('entities', []))} entities\")\n",
    "print(f\"Detected {len(patterns)} temporal patterns\")\n",
    "print(f\"Connectivity: {connectivity.get('is_connected', False)}\")\n",
    "print(f\"Inferred {len(correlations)} correlations\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Detect Campaigns\n",
    "\n",
    "Detect threat campaigns using graph analysis and inference.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "campaigns = []\n",
    "\n",
    "if len(patterns) > 0:\n",
    "    campaigns.append({\n",
    "        \"campaign_id\": \"campaign_1\",\n",
    "        \"description\": \"Detected threat campaign based on temporal patterns\",\n",
    "        \"iocs\": [e.get(\"id\") for e in threat_entities[:3]],\n",
    "        \"severity\": \"high\",\n",
    "        \"patterns\": len(patterns)\n",
    "    })\n",
    "\n",
    "if correlations:\n",
    "    campaigns.append({\n",
    "        \"campaign_id\": \"campaign_2\",\n",
    "        \"description\": \"Detected campaign from inference correlations\",\n",
    "        \"iocs\": [e.get(\"id\") for e in threat_entities[:2]],\n",
    "        \"severity\": \"medium\",\n",
    "        \"correlations\": len(correlations)\n",
    "    })\n",
    "\n",
    "print(f\"Detected {len(campaigns)} threat campaigns\")\n",
    "for campaign in campaigns:\n",
    "    print(f\"  Campaign: {campaign['campaign_id']} - Severity: {campaign['severity']}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Quality Assessment and Provenance\n",
    "\n",
    "Assess graph quality and track provenance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "provenance_tracker = ProvenanceTracker()\n",
    "conflict_detector = ConflictDetector()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(threat_kg)\n",
    "\n",
    "for entity in threat_entities:\n",
    "    provenance_tracker.track_entity(entity.get(\"id\"), \"threat_feed\", entity)\n",
    "\n",
    "conflicts = conflict_detector.detect_value_conflicts(threat_entities, \"name\")\n",
    "\n",
    "print(f\"Graph quality score: {quality_score.get('overall_score', 0):.3f}\")\n",
    "print(f\"Tracked provenance for {len(threat_entities)} entities\")\n",
    "print(f\"Detected {len(conflicts)} conflicts\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Generate Reports\n",
    "\n",
    "Generate threat intelligence reports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "rdf_exporter.export_knowledge_graph(threat_kg, os.path.join(temp_dir, \"threats.rdf\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Threat correlation analysis detected {len(campaigns)} campaigns\",\n",
    "    \"iocs\": len(threat_entities),\n",
    "    \"campaigns\": campaigns,\n",
    "    \"quality_score\": quality_score.get('overall_score', 0)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "print(f\"Report length: {len(report)} characters\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Visualize Results\n",
    "\n",
    "Visualize threat correlation results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(threat_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(threat_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Multi-source ingestion → Extraction → Temporal KG → Correlation → Campaign Detection → Quality → Reports → Visualization\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
