{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Incident_Analysis.ipynb)\n",
    "\n",
    "# Incident Analysis Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete security incident analysis pipeline: ingest security logs from multiple sources (files, databases, streams), parse structured and unstructured logs, extract security entities, build knowledge graph, analyze relationships, detect anomalies, and generate incident reports.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, RepoIngestor, EmailIngestor, MCPIngestor\n",
    "- **Parsing**: JSONParser, XMLParser, StructuredDataParser, DocumentParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "- **KG**: GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer, CentralityCalculator\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor, ConflictDetector, ProvenanceTracker\n",
    "- **Export**: JSONExporter, RDFExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, AnalyticsVisualizer, TemporalVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Multiple Security Sources → Parse Logs → Extract Security Entities → Build Incident KG → Analyze Relationships → Detect Anomalies → Generate Reports → Visualize**\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Ingest Security Logs from Multiple Sources\n",
    "\n",
    "Ingest security logs from files, databases, streams, and threat intelligence feeds.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, DBIngestor, StreamIngestor, FeedIngestor\n",
    "from semantica.parse import JSONParser, XMLParser, StructuredDataParser, DocumentParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "from semantica.kg import GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer, CentralityCalculator\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    
    "from semantica.conflicts import ConflictDetector\n",
    "from semantica.kg import ProvenanceTracker\n",
    "from semantica.export import JSONExporter, RDFExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, AnalyticsVisualizer, TemporalVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "stream_ingestor = StreamIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "\n",
    "json_parser = JSONParser()\n",
    "xml_parser = XMLParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "document_parser = DocumentParser()\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Real-world security log formats\n",
    "security_logs_json = os.path.join(temp_dir, \"security_logs.json\")\n",
    "security_logs_data = [\n",
    "    {\n",
    "        \"timestamp\": (datetime.now() - timedelta(hours=2)).isoformat(),\n",
    "        \"source_ip\": \"192.168.1.100\",\n",
    "        \"destination_ip\": \"10.0.0.50\",\n",
    "        \"event_type\": \"failed_login\",\n",
    "        \"user\": \"admin\",\n",
    "        \"severity\": \"medium\",\n",
    "        \"message\": \"Multiple failed login attempts detected\"\n",
    "    },\n",
    "    {\n",
    "        \"timestamp\": (datetime.now() - timedelta(hours=1)).isoformat(),\n",
    "        \"source_ip\": \"203.0.113.45\",\n",
    "        \"destination_ip\": \"10.0.0.50\",\n",
    "        \"event_type\": \"port_scan\",\n",
    "        \"severity\": \"high\",\n",
    "        \"message\": \"Port scanning activity detected from external IP\"\n",
    "    },\n",
    "    {\n",
    "        \"timestamp\": (datetime.now() - timedelta(minutes=30)).isoformat(),\n",
    "        \"source_ip\": \"192.168.1.100\",\n",
    "        \"destination_ip\": \"10.0.0.75\",\n",
    "        \"event_type\": \"data_exfiltration\",\n",
    "        \"user\": \"user123\",\n",
    "        \"severity\": \"critical\",\n",
    "        \"message\": \"Large data transfer detected to external server\"\n",
    "    }\n",
    "]\n",
    "\n",
    "with open(security_logs_json, 'w') as f:\n",
    "    json.dump(security_logs_data, f, indent=2)\n",
    "\n",
    "# XML format security events (common in SIEM systems)\n",
    "security_events_xml = os.path.join(temp_dir, \"security_events.xml\")\n",
    "xml_content = \"\"\"<?xml version=\"1.0\"?>\n",
    "<security_events>\n",
    "    <event>\n",
    "        <timestamp>2024-01-15T14:30:00</timestamp>\n",
    "        <source_ip>172.16.0.10</source_ip>\n",
    "        <destination_ip>10.0.0.50</destination_ip>\n",
    "        <event_type>malware_detection</event_type>\n",
    "        <severity>high</severity>\n",
    "        <description>Malware signature detected in file transfer</description>\n",
    "    </event>\n",
    "    <event>\n",
    "        <timestamp>2024-01-15T15:00:00</timestamp>\n",
    "        <source_ip>192.168.1.200</source_ip>\n",
    "        <destination_ip>10.0.0.50</destination_ip>\n",
    "        <event_type>unauthorized_access</event_type>\n",
    "        <severity>critical</severity>\n",
    "        <description>Unauthorized access attempt to restricted resource</description>\n",
    "    </event>\n",
    "</security_events>\"\"\"\n",
    "\n",
    "with open(security_events_xml, 'w') as f:\n",
    "    f.write(xml_content)\n",
    "\n",
    "# Ingest from files\n",
    "file_objects = file_ingestor.ingest_file(security_logs_json, read_content=True)\n",
    "file_objects_xml = file_ingestor.ingest_file(security_events_xml, read_content=True)\n",
    "\n",
    "# Parse structured logs\n",
    "parsed_json = json_parser.parse(security_logs_json)\n",
    "parsed_xml = xml_parser.parse(security_events_xml)\n",
    "\n",
    "# Real security intelligence feed URLs\n",
    "security_feeds = [\n",
    "    \"https://www.cisa.gov/news.xml\",  # CISA Security Advisories\n",
    "    \"https://www.us-cert.gov/ncas/alerts.xml\",  # US-CERT Alerts\n",
    "    \"https://feeds.feedburner.com/SecurityWeek\",  # Security Week\n",
    "    \"https://www.darkreading.com/rss.xml\"  # Dark Reading\n",
    "]\n",
    "\n",
    "threat_feed_list = []\n",
    "for feed_url in security_feeds:\n",
    "    threat_feed = feed_ingestor.ingest_feed(feed_url)\n",
    "    if threat_feed:\n",
    "        threat_feed_list.append(threat_feed)\n",
    "        print(f\"  Ingested feed: {feed_url}\")\n",
    "\n",
    "print(f\"Ingested {len([file_objects]) if file_objects else 0} JSON log files\")\n",
    "print(f\"Ingested {len([file_objects_xml]) if file_objects_xml else 0} XML event files\")\n",
    "print(f\"Parsed {len(parsed_json.data) if parsed_json and parsed_json.data else 0} JSON log entries\")\n",
    "print(f\"Parsed {len(parsed_xml.elements) if parsed_xml else 0} XML event elements\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract Security Entities and Relationships\n",
    "\n",
    "Extract security entities (IPs, users, events) and relationships from parsed logs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "event_detector = EventDetector()\n",
    "triple_extractor = TripleExtractor()\n",
    "\n",
    "all_entities = []\n",
    "all_relationships = []\n",
    "all_events = []\n",
    "\n",
    "# Extract from JSON logs\n",
    "if parsed_json and parsed_json.data:\n",
    "    for log_entry in parsed_json.data:\n",
    "        if isinstance(log_entry, dict):\n",
    "            log_text = f\"{log_entry.get('event_type', '')} from {log_entry.get('source_ip', '')} to {log_entry.get('destination_ip', '')}: {log_entry.get('message', '')}\"\n",
    "            \n",
    "            entities = ner_extractor.extract(log_text)\n",
    "            all_entities.extend(entities)\n",
    "            \n",
    "            relationships = relation_extractor.extract(log_text, entities)\n",
    "            all_relationships.extend(relationships)\n",
    "            \n",
    "            events = event_detector.detect_events(log_text)\n",
    "            all_events.extend(events)\n",
    "\n",
    "# Extract from XML events\n",
    "if parsed_xml and parsed_xml.elements:\n",
    "    for elem in parsed_xml.elements:\n",
    "        if hasattr(elem, 'text') and elem.text:\n",
    "            entities = ner_extractor.extract(elem.text)\n",
    "            all_entities.extend(entities)\n",
    "            \n",
    "            relationships = relation_extractor.extract(elem.text, entities)\n",
    "            all_relationships.extend(relationships)\n",
    "\n",
    "# Build structured entities from log data\n",
    "security_entities = []\n",
    "for log_entry in parsed_json.data if parsed_json and parsed_json.data else []:\n",
    "    if isinstance(log_entry, dict):\n",
    "        security_entities.append({\n",
    "            \"id\": log_entry.get(\"source_ip\", \"\"),\n",
    "            \"type\": \"IP_Address\",\n",
    "            \"name\": log_entry.get(\"source_ip\", \"\"),\n",
    "            \"properties\": {\"source\": \"security_logs\"}\n",
    "        })\n",
    "        security_entities.append({\n",
    "            \"id\": log_entry.get(\"destination_ip\", \"\"),\n",
    "            \"type\": \"IP_Address\",\n",
    "            \"name\": log_entry.get(\"destination_ip\", \"\"),\n",
    "            \"properties\": {\"source\": \"security_logs\"}\n",
    "        })\n",
    "        if log_entry.get(\"user\"):\n",
    "            security_entities.append({\n",
    "                \"id\": log_entry.get(\"user\", \"\"),\n",
    "                \"type\": \"User\",\n",
    "                \"name\": log_entry.get(\"user\", \"\"),\n",
    "                \"properties\": {\"source\": \"security_logs\"}\n",
    "            })\n",
    "        security_entities.append({\n",
    "            \"id\": log_entry.get(\"event_type\", \"\"),\n",
    "            \"type\": \"Security_Event\",\n",
    "            \"name\": log_entry.get(\"event_type\", \"\"),\n",
    "            \"properties\": {\n",
    "                \"severity\": log_entry.get(\"severity\", \"\"),\n",
    "                \"timestamp\": log_entry.get(\"timestamp\", \"\"),\n",
    "                \"message\": log_entry.get(\"message\", \"\")\n",
    "            }\n",
    "        })\n",
    "\n",
    "incident_relationships = []\n",
    "for log_entry in parsed_json.data if parsed_json and parsed_json.data else []:\n",
    "    if isinstance(log_entry, dict):\n",
    "        incident_relationships.append({\n",
    "            \"source\": log_entry.get(\"source_ip\", \"\"),\n",
    "            \"target\": log_entry.get(\"event_type\", \"\"),\n",
    "            \"type\": \"triggered\",\n",
    "            \"properties\": {\"timestamp\": log_entry.get(\"timestamp\", \"\")}\n",
    "        })\n",
    "        incident_relationships.append({\n",
    "            \"source\": log_entry.get(\"event_type\", \"\"),\n",
    "            \"target\": log_entry.get(\"destination_ip\", \"\"),\n",
    "            \"type\": \"targeted\",\n",
    "            \"properties\": {\"timestamp\": log_entry.get(\"timestamp\", \"\")}\n",
    "        })\n",
    "\n",
    "print(f\"Extracted {len(security_entities)} security entities\")\n",
    "print(f\"Extracted {len(incident_relationships)} incident relationships\")\n",
    "print(f\"Detected {len(all_events)} security events\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build Incident Knowledge Graph\n",
    "\n",
    "Build a knowledge graph from security entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "centrality_calculator = CentralityCalculator()\n",
    "provenance_tracker = ProvenanceTracker()\n",
    "\n",
    "incident_kg = builder.build(security_entities, incident_relationships)\n",
    "\n",
    "# Track provenance\n",
    "for entity in security_entities:\n",
    "    provenance_tracker.track_entity(entity.get(\"id\"), entity.get(\"properties\", {}).get(\"source\", \"unknown\"), entity)\n",
    "\n",
    "# Analyze graph structure\n",
    "metrics = graph_analyzer.compute_metrics(incident_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(incident_kg)\n",
    "centrality_result = centrality_calculator.calculate_degree_centrality(incident_kg)\n",
    "centrality_scores = centrality_result.get('centrality', {})\n",
    "\n",
    "print(f\"Built incident knowledge graph\")\n",
    "print(f\"  Entities: {len(incident_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(incident_kg.get('relationships', []))}\")\n",
    "print(f\"  Graph density: {metrics.get('density', 0):.3f}\")\n",
    "print(f\"  Connected components: {len(connectivity.get('components', []))}\")\n",
    "print(f\"  Central entities: {len([e for e, score in centrality_scores.items() if score > 0])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Analyze Relationships and Detect Anomalies\n",
    "\n",
    "Analyze security relationships and detect anomalous patterns.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "conflict_detector = ConflictDetector()\n",
    "\n",
    "# Define security rules\n",
    "inference_engine.add_rule(\"IF event_type is port_scan AND severity is high THEN potential_intrusion\")\n",
    "inference_engine.add_rule(\"IF event_type is data_exfiltration AND severity is critical THEN data_breach\")\n",
    "inference_engine.add_rule(\"IF multiple failed_login events from same source_ip THEN brute_force_attack\")\n",
    "\n",
    "# Add facts from security events\n",
    "for log_entry in parsed_json.data if parsed_json and parsed_json.data else []:\n",
    "    if isinstance(log_entry, dict):\n",
    "        inference_engine.add_fact({\n",
    "            \"event_type\": log_entry.get(\"event_type\", \"\"),\n",
    "            \"severity\": log_entry.get(\"severity\", \"\"),\n",
    "            \"source_ip\": log_entry.get(\"source_ip\", \"\")\n",
    "        })\n",
    "\n",
    "# Run inference\n",
    "inferred_threats = inference_engine.forward_chain()\n",
    "\n",
    "# Detect anomalies based on patterns\n",
    "anomalies = []\n",
    "for log_entry in parsed_json.data if parsed_json and parsed_json.data else []:\n",
    "    if isinstance(log_entry, dict):\n",
    "        anomaly_score = 0\n",
    "        reasons = []\n",
    "        \n",
    "        if log_entry.get(\"severity\") == \"critical\":\n",
    "            anomaly_score += 5\n",
    "            reasons.append(\"Critical severity event\")\n",
    "        \n",
    "        if log_entry.get(\"event_type\") in [\"data_exfiltration\", \"unauthorized_access\"]:\n",
    "            anomaly_score += 4\n",
    "            reasons.append(\"High-risk event type\")\n",
    "        \n",
    "        if log_entry.get(\"severity\") == \"high\" and log_entry.get(\"event_type\") == \"port_scan\":\n",
    "            anomaly_score += 3\n",
    "            reasons.append(\"Port scanning detected\")\n",
    "        \n",
    "        if anomaly_score >= 3:\n",
    "            anomalies.append({\n",
    "                \"event\": log_entry.get(\"event_type\", \"\"),\n",
    "                \"source_ip\": log_entry.get(\"source_ip\", \"\"),\n",
    "                \"severity\": log_entry.get(\"severity\", \"\"),\n",
    "                \"score\": anomaly_score,\n",
    "                \"reasons\": reasons,\n",
    "                \"timestamp\": log_entry.get(\"timestamp\", \"\")\n",
    "            })\n",
    "\n",
    "# Detect conflicts in security data\n",
    "conflicts = conflict_detector.detect_value_conflicts(security_entities, \"name\")\n",
    "\n",
    "print(f\"Analyzed security relationships\")\n",
    "print(f\"Inferred {len(inferred_threats)} potential threats\")\n",
    "print(f\"Detected {len(anomalies)} anomalies\")\n",
    "print(f\"Found {len(conflicts)} data conflicts\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Generate Incident Reports\n",
    "\n",
    "Generate comprehensive incident analysis reports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(incident_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(incident_kg, os.path.join(temp_dir, \"incident_kg.json\"))\n",
    "rdf_exporter.export_knowledge_graph(incident_kg, os.path.join(temp_dir, \"incident_kg.rdf\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Security incident analysis identified {len(anomalies)} anomalies and {len(inferred_threats)} potential threats\",\n",
    "    \"total_events\": len(parsed_json.data) if parsed_json and parsed_json.data else 0,\n",
    "    \"anomalies\": len(anomalies),\n",
    "    \"threats\": len(inferred_threats),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0),\n",
    "    \"critical_events\": len([e for e in anomalies if e.get('severity') == 'critical'])\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "print(f\"Report length: {len(report)} characters\")\n",
    "print(f\"Graph quality score: {quality_score.get('overall_score', 0):.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Visualize Security Incidents\n",
    "\n",
    "Visualize incident knowledge graph and security patterns.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kg_visualizer = KGVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(incident_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(incident_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(incident_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Multiple Security Sources → Parse Logs → Extract Entities → Build KG → Analyze → Detect Anomalies → Reports → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
