{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/04_Threat_Intelligence_Hybrid_RAG.ipynb)\n",
    "\n",
    "# Threat Intelligence Hybrid RAG Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete threat intelligence hybrid RAG pipeline: ingest threat intelligence from multiple sources (files, web, feeds), extract threat entities, build knowledge graph, generate embeddings, set up hybrid search (vector + temporal KG), and query threats using advanced RAG.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, DBIngestor, MCPIngestor\n",
    "- **Parsing**: JSONParser, XMLParser, HTMLParser, DocumentParser, MCPParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "- **KG**: GraphBuilder, TemporalGraphQuery, GraphAnalyzer, ConnectivityAnalyzer\n",
    "- **Embeddings**: EmbeddingGenerator, TextEmbedder\n",
    "- **Vector Store**: VectorStore, HybridSearch\n",
    "- **Context**: ContextRetriever, ContextGraphBuilder\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Export**: JSONExporter, RDFExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Multi-Source Threat Intel (Files, Web, Feeds, MCP) → Parse → Extract Entities → Build KG → Generate Embeddings → Vector Store → Hybrid RAG Setup → Query Threats → Generate Reports → Visualize**\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Multi-Source Threat Intelligence Ingestion\n",
    "\n",
    "Ingest threat intelligence from files, web sources, and feeds.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, WebIngestor, FeedIngestor, DBIngestor, MCPIngestor, ingest_mcp\n",
    "from semantica.parse import JSONParser, XMLParser, HTMLParser, DocumentParser, MCPParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "from semantica.kg import GraphBuilder, TemporalGraphQuery, GraphAnalyzer, ConnectivityAnalyzer\n",
    "from semantica.embeddings import EmbeddingGenerator, TextEmbedder\n",
    "from semantica.vector_store import VectorStore, HybridSearch\n",
    "from semantica.context import ContextRetriever, ContextGraphBuilder\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    "from semantica.export import JSONExporter, RDFExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "mcp_ingestor = MCPIngestor()\n",
    "\n",
    "json_parser = JSONParser()\n",
    "xml_parser = XMLParser()\n",
    "html_parser = HTMLParser()\n",
    "document_parser = DocumentParser()\n",
    "mcp_parser = MCPParser()\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Real-world threat intelligence formats\n",
    "threat_intel_json = os.path.join(temp_dir, \"threat_intel.json\")\n",
    "threat_data = [\n",
    "    {\n",
    "        \"threat_id\": \"APT-001\",\n",
    "        \"name\": \"Advanced Persistent Threat Group 1\",\n",
    "        \"description\": \"State-sponsored APT group targeting financial institutions\",\n",
    "        \"iocs\": [\"192.168.1.100\", \"malicious-domain.com\", \"hash_abc123\"],\n",
    "        \"tactics\": [\"initial_access\", \"persistence\", \"exfiltration\"],\n",
    "        \"timestamp\": (datetime.now() - timedelta(days=7)).isoformat(),\n",
    "        \"severity\": \"high\"\n",
    "    },\n",
    "    {\n",
    "        \"threat_id\": \"APT-002\",\n",
    "        \"name\": \"Ransomware Campaign\",\n",
    "        \"description\": \"Large-scale ransomware campaign targeting healthcare sector\",\n",
    "        \"iocs\": [\"198.51.100.50\", \"ransomware-domain.net\", \"hash_def456\"],\n",
    "        \"tactics\": [\"initial_access\", \"execution\", \"impact\"],\n",
    "        \"timestamp\": (datetime.now() - timedelta(days=3)).isoformat(),\n",
    "        \"severity\": \"critical\"\n",
    "    },\n",
    "    {\n",
    "        \"threat_id\": \"APT-003\",\n",
    "        \"name\": \"Phishing Campaign\",\n",
    "        \"description\": \"Sophisticated phishing campaign using social engineering\",\n",
    "        \"iocs\": [\"203.0.113.75\", \"phishing-site.org\", \"hash_ghi789\"],\n",
    "        \"tactics\": [\"initial_access\", \"collection\"],\n",
    "        \"timestamp\": (datetime.now() - timedelta(days=1)).isoformat(),\n",
    "        \"severity\": \"medium\"\n",
    "    }\n",
    "]\n",
    "\n",
    "with open(threat_intel_json, 'w') as f:\n",
    "    json.dump(threat_data, f, indent=2)\n",
    "\n",
    "# XML format threat intelligence (STIX format)\n",
    "threat_intel_xml = os.path.join(temp_dir, \"threat_intel.xml\")\n",
    "xml_content = \"\"\"<?xml version=\"1.0\"?>\n",
    "<threat_intelligence>\n",
    "    <indicator>\n",
    "        <id>IOC-001</id>\n",
    "        <type>IP</type>\n",
    "        <value>172.16.0.50</value>\n",
    "        <threat_type>malware</threat_type>\n",
    "        <timestamp>2024-01-15T10:00:00</timestamp>\n",
    "    </indicator>\n",
    "    <indicator>\n",
    "        <id>IOC-002</id>\n",
    "        <type>Domain</type>\n",
    "        <value>suspicious-domain.com</value>\n",
    "        <threat_type>phishing</threat_type>\n",
    "        <timestamp>2024-01-15T11:00:00</timestamp>\n",
    "    </indicator>\n",
    "</threat_intelligence>\"\"\"\n",
    "\n",
    "with open(threat_intel_xml, 'w') as f:\n",
    "    f.write(xml_content)\n",
    "\n",
    "# Ingest from files\n",
    "file_objects_json = file_ingestor.ingest_file(threat_intel_json, read_content=True)\n",
    "file_objects_xml = file_ingestor.ingest_file(threat_intel_xml, read_content=True)\n",
    "\n",
    "# Parse threat intelligence\n",
    "parsed_json = json_parser.parse(threat_intel_json)\n",
    "parsed_xml = xml_parser.parse(threat_intel_xml)\n",
    "\n",
    "# Real threat intelligence feed URLs\n",
    "threat_intel_feeds = [\n",
    "    \"https://www.cisa.gov/news.xml\",  # CISA Security Advisories\n",
    "    \"https://www.us-cert.gov/ncas/alerts.xml\",  # US-CERT Alerts\n",
    "    \"https://feeds.feedburner.com/SecurityWeek\",  # Security Week\n",
    "    \"https://www.darkreading.com/rss.xml\",  # Dark Reading\n",
    "    \"https://krebsonsecurity.com/feed/\"  # Krebs on Security\n",
    "]\n",
    "\n",
    "threat_feed_list = []\n",
    "for feed_url in threat_intel_feeds:\n",
    "    threat_feed = feed_ingestor.ingest_feed(feed_url)\n",
    "    if threat_feed:\n",
    "        threat_feed_list.append(threat_feed)\n",
    "        print(f\"  Ingested feed: {feed_url}\")\n",
    "        print(f\"  Items: {len(threat_feed.items) if hasattr(threat_feed, 'items') else 0}\")\n",
    "\n",
    "# Real web sources for threat intelligence\n",
    "threat_web_sources = [\n",
    "    \"https://api.github.com/repos/mitre/cti\",  # MITRE ATT&CK Framework\n",
    "    \"https://www.cisa.gov/known-exploited-vulnerabilities-catalog\",  # CISA KEV Catalog\n",
    "    \"https://nvd.nist.gov/vuln/search\"  # NIST NVD\n",
    "]\n",
    "\n",
    "web_content_list = []\n",
    "for web_url in threat_web_sources[:1]:  # Process first URL\n",
    "    web_content = web_ingestor.ingest_url(web_url)\n",
    "    if web_content:\n",
    "        web_content_list.append(web_content)\n",
    "        print(f\"  Ingested web source: {web_url}\")\n",
    "\n",
    "# Optional: Ingest from MCP server\n",
    "# Users can bring their own threat intelligence MCP server via URL\n",
    "mcp_threat_data = []\n",
    "# Connect to threat intelligence MCP server via URL\n",
    "# Example: http://localhost:8000/mcp or https://api.example.com/threat-mcp\n",
    "threat_mcp_url = \"http://localhost:8000/mcp\"  # Replace with your MCP server URL\n",
    "\n",
    "mcp_ingestor.connect(\n",
    "    \"threat_mcp_server\",\n",
    "    url=threat_mcp_url,\n",
    "    headers={\n",
    "        \"Authorization\": \"Bearer your_token\",\n",
    "        \"X-API-Key\": \"your_api_key\"\n",
    "    } if \"api.example.com\" in threat_mcp_url else {}\n",
    ")\n",
    "\n",
    "# Ingest threat indicators from MCP server\n",
    "mcp_data = mcp_ingestor.ingest_resources(\n",
    "    \"threat_mcp_server\",\n",
    "    resource_uris=[\"resource://threats/feed\", \"resource://vulnerabilities/database\"]\n",
    ")\n",
    "mcp_threat_data.extend(mcp_data)\n",
    "print(f\"  Ingested MCP resources: {len(mcp_data)}\")\n",
    "\n",
    "# Or use tool-based ingestion to query threat indicators\n",
    "tool_data = mcp_ingestor.ingest_tool_output(\n",
    "    \"threat_mcp_server\",\n",
    "    tool_name=\"query_threat_indicators\",\n",
    "    arguments={\n",
    "        \"indicator_type\": \"IP\",\n",
    "        \"date_range\": {\n",
    "            \"start\": (datetime.now() - timedelta(days=7)).isoformat(),\n",
    "            \"end\": datetime.now().isoformat()\n",
    "        }\n",
    "    }\n",
    ")\n",
    "if tool_data:\n",
    "    mcp_threat_data.append(tool_data)\n",
    "    print(f\"  Retrieved tool data\")\n",
    "\n",
    "# Parse MCP responses and merge with existing threat data\n",
    "for mcp_item in mcp_threat_data:\n",
    "    parsed_mcp = mcp_parser.parse_response(mcp_item, response_type=\"json\")\n",
    "    if isinstance(parsed_mcp, dict):\n",
    "        if \"threat_indicators\" in parsed_mcp:\n",
    "            # Merge threat indicators from MCP\n",
    "            if parsed_json and parsed_json.data:\n",
    "                parsed_json.data.extend(parsed_mcp.get(\"threat_indicators\", []))\n",
    "            else:\n",
    "                parsed_json.data = parsed_mcp.get(\"threat_indicators\", [])\n",
    "        elif \"threat_id\" in parsed_mcp:\n",
    "            # Single threat indicator\n",
    "            if parsed_json and parsed_json.data:\n",
    "                parsed_json.data.append(parsed_mcp)\n",
    "            else:\n",
    "                parsed_json.data = [parsed_mcp]\n",
    "        print(f\"  Parsed MCP item\")\n",
    "\n",
    "mcp_ingestor.disconnect(\"threat_mcp_server\")\n",
    "print(f\"  Disconnected from MCP server\")\n",
    "\n",
    "print(f\"Ingested {len([file_objects_json]) if file_objects_json else 0} JSON threat intelligence files\")\n",
    "print(f\"Ingested {len([file_objects_xml]) if file_objects_xml else 0} XML threat intelligence files\")\n",
    "print(f\"Parsed {len(parsed_json.data) if parsed_json and parsed_json.data else 0} JSON threat entries\")\n",
    "print(f\"Parsed {len(parsed_xml.elements) if parsed_xml else 0} XML indicator elements\")\n",
    "print(f\"MCP server sources: {len(mcp_threat_data)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract Threat Intelligence Entities\n",
    "\n",
    "Extract threat entities, IOCs, and relationships from threat intelligence data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "event_detector = EventDetector()\n",
    "triple_extractor = TripleExtractor()\n",
    "\n",
    "threat_entities = []\n",
    "threat_relationships = []\n",
    "all_documents = []\n",
    "\n",
    "# Extract from JSON threat intelligence\n",
    "if parsed_json and parsed_json.data:\n",
    "    for threat in parsed_json.data:\n",
    "        if isinstance(threat, dict):\n",
    "            threat_text = f\"{threat.get('name', '')}: {threat.get('description', '')}\"\n",
    "            all_documents.append(threat_text)\n",
    "            \n",
    "            threat_entities.append({\n",
    "                \"id\": threat.get(\"threat_id\", \"\"),\n",
    "                \"type\": \"Threat_Actor\",\n",
    "                \"name\": threat.get(\"name\", \"\"),\n",
    "                \"properties\": {\n",
    "                    \"description\": threat.get(\"description\", \"\"),\n",
    "                    \"severity\": threat.get(\"severity\", \"\"),\n",
    "                    \"timestamp\": threat.get(\"timestamp\", \"\")\n",
    "                }\n",
    "            })\n",
    "            \n",
    "            for ioc in threat.get(\"iocs\", []):\n",
    "                threat_entities.append({\n",
    "                    \"id\": ioc,\n",
    "                    \"type\": \"IOC\",\n",
    "                    \"name\": ioc,\n",
    "                    \"properties\": {\n",
    "                        \"threat_id\": threat.get(\"threat_id\", \"\"),\n",
    "                        \"timestamp\": threat.get(\"timestamp\", \"\")\n",
    "                    }\n",
    "                })\n",
    "                threat_relationships.append({\n",
    "                    \"source\": threat.get(\"threat_id\", \"\"),\n",
    "                    \"target\": ioc,\n",
    "                    \"type\": \"uses\",\n",
    "                    \"properties\": {\"timestamp\": threat.get(\"timestamp\", \"\")}\n",
    "                })\n",
    "            \n",
    "            for tactic in threat.get(\"tactics\", []):\n",
    "                threat_entities.append({\n",
    "                    \"id\": tactic,\n",
    "                    \"type\": \"Tactic\",\n",
    "                    \"name\": tactic,\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "                threat_relationships.append({\n",
    "                    \"source\": threat.get(\"threat_id\", \"\"),\n",
    "                    \"target\": tactic,\n",
    "                    \"type\": \"employs\",\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "\n",
    "# Extract from XML indicators\n",
    "if parsed_xml and parsed_xml.elements:\n",
    "    for elem in parsed_xml.elements:\n",
    "        if hasattr(elem, 'text') and elem.text:\n",
    "            entities = ner_extractor.extract(elem.text)\n",
    "            threat_entities.extend(entities)\n",
    "\n",
    "print(f\"Extracted {len(threat_entities)} threat intelligence entities\")\n",
    "print(f\"Extracted {len(threat_relationships)} threat relationships\")\n",
    "print(f\"Collected {len(all_documents)} threat intelligence documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build Threat Intelligence Knowledge Graph\n",
    "\n",
    "Build knowledge graph from threat entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "temporal_query = TemporalGraphQuery()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "\n",
    "threat_kg = builder.build(threat_entities, threat_relationships)\n",
    "\n",
    "# Analyze graph structure\n",
    "metrics = graph_analyzer.compute_metrics(threat_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(threat_kg)\n",
    "\n",
    "print(f\"Built threat intelligence knowledge graph\")\n",
    "print(f\"  Entities: {len(threat_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(threat_kg.get('relationships', []))}\")\n",
    "print(f\"  Graph density: {metrics.get('density', 0):.3f}\")\n",
    "print(f\"  Connected components: {len(connectivity.get('components', []))}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Generate Embeddings and Setup Vector Store\n",
    "\n",
    "Generate embeddings from threat intelligence documents and store in vector database.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_generator = EmbeddingGenerator()\n",
    "text_embedder = TextEmbedder()\n",
    "vector_store = VectorStore()\n",
    "hybrid_search = HybridSearch()\n",
    "\n",
    "# Generate embeddings for threat intelligence documents\n",
    "embeddings = embedding_generator.generate_embeddings(all_documents, data_type=\"text\")\n",
    "\n",
    "# Prepare metadata for vector store\n",
    "metadata = []\n",
    "for i, doc in enumerate(all_documents):\n",
    "    metadata.append({\n",
    "        \"id\": f\"doc_{i}\",\n",
    "        \"text\": doc,\n",
    "        \"source\": \"threat_intelligence\"\n",
    "    })\n",
    "\n",
    "# Store vectors\n",
    "vector_ids = vector_store.store_vectors(embeddings, metadata)\n",
    "\n",
    "print(f\"Generated embeddings for {len(all_documents)} documents\")\n",
    "print(f\"Stored {len(vector_ids)} vectors in vector store\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Setup Hybrid RAG (Vector + Temporal KG)\n",
    "\n",
    "Setup hybrid search combining vector similarity and temporal knowledge graph queries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context_retriever = ContextRetriever()\n",
    "context_graph_builder = ContextGraphBuilder()\n",
    "\n",
    "# Setup context retriever with KG and vector store\n",
    "context_retriever = ContextRetriever(\n",
    "    knowledge_graph=threat_kg,\n",
    "    vector_store=vector_store\n",
    ")\n",
    "\n",
    "print(f\"  Knowledge graph: {len(threat_kg.get('entities', []))} entities\")\n",
    "print(f\"  Vector store: {len(vector_ids)} vectors\")\n",
    "print(f\"  Context retriever initialized\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Query Threats Using Hybrid RAG\n",
    "\n",
    "Query threat intelligence using hybrid search (vector + temporal KG).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Query examples\n",
    "queries = [\n",
    "    \"What are the latest APT threats?\",\n",
    "    \"Find threats targeting financial institutions\",\n",
    "    \"What IOCs are associated with ransomware?\"\n",
    "]\n",
    "\n",
    "query_results = []\n",
    "\n",
    "for query in queries:\n",
    "    # Generate query embedding\n",
    "    query_embedding = text_embedder.embed_text(query)\n",
    "    \n",
    "    # Vector search\n",
    "    vector_results = vector_store.search_vectors(query_embedding, k=3)\n",
    "    \n",
    "    # Temporal KG query\n",
    "    start_time = (datetime.now() - timedelta(days=30)).isoformat()\n",
    "    end_time = datetime.now().isoformat()\n",
    "    \n",
    "    temporal_results = temporal_query.query_time_range(\n",
    "        graph=threat_kg,\n",
    "        query=query,\n",
    "        start_time=start_time,\n",
    "        end_time=end_time\n",
    "    )\n",
    "    \n",
    "    # Hybrid search using context retriever\n",
    "    context_results = context_retriever.retrieve(\n",
    "        query=query,\n",
    "        top_k=3,\n",
    "        use_graph_expansion=True\n",
    "    )\n",
    "    \n",
    "    query_results.append({\n",
    "        \"query\": query,\n",
    "        \"vector_results\": len(vector_results),\n",
    "        \"temporal_results\": len(temporal_results.get('entities', [])),\n",
    "        \"context_results\": len(context_results) if context_results else 0\n",
    "    })\n",
    "\n",
    "# Inference for threat analysis\n",
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "inference_engine.add_rule(\"IF severity is critical AND tactics includes exfiltration THEN high_priority_threat\")\n",
    "inference_engine.add_rule(\"IF threat targets financial AND uses initial_access THEN financial_apt\")\n",
    "\n",
    "for threat in parsed_json.data if parsed_json and parsed_json.data else []:\n",
    "    if isinstance(threat, dict):\n",
    "        inference_engine.add_fact({\n",
    "            \"threat_id\": threat.get(\"threat_id\", \"\"),\n",
    "            \"severity\": threat.get(\"severity\", \"\"),\n",
    "            \"tactics\": threat.get(\"tactics\", [])\n",
    "        })\n",
    "\n",
    "threat_insights = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"Processed {len(queries)} threat intelligence queries\")\n",
    "for result in query_results:\n",
    "    print(f\"  Query: '{result['query']}' - Vector: {result['vector_results']}, Temporal: {result['temporal_results']}, Context: {result['context_results']}\")\n",
    "print(f\"Generated {len(threat_insights)} threat insights from inference\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(threat_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(threat_kg, os.path.join(temp_dir, \"threat_kg.json\"))\n",
    "rdf_exporter.export_knowledge_graph(threat_kg, os.path.join(temp_dir, \"threat_kg.rdf\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Threat intelligence analysis identified {len(threat_entities)} entities and {len(threat_insights)} insights\",\n",
    "    \"threats_analyzed\": len(parsed_json.data) if parsed_json and parsed_json.data else 0,\n",
    "    \"iocs\": len([e for e in threat_entities if e.get(\"type\") == \"IOC\"]),\n",
    "    \"insights\": len(threat_insights),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0),\n",
    "    \"critical_threats\": len([t for t in parsed_json.data if isinstance(t, dict) and t.get(\"severity\") == \"critical\"]) if parsed_json and parsed_json.data else 0\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "print(f\"Report length: {len(report)} characters\")\n",
    "print(f\"Graph quality score: {quality_score.get('overall_score', 0):.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Visualize Threat Intelligence\n",
    "\n",
    "Visualize threat intelligence knowledge graph and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kg_visualizer = KGVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(threat_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(threat_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(threat_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Multi-Source Threat Intel → Parse → Extract → Build KG → Embeddings → Vector Store → Hybrid RAG → Query → Reports → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
