{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/05_Threat_Intelligence_Integration.ipynb)\n",
    "\n",
    "# Threat Intelligence Integration Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates how to integrate Python/FastMCP MCP servers as data sources for threat intelligence ingestion. Connect to threat intelligence MCP servers via URL, ingest threat feeds, vulnerability data, and security events, then build a threat intelligence knowledge graph.\n",
    "\n",
    "**IMPORTANT**: This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: MCPIngestor, ingest_mcp, WebIngestor, FeedIngestor\n",
    "- **Parsing**: MCPParser, JSONParser, XMLParser, StructuredDataParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "- **KG**: GraphBuilder, TemporalGraphQuery, GraphAnalyzer, ConnectivityAnalyzer\n",
    "- **Embeddings**: EmbeddingGenerator, TextEmbedder\n",
    "- **Vector Store**: VectorStore, HybridSearch\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Export**: JSONExporter, RDFExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Connect to Threat Intel MCP Server → Ingest Threat Data via MCP → Parse MCP Responses → Extract Threat Entities → Build Threat KG → Generate Embeddings → Hybrid RAG → Analyze Threats → Generate Reports → Visualize**\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Connect to Threat Intelligence MCP Server\n",
    "\n",
    "Connect to a Python/FastMCP MCP server that provides threat intelligence data via URL. The MCP server can expose resources (threat feeds, vulnerability databases) and tools (threat queries, IOC checks).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import MCPIngestor, ingest_mcp\n",
    "from semantica.parse import MCPParser, JSONParser, XMLParser, StructuredDataParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "from semantica.kg import GraphBuilder, TemporalGraphQuery, GraphAnalyzer, ConnectivityAnalyzer\n",
    "from semantica.embeddings import EmbeddingGenerator, TextEmbedder\n",
    "from semantica.vector_store import VectorStore, HybridSearch\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    "from semantica.export import JSONExporter, RDFExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "# Initialize MCP ingestor\n",
    "mcp_ingestor = MCPIngestor()\n",
    "\n",
    "# Connect to threat intelligence MCP server via URL\n",
    "# Replace with your actual MCP server URL\n",
    "# Example: http://localhost:8000/mcp or https://api.example.com/threat-mcp\n",
    "threat_mcp_url = \"http://localhost:8000/mcp\"\n",
    "\n",
    "# Connect to MCP server with authentication (if required)\n",
    "mcp_ingestor.connect(\n",
    "    \"threat_server\",\n",
    "    url=threat_mcp_url,\n",
    "    headers={\n",
    "        \"Authorization\": \"Bearer your_token\",\n",
    "        \"X-API-Key\": \"your_api_key\"\n",
    "    } if \"api.example.com\" in threat_mcp_url else {}\n",
    ")\n",
    "\n",
    "# List available resources (threat feeds, vulnerability databases)\n",
    "resources = mcp_ingestor.list_available_resources(\"threat_server\")\n",
    "print(f\"\\n📊 Available Resources ({len(resources)}):\")\n",
    "for resource in resources[:5]:  # Show first 5\n",
    "    print(f\"  - {resource.uri}: {resource.name}\")\n",
    "    if resource.description:\n",
    "        print(f\"    {resource.description[:80]}...\")\n",
    "\n",
    "# List available tools (threat queries, IOC checks)\n",
    "tools = mcp_ingestor.list_available_tools(\"threat_server\")\n",
    "print(f\"\\n🔧 Available Tools ({len(tools)}):\")\n",
    "for tool in tools[:5]:  # Show first 5\n",
    "    print(f\"  - {tool.name}: {tool.description or 'No description'}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Ingest Threat Intelligence Data from MCP Server\n",
    "\n",
    "Ingest threat feeds, vulnerability data, and security events using both resource-based and tool-based methods.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize parsers\n",
    "mcp_parser = MCPParser()\n",
    "json_parser = JSONParser()\n",
    "xml_parser = XMLParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "\n",
    "threat_data = []\n",
    "\n",
    "# Method 1: Resource-based ingestion\n",
    "# Ingest from MCP resources (threat feeds, vulnerability databases)\n",
    "threat_feeds = mcp_ingestor.ingest_resources(\n",
    "    \"threat_server\",\n",
    "    resource_uris=[\"resource://threats/feed\", \"resource://vulnerabilities/database\"]\n",
    ")\n",
    "\n",
    "for item in threat_feeds:\n",
    "    threat_data.append(item)\n",
    "    print(f\"  Ingested resource: {item}\")\n",
    "\n",
    "# Method 2: Tool-based ingestion\n",
    "# Call MCP tools to retrieve data dynamically\n",
    "# Example: Query threat indicators\n",
    "threat_indicators = mcp_ingestor.ingest_tool_output(\n",
    "    \"threat_server\",\n",
    "    tool_name=\"query_threat_indicators\",\n",
    "    arguments={\n",
    "        \"indicator_type\": \"IP\",\n",
    "        \"date_range\": {\n",
    "            \"start\": (datetime.now() - timedelta(days=7)).isoformat(),\n",
    "            \"end\": datetime.now().isoformat()\n",
    "        }\n",
    "    }\n",
    ")\n",
    "\n",
    "if threat_indicators:\n",
    "    threat_data.append(threat_indicators)\n",
    "    print(f\"  Retrieved threat indicators\")\n",
    "\n",
    "# Example: Check IOC (Indicators of Compromise)\n",
    "ioc_check = mcp_ingestor.ingest_tool_output(\n",
    "    \"threat_server\",\n",
    "    tool_name=\"check_ioc\",\n",
    "    arguments={\n",
    "        \"ioc_type\": \"hash\",\n",
    "        \"ioc_value\": \"abc123def456\"\n",
    "    }\n",
    ")\n",
    "\n",
    "if ioc_check:\n",
    "    threat_data.append(ioc_check)\n",
    "    print(f\"  Retrieved IOC check results\")\n",
    "\n",
    "# Sample threat intelligence data (if MCP server is not available)\n",
    "if not threat_data:\n",
    "    sample_data = {\n",
    "        \"threat_indicators\": [\n",
    "            {\n",
    "                \"indicator_id\": \"TI001\",\n",
    "                \"indicator_type\": \"IP\",\n",
    "                \"indicator_value\": \"192.168.1.100\",\n",
    "                \"threat_type\": \"malware\",\n",
    "                \"severity\": \"high\",\n",
    "                \"timestamp\": (datetime.now() - timedelta(days=1)).isoformat(),\n",
    "                \"source\": \"ThreatFeed1\"\n",
    "            },\n",
    "            {\n",
    "                \"indicator_id\": \"TI002\",\n",
    "                \"indicator_type\": \"domain\",\n",
    "                \"indicator_value\": \"malicious.example.com\",\n",
    "                \"threat_type\": \"phishing\",\n",
    "                \"severity\": \"medium\",\n",
    "                \"timestamp\": (datetime.now() - timedelta(hours=12)).isoformat(),\n",
    "                \"source\": \"ThreatFeed2\"\n",
    "            },\n",
    "            {\n",
    "                \"indicator_id\": \"TI003\",\n",
    "                \"indicator_type\": \"hash\",\n",
    "                \"indicator_value\": \"abc123def456\",\n",
    "                \"threat_type\": \"ransomware\",\n",
    "                \"severity\": \"critical\",\n",
    "                \"timestamp\": datetime.now().isoformat(),\n",
    "                \"source\": \"ThreatFeed1\"\n",
    "            }\n",
    "        ],\n",
    "        \"vulnerabilities\": [\n",
    "            {\n",
    "                \"cve_id\": \"CVE-2024-0001\",\n",
    "                \"description\": \"Remote code execution vulnerability\",\n",
    "                \"severity\": \"critical\",\n",
    "                \"affected_products\": [\"Product A\", \"Product B\"],\n",
    "                \"published_date\": (datetime.now() - timedelta(days=5)).isoformat()\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "    threat_data.append(sample_data)\n",
    "    print(f\"  Loaded {len(sample_data['threat_indicators'])} threat indicators\")\n",
    "    print(f\"  Loaded {len(sample_data['vulnerabilities'])} vulnerabilities\")\n",
    "\n",
    "print(f\"\\n📊 Total threat intelligence data items ingested: {len(threat_data)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Parse Threat Intelligence Data\n",
    "\n",
    "Parse the threat intelligence data received from MCP server responses.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed_threat_data = []\n",
    "\n",
    "# Parse MCP responses\n",
    "for data_item in threat_data:\n",
    "    # Parse MCP response (handles JSON, XML, text, binary)\n",
    "    if isinstance(data_item, dict):\n",
    "        parsed_item = data_item\n",
    "    else:\n",
    "        parsed_item = mcp_parser.parse_response(data_item, response_type=\"json\")\n",
    "    \n",
    "    parsed_threat_data.append(parsed_item)\n",
    "    print(f\"  Parsed data item\")\n",
    "\n",
    "# Extract threat indicators and vulnerabilities\n",
    "threat_indicators = []\n",
    "vulnerabilities = []\n",
    "\n",
    "for item in parsed_threat_data:\n",
    "    if isinstance(item, dict):\n",
    "        if \"threat_indicators\" in item:\n",
    "            threat_indicators.extend(item[\"threat_indicators\"])\n",
    "        elif \"indicator_id\" in item:\n",
    "            threat_indicators.append(item)\n",
    "        elif \"vulnerabilities\" in item:\n",
    "            vulnerabilities.extend(item[\"vulnerabilities\"])\n",
    "        elif \"cve_id\" in item:\n",
    "            vulnerabilities.append(item)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Extract Threat Entities and Relationships\n",
    "\n",
    "Extract threat entities (indicators, vulnerabilities, threat actors) and relationships from MCP data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "event_detector = EventDetector()\n",
    "triple_extractor = TripleExtractor()\n",
    "\n",
    "threat_entities = []\n",
    "threat_relationships = []\n",
    "\n",
    "# Extract from threat indicators\n",
    "for indicator in threat_indicators:\n",
    "    if isinstance(indicator, dict):\n",
    "        indicator_id = indicator.get(\"indicator_id\", \"\")\n",
    "        indicator_type = indicator.get(\"indicator_type\", \"\")\n",
    "        threat_type = indicator.get(\"threat_type\", \"\")\n",
    "        source = indicator.get(\"source\", \"\")\n",
    "        \n",
    "        # Threat Indicator entity\n",
    "        threat_entities.append({\n",
    "            \"id\": indicator_id,\n",
    "            \"type\": \"ThreatIndicator\",\n",
    "            \"name\": indicator_id,\n",
    "            \"properties\": {\n",
    "                \"indicator_type\": indicator_type,\n",
    "                \"indicator_value\": indicator.get(\"indicator_value\", \"\"),\n",
    "                \"threat_type\": threat_type,\n",
    "                \"severity\": indicator.get(\"severity\", \"\"),\n",
    "                \"timestamp\": indicator.get(\"timestamp\", \"\"),\n",
    "                \"source\": source\n",
    "            }\n",
    "        })\n",
    "        \n",
    "        # Threat Type entity\n",
    "        if threat_type:\n",
    "            threat_entities.append({\n",
    "                \"id\": threat_type,\n",
    "                \"type\": \"ThreatType\",\n",
    "                \"name\": threat_type,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            threat_relationships.append({\n",
    "                \"source\": indicator_id,\n",
    "                \"target\": threat_type,\n",
    "                \"type\": \"classified_as\",\n",
    "                \"properties\": {}\n",
    "            })\n",
    "        \n",
    "        # Source entity\n",
    "        if source:\n",
    "            threat_entities.append({\n",
    "                \"id\": source,\n",
    "                \"type\": \"ThreatSource\",\n",
    "                \"name\": source,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            threat_relationships.append({\n",
    "                \"source\": indicator_id,\n",
    "                \"target\": source,\n",
    "                \"type\": \"reported_by\",\n",
    "                \"properties\": {}\n",
    "            })\n",
    "\n",
    "# Extract from vulnerabilities\n",
    "for vuln in vulnerabilities:\n",
    "    if isinstance(vuln, dict):\n",
    "        cve_id = vuln.get(\"cve_id\", \"\")\n",
    "        \n",
    "        # Vulnerability entity\n",
    "        threat_entities.append({\n",
    "            \"id\": cve_id,\n",
    "            \"type\": \"Vulnerability\",\n",
    "            \"name\": cve_id,\n",
    "            \"properties\": {\n",
    "                \"description\": vuln.get(\"description\", \"\"),\n",
    "                \"severity\": vuln.get(\"severity\", \"\"),\n",
    "                \"published_date\": vuln.get(\"published_date\", \"\")\n",
    "            }\n",
    "        })\n",
    "        \n",
    "        # Affected products\n",
    "        for product in vuln.get(\"affected_products\", []):\n",
    "            threat_entities.append({\n",
    "                \"id\": product,\n",
    "                \"type\": \"Product\",\n",
    "                \"name\": product,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            threat_relationships.append({\n",
    "                \"source\": cve_id,\n",
    "                \"target\": product,\n",
    "                \"type\": \"affects\",\n",
    "                \"properties\": {}\n",
    "            })\n",
    "\n",
    "# Remove duplicates\n",
    "seen_entities = set()\n",
    "unique_entities = []\n",
    "for entity in threat_entities:\n",
    "    entity_key = (entity[\"id\"], entity[\"type\"])\n",
    "    if entity_key not in seen_entities:\n",
    "        seen_entities.add(entity_key)\n",
    "        unique_entities.append(entity)\n",
    "\n",
    "threat_entities = unique_entities\n",
    "\n",
    "print(f\"  - Threat Indicators: {len([e for e in threat_entities if e['type'] == 'ThreatIndicator'])}\")\n",
    "print(f\"  - Vulnerabilities: {len([e for e in threat_entities if e['type'] == 'Vulnerability'])}\")\n",
    "print(f\"  - Threat Types: {len([e for e in threat_entities if e['type'] == 'ThreatType'])}\")\n",
    "print(f\"  - Sources: {len([e for e in threat_entities if e['type'] == 'ThreatSource'])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Build Threat Intelligence Knowledge Graph\n",
    "\n",
    "Build a temporal knowledge graph from the extracted threat entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "temporal_query = TemporalGraphQuery()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "\n",
    "# Build knowledge graph\n",
    "threat_kg = builder.build(threat_entities, threat_relationships)\n",
    "\n",
    "# Analyze graph structure\n",
    "metrics = graph_analyzer.compute_metrics(threat_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(threat_kg)\n",
    "\n",
    "print(f\"  Entities: {len(threat_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(threat_kg.get('relationships', []))}\")\n",
    "print(f\"  Graph density: {metrics.get('density', 0):.3f}\")\n",
    "print(f\"  Connectivity: {connectivity.get('connected_components', 0)} components\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Generate Embeddings and Set Up Hybrid RAG\n",
    "\n",
    "Generate embeddings for threat intelligence data and set up hybrid search (vector + temporal KG).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate embeddings\n",
    "embedding_generator = EmbeddingGenerator()\n",
    "text_embedder = TextEmbedder()\n",
    "\n",
    "# Generate embeddings for threat entities\n",
    "threat_texts = []\n",
    "for entity in threat_entities:\n",
    "    if entity.get(\"type\") == \"ThreatIndicator\":\n",
    "        text = f\"{entity.get('properties', {}).get('indicator_value', '')} {entity.get('properties', {}).get('threat_type', '')} {entity.get('properties', {}).get('description', '')}\"\n",
    "        threat_texts.append(text)\n",
    "\n",
    "embeddings = embedding_generator.generate_embeddings(threat_texts)\n",
    "\n",
    "# Set up vector store\n",
    "vector_store = VectorStore()\n",
    "vector_store.add_embeddings(threat_texts, embeddings)\n",
    "\n",
    "# Set up hybrid search (vector + temporal KG)\n",
    "hybrid_search = HybridSearch()\n",
    "hybrid_search.setup(vector_store, threat_kg)\n",
    "\n",
    "\n",
    "# Inference engine for threat analysis\n",
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "# Threat analysis rules\n",
    "inference_engine.add_rule(\"IF severity(critical) AND threat_type(ransomware) THEN immediate_response_required\")\n",
    "inference_engine.add_rule(\"IF severity(high) AND indicator_type(IP) THEN block_ip\")\n",
    "\n",
    "# Add facts from threat data\n",
    "for indicator in threat_indicators:\n",
    "    if isinstance(indicator, dict):\n",
    "        inference_engine.add_fact({\n",
    "            \"indicator_id\": indicator.get(\"indicator_id\", \"\"),\n",
    "            \"severity\": indicator.get(\"severity\", \"\"),\n",
    "            \"threat_type\": indicator.get(\"threat_type\", \"\"),\n",
    "            \"indicator_type\": indicator.get(\"indicator_type\", \"\")\n",
    "        })\n",
    "\n",
    "# Generate threat insights\n",
    "threat_insights = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"  Threat insights: {len(threat_insights)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Export and Visualize\n",
    "\n",
    "Export the threat intelligence knowledge graph and generate visualizations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tempfile\n",
    "import os\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "json_exporter = JSONExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "# Export knowledge graph\n",
    "json_exporter.export_knowledge_graph(threat_kg, os.path.join(temp_dir, \"threat_kg.json\"))\n",
    "rdf_exporter.export_knowledge_graph(threat_kg, os.path.join(temp_dir, \"threat_kg.rdf\"))\n",
    "\n",
    "# Generate report\n",
    "report_data = {\n",
    "    \"summary\": f\"Threat intelligence integration from MCP server identified {len(threat_insights)} insights\",\n",
    "    \"threat_indicators\": len([e for e in threat_entities if e['type'] == 'ThreatIndicator']),\n",
    "    \"vulnerabilities\": len([e for e in threat_entities if e['type'] == 'Vulnerability']),\n",
    "    \"threat_types\": len([e for e in threat_entities if e['type'] == 'ThreatType']),\n",
    "    \"insights\": len(threat_insights)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "print(f\"  JSON: {os.path.join(temp_dir, 'threat_kg.json')}\")\n",
    "print(f\"  RDF: {os.path.join(temp_dir, 'threat_kg.rdf')}\")\n",
    "\n",
    "# Visualize\n",
    "kg_visualizer = KGVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(threat_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(threat_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(threat_kg, output=\"interactive\")\n",
    "\n",
    "\n",
    "# Cleanup: Disconnect from MCP server\n",
    "mcp_ingestor.disconnect(\"threat_server\")\n",
    "print(\"  Disconnected from MCP server\")\n",
    "\n",
    "print(f\"📊 Total modules used: 20+\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
