{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb)\n",
    "\n",
    "# Threat Intelligence Hybrid RAG - Vector + Graph Retrieval\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates **threat intelligence hybrid RAG** using Semantica with focus on **hybrid search**, **vector + graph retrieval**, and **context-aware queries**. The pipeline combines vector search with temporal knowledge graphs for advanced threat intelligence querying.\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **Hybrid RAG**: Combines vector similarity search with knowledge graph traversal\n",
    "- **Vector + Graph Retrieval**: Uses both vector embeddings and graph relationships\n",
    "- **Context-Aware Queries**: Provides context-aware retrieval for threat intelligence\n",
    "- **Temporal Knowledge Graphs**: Builds temporal KGs for threat timeline analysis\n",
    "- **Multi-hop Reasoning**: Follows relationships across the graph for deeper context\n",
    "- **Comprehensive Data Sources**: Multiple threat intelligence feeds, APIs, and databases\n",
    "- **Modular Architecture**: Direct use of Semantica modules without core orchestrator\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Ingest threat intelligence data from multiple sources\n",
    "- Extract threat entities (IOCs, Campaigns, Threats, Actors, TTPs, Malware)\n",
    "- Build temporal threat intelligence knowledge graphs\n",
    "- Generate embeddings and populate vector stores\n",
    "- Perform hybrid vector + graph queries\n",
    "- Analyze threat networks using graph analytics\n",
    "- Store and query threat intelligence using vector stores and graph stores\n",
    "\n",
    "### Pipeline Flow\n",
    "\n",
    "```mermaid\n",
    "graph TD\n",
    "    A[Data Ingestion] --> B[Document Parsing]\n",
    "    B --> C[Text Processing]\n",
    "    C --> D[Entity Extraction]\n",
    "    D --> E[Relationship Extraction]\n",
    "    E --> F[Deduplication]\n",
    "    F --> G[Conflict Detection]\n",
    "    G --> H[Temporal Knowledge Graph]\n",
    "    H --> I[Embeddings]\n",
    "    I --> J[Vector Store]\n",
    "    H --> K[Temporal Queries]\n",
    "    K --> L[Graph Analytics]\n",
    "    L --> M[GraphRAG Queries]\n",
    "    J --> M\n",
    "    H --> N[Reasoning & Threat]\n",
    "    M --> O[Visualization]\n",
    "    N --> O\n",
    "    H --> P[Graph Store]\n",
    "    P --> O\n",
    "    O --> Q[Export]\n",
    "```\n",
    "\n",
    "\n",
    "## Installation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration & Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
    "\n",
    "# Configuration constants\n",
    "EMBEDDING_DIMENSION = 384\n",
    "EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n",
    "CHUNK_SIZE = 1000\n",
    "CHUNK_OVERLAP = 200\n",
    "TEMPORAL_GRANULARITY = \"day\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting Threat Intelligence Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n",
    "import os\n",
    "from contextlib import redirect_stderr\n",
    "from io import StringIO\n",
    "\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "feed_sources = [\n",
    "    # Threat Intelligence RSS Feeds\n",
    "    (\"US-CERT Alerts\", \"https://www.us-cert.gov/ncas/alerts.xml\"),\n",
    "    (\"SANS ISC\", \"https://isc.sans.edu/rssfeed.xml\"),\n",
    "    (\"Krebs on Security\", \"https://krebsonsecurity.com/feed/\"),\n",
    "    (\"ThreatPost\", \"https://threatpost.com/feed/\"),\n",
    "    (\"BleepingComputer\", \"https://www.bleepingcomputer.com/feed/\"),\n",
    "    (\"SecurityWeek\", \"https://www.securityweek.com/rss\"),\n",
    "]\n",
    "\n",
    "feed_ingestor = FeedIngestor()\n",
    "all_documents = []\n",
    "\n",
    "print(f\"Ingesting from {len(feed_sources)} feed sources...\")\n",
    "for i, (feed_name, feed_url) in enumerate(feed_sources, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
    "        \n",
    "        feed_count = 0\n",
    "        for item in feed_data.items:\n",
    "            if not item.content:\n",
    "                item.content = item.description or item.title or \"\"\n",
    "            if item.content:\n",
    "                if not hasattr(item, 'metadata'):\n",
    "                    item.metadata = {}\n",
    "                item.metadata['source'] = feed_name\n",
    "                all_documents.append(item)\n",
    "                feed_count += 1\n",
    "        \n",
    "        if feed_count > 0:\n",
    "            print(f\"  [{i}/{len(feed_sources)}] {feed_name}: {feed_count} documents\")\n",
    "    except Exception:\n",
    "        continue\n",
    "\n",
    "if not all_documents:\n",
    "    threat_data = \"\"\"\n",
    "    IOC: IP address 192.168.1.50 associated with APT28 campaign.\n",
    "    Threat actor APT28 uses TTP: Spear phishing and credential harvesting.\n",
    "    Campaign Operation GhostShell targets financial institutions.\n",
    "    Malware sample hash: abc123def456 linked to APT28 infrastructure.\n",
    "    IOC: Domain example-malicious.com linked to APT29 operations.\n",
    "    Threat actor APT29 uses TTP: Watering hole attacks and lateral movement.\n",
    "    Campaign Operation SolarWinds targets technology companies.\n",
    "    IOC: File hash xyz789ghi012 associated with ransomware group.\n",
    "    \"\"\"\n",
    "    with open(\"data/threat_intel.txt\", \"w\") as f:\n",
    "        f.write(threat_data)\n",
    "    file_ingestor = FileIngestor()\n",
    "    all_documents = file_ingestor.ingest(\"data/threat_intel.txt\")\n",
    "\n",
    "documents = all_documents\n",
    "print(f\"Ingested {len(documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing Threat Intelligence Documents\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.parse import DocumentParser\n",
    "\n",
    "parser = DocumentParser()\n",
    "\n",
    "print(f\"Parsing {len(documents)} documents...\")\n",
    "parsed_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    try:\n",
    "        parsed = parser.parse(\n",
    "            doc.content if hasattr(doc, 'content') else str(doc),\n",
    "            content_type=\"text\"\n",
    "        )\n",
    "        parsed_documents.append(parsed)\n",
    "    except Exception:\n",
    "        parsed_documents.append(doc)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Parsed {i}/{len(documents)} documents...\")\n",
    "\n",
    "documents = parsed_documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizing and Chunking Threat Intelligence Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.split import TextSplitter\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "# Use entity-aware chunking to preserve threat entity boundaries for GraphRAG\n",
    "splitter = TextSplitter(\n",
    "    method=\"entity_aware\",\n",
    "    ner_method=\"spacy\",\n",
    "    chunk_size=CHUNK_SIZE,\n",
    "    chunk_overlap=CHUNK_OVERLAP\n",
    ")\n",
    "\n",
    "print(f\"Normalizing {len(documents)} documents...\")\n",
    "normalized_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    normalized_text = normalizer.normalize(\n",
    "        doc.content if hasattr(doc, 'content') else str(doc),\n",
    "        clean_html=True,\n",
    "        normalize_entities=True,\n",
    "        remove_extra_whitespace=True,\n",
    "        lowercase=False\n",
    "    )\n",
    "    normalized_documents.append(normalized_text)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Normalized {i}/{len(documents)} documents...\")\n",
    "\n",
    "print(f\"Chunking {len(normalized_documents)} documents...\")\n",
    "chunked_documents = []\n",
    "for i, doc_text in enumerate(normalized_documents, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            chunks = splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    except Exception:\n",
    "        simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "        chunks = simple_splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    if i % 50 == 0 or i == len(normalized_documents):\n",
    "        print(f\"  Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n",
    "\n",
    "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "entity_extractor = NERExtractor(\n",
    "    method=\"ml\",  \n",
    "    model=\"en_core_web_sm\"\n",
    ")\n",
    "\n",
    "all_entities = []\n",
    "print(f\"Extracting entities from {len(chunked_documents)} chunks using ML-based extraction...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        entities = entity_extractor.extract_entities(chunk_text)\n",
    "        # Filter entities by threat intelligence types\n",
    "        filtered_entities = [\n",
    "            e for e in entities \n",
    "            if any(entity_type.lower() in e.label.lower() for entity_type in [\"IOC\", \"Campaign\", \"Threat\", \"Actor\", \"TTP\", \"Malware\", \"ORG\", \"PERSON\", \"GPE\"])\n",
    "        ]\n",
    "        all_entities.extend(filtered_entities)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_entities)} entities found)\")\n",
    "\n",
    "# Map spaCy entity types to threat intelligence types\n",
    "iocs = [e for e in all_entities if \"ioc\" in e.label.lower() or e.text.startswith((\"http\", \"192\", \"10.\", \"172.\"))]\n",
    "actors = [e for e in all_entities if e.label in [\"PERSON\", \"ORG\"] or \"actor\" in e.label.lower()]\n",
    "campaigns = [e for e in all_entities if \"campaign\" in e.label.lower() or \"campaign\" in e.text.lower()]\n",
    "ttps = [e for e in all_entities if \"ttp\" in e.label.lower() or \"technique\" in e.label.lower()]\n",
    "\n",
    "print(f\"Extracted {len(iocs)} IOCs, {len(actors)} actors, {len(campaigns)} campaigns, {len(ttps)} TTPs\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Threat Relationships\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"dependency\",  \n",
    "    model=\"en_core_web_sm\",  \n",
    "    confidence_threshold=0.5,  \n",
    "    max_distance=50  \n",
    ")\n",
    "\n",
    "all_relationships = []\n",
    "print(f\"Extracting relationships from {len(chunked_documents)} chunks using ML-based dependency parsing...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        # Extract relationships using dependency parsing\n",
    "        relationships = relation_extractor.extract_relations(\n",
    "            chunk_text,\n",
    "            entities=all_entities,\n",
    "            relation_types=[\"associated_with\", \"uses\", \"targets\", \"linked_to\", \"part_of\", \"employs\"]\n",
    "        )\n",
    "        all_relationships.extend(relationships)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n",
    "\n",
    "print(f\"Extracted {len(all_relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resolving Duplicate IOCs and Actors\n",
    "\n",
    "**Best Approach & Methods:**\n",
    "\n",
    "• **Multi-Factor Detection**: `DuplicateDetector` with Jaro-Winkler similarity (0.85 threshold) + property/type matching for high-precision duplicate identification\n",
    "\n",
    "• **Keep Most Complete Merge**: `EntityMerger` with `strategy=\"keep_most_complete\"` preserves entities with maximum information (properties, relationships, metadata)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.deduplication import DuplicateDetector, EntityMerger\n",
    "from semantica.semantic_extract import Entity\n",
    "\n",
    "# Convert Entity objects to dictionaries for deduplication module\n",
    "print(f\"Converting {len(all_entities)} entities to dictionaries...\")\n",
    "entity_dicts = [\n",
    "    {\n",
    "        \"id\": f\"entity_{i}\",\n",
    "        \"name\": e.text,\n",
    "        \"type\": e.label,\n",
    "        \"start_char\": e.start_char,\n",
    "        \"end_char\": e.end_char,\n",
    "        \"confidence\": e.confidence,\n",
    "        \"metadata\": e.metadata if hasattr(e, 'metadata') else {}\n",
    "    }\n",
    "    for i, e in enumerate(all_entities)\n",
    "]\n",
    "\n",
    "# Use DuplicateDetector with similarity threshold for duplicate detection\n",
    "# Progress tracking is built-in: automatically shows similarity calculation, \n",
    "# duplicate candidate creation, and group formation progress\n",
    "duplicate_detector = DuplicateDetector(\n",
    "    similarity_threshold=0.85,  # Jaro-Winkler similarity threshold\n",
    "    confidence_threshold=0.7  # Minimum confidence for duplicate candidates\n",
    ")\n",
    "\n",
    "print(f\"Detecting duplicates in {len(entity_dicts)} entities...\")\n",
    "# Progress tracking automatically displays:\n",
    "# - Similarity calculation progress (comparing entity pairs)\n",
    "# - Duplicate candidate creation progress\n",
    "# - Duplicate group formation progress\n",
    "duplicate_groups = duplicate_detector.detect_duplicate_groups(entity_dicts)\n",
    "\n",
    "print(f\"Detected {len(duplicate_groups)} duplicate groups\")\n",
    "\n",
    "# Use EntityMerger to merge duplicates using keep_most_complete strategy\n",
    "# Progress tracking is built-in: automatically shows duplicate detection \n",
    "# and merge operations progress\n",
    "entity_merger = EntityMerger(preserve_provenance=True)\n",
    "\n",
    "print(f\"Merging duplicates using keep_most_complete strategy...\")\n",
    "# Progress tracking automatically displays:\n",
    "# - Duplicate group detection progress\n",
    "# - Merge operations progress (for each group being merged)\n",
    "merge_operations = entity_merger.merge_duplicates(\n",
    "    entity_dicts,\n",
    "    strategy=\"keep_most_complete\",  # Preserve entity with most information\n",
    "    threshold=0.85\n",
    ")\n",
    "\n",
    "# Extract merged entities from merge operations\n",
    "merged_entity_dicts = []\n",
    "merged_ids = set()\n",
    "\n",
    "for op in merge_operations:\n",
    "    merged_entity_dicts.append(op.merged_entity)\n",
    "    # Track all source entity IDs that were merged\n",
    "    for source in op.source_entities:\n",
    "        merged_ids.add(source.get(\"id\") or source.get(\"name\"))\n",
    "\n",
    "# Add entities that weren't merged (singletons)\n",
    "for entity in entity_dicts:\n",
    "    entity_id = entity.get(\"id\") or entity.get(\"name\")\n",
    "    if entity_id not in merged_ids:\n",
    "        merged_entity_dicts.append(entity)\n",
    "\n",
    "# Convert back to Entity objects\n",
    "merged_entities = [\n",
    "    Entity(\n",
    "        text=e.get(\"name\", \"\"),\n",
    "        label=e.get(\"type\", \"\"),\n",
    "        start_char=e.get(\"start_char\", 0),\n",
    "        end_char=e.get(\"end_char\", 0),\n",
    "        confidence=e.get(\"confidence\", 1.0),\n",
    "        metadata=e.get(\"metadata\", {})\n",
    "    )\n",
    "    for e in merged_entity_dicts\n",
    "]\n",
    "\n",
    "print(f\"Deduplicated {len(entity_dicts)} entities to {len(merged_entities)} unique entities\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detecting Threat Intelligence Conflicts\n",
    "\n",
    "**Best Approach & Methods:**\n",
    "\n",
    "• **Type Conflict Detection**: `method=\"type\"` identifies conflicting entity classifications (e.g., IOC as both \"Malware\" and \"Threat\")\n",
    "\n",
    "• **Highest Confidence Resolution**: `strategy=\"highest_confidence\"` automatically resolves conflicts by prioritizing the most confident source\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
    "\n",
    "conflict_detector = ConflictDetector()\n",
    "conflict_resolver = ConflictResolver()\n",
    "\n",
    "# Convert Entity objects to dictionaries for conflict detection\n",
    "entity_dicts = [\n",
    "    {\n",
    "        \"id\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"text\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"label\": e.label if hasattr(e, 'label') else \"ENTITY\",\n",
    "        \"type\": e.label if hasattr(e, 'label') else \"ENTITY\",\n",
    "        \"confidence\": e.confidence if hasattr(e, 'confidence') else 1.0,\n",
    "        \"metadata\": e.metadata if hasattr(e, 'metadata') else {}\n",
    "    }\n",
    "    for e in all_entities\n",
    "]\n",
    "\n",
    "print(f\"Detecting type conflicts in {len(entity_dicts)} entities...\")\n",
    "conflicts = conflict_detector.detect_type_conflicts(entity_dicts)\n",
    "\n",
    "print(f\"Detected {len(conflicts)} type conflicts\")\n",
    "\n",
    "if conflicts:\n",
    "    print(f\"Resolving conflicts using highest_confidence strategy...\")\n",
    "    resolved = conflict_resolver.resolve_conflicts(\n",
    "        conflicts,\n",
    "        strategy=\"highest_confidence\"\n",
    "    )\n",
    "    print(f\"Resolved {len(resolved)} conflicts\")\n",
    "else:\n",
    "    print(\"No conflicts detected\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Temporal Threat Intelligence Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "graph_builder = GraphBuilder(\n",
    "    merge_entities=False,\n",
    "    resolve_conflicts=False,\n",
    "    entity_resolution_strategy=\"fuzzy\",\n",
    "    enable_temporal=True,\n",
    "    temporal_granularity=TEMPORAL_GRANULARITY\n",
    ")\n",
    "\n",
    "print(f\"Building knowledge graph...\")\n",
    "kg_sources = [{\n",
    "    \"entities\": [{\"text\": e.text, \"type\": e.label, \"confidence\": e.confidence} for e in merged_entities],\n",
    "    \"relationships\": [{\"source\": r.subject.text, \"target\": r.object.text, \"type\": r.predicate, \"confidence\": r.confidence} for r in all_relationships]\n",
    "}]\n",
    "\n",
    "kg = graph_builder.build(kg_sources)\n",
    "\n",
    "entities_count = len(kg.get('entities', []))\n",
    "relationships_count = len(kg.get('relationships', []))\n",
    "print(f\"Graph: {entities_count} entities, {relationships_count} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Embeddings for IOCs and Threats\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.embeddings import EmbeddingGenerator\n",
    "\n",
    "embedding_gen = EmbeddingGenerator(\n",
    "    provider=\"sentence_transformers\",\n",
    "    model=EMBEDDING_MODEL\n",
    ")\n",
    "\n",
    "ioc_texts = [f\"{ioc.text} {getattr(ioc, 'description', '')}\" for ioc in iocs]\n",
    "ioc_embeddings = embedding_gen.generate_embeddings(ioc_texts)\n",
    "\n",
    "actor_texts = [f\"{actor.text} {getattr(actor, 'description', '')}\" for actor in actors]\n",
    "actor_embeddings = embedding_gen.generate_embeddings(actor_texts)\n",
    "\n",
    "print(f\"Generated {len(ioc_embeddings)} IOC embeddings and {len(actor_embeddings)} actor embeddings\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populating Vector Store\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
    "\n",
    "print(f\"Storing {len(ioc_embeddings)} IOC vectors and {len(actor_embeddings)} actor vectors...\")\n",
    "ioc_ids = vector_store.store_vectors(\n",
    "    vectors=ioc_embeddings,\n",
    "    metadata=[{\"type\": \"ioc\", \"name\": ioc.text, \"label\": ioc.label} for ioc in iocs]\n",
    ")\n",
    "\n",
    "actor_ids = vector_store.store_vectors(\n",
    "    vectors=actor_embeddings,\n",
    "    metadata=[{\"type\": \"actor\", \"name\": actor.text, \"label\": actor.label} for actor in actors]\n",
    ")\n",
    "\n",
    "print(f\"Stored {len(ioc_ids)} IOC vectors and {len(actor_ids)} actor vectors\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Temporal Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import TemporalGraphQuery\n",
    "\n",
    "temporal_query = TemporalGraphQuery(\n",
    "    enable_temporal_reasoning=True,\n",
    "    temporal_granularity=TEMPORAL_GRANULARITY\n",
    ")\n",
    "\n",
    "query_results = temporal_query.query_at_time(\n",
    "    kg,\n",
    "    query={\"type\": \"Campaign\"},\n",
    "    at_time=\"2024-01-01\"\n",
    ")\n",
    "\n",
    "evolution = temporal_query.analyze_evolution(kg)\n",
    "temporal_patterns = temporal_query.query_temporal_pattern(kg, pattern=\"sequence\")\n",
    "\n",
    "print(f\"Temporal queries: {len(query_results)} campaigns at query time\")\n",
    "print(f\"Temporal patterns detected: {temporal_patterns.get('num_patterns', 0)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Threat Network Structure\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphAnalyzer, CentralityCalculator, CommunityDetector\n",
    "\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "centrality_calc = CentralityCalculator()\n",
    "community_detector = CommunityDetector()\n",
    "\n",
    "analysis = graph_analyzer.analyze_graph(kg)\n",
    "\n",
    "degree_centrality = centrality_calc.calculate_degree_centrality(kg)\n",
    "betweenness_centrality = centrality_calc.calculate_betweenness_centrality(kg)\n",
    "\n",
    "communities = community_detector.detect_communities(kg, method=\"louvain\")\n",
    "connectivity = graph_analyzer.analyze_connectivity(kg)\n",
    "\n",
    "print(f\"Graph analytics:\")\n",
    "print(f\"  - Communities: {len(communities)}\")\n",
    "print(f\"  - Connected components: {len(connectivity.get('components', []))}\")\n",
    "print(f\"  - Graph density: {analysis.get('density', 0):.3f}\")\n",
    "print(f\"  - Central nodes (degree): {len(degree_centrality)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GraphRAG: Hybrid Vector + Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.context import AgentContext\n",
    "from semantica.llms import Groq\n",
    "import os\n",
    "\n",
    "context = AgentContext(\n",
    "    vector_store=vector_store, \n",
    "    knowledge_graph=kg,\n",
    "    max_expansion_hops=3,\n",
    "    hybrid_alpha=0.7\n",
    ")\n",
    "\n",
    "llm = Groq(model=\"llama-3.1-8b-instant\", api_key=os.getenv(\"GROQ_API_KEY\"))\n",
    "\n",
    "# First, explore what threat actors/entities are in the graph\n",
    "print(\"Exploring knowledge graph for threat actors and entities...\\n\")\n",
    "kg_entities = kg.get('entities', [])\n",
    "kg_relationships = kg.get('relationships', [])\n",
    "\n",
    "# Find threat-related entities\n",
    "threat_keywords = ['APT', 'malware', 'threat', 'actor', 'campaign', 'attack', 'vulnerability', 'exploit']\n",
    "threat_entities = []\n",
    "for e in kg_entities:\n",
    "    text = str(e.get('text', '') or e.get('name', '')).upper()\n",
    "    entity_type = str(e.get('type', '')).upper()\n",
    "    if any(kw in text or kw in entity_type for kw in threat_keywords):\n",
    "        threat_entities.append(e)\n",
    "\n",
    "print(f\"Found {len(threat_entities)} threat-related entities:\")\n",
    "for e in threat_entities[:10]:\n",
    "    print(f\"  - {e.get('text') or e.get('name')} (type: {e.get('type')})\")\n",
    "\n",
    "# Check for APT28 specifically\n",
    "apt28_entities = [\n",
    "    e for e in kg_entities \n",
    "    if 'APT28' in str(e.get('text', '')).upper() or \n",
    "       'APT28' in str(e.get('name', '')).upper() or\n",
    "       'FANCY BEAR' in str(e.get('text', '')).upper()\n",
    "]\n",
    "\n",
    "print(f\"\\nSearching for APT28/Fancy Bear: {'Found' if apt28_entities else 'Not found in knowledge graph'}\")\n",
    "if apt28_entities:\n",
    "    for e in apt28_entities:\n",
    "        print(f\"  - {e.get('text') or e.get('name')} (type: {e.get('type')})\")\n",
    "\n",
    "# Try broader query if APT28 not found\n",
    "query = \"What threats are associated with APT28?\" if apt28_entities else \"What are the main cybersecurity threats and threat actors mentioned?\"\n",
    "\n",
    "print(f\"\\n{'='*80}\")\n",
    "print(f\"Query: {query}\")\n",
    "print(f\"{'='*80}\\n\")\n",
    "\n",
    "# Use multi-hop reasoning with improved prompt\n",
    "result = context.query_with_reasoning(\n",
    "    query=query,\n",
    "    llm_provider=llm,\n",
    "    max_results=20,\n",
    "    max_hops=3,\n",
    "    min_score=0.15\n",
    ")\n",
    "\n",
    "print(\"=\" * 80)\n",
    "print(\"Generated Answer (with Multi-hop Reasoning):\")\n",
    "print(\"=\" * 80)\n",
    "response = result.get('response', 'No response generated')\n",
    "\n",
    "# If APT28 not found, enhance the response\n",
    "if not apt28_entities and 'APT28' in query:\n",
    "    response += f\"\\n\\nNote: APT28 (Fancy Bear) was not found in the current knowledge graph. \"\n",
    "    response += f\"The graph contains {len(threat_entities)} threat-related entities. \"\n",
    "    response += \"Consider ingesting more threat intelligence feeds that mention APT28.\"\n",
    "\n",
    "print(response)\n",
    "print(\"\\n\" + \"=\" * 80)\n",
    "\n",
    "print(f\"\\nReasoning Details:\")\n",
    "print(f\"- Confidence: {result.get('confidence', 0):.3f}\")\n",
    "print(f\"- Sources: {result.get('num_sources', 0)}\")\n",
    "print(f\"- Reasoning Paths: {result.get('num_reasoning_paths', 0)}\")\n",
    "print(f\"- Total entities in graph: {len(kg_entities)}\")\n",
    "print(f\"- Total relationships in graph: {len(kg_relationships)}\")\n",
    "\n",
    "if result.get('sources'):\n",
    "    print(f\"\\nTop Sources:\")\n",
    "    for i, source in enumerate(result['sources'][:5], 1):\n",
    "        content = source.get('content', '')[:200] if isinstance(source, dict) else str(source)[:200]\n",
    "        score = source.get('score', 0) if isinstance(source, dict) else 0\n",
    "        print(f\"  {i}. Score: {score:.3f}\")\n",
    "        print(f\"     {content}...\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reasoning and Threat Analysis\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.reasoning import Reasoner\n",
    "from semantica.kg import ConnectivityAnalyzer\n",
    "\n",
    "# Rule-based inference\n",
    "reasoner = Reasoner()\n",
    "reasoner.add_rule(\"IF IOC associated_with Campaign AND Campaign uses TTP THEN IOC linked_to TTP\")\n",
    "reasoner.add_rule(\"IF Actor uses TTP AND TTP targets Campaign THEN Actor part_of Campaign\")\n",
    "\n",
    "inferred_facts = reasoner.infer_facts(kg)\n",
    "print(f\"Inferred {len(inferred_facts)} facts from rules\")\n",
    "\n",
    "# Analyze connectivity using ConnectivityAnalyzer class\n",
    "connectivity = ConnectivityAnalyzer()\n",
    "connectivity_result = connectivity.analyze_connectivity(kg)\n",
    "print(f\"\\nGraph connectivity: {connectivity_result.get('num_components', 0)} components\")\n",
    "print(f\"Graph density: {connectivity_result.get('density', 0):.3f}\")\n",
    "\n",
    "# Find paths between entity types\n",
    "kg_entities = kg.get('entities', [])\n",
    "actors = [e for e in kg_entities if 'actor' in str(e.get('type', '')).lower()][:3]\n",
    "iocs = [e for e in kg_entities if 'ioc' in str(e.get('type', '')).lower()][:3]\n",
    "\n",
    "threat_paths = []\n",
    "for actor in actors:\n",
    "    for ioc in iocs:\n",
    "        actor_id = actor.get('id') or actor.get('text')\n",
    "        ioc_id = ioc.get('id') or ioc.get('text')\n",
    "        if actor_id and ioc_id:\n",
    "            path_result = connectivity.calculate_shortest_paths(\n",
    "                kg,\n",
    "                source=actor_id,\n",
    "                target=ioc_id\n",
    "            )\n",
    "            if path_result.get('exists'):\n",
    "                threat_paths.append(path_result)\n",
    "\n",
    "print(f\"\\nFound {len(threat_paths)} threat paths between Actor and IOC entities\")\n",
    "if threat_paths:\n",
    "    for i, path in enumerate(threat_paths[:3], 1):\n",
    "        print(f\"  Path {i}: distance={path.get('distance', -1)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing Threat Intelligence Graph (Optional)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.graph_store import GraphStore\n",
    "\n",
    "# Optional: Store to persistent graph database\n",
    "# graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n",
    "# graph_store.store_graph(kg)\n",
    "\n",
    "print(\"Graph store configured (commented out for demo)\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the Threat Intelligence Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.visualization import KGVisualizer\n",
    "import plotly.io as pio\n",
    "\n",
    "# Configure Plotly for Colab\n",
    "pio.renderers.default = \"colab\"\n",
    "\n",
    "visualizer = KGVisualizer(layout=\"force\", node_size=20)\n",
    "fig = visualizer.visualize_network(\n",
    "    kg,\n",
    "    output=\"interactive\",\n",
    "    node_color_by=\"type\"\n",
    ")\n",
    "\n",
    "# Display in Colab\n",
    "if fig:\n",
    "    fig.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.export import GraphExporter\n",
    "\n",
    "exporter = GraphExporter()\n",
    "exporter.export(kg, output_path=\"threat_intelligence_kg.json\", format=\"json\")\n",
    "exporter.export(kg, output_path=\"threat_intelligence_kg.graphml\", format=\"graphml\")\n",
    "\n",
    "print(\"Exported threat intelligence knowledge graph to JSON and GraphML formats\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
