{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)\n",
    "\n",
    "# Real-Time Anomaly Detection - Stream Processing & Temporal KGs\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates **real-time anomaly detection** using Semantica with focus on **stream ingestion**, **temporal knowledge graphs**, and **pattern detection**. The pipeline streams security logs in real-time, builds temporal knowledge graphs, and detects anomalies using pattern detection.\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **Stream Processing**: Emphasizes real-time log streaming and processing\n",
    "- **Temporal Knowledge Graphs**: Builds temporal KGs to track events over time\n",
    "- **Pattern Detection**: Uses graph patterns to identify anomalies\n",
    "- **Automated Alerting**: Generates alerts for detected anomalies\n",
    "- **Comprehensive Data Sources**: Multiple security RSS feeds, APIs, and databases\n",
    "- **Modular Architecture**: Direct use of Semantica modules without core orchestrator\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Ingest security data from multiple sources (RSS feeds, APIs, streams)\n",
    "- Extract security entities (Logs, Events, IPs, Users, Alerts, Attacks)\n",
    "- Build temporal security knowledge graphs\n",
    "- Perform temporal queries and pattern detection\n",
    "- Detect anomalies using graph reasoning\n",
    "- Store and query security data using vector stores and graph stores\n",
    "\n",
    "### Pipeline Flow\n",
    "\n",
    "```mermaid\n",
    "graph TD\n",
    "    A[Data Ingestion] --> B[Document Parsing]\n",
    "    B --> C[Text Processing]\n",
    "    C --> D[Entity Extraction]\n",
    "    D --> E[Relationship Extraction]\n",
    "    E --> F[Deduplication]\n",
    "    F --> G[Conflict Detection]\n",
    "    G --> H[Temporal Knowledge Graph]\n",
    "    H --> I[Embeddings]\n",
    "    I --> J[Vector Store]\n",
    "    H --> K[Temporal Queries]\n",
    "    K --> L[Pattern Detection]\n",
    "    L --> M[Reasoning & Anomaly]\n",
    "    J --> N[GraphRAG Queries]\n",
    "    M --> N\n",
    "    H --> O[Graph Store]\n",
    "    N --> P[Visualization]\n",
    "    O --> P\n",
    "    P --> Q[Export]\n",
    "```\n",
    "\n",
    "\n",
    "## Installation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration & Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
    "\n",
    "# Configuration constants\n",
    "EMBEDDING_DIMENSION = 384\n",
    "EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n",
    "CHUNK_SIZE = 500\n",
    "CHUNK_OVERLAP = 50\n",
    "TEMPORAL_GRANULARITY = \"minute\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting Security Data from Multiple Sources\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FeedIngestor, StreamIngestor, FileIngestor\n",
    "import os\n",
    "from contextlib import redirect_stderr\n",
    "from io import StringIO\n",
    "\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "feed_sources = [\n",
    "    # Security RSS Feeds\n",
    "    (\"US-CERT Alerts\", \"https://www.us-cert.gov/ncas/alerts.xml\"),\n",
    "    (\"SANS ISC\", \"https://isc.sans.edu/rssfeed.xml\"),\n",
    "    (\"Krebs on Security\", \"https://krebsonsecurity.com/feed/\"),\n",
    "    (\"ThreatPost\", \"https://threatpost.com/feed/\"),\n",
    "    (\"BleepingComputer\", \"https://www.bleepingcomputer.com/feed/\"),\n",
    "]\n",
    "\n",
    "feed_ingestor = FeedIngestor()\n",
    "all_documents = []\n",
    "\n",
    "print(f\"Ingesting from {len(feed_sources)} feed sources...\")\n",
    "for i, (feed_name, feed_url) in enumerate(feed_sources, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
    "        \n",
    "        feed_count = 0\n",
    "        for item in feed_data.items:\n",
    "            if not item.content:\n",
    "                item.content = item.description or item.title or \"\"\n",
    "            if item.content:\n",
    "                if not hasattr(item, 'metadata'):\n",
    "                    item.metadata = {}\n",
    "                item.metadata['source'] = feed_name\n",
    "                all_documents.append(item)\n",
    "                feed_count += 1\n",
    "        \n",
    "        if feed_count > 0:\n",
    "            print(f\"  [{i}/{len(feed_sources)}] {feed_name}: {feed_count} documents\")\n",
    "    except Exception:\n",
    "        continue\n",
    "\n",
    "# Simulate stream ingestion (in production, use actual Kafka/WebSocket)\n",
    "if not all_documents:\n",
    "    security_logs = \"\"\"\n",
    "    2024-01-01 10:00:00 - Login attempt from IP 192.168.1.100 user admin\n",
    "    2024-01-01 10:01:00 - Failed login from IP 192.168.1.100 user admin\n",
    "    2024-01-01 10:02:00 - Multiple failed logins from IP 192.168.1.100 user admin\n",
    "    2024-01-01 10:03:00 - Unusual activity detected from IP 192.168.1.100\n",
    "    2024-01-01 10:04:00 - Alert: Potential brute force attack from IP 192.168.1.100\n",
    "    2024-01-01 10:05:00 - Login attempt from IP 192.168.1.101 user test\n",
    "    2024-01-01 10:06:00 - Suspicious file access from IP 192.168.1.102\n",
    "    2024-01-01 10:07:00 - Multiple connection attempts from IP 192.168.1.103\n",
    "    \"\"\"\n",
    "    with open(\"data/security_logs.txt\", \"w\") as f:\n",
    "        f.write(security_logs)\n",
    "    file_ingestor = FileIngestor()\n",
    "    all_documents = file_ingestor.ingest(\"data/security_logs.txt\")\n",
    "\n",
    "documents = all_documents\n",
    "print(f\"Ingested {len(documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.parse import DocumentParser\n",
    "\n",
    "parser = DocumentParser()\n",
    "\n",
    "print(f\"Parsing {len(documents)} documents...\")\n",
    "parsed_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    try:\n",
    "        parsed = parser.parse(\n",
    "            doc.content if hasattr(doc, 'content') else str(doc),\n",
    "            content_type=\"text\"\n",
    "        )\n",
    "        parsed_documents.append(parsed)\n",
    "    except Exception:\n",
    "        parsed_documents.append(doc)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Parsed {i}/{len(documents)} documents...\")\n",
    "\n",
    "documents = parsed_documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizing and Chunking Security Logs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.split import TextSplitter\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "# Use sentence chunking for log line boundaries (structured logs)\n",
    "splitter = TextSplitter(method=\"sentence\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "\n",
    "print(f\"Normalizing {len(documents)} documents...\")\n",
    "normalized_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    normalized_text = normalizer.normalize(\n",
    "        doc.content if hasattr(doc, 'content') else str(doc),\n",
    "        clean_html=True,\n",
    "        normalize_entities=True,\n",
    "        remove_extra_whitespace=True,\n",
    "        lowercase=False\n",
    "    )\n",
    "    normalized_documents.append(normalized_text)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Normalized {i}/{len(documents)} documents...\")\n",
    "\n",
    "print(f\"Chunking {len(normalized_documents)} documents...\")\n",
    "chunked_documents = []\n",
    "for i, doc_text in enumerate(normalized_documents, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            chunks = splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    except Exception:\n",
    "        simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "        chunks = simple_splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    if i % 50 == 0 or i == len(normalized_documents):\n",
    "        print(f\"  Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n",
    "\n",
    "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Security Entities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "security_patterns = {\n",
    "    \"IP\": r\"\\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\b\",\n",
    "    \"User\": r\"\\buser\\s+([a-zA-Z0-9_\\-\\.]+)\\b\",\n",
    "    \"Alert\": r\"\\b(?:alert|warning|alarm):\\s*([^\\n\\.]+)|\\b(?:alert|warning|alarm)\\s+(?:detected|triggered|generated|raised)\\b\",\n",
    "    \"Event\": r\"\\b(?:login|access|connection|request|attempt|failed|successful|suspicious|unusual)\\s+(?:event|attempt|request|activity|access)\\b\",\n",
    "    \"Log\": r\"\\b\\d{4}-\\d{2}-\\d{2}\\s+\\d{2}:\\d{2}:\\d{2}\\s+-\\s+([^\\n]+)\",\n",
    "    \"Attack\": r\"\\b(?:attack|breach|intrusion|exploit|malware|virus|ransomware|phishing|brute\\s+force|ddos)\\b\",\n",
    "}\n",
    "\n",
    "entity_extractor = NERExtractor(method=\"regex\", patterns=security_patterns)\n",
    "\n",
    "all_entities = []\n",
    "print(f\"Extracting entities from {len(chunked_documents)} chunks...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        entities = entity_extractor.extract_entities(\n",
    "            chunk_text,\n",
    "            entity_types=[\"Log\", \"Event\", \"IP\", \"User\", \"Alert\", \"Attack\"]\n",
    "        )\n",
    "        all_entities.extend(entities)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_entities)} entities found)\")\n",
    "\n",
    "ips = [e for e in all_entities if e.label == \"IP\" or \"ip\" in e.label.lower()]\n",
    "users = [e for e in all_entities if e.label == \"User\" or \"user\" in e.label.lower()]\n",
    "alerts = [e for e in all_entities if e.label == \"Alert\" or \"alert\" in e.label.lower()]\n",
    "\n",
    "print(f\"Extracted {len(ips)} IPs, {len(users)} users, {len(alerts)} alerts\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Security Relationships\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "# Filter entities to only meaningful security entities\n",
    "filtered_entities = [\n",
    "    e for e in all_entities \n",
    "    if e.label in [\"IP\", \"User\", \"Alert\", \"Attack\", \"Event\", \"Log\"] \n",
    "    and len(e.text) > 2\n",
    "    and e.text.lower() not in [\"to\", \"from\", \"should\", \"would\", \"choices\", \"connects\"]\n",
    "]\n",
    "\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"cooccurrence\",\n",
    "    max_distance=60,\n",
    "    confidence_threshold=0.6\n",
    ")\n",
    "\n",
    "# Deduplicate relationships\n",
    "seen_relationships = set()\n",
    "all_relationships = []\n",
    "print(f\"Extracting relationships from {len(chunked_documents)} chunks using {len(filtered_entities)} filtered entities...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        relationships = relation_extractor.extract_relations(\n",
    "            chunk_text,\n",
    "            entities=filtered_entities,\n",
    "            relation_types=[\"from\", \"attempts\", \"triggers\", \"detects\", \"associated_with\", \"causes\"]\n",
    "        )\n",
    "        # Deduplicate based on subject, predicate, object\n",
    "        for rel in relationships:\n",
    "            rel_key = (rel.subject.text, rel.predicate, rel.object.text)\n",
    "            if rel_key not in seen_relationships:\n",
    "                seen_relationships.add(rel_key)\n",
    "                all_relationships.append(rel)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n",
    "\n",
    "print(f\"Extracted {len(all_relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detecting Security Conflicts\n",
    "\n",
    "- **Using entity-wide conflict detection** to identify all types of conflicts (value, type, relationship, temporal) across security entities from multiple sources. \n",
    "- **Voting resolution strategy** selects the majority consensus value, ensuring reliability through multi-source agreement for security event data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
    "\n",
    "conflict_detector = ConflictDetector()\n",
    "conflict_resolver = ConflictResolver()\n",
    "\n",
    "# Convert entities to dictionaries for conflict detection\n",
    "entity_dicts = [\n",
    "    {\n",
    "        \"id\": e.text,\n",
    "        \"text\": e.text,\n",
    "        \"label\": e.label,\n",
    "        \"type\": e.label,\n",
    "        \"confidence\": e.confidence if hasattr(e, 'confidence') else 1.0,\n",
    "        \"metadata\": e.metadata if hasattr(e, 'metadata') else {}\n",
    "    }\n",
    "    for e in all_entities\n",
    "]\n",
    "\n",
    "# Convert relationships to dictionaries for conflict detection\n",
    "relationship_dicts = [\n",
    "    {\n",
    "        \"id\": f\"{r.subject.text}_{r.predicate}_{r.object.text}\",\n",
    "        \"source_id\": r.subject.text,\n",
    "        \"target_id\": r.object.text,\n",
    "        \"type\": r.predicate,\n",
    "        \"subject\": r.subject.text,\n",
    "        \"object\": r.object.text,\n",
    "        \"predicate\": r.predicate,\n",
    "        \"confidence\": r.confidence if hasattr(r, 'confidence') else 1.0\n",
    "    }\n",
    "    for r in all_relationships\n",
    "]\n",
    "\n",
    "print(f\"Detecting conflicts in {len(entity_dicts)} entities and {len(relationship_dicts)} relationships...\")\n",
    "\n",
    "# Detect entity conflicts (value, type, temporal)\n",
    "value_conflicts = conflict_detector.detect_value_conflicts(entity_dicts, property_name=\"label\")\n",
    "type_conflicts = conflict_detector.detect_type_conflicts(entity_dicts)\n",
    "temporal_conflicts = conflict_detector.detect_temporal_conflicts(entity_dicts)\n",
    "entity_conflicts = value_conflicts + type_conflicts + temporal_conflicts\n",
    "\n",
    "# Detect relationship conflicts\n",
    "relationship_conflicts = conflict_detector.detect_relationship_conflicts(relationship_dicts)\n",
    "\n",
    "# Combine all conflicts\n",
    "all_conflicts = entity_conflicts + relationship_conflicts\n",
    "\n",
    "print(f\"Detected {len(all_conflicts)} conflicts ({len(entity_conflicts)} entity, {len(relationship_conflicts)} relationship)\")\n",
    "\n",
    "if all_conflicts:\n",
    "    print(f\"Resolving conflicts using voting strategy...\")\n",
    "    resolved = conflict_resolver.resolve_conflicts(\n",
    "        all_conflicts,\n",
    "        strategy=\"voting\"\n",
    "    )\n",
    "    print(f\"Resolved {len(resolved)} conflicts\")\n",
    "else:\n",
    "    print(\"No conflicts detected\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Temporal Security Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "graph_builder = GraphBuilder(\n",
    "    merge_entities=False,\n",
    "    resolve_conflicts=False,\n",
    "    entity_resolution_strategy=\"fuzzy\",\n",
    "    enable_temporal=True,\n",
    "    temporal_granularity=TEMPORAL_GRANULARITY\n",
    ")\n",
    "\n",
    "print(f\"Building knowledge graph...\")\n",
    "kg_sources = [{\n",
    "    \"entities\": [{\"text\": e.text, \"type\": e.label, \"confidence\": e.confidence if hasattr(e, 'confidence') else 1.0} for e in all_entities],\n",
    "    \"relationships\": [{\"source\": r.subject.text, \"target\": r.object.text, \"type\": r.predicate, \"confidence\": r.confidence if hasattr(r, 'confidence') else 1.0} for r in all_relationships]\n",
    "}]\n",
    "\n",
    "kg = graph_builder.build(kg_sources)\n",
    "\n",
    "entities_count = len(kg.get('entities', []))\n",
    "relationships_count = len(kg.get('relationships', []))\n",
    "print(f\"Graph: {entities_count} entities, {relationships_count} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Embeddings for Events and IPs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.embeddings import EmbeddingGenerator\n",
    "\n",
    "embedding_gen = EmbeddingGenerator(\n",
    "    provider=\"sentence_transformers\",\n",
    "    model=EMBEDDING_MODEL\n",
    ")\n",
    "\n",
    "event_entities = [e for e in all_entities if e.label in [\"Event\", \"Log\", \"Alert\"]]\n",
    "print(f\"Generating embeddings for {len(event_entities)} events and {len(ips)} IPs...\")\n",
    "event_texts = [e.text for e in event_entities]\n",
    "event_embeddings = embedding_gen.generate_embeddings(event_texts)\n",
    "\n",
    "ip_texts = [ip.text for ip in ips]\n",
    "ip_embeddings = embedding_gen.generate_embeddings(ip_texts)\n",
    "\n",
    "print(f\"Generated {len(event_embeddings)} event embeddings and {len(ip_embeddings)} IP embeddings\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populating Vector Store\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
    "\n",
    "print(f\"Storing {len(event_embeddings)} event vectors and {len(ip_embeddings)} IP vectors...\")\n",
    "event_ids = vector_store.store_vectors(\n",
    "    vectors=event_embeddings,\n",
    "    metadata=[{\"type\": \"event\", \"name\": e.text, \"label\": e.label} for e in event_entities]\n",
    ")\n",
    "\n",
    "ip_ids = vector_store.store_vectors(\n",
    "    vectors=ip_embeddings,\n",
    "    metadata=[{\"type\": \"ip\", \"name\": ip.text, \"label\": ip.label} for ip in ips]\n",
    ")\n",
    "\n",
    "print(f\"Stored {len(event_ids)} event vectors and {len(ip_ids)} IP vectors\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Temporal Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import TemporalGraphQuery\n",
    "\n",
    "temporal_query = TemporalGraphQuery(\n",
    "    enable_temporal_reasoning=True,\n",
    "    temporal_granularity=TEMPORAL_GRANULARITY\n",
    ")\n",
    "\n",
    "query_results = temporal_query.query_at_time(\n",
    "    kg,\n",
    "    query={\"type\": \"Alert\"},\n",
    "    at_time=\"2024-01-01 10:04:00\"\n",
    ")\n",
    "\n",
    "evolution = temporal_query.analyze_evolution(kg)\n",
    "temporal_patterns = temporal_query.query_temporal_pattern(kg, pattern=\"sequence\")\n",
    "\n",
    "print(f\"Temporal queries: {query_results.get('num_relationships', 0)} alerts at query time\")\n",
    "print(f\"Temporal patterns detected: {temporal_patterns.get('num_patterns', 0)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detecting Anomaly Patterns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphAnalyzer\n",
    "\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "\n",
    "# Detect suspicious IPs\n",
    "suspicious_ips = []\n",
    "for entity in kg.get(\"entities\", []):\n",
    "    if entity.get(\"type\") == \"IP\":\n",
    "        related_rels = [r for r in kg.get(\"relationships\", []) \n",
    "                        if r.get(\"source\") == entity.get(\"id\") or r.get(\"target\") == entity.get(\"id\")]\n",
    "        if any(\"alert\" in str(r.get(\"type\", \"\")).lower() or \"attack\" in str(r.get(\"type\", \"\")).lower() \n",
    "               for r in related_rels):\n",
    "            suspicious_ips.append(entity)\n",
    "\n",
    "# Find all Alert entities\n",
    "alert_entities = [e for e in kg.get(\"entities\", []) if e.get(\"type\") == \"Alert\"]\n",
    "alert_ids = [e.get(\"id\") for e in alert_entities if e.get(\"id\")]\n",
    "\n",
    "# Detect anomaly patterns (multiple failed logins, unusual activity)\n",
    "anomaly_patterns = []\n",
    "for ip in ips[:10]:\n",
    "    ip_name = ip.text\n",
    "    ip_id = ip_name  # Use text as ID if entity ID not available\n",
    "    \n",
    "    # Find paths from IP to each Alert entity\n",
    "    paths_found = []\n",
    "    for alert_id in alert_ids:\n",
    "        path_result = graph_analyzer.connectivity_analyzer.calculate_shortest_paths(\n",
    "            kg,\n",
    "            source=ip_id,\n",
    "            target=alert_id\n",
    "        )\n",
    "        # Check if path exists and is within max_hops (2)\n",
    "        if path_result.get(\"exists\") and path_result.get(\"distance\", -1) <= 2:\n",
    "            paths_found.append(path_result)\n",
    "    \n",
    "    if len(paths_found) > 0:\n",
    "        anomaly_patterns.append({\n",
    "            'ip': ip_name,\n",
    "            'alert_count': len(paths_found),\n",
    "            'pattern': 'suspicious_activity'\n",
    "        })\n",
    "\n",
    "print(f\"Pattern detection: {len(anomaly_patterns)} anomaly patterns found\")\n",
    "print(f\"Suspicious IPs: {len(suspicious_ips)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reasoning and Anomaly Detection\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.reasoning import Reasoner\n",
    "from semantica.kg import GraphAnalyzer\n",
    "\n",
    "reasoner = Reasoner()\n",
    "\n",
    "reasoner.add_rule(\"IF IP attempts Event AND Event type failed_login AND Event count > 3 THEN IP triggers Alert\")\n",
    "reasoner.add_rule(\"IF User from IP AND IP triggers Alert THEN User associated_with Alert\")\n",
    "\n",
    "inferred_facts = reasoner.infer_facts(kg)\n",
    "\n",
    "# Find paths from IP entities to Alert entities\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "ip_entities = [e for e in kg.get(\"entities\", []) if e.get(\"type\") == \"IP\"]\n",
    "alert_entities = [e for e in kg.get(\"entities\", []) if e.get(\"type\") == \"Alert\"]\n",
    "\n",
    "anomaly_paths = []\n",
    "for ip_entity in ip_entities:\n",
    "    ip_id = ip_entity.get(\"id\") or ip_entity.get(\"name\", \"\")\n",
    "    for alert_entity in alert_entities:\n",
    "        alert_id = alert_entity.get(\"id\") or alert_entity.get(\"name\", \"\")\n",
    "        if ip_id and alert_id:\n",
    "            path_result = graph_analyzer.connectivity_analyzer.calculate_shortest_paths(\n",
    "                kg,\n",
    "                source=ip_id,\n",
    "                target=alert_id\n",
    "            )\n",
    "            # Check if path exists and is within max_hops (2)\n",
    "            if path_result.get(\"exists\") and path_result.get(\"distance\", -1) <= 2:\n",
    "                anomaly_paths.append(path_result)\n",
    "\n",
    "print(f\"Inferred {len(inferred_facts)} facts\")\n",
    "print(f\"Found {len(anomaly_paths)} anomaly paths\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing Security Knowledge Graph (Optional)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.graph_store import GraphStore\n",
    "\n",
    "# Optional: Store to persistent graph database\n",
    "# graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n",
    "# graph_store.store_graph(kg)\n",
    "\n",
    "print(\"Graph store configured (commented out for demo)\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GraphRAG: Hybrid Vector + Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.context import AgentContext\n",
    "from semantica.llms import Groq\n",
    "import os\n",
    "\n",
    "context = AgentContext(vector_store=vector_store, knowledge_graph=kg)\n",
    "\n",
    "# Initialize LLM provider\n",
    "llm_provider = Groq(\n",
    "    model=\"llama-3.1-8b-instant\",\n",
    "    api_key=os.getenv(\"GROQ_API_KEY\")\n",
    ")\n",
    "\n",
    "query = \"What IPs are associated with security alerts?\"\n",
    "result = context.query_with_reasoning(\n",
    "    query=query,\n",
    "    llm_provider=llm_provider,\n",
    "    max_results=10,\n",
    "    max_hops=2\n",
    ")\n",
    "\n",
    "print(f\"GraphRAG Query with Reasoning: '{query}'\\n\")\n",
    "print(\"=\" * 80)\n",
    "print(f\"\\nGenerated Response:\\n{result['response']}\\n\")\n",
    "print(\"=\" * 80)\n",
    "if result.get('reasoning_path'):\n",
    "    print(f\"\\nReasoning Path:\\n{result['reasoning_path']}\\n\")\n",
    "print(f\"Confidence: {result.get('confidence', 0):.3f}\")\n",
    "print(f\"Sources Used: {result.get('num_sources', 0)}\")\n",
    "print(f\"Reasoning Paths Found: {result.get('num_reasoning_paths', 0)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the Temporal Security Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.visualization import KGVisualizer\n",
    "\n",
    "# Create visualizer with optimized configuration for better layout\n",
    "visualizer = KGVisualizer(\n",
    "    layout=\"force\",  # Force-directed layout for better node distribution\n",
    "    node_size=15,  # Slightly smaller nodes for better visibility\n",
    "    color_scheme=\"vibrant\"  # Use vibrant color scheme for better distinction\n",
    ")\n",
    "\n",
    "# Create interactive visualization with improved layout parameters\n",
    "fig = visualizer.visualize_network(\n",
    "    kg,\n",
    "    output=\"interactive\",  # Interactive visualization in notebook\n",
    "    node_color_by=\"type\",  # Color nodes by entity type\n",
    "    hover_data=[\"type\"],  # Show entity type in hover tooltip\n",
    "    algorithm=\"kamada_kawai\",  # Use Kamada-Kawai algorithm for better layout (more stable than spring)\n",
    "    k=2.0,  # Optimal distance between nodes (larger = more spread out)\n",
    "    iterations=100,  # More iterations for better convergence\n",
    "    seed=42  # Fixed seed for reproducible layouts\n",
    ")\n",
    "\n",
    "# Update layout for better appearance\n",
    "fig.update_layout(\n",
    "    title=\"Anomaly Detection Knowledge Graph\",\n",
    "    width=1200,  # Wider view\n",
    "    height=800,  # Taller view\n",
    "    font=dict(size=12),\n",
    "    hovermode=\"closest\"\n",
    ")\n",
    "\n",
    "# Display the interactive figure\n",
    "fig.show()\n",
    "\n",
    "print(\"Interactive visualization displayed above\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.export import GraphExporter, CSVExporter\n",
    "\n",
    "# Export to supported graph formats\n",
    "graph_exporter = GraphExporter()\n",
    "graph_exporter.export(kg, output_path=\"anomaly_detection_kg.json\", format=\"json\")\n",
    "graph_exporter.export(kg, output_path=\"anomaly_detection_kg.graphml\", format=\"graphml\")\n",
    "\n",
    "# Export knowledge graph to CSV using Semantica CSVExporter\n",
    "csv_exporter = CSVExporter()\n",
    "csv_exporter.export_knowledge_graph(kg, \"anomaly_detection_kg\")\n",
    "# This creates: anomaly_detection_kg_entities.csv and anomaly_detection_kg_relationships.csv\n",
    "\n",
    "# Export only Alert entities to a separate CSV file\n",
    "alerts = [e for e in kg.get(\"entities\", []) if e.get(\"type\") == \"Alert\"]\n",
    "if alerts:\n",
    "    csv_exporter.export_entities(alerts, \"anomaly_detection_alerts.csv\")\n",
    "\n",
    "print(\"Exported knowledge graph to JSON and GraphML formats\")\n",
    "print(\"Exported knowledge graph entities and relationships to CSV\")\n",
    "if alerts:\n",
    "    print(f\"Exported {len(alerts)} Alert entities to anomaly_detection_alerts.csv\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
