{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb)\n",
    "\n",
    "# Criminal Network Analysis with Semantica\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete criminal network analysis pipeline using **Semantica as the core framework** with agent-based workflows, MCP integration, and comprehensive graph analytics. The pipeline ingests data from multiple sources (police reports, court records, surveillance data), builds knowledge graphs, performs network analysis, and generates intelligence reports.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### Why Semantica?\n",
    "\n",
    "Semantica provides a complete framework for criminal network analysis:\n",
    "\n",
    "- **Multi-Source Data Ingestion**: Ingest from files, databases, web sources, and MCP-enabled external APIs\n",
    "- **MCP Integration**: Access public records databases, court records APIs, and real-time data streams\n",
    "- **Agent-Based Workflows**: Autonomous agents with persistent memory for coordinated intelligence gathering\n",
    "- **Graph Analytics**: Comprehensive network analysis with centrality measures and community detection\n",
    "- **GraphRAG**: Hybrid retrieval combining knowledge graph queries with vector similarity search\n",
    "- **Pattern Detection**: Identify suspicious patterns and relationships in criminal networks\n",
    "- **Intelligence Reporting**: Generate professional HTML reports with actionable insights\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- Ingest criminal network data from multiple sources including MCP-enabled databases\n",
    "- Autonomous agent-based data gathering with persistent memory\n",
    "- Extract entities (suspects, organizations, locations, events) and relationships\n",
    "- Build temporal knowledge graphs for criminal network analysis\n",
    "- Comprehensive graph analytics (PageRank, Betweenness, Closeness, Eigenvector, Louvain)\n",
    "- Hybrid RAG for intelligence queries combining KG and vector search\n",
    "- Pattern detection for identifying key players and suspicious activities\n",
    "- Professional intelligence report generation\n",
    "\n",
    "### Semantica Modules Used (25+)\n",
    "\n",
    "- **Ingest**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor (multi-source data ingestion)\n",
    "- **MCP Integration**: MCP browser tools and resources for external data access\n",
    "- **Parse**: StructuredDataParser, CSVParser, JSONParser, DocumentParser\n",
    "- **Normalize**: TextNormalizer, DataNormalizer\n",
    "- **Semantic Extract**: NERExtractor, RelationExtractor, TripleExtractor, EventDetector\n",
    "- **KG**: GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer, TemporalGraphQuery\n",
    "- **Graph Store**: GraphStore with Neo4j/FalkorDB for persistent criminal network storage\n",
    "- **Graph Analytics**: PageRank, Betweenness, Closeness, Eigenvector centrality, Louvain community detection\n",
    "- **Embeddings**: EmbeddingGenerator, TextEmbedder\n",
    "- **Vector Store**: VectorStore, HybridSearch, MetadataFilter\n",
    "- **Context**: AgentMemory, ContextRetriever, ContextGraphBuilder\n",
    "- **Pipeline**: PipelineBuilder, ExecutionEngine, ParallelismManager\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Export**: ReportGenerator, JSONExporter\n",
    "- **Visualization**: KGVisualizer, AnalyticsVisualizer, TemporalVisualizer\n",
    "- **Deduplication**: DuplicateDetector, EntityMerger\n",
    "\n",
    "### Pipeline Overview\n",
    "\n",
    "**Data Sources (Police Reports, Court Records, Surveillance) → MCP Integration → Agent-Based Data Gathering → Parse → Extract Entities/Relationships → Build Criminal Network KG → Graph Analytics → GraphRAG → Agent Analysis → Pattern Detection → Generate Intelligence Report → Visualize**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Setup and Import Semantica Modules\n",
    "\n",
    "Import all necessary Semantica modules for criminal network analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import all Semantica modules for criminal network analysis\n",
    "from semantica.ingest import FileIngestor, DBIngestor, WebIngestor, StreamIngestor\n",
    "from semantica.parse import StructuredDataParser, CSVParser, JSONParser, DocumentParser\n",
    "from semantica.normalize import TextNormalizer, DataNormalizer\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, TripleExtractor, EventDetector\n",
    "from semantica.kg import GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer, TemporalGraphQuery\n",
    "from semantica.embeddings import EmbeddingGenerator, TextEmbedder\n",
    "from semantica.vector_store import VectorStore, HybridSearch, MetadataFilter\n",
    "from semantica.context import AgentMemory, ContextRetriever, ContextGraphBuilder\n",
    "from semantica.pipeline import PipelineBuilder, ExecutionEngine, ParallelismManager\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    "from semantica.export import ReportGenerator, JSONExporter\n",
    "from semantica.visualization import KGVisualizer, AnalyticsVisualizer, TemporalVisualizer\n",
    "from semantica.deduplication import DuplicateDetector, EntityMerger\n",
    "\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "import numpy as np\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Initialize Agent Memory and Setup Agents\n",
    "\n",
    "Set up AgentMemory for persistent context and initialize specialized agents for criminal network analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize vector store for agent memory\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=768)\n",
    "embedding_generator = EmbeddingGenerator()\n",
    "\n",
    "# Initialize agent memory for persistent context\n",
    "agent_memory = AgentMemory(\n",
    "    vector_store=vector_store,\n",
    "    retention_policy=\"unlimited\",\n",
    "    max_memory_size=10000\n",
    ")\n",
    "\n",
    "# Initialize Semantica modules for agents\n",
    "file_ingestor = FileIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "inference_engine = InferenceEngine()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Ingest Criminal Network Data with MCP Integration\n",
    "\n",
    "Ingest data from multiple sources including police reports, court records, and surveillance data. Use MCP for accessing external databases and APIs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create temporary directory for sample data\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample criminal network data - police reports\n",
    "police_reports_data = {\n",
    "    \"reports\": [\n",
    "        {\n",
    "            \"report_id\": \"PR001\",\n",
    "            \"date\": \"2024-01-15\",\n",
    "            \"suspects\": [\"John Doe\", \"Jane Smith\"],\n",
    "            \"location\": \"Downtown District\",\n",
    "            \"incident_type\": \"Drug Trafficking\",\n",
    "            \"description\": \"Multiple suspects observed in suspicious activity. Connection to known criminal organization.\"\n",
    "        },\n",
    "        {\n",
    "            \"report_id\": \"PR002\",\n",
    "            \"date\": \"2024-02-10\",\n",
    "            \"suspects\": [\"Jane Smith\", \"Bob Johnson\"],\n",
    "            \"location\": \"Warehouse District\",\n",
    "            \"incident_type\": \"Money Laundering\",\n",
    "            \"description\": \"Financial transactions linked to criminal network. Multiple bank accounts involved.\"\n",
    "        },\n",
    "        {\n",
    "            \"report_id\": \"PR003\",\n",
    "            \"date\": \"2024-03-05\",\n",
    "            \"suspects\": [\"John Doe\", \"Alice Brown\", \"Charlie Wilson\"],\n",
    "            \"location\": \"Port Area\",\n",
    "            \"incident_type\": \"Smuggling\",\n",
    "            \"description\": \"Cargo shipment investigation reveals connections to international criminal network.\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "# Sample court records data\n",
    "court_records_data = {\n",
    "    \"cases\": [\n",
    "        {\n",
    "            \"case_id\": \"CR001\",\n",
    "            \"defendant\": \"John Doe\",\n",
    "            \"charges\": [\"Drug Trafficking\", \"Conspiracy\"],\n",
    "            \"date\": \"2024-01-20\",\n",
    "            \"outcome\": \"Pending\",\n",
    "            \"co_defendants\": [\"Jane Smith\"]\n",
    "        },\n",
    "        {\n",
    "            \"case_id\": \"CR002\",\n",
    "            \"defendant\": \"Jane Smith\",\n",
    "            \"charges\": [\"Money Laundering\", \"Racketeering\"],\n",
    "            \"date\": \"2024-02-15\",\n",
    "            \"outcome\": \"Pending\",\n",
    "            \"co_defendants\": [\"Bob Johnson\"]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "# Sample surveillance data\n",
    "surveillance_data = {\n",
    "    \"observations\": [\n",
    "        {\n",
    "            \"observation_id\": \"SV001\",\n",
    "            \"date\": \"2024-01-18\",\n",
    "            \"subjects\": [\"John Doe\", \"Jane Smith\"],\n",
    "            \"location\": \"Restaurant XYZ\",\n",
    "            \"duration_minutes\": 45,\n",
    "            \"notes\": \"Extended meeting between known associates\"\n",
    "        },\n",
    "        {\n",
    "            \"observation_id\": \"SV002\",\n",
    "            \"date\": \"2024-02-12\",\n",
    "            \"subjects\": [\"Jane Smith\", \"Bob Johnson\"],\n",
    "            \"location\": \"Business Office ABC\",\n",
    "            \"duration_minutes\": 120,\n",
    "            \"notes\": \"Business meeting with financial documents exchange\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "# Save sample data files\n",
    "police_reports_file = os.path.join(temp_dir, \"police_reports.json\")\n",
    "court_records_file = os.path.join(temp_dir, \"court_records.json\")\n",
    "surveillance_file = os.path.join(temp_dir, \"surveillance.json\")\n",
    "\n",
    "with open(police_reports_file, 'w') as f:\n",
    "    json.dump(police_reports_data, f, indent=2)\n",
    "with open(court_records_file, 'w') as f:\n",
    "    json.dump(court_records_data, f, indent=2)\n",
    "with open(surveillance_file, 'w') as f:\n",
    "    json.dump(surveillance_data, f, indent=2)\n",
    "\n",
    "# Ingest files using Semantica\n",
    "police_data = file_ingestor.ingest_file(police_reports_file, read_content=True)\n",
    "court_data = file_ingestor.ingest_file(court_records_file, read_content=True)\n",
    "surveillance_data_obj = file_ingestor.ingest_file(surveillance_file, read_content=True)\n",
    "\n",
    "# Store in agent memory\n",
    "agent_memory.store(\n",
    "    \"Ingested police reports, court records, and surveillance data\",\n",
    "    metadata={\n",
    "        \"type\": \"data_ingestion\",\n",
    "        \"sources\": [\"police_reports\", \"court_records\", \"surveillance\"],\n",
    "        \"timestamp\": datetime.now().isoformat()\n",
    "    }\n",
    ")\n",
    "\n",
    "print(f\"  - Police reports: {len(police_reports_data['reports'])}\")\n",
    "print(f\"  - Court records: {len(court_records_data['cases'])}\")\n",
    "print(f\"  - Surveillance observations: {len(surveillance_data['observations'])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize parsers and normalizers\n",
    "json_parser = JSONParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "text_normalizer = TextNormalizer()\n",
    "data_normalizer = DataNormalizer()\n",
    "\n",
    "# Parse all data sources\n",
    "parsed_police = json_parser.parse(police_reports_file)\n",
    "parsed_court = json_parser.parse(court_records_file)\n",
    "parsed_surveillance = json_parser.parse(surveillance_file)\n",
    "\n",
    "# Combine all parsed data\n",
    "all_data = {\n",
    "    \"police_reports\": parsed_police.data if hasattr(parsed_police, 'data') else parsed_police,\n",
    "    \"court_records\": parsed_court.data if hasattr(parsed_court, 'data') else parsed_court,\n",
    "    \"surveillance\": parsed_surveillance.data if hasattr(parsed_surveillance, 'data') else parsed_surveillance\n",
    "}\n",
    "\n",
    "print(f\"  - Police reports parsed: {len(all_data['police_reports'].get('reports', [])) if isinstance(all_data['police_reports'], dict) else 0}\")\n",
    "print(f\"  - Court records parsed: {len(all_data['court_records'].get('cases', [])) if isinstance(all_data['court_records'], dict) else 0}\")\n",
    "print(f\"  - Surveillance parsed: {len(all_data['surveillance'].get('observations', [])) if isinstance(all_data['surveillance'], dict) else 0}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Extract Entities and Relationships\n",
    "\n",
    "Extract criminal network entities (suspects, organizations, locations, events) and relationships using Semantica's extraction modules.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize extractors\n",
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "triple_extractor = TripleExtractor()\n",
    "event_detector = EventDetector()\n",
    "\n",
    "# Extract entities and relationships\n",
    "criminal_entities = []\n",
    "criminal_relationships = []\n",
    "entity_id_map = {}  # Map names to IDs\n",
    "\n",
    "# Process police reports\n",
    "if isinstance(all_data['police_reports'], dict):\n",
    "    for report in all_data['police_reports'].get('reports', []):\n",
    "        # Extract suspects as entities\n",
    "        for suspect in report.get('suspects', []):\n",
    "            if suspect not in entity_id_map:\n",
    "                entity_id = f\"PERSON_{len(entity_id_map) + 1}\"\n",
    "                entity_id_map[suspect] = entity_id\n",
    "                criminal_entities.append({\n",
    "                    \"id\": entity_id,\n",
    "                    \"type\": \"Person\",\n",
    "                    \"name\": suspect,\n",
    "                    \"properties\": {\n",
    "                        \"source\": \"police_report\",\n",
    "                        \"report_id\": report.get('report_id')\n",
    "                    }\n",
    "                })\n",
    "        \n",
    "        # Extract location\n",
    "        location = report.get('location')\n",
    "        if location:\n",
    "            loc_id = f\"LOC_{location.replace(' ', '_')}\"\n",
    "            if loc_id not in [e.get('id') for e in criminal_entities]:\n",
    "                criminal_entities.append({\n",
    "                    \"id\": loc_id,\n",
    "                    \"type\": \"Location\",\n",
    "                    \"name\": location,\n",
    "                    \"properties\": {\"source\": \"police_report\"}\n",
    "                })\n",
    "        \n",
    "        # Create relationships between suspects\n",
    "        suspects = report.get('suspects', [])\n",
    "        for i, suspect1 in enumerate(suspects):\n",
    "            for suspect2 in suspects[i+1:]:\n",
    "                if suspect1 in entity_id_map and suspect2 in entity_id_map:\n",
    "                    criminal_relationships.append({\n",
    "                        \"source\": entity_id_map[suspect1],\n",
    "                        \"target\": entity_id_map[suspect2],\n",
    "                        \"type\": \"associated_with\",\n",
    "                        \"properties\": {\n",
    "                            \"incident_type\": report.get('incident_type'),\n",
    "                            \"date\": report.get('date'),\n",
    "                            \"source\": \"police_report\",\n",
    "                            \"report_id\": report.get('report_id')\n",
    "                        }\n",
    "                    })\n",
    "\n",
    "# Process court records\n",
    "if isinstance(all_data['court_records'], dict):\n",
    "    for case in all_data['court_records'].get('cases', []):\n",
    "        defendant = case.get('defendant')\n",
    "        if defendant and defendant in entity_id_map:\n",
    "            # Add case information to entity\n",
    "            for entity in criminal_entities:\n",
    "                if entity.get('id') == entity_id_map[defendant]:\n",
    "                    entity['properties']['cases'] = entity['properties'].get('cases', [])\n",
    "                    entity['properties']['cases'].append({\n",
    "                        \"case_id\": case.get('case_id'),\n",
    "                        \"charges\": case.get('charges'),\n",
    "                        \"outcome\": case.get('outcome')\n",
    "                    })\n",
    "            \n",
    "            # Create relationships with co-defendants\n",
    "            for co_def in case.get('co_defendants', []):\n",
    "                if co_def in entity_id_map:\n",
    "                    criminal_relationships.append({\n",
    "                        \"source\": entity_id_map[defendant],\n",
    "                        \"target\": entity_id_map[co_def],\n",
    "                        \"type\": \"co_defendant\",\n",
    "                        \"properties\": {\n",
    "                            \"case_id\": case.get('case_id'),\n",
    "                            \"date\": case.get('date')\n",
    "                        }\n",
    "                    })\n",
    "\n",
    "# Process surveillance data\n",
    "if isinstance(all_data['surveillance'], dict):\n",
    "    for obs in all_data['surveillance'].get('observations', []):\n",
    "        subjects = obs.get('subjects', [])\n",
    "        for i, subject1 in enumerate(subjects):\n",
    "            for subject2 in subjects[i+1:]:\n",
    "                if subject1 in entity_id_map and subject2 in entity_id_map:\n",
    "                    criminal_relationships.append({\n",
    "                        \"source\": entity_id_map[subject1],\n",
    "                        \"target\": entity_id_map[subject2],\n",
    "                        \"type\": \"observed_with\",\n",
    "                        \"properties\": {\n",
    "                            \"location\": obs.get('location'),\n",
    "                            \"date\": obs.get('date'),\n",
    "                            \"duration_minutes\": obs.get('duration_minutes')\n",
    "                        }\n",
    "                    })\n",
    "\n",
    "print(f\"  - Entities: {len(criminal_entities)}\")\n",
    "print(f\"  - Relationships: {len(criminal_relationships)}\")\n",
    "print(f\"  - Unique persons: {len([e for e in criminal_entities if e.get('type') == 'Person'])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Build Criminal Network Knowledge Graph\n",
    "\n",
    "Build the knowledge graph from extracted entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize graph builder\n",
    "graph_builder = GraphBuilder()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "\n",
    "# Build criminal network knowledge graph\n",
    "criminal_kg = graph_builder.build(criminal_entities, criminal_relationships)\n",
    "\n",
    "# Analyze connectivity\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(criminal_kg)\n",
    "\n",
    "# Store in agent memory\n",
    "agent_memory.store(\n",
    "    f\"Built criminal network knowledge graph with {len(criminal_entities)} entities and {len(criminal_relationships)} relationships\",\n",
    "    metadata={\n",
    "        \"type\": \"knowledge_graph\",\n",
    "        \"entity_count\": len(criminal_entities),\n",
    "        \"relationship_count\": len(criminal_relationships)\n",
    "    },\n",
    "    entities=criminal_entities,\n",
    "    relationships=criminal_relationships\n",
    ")\n",
    "\n",
    "print(f\"  - Entities: {len(criminal_kg.get('entities', []))}\")\n",
    "print(f\"  - Relationships: {len(criminal_kg.get('relationships', []))}\")\n",
    "print(f\"  - Connected components: {connectivity.get('num_components', 0)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Persist Criminal Network to Graph Database\n",
    "\n",
    "Store the criminal network knowledge graph in a persistent graph database for real-time queries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.graph_store import GraphStore\n",
    "\n",
    "# Initialize graph store for persistent criminal network storage\n",
    "# For production: use Neo4j for enterprise features or FalkorDB for real-time queries\n",
    "\n",
    "# Option 1: Neo4j\n",
    "graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n",
    "\n",
    "# Option 2: FalkorDB\n",
    "# graph_store = GraphStore(backend=\"falkordb\", host=\"localhost\", port=6379, graph_name=\"criminal_network\")\n",
    "\n",
    "graph_store.connect()\n",
    "\n",
    "# Store entities as nodes\n",
    "entity_node_map = {}\n",
    "for entity in criminal_entities:\n",
    "    node = graph_store.create_node(\n",
    "        labels=[entity[\"type\"]],\n",
    "        properties={\n",
    "            \"original_id\": entity[\"id\"],\n",
    "            \"name\": entity[\"name\"],\n",
    "            **entity.get(\"properties\", {})\n",
    "        }\n",
    "    )\n",
    "    entity_node_map[entity[\"id\"]] = node.get(\"id\")\n",
    "\n",
    "# Store relationships\n",
    "for rel in criminal_relationships:\n",
    "    if rel[\"source\"] in entity_node_map and rel[\"target\"] in entity_node_map:\n",
    "        graph_store.create_relationship(\n",
    "            start_node_id=entity_node_map[rel[\"source\"]],\n",
    "            end_node_id=entity_node_map[rel[\"target\"]],\n",
    "            rel_type=rel[\"type\"].upper(),\n",
    "            properties=rel.get(\"properties\", {})\n",
    "        )\n",
    "\n",
    "\n",
    "# Query for key suspects using Cypher\n",
    "suspect_query = \"\"\"\n",
    "    MATCH (p:Person)-[r]->(other)\n",
    "    WITH p, count(r) as connections\n",
    "    ORDER BY connections DESC\n",
    "    LIMIT 5\n",
    "    RETURN p.name as suspect, connections\n",
    "\"\"\"\n",
    "suspect_results = graph_store.execute_query(suspect_query)\n",
    "\n",
    "# Store in agent memory\n",
    "agent_memory.store(\n",
    "    f\"Criminal network stored in graph database with {len(entity_node_map)} entities\",\n",
    "    metadata={\n",
    "        \"type\": \"graph_store\",\n",
    "        \"entity_count\": len(entity_node_map),\n",
    "        \"relationship_count\": len(criminal_relationships)\n",
    "    }\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Perform Comprehensive Graph Analytics\n",
    "\n",
    "Perform all graph analytics including centrality measures, community detection, and connectivity analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform comprehensive graph analytics\n",
    "# 1. Centrality measures\n",
    "pagerank_centrality = graph_analyzer.compute_centrality(criminal_kg, method=\"pagerank\")\n",
    "betweenness_centrality = graph_analyzer.compute_centrality(criminal_kg, method=\"betweenness\")\n",
    "closeness_centrality = graph_analyzer.compute_centrality(criminal_kg, method=\"closeness\")\n",
    "eigenvector_centrality = graph_analyzer.compute_centrality(criminal_kg, method=\"eigenvector\")\n",
    "\n",
    "# 2. Community detection\n",
    "communities = graph_analyzer.detect_communities(criminal_kg, method=\"louvain\")\n",
    "\n",
    "# 3. Graph metrics\n",
    "kg_metrics = graph_analyzer.compute_metrics(criminal_kg)\n",
    "\n",
    "# 4. Identify key players (high centrality)\n",
    "key_players_pagerank = sorted(\n",
    "    [(node, score) for node, score in pagerank_centrality.items()],\n",
    "    key=lambda x: x[1],\n",
    "    reverse=True\n",
    ")[:5]\n",
    "\n",
    "key_players_betweenness = sorted(\n",
    "    [(node, score) for node, score in betweenness_centrality.items()],\n",
    "    key=lambda x: x[1],\n",
    "    reverse=True\n",
    ")[:5]\n",
    "\n",
    "# Use graph store for analytics queries\n",
    "centrality_query = \"\"\"\n",
    "    MATCH (p:Person)-[r]-()\n",
    "    WITH p, count(r) as degree\n",
    "    RETURN p.name as suspect, degree\n",
    "    ORDER BY degree DESC\n",
    "    LIMIT 5\n",
    "\"\"\"\n",
    "centrality_results = graph_store.execute_query(centrality_query)\n",
    "\n",
    "# Store analytics in agent memory\n",
    "agent_memory.store(\n",
    "    f\"Graph analytics completed: {len(key_players_pagerank)} key players identified\",\n",
    "    metadata={\n",
    "        \"type\": \"graph_analytics\",\n",
    "        \"key_players_count\": len(key_players_pagerank),\n",
    "        \"communities\": communities.get('num_communities', 0) if isinstance(communities, dict) else 0\n",
    "    }\n",
    ")\n",
    "\n",
    "print(f\"  - PageRank centrality: {len(pagerank_centrality)} entities\")\n",
    "print(f\"  - Betweenness centrality: {len(betweenness_centrality)} entities\")\n",
    "print(f\"  - Closeness centrality: {len(closeness_centrality)} entities\")\n",
    "print(f\"  - Eigenvector centrality: {len(eigenvector_centrality)} entities\")\n",
    "print(f\"  - Communities detected: {communities.get('num_communities', 0) if isinstance(communities, dict) else 0}\")\n",
    "print(f\"\\nTop Key Players (PageRank):\")\n",
    "for entity_id, score in key_players_pagerank:\n",
    "    entity_name = next((e.get('name', '') for e in criminal_entities if e.get('id') == entity_id), 'Unknown')\n",
    "    print(f\"  - {entity_name} ({entity_id}): {score:.4f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9: Agent-Based Analysis Workflows\n",
    "\n",
    "Define and execute specialized agents for criminal network analysis using Pipeline coordination.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize pipeline modules\n",
    "pipeline_builder = PipelineBuilder()\n",
    "execution_engine = ExecutionEngine()\n",
    "parallelism_manager = ParallelismManager(max_workers=4)\n",
    "\n",
    "# Define specialized agents\n",
    "\n",
    "# Agent 1: Data Gathering Agent\n",
    "def agent_data_gathering(sources, memory):\n",
    "    \"\"\"Autonomous agent for data gathering with memory.\"\"\"\n",
    "    # Retrieve relevant context from memory\n",
    "    context = memory.retrieve(\"criminal network data\", max_results=5)\n",
    "    # Simulate data gathering\n",
    "    gathered_data = {\n",
    "        \"sources_processed\": len(sources),\n",
    "        \"context_items\": len(context)\n",
    "    }\n",
    "    memory.store(\n",
    "        f\"Data gathering agent processed {len(sources)} sources\",\n",
    "        metadata={\"agent\": \"data_gathering\", \"sources_count\": len(sources)}\n",
    "    )\n",
    "    return gathered_data\n",
    "\n",
    "# Agent 2: Network Analysis Agent\n",
    "def agent_network_analysis(graph, analyzer, memory):\n",
    "    \"\"\"Agent for network structure analysis.\"\"\"\n",
    "    metrics = analyzer.compute_metrics(graph)\n",
    "    centrality = analyzer.compute_centrality(graph, method=\"pagerank\")\n",
    "    communities = analyzer.detect_communities(graph, method=\"louvain\")\n",
    "    \n",
    "    memory.store(\n",
    "        f\"Network analysis completed: {communities.get('num_communities', 0) if isinstance(communities, dict) else 0} communities detected\",\n",
    "        metadata={\"agent\": \"network_analysis\", \"communities\": communities.get('num_communities', 0) if isinstance(communities, dict) else 0}\n",
    "    )\n",
    "    return {\"metrics\": metrics, \"centrality\": centrality, \"communities\": communities}\n",
    "\n",
    "# Agent 3: Pattern Detection Agent\n",
    "def agent_pattern_detection(graph, inference_engine, memory):\n",
    "    \"\"\"Agent for identifying suspicious patterns.\"\"\"\n",
    "    # Define pattern detection rules\n",
    "    pattern_rules = [\n",
    "        {\n",
    "            \"rule_id\": \"high_centrality_suspect\",\n",
    "            \"condition\": \"IF entity has high_pagerank AND entity has multiple_relationships THEN entity is key_player\",\n",
    "            \"action\": \"flag_key_player\"\n",
    "        },\n",
    "        {\n",
    "            \"rule_id\": \"dense_network\",\n",
    "            \"condition\": \"IF entities form dense_cluster AND high_communication THEN entities form_criminal_ring\",\n",
    "            \"action\": \"identify_criminal_ring\"\n",
    "        }\n",
    "    ]\n",
    "    \n",
    "    patterns = inference_engine.infer(knowledge_graph=graph, rules=pattern_rules)\n",
    "    memory.store(\n",
    "        f\"Pattern detection identified {len(patterns) if isinstance(patterns, list) else 1} patterns\",\n",
    "        metadata={\"agent\": \"pattern_detection\", \"patterns_count\": len(patterns) if isinstance(patterns, list) else 1}\n",
    "    )\n",
    "    return {\"patterns\": patterns, \"rules\": pattern_rules}\n",
    "\n",
    "# Agent 4: Report Generation Agent\n",
    "def agent_report_generation(analysis_results, memory):\n",
    "    \"\"\"Agent for compiling intelligence reports.\"\"\"\n",
    "    # Retrieve all relevant context\n",
    "    context = memory.retrieve(\"criminal network\", max_results=10)\n",
    "    report_data = {\n",
    "        \"analysis_results\": analysis_results,\n",
    "        \"context_items\": len(context),\n",
    "        \"timestamp\": datetime.now().isoformat()\n",
    "    }\n",
    "    memory.store(\n",
    "        \"Report generation agent compiled intelligence report\",\n",
    "        metadata={\"agent\": \"report_generation\", \"context_items\": len(context)}\n",
    "    )\n",
    "    return report_data\n",
    "\n",
    "def data_gathering_handler(data, **config):\n",
    "    sources = data.get(\"sources\", [])\n",
    "    memory = data.get(\"memory\")\n",
    "    r = agent_data_gathering(sources, memory)\n",
    "    return {**data, \"gathered\": r}\n",
    "def network_analysis_handler(data, **config):\n",
    "    graph = data.get(\"criminal_kg\")\n",
    "    analyzer = data.get(\"graph_analyzer\")\n",
    "    memory = data.get(\"memory\")\n",
    "    r = agent_network_analysis(graph, analyzer, memory)\n",
    "    return {**data, \"network_analysis\": r}\n",
    "def pattern_detection_handler(data, **config):\n",
    "    graph = data.get(\"criminal_kg\")\n",
    "    inf = data.get(\"inference_engine\")\n",
    "    memory = data.get(\"memory\")\n",
    "    r = agent_pattern_detection(graph, inf, memory)\n",
    "    return {**data, \"patterns\": r}\n",
    "def report_generation_handler(data, **config):\n",
    "    memory = data.get(\"memory\")\n",
    "    analysis_results = {\n",
    "        \"network\": data.get(\"network_analysis\"),\n",
    "        \"patterns\": data.get(\"patterns\")\n",
    "    }\n",
    "    r = agent_report_generation(analysis_results, memory)\n",
    "    return {**data, \"report\": r}\n",
    "criminal_network_pipeline = (\n",
    "    pipeline_builder\n",
    "    .add_step(\"data_gathering\", \"ingest\", handler=data_gathering_handler)\n",
    "    .add_step(\"network_analysis\", \"analyze_graph\", dependencies=[\"data_gathering\"], handler=network_analysis_handler)\n",
    "    .add_step(\"pattern_detection\", \"analysis\", dependencies=[\"network_analysis\"], handler=pattern_detection_handler)\n",
    "    .add_step(\"report_generation\", \"report\", dependencies=[\"pattern_detection\"], handler=report_generation_handler)\n",
    ")\n",
    ".build()\n",
    "input_data = {\"sources\": [police_reports_file, court_records_file, surveillance_file], \"criminal_kg\": criminal_kg, \"graph_analyzer\": graph_analyzer, \"inference_engine\": inference_engine, \"memory\": agent_memory}\n",
    "pipeline_result = execution_engine.execute_pipeline(criminal_network_pipeline, data=input_data, parallel=True)\n",
    "\n",
    "print(f\"  - Pipeline steps: {len(criminal_network_pipeline.steps)}\")\n",
    "print(f\"  - Parallel execution: Enabled\")\n",
    "print(f\"  - Execution status: {pipeline_result.success if hasattr(pipeline_result, 'success') else 'Completed'}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize visualizers\n",
    "kg_visualizer = KGVisualizer(layout=\"force\", color_scheme=\"vibrant\")\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "\n",
    "# Visualize criminal network\n",
    "network_fig = kg_visualizer.visualize_network(\n",
    "    criminal_kg,\n",
    "    output=\"interactive\"\n",
    ")\n",
    "\n",
    "# Visualize communities\n",
    "communities_fig = kg_visualizer.visualize_communities(\n",
    "    criminal_kg,\n",
    "    communities,\n",
    "    output=\"interactive\"\n",
    ")\n",
    "\n",
    "# Visualize centrality rankings\n",
    "centrality_fig = analytics_visualizer.visualize_centrality_rankings(\n",
    "    pagerank_centrality,\n",
    "    centrality_type=\"pagerank\",\n",
    "    top_n=10,\n",
    "    output=\"interactive\"\n",
    ")\n",
    "\n",
    "# Visualize centrality comparison\n",
    "centrality_comparison = {\n",
    "    \"pagerank\": pagerank_centrality,\n",
    "    \"betweenness\": betweenness_centrality,\n",
    "    \"closeness\": closeness_centrality\n",
    "}\n",
    "comparison_fig = analytics_visualizer.visualize_centrality_comparison(\n",
    "    centrality_comparison,\n",
    "    top_n=10,\n",
    "    output=\"interactive\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 11: Generate Intelligence Report\n",
    "\n",
    "Generate a comprehensive intelligence report on the criminal network structure.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize report generator\n",
    "report_generator = ReportGenerator()\n",
    "json_exporter = JSONExporter()\n",
    "\n",
    "# Prepare intelligence report data\n",
    "intelligence_report_data = {\n",
    "    \"title\": \"Criminal Network Analysis Intelligence Report\",\n",
    "    \"executive_summary\": \"Analysis of criminal network structure, key players, communities, and patterns\",\n",
    "    \"knowledge_graph_metrics\": kg_metrics,\n",
    "    \"communities\": communities,\n",
    "    \"key_players\": {\n",
    "        \"pagerank\": key_players_pagerank,\n",
    "        \"betweenness\": key_players_betweenness\n",
    "    },\n",
    "    \"centrality_measures\": {\n",
    "        \"pagerank\": pagerank_centrality,\n",
    "        \"betweenness\": betweenness_centrality,\n",
    "        \"closeness\": closeness_centrality,\n",
    "        \"eigenvector\": eigenvector_centrality\n",
    "    },\n",
    "    \"network_structure\": {\n",
    "        \"entities\": len(criminal_entities),\n",
    "        \"relationships\": len(criminal_relationships),\n",
    "        \"density\": kg_metrics.get('density', 0) if isinstance(kg_metrics, dict) else 0,\n",
    "        \"components\": connectivity.get('num_components', 0)\n",
    "    },\n",
    "    \"agent_memory_stats\": agent_memory.get_statistics(),\n",
    "    \"graphrag_capabilities\": {\n",
    "        \"embeddings_count\": len(entity_embeddings),\n",
    "        \"hybrid_search\": True\n",
    "    }\n",
    "}\n",
    "\n",
    "# Generate professional HTML report\n",
    "intelligence_report_file = os.path.join(temp_dir, \"criminal_network_intelligence_report.html\")\n",
    "report_generator.generate_report(\n",
    "    intelligence_report_data,\n",
    "    intelligence_report_file,\n",
    "    format=\"html\"\n",
    ")\n",
    "\n",
    "# Export network data as JSON\n",
    "network_json_file = os.path.join(temp_dir, \"criminal_network_data.json\")\n",
    "json_exporter.export(criminal_kg, network_json_file)\n",
    "\n",
    "print(f\"  - Intelligence report HTML: {intelligence_report_file}\")\n",
    "print(f\"  - Network data JSON: {network_json_file}\")\n",
    "print(f\"  - Report includes: Executive summary, metrics, key players, communities, agent memory stats\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion and Best Practices\n",
    "\n",
    "### Key Takeaways\n",
    "\n",
    "1. **Semantica as Core Framework**: This notebook demonstrated using Semantica as the exclusive framework for criminal network analysis\n",
    "2. **Agent-Based Workflows**: Semantica's Pipeline module enables parallel agent coordination with persistent memory\n",
    "3. **MCP Integration**: MCP can be used for accessing external databases and APIs for real-time data\n",
    "4. **Graph Analytics**: Semantica's GraphAnalyzer provides comprehensive algorithms (PageRank, Betweenness, Closeness, Eigenvector, Louvain)\n",
    "5. **Hybrid RAG**: Semantica's HybridSearch combines knowledge graph queries with vector similarity search\n",
    "6. **Agent Memory**: Semantica's AgentMemory provides persistent context across agent interactions\n",
    "7. **Professional Reports**: Semantica's ReportGenerator creates professional HTML intelligence reports\n",
    "\n",
    "### Semantica-Specific Performance Considerations\n",
    "\n",
    "- **Graph Analytics**: Use Semantica's GraphAnalyzer for efficient community detection and centrality calculations on large networks\n",
    "- **Parallel Execution**: Leverage Semantica's ParallelismManager for concurrent agent execution\n",
    "- **Vector Search**: Use Semantica's HybridSearch for efficient entity similarity search combined with graph queries\n",
    "- **Agent Memory**: Utilize Semantica's AgentMemory for persistent context across agent workflows\n",
    "\n",
    "### Deployment Recommendations Using Semantica\n",
    "\n",
    "1. **Production Setup**:\n",
    "   - Use Semantica's configuration management for data source settings\n",
    "   - Leverage Semantica's Pipeline module for automated intelligence workflows\n",
    "   - Use Semantica's export modules for report persistence\n",
    "\n",
    "2. **Scalability**:\n",
    "   - Use Semantica's batch processing for large-scale network data\n",
    "   - Leverage Semantica's graph analytics optimizations\n",
    "   - Utilize Semantica's parallel execution for concurrent analysis\n",
    "\n",
    "3. **Quality Assurance**:\n",
    "   - Use Semantica's Deduplication modules for entity resolution\n",
    "   - Leverage Semantica's ExplanationGenerator for report traceability\n",
    "   - Utilize Semantica's quality modules for data validation\n",
    "\n",
    "### How Semantica's Architecture Benefits Criminal Network Analysis\n",
    "\n",
    "- **Comprehensive Analytics**: Semantica's GraphAnalyzer provides all necessary algorithms in one framework\n",
    "- **Unified Pipeline**: Semantica's Pipeline module orchestrates complex multi-agent workflows\n",
    "- **Agent Memory**: Semantica's AgentMemory enables persistent context for intelligent agents\n",
    "- **Hybrid RAG**: Semantica's HybridSearch combines graph and vector search for comprehensive intelligence queries\n",
    "- **Extensibility**: Semantica's registry system enables custom analysis methods\n",
    "- **Integration**: Semantica's unified framework simplifies integration with existing systems\n",
    "- **Performance**: Semantica's optimized algorithms handle large-scale network analysis efficiently\n",
    "- **Explainability**: Semantica's ExplanationGenerator provides traceable intelligence reports\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
