{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/03_Fraud_Detection.ipynb)\n",
    "\n",
    "# Fraud Detection Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete fraud detection pipeline for finance: ingest transaction streams, build temporal knowledge graph, detect fraud patterns, perform anomaly detection, and generate alerts.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n",
    "- **Parsing**: StructuredDataParser, DocumentParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector\n",
    "- **KG**: GraphBuilder, TemporalPatternDetector, GraphAnalyzer\n",
    "- **Graph Store**: GraphStore with Neo4j/FalkorDB for persistent fraud graph\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor, AutomatedFixer\n",
    "- **Export**: JSONExporter, CSVExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Transaction Stream → Parse → Extract → Build Temporal KG → Store in Graph DB → Detect Patterns → Anomaly Detection → Generate Alerts → Visualize**\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Process Transactions\n",
    "\n",
    "Ingest and parse transaction data from multiple sources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, StreamIngestor, DBIngestor\n",
    "from semantica.parse import StructuredDataParser, DocumentParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector\n",
    "from semantica.kg import GraphBuilder, TemporalPatternDetector, GraphAnalyzer\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    
    "from semantica.export import JSONExporter, CSVExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "stream_ingestor = StreamIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "structured_parser = StructuredDataParser()\n",
    "document_parser = DocumentParser()\n",
    "\n",
    "# Real streaming sources for transaction monitoring\n",
    "stream_sources = [\n",
    "    {\n",
    "        \"type\": \"kafka\",\n",
    "        \"topic\": \"transactions\",\n",
    "        \"bootstrap_servers\": [\"localhost:9092\"],\n",
    "        \"consumer_config\": {\"group_id\": \"fraud_detection\"}\n",
    "    },\n",
    "    {\n",
    "        \"type\": \"rabbitmq\",\n",
    "        \"queue\": \"payment_events\",\n",
    "        \"connection_url\": \"amqp://user:password@localhost:5672/\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Real database connection for transaction data\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/transactions_db\"\n",
    "db_query = \"SELECT transaction_id, user_id, amount, merchant, location, timestamp, device FROM transactions WHERE timestamp > NOW() - INTERVAL '24 hours' ORDER BY timestamp DESC LIMIT 10000\"\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "transactions_file = os.path.join(temp_dir, \"transactions.json\")\n",
    "transactions_data = [\n",
    "    {\"transaction_id\": \"txn_001\", \"user_id\": \"user_123\", \"amount\": 150.00, \"merchant\": \"Online Store\", \"location\": \"New York\", \"timestamp\": (datetime.now() - timedelta(hours=1)).isoformat(), \"device\": \"mobile\"},\n",
    "    {\"transaction_id\": \"txn_002\", \"user_id\": \"user_123\", \"amount\": 2500.00, \"merchant\": \"Luxury Store\", \"location\": \"Paris\", \"timestamp\": (datetime.now() - timedelta(minutes=30)).isoformat(), \"device\": \"web\"},\n",
    "    {\"transaction_id\": \"txn_003\", \"user_id\": \"user_456\", \"amount\": 50.00, \"merchant\": \"Grocery Store\", \"location\": \"San Francisco\", \"timestamp\": (datetime.now() - timedelta(minutes=15)).isoformat(), \"device\": \"mobile\"},\n",
    "    {\"transaction_id\": \"txn_004\", \"user_id\": \"user_123\", \"amount\": 5000.00, \"merchant\": \"Electronics Store\", \"location\": \"Tokyo\", \"timestamp\": (datetime.now() - timedelta(minutes=5)).isoformat(), \"device\": \"mobile\"}\n",
    "]\n",
    "\n",
    "with open(transactions_file, 'w') as f:\n",
    "    json.dump(transactions_data, f)\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(transactions_file, read_content=True)\n",
    "parsed_data = structured_parser.parse_json(transactions_file)\n",
    "\n",
    "transaction_stream = []\n",
    "for txn in parsed_data.get(\"data\", transactions_data):\n",
    "    if isinstance(txn, dict):\n",
    "        txn_copy = txn.copy()\n",
    "        if \"timestamp\" in txn_copy and isinstance(txn_copy[\"timestamp\"], str):\n",
    "            txn_copy[\"timestamp\"] = datetime.fromisoformat(txn_copy[\"timestamp\"])\n",
    "        transaction_stream.append(txn_copy)\n",
    "\n",
    "print(f\"Ingested {len(file_objects)} transaction files\")\n",
    "print(f\"Parsed {len(transaction_stream)} transactions\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Build Temporal Transaction Knowledge Graph\n",
    "\n",
    "Build a temporal knowledge graph from transaction data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "\n",
    "transaction_entities = []\n",
    "relationships = []\n",
    "\n",
    "for txn in transaction_stream:\n",
    "    txn_id = txn[\"transaction_id\"]\n",
    "    user_id = txn[\"user_id\"]\n",
    "    merchant = txn[\"merchant\"]\n",
    "    location = txn[\"location\"]\n",
    "\n",
    "    transaction_entities.append({\n",
    "        \"id\": txn_id,\n",
    "        \"type\": \"Transaction\",\n",
    "        \"name\": txn_id,\n",
    "        \"properties\": {\n",
    "            \"amount\": txn[\"amount\"],\n",
    "            \"timestamp\": txn[\"timestamp\"].isoformat() if isinstance(txn[\"timestamp\"], datetime) else txn[\"timestamp\"],\n",
    "            \"device\": txn[\"device\"]\n",
    "        }\n",
    "    })\n",
    "\n",
    "    transaction_entities.append({\n",
    "        \"id\": user_id,\n",
    "        \"type\": \"User\",\n",
    "        \"name\": user_id,\n",
    "        \"properties\": {}\n",
    "    })\n",
    "\n",
    "    transaction_entities.append({\n",
    "        \"id\": merchant,\n",
    "        \"type\": \"Merchant\",\n",
    "        \"name\": merchant,\n",
    "        \"properties\": {}\n",
    "    })\n",
    "\n",
    "    transaction_entities.append({\n",
    "        \"id\": location,\n",
    "        \"type\": \"Location\",\n",
    "        \"name\": location,\n",
    "        \"properties\": {}\n",
    "    })\n",
    "\n",
    "    relationships.append({\n",
    "        \"source\": user_id,\n",
    "        \"target\": txn_id,\n",
    "        \"type\": \"performed\",\n",
    "        \"properties\": {\"timestamp\": txn[\"timestamp\"].isoformat() if isinstance(txn[\"timestamp\"], datetime) else txn[\"timestamp\"]}\n",
    "    })\n",
    "\n",
    "    relationships.append({\n",
    "        \"source\": txn_id,\n",
    "        \"target\": merchant,\n",
    "        \"type\": \"at_merchant\",\n",
    "        \"properties\": {}\n",
    "    })\n",
    "\n",
    "    relationships.append({\n",
    "        \"source\": txn_id,\n",
    "        \"target\": location,\n",
    "        \"type\": \"in_location\",\n",
    "        \"properties\": {}\n",
    "    })\n",
    "\n",
    "transaction_kg = builder.build(transaction_entities, relationships)\n",
    "\n",
    "print(f\"Built temporal knowledge graph with {len(transaction_entities)} entities and {len(relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Store in Graph Database\n",
    "\n",
    "Store the transaction graph in a persistent graph database for real-time fraud queries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.graph_store import GraphStore\n",
    "\n",
    "# Initialize graph store - FalkorDB is ideal for real-time fraud detection\n",
    "# For production: use FalkorDB with Redis for ultra-fast queries\n",
    "# graph_store = GraphStore(backend=\"falkordb\", host=\"localhost\", port=6379, graph_name=\"fraud_graph\")\n",
    "\n",
    "# Option 1: Neo4j (requires Neo4j server running)\n",
    "    graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n",
    "    graph_store.connect()\n",
    "\n",
    "# Store transaction entities in graph database\n",
    "node_id_map = {}\n",
    "\n",
    "for entity in transaction_entities:\n",
    "    node = graph_store.create_node(\n",
    "        labels=[entity[\"type\"]],\n",
    "        properties={\n",
    "            \"name\": entity[\"name\"],\n",
    "            \"original_id\": entity[\"id\"],\n",
    "            **entity.get(\"properties\", {})\n",
    "        }\n",
    "    )\n",
    "    node_id_map[entity[\"id\"]] = node.get(\"id\")\n",
    "\n",
    "print(f\"Stored {len(transaction_entities)} entities in graph database\")\n",
    "\n",
    "# Store relationships\n",
    "for rel in relationships:\n",
    "    if rel[\"source\"] in node_id_map and rel[\"target\"] in node_id_map:\n",
    "        graph_store.create_relationship(\n",
    "            start_node_id=node_id_map[rel[\"source\"]],\n",
    "            end_node_id=node_id_map[rel[\"target\"]],\n",
    "            rel_type=rel[\"type\"],\n",
    "            properties=rel.get(\"properties\", {})\n",
    "        )\n",
    "\n",
    "print(f\"Stored {len(relationships)} relationships in graph database\")\n",
    "\n",
    "# Query for high-value transactions using Cypher\n",
    "high_value_query = \"\"\"\n",
    "    MATCH (t:Transaction)\n",
    "    WHERE t.amount > 1000\n",
    "    RETURN t.name as transaction_id, t.amount as amount\n",
    "\"\"\"\n",
    "high_value_results = graph_store.execute_query(high_value_query)\n",
    "print(f\"Found {len(high_value_results.get('records', []))} high-value transactions\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Detect Fraud Patterns\n",
    "\n",
    "Detect fraud patterns using graph queries and temporal analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use graph queries to detect fraud patterns\n",
    "pattern_detector = TemporalPatternDetector()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "\n",
    "# Query for users with multiple transactions in short time\n",
    "rapid_transactions_query = \"\"\"\n",
    "    MATCH (u:User)-[:performed]->(t:Transaction)\n",
    "    WITH u, count(t) as txn_count\n",
    "    WHERE txn_count > 2\n",
    "    RETURN u.name as user_id, txn_count\n",
    "\"\"\"\n",
    "rapid_txn_results = graph_store.execute_query(rapid_transactions_query)\n",
    "\n",
    "# Query for transactions in multiple locations\n",
    "location_query = \"\"\"\n",
    "    MATCH (u:User)-[:performed]->(t:Transaction)-[:in_location]->(l:Location)\n",
    "    WITH u, collect(DISTINCT l.name) as locations\n",
    "    WHERE size(locations) > 1\n",
    "    RETURN u.name as user_id, locations\n",
    "\"\"\"\n",
    "location_results = graph_store.execute_query(location_query)\n",
    "\n",
    "# Detect fraud patterns from analysis\n",
    "fraud_patterns = []\n",
    "user_transactions = {}\n",
    "for txn in transaction_stream:\n",
    "    user_id = txn[\"user_id\"]\n",
    "    if user_id not in user_transactions:\n",
    "        user_transactions[user_id] = []\n",
    "    user_transactions[user_id].append(txn)\n",
    "\n",
    "for user_id, txns in user_transactions.items():\n",
    "    if len(txns) > 1:\n",
    "        amounts = [t[\"amount\"] for t in txns]\n",
    "        locations = [t[\"location\"] for t in txns]\n",
    "        \n",
    "        if max(amounts) > 1000:\n",
    "            fraud_patterns.append({\n",
    "                \"type\": \"high_value_transaction\",\n",
    "                \"user_id\": user_id,\n",
    "                \"amount\": max(amounts),\n",
    "                \"severity\": \"medium\"\n",
    "            })\n",
    "\n",
    "        if len(set(locations)) > 2:\n",
    "            fraud_patterns.append({\n",
    "                \"type\": \"rapid_location_change\",\n",
    "                \"user_id\": user_id,\n",
    "                \"locations\": list(set(locations)),\n",
    "                \"severity\": \"high\"\n",
    "            })\n",
    "\n",
    "print(f\"Detected {len(fraud_patterns)} fraud patterns using graph analysis\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Generate Alerts and Reports\n",
    "\n",
    "Generate fraud alerts and reports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_exporter = JSONExporter()\n",
    "csv_exporter = CSVExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "json_exporter.export_knowledge_graph(transaction_kg, os.path.join(temp_dir, \"transactions.json\"))\n",
    "csv_exporter.export_entities(transaction_entities, os.path.join(temp_dir, \"entities.csv\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Fraud detection analysis identified {len(anomalies)} suspicious transactions\",\n",
    "    \"fraud_patterns\": len(fraud_patterns),\n",
    "    \"anomalies\": len(anomalies),\n",
    "    \"transactions_analyzed\": len(transaction_stream)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "print(f\"Report length: {len(report)} characters\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Quality Assessment and Visualization\n",
    "\n",
    "Assess graph quality and visualize results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "automated_fixer = AutomatedFixer()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(transaction_kg)\n",
    "\n",
    "kg_visualizer = KGVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(transaction_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(transaction_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(transaction_kg, output=\"interactive\")\n",
    "\n",
    "# Get final graph store statistics\n",
    "graph_stats = graph_store.get_stats()\n",
    "print(f\"Graph store statistics: {graph_stats}\")\n",
    "\n",
    "# Close graph store connection\n",
    "graph_store.close()\n",
    "\n",
    "print(f\"Graph quality score: {quality_score.get('overall_score', 0):.3f}\")\n",
    "print(f\"Total modules used: 20+ (including GraphStore)\")\n",
    "print(f\"Pipeline complete: Transaction Stream → Parse → Extract → Temporal KG → Store in Graph DB → Pattern Detection → Anomaly Detection → Reports → Visualization\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
