{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb)\n",
    "\n",
    "# Fraud Detection - Temporal KGs & Pattern Detection\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates **fraud detection** using Semantica with focus on **temporal knowledge graphs**, **anomaly detection**, and **pattern recognition**. The pipeline analyzes transaction streams using temporal knowledge graphs to detect fraud patterns and anomalies in real-time.\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **Temporal Knowledge Graphs**: Builds temporal KGs to track transaction patterns over time\n",
    "- **Pattern Detection**: Uses TemporalPatternDetector and reasoning for fraud detection\n",
    "- **Anomaly Detection**: Uses graph-based pattern recognition to identify fraud\n",
    "- **Conflict Detection**: Detects conflicting transaction data from multiple sources\n",
    "- **Real-Time Stream Processing**: Demonstrates real-time transaction stream processing\n",
    "- **Comprehensive Data Sources**: Multiple transaction streams, APIs, and fraud databases\n",
    "- **Modular Architecture**: Direct use of Semantica modules without core orchestrator\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Ingest transaction data from streams and APIs\n",
    "- Extract transaction entities (Transactions, Accounts, Devices, Patterns, Anomalies)\n",
    "- Build temporal transaction knowledge graphs\n",
    "- Perform temporal queries and pattern detection\n",
    "- Detect fraud patterns using graph reasoning\n",
    "- Analyze transaction networks using graph analytics\n",
    "- Store and query transaction data using vector stores\n",
    "\n",
    "### Pipeline Flow\n",
    "\n",
    "```mermaid\n",
    "graph TD\n",
    "    A[Data Ingestion] --> B[Document Parsing]\n",
    "    B --> C[Text Processing]\n",
    "    C --> D[Entity Extraction]\n",
    "    D --> E[Relationship Extraction]\n",
    "    E --> F[Deduplication]\n",
    "    F --> G[Conflict Detection]\n",
    "    G --> H[Temporal Knowledge Graph]\n",
    "    H --> I[Embeddings]\n",
    "    I --> J[Vector Store]\n",
    "    H --> K[Temporal Queries]\n",
    "    K --> L[Temporal Pattern Detection]\n",
    "    L --> M[Reasoning & Fraud]\n",
    "    M --> N[Graph Analytics]\n",
    "    J --> O[GraphRAG Queries]\n",
    "    N --> O\n",
    "    O --> P[Visualization]\n",
    "    P --> Q[Export]\n",
    "```\n",
    "\n",
    "## Installation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration & Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
    "\n",
    "# Configuration constants\n",
    "EMBEDDING_DIMENSION = 384\n",
    "EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n",
    "CHUNK_SIZE = 500\n",
    "CHUNK_OVERLAP = 50\n",
    "TEMPORAL_GRANULARITY = \"minute\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting Transaction Data from Streams\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import StreamIngestor, WebIngestor, FileIngestor\n",
    "import os\n",
    "from contextlib import redirect_stderr\n",
    "from io import StringIO\n",
    "\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "all_documents = []\n",
    "\n",
    "# ============================================================================\n",
    "# REAL DATA SOURCE 1: CSV Transaction Data\n",
    "# ============================================================================\n",
    "# Ingest transaction data from CSV file with comprehensive fraud indicators\n",
    "print(\"📊 Ingesting transaction data from CSV...\")\n",
    "csv_path = \"data/transactions.csv\"\n",
    "file_ingestor = FileIngestor()\n",
    "\n",
    "try:\n",
    "    csv_documents = file_ingestor.ingest(csv_path)\n",
    "    for doc in csv_documents:\n",
    "        if not hasattr(doc, 'metadata'):\n",
    "            doc.metadata = {}\n",
    "        doc.metadata['source'] = 'Transaction CSV'\n",
    "        doc.metadata['data_type'] = 'transaction'\n",
    "        all_documents.append(doc)\n",
    "    print(f\"  ✅ Loaded {len(csv_documents)} documents from CSV\")\n",
    "except Exception as e:\n",
    "    print(f\"  ⚠️  CSV ingestion failed: {e}\")\n",
    "\n",
    "# ============================================================================\n",
    "# REAL DATA SOURCE 2: JSON Account & Fraud Pattern Data\n",
    "# ============================================================================\n",
    "# Ingest account metadata, device information, and fraud patterns from JSON\n",
    "print(\"📊 Ingesting account and fraud pattern data from JSON...\")\n",
    "json_path = \"data/accounts.json\"\n",
    "\n",
    "try:\n",
    "    json_documents = file_ingestor.ingest(json_path)\n",
    "    for doc in json_documents:\n",
    "        if not hasattr(doc, 'metadata'):\n",
    "            doc.metadata = {}\n",
    "        doc.metadata['source'] = 'Account JSON'\n",
    "        doc.metadata['data_type'] = 'account_metadata'\n",
    "        all_documents.append(doc)\n",
    "    print(f\"  ✅ Loaded {len(json_documents)} documents from JSON\")\n",
    "except Exception as e:\n",
    "    print(f\"  ⚠️  JSON ingestion failed: {e}\")\n",
    "\n",
    "# ============================================================================\n",
    "# REAL DATA SOURCE 3: External Payment Processor API (Example)\n",
    "# ============================================================================\n",
    "# In production, you would use real APIs like:\n",
    "# - Stripe API: https://api.stripe.com/v1/charges\n",
    "# - PayPal API: https://api.paypal.com/v1/payments\n",
    "# - Square API: https://connect.squareup.com/v2/payments\n",
    "# \n",
    "# For demonstration, we'll use a mock API endpoint that returns transaction data\n",
    "# In production, replace with actual API endpoints and authentication\n",
    "print(\"📊 Attempting to ingest from payment processor API...\")\n",
    "payment_apis = [\n",
    "    \"https://api.stripe.com/v1/charges\",  # Stripe (requires API key)\n",
    "    \"https://api.paypal.com/v1/payments\",  # PayPal (requires OAuth)\n",
    "    # Add your actual API endpoints here\n",
    "]\n",
    "\n",
    "web_ingestor = WebIngestor()\n",
    "api_success = False\n",
    "\n",
    "for api_url in payment_apis:\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            api_documents = web_ingestor.ingest(api_url, method=\"url\")\n",
    "        if api_documents:\n",
    "            for doc in api_documents:\n",
    "                if not hasattr(doc, 'metadata'):\n",
    "                    doc.metadata = {}\n",
    "                doc.metadata['source'] = f'Payment API ({api_url.split(\"//\")[1].split(\"/\")[0]})'\n",
    "                doc.metadata['data_type'] = 'api_transaction'\n",
    "                all_documents.append(doc)\n",
    "            print(f\"  ✅ Loaded {len(api_documents)} documents from {api_url}\")\n",
    "            api_success = True\n",
    "            break\n",
    "    except Exception:\n",
    "        continue\n",
    "\n",
    "if not api_success:\n",
    "    print(\"  ℹ️  API endpoints require authentication. Using local data sources.\")\n",
    "\n",
    "# ============================================================================\n",
    "# REAL DATA SOURCE 4: Stream Ingestion (Kafka/RabbitMQ)\n",
    "# ============================================================================\n",
    "# For real-time fraud detection, ingest from message streams\n",
    "# Example Kafka configuration:\n",
    "# stream_config = {\n",
    "#     \"bootstrap_servers\": \"localhost:9092\",\n",
    "#     \"topic\": \"transactions\",\n",
    "#     \"group_id\": \"fraud_detection\"\n",
    "# }\n",
    "# stream_ingestor = StreamIngestor()\n",
    "# stream_documents = stream_ingestor.ingest(stream_config, method=\"kafka\")\n",
    "print(\"📊 Stream ingestion (Kafka/RabbitMQ) - Configure in production\")\n",
    "\n",
    "# ============================================================================\n",
    "# PUBLIC FRAUD DETECTION DATASETS (References)\n",
    "# ============================================================================\n",
    "# For additional training and testing, consider these public datasets:\n",
    "# \n",
    "# 1. Credit Card Fraud Detection (Kaggle)\n",
    "#    URL: https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud\n",
    "#    - 284,807 transactions, 492 fraudulent (0.172% fraud rate)\n",
    "#    - Download and ingest: file_ingestor.ingest(\"creditcard.csv\")\n",
    "#\n",
    "# 2. IEEE-CIS Fraud Detection (Kaggle)\n",
    "#    URL: https://www.kaggle.com/competitions/ieee-fraud-detection\n",
    "#    - 590,540 transactions, ~3.5% fraudulent\n",
    "#    - 431 features (400 numerical, 31 categorical)\n",
    "#\n",
    "# 3. PaySim Synthetic Financial Dataset\n",
    "#    URL: https://www.kaggle.com/datasets/ealaxi/paysim1\n",
    "#    - 6,000,000 mobile money transactions\n",
    "#    - ~0.14% fraud rate\n",
    "#\n",
    "# 4. UCI Credit Card Dataset\n",
    "#    URL: https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients\n",
    "#    - 30,000 credit card clients\n",
    "#    - 23 features including payment history\n",
    "#\n",
    "# To use these datasets:\n",
    "# 1. Download the dataset files\n",
    "# 2. Place them in the data/ directory\n",
    "# 3. Use FileIngestor to load: file_ingestor.ingest(\"data/dataset.csv\")\n",
    "\n",
    "# ============================================================================\n",
    "# Fallback: Generate sample data if no sources available\n",
    "# ============================================================================\n",
    "if not all_documents:\n",
    "    print(\"⚠️  No data sources found. Generating sample transaction data...\")\n",
    "    tx_data = \"\"\"\n",
    "    2024-01-01 10:00:00 - Transaction $1000 from Account A123 to Account B456\n",
    "    2024-01-01 10:01:00 - Transaction $5000 from Account A123 to Account C789\n",
    "    2024-01-01 10:02:00 - Transaction $10000 from Account A123 to Account D012 (unusual pattern)\n",
    "    2024-01-01 10:03:00 - Multiple rapid transactions from Account A123 (suspicious)\n",
    "    2024-01-01 10:04:00 - Transaction $2000 from Account B456 to Account E789\n",
    "    2024-01-01 10:05:00 - Large transaction $50000 from Account A123 to Account F012 (fraud alert)\n",
    "    2024-01-01 10:06:00 - Transaction $1500 from Account C789 to Account G345\n",
    "    2024-01-01 10:07:00 - Unusual device login from Account A123 (suspicious activity)\n",
    "    \"\"\"\n",
    "    with open(\"data/transactions.txt\", \"w\") as f:\n",
    "        f.write(tx_data)\n",
    "    file_ingestor = FileIngestor()\n",
    "    all_documents = file_ingestor.ingest(\"data/transactions.txt\")\n",
    "\n",
    "documents = all_documents\n",
    "print(f\"\\n✅ Total ingested: {len(documents)} documents from {len(set(doc.metadata.get('source', 'unknown') for doc in documents))} sources\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing Transaction Documents\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.parse import DocumentParser\n",
    "\n",
    "parser = DocumentParser()\n",
    "\n",
    "print(f\"Parsing {len(documents)} documents...\")\n",
    "parsed_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    try:\n",
    "        parsed = parser.parse(\n",
    "            doc.content if hasattr(doc, 'content') else str(doc),\n",
    "            content_type=\"text\"\n",
    "        )\n",
    "        parsed_documents.append(parsed)\n",
    "    except Exception:\n",
    "        parsed_documents.append(doc)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Parsed {i}/{len(documents)} documents...\")\n",
    "\n",
    "documents = parsed_documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizing and Chunking Transaction Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.split import TextSplitter\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "# Use sentence chunking for transaction logs\n",
    "splitter = TextSplitter(method=\"sentence\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "\n",
    "print(f\"Normalizing {len(documents)} documents...\")\n",
    "normalized_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    normalized_text = normalizer.normalize(\n",
    "        doc.content if hasattr(doc, 'content') else str(doc),\n",
    "        clean_html=True,\n",
    "        normalize_entities=True,\n",
    "        normalize_numbers=True,\n",
    "        remove_extra_whitespace=True,\n",
    "        lowercase=False\n",
    "    )\n",
    "    normalized_documents.append(normalized_text)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Normalized {i}/{len(documents)} documents...\")\n",
    "\n",
    "print(f\"Chunking {len(normalized_documents)} documents...\")\n",
    "chunked_documents = []\n",
    "for i, doc_text in enumerate(normalized_documents, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            chunks = splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    except Exception:\n",
    "        simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "        chunks = simple_splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    if i % 50 == 0 or i == len(normalized_documents):\n",
    "        print(f\"  Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n",
    "\n",
    "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Transaction Entities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "entity_extractor = NERExtractor(\n",
    "    method=\"llm\",\n",
    "    provider=\"groq\",\n",
    "    llm_model=\"llama-3.1-8b-instant\",\n",
    "    temperature=0.0\n",
    ")\n",
    "\n",
    "all_entities = []\n",
    "print(f\"Extracting entities from {len(chunked_documents)} chunks...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        entities = entity_extractor.extract_entities(\n",
    "            chunk_text,\n",
    "            entity_types=[\"Transaction\", \"Account\", \"Device\", \"Pattern\", \"Anomaly\"]\n",
    "        )\n",
    "        all_entities.extend(entities)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_entities)} entities found)\")\n",
    "\n",
    "transactions = [e for e in all_entities if e.label == \"Transaction\" or \"transaction\" in e.label.lower()]\n",
    "accounts = [e for e in all_entities if e.label == \"Account\" or \"account\" in e.label.lower()]\n",
    "anomalies = [e for e in all_entities if e.label in [\"Anomaly\", \"Pattern\"] or \"anomaly\" in e.label.lower() or \"pattern\" in e.label.lower()]\n",
    "\n",
    "print(f\"Extracted {len(transactions)} transactions, {len(accounts)} accounts, {len(anomalies)} anomalies/patterns\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Transaction Relationships\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"llm\",\n",
    "    provider=\"groq\",\n",
    "    llm_model=\"llama-3.1-8b-instant\",\n",
    "    temperature=0.0\n",
    ")\n",
    "\n",
    "all_relationships = []\n",
    "print(f\"Extracting relationships from {len(chunked_documents)} chunks...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        relationships = relation_extractor.extract_relations(\n",
    "            chunk_text,\n",
    "            entities=all_entities,\n",
    "            relation_types=[\"from\", \"to\", \"triggers\", \"detects\", \"associated_with\", \"causes\"]\n",
    "        )\n",
    "        all_relationships.extend(relationships)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n",
    "\n",
    "print(f\"Extracted {len(all_relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detecting Transaction Conflicts\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
    "from semantica.conflicts.methods import detect_conflicts\n",
    "\n",
    "# Use logical conflict detection for fraud rules\n",
    "# expert_review strategy flags conflicts for manual review by fraud analysts\n",
    "conflict_detector = ConflictDetector()\n",
    "conflict_resolver = ConflictResolver()\n",
    "\n",
    "# Convert all entities to dictionaries for conflict detection\n",
    "print(f\"Converting {len(all_entities)} entities to dictionaries...\")\n",
    "entity_dicts = []\n",
    "for e in all_entities:\n",
    "    entity_dict = {\n",
    "        \"id\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"name\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"text\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"type\": e.label if hasattr(e, 'label') else \"ENTITY\",\n",
    "        \"label\": e.label if hasattr(e, 'label') else \"ENTITY\",\n",
    "        \"confidence\": getattr(e, 'confidence', 1.0),\n",
    "        \"metadata\": getattr(e, 'metadata', {})\n",
    "    }\n",
    "    entity_dicts.append(entity_dict)\n",
    "\n",
    "# Convert all relationships to dictionaries for conflict detection\n",
    "print(f\"Converting {len(all_relationships)} relationships to dictionaries...\")\n",
    "relationship_dicts = []\n",
    "for r in all_relationships:\n",
    "    # Handle different relationship object formats\n",
    "    if hasattr(r, 'subject') and hasattr(r, 'predicate') and hasattr(r, 'object'):\n",
    "        # Relation object format: subject, predicate, object\n",
    "        source = r.subject.text if hasattr(r.subject, 'text') else str(r.subject)\n",
    "        target = r.object.text if hasattr(r.object, 'text') else str(r.object)\n",
    "        rel_type = r.predicate if isinstance(r.predicate, str) else str(r.predicate)\n",
    "    elif hasattr(r, 'source') and hasattr(r, 'target'):\n",
    "        # Alternative format: source, target, type/label\n",
    "        source = r.source.text if hasattr(r.source, 'text') else str(r.source)\n",
    "        target = r.target.text if hasattr(r.target, 'text') else str(r.target)\n",
    "        rel_type = getattr(r, 'label', getattr(r, 'type', 'RELATED_TO'))\n",
    "    else:\n",
    "        # Fallback: try to extract from dict-like object\n",
    "        source = getattr(r, 'source_id', getattr(r, 'source', 'UNKNOWN'))\n",
    "        target = getattr(r, 'target_id', getattr(r, 'target', 'UNKNOWN'))\n",
    "        rel_type = getattr(r, 'type', getattr(r, 'label', 'RELATED_TO'))\n",
    "    \n",
    "    relationship_dict = {\n",
    "        \"id\": f\"{source}_{rel_type}_{target}\",\n",
    "        \"source_id\": source,\n",
    "        \"target_id\": target,\n",
    "        \"type\": rel_type,\n",
    "        \"confidence\": getattr(r, 'confidence', 1.0),\n",
    "        \"metadata\": getattr(r, 'metadata', {})\n",
    "    }\n",
    "    relationship_dicts.append(relationship_dict)\n",
    "\n",
    "print(f\"Detecting logical conflicts in {len(entity_dicts)} entities and {len(relationship_dicts)} relationships...\")\n",
    "\n",
    "# Detect logical conflicts (e.g., conflicting fraud indicators)\n",
    "# Use the standalone function from methods module which accepts method as keyword argument\n",
    "conflicts = detect_conflicts(entity_dicts, method=\"logical\")\n",
    "\n",
    "# Also detect relationship conflicts\n",
    "relationship_conflicts = conflict_detector.detect_relationship_conflicts(relationship_dicts)\n",
    "all_conflicts = conflicts + relationship_conflicts\n",
    "\n",
    "print(f\"Detected {len(conflicts)} entity conflicts and {len(relationship_conflicts)} relationship conflicts (total: {len(all_conflicts)} conflicts)\")\n",
    "\n",
    "if all_conflicts:\n",
    "    print(f\"Resolving conflicts using expert_review strategy...\")\n",
    "    resolved = conflict_resolver.resolve_conflicts(\n",
    "        all_conflicts,\n",
    "        strategy=\"expert_review\"  # Manual review by fraud analysts\n",
    "    )\n",
    "    print(f\"Resolved {len(resolved)} conflicts (flagged for expert review)\")\n",
    "else:\n",
    "    print(\"No conflicts detected\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Temporal Transaction Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "graph_builder = GraphBuilder(\n",
    "    merge_entities=True,\n",
    "    resolve_conflicts=True,\n",
    "    entity_resolution_strategy=\"fuzzy\",\n",
    "    enable_temporal=True,\n",
    "    temporal_granularity=TEMPORAL_GRANULARITY\n",
    ")\n",
    "\n",
    "print(f\"Building knowledge graph from {len(all_entities)} entities and {len(all_relationships)} relationships...\")\n",
    "\n",
    "# GraphBuilder handles Entity and Relationship objects directly\n",
    "kg = graph_builder.build({\n",
    "    \"entities\": all_entities,\n",
    "    \"relationships\": all_relationships\n",
    "})\n",
    "\n",
    "entities_count = len(kg.get('entities', []))\n",
    "relationships_count = len(kg.get('relationships', []))\n",
    "print(f\"Graph: {entities_count} entities, {relationships_count} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Embeddings for Transactions and Accounts\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.embeddings import EmbeddingGenerator\n",
    "\n",
    "embedding_gen = EmbeddingGenerator(\n",
    "    provider=\"sentence_transformers\",\n",
    "    model=EMBEDDING_MODEL\n",
    ")\n",
    "\n",
    "print(f\"Generating embeddings for {len(transactions)} transactions and {len(accounts)} accounts...\")\n",
    "transaction_texts = [t.text for t in transactions]\n",
    "transaction_embeddings = embedding_gen.generate_embeddings(transaction_texts)\n",
    "\n",
    "account_texts = [a.text for a in accounts]\n",
    "account_embeddings = embedding_gen.generate_embeddings(account_texts)\n",
    "\n",
    "print(f\"Generated {len(transaction_embeddings)} transaction embeddings and {len(account_embeddings)} account embeddings\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populating Vector Store\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
    "\n",
    "print(f\"Storing {len(transaction_embeddings)} transaction vectors and {len(account_embeddings)} account vectors...\")\n",
    "transaction_ids = vector_store.store_vectors(\n",
    "    vectors=transaction_embeddings,\n",
    "    metadata=[{\"type\": \"transaction\", \"name\": t.text, \"label\": t.label} for t in transactions]\n",
    ")\n",
    "\n",
    "account_ids = vector_store.store_vectors(\n",
    "    vectors=account_embeddings,\n",
    "    metadata=[{\"type\": \"account\", \"name\": a.text, \"label\": a.label} for a in accounts]\n",
    ")\n",
    "\n",
    "print(f\"Stored {len(transaction_ids)} transaction vectors and {len(account_ids)} account vectors\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Temporal Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import TemporalGraphQuery\n",
    "\n",
    "temporal_query = TemporalGraphQuery(\n",
    "    enable_temporal_reasoning=True,\n",
    "    temporal_granularity=TEMPORAL_GRANULARITY\n",
    ")\n",
    "\n",
    "query_results = temporal_query.query_at_time(\n",
    "    kg,\n",
    "    query={\"type\": \"Transaction\"},\n",
    "    at_time=\"2024-01-01 10:05:00\"\n",
    ")\n",
    "\n",
    "evolution = temporal_query.analyze_evolution(kg)\n",
    "pattern_results = temporal_query.query_temporal_pattern(kg, pattern=\"sequence\")\n",
    "\n",
    "print(f\"Temporal queries: {len(query_results.get('entities', []))} transactions at query time\")\n",
    "print(f\"Temporal patterns detected: {pattern_results.get('num_patterns', 0)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Temporal Pattern Detection\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import TemporalPatternDetector\n",
    "\n",
    "pattern_detector = TemporalPatternDetector()\n",
    "\n",
    "# Detect temporal fraud patterns (using sequence pattern detection)\n",
    "fraud_patterns = pattern_detector.detect_temporal_patterns(kg, pattern_type=\"sequence\")\n",
    "\n",
    "# Detect sequence patterns (rapid transactions, unusual timing)\n",
    "sequence_patterns = pattern_detector.detect_temporal_patterns(kg, pattern_type=\"sequence\")\n",
    "\n",
    "print(f\"Detected {len(fraud_patterns)} fraud patterns\")\n",
    "print(f\"Detected {len(sequence_patterns)} sequence patterns\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reasoning and Fraud Detection\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.reasoning import Reasoner\n",
    "from semantica.kg import ConnectivityAnalyzer\n",
    "\n",
    "reasoner = Reasoner()\n",
    "\n",
    "reasoner.add_rule(\"IF Account from Transaction AND Transaction amount > 10000 AND Transaction count > 3 THEN Account triggers Anomaly\")\n",
    "reasoner.add_rule(\"IF Transaction from Account AND Account triggers Anomaly THEN Transaction associated_with Pattern\")\n",
    "\n",
    "inferred_facts = reasoner.infer_facts(kg)\n",
    "\n",
    "# Use Semantica's built-in analyze_connectivity for path finding\n",
    "accounts = [e for e in kg.get('entities', []) if e.get('type') == 'Account']\n",
    "anomalies = [e for e in kg.get('entities', []) if e.get('type') == 'Anomaly']\n",
    "\n",
    "fraud_paths = []\n",
    "if accounts and anomalies:\n",
    "    account_id = accounts[0].get('id') or accounts[0].get('text') or accounts[0].get('name')\n",
    "    anomaly_id = anomalies[0].get('id') or anomalies[0].get('text') or anomalies[0].get('name')\n",
    "    if account_id and anomaly_id:\n",
    "        path_result = analyze_connectivity(kg, method=\"paths\", source=account_id, target=anomaly_id)\n",
    "        if path_result.get('exists'):\n",
    "            fraud_paths = [path_result]\n",
    "\n",
    "print(f\"Inferred {len(inferred_facts)} facts\")\n",
    "print(f\"Found {len(fraud_paths)} fraud paths\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Transaction Network Structure\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphAnalyzer, CommunityDetector\n",
    "\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "community_detector = CommunityDetector()\n",
    "\n",
    "analysis = graph_analyzer.analyze_graph(kg)\n",
    "\n",
    "communities = community_detector.detect_communities(kg, method=\"louvain\")\n",
    "connectivity = graph_analyzer.analyze_connectivity(kg)\n",
    "\n",
    "# Detect suspicious account communities\n",
    "suspicious_communities = []\n",
    "for community in communities:\n",
    "    community_accounts = [e for e in kg.get(\"entities\", []) \n",
    "                          if e.get(\"id\") in community and e.get(\"type\") == \"Account\"]\n",
    "    if len(community_accounts) > 0:\n",
    "        # Check if community has suspicious patterns\n",
    "        suspicious_communities.append({\n",
    "            \"community_id\": len(suspicious_communities),\n",
    "            \"account_count\": len(community_accounts)\n",
    "        })\n",
    "\n",
    "print(f\"Graph analytics:\")\n",
    "print(f\"  - Communities: {len(communities)}\")\n",
    "print(f\"  - Connected components: {len(connectivity.get('components', []))}\")\n",
    "print(f\"  - Graph density: {analysis.get('density', 0):.3f}\")\n",
    "print(f\"  - Suspicious communities: {len(suspicious_communities)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GraphRAG: Hybrid Vector + Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.context import AgentContext, ContextGraph, ContextRetriever\n",
    "from semantica.llms import Groq\n",
    "\n",
    "# Initialize AgentContext with knowledge graph for GraphRAG\n",
    "context = AgentContext(\n",
    "    vector_store=vector_store,\n",
    "    knowledge_graph=kg,\n",
    "    hybrid_alpha=0.7,  # 70% graph, 30% vector\n",
    "    use_graph_expansion=True\n",
    ")\n",
    "\n",
    "# Build context graph using ContextGraph directly\n",
    "print(\"Building context graph from knowledge graph...\")\n",
    "context_graph = ContextGraph()\n",
    "\n",
    "# Convert KG entities and relationships to context graph format\n",
    "kg_entities = kg.get('entities', [])[:50]\n",
    "kg_relationships = kg.get('relationships', [])[:100]\n",
    "\n",
    "# Build context graph from entities and relationships\n",
    "graph_result = context_graph.build_from_entities_and_relationships(\n",
    "    entities=kg_entities,\n",
    "    relationships=kg_relationships\n",
    ")\n",
    "\n",
    "print(f\"Context graph built: {len(context_graph.nodes)} nodes, {len(context_graph.edges)} edges\")\n",
    "\n",
    "# Store transaction data in context graph for better retrieval\n",
    "print(\"\\nStoring transaction data in context graph...\")\n",
    "for i, entity in enumerate(kg.get('entities', [])[:20]):  # Store sample entities\n",
    "    entity_text = f\"{entity.get('text', entity.get('name', ''))} is a {entity.get('type', 'Entity')}\"\n",
    "    context.store(\n",
    "        content=entity_text,\n",
    "        metadata={\"type\": entity.get('type'), \"source\": \"fraud_detection\"},\n",
    "        entities=[entity],\n",
    "        extract_entities=False,  # Already extracted\n",
    "        link_entities=True\n",
    "    )\n",
    "\n",
    "# Get context graph statistics\n",
    "stats = context.stats()\n",
    "print(f\"\\nContext Graph Statistics:\")\n",
    "print(f\"  - Total memories: {stats.get('total_memories', 0)}\")\n",
    "print(f\"  - Graph nodes: {stats.get('graph_nodes', 0)}\")\n",
    "print(f\"  - Graph edges: {stats.get('graph_edges', 0)}\")\n",
    "\n",
    "# Initialize Groq LLM for reasoning\n",
    "llm = Groq(\n",
    "    model=\"llama-3.1-8b-instant\",\n",
    "    api_key=os.getenv(\"GROQ_API_KEY\")\n",
    ")\n",
    "\n",
    "# Query with multi-hop reasoning using Groq LLM and context graph\n",
    "queries = [\n",
    "    \"What accounts have suspicious transaction patterns?\",\n",
    "    \"Which accounts show signs of fraud based on device changes?\",\n",
    "    \"What are the relationships between fraudulent transactions and accounts?\"\n",
    "]\n",
    "\n",
    "print(\"\\n\" + \"=\" * 80)\n",
    "print(\"GraphRAG with Multi-Hop Reasoning (Groq LLM + Context Graph)\")\n",
    "print(\"=\" * 80)\n",
    "\n",
    "for query in queries:\n",
    "    print(f\"\\n{'='*80}\")\n",
    "    print(f\"Query: {query}\")\n",
    "    print(f\"{'='*80}\\n\")\n",
    "    \n",
    "    # Use query_with_reasoning for better responses with context graph\n",
    "    result = context.query_with_reasoning(\n",
    "        query=query,\n",
    "        llm_provider=llm,\n",
    "        max_results=15,\n",
    "        max_hops=3,  # Multi-hop reasoning through context graph\n",
    "        min_score=0.2\n",
    "    )\n",
    "    \n",
    "    print(f\"Generated Response:\\n{result.get('response', 'No response')}\\n\")\n",
    "    \n",
    "    if result.get('reasoning_path'):\n",
    "        print(f\"Reasoning Path:\\n{result.get('reasoning_path')}\\n\")\n",
    "    \n",
    "    print(f\"Confidence: {result.get('confidence', 0):.3f}\")\n",
    "    print(f\"Sources Used: {result.get('num_sources', 0)}\")\n",
    "    print(f\"Reasoning Paths: {result.get('num_reasoning_paths', 0)}\")\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the Temporal Fraud Detection Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.visualization import TemporalVisualizer\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "# Prepare temporal KG with timestamps for interactive visualization\n",
    "# Extract timestamps from entities (if they have temporal metadata)\n",
    "timestamps = {}\n",
    "entities = kg.get('entities', [])\n",
    "relationships = kg.get('relationships', [])\n",
    "\n",
    "# Build timestamps map from entity metadata or relationships\n",
    "for entity in entities:\n",
    "    entity_id = entity.get('id') or entity.get('text') or entity.get('name', '')\n",
    "    if entity_id:\n",
    "        # Extract timestamp from entity metadata if available\n",
    "        entity_times = []\n",
    "        if 'timestamp' in entity:\n",
    "            entity_times.append(entity['timestamp'])\n",
    "        elif 'temporal' in entity:\n",
    "            entity_times.extend(entity.get('temporal', []))\n",
    "        else:\n",
    "            # Use relationships to infer timestamps\n",
    "            for rel in relationships:\n",
    "                if rel.get('source') == entity_id or rel.get('target') == entity_id:\n",
    "                    if 'timestamp' in rel:\n",
    "                        entity_times.append(rel['timestamp'])\n",
    "        \n",
    "        if entity_times:\n",
    "            timestamps[entity_id] = sorted(list(set(entity_times)))\n",
    "\n",
    "# If no timestamps found, create synthetic timestamps based on entity order\n",
    "if not timestamps:\n",
    "    base_time = datetime(2024, 1, 1, 10, 0, 0)\n",
    "    for i, entity in enumerate(entities[:50]):  # Limit to first 50 for performance\n",
    "        entity_id = entity.get('id') or entity.get('text') or entity.get('name', '')\n",
    "        if entity_id:\n",
    "            # Assign timestamps in sequence\n",
    "            entity_time = base_time + timedelta(minutes=i)\n",
    "            timestamps[entity_id] = [entity_time.strftime(\"%Y-%m-%d %H:%M:%S\")]\n",
    "\n",
    "# Create temporal KG structure\n",
    "temporal_kg = {\n",
    "    \"entities\": entities[:50],  # Limit for performance\n",
    "    \"relationships\": relationships[:100],  # Limit for performance\n",
    "    \"timestamps\": timestamps\n",
    "}\n",
    "\n",
    "# Initialize TemporalVisualizer\n",
    "temporal_viz = TemporalVisualizer()\n",
    "\n",
    "print(\"Generating interactive temporal dashboard...\")\n",
    "# Create interactive temporal dashboard\n",
    "dashboard_fig = temporal_viz.visualize_temporal_dashboard(\n",
    "    temporal_kg,\n",
    "    output=\"interactive\",\n",
    "    title=\"Fraud Detection - Temporal Knowledge Graph Dashboard\"\n",
    ")\n",
    "\n",
    "# Display the interactive figure\n",
    "if dashboard_fig:\n",
    "    dashboard_fig.show()\n",
    "    print(\"\\n✅ Interactive temporal dashboard displayed above\")\n",
    "else:\n",
    "    print(\"⚠️  Dashboard generation failed\")\n",
    "\n",
    "print(\"\\nGenerating interactive network evolution animation...\")\n",
    "# Create interactive network evolution animation\n",
    "evolution_fig = temporal_viz.visualize_network_evolution(\n",
    "    temporal_kg,\n",
    "    output=\"interactive\",\n",
    "    title=\"Fraud Detection - Network Evolution Over Time\"\n",
    ")\n",
    "\n",
    "# Display the interactive animation\n",
    "if evolution_fig:\n",
    "    evolution_fig.show()\n",
    "    print(\"\\n✅ Interactive network evolution animation displayed above\")\n",
    "else:\n",
    "    print(\"⚠️  Network evolution visualization failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.export import GraphExporter, CSVExporter\n",
    "\n",
    "# Export to JSON and GraphML using GraphExporter\n",
    "graph_exporter = GraphExporter()\n",
    "graph_exporter.export(kg, output_path=\"fraud_detection_kg.json\", format=\"json\")\n",
    "graph_exporter.export(kg, output_path=\"fraud_detection_kg.graphml\", format=\"graphml\")\n",
    "\n",
    "# Export to CSV using CSVExporter\n",
    "csv_exporter = CSVExporter()\n",
    "csv_exporter.export_knowledge_graph(kg, \"fraud_detection_alerts\")\n",
    "# Creates: fraud_detection_alerts_entities.csv, fraud_detection_alerts_relationships.csv\n",
    "\n",
    "print(\"✅ Exported fraud detection knowledge graph:\")\n",
    "print(\"   - JSON: fraud_detection_kg.json\")\n",
    "print(\"   - GraphML: fraud_detection_kg.graphml\")\n",
    "print(\"   - CSV entities: fraud_detection_alerts_entities.csv\")\n",
    "print(\"   - CSV relationships: fraud_detection_alerts_relationships.csv\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
