{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb)\n", "\n", "# Fraud Detection - Temporal KGs & Pattern Detection\n", "\n", "## Overview\n", "\n", "This notebook demonstrates **fraud detection** using Semantica with focus on **temporal knowledge graphs**, **anomaly detection**, and **pattern recognition**. The pipeline analyzes transaction streams using temporal knowledge graphs to detect fraud patterns and anomalies in real-time.\n", "\n", "### Key Features\n", "\n", "- **Temporal Knowledge Graphs**: Builds temporal KGs to track transaction patterns over time\n", "- **Pattern Detection**: Uses TemporalPatternDetector and reasoning for fraud detection\n", "- **Anomaly Detection**: Uses graph-based pattern recognition to identify fraud\n", "- **Conflict Detection**: Detects conflicting transaction data from multiple sources\n", "- **Real-Time Stream Processing**: Demonstrates real-time transaction stream processing\n", "- **Comprehensive Data Sources**: Multiple transaction streams, APIs, and fraud databases\n", "- **Modular Architecture**: Direct use of Semantica modules without core orchestrator\n", "\n", "### Learning Objectives\n", "\n", "- Ingest transaction data from streams and APIs\n", "- Extract transaction entities (Transactions, Accounts, Devices, Patterns, Anomalies)\n", "- Build temporal transaction knowledge graphs\n", "- Perform temporal queries and pattern detection\n", "- Detect fraud patterns using graph reasoning\n", "- Analyze transaction networks using graph analytics\n", "- Store and query transaction data using vector stores\n", "\n", "### Pipeline Flow\n", "\n", "```mermaid\n", "graph TD\n", " A[Data Ingestion] --> B[Document Parsing]\n", " B --> C[Text Processing]\n", " C --> D[Entity Extraction]\n", " D --> E[Relationship Extraction]\n", " E --> F[Deduplication]\n", " F --> G[Conflict Detection]\n", " G --> H[Temporal Knowledge Graph]\n", " H --> I[Embeddings]\n", " I --> J[Vector Store]\n", " H --> K[Temporal Queries]\n", " K --> L[Temporal Pattern Detection]\n", " L --> M[Reasoning & Fraud]\n", " M --> N[Graph Analytics]\n", " J --> O[GraphRAG Queries]\n", " N --> O\n", " O --> P[Visualization]\n", " P --> Q[Export]\n", "```\n", "\n", "## Installation\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Configuration & Setup\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n", "\n", "# Configuration constants\n", "EMBEDDING_DIMENSION = 384\n", "EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n", "CHUNK_SIZE = 500\n", "CHUNK_OVERLAP = 50\n", "TEMPORAL_GRANULARITY = \"minute\"\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Ingesting Transaction Data from Streams\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.ingest import StreamIngestor, WebIngestor, FileIngestor\n", "import os\n", "from contextlib import redirect_stderr\n", "from io import StringIO\n", "\n", "os.makedirs(\"data\", exist_ok=True)\n", "\n", "all_documents = []\n", "\n", "# ============================================================================\n", "# REAL DATA SOURCE 1: CSV Transaction Data\n", "# ============================================================================\n", "# Ingest transaction data from CSV file with comprehensive fraud indicators\n", "print(\"šŸ“Š Ingesting transaction data from CSV...\")\n", "csv_path = \"data/transactions.csv\"\n", "file_ingestor = FileIngestor()\n", "\n", "try:\n", " csv_documents = file_ingestor.ingest(csv_path)\n", " for doc in csv_documents:\n", " if not hasattr(doc, 'metadata'):\n", " doc.metadata = {}\n", " doc.metadata['source'] = 'Transaction CSV'\n", " doc.metadata['data_type'] = 'transaction'\n", " all_documents.append(doc)\n", " print(f\" āœ… Loaded {len(csv_documents)} documents from CSV\")\n", "except Exception as e:\n", " print(f\" āš ļø CSV ingestion failed: {e}\")\n", "\n", "# ============================================================================\n", "# REAL DATA SOURCE 2: JSON Account & Fraud Pattern Data\n", "# ============================================================================\n", "# Ingest account metadata, device information, and fraud patterns from JSON\n", "print(\"šŸ“Š Ingesting account and fraud pattern data from JSON...\")\n", "json_path = \"data/accounts.json\"\n", "\n", "try:\n", " json_documents = file_ingestor.ingest(json_path)\n", " for doc in json_documents:\n", " if not hasattr(doc, 'metadata'):\n", " doc.metadata = {}\n", " doc.metadata['source'] = 'Account JSON'\n", " doc.metadata['data_type'] = 'account_metadata'\n", " all_documents.append(doc)\n", " print(f\" āœ… Loaded {len(json_documents)} documents from JSON\")\n", "except Exception as e:\n", " print(f\" āš ļø JSON ingestion failed: {e}\")\n", "\n", "# ============================================================================\n", "# REAL DATA SOURCE 3: External Payment Processor API (Example)\n", "# ============================================================================\n", "# In production, you would use real APIs like:\n", "# - Stripe API: https://api.stripe.com/v1/charges\n", "# - PayPal API: https://api.paypal.com/v1/payments\n", "# - Square API: https://connect.squareup.com/v2/payments\n", "# \n", "# For demonstration, we'll use a mock API endpoint that returns transaction data\n", "# In production, replace with actual API endpoints and authentication\n", "print(\"šŸ“Š Attempting to ingest from payment processor API...\")\n", "payment_apis = [\n", " \"https://api.stripe.com/v1/charges\", # Stripe (requires API key)\n", " \"https://api.paypal.com/v1/payments\", # PayPal (requires OAuth)\n", " # Add your actual API endpoints here\n", "]\n", "\n", "web_ingestor = WebIngestor()\n", "api_success = False\n", "\n", "for api_url in payment_apis:\n", " try:\n", " with redirect_stderr(StringIO()):\n", " api_documents = web_ingestor.ingest(api_url, method=\"url\")\n", " if api_documents:\n", " for doc in api_documents:\n", " if not hasattr(doc, 'metadata'):\n", " doc.metadata = {}\n", " doc.metadata['source'] = f'Payment API ({api_url.split(\"//\")[1].split(\"/\")[0]})'\n", " doc.metadata['data_type'] = 'api_transaction'\n", " all_documents.append(doc)\n", " print(f\" āœ… Loaded {len(api_documents)} documents from {api_url}\")\n", " api_success = True\n", " break\n", " except Exception:\n", " continue\n", "\n", "if not api_success:\n", " print(\" ā„¹ļø API endpoints require authentication. Using local data sources.\")\n", "\n", "# ============================================================================\n", "# REAL DATA SOURCE 4: Stream Ingestion (Kafka/RabbitMQ)\n", "# ============================================================================\n", "# For real-time fraud detection, ingest from message streams\n", "# Example Kafka configuration:\n", "# stream_config = {\n", "# \"bootstrap_servers\": \"localhost:9092\",\n", "# \"topic\": \"transactions\",\n", "# \"group_id\": \"fraud_detection\"\n", "# }\n", "# stream_ingestor = StreamIngestor()\n", "# stream_documents = stream_ingestor.ingest(stream_config, method=\"kafka\")\n", "print(\"šŸ“Š Stream ingestion (Kafka/RabbitMQ) - Configure in production\")\n", "\n", "# ============================================================================\n", "# PUBLIC FRAUD DETECTION DATASETS (References)\n", "# ============================================================================\n", "# For additional training and testing, consider these public datasets:\n", "# \n", "# 1. Credit Card Fraud Detection (Kaggle)\n", "# URL: https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud\n", "# - 284,807 transactions, 492 fraudulent (0.172% fraud rate)\n", "# - Download and ingest: file_ingestor.ingest(\"creditcard.csv\")\n", "#\n", "# 2. IEEE-CIS Fraud Detection (Kaggle)\n", "# URL: https://www.kaggle.com/competitions/ieee-fraud-detection\n", "# - 590,540 transactions, ~3.5% fraudulent\n", "# - 431 features (400 numerical, 31 categorical)\n", "#\n", "# 3. PaySim Synthetic Financial Dataset\n", "# URL: https://www.kaggle.com/datasets/ealaxi/paysim1\n", "# - 6,000,000 mobile money transactions\n", "# - ~0.14% fraud rate\n", "#\n", "# 4. UCI Credit Card Dataset\n", "# URL: https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients\n", "# - 30,000 credit card clients\n", "# - 23 features including payment history\n", "#\n", "# To use these datasets:\n", "# 1. Download the dataset files\n", "# 2. Place them in the data/ directory\n", "# 3. Use FileIngestor to load: file_ingestor.ingest(\"data/dataset.csv\")\n", "\n", "# ============================================================================\n", "# Fallback: Generate sample data if no sources available\n", "# ============================================================================\n", "if not all_documents:\n", " print(\"āš ļø No data sources found. Generating sample transaction data...\")\n", " tx_data = \"\"\"\n", " 2024-01-01 10:00:00 - Transaction $1000 from Account A123 to Account B456\n", " 2024-01-01 10:01:00 - Transaction $5000 from Account A123 to Account C789\n", " 2024-01-01 10:02:00 - Transaction $10000 from Account A123 to Account D012 (unusual pattern)\n", " 2024-01-01 10:03:00 - Multiple rapid transactions from Account A123 (suspicious)\n", " 2024-01-01 10:04:00 - Transaction $2000 from Account B456 to Account E789\n", " 2024-01-01 10:05:00 - Large transaction $50000 from Account A123 to Account F012 (fraud alert)\n", " 2024-01-01 10:06:00 - Transaction $1500 from Account C789 to Account G345\n", " 2024-01-01 10:07:00 - Unusual device login from Account A123 (suspicious activity)\n", " \"\"\"\n", " with open(\"data/transactions.txt\", \"w\") as f:\n", " f.write(tx_data)\n", " file_ingestor = FileIngestor()\n", " all_documents = file_ingestor.ingest(\"data/transactions.txt\")\n", "\n", "documents = all_documents\n", "print(f\"\\nāœ… Total ingested: {len(documents)} documents from {len(set(doc.metadata.get('source', 'unknown') for doc in documents))} sources\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parsing Transaction Documents\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.parse import DocumentParser\n", "\n", "parser = DocumentParser()\n", "\n", "print(f\"Parsing {len(documents)} documents...\")\n", "parsed_documents = []\n", "for i, doc in enumerate(documents, 1):\n", " try:\n", " parsed = parser.parse(\n", " doc.content if hasattr(doc, 'content') else str(doc),\n", " content_type=\"text\"\n", " )\n", " parsed_documents.append(parsed)\n", " except Exception:\n", " parsed_documents.append(doc)\n", " if i % 50 == 0 or i == len(documents):\n", " print(f\" Parsed {i}/{len(documents)} documents...\")\n", "\n", "documents = parsed_documents\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Normalizing and Chunking Transaction Data\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.normalize import TextNormalizer\n", "from semantica.split import TextSplitter\n", "\n", "normalizer = TextNormalizer()\n", "# Use sentence chunking for transaction logs\n", "splitter = TextSplitter(method=\"sentence\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n", "\n", "print(f\"Normalizing {len(documents)} documents...\")\n", "normalized_documents = []\n", "for i, doc in enumerate(documents, 1):\n", " normalized_text = normalizer.normalize(\n", " doc.content if hasattr(doc, 'content') else str(doc),\n", " clean_html=True,\n", " normalize_entities=True,\n", " normalize_numbers=True,\n", " remove_extra_whitespace=True,\n", " lowercase=False\n", " )\n", " normalized_documents.append(normalized_text)\n", " if i % 50 == 0 or i == len(documents):\n", " print(f\" Normalized {i}/{len(documents)} documents...\")\n", "\n", "print(f\"Chunking {len(normalized_documents)} documents...\")\n", "chunked_documents = []\n", "for i, doc_text in enumerate(normalized_documents, 1):\n", " try:\n", " with redirect_stderr(StringIO()):\n", " chunks = splitter.split(doc_text)\n", " chunked_documents.extend(chunks)\n", " except Exception:\n", " simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n", " chunks = simple_splitter.split(doc_text)\n", " chunked_documents.extend(chunks)\n", " if i % 50 == 0 or i == len(normalized_documents):\n", " print(f\" Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n", "\n", "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Extracting Transaction Entities\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.semantic_extract import NERExtractor\n", "\n", "entity_extractor = NERExtractor(\n", " method=\"llm\",\n", " provider=\"groq\",\n", " llm_model=\"llama-3.1-8b-instant\",\n", " temperature=0.0\n", ")\n", "\n", "all_entities = []\n", "print(f\"Extracting entities from {len(chunked_documents)} chunks...\")\n", "for i, chunk in enumerate(chunked_documents, 1):\n", " chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n", " try:\n", " entities = entity_extractor.extract_entities(\n", " chunk_text,\n", " entity_types=[\"Transaction\", \"Account\", \"Device\", \"Pattern\", \"Anomaly\"]\n", " )\n", " all_entities.extend(entities)\n", " except Exception:\n", " continue\n", " \n", " if i % 20 == 0 or i == len(chunked_documents):\n", " print(f\" Processed {i}/{len(chunked_documents)} chunks ({len(all_entities)} entities found)\")\n", "\n", "transactions = [e for e in all_entities if e.label == \"Transaction\" or \"transaction\" in e.label.lower()]\n", "accounts = [e for e in all_entities if e.label == \"Account\" or \"account\" in e.label.lower()]\n", "anomalies = [e for e in all_entities if e.label in [\"Anomaly\", \"Pattern\"] or \"anomaly\" in e.label.lower() or \"pattern\" in e.label.lower()]\n", "\n", "print(f\"Extracted {len(transactions)} transactions, {len(accounts)} accounts, {len(anomalies)} anomalies/patterns\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Extracting Transaction Relationships\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.semantic_extract import RelationExtractor\n", "\n", "relation_extractor = RelationExtractor(\n", " method=\"llm\",\n", " provider=\"groq\",\n", " llm_model=\"llama-3.1-8b-instant\",\n", " temperature=0.0\n", ")\n", "\n", "all_relationships = []\n", "print(f\"Extracting relationships from {len(chunked_documents)} chunks...\")\n", "for i, chunk in enumerate(chunked_documents, 1):\n", " chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n", " try:\n", " relationships = relation_extractor.extract_relations(\n", " chunk_text,\n", " entities=all_entities,\n", " relation_types=[\"from\", \"to\", \"triggers\", \"detects\", \"associated_with\", \"causes\"]\n", " )\n", " all_relationships.extend(relationships)\n", " except Exception:\n", " continue\n", " \n", " if i % 20 == 0 or i == len(chunked_documents):\n", " print(f\" Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n", "\n", "print(f\"Extracted {len(all_relationships)} relationships\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Detecting Transaction Conflicts\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.conflicts import ConflictDetector, ConflictResolver\n", "from semantica.conflicts.methods import detect_conflicts\n", "\n", "# Use logical conflict detection for fraud rules\n", "# expert_review strategy flags conflicts for manual review by fraud analysts\n", "conflict_detector = ConflictDetector()\n", "conflict_resolver = ConflictResolver()\n", "\n", "# Convert all entities to dictionaries for conflict detection\n", "print(f\"Converting {len(all_entities)} entities to dictionaries...\")\n", "entity_dicts = []\n", "for e in all_entities:\n", " entity_dict = {\n", " \"id\": e.text if hasattr(e, 'text') else str(e),\n", " \"name\": e.text if hasattr(e, 'text') else str(e),\n", " \"text\": e.text if hasattr(e, 'text') else str(e),\n", " \"type\": e.label if hasattr(e, 'label') else \"ENTITY\",\n", " \"label\": e.label if hasattr(e, 'label') else \"ENTITY\",\n", " \"confidence\": getattr(e, 'confidence', 1.0),\n", " \"metadata\": getattr(e, 'metadata', {})\n", " }\n", " entity_dicts.append(entity_dict)\n", "\n", "# Convert all relationships to dictionaries for conflict detection\n", "print(f\"Converting {len(all_relationships)} relationships to dictionaries...\")\n", "relationship_dicts = []\n", "for r in all_relationships:\n", " # Handle different relationship object formats\n", " if hasattr(r, 'subject') and hasattr(r, 'predicate') and hasattr(r, 'object'):\n", " # Relation object format: subject, predicate, object\n", " source = r.subject.text if hasattr(r.subject, 'text') else str(r.subject)\n", " target = r.object.text if hasattr(r.object, 'text') else str(r.object)\n", " rel_type = r.predicate if isinstance(r.predicate, str) else str(r.predicate)\n", " elif hasattr(r, 'source') and hasattr(r, 'target'):\n", " # Alternative format: source, target, type/label\n", " source = r.source.text if hasattr(r.source, 'text') else str(r.source)\n", " target = r.target.text if hasattr(r.target, 'text') else str(r.target)\n", " rel_type = getattr(r, 'label', getattr(r, 'type', 'RELATED_TO'))\n", " else:\n", " # Fallback: try to extract from dict-like object\n", " source = getattr(r, 'source_id', getattr(r, 'source', 'UNKNOWN'))\n", " target = getattr(r, 'target_id', getattr(r, 'target', 'UNKNOWN'))\n", " rel_type = getattr(r, 'type', getattr(r, 'label', 'RELATED_TO'))\n", " \n", " relationship_dict = {\n", " \"id\": f\"{source}_{rel_type}_{target}\",\n", " \"source_id\": source,\n", " \"target_id\": target,\n", " \"type\": rel_type,\n", " \"confidence\": getattr(r, 'confidence', 1.0),\n", " \"metadata\": getattr(r, 'metadata', {})\n", " }\n", " relationship_dicts.append(relationship_dict)\n", "\n", "print(f\"Detecting logical conflicts in {len(entity_dicts)} entities and {len(relationship_dicts)} relationships...\")\n", "\n", "# Detect logical conflicts (e.g., conflicting fraud indicators)\n", "# Use the standalone function from methods module which accepts method as keyword argument\n", "conflicts = detect_conflicts(entity_dicts, method=\"logical\")\n", "\n", "# Also detect relationship conflicts\n", "relationship_conflicts = conflict_detector.detect_relationship_conflicts(relationship_dicts)\n", "all_conflicts = conflicts + relationship_conflicts\n", "\n", "print(f\"Detected {len(conflicts)} entity conflicts and {len(relationship_conflicts)} relationship conflicts (total: {len(all_conflicts)} conflicts)\")\n", "\n", "if all_conflicts:\n", " print(f\"Resolving conflicts using expert_review strategy...\")\n", " resolved = conflict_resolver.resolve_conflicts(\n", " all_conflicts,\n", " strategy=\"expert_review\" # Manual review by fraud analysts\n", " )\n", " print(f\"Resolved {len(resolved)} conflicts (flagged for expert review)\")\n", "else:\n", " print(\"No conflicts detected\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Building Temporal Transaction Knowledge Graph\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.kg import GraphBuilder\n", "\n", "graph_builder = GraphBuilder(\n", " merge_entities=True,\n", " resolve_conflicts=True,\n", " entity_resolution_strategy=\"fuzzy\",\n", " enable_temporal=True,\n", " temporal_granularity=TEMPORAL_GRANULARITY\n", ")\n", "\n", "print(f\"Building knowledge graph from {len(all_entities)} entities and {len(all_relationships)} relationships...\")\n", "\n", "# GraphBuilder handles Entity and Relationship objects directly\n", "kg = graph_builder.build({\n", " \"entities\": all_entities,\n", " \"relationships\": all_relationships\n", "})\n", "\n", "entities_count = len(kg.get('entities', []))\n", "relationships_count = len(kg.get('relationships', []))\n", "print(f\"Graph: {entities_count} entities, {relationships_count} relationships\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generating Embeddings for Transactions and Accounts\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.embeddings import EmbeddingGenerator\n", "\n", "embedding_gen = EmbeddingGenerator(\n", " provider=\"sentence_transformers\",\n", " model=EMBEDDING_MODEL\n", ")\n", "\n", "print(f\"Generating embeddings for {len(transactions)} transactions and {len(accounts)} accounts...\")\n", "transaction_texts = [t.text for t in transactions]\n", "transaction_embeddings = embedding_gen.generate_embeddings(transaction_texts)\n", "\n", "account_texts = [a.text for a in accounts]\n", "account_embeddings = embedding_gen.generate_embeddings(account_texts)\n", "\n", "print(f\"Generated {len(transaction_embeddings)} transaction embeddings and {len(account_embeddings)} account embeddings\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Populating Vector Store\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.vector_store import VectorStore\n", "\n", "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n", "\n", "print(f\"Storing {len(transaction_embeddings)} transaction vectors and {len(account_embeddings)} account vectors...\")\n", "transaction_ids = vector_store.store_vectors(\n", " vectors=transaction_embeddings,\n", " metadata=[{\"type\": \"transaction\", \"name\": t.text, \"label\": t.label} for t in transactions]\n", ")\n", "\n", "account_ids = vector_store.store_vectors(\n", " vectors=account_embeddings,\n", " metadata=[{\"type\": \"account\", \"name\": a.text, \"label\": a.label} for a in accounts]\n", ")\n", "\n", "print(f\"Stored {len(transaction_ids)} transaction vectors and {len(account_ids)} account vectors\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Temporal Graph Queries\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.kg import TemporalGraphQuery\n", "\n", "temporal_query = TemporalGraphQuery(\n", " enable_temporal_reasoning=True,\n", " temporal_granularity=TEMPORAL_GRANULARITY\n", ")\n", "\n", "query_results = temporal_query.query_at_time(\n", " kg,\n", " query={\"type\": \"Transaction\"},\n", " at_time=\"2024-01-01 10:05:00\"\n", ")\n", "\n", "evolution = temporal_query.analyze_evolution(kg)\n", "pattern_results = temporal_query.query_temporal_pattern(kg, pattern=\"sequence\")\n", "\n", "print(f\"Temporal queries: {len(query_results.get('entities', []))} transactions at query time\")\n", "print(f\"Temporal patterns detected: {pattern_results.get('num_patterns', 0)}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Temporal Pattern Detection\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.kg import TemporalPatternDetector\n", "\n", "pattern_detector = TemporalPatternDetector()\n", "\n", "# Detect temporal fraud patterns (using sequence pattern detection)\n", "fraud_patterns = pattern_detector.detect_temporal_patterns(kg, pattern_type=\"sequence\")\n", "\n", "# Detect sequence patterns (rapid transactions, unusual timing)\n", "sequence_patterns = pattern_detector.detect_temporal_patterns(kg, pattern_type=\"sequence\")\n", "\n", "print(f\"Detected {len(fraud_patterns)} fraud patterns\")\n", "print(f\"Detected {len(sequence_patterns)} sequence patterns\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reasoning and Fraud Detection\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.reasoning import Reasoner\n", "from semantica.kg import ConnectivityAnalyzer\n", "\n", "reasoner = Reasoner()\n", "\n", "reasoner.add_rule(\"IF Account from Transaction AND Transaction amount > 10000 AND Transaction count > 3 THEN Account triggers Anomaly\")\n", "reasoner.add_rule(\"IF Transaction from Account AND Account triggers Anomaly THEN Transaction associated_with Pattern\")\n", "\n", "inferred_facts = reasoner.infer_facts(kg)\n", "\n", "# Use Semantica's built-in analyze_connectivity for path finding\n", "accounts = [e for e in kg.get('entities', []) if e.get('type') == 'Account']\n", "anomalies = [e for e in kg.get('entities', []) if e.get('type') == 'Anomaly']\n", "\n", "fraud_paths = []\n", "if accounts and anomalies:\n", " account_id = accounts[0].get('id') or accounts[0].get('text') or accounts[0].get('name')\n", " anomaly_id = anomalies[0].get('id') or anomalies[0].get('text') or anomalies[0].get('name')\n", " if account_id and anomaly_id:\n", " path_result = analyze_connectivity(kg, method=\"paths\", source=account_id, target=anomaly_id)\n", " if path_result.get('exists'):\n", " fraud_paths = [path_result]\n", "\n", "print(f\"Inferred {len(inferred_facts)} facts\")\n", "print(f\"Found {len(fraud_paths)} fraud paths\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Analyzing Transaction Network Structure\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.kg import GraphAnalyzer, CommunityDetector\n", "\n", "graph_analyzer = GraphAnalyzer()\n", "community_detector = CommunityDetector()\n", "\n", "analysis = graph_analyzer.analyze_graph(kg)\n", "\n", "communities = community_detector.detect_communities(kg, method=\"louvain\")\n", "connectivity = graph_analyzer.analyze_connectivity(kg)\n", "\n", "# Detect suspicious account communities\n", "suspicious_communities = []\n", "for community in communities:\n", " community_accounts = [e for e in kg.get(\"entities\", []) \n", " if e.get(\"id\") in community and e.get(\"type\") == \"Account\"]\n", " if len(community_accounts) > 0:\n", " # Check if community has suspicious patterns\n", " suspicious_communities.append({\n", " \"community_id\": len(suspicious_communities),\n", " \"account_count\": len(community_accounts)\n", " })\n", "\n", "print(f\"Graph analytics:\")\n", "print(f\" - Communities: {len(communities)}\")\n", "print(f\" - Connected components: {len(connectivity.get('components', []))}\")\n", "print(f\" - Graph density: {analysis.get('density', 0):.3f}\")\n", "print(f\" - Suspicious communities: {len(suspicious_communities)}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## GraphRAG: Hybrid Vector + Graph Queries\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.context import AgentContext, ContextGraph, ContextRetriever\n", "from semantica.llms import Groq\n", "\n", "# Initialize AgentContext with knowledge graph for GraphRAG\n", "context = AgentContext(\n", " vector_store=vector_store,\n", " knowledge_graph=kg,\n", " hybrid_alpha=0.7, # 70% graph, 30% vector\n", " use_graph_expansion=True\n", ")\n", "\n", "# Build context graph using ContextGraph directly\n", "print(\"Building context graph from knowledge graph...\")\n", "context_graph = ContextGraph()\n", "\n", "# Convert KG entities and relationships to context graph format\n", "kg_entities = kg.get('entities', [])[:50]\n", "kg_relationships = kg.get('relationships', [])[:100]\n", "\n", "# Build context graph from entities and relationships\n", "graph_result = context_graph.build_from_entities_and_relationships(\n", " entities=kg_entities,\n", " relationships=kg_relationships\n", ")\n", "\n", "print(f\"Context graph built: {len(context_graph.nodes)} nodes, {len(context_graph.edges)} edges\")\n", "\n", "# Store transaction data in context graph for better retrieval\n", "print(\"\\nStoring transaction data in context graph...\")\n", "for i, entity in enumerate(kg.get('entities', [])[:20]): # Store sample entities\n", " entity_text = f\"{entity.get('text', entity.get('name', ''))} is a {entity.get('type', 'Entity')}\"\n", " context.store(\n", " content=entity_text,\n", " metadata={\"type\": entity.get('type'), \"source\": \"fraud_detection\"},\n", " entities=[entity],\n", " extract_entities=False, # Already extracted\n", " link_entities=True\n", " )\n", "\n", "# Get context graph statistics\n", "stats = context.stats()\n", "print(f\"\\nContext Graph Statistics:\")\n", "print(f\" - Total memories: {stats.get('total_memories', 0)}\")\n", "print(f\" - Graph nodes: {stats.get('graph_nodes', 0)}\")\n", "print(f\" - Graph edges: {stats.get('graph_edges', 0)}\")\n", "\n", "# Initialize Groq LLM for reasoning\n", "llm = Groq(\n", " model=\"llama-3.1-8b-instant\",\n", " api_key=os.getenv(\"GROQ_API_KEY\")\n", ")\n", "\n", "# Query with multi-hop reasoning using Groq LLM and context graph\n", "queries = [\n", " \"What accounts have suspicious transaction patterns?\",\n", " \"Which accounts show signs of fraud based on device changes?\",\n", " \"What are the relationships between fraudulent transactions and accounts?\"\n", "]\n", "\n", "print(\"\\n\" + \"=\" * 80)\n", "print(\"GraphRAG with Multi-Hop Reasoning (Groq LLM + Context Graph)\")\n", "print(\"=\" * 80)\n", "\n", "for query in queries:\n", " print(f\"\\n{'='*80}\")\n", " print(f\"Query: {query}\")\n", " print(f\"{'='*80}\\n\")\n", " \n", " # Use query_with_reasoning for better responses with context graph\n", " result = context.query_with_reasoning(\n", " query=query,\n", " llm_provider=llm,\n", " max_results=15,\n", " max_hops=3, # Multi-hop reasoning through context graph\n", " min_score=0.2\n", " )\n", " \n", " print(f\"Generated Response:\\n{result.get('response', 'No response')}\\n\")\n", " \n", " if result.get('reasoning_path'):\n", " print(f\"Reasoning Path:\\n{result.get('reasoning_path')}\\n\")\n", " \n", " print(f\"Confidence: {result.get('confidence', 0):.3f}\")\n", " print(f\"Sources Used: {result.get('num_sources', 0)}\")\n", " print(f\"Reasoning Paths: {result.get('num_reasoning_paths', 0)}\")\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Visualizing the Temporal Fraud Detection Knowledge Graph\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.visualization import TemporalVisualizer\n", "from datetime import datetime, timedelta\n", "\n", "# Prepare temporal KG with timestamps for interactive visualization\n", "# Extract timestamps from entities (if they have temporal metadata)\n", "timestamps = {}\n", "entities = kg.get('entities', [])\n", "relationships = kg.get('relationships', [])\n", "\n", "# Build timestamps map from entity metadata or relationships\n", "for entity in entities:\n", " entity_id = entity.get('id') or entity.get('text') or entity.get('name', '')\n", " if entity_id:\n", " # Extract timestamp from entity metadata if available\n", " entity_times = []\n", " if 'timestamp' in entity:\n", " entity_times.append(entity['timestamp'])\n", " elif 'temporal' in entity:\n", " entity_times.extend(entity.get('temporal', []))\n", " else:\n", " # Use relationships to infer timestamps\n", " for rel in relationships:\n", " if rel.get('source') == entity_id or rel.get('target') == entity_id:\n", " if 'timestamp' in rel:\n", " entity_times.append(rel['timestamp'])\n", " \n", " if entity_times:\n", " timestamps[entity_id] = sorted(list(set(entity_times)))\n", "\n", "# If no timestamps found, create synthetic timestamps based on entity order\n", "if not timestamps:\n", " base_time = datetime(2024, 1, 1, 10, 0, 0)\n", " for i, entity in enumerate(entities[:50]): # Limit to first 50 for performance\n", " entity_id = entity.get('id') or entity.get('text') or entity.get('name', '')\n", " if entity_id:\n", " # Assign timestamps in sequence\n", " entity_time = base_time + timedelta(minutes=i)\n", " timestamps[entity_id] = [entity_time.strftime(\"%Y-%m-%d %H:%M:%S\")]\n", "\n", "# Create temporal KG structure\n", "temporal_kg = {\n", " \"entities\": entities[:50], # Limit for performance\n", " \"relationships\": relationships[:100], # Limit for performance\n", " \"timestamps\": timestamps\n", "}\n", "\n", "# Initialize TemporalVisualizer\n", "temporal_viz = TemporalVisualizer()\n", "\n", "print(\"Generating interactive temporal dashboard...\")\n", "# Create interactive temporal dashboard\n", "dashboard_fig = temporal_viz.visualize_temporal_dashboard(\n", " temporal_kg,\n", " output=\"interactive\",\n", " title=\"Fraud Detection - Temporal Knowledge Graph Dashboard\"\n", ")\n", "\n", "# Display the interactive figure\n", "if dashboard_fig:\n", " dashboard_fig.show()\n", " print(\"\\nāœ… Interactive temporal dashboard displayed above\")\n", "else:\n", " print(\"āš ļø Dashboard generation failed\")\n", "\n", "print(\"\\nGenerating interactive network evolution animation...\")\n", "# Create interactive network evolution animation\n", "evolution_fig = temporal_viz.visualize_network_evolution(\n", " temporal_kg,\n", " output=\"interactive\",\n", " title=\"Fraud Detection - Network Evolution Over Time\"\n", ")\n", "\n", "# Display the interactive animation\n", "if evolution_fig:\n", " evolution_fig.show()\n", " print(\"\\nāœ… Interactive network evolution animation displayed above\")\n", "else:\n", " print(\"āš ļø Network evolution visualization failed\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exporting Results\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from semantica.export import GraphExporter, CSVExporter\n", "\n", "# Export to JSON and GraphML using GraphExporter\n", "graph_exporter = GraphExporter()\n", "graph_exporter.export(kg, output_path=\"fraud_detection_kg.json\", format=\"json\")\n", "graph_exporter.export(kg, output_path=\"fraud_detection_kg.graphml\", format=\"graphml\")\n", "\n", "# Export to CSV using CSVExporter\n", "csv_exporter = CSVExporter()\n", "csv_exporter.export_knowledge_graph(kg, \"fraud_detection_alerts\")\n", "# Creates: fraud_detection_alerts_entities.csv, fraud_detection_alerts_relationships.csv\n", "\n", "print(\"āœ… Exported fraud detection knowledge graph:\")\n", "print(\" - JSON: fraud_detection_kg.json\")\n", "print(\" - GraphML: fraud_detection_kg.graphml\")\n", "print(\" - CSV entities: fraud_detection_alerts_entities.csv\")\n", "print(\" - CSV relationships: fraud_detection_alerts_relationships.csv\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 2 }