{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb)\n",
    "\n",
    "# Transaction Network Analysis - Pattern Detection & Graph Analytics\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates **blockchain transaction network analysis** using Semantica with focus on **pattern detection**, **network analytics**, and **real-time processing**. The pipeline analyzes blockchain transaction networks to detect patterns, identify whale movements, and analyze token flows.\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **Pattern Detection**: Emphasizes graph analytics for transaction pattern recognition\n",
    "- **Network Analytics**: Uses centrality measures and community detection\n",
    "- **Temporal Analysis**: Time-aware queries and transaction evolution tracking\n",
    "- **Whale Tracking**: Identifies large transaction movements\n",
    "- **Flow Analysis**: Analyzes token flows through the network\n",
    "- **Comprehensive Data Sources**: Multiple blockchain APIs, analytics platforms, and databases\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Ingest blockchain transaction data from multiple sources\n",
    "- Extract transaction entities (Transactions, Wallets, Addresses, Blocks, Flows)\n",
    "- Build temporal transaction network graphs\n",
    "- Perform graph analytics (centrality, communities, connectivity)\n",
    "- Detect patterns and whale movements\n",
    "- Analyze token flows and transaction paths\n",
    "- Store and query transaction data using vector stores and graph stores\n",
    "\n",
    "### Pipeline Flow\n",
    "\n",
    "```mermaid\n",
    "graph TD\n",
    "    A[Data Ingestion] --> B[Document Parsing]\n",
    "    B --> C[Text Processing]\n",
    "    C --> D[Entity Extraction]\n",
    "    D --> E[Relationship Extraction]\n",
    "    E --> F[Deduplication]\n",
    "    F --> G[Conflict Detection]\n",
    "    G --> H[Transaction Network Graph]\n",
    "    H --> I[Embeddings]\n",
    "    I --> J[Vector Store]\n",
    "    H --> K[Graph Analytics]\n",
    "    K --> L[Temporal Queries]\n",
    "    L --> M[Pattern Detection]\n",
    "    M --> N[Flow Analysis]\n",
    "    J --> O[GraphRAG Queries]\n",
    "    H --> P[Graph Store]\n",
    "    O --> Q[Visualization]\n",
    "    P --> Q\n",
    "    Q --> R[Export]\n",
    "```\n",
    "\n",
    "## Installation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration & Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
    "\n",
    "# Configuration constants\n",
    "EMBEDDING_DIMENSION = 384\n",
    "EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n",
    "CHUNK_SIZE = 1000\n",
    "CHUNK_OVERLAP = 200\n",
    "TEMPORAL_GRANULARITY = \"day\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting Blockchain Transaction Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import WebIngestor, FileIngestor, FeedIngestor\n",
    "import os\n",
    "from contextlib import redirect_stderr\n",
    "from io import StringIO\n",
    "\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "# Blockchain and crypto news RSS feeds\n",
    "feed_sources = [\n",
    "    (\"CoinDesk\", \"https://www.coindesk.com/arc/outboundfeeds/rss/\"),\n",
    "    (\"CoinTelegraph\", \"https://cointelegraph.com/rss\"),\n",
    "    (\"Decrypt\", \"https://decrypt.co/feed\"),\n",
    "    (\"The Block\", \"https://www.theblock.co/rss.xml\"),\n",
    "    (\"CryptoSlate\", \"https://cryptoslate.com/feed/\"),\n",
    "    (\"CryptoNews\", \"https://cryptonews.com/news/feed/\"),\n",
    "    (\"Bitcoin Magazine\", \"https://bitcoinmagazine.com/.rss/full/\"),\n",
    "    (\"Ethereum News\", \"https://ethereum.org/en/feed.xml\"),\n",
    "]\n",
    "\n",
    "# Blockchain data and analytics web sources\n",
    "web_sources = [\n",
    "    (\"Blockchain.com Stats\", \"https://www.blockchain.com/explorer\"),\n",
    "    (\"Etherscan\", \"https://etherscan.io/\"),\n",
    "    (\"Bitcoin Explorer\", \"https://blockstream.info/\"),\n",
    "]\n",
    "\n",
    "# Initialize ingestors\n",
    "feed_ingestor = FeedIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "all_documents = []\n",
    "\n",
    "# Ingest from RSS feeds\n",
    "print(f\"Ingesting from {len(feed_sources)} RSS feed sources...\")\n",
    "for i, (feed_name, feed_url) in enumerate(feed_sources, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
    "        \n",
    "        feed_count = 0\n",
    "        for item in feed_data.items:\n",
    "            if not item.content:\n",
    "                item.content = item.description or item.title or \"\"\n",
    "            if item.content:\n",
    "                if not hasattr(item, 'metadata'):\n",
    "                    item.metadata = {}\n",
    "                item.metadata['source'] = feed_name\n",
    "                item.metadata['type'] = 'feed'\n",
    "                all_documents.append(item)\n",
    "                feed_count += 1\n",
    "        \n",
    "        if feed_count > 0:\n",
    "            print(f\"  [{i}/{len(feed_sources)}] {feed_name}: {feed_count} documents\")\n",
    "    except Exception as e:\n",
    "        if i <= 3:  # Show first few errors\n",
    "            print(f\"  Warning: {feed_name} failed: {str(e)[:50]}\")\n",
    "        continue\n",
    "\n",
    "# Ingest from web sources (transaction-related pages)\n",
    "print(f\"\\nIngesting from {len(web_sources)} web sources...\")\n",
    "for i, (web_name, web_url) in enumerate(web_sources, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            web_documents = web_ingestor.ingest(web_url, method=\"url\")\n",
    "        \n",
    "        web_count = 0\n",
    "        for doc in web_documents:\n",
    "            if not hasattr(doc, 'metadata'):\n",
    "                doc.metadata = {}\n",
    "            doc.metadata['source'] = web_name\n",
    "            doc.metadata['type'] = 'web'\n",
    "            all_documents.append(doc)\n",
    "            web_count += 1\n",
    "        \n",
    "        if web_count > 0:\n",
    "            print(f\"  [{i}/{len(web_sources)}] {web_name}: {web_count} documents\")\n",
    "    except Exception as e:\n",
    "        if i <= 2:  # Show first few errors\n",
    "            print(f\"  Warning: {web_name} failed: {str(e)[:50]}\")\n",
    "        continue\n",
    "\n",
    "# Fallback to sample transaction data if no documents ingested\n",
    "if not all_documents:\n",
    "    print(\"\\n⚠️ No documents ingested from feeds/web sources. Using sample transaction data...\")\n",
    "    tx_data = \"\"\"\n",
    "    Transaction 0x123 transfers 1000 ETH from wallet 0xABC to wallet 0xDEF at block 18500000.\n",
    "    Transaction 0x456 transfers 500 BTC from wallet 0xGHI to wallet 0xJKL at block 18500001.\n",
    "    Large transaction 0x789 moves 10000 ETH (whale movement) from wallet 0xMNO to wallet 0xPQR at block 18500002.\n",
    "    Transaction 0xabc transfers 200 USDT from wallet 0xSTU to wallet 0xVWX at block 18500003.\n",
    "    Transaction 0xdef transfers 5000 ETH from wallet 0xYZA to wallet 0xBCD at block 18500004.\n",
    "    Transaction 0x111 transfers 3000 DAI from wallet 0xEFG to wallet 0xHIJ at block 18500005.\n",
    "    Transaction 0x222 transfers 1500 USDC from wallet 0xKLM to wallet 0xNOP at block 18500006.\n",
    "    Transaction 0x333 transfers 2500 LINK from wallet 0xQRS to wallet 0xTUV at block 18500007.\n",
    "    Transaction 0x444 transfers 8000 MATIC from wallet 0xWXY to wallet 0xZAB at block 18500008.\n",
    "    Transaction 0x555 transfers 12000 UNI from wallet 0xCDE to wallet 0xFGH at block 18500009.\n",
    "    \"\"\"\n",
    "    with open(\"data/transactions.txt\", \"w\") as f:\n",
    "        f.write(tx_data)\n",
    "    file_ingestor = FileIngestor()\n",
    "    all_documents = file_ingestor.ingest(\"data/transactions.txt\")\n",
    "    for doc in all_documents:\n",
    "        if not hasattr(doc, 'metadata'):\n",
    "            doc.metadata = {}\n",
    "        doc.metadata['source'] = 'Sample Data'\n",
    "        doc.metadata['type'] = 'sample'\n",
    "\n",
    "documents = all_documents\n",
    "\n",
    "# Count unique sources properly handling different document types\n",
    "unique_sources = set()\n",
    "for d in documents:\n",
    "    if hasattr(d, 'metadata') and d.metadata:\n",
    "        source = d.metadata.get('source', 'Unknown')\n",
    "        unique_sources.add(source)\n",
    "    elif isinstance(d, dict) and 'metadata' in d:\n",
    "        source = d['metadata'].get('source', 'Unknown')\n",
    "        unique_sources.add(source)\n",
    "\n",
    "print(f\"\\n✅ Total ingested: {len(documents)} documents\")\n",
    "print(f\"   Sources: {len(unique_sources)} unique sources\")\n",
    "if unique_sources:\n",
    "    print(f\"   Source list: {', '.join(sorted(unique_sources))}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing Transaction Documents\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.parse import DocumentParser\n",
    "\n",
    "parser = DocumentParser()\n",
    "\n",
    "print(f\"Parsing {len(documents)} documents...\")\n",
    "parsed_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    try:\n",
    "        parsed = parser.parse(\n",
    "            doc.content if hasattr(doc, 'content') else str(doc),\n",
    "            content_type=\"text\"\n",
    "        )\n",
    "        parsed_documents.append(parsed)\n",
    "    except Exception:\n",
    "        parsed_documents.append(doc)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Parsed {i}/{len(documents)} documents...\")\n",
    "\n",
    "documents = parsed_documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizing and Chunking Transaction Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.split import TextSplitter\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "splitter = TextSplitter(\n",
    "    method=\"entity_aware\",\n",
    "    ner_method=\"spacy\",\n",
    "    chunk_size=CHUNK_SIZE,\n",
    "    chunk_overlap=CHUNK_OVERLAP\n",
    ")\n",
    "\n",
    "print(f\"Normalizing {len(documents)} documents...\")\n",
    "normalized_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    normalized_text = normalizer.normalize(\n",
    "        doc.content if hasattr(doc, 'content') else str(doc),\n",
    "        clean_html=True,\n",
    "        normalize_entities=True,\n",
    "        remove_extra_whitespace=True,\n",
    "        lowercase=False\n",
    "    )\n",
    "    normalized_documents.append(normalized_text)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Normalized {i}/{len(documents)} documents...\")\n",
    "\n",
    "print(f\"Chunking {len(normalized_documents)} documents...\")\n",
    "chunked_documents = []\n",
    "for i, doc_text in enumerate(normalized_documents, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            chunks = splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    except Exception:\n",
    "        simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "        chunks = simple_splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    if i % 50 == 0 or i == len(normalized_documents):\n",
    "        print(f\"  Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n",
    "\n",
    "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Transaction Entities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "# Initialize NERExtractor with ML method only (spaCy)\n",
    "# Note: ML method extracts standard NER labels (PERSON, ORG, GPE, MONEY, etc.)\n",
    "entity_extractor = NERExtractor(\n",
    "    method=[\"ml\"],\n",
    "    min_confidence=0.5\n",
    ")\n",
    "\n",
    "# Extract all entities using Semantica's extract() method - handles batch processing\n",
    "print(f\"Extracting entities from {len(chunked_documents)} chunks using ML (spaCy)...\")\n",
    "batch_results = entity_extractor.extract(chunked_documents)\n",
    "\n",
    "# Flatten results (extract() returns List[List[Entity]] for batch input)\n",
    "all_entities = [entity for entity_list in batch_results for entity in entity_list]\n",
    "\n",
    "# Use Semantica's classify_entities to group by standard labels\n",
    "classified = entity_extractor.classify_entities(all_entities)\n",
    "\n",
    "# Filter entities for blockchain transaction domain\n",
    "# Look for transaction hashes, wallet addresses, block numbers, and crypto tokens\n",
    "transaction_keywords = [\"transaction\", \"tx\", \"0x\", \"transfer\", \"sent\", \"received\"]\n",
    "wallet_keywords = [\"wallet\", \"address\", \"0x\", \"account\"]\n",
    "block_keywords = [\"block\", \"blockchain\", \"height\", \"block number\"]\n",
    "\n",
    "transactions = [\n",
    "    e for e in all_entities \n",
    "    if any(kw in e.text.lower() for kw in transaction_keywords) \n",
    "    or e.label == \"MONEY\"  # Money entities often represent transactions\n",
    "]\n",
    "wallets = [\n",
    "    e for e in all_entities \n",
    "    if any(kw in e.text.lower() for kw in wallet_keywords)\n",
    "    or (len(e.text) >= 26 and e.text.startswith(\"0x\"))  # Ethereum addresses\n",
    "]\n",
    "blocks = [\n",
    "    e for e in all_entities \n",
    "    if any(kw in e.text.lower() for kw in block_keywords)\n",
    "    or e.label == \"CARDINAL\"  # Block numbers are often cardinal numbers\n",
    "]\n",
    "\n",
    "print(f\"\\n✅ Extraction complete!\")\n",
    "print(f\"   Total entities: {len(all_entities)}\")\n",
    "print(f\"   Standard labels: {list(classified.keys())}\")\n",
    "print(f\"   Transactions (filtered): {len(transactions)}\")\n",
    "print(f\"   Wallets/Addresses (filtered): {len(wallets)}\")\n",
    "print(f\"   Blocks (filtered): {len(blocks)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Transaction Relationships\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "# Use ML-based dependency parsing to avoid rate limits\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"dependency\",  # ML/NLP method - no API calls needed\n",
    "    verbose=True\n",
    ")\n",
    "\n",
    "all_relationships = []\n",
    "error_count = 0\n",
    "print(f\"Extracting relationships from {len(chunked_documents)} chunks using ML (dependency parsing)...\")\n",
    "\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        relationships = relation_extractor.extract_relations(\n",
    "            chunk_text,\n",
    "            entities=all_entities,\n",
    "            relation_types=[\"transfers\", \"from\", \"to\", \"in_block\", \"contains\", \"flows_to\"],\n",
    "            verbose=True\n",
    "        )\n",
    "        all_relationships.extend(relationships)\n",
    "    except Exception as e:\n",
    "        error_count += 1\n",
    "        if error_count <= 3:\n",
    "            print(f\"  Warning: Error on chunk {i}: {str(e)[:100]}\")\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n",
    "\n",
    "if error_count > 0:\n",
    "    print(f\"  Note: {error_count} chunks had errors during relation extraction\")\n",
    "\n",
    "print(f\"\\n✅ Extracted {len(all_relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detecting Transaction Conflicts\n",
    "\n",
    "-  **Multi-Type Detection**: Detects entity, relationship, and temporal conflicts across transaction network\n",
    "- **Most Recent Strategy**: Uses `most_recent` resolution for transaction data (most accurate for blockchain)\n",
    "- **Source-Aware Resolution**: Considers source reliability and confidence scores for conflict resolution\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
    "\n",
    "# Initialize conflict detection and resolution\n",
    "conflict_detector = ConflictDetector()\n",
    "conflict_resolver = ConflictResolver()\n",
    "\n",
    "# Use Semantica's conflict detection methods directly\n",
    "# Detects entity, relationship, and temporal conflicts\n",
    "print(f\"Detecting conflicts in {len(all_entities)} entities and {len(all_relationships)} relationships...\")\n",
    "\n",
    "# Convert to dict format for conflict detection (Semantica expects dicts)\n",
    "entity_dicts = [{\"id\": e.text, \"text\": e.text, \"type\": e.label, \"confidence\": getattr(e, 'confidence', 1.0)} for e in all_entities]\n",
    "relationship_dicts = [{\"id\": f\"{r.subject.text}_{r.predicate}_{r.object.text}\", \"source_id\": r.subject.text, \"target_id\": r.object.text, \"type\": r.predicate} for r in all_relationships]\n",
    "\n",
    "# Detect all conflict types using Semantica's methods\n",
    "all_conflicts = []\n",
    "all_conflicts.extend(conflict_detector.detect_entity_conflicts(entity_dicts))\n",
    "all_conflicts.extend(conflict_detector.detect_relationship_conflicts(relationship_dicts))\n",
    "all_conflicts.extend(conflict_detector.detect_temporal_conflicts(entity_dicts))\n",
    "\n",
    "print(f\"Detected {len(all_conflicts)} total conflicts\")\n",
    "\n",
    "# Resolve conflicts using best strategy for transaction networks\n",
    "if all_conflicts:\n",
    "    print(f\"Resolving conflicts using 'most_recent' strategy (best for transaction data)...\")\n",
    "    resolved = conflict_resolver.resolve_conflicts(all_conflicts, strategy=\"most_recent\")\n",
    "    resolved_count = len([r for r in resolved if r.resolved])\n",
    "    print(f\"✅ Resolved {resolved_count}/{len(all_conflicts)} conflicts\")\n",
    "else:\n",
    "    print(\"✅ No conflicts detected - data is consistent\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Temporal Transaction Network Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "# Conflicts already resolved - disable expensive operations\n",
    "graph_builder = GraphBuilder(\n",
    "    merge_entities=False,  # Skip entity merging (already done in conflict resolution)\n",
    "    resolve_conflicts=False,  # Conflicts already resolved\n",
    "    entity_resolution_strategy=\"exact\",  \n",
    "    enable_temporal=True,\n",
    "    temporal_granularity=TEMPORAL_GRANULARITY,\n",
    "    track_history=True,  \n",
    "    version_snapshots=True \n",
    ")\n",
    "\n",
    "# Build graph - Semantica's build() method automatically shows progress and ETA\n",
    "kg_sources = [{\n",
    "    \"entities\": [{\"text\": e.text, \"type\": e.label, \"confidence\": getattr(e, 'confidence', 1.0)} for e in all_entities],\n",
    "    \"relationships\": [{\"source\": r.subject.text, \"target\": r.object.text, \"type\": r.predicate, \"confidence\": getattr(r, 'confidence', 1.0)} for r in all_relationships]\n",
    "}]\n",
    "\n",
    "kg = graph_builder.build(kg_sources)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Embeddings for Transactions and Wallets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.embeddings import EmbeddingGenerator\n",
    "\n",
    "embedding_gen = EmbeddingGenerator(\n",
    "    provider=\"sentence_transformers\",\n",
    "    model=EMBEDDING_MODEL\n",
    ")\n",
    "\n",
    "print(f\"Generating embeddings for {len(transactions)} transactions and {len(wallets)} wallets...\")\n",
    "transaction_texts = [t.text for t in transactions]\n",
    "transaction_embeddings = embedding_gen.generate_embeddings(transaction_texts)\n",
    "\n",
    "wallet_texts = [w.text for w in wallets]\n",
    "wallet_embeddings = embedding_gen.generate_embeddings(wallet_texts)\n",
    "\n",
    "print(f\"Generated {len(transaction_embeddings)} transaction embeddings and {len(wallet_embeddings)} wallet embeddings\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populating Vector Store\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
    "\n",
    "print(f\"Storing {len(transaction_embeddings)} transaction vectors and {len(wallet_embeddings)} wallet vectors...\")\n",
    "transaction_ids = vector_store.store_vectors(\n",
    "    vectors=transaction_embeddings,\n",
    "    metadata=[{\"type\": \"transaction\", \"name\": t.text, \"label\": t.label} for t in transactions]\n",
    ")\n",
    "\n",
    "wallet_ids = vector_store.store_vectors(\n",
    "    vectors=wallet_embeddings,\n",
    "    metadata=[{\"type\": \"wallet\", \"name\": w.text, \"label\": w.label} for w in wallets]\n",
    ")\n",
    "\n",
    "print(f\"Stored {len(transaction_ids)} transaction vectors and {len(wallet_ids)} wallet vectors\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Graph Structure\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphAnalyzer, CentralityCalculator, CommunityDetector\n",
    "\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "centrality_calc = CentralityCalculator()\n",
    "community_detector = CommunityDetector()\n",
    "\n",
    "analysis = graph_analyzer.analyze_graph(kg)\n",
    "\n",
    "degree_centrality = centrality_calc.calculate_degree_centrality(kg)\n",
    "betweenness_centrality = centrality_calc.calculate_betweenness_centrality(kg)\n",
    "closeness_centrality = centrality_calc.calculate_closeness_centrality(kg)\n",
    "\n",
    "communities = community_detector.detect_communities(kg, method=\"louvain\")\n",
    "connectivity = graph_analyzer.analyze_connectivity(kg)\n",
    "\n",
    "print(f\"Graph analytics:\")\n",
    "print(f\"  - Communities: {len(communities)}\")\n",
    "print(f\"  - Connected components: {len(connectivity.get('components', []))}\")\n",
    "print(f\"  - Graph density: {analysis.get('density', 0):.3f}\")\n",
    "print(f\"  - Central nodes (degree): {len(degree_centrality)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Temporal Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import TemporalGraphQuery\n",
    "\n",
    "temporal_query = TemporalGraphQuery(\n",
    "    enable_temporal_reasoning=True,\n",
    "    temporal_granularity=TEMPORAL_GRANULARITY\n",
    ")\n",
    "\n",
    "query_results = temporal_query.query_at_time(\n",
    "    kg,\n",
    "    query={\"type\": \"Transaction\"},\n",
    "    at_time=\"2024-01-01\"\n",
    ")\n",
    "\n",
    "evolution = temporal_query.analyze_evolution(kg)\n",
    "temporal_patterns = temporal_query.query_temporal_pattern(kg, pattern=\"sequence\")\n",
    "\n",
    "print(f\"Temporal queries: {query_results.get('num_relationships', 0)} relationships at query time\")\n",
    "print(f\"Temporal patterns detected: {temporal_patterns.get('num_patterns', 0)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detecting Patterns and Whale Movements\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Detect whale movements (large transactions)\n",
    "whale_wallets = []\n",
    "for entity in kg.get(\"entities\", []):\n",
    "    if entity.get(\"type\") in [\"Wallet\", \"Address\"]:\n",
    "        # Check for large transaction relationships\n",
    "        related_rels = [r for r in kg.get(\"relationships\", []) \n",
    "                        if r.get(\"source\") == entity.get(\"id\") or r.get(\"target\") == entity.get(\"id\")]\n",
    "        if any(\"large\" in str(r.get(\"type\", \"\")).lower() or \"whale\" in str(r.get(\"type\", \"\")).lower() \n",
    "               for r in related_rels):\n",
    "            whale_wallets.append(entity)\n",
    "\n",
    "# Detect suspicious patterns (high frequency transactions)\n",
    "suspicious_patterns = []\n",
    "for wallet in wallets[:10]:\n",
    "    wallet_name = wallet.text\n",
    "    # Find all relationships where wallet is source or target, and connected to Transaction entities\n",
    "    wallet_relationships = [\n",
    "        r for r in kg.get(\"relationships\", [])\n",
    "        if (r.get(\"source\") == wallet_name or r.get(\"target\") == wallet_name)\n",
    "    ]\n",
    "    # Find connected Transaction entities\n",
    "    transaction_ids = set()\n",
    "    for rel in wallet_relationships:\n",
    "        other_entity_id = rel.get(\"target\") if rel.get(\"source\") == wallet_name else rel.get(\"source\")\n",
    "        # Check if the other entity is a Transaction\n",
    "        other_entity = next((e for e in kg.get(\"entities\", []) if e.get(\"id\") == other_entity_id), None)\n",
    "        if other_entity and other_entity.get(\"type\") == \"Transaction\":\n",
    "            transaction_ids.add(other_entity_id)\n",
    "    \n",
    "    transaction_count = len(transaction_ids)\n",
    "    if transaction_count > 5:  # High transaction frequency\n",
    "        suspicious_patterns.append({\n",
    "            'wallet': wallet_name,\n",
    "            'transaction_count': transaction_count\n",
    "        })\n",
    "\n",
    "print(f\"Whale tracking: {len(whale_wallets)} large transaction wallets identified\")\n",
    "print(f\"Suspicious patterns: {len(suspicious_patterns)} high-frequency wallets\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Token Flows\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Analyze token flows through the network\n",
    "from collections import deque\n",
    "\n",
    "flow_analysis = []\n",
    "for transaction in transactions[:10]:\n",
    "    tx_name = transaction.text\n",
    "    \n",
    "    # Build adjacency list from relationships\n",
    "    adjacency = {}\n",
    "    for rel in kg.get(\"relationships\", []):\n",
    "        source = rel.get(\"source\")\n",
    "        target = rel.get(\"target\")\n",
    "        if source and target:\n",
    "            if source not in adjacency:\n",
    "                adjacency[source] = []\n",
    "            if target not in adjacency[source]:\n",
    "                adjacency[source].append(target)\n",
    "            if target not in adjacency:\n",
    "                adjacency[target] = []\n",
    "            if source not in adjacency[target]:\n",
    "                adjacency[target].append(source)\n",
    "    \n",
    "    # BFS to find wallets within 2 hops\n",
    "    if tx_name not in adjacency:\n",
    "        continue\n",
    "    \n",
    "    queue = deque([(tx_name, [tx_name], 0)])\n",
    "    visited = {tx_name}\n",
    "    paths_to_wallets = []\n",
    "    \n",
    "    while queue:\n",
    "        node, path, hops = queue.popleft()\n",
    "        \n",
    "        if hops > 2:  # Max 2 hops\n",
    "            continue\n",
    "        \n",
    "        # Check if current node is a Wallet or Address\n",
    "        entity = next((e for e in kg.get(\"entities\", []) if e.get(\"id\") == node), None)\n",
    "        if entity and entity.get(\"type\") in [\"Wallet\", \"Address\"] and node != tx_name:\n",
    "            paths_to_wallets.append({\n",
    "                'transaction': tx_name,\n",
    "                'flow_path': path,\n",
    "                'target': node,\n",
    "                'path_length': len(path) - 1\n",
    "            })\n",
    "        \n",
    "        # Continue BFS\n",
    "        for neighbor in adjacency.get(node, []):\n",
    "            if neighbor not in visited:\n",
    "                visited.add(neighbor)\n",
    "                queue.append((neighbor, path + [neighbor], hops + 1))\n",
    "    \n",
    "    flow_analysis.extend(paths_to_wallets)\n",
    "\n",
    "flow_analysis.sort(key=lambda x: x['path_length'])\n",
    "\n",
    "print(f\"Flow analysis: {len(flow_analysis)} token flow paths identified\")\n",
    "for i, flow in enumerate(flow_analysis[:5], 1):\n",
    "    print(f\"{i}. {flow['transaction']} -> {flow['target']} (path length: {flow['path_length']})\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing Transaction Network (Optional)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.graph_store import GraphStore\n",
    "\n",
    "# Optional: Store to persistent graph database\n",
    "# graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n",
    "# graph_store.store_graph(kg)\n",
    "\n",
    "print(\"Graph store configured (commented out for demo)\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GraphRAG: Hybrid Vector + Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.context import AgentContext\n",
    "\n",
    "context = AgentContext(vector_store=vector_store, knowledge_graph=kg)\n",
    "\n",
    "query = \"What are the largest transactions?\"\n",
    "results = context.retrieve(\n",
    "    query,\n",
    "    max_results=10,\n",
    "    use_graph=True,\n",
    "    expand_graph=True,\n",
    "    include_entities=True,\n",
    "    include_relationships=True\n",
    ")\n",
    "\n",
    "print(f\"GraphRAG query: '{query}'\")\n",
    "print(f\"\\nRetrieved {len(results)} results:\\n\")\n",
    "for i, result in enumerate(results[:5], 1):\n",
    "    print(f\"{i}. Score: {result.get('score', 0):.3f}\")\n",
    "    print(f\"   Content: {result.get('content', '')[:200]}...\")\n",
    "    if result.get('related_entities'):\n",
    "        print(f\"   Related entities: {len(result['related_entities'])}\")\n",
    "    print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the Transaction Network\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.visualization import KGVisualizer\n",
    "\n",
    "# Create visualizer with force-directed layout for better interactive visualization\n",
    "visualizer = KGVisualizer(\n",
    "    layout=\"force\",  # Force-directed layout for better node distribution\n",
    "    color_scheme=\"vibrant\",  # Better color scheme\n",
    "    node_size=15,\n",
    "    edge_width=1.5\n",
    ")\n",
    "\n",
    "# Create interactive network visualization\n",
    "fig = visualizer.visualize_network(\n",
    "    kg,\n",
    "    output=\"interactive\",  # Interactive Plotly visualization\n",
    "    file_path=\"transaction_network.html\",  # Also save to HTML file\n",
    "    node_color_by=\"type\",  # Color nodes by entity type\n",
    "    hover_data=[\"type\", \"label\"]  # Show type and label in hover tooltip\n",
    ")\n",
    "\n",
    "# Display the interactive figure in the notebook\n",
    "fig.show()\n",
    "\n",
    "print(\"✅ Interactive visualization displayed above\")\n",
    "print(\"📁 Visualization also saved to transaction_network.html\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.export import GraphExporter, export_csv\n",
    "\n",
    "# Export to graph formats using GraphExporter\n",
    "graph_exporter = GraphExporter()\n",
    "graph_exporter.export(kg, file_path=\"transaction_network.json\", format=\"json\")\n",
    "graph_exporter.export(kg, file_path=\"transaction_network.graphml\", format=\"graphml\")\n",
    "graph_exporter.export(kg, file_path=\"transaction_network.gexf\", format=\"gexf\")\n",
    "graph_exporter.export(kg, file_path=\"transaction_network.dot\", format=\"dot\")\n",
    "\n",
    "# Export to CSV using export_csv convenience function\n",
    "# This creates separate CSV files for entities and relationships\n",
    "export_csv(kg, \"transaction_network\")\n",
    "\n",
    "print(\"✅ Exported transaction network to multiple formats:\")\n",
    "print(\"   - JSON: transaction_network.json\")\n",
    "print(\"   - GraphML: transaction_network.graphml\")\n",
    "print(\"   - GEXF: transaction_network.gexf\")\n",
    "print(\"   - DOT: transaction_network.dot\")\n",
    "print(\"   - CSV: transaction_network_entities.csv, transaction_network_relationships.csv\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
