{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb)\n",
    "\n",
    "# DeFi Protocol Intelligence - Risk Assessment & Ontology Reasoning\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates **DeFi protocol intelligence** using Semantica with focus on **risk assessment**, **ontology-based reasoning**, and **relationship analysis**. The pipeline ingests DeFi data from multiple sources, extracts protocol entities, builds knowledge graphs, and assesses risks using graph reasoning.\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **Risk Assessment Focus**: Emphasizes KG construction and reasoning for risk evaluation\n",
    "- **Ontology-Based Reasoning**: Uses domain ontologies for DeFi protocol analysis\n",
    "- **Relationship Analysis**: Analyzes protocol relationships and dependencies\n",
    "- **Comprehensive Data Sources**: Multiple RSS feeds, APIs, and databases\n",
    "- **Modular Architecture**: Direct use of Semantica modules without core orchestrator\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Ingest DeFi data from multiple sources (RSS feeds, APIs, databases)\n",
    "- Extract DeFi entities (Protocols, Tokens, Pools, Transactions, Risks)\n",
    "- Build and analyze DeFi knowledge graphs\n",
    "- Generate and utilize DeFi ontologies\n",
    "- Perform risk assessment using graph reasoning\n",
    "- Store and query DeFi data using vector stores and graph stores\n",
    "\n",
    "### Pipeline Flow\n",
    "\n",
    "```mermaid\n",
    "graph TD\n",
    "    A[Data Ingestion] --> B[Document Parsing]\n",
    "    B --> C[Text Processing]\n",
    "    C --> D[Entity Extraction]\n",
    "    D --> E[Relationship Extraction]\n",
    "    E --> F[Deduplication]\n",
    "    F --> G[Conflict Detection]\n",
    "    G --> H[Knowledge Graph]\n",
    "    H --> I[Embeddings]\n",
    "    I --> J[Vector Store]\n",
    "    H --> K[Ontology Generation]\n",
    "    K --> L[Reasoning & Risk]\n",
    "    J --> M[GraphRAG Queries]\n",
    "    L --> M\n",
    "    H --> N[Graph Store]\n",
    "    K --> O[Triplet Store]\n",
    "    M --> P[Visualization]\n",
    "    N --> P\n",
    "    O --> P\n",
    "    P --> Q[Export]\n",
    "```\n",
    "\n",
    "## Installation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration & Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
    "\n",
    "# Configuration constants\n",
    "EMBEDDING_DIMENSION = 384\n",
    "EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n",
    "CHUNK_SIZE = 1000\n",
    "CHUNK_OVERLAP = 200\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting DeFi Data from Multiple Sources\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FeedIngestor, FileIngestor, WebIngestor\n",
    "import os\n",
    "from contextlib import redirect_stderr\n",
    "from io import StringIO\n",
    "\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "feed_sources = [\n",
    "    # Crypto News RSS Feeds\n",
    "    (\"CoinDesk\", \"https://www.coindesk.com/arc/outboundfeeds/rss/\"),\n",
    "    (\"CoinTelegraph\", \"https://cointelegraph.com/rss\"),\n",
    "    (\"Decrypt\", \"https://decrypt.co/feed\"),\n",
    "    (\"The Block\", \"https://www.theblock.co/rss.xml\"),\n",
    "    (\"CryptoSlate\", \"https://cryptoslate.com/feed/\"),\n",
    "    (\"CryptoNews\", \"https://cryptonews.com/news/feed/\"),\n",
    "]\n",
    "\n",
    "feed_ingestor = FeedIngestor()\n",
    "all_documents = []\n",
    "\n",
    "print(f\"Ingesting from {len(feed_sources)} feed sources...\")\n",
    "for i, (feed_name, feed_url) in enumerate(feed_sources, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
    "        \n",
    "        feed_count = 0\n",
    "        for item in feed_data.items:\n",
    "            if not item.content:\n",
    "                item.content = item.description or item.title or \"\"\n",
    "            if item.content:\n",
    "                if not hasattr(item, 'metadata'):\n",
    "                    item.metadata = {}\n",
    "                item.metadata['source'] = feed_name\n",
    "                all_documents.append(item)\n",
    "                feed_count += 1\n",
    "        \n",
    "        if feed_count > 0:\n",
    "            print(f\"  [{i}/{len(feed_sources)}] {feed_name}: {feed_count} documents\")\n",
    "    except Exception:\n",
    "        continue\n",
    "\n",
    "if not all_documents:\n",
    "    defi_data = \"\"\"\n",
    "    Uniswap is a decentralized exchange protocol with high liquidity pools. It uses automated market makers (AMMs) for token swaps.\n",
    "    Aave is a lending protocol that offers variable and stable interest rates. Users can deposit assets to earn yield.\n",
    "    Compound is a money market protocol for lending and borrowing cryptocurrencies. It uses algorithmic interest rates.\n",
    "    MakerDAO uses collateralized debt positions (CDPs) for stablecoin generation. DAI is the stablecoin created.\n",
    "    Curve Finance is a decentralized exchange optimized for stablecoin trading with low slippage.\n",
    "    Yearn Finance aggregates yield farming strategies across multiple DeFi protocols.\n",
    "    SushiSwap is a decentralized exchange and automated market maker with yield farming features.\n",
    "    Balancer is a protocol for programmable liquidity and automated portfolio management.\n",
    "    \"\"\"\n",
    "    with open(\"data/defi_protocols.txt\", \"w\") as f:\n",
    "        f.write(defi_data)\n",
    "    file_ingestor = FileIngestor()\n",
    "    all_documents = file_ingestor.ingest(\"data/defi_protocols.txt\")\n",
    "\n",
    "documents = all_documents\n",
    "print(f\"Ingested {len(documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.parse import DocumentParser\n",
    "\n",
    "parser = DocumentParser()\n",
    "\n",
    "print(f\"Parsing {len(documents)} documents...\")\n",
    "parsed_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    try:\n",
    "        parsed = parser.parse(\n",
    "            doc.content if hasattr(doc, 'content') else str(doc),\n",
    "            content_type=\"text\"\n",
    "        )\n",
    "        parsed_documents.append(parsed)\n",
    "    except Exception:\n",
    "        parsed_documents.append(doc)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Parsed {i}/{len(documents)} documents...\")\n",
    "\n",
    "documents = parsed_documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizing and Chunking DeFi Documents\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.split import TextSplitter\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "splitter = TextSplitter(\n",
    "    method=\"entity_aware\",\n",
    "    ner_method=\"spacy\",\n",
    "    chunk_size=CHUNK_SIZE,\n",
    "    chunk_overlap=CHUNK_OVERLAP\n",
    ")\n",
    "\n",
    "print(f\"Normalizing {len(documents)} documents...\")\n",
    "normalized_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    normalized_text = normalizer.normalize(\n",
    "        doc.content if hasattr(doc, 'content') else str(doc),\n",
    "        clean_html=True,\n",
    "        normalize_entities=True,\n",
    "        remove_extra_whitespace=True,\n",
    "        lowercase=False\n",
    "    )\n",
    "    normalized_documents.append(normalized_text)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Normalized {i}/{len(documents)} documents...\")\n",
    "\n",
    "print(f\"Chunking {len(normalized_documents)} documents...\")\n",
    "chunked_documents = []\n",
    "for i, doc_text in enumerate(normalized_documents, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            chunks = splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    except Exception:\n",
    "        simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "        chunks = simple_splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    if i % 50 == 0 or i == len(normalized_documents):\n",
    "        print(f\"  Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n",
    "\n",
    "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting DeFi Entities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "# Initialize NERExtractor with ML method only (spaCy)\n",
    "# Note: ML method extracts standard NER labels (PERSON, ORG, GPE, etc.)\n",
    "entity_extractor = NERExtractor(\n",
    "    method=[\"ml\"],\n",
    "    min_confidence=0.5\n",
    ")\n",
    "\n",
    "# Extract all entities (ML method doesn't support custom entity types)\n",
    "# We'll filter/classify them after extraction\n",
    "print(f\"Extracting entities from {len(chunked_documents)} chunks using ML (spaCy)...\")\n",
    "batch_results = entity_extractor.extract(chunked_documents)\n",
    "\n",
    "# Flatten results (extract() returns List[List[Entity]] for batch input)\n",
    "all_entities = [entity for entity_list in batch_results for entity in entity_list]\n",
    "\n",
    "# Use Semantica's classify_entities to group by standard labels\n",
    "classified = entity_extractor.classify_entities(all_entities)\n",
    "\n",
    "# Filter entities for DeFi domain - look for protocol/token names in ORG entities\n",
    "# and common DeFi-related terms\n",
    "protocol_keywords = [\"uniswap\", \"aave\", \"compound\", \"makerdao\", \"curve\", \"yearn\", \n",
    "                     \"sushiswap\", \"balancer\", \"protocol\", \"defi\", \"dapp\"]\n",
    "token_keywords = [\"token\", \"coin\", \"crypto\", \"btc\", \"eth\", \"dai\", \"usdc\", \"usdt\"]\n",
    "risk_keywords = [\"risk\", \"vulnerability\", \"exploit\", \"hack\", \"attack\", \"breach\"]\n",
    "\n",
    "protocols = [\n",
    "    e for e in all_entities \n",
    "    if e.label == \"ORG\" or any(kw in e.text.lower() for kw in protocol_keywords)\n",
    "]\n",
    "tokens = [\n",
    "    e for e in all_entities \n",
    "    if any(kw in e.text.lower() for kw in token_keywords) or e.label == \"MONEY\"\n",
    "]\n",
    "risks = [\n",
    "    e for e in all_entities \n",
    "    if any(kw in e.text.lower() for kw in risk_keywords)\n",
    "]\n",
    "\n",
    "print(f\"\\n✅ Extraction complete!\")\n",
    "print(f\"   Total entities: {len(all_entities)}\")\n",
    "print(f\"   Standard labels: {list(classified.keys())}\")\n",
    "print(f\"   Protocols (filtered): {len(protocols)}\")\n",
    "print(f\"   Tokens (filtered): {len(tokens)}\")\n",
    "print(f\"   Risks (filtered): {len(risks)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting DeFi Relationships\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "# Use ML-based dependency parsing to avoid rate limits\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"dependency\",  # ML/NLP method - no API calls needed\n",
    "    verbose=True\n",
    ")\n",
    "\n",
    "all_relationships = []\n",
    "error_count = 0\n",
    "print(f\"Extracting relationships from {len(chunked_documents)} chunks...\")\n",
    "\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        relationships = relation_extractor.extract_relations(\n",
    "            chunk_text,\n",
    "            entities=all_entities,\n",
    "            relation_types=[\"uses\", \"governs\", \"provides\", \"has_risk\", \"interacts_with\", \"depends_on\"],\n",
    "            verbose=True\n",
    "        )\n",
    "        all_relationships.extend(relationships)\n",
    "    except Exception as e:\n",
    "        error_count += 1\n",
    "        if error_count <= 3:\n",
    "            print(f\"  Warning: Error on chunk {i}: {str(e)[:100]}\")\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n",
    "\n",
    "if error_count > 0:\n",
    "    print(f\"  Note: {error_count} chunks had errors during relation extraction\")\n",
    "\n",
    "print(f\"Extracted {len(all_relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resolving Duplicate Entities\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detecting and Resolving Conflicts\n",
    "\n",
    "• **Entity & Relationship Conflict Detection**: Detects conflicts in both entity properties (protocol names, addresses) and relationships (protocol interactions, dependencies) from multiple data sources to ensure data consistency across the DeFi knowledge graph.\n",
    "\n",
    "• **Credibility-Weighted Resolution**: Uses credibility-weighted strategy that considers source reliability and extraction confidence scores, prioritizing high-confidence sources for critical DeFi protocol information while aggregating evidence from multiple sources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
    "\n",
    "conflict_detector = ConflictDetector()\n",
    "conflict_resolver = ConflictResolver()\n",
    "\n",
    "# Convert entities to dictionaries for conflict detection\n",
    "entity_dicts = [\n",
    "    {\n",
    "        \"id\": e.id if hasattr(e, 'id') else e.text,\n",
    "        \"text\": e.text,\n",
    "        \"label\": e.label,\n",
    "        \"type\": e.label,\n",
    "        \"confidence\": e.confidence if hasattr(e, 'confidence') else 1.0,\n",
    "        \"metadata\": e.metadata if hasattr(e, 'metadata') else {}\n",
    "    }\n",
    "    for e in all_entities\n",
    "]\n",
    "\n",
    "# Convert relationships to dictionaries for conflict detection\n",
    "relationship_dicts = [\n",
    "    {\n",
    "        \"id\": f\"{r.subject.text}_{r.predicate}_{r.object.text}\",\n",
    "        \"source_id\": r.subject.text,\n",
    "        \"target_id\": r.object.text,\n",
    "        \"type\": r.predicate,\n",
    "        \"subject\": r.subject.text,\n",
    "        \"object\": r.object.text,\n",
    "        \"predicate\": r.predicate,\n",
    "        \"confidence\": r.confidence if hasattr(r, 'confidence') else 1.0,\n",
    "        \"metadata\": r.metadata if hasattr(r, 'metadata') else {}\n",
    "    }\n",
    "    for r in all_relationships\n",
    "]\n",
    "\n",
    "# Detect conflicts in both entities and relationships\n",
    "all_conflicts = []\n",
    "\n",
    "# 1. Detect entity conflicts (duplicate protocols, conflicting properties)\n",
    "print(f\"Detecting entity conflicts in {len(entity_dicts)} entities...\")\n",
    "entity_conflicts = conflict_detector.detect_entity_conflicts(entity_dicts)\n",
    "all_conflicts.extend(entity_conflicts)\n",
    "print(f\"Detected {len(entity_conflicts)} entity conflicts\")\n",
    "\n",
    "# 2. Detect relationship conflicts (conflicting protocol interactions)\n",
    "print(f\"Detecting relationship conflicts in {len(relationship_dicts)} relationships...\")\n",
    "relationship_conflicts = conflict_detector.detect_relationship_conflicts(relationship_dicts)\n",
    "all_conflicts.extend(relationship_conflicts)\n",
    "print(f\"Detected {len(relationship_conflicts)} relationship conflicts\")\n",
    "\n",
    "# Resolve all conflicts using credibility-weighted strategy\n",
    "resolved_entities = entity_dicts.copy()\n",
    "resolved_relationships = relationship_dicts.copy()\n",
    "\n",
    "if all_conflicts:\n",
    "    print(f\"Resolving {len(all_conflicts)} conflicts using credibility-weighted strategy...\")\n",
    "    resolved = conflict_resolver.resolve_conflicts(\n",
    "        all_conflicts,\n",
    "        strategy=\"credibility_weighted\"  # Weight by source credibility and confidence\n",
    "    )\n",
    "    \n",
    "    # Apply resolved values back to entities and relationships\n",
    "    for result in resolved:\n",
    "        if result.resolved and result.resolved_value is not None:\n",
    "            if result.metadata.get(\"entity_id\"):\n",
    "                # Entity conflict - update entity\n",
    "                entity_id = result.metadata.get(\"entity_id\")\n",
    "                property_name = result.metadata.get(\"property_name\")\n",
    "                for entity in resolved_entities:\n",
    "                    if entity.get(\"id\") == entity_id and property_name:\n",
    "                        entity[property_name] = result.resolved_value\n",
    "            elif result.metadata.get(\"relationship_id\"):\n",
    "                # Relationship conflict - update relationship\n",
    "                rel_id = result.metadata.get(\"relationship_id\")\n",
    "                property_name = result.metadata.get(\"property_name\")\n",
    "                for rel in resolved_relationships:\n",
    "                    if rel.get(\"id\") == rel_id and property_name:\n",
    "                        rel[property_name] = result.resolved_value\n",
    "    \n",
    "    print(f\"Resolved {len([r for r in resolved if r.resolved])} conflicts\")\n",
    "    print(f\"Applied resolutions to {len(resolved_entities)} entities and {len(resolved_relationships)} relationships\")\n",
    "else:\n",
    "    print(\"No conflicts detected\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building DeFi Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "# Conflicts already resolved - disable conflict detection in GraphBuilder\n",
    "graph_builder = GraphBuilder(\n",
    "    entity_resolution_strategy=\"fuzzy\",\n",
    "    resolve_conflicts=False  # Conflicts already resolved in previous cell\n",
    ")\n",
    "\n",
    "kg_sources = [{\n",
    "    \"entities\": [\n",
    "        {\"id\": e.get(\"id\", e.get(\"text\")), \"text\": e.get(\"text\"), \"type\": e.get(\"type\", e.get(\"label\"))}\n",
    "        for e in resolved_entities\n",
    "    ],\n",
    "    \"relationships\": [\n",
    "        {\n",
    "            \"source\": r.get(\"source_id\", r.get(\"subject\")),\n",
    "            \"target\": r.get(\"target_id\", r.get(\"object\")),\n",
    "            \"type\": r.get(\"type\", r.get(\"predicate\"))\n",
    "        }\n",
    "        for r in resolved_relationships\n",
    "    ]\n",
    "}]\n",
    "\n",
    "kg = graph_builder.build(kg_sources)\n",
    "\n",
    "entities_count = len(kg.get('entities', []))\n",
    "relationships_count = len(kg.get('relationships', []))\n",
    "print(f\"Graph: {entities_count} entities, {relationships_count} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Embeddings for Protocols and Tokens\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.embeddings import EmbeddingGenerator\n",
    "\n",
    "embedding_gen = EmbeddingGenerator(\n",
    "    provider=\"sentence_transformers\",\n",
    "    model=EMBEDDING_MODEL\n",
    ")\n",
    "\n",
    "print(f\"Generating embeddings for {len(protocols)} protocols and {len(tokens)} tokens...\")\n",
    "protocol_texts = [p.text for p in protocols]\n",
    "protocol_embeddings = embedding_gen.generate_embeddings(protocol_texts)\n",
    "\n",
    "token_texts = [t.text for t in tokens]\n",
    "token_embeddings = embedding_gen.generate_embeddings(token_texts)\n",
    "\n",
    "print(f\"Generated {len(protocol_embeddings)} protocol embeddings and {len(token_embeddings)} token embeddings\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populating Vector Store\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
    "\n",
    "print(f\"Storing {len(protocol_embeddings)} protocol vectors and {len(token_embeddings)} token vectors...\")\n",
    "protocol_ids = vector_store.store_vectors(\n",
    "    vectors=protocol_embeddings,\n",
    "    metadata=[{\"type\": \"protocol\", \"name\": p.text, \"label\": p.label} for p in protocols]\n",
    ")\n",
    "\n",
    "token_ids = vector_store.store_vectors(\n",
    "    vectors=token_embeddings,\n",
    "    metadata=[{\"type\": \"token\", \"name\": t.text, \"label\": t.label} for t in tokens]\n",
    ")\n",
    "\n",
    "print(f\"Stored {len(protocol_ids)} protocol vectors and {len(token_ids)} token vectors\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating DeFi Ontology\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ontology import OntologyGenerator\n",
    "\n",
    "ontology_gen = OntologyGenerator(base_uri=\"https://defi.example.org/ontology/\")\n",
    "ontology = ontology_gen.generate_from_graph(kg)\n",
    "\n",
    "print(f\"Generated DeFi ontology with {len(ontology.get('classes', []))} classes\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reasoning and Risk Assessment\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.reasoning import Reasoner\n",
    "from semantica.kg import GraphAnalyzer\n",
    "\n",
    "reasoner = Reasoner()\n",
    "reasoner.add_rule(\"IF Protocol has_risk Risk AND Risk severity high THEN Protocol risk_level critical\")\n",
    "reasoner.add_rule(\"IF Protocol depends_on Protocol AND Protocol has_risk Risk THEN Protocol inherits Risk\")\n",
    "\n",
    "inferred_facts = reasoner.infer_facts(kg)\n",
    "\n",
    "# Find paths from Protocols to Risks using GraphAnalyzer\n",
    "graph_analyzer = GraphAnalyzer(kg)\n",
    "protocols = [e.get(\"id\") or e.get(\"text\") for e in kg.get(\"entities\", []) if e.get(\"type\") == \"Protocol\"]\n",
    "risks = [e.get(\"id\") or e.get(\"text\") for e in kg.get(\"entities\", []) if e.get(\"type\") == \"Risk\"]\n",
    "\n",
    "risk_paths = []\n",
    "for protocol in protocols[:10]:\n",
    "    for risk in risks[:5]:\n",
    "        path = graph_analyzer.connectivity_analyzer.calculate_shortest_paths(kg, source=protocol, target=risk)\n",
    "        if path.get(\"exists\") and path.get(\"distance\", -1) <= 2:\n",
    "            risk_paths.append(path)\n",
    "\n",
    "print(f\"Inferred {len(inferred_facts)} facts\")\n",
    "print(f\"Found {len(risk_paths)} risk paths\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing Knowledge Graph (Optional)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.graph_store import GraphStore\n",
    "\n",
    "# Optional: Store to persistent graph database\n",
    "# graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n",
    "# graph_store.store_graph(kg)\n",
    "\n",
    "print(\"Graph store configured (commented out for demo)\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing Ontology as RDF Triplets (Optional)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.triplet_store import TripletStore\n",
    "\n",
    "# Store knowledge graph and ontology as RDF triplets\n",
    "# Note: Requires Blazegraph running on localhost:9999\n",
    "try:\n",
    "    triplet_store = TripletStore(backend=\"blazegraph\", endpoint=\"http://localhost:9999/blazegraph\")\n",
    "    result = triplet_store.store(knowledge_graph=kg, ontology=ontology)\n",
    "    \n",
    "    if result.get('success'):\n",
    "        print(f\"✓ Stored {result.get('processed', 0)}/{result.get('total', 0)} triplets successfully\")\n",
    "    else:\n",
    "        print(f\"⚠ Stored {result.get('processed', 0)}/{result.get('total', 0)} triplets ({result.get('failed', 0)} failed)\")\n",
    "except Exception as e:\n",
    "    print(f\"⚠ Could not connect to Blazegraph: {str(e)[:100]}\")\n",
    "    print(\"   To use triplet store, start Blazegraph on localhost:9999\")\n",
    "    print(\"   Skipping triplet storage for this demo\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GraphRAG: Hybrid Vector + Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.context import AgentContext\n",
    "\n",
    "context = AgentContext(vector_store=vector_store, knowledge_graph=kg)\n",
    "\n",
    "query = \"What protocols have high risk?\"\n",
    "results = context.retrieve(\n",
    "    query,\n",
    "    max_results=10,\n",
    "    use_graph=True,\n",
    "    expand_graph=True,\n",
    "    include_entities=True,\n",
    "    include_relationships=True\n",
    ")\n",
    "\n",
    "print(f\"GraphRAG query: '{query}'\")\n",
    "print(f\"\\nRetrieved {len(results)} results:\\n\")\n",
    "for i, result in enumerate(results[:5], 1):\n",
    "    print(f\"{i}. Score: {result.get('score', 0):.3f}\")\n",
    "    print(f\"   Content: {result.get('content', '')[:200]}...\")\n",
    "    if result.get('related_entities'):\n",
    "        print(f\"   Related entities: {len(result['related_entities'])}\")\n",
    "    print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the DeFi Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.visualization import KGVisualizer\n",
    "import plotly.graph_objects as go\n",
    "\n",
    "# Print graph statistics for context\n",
    "num_entities = len(kg.get(\"entities\", []))\n",
    "num_relationships = len(kg.get(\"relationships\", []))\n",
    "print(f\"📊 Knowledge Graph Statistics:\")\n",
    "print(f\"   Entities: {num_entities}\")\n",
    "print(f\"   Relationships: {num_relationships}\")\n",
    "print(f\"   Density: {num_relationships / max(num_entities * (num_entities - 1) / 2, 1):.4f}\\n\")\n",
    "\n",
    "# Create visualizer with highly optimized settings for clarity and interactivity\n",
    "visualizer = KGVisualizer(\n",
    "    layout=\"force\",\n",
    "    node_size=20,  # Larger nodes for better visibility\n",
    "    edge_width=1.5,\n",
    "    color_scheme=\"vibrant\",  # Colorful and distinct\n",
    "    k=3.0,  # More spacing between nodes\n",
    "    iterations=150,  # More iterations for stable layout\n",
    "    temperature=0.7,\n",
    "    cooling_factor=0.99\n",
    ")\n",
    "\n",
    "# Generate interactive Plotly figure with maximum interactivity\n",
    "fig = visualizer.visualize_network(\n",
    "    kg,\n",
    "    output=\"interactive\",\n",
    "    algorithm=\"kamada_kawai\",  # Best algorithm for complex graphs\n",
    "    node_color_by=\"type\",  # Color by entity type\n",
    "    node_size_by=None,\n",
    "    hover_data=[\"type\", \"id\", \"label\"],  # Rich hover information\n",
    "    scale=2.5,  # Large scale for clear spacing\n",
    "    seed=42,  # Reproducible\n",
    "    show_detailed_edges=True  # Show edge labels\n",
    ")\n",
    "\n",
    "# Enhance the visualization with better interactivity and explanations\n",
    "if fig:\n",
    "    # Update layout with comprehensive interactive features\n",
    "    fig.update_layout(\n",
    "        title={\n",
    "            \"text\": \"🔗 DeFi Protocol Knowledge Graph - Interactive Visualization\",\n",
    "            \"x\": 0.5,\n",
    "            \"xanchor\": \"center\",\n",
    "            \"font\": {\"size\": 20, \"color\": \"#2c3e50\"}\n",
    "        },\n",
    "        showlegend=True,\n",
    "        hovermode=\"closest\",  # Show closest node on hover\n",
    "        margin=dict(b=40, l=40, r=40, t=80),\n",
    "        xaxis=dict(\n",
    "            showgrid=False,\n",
    "            zeroline=False,\n",
    "            showticklabels=False,\n",
    "            title=\"\"\n",
    "        ),\n",
    "        yaxis=dict(\n",
    "            showgrid=False,\n",
    "            zeroline=False,\n",
    "            showticklabels=False,\n",
    "            title=\"\"\n",
    "        ),\n",
    "        plot_bgcolor=\"rgba(250, 250, 250, 1)\",  # Light gray background\n",
    "        paper_bgcolor=\"white\",\n",
    "        font=dict(family=\"Arial, sans-serif\", size=12),\n",
    "        # Add annotations for explanation\n",
    "        annotations=[\n",
    "            dict(\n",
    "                text=\"💡 <b>How to interact:</b><br>\"\n",
    "                     \"• Hover over nodes to see details<br>\"\n",
    "                     \"• Click and drag to pan<br>\"\n",
    "                     \"• Use mouse wheel to zoom<br>\"\n",
    "                     \"• Double-click to reset view<br>\"\n",
    "                     \"• Colors represent entity types\",\n",
    "                xref=\"paper\",\n",
    "                yref=\"paper\",\n",
    "                x=0.02,\n",
    "                y=0.98,\n",
    "                xanchor=\"left\",\n",
    "                yanchor=\"top\",\n",
    "                bgcolor=\"rgba(255, 255, 255, 0.9)\",\n",
    "                bordercolor=\"rgba(0, 0, 0, 0.2)\",\n",
    "                borderwidth=1,\n",
    "                font=dict(size=10, color=\"#34495e\"),\n",
    "                showarrow=False\n",
    "            )\n",
    "        ],\n",
    "        # Enhanced hover template\n",
    "        hoverlabel=dict(\n",
    "            bgcolor=\"rgba(255, 255, 255, 0.95)\",\n",
    "            bordercolor=\"#3498db\",\n",
    "            font_size=12,\n",
    "            font_family=\"Arial\"\n",
    "        ),\n",
    "        # Make it more responsive\n",
    "        autosize=True,\n",
    "        height=800,\n",
    "        width=None\n",
    "    )\n",
    "    \n",
    "    # Update traces for better interactivity\n",
    "    for trace in fig.data:\n",
    "        if hasattr(trace, 'marker'):\n",
    "            # Enhance node visibility\n",
    "            trace.marker.line.width = 2\n",
    "            trace.marker.line.color = \"white\"\n",
    "            trace.marker.opacity = 0.9\n",
    "        if hasattr(trace, 'text'):\n",
    "            # Make labels more readable\n",
    "            trace.textfont.size = 11\n",
    "            trace.textfont.color = \"#2c3e50\"\n",
    "            trace.textposition = \"middle center\"\n",
    "    \n",
    "    # Add modebar with useful tools\n",
    "    fig.update_layout(\n",
    "        modebar_add=[\n",
    "            \"zoom2d\",\n",
    "            \"pan2d\",\n",
    "            \"select2d\",\n",
    "            \"lasso2d\",\n",
    "            \"zoomIn2d\",\n",
    "            \"zoomOut2d\",\n",
    "            \"autoScale2d\",\n",
    "            \"resetScale2d\"\n",
    "        ]\n",
    "    )\n",
    "    \n",
    "    # Display the enhanced interactive graph\n",
    "    fig.show(config={\n",
    "        \"displayModeBar\": True,\n",
    "        \"displaylogo\": False,\n",
    "        \"modeBarButtonsToAdd\": [\"drawline\", \"drawopenpath\", \"drawclosedpath\", \"drawcircle\", \"drawrect\", \"eraseshape\"],\n",
    "        \"toImageButtonOptions\": {\n",
    "            \"format\": \"png\",\n",
    "            \"filename\": \"defi_kg\",\n",
    "            \"height\": 800,\n",
    "            \"width\": 1200,\n",
    "            \"scale\": 2\n",
    "        }\n",
    "    })\n",
    "    \n",
    "    print(\"\\n✅ Interactive visualization displayed!\")\n",
    "    print(\"   Use the toolbar above to zoom, pan, and interact with the graph\")\n",
    "else:\n",
    "    print(\"⚠️ Could not generate visualization\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.export import GraphExporter, RDFExporter\n",
    "\n",
    "# Export knowledge graph to graph formats\n",
    "graph_exporter = GraphExporter()\n",
    "graph_exporter.export(kg, output_path=\"defi_protocol_kg.json\", format=\"json\")\n",
    "graph_exporter.export(kg, output_path=\"defi_protocol_kg.graphml\", format=\"graphml\")\n",
    "\n",
    "# Export ontology to RDF/TTL format using RDFExporter\n",
    "rdf_exporter = RDFExporter()\n",
    "rdf_string = rdf_exporter.export_to_rdf(ontology, format=\"turtle\")\n",
    "with open(\"defi_ontology.ttl\", \"w\", encoding=\"utf-8\") as f:\n",
    "    f.write(rdf_string)\n",
    "\n",
    "print(\"✅ Exported knowledge graph to JSON and GraphML formats\")\n",
    "print(\"✅ Exported ontology to RDF/TTL format\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
