{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)\n",
    "\n",
    "# GraphRAG Complete - End-to-End Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a **complete end-to-end GraphRAG (Graph-based Retrieval Augmented Generation) system** using the Semantica framework. \n",
    "\n",
    "GraphRAG combines the power of:\n",
    "- **Vector Search**: Semantic similarity matching for finding relevant text chunks\n",
    "- **Knowledge Graphs**: Structured representation of entities and their relationships\n",
    "- **Graph Traversal**: Multi-hop reasoning across entity connections\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **Real-World Data**: Uses actual data sources via RSS feeds, web scraping, and local files (NO mock data)\n",
    "- **Complete Pipeline**: From data ingestion to LLM-powered question answering\n",
    "- **Hybrid Retrieval**: Combines vector similarity search with knowledge graph traversal\n",
    "- **Multi-hop Reasoning**: Follows relationships across the graph for deeper context\n",
    "\n",
    "### What You'll Learn\n",
    "\n",
    "- How to ingest real-world data from multiple sources (RSS feeds, web, local files)\n",
    "- How to build knowledge graphs from unstructured text\n",
    "- How to implement hybrid search combining vectors and graphs\n",
    "- How to use ContextRetriever for intelligent context expansion\n",
    "- How to integrate LLMs with GraphRAG for question answering\n",
    "- How to visualize and export knowledge graphs\n",
    "\n",
    "### Pipeline Architecture\n",
    "\n",
    "The complete GraphRAG pipeline consists of 7 phases:\n",
    "\n",
    "1. **Phase 0**: Setup & Foundation Seeding\n",
    "2. **Phase 1**: Multi-Source Ingestion\n",
    "3. **Phase 2**: Document Processing & Chunking\n",
    "4. **Phase 3**: Comprehensive Semantic Extraction\n",
    "5. **Phase 4**: Knowledge Graph Construction & Refinement\n",
    "6. **Phase 5**: Vector Store Population\n",
    "7. **Phase 6**: GraphRAG Query System\n",
    "8. **Phase 7**: Visualization & Export\n",
    "\n",
    "---\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica and required dependencies:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Ignoring invalid distribution ~gno (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\threading.py:986: ResourceWarning:\n",
      "\n",
      "unclosed file <_io.BufferedWriter name=3>\n",
      "\n",
      "c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\threading.py:986: ResourceWarning:\n",
      "\n",
      "unclosed file <_io.BufferedReader name=4>\n",
      "\n",
      "c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\threading.py:986: ResourceWarning:\n",
      "\n",
      "unclosed file <_io.BufferedReader name=5>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Install Semantica and all required dependencies\n",
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phase 0: Setup & Configuration\n",
    "\n",
    "In this phase, we configure Semantica with all necessary components for the GraphRAG pipeline.\n",
    "\n",
    "### Configuration Components\n",
    "\n",
    "- **Embedding Provider**: Sentence Transformers for generating text embeddings\n",
    "- **Extraction Provider**: Groq LLM for entity and relationship extraction\n",
    "- **Inference Provider**: Groq LLM for answer generation\n",
    "- **Vector Store**: FAISS for efficient vector similarity search\n",
    "- **Knowledge Graph**: NetworkX backend for graph operations\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Core imports will be added in cells where they're first used\n",
    "import os\n",
    "import json\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set API Keys\n",
    "\n",
    "Configure API keys for LLM providers. In production, use environment variables.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up API keys\n",
    "# Note: In production, use environment variables: export GROQ_API_KEY=\"your-key\"\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"Your Groq API\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Semantica\n",
    "\n",
    "Create a configuration dictionary specifying:\n",
    "- Embedding model and provider\n",
    "- Extraction model and provider  \n",
    "- Inference model and provider\n",
    "- Vector store backend and dimensions\n",
    "- Knowledge graph backend and settings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create configuration dictionary\n",
    "config_dict = {\n",
    "    \"project_name\": \"GraphRAG_Complete\",\n",
    "    \n",
    "    # Embedding configuration\n",
    "    \"embedding\": {\n",
    "        \"provider\": \"sentence_transformers\", \n",
    "        \"model\": \"all-MiniLM-L6-v2\"  # 384-dimensional embeddings\n",
    "    }, \n",
    "    \n",
    "    # Extraction configuration (for NER and relation extraction)\n",
    "    \"extraction\": {\n",
    "        \"provider\": \"groq\", \n",
    "        \"model\": \"llama-3.1-8b-instant\", \n",
    "        \"temperature\": 0.0  # Deterministic extraction\n",
    "    },\n",
    "    \n",
    "    # Inference configuration (for answer generation)\n",
    "    \"inference\": {\n",
    "        \"provider\": \"groq\",\n",
    "        \"model\": \"llama-3.3-70b-versatile\"\n",
    "    },\n",
    "    \n",
    "    # Vector store configuration\n",
    "    \"vector_store\": {\n",
    "        \"provider\": \"faiss\", \n",
    "        \"dimension\": 384  # Must match embedding dimension\n",
    "    },\n",
    "    \n",
    "    # Knowledge graph configuration\n",
    "    \"knowledge_graph\": {\n",
    "        \"backend\": \"networkx\", \n",
    "        \"merge_entities\": True  # Automatically merge duplicate entities\n",
    "    }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Core Components\n",
    "\n",
    "Initialize the main Semantica core and vector store using the configuration.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "fastembed not available. Install with: pip install fastembed. Using fallback embedding method.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration complete. Semantica initialized.\n",
      "  - Embedding dimension: 384\n",
      "  - Vector store backend: FAISS\n",
      "  - Knowledge graph backend: NetworkX\n"
     ]
    }
   ],
   "source": [
    "from semantica.core import Semantica, ConfigManager\n",
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "# Load configuration and initialize Semantica core\n",
    "config = ConfigManager().load_from_dict(config_dict)\n",
    "core = Semantica(config=config)\n",
    "\n",
    "# Initialize vector store with matching dimension\n",
    "vs = VectorStore(backend=\"faiss\", dimension=384)\n",
    "\n",
    "print(\"Configuration complete. Semantica initialized.\")\n",
    "print(f\"  - Embedding dimension: 384\")\n",
    "print(f\"  - Vector store backend: FAISS\")\n",
    "print(f\"  - Knowledge graph backend: NetworkX\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phase 0.1: Foundation Seeding\n",
    "\n",
    "Seed the knowledge graph with verified ground truth data. This provides a foundation of known facts that the system can build upon.\n",
    "\n",
    "### Why Foundation Seeding?\n",
    "\n",
    "- **Quality Assurance**: Start with verified, accurate knowledge\n",
    "- **Domain Expertise**: Incorporate expert knowledge into the graph\n",
    "- **Better Extraction**: Guide entity extraction with known entities\n",
    "- **Conflict Resolution**: Use seed data as reference for resolving conflicts\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define foundation data (ground truth)\n",
    "foundation_data = {\n",
    "    \"entities\": [\n",
    "        {\n",
    "            \"id\": \"hyaluronic_acid\", \n",
    "            \"name\": \"Hyaluronic Acid\", \n",
    "            \"type\": \"Ingredient\", \n",
    "            \"properties\": {\"role\": \"Humectant\"}\n",
    "        },\n",
    "        {\n",
    "            \"id\": \"retinol\", \n",
    "            \"name\": \"Retinol\", \n",
    "            \"type\": \"Ingredient\", \n",
    "            \"properties\": {\"role\": \"Anti-aging actives\"}\n",
    "        },\n",
    "        {\n",
    "            \"id\": \"niacinamide\", \n",
    "            \"name\": \"Niacinamide\", \n",
    "            \"type\": \"Ingredient\", \n",
    "            \"properties\": {\"role\": \"Barrier repair\"}\n",
    "        }\n",
    "    ],\n",
    "    \"relationships\": [\n",
    "        {\n",
    "            \"source\": \"hyaluronic_acid\", \n",
    "            \"target\": \"niacinamide\", \n",
    "            \"type\": \"COMPLEMENTS\", \n",
    "            \"properties\": {\"benefit\": \"Hydration + Barrier\"}\n",
    "        }\n",
    "    ]\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save and Register Seed Data\n",
    "\n",
    "Save the foundation data to a JSON file and register it with SeedDataManager.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Foundation data saved to skincare_base.json\n"
     ]
    }
   ],
   "source": [
    "# Save foundation data to JSON file\n",
    "with open(\"skincare_base.json\", \"w\") as f:\n",
    "    json.dump(foundation_data, f, indent=2)\n",
    "\n",
    "print(\"Foundation data saved to skincare_base.json\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Foundation Graph\n",
    "\n",
    "Use SeedDataManager to load the seed data and create the foundation graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phase 0.1 Complete. Seeded 3 primary nodes and 1 relationships.\n",
      "  - Foundation graph created successfully\n",
      "  - Seed data will be merged with extracted data in Phase 4\n"
     ]
    }
   ],
   "source": [
    "from semantica.seed import SeedDataManager\n",
    "\n",
    "# Initialize SeedDataManager\n",
    "seed_manager = SeedDataManager()\n",
    "\n",
    "# Register the seed data source\n",
    "seed_manager.register_source(\"core_ontology\", \"json\", \"skincare_base.json\")\n",
    "\n",
    "# Create foundation graph from seed data\n",
    "foundation_graph = seed_manager.create_foundation_graph()\n",
    "\n",
    "print(f\"Phase 0.1 Complete. Seeded {len(foundation_data['entities'])} primary nodes and {len(foundation_data['relationships'])} relationships.\")\n",
    "print(f\"  - Foundation graph created successfully\")\n",
    "print(f\"  - Seed data will be merged with extracted data in Phase 4\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phase 1: Multi-Source Ingestion\n",
    "\n",
    "Ingest real-world data from multiple sources to build a comprehensive knowledge base.\n",
    "\n",
    "### Data Sources\n",
    "\n",
    "- **RSS/Atom Feeds**: Real-time updates from blogs and news sites\n",
    "- **Web Pages**: Structured content from websites\n",
    "- **Local Files**: Expert guides and documentation\n",
    "\n",
    "### Components\n",
    "\n",
    "- FeedIngestor, WebIngestor, FileIngestor for ingestion\n",
    "- TextNormalizer for content cleaning and normalization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import ingestion and normalization modules\n",
    "from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n",
    "from semantica.normalize import TextNormalizer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Ingestion Components\n",
    "\n",
    "Create instances of the ingestion classes and text normalizer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingestion components initialized.\n"
     ]
    }
   ],
   "source": [
    "from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n",
    "from semantica.normalize import TextNormalizer\n",
    "\n",
    "# Initialize ingestion components\n",
    "normalizer = TextNormalizer()\n",
    "feed_ingestor = FeedIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "file_ingestor = FileIngestor()\n",
    "\n",
    "# Container for all ingested content\n",
    "all_content = []\n",
    "\n",
    "print(\"Ingestion components initialized.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingest RSS/Atom Feeds\n",
    "\n",
    "RSS and Atom feeds provide structured, regularly updated content from blogs and news sites.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting from RSS/Atom feeds...\n",
      "  Processing: https://makeupandbeautyblog.com/feed\n",
      "    Successfully ingested 3 items\n",
      "  Processing: https://www.drbaileyskincare.com/blogs/blog.atom\n",
      "    Successfully ingested 3 items\n",
      "\n",
      "Total feed items ingested: 6\n"
     ]
    }
   ],
   "source": [
    "# Define feed URLs\n",
    "feed_urls = [\n",
    "    \"https://makeupandbeautyblog.com/feed\",\n",
    "    \"https://www.drbaileyskincare.com/blogs/blog.atom\"\n",
    "]\n",
    "\n",
    "print(\"Ingesting from RSS/Atom feeds...\")\n",
    "for url in feed_urls:\n",
    "    try:\n",
    "        print(f\"  Processing: {url}\")\n",
    "        feed_data = feed_ingestor.ingest_feed(url)\n",
    "        \n",
    "        # Extract content from feed items (limit to 3 per feed to avoid rate limits)\n",
    "        for item in feed_data.items[:3]:\n",
    "            text = item.content or item.description or item.title\n",
    "            if text:\n",
    "                all_content.append(text)\n",
    "        print(f\"    Successfully ingested {min(3, len(feed_data.items))} items\")\n",
    "    except Exception as e:\n",
    "        print(f\"    Warning: Failed to ingest {url}: {e}\")\n",
    "\n",
    "print(f\"\\nTotal feed items ingested: {len([c for c in all_content if c])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingest Web Pages\n",
    "\n",
    "Web ingestion extracts content from specific web pages, including Wikipedia articles and documentation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting from web pages...\n",
      "  Processing: https://en.wikipedia.org/wiki/Retinol\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "URL https://en.wikipedia.org/wiki/Retinol blocked by robots.txt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Warning: Failed to ingest https://en.wikipedia.org/wiki/Retinol: URL blocked by robots.txt: https://en.wikipedia.org/wiki/Retinol\n",
      "\n",
      "Total web pages ingested: 6\n"
     ]
    }
   ],
   "source": [
    "# Define web URLs to ingest\n",
    "web_urls = [\n",
    "    \"https://en.wikipedia.org/wiki/Retinol\"\n",
    "]\n",
    "\n",
    "print(\"Ingesting from web pages...\")\n",
    "for url in web_urls:\n",
    "    try:\n",
    "        print(f\"  Processing: {url}\")\n",
    "        content = web_ingestor.ingest_url(url)\n",
    "        if content and content.text:\n",
    "            all_content.append(content.text)\n",
    "            print(f\"    Successfully ingested content ({len(content.text)} characters)\")\n",
    "    except Exception as e:\n",
    "        print(f\"    Warning: Failed to ingest {url}: {e}\")\n",
    "\n",
    "print(f\"\\nTotal web pages ingested: {len([c for c in all_content if c])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingest Local Files\n",
    "\n",
    "Local files provide expert knowledge and documentation that may not be available online.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting from local files...\n",
      "  Processing: expert_skincare_guide.txt\n",
      "    Successfully ingested expert_skincare_guide.txt\n",
      "  Processing: data/sample_graphrag_paper.txt\n",
      "    Successfully ingested data/sample_graphrag_paper.txt\n",
      "\n",
      "Total local files processed: 8\n"
     ]
    }
   ],
   "source": [
    "# Define local files to ingest\n",
    "local_files = [\n",
    "    \"expert_skincare_guide.txt\",\n",
    "    \"data/sample_graphrag_paper.txt\"\n",
    "]\n",
    "\n",
    "print(\"Ingesting from local files...\")\n",
    "for file_path in local_files:\n",
    "    try:\n",
    "        if os.path.exists(file_path):\n",
    "            print(f\"  Processing: {file_path}\")\n",
    "            file_data = file_ingestor.ingest_file(file_path)\n",
    "            # ingest_file returns a single FileObject, not a list\n",
    "            if hasattr(file_data, 'text') and file_data.text:\n",
    "                all_content.append(file_data.text)\n",
    "            elif hasattr(file_data, 'content') and file_data.content:\n",
    "                # If content is bytes, decode it\n",
    "                if isinstance(file_data.content, bytes):\n",
    "                    try:\n",
    "                        text = file_data.content.decode('utf-8')\n",
    "                        all_content.append(text)\n",
    "                    except UnicodeDecodeError:\n",
    "                        print(f\"    Warning: Could not decode content from {file_path}\")\n",
    "            print(f\"    Successfully ingested {file_path}\")\n",
    "        else:\n",
    "            print(f\"    Warning: File not found: {file_path}\")\n",
    "    except Exception as e:\n",
    "        print(f\"    Warning: Failed to ingest {file_path}: {e}\")\n",
    "\n",
    "print(f\"\\nTotal local files processed: {len([c for c in all_content if c])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalize Content\n",
    "\n",
    "Text normalization cleans and standardizes the ingested content:\n",
    "- Removes extra whitespace\n",
    "- Normalizes encoding\n",
    "- Standardizes formatting\n",
    "- Filters out very short content\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalizing content...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Phase 1 Complete. Ingested 8 documents from multiple sources.\n",
      "  - Feed items: 2 feeds processed\n",
      "  - Web pages: 1 URLs processed\n",
      "  - Local files: 2 files processed\n",
      "  - Total normalized documents: 8\n"
     ]
    }
   ],
   "source": [
    "# Normalize all ingested content\n",
    "print(\"Normalizing content...\")\n",
    "normalized_content = []\n",
    "\n",
    "for text in all_content:\n",
    "    if text and len(text) > 50:  # Filter out very short content\n",
    "        normalized_text = normalizer.normalize(text)\n",
    "        normalized_content.append(normalized_text)\n",
    "\n",
    "print(f\"\\nPhase 1 Complete. Ingested {len(normalized_content)} documents from multiple sources.\")\n",
    "print(f\"  - Feed items: {len(feed_urls)} feeds processed\")\n",
    "print(f\"  - Web pages: {len(web_urls)} URLs processed\")\n",
    "print(f\"  - Local files: {len(local_files)} files processed\")\n",
    "print(f\"  - Total normalized documents: {len(normalized_content)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phase 2: Document Processing & Chunking\n",
    "\n",
    "Split documents into semantic chunks that preserve entity boundaries and maintain context.\n",
    "\n",
    "### Why Chunking?\n",
    "\n",
    "- **Processing Efficiency**: LLMs have token limits\n",
    "- **Context Preservation**: Maintain semantic coherence within chunks\n",
    "- **Entity Awareness**: Preserve entity boundaries to avoid splitting important relationships\n",
    "\n",
    "### Chunking Strategy\n",
    "\n",
    "- **EntityAwareChunker**: Intelligently splits text while preserving entity mentions\n",
    "- **Chunk Size**: 1000 characters (balance between context and processing efficiency)\n",
    "- **Overlap**: 200 characters (ensures continuity between chunks)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import chunking module\n",
    "from semantica.split import EntityAwareChunker\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Chunker\n",
    "\n",
    "Configure the EntityAwareChunker with appropriate chunk size and overlap.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EntityAwareChunker initialized.\n",
      "  - Chunk size: 1000 characters\n",
      "  - Overlap: 200 characters\n"
     ]
    }
   ],
   "source": [
    "from semantica.split import EntityAwareChunker\n",
    "\n",
    "# Initialize EntityAwareChunker\n",
    "# - chunk_size: Maximum characters per chunk (1000)\n",
    "# - chunk_overlap: Characters to overlap between chunks (200) for context continuity\n",
    "chunker = EntityAwareChunker(chunk_size=1000, chunk_overlap=200)\n",
    "\n",
    "print(\"EntityAwareChunker initialized.\")\n",
    "print(f\"  - Chunk size: 1000 characters\")\n",
    "print(f\"  - Overlap: 200 characters\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chunk All Documents\n",
    "\n",
    "Process all normalized documents into semantic chunks.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chunking documents...\n",
      "  Document 1: 2 chunks created\n",
      "  Document 2: 2 chunks created\n",
      "  Document 3: 2 chunks created\n",
      "  Document 4: 17 chunks created\n",
      "  Document 5: 26 chunks created\n",
      "  Document 6: 24 chunks created\n",
      "  Document 7: 1 chunks created\n",
      "  Document 8: 2 chunks created\n",
      "\n",
      "Phase 2 Complete. Generated 76 semantic chunks.\n",
      "  - Average chunk size: 2008 characters\n",
      "  - Total chunks: 76\n"
     ]
    }
   ],
   "source": [
    "# Chunk all normalized documents\n",
    "all_chunks = []\n",
    "\n",
    "print(\"Chunking documents...\")\n",
    "for i, text in enumerate(normalized_content, 1):\n",
    "    if text:\n",
    "        chunks = chunker.chunk(text)\n",
    "        all_chunks.extend(chunks)\n",
    "        print(f\"  Document {i}: {len(chunks)} chunks created\")\n",
    "\n",
    "print(f\"\\nPhase 2 Complete. Generated {len(all_chunks)} semantic chunks.\")\n",
    "if all_chunks:\n",
    "    avg_size = sum(len(str(c.text)) for c in all_chunks) / len(all_chunks)\n",
    "    print(f\"  - Average chunk size: {avg_size:.0f} characters\")\n",
    "    print(f\"  - Total chunks: {len(all_chunks)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phase 3: Semantic Extraction\n",
    "\n",
    "Extract structured knowledge from unstructured text using multiple extraction methods.\n",
    "\n",
    "### Extraction Methods\n",
    "\n",
    "1. **Named Entity Recognition (NER)**: Identify entities (people, places, concepts)\n",
    "2. **Relation Extraction**: Find relationships between entities\n",
    "3. **Triplet Extraction**: Extract subject-predicate-object triplets (RDF format)\n",
    "4. **Event Detection**: Identify events and their participants\n",
    "5. **Coreference Resolution**: Link pronouns and references to entities\n",
    "\n",
    "Multiple methods provide comprehensive coverage and increased reliability.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import semantic extraction modules\n",
    "from semantica.semantic_extract import (\n",
    "    NERExtractor, \n",
    "    RelationExtractor, \n",
    "    TripletExtractor,\n",
    "    EventDetector,\n",
    "    CoreferenceResolver\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Extractors\n",
    "\n",
    "Create instances of all extraction classes with appropriate configurations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All extractors initialized successfully.\n",
      "  - NER Extractor: Ready\n",
      "  - Relation Extractor: Ready\n",
      "  - Triplet Extractor: Ready\n",
      "  - Event Detector: Ready\n",
      "  - Coreference Resolver: Ready\n"
     ]
    }
   ],
   "source": [
    "from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
    "\n",
    "# Initialize Named Entity Recognition extractor\n",
    "# Uses LLM method with Groq for high-quality entity extraction\n",
    "ner = NERExtractor(\n",
    "    method=\"llm\", \n",
    "    provider=\"groq\", \n",
    "    model=\"llama-3.1-8b-instant\"\n",
    ")\n",
    "\n",
    "# Initialize Relation Extraction extractor\n",
    "# Extracts relationships between entities\n",
    "rel_ext = RelationExtractor(\n",
    "    method=\"llm\", \n",
    "    provider=\"groq\", \n",
    "    model=\"llama-3.1-8b-instant\"\n",
    ")\n",
    "\n",
    "# Initialize Triplet Extraction extractor\n",
    "# Extracts RDF-style triplets (subject-predicate-object)\n",
    "triplet_ext = TripletExtractor(\n",
    "    method=\"llm\", \n",
    "    provider=\"groq\", \n",
    "    model=\"llama-3.1-8b-instant\"\n",
    ")\n",
    "\n",
    "# Initialize Event Detection extractor\n",
    "# Detects events and their participants\n",
    "event_detector = EventDetector()\n",
    "\n",
    "# Initialize Coreference Resolution\n",
    "# Links pronouns and references to entities\n",
    "coref_resolver = CoreferenceResolver()\n",
    "\n",
    "print(\"All extractors initialized successfully.\")\n",
    "print(\"  - NER Extractor: Ready\")\n",
    "print(\"  - Relation Extractor: Ready\")\n",
    "print(\"  - Triplet Extractor: Ready\")\n",
    "print(\"  - Event Detector: Ready\")\n",
    "print(\"  - Coreference Resolver: Ready\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Results Container\n",
    "\n",
    "Create a dictionary to store all extraction results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results container initialized.\n"
     ]
    }
   ],
   "source": [
    "# Container for all extraction results\n",
    "combined_results = {\n",
    "    \"entities\": [],\n",
    "    \"relationships\": [],\n",
    "    \"triplets\": [],\n",
    "    \"events\": []\n",
    "}\n",
    "\n",
    "print(\"Results container initialized.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract Named Entities\n",
    "\n",
    "Named Entity Recognition identifies entities such as:\n",
    "- **Persons**: People, characters\n",
    "- **Organizations**: Companies, institutions\n",
    "- **Locations**: Places, regions\n",
    "- **Concepts**: Ideas, topics\n",
    "- **Products**: Items, ingredients\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting semantic information from chunks...\n",
      "Processing 10 chunks...\n",
      "\n",
      "Chunk 1/10:\n",
      "  Extracting entities...\n",
      "    Found 3 entities\n",
      "  Extracting relationships...\n",
      "    Found 2 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['makeup and beauty blog', 'monday poll', 'december 15, 2025']\n",
      "    Found 9 triplets\n",
      "  Detecting events...\n",
      "    Found 0 events\n",
      "\n",
      "Chunk 2/10:\n",
      "  Extracting entities...\n",
      "    Found 8 entities\n",
      "  Extracting relationships...\n",
      "    Found 3 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['jo malone', 'laneige', 'ulta', 'trader joe', 'monday', '2007', 'makeup and beauty blog', 'makeup and beauty blog monday poll, vol. 893']\n",
      "    Found 198 triplets\n",
      "  Detecting events...\n",
      "    Found 1 events\n",
      "\n",
      "Chunk 3/10:\n",
      "  Extracting entities...\n",
      "    Found 5 entities\n",
      "  Extracting relationships...\n",
      "    Found 4 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['makeup and beauty blog', 'monday poll', 'makeup and beauty blog monday poll', 'vol. 892', '2007']\n",
      "    Found 5 triplets\n",
      "  Detecting events...\n",
      "    Found 0 events\n",
      "\n",
      "Chunk 4/10:\n",
      "  Extracting entities...\n",
      "    Found 5 entities\n",
      "  Extracting relationships...\n",
      "    Found 2 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['holiday', 'mbb', 'makeup and beauty blog']\n",
      "    Found 9 triplets\n",
      "  Detecting events...\n",
      "    Found 3 events\n",
      "\n",
      "Chunk 5/10:\n",
      "  Extracting entities...\n",
      "    Found 7 entities\n",
      "  Extracting relationships...\n",
      "    Found 3 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['makeup and beauty blog', 'monday poll', 'vol. 891', 'hawaii', 'lanikai', '2024', '2007']\n",
      "    Found 18 triplets\n",
      "  Detecting events...\n",
      "    Found 0 events\n",
      "\n",
      "Chunk 6/10:\n",
      "  Extracting entities...\n",
      "    Found 4 entities\n",
      "  Extracting relationships...\n",
      "    Found 1 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['cavallo point', 'sausalito', 'rosie', 'marnie']\n",
      "    Found 11 triplets\n",
      "  Detecting events...\n",
      "    Found 3 events\n",
      "\n",
      "Chunk 7/10:\n",
      "  Extracting entities...\n",
      "    Found 207 entities\n",
      "  Extracting relationships...\n",
      "    Found 0 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['rosacea', 'dr. bailey', \"dr. bailey's skincare\"]\n",
      "    Found 25 triplets\n",
      "  Detecting events...\n",
      "    Found 0 events\n",
      "\n",
      "Chunk 8/10:\n",
      "  Extracting entities...\n",
      "    Found 9 entities\n",
      "  Extracting relationships...\n",
      "    Found 5 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['dr. bailey', 'green tea antioxidant skin therapy', 'rosacea therapy skin care kit', 'rosacea treatment routine', 'benzoyl peroxide cream', \"dr. bailey's skincare\"]\n",
      "    Found 7 triplets\n",
      "  Detecting events...\n",
      "    Found 0 events\n",
      "\n",
      "Chunk 9/10:\n",
      "  Extracting entities...\n",
      "    Found 7 entities\n",
      "  Extracting relationships...\n",
      "    Found 10 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['facial dandruff', 'rosacea', 'calming zinc soap', 'green tea antioxidant skin therapy', 'clotrimazole', \"dr. bailey's skincare\", 'dr. bailey']\n",
      "    Found 12 triplets\n",
      "  Detecting events...\n",
      "    Found 0 events\n",
      "\n",
      "Chunk 10/10:\n",
      "  Extracting entities...\n",
      "    Found 6 entities\n",
      "  Extracting relationships...\n",
      "    Found 5 relationships\n",
      "  Extracting triplets...\n",
      "DEBUG: Entity map keys: ['hydrate', 'rosacea', 'daily moisturizing face cream', 'omega enriched face booster oil', 'sea buckthorn', 'castor seed oil']\n",
      "    Found 11 triplets\n",
      "  Detecting events...\n",
      "    Found 0 events\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Process first 10 chunks to avoid rate limits\n",
    "print(\"Extracting semantic information from chunks...\")\n",
    "print(f\"Processing {min(10, len(all_chunks))} chunks...\\n\")\n",
    "\n",
    "for i, chunk in enumerate(all_chunks[:10]):\n",
    "    txt = str(chunk.text)\n",
    "    if len(txt) < 50:\n",
    "        continue\n",
    "        \n",
    "    print(f\"Chunk {i+1}/{min(10, len(all_chunks))}:\")\n",
    "    \n",
    "    try:\n",
    "        # Step 1: Named Entity Recognition\n",
    "        print(f\"  Extracting entities...\")\n",
    "        entities = ner.extract(txt)\n",
    "        \n",
    "        for e in entities:\n",
    "            combined_results[\"entities\"].append({\n",
    "                \"name\": e.text, \n",
    "                \"type\": e.label, \n",
    "                \"id\": e.text.lower().replace(' ', '_').replace('-', '_'),\n",
    "                \"confidence\": getattr(e, 'confidence', 0.8)\n",
    "            })\n",
    "        \n",
    "        print(f\"    Found {len(entities)} entities\")\n",
    "        \n",
    "        # Step 2: Relation Extraction (requires entities)\n",
    "        if entities:\n",
    "            print(f\"  Extracting relationships...\")\n",
    "            try:\n",
    "                relations = rel_ext.extract(txt, entities=entities)\n",
    "                \n",
    "                for r in relations:\n",
    "                    combined_results[\"relationships\"].append({\n",
    "                        \"source\": r.subject, \n",
    "                        \"target\": r.object, \n",
    "                        \"type\": r.predicate,\n",
    "                        \"confidence\": getattr(r, 'confidence', 0.7)\n",
    "                    })\n",
    "                \n",
    "                print(f\"    Found {len(relations)} relationships\")\n",
    "            except Exception as e:\n",
    "                print(f\"    Warning: Error extracting relationships: {e}\")\n",
    "        else:\n",
    "            print(f\"  Skipping relationship extraction (no entities found)\")\n",
    "        \n",
    "        # Step 3: Triplet Extraction\n",
    "        print(f\"  Extracting triplets...\")\n",
    "        try:\n",
    "            triplets = triplet_ext.extract_triplets(txt, entities=entities if entities else None)\n",
    "            \n",
    "            for t in triplets:\n",
    "                combined_results[\"triplets\"].append({\n",
    "                    \"subject\": t.subject,\n",
    "                    \"predicate\": t.predicate,\n",
    "                    \"object\": t.object\n",
    "                })\n",
    "            \n",
    "            print(f\"    Found {len(triplets)} triplets\")\n",
    "        except Exception as e:\n",
    "            print(f\"    Warning: Error extracting triplets: {e}\")\n",
    "        \n",
    "        # Step 4: Event Detection\n",
    "        print(f\"  Detecting events...\")\n",
    "        try:\n",
    "            events = event_detector.detect_events(txt)\n",
    "            \n",
    "            for evt in events:\n",
    "                combined_results[\"events\"].append({\n",
    "                    \"type\": evt.event_type,\n",
    "                    \"text\": evt.text,\n",
    "                    \"participants\": evt.participants\n",
    "                })\n",
    "            \n",
    "            print(f\"    Found {len(events)} events\")\n",
    "        except Exception as e:\n",
    "            print(f\"    Warning: Error detecting events: {e}\")\n",
    "        \n",
    "        print()  # Blank line between chunks\n",
    "        \n",
    "    except Exception as e:\n",
    "        print(f\"  Warning: Error processing chunk: {e}\\n\")\n",
    "        continue\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extraction Process Overview\n",
    "\n",
    "The extraction loop processes each chunk through four steps:\n",
    "\n",
    "1. **Named Entity Recognition**: Identifies entities in the text\n",
    "2. **Relation Extraction**: Finds relationships between entities (requires entities from step 1)\n",
    "3. **Triplet Extraction**: Extracts RDF-style triplets\n",
    "4. **Event Detection**: Identifies temporal events and participants\n",
    "\n",
    "All extraction results are stored in the `combined_results` dictionary.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the Extraction Results\n",
    "\n",
    "Each extraction method produces different types of structured data:\n",
    "\n",
    "- **Entities**: Individual concepts, people, places, products\n",
    "- **Relationships**: Connections between entities (e.g., \"Retinol COMPLEMENTS Niacinamide\")\n",
    "- **Triplets**: RDF-style facts (subject-predicate-object)\n",
    "- **Events**: Temporal occurrences with participants\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extraction Summary\n",
    "\n",
    "Review the extraction results and statistics.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============================================================\n",
      "Phase 3 Complete - Extraction Summary\n",
      "============================================================\n",
      "Entities extracted: 261\n",
      "Relationships extracted: 35\n",
      "Triplets extracted: 305\n",
      "Events detected: 7\n",
      "============================================================\n",
      "\n",
      "Sample entities:\n",
      "  - Makeup and Beauty Blog (ORG)\n",
      "  - Monday Poll (EVENT)\n",
      "  - December 15, 2025 (DATE)\n",
      "  - Jo Malone (ORG)\n",
      "  - Laneige (ORG)\n",
      "\n",
      "Sample relationships:\n",
      "  - Entity(text='Makeup and Beauty Blog', label='ORG', start_char=0, end_char=20, confidence=1.0, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'}) --[hosts]--> Entity(text='Monday Poll', label='EVENT', start_char=20, end_char=32, confidence=1.0, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'})\n",
      "  - Entity(text='Monday Poll', label='EVENT', start_char=20, end_char=32, confidence=1.0, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'}) --[occurs on]--> Entity(text='December 15, 2025', label='DATE', start_char=53, end_char=71, confidence=1.0, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'})\n",
      "  - Entity(text='Makeup and Beauty Blog', label='ORG', start_char=187, end_char=207, confidence=1, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'}) --[publishes]--> Entity(text='Makeup and Beauty Blog Monday Poll, Vol. 893', label='EVENT', start_char=208, end_char=238, confidence=1, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'})\n",
      "  - Entity(text='Makeup and Beauty Blog', label='ORG', start_char=187, end_char=207, confidence=1, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'}) --[has]--> Entity(text='Monday', label='DATE', start_char=35, end_char=43, confidence=1, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'})\n",
      "  - Entity(text='Makeup and Beauty Blog', label='ORG', start_char=187, end_char=207, confidence=1, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'}) --[has]--> Entity(text='2007', label='DATE', start_char=57, end_char=65, confidence=1, metadata={'provider': 'groq', 'model': 'llama-3.1-8b-instant', 'extraction_method': 'llm'})\n"
     ]
    }
   ],
   "source": [
    "# Display extraction summary\n",
    "print(\"=\" * 60)\n",
    "print(\"Phase 3 Complete - Extraction Summary\")\n",
    "print(\"=\" * 60)\n",
    "print(f\"Entities extracted: {len(combined_results['entities'])}\")\n",
    "print(f\"Relationships extracted: {len(combined_results['relationships'])}\")\n",
    "print(f\"Triplets extracted: {len(combined_results['triplets'])}\")\n",
    "print(f\"Events detected: {len(combined_results['events'])}\")\n",
    "print(\"=\" * 60)\n",
    "\n",
    "# Show sample entities\n",
    "if combined_results['entities']:\n",
    "    print(\"\\nSample entities:\")\n",
    "    for entity in combined_results['entities'][:5]:\n",
    "        print(f\"  - {entity['name']} ({entity['type']})\")\n",
    "\n",
    "# Show sample relationships\n",
    "if combined_results['relationships']:\n",
    "    print(\"\\nSample relationships:\")\n",
    "    for rel in combined_results['relationships'][:5]:\n",
    "        print(f\"  - {rel['source']} --[{rel['type']}]--> {rel['target']}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phase 4: Knowledge Graph Construction & Refinement\n",
    "\n",
    "Build the knowledge graph from extracted entities and relationships, then refine it through entity resolution and analysis.\n",
    "\n",
    "### Graph Construction Process\n",
    "\n",
    "1. Graph building from entities and relationships\n",
    "2. Entity resolution to deduplicate and merge similar entities\n",
    "3. Graph analysis of structure and properties\n",
    "4. Centrality calculation to identify important entities\n",
    "5. Community detection to find clusters of related entities\n",
    "\n",
    "Entity resolution merges duplicate entities (e.g., \"Retinol\" and \"retinol\") to improve graph quality.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Initial Knowledge Graph\n",
    "\n",
    "Use GraphBuilder to create the initial graph structure from extracted entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building knowledge graph...\n",
      "Initial graph statistics:\n",
      "  - Entities: 20\n",
      "  - Relationships: 35\n",
      "  - Metadata: {'num_entities': 20, 'num_relationships': 35, 'temporal_enabled': False, 'timestamp': '2025-12-24T12:46:41.535755', 'entity_resolution_applied': True}\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "# Initialize GraphBuilder with entity merging enabled\n",
    "gb = GraphBuilder(merge_entities=True)\n",
    "\n",
    "# Build knowledge graph from extraction results\n",
    "print(\"Building knowledge graph...\")\n",
    "kg = gb.build(sources=[combined_results])\n",
    "\n",
    "print(f\"Initial graph statistics:\")\n",
    "print(f\"  - Entities: {len(kg.get('entities', []))}\")\n",
    "print(f\"  - Relationships: {len(kg.get('relationships', []))}\")\n",
    "print(f\"  - Metadata: {kg.get('metadata', {})}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resolve Entities (Deduplication)\n",
    "\n",
    "Entity resolution merges duplicate entities using semantic similarity. This is crucial for:\n",
    "- **Consistency**: Ensure each real-world entity appears only once\n",
    "- **Accuracy**: Improve graph quality by removing duplicates\n",
    "- **Information Aggregation**: Combine properties from multiple mentions\n",
    "- **Quality**: Create a cleaner, more accurate graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Resolving entities (deduplication)...\n",
      "  Method: Semantic similarity matching\n",
      "  Threshold: 0.85 (85% similarity)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Entity resolution complete:\n",
      "  - Original entities: 20\n",
      "  - Resolved entities: 20\n",
      "  - Entities merged: 0\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import EntityResolver\n",
    "\n",
    "# Initialize EntityResolver\n",
    "# similarity_threshold: Minimum similarity (0.85 = 85%) to consider entities as duplicates\n",
    "resolver = EntityResolver(similarity_threshold=0.85)\n",
    "\n",
    "print(\"Resolving entities (deduplication)...\")\n",
    "print(f\"  Method: Semantic similarity matching\")\n",
    "print(f\"  Threshold: 0.85 (85% similarity)\")\n",
    "\n",
    "# Resolve entities using semantic method\n",
    "resolved_entities = resolver.resolve_entities(\n",
    "    kg.get('entities', []), \n",
    ")\n",
    "\n",
    "# Create final graph with resolved entities\n",
    "kg_final = {\n",
    "    **kg,\n",
    "    'entities': resolved_entities\n",
    "}\n",
    "\n",
    "print(f\"\\nEntity resolution complete:\")\n",
    "print(f\"  - Original entities: {len(kg.get('entities', []))}\")\n",
    "print(f\"  - Resolved entities: {len(kg_final['entities'])}\")\n",
    "print(f\"  - Entities merged: {len(kg.get('entities', [])) - len(kg_final['entities'])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyze Graph Structure\n",
    "\n",
    "Graph analysis provides insights into the graph's topology and connectivity.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Analyzing graph structure...\n",
      "\n",
      "Graph structure metrics:\n",
      "  - Graph density: 0.0900\n",
      "  - Connected components: 0\n",
      "  - Average degree: 2.20\n",
      "  - Total nodes: 25\n",
      "  - Total edges: 27\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import GraphAnalyzer\n",
    "\n",
    "# Initialize GraphAnalyzer\n",
    "analyzer = GraphAnalyzer()\n",
    "\n",
    "print(\"Analyzing graph structure...\")\n",
    "\n",
    "# Perform comprehensive graph analysis\n",
    "analysis = analyzer.analyze_graph(kg_final)\n",
    "\n",
    "# Extract metrics from nested structure\n",
    "metrics = analysis.get('metrics', {})\n",
    "connectivity = analysis.get('connectivity', {})\n",
    "\n",
    "print(f\"\\nGraph structure metrics:\")\n",
    "print(f\"  - Graph density: {metrics.get('density', 0):.4f}\")\n",
    "print(f\"  - Connected components: {connectivity.get('connected_components', 0)}\")\n",
    "print(f\"  - Average degree: {metrics.get('avg_degree', 0):.2f}\")\n",
    "print(f\"  - Total nodes: {metrics.get('num_nodes', 0)}\")\n",
    "print(f\"  - Total edges: {metrics.get('num_edges', 0)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate Centrality Measures\n",
    "\n",
    "Centrality measures identify the most important entities in the graph:\n",
    "- **Degree Centrality**: Entities with many connections\n",
    "- **Betweenness Centrality**: Entities that bridge different parts of the graph\n",
    "- **Closeness Centrality**: Entities that are close to all other entities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating centrality measures...\n",
      "\n",
      "Top 5 entities by degree centrality:\n",
      "  1. Makeup and Beauty Blog: 0.4167\n",
      "  2. Green Tea Antioxidant Skin Therapy: 0.2083\n",
      "  3. Dr. Bailey: 0.1667\n",
      "  4. Calming Zinc Soap: 0.1667\n",
      "  5. Rosacea Therapy Skin Care Kit: 0.1250\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import CentralityCalculator\n",
    "\n",
    "# Initialize CentralityCalculator\n",
    "centrality_calc = CentralityCalculator()\n",
    "\n",
    "print(\"Calculating centrality measures...\")\n",
    "\n",
    "# Calculate degree centrality (simplest and most common)\n",
    "# This returns a dictionary containing 'centrality', 'rankings', etc.\n",
    "result = centrality_calc.calculate_degree_centrality(kg_final)\n",
    "\n",
    "if result and \"rankings\" in result:\n",
    "    # Use the pre-computed rankings (highest scores first)\n",
    "    top_entities = result[\"rankings\"][:5]\n",
    "\n",
    "    print(f\"\\nTop 5 entities by degree centrality:\")\n",
    "    for rank, item in enumerate(top_entities, 1):\n",
    "        # item['node'] is the entity ID, item['score'] is the normalized score\n",
    "        print(f\"  {rank}. {item['node']}: {item['score']:.4f}\")\n",
    "else:\n",
    "    print(\"  No centrality data available\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect Communities\n",
    "\n",
    "Community detection identifies clusters of closely connected entities, revealing:\n",
    "- **Thematic Groups**: Entities that belong to the same topic\n",
    "- **Subnetworks**: Dense clusters within the graph\n",
    "- **Domain Structure**: How knowledge is organized\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detecting communities...\n",
      "  Method: Louvain algorithm (greedy modularity optimization)\n",
      "\n",
      "Community detection results:\n",
      "  - Total communities found: 4\n",
      "  - Community 1: 10 entities\n",
      "    Sample entities: 2007, Makeup and Beauty Blog Monday Poll, Monday Poll\n",
      "  - Community 2: 8 entities\n",
      "    Sample entities: Dr. Bailey, rosacea, Facial dandruff\n",
      "  - Community 3: 5 entities\n",
      "    Sample entities: Omega Enriched Face Booster Oil, sea buckthorn, Hydrate\n"
     ]
    }
   ],
   "source": [
    "# Initialize CommunityDetector\n",
    "from semantica.kg import CommunityDetector\n",
    "community_detector = CommunityDetector()\n",
    "\n",
    "print(\"Detecting communities...\")\n",
    "print(\"  Method: Louvain algorithm (greedy modularity optimization)\")\n",
    "\n",
    "# Detect communities using Louvain algorithm\n",
    "result = community_detector.detect_communities(\n",
    "    kg_final, \n",
    "    method=\"louvain\"\n",
    ")\n",
    "\n",
    "if result and \"communities\" in result:\n",
    "    communities = result[\"communities\"]\n",
    "    print(f\"\\nCommunity detection results:\")\n",
    "    print(f\"  - Total communities found: {len(communities)}\")\n",
    "    \n",
    "    # Show top 3 communities\n",
    "    sorted_communities = sorted(communities, key=len, reverse=True)\n",
    "    for i, community in enumerate(sorted_communities[:3], 1):\n",
    "        print(f\"  - Community {i}: {len(community)} entities\")\n",
    "        # Show sample entities from this community\n",
    "        sample_entities = list(community)[:3]\n",
    "        print(f\"    Sample entities: {', '.join(sample_entities)}\")\n",
    "else:\n",
    "    print(\"  No communities detected\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phase 4 Summary\n",
    "\n",
    "Review the final graph statistics and quality metrics.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============================================================\n",
      "Phase 4 Complete - Knowledge Graph Summary\n",
      "============================================================\n",
      "Final graph contains:\n",
      "  - Entities: 20\n",
      "  - Relationships: 35\n",
      "  - Graph density: 0.0900\n",
      "  - Connected components: 0\n",
      "  - Communities: 4\n",
      "============================================================\n"
     ]
    }
   ],
   "source": [
    "print(\"=\" * 60)\n",
    "print(\"Phase 4 Complete - Knowledge Graph Summary\")\n",
    "print(\"=\" * 60)\n",
    "# Extract metrics from nested structure (if not already extracted)\n",
    "metrics = analysis.get('metrics', {})\n",
    "connectivity = analysis.get('connectivity', {})\n",
    "\n",
    "print(f\"Final graph contains:\")\n",
    "print(f\"  - Entities: {len(kg_final['entities'])}\")\n",
    "print(f\"  - Relationships: {len(kg_final.get('relationships', []))}\")\n",
    "print(f\"  - Graph density: {metrics.get('density', 0):.4f}\")\n",
    "print(f\"  - Connected components: {connectivity.get('connected_components', 0)}\")\n",
    "print(f\"  - Communities: {len(communities) if communities else 0}\")\n",
    "print(\"=\" * 60)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phase 5: Vector Store Population\n",
    "\n",
    "Generate embeddings for all document chunks and store them in the vector database for hybrid retrieval.\n",
    "\n",
    "### Why Vector Store?\n",
    "\n",
    "- **Semantic Search**: Find text chunks by meaning, not just keywords\n",
    "- **Hybrid Retrieval**: Combine with graph traversal for comprehensive results\n",
    "- **Fast Lookup**: Efficient similarity search using FAISS\n",
    "\n",
    "### Embedding Process\n",
    "\n",
    "1. Generate embeddings for all chunks using the configured embedding model\n",
    "2. Store vectors with metadata linking back to original chunks\n",
    "3. Enable fast similarity search for retrieval\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing texts for embedding...\n",
      "  - Total chunks to embed: 76\n",
      "  - Embedding model: all-MiniLM-L6-v2\n",
      "  - Expected dimension: 384\n"
     ]
    }
   ],
   "source": [
    "# Prepare texts for embedding generation\n",
    "print(\"Preparing texts for embedding...\")\n",
    "texts = [str(c.text) for c in all_chunks]\n",
    "\n",
    "print(f\"  - Total chunks to embed: {len(texts)}\")\n",
    "print(f\"  - Embedding model: all-MiniLM-L6-v2\")\n",
    "print(f\"  - Expected dimension: 384\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating embeddings...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "fastembed not available. Install with: pip install fastembed. Using fallback embedding method.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embeddings generated successfully:\n",
      "  - Total embeddings: 76\n",
      "  - Embedding dimension: 128\n"
     ]
    }
   ],
   "source": [
    "# Generate embeddings using the configured embedding generator\n",
    "print(\"Generating embeddings...\")\n",
    "embeddings = core.embedding_generator.generate_embeddings(texts)\n",
    "\n",
    "print(f\"Embeddings generated successfully:\")\n",
    "print(f\"  - Total embeddings: {len(embeddings)}\")\n",
    "print(f\"  - Embedding dimension: {embeddings.shape[1] if len(embeddings) > 0 else 0}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Metadata\n",
    "\n",
    "Create metadata for each vector that links it back to the original chunk and provides context.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing metadata...\n",
      "  - Metadata entries created: 76\n"
     ]
    }
   ],
   "source": [
    "# Create metadata for each vector\n",
    "metadata_list = []\n",
    "\n",
    "print(\"Preparing metadata...\")\n",
    "for i, chunk in enumerate(all_chunks):\n",
    "    metadata_list.append({\n",
    "        \"text\": str(chunk.text),\n",
    "        \"chunk_id\": i,\n",
    "        \"source\": \"multi_source_ingestion\",\n",
    "        \"chunk_length\": len(str(chunk.text))\n",
    "    })\n",
    "\n",
    "print(f\"  - Metadata entries created: {len(metadata_list)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store Vectors\n",
    "\n",
    "Store all embeddings in the FAISS vector store for fast similarity search.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Storing vectors in vector store...\n",
      "\n",
      "Phase 5 Complete. Vector store populated successfully.\n",
      "  - Vectors stored: 76\n",
      "  - Vector store backend: FAISS\n",
      "  - Embedding dimension: 128\n",
      "  - Ready for similarity search\n"
     ]
    }
   ],
   "source": [
    "# Store vectors in FAISS vector store\n",
    "print(\"Storing vectors in vector store...\")\n",
    "vs.store_vectors(vectors=embeddings, metadata=metadata_list)\n",
    "\n",
    "print(f\"\\nPhase 5 Complete. Vector store populated successfully.\")\n",
    "print(f\"  - Vectors stored: {len(embeddings)}\")\n",
    "print(f\"  - Vector store backend: FAISS\")\n",
    "print(f\"  - Embedding dimension: {embeddings.shape[1] if len(embeddings) > 0 else 0}\")\n",
    "print(f\"  - Ready for similarity search\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phase 6: GraphRAG Query System\n",
    "\n",
    "Implement hybrid retrieval that combines vector search with graph traversal.\n",
    "\n",
    "### Hybrid Retrieval\n",
    "\n",
    "- **Vector Search**: Finds semantically similar text chunks\n",
    "- **Graph Traversal**: Follows entity relationships to find connected information\n",
    "\n",
    "### AgentContext Features\n",
    "\n",
    "- Auto-detection of RAG vs GraphRAG based on available components\n",
    "- Graph expansion with configurable hop limits\n",
    "- Hybrid scoring combining vector and graph scores\n",
    "- Entity linking to connect query terms to graph entities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import context and provider modules\n",
    "from semantica.context import AgentContext, EntityLinker\n",
    "from semantica.semantic_extract.providers import create_provider\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize AgentContext\n",
    "\n",
    "AgentContext is the main interface for GraphRAG. It orchestrates:\n",
    "- Vector store queries\n",
    "- Knowledge graph traversal\n",
    "- Hybrid scoring and ranking\n",
    "- Context aggregation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing AgentContext for GraphRAG...\n",
      "AgentContext initialized successfully.\n",
      "  - Graph expansion: Enabled\n",
      "  - Max expansion hops: 2\n",
      "  - Hybrid alpha: 0.6 (60% graph, 40% vector)\n"
     ]
    }
   ],
   "source": [
    "from semantica.context import AgentContext\n",
    "\n",
    "# Initialize AgentContext with hybrid retrieval enabled\n",
    "print(\"Initializing AgentContext for GraphRAG...\")\n",
    "\n",
    "ctx = AgentContext(\n",
    "    vector_store=vs,                    # Vector store for semantic search\n",
    "    knowledge_graph=kg_final,           # Knowledge graph for relationship traversal\n",
    "    use_graph_expansion=True,           # Enable graph traversal\n",
    "    max_expansion_hops=2,               # Traverse up to 2 hops from initial entities\n",
    "    hybrid_alpha=0.6                    # 60% weight on graph, 40% on vector\n",
    ")\n",
    "\n",
    "print(\"AgentContext initialized successfully.\")\n",
    "print(f\"  - Graph expansion: Enabled\")\n",
    "print(f\"  - Max expansion hops: 2\")\n",
    "print(f\"  - Hybrid alpha: 0.6 (60% graph, 40% vector)\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize LLM Provider\n",
    "\n",
    "Set up the LLM provider for generating final answers from retrieved context.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LLM provider initialized.\n",
      "  - Provider: Groq\n",
      "  - Model: llama-3.3-70b-versatile\n",
      "  - Ready for answer generation\n"
     ]
    }
   ],
   "source": [
    "# Initialize LLM provider for answer generation\n",
    "llm_provider = create_provider(\"groq\", model=\"llama-3.3-70b-versatile\")\n",
    "\n",
    "print(\"LLM provider initialized.\")\n",
    "print(f\"  - Provider: Groq\")\n",
    "print(f\"  - Model: llama-3.3-70b-versatile\")\n",
    "print(f\"  - Ready for answer generation\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Entity Linker\n",
    "\n",
    "EntityLinker resolves text mentions to entities in the knowledge graph, enabling better query understanding.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EntityLinker initialized.\n",
      "  - Knowledge graph: Linked\n",
      "  - Ready for entity resolution in queries\n"
     ]
    }
   ],
   "source": [
    "# Initialize EntityLinker\n",
    "linker = EntityLinker(knowledge_graph=kg_final)\n",
    "\n",
    "print(\"EntityLinker initialized.\")\n",
    "print(f\"  - Knowledge graph: Linked\")\n",
    "print(f\"  - Ready for entity resolution in queries\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Query Example\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================================================\n",
      "GRAPH RAG QUERY PROCESSING\n",
      "======================================================================\n",
      "Query: What ingredients synergize with Retinol to prevent irritation?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Define the user query\n",
    "user_query = \"What ingredients synergize with Retinol to prevent irritation?\"\n",
    "\n",
    "print(\"=\" * 70)\n",
    "print(\"GRAPH RAG QUERY PROCESSING\")\n",
    "print(\"=\" * 70)\n",
    "print(f\"Query: {user_query}\\n\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve Context Using Hybrid Search\n",
    "\n",
    "Use AgentContext to retrieve relevant context using both vector search and graph traversal.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieving context using hybrid search...\n",
      "  - Vector search: Finding semantically similar chunks\n",
      "  - Graph traversal: Following entity relationships\n",
      "  - Max results: 5\n",
      "  - Graph expansion: Enabled (2 hops)\n",
      "\n",
      "Retrieved 1 context results.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Retrieve context using hybrid retrieval\n",
    "print(\"Retrieving context using hybrid search...\")\n",
    "print(\"  - Vector search: Finding semantically similar chunks\")\n",
    "print(\"  - Graph traversal: Following entity relationships\")\n",
    "print(\"  - Max results: 5\")\n",
    "print(\"  - Graph expansion: Enabled (2 hops)\\n\")\n",
    "\n",
    "context_results = ctx.retrieve(\n",
    "    user_query, \n",
    "    max_results=5,\n",
    "    use_graph=True,              # Enable graph-based retrieval\n",
    "    expand_graph=True,           # Expand to related entities\n",
    "    include_entities=True,       # Include related entities in results\n",
    "    include_relationships=True   # Include relationships in results\n",
    ")\n",
    "\n",
    "print(f\"Retrieved {len(context_results)} context results.\\n\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display Retrieved Context\n",
    "\n",
    "Review the retrieved context and multi-hop connections discovered by GraphRAG.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================================================\n",
      "MULTI-HOP CONTEXT DISCOVERED\n",
      "======================================================================\n",
      "\n",
      "Result 1:\n",
      "  Content: ...\n",
      "  Relevance Score: 0.7722\n",
      "\n",
      "======================================================================\n"
     ]
    }
   ],
   "source": [
    "if context_results:\n",
    "    print(\"=\" * 70)\n",
    "    print(\"MULTI-HOP CONTEXT DISCOVERED\")\n",
    "    print(\"=\" * 70)\n",
    "    \n",
    "    for i, res in enumerate(context_results, 1):\n",
    "        print(f\"\\nResult {i}:\")\n",
    "        print(f\"  Content: {res.get('content', '')[:150]}...\")\n",
    "        print(f\"  Relevance Score: {res.get('score', 0):.4f}\")\n",
    "        \n",
    "        # Display related entities (multi-hop connections)\n",
    "        if res.get('related_entities'):\n",
    "            print(f\"  Related Entities ({len(res['related_entities'])}):\")\n",
    "            for entity in res['related_entities'][:3]:\n",
    "                entity_name = entity.get('name', entity.get('content', 'Unknown'))\n",
    "                print(f\"    - {entity_name}\")\n",
    "        \n",
    "        # Display related relationships\n",
    "        if res.get('related_relationships'):\n",
    "            print(f\"  Related Relationships: {len(res['related_relationships'])}\")\n",
    "    \n",
    "    print(\"\\n\" + \"=\" * 70)\n",
    "else:\n",
    "    print(\"No relevant context found.\")\n",
    "    print(\"This may indicate:\")\n",
    "    print(\"  - The knowledge graph doesn't contain relevant information\")\n",
    "    print(\"  - Try a different query\")\n",
    "    print(\"  - Check if entities in the query exist in the graph\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate Final Answer\n",
    "\n",
    "Use the LLM to synthesize a comprehensive answer from the retrieved context.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================================================\n",
      "GENERATING FINAL ANSWER\n",
      "======================================================================\n",
      "Using LLM to synthesize answer from retrieved context...\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To prevent irritation when using Retinol, it's essential to combine it with ingredients that synergize well and help mitigate potential side effects such as redness, dryness, and sensitivity. Here are some key ingredients known to work harmoniously with Retinol:\n",
      "\n",
      "1. **Hyaluronic Acid**: This natural humectant is renowned for its ability to retain moisture in the skin. When used alongside Retinol, Hyaluronic Acid helps to counterbalance the drying effects that Retinol can sometimes cause, ensuring the skin remains hydrated and comfortable.\n",
      "\n",
      "2. **Niacinamide**: A form of Vitamin B3, Niacinamide is celebrated for its skin-restoring abilities. It can improve the skin's barrier function, reduce inflammation, and enhance the skin's natural ceramide production. When paired with Retinol, Niacinamide can help reduce the risk of irritation and support the skin's overall health and resilience.\n",
      "\n",
      "3. **Ceramides**: These are vital components of the skin's barrier, helping to lock in moisture and protect the skin from environmental stressors. Combining Ceramides with Retinol can help repair and restore the skin's barrier, minimizing the potential for irritation and dryness associated with Retinol use.\n",
      "\n",
      "4. **Glycerin**: Similar to Hyaluronic Acid, Glycerin is a humectant that attracts and retains moisture in the skin. Its inclusion in a Retinol regimen can help maintain the skin's moisture balance, reducing the likelihood of dryness and irritation.\n",
      "\n",
      "5. **Squalane**: This naturally occurring lipid is highly compatible with the skin and serves as an excellent moisturizer. Squalane can help protect the skin from water loss, reducing the potential for Retinol-induced dryness and irritation, and leaving the skin feeling soft and supple.\n",
      "\n",
      "6. **Peptides**: Short chains of amino acids, Peptides can stimulate collagen production and improve skin elasticity. When used in conjunction with Retinol, Peptides may help in reducing the appearance of fine lines and wrinkles while supporting the skin's natural repair processes, potentially minimizing irritation.\n",
      "\n",
      "7. **Vitamin E**: Known for its antioxidant properties, Vitamin E can help protect the skin from oxidative stress and damage caused by free radicals. Combining Vitamin E with Retinol may enhance the skin's tolerance to Retinol by reducing oxidative stress, thereby potentially lowering the risk of irritation.\n",
      "\n",
      "8. **Aloe Vera**: Aloe Vera is widely recognized for its soothing and calming effects on the skin. It can help mitigate irritation and redness, making it a beneficial ingredient to use alongside Retinol, especially for sensitive skin types.\n",
      "\n",
      "When incorporating these ingredients into your skincare routine with Retinol, it's crucial to follow a few guidelines:\n",
      "- Start with a lower concentration of Retinol and gradually increase as your skin becomes more tolerant.\n",
      "- Apply moisturizing products containing these synergistic ingredients generously to help buffer the skin against potential irritation.\n",
      "- Be patient, as it may take some time for your skin to adjust to Retinol, even with the use of soothing ingredients.\n",
      "\n",
      "Remember, everyone's skin is different, so it's essential to monitor your skin's response to Retinol and adjust your skincare regimen accordingly. If irritation persists, consider consulting with a dermatologist for personalized advice.\n"
     ]
    }
   ],
   "source": [
    "if context_results:\n",
    "    # Combine all retrieved context\n",
    "    context_text = \"\\n\\n\".join([r.get('content', '') for r in context_results])\n",
    "    \n",
    "    # Create prompt for LLM\n",
    "    prompt = f\"\"\"Based on the following context from a knowledge graph, answer the user query accurately and comprehensively.\n",
    "\n",
    "Context:\n",
    "{context_text}\n",
    "\n",
    "Query: {user_query}\n",
    "\n",
    "Provide a detailed answer based on the context above:\"\"\"\n",
    "    \n",
    "    print(\"=\" * 70)\n",
    "    print(\"GENERATING FINAL ANSWER\")\n",
    "    print(\"=\" * 70)\n",
    "    print(\"Using LLM to synthesize answer from retrieved context...\\n\")\n",
    "    \n",
    "    try:\n",
    "        final_answer = llm_provider.generate(prompt, temperature=0.3)\n",
    "        print(final_answer)\n",
    "    except Exception as e:\n",
    "        print(f\"Warning: LLM generation failed: {e}\")\n",
    "        print(\"\\nHowever, we successfully retrieved relevant context using GraphRAG!\")\n",
    "        print(\"The context above can be used to answer the query manually.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phase 7: Visualization & Export\n",
    "\n",
    "Visualize the knowledge graph and export it in various formats for analysis and sharing.\n",
    "\n",
    "### Visualization\n",
    "\n",
    "- Interactive network visualization with multiple layout algorithms (spring, circular, hierarchical)\n",
    "\n",
    "### Export Formats\n",
    "\n",
    "- **JSON**: Human-readable, easy to process\n",
    "- **GraphML**: Standard graph format, compatible with many tools\n",
    "- **RDF**: Semantic web format, supports SPARQL queries\n",
    "- **CSV**: Tabular format for analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize Knowledge Graph\n",
    "\n",
    "Create a visual representation of the knowledge graph showing entities and their relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Visualizing knowledge graph...\n",
      "  - Layout: Spring (force-directed)\n",
      "  - Title: GraphRAG Knowledge Graph\n",
      "Graph visualization complete.\n"
     ]
    }
   ],
   "source": [
    "from semantica.visualization import KGVisualizer\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Initialize KGVisualizer\n",
    "viz = KGVisualizer()\n",
    "\n",
    "print(\"Visualizing knowledge graph...\")\n",
    "print(\"  - Layout: Spring (force-directed)\")\n",
    "print(\"  - Title: GraphRAG Knowledge Graph\")\n",
    "\n",
    "try:\n",
    "    viz.visualize_network(\n",
    "        kg_final, \n",
    "        layout=\"spring\", \n",
    "        title=\"GraphRAG Knowledge Graph\",\n",
    "        output=\"static\"\n",
    "    )\n",
    "    plt.show()\n",
    "    print(\"Graph visualization complete.\")\n",
    "except Exception as e:\n",
    "    print(f\"Warning: Visualization error: {e}\")\n",
    "    print(\"Graph structure:\")\n",
    "    print(f\"  - Entities: {len(kg_final.get('entities', []))}\")\n",
    "    print(f\"  - Relationships: {len(kg_final.get('relationships', []))}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Export Knowledge Graph\n",
    "\n",
    "Export the graph in multiple formats for different use cases.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Exporting knowledge graph...\n",
      "  Exported to JSON: graphrag_kg.json\n",
      "  Exported to GraphML: graphrag_kg.graphml\n",
      "\n",
      "Phase 7 Complete. Graph visualized and exported.\n"
     ]
    }
   ],
   "source": [
    "from semantica.export import GraphExporter\n",
    "\n",
    "# Initialize GraphExporter\n",
    "exporter = GraphExporter()\n",
    "\n",
    "print(\"\\nExporting knowledge graph...\")\n",
    "\n",
    "# Export to JSON (human-readable, easy to process)\n",
    "try:\n",
    "    exporter.export_knowledge_graph(kg_final, \"graphrag_kg.json\", format=\"json\")\n",
    "    print(\"  Exported to JSON: graphrag_kg.json\")\n",
    "except Exception as e:\n",
    "    print(f\"  Warning: JSON export error: {e}\")\n",
    "\n",
    "# Export to GraphML (standard graph format)\n",
    "try:\n",
    "    exporter.export_knowledge_graph(kg_final, \"graphrag_kg.graphml\", format=\"graphml\")\n",
    "    print(\"  Exported to GraphML: graphrag_kg.graphml\")\n",
    "except Exception as e:\n",
    "    print(f\"  Warning: GraphML export error: {e}\")\n",
    "\n",
    "print(\"\\nPhase 7 Complete. Graph visualized and exported.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "This notebook demonstrated a complete GraphRAG pipeline from data ingestion to question answering.\n",
    "\n",
    "### Pipeline Overview\n",
    "\n",
    "The complete pipeline consisted of 7 phases:\n",
    "\n",
    "1. **Phase 0**: Setup & Foundation Seeding\n",
    "   - Configured Semantica with all components\n",
    "   - Seeded knowledge graph with ground truth data\n",
    "\n",
    "2. **Phase 1**: Multi-Source Ingestion\n",
    "   - Ingested from RSS feeds, web pages, and local files\n",
    "   - Normalized all content for processing\n",
    "\n",
    "3. **Phase 2**: Document Processing & Chunking\n",
    "   - Split documents into semantic chunks\n",
    "   - Preserved entity boundaries\n",
    "\n",
    "4. **Phase 3**: Comprehensive Semantic Extraction\n",
    "   - Extracted entities using NER\n",
    "   - Extracted relationships between entities\n",
    "   - Extracted RDF triplets\n",
    "   - Detected events and their participants\n",
    "\n",
    "5. **Phase 4**: Knowledge Graph Construction & Refinement\n",
    "   - Built graph from extracted data\n",
    "   - Resolved duplicate entities\n",
    "   - Analyzed graph structure\n",
    "   - Calculated centrality measures\n",
    "   - Detected communities\n",
    "\n",
    "6. **Phase 5**: Vector Store Population\n",
    "   - Generated embeddings for all chunks\n",
    "   - Stored vectors with metadata\n",
    "\n",
    "7. **Phase 6**: GraphRAG Query System\n",
    "   - Implemented hybrid retrieval\n",
    "   - Demonstrated multi-hop reasoning\n",
    "   - Generated answers using LLM\n",
    "\n",
    "8. **Phase 7**: Visualization & Export\n",
    "   - Visualized knowledge graph\n",
    "   - Exported in multiple formats\n",
    "\n",
    "### Key Takeaways\n",
    "\n",
    "- **GraphRAG** combines the best of vector search and graph traversal\n",
    "- **Multi-hop reasoning** allows answering complex questions that require connecting multiple pieces of information\n",
    "- **Hybrid retrieval** provides more contextually accurate results than vector-only search\n",
    "- **Semantica framework** provides all the tools needed for production-ready GraphRAG systems\n",
    "\n",
    "### Next Steps\n",
    "\n",
    "- Experiment with different queries to explore the knowledge graph\n",
    "- Add more data sources to expand the knowledge base\n",
    "- Fine-tune extraction parameters for your specific domain\n",
    "- Explore advanced features like temporal reasoning and conflict resolution\n",
    "- Integrate with persistent graph stores (Neo4j, ArangoDB) for production use\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
