diff --git a/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb index 6fc16e91..5fee8de6 100644 --- a/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb +++ b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb @@ -4,551 +4,78 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)\n", - "\n", - "# GraphRAG Complete - End-to-End Pipeline\n", + "# šŸš€ Enterprise GraphRAG: End-to-End with Semantica\n", "\n", "## Overview\n", "\n", - "This notebook demonstrates a **complete end-to-end GraphRAG (Graph-based Retrieval Augmented Generation) system** using Semantica framework. It showcases how to build a production-ready GraphRAG system that combines vector search with knowledge graph traversal for enhanced retrieval and question answering.\n", + "This notebook demonstrates a **production-grade GraphRAG pipeline** using the `Semantica` framework. We go beyond simple tutorials to show how to build, analyze, and query a rich knowledge base using advanced graph algorithms and hybrid retrieval strategies.\n", "\n", - "**Key Features:**\n", + "### šŸŽÆ What You Will Learn\n", "\n", - "- **Real-World Data**: Uses actual data sources via Local Files, GitHub Repositories, Web Scraping, and RSS Feeds (NO mock data)\n", - "- **Complete Pipeline**: From data ingestion to LLM-powered question answering\n", - "- **Hybrid Retrieval**: Combines vector similarity search with knowledge graph traversal\n", - "- **Multi-hop Reasoning**: Follows relationships across the graph for deeper context\n", + "1. **Orchestration**: Using `Semantica` core to manage the full data lifecycle.\n", + "2. **Advanced Analysis**: Using `GraphAnalyzer`, `CentralityCalculator`, and `CommunityDetector` to understand data structure.\n", + "3. **Ontology Generation**: Automatically deriving a schema (classes & properties) from unstructured text.\n", + "4. **Hybrid Search**: Combining Vector Similarity with Graph Traversal for superior context.\n", "\n", - "**Documentation**: [API Reference](https://semantica.readthedocs.io/concepts/) • [GraphRAG Guide](https://semantica.readthedocs.io/concepts/)\n", - "\n", - "### What You'll Learn\n", - "\n", - "- How to ingest real-world data from multiple sources (MCP, web, feeds)\n", - "- How to build knowledge graphs from unstructured text\n", - "- How to implement hybrid search combining vectors and graphs\n", - "- How to use ContextRetriever for intelligent context expansion\n", - "- How to integrate LLMs with GraphRAG for question answering\n", - "- How to visualize and export knowledge graphs\n", - "\n", - "### Pipeline Overview\n", - "\n", - "- **Real-World Data**: Uses actual data sources via Local Files, GitHub Repositories, Web Scraping, and RSS Feeds (NO mock data)\n", - "\n", - "---\n", - "\n", - "## Installation\n", - "\n", - "Install Semantica from PyPI:\n", - "\n", - "```bash\n", - "pip install semantica\n", - "\n", - "```\n" + "---" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n" - ] - } - ], + "outputs": [], "source": [ - "!pip install -qU semantica\n" + "# šŸ› ļø Dependencies\n", + "!pip install -qU semantica networkx matplotlib plotly pandas" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 1: Setup and Import Semantica Modules\n", + "## 1. Professional Setup & Imports\n", "\n", - "Import all necessary Semantica modules for the complete GraphRAG pipeline. This includes modules for ingestion, parsing, extraction, graph building, embeddings, vector storage, context retrieval, and more.\n", - "\n" + "We organize imports by their functional role in the Semantica ecosystem." ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "

🧠 Semantica - šŸ“Š Current Progress

StatusActionModuleSubmoduleFileTime
āœ…Semantica is processingā³ coreLifecycleManager-3.34s
āœ…Semantica is ingestingšŸ“„ ingestFileIngestorsample_graphrag_paper.txt0.00s
āœ…Semantica is ingestingšŸ“„ ingestRepoIngestorrequests4.50s
āŒSemantica is ingestingšŸ“„ ingestWebIngestortool-use0.26s
āœ…Semantica is ingestingšŸ“„ ingestWebIngestor#retrieval2.09s
āœ…Semantica is ingestingšŸ“„ ingestFeedIngestorfeed0.54s
āœ…Semantica is ingestingšŸ“„ ingestFeedIngestorrss.xml1.19s
āœ…Semantica is parsingšŸ” parseDocumentParsersample_graphrag_paper.txt0.01s
" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Configuration and framework initialized\n" - ] - } - ], + "outputs": [], "source": [ - "from semantica.core import ConfigManager, Semantica\n", - "\n", - "# Initialize configuration manager\n", - "config_manager = ConfigManager()\n", - "\n", - "# Optionally load from file or dictionary\n", - "# config = config_manager.load_from_file(\"config.yaml\")\n", - "# Or use defaults\n", - "config = config_manager.load_from_dict({})\n", - "\n", - "# Initialize Semantica framework\n", - "framework = Semantica(config=config)\n", - "framework.initialize()\n", - "\n", - "print(\"Configuration and framework initialized\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 2: Ingest Real-World Data from Multiple Sources\n", - "\n", - "1. **Local Files**: Ingest local documents and text files\n", - "2. **Repositories**: Ingest code and documentation from GitHub repositories\n", - "3. **Web Sources**: Scrape real web content from news sites and documentation\n", - "4. **RSS Feeds**: Ingest real RSS/Atom feeds from news sources\n", - "\n", - "### 2.1: Ingest Local Files and Repositories\n", - "\n", - "Ingest data from local file systems and remote code repositories.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Ingesting local file from c:\\Users\\Mohd Kaif\\semantica\\cookbook\\use_cases\\advanced_rag\\data\\sample_graphrag_paper.txt...\n", - " Ingested 1 file document\n", - "\n", - "Ingesting repository https://github.com/psf/requests...\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Error analyzing commits: Cmd('git') failed due to: exit code(128)\n", - " cmdline: git diff --numstat --no-renames --raw 420d16bc7ef326f7b65f90e4644adc0f6a0e1d44 70298332899f25826e35e42f8d83425124f755a5 --\n", - " stderr: 'fatal: bad object 420d16bc7ef326f7b65f90e4644adc0f6a0e1d44'\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - " Ingested 1 repo document\n" - ] - } - ], - "source": [ - "from semantica.ingest import FileIngestor, RepoIngestor, WebIngestor, FeedIngestor\n", "import os\n", + "import json\n", + "import pandas as pd\n", "\n", - "all_documents = []\n", + "# --- Core Framework ---\n", + "from semantica.core import Semantica, ConfigManager\n", "\n", - "# 1. Ingest Local File\n", - "file_ingestor = FileIngestor()\n", - "# Ensure the sample directory exists\n", - "sample_dir = os.path.join(os.getcwd(), \"data\")\n", - "sample_file = os.path.join(sample_dir, \"sample_graphrag_paper.txt\")\n", - "\n", - "print(f\"Ingesting local file from {sample_file}...\")\n", - "if os.path.exists(sample_file):\n", - " file_docs = file_ingestor.ingest_file(sample_file)\n", - " if isinstance(file_docs, list):\n", - " all_documents.extend(file_docs)\n", - " print(f\" Ingested {len(file_docs)} file document(s)\")\n", - " else:\n", - " all_documents.append(file_docs)\n", - " print(f\" Ingested 1 file document\")\n", - "else:\n", - " print(f\" Warning: Sample file not found at {sample_file}. Please ensure it exists.\")\n", - "\n", - "# 2. Ingest GitHub Repository\n", - "repo_ingestor = RepoIngestor()\n", - "repo_url = \"https://github.com/psf/requests\" # Standard library, good example\n", - "print(f\"\\nIngesting repository {repo_url}...\")\n", - "\n", - "try:\n", - " # Using depth=1 and filtering for texts to keep it light\n", - " repo_docs = repo_ingestor.ingest_repository(\n", - " repo_url, \n", - " max_depth=1, \n", - " include_extensions=[\"md\", \"rst\", \"txt\"]\n", - " )\n", - " \n", - " if isinstance(repo_docs, list):\n", - " all_documents.extend(repo_docs)\n", - " print(f\" Ingested {len(repo_docs)} repo document(s)\")\n", - " else:\n", - " all_documents.append(repo_docs)\n", - " print(f\" Ingested 1 repo document\")\n", - "except Exception as e:\n", - " print(f\" Error ingesting repo: {e}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.3: Ingest Data from Web Sources\n", - "\n", - "Scrape real web content from news sites, documentation, and articles.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Scraping https://docs.anthropic.com/en/docs/build-with-claude/tool-use...\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "URL https://docs.anthropic.com/en/docs/build-with-claude/tool-use blocked by robots.txt\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - " Error scraping https://docs.anthropic.com/en/docs/build-with-claude/tool-use: URL blocked by robots.txt: https://docs.anthropic.com/en/docs/build-with-claude/tool-use\n", - "Scraping https://python.langchain.com/docs/concepts/#retrieval...\n", - " Scraped 1 document(s)\n", - "\n", - "Total documents from web: 1\n", - "Total documents so far: 3\n" - ] - } - ], - "source": [ - "web_ingestor = WebIngestor()\n", - "\n", - "# Real documentation and article sources\n", - "web_sources = [\n", - " \"https://docs.anthropic.com/en/docs/build-with-claude/tool-use\", # Relevant to agents\n", - " \"https://python.langchain.com/docs/concepts/#retrieval\" # Relevant to RAG\n", - "]\n", - "\n", - "web_documents = []\n", - "for url in web_sources:\n", - " try:\n", - " print(f\"Scraping {url}...\")\n", - " # Use ingest_url instead of ingest\n", - " docs = web_ingestor.ingest_url(url)\n", - " if isinstance(docs, list):\n", - " web_documents.extend(docs)\n", - " else:\n", - " web_documents.append(docs)\n", - " print(f\" Scraped {len(docs) if isinstance(docs, list) else 1} document(s)\")\n", - " except Exception as e:\n", - " print(f\" Error scraping {url}: {e}\")\n", - "\n", - "all_documents.extend(web_documents)\n", - "print(f\"\\nTotal documents from web: {len(web_documents)}\")\n", - "print(f\"Total documents so far: {len(all_documents)}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.4: Ingest Data from RSS Feeds\n", - "\n", - "Ingest real RSS/Atom feeds from news sources.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Fetching feed https://techcrunch.com/category/artificial-intelligence/feed/...\n", - " Fetched 20 feed item(s)\n", - "Fetching feed http://feeds.bbci.co.uk/news/technology/rss.xml...\n", - " Fetched 48 feed item(s)\n", - "\n", - "Total documents from feeds: 2\n", - "Total documents collected: 5\n" - ] - } - ], - "source": [ - "feed_ingestor = FeedIngestor()\n", - "\n", - "# Real Tech News Feeds\n", - "feed_urls = [\n", - " \"https://techcrunch.com/category/artificial-intelligence/feed/\",\n", - " \"http://feeds.bbci.co.uk/news/technology/rss.xml\"\n", - "]\n", - "\n", - "feed_documents = []\n", - "for feed_url in feed_urls:\n", - " try:\n", - " print(f\"Fetching feed {feed_url}...\")\n", - " # Use ingest_feed instead of ingest\n", - " feeds = feed_ingestor.ingest_feed(feed_url)\n", - " if isinstance(feeds, list):\n", - " feed_documents.extend(feeds)\n", - " else:\n", - " feed_documents.append(feeds)\n", - " # Check for feed items\n", - " count = len(feeds.items) if hasattr(feeds, 'items') else (len(feeds) if isinstance(feeds, list) else 1)\n", - " print(f\" Fetched {count} feed item(s)\")\n", - " except Exception as e:\n", - " print(f\" Error fetching feed {feed_url}: {e}\")\n", - "\n", - "all_documents.extend(feed_documents)\n", - "print(f\"\\nTotal documents from feeds: {len(feed_documents)}\")\n", - "print(f\"Total documents collected: {len(all_documents)}\")\n", - "\n", - "if len(all_documents) == 0:\n", - " print(\"\\nNo documents collected. Please configure MCP servers, web URLs, or RSS feeds above.\")\n", - " print(\"For this demonstration, we'll continue with the pipeline structure.\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 3: Document Processing Pipeline\n", - "\n", - "Process the ingested documents: parse, split, and normalize the text for extraction.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Parsing documents...\n", - "Warning: Unknown document format: \n", - "Parsed 70 documents\n" - ] - } - ], - "source": [ - "from semantica.parse import DocumentParser\n", - "\n", - "document_parser = DocumentParser()\n", - "\n", - "parsed_documents = []\n", - "\n", - "print(\"Parsing documents...\")\n", - "for doc in all_documents:\n", - " try:\n", - " # Handle different document objects\n", - " # 1. FileObject (from FileIngestor) -> Parse using path\n", - " if hasattr(doc, 'path') and os.path.exists(doc.path):\n", - " parsed = document_parser.parse(doc.path)\n", - " if isinstance(parsed, list):\n", - " parsed_documents.extend(parsed)\n", - " else:\n", - " parsed_documents.append(parsed)\n", - " \n", - " # 2. WebContent (from WebIngestor) -> Already has text\n", - " elif hasattr(doc, 'text') and hasattr(doc, 'url'):\n", - " # It's already parsed content, just use it\n", - " # Ensure it has a 'content' attribute for downstream steps\n", - " if not hasattr(doc, 'content') and hasattr(doc, 'text'):\n", - " doc.content = doc.text\n", - " parsed_documents.append(doc)\n", - " \n", - " # 3. FeedData (from FeedIngestor) -> Has items list\n", - " elif hasattr(doc, 'items') and isinstance(doc.items, list):\n", - " for item in doc.items:\n", - " # Feed items usually have .content, .description, or .summary\n", - " # We'll normalize to .content\n", - " if hasattr(item, 'content'):\n", - " pass # Good\n", - " elif hasattr(item, 'description'):\n", - " item.content = item.description\n", - " elif hasattr(item, 'summary'):\n", - " item.content = item.summary\n", - " parsed_documents.append(item)\n", - "\n", - " # 4. Standard path string\n", - " elif isinstance(doc, str) and os.path.exists(doc):\n", - " parsed = document_parser.parse(doc)\n", - " if isinstance(parsed, list):\n", - " parsed_documents.extend(parsed)\n", - " else:\n", - " parsed_documents.append(parsed)\n", - " \n", - " else:\n", - " # Fallback or already a usable document object\n", - " if hasattr(doc, 'content') or hasattr(doc, 'text'):\n", - " if not hasattr(doc, 'content'): doc.content = doc.text\n", - " parsed_documents.append(doc)\n", - " else:\n", - " print(f\"Warning: Unknown document format: {type(doc)}\")\n", - "\n", - " except Exception as e:\n", - " print(f\"Error parsing document: {e}\")\n", - " continue\n", - "\n", - "print(f\"Parsed {len(parsed_documents)} documents\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.2: Split Documents Using Dual Chunking Strategy\n", - "\n", - "For GraphRAG, we use **two different chunking methods** optimized for different stores:\n", - "\n", - "**For Vector Store** (semantic similarity search):\n", - "- **Semantic Chunking**: Uses embeddings to find natural semantic boundaries\n", - "- Better for vector similarity search and retrieval\n", - "\n", - "**For Graph Store** (knowledge structure preservation):\n", - "- **Entity-Aware Chunking**: Preserves entity boundaries (prevents splitting entities)\n", - "- **Relation-Aware Chunking**: Preserves relationship triplets (keeps subject-predicate-object together)\n", - "- **Graph-Based Chunking**: Uses existing graph structure for optimal chunking\n", - "\n", - "We'll create chunks optimized for each store type.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Extracting entities, relationships, and triplets...\n" - ] - }, - { - "ename": "NameError", - "evalue": "name 'normalized_documents' is not defined", - "output_type": "error", - "traceback": [ - "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)", - "Cell \u001b[1;32mIn[7], line 13\u001b[0m\n\u001b[0;32m 10\u001b[0m flat_relationships \u001b[38;5;241m=\u001b[39m []\n\u001b[0;32m 11\u001b[0m flat_triplets \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m---> 13\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m doc \u001b[38;5;129;01min\u001b[39;00m \u001b[43mnormalized_documents\u001b[49m:\n\u001b[0;32m 14\u001b[0m text \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mstr\u001b[39m(doc\u001b[38;5;241m.\u001b[39mcontent) \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(doc, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcontent\u001b[39m\u001b[38;5;124m'\u001b[39m) \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28mstr\u001b[39m(doc)\n\u001b[0;32m 16\u001b[0m entities \u001b[38;5;241m=\u001b[39m ner\u001b[38;5;241m.\u001b[39mextract_entities(text)\n", - "\u001b[1;31mNameError\u001b[0m: name 'normalized_documents' is not defined" - ] - } - ], - "source": [ - "from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor, TripletExtractor\n", - "\n", - "print(\"Extracting entities, relationships, and triplets...\")\n", - "\n", - "ner = NamedEntityRecognizer()\n", - "rel_extractor = RelationExtractor()\n", - "triplet_extractor = TripletExtractor()\n", - "\n", - "flat_entities = []\n", - "flat_relationships = []\n", - "flat_triplets = []\n", - "\n", - "for doc in normalized_documents:\n", - " text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", - " \n", - " entities = ner.extract_entities(text)\n", - " flat_entities.extend(entities if isinstance(entities, list) else [entities])\n", - " \n", - " relations = rel_extractor.extract_relations(text, entities=entities)\n", - " flat_relationships.extend(relations if isinstance(relations, list) else [relations])\n", - " \n", - " triplets = triplet_extractor.extract_triplets(text, entities=entities, relationships=relations)\n", - " flat_triplets.extend(triplets if isinstance(triplets, list) else [triplets])\n", - "\n", - "print(f\"Extracted {len(flat_entities)} entities\")\n", - "print(f\"Extracted {len(flat_relationships)} relationships\")\n", - "print(f\"Extracted {len(flat_triplets)} triplets\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.2.1: Graph-Based Chunking (Iterative Refinement)\n", - "\n", - "After building the knowledge graph, we can use graph-based chunking to refine chunks based on graph structure. This is useful for re-chunking or optimizing existing chunks.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "graph_chunker = GraphBasedChunker(\n", - " chunk_size=1000,\n", - " chunk_overlap=200,\n", - " strategy=\"community\",\n", - " algorithm=\"louvain\"\n", + "# --- Knowledge Graph & Analytics ---\n", + "from semantica.kg import (\n", + " GraphBuilder, \n", + " GraphAnalyzer, \n", + " CentralityCalculator, \n", + " CommunityDetector\n", ")\n", "\n", - "print(\"Graph-based chunker initialized\")\n" + "# --- Ontology & Schema ---\n", + "from semantica.ontology import OntologyGenerator\n", + "\n", + "# --- Visualization ---\n", + "from semantica.visualization import KGVisualizer\n", + "\n", + "print(\"āœ… Semantica Enterprise Modules Loaded.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### 3.3: Normalize Text\n", + "## 2. Realistic Data Simulation\n", "\n", - "Clean and normalize text for better extraction quality.\n", - "\n" + "Instead of \"Hello World\", let's use a dense technical text. We'll simulate ingesting a research paper abstract on **\"Graph Neural Networks for Knowledge Graph Completion\"**. This provides rich entities (Models, Metrics, Concepts) and relationships." ] }, { @@ -557,43 +84,40 @@ "metadata": {}, "outputs": [], "source": [ - "from semantica.normalize import TextNormalizer\n", + "research_paper_text = \"\"\"\n", + "Title: Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs.\n", + "Abstract:\n", + "Knowledge Graphs (KGs) are collections of structured facts, often represented as triplets (head, relation, tail).\n", + "However, most real-world KGs like Freebase, DBPedia, and YAGO are incomplete.\n", + "Link Prediction is the task of inferring missing facts based on existing ones.\n", + "We propose a new model, Graph Attention Network for Knowledge Graphs (GAT-KG), which utilizes a multi-head attention mechanism.\n", + "Unlike TransE or DistMult which rely on translational assumptions, GAT-KG captures the structural information of the local neighborhood.\n", + "Our encoder aggregates messages from neighbors, assigning different attention weights to different relationships.\n", + "The decoder uses a standard ConvKB layer to score the plausibility of triplets.\n", + "We evaluate our method on standard benchmarks: FB15k-237 and WN18RR.\n", + "Results show that GAT-KG outperforms state-of-the-art baselines like RotatE and TuckER by 5% in Hits@10.\n", + "Ablation studies confirm that the attention mechanism is crucial for handling sparse graphs.\n", + "\"\"\"\n", "\n", - "text_normalizer = TextNormalizer()\n", + "data_dir = os.path.join(os.getcwd(), \"data\")\n", + "os.makedirs(data_dir, exist_ok=True)\n", + "source_path = os.path.join(data_dir, \"gnn_research.txt\")\n", "\n", - "print(\"Normalizing vector store chunks...\")\n", - "normalized_vector_chunks = []\n", - "for chunk in vector_store_chunks:\n", - " normalized = text_normalizer.normalize_text(chunk)\n", - " if isinstance(normalized, list):\n", - " normalized_vector_chunks.extend(normalized)\n", - " else:\n", - " normalized_vector_chunks.append(normalized)\n", + "with open(source_path, \"w\", encoding=\"utf-8\") as f:\n", + " f.write(research_paper_text)\n", "\n", - "print(\"Normalizing graph store chunks...\")\n", - "normalized_graph_chunks = []\n", - "for chunk in graph_store_chunks:\n", - " normalized = text_normalizer.normalize_text(chunk)\n", - " if isinstance(normalized, list):\n", - " normalized_graph_chunks.extend(normalized)\n", - " else:\n", - " normalized_graph_chunks.append(normalized)\n", - "\n", - "normalized_documents = normalized_vector_chunks + normalized_graph_chunks\n", - "print(f\"Normalized {len(normalized_documents)} chunks\")\n", - "print(f\" Vector store chunks: {len(normalized_vector_chunks)}\")\n", - "print(f\" Graph store chunks: {len(normalized_graph_chunks)}\")\n", - "print(\"Document processing complete!\")\n" + "print(f\"šŸ“„ Research paper saved to: {source_path}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 4: Semantic Extraction\n", + "## 3. Configuration Management\n", "\n", - "Extract entities, relationships, and triplets from the processed documents. This is the foundation for building the knowledge graph.\n", - "\n" + "We configure the system for **precision**. \n", + "* `entity_resolution_strategy: fuzzy`: Handles minor typos (e.g., \"Conv-KB\" vs \"ConvKB\").\n", + "* `extraction`: We specify a capable LLM model to ensure we catch complex technical terms." ] }, { @@ -602,1071 +126,36 @@ "metadata": {}, "outputs": [], "source": [ - "from semantica.semantic_extract import build as extract_build\n", - "\n", - "print(\"Extracting entities, relationships, and triplets...\")\n", - "\n", - "extraction_result = extract_build(\n", - " text=[str(doc.content) if hasattr(doc, 'content') else str(doc) for doc in normalized_documents],\n", - " extract_entities=True,\n", - " extract_relations=True,\n", - " extract_triplets=True\n", - ")\n", - "\n", - "flat_entities = extraction_result.get('entities', [])\n", - "flat_relationships = extraction_result.get('relationships', [])\n", - "flat_triplets = extraction_result.get('triplets', [])\n", - "\n", - "print(f\"Extracted {len(flat_entities)} entities\")\n", - "print(f\"Extracted {len(flat_relationships)} relationships\")\n", - "print(f\"Extracted {len(flat_triplets)} triplets\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(f\"\\nExtraction Summary:\")\n", - "print(f\"Entities: {len(flat_entities)}\")\n", - "print(f\"Relationships: {len(flat_relationships)}\")\n", - "print(f\"Triplets: {len(flat_triplets)}\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 5: Knowledge Graph Construction\n", - "\n", - "Build the knowledge graph from extracted entities and relationships. Apply quality assurance measures including deduplication and entity resolution.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.kg.methods import build_kg, resolve_entities, deduplicate_graph\n", - "\n", - "print(\"Deduplicating and resolving entities...\")\n", - "\n", - "deduplicated_result = deduplicate_graph(flat_entities, method=\"default\")\n", - "deduplicated_entities = deduplicated_result.get('entities', flat_entities)\n", - "\n", - "resolved_result = resolve_entities(deduplicated_entities, method=\"fuzzy\")\n", - "resolved_entities = resolved_result.get('entities', deduplicated_entities)\n", - "\n", - "print(f\"Deduplicated: {len(flat_entities)} → {len(deduplicated_entities)} entities\")\n", - "print(f\"Resolved: {len(deduplicated_entities)} → {len(resolved_entities)} entities\")\n", - "\n", - "print(\"Building knowledge graph...\")\n", - "\n", - "kg_result = build_kg(\n", - " sources=[{\n", - " 'entities': resolved_entities,\n", - " 'relationships': flat_relationships,\n", - " 'triplets': flat_triplets\n", - " }],\n", - " method=\"default\",\n", - " merge_entities=True,\n", - " resolve_conflicts=True\n", - ")\n", - "\n", - "knowledge_graph = kg_result.get('graph')\n", - "\n", - "print(f\"Knowledge graph built!\")\n", - "print(f\"Nodes: {knowledge_graph.number_of_nodes()}\")\n", - "print(f\"Edges: {knowledge_graph.number_of_edges()}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5.2: Analyze Knowledge Graph\n", - "\n", - "Analyze the graph structure to understand its properties and quality.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5.3: Refine Chunks Using Graph-Based Chunking\n", - "\n", - "After building the knowledge graph, we can use graph-based chunking to refine chunks based on graph structure. This creates chunks that align with graph communities or centrality.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "if knowledge_graph and knowledge_graph.number_of_nodes() > 0:\n", - " print(\"Refining chunks using graph-based chunking...\")\n", - " \n", - " refined_chunks = []\n", - " \n", - " for i, doc in enumerate(parsed_documents[:5]):\n", - " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", - " if doc_text.strip():\n", - " try:\n", - " graph_chunks = graph_chunker.chunk(\n", - " doc_text,\n", - " graph=knowledge_graph\n", - " )\n", - " \n", - " if isinstance(graph_chunks, list):\n", - " for chunk in graph_chunks:\n", - " if hasattr(chunk, 'metadata'):\n", - " chunk.metadata['chunking_method'] = 'graph_based'\n", - " chunk.metadata['source_doc'] = i\n", - " refined_chunks.extend(graph_chunks)\n", - " else:\n", - " refined_chunks.append(graph_chunks)\n", - " except Exception as e:\n", - " print(f\"Note: Graph-based chunking not available for doc {i}, using original chunks\")\n", - " continue\n", - " \n", - " if refined_chunks:\n", - " print(f\"Created {len(refined_chunks)} graph-based refined chunks\")\n", - " print(\"These chunks are aligned with graph communities/structure\")\n", - " else:\n", - " print(\"Using original entity/relation-aware chunks\")\n", - "else:\n", - " print(\"Graph is empty, using original entity/relation-aware chunks\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.kg.methods import analyze_graph, calculate_centrality, detect_communities, analyze_connectivity\n", - "\n", - "print(\"Analyzing knowledge graph...\")\n", - "\n", - "graph_metrics = analyze_graph(knowledge_graph, method=\"default\")\n", - "print(f\"\\nGraph Metrics:\")\n", - "print(f\"Nodes: {graph_metrics.get('nodes', 0)}\")\n", - "print(f\"Edges: {graph_metrics.get('edges', 0)}\")\n", - "print(f\"Density: {graph_metrics.get('density', 0):.4f}\")\n", - "\n", - "connectivity = analyze_connectivity(knowledge_graph, method=\"default\")\n", - "print(f\"\\nConnectivity:\")\n", - "print(f\"Connected Components: {connectivity.get('connected_components', 0)}\")\n", - "print(f\"Largest Component Size: {connectivity.get('largest_component_size', 0)}\")\n", - "\n", - "if knowledge_graph.number_of_nodes() > 0:\n", - " centrality = calculate_centrality(knowledge_graph, method='pagerank')\n", - " top_nodes = sorted(centrality.items(), key=lambda x: x[1], reverse=True)[:5]\n", - " print(f\"\\nTop 5 Central Nodes (PageRank):\")\n", - " for node, score in top_nodes:\n", - " print(f\" {node}: {score:.4f}\")\n", - " \n", - " communities = detect_communities(knowledge_graph, method='louvain')\n", - " print(f\"\\nCommunities Detected: {len(communities)}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5.3: Store Knowledge Graph (Optional)\n", - "\n", - "Optionally persist the knowledge graph to a graph database for long-term storage.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Optional: Store graph in persistent graph database\n", - "# Uncomment to use Neo4j\n", - "# graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n", - "# graph_store.connect()\n", - "# \n", - "# # Store nodes and track node ID mapping\n", - "# node_id_map = {}\n", - "# for node_id, node_data in knowledge_graph.nodes(data=True):\n", - "# labels = [node_data.get('type', 'Entity')]\n", - "# properties = {k: v for k, v in node_data.items() if k != 'type'}\n", - "# created_node = graph_store.create_node(labels, properties)\n", - "# node_id_map[node_id] = created_node.get(\"id\")\n", - "# \n", - "# # Store relationships using mapped node IDs\n", - "# for source, target, edge_data in knowledge_graph.edges(data=True):\n", - "# if source in node_id_map and target in node_id_map:\n", - "# rel_type = edge_data.get('type', 'RELATED_TO')\n", - "# properties = {k: v for k, v in edge_data.items() if k != 'type'}\n", - "# graph_store.create_relationship(\n", - "# start_node_id=node_id_map[source],\n", - "# end_node_id=node_id_map[target],\n", - "# rel_type=rel_type,\n", - "# properties=properties\n", - "# )\n", - "# \n", - "# graph_store.close()\n", - "# print(\"Knowledge graph stored in database\")\n", - "print(\"Graph storage is optional. The in-memory graph is ready for GraphRAG.\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 6: Embedding Generation\n", - "\n", - "Generate vector embeddings for documents, entities, and relationships. These embeddings enable semantic search and similarity calculations.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.embeddings import EmbeddingGenerator\n", - "\n", - "embedding_generator = EmbeddingGenerator()\n", - "\n", - "print(\"Generating embeddings for vector store chunks (semantic chunks)...\")\n", - "vector_chunk_embeddings = {}\n", - "\n", - "for i, chunk in enumerate(normalized_vector_chunks):\n", - " text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n", - " if text.strip():\n", - " embedding = embedding_generator.generate_embeddings(text, data_type=\"text\")\n", - " vector_chunk_embeddings[f\"vector_chunk_{i}\"] = {\n", - " 'embedding': embedding,\n", - " 'text': text,\n", - " 'chunking_method': 'semantic',\n", - " 'store_type': 'vector'\n", - " }\n", - "\n", - "print(f\"Generated {len(vector_chunk_embeddings)} vector store chunk embeddings\")\n", - "\n", - "print(\"\\nGenerating embeddings for graph store chunks (graph-aware chunks)...\")\n", - "graph_chunk_embeddings = {}\n", - "\n", - "for i, chunk in enumerate(normalized_graph_chunks):\n", - " text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n", - " if text.strip():\n", - " embedding = embedding_generator.generate_embeddings(text, data_type=\"text\")\n", - " graph_chunk_embeddings[f\"graph_chunk_{i}\"] = {\n", - " 'embedding': embedding,\n", - " 'text': text,\n", - " 'chunking_method': 'graph_aware',\n", - " 'store_type': 'graph'\n", - " }\n", - "\n", - "print(f\"Generated {len(graph_chunk_embeddings)} graph store chunk embeddings\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 6.2: Generate Entity Embeddings\n", - "\n", - "Generate embeddings for entities to enable entity-based semantic search.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Generate embeddings for entities\n", - "print(\"Generating embeddings for entities...\")\n", - "entity_embeddings = {}\n", - "\n", - "for entity in resolved_entities[:100]: # Limit to first 100 for demo\n", - " if isinstance(entity, dict):\n", - " entity_text = entity.get('text', entity.get('name', str(entity)))\n", - " else:\n", - " entity_text = str(entity)\n", - " \n", - " if entity_text.strip():\n", - " embedding = embedding_generator.generate_embeddings(entity_text, data_type=\"text\")\n", - " entity_id = entity.get('id', entity.get('text', str(entity))) if isinstance(entity, dict) else str(entity)\n", - " entity_embeddings[entity_id] = {\n", - " 'embedding': embedding,\n", - " 'text': entity_text\n", - " }\n", - "\n", - "print(f\"Generated {len(entity_embeddings)} entity embeddings\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 7: Vector Store Setup\n", - "\n", - "Store embeddings in a vector store for fast similarity search and retrieval.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.vector_store import VectorStore, HybridSearch\n", - "from semantica.graph_store import GraphStore\n", - "\n", - "vector_store = VectorStore()\n", - "\n", - "vectors = []\n", - "metadata_list = []\n", - "ids = []\n", - "\n", - "print(\"Storing semantic chunks in vector store...\")\n", - "for chunk_id, chunk_data in vector_chunk_embeddings.items():\n", - " vectors.append(chunk_data['embedding'])\n", - " metadata_list.append({\n", - " 'type': 'chunk',\n", - " 'chunking_method': 'semantic',\n", - " 'store_type': 'vector',\n", - " 'text': chunk_data['text'][:200]\n", - " })\n", - " ids.append(chunk_id)\n", - "\n", - "for entity_id, entity_data in entity_embeddings.items():\n", - " vectors.append(entity_data['embedding'])\n", - " metadata_list.append({'type': 'entity', 'text': entity_data['text']})\n", - " ids.append(entity_id)\n", - "\n", - "if vectors:\n", - " vector_store.store(vectors=vectors, metadata=metadata_list, ids=ids)\n", - " print(f\"Stored {len(vectors)} vectors in vector store\")\n", - " print(f\" Semantic chunks: {len(vector_chunk_embeddings)}\")\n", - " print(f\" Entities: {len(entity_embeddings)}\")\n", - "else:\n", - " print(\"No vectors to store\")\n", - "\n", - "print(\"\\nStoring graph-aware chunks in graph store...\")\n", - "# Option 1: Neo4j (requires Neo4j server running)\n", - "graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n", - "graph_store.connect()\n", - "\n", - "for i, chunk in enumerate(graph_store_chunks):\n", - " chunk_text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n", - " if chunk_text.strip():\n", - " chunk_metadata = {\n", - " 'chunking_method': 'graph_aware',\n", - " 'store_type': 'graph',\n", - " 'text': chunk_text[:500]\n", - " }\n", - " if hasattr(chunk, 'metadata'):\n", - " if chunk.metadata.get('entities'):\n", - " chunk_metadata['entities'] = chunk.metadata['entities']\n", - " if chunk.metadata.get('relationships'):\n", - " chunk_metadata['relationships'] = chunk.metadata['relationships']\n", - " \n", - " graph_store.create_node(\n", - " labels=['Chunk'],\n", - " properties={\n", - " 'id': f\"graph_chunk_{i}\",\n", - " **chunk_metadata\n", - " }\n", - " )\n", - "\n", - "print(f\"Stored {len(graph_store_chunks)} graph-aware chunks in graph store\")\n", - "graph_store.close()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 8: Hybrid Search Implementation\n", - "\n", - "Implement hybrid search that combines vector similarity search with knowledge graph traversal for enhanced retrieval.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.vector_store import HybridSearch\n", - "\n", - "hybrid_search = HybridSearch(vector_store=vector_store)\n", - "\n", - "def perform_hybrid_search(query: str, top_k: int = 10):\n", - " query_embedding = embedding_generator.generate_embeddings(query, data_type=\"text\")\n", - " vector_results = vector_store.search(\n", - " query_vector=query_embedding,\n", - " top_k=top_k * 2\n", - " )\n", - " \n", - " # Graph-based search (if query contains entity mentions)\n", - " graph_results = []\n", - " if knowledge_graph.number_of_nodes() > 0:\n", - " # Extract entities from query\n", - " query_entities = ner_extractor.extract(query)\n", - " if query_entities:\n", - " # Find related nodes in graph\n", - " for entity in query_entities:\n", - " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n", - " # Search for entity in graph\n", - " for node in knowledge_graph.nodes():\n", - " if entity_text.lower() in str(node).lower():\n", - " # Get neighbors\n", - " neighbors = list(knowledge_graph.neighbors(node))\n", - " for neighbor in neighbors[:5]: # Limit neighbors\n", - " graph_results.append({\n", - " 'id': f\"graph_{node}_{neighbor}\",\n", - " 'content': f\"{node} -> {neighbor}\",\n", - " 'score': 0.7, # Graph relevance score\n", - " 'source': 'graph'\n", - " })\n", - " \n", - " # Combine and rank results using hybrid search\n", - " all_results = vector_results + graph_results\n", - " \n", - " # Use hybrid search ranker\n", - " if all_results:\n", - " ranked_results = hybrid_search.ranker.rank([all_results], top_k=top_k)\n", - " return ranked_results[:top_k]\n", - " \n", - " return []\n", - "\n", - "# Test hybrid search\n", - "test_query = \"artificial intelligence and machine learning\"\n", - "print(f\"Testing hybrid search with query: '{test_query}'\")\n", - "search_results = perform_hybrid_search(test_query, top_k=5)\n", - "\n", - "print(f\"Search Results ({len(search_results)}):\")\n", - "for i, result in enumerate(search_results[:5], 1):\n", - " print(f\"\\n{i}. Score: {result.get('score', 0):.4f}\")\n", - " print(f\" Source: {result.get('source', 'unknown')}\")\n", - " content = result.get('content', result.get('text', 'N/A'))\n", - " print(f\" Content: {content[:100]}...\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.context import ContextRetriever, ContextGraphBuilder, AgentMemory\n", - "from semantica.context.methods import retrieve_context, build_context_graph\n", - "\n", - "agent_memory = AgentMemory(\n", - " vector_store=vector_store,\n", - " knowledge_graph=knowledge_graph\n", - ")\n", - "\n", - "context_retriever = ContextRetriever(\n", - " memory_store=agent_memory,\n", - " knowledge_graph=knowledge_graph,\n", - " vector_store=vector_store,\n", - " use_graph_expansion=True,\n", - " max_expansion_hops=2,\n", - " hybrid_alpha=0.5\n", - ")\n", - "\n", - "context_graph_builder = ContextGraphBuilder()\n", - "\n", - "print(\"Context retrieval system initialized\")\n", - "print(f\"Graph expansion: Enabled (max {context_retriever.max_expansion_hops} hops)\")\n", - "print(f\"Hybrid alpha: {context_retriever.hybrid_alpha} (0=vector only, 1=graph only)\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 9.2: Retrieve Context with Graph Expansion\n", - "\n", - "Retrieve context using hybrid approach with graph expansion for multi-hop reasoning.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def retrieve_context_for_query(query: str, max_results: int = 10):\n", - " print(f\"\\nRetrieving context for: '{query}'\")\n", - " \n", - " retrieved_contexts = retrieve_context(\n", - " query=query,\n", - " method=\"hybrid\",\n", - " max_results=max_results,\n", - " knowledge_graph=knowledge_graph,\n", - " vector_store=vector_store,\n", - " use_graph_expansion=True,\n", - " max_hops=2\n", - " )\n", - " \n", - " print(f\"Retrieved {len(retrieved_contexts)} context items\")\n", - " \n", - " for i, ctx in enumerate(retrieved_contexts[:5], 1):\n", - " print(f\"\\n{i}. Relevance: {ctx.score:.4f}\")\n", - " print(f\" Source: {ctx.source}\")\n", - " print(f\" Content: {ctx.content[:150]}...\")\n", - " if hasattr(ctx, 'related_entities') and ctx.related_entities:\n", - " print(f\" Related entities: {len(ctx.related_entities)}\")\n", - " if hasattr(ctx, 'related_relationships') and ctx.related_relationships:\n", - " print(f\" Related relationships: {len(ctx.related_relationships)}\")\n", - " \n", - " return retrieved_contexts\n", - "\n", - "test_query = \"What are the relationships between AI and machine learning?\"\n", - "contexts = retrieve_context_for_query(test_query, max_results=10)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 9.3: Build Context Graph\n", - "\n", - "Build a context graph from retrieved contexts to visualize relationships.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "if contexts:\n", - " context_graph = build_context_graph(\n", - " contexts=contexts,\n", - " method=\"entities_relationships\"\n", - " )\n", - " \n", - " print(f\"Context Graph:\")\n", - " print(f\"Nodes: {context_graph.number_of_nodes()}\")\n", - " print(f\"Edges: {context_graph.number_of_edges()}\")\n", - " \n", - " if context_graph.number_of_nodes() > 0:\n", - " print(f\"\\nSample Context Graph Nodes:\")\n", - " for i, node in enumerate(list(context_graph.nodes())[:5], 1):\n", - " print(f\" {i}. {node}\")\n", - "else:\n", - " print(\"No contexts to build graph from\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 10: GraphRAG Query System\n", - "\n", - "Build a complete GraphRAG query processing pipeline that handles different types of queries and prepares context for LLM integration.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.semantic_extract import NERExtractor\n", - "\n", - "class GraphRAGQuerySystem:\n", - " def __init__(self, context_retriever, knowledge_graph, vector_store):\n", - " self.context_retriever = context_retriever\n", - " self.knowledge_graph = knowledge_graph\n", - " self.vector_store = vector_store\n", - " self.ner_extractor = NERExtractor()\n", - " \n", - " def process_query(self, query: str, max_context: int = 10):\n", - " \"\"\"\n", - " Process a query through the complete GraphRAG pipeline.\n", - " \n", - " Steps:\n", - " 1. Parse user query\n", - " 2. Extract query entities\n", - " 3. Perform hybrid search (vector + graph)\n", - " 4. Retrieve relevant context\n", - " 5. Expand context with graph relationships\n", - " 6. Prepare context for LLM\n", - " \"\"\"\n", - " print(f\"Processing query: '{query}'\")\n", - " \n", - " # Step 1: Extract entities from query\n", - " query_entities = self.ner_extractor.extract(query)\n", - " print(f\"Extracted {len(query_entities)} entities from query\")\n", - " \n", - " # Step 2: Retrieve context\n", - " contexts = self.context_retriever.retrieve(\n", - " query=query,\n", - " max_results=max_context,\n", - " use_graph_expansion=True,\n", - " max_hops=2\n", - " )\n", - " \n", - " # Step 3: Expand context with graph relationships\n", - " expanded_context = self._expand_context_with_graph(contexts, query_entities)\n", - " \n", - " # Step 4: Prepare context for LLM\n", - " llm_context = self._prepare_llm_context(expanded_context, query)\n", - " \n", - " return {\n", - " 'query': query,\n", - " 'query_entities': query_entities,\n", - " 'contexts': contexts,\n", - " 'expanded_context': expanded_context,\n", - " 'llm_context': llm_context\n", - " }\n", - " \n", - " def _expand_context_with_graph(self, contexts, query_entities):\n", - " \"\"\"Expand context by following graph relationships.\"\"\"\n", - " expanded = []\n", - " \n", - " for ctx in contexts:\n", - " expanded.append(ctx)\n", - " \n", - " # Add related entities from graph\n", - " if ctx.related_entities:\n", - " for entity in ctx.related_entities[:3]: # Limit expansion\n", - " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n", - " # Find in graph and get neighbors\n", - " for node in self.knowledge_graph.nodes():\n", - " if entity_text.lower() in str(node).lower():\n", - " neighbors = list(self.knowledge_graph.neighbors(node))[:2]\n", - " for neighbor in neighbors:\n", - " expanded.append({\n", - " 'content': f\"Related: {node} -> {neighbor}\",\n", - " 'score': 0.6,\n", - " 'source': 'graph_expansion'\n", - " })\n", - " \n", - " return expanded\n", - " \n", - " def _prepare_llm_context(self, contexts, query):\n", - " \"\"\"Prepare formatted context for LLM.\"\"\"\n", - " context_text = f\"Query: {query}\\n\\nRelevant Context:\\n\\n\"\n", - " \n", - " for i, ctx in enumerate(contexts[:10], 1):\n", - " content = ctx.content if hasattr(ctx, 'content') else ctx.get('content', str(ctx))\n", - " score = ctx.score if hasattr(ctx, 'score') else ctx.get('score', 0)\n", - " source = ctx.source if hasattr(ctx, 'source') else ctx.get('source', 'unknown')\n", - " \n", - " context_text += f\"{i}. [Relevance: {score:.3f}, Source: {source}]\\n\"\n", - " context_text += f\"{content[:300]}...\\n\\n\"\n", - " \n", - " return context_text\n", - "\n", - "# Initialize GraphRAG query system\n", - "graphrag_system = GraphRAGQuerySystem(\n", - " context_retriever=context_retriever,\n", - " knowledge_graph=knowledge_graph,\n", - " vector_store=vector_store\n", - ")\n", - "\n", - "print(f\"GraphRAG query system initialized\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example queries\n", - "example_queries = [\n", - " \"What is artificial intelligence?\", # Factual question\n", - " \"How are AI and machine learning related?\", # Relationship query\n", - " \"What are the applications of deep learning in healthcare?\", # Complex multi-hop query\n", - "]\n", - "\n", - "# Process each query\n", - "query_results = {}\n", - "for query in example_queries:\n", - " print(f\"\\n{'='*60}\")\n", - " result = graphrag_system.process_query(query, max_context=10)\n", - " query_results[query] = result\n", - " \n", - " print(f\"Prepared LLM Context ({len(result['llm_context'])} chars):\")\n", - " print(result['llm_context'][:500] + \"...\")\n", - "\n", - "print(f\"\\nProcessed {len(query_results)} queries\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 11: LLM Integration\n", - "\n", - "Integrate with LLM (OpenAI, Anthropic, or local) to generate answers using the retrieved GraphRAG context.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# LLM Integration\n", - "# This section demonstrates how to integrate with LLMs using the retrieved context\n", - "\n", - "def generate_answer_with_llm(query: str, llm_context: str, llm_provider: str = \"openai\"):\n", - " \"\"\"\n", - " Generate answer using LLM with GraphRAG context.\n", - " \n", - " Supports OpenAI, Anthropic, or local LLMs.\n", - " \"\"\"\n", - " # Build prompt\n", - " prompt = f\"\"\"You are an AI assistant with access to a knowledge graph and retrieved context.\n", - "\n", - "Context from Knowledge Graph:\n", - "{llm_context}\n", - "\n", - "Question: {query}\n", - "\n", - "Based on the context provided above, please answer the question. If the context doesn't contain enough information, say so. Cite specific entities or relationships from the context when relevant.\n", - "\n", - "Answer:\"\"\"\n", - " \n", - " # Here you would call your LLM\n", - " # Example with OpenAI (uncomment and configure):\n", - " # try:\n", - " # from openai import OpenAI\n", - " # client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))\n", - " # response = client.chat.completions.create(\n", - " # model=\"gpt-4\",\n", - " # messages=[\n", - " # {\"role\": \"system\", \"content\": \"You are a helpful assistant with access to knowledge graphs.\"},\n", - " # {\"role\": \"user\", \"content\": prompt}\n", - " # ],\n", - " # temperature=0.7\n", - " # )\n", - " # return response.choices[0].message.content\n", - " # except Exception as e:\n", - " # return f\"Error calling LLM: {e}\"\n", - " \n", - " # For demonstration, return the prompt structure\n", - " return f\"[LLM Answer would be generated here using the context above]\"\n", - "\n", - "# Example: Generate answer for a query\n", - "if query_results:\n", - " sample_query = list(query_results.keys())[0]\n", - " sample_result = query_results[sample_query]\n", - " \n", - " print(f\"Generating answer for: '{sample_query}'\")\n", - " answer = generate_answer_with_llm(\n", - " query=sample_query,\n", - " llm_context=sample_result['llm_context']\n", - " )\n", - " \n", - " print(f\"\\nAnswer:\")\n", - " print(answer)\n", - " print(f\"\\nContext Statistics:\")\n", - " print(f\" Context items: {len(sample_result['contexts'])}\")\n", - " print(f\" Expanded context: {len(sample_result['expanded_context'])}\")\n", - " print(f\" Query entities: {len(sample_result['query_entities'])}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 11.2: Source Attribution and Explainability\n", - "\n", - "Show which parts of the knowledge graph contributed to the answer for explainability.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def explain_answer_sources(query_result):\n", - " \"\"\"\n", - " Explain which sources contributed to the answer.\n", - " \"\"\"\n", - " print(f\"Answer Sources and Attribution:\")\n", - " print(f\"Query: {query_result['query']}\")\n", - " print(f\"\\nRetrieved Context Sources:\")\n", - " \n", - " sources = {}\n", - " for ctx in query_result['contexts']:\n", - " source = ctx.source if hasattr(ctx, 'source') else ctx.get('source', 'unknown')\n", - " sources[source] = sources.get(source, 0) + 1\n", - " \n", - " for source, count in sources.items():\n", - " print(f\" {source}: {count} context items\")\n", - " \n", - " print(f\"\\nGraph Entities Involved:\")\n", - " for entity in query_result['query_entities'][:5]:\n", - " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n", - " print(f\" - {entity_text}\")\n", - " \n", - " print(f\"\\nContext Expansion:\")\n", - " print(f\" Original contexts: {len(query_result['contexts'])}\")\n", - " print(f\" Expanded contexts: {len(query_result['expanded_context'])}\")\n", - " print(f\" Expansion ratio: {len(query_result['expanded_context']) / max(len(query_result['contexts']), 1):.2f}x\")\n", - "\n", - "# Explain sources for sample query\n", - "if query_results:\n", - " explain_answer_sources(list(query_results.values())[0])\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 12: Advanced Features\n", - "\n", - "Demonstrate advanced features including reasoning, quality assessment, and visualization.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.reasoning import InferenceEngine, RuleManager\n", - "\n", - "# Advanced Feature 1: Reasoning with Inference Engine\n", - "print(f\"Advanced Feature: Logical Reasoning\")\n", - "inference_engine = InferenceEngine()\n", - "rule_manager = RuleManager()\n", - "\n", - "# Example: Add inference rules\n", - "# inference_engine.add_rule(\"IF entity A works_for entity B AND entity B located_in entity C THEN entity A located_in entity C\")\n", - "# # inference_engine.add_fact(...) # Add facts first\n", - "# new_facts = inference_engine.forward_chain()\n", - "# print(f\"Inferred {len(new_facts)} new facts\")\n", - "\n", - "print(f\"Reasoning can infer new relationships from existing knowledge\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 12.2: Visualize Knowledge Graph\n", - "\n", - "Visualize the knowledge graph to understand its structure.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.visualization import KGVisualizer, AnalyticsVisualizer\n", - "\n", - "# Initialize visualizer\n", - "kg_visualizer = KGVisualizer()\n", - "analytics_visualizer = AnalyticsVisualizer()\n", - "\n", - "# Visualize knowledge graph\n", - "if knowledge_graph.number_of_nodes() > 0:\n", - " print(f\"Visualizing knowledge graph...\")\n", - " \n", - " # Create visualization\n", - " # Uncomment to generate visualization\n", - " # visualization = kg_visualizer.visualize(\n", - " # knowledge_graph,\n", - " # output_path=\"graphrag_visualization.html\",\n", - " # layout=\"spring\",\n", - " # show_labels=True\n", - " # )\n", - " # print(f\"Visualization saved to graphrag_visualization.html\")\n", - " \n", - " print(f\"Graph Statistics for Visualization:\")\n", - " print(f\" Nodes: {knowledge_graph.number_of_nodes()}\")\n", - " print(f\" Edges: {knowledge_graph.number_of_edges()}\")\n", - " print(f\" Node types: {len(set(n.get('type', 'Unknown') for _, n in knowledge_graph.nodes(data=True)))}\")\n", - " print(f\" Edge types: {len(set(e.get('type', 'Unknown') for _, _, e in knowledge_graph.edges(data=True)))}\")\n", - " \n", - " # Analytics visualization\n", - " # analytics_viz = analytics_visualizer.visualize(\n", - " # knowledge_graph,\n", - " # metrics=['centrality', 'communities'],\n", - " # output_path=\"graphrag_analytics.html\"\n", - " # )\n", - " # print(f\"Analytics visualization saved\")\n", - "else:\n", - " print(f\"Graph is empty, skipping visualization\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 13: Complete End-to-End Example\n", - "\n", - "Demonstrate a complete end-to-end GraphRAG workflow with real-world data, showing the full pipeline from ingestion to answer generation.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def complete_graphrag_workflow(query: str):\n", - " \"\"\"\n", - " Complete GraphRAG workflow from query to answer.\n", - " \"\"\"\n", - " print(f\"\\n{'='*70}\")\n", - " print(f\"Complete GraphRAG Workflow\")\n", - " print(f\"{'='*70}\")\n", - " print(f\"Query: {query}\\n\")\n", - " \n", - " # Step 1: Process query\n", - " print(\"Step 1: Processing query...\")\n", - " result = graphrag_system.process_query(query, max_context=10)\n", - " \n", - " # Step 2: Generate answer\n", - " print(\"\\nStep 2: Generating answer with LLM...\")\n", - " answer = generate_answer_with_llm(query, result['llm_context'])\n", - " \n", - " # Step 3: Explain sources\n", - " print(\"\\nStep 3: Explaining sources...\")\n", - " explain_answer_sources(result)\n", - " \n", - " # Step 4: Show performance metrics\n", - " print(\"\\nStep 4: Performance Metrics:\")\n", - " print(f\" Context retrieval time: <1s (simulated)\")\n", - " print(f\" Context items retrieved: {len(result['contexts'])}\")\n", - " print(f\" Graph expansion hops: 2\")\n", - " print(f\" Total context size: {len(result['llm_context'])} characters\")\n", - " \n", - " return {\n", - " 'query': query,\n", - " 'answer': answer,\n", - " 'contexts': result['contexts'],\n", - " 'metrics': {\n", - " 'context_items': len(result['contexts']),\n", - " 'expanded_items': len(result['expanded_context']),\n", - " 'query_entities': len(result['query_entities'])\n", - " }\n", - " }\n", - "\n", - "# Run complete workflow example\n", - "if len(all_documents) > 0 or knowledge_graph.number_of_nodes() > 0:\n", - " example_query = \"What are the main concepts and their relationships?\"\n", - " workflow_result = complete_graphrag_workflow(example_query)\n", - " \n", - " print(f\"\\nComplete workflow executed successfully!\")\n", - " print(f\"Final Results:\")\n", - " print(f\" Query processed: āœ“\")\n", - " print(f\" Context retrieved: {workflow_result['metrics']['context_items']} items\")\n", - " print(f\" Answer generated: āœ“\")\n", - "else:\n", - " print(\"Configure data sources above to run complete workflow with real data\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(f\"Comparison: Traditional RAG vs GraphRAG\\n\")\n", - "\n", - "comparison = {\n", - " \"Traditional RAG\": {\n", - " \"Retrieval\": \"Vector similarity only\",\n", - " \"Context\": \"Flat document chunks\",\n", - " \"Relationships\": \"Not captured\",\n", - " \"Multi-hop\": \"Not supported\",\n", - " \"Explainability\": \"Limited (source documents only)\"\n", + "config_data = {\n", + " \"project_name\": \"ResearchGraph\",\n", + " \"embedding\": {\n", + " \"provider\": \"openai\",\n", + " \"model\": \"text-embedding-3-small\"\n", " },\n", - " \"GraphRAG\": {\n", - " \"Retrieval\": \"Vector + Graph traversal\",\n", - " \"Context\": \"Structured knowledge graph\",\n", - " \"Relationships\": \"Explicitly modeled\",\n", - " \"Multi-hop\": \"Supported (graph expansion)\",\n", - " \"Explainability\": \"High (entities, relationships, paths)\"\n", + " \"knowledge_graph\": {\n", + " \"backend\": \"networkx\",\n", + " \"merge_entities\": True,\n", + " \"entity_resolution_strategy\": \"fuzzy\",\n", + " \"similarity_threshold\": 0.85\n", + " },\n", + " \"extraction\": {\n", + " \"model\": \"gpt-4o\",\n", + " \"temperature\": 0.0\n", " }\n", "}\n", "\n", - "print(\"Feature Comparison:\")\n", - "print(f\"{'Feature':<20} {'Traditional RAG':<25} {'GraphRAG':<25}\")\n", - "print(\"-\" * 70)\n", - "\n", - "for feature in comparison[\"Traditional RAG\"].keys():\n", - " trad = comparison[\"Traditional RAG\"][feature]\n", - " graph = comparison[\"GraphRAG\"][feature]\n", - " print(f\"{feature:<20} {trad:<25} {graph:<25}\")\n", - "\n", - "print(\"\\nGraphRAG Advantages:\")\n", - "print(f\" • Better handling of complex queries requiring relationship understanding\")\n", - "print(f\" • Multi-hop reasoning across entities\")\n", - "print(f\" • More accurate answers through structured knowledge\")\n", - "print(f\" • Better explainability with graph paths\")\n", - "print(f\" • Reduced hallucinations through graph validation\")\n" + "config = ConfigManager().load_from_dict(config_data)\n", + "semantica = Semantica(config=config)\n", + "semantica.initialize()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 14: Export and Persistence\n", + "## 4. Building the Knowledge Base\n", "\n", - "Export the knowledge graph and save the vector store for reuse and sharing.\n", - "\n" + "We execute the orchestrated pipeline. This single call replaces manual parsing, cleaning, chunking, NER, RE, and embedding generation." ] }, { @@ -1675,108 +164,183 @@ "metadata": {}, "outputs": [], "source": [ - "from semantica.export import JSONExporter, RDFExporter, CSVExporter\n", + "print(\"šŸš€ Building Knowledge Base from Research Paper...\")\n", "\n", - "json_exporter = JSONExporter()\n", - "rdf_exporter = RDFExporter()\n", - "csv_exporter = CSVExporter()\n", + "result = semantica.build_knowledge_base(\n", + " sources=[source_path],\n", + " embeddings=True,\n", + " graph=True,\n", + " normalize=True\n", + ")\n", "\n", - "# Export knowledge graph to JSON\n", - "if knowledge_graph.number_of_nodes() > 0:\n", - " print(f\"Exporting knowledge graph...\")\n", - " \n", - " # Export to JSON\n", - " json_output = json_exporter.export(knowledge_graph, \"graphrag_knowledge_graph.json\")\n", - " print(f\"Exported to JSON: graphrag_knowledge_graph.json\")\n", - " \n", - " # Export to RDF\n", - " rdf_output = rdf_exporter.export(knowledge_graph, \"graphrag_knowledge_graph.rdf\")\n", - " print(f\"Exported to RDF: graphrag_knowledge_graph.rdf\")\n", - " \n", - " # Export entities to CSV\n", - " entities_data = []\n", - " for entity in resolved_entities[:100]: # Limit for demo\n", - " if isinstance(entity, dict):\n", - " entities_data.append({\n", - " 'id': entity.get('id', ''),\n", - " 'text': entity.get('text', entity.get('name', '')),\n", - " 'type': entity.get('type', 'Unknown')\n", - " })\n", - " \n", - " if entities_data:\n", - " csv_output = csv_exporter.export(entities_data, \"graphrag_entities.csv\")\n", - " print(f\"Exported entities to CSV: graphrag_entities.csv\")\n", - " \n", - " print(f\"\\nExport Summary:\")\n", - " print(f\" Nodes exported: {knowledge_graph.number_of_nodes()}\")\n", - " print(f\" Edges exported: {knowledge_graph.number_of_edges()}\")\n", - " print(f\" Entities exported: {len(entities_data)}\")\n", - "else:\n", - " print(f\"Graph is empty, skipping export\")\n", - "\n", - "# Save vector store (if supported)\n", - "print(\"\\nVector Store:\")\n", - "print(f\" Vectors stored: āœ“\")\n", - "print(f\" Metadata stored: āœ“\")\n", - "print(f\" Ready for reuse: āœ“\")\n" + "kg = result[\"knowledge_graph\"]\n", + "print(f\"\\nāœ… Extraction Complete.\")\n", + "print(f\"Nodes: {kg.number_of_nodes()} | Edges: {kg.number_of_edges()}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Summary and Next Steps\n", + "## 5. Advanced Graph Analytics\n", "\n", - "### What We Built\n", + "A knowledge graph is more than just storage; it's a structure we can analyze to find insights.\n", "\n", - "This notebook demonstrated a **complete end-to-end GraphRAG system** using Semantica:\n", + "### A. Centrality (Finding Key Concepts)\n", + "Who are the \"VIPs\" in this paper? We use Degree Centrality to find the most connected entities." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "centrality_calc = CentralityCalculator()\n", + "centrality_scores = centrality_calc.calculate_degree_centrality(kg)\n", "\n", - "1. **Real-World Data Ingestion**: MCP servers, web scraping, RSS feeds\n", - "2. **Document Processing**: Parsing, splitting, normalization\n", - "3. **Semantic Extraction**: Entities, relationships, triplets\n", - "4. **Knowledge Graph Construction**: With quality assurance\n", - "5. **Embedding Generation**: For documents and entities\n", - "6. **Vector Store**: Fast similarity search\n", - "7. **Hybrid Search**: Combining vectors and graphs\n", - "8. **Context Retrieval**: With graph expansion\n", - "9. **GraphRAG Query System**: Complete query processing\n", - "10. **LLM Integration**: Answer generation with context\n", - "11. **Advanced Features**: Reasoning, quality, visualization\n", - "12. **Export**: Persistence and sharing\n", - "\n", - "### Key Takeaways\n", - "\n", - "- **GraphRAG** combines the best of vector search and knowledge graphs\n", - "- **Multi-hop reasoning** enables deeper understanding\n", - "- **Real-world data** makes the system production-ready\n", - "- **Semantica** provides all modules needed for GraphRAG\n", - "\n", - "### Next Steps\n", - "\n", - "1. **Configure Real Data Sources**: Set up MCP servers, web URLs, or RSS feeds\n", - "2. **Customize Extraction**: Adjust entity and relationship extraction for your domain\n", - "3. **Tune Hybrid Search**: Experiment with `hybrid_alpha` for your use case\n", - "4. **Add More LLMs**: Integrate with Anthropic, local models, or other providers\n", - "5. **Scale Up**: Process larger datasets and optimize performance\n", - "6. **Deploy**: Build production GraphRAG applications\n", - "\n", - "### Resources\n", - "\n", - "- [Semantica Documentation](https://semantica.readthedocs.io/)\n", - "- [GraphRAG Concepts](https://semantica.readthedocs.io/concepts/)\n", - "- [API Reference](https://semantica.readthedocs.io/reference/)\n", - "- [More Examples](https://semantica.readthedocs.io/cookbook/)\n", - "\n", - "---\n", - "\n", - "**Congratulations!** You've built a complete GraphRAG system with Semantica!\n", - "\n" + "# Convert to DataFrame for nice display\n", + "df_centrality = pd.DataFrame(centrality_scores.get(\"rankings\", []))\n", + "if not df_centrality.empty:\n", + " print(\"šŸ† Top 5 Most Central Concepts:\")\n", + " print(df_centrality.head(5)[[\"node\", \"score\"]].to_string(index=False))" ] }, { "cell_type": "markdown", "metadata": {}, - "source": [] + "source": [ + "### B. Community Detection (Topic Modeling)\n", + "Can we group these entities into topics? We use the **Louvain Algorithm** to detect communities within the graph." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "detector = CommunityDetector()\n", + "communities_result = detector.detect_communities(kg, algorithm=\"louvain\")\n", + "communities = communities_result.get(\"communities\", [])\n", + "\n", + "print(f\"šŸ” Detected {len(communities)} distinct communities (topics).\")\n", + "\n", + "for i, comm in enumerate(communities[:3]): # Show first 3\n", + " print(f\"\\nGroup {i+1}: {', '.join(list(comm)[:5])}...\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 6. Ontology Generation\n", + "\n", + "One of Semantica's most powerful features is **Reverse Engineering the Schema**. We can look at the graph we just built and ask: \"What is the underlying data model?\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ontology_gen = OntologyGenerator()\n", + "ontology = ontology_gen.generate_from_graph(kg)\n", + "\n", + "print(\"🧬 Inferred Ontology Classes:\")\n", + "if \"classes\" in ontology:\n", + " for cls in ontology[\"classes\"][:5]:\n", + " print(f" - {cls.get('name')}: {cls.get('description', 'No description inferred')}")\n", + "else:\n", + " print(\" (Ontology generation requires sufficient data volume for inference)\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 7. Hybrid GraphRAG Inference\n", + "\n", + "Finally, we use the system to answer a complex question. We simulate the logic of a Hybrid Search:\n", + "1. **Vector**: Find the relevant text chunk.\n", + "2. **Graph**: Find the *neighbors* of the key entities to understand the context (e.g., what benchmarks were used?)\n", + "3. **Synthesis**: (Conceptually) combine these for the LLM." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"What benchmarks were used to evaluate GAT-KG?\"\n", + "print(f\"ā“ Query: '{query}'\")\n", + "\n", + "# 1. Vector Search\n", + "vector_store = semantica.vector_store\n", + "embedding_gen = semantica.embedding_generator\n", + "q_vec = embedding_gen.generate_embeddings(query, data_type=\"text\")\n", + "vec_results = vector_store.search(q_vec, k=1)\n", + "\n", + "print(\"\\nšŸ”¹ [Vector Retrieval] Found Context:\")\n", + "top_chunk = vec_results[0].get('metadata', {}).get('text', '') if vec_results else \"No match\"\n", + "print(f\" \\\"{top_chunk[:100]}...\\\"\")\n", + "\n", + "# 2. Graph Traversal\n", + "print(\"\\nšŸ”ø [Graph Retrieval] Exploring Neighborhoods:\")\n", + "# Find 'GAT-KG' in the graph and see what it connects to\n", + "target_node = None\n", + "for node in kg.nodes():\n", + " if \"GAT-KG\" in str(node):\n", + " target_node = node\n", + " break\n", + "\n", + "if target_node:\n", + " neighbors = list(kg[target_node])\n", + " print(f" Entity '{target_node}' is connected to:")\n", + " for n in neighbors:\n", + " relation = kg[target_node][n].get('type', 'related_to')\n", + " print(f" --[{relation}]--> {n}")\n", + "else:\n", + " print(\" (Entity 'GAT-KG' not explicitly found in graph nodes)\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 8. Visualization\n", + "\n", + "We visualize the final graph structure." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "viz = KGVisualizer()\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import networkx as nx\n", + "\n", + "plt.figure(figsize=(14, 10))\n", + "pos = nx.spring_layout(kg, k=0.8, seed=42)\n", + "\n", + "# Draw with more style\n", + "nx.draw_networkx_nodes(kg, pos, node_size=1500, node_color=\"#6a0dad\", alpha=0.8) # Semantica Purple\n", + "nx.draw_networkx_edges(kg, pos, width=1.5, alpha=0.4, edge_color=\"gray\", arrows=True)\n", + "nx.draw_networkx_labels(kg, pos, font_size=9, font_color=\"white\", font_weight=\"bold\")\n", + "\n", + "edge_labels = nx.get_edge_attributes(kg, 'type')\n", + "nx.draw_networkx_edge_labels(kg, pos, edge_labels=edge_labels, font_size=7)\n", + "\n", + "plt.title(\"Research Paper Knowledge Graph\", fontsize=20)\n", + "plt.axis('off')\n", + "plt.show()" + ] } ], "metadata": { @@ -1795,9 +359,9 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.9" + "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file