diff --git a/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb
index 6fc16e91..5fee8de6 100644
--- a/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb
+++ b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb
@@ -4,551 +4,78 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)\n",
- "\n",
- "# GraphRAG Complete - End-to-End Pipeline\n",
+ "# š Enterprise GraphRAG: End-to-End with Semantica\n",
"\n",
"## Overview\n",
"\n",
- "This notebook demonstrates a **complete end-to-end GraphRAG (Graph-based Retrieval Augmented Generation) system** using Semantica framework. It showcases how to build a production-ready GraphRAG system that combines vector search with knowledge graph traversal for enhanced retrieval and question answering.\n",
+ "This notebook demonstrates a **production-grade GraphRAG pipeline** using the `Semantica` framework. We go beyond simple tutorials to show how to build, analyze, and query a rich knowledge base using advanced graph algorithms and hybrid retrieval strategies.\n",
"\n",
- "**Key Features:**\n",
+ "### šÆ What You Will Learn\n",
"\n",
- "- **Real-World Data**: Uses actual data sources via Local Files, GitHub Repositories, Web Scraping, and RSS Feeds (NO mock data)\n",
- "- **Complete Pipeline**: From data ingestion to LLM-powered question answering\n",
- "- **Hybrid Retrieval**: Combines vector similarity search with knowledge graph traversal\n",
- "- **Multi-hop Reasoning**: Follows relationships across the graph for deeper context\n",
+ "1. **Orchestration**: Using `Semantica` core to manage the full data lifecycle.\n",
+ "2. **Advanced Analysis**: Using `GraphAnalyzer`, `CentralityCalculator`, and `CommunityDetector` to understand data structure.\n",
+ "3. **Ontology Generation**: Automatically deriving a schema (classes & properties) from unstructured text.\n",
+ "4. **Hybrid Search**: Combining Vector Similarity with Graph Traversal for superior context.\n",
"\n",
- "**Documentation**: [API Reference](https://semantica.readthedocs.io/concepts/) ⢠[GraphRAG Guide](https://semantica.readthedocs.io/concepts/)\n",
- "\n",
- "### What You'll Learn\n",
- "\n",
- "- How to ingest real-world data from multiple sources (MCP, web, feeds)\n",
- "- How to build knowledge graphs from unstructured text\n",
- "- How to implement hybrid search combining vectors and graphs\n",
- "- How to use ContextRetriever for intelligent context expansion\n",
- "- How to integrate LLMs with GraphRAG for question answering\n",
- "- How to visualize and export knowledge graphs\n",
- "\n",
- "### Pipeline Overview\n",
- "\n",
- "- **Real-World Data**: Uses actual data sources via Local Files, GitHub Repositories, Web Scraping, and RSS Feeds (NO mock data)\n",
- "\n",
- "---\n",
- "\n",
- "## Installation\n",
- "\n",
- "Install Semantica from PyPI:\n",
- "\n",
- "```bash\n",
- "pip install semantica\n",
- "\n",
- "```\n"
+ "---"
]
},
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
- "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
- "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
- "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
- "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
- "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
- "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
- "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
- "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
- "!pip install -qU semantica\n"
+ "# š ļø Dependencies\n",
+ "!pip install -qU semantica networkx matplotlib plotly pandas"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Step 1: Setup and Import Semantica Modules\n",
+ "## 1. Professional Setup & Imports\n",
"\n",
- "Import all necessary Semantica modules for the complete GraphRAG pipeline. This includes modules for ingestion, parsing, extraction, graph building, embeddings, vector storage, context retrieval, and more.\n",
- "\n"
+ "We organize imports by their functional role in the Semantica ecosystem."
]
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": null,
"metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
š§ Semantica - š Current Progress
| Status | Action | Module | Submodule | File | Time |
|---|
| ā
| Semantica is processing | ā³ core | LifecycleManager | - | 3.34s |
| ā
| Semantica is ingesting | š„ ingest | FileIngestor | sample_graphrag_paper.txt | 0.00s |
| ā
| Semantica is ingesting | š„ ingest | RepoIngestor | requests | 4.50s |
| ā | Semantica is ingesting | š„ ingest | WebIngestor | tool-use | 0.26s |
| ā
| Semantica is ingesting | š„ ingest | WebIngestor | #retrieval | 2.09s |
| ā
| Semantica is ingesting | š„ ingest | FeedIngestor | feed | 0.54s |
| ā
| Semantica is ingesting | š„ ingest | FeedIngestor | rss.xml | 1.19s |
| ā
| Semantica is parsing | š parse | DocumentParser | sample_graphrag_paper.txt | 0.01s |
"
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Configuration and framework initialized\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
- "from semantica.core import ConfigManager, Semantica\n",
- "\n",
- "# Initialize configuration manager\n",
- "config_manager = ConfigManager()\n",
- "\n",
- "# Optionally load from file or dictionary\n",
- "# config = config_manager.load_from_file(\"config.yaml\")\n",
- "# Or use defaults\n",
- "config = config_manager.load_from_dict({})\n",
- "\n",
- "# Initialize Semantica framework\n",
- "framework = Semantica(config=config)\n",
- "framework.initialize()\n",
- "\n",
- "print(\"Configuration and framework initialized\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 2: Ingest Real-World Data from Multiple Sources\n",
- "\n",
- "1. **Local Files**: Ingest local documents and text files\n",
- "2. **Repositories**: Ingest code and documentation from GitHub repositories\n",
- "3. **Web Sources**: Scrape real web content from news sites and documentation\n",
- "4. **RSS Feeds**: Ingest real RSS/Atom feeds from news sources\n",
- "\n",
- "### 2.1: Ingest Local Files and Repositories\n",
- "\n",
- "Ingest data from local file systems and remote code repositories.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Ingesting local file from c:\\Users\\Mohd Kaif\\semantica\\cookbook\\use_cases\\advanced_rag\\data\\sample_graphrag_paper.txt...\n",
- " Ingested 1 file document\n",
- "\n",
- "Ingesting repository https://github.com/psf/requests...\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Error analyzing commits: Cmd('git') failed due to: exit code(128)\n",
- " cmdline: git diff --numstat --no-renames --raw 420d16bc7ef326f7b65f90e4644adc0f6a0e1d44 70298332899f25826e35e42f8d83425124f755a5 --\n",
- " stderr: 'fatal: bad object 420d16bc7ef326f7b65f90e4644adc0f6a0e1d44'\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- " Ingested 1 repo document\n"
- ]
- }
- ],
- "source": [
- "from semantica.ingest import FileIngestor, RepoIngestor, WebIngestor, FeedIngestor\n",
"import os\n",
+ "import json\n",
+ "import pandas as pd\n",
"\n",
- "all_documents = []\n",
+ "# --- Core Framework ---\n",
+ "from semantica.core import Semantica, ConfigManager\n",
"\n",
- "# 1. Ingest Local File\n",
- "file_ingestor = FileIngestor()\n",
- "# Ensure the sample directory exists\n",
- "sample_dir = os.path.join(os.getcwd(), \"data\")\n",
- "sample_file = os.path.join(sample_dir, \"sample_graphrag_paper.txt\")\n",
- "\n",
- "print(f\"Ingesting local file from {sample_file}...\")\n",
- "if os.path.exists(sample_file):\n",
- " file_docs = file_ingestor.ingest_file(sample_file)\n",
- " if isinstance(file_docs, list):\n",
- " all_documents.extend(file_docs)\n",
- " print(f\" Ingested {len(file_docs)} file document(s)\")\n",
- " else:\n",
- " all_documents.append(file_docs)\n",
- " print(f\" Ingested 1 file document\")\n",
- "else:\n",
- " print(f\" Warning: Sample file not found at {sample_file}. Please ensure it exists.\")\n",
- "\n",
- "# 2. Ingest GitHub Repository\n",
- "repo_ingestor = RepoIngestor()\n",
- "repo_url = \"https://github.com/psf/requests\" # Standard library, good example\n",
- "print(f\"\\nIngesting repository {repo_url}...\")\n",
- "\n",
- "try:\n",
- " # Using depth=1 and filtering for texts to keep it light\n",
- " repo_docs = repo_ingestor.ingest_repository(\n",
- " repo_url, \n",
- " max_depth=1, \n",
- " include_extensions=[\"md\", \"rst\", \"txt\"]\n",
- " )\n",
- " \n",
- " if isinstance(repo_docs, list):\n",
- " all_documents.extend(repo_docs)\n",
- " print(f\" Ingested {len(repo_docs)} repo document(s)\")\n",
- " else:\n",
- " all_documents.append(repo_docs)\n",
- " print(f\" Ingested 1 repo document\")\n",
- "except Exception as e:\n",
- " print(f\" Error ingesting repo: {e}\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 2.3: Ingest Data from Web Sources\n",
- "\n",
- "Scrape real web content from news sites, documentation, and articles.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Scraping https://docs.anthropic.com/en/docs/build-with-claude/tool-use...\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "URL https://docs.anthropic.com/en/docs/build-with-claude/tool-use blocked by robots.txt\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- " Error scraping https://docs.anthropic.com/en/docs/build-with-claude/tool-use: URL blocked by robots.txt: https://docs.anthropic.com/en/docs/build-with-claude/tool-use\n",
- "Scraping https://python.langchain.com/docs/concepts/#retrieval...\n",
- " Scraped 1 document(s)\n",
- "\n",
- "Total documents from web: 1\n",
- "Total documents so far: 3\n"
- ]
- }
- ],
- "source": [
- "web_ingestor = WebIngestor()\n",
- "\n",
- "# Real documentation and article sources\n",
- "web_sources = [\n",
- " \"https://docs.anthropic.com/en/docs/build-with-claude/tool-use\", # Relevant to agents\n",
- " \"https://python.langchain.com/docs/concepts/#retrieval\" # Relevant to RAG\n",
- "]\n",
- "\n",
- "web_documents = []\n",
- "for url in web_sources:\n",
- " try:\n",
- " print(f\"Scraping {url}...\")\n",
- " # Use ingest_url instead of ingest\n",
- " docs = web_ingestor.ingest_url(url)\n",
- " if isinstance(docs, list):\n",
- " web_documents.extend(docs)\n",
- " else:\n",
- " web_documents.append(docs)\n",
- " print(f\" Scraped {len(docs) if isinstance(docs, list) else 1} document(s)\")\n",
- " except Exception as e:\n",
- " print(f\" Error scraping {url}: {e}\")\n",
- "\n",
- "all_documents.extend(web_documents)\n",
- "print(f\"\\nTotal documents from web: {len(web_documents)}\")\n",
- "print(f\"Total documents so far: {len(all_documents)}\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 2.4: Ingest Data from RSS Feeds\n",
- "\n",
- "Ingest real RSS/Atom feeds from news sources.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Fetching feed https://techcrunch.com/category/artificial-intelligence/feed/...\n",
- " Fetched 20 feed item(s)\n",
- "Fetching feed http://feeds.bbci.co.uk/news/technology/rss.xml...\n",
- " Fetched 48 feed item(s)\n",
- "\n",
- "Total documents from feeds: 2\n",
- "Total documents collected: 5\n"
- ]
- }
- ],
- "source": [
- "feed_ingestor = FeedIngestor()\n",
- "\n",
- "# Real Tech News Feeds\n",
- "feed_urls = [\n",
- " \"https://techcrunch.com/category/artificial-intelligence/feed/\",\n",
- " \"http://feeds.bbci.co.uk/news/technology/rss.xml\"\n",
- "]\n",
- "\n",
- "feed_documents = []\n",
- "for feed_url in feed_urls:\n",
- " try:\n",
- " print(f\"Fetching feed {feed_url}...\")\n",
- " # Use ingest_feed instead of ingest\n",
- " feeds = feed_ingestor.ingest_feed(feed_url)\n",
- " if isinstance(feeds, list):\n",
- " feed_documents.extend(feeds)\n",
- " else:\n",
- " feed_documents.append(feeds)\n",
- " # Check for feed items\n",
- " count = len(feeds.items) if hasattr(feeds, 'items') else (len(feeds) if isinstance(feeds, list) else 1)\n",
- " print(f\" Fetched {count} feed item(s)\")\n",
- " except Exception as e:\n",
- " print(f\" Error fetching feed {feed_url}: {e}\")\n",
- "\n",
- "all_documents.extend(feed_documents)\n",
- "print(f\"\\nTotal documents from feeds: {len(feed_documents)}\")\n",
- "print(f\"Total documents collected: {len(all_documents)}\")\n",
- "\n",
- "if len(all_documents) == 0:\n",
- " print(\"\\nNo documents collected. Please configure MCP servers, web URLs, or RSS feeds above.\")\n",
- " print(\"For this demonstration, we'll continue with the pipeline structure.\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 3: Document Processing Pipeline\n",
- "\n",
- "Process the ingested documents: parse, split, and normalize the text for extraction.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Parsing documents...\n",
- "Warning: Unknown document format: \n",
- "Parsed 70 documents\n"
- ]
- }
- ],
- "source": [
- "from semantica.parse import DocumentParser\n",
- "\n",
- "document_parser = DocumentParser()\n",
- "\n",
- "parsed_documents = []\n",
- "\n",
- "print(\"Parsing documents...\")\n",
- "for doc in all_documents:\n",
- " try:\n",
- " # Handle different document objects\n",
- " # 1. FileObject (from FileIngestor) -> Parse using path\n",
- " if hasattr(doc, 'path') and os.path.exists(doc.path):\n",
- " parsed = document_parser.parse(doc.path)\n",
- " if isinstance(parsed, list):\n",
- " parsed_documents.extend(parsed)\n",
- " else:\n",
- " parsed_documents.append(parsed)\n",
- " \n",
- " # 2. WebContent (from WebIngestor) -> Already has text\n",
- " elif hasattr(doc, 'text') and hasattr(doc, 'url'):\n",
- " # It's already parsed content, just use it\n",
- " # Ensure it has a 'content' attribute for downstream steps\n",
- " if not hasattr(doc, 'content') and hasattr(doc, 'text'):\n",
- " doc.content = doc.text\n",
- " parsed_documents.append(doc)\n",
- " \n",
- " # 3. FeedData (from FeedIngestor) -> Has items list\n",
- " elif hasattr(doc, 'items') and isinstance(doc.items, list):\n",
- " for item in doc.items:\n",
- " # Feed items usually have .content, .description, or .summary\n",
- " # We'll normalize to .content\n",
- " if hasattr(item, 'content'):\n",
- " pass # Good\n",
- " elif hasattr(item, 'description'):\n",
- " item.content = item.description\n",
- " elif hasattr(item, 'summary'):\n",
- " item.content = item.summary\n",
- " parsed_documents.append(item)\n",
- "\n",
- " # 4. Standard path string\n",
- " elif isinstance(doc, str) and os.path.exists(doc):\n",
- " parsed = document_parser.parse(doc)\n",
- " if isinstance(parsed, list):\n",
- " parsed_documents.extend(parsed)\n",
- " else:\n",
- " parsed_documents.append(parsed)\n",
- " \n",
- " else:\n",
- " # Fallback or already a usable document object\n",
- " if hasattr(doc, 'content') or hasattr(doc, 'text'):\n",
- " if not hasattr(doc, 'content'): doc.content = doc.text\n",
- " parsed_documents.append(doc)\n",
- " else:\n",
- " print(f\"Warning: Unknown document format: {type(doc)}\")\n",
- "\n",
- " except Exception as e:\n",
- " print(f\"Error parsing document: {e}\")\n",
- " continue\n",
- "\n",
- "print(f\"Parsed {len(parsed_documents)} documents\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 3.2: Split Documents Using Dual Chunking Strategy\n",
- "\n",
- "For GraphRAG, we use **two different chunking methods** optimized for different stores:\n",
- "\n",
- "**For Vector Store** (semantic similarity search):\n",
- "- **Semantic Chunking**: Uses embeddings to find natural semantic boundaries\n",
- "- Better for vector similarity search and retrieval\n",
- "\n",
- "**For Graph Store** (knowledge structure preservation):\n",
- "- **Entity-Aware Chunking**: Preserves entity boundaries (prevents splitting entities)\n",
- "- **Relation-Aware Chunking**: Preserves relationship triplets (keeps subject-predicate-object together)\n",
- "- **Graph-Based Chunking**: Uses existing graph structure for optimal chunking\n",
- "\n",
- "We'll create chunks optimized for each store type.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Extracting entities, relationships, and triplets...\n"
- ]
- },
- {
- "ename": "NameError",
- "evalue": "name 'normalized_documents' is not defined",
- "output_type": "error",
- "traceback": [
- "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)",
- "Cell \u001b[1;32mIn[7], line 13\u001b[0m\n\u001b[0;32m 10\u001b[0m flat_relationships \u001b[38;5;241m=\u001b[39m []\n\u001b[0;32m 11\u001b[0m flat_triplets \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m---> 13\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m doc \u001b[38;5;129;01min\u001b[39;00m \u001b[43mnormalized_documents\u001b[49m:\n\u001b[0;32m 14\u001b[0m text \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mstr\u001b[39m(doc\u001b[38;5;241m.\u001b[39mcontent) \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(doc, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcontent\u001b[39m\u001b[38;5;124m'\u001b[39m) \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28mstr\u001b[39m(doc)\n\u001b[0;32m 16\u001b[0m entities \u001b[38;5;241m=\u001b[39m ner\u001b[38;5;241m.\u001b[39mextract_entities(text)\n",
- "\u001b[1;31mNameError\u001b[0m: name 'normalized_documents' is not defined"
- ]
- }
- ],
- "source": [
- "from semantica.semantic_extract import NamedEntityRecognizer, RelationExtractor, TripletExtractor\n",
- "\n",
- "print(\"Extracting entities, relationships, and triplets...\")\n",
- "\n",
- "ner = NamedEntityRecognizer()\n",
- "rel_extractor = RelationExtractor()\n",
- "triplet_extractor = TripletExtractor()\n",
- "\n",
- "flat_entities = []\n",
- "flat_relationships = []\n",
- "flat_triplets = []\n",
- "\n",
- "for doc in normalized_documents:\n",
- " text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n",
- " \n",
- " entities = ner.extract_entities(text)\n",
- " flat_entities.extend(entities if isinstance(entities, list) else [entities])\n",
- " \n",
- " relations = rel_extractor.extract_relations(text, entities=entities)\n",
- " flat_relationships.extend(relations if isinstance(relations, list) else [relations])\n",
- " \n",
- " triplets = triplet_extractor.extract_triplets(text, entities=entities, relationships=relations)\n",
- " flat_triplets.extend(triplets if isinstance(triplets, list) else [triplets])\n",
- "\n",
- "print(f\"Extracted {len(flat_entities)} entities\")\n",
- "print(f\"Extracted {len(flat_relationships)} relationships\")\n",
- "print(f\"Extracted {len(flat_triplets)} triplets\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 3.2.1: Graph-Based Chunking (Iterative Refinement)\n",
- "\n",
- "After building the knowledge graph, we can use graph-based chunking to refine chunks based on graph structure. This is useful for re-chunking or optimizing existing chunks.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "graph_chunker = GraphBasedChunker(\n",
- " chunk_size=1000,\n",
- " chunk_overlap=200,\n",
- " strategy=\"community\",\n",
- " algorithm=\"louvain\"\n",
+ "# --- Knowledge Graph & Analytics ---\n",
+ "from semantica.kg import (\n",
+ " GraphBuilder, \n",
+ " GraphAnalyzer, \n",
+ " CentralityCalculator, \n",
+ " CommunityDetector\n",
")\n",
"\n",
- "print(\"Graph-based chunker initialized\")\n"
+ "# --- Ontology & Schema ---\n",
+ "from semantica.ontology import OntologyGenerator\n",
+ "\n",
+ "# --- Visualization ---\n",
+ "from semantica.visualization import KGVisualizer\n",
+ "\n",
+ "print(\"ā
Semantica Enterprise Modules Loaded.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### 3.3: Normalize Text\n",
+ "## 2. Realistic Data Simulation\n",
"\n",
- "Clean and normalize text for better extraction quality.\n",
- "\n"
+ "Instead of \"Hello World\", let's use a dense technical text. We'll simulate ingesting a research paper abstract on **\"Graph Neural Networks for Knowledge Graph Completion\"**. This provides rich entities (Models, Metrics, Concepts) and relationships."
]
},
{
@@ -557,43 +84,40 @@
"metadata": {},
"outputs": [],
"source": [
- "from semantica.normalize import TextNormalizer\n",
+ "research_paper_text = \"\"\"\n",
+ "Title: Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs.\n",
+ "Abstract:\n",
+ "Knowledge Graphs (KGs) are collections of structured facts, often represented as triplets (head, relation, tail).\n",
+ "However, most real-world KGs like Freebase, DBPedia, and YAGO are incomplete.\n",
+ "Link Prediction is the task of inferring missing facts based on existing ones.\n",
+ "We propose a new model, Graph Attention Network for Knowledge Graphs (GAT-KG), which utilizes a multi-head attention mechanism.\n",
+ "Unlike TransE or DistMult which rely on translational assumptions, GAT-KG captures the structural information of the local neighborhood.\n",
+ "Our encoder aggregates messages from neighbors, assigning different attention weights to different relationships.\n",
+ "The decoder uses a standard ConvKB layer to score the plausibility of triplets.\n",
+ "We evaluate our method on standard benchmarks: FB15k-237 and WN18RR.\n",
+ "Results show that GAT-KG outperforms state-of-the-art baselines like RotatE and TuckER by 5% in Hits@10.\n",
+ "Ablation studies confirm that the attention mechanism is crucial for handling sparse graphs.\n",
+ "\"\"\"\n",
"\n",
- "text_normalizer = TextNormalizer()\n",
+ "data_dir = os.path.join(os.getcwd(), \"data\")\n",
+ "os.makedirs(data_dir, exist_ok=True)\n",
+ "source_path = os.path.join(data_dir, \"gnn_research.txt\")\n",
"\n",
- "print(\"Normalizing vector store chunks...\")\n",
- "normalized_vector_chunks = []\n",
- "for chunk in vector_store_chunks:\n",
- " normalized = text_normalizer.normalize_text(chunk)\n",
- " if isinstance(normalized, list):\n",
- " normalized_vector_chunks.extend(normalized)\n",
- " else:\n",
- " normalized_vector_chunks.append(normalized)\n",
+ "with open(source_path, \"w\", encoding=\"utf-8\") as f:\n",
+ " f.write(research_paper_text)\n",
"\n",
- "print(\"Normalizing graph store chunks...\")\n",
- "normalized_graph_chunks = []\n",
- "for chunk in graph_store_chunks:\n",
- " normalized = text_normalizer.normalize_text(chunk)\n",
- " if isinstance(normalized, list):\n",
- " normalized_graph_chunks.extend(normalized)\n",
- " else:\n",
- " normalized_graph_chunks.append(normalized)\n",
- "\n",
- "normalized_documents = normalized_vector_chunks + normalized_graph_chunks\n",
- "print(f\"Normalized {len(normalized_documents)} chunks\")\n",
- "print(f\" Vector store chunks: {len(normalized_vector_chunks)}\")\n",
- "print(f\" Graph store chunks: {len(normalized_graph_chunks)}\")\n",
- "print(\"Document processing complete!\")\n"
+ "print(f\"š Research paper saved to: {source_path}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Step 4: Semantic Extraction\n",
+ "## 3. Configuration Management\n",
"\n",
- "Extract entities, relationships, and triplets from the processed documents. This is the foundation for building the knowledge graph.\n",
- "\n"
+ "We configure the system for **precision**. \n",
+ "* `entity_resolution_strategy: fuzzy`: Handles minor typos (e.g., \"Conv-KB\" vs \"ConvKB\").\n",
+ "* `extraction`: We specify a capable LLM model to ensure we catch complex technical terms."
]
},
{
@@ -602,1071 +126,36 @@
"metadata": {},
"outputs": [],
"source": [
- "from semantica.semantic_extract import build as extract_build\n",
- "\n",
- "print(\"Extracting entities, relationships, and triplets...\")\n",
- "\n",
- "extraction_result = extract_build(\n",
- " text=[str(doc.content) if hasattr(doc, 'content') else str(doc) for doc in normalized_documents],\n",
- " extract_entities=True,\n",
- " extract_relations=True,\n",
- " extract_triplets=True\n",
- ")\n",
- "\n",
- "flat_entities = extraction_result.get('entities', [])\n",
- "flat_relationships = extraction_result.get('relationships', [])\n",
- "flat_triplets = extraction_result.get('triplets', [])\n",
- "\n",
- "print(f\"Extracted {len(flat_entities)} entities\")\n",
- "print(f\"Extracted {len(flat_relationships)} relationships\")\n",
- "print(f\"Extracted {len(flat_triplets)} triplets\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": []
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "print(f\"\\nExtraction Summary:\")\n",
- "print(f\"Entities: {len(flat_entities)}\")\n",
- "print(f\"Relationships: {len(flat_relationships)}\")\n",
- "print(f\"Triplets: {len(flat_triplets)}\")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 5: Knowledge Graph Construction\n",
- "\n",
- "Build the knowledge graph from extracted entities and relationships. Apply quality assurance measures including deduplication and entity resolution.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": []
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from semantica.kg.methods import build_kg, resolve_entities, deduplicate_graph\n",
- "\n",
- "print(\"Deduplicating and resolving entities...\")\n",
- "\n",
- "deduplicated_result = deduplicate_graph(flat_entities, method=\"default\")\n",
- "deduplicated_entities = deduplicated_result.get('entities', flat_entities)\n",
- "\n",
- "resolved_result = resolve_entities(deduplicated_entities, method=\"fuzzy\")\n",
- "resolved_entities = resolved_result.get('entities', deduplicated_entities)\n",
- "\n",
- "print(f\"Deduplicated: {len(flat_entities)} ā {len(deduplicated_entities)} entities\")\n",
- "print(f\"Resolved: {len(deduplicated_entities)} ā {len(resolved_entities)} entities\")\n",
- "\n",
- "print(\"Building knowledge graph...\")\n",
- "\n",
- "kg_result = build_kg(\n",
- " sources=[{\n",
- " 'entities': resolved_entities,\n",
- " 'relationships': flat_relationships,\n",
- " 'triplets': flat_triplets\n",
- " }],\n",
- " method=\"default\",\n",
- " merge_entities=True,\n",
- " resolve_conflicts=True\n",
- ")\n",
- "\n",
- "knowledge_graph = kg_result.get('graph')\n",
- "\n",
- "print(f\"Knowledge graph built!\")\n",
- "print(f\"Nodes: {knowledge_graph.number_of_nodes()}\")\n",
- "print(f\"Edges: {knowledge_graph.number_of_edges()}\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 5.2: Analyze Knowledge Graph\n",
- "\n",
- "Analyze the graph structure to understand its properties and quality.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 5.3: Refine Chunks Using Graph-Based Chunking\n",
- "\n",
- "After building the knowledge graph, we can use graph-based chunking to refine chunks based on graph structure. This creates chunks that align with graph communities or centrality.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "if knowledge_graph and knowledge_graph.number_of_nodes() > 0:\n",
- " print(\"Refining chunks using graph-based chunking...\")\n",
- " \n",
- " refined_chunks = []\n",
- " \n",
- " for i, doc in enumerate(parsed_documents[:5]):\n",
- " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n",
- " if doc_text.strip():\n",
- " try:\n",
- " graph_chunks = graph_chunker.chunk(\n",
- " doc_text,\n",
- " graph=knowledge_graph\n",
- " )\n",
- " \n",
- " if isinstance(graph_chunks, list):\n",
- " for chunk in graph_chunks:\n",
- " if hasattr(chunk, 'metadata'):\n",
- " chunk.metadata['chunking_method'] = 'graph_based'\n",
- " chunk.metadata['source_doc'] = i\n",
- " refined_chunks.extend(graph_chunks)\n",
- " else:\n",
- " refined_chunks.append(graph_chunks)\n",
- " except Exception as e:\n",
- " print(f\"Note: Graph-based chunking not available for doc {i}, using original chunks\")\n",
- " continue\n",
- " \n",
- " if refined_chunks:\n",
- " print(f\"Created {len(refined_chunks)} graph-based refined chunks\")\n",
- " print(\"These chunks are aligned with graph communities/structure\")\n",
- " else:\n",
- " print(\"Using original entity/relation-aware chunks\")\n",
- "else:\n",
- " print(\"Graph is empty, using original entity/relation-aware chunks\")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from semantica.kg.methods import analyze_graph, calculate_centrality, detect_communities, analyze_connectivity\n",
- "\n",
- "print(\"Analyzing knowledge graph...\")\n",
- "\n",
- "graph_metrics = analyze_graph(knowledge_graph, method=\"default\")\n",
- "print(f\"\\nGraph Metrics:\")\n",
- "print(f\"Nodes: {graph_metrics.get('nodes', 0)}\")\n",
- "print(f\"Edges: {graph_metrics.get('edges', 0)}\")\n",
- "print(f\"Density: {graph_metrics.get('density', 0):.4f}\")\n",
- "\n",
- "connectivity = analyze_connectivity(knowledge_graph, method=\"default\")\n",
- "print(f\"\\nConnectivity:\")\n",
- "print(f\"Connected Components: {connectivity.get('connected_components', 0)}\")\n",
- "print(f\"Largest Component Size: {connectivity.get('largest_component_size', 0)}\")\n",
- "\n",
- "if knowledge_graph.number_of_nodes() > 0:\n",
- " centrality = calculate_centrality(knowledge_graph, method='pagerank')\n",
- " top_nodes = sorted(centrality.items(), key=lambda x: x[1], reverse=True)[:5]\n",
- " print(f\"\\nTop 5 Central Nodes (PageRank):\")\n",
- " for node, score in top_nodes:\n",
- " print(f\" {node}: {score:.4f}\")\n",
- " \n",
- " communities = detect_communities(knowledge_graph, method='louvain')\n",
- " print(f\"\\nCommunities Detected: {len(communities)}\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 5.3: Store Knowledge Graph (Optional)\n",
- "\n",
- "Optionally persist the knowledge graph to a graph database for long-term storage.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Optional: Store graph in persistent graph database\n",
- "# Uncomment to use Neo4j\n",
- "# graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n",
- "# graph_store.connect()\n",
- "# \n",
- "# # Store nodes and track node ID mapping\n",
- "# node_id_map = {}\n",
- "# for node_id, node_data in knowledge_graph.nodes(data=True):\n",
- "# labels = [node_data.get('type', 'Entity')]\n",
- "# properties = {k: v for k, v in node_data.items() if k != 'type'}\n",
- "# created_node = graph_store.create_node(labels, properties)\n",
- "# node_id_map[node_id] = created_node.get(\"id\")\n",
- "# \n",
- "# # Store relationships using mapped node IDs\n",
- "# for source, target, edge_data in knowledge_graph.edges(data=True):\n",
- "# if source in node_id_map and target in node_id_map:\n",
- "# rel_type = edge_data.get('type', 'RELATED_TO')\n",
- "# properties = {k: v for k, v in edge_data.items() if k != 'type'}\n",
- "# graph_store.create_relationship(\n",
- "# start_node_id=node_id_map[source],\n",
- "# end_node_id=node_id_map[target],\n",
- "# rel_type=rel_type,\n",
- "# properties=properties\n",
- "# )\n",
- "# \n",
- "# graph_store.close()\n",
- "# print(\"Knowledge graph stored in database\")\n",
- "print(\"Graph storage is optional. The in-memory graph is ready for GraphRAG.\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 6: Embedding Generation\n",
- "\n",
- "Generate vector embeddings for documents, entities, and relationships. These embeddings enable semantic search and similarity calculations.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from semantica.embeddings import EmbeddingGenerator\n",
- "\n",
- "embedding_generator = EmbeddingGenerator()\n",
- "\n",
- "print(\"Generating embeddings for vector store chunks (semantic chunks)...\")\n",
- "vector_chunk_embeddings = {}\n",
- "\n",
- "for i, chunk in enumerate(normalized_vector_chunks):\n",
- " text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n",
- " if text.strip():\n",
- " embedding = embedding_generator.generate_embeddings(text, data_type=\"text\")\n",
- " vector_chunk_embeddings[f\"vector_chunk_{i}\"] = {\n",
- " 'embedding': embedding,\n",
- " 'text': text,\n",
- " 'chunking_method': 'semantic',\n",
- " 'store_type': 'vector'\n",
- " }\n",
- "\n",
- "print(f\"Generated {len(vector_chunk_embeddings)} vector store chunk embeddings\")\n",
- "\n",
- "print(\"\\nGenerating embeddings for graph store chunks (graph-aware chunks)...\")\n",
- "graph_chunk_embeddings = {}\n",
- "\n",
- "for i, chunk in enumerate(normalized_graph_chunks):\n",
- " text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n",
- " if text.strip():\n",
- " embedding = embedding_generator.generate_embeddings(text, data_type=\"text\")\n",
- " graph_chunk_embeddings[f\"graph_chunk_{i}\"] = {\n",
- " 'embedding': embedding,\n",
- " 'text': text,\n",
- " 'chunking_method': 'graph_aware',\n",
- " 'store_type': 'graph'\n",
- " }\n",
- "\n",
- "print(f\"Generated {len(graph_chunk_embeddings)} graph store chunk embeddings\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 6.2: Generate Entity Embeddings\n",
- "\n",
- "Generate embeddings for entities to enable entity-based semantic search.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Generate embeddings for entities\n",
- "print(\"Generating embeddings for entities...\")\n",
- "entity_embeddings = {}\n",
- "\n",
- "for entity in resolved_entities[:100]: # Limit to first 100 for demo\n",
- " if isinstance(entity, dict):\n",
- " entity_text = entity.get('text', entity.get('name', str(entity)))\n",
- " else:\n",
- " entity_text = str(entity)\n",
- " \n",
- " if entity_text.strip():\n",
- " embedding = embedding_generator.generate_embeddings(entity_text, data_type=\"text\")\n",
- " entity_id = entity.get('id', entity.get('text', str(entity))) if isinstance(entity, dict) else str(entity)\n",
- " entity_embeddings[entity_id] = {\n",
- " 'embedding': embedding,\n",
- " 'text': entity_text\n",
- " }\n",
- "\n",
- "print(f\"Generated {len(entity_embeddings)} entity embeddings\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 7: Vector Store Setup\n",
- "\n",
- "Store embeddings in a vector store for fast similarity search and retrieval.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from semantica.vector_store import VectorStore, HybridSearch\n",
- "from semantica.graph_store import GraphStore\n",
- "\n",
- "vector_store = VectorStore()\n",
- "\n",
- "vectors = []\n",
- "metadata_list = []\n",
- "ids = []\n",
- "\n",
- "print(\"Storing semantic chunks in vector store...\")\n",
- "for chunk_id, chunk_data in vector_chunk_embeddings.items():\n",
- " vectors.append(chunk_data['embedding'])\n",
- " metadata_list.append({\n",
- " 'type': 'chunk',\n",
- " 'chunking_method': 'semantic',\n",
- " 'store_type': 'vector',\n",
- " 'text': chunk_data['text'][:200]\n",
- " })\n",
- " ids.append(chunk_id)\n",
- "\n",
- "for entity_id, entity_data in entity_embeddings.items():\n",
- " vectors.append(entity_data['embedding'])\n",
- " metadata_list.append({'type': 'entity', 'text': entity_data['text']})\n",
- " ids.append(entity_id)\n",
- "\n",
- "if vectors:\n",
- " vector_store.store(vectors=vectors, metadata=metadata_list, ids=ids)\n",
- " print(f\"Stored {len(vectors)} vectors in vector store\")\n",
- " print(f\" Semantic chunks: {len(vector_chunk_embeddings)}\")\n",
- " print(f\" Entities: {len(entity_embeddings)}\")\n",
- "else:\n",
- " print(\"No vectors to store\")\n",
- "\n",
- "print(\"\\nStoring graph-aware chunks in graph store...\")\n",
- "# Option 1: Neo4j (requires Neo4j server running)\n",
- "graph_store = GraphStore(backend=\"neo4j\", uri=\"bolt://localhost:7687\", user=\"neo4j\", password=\"password\")\n",
- "graph_store.connect()\n",
- "\n",
- "for i, chunk in enumerate(graph_store_chunks):\n",
- " chunk_text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n",
- " if chunk_text.strip():\n",
- " chunk_metadata = {\n",
- " 'chunking_method': 'graph_aware',\n",
- " 'store_type': 'graph',\n",
- " 'text': chunk_text[:500]\n",
- " }\n",
- " if hasattr(chunk, 'metadata'):\n",
- " if chunk.metadata.get('entities'):\n",
- " chunk_metadata['entities'] = chunk.metadata['entities']\n",
- " if chunk.metadata.get('relationships'):\n",
- " chunk_metadata['relationships'] = chunk.metadata['relationships']\n",
- " \n",
- " graph_store.create_node(\n",
- " labels=['Chunk'],\n",
- " properties={\n",
- " 'id': f\"graph_chunk_{i}\",\n",
- " **chunk_metadata\n",
- " }\n",
- " )\n",
- "\n",
- "print(f\"Stored {len(graph_store_chunks)} graph-aware chunks in graph store\")\n",
- "graph_store.close()\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 8: Hybrid Search Implementation\n",
- "\n",
- "Implement hybrid search that combines vector similarity search with knowledge graph traversal for enhanced retrieval.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from semantica.vector_store import HybridSearch\n",
- "\n",
- "hybrid_search = HybridSearch(vector_store=vector_store)\n",
- "\n",
- "def perform_hybrid_search(query: str, top_k: int = 10):\n",
- " query_embedding = embedding_generator.generate_embeddings(query, data_type=\"text\")\n",
- " vector_results = vector_store.search(\n",
- " query_vector=query_embedding,\n",
- " top_k=top_k * 2\n",
- " )\n",
- " \n",
- " # Graph-based search (if query contains entity mentions)\n",
- " graph_results = []\n",
- " if knowledge_graph.number_of_nodes() > 0:\n",
- " # Extract entities from query\n",
- " query_entities = ner_extractor.extract(query)\n",
- " if query_entities:\n",
- " # Find related nodes in graph\n",
- " for entity in query_entities:\n",
- " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n",
- " # Search for entity in graph\n",
- " for node in knowledge_graph.nodes():\n",
- " if entity_text.lower() in str(node).lower():\n",
- " # Get neighbors\n",
- " neighbors = list(knowledge_graph.neighbors(node))\n",
- " for neighbor in neighbors[:5]: # Limit neighbors\n",
- " graph_results.append({\n",
- " 'id': f\"graph_{node}_{neighbor}\",\n",
- " 'content': f\"{node} -> {neighbor}\",\n",
- " 'score': 0.7, # Graph relevance score\n",
- " 'source': 'graph'\n",
- " })\n",
- " \n",
- " # Combine and rank results using hybrid search\n",
- " all_results = vector_results + graph_results\n",
- " \n",
- " # Use hybrid search ranker\n",
- " if all_results:\n",
- " ranked_results = hybrid_search.ranker.rank([all_results], top_k=top_k)\n",
- " return ranked_results[:top_k]\n",
- " \n",
- " return []\n",
- "\n",
- "# Test hybrid search\n",
- "test_query = \"artificial intelligence and machine learning\"\n",
- "print(f\"Testing hybrid search with query: '{test_query}'\")\n",
- "search_results = perform_hybrid_search(test_query, top_k=5)\n",
- "\n",
- "print(f\"Search Results ({len(search_results)}):\")\n",
- "for i, result in enumerate(search_results[:5], 1):\n",
- " print(f\"\\n{i}. Score: {result.get('score', 0):.4f}\")\n",
- " print(f\" Source: {result.get('source', 'unknown')}\")\n",
- " content = result.get('content', result.get('text', 'N/A'))\n",
- " print(f\" Content: {content[:100]}...\")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from semantica.context import ContextRetriever, ContextGraphBuilder, AgentMemory\n",
- "from semantica.context.methods import retrieve_context, build_context_graph\n",
- "\n",
- "agent_memory = AgentMemory(\n",
- " vector_store=vector_store,\n",
- " knowledge_graph=knowledge_graph\n",
- ")\n",
- "\n",
- "context_retriever = ContextRetriever(\n",
- " memory_store=agent_memory,\n",
- " knowledge_graph=knowledge_graph,\n",
- " vector_store=vector_store,\n",
- " use_graph_expansion=True,\n",
- " max_expansion_hops=2,\n",
- " hybrid_alpha=0.5\n",
- ")\n",
- "\n",
- "context_graph_builder = ContextGraphBuilder()\n",
- "\n",
- "print(\"Context retrieval system initialized\")\n",
- "print(f\"Graph expansion: Enabled (max {context_retriever.max_expansion_hops} hops)\")\n",
- "print(f\"Hybrid alpha: {context_retriever.hybrid_alpha} (0=vector only, 1=graph only)\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 9.2: Retrieve Context with Graph Expansion\n",
- "\n",
- "Retrieve context using hybrid approach with graph expansion for multi-hop reasoning.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def retrieve_context_for_query(query: str, max_results: int = 10):\n",
- " print(f\"\\nRetrieving context for: '{query}'\")\n",
- " \n",
- " retrieved_contexts = retrieve_context(\n",
- " query=query,\n",
- " method=\"hybrid\",\n",
- " max_results=max_results,\n",
- " knowledge_graph=knowledge_graph,\n",
- " vector_store=vector_store,\n",
- " use_graph_expansion=True,\n",
- " max_hops=2\n",
- " )\n",
- " \n",
- " print(f\"Retrieved {len(retrieved_contexts)} context items\")\n",
- " \n",
- " for i, ctx in enumerate(retrieved_contexts[:5], 1):\n",
- " print(f\"\\n{i}. Relevance: {ctx.score:.4f}\")\n",
- " print(f\" Source: {ctx.source}\")\n",
- " print(f\" Content: {ctx.content[:150]}...\")\n",
- " if hasattr(ctx, 'related_entities') and ctx.related_entities:\n",
- " print(f\" Related entities: {len(ctx.related_entities)}\")\n",
- " if hasattr(ctx, 'related_relationships') and ctx.related_relationships:\n",
- " print(f\" Related relationships: {len(ctx.related_relationships)}\")\n",
- " \n",
- " return retrieved_contexts\n",
- "\n",
- "test_query = \"What are the relationships between AI and machine learning?\"\n",
- "contexts = retrieve_context_for_query(test_query, max_results=10)\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": []
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 9.3: Build Context Graph\n",
- "\n",
- "Build a context graph from retrieved contexts to visualize relationships.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "if contexts:\n",
- " context_graph = build_context_graph(\n",
- " contexts=contexts,\n",
- " method=\"entities_relationships\"\n",
- " )\n",
- " \n",
- " print(f\"Context Graph:\")\n",
- " print(f\"Nodes: {context_graph.number_of_nodes()}\")\n",
- " print(f\"Edges: {context_graph.number_of_edges()}\")\n",
- " \n",
- " if context_graph.number_of_nodes() > 0:\n",
- " print(f\"\\nSample Context Graph Nodes:\")\n",
- " for i, node in enumerate(list(context_graph.nodes())[:5], 1):\n",
- " print(f\" {i}. {node}\")\n",
- "else:\n",
- " print(\"No contexts to build graph from\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 10: GraphRAG Query System\n",
- "\n",
- "Build a complete GraphRAG query processing pipeline that handles different types of queries and prepares context for LLM integration.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from semantica.semantic_extract import NERExtractor\n",
- "\n",
- "class GraphRAGQuerySystem:\n",
- " def __init__(self, context_retriever, knowledge_graph, vector_store):\n",
- " self.context_retriever = context_retriever\n",
- " self.knowledge_graph = knowledge_graph\n",
- " self.vector_store = vector_store\n",
- " self.ner_extractor = NERExtractor()\n",
- " \n",
- " def process_query(self, query: str, max_context: int = 10):\n",
- " \"\"\"\n",
- " Process a query through the complete GraphRAG pipeline.\n",
- " \n",
- " Steps:\n",
- " 1. Parse user query\n",
- " 2. Extract query entities\n",
- " 3. Perform hybrid search (vector + graph)\n",
- " 4. Retrieve relevant context\n",
- " 5. Expand context with graph relationships\n",
- " 6. Prepare context for LLM\n",
- " \"\"\"\n",
- " print(f\"Processing query: '{query}'\")\n",
- " \n",
- " # Step 1: Extract entities from query\n",
- " query_entities = self.ner_extractor.extract(query)\n",
- " print(f\"Extracted {len(query_entities)} entities from query\")\n",
- " \n",
- " # Step 2: Retrieve context\n",
- " contexts = self.context_retriever.retrieve(\n",
- " query=query,\n",
- " max_results=max_context,\n",
- " use_graph_expansion=True,\n",
- " max_hops=2\n",
- " )\n",
- " \n",
- " # Step 3: Expand context with graph relationships\n",
- " expanded_context = self._expand_context_with_graph(contexts, query_entities)\n",
- " \n",
- " # Step 4: Prepare context for LLM\n",
- " llm_context = self._prepare_llm_context(expanded_context, query)\n",
- " \n",
- " return {\n",
- " 'query': query,\n",
- " 'query_entities': query_entities,\n",
- " 'contexts': contexts,\n",
- " 'expanded_context': expanded_context,\n",
- " 'llm_context': llm_context\n",
- " }\n",
- " \n",
- " def _expand_context_with_graph(self, contexts, query_entities):\n",
- " \"\"\"Expand context by following graph relationships.\"\"\"\n",
- " expanded = []\n",
- " \n",
- " for ctx in contexts:\n",
- " expanded.append(ctx)\n",
- " \n",
- " # Add related entities from graph\n",
- " if ctx.related_entities:\n",
- " for entity in ctx.related_entities[:3]: # Limit expansion\n",
- " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n",
- " # Find in graph and get neighbors\n",
- " for node in self.knowledge_graph.nodes():\n",
- " if entity_text.lower() in str(node).lower():\n",
- " neighbors = list(self.knowledge_graph.neighbors(node))[:2]\n",
- " for neighbor in neighbors:\n",
- " expanded.append({\n",
- " 'content': f\"Related: {node} -> {neighbor}\",\n",
- " 'score': 0.6,\n",
- " 'source': 'graph_expansion'\n",
- " })\n",
- " \n",
- " return expanded\n",
- " \n",
- " def _prepare_llm_context(self, contexts, query):\n",
- " \"\"\"Prepare formatted context for LLM.\"\"\"\n",
- " context_text = f\"Query: {query}\\n\\nRelevant Context:\\n\\n\"\n",
- " \n",
- " for i, ctx in enumerate(contexts[:10], 1):\n",
- " content = ctx.content if hasattr(ctx, 'content') else ctx.get('content', str(ctx))\n",
- " score = ctx.score if hasattr(ctx, 'score') else ctx.get('score', 0)\n",
- " source = ctx.source if hasattr(ctx, 'source') else ctx.get('source', 'unknown')\n",
- " \n",
- " context_text += f\"{i}. [Relevance: {score:.3f}, Source: {source}]\\n\"\n",
- " context_text += f\"{content[:300]}...\\n\\n\"\n",
- " \n",
- " return context_text\n",
- "\n",
- "# Initialize GraphRAG query system\n",
- "graphrag_system = GraphRAGQuerySystem(\n",
- " context_retriever=context_retriever,\n",
- " knowledge_graph=knowledge_graph,\n",
- " vector_store=vector_store\n",
- ")\n",
- "\n",
- "print(f\"GraphRAG query system initialized\")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Example queries\n",
- "example_queries = [\n",
- " \"What is artificial intelligence?\", # Factual question\n",
- " \"How are AI and machine learning related?\", # Relationship query\n",
- " \"What are the applications of deep learning in healthcare?\", # Complex multi-hop query\n",
- "]\n",
- "\n",
- "# Process each query\n",
- "query_results = {}\n",
- "for query in example_queries:\n",
- " print(f\"\\n{'='*60}\")\n",
- " result = graphrag_system.process_query(query, max_context=10)\n",
- " query_results[query] = result\n",
- " \n",
- " print(f\"Prepared LLM Context ({len(result['llm_context'])} chars):\")\n",
- " print(result['llm_context'][:500] + \"...\")\n",
- "\n",
- "print(f\"\\nProcessed {len(query_results)} queries\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 11: LLM Integration\n",
- "\n",
- "Integrate with LLM (OpenAI, Anthropic, or local) to generate answers using the retrieved GraphRAG context.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# LLM Integration\n",
- "# This section demonstrates how to integrate with LLMs using the retrieved context\n",
- "\n",
- "def generate_answer_with_llm(query: str, llm_context: str, llm_provider: str = \"openai\"):\n",
- " \"\"\"\n",
- " Generate answer using LLM with GraphRAG context.\n",
- " \n",
- " Supports OpenAI, Anthropic, or local LLMs.\n",
- " \"\"\"\n",
- " # Build prompt\n",
- " prompt = f\"\"\"You are an AI assistant with access to a knowledge graph and retrieved context.\n",
- "\n",
- "Context from Knowledge Graph:\n",
- "{llm_context}\n",
- "\n",
- "Question: {query}\n",
- "\n",
- "Based on the context provided above, please answer the question. If the context doesn't contain enough information, say so. Cite specific entities or relationships from the context when relevant.\n",
- "\n",
- "Answer:\"\"\"\n",
- " \n",
- " # Here you would call your LLM\n",
- " # Example with OpenAI (uncomment and configure):\n",
- " # try:\n",
- " # from openai import OpenAI\n",
- " # client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))\n",
- " # response = client.chat.completions.create(\n",
- " # model=\"gpt-4\",\n",
- " # messages=[\n",
- " # {\"role\": \"system\", \"content\": \"You are a helpful assistant with access to knowledge graphs.\"},\n",
- " # {\"role\": \"user\", \"content\": prompt}\n",
- " # ],\n",
- " # temperature=0.7\n",
- " # )\n",
- " # return response.choices[0].message.content\n",
- " # except Exception as e:\n",
- " # return f\"Error calling LLM: {e}\"\n",
- " \n",
- " # For demonstration, return the prompt structure\n",
- " return f\"[LLM Answer would be generated here using the context above]\"\n",
- "\n",
- "# Example: Generate answer for a query\n",
- "if query_results:\n",
- " sample_query = list(query_results.keys())[0]\n",
- " sample_result = query_results[sample_query]\n",
- " \n",
- " print(f\"Generating answer for: '{sample_query}'\")\n",
- " answer = generate_answer_with_llm(\n",
- " query=sample_query,\n",
- " llm_context=sample_result['llm_context']\n",
- " )\n",
- " \n",
- " print(f\"\\nAnswer:\")\n",
- " print(answer)\n",
- " print(f\"\\nContext Statistics:\")\n",
- " print(f\" Context items: {len(sample_result['contexts'])}\")\n",
- " print(f\" Expanded context: {len(sample_result['expanded_context'])}\")\n",
- " print(f\" Query entities: {len(sample_result['query_entities'])}\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 11.2: Source Attribution and Explainability\n",
- "\n",
- "Show which parts of the knowledge graph contributed to the answer for explainability.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": []
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def explain_answer_sources(query_result):\n",
- " \"\"\"\n",
- " Explain which sources contributed to the answer.\n",
- " \"\"\"\n",
- " print(f\"Answer Sources and Attribution:\")\n",
- " print(f\"Query: {query_result['query']}\")\n",
- " print(f\"\\nRetrieved Context Sources:\")\n",
- " \n",
- " sources = {}\n",
- " for ctx in query_result['contexts']:\n",
- " source = ctx.source if hasattr(ctx, 'source') else ctx.get('source', 'unknown')\n",
- " sources[source] = sources.get(source, 0) + 1\n",
- " \n",
- " for source, count in sources.items():\n",
- " print(f\" {source}: {count} context items\")\n",
- " \n",
- " print(f\"\\nGraph Entities Involved:\")\n",
- " for entity in query_result['query_entities'][:5]:\n",
- " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n",
- " print(f\" - {entity_text}\")\n",
- " \n",
- " print(f\"\\nContext Expansion:\")\n",
- " print(f\" Original contexts: {len(query_result['contexts'])}\")\n",
- " print(f\" Expanded contexts: {len(query_result['expanded_context'])}\")\n",
- " print(f\" Expansion ratio: {len(query_result['expanded_context']) / max(len(query_result['contexts']), 1):.2f}x\")\n",
- "\n",
- "# Explain sources for sample query\n",
- "if query_results:\n",
- " explain_answer_sources(list(query_results.values())[0])\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 12: Advanced Features\n",
- "\n",
- "Demonstrate advanced features including reasoning, quality assessment, and visualization.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from semantica.reasoning import InferenceEngine, RuleManager\n",
- "\n",
- "# Advanced Feature 1: Reasoning with Inference Engine\n",
- "print(f\"Advanced Feature: Logical Reasoning\")\n",
- "inference_engine = InferenceEngine()\n",
- "rule_manager = RuleManager()\n",
- "\n",
- "# Example: Add inference rules\n",
- "# inference_engine.add_rule(\"IF entity A works_for entity B AND entity B located_in entity C THEN entity A located_in entity C\")\n",
- "# # inference_engine.add_fact(...) # Add facts first\n",
- "# new_facts = inference_engine.forward_chain()\n",
- "# print(f\"Inferred {len(new_facts)} new facts\")\n",
- "\n",
- "print(f\"Reasoning can infer new relationships from existing knowledge\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 12.2: Visualize Knowledge Graph\n",
- "\n",
- "Visualize the knowledge graph to understand its structure.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from semantica.visualization import KGVisualizer, AnalyticsVisualizer\n",
- "\n",
- "# Initialize visualizer\n",
- "kg_visualizer = KGVisualizer()\n",
- "analytics_visualizer = AnalyticsVisualizer()\n",
- "\n",
- "# Visualize knowledge graph\n",
- "if knowledge_graph.number_of_nodes() > 0:\n",
- " print(f\"Visualizing knowledge graph...\")\n",
- " \n",
- " # Create visualization\n",
- " # Uncomment to generate visualization\n",
- " # visualization = kg_visualizer.visualize(\n",
- " # knowledge_graph,\n",
- " # output_path=\"graphrag_visualization.html\",\n",
- " # layout=\"spring\",\n",
- " # show_labels=True\n",
- " # )\n",
- " # print(f\"Visualization saved to graphrag_visualization.html\")\n",
- " \n",
- " print(f\"Graph Statistics for Visualization:\")\n",
- " print(f\" Nodes: {knowledge_graph.number_of_nodes()}\")\n",
- " print(f\" Edges: {knowledge_graph.number_of_edges()}\")\n",
- " print(f\" Node types: {len(set(n.get('type', 'Unknown') for _, n in knowledge_graph.nodes(data=True)))}\")\n",
- " print(f\" Edge types: {len(set(e.get('type', 'Unknown') for _, _, e in knowledge_graph.edges(data=True)))}\")\n",
- " \n",
- " # Analytics visualization\n",
- " # analytics_viz = analytics_visualizer.visualize(\n",
- " # knowledge_graph,\n",
- " # metrics=['centrality', 'communities'],\n",
- " # output_path=\"graphrag_analytics.html\"\n",
- " # )\n",
- " # print(f\"Analytics visualization saved\")\n",
- "else:\n",
- " print(f\"Graph is empty, skipping visualization\")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Step 13: Complete End-to-End Example\n",
- "\n",
- "Demonstrate a complete end-to-end GraphRAG workflow with real-world data, showing the full pipeline from ingestion to answer generation.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def complete_graphrag_workflow(query: str):\n",
- " \"\"\"\n",
- " Complete GraphRAG workflow from query to answer.\n",
- " \"\"\"\n",
- " print(f\"\\n{'='*70}\")\n",
- " print(f\"Complete GraphRAG Workflow\")\n",
- " print(f\"{'='*70}\")\n",
- " print(f\"Query: {query}\\n\")\n",
- " \n",
- " # Step 1: Process query\n",
- " print(\"Step 1: Processing query...\")\n",
- " result = graphrag_system.process_query(query, max_context=10)\n",
- " \n",
- " # Step 2: Generate answer\n",
- " print(\"\\nStep 2: Generating answer with LLM...\")\n",
- " answer = generate_answer_with_llm(query, result['llm_context'])\n",
- " \n",
- " # Step 3: Explain sources\n",
- " print(\"\\nStep 3: Explaining sources...\")\n",
- " explain_answer_sources(result)\n",
- " \n",
- " # Step 4: Show performance metrics\n",
- " print(\"\\nStep 4: Performance Metrics:\")\n",
- " print(f\" Context retrieval time: <1s (simulated)\")\n",
- " print(f\" Context items retrieved: {len(result['contexts'])}\")\n",
- " print(f\" Graph expansion hops: 2\")\n",
- " print(f\" Total context size: {len(result['llm_context'])} characters\")\n",
- " \n",
- " return {\n",
- " 'query': query,\n",
- " 'answer': answer,\n",
- " 'contexts': result['contexts'],\n",
- " 'metrics': {\n",
- " 'context_items': len(result['contexts']),\n",
- " 'expanded_items': len(result['expanded_context']),\n",
- " 'query_entities': len(result['query_entities'])\n",
- " }\n",
- " }\n",
- "\n",
- "# Run complete workflow example\n",
- "if len(all_documents) > 0 or knowledge_graph.number_of_nodes() > 0:\n",
- " example_query = \"What are the main concepts and their relationships?\"\n",
- " workflow_result = complete_graphrag_workflow(example_query)\n",
- " \n",
- " print(f\"\\nComplete workflow executed successfully!\")\n",
- " print(f\"Final Results:\")\n",
- " print(f\" Query processed: ā\")\n",
- " print(f\" Context retrieved: {workflow_result['metrics']['context_items']} items\")\n",
- " print(f\" Answer generated: ā\")\n",
- "else:\n",
- " print(\"Configure data sources above to run complete workflow with real data\")\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "print(f\"Comparison: Traditional RAG vs GraphRAG\\n\")\n",
- "\n",
- "comparison = {\n",
- " \"Traditional RAG\": {\n",
- " \"Retrieval\": \"Vector similarity only\",\n",
- " \"Context\": \"Flat document chunks\",\n",
- " \"Relationships\": \"Not captured\",\n",
- " \"Multi-hop\": \"Not supported\",\n",
- " \"Explainability\": \"Limited (source documents only)\"\n",
+ "config_data = {\n",
+ " \"project_name\": \"ResearchGraph\",\n",
+ " \"embedding\": {\n",
+ " \"provider\": \"openai\",\n",
+ " \"model\": \"text-embedding-3-small\"\n",
" },\n",
- " \"GraphRAG\": {\n",
- " \"Retrieval\": \"Vector + Graph traversal\",\n",
- " \"Context\": \"Structured knowledge graph\",\n",
- " \"Relationships\": \"Explicitly modeled\",\n",
- " \"Multi-hop\": \"Supported (graph expansion)\",\n",
- " \"Explainability\": \"High (entities, relationships, paths)\"\n",
+ " \"knowledge_graph\": {\n",
+ " \"backend\": \"networkx\",\n",
+ " \"merge_entities\": True,\n",
+ " \"entity_resolution_strategy\": \"fuzzy\",\n",
+ " \"similarity_threshold\": 0.85\n",
+ " },\n",
+ " \"extraction\": {\n",
+ " \"model\": \"gpt-4o\",\n",
+ " \"temperature\": 0.0\n",
" }\n",
"}\n",
"\n",
- "print(\"Feature Comparison:\")\n",
- "print(f\"{'Feature':<20} {'Traditional RAG':<25} {'GraphRAG':<25}\")\n",
- "print(\"-\" * 70)\n",
- "\n",
- "for feature in comparison[\"Traditional RAG\"].keys():\n",
- " trad = comparison[\"Traditional RAG\"][feature]\n",
- " graph = comparison[\"GraphRAG\"][feature]\n",
- " print(f\"{feature:<20} {trad:<25} {graph:<25}\")\n",
- "\n",
- "print(\"\\nGraphRAG Advantages:\")\n",
- "print(f\" ⢠Better handling of complex queries requiring relationship understanding\")\n",
- "print(f\" ⢠Multi-hop reasoning across entities\")\n",
- "print(f\" ⢠More accurate answers through structured knowledge\")\n",
- "print(f\" ⢠Better explainability with graph paths\")\n",
- "print(f\" ⢠Reduced hallucinations through graph validation\")\n"
+ "config = ConfigManager().load_from_dict(config_data)\n",
+ "semantica = Semantica(config=config)\n",
+ "semantica.initialize()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Step 14: Export and Persistence\n",
+ "## 4. Building the Knowledge Base\n",
"\n",
- "Export the knowledge graph and save the vector store for reuse and sharing.\n",
- "\n"
+ "We execute the orchestrated pipeline. This single call replaces manual parsing, cleaning, chunking, NER, RE, and embedding generation."
]
},
{
@@ -1675,108 +164,183 @@
"metadata": {},
"outputs": [],
"source": [
- "from semantica.export import JSONExporter, RDFExporter, CSVExporter\n",
+ "print(\"š Building Knowledge Base from Research Paper...\")\n",
"\n",
- "json_exporter = JSONExporter()\n",
- "rdf_exporter = RDFExporter()\n",
- "csv_exporter = CSVExporter()\n",
+ "result = semantica.build_knowledge_base(\n",
+ " sources=[source_path],\n",
+ " embeddings=True,\n",
+ " graph=True,\n",
+ " normalize=True\n",
+ ")\n",
"\n",
- "# Export knowledge graph to JSON\n",
- "if knowledge_graph.number_of_nodes() > 0:\n",
- " print(f\"Exporting knowledge graph...\")\n",
- " \n",
- " # Export to JSON\n",
- " json_output = json_exporter.export(knowledge_graph, \"graphrag_knowledge_graph.json\")\n",
- " print(f\"Exported to JSON: graphrag_knowledge_graph.json\")\n",
- " \n",
- " # Export to RDF\n",
- " rdf_output = rdf_exporter.export(knowledge_graph, \"graphrag_knowledge_graph.rdf\")\n",
- " print(f\"Exported to RDF: graphrag_knowledge_graph.rdf\")\n",
- " \n",
- " # Export entities to CSV\n",
- " entities_data = []\n",
- " for entity in resolved_entities[:100]: # Limit for demo\n",
- " if isinstance(entity, dict):\n",
- " entities_data.append({\n",
- " 'id': entity.get('id', ''),\n",
- " 'text': entity.get('text', entity.get('name', '')),\n",
- " 'type': entity.get('type', 'Unknown')\n",
- " })\n",
- " \n",
- " if entities_data:\n",
- " csv_output = csv_exporter.export(entities_data, \"graphrag_entities.csv\")\n",
- " print(f\"Exported entities to CSV: graphrag_entities.csv\")\n",
- " \n",
- " print(f\"\\nExport Summary:\")\n",
- " print(f\" Nodes exported: {knowledge_graph.number_of_nodes()}\")\n",
- " print(f\" Edges exported: {knowledge_graph.number_of_edges()}\")\n",
- " print(f\" Entities exported: {len(entities_data)}\")\n",
- "else:\n",
- " print(f\"Graph is empty, skipping export\")\n",
- "\n",
- "# Save vector store (if supported)\n",
- "print(\"\\nVector Store:\")\n",
- "print(f\" Vectors stored: ā\")\n",
- "print(f\" Metadata stored: ā\")\n",
- "print(f\" Ready for reuse: ā\")\n"
+ "kg = result[\"knowledge_graph\"]\n",
+ "print(f\"\\nā
Extraction Complete.\")\n",
+ "print(f\"Nodes: {kg.number_of_nodes()} | Edges: {kg.number_of_edges()}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Summary and Next Steps\n",
+ "## 5. Advanced Graph Analytics\n",
"\n",
- "### What We Built\n",
+ "A knowledge graph is more than just storage; it's a structure we can analyze to find insights.\n",
"\n",
- "This notebook demonstrated a **complete end-to-end GraphRAG system** using Semantica:\n",
+ "### A. Centrality (Finding Key Concepts)\n",
+ "Who are the \"VIPs\" in this paper? We use Degree Centrality to find the most connected entities."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "centrality_calc = CentralityCalculator()\n",
+ "centrality_scores = centrality_calc.calculate_degree_centrality(kg)\n",
"\n",
- "1. **Real-World Data Ingestion**: MCP servers, web scraping, RSS feeds\n",
- "2. **Document Processing**: Parsing, splitting, normalization\n",
- "3. **Semantic Extraction**: Entities, relationships, triplets\n",
- "4. **Knowledge Graph Construction**: With quality assurance\n",
- "5. **Embedding Generation**: For documents and entities\n",
- "6. **Vector Store**: Fast similarity search\n",
- "7. **Hybrid Search**: Combining vectors and graphs\n",
- "8. **Context Retrieval**: With graph expansion\n",
- "9. **GraphRAG Query System**: Complete query processing\n",
- "10. **LLM Integration**: Answer generation with context\n",
- "11. **Advanced Features**: Reasoning, quality, visualization\n",
- "12. **Export**: Persistence and sharing\n",
- "\n",
- "### Key Takeaways\n",
- "\n",
- "- **GraphRAG** combines the best of vector search and knowledge graphs\n",
- "- **Multi-hop reasoning** enables deeper understanding\n",
- "- **Real-world data** makes the system production-ready\n",
- "- **Semantica** provides all modules needed for GraphRAG\n",
- "\n",
- "### Next Steps\n",
- "\n",
- "1. **Configure Real Data Sources**: Set up MCP servers, web URLs, or RSS feeds\n",
- "2. **Customize Extraction**: Adjust entity and relationship extraction for your domain\n",
- "3. **Tune Hybrid Search**: Experiment with `hybrid_alpha` for your use case\n",
- "4. **Add More LLMs**: Integrate with Anthropic, local models, or other providers\n",
- "5. **Scale Up**: Process larger datasets and optimize performance\n",
- "6. **Deploy**: Build production GraphRAG applications\n",
- "\n",
- "### Resources\n",
- "\n",
- "- [Semantica Documentation](https://semantica.readthedocs.io/)\n",
- "- [GraphRAG Concepts](https://semantica.readthedocs.io/concepts/)\n",
- "- [API Reference](https://semantica.readthedocs.io/reference/)\n",
- "- [More Examples](https://semantica.readthedocs.io/cookbook/)\n",
- "\n",
- "---\n",
- "\n",
- "**Congratulations!** You've built a complete GraphRAG system with Semantica!\n",
- "\n"
+ "# Convert to DataFrame for nice display\n",
+ "df_centrality = pd.DataFrame(centrality_scores.get(\"rankings\", []))\n",
+ "if not df_centrality.empty:\n",
+ " print(\"š Top 5 Most Central Concepts:\")\n",
+ " print(df_centrality.head(5)[[\"node\", \"score\"]].to_string(index=False))"
]
},
{
"cell_type": "markdown",
"metadata": {},
- "source": []
+ "source": [
+ "### B. Community Detection (Topic Modeling)\n",
+ "Can we group these entities into topics? We use the **Louvain Algorithm** to detect communities within the graph."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "detector = CommunityDetector()\n",
+ "communities_result = detector.detect_communities(kg, algorithm=\"louvain\")\n",
+ "communities = communities_result.get(\"communities\", [])\n",
+ "\n",
+ "print(f\"š Detected {len(communities)} distinct communities (topics).\")\n",
+ "\n",
+ "for i, comm in enumerate(communities[:3]): # Show first 3\n",
+ " print(f\"\\nGroup {i+1}: {', '.join(list(comm)[:5])}...\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 6. Ontology Generation\n",
+ "\n",
+ "One of Semantica's most powerful features is **Reverse Engineering the Schema**. We can look at the graph we just built and ask: \"What is the underlying data model?\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ontology_gen = OntologyGenerator()\n",
+ "ontology = ontology_gen.generate_from_graph(kg)\n",
+ "\n",
+ "print(\"𧬠Inferred Ontology Classes:\")\n",
+ "if \"classes\" in ontology:\n",
+ " for cls in ontology[\"classes\"][:5]:\n",
+ " print(f" - {cls.get('name')}: {cls.get('description', 'No description inferred')}")\n",
+ "else:\n",
+ " print(\" (Ontology generation requires sufficient data volume for inference)\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 7. Hybrid GraphRAG Inference\n",
+ "\n",
+ "Finally, we use the system to answer a complex question. We simulate the logic of a Hybrid Search:\n",
+ "1. **Vector**: Find the relevant text chunk.\n",
+ "2. **Graph**: Find the *neighbors* of the key entities to understand the context (e.g., what benchmarks were used?)\n",
+ "3. **Synthesis**: (Conceptually) combine these for the LLM."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "query = \"What benchmarks were used to evaluate GAT-KG?\"\n",
+ "print(f\"ā Query: '{query}'\")\n",
+ "\n",
+ "# 1. Vector Search\n",
+ "vector_store = semantica.vector_store\n",
+ "embedding_gen = semantica.embedding_generator\n",
+ "q_vec = embedding_gen.generate_embeddings(query, data_type=\"text\")\n",
+ "vec_results = vector_store.search(q_vec, k=1)\n",
+ "\n",
+ "print(\"\\nš¹ [Vector Retrieval] Found Context:\")\n",
+ "top_chunk = vec_results[0].get('metadata', {}).get('text', '') if vec_results else \"No match\"\n",
+ "print(f\" \\\"{top_chunk[:100]}...\\\"\")\n",
+ "\n",
+ "# 2. Graph Traversal\n",
+ "print(\"\\nšø [Graph Retrieval] Exploring Neighborhoods:\")\n",
+ "# Find 'GAT-KG' in the graph and see what it connects to\n",
+ "target_node = None\n",
+ "for node in kg.nodes():\n",
+ " if \"GAT-KG\" in str(node):\n",
+ " target_node = node\n",
+ " break\n",
+ "\n",
+ "if target_node:\n",
+ " neighbors = list(kg[target_node])\n",
+ " print(f" Entity '{target_node}' is connected to:")\n",
+ " for n in neighbors:\n",
+ " relation = kg[target_node][n].get('type', 'related_to')\n",
+ " print(f" --[{relation}]--> {n}")\n",
+ "else:\n",
+ " print(\" (Entity 'GAT-KG' not explicitly found in graph nodes)\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 8. Visualization\n",
+ "\n",
+ "We visualize the final graph structure."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "viz = KGVisualizer()\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "import networkx as nx\n",
+ "\n",
+ "plt.figure(figsize=(14, 10))\n",
+ "pos = nx.spring_layout(kg, k=0.8, seed=42)\n",
+ "\n",
+ "# Draw with more style\n",
+ "nx.draw_networkx_nodes(kg, pos, node_size=1500, node_color=\"#6a0dad\", alpha=0.8) # Semantica Purple\n",
+ "nx.draw_networkx_edges(kg, pos, width=1.5, alpha=0.4, edge_color=\"gray\", arrows=True)\n",
+ "nx.draw_networkx_labels(kg, pos, font_size=9, font_color=\"white\", font_weight=\"bold\")\n",
+ "\n",
+ "edge_labels = nx.get_edge_attributes(kg, 'type')\n",
+ "nx.draw_networkx_edge_labels(kg, pos, edge_labels=edge_labels, font_size=7)\n",
+ "\n",
+ "plt.title(\"Research Paper Knowledge Graph\", fontsize=20)\n",
+ "plt.axis('off')\n",
+ "plt.show()"
+ ]
}
],
"metadata": {
@@ -1795,9 +359,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.9"
+ "version": "3.11.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
-}
+}
\ No newline at end of file