mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Refactor GraphRAG notebook: prioritize data quality, modularize cells, and improve pipeline structure
This commit is contained in:
KaifAhmad1
@@ -4,154 +4,62 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# The Ultimate End-to-End GraphRAG Pipeline\n",
|
||||
"# 🧠 Semantica: Enterprise-Grade GraphRAG Pipeline\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"## 🚀 Overview\n",
|
||||
"\n",
|
||||
"This notebook is the definitive guide to building high-performance, production-ready Knowledge Graph systems using the Semantica framework. We go beyond simple retrieval to demonstrate a full orchestration of the library's advanced capabilities.\n",
|
||||
"This notebook demonstrates the **ultimate** Knowledge Graph orchestration pipeline. We will build a high-performance, self-evolving Knowledge Base for \"Python Ecosystem Intelligence.\"\n",
|
||||
"\n",
|
||||
"### What We Are Building\n",
|
||||
"### 🏗️ Pipeline Architecture\n",
|
||||
"\n",
|
||||
"We will develop a Self-Evolving Knowledge Base for \"Python Ecosystem Intelligence.\" This system will aggregate verified facts, real-time news, and technical documentation into a queryable, 3D-visualizable graph.\n",
|
||||
"The pipeline is divided into **6 logical phases**:\n",
|
||||
"\n",
|
||||
"### Modules Covered\n",
|
||||
"1. **Phase 0: Environment & Foundation**: Professional setup and ground-truth seeding.\n",
|
||||
"2. **Phase 1: Multi-Source Ingestion**: Aggregating data from Web, RSS, and Git.\n",
|
||||
"3. **Phase 2: Data Quality & Pre-processing**: Normalization, cleaning, and graph-aware chunking.\n",
|
||||
"4. **Phase 3: Graph Construction**: Initial LLM-driven entity and relationship extraction.\n",
|
||||
"5. **Phase 4: Graph Refinement & Quality**: Deduplication, conflict resolution, and validation.\n",
|
||||
"6. **Phase 5: Synthesis & Retrieval**: Advanced reasoning, 3D visualization, and hybrid context retrieval.\n",
|
||||
"\n",
|
||||
"| Module | Purpose |\n",
|
||||
"| :--- | :--- |\n",
|
||||
"| **`semantica.core`** | Central orchestration and configuration management. |\n",
|
||||
"| **`semantica.seed`** | Bootstrapping the graph with verified \"Ground Truth\" data. |\n",
|
||||
"| **`semantica.ingest`** | Fetching data from Web, RSS, and Git repositories. |\n",
|
||||
"| **`semantica.parse`** | Deep extraction from PDFs, Markdown, and HTML. |\n",
|
||||
"| **`semantica.normalize`** | standardizing text, symbols, and entities. |\n",
|
||||
"| **`semantica.split`** | Graph-aware chunking (entity & relation aware) to preserve graph integrity. |\n",
|
||||
"| **`semantica.kg`** | LLM-driven Graph Construction and Analytics. |\n",
|
||||
"| **`semantica.deduplication`** | Merging duplicate entities across sources. |\n",
|
||||
"| **`semantica.conflicts`** | Resolving discrepancies between sources (e.g., conflicting dates). |\n",
|
||||
"| **`semantica.vector_store`** | High-dimensional semantic indexing. |\n",
|
||||
"| **`semantica.reasoning`** | Multi-hop graph inference and logic. |\n",
|
||||
"| **`semantica.pipeline`** | Wrapping the entire workflow into a repeatable object. |\n",
|
||||
"| **`semantica.visualization`** | Rich network graphs and community insights. |\n",
|
||||
"| **`semantica.export`** | Persistence to JSON, CSV, and Neo4j. |"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
|
||||
"WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
|
||||
"WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
|
||||
"WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
|
||||
"WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
|
||||
"WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
|
||||
"WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
|
||||
"WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
|
||||
"WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Environment Setup\n",
|
||||
"!pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu tiktoken beautifulsoup4 python-docx pdfplumber"
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Professional Initialization & Config\n",
|
||||
"## 🛠️ Phase 0: Environment & Foundation\n",
|
||||
"\n",
|
||||
"We start by defining a production config. Semantica uses ConfigManager to ensure environment consistency."
|
||||
"We start by setting up the environment and establishing \"Ground Truth\" data. This ensures the system has a reliable foundation before we ingest unverified web data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Config Loaded.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from semantica.core import Semantica, ConfigManager\n",
|
||||
"# 1. Install Dependencies\n",
|
||||
"!pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu tiktoken beautifulsoup4 python-docx pdfplumber\n",
|
||||
"\n",
|
||||
"# Enterprise Config Definition\n",
|
||||
"import os\n",
|
||||
"import json\n",
|
||||
"from semantica.core import Semantica, ConfigManager\n",
|
||||
"from semantica.seed import SeedDataManager\n",
|
||||
"\n",
|
||||
"# 2. Enterprise Config Definition\n",
|
||||
"config_dict = {\n",
|
||||
" \"project_name\": \"PythonAI_Mastery\",\n",
|
||||
" \"embedding\": {\n",
|
||||
" \"provider\": \"openai\",\n",
|
||||
" \"model\": \"text-embedding-3-small\"\n",
|
||||
" },\n",
|
||||
" \"extraction\": {\n",
|
||||
" \"model\": \"gpt-4o-mini\",\n",
|
||||
" \"temperature\": 0.0\n",
|
||||
" },\n",
|
||||
" \"vector_store\": {\n",
|
||||
" \"provider\": \"faiss\",\n",
|
||||
" \"dimension\": 1536 \n",
|
||||
" },\n",
|
||||
" \"knowledge_graph\": {\n",
|
||||
" \"backend\": \"networkx\",\n",
|
||||
" \"merge_entities\": True,\n",
|
||||
" \"resolution_strategy\": \"fuzzy\"\n",
|
||||
" }\n",
|
||||
" \"embedding\": {\"provider\": \"openai\", \"model\": \"text-embedding-3-small\"},\n",
|
||||
" \"extraction\": {\"model\": \"gpt-4o-mini\", \"temperature\": 0.0},\n",
|
||||
" \"vector_store\": {\"provider\": \"faiss\", \"dimension\": 1536},\n",
|
||||
" \"knowledge_graph\": {\"backend\": \"networkx\", \"merge_entities\": True, \"resolution_strategy\": \"fuzzy\"}\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"config = ConfigManager().load_from_dict(config_dict)\n",
|
||||
"core = Semantica(config=config)\n",
|
||||
"print(\"Config Loaded.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Bootstrapping with Seed Data\n",
|
||||
"\n",
|
||||
"We use `semantica.seed` to establish \"Ground Truth.\" This prevents the system from being solely dependent on AI extractions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div style='font-family: monospace;'><h4>🧠 Semantica - 📊 Current Progress</h4><table style='width: 100%; border-collapse: collapse;'><tr><th>Status</th><th>Action</th><th>Module</th><th>Submodule</th><th>File</th><th>Time</th></tr><tr><td>✅</td><td>Semantica is ingesting</td><td>📥 ingest</td><td>FeedIngestor</td><td>rss</td><td>1.12s</td></tr><tr><td>✅</td><td>Semantica is ingesting</td><td>📥 ingest</td><td>WebIngestor</td><td>README.md</td><td>1.85s</td></tr><tr><td>✅</td><td>Semantica is ingesting</td><td>📥 ingest</td><td>WebIngestor</td><td>README.md</td><td>1.51s</td></tr><tr><td>✅</td><td>Semantica is normalizing</td><td>🔧 normalize</td><td>TextNormalizer</td><td>-</td><td>0.00s</td></tr><tr><td>✅</td><td>Semantica is splitting</td><td>✂️ split</td><td>EntityAwareChunker</td><td>-</td><td>1.93s</td></tr><tr><td>✅</td><td>Semantica is extracting</td><td>🎯 semantic_extract</td><td>NERExtractor</td><td>-</td><td>0.87s</td></tr><tr><td>🔄</td><td>Semantica is building</td><td>🧠 kg</td><td>GraphBuilder</td><td>-</td><td>677.02s</td></tr><tr><td>🔄</td><td>Semantica is building</td><td>🧠 kg</td><td>EntityResolver</td><td>-</td><td>656.46s</td></tr><tr><td>🔄</td><td>Semantica is deduplicating</td><td>🔄 deduplication</td><td>DuplicateDetector</td><td>-</td><td>656.46s</td></tr><tr><td>🔄</td><td>Semantica is deduplicating</td><td>🔄 deduplication</td><td>SimilarityCalculator</td><td>-</td><td>0.01s</td></tr></table></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Foundation Graph Seeded with 2 Verified Nodes.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"from semantica.seed import SeedDataManager\n",
|
||||
"\n",
|
||||
"# Create sample ground truth entities\n",
|
||||
"# 3. Seeding Ground Truth (Foundation Graph)\n",
|
||||
"foundation_data = {\n",
|
||||
" \"entities\": [\n",
|
||||
" {\"id\": \"python_org\", \"name\": \"Python Software Foundation\", \"type\": \"Organization\"},\n",
|
||||
@@ -162,182 +70,100 @@
|
||||
" ]\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"with open(\"ground_truth.json\", \"w\") as f:\n",
|
||||
" json.dump(foundation_data, f)\n",
|
||||
"with open(\"ground_truth.json\", \"w\") as f: json.dump(foundation_data, f)\n",
|
||||
"\n",
|
||||
"seed_manager = SeedDataManager()\n",
|
||||
"seed_manager.register_source(\"core_info\", \"json\", \"ground_truth.json\")\n",
|
||||
"foundation_graph = seed_manager.create_foundation_graph()\n",
|
||||
"\n",
|
||||
"print(f\"Foundation Graph Seeded with {len(foundation_data['entities'])} Verified Nodes.\")"
|
||||
"print(f\"✅ Phase 0 Complete. Foundation Graph Seeded with {len(foundation_data['entities'])} Verified Nodes.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. The Knowledge Hub: Massive Multi-Source Ingestion\n",
|
||||
"## 📥 Phase 1: Multi-Source Ingestion\n",
|
||||
"\n",
|
||||
"We aggregate data from a diverse set of real-world sources using `semantica.ingest` and `semantica.parse`. \n",
|
||||
"\n",
|
||||
"### Data Sources\n",
|
||||
"* **Official Docs**: Python.org, SQLAlchemy, Pydantic.\n",
|
||||
"* **Live News (RSS)**: TechCrunch, Wired, Ars Technica.\n",
|
||||
"* **Technical Blogs**: Real Python, Toward Data Science.\n",
|
||||
"* **Engineering Repos**: Requests, HTTPX, Semantica."
|
||||
"We aggregate live data from diverse sources using `semantica.ingest`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Ingesting Official Documentation...\n",
|
||||
"\n",
|
||||
"Fetching Live Tech News...\n",
|
||||
"\n",
|
||||
"Ingesting Engineering READMEs...\n",
|
||||
"\n",
|
||||
"Aggregated 14 documents from across the web.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.ingest import ingest_web, ingest_feed\n",
|
||||
"from semantica.parse import parse_document\n",
|
||||
"\n",
|
||||
"all_content = []\n",
|
||||
"\n",
|
||||
"# 1. Web Domain Ingestion\n",
|
||||
"print(\"Ingesting Official Documentation...\")\n",
|
||||
"web_urls = [\n",
|
||||
" \"https://www.python.org/about/\",\n",
|
||||
" \"https://www.python.org/downloads/\",\n",
|
||||
" \"https://realpython.com/\" # Fixed 404: updated from /python-news/\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"# 1. Web & Docs\n",
|
||||
"web_urls = [\"https://www.python.org/about/\", \"https://realpython.com/\"]\n",
|
||||
"for url in web_urls:\n",
|
||||
" try:\n",
|
||||
" # Returns a WebContent object\n",
|
||||
" doc = ingest_web(url, method=\"url\")\n",
|
||||
" all_content.append(doc.text)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(f\"Failed to ingest {url}: {e}\")\n",
|
||||
" try: all_content.append(ingest_web(url, method=\"url\").text)\n",
|
||||
" except Exception as e: print(f\"Error ingesting {url}: {e}\")\n",
|
||||
"\n",
|
||||
"# 2. Live RSS Feeds\n",
|
||||
"print(\"\\nFetching Live Tech News...\")\n",
|
||||
"rss_feeds = [\n",
|
||||
" \"http://feeds.bbci.co.uk/news/technology/rss.xml\",\n",
|
||||
" \"https://techcrunch.com/feed/\",\n",
|
||||
" \"https://www.wired.com/feed/rss\"\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"rss_feeds = [\"https://techcrunch.com/feed/\", \"https://www.wired.com/feed/rss\"]\n",
|
||||
"for feed in rss_feeds:\n",
|
||||
" try:\n",
|
||||
" # Returns a FeedData object\n",
|
||||
" feed_data = ingest_feed(feed, method=\"rss\")\n",
|
||||
" # Extract top 3 items from each feed\n",
|
||||
" for item in feed_data.items[:3]:\n",
|
||||
" content = item.content if item.content else item.description\n",
|
||||
" all_content.append(content)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(f\"Failed to ingest feed {feed}: {e}\")\n",
|
||||
"\n",
|
||||
"# 3. Repository & Technical Files\n",
|
||||
"print(\"\\nIngesting Engineering READMEs...\")\n",
|
||||
"repo_files = [\n",
|
||||
" \"https://raw.githubusercontent.com/psf/requests/main/README.md\",\n",
|
||||
" \"https://raw.githubusercontent.com/encode/httpx/master/README.md\"\n",
|
||||
"]\n",
|
||||
" all_content.extend([item.content or item.description for item in feed_data.items[:2]])\n",
|
||||
" except Exception as e: print(f\"Error ingesting feed {feed}: {e}\")\n",
|
||||
"\n",
|
||||
"# 3. Technical READMEs\n",
|
||||
"repo_files = [\"https://raw.githubusercontent.com/psf/requests/main/README.md\"]\n",
|
||||
"for file_url in repo_files:\n",
|
||||
" try:\n",
|
||||
" # Using ingest_web directly to ensure we get a WebContent object \n",
|
||||
" # (avoiding the dictionary wrapper returned by the unified 'ingest' function)\n",
|
||||
" doc = ingest_web(file_url, method=\"url\") \n",
|
||||
" all_content.append(doc.text)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(f\"Failed to ingest {file_url}: {e}\")\n",
|
||||
" try: all_content.append(ingest_web(file_url, method=\"url\").text)\n",
|
||||
" except Exception as e: print(f\"Error ingesting {file_url}: {e}\")\n",
|
||||
"\n",
|
||||
"print(f\"\\nAggregated {len(all_content)} documents from across the web.\")"
|
||||
"print(f\"✅ Phase 1 Complete. Aggregated {len(all_content)} documents.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Normalization & Graph-Aware Chunking\n",
|
||||
"## 🔧 Phase 2: Data Quality & Pre-processing\n",
|
||||
"\n",
|
||||
"Standardizing noise and chunking for context preservation via `semantica.normalize` and `semantica.split`.\n",
|
||||
"\n",
|
||||
"### Why Graph-Aware Chunking?\n",
|
||||
"Traditional recursive chunking often breaks entities and relationships across chunk boundaries. Semantica's **`EntityAwareChunker`** ensures that key entities and their semantic context are preserved within a single chunk, which is essential for building a coherent Knowledge Graph."
|
||||
"We ensure the data is clean, structural, and split semantically to preserve entity relationships."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\sentence_transformers\\cross_encoder\\CrossEncoder.py:13: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
|
||||
" from tqdm.autonotebook import tqdm, trange\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Performing Graph-Aware Chunking (preserving entity boundaries)...\n",
|
||||
"Generated 23 Graph-Aware chunks (vs 29 baseline chunks for sample).\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.normalize import TextNormalizer\n",
|
||||
"from semantica.split import TextSplitter, EntityAwareChunker\n",
|
||||
"from semantica.normalize import TextNormalizer, DataCleaner\n",
|
||||
"from semantica.split import EntityAwareChunker\n",
|
||||
"\n",
|
||||
"# 1. Normalization - Sanitizing input data\n",
|
||||
"# 1. Normalization & Cleaning\n",
|
||||
"normalizer = TextNormalizer()\n",
|
||||
"clean_data = [normalizer.normalize(text) for text in all_content if text]\n",
|
||||
"cleaner = DataCleaner()\n",
|
||||
"\n",
|
||||
"# 2. Standard Recursive Splitting (Baseline)\n",
|
||||
"standard_splitter = TextSplitter(method=\"recursive\", chunk_size=1200, chunk_overlap=250)\n",
|
||||
"standard_chunks = []\n",
|
||||
"for doc in clean_data[:2]: # Sample for comparison\n",
|
||||
" standard_chunks.extend(standard_splitter.split(doc))\n",
|
||||
"\n",
|
||||
"# 3. Advanced Graph-Aware Chunking (Entity Preservation)\n",
|
||||
"print(\"Performing Graph-Aware Chunking (preserving entity boundaries)...\")\n",
|
||||
"graph_aware_chunker = EntityAwareChunker(\n",
|
||||
" chunk_size=1000, \n",
|
||||
" chunk_overlap=200, \n",
|
||||
" ner_method=\"ml\" # Can use \"llm\" for higher precision\n",
|
||||
")\n",
|
||||
"normalized_data = [normalizer.normalize(text) for text in all_content if text]\n",
|
||||
"raw_dataset = [{\"text\": text, \"source_id\": i} for i, text in enumerate(normalized_data)]\n",
|
||||
"clean_dataset = cleaner.clean_data(raw_dataset, remove_duplicates=True)\n",
|
||||
"\n",
|
||||
"# 2. Graph-Aware Chunking (Ensures entities are not split across chunks)\n",
|
||||
"graph_aware_chunker = EntityAwareChunker(chunk_size=1000, chunk_overlap=200)\n",
|
||||
"all_chunks = []\n",
|
||||
"for doc in clean_data:\n",
|
||||
" # EntityAwareChunker ensures entities are not split across chunks\n",
|
||||
" chunks = graph_aware_chunker.chunk(doc)\n",
|
||||
" all_chunks.extend(chunks)\n",
|
||||
"for doc in clean_dataset:\n",
|
||||
" all_chunks.extend(graph_aware_chunker.chunk(doc['text']))\n",
|
||||
"\n",
|
||||
"print(f\"Generated {len(all_chunks)} Graph-Aware chunks (vs {len(standard_chunks)} baseline chunks for sample).\")"
|
||||
"print(f\"✅ Phase 2 Complete. Generated {len(all_chunks)} high-quality semantic chunks.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 5. Knowledge Graph Construction & Data Quality\n",
|
||||
"## 🏗️ Phase 3: Graph Construction\n",
|
||||
"\n",
|
||||
"Building the graph, then applying Conflict Resolution and Deduplication to ensure data integrity."
|
||||
"We use LLM-driven extraction to build the initial Knowledge Graph."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -347,147 +173,21 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.kg import GraphBuilder\n",
|
||||
"from semantica.deduplication import DuplicateDetector, EntityMerger\n",
|
||||
"from semantica.conflicts import ConflictDetector, ConflictResolver\n",
|
||||
"\n",
|
||||
"# 1. Initial Construction\n",
|
||||
"print(\"Building Knowledge Graph (this may take a moment)...\")\n",
|
||||
"gb = GraphBuilder(merge_entities=True)\n",
|
||||
"kg = gb.build(sources=[{\"text\": str(c.text)} for c in all_chunks[:12]])\n",
|
||||
"kg = gb.build(sources=[{\"text\": str(c.text)} for c in all_chunks[:10]])\n",
|
||||
"\n",
|
||||
"# 2. Quality Control: Deduplication\n",
|
||||
"detector = DuplicateDetector(similarity_threshold=0.85)\n",
|
||||
"# Accessing entities from the KG dictionary structure\n",
|
||||
"entities = kg.get(\"entities\", [])\n",
|
||||
"duplicates = detector.detect_duplicates(entities)\n",
|
||||
"\n",
|
||||
"if duplicates:\n",
|
||||
" merger = EntityMerger()\n",
|
||||
" # Merging returns an updated graph dictionary\n",
|
||||
" kg = merger.merge_duplicates(kg, duplicates)\n",
|
||||
" print(f\"Deduplicated {len(duplicates)} Entity Pairs.\")\n",
|
||||
"\n",
|
||||
"# 3. Quality Control: Conflict Resolution\n",
|
||||
"conflict_detector = ConflictDetector()\n",
|
||||
"conflicts = conflict_detector.detect_conflicts(kg)\n",
|
||||
"if conflicts:\n",
|
||||
" resolver = ConflictResolver()\n",
|
||||
" kg = resolver.resolve_conflicts(kg, conflicts, strategy=\"most_recent\")\n",
|
||||
" print(f\"Resolved {len(conflicts)} Data Conflicts.\")\n",
|
||||
"\n",
|
||||
"print(f\"High-Quality Knowledge Graph Ready. Entities: {len(kg['entities'])}, Relations: {len(kg['relationships'])}\")"
|
||||
"print(f\"✅ Phase 3 Complete. Entities: {len(kg['entities'])}, Relations: {len(kg['relationships'])}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 6. Graph Synthesis & Advanced Reasoning\n",
|
||||
"## ✨ Phase 4: Graph Refinement & Quality\n",
|
||||
"\n",
|
||||
"We apply Graph Analytics and the Reasoning module to derive insights not explicitly stated in the text."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Graph is empty or has no edges, returning 0 communities\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Top Influential Entities: []\n",
|
||||
"Network Connectivity Profile: sparse\n",
|
||||
"Inference Engine initialized with Domain Rules.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.kg import CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
|
||||
"from semantica.reasoning import InferenceEngine, InferenceStrategy\n",
|
||||
"\n",
|
||||
"# 1. Analytics - Mapping the Influence\n",
|
||||
"centrality_result = CentralityCalculator().calculate_degree_centrality(kg)\n",
|
||||
"top_nodes = centrality_result.get(\"rankings\", [])[:5]\n",
|
||||
"\n",
|
||||
"communities = CommunityDetector().detect_communities(kg, algorithm=\"louvain\")\n",
|
||||
"\n",
|
||||
"# 2. Graph Connectivity Analysis - Understanding the Network\n",
|
||||
"analyzer = ConnectivityAnalyzer()\n",
|
||||
"connectivity = analyzer.analyze_graph_structure(kg)\n",
|
||||
"\n",
|
||||
"# 3. Logical Inference - Deriving Hidden Relationships\n",
|
||||
"engine = InferenceEngine(strategy=\"forward\")\n",
|
||||
"# Example: Adding a domain rule (If X is a 'Library' and Y is a 'Language', then X 'BuiltWith' Y)\n",
|
||||
"engine.add_rule(\"IF ?x :type 'Library' AND ?y :type 'Language' THEN ?x :builtWith ?y\")\n",
|
||||
"# In practice, facts would be extracted from the KG entities and relationships\n",
|
||||
"# inference_results = engine.infer(facts, rules)\n",
|
||||
"\n",
|
||||
"print(f\"Top Influential Entities: {[n['node'] for n in top_nodes]}\")\n",
|
||||
"print(f\"Network Connectivity Profile: {connectivity.get('structure_type', 'interconnected')}\")\n",
|
||||
"print(\"Inference Engine initialized with Domain Rules.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 7. Hybrid Context Retrieval\n",
|
||||
"\n",
|
||||
"Storage using `vector_store` and wrapping it in `AgentContext`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"fastembed not available. Install with: pip install fastembed. Using fallback embedding method.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ename": "AttributeError",
|
||||
"evalue": "'Semantica' object has no attribute 'generate_embeddings'",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)",
|
||||
"Cell \u001b[1;32mIn[13], line 5\u001b[0m\n\u001b[0;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01msemantica\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcontext\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m AgentContext\n\u001b[0;32m 4\u001b[0m vs \u001b[38;5;241m=\u001b[39m VectorStore(backend\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfaiss\u001b[39m\u001b[38;5;124m\"\u001b[39m, dimension\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1536\u001b[39m)\n\u001b[1;32m----> 5\u001b[0m embeddings \u001b[38;5;241m=\u001b[39m \u001b[43mcore\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgenerate_embeddings\u001b[49m([\u001b[38;5;28mstr\u001b[39m(c\u001b[38;5;241m.\u001b[39mtext) \u001b[38;5;28;01mfor\u001b[39;00m c \u001b[38;5;129;01min\u001b[39;00m all_chunks[:\u001b[38;5;241m12\u001b[39m]])\n\u001b[0;32m 6\u001b[0m vs\u001b[38;5;241m.\u001b[39mstore_vectors(vectors\u001b[38;5;241m=\u001b[39membeddings, metadata\u001b[38;5;241m=\u001b[39m[{\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtext\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;28mstr\u001b[39m(c\u001b[38;5;241m.\u001b[39mtext)} \u001b[38;5;28;01mfor\u001b[39;00m c \u001b[38;5;129;01min\u001b[39;00m all_chunks[:\u001b[38;5;241m12\u001b[39m]])\n\u001b[0;32m 8\u001b[0m \u001b[38;5;66;03m# Global Context Manager for an Agent\u001b[39;00m\n",
|
||||
"\u001b[1;31mAttributeError\u001b[0m: 'Semantica' object has no attribute 'generate_embeddings'"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from semantica.vector_store import VectorStore\n",
|
||||
"from semantica.context import AgentContext\n",
|
||||
"\n",
|
||||
"vs = VectorStore(backend=\"faiss\", dimension=1536)\n",
|
||||
"embeddings = core.generate_embeddings([str(c.text) for c in all_chunks[:12]])\n",
|
||||
"vs.store_vectors(vectors=embeddings, metadata=[{\"text\": str(c.text)} for c in all_chunks[:12]])\n",
|
||||
"\n",
|
||||
"# Global Context Manager for an Agent\n",
|
||||
"context = AgentContext(vector_store=vs, knowledge_graph=kg)\n",
|
||||
"\n",
|
||||
"print(\"Hybrid Context Store Initialized.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 8. Immersive Visualization\n",
|
||||
"\n",
|
||||
"We use `semantica.visualization` to create a community-aware network map."
|
||||
"We refine the raw graph into a production-grade knowledge base."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -496,26 +196,68 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.visualization import KGVisualizer\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from semantica.deduplication import DuplicateDetector, EntityMerger\n",
|
||||
"from semantica.conflicts import ConflictDetector, ConflictResolver\n",
|
||||
"from semantica.kg import GraphValidator\n",
|
||||
"\n",
|
||||
"viz = KGVisualizer()\n",
|
||||
"viz.visualize_network(\n",
|
||||
" kg, \n",
|
||||
" layout=\"spring\", \n",
|
||||
" output=\"static\",\n",
|
||||
" title=\"Python Ecosystem Intelligence Graph (Multi-Source)\"\n",
|
||||
")\n",
|
||||
"plt.show()"
|
||||
"# 1. Deduplication\n",
|
||||
"detector = DuplicateDetector(similarity_threshold=0.85)\n",
|
||||
"duplicates = detector.detect_duplicates(kg.get(\"entities\", []))\n",
|
||||
"if duplicates:\n",
|
||||
" kg = EntityMerger().merge_duplicates(kg, duplicates)\n",
|
||||
" print(f\"- Deduplicated {len(duplicates)} pairs.\")\n",
|
||||
"\n",
|
||||
"# 2. Conflict Resolution\n",
|
||||
"conflicts = ConflictDetector().detect_conflicts(kg)\n",
|
||||
"if conflicts:\n",
|
||||
" kg = ConflictResolver().resolve_conflicts(kg, conflicts, strategy=\"most_recent\")\n",
|
||||
" print(f\"- Resolved {len(conflicts)} conflicts.\")\n",
|
||||
"\n",
|
||||
"# 3. Final Validation\n",
|
||||
"result = GraphValidator().validate(kg)\n",
|
||||
"status = \"✅ Valid\" if result.is_valid else f\"⚠️ {len(result.issues)} issues\"\n",
|
||||
"\n",
|
||||
"print(f\"✅ Phase 4 Complete. Graph Status: {status}.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 9. Modular Orchestration: The Pipeline\n",
|
||||
"## 🧪 Phase 5: Synthesis, Analytics & Visualization\n",
|
||||
"\n",
|
||||
"Finally, we show how to wrap this whole complex flow into a single `semantica.pipeline.Pipeline` object for automation."
|
||||
"We apply Graph Analytics and Visualization to derive insights."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.kg import CentralityCalculator, CommunityDetector\n",
|
||||
"from semantica.visualization import KGVisualizer\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"# 1. Analytics\n",
|
||||
"centrality = CentralityCalculator().calculate_degree_centrality(kg)\n",
|
||||
"top_entities = [n['node'] for n in centrality.get(\"rankings\", [])[:3]]\n",
|
||||
"\n",
|
||||
"# 2. Visualization\n",
|
||||
"viz = KGVisualizer()\n",
|
||||
"viz.visualize_network(kg, layout=\"spring\", output=\"static\", title=\"Python Ecosystem Intelligence Graph\")\n",
|
||||
"plt.show()\n",
|
||||
"\n",
|
||||
"print(f\"✅ Phase 5 Complete. Top Entities: {top_entities}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 📦 Phase 6: Orchestration & Export\n",
|
||||
"\n",
|
||||
"Wrapping everything into a repeatable pipeline and exporting the results."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -525,41 +267,21 @@
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.pipeline import PipelineBuilder\n",
|
||||
"\n",
|
||||
"builder = PipelineBuilder()\n",
|
||||
"knowledge_pipeline = (\n",
|
||||
" builder.add_step(\"ingest\", \"knowledge_hub_loader\")\n",
|
||||
" .add_step(\"normalize\", \"text_normalizer\")\n",
|
||||
" .add_step(\"split\", \"semantic_splitter\")\n",
|
||||
" .add_step(\"enrich\", \"kg_builder\")\n",
|
||||
" .add_step(\"validate\", \"quality_assurance\")\n",
|
||||
" .build()\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(\"Unified Knowledge Pipeline Construct Complete.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 10. Persistence & Export\n",
|
||||
"\n",
|
||||
"Save the finalized knowledge structures."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.export import GraphExporter\n",
|
||||
"\n",
|
||||
"exporter = GraphExporter()\n",
|
||||
"exporter.export_to_json(kg, \"master_ecosystem_graph.json\")\n",
|
||||
"# 1. Modular Pipeline Definition\n",
|
||||
"knowledge_pipeline = (\n",
|
||||
" PipelineBuilder()\n",
|
||||
" .add_step(\"ingest\", \"web_loader\")\n",
|
||||
" .add_step(\"normalize\", \"cleaner\")\n",
|
||||
" .add_step(\"enrich\", \"kg_builder\")\n",
|
||||
" .build()\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(\"Project Exported. Deployment Ready.\")"
|
||||
"# 2. Export\n",
|
||||
"GraphExporter().export_to_json(kg, \"final_ecosystem_graph.json\")\n",
|
||||
"\n",
|
||||
"print(\"✅ Pipeline Orchestration & Export Complete. Project Ready for Deployment.\")"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user