diff --git a/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb index b92b9f5d..4a783f8c 100644 --- a/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb +++ b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb @@ -4,154 +4,62 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# The Ultimate End-to-End GraphRAG Pipeline\n", + "# ๐Ÿง  Semantica: Enterprise-Grade GraphRAG Pipeline\n", "\n", - "## Overview\n", + "## ๐Ÿš€ Overview\n", "\n", - "This notebook is the definitive guide to building high-performance, production-ready Knowledge Graph systems using the Semantica framework. We go beyond simple retrieval to demonstrate a full orchestration of the library's advanced capabilities.\n", + "This notebook demonstrates the **ultimate** Knowledge Graph orchestration pipeline. We will build a high-performance, self-evolving Knowledge Base for \"Python Ecosystem Intelligence.\"\n", "\n", - "### What We Are Building\n", + "### ๐Ÿ—๏ธ Pipeline Architecture\n", "\n", - "We will develop a Self-Evolving Knowledge Base for \"Python Ecosystem Intelligence.\" This system will aggregate verified facts, real-time news, and technical documentation into a queryable, 3D-visualizable graph.\n", + "The pipeline is divided into **6 logical phases**:\n", "\n", - "### Modules Covered\n", + "1. **Phase 0: Environment & Foundation**: Professional setup and ground-truth seeding.\n", + "2. **Phase 1: Multi-Source Ingestion**: Aggregating data from Web, RSS, and Git.\n", + "3. **Phase 2: Data Quality & Pre-processing**: Normalization, cleaning, and graph-aware chunking.\n", + "4. **Phase 3: Graph Construction**: Initial LLM-driven entity and relationship extraction.\n", + "5. **Phase 4: Graph Refinement & Quality**: Deduplication, conflict resolution, and validation.\n", + "6. **Phase 5: Synthesis & Retrieval**: Advanced reasoning, 3D visualization, and hybrid context retrieval.\n", "\n", - "| Module | Purpose |\n", - "| :--- | :--- |\n", - "| **`semantica.core`** | Central orchestration and configuration management. |\n", - "| **`semantica.seed`** | Bootstrapping the graph with verified \"Ground Truth\" data. |\n", - "| **`semantica.ingest`** | Fetching data from Web, RSS, and Git repositories. |\n", - "| **`semantica.parse`** | Deep extraction from PDFs, Markdown, and HTML. |\n", - "| **`semantica.normalize`** | standardizing text, symbols, and entities. |\n", - "| **`semantica.split`** | Graph-aware chunking (entity & relation aware) to preserve graph integrity. |\n", - "| **`semantica.kg`** | LLM-driven Graph Construction and Analytics. |\n", - "| **`semantica.deduplication`** | Merging duplicate entities across sources. |\n", - "| **`semantica.conflicts`** | Resolving discrepancies between sources (e.g., conflicting dates). |\n", - "| **`semantica.vector_store`** | High-dimensional semantic indexing. |\n", - "| **`semantica.reasoning`** | Multi-hop graph inference and logic. |\n", - "| **`semantica.pipeline`** | Wrapping the entire workflow into a repeatable object. |\n", - "| **`semantica.visualization`** | Rich network graphs and community insights. |\n", - "| **`semantica.export`** | Persistence to JSON, CSV, and Neo4j. |" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", - "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n" - ] - } - ], - "source": [ - "# Environment Setup\n", - "!pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu tiktoken beautifulsoup4 python-docx pdfplumber" + "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## 1. Professional Initialization & Config\n", + "## ๐Ÿ› ๏ธ Phase 0: Environment & Foundation\n", "\n", - "We start by defining a production config. Semantica uses ConfigManager to ensure environment consistency." + "We start by setting up the environment and establishing \"Ground Truth\" data. This ensures the system has a reliable foundation before we ingest unverified web data." ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Config Loaded.\n" - ] - } - ], + "outputs": [], "source": [ - "import os\n", - "from semantica.core import Semantica, ConfigManager\n", + "# 1. Install Dependencies\n", + "!pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu tiktoken beautifulsoup4 python-docx pdfplumber\n", "\n", - "# Enterprise Config Definition\n", + "import os\n", + "import json\n", + "from semantica.core import Semantica, ConfigManager\n", + "from semantica.seed import SeedDataManager\n", + "\n", + "# 2. Enterprise Config Definition\n", "config_dict = {\n", " \"project_name\": \"PythonAI_Mastery\",\n", - " \"embedding\": {\n", - " \"provider\": \"openai\",\n", - " \"model\": \"text-embedding-3-small\"\n", - " },\n", - " \"extraction\": {\n", - " \"model\": \"gpt-4o-mini\",\n", - " \"temperature\": 0.0\n", - " },\n", - " \"vector_store\": {\n", - " \"provider\": \"faiss\",\n", - " \"dimension\": 1536 \n", - " },\n", - " \"knowledge_graph\": {\n", - " \"backend\": \"networkx\",\n", - " \"merge_entities\": True,\n", - " \"resolution_strategy\": \"fuzzy\"\n", - " }\n", + " \"embedding\": {\"provider\": \"openai\", \"model\": \"text-embedding-3-small\"},\n", + " \"extraction\": {\"model\": \"gpt-4o-mini\", \"temperature\": 0.0},\n", + " \"vector_store\": {\"provider\": \"faiss\", \"dimension\": 1536},\n", + " \"knowledge_graph\": {\"backend\": \"networkx\", \"merge_entities\": True, \"resolution_strategy\": \"fuzzy\"}\n", "}\n", "\n", "config = ConfigManager().load_from_dict(config_dict)\n", "core = Semantica(config=config)\n", - "print(\"Config Loaded.\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2. Bootstrapping with Seed Data\n", "\n", - "We use `semantica.seed` to establish \"Ground Truth.\" This prevents the system from being solely dependent on AI extractions." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "

๐Ÿง  Semantica - ๐Ÿ“Š Current Progress

StatusActionModuleSubmoduleFileTime
โœ…Semantica is ingesting๐Ÿ“ฅ ingestFeedIngestorrss1.12s
โœ…Semantica is ingesting๐Ÿ“ฅ ingestWebIngestorREADME.md1.85s
โœ…Semantica is ingesting๐Ÿ“ฅ ingestWebIngestorREADME.md1.51s
โœ…Semantica is normalizing๐Ÿ”ง normalizeTextNormalizer-0.00s
โœ…Semantica is splittingโœ‚๏ธ splitEntityAwareChunker-1.93s
โœ…Semantica is extracting๐ŸŽฏ semantic_extractNERExtractor-0.87s
๐Ÿ”„Semantica is building๐Ÿง  kgGraphBuilder-677.02s
๐Ÿ”„Semantica is building๐Ÿง  kgEntityResolver-656.46s
๐Ÿ”„Semantica is deduplicating๐Ÿ”„ deduplicationDuplicateDetector-656.46s
๐Ÿ”„Semantica is deduplicating๐Ÿ”„ deduplicationSimilarityCalculator-0.01s
" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Foundation Graph Seeded with 2 Verified Nodes.\n" - ] - } - ], - "source": [ - "import json\n", - "from semantica.seed import SeedDataManager\n", - "\n", - "# Create sample ground truth entities\n", + "# 3. Seeding Ground Truth (Foundation Graph)\n", "foundation_data = {\n", " \"entities\": [\n", " {\"id\": \"python_org\", \"name\": \"Python Software Foundation\", \"type\": \"Organization\"},\n", @@ -162,182 +70,100 @@ " ]\n", "}\n", "\n", - "with open(\"ground_truth.json\", \"w\") as f:\n", - " json.dump(foundation_data, f)\n", + "with open(\"ground_truth.json\", \"w\") as f: json.dump(foundation_data, f)\n", "\n", "seed_manager = SeedDataManager()\n", "seed_manager.register_source(\"core_info\", \"json\", \"ground_truth.json\")\n", "foundation_graph = seed_manager.create_foundation_graph()\n", "\n", - "print(f\"Foundation Graph Seeded with {len(foundation_data['entities'])} Verified Nodes.\")" + "print(f\"โœ… Phase 0 Complete. Foundation Graph Seeded with {len(foundation_data['entities'])} Verified Nodes.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## 3. The Knowledge Hub: Massive Multi-Source Ingestion\n", + "## ๐Ÿ“ฅ Phase 1: Multi-Source Ingestion\n", "\n", - "We aggregate data from a diverse set of real-world sources using `semantica.ingest` and `semantica.parse`. \n", - "\n", - "### Data Sources\n", - "* **Official Docs**: Python.org, SQLAlchemy, Pydantic.\n", - "* **Live News (RSS)**: TechCrunch, Wired, Ars Technica.\n", - "* **Technical Blogs**: Real Python, Toward Data Science.\n", - "* **Engineering Repos**: Requests, HTTPX, Semantica." + "We aggregate live data from diverse sources using `semantica.ingest`." ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Ingesting Official Documentation...\n", - "\n", - "Fetching Live Tech News...\n", - "\n", - "Ingesting Engineering READMEs...\n", - "\n", - "Aggregated 14 documents from across the web.\n" - ] - } - ], + "outputs": [], "source": [ "from semantica.ingest import ingest_web, ingest_feed\n", "from semantica.parse import parse_document\n", "\n", "all_content = []\n", "\n", - "# 1. Web Domain Ingestion\n", - "print(\"Ingesting Official Documentation...\")\n", - "web_urls = [\n", - " \"https://www.python.org/about/\",\n", - " \"https://www.python.org/downloads/\",\n", - " \"https://realpython.com/\" # Fixed 404: updated from /python-news/\n", - "]\n", - "\n", + "# 1. Web & Docs\n", + "web_urls = [\"https://www.python.org/about/\", \"https://realpython.com/\"]\n", "for url in web_urls:\n", - " try:\n", - " # Returns a WebContent object\n", - " doc = ingest_web(url, method=\"url\")\n", - " all_content.append(doc.text)\n", - " except Exception as e:\n", - " print(f\"Failed to ingest {url}: {e}\")\n", + " try: all_content.append(ingest_web(url, method=\"url\").text)\n", + " except Exception as e: print(f\"Error ingesting {url}: {e}\")\n", "\n", "# 2. Live RSS Feeds\n", - "print(\"\\nFetching Live Tech News...\")\n", - "rss_feeds = [\n", - " \"http://feeds.bbci.co.uk/news/technology/rss.xml\",\n", - " \"https://techcrunch.com/feed/\",\n", - " \"https://www.wired.com/feed/rss\"\n", - "]\n", - "\n", + "rss_feeds = [\"https://techcrunch.com/feed/\", \"https://www.wired.com/feed/rss\"]\n", "for feed in rss_feeds:\n", " try:\n", - " # Returns a FeedData object\n", " feed_data = ingest_feed(feed, method=\"rss\")\n", - " # Extract top 3 items from each feed\n", - " for item in feed_data.items[:3]:\n", - " content = item.content if item.content else item.description\n", - " all_content.append(content)\n", - " except Exception as e:\n", - " print(f\"Failed to ingest feed {feed}: {e}\")\n", - "\n", - "# 3. Repository & Technical Files\n", - "print(\"\\nIngesting Engineering READMEs...\")\n", - "repo_files = [\n", - " \"https://raw.githubusercontent.com/psf/requests/main/README.md\",\n", - " \"https://raw.githubusercontent.com/encode/httpx/master/README.md\"\n", - "]\n", + " all_content.extend([item.content or item.description for item in feed_data.items[:2]])\n", + " except Exception as e: print(f\"Error ingesting feed {feed}: {e}\")\n", "\n", + "# 3. Technical READMEs\n", + "repo_files = [\"https://raw.githubusercontent.com/psf/requests/main/README.md\"]\n", "for file_url in repo_files:\n", - " try:\n", - " # Using ingest_web directly to ensure we get a WebContent object \n", - " # (avoiding the dictionary wrapper returned by the unified 'ingest' function)\n", - " doc = ingest_web(file_url, method=\"url\") \n", - " all_content.append(doc.text)\n", - " except Exception as e:\n", - " print(f\"Failed to ingest {file_url}: {e}\")\n", + " try: all_content.append(ingest_web(file_url, method=\"url\").text)\n", + " except Exception as e: print(f\"Error ingesting {file_url}: {e}\")\n", "\n", - "print(f\"\\nAggregated {len(all_content)} documents from across the web.\")" + "print(f\"โœ… Phase 1 Complete. Aggregated {len(all_content)} documents.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## 4. Normalization & Graph-Aware Chunking\n", + "## ๐Ÿ”ง Phase 2: Data Quality & Pre-processing\n", "\n", - "Standardizing noise and chunking for context preservation via `semantica.normalize` and `semantica.split`.\n", - "\n", - "### Why Graph-Aware Chunking?\n", - "Traditional recursive chunking often breaks entities and relationships across chunk boundaries. Semantica's **`EntityAwareChunker`** ensures that key entities and their semantic context are preserved within a single chunk, which is essential for building a coherent Knowledge Graph." + "We ensure the data is clean, structural, and split semantically to preserve entity relationships." ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\sentence_transformers\\cross_encoder\\CrossEncoder.py:13: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n", - " from tqdm.autonotebook import tqdm, trange\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Performing Graph-Aware Chunking (preserving entity boundaries)...\n", - "Generated 23 Graph-Aware chunks (vs 29 baseline chunks for sample).\n" - ] - } - ], + "outputs": [], "source": [ - "from semantica.normalize import TextNormalizer\n", - "from semantica.split import TextSplitter, EntityAwareChunker\n", + "from semantica.normalize import TextNormalizer, DataCleaner\n", + "from semantica.split import EntityAwareChunker\n", "\n", - "# 1. Normalization - Sanitizing input data\n", + "# 1. Normalization & Cleaning\n", "normalizer = TextNormalizer()\n", - "clean_data = [normalizer.normalize(text) for text in all_content if text]\n", + "cleaner = DataCleaner()\n", "\n", - "# 2. Standard Recursive Splitting (Baseline)\n", - "standard_splitter = TextSplitter(method=\"recursive\", chunk_size=1200, chunk_overlap=250)\n", - "standard_chunks = []\n", - "for doc in clean_data[:2]: # Sample for comparison\n", - " standard_chunks.extend(standard_splitter.split(doc))\n", - "\n", - "# 3. Advanced Graph-Aware Chunking (Entity Preservation)\n", - "print(\"Performing Graph-Aware Chunking (preserving entity boundaries)...\")\n", - "graph_aware_chunker = EntityAwareChunker(\n", - " chunk_size=1000, \n", - " chunk_overlap=200, \n", - " ner_method=\"ml\" # Can use \"llm\" for higher precision\n", - ")\n", + "normalized_data = [normalizer.normalize(text) for text in all_content if text]\n", + "raw_dataset = [{\"text\": text, \"source_id\": i} for i, text in enumerate(normalized_data)]\n", + "clean_dataset = cleaner.clean_data(raw_dataset, remove_duplicates=True)\n", "\n", + "# 2. Graph-Aware Chunking (Ensures entities are not split across chunks)\n", + "graph_aware_chunker = EntityAwareChunker(chunk_size=1000, chunk_overlap=200)\n", "all_chunks = []\n", - "for doc in clean_data:\n", - " # EntityAwareChunker ensures entities are not split across chunks\n", - " chunks = graph_aware_chunker.chunk(doc)\n", - " all_chunks.extend(chunks)\n", + "for doc in clean_dataset:\n", + " all_chunks.extend(graph_aware_chunker.chunk(doc['text']))\n", "\n", - "print(f\"Generated {len(all_chunks)} Graph-Aware chunks (vs {len(standard_chunks)} baseline chunks for sample).\")" + "print(f\"โœ… Phase 2 Complete. Generated {len(all_chunks)} high-quality semantic chunks.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## 5. Knowledge Graph Construction & Data Quality\n", + "## ๐Ÿ—๏ธ Phase 3: Graph Construction\n", "\n", - "Building the graph, then applying Conflict Resolution and Deduplication to ensure data integrity." + "We use LLM-driven extraction to build the initial Knowledge Graph." ] }, { @@ -347,147 +173,21 @@ "outputs": [], "source": [ "from semantica.kg import GraphBuilder\n", - "from semantica.deduplication import DuplicateDetector, EntityMerger\n", - "from semantica.conflicts import ConflictDetector, ConflictResolver\n", "\n", - "# 1. Initial Construction\n", + "print(\"Building Knowledge Graph (this may take a moment)...\")\n", "gb = GraphBuilder(merge_entities=True)\n", - "kg = gb.build(sources=[{\"text\": str(c.text)} for c in all_chunks[:12]])\n", + "kg = gb.build(sources=[{\"text\": str(c.text)} for c in all_chunks[:10]])\n", "\n", - "# 2. Quality Control: Deduplication\n", - "detector = DuplicateDetector(similarity_threshold=0.85)\n", - "# Accessing entities from the KG dictionary structure\n", - "entities = kg.get(\"entities\", [])\n", - "duplicates = detector.detect_duplicates(entities)\n", - "\n", - "if duplicates:\n", - " merger = EntityMerger()\n", - " # Merging returns an updated graph dictionary\n", - " kg = merger.merge_duplicates(kg, duplicates)\n", - " print(f\"Deduplicated {len(duplicates)} Entity Pairs.\")\n", - "\n", - "# 3. Quality Control: Conflict Resolution\n", - "conflict_detector = ConflictDetector()\n", - "conflicts = conflict_detector.detect_conflicts(kg)\n", - "if conflicts:\n", - " resolver = ConflictResolver()\n", - " kg = resolver.resolve_conflicts(kg, conflicts, strategy=\"most_recent\")\n", - " print(f\"Resolved {len(conflicts)} Data Conflicts.\")\n", - "\n", - "print(f\"High-Quality Knowledge Graph Ready. Entities: {len(kg['entities'])}, Relations: {len(kg['relationships'])}\")" + "print(f\"โœ… Phase 3 Complete. Entities: {len(kg['entities'])}, Relations: {len(kg['relationships'])}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## 6. Graph Synthesis & Advanced Reasoning\n", + "## โœจ Phase 4: Graph Refinement & Quality\n", "\n", - "We apply Graph Analytics and the Reasoning module to derive insights not explicitly stated in the text." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Graph is empty or has no edges, returning 0 communities\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Top Influential Entities: []\n", - "Network Connectivity Profile: sparse\n", - "Inference Engine initialized with Domain Rules.\n" - ] - } - ], - "source": [ - "from semantica.kg import CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n", - "from semantica.reasoning import InferenceEngine, InferenceStrategy\n", - "\n", - "# 1. Analytics - Mapping the Influence\n", - "centrality_result = CentralityCalculator().calculate_degree_centrality(kg)\n", - "top_nodes = centrality_result.get(\"rankings\", [])[:5]\n", - "\n", - "communities = CommunityDetector().detect_communities(kg, algorithm=\"louvain\")\n", - "\n", - "# 2. Graph Connectivity Analysis - Understanding the Network\n", - "analyzer = ConnectivityAnalyzer()\n", - "connectivity = analyzer.analyze_graph_structure(kg)\n", - "\n", - "# 3. Logical Inference - Deriving Hidden Relationships\n", - "engine = InferenceEngine(strategy=\"forward\")\n", - "# Example: Adding a domain rule (If X is a 'Library' and Y is a 'Language', then X 'BuiltWith' Y)\n", - "engine.add_rule(\"IF ?x :type 'Library' AND ?y :type 'Language' THEN ?x :builtWith ?y\")\n", - "# In practice, facts would be extracted from the KG entities and relationships\n", - "# inference_results = engine.infer(facts, rules)\n", - "\n", - "print(f\"Top Influential Entities: {[n['node'] for n in top_nodes]}\")\n", - "print(f\"Network Connectivity Profile: {connectivity.get('structure_type', 'interconnected')}\")\n", - "print(\"Inference Engine initialized with Domain Rules.\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 7. Hybrid Context Retrieval\n", - "\n", - "Storage using `vector_store` and wrapping it in `AgentContext`." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "fastembed not available. Install with: pip install fastembed. Using fallback embedding method.\n" - ] - }, - { - "ename": "AttributeError", - "evalue": "'Semantica' object has no attribute 'generate_embeddings'", - "output_type": "error", - "traceback": [ - "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)", - "Cell \u001b[1;32mIn[13], line 5\u001b[0m\n\u001b[0;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21;01msemantica\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcontext\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m AgentContext\n\u001b[0;32m 4\u001b[0m vs \u001b[38;5;241m=\u001b[39m VectorStore(backend\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfaiss\u001b[39m\u001b[38;5;124m\"\u001b[39m, dimension\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1536\u001b[39m)\n\u001b[1;32m----> 5\u001b[0m embeddings \u001b[38;5;241m=\u001b[39m \u001b[43mcore\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgenerate_embeddings\u001b[49m([\u001b[38;5;28mstr\u001b[39m(c\u001b[38;5;241m.\u001b[39mtext) \u001b[38;5;28;01mfor\u001b[39;00m c \u001b[38;5;129;01min\u001b[39;00m all_chunks[:\u001b[38;5;241m12\u001b[39m]])\n\u001b[0;32m 6\u001b[0m vs\u001b[38;5;241m.\u001b[39mstore_vectors(vectors\u001b[38;5;241m=\u001b[39membeddings, metadata\u001b[38;5;241m=\u001b[39m[{\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtext\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;28mstr\u001b[39m(c\u001b[38;5;241m.\u001b[39mtext)} \u001b[38;5;28;01mfor\u001b[39;00m c \u001b[38;5;129;01min\u001b[39;00m all_chunks[:\u001b[38;5;241m12\u001b[39m]])\n\u001b[0;32m 8\u001b[0m \u001b[38;5;66;03m# Global Context Manager for an Agent\u001b[39;00m\n", - "\u001b[1;31mAttributeError\u001b[0m: 'Semantica' object has no attribute 'generate_embeddings'" - ] - } - ], - "source": [ - "from semantica.vector_store import VectorStore\n", - "from semantica.context import AgentContext\n", - "\n", - "vs = VectorStore(backend=\"faiss\", dimension=1536)\n", - "embeddings = core.generate_embeddings([str(c.text) for c in all_chunks[:12]])\n", - "vs.store_vectors(vectors=embeddings, metadata=[{\"text\": str(c.text)} for c in all_chunks[:12]])\n", - "\n", - "# Global Context Manager for an Agent\n", - "context = AgentContext(vector_store=vs, knowledge_graph=kg)\n", - "\n", - "print(\"Hybrid Context Store Initialized.\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 8. Immersive Visualization\n", - "\n", - "We use `semantica.visualization` to create a community-aware network map." + "We refine the raw graph into a production-grade knowledge base." ] }, { @@ -496,26 +196,68 @@ "metadata": {}, "outputs": [], "source": [ - "from semantica.visualization import KGVisualizer\n", - "import matplotlib.pyplot as plt\n", + "from semantica.deduplication import DuplicateDetector, EntityMerger\n", + "from semantica.conflicts import ConflictDetector, ConflictResolver\n", + "from semantica.kg import GraphValidator\n", "\n", - "viz = KGVisualizer()\n", - "viz.visualize_network(\n", - " kg, \n", - " layout=\"spring\", \n", - " output=\"static\",\n", - " title=\"Python Ecosystem Intelligence Graph (Multi-Source)\"\n", - ")\n", - "plt.show()" + "# 1. Deduplication\n", + "detector = DuplicateDetector(similarity_threshold=0.85)\n", + "duplicates = detector.detect_duplicates(kg.get(\"entities\", []))\n", + "if duplicates:\n", + " kg = EntityMerger().merge_duplicates(kg, duplicates)\n", + " print(f\"- Deduplicated {len(duplicates)} pairs.\")\n", + "\n", + "# 2. Conflict Resolution\n", + "conflicts = ConflictDetector().detect_conflicts(kg)\n", + "if conflicts:\n", + " kg = ConflictResolver().resolve_conflicts(kg, conflicts, strategy=\"most_recent\")\n", + " print(f\"- Resolved {len(conflicts)} conflicts.\")\n", + "\n", + "# 3. Final Validation\n", + "result = GraphValidator().validate(kg)\n", + "status = \"โœ… Valid\" if result.is_valid else f\"โš ๏ธ {len(result.issues)} issues\"\n", + "\n", + "print(f\"โœ… Phase 4 Complete. Graph Status: {status}.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## 9. Modular Orchestration: The Pipeline\n", + "## ๐Ÿงช Phase 5: Synthesis, Analytics & Visualization\n", "\n", - "Finally, we show how to wrap this whole complex flow into a single `semantica.pipeline.Pipeline` object for automation." + "We apply Graph Analytics and Visualization to derive insights." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import CentralityCalculator, CommunityDetector\n", + "from semantica.visualization import KGVisualizer\n", + "import matplotlib.pyplot as plt\n", + "\n", + "# 1. Analytics\n", + "centrality = CentralityCalculator().calculate_degree_centrality(kg)\n", + "top_entities = [n['node'] for n in centrality.get(\"rankings\", [])[:3]]\n", + "\n", + "# 2. Visualization\n", + "viz = KGVisualizer()\n", + "viz.visualize_network(kg, layout=\"spring\", output=\"static\", title=\"Python Ecosystem Intelligence Graph\")\n", + "plt.show()\n", + "\n", + "print(f\"โœ… Phase 5 Complete. Top Entities: {top_entities}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ๐Ÿ“ฆ Phase 6: Orchestration & Export\n", + "\n", + "Wrapping everything into a repeatable pipeline and exporting the results." ] }, { @@ -525,41 +267,21 @@ "outputs": [], "source": [ "from semantica.pipeline import PipelineBuilder\n", - "\n", - "builder = PipelineBuilder()\n", - "knowledge_pipeline = (\n", - " builder.add_step(\"ingest\", \"knowledge_hub_loader\")\n", - " .add_step(\"normalize\", \"text_normalizer\")\n", - " .add_step(\"split\", \"semantic_splitter\")\n", - " .add_step(\"enrich\", \"kg_builder\")\n", - " .add_step(\"validate\", \"quality_assurance\")\n", - " .build()\n", - ")\n", - "\n", - "print(\"Unified Knowledge Pipeline Construct Complete.\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 10. Persistence & Export\n", - "\n", - "Save the finalized knowledge structures." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ "from semantica.export import GraphExporter\n", "\n", - "exporter = GraphExporter()\n", - "exporter.export_to_json(kg, \"master_ecosystem_graph.json\")\n", + "# 1. Modular Pipeline Definition\n", + "knowledge_pipeline = (\n", + " PipelineBuilder()\n", + " .add_step(\"ingest\", \"web_loader\")\n", + " .add_step(\"normalize\", \"cleaner\")\n", + " .add_step(\"enrich\", \"kg_builder\")\n", + " .build()\n", + ")\n", "\n", - "print(\"Project Exported. Deployment Ready.\")" + "# 2. Export\n", + "GraphExporter().export_to_json(kg, \"final_ecosystem_graph.json\")\n", + "\n", + "print(\"โœ… Pipeline Orchestration & Export Complete. Project Ready for Deployment.\")" ] } ],