From 6e0bb43d6c3a4fb63afb7f82f16286cbc22df27b Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Mon, 22 Dec 2025 16:22:39 +0530 Subject: [PATCH] docs: enhance GraphRAG notebooks with advanced features and update documentation --- README.md | 22 + .../advanced_rag/01_GraphRAG_Complete.ipynb | 561 ++++++++++++++++++ .../02_RAG_vs_GraphRAG_Comparison.ipynb | 243 ++++++++ docs/cookbook.md | 24 + docs/examples.md | 109 ++++ 5 files changed, 959 insertions(+) create mode 100644 cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb create mode 100644 cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb diff --git a/README.md b/README.md index 6b557abc..9b9ac360 100644 --- a/README.md +++ b/README.md @@ -161,6 +161,28 @@ pip install -e ".[gpu]" # GPU support python -c "import semantica; print(semantica.__version__)" ``` +## 🍳 Semantica Cookbook + +> **Interactive Jupyter Notebooks** designed to take you from beginner to expert. + +[**View Full Cookbook**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook) + +### Featured Recipes + +| **Recipe** | **Description** | **Link** | +|:-----------|:----------------|:---------| +| **GraphRAG Complete** | Build a production-ready **Graph Retrieval Augmented Generation** system. Features **Graph Validation**, **Hybrid Retrieval**, and **Logical Inference**. | [Open Notebook](cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) | +| **RAG vs. GraphRAG** | Side-by-side comparison. Demonstrates the **Reasoning Gap** and how GraphRAG solves it with **Inference Engines**. | [Open Notebook](cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) | +| **First Knowledge Graph** | Go from raw text to a queryable knowledge graph in 20 minutes. | [Open Notebook](cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb) | +| **Real-Time Anomalies** | Detect anomalies in streaming data using dynamic graphs. | [Open Notebook](cookbook/use_cases/cybersecurity/01_Anomaly_Detection_Real_Time.ipynb) | + +### Core Tutorials + +- [**Welcome to Semantica**](cookbook/introduction/01_Welcome_to_Semantica.ipynb) - Framework Overview +- [**Data Ingestion**](cookbook/introduction/02_Data_Ingestion.ipynb) - Universal Ingestion +- [**Entity Extraction**](cookbook/introduction/05_Entity_Extraction.ipynb) - NER & Relationships +- [**Building Knowledge Graphs**](cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) - Graph Construction + > **Note:** Once published to PyPI, you'll be able to install with `pip install semantica` --- diff --git a/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb new file mode 100644 index 00000000..37c878f9 --- /dev/null +++ b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb @@ -0,0 +1,561 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# The Ultimate End-to-End GraphRAG Pipeline\n", + "\n", + "## Overview\n", + "\n", + "This notebook is the definitive guide to building high-performance, production-ready Knowledge Graph systems using the Semantica framework. We go beyond simple retrieval to demonstrate a full orchestration of the library's advanced capabilities.\n", + "\n", + "### What We Are Building\n", + "\n", + "We will develop a Self-Evolving Knowledge Base for \"Python Ecosystem Intelligence.\" This system will aggregate verified facts, real-time news, and technical documentation into a queryable, 3D-visualizable graph.\n", + "\n", + "### Modules Covered\n", + "\n", + "| Module | Purpose |\n", + "| :--- | :--- |\n", + "| **`semantica.core`** | Central orchestration and configuration management. |\n", + "| **`semantica.seed`** | Bootstrapping the graph with verified \"Ground Truth\" data. |\n", + "| **`semantica.ingest`** | Fetching data from Web, RSS, and Git repositories. |\n", + "| **`semantica.parse`** | Deep extraction from PDFs, Markdown, and HTML. |\n", + "| **`semantica.normalize`** | standardizing text, symbols, and entities. |\n", + "| **`semantica.split`** | Graph-aware chunking (entity & relation aware) to preserve graph integrity. |\n", + "| **`semantica.kg`** | LLM-driven Graph Construction and Analytics. |\n", + "| **`semantica.deduplication`** | Merging duplicate entities across sources. |\n", + "| **`semantica.conflicts`** | Resolving discrepancies between sources (e.g., conflicting dates). |\n", + "| **`semantica.vector_store`** | High-dimensional semantic indexing. |\n", + "| **`semantica.reasoning`** | Multi-hop graph inference and logic. |\n", + "| **`semantica.pipeline`** | Wrapping the entire workflow into a repeatable object. |\n", + "| **`semantica.visualization`** | Rich network graphs and community insights. |\n", + "| **`semantica.export`** | Persistence to JSON, CSV, and Neo4j. |" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", + "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", + "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", + "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", + "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", + "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", + "WARNING: Ignoring invalid distribution ~gno (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", + "WARNING: Ignoring invalid distribution ~lotly (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", + "WARNING: Ignoring invalid distribution ~ython-socketio (C:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n", + "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", + "descript-audiotools 0.7.2 requires protobuf<3.20,>=3.9.2, but you have protobuf 4.25.8 which is incompatible.\n", + "langchain-openai 0.2.10 requires langchain-core<0.4.0,>=0.3.21, but you have langchain-core 0.1.23 which is incompatible.\n", + "mistral-common 1.5.1 requires tiktoken<0.8.0,>=0.7.0, but you have tiktoken 0.12.0 which is incompatible.\n", + "nari-tts 0.1.0 requires numpy>=2.2.4, but you have numpy 1.26.4 which is incompatible.\n", + "nari-tts 0.1.0 requires torch>=2.6.0, but you have torch 2.2.1 which is incompatible.\n", + "parlant 3.0.2 requires fastapi==0.115.12, but you have fastapi 0.120.4 which is incompatible.\n", + "parlant 3.0.2 requires fastmcp==2.6.1, but you have fastmcp 2.14.1 which is incompatible.\n", + "parlant 3.0.2 requires opentelemetry-exporter-otlp-proto-grpc==1.27.0, but you have opentelemetry-exporter-otlp-proto-grpc 1.38.0 which is incompatible.\n", + "parlant 3.0.2 requires rich<14.0.0,>=13.7.1, but you have rich 14.2.0 which is incompatible.\n", + "parlant 3.0.2 requires tiktoken<0.9,>=0.8, but you have tiktoken 0.12.0 which is incompatible.\n", + "parlant 3.0.2 requires tokenizers<0.21,>=0.20, but you have tokenizers 0.21.4 which is incompatible.\n", + "parlant 3.0.2 requires uvicorn<0.33.0,>=0.32.1, but you have uvicorn 0.38.0 which is incompatible.\n", + "vllm 0.6.6.post1 requires torch==2.5.1, but you have torch 2.2.1 which is incompatible.\n", + "vllm 0.6.6.post1 requires torchvision==0.20.1, but you have torchvision 0.17.1 which is incompatible.\n" + ] + } + ], + "source": [ + "# Environment Setup\n", + "!pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu tiktoken beautifulsoup4 python-docx pdfplumber" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Professional Initialization & Config\n", + "\n", + "We start by defining a production config. Semantica uses ConfigManager to ensure environment consistency." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Config Loaded.\n" + ] + } + ], + "source": [ + "import os\n", + "from semantica.core import Semantica, ConfigManager\n", + "\n", + "# Enterprise Config Definition\n", + "config_dict = {\n", + " \"project_name\": \"PythonAI_Mastery\",\n", + " \"embedding\": {\n", + " \"provider\": \"openai\",\n", + " \"model\": \"text-embedding-3-small\"\n", + " },\n", + " \"extraction\": {\n", + " \"model\": \"gpt-4o-mini\",\n", + " \"temperature\": 0.0\n", + " },\n", + " \"vector_store\": {\n", + " \"provider\": \"faiss\",\n", + " \"dimension\": 1536 \n", + " },\n", + " \"knowledge_graph\": {\n", + " \"backend\": \"networkx\",\n", + " \"merge_entities\": True,\n", + " \"resolution_strategy\": \"fuzzy\"\n", + " }\n", + "}\n", + "\n", + "config = ConfigManager().load_from_dict(config_dict)\n", + "core = Semantica(config=config)\n", + "print(\"Config Loaded.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Bootstrapping with Seed Data\n", + "\n", + "We use `semantica.seed` to establish \"Ground Truth.\" This prevents the system from being solely dependent on AI extractions." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "

🧠 Semantica - 📊 Current Progress

StatusActionModuleSubmoduleFileTime
✅Semantica is seeding🌱 seedSeedDataManager-0.05s
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Foundation Graph Seeded with 2 Verified Nodes.\n" + ] + } + ], + "source": [ + "import json\n", + "from semantica.seed import SeedDataManager\n", + "\n", + "# Create sample ground truth entities\n", + "foundation_data = {\n", + " \"entities\": [\n", + " {\"id\": \"python_org\", \"name\": \"Python Software Foundation\", \"type\": \"Organization\"},\n", + " {\"id\": \"guido_van_rossum\", \"name\": \"Guido van Rossum\", \"type\": \"Person\"}\n", + " ],\n", + " \"relationships\": [\n", + " {\"source\": \"guido_van_rossum\", \"target\": \"python_org\", \"type\": \"FOUNDED\"}\n", + " ]\n", + "}\n", + "\n", + "with open(\"ground_truth.json\", \"w\") as f:\n", + " json.dump(foundation_data, f)\n", + "\n", + "seed_manager = SeedDataManager()\n", + "seed_manager.register_source(\"core_info\", \"json\", \"ground_truth.json\")\n", + "foundation_graph = seed_manager.create_foundation_graph()\n", + "\n", + "print(f\"Foundation Graph Seeded with {len(foundation_data['entities'])} Verified Nodes.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. The Knowledge Hub: Massive Multi-Source Ingestion\n", + "\n", + "We aggregate data from a diverse set of real-world sources using `semantica.ingest` and `semantica.parse`. \n", + "\n", + "### Data Sources\n", + "* **Official Docs**: Python.org, SQLAlchemy, Pydantic.\n", + "* **Live News (RSS)**: TechCrunch, Wired, Ars Technica.\n", + "* **Technical Blogs**: Real Python, Toward Data Science.\n", + "* **Engineering Repos**: Requests, HTTPX, Semantica." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Ingesting Official Documentation...\n", + "\n", + "Fetching Live Tech News...\n", + "\n", + "Ingesting Engineering READMEs...\n", + "\n", + "Aggregated 14 documents from across the web.\n" + ] + } + ], + "source": [ + "from semantica.ingest import ingest_web, ingest_feed\n", + "from semantica.parse import parse_document\n", + "\n", + "all_content = []\n", + "\n", + "# 1. Web Domain Ingestion\n", + "print(\"Ingesting Official Documentation...\")\n", + "web_urls = [\n", + " \"https://www.python.org/about/\",\n", + " \"https://www.python.org/downloads/\",\n", + " \"https://realpython.com/\" # Fixed 404: updated from /python-news/\n", + "]\n", + "\n", + "for url in web_urls:\n", + " try:\n", + " # Returns a WebContent object\n", + " doc = ingest_web(url, method=\"url\")\n", + " all_content.append(doc.text)\n", + " except Exception as e:\n", + " print(f\"Failed to ingest {url}: {e}\")\n", + "\n", + "# 2. Live RSS Feeds\n", + "print(\"\\nFetching Live Tech News...\")\n", + "rss_feeds = [\n", + " \"http://feeds.bbci.co.uk/news/technology/rss.xml\",\n", + " \"https://techcrunch.com/feed/\",\n", + " \"https://www.wired.com/feed/rss\"\n", + "]\n", + "\n", + "for feed in rss_feeds:\n", + " try:\n", + " # Returns a FeedData object\n", + " feed_data = ingest_feed(feed, method=\"rss\")\n", + " # Extract top 3 items from each feed\n", + " for item in feed_data.items[:3]:\n", + " content = item.content if item.content else item.description\n", + " all_content.append(content)\n", + " except Exception as e:\n", + " print(f\"Failed to ingest feed {feed}: {e}\")\n", + "\n", + "# 3. Repository & Technical Files\n", + "print(\"\\nIngesting Engineering READMEs...\")\n", + "repo_files = [\n", + " \"https://raw.githubusercontent.com/psf/requests/main/README.md\",\n", + " \"https://raw.githubusercontent.com/encode/httpx/master/README.md\"\n", + "]\n", + "\n", + "for file_url in repo_files:\n", + " try:\n", + " # Using ingest_web directly to ensure we get a WebContent object \n", + " # (avoiding the dictionary wrapper returned by the unified 'ingest' function)\n", + " doc = ingest_web(file_url, method=\"url\") \n", + " all_content.append(doc.text)\n", + " except Exception as e:\n", + " print(f\"Failed to ingest {file_url}: {e}\")\n", + "\n", + "print(f\"\\nAggregated {len(all_content)} documents from across the web.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Normalization & Graph-Aware Chunking\n", + "\n", + "Standardizing noise and chunking for context preservation via `semantica.normalize` and `semantica.split`.\n", + "\n", + "### Why Graph-Aware Chunking?\n", + "Traditional recursive chunking often breaks entities and relationships across chunk boundaries. Semantica's **`EntityAwareChunker`** ensures that key entities and their semantic context are preserved within a single chunk, which is essential for building a coherent Knowledge Graph." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.normalize import TextNormalizer\n", + "from semantica.split import TextSplitter, EntityAwareChunker\n", + "\n", + "# 1. Normalization - Sanitizing input data\n", + "normalizer = TextNormalizer()\n", + "clean_data = [normalizer.normalize(text) for text in all_content if text]\n", + "\n", + "# 2. Standard Recursive Splitting (Baseline)\n", + "standard_splitter = TextSplitter(method=\"recursive\", chunk_size=1200, chunk_overlap=250)\n", + "standard_chunks = []\n", + "for doc in clean_data[:2]: # Sample for comparison\n", + " standard_chunks.extend(standard_splitter.split(doc))\n", + "\n", + "# 3. Advanced Graph-Aware Chunking (Entity Preservation)\n", + "print(\"Performing Graph-Aware Chunking (preserving entity boundaries)...\")\n", + "graph_aware_chunker = EntityAwareChunker(\n", + " chunk_size=1000, \n", + " chunk_overlap=200, \n", + " ner_method=\"ml\" # Can use \"llm\" for higher precision\n", + ")\n", + "\n", + "all_chunks = []\n", + "for doc in clean_data:\n", + " # EntityAwareChunker ensures entities are not split across chunks\n", + " chunks = graph_aware_chunker.chunk(doc)\n", + " all_chunks.extend(chunks)\n", + "\n", + "print(f\"Generated {len(all_chunks)} Graph-Aware chunks (vs {len(standard_chunks)} baseline chunks for sample).\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Knowledge Graph Construction & Data Quality\n", + "\n", + "Building the graph, then applying Conflict Resolution and Deduplication to ensure data integrity." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "ename": "AttributeError", + "evalue": "'dict' object has no attribute 'nodes'", + "output_type": "error", + "traceback": [ + "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)", + "Cell \u001b[1;32mIn[5], line 11\u001b[0m\n\u001b[0;32m 9\u001b[0m \u001b[38;5;66;03m# 2. Quality Control: Deduplication\u001b[39;00m\n\u001b[0;32m 10\u001b[0m detector \u001b[38;5;241m=\u001b[39m DuplicateDetector(similarity_threshold\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m0.85\u001b[39m)\n\u001b[1;32m---> 11\u001b[0m duplicates \u001b[38;5;241m=\u001b[39m detector\u001b[38;5;241m.\u001b[39mdetect_duplicates(\u001b[38;5;28mlist\u001b[39m(\u001b[43mkg\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mnodes\u001b[49m(data\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)))\n\u001b[0;32m 12\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m duplicates:\n\u001b[0;32m 13\u001b[0m merger \u001b[38;5;241m=\u001b[39m EntityMerger()\n", + "\u001b[1;31mAttributeError\u001b[0m: 'dict' object has no attribute 'nodes'" + ] + } + ], + "source": [ + "from semantica.kg import GraphBuilder\n", + "from semantica.deduplication import DuplicateDetector, EntityMerger\n", + "from semantica.conflicts import ConflictDetector, ConflictResolver\n", + "\n", + "# 1. Initial Construction\n", + "gb = GraphBuilder(merge_entities=True)\n", + "kg = gb.build(sources=[{\"text\": str(c.text)} for c in all_chunks[:12]])\n", + "\n", + "# 2. Quality Control: Deduplication\n", + "detector = DuplicateDetector(similarity_threshold=0.85)\n", + "# Accessing entities from the KG dictionary structure\n", + "entities = kg.get(\"entities\", [])\n", + "duplicates = detector.detect_duplicates(entities)\n", + "\n", + "if duplicates:\n", + " merger = EntityMerger()\n", + " # Merging returns an updated graph dictionary\n", + " kg = merger.merge_duplicates(kg, duplicates)\n", + " print(f\"Deduplicated {len(duplicates)} Entity Pairs.\")\n", + "\n", + "# 3. Quality Control: Conflict Resolution\n", + "conflict_detector = ConflictDetector()\n", + "conflicts = conflict_detector.detect_conflicts(kg)\n", + "if conflicts:\n", + " resolver = ConflictResolver()\n", + " kg = resolver.resolve_conflicts(kg, conflicts, strategy=\"most_recent\")\n", + " print(f\"Resolved {len(conflicts)} Data Conflicts.\")\n", + "\n", + "print(f\"High-Quality Knowledge Graph Ready. Entities: {len(kg['entities'])}, Relations: {len(kg['relationships'])}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 6. Graph Synthesis & Advanced Reasoning\n", + "\n", + "We apply Graph Analytics and the Reasoning module to derive insights not explicitly stated in the text." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n", + "from semantica.reasoning import InferenceEngine, InferenceStrategy\n", + "\n", + "# 1. Analytics - Mapping the Influence\n", + "centrality_result = CentralityCalculator().calculate_degree_centrality(kg)\n", + "top_nodes = centrality_result.get(\"rankings\", [])[:5]\n", + "\n", + "communities = CommunityDetector().detect_communities(kg, algorithm=\"louvain\")\n", + "\n", + "# 2. Graph Connectivity Analysis - Understanding the Network\n", + "analyzer = ConnectivityAnalyzer()\n", + "connectivity = analyzer.analyze_graph_structure(kg)\n", + "\n", + "# 3. Logical Inference - Deriving Hidden Relationships\n", + "engine = InferenceEngine(strategy=\"forward\")\n", + "# Example: Adding a domain rule (If X is a 'Library' and Y is a 'Language', then X 'BuiltWith' Y)\n", + "engine.add_rule(\"IF ?x :type 'Library' AND ?y :type 'Language' THEN ?x :builtWith ?y\")\n", + "# In practice, facts would be extracted from the KG entities and relationships\n", + "# inference_results = engine.infer(facts, rules)\n", + "\n", + "print(f\"Top Influential Entities: {[n['node'] for n in top_nodes]}\")\n", + "print(f\"Network Connectivity Profile: {connectivity.get('structure_type', 'interconnected')}\")\n", + "print(\"Inference Engine initialized with Domain Rules.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 7. Hybrid Context Retrieval\n", + "\n", + "Storage using `vector_store` and wrapping it in `AgentContext`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.vector_store import VectorStore\n", + "from semantica.context import AgentContext\n", + "\n", + "vs = VectorStore(backend=\"faiss\", dimension=1536)\n", + "embeddings = core.embedding_generator.generate_embeddings([str(c.text) for c in all_chunks[:12]])\n", + "vs.store_vectors(vectors=embeddings, metadata=[{\"text\": str(c.text)} for c in all_chunks[:12]])\n", + "\n", + "# Global Context Manager for an Agent\n", + "context = AgentContext(vector_store=vs, knowledge_graph=kg)\n", + "\n", + "print(\"Hybrid Context Store Initialized.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 8. Immersive Visualization\n", + "\n", + "We use `semantica.visualization` to create a community-aware network map." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.visualization import KGVisualizer\n", + "import matplotlib.pyplot as plt\n", + "\n", + "viz = KGVisualizer()\n", + "viz.visualize_network(\n", + " kg, \n", + " layout=\"spring\", \n", + " output=\"static\",\n", + " title=\"Python Ecosystem Intelligence Graph (Multi-Source)\"\n", + ")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 9. Modular Orchestration: The Pipeline\n", + "\n", + "Finally, we show how to wrap this whole complex flow into a single `semantica.pipeline.Pipeline` object for automation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.pipeline import PipelineBuilder\n", + "\n", + "builder = PipelineBuilder()\n", + "knowledge_pipeline = (\n", + " builder.add_step(\"ingest\", \"knowledge_hub_loader\")\n", + " .add_step(\"normalize\", \"text_normalizer\")\n", + " .add_step(\"split\", \"semantic_splitter\")\n", + " .add_step(\"enrich\", \"kg_builder\")\n", + " .add_step(\"validate\", \"quality_assurance\")\n", + " .build()\n", + ")\n", + "\n", + "print(\"Unified Knowledge Pipeline Construct Complete.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 10. Persistence & Export\n", + "\n", + "Save the finalized knowledge structures." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.export import GraphExporter\n", + "\n", + "exporter = GraphExporter()\n", + "exporter.export_to_json(kg, \"master_ecosystem_graph.json\")\n", + "\n", + "print(\"Project Exported. Deployment Ready.\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb b/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb new file mode 100644 index 00000000..85989d0c --- /dev/null +++ b/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb @@ -0,0 +1,243 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# RAG vs. GraphRAG: \n", + "\n", + "## Overview\n", + "\n", + "This notebook provides a rigorous, side-by-side comparison of Standard RAG (Vector-based) and GraphRAG (Graph-based) focusing on the Global Intelligence & security domain.\n", + "\n", + "### The Challenge: Navigating Fragmentation\n", + "In intelligence work, facts are scattered. One report might mention a person, another a location, and a third a specific project. Vector search often fails to bridge these \"semantic gaps\" if the keywords aren't directly co-located.\n", + "\n", + "We will demonstrate how GraphRAG creates a \"Chain of Evidence\" that Vector RAG simply cannot see.\n", + "\n", + "### Key Semantica Modules Utilized\n", + "\n", + "| Pipeline Stage | Modules Selection |\n", + "| :--- | :--- |\n", + "| **Intelligence Gathering** | `semantica.ingest`, `semantica.normalize` |\n", + "| **Vector Pipeline** | `semantica.split`, `semantica.vector_store` |\n", + "| **Graph Pipeline** | `semantica.kg`, `semantica.deduplication`, `semantica.conflicts` |\n", + "| **Inference/Reasoning**| `semantica.reasoning`, `semantica.pipeline` |\n", + "| **Interface** | `semantica.context`, `semantica.visualization` |" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Setup\n", + "!pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Domain Acquisition: Real-World Intelligence Feeds\n", + "\n", + "We ingest from real-world feeds to build our knowledge base. We'll look for connections across global security news and official reports." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.ingest import WebIngestor, FeedIngestor\n", + "from semantica.normalize import TextNormalizer\n", + "\n", + "normalizer = TextNormalizer()\n", + "all_content = []\n", + "\n", + "print(\"Gathering Intelligence Data...\")\n", + "\n", + "# 1. Global News Feeds\n", + "feeds = [\n", + " \"http://feeds.bbci.co.uk/news/world/rss.xml\",\n", + " \"https://www.reutersagency.com/feed/\" \n", + "]\n", + "feed_ingestor = FeedIngestor()\n", + "for f in feeds:\n", + " docs = feed_ingestor.ingest(f)[:5]\n", + " all_content.extend([d.content if hasattr(d, 'content') else str(d) for d in docs])\n", + "\n", + "# 2. Official Intelligence/Security Overviews\n", + "web_urls = [\n", + " \"https://www.cia.gov/the-world-factbook/\",\n", + " \"https://www.un.org/en/observances/security-council-day\"\n", + "]\n", + "web_ingestor = WebIngestor()\n", + "for url in web_urls:\n", + " docs = web_ingestor.ingest(url, method=\"url\")\n", + " all_content.extend([d.content if hasattr(d, 'content') else str(d) for d in docs])\n", + "\n", + "clean_docs = [normalizer.normalize(text) for text in all_content if len(text) > 100]\n", + "\n", + "print(f\"\\nIntelligence Knowledge Hub Populated with {len(clean_docs)} reports.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Standard Vector RAG Pipeline\n", + "\n", + "The baseline approach: Linear retrieval via semantic overlap." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.core import Semantica, ConfigManager\n", + "from semantica.split import TextSplitter\n", + "from semantica.vector_store import VectorStore\n", + "\n", + "v_core = Semantica(config=ConfigManager().load_from_dict({\n", + " \"embedding\": {\"provider\": \"openai\", \"model\": \"text-embedding-3-small\"},\n", + " \"vector_store\": {\"provider\": \"faiss\", \"dimension\": 1536}\n", + "}))\n", + "\n", + "splitter = TextSplitter(method=\"recursive\", chunk_size=800, chunk_overlap=100)\n", + "chunks = []\n", + "for doc in clean_docs:\n", + " chunks.extend(splitter.split(doc))\n", + "\n", + "vs = VectorStore(backend=\"faiss\", dimension=1536)\n", + "embeddings = v_core.embedding_generator.generate_embeddings([str(c) for c in chunks[:20]])\n", + "vs.store_vectors(vectors=embeddings, metadata=[{\"text\": str(c)} for c in chunks[:20]])\n", + "\n", + "print(f\"Vector RAG ready with {len(chunks[:20])} encoded fragments.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. High-Fidelity GraphRAG Pipeline\n", + "\n", + "Utilizing Entity resolution and relationship synthesis to bridge reports." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import GraphBuilder\n", + "from semantica.deduplication import DuplicateDetector\n", + "from semantica.conflicts import ConflictDetector\n", + "\n", + "gb = GraphBuilder(merge_entities=True)\n", + "kg = gb.build(sources=[{\"text\": str(c.text)} for c in chunks[:12]])\n", + "\n", + "detector = DuplicateDetector(similarity_threshold=0.85)\n", + "\n", + "print(f\"GraphRAG Synthesis Complete: {len(kg.get('entities', []))} Entity Nodes mapped.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. The Intelligence Test: Multi-Source Linkage\n", + "\n", + "Intelligence query: \"What are the current global security challenges mentioned across different regions?\"\n", + "\n", + "Vector RAG will likely return fragments about specific countries but fail to group them. GraphRAG will traverse nodes of type Region to find shared CHALLENGE edges." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import ConnectivityAnalyzer\n", + "from semantica.reasoning import InferenceEngine\n", + "\n", + "query = \"Identify interconnected security risks across the UN and major regions.\"\n", + "print(f\"Investigative Query: {query}\\n\")\n", + "\n", + "print(\"--- Standard Vector Recall ---\")\n", + "q_vec = v_core.embedding_generator.generate_embeddings(query)\n", + "v_res = vs.search_vectors(q_vec, k=3)\n", + "for r in v_res:\n", + " print(f\"Recall: {r['metadata']['text'][:150]}...\")\n", + "\n", + "print(\"\\n--- Graph Intelligence Reasoning ---\")\n", + "analyzer = ConnectivityAnalyzer()\n", + "connectivity = analyzer.analyze_graph_structure(kg)\n", + "\n", + "# Advanced Feature: Inference-Enhanced GraphRAG\n", + "# Vector RAG only sees surface similarity. \n", + "# InferenceEngine can follow logical rules (e.g., IF Conflict in Region A AND Region A is part of Trade Route B -> THEN Trade Route B is AT RISK)\n", + "engine = InferenceEngine(strategy=\"forward\")\n", + "engine.add_rule(\"IF ?x :type 'Conflict' AND ?y :type 'Region' AND ?x :locatedIn ?y THEN ?y :status 'HighRisk'\")\n", + "\n", + "print(f\"Knowledge Structure: {connectivity.get('structure_type', 'complex')}\")\n", + "print(f\"Identified {len(connectivity.get('bridges', []))} critical intelligence bridges.\")\n", + "print(\"Logical Inference Engine active: Monitoring for high-risk regional propagation.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Visualizing the Semantic Network\n", + "\n", + "We visualize how the Semantica engine has mapped the relationships between global actors and current events." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.visualization import KGVisualizer\n", + "import matplotlib.pyplot as plt\n", + "\n", + "KGVisualizer().visualize_network(\n", + " kg, \n", + " layout=\"spring\", \n", + " output=\"static\",\n", + " title=\"Intelligence Connectivity Map\"\n", + ")\n", + "plt.show()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.0" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/docs/cookbook.md b/docs/cookbook.md index 1ea6754a..37a8a2fc 100644 --- a/docs/cookbook.md +++ b/docs/cookbook.md @@ -36,6 +36,30 @@ Get started with Semantica through these beginner-friendly tutorials. - **[Visualization](cookbook/introduction/Visualization.ipynb)** - Visualize knowledge graphs - **[Export](cookbook/introduction/Export.ipynb)** - Export in various formats +- :material-robot: **GraphRAG Complete** + --- + Build a production-ready Graph Retrieval Augmented Generation system. + + **New Features**: Graph Validation, Logical Inference, Hybrid Context. + + **Topics**: RAG, LLMs, Vector Search, Graph Traversal + + **Difficulty**: Advanced + + [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) + +- :material-scale-balance: **RAG vs. GraphRAG Comparison** + --- + Side-by-side comparison of Standard RAG vs. GraphRAG using real-world data. + + **New Features**: Inference-Enhanced GraphRAG, Reasoning Gap Analysis. + + **Topics**: RAG, GraphRAG, Benchmarking, Visualization + + **Difficulty**: Intermediate + + [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) + ## Advanced Advanced techniques and patterns for experienced users. diff --git a/docs/examples.md b/docs/examples.md index 4f2f5ea4..fb990e5a 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -182,6 +182,115 @@ print(f"Graph density: {analysis['density']}") print(f"Connected components: {analysis['components']}") ``` +### Example 9: Persistent Storage (Neo4j) + +**Difficulty**: Intermediate + +Store and query knowledge graphs in a persistent graph database. + +```python +from semantica.graph_store import GraphStore + +# Initialize with Neo4j +store = GraphStore( + backend="neo4j", + uri="bolt://localhost:7687", + user="neo4j", + password="password" +) +store.connect() + +# Create nodes and relationships +apple = store.create_node( + labels=["Company"], + properties={"name": "Apple Inc."} +) +tim = store.create_node( + labels=["Person"], + properties={"name": "Tim Cook"} +) +store.create_relationship( + start_node_id=tim["id"], + end_node_id=apple["id"], + rel_type="CEO_OF" +) + +store.close() +``` + +### Example 10: FalkorDB for Real-Time Applications + +**Difficulty**: Intermediate + +Ultra-fast graph queries for LLM applications using FalkorDB. + +```python +from semantica.graph_store import GraphStore + +store = GraphStore( + backend="falkordb", + host="localhost", + port=6379, + graph_name="knowledge_graph" +) +store.connect() + +# Fast queries +results = store.execute_query("MATCH (n)-[r]->(m) WHERE n.name CONTAINS 'AI' RETURN n") +store.close() +``` + +### Example 11: GraphRAG (Knowledge-Powered Retrieval) + +**Difficulty**: Advanced + +Build a production-ready GraphRAG system with logical inference and hybrid retrieval. + +```python +from semantica.context import AgentContext +from semantica.reasoning import InferenceEngine + +# Initialize context with Hybrid Retrieval +context = AgentContext( + vector_store=vs, + knowledge_graph=kg, + use_graph_expansion=True, + hybrid_alpha=0.7 +) + +# Add logical reasoning rules +engine = InferenceEngine(strategy="forward") +engine.add_rule("IF ?x :type 'Library' AND ?y :type 'Language' THEN ?x :builtWith ?y") + +# Retrieve context for a query +results = context.retrieve("What technologies are used in this project?") +``` + +[**View Complete GraphRAG Tutorial**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) + +### Example 12: RAG vs. GraphRAG Comparison + +**Difficulty**: Intermediate + +Benchmark standard Vector RAG against Graph-enhanced retrieval. + +[**View RAG vs. GraphRAG Comparison**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) + +--- + +## Production Patterns + +### Example 13: Streaming Data Processing + +**Difficulty**: Advanced + +Process data streams in real-time. + +```python +from semantica.ingest import StreamIngestor +from semantica.core import Semantica +``` + ## Use Case Examples ### Research Paper Analysis