diff --git a/cookbook/introduction/01_Welcome_to_Semantica.ipynb b/cookbook/introduction/01_Welcome_to_Semantica.ipynb index 16250f8f..2616efe8 100644 --- a/cookbook/introduction/01_Welcome_to_Semantica.ipynb +++ b/cookbook/introduction/01_Welcome_to_Semantica.ipynb @@ -1,614 +1,647 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)\n", - "\n", - "# Welcome to Semantica\n", - "\n", - "## Overview\n", - "\n", - "This notebook introduces you to the **Semantica framework** - a comprehensive knowledge graph and semantic processing framework for building production-ready semantic AI applications.\n", - "\n", - "**Documentation**: [Getting Started](https://semantica.readthedocs.io/getting-started/) β€’ [Concepts](https://semantica.readthedocs.io/concepts/) β€’ [API Reference](https://semantica.readthedocs.io/reference/)\n", - "\n", - "### What You'll Learn\n", - "\n", - "- What Semantica is and why it's useful\n", - "- How to install and configure the framework\n", - "- Understanding the framework architecture\n", - "- Key concepts and terminology\n", - "- Next steps for getting started\n", - "\n", - "## What is Semantica?\n", - "\n", - "**Semantica** is a production-ready framework for:\n", - "\n", - "- **Building Knowledge Graphs**: Transform unstructured data into structured knowledge graphs\n", - "- **Semantic Processing**: Extract entities, relationships, and meaning from text, images, and audio\n", - "- **GraphRAG**: Graph-based retrieval augmented generation\n", - "- **Temporal Analysis**: Time-aware knowledge graphs\n", - "- **Multi-Modal Processing**: Handle text, images, audio, and structured data\n", - "- **Enterprise Features**: Quality assurance, conflict resolution, ontology generation\n", - "\n", - "### Use Cases\n", - "\n", - "- Threat intelligence and cybersecurity\n", - "- Healthcare and medical research\n", - "- Financial analysis and fraud detection\n", - "- Supply chain optimization\n", - "- Research and knowledge management\n", - "- Multi-agent AI systems\n", - "\n", - "\n", - "## Installation & Setup\n", - "\n", - "### Prerequisites\n", - "\n", - "Before installing Semantica, ensure you have:\n", - "- Python 3.8 or higher\n", - "- pip package manager\n", - "- (Optional) Virtual environment for isolation\n", - "\n", - "### Installation Methods\n", - "\n", - "```bash\n", - "# Method 1: Install from PyPI (Recommended)\n", - "pip install semantica\n", - "\n", - "# Or install with all optional dependencies:\n", - "pip install semantica[all]\n", - "\n", - "# Method 2: Install from source (development version)\n", - "git clone https://github.com/Hawksight-AI/semantica.git\n", - "cd semantica\n", - "pip install -e .\n", - "\n", - "# Or with all optional dependencies:\n", - "pip install -e \".[all]\"\n", - "\n", - "# Verify installation\n", - "import semantica\n", - "print(semantica.__version__)\n", - "```\n", - "\n", - "### Configuration\n", - "\n", - "```bash\n", - "# Set up environment variables for API keys and configuration\n", - "# export SEMANTICA_API_KEY=your_openai_key\n", - "# export SEMANTICA_EMBEDDING_PROVIDER=openai\n", - "# export SEMANTICA_MODEL_NAME=gpt-4\n", - "\n", - "# Or use a config file (config.yaml):\n", - "# api_keys:\n", - "# openai: your_key_here\n", - "# anthropic: your_key_here\n", - "# embedding:\n", - "# provider: openai\n", - "# model: text-embedding-3-large\n", - "# dimensions: 3072\n", - "# knowledge_graph:\n", - "# backend: networkx # or neo4j, arangodb\n", - "# temporal: true\n", - "```\n", - "\n", - "---\n", - "\n", - "## Framework Architecture Overview\n", - "\n", - "Semantica is organized into modular components, each handling a specific aspect of semantic processing:\n", - "\n", - "### 1. INGEST MODULE - Data Ingestion\n", - "**Purpose**: Ingest data from various sources\n", - "**Components**:\n", - "- `FileIngestor`: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n", - "- `WebIngestor`: Scrape and ingest web pages\n", - "- `FeedIngestor`: Process RSS/Atom feeds\n", - "- `StreamIngestor`: Real-time data streaming (Kafka, RabbitMQ, Kinesis, Pulsar)\n", - "- `DBIngestor`: Database queries and ingestion (PostgreSQL, MySQL, SQLite, Oracle, SQL Server)\n", - "- `EmailIngestor`: Process email messages (IMAP, POP3)\n", - "- `RepoIngestor`: Git repository analysis\n", - "- `MCPIngestor`: Model Context Protocol server integration\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.ingest import FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n", - "file_ingestor = FileIngestor()\n", - "web_ingestor = WebIngestor()\n", - "documents = file_ingestor.ingest(\"data/\")\n", - "web_docs = web_ingestor.ingest(\"https://example.com\")\n", - "```\n", - "\n", - "### 2. PARSE MODULE - Document Parsing\n", - "**Purpose**: Parse and extract content from various formats\n", - "**Components**:\n", - "- `DocumentParser`: Main parser orchestrator\n", - "- `PDFParser`: Extract text, tables, images from PDFs\n", - "- `DOCXParser`: Parse Word documents\n", - "- `HTMLParser`: Extract content from HTML\n", - "- `JSONParser`: Parse structured JSON data\n", - "- `ExcelParser`: Process spreadsheets\n", - "- `ImageParser`: OCR and image analysis\n", - "- `CodeParser`: Parse source code files\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.parse import DocumentParser\n", - "parser = DocumentParser()\n", - "parsed_docs = parser.parse(documents)\n", - "```\n", - "\n", - "### 3. NORMALIZE MODULE - Text Normalization\n", - "**Purpose**: Clean and normalize text for processing\n", - "**Components**:\n", - "- `TextNormalizer`: Main normalization orchestrator\n", - "- `TextCleaner`: Remove noise, fix encoding\n", - "- `DataCleaner`: Clean structured data\n", - "- `EntityNormalizer`: Normalize entity names\n", - "- `DateNormalizer`: Standardize date formats\n", - "- `NumberNormalizer`: Normalize numeric values\n", - "- `LanguageDetector`: Detect document language\n", - "- `EncodingHandler`: Handle character encoding\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.normalize import TextNormalizer\n", - "normalizer = TextNormalizer()\n", - "normalized = normalizer.normalize(parsed_docs)\n", - "```\n", - "\n", - "### 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n", - "**Purpose**: Extract entities, relationships, and semantic information\n", - "**Components**:\n", - "- `NERExtractor`: Named Entity Recognition\n", - "- `RelationExtractor`: Extract relationships between entities\n", - "- `SemanticAnalyzer`: Deep semantic analysis\n", - "- `SemanticNetworkExtractor`: Extract semantic networks\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.semantic_extract import NERExtractor, RelationExtractor\n", - "extractor = NERExtractor()\n", - "entities = extractor.extract(normalized_docs)\n", - "relation_extractor = RelationExtractor()\n", - "relationships = relation_extractor.extract(normalized_docs, entities)\n", - "```\n", - "\n", - "### 5. KG MODULE - Knowledge Graph Construction\n", - "**Purpose**: Build and manage knowledge graphs\n", - "**Components**:\n", - "- `GraphBuilder`: Construct knowledge graphs from entities/relationships\n", - "- `GraphAnalyzer`: Analyze graph structure and properties\n", - "- `GraphValidator`: Validate graph quality and consistency\n", - "- `EntityResolver`: Resolve entity conflicts and duplicates\n", - "- `ConflictDetector`: Detect conflicting information\n", - "- `CentralityCalculator`: Calculate node importance metrics\n", - "- `CommunityDetector`: Detect communities in graphs\n", - "- `ConnectivityAnalyzer`: Analyze graph connectivity\n", - "- `TemporalQuery`: Query temporal knowledge graphs\n", - "- `Deduplicator`: Remove duplicate entities/relationships\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.kg import GraphBuilder, GraphAnalyzer\n", - "builder = GraphBuilder()\n", - "kg = builder.build(entities, relationships)\n", - "analyzer = GraphAnalyzer()\n", - "metrics = analyzer.analyze(kg)\n", - "```\n", - "\n", - "### 6. EMBEDDINGS MODULE - Embedding Generation\n", - "**Purpose**: Generate vector embeddings for various data types\n", - "**Components**:\n", - "- `EmbeddingGenerator`: Main embedding orchestrator\n", - "- `TextEmbedder`: Generate text embeddings\n", - "- `ImageEmbedder`: Generate image embeddings\n", - "- `AudioEmbedder`: Generate audio embeddings\n", - "- `MultimodalEmbedder`: Combine multiple modalities\n", - "- `EmbeddingOptimizer`: Optimize embedding quality\n", - "- `ProviderAdapters`: Support for OpenAI, Cohere, etc.\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.embeddings import EmbeddingGenerator\n", - "generator = EmbeddingGenerator()\n", - "embeddings = generator.generate(documents)\n", - "```\n", - "\n", - "### 7. VECTOR_STORE MODULE - Vector Database Operations\n", - "**Purpose**: Store and search vector embeddings\n", - "**Components**:\n", - "- `VectorStore`: Main vector store interface\n", - "- `FAISSAdapter`: FAISS integration\n", - "- `HybridSearch`: Combine vector and keyword search\n", - "- `VectorRetriever`: Retrieve relevant vectors\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.vector_store import VectorStore, HybridSearch\n", - "vector_store = VectorStore()\n", - "vector_store.store(embeddings, documents, metadata)\n", - "hybrid_search = HybridSearch(vector_store)\n", - "results = hybrid_search.search(query, top_k=10)\n", - "```\n", - "\n", - "### 8. GRAPH_STORE MODULE - Persistent Graph Database Operations\n", - "**Purpose**: Store and query property graphs in Neo4j, KuzuDB, or FalkorDB\n", - "**Components**:\n", - "- `GraphStore`: Main graph store interface\n", - "- `Neo4jAdapter`: Neo4j integration (enterprise features)\n", - "- `KuzuAdapter`: KuzuDB integration (embedded, no server)\n", - "- `FalkorDBAdapter`: FalkorDB integration (Redis-based, ultra-fast)\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.graph_store import GraphStore\n", - "store = GraphStore(backend=\"kuzu\", database_path=\"./my_graph_db\")\n", - "store.connect()\n", - "node = store.create_node([\"Person\"], {\"name\": \"John\", \"age\": 30})\n", - "store.create_relationship(node[\"id\"], other_id, \"KNOWS\", {\"since\": 2020})\n", - "results = store.execute_query(\"MATCH (p:Person) RETURN p.name\")\n", - "store.close()\n", - "```\n", - "\n", - "### 9. REASONING MODULE - Inference and Reasoning\n", - "**Purpose**: Perform logical inference and reasoning\n", - "**Components**:\n", - "- `InferenceEngine`: Main inference orchestrator\n", - "- `RuleManager`: Manage inference rules\n", - "- `DeductiveReasoner`: Deductive reasoning\n", - "- `AbductiveReasoner`: Abductive reasoning\n", - "- `ExplanationGenerator`: Generate explanations for inferences\n", - "- `RETEEngine`: RETE algorithm for rule matching\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.reasoning import InferenceEngine, RuleManager\n", - "inference_engine = InferenceEngine()\n", - "rule_manager = RuleManager()\n", - "new_facts = inference_engine.forward_chain(kg, rule_manager)\n", - "```\n", - "\n", - "### 10. ONTOLOGY MODULE - Ontology Generation\n", - "**Purpose**: Generate and manage ontologies\n", - "**Components**:\n", - "- `OntologyGenerator`: Generate ontologies from knowledge graphs\n", - "- `OntologyValidator`: Validate ontology structure\n", - "- `OWLGenerator`: Generate OWL format ontologies\n", - "- `PropertyGenerator`: Generate ontology properties\n", - "- `ClassInferrer`: Infer ontology classes\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.ontology import OntologyGenerator\n", - "generator = OntologyGenerator()\n", - "ontology = generator.generate_from_graph(kg)\n", - "```\n", - "\n", - "### 11. EXPORT MODULE - Data Export\n", - "**Purpose**: Export data in various formats\n", - "**Components**:\n", - "- `JSONExporter`: Export to JSON\n", - "- `RDFExporter`: Export to RDF/XML\n", - "- `CSVExporter`: Export to CSV\n", - "- `GraphExporter`: Export to graph formats (GraphML, GEXF)\n", - "- `OWLExporter`: Export to OWL\n", - "- `VectorExporter`: Export vectors\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.export import JSONExporter, RDFExporter\n", - "json_exporter = JSONExporter()\n", - "json_exporter.export(kg, \"output.json\")\n", - "```\n", - "\n", - "### 12. VISUALIZATION MODULE - Graph Visualization\n", - "**Purpose**: Visualize knowledge graphs and analytics\n", - "**Components**:\n", - "- `KGVisualizer`: Visualize knowledge graphs\n", - "- `EmbeddingVisualizer`: Visualize embeddings (t-SNE, PCA, UMAP)\n", - "- `QualityVisualizer`: Visualize quality metrics\n", - "- `AnalyticsVisualizer`: Visualize graph analytics\n", - "- `TemporalVisualizer`: Visualize temporal data\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.visualization import KGVisualizer\n", - "visualizer = KGVisualizer()\n", - "visualizer.visualize(kg)\n", - "```\n", - "\n", - "### 13. PIPELINE MODULE - Pipeline Orchestration\n", - "**Purpose**: Build and execute processing pipelines\n", - "**Components**:\n", - "- `PipelineBuilder`: Build complex pipelines\n", - "- `ExecutionEngine`: Execute pipelines\n", - "- `FailureHandler`: Handle pipeline failures\n", - "- `ParallelismManager`: Enable parallel processing\n", - "- `ResourceScheduler`: Schedule resources\n", - "\n", - "**Example**:\n", - "```python\n", - "from semantica.pipeline import PipelineBuilder\n", - "builder = PipelineBuilder()\n", - "pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n", - " .add_step(\"parse\", DocumentParser()) \\\\\n", - " .build()\n", - "```\n", - "\n", - "---\n", - "\n", - "## Key Concepts Explained\n", - "\n", - "Understanding these concepts is crucial for working with Semantica:\n", - "\n", - "### 1. KNOWLEDGE GRAPHS\n", - "**Definition**: A knowledge graph is a structured representation of entities (nodes) and their relationships (edges) with properties and attributes.\n", - "\n", - "**Structure**:\n", - "- **Nodes**: Represent entities (people, places, concepts, events)\n", - "- **Edges**: Represent relationships (works_for, located_in, causes)\n", - "- **Properties**: Attributes of entities and relationships\n", - "- **Metadata**: Additional information (sources, timestamps, confidence)\n", - "\n", - "**Example**:\n", - "- Entity: \"John Doe\" (Person)\n", - "- Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n", - "- Properties: `{start_date: \"2020-01-01\", role: \"Engineer\"}`\n", - "\n", - "**Benefits**:\n", - "- Structured representation of unstructured data\n", - "- Enables complex queries and reasoning\n", - "- Supports temporal tracking\n", - "- Facilitates knowledge discovery\n", - "\n", - "### 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n", - "**Definition**: The process of identifying and classifying named entities in text into predefined categories.\n", - "\n", - "**Entity Types**:\n", - "- **Person**: Names of people\n", - "- **Organization**: Companies, institutions\n", - "- **Location**: Places, geographic entities\n", - "- **Date/Time**: Temporal expressions\n", - "- **Money**: Monetary values\n", - "- **Product**: Products and services\n", - "- **Event**: Events and occurrences\n", - "- **Custom**: Domain-specific entities\n", - "\n", - "**Example**:\n", - "Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n", - "Entities:\n", - "- \"Apple Inc.\" -> Organization\n", - "- \"Steve Jobs\" -> Person\n", - "- \"Cupertino, California\" -> Location\n", - "\n", - "**Methods**:\n", - "- Rule-based: Pattern matching\n", - "- Machine Learning: Trained models (spaCy, transformers)\n", - "- LLM-based: Using large language models\n", - "\n", - "### 3. RELATIONSHIP EXTRACTION\n", - "**Definition**: Identifying and extracting relationships between entities in text.\n", - "\n", - "**Relationship Types**:\n", - "- **Semantic**: \"works_for\", \"located_in\", \"causes\"\n", - "- **Temporal**: \"before\", \"after\", \"during\"\n", - "- **Causal**: \"causes\", \"results_in\", \"prevents\"\n", - "- **Hierarchical**: \"part_of\", \"subclass_of\", \"instance_of\"\n", - "\n", - "**Example**:\n", - "Text: \"John works for Acme Corp in New York.\"\n", - "Relationships:\n", - "- (John, works_for, Acme Corp)\n", - "- (Acme Corp, located_in, New York)\n", - "\n", - "**Methods**:\n", - "- Pattern matching\n", - "- Dependency parsing\n", - "- Machine learning models\n", - "- LLM-based extraction\n", - "\n", - "### 4. EMBEDDINGS\n", - "**Definition**: Dense vector representations of text, images, or other data that capture semantic meaning in a continuous vector space.\n", - "\n", - "**Properties**:\n", - "- Similar entities have similar embeddings (close in vector space)\n", - "- Enable semantic search and similarity calculations\n", - "- Fixed or variable dimensions (typically 128-4096)\n", - "\n", - "**Example**:\n", - "Text: \"machine learning\"\n", - "Embedding: `[0.123, -0.456, 0.789, ..., 0.234]` (vector of 1536 dimensions)\n", - "\n", - "**Use Cases**:\n", - "- Semantic search\n", - "- Clustering and classification\n", - "- Recommendation systems\n", - "- Anomaly detection\n", - "\n", - "### 5. TEMPORAL GRAPHS\n", - "**Definition**: Knowledge graphs that track changes over time, allowing queries about the state of the graph at specific time points.\n", - "\n", - "**Features**:\n", - "- Timestamps on entities and relationships\n", - "- Version history\n", - "- Time-point queries\n", - "- Temporal pattern detection\n", - "\n", - "**Example**:\n", - "- Entity: \"Company X\"\n", - "- Relationship: (Company X, has_CEO, Person Y)\n", - "- Temporal: `valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"`\n", - "\n", - "**Use Cases**:\n", - "- Tracking organizational changes\n", - "- Monitoring system evolution\n", - "- Analyzing trends over time\n", - "- Historical analysis\n", - "\n", - "### 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n", - "**Definition**: An advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs.\n", - "\n", - "**Components**:\n", - "- **Vector Store**: For semantic similarity search\n", - "- **Knowledge Graph**: For structured relationship traversal\n", - "- **Hybrid Search**: Combines both approaches\n", - "- **LLM Integration**: Uses retrieved context for generation\n", - "\n", - "**Advantages over Traditional RAG**:\n", - "- Better handling of complex queries\n", - "- Relationship-aware retrieval\n", - "- Reduced hallucinations\n", - "- More accurate answers\n", - "\n", - "**Example Workflow**:\n", - "1. Query: \"Who worked with John at Acme Corp?\"\n", - "2. Vector search finds relevant documents\n", - "3. Knowledge graph traversal finds relationships\n", - "4. Combined context sent to LLM\n", - "5. LLM generates accurate answer using both sources\n", - "\n", - "### 7. ONTOLOGY\n", - "**Definition**: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language).\n", - "\n", - "**Components**:\n", - "- **Classes**: Categories of entities\n", - "- **Properties**: Relationships and attributes\n", - "- **Individuals**: Specific instances\n", - "- **Axioms**: Rules and constraints\n", - "\n", - "**Example**:\n", - "- Class: Person\n", - "- SubClass: Employee, Customer\n", - "- Property: worksFor (domain: Person, range: Organization)\n", - "\n", - "**Use Cases**:\n", - "- Standardize domain knowledge\n", - "- Enable reasoning\n", - "- Facilitate data integration\n", - "- Support semantic web\n", - "\n", - "### 8. QUALITY ASSURANCE\n", - "**Definition**: Processes and metrics to ensure knowledge graph quality, including completeness, consistency, and accuracy.\n", - "\n", - "**Metrics**:\n", - "- **Completeness**: Percentage of entities with required properties\n", - "- **Consistency**: Absence of contradictions\n", - "- **Accuracy**: Correctness of extracted information\n", - "- **Coverage**: Breadth of domain coverage\n", - "\n", - "**Methods**:\n", - "- Validation rules\n", - "- Automated quality checks\n", - "- Conflict detection\n", - "- Source verification\n", - "---\n", - "\n", - "## Next Steps\n", - "\n", - "Now that you understand the basics, here are recommended next steps:\n", - "\n", - "1. **Your First Knowledge Graph** (`01_Your_First_Knowledge_Graph.ipynb`)\n", - " - Build your first knowledge graph from a document\n", - " - Learn the basic workflow\n", - "\n", - "2. **Configuration Basics** (`02_Configuration_Basics.ipynb`)\n", - " - Set up configuration files\n", - " - Configure API keys and providers\n", - "\n", - "3. **Core Workflows** (`01_core_workflows/`)\n", - " - Learn common patterns and workflows\n", - " - Start with \"From Unstructured to Structured\"\n", - "\n", - "4. **Use Cases** (`03_use_cases/`)\n", - " - Explore domain-specific applications\n", - " - Find examples relevant to your domain\n", - "\n", - "---\n", - "\n", - "## Best Practices\n", - "\n", - "### 1. START SMALL\n", - "- Begin with simple documents\n", - "- Validate each step before moving forward\n", - "- Build incrementally\n", - "\n", - "### 2. CONFIGURE PROPERLY\n", - "- Use environment variables for sensitive data\n", - "- Set up proper logging\n", - "- Configure appropriate model sizes\n", - "\n", - "### 3. VALIDATE DATA\n", - "- Always validate extracted entities\n", - "- Check relationship quality\n", - "- Use quality assurance tools\n", - "\n", - "### 4. HANDLE ERRORS\n", - "- Implement error handling\n", - "- Use retry mechanisms\n", - "- Log errors for debugging\n", - "\n", - "### 5. OPTIMIZE PERFORMANCE\n", - "- Use batch processing for large datasets\n", - "- Enable parallel processing where possible\n", - "- Cache embeddings and results\n", - "\n", - "### 6. DOCUMENT YOUR WORKFLOWS\n", - "- Document data sources\n", - "- Track processing steps\n", - "- Maintain metadata\n", - "\n", - "---\n", - "\n", - "## Troubleshooting\n", - "\n", - "Common issues and solutions:\n", - "\n", - "### Issue 1: Import Errors\n", - "**Solution**:\n", - "- Ensure Semantica is properly installed\n", - "- Check Python version (3.8+)\n", - "- Verify virtual environment is activated\n", - "- Install missing dependencies: `pip install -r requirements.txt`\n", - "\n", - "### Issue 2: API Key Errors\n", - "**Solution**:\n", - "- Set environment variables: `export SEMANTICA_API_KEY=your_key`\n", - "- Check config file for correct key format\n", - "- Verify API key is valid and has sufficient credits\n", - "\n", - "### Issue 3: Memory Issues\n", - "**Solution**:\n", - "- Process documents in batches\n", - "- Use smaller embedding models\n", - "- Enable garbage collection\n", - "- Consider using streaming for large datasets\n", - "\n", - "### Issue 4: Low Quality Extractions\n", - "**Solution**:\n", - "- Preprocess and normalize text\n", - "- Use domain-specific models\n", - "- Adjust extraction parameters\n", - "- Validate and clean extracted entities\n", - "\n", - "### Issue 5: Slow Processing\n", - "**Solution**:\n", - "- Enable parallel processing\n", - "- Use GPU acceleration if available\n", - "- Cache intermediate results\n", - "- Optimize batch sizes\n", - "\n" - ] - } - ], - "metadata": { - "language_info": { - "name": "python" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)\n", + "\n", + "# Welcome to Semantica\n", + "\n", + "## Overview\n", + "\n", + "This notebook introduces you to the **Semantica framework** - a comprehensive knowledge graph and semantic processing framework for building production-ready semantic AI applications.\n", + "\n", + "**Documentation**: [Getting Started](https://semantica.readthedocs.io/getting-started/) β€’ [Concepts](https://semantica.readthedocs.io/concepts/) β€’ [API Reference](https://semantica.readthedocs.io/reference/)\n", + "\n", + "### What You'll Learn\n", + "\n", + "- What Semantica is and why it's useful\n", + "- How to install and configure the framework\n", + "- Understanding the framework architecture\n", + "- Key concepts and terminology\n", + "- Next steps for getting started\n", + "\n", + "## What is Semantica?\n", + "\n", + "**Semantica** is a production-ready framework for:\n", + "\n", + "- **Building Knowledge Graphs**: Transform unstructured data into structured knowledge graphs\n", + "- **Semantic Processing**: Extract entities, relationships, and meaning from text, images, and audio\n", + "- **GraphRAG**: Graph-based retrieval augmented generation\n", + "- **Temporal Analysis**: Time-aware knowledge graphs\n", + "- **Multi-Modal Processing**: Handle text, images, audio, and structured data\n", + "- **Enterprise Features**: Quality assurance, conflict resolution, ontology generation\n", + "\n", + "### Use Cases\n", + "\n", + "- Threat intelligence and cybersecurity\n", + "- Healthcare and medical research\n", + "- Financial analysis and fraud detection\n", + "- Supply chain optimization\n", + "- Research and knowledge management\n", + "- Multi-agent AI systems\n", + "\n", + "\n", + "## Installation & Setup\n", + "\n", + "### Prerequisites\n", + "\n", + "Before installing Semantica, ensure you have:\n", + "- Python 3.8 or higher\n", + "- pip package manager\n", + "- (Optional) Virtual environment for isolation\n", + "\n", + "### Installation Methods\n", + "\n", + "```bash\n", + "# Method 1: Install from PyPI (Recommended)\n", + "pip install semantica\n", + "\n", + "# Or install with all optional dependencies:\n", + "pip install semantica[all]\n", + "\n", + "# Method 2: Install from source (development version)\n", + "git clone https://github.com/Hawksight-AI/semantica.git\n", + "cd semantica\n", + "pip install -e .\n", + "\n", + "# Or with all optional dependencies:\n", + "pip install -e \".[all]\"\n", + "\n", + "# Verify installation\n", + "import semantica\n", + "print(semantica.__version__)\n", + "```\n", + "\n", + "### Configuration\n", + "\n", + "```bash\n", + "# Set up environment variables for API keys and configuration\n", + "# export SEMANTICA_API_KEY=your_openai_key\n", + "# export SEMANTICA_EMBEDDING_PROVIDER=openai\n", + "# export SEMANTICA_MODEL_NAME=gpt-4\n", + "\n", + "# Or use a config file (config.yaml):\n", + "# api_keys:\n", + "# openai: your_key_here\n", + "# anthropic: your_key_here\n", + "# embedding:\n", + "# provider: openai\n", + "# model: text-embedding-3-large\n", + "# dimensions: 3072\n", + "# knowledge_graph:\n", + "# backend: networkx # or neo4j, arangodb\n", + "# temporal: true\n", + "```\n", + "\n", + "---\n", + "\n", + "## Framework Architecture Overview\n", + "\n", + "Semantica is organized into modular components, each handling a specific aspect of semantic processing:\n", + "\n", + "### 1. INGEST MODULE - Data Ingestion\n", + "**Purpose**: Ingest data from various sources\n", + "**Components**:\n", + "- `FileIngestor`: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n", + "- `WebIngestor`: Scrape and ingest web pages\n", + "- `FeedIngestor`: Process RSS/Atom feeds\n", + "- `StreamIngestor`: Real-time data streaming (Kafka, RabbitMQ, Kinesis, Pulsar)\n", + "- `DBIngestor`: Database queries and ingestion (PostgreSQL, MySQL, SQLite, Oracle, SQL Server)\n", + "- `EmailIngestor`: Process email messages (IMAP, POP3)\n", + "- `RepoIngestor`: Git repository analysis\n", + "- `MCPIngestor`: Model Context Protocol server integration\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.ingest import FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n", + "file_ingestor = FileIngestor()\n", + "web_ingestor = WebIngestor()\n", + "documents = file_ingestor.ingest(\"data/\")\n", + "web_docs = web_ingestor.ingest(\"https://example.com\")\n", + "```\n", + "\n", + "### 2. PARSE MODULE - Document Parsing\n", + "**Purpose**: Parse and extract content from various formats\n", + "**Components**:\n", + "- `DocumentParser`: Main parser orchestrator\n", + "- `PDFParser`: Extract text, tables, images from PDFs\n", + "- `DOCXParser`: Parse Word documents\n", + "- `HTMLParser`: Extract content from HTML\n", + "- `JSONParser`: Parse structured JSON data\n", + "- `ExcelParser`: Process spreadsheets\n", + "- `ImageParser`: OCR and image analysis\n", + "- `CodeParser`: Parse source code files\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.parse import DocumentParser\n", + "parser = DocumentParser()\n", + "parsed_docs = parser.parse(documents)\n", + "```\n", + "\n", + "### 3. NORMALIZE MODULE - Text Normalization\n", + "**Purpose**: Clean and normalize text for processing\n", + "**Components**:\n", + "- `TextNormalizer`: Main normalization orchestrator\n", + "- `TextCleaner`: Remove noise, fix encoding\n", + "- `DataCleaner`: Clean structured data\n", + "- `EntityNormalizer`: Normalize entity names\n", + "- `DateNormalizer`: Standardize date formats\n", + "- `NumberNormalizer`: Normalize numeric values\n", + "- `LanguageDetector`: Detect document language\n", + "- `EncodingHandler`: Handle character encoding\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.normalize import TextNormalizer\n", + "normalizer = TextNormalizer()\n", + "normalized = normalizer.normalize(parsed_docs)\n", + "```\n", + "\n", + "### 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n", + "**Purpose**: Extract entities, relationships, and semantic information\n", + "**Components**:\n", + "- `NERExtractor`: Named Entity Recognition\n", + "- `RelationExtractor`: Extract relationships between entities\n", + "- `SemanticAnalyzer`: Deep semantic analysis\n", + "- `SemanticNetworkExtractor`: Extract semantic networks\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.semantic_extract import NERExtractor, RelationExtractor\n", + "extractor = NERExtractor()\n", + "entities = extractor.extract(normalized_docs)\n", + "relation_extractor = RelationExtractor()\n", + "relationships = relation_extractor.extract(normalized_docs, entities)\n", + "```\n", + "\n", + "### 5. KG MODULE - Knowledge Graph Construction\n", + "**Purpose**: Build and manage knowledge graphs\n", + "**Components**:\n", + "- `GraphBuilder`: Construct knowledge graphs from entities/relationships\n", + "- `GraphAnalyzer`: Analyze graph structure and properties\n", + "- `GraphValidator`: Validate graph quality and consistency\n", + "- `EntityResolver`: Resolve entity conflicts and duplicates\n", + "- `ConflictDetector`: Detect conflicting information\n", + "- `CentralityCalculator`: Calculate node importance metrics\n", + "- `CommunityDetector`: Detect communities in graphs\n", + "- `ConnectivityAnalyzer`: Analyze graph connectivity\n", + "- `TemporalQuery`: Query temporal knowledge graphs\n", + "- `Deduplicator`: Remove duplicate entities/relationships\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.kg import GraphBuilder, GraphAnalyzer\n", + "builder = GraphBuilder()\n", + "kg = builder.build(entities, relationships)\n", + "analyzer = GraphAnalyzer()\n", + "metrics = analyzer.analyze(kg)\n", + "```\n", + "\n", + "### 6. EMBEDDINGS MODULE - Embedding Generation\n", + "**Purpose**: Generate vector embeddings for various data types\n", + "**Components**:\n", + "- `EmbeddingGenerator`: Main embedding orchestrator\n", + "- `TextEmbedder`: Generate text embeddings\n", + "- `ImageEmbedder`: Generate image embeddings\n", + "- `AudioEmbedder`: Generate audio embeddings\n", + "- `MultimodalEmbedder`: Combine multiple modalities\n", + "- `EmbeddingOptimizer`: Optimize embedding quality\n", + "- `ProviderAdapters`: Support for OpenAI, Cohere, etc.\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.embeddings import EmbeddingGenerator\n", + "generator = EmbeddingGenerator()\n", + "embeddings = generator.generate(documents)\n", + "```\n", + "\n", + "### 7. VECTOR_STORE MODULE - Vector Database Operations\n", + "**Purpose**: Store and search vector embeddings\n", + "**Components**:\n", + "- `VectorStore`: Main vector store interface\n", + "- `FAISSAdapter`: FAISS integration\n", + "- `HybridSearch`: Combine vector and keyword search\n", + "- `VectorRetriever`: Retrieve relevant vectors\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.vector_store import VectorStore, HybridSearch\n", + "vector_store = VectorStore()\n", + "vector_store.store(embeddings, documents, metadata)\n", + "hybrid_search = HybridSearch(vector_store)\n", + "results = hybrid_search.search(query, top_k=10)\n", + "```\n", + "\n", + "### 8. GRAPH_STORE MODULE - Persistent Graph Database Operations\n", + "**Purpose**: Store and query property graphs in Neo4j, KuzuDB, or FalkorDB\n", + "**Components**:\n", + "- `GraphStore`: Main graph store interface\n", + "- `Neo4jAdapter`: Neo4j integration (enterprise features)\n", + "- `KuzuAdapter`: KuzuDB integration (embedded, no server)\n", + "- `FalkorDBAdapter`: FalkorDB integration (Redis-based, ultra-fast)\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.graph_store import GraphStore\n", + "store = GraphStore(backend=\"kuzu\", database_path=\"./my_graph_db\")\n", + "store.connect()\n", + "node = store.create_node([\"Person\"], {\"name\": \"John\", \"age\": 30})\n", + "store.create_relationship(node[\"id\"], other_id, \"KNOWS\", {\"since\": 2020})\n", + "results = store.execute_query(\"MATCH (p:Person) RETURN p.name\")\n", + "store.close()\n", + "```\n", + "\n", + "### 9. REASONING MODULE - Inference and Reasoning\n", + "**Purpose**: Perform logical inference and reasoning\n", + "**Components**:\n", + "- `InferenceEngine`: Main inference orchestrator\n", + "- `RuleManager`: Manage inference rules\n", + "- `DeductiveReasoner`: Deductive reasoning\n", + "- `AbductiveReasoner`: Abductive reasoning\n", + "- `ExplanationGenerator`: Generate explanations for inferences\n", + "- `RETEEngine`: RETE algorithm for rule matching\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.reasoning import InferenceEngine, RuleManager\n", + "inference_engine = InferenceEngine()\n", + "rule_manager = RuleManager()\n", + "new_facts = inference_engine.forward_chain(kg, rule_manager)\n", + "```\n", + "\n", + "### 10. ONTOLOGY MODULE - Ontology Generation\n", + "**Purpose**: Generate and manage ontologies\n", + "**Components**:\n", + "- `OntologyGenerator`: Generate ontologies from knowledge graphs\n", + "- `OntologyValidator`: Validate ontology structure\n", + "- `OWLGenerator`: Generate OWL format ontologies\n", + "- `PropertyGenerator`: Generate ontology properties\n", + "- `ClassInferrer`: Infer ontology classes\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.ontology import OntologyGenerator\n", + "generator = OntologyGenerator()\n", + "ontology = generator.generate_from_graph(kg)\n", + "```\n", + "\n", + "### 11. EXPORT MODULE - Data Export\n", + "**Purpose**: Export data in various formats\n", + "**Components**:\n", + "- `JSONExporter`: Export to JSON\n", + "- `RDFExporter`: Export to RDF/XML\n", + "- `CSVExporter`: Export to CSV\n", + "- `GraphExporter`: Export to graph formats (GraphML, GEXF)\n", + "- `OWLExporter`: Export to OWL\n", + "- `VectorExporter`: Export vectors\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.export import JSONExporter, RDFExporter\n", + "json_exporter = JSONExporter()\n", + "json_exporter.export(kg, \"output.json\")\n", + "```\n", + "\n", + "### 12. VISUALIZATION MODULE - Graph Visualization\n", + "**Purpose**: Visualize knowledge graphs and analytics\n", + "**Components**:\n", + "- `KGVisualizer`: Visualize knowledge graphs\n", + "- `EmbeddingVisualizer`: Visualize embeddings (t-SNE, PCA, UMAP)\n", + "- `QualityVisualizer`: Visualize quality metrics\n", + "- `AnalyticsVisualizer`: Visualize graph analytics\n", + "- `TemporalVisualizer`: Visualize temporal data\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.visualization import KGVisualizer\n", + "visualizer = KGVisualizer()\n", + "visualizer.visualize(kg)\n", + "```\n", + "\n", + "### 13. PIPELINE MODULE - Pipeline Orchestration\n", + "**Purpose**: Build and execute processing pipelines\n", + "**Components**:\n", + "- `PipelineBuilder`: Build complex pipelines\n", + "- `ExecutionEngine`: Execute pipelines\n", + "- `FailureHandler`: Handle pipeline failures\n", + "- `ParallelismManager`: Enable parallel processing\n", + "- `ResourceScheduler`: Schedule resources\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.pipeline import PipelineBuilder\n", + "builder = PipelineBuilder()\n", + "pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n", + " .add_step(\"parse\", DocumentParser()) \\\\\n", + " .build()\n", + "```\n", + "\n", + "### 14. CORE MODULE - Framework Orchestration\n", + "**Purpose**: Framework initialization, configuration, lifecycle management, and plugin system\n", + "**Components**:\n", + "- `Semantica`: Main framework class coordinating all components\n", + "- `ConfigManager`: Configuration loading, validation, and management\n", + "- `Config`: Configuration data class with validation\n", + "- `LifecycleManager`: System lifecycle management with hooks and health monitoring\n", + "- `PluginRegistry`: Dynamic plugin discovery and loading\n", + "- `MethodRegistry`: Registry for custom orchestration methods\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.core import Semantica, ConfigManager\n", + "\n", + "# Load configuration\n", + "config_manager = ConfigManager()\n", + "config = config_manager.load_from_file(\"config.yaml\")\n", + "\n", + "# Initialize framework\n", + "framework = Semantica(config=config)\n", + "framework.initialize()\n", + "\n", + "# Build knowledge base\n", + "result = framework.build_knowledge_base(\n", + " sources=[\"doc1.pdf\", \"doc2.docx\"],\n", + " embeddings=True,\n", + " graph=True\n", + ")\n", + "\n", + "# Shutdown gracefully\n", + "framework.shutdown()\n", + "```\n", + "\n", + "---\n", + "\n", + "## Key Concepts Explained\n", + "\n", + "Understanding these concepts is crucial for working with Semantica:\n", + "\n", + "### 1. KNOWLEDGE GRAPHS\n", + "**Definition**: A knowledge graph is a structured representation of entities (nodes) and their relationships (edges) with properties and attributes.\n", + "\n", + "**Structure**:\n", + "- **Nodes**: Represent entities (people, places, concepts, events)\n", + "- **Edges**: Represent relationships (works_for, located_in, causes)\n", + "- **Properties**: Attributes of entities and relationships\n", + "- **Metadata**: Additional information (sources, timestamps, confidence)\n", + "\n", + "**Example**:\n", + "- Entity: \"John Doe\" (Person)\n", + "- Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n", + "- Properties: `{start_date: \"2020-01-01\", role: \"Engineer\"}`\n", + "\n", + "**Benefits**:\n", + "- Structured representation of unstructured data\n", + "- Enables complex queries and reasoning\n", + "- Supports temporal tracking\n", + "- Facilitates knowledge discovery\n", + "\n", + "### 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n", + "**Definition**: The process of identifying and classifying named entities in text into predefined categories.\n", + "\n", + "**Entity Types**:\n", + "- **Person**: Names of people\n", + "- **Organization**: Companies, institutions\n", + "- **Location**: Places, geographic entities\n", + "- **Date/Time**: Temporal expressions\n", + "- **Money**: Monetary values\n", + "- **Product**: Products and services\n", + "- **Event**: Events and occurrences\n", + "- **Custom**: Domain-specific entities\n", + "\n", + "**Example**:\n", + "Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n", + "Entities:\n", + "- \"Apple Inc.\" -> Organization\n", + "- \"Steve Jobs\" -> Person\n", + "- \"Cupertino, California\" -> Location\n", + "\n", + "**Methods**:\n", + "- Rule-based: Pattern matching\n", + "- Machine Learning: Trained models (spaCy, transformers)\n", + "- LLM-based: Using large language models\n", + "\n", + "### 3. RELATIONSHIP EXTRACTION\n", + "**Definition**: Identifying and extracting relationships between entities in text.\n", + "\n", + "**Relationship Types**:\n", + "- **Semantic**: \"works_for\", \"located_in\", \"causes\"\n", + "- **Temporal**: \"before\", \"after\", \"during\"\n", + "- **Causal**: \"causes\", \"results_in\", \"prevents\"\n", + "- **Hierarchical**: \"part_of\", \"subclass_of\", \"instance_of\"\n", + "\n", + "**Example**:\n", + "Text: \"John works for Acme Corp in New York.\"\n", + "Relationships:\n", + "- (John, works_for, Acme Corp)\n", + "- (Acme Corp, located_in, New York)\n", + "\n", + "**Methods**:\n", + "- Pattern matching\n", + "- Dependency parsing\n", + "- Machine learning models\n", + "- LLM-based extraction\n", + "\n", + "### 4. EMBEDDINGS\n", + "**Definition**: Dense vector representations of text, images, or other data that capture semantic meaning in a continuous vector space.\n", + "\n", + "**Properties**:\n", + "- Similar entities have similar embeddings (close in vector space)\n", + "- Enable semantic search and similarity calculations\n", + "- Fixed or variable dimensions (typically 128-4096)\n", + "\n", + "**Example**:\n", + "Text: \"machine learning\"\n", + "Embedding: `[0.123, -0.456, 0.789, ..., 0.234]` (vector of 1536 dimensions)\n", + "\n", + "**Use Cases**:\n", + "- Semantic search\n", + "- Clustering and classification\n", + "- Recommendation systems\n", + "- Anomaly detection\n", + "\n", + "### 5. TEMPORAL GRAPHS\n", + "**Definition**: Knowledge graphs that track changes over time, allowing queries about the state of the graph at specific time points.\n", + "\n", + "**Features**:\n", + "- Timestamps on entities and relationships\n", + "- Version history\n", + "- Time-point queries\n", + "- Temporal pattern detection\n", + "\n", + "**Example**:\n", + "- Entity: \"Company X\"\n", + "- Relationship: (Company X, has_CEO, Person Y)\n", + "- Temporal: `valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"`\n", + "\n", + "**Use Cases**:\n", + "- Tracking organizational changes\n", + "- Monitoring system evolution\n", + "- Analyzing trends over time\n", + "- Historical analysis\n", + "\n", + "### 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n", + "**Definition**: An advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs.\n", + "\n", + "**Components**:\n", + "- **Vector Store**: For semantic similarity search\n", + "- **Knowledge Graph**: For structured relationship traversal\n", + "- **Hybrid Search**: Combines both approaches\n", + "- **LLM Integration**: Uses retrieved context for generation\n", + "\n", + "**Advantages over Traditional RAG**:\n", + "- Better handling of complex queries\n", + "- Relationship-aware retrieval\n", + "- Reduced hallucinations\n", + "- More accurate answers\n", + "\n", + "**Example Workflow**:\n", + "1. Query: \"Who worked with John at Acme Corp?\"\n", + "2. Vector search finds relevant documents\n", + "3. Knowledge graph traversal finds relationships\n", + "4. Combined context sent to LLM\n", + "5. LLM generates accurate answer using both sources\n", + "\n", + "### 7. ONTOLOGY\n", + "**Definition**: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language).\n", + "\n", + "**Components**:\n", + "- **Classes**: Categories of entities\n", + "- **Properties**: Relationships and attributes\n", + "- **Individuals**: Specific instances\n", + "- **Axioms**: Rules and constraints\n", + "\n", + "**Example**:\n", + "- Class: Person\n", + "- SubClass: Employee, Customer\n", + "- Property: worksFor (domain: Person, range: Organization)\n", + "\n", + "**Use Cases**:\n", + "- Standardize domain knowledge\n", + "- Enable reasoning\n", + "- Facilitate data integration\n", + "- Support semantic web\n", + "\n", + "### 8. QUALITY ASSURANCE\n", + "**Definition**: Processes and metrics to ensure knowledge graph quality, including completeness, consistency, and accuracy.\n", + "\n", + "**Metrics**:\n", + "- **Completeness**: Percentage of entities with required properties\n", + "- **Consistency**: Absence of contradictions\n", + "- **Accuracy**: Correctness of extracted information\n", + "- **Coverage**: Breadth of domain coverage\n", + "\n", + "**Methods**:\n", + "- Validation rules\n", + "- Automated quality checks\n", + "- Conflict detection\n", + "- Source verification\n", + "---\n", + "\n", + "## Next Steps\n", + "\n", + "Now that you understand the basics, here are recommended next steps:\n", + "\n", + "1. **Your First Knowledge Graph** (`01_Your_First_Knowledge_Graph.ipynb`)\n", + " - Build your first knowledge graph from a document\n", + " - Learn the basic workflow\n", + "\n", + "2. **Configuration Basics** (`02_Configuration_Basics.ipynb`)\n", + " - Set up configuration files\n", + " - Configure API keys and providers\n", + "\n", + "3. **Core Workflows** (`01_core_workflows/`)\n", + " - Learn common patterns and workflows\n", + " - Start with \"From Unstructured to Structured\"\n", + "\n", + "4. **Use Cases** (`03_use_cases/`)\n", + " - Explore domain-specific applications\n", + " - Find examples relevant to your domain\n", + "\n", + "---\n", + "\n", + "## Best Practices\n", + "\n", + "### 1. START SMALL\n", + "- Begin with simple documents\n", + "- Validate each step before moving forward\n", + "- Build incrementally\n", + "\n", + "### 2. CONFIGURE PROPERLY\n", + "- Use environment variables for sensitive data\n", + "- Set up proper logging\n", + "- Configure appropriate model sizes\n", + "\n", + "### 3. VALIDATE DATA\n", + "- Always validate extracted entities\n", + "- Check relationship quality\n", + "- Use quality assurance tools\n", + "\n", + "### 4. HANDLE ERRORS\n", + "- Implement error handling\n", + "- Use retry mechanisms\n", + "- Log errors for debugging\n", + "\n", + "### 5. OPTIMIZE PERFORMANCE\n", + "- Use batch processing for large datasets\n", + "- Enable parallel processing where possible\n", + "- Cache embeddings and results\n", + "\n", + "### 6. DOCUMENT YOUR WORKFLOWS\n", + "- Document data sources\n", + "- Track processing steps\n", + "- Maintain metadata\n", + "\n", + "---\n", + "\n", + "## Troubleshooting\n", + "\n", + "Common issues and solutions:\n", + "\n", + "### Issue 1: Import Errors\n", + "**Solution**:\n", + "- Ensure Semantica is properly installed\n", + "- Check Python version (3.8+)\n", + "- Verify virtual environment is activated\n", + "- Install missing dependencies: `pip install -r requirements.txt`\n", + "\n", + "### Issue 2: API Key Errors\n", + "**Solution**:\n", + "- Set environment variables: `export SEMANTICA_API_KEY=your_key`\n", + "- Check config file for correct key format\n", + "- Verify API key is valid and has sufficient credits\n", + "\n", + "### Issue 3: Memory Issues\n", + "**Solution**:\n", + "- Process documents in batches\n", + "- Use smaller embedding models\n", + "- Enable garbage collection\n", + "- Consider using streaming for large datasets\n", + "\n", + "### Issue 4: Low Quality Extractions\n", + "**Solution**:\n", + "- Preprocess and normalize text\n", + "- Use domain-specific models\n", + "- Adjust extraction parameters\n", + "- Validate and clean extracted entities\n", + "\n", + "### Issue 5: Slow Processing\n", + "**Solution**:\n", + "- Enable parallel processing\n", + "- Use GPU acceleration if available\n", + "- Cache intermediate results\n", + "- Optimize batch sizes\n", + "\n" + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 } diff --git a/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb b/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb index 236f0550..8eea1492 100644 --- a/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb +++ b/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb @@ -1,267 +1,289 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Your_First_Knowledge_Graph.ipynb)\n", - "\n", - "# πŸš€ Your First Knowledge Graph\n", - "\n", - "## Overview\n", - "\n", - "This notebook walks you through creating your first knowledge graph from a simple document. You'll learn the complete end-to-end workflow from ingesting a file to visualizing the resulting knowledge graph.\n", - "\n", - "> [!TIP]\n", - "> This is the perfect starting point if you are new to Semantica. No prior knowledge of knowledge graphs is required!\n", - "\n", - "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/kg/)\n", - "\n", - "### 🎯 Learning Objectives\n", - "\n", - "- **Understand the Workflow**: Learn the `File β†’ Parse β†’ Extract β†’ Graph` pipeline\n", - "- **Ingest Data**: Load documents using `FileIngestor`\n", - "- **Parse Content**: Extract text using `DocumentParser`\n", - "- **Extract Knowledge**: Identify entities using `NERExtractor`\n", - "- **Build Graph**: Construct a graph using `GraphBuilder`\n", - "- **Visualize**: See your graph come to life with `KGVisualizer`\n", - "\n", - "## Installation\n", - "\n", - "Install Semantica from PyPI:\n", - "\n", - "```bash\n", - "pip install semantica\n", - "# Or with all optional dependencies:\n", - "pip install semantica[all]\n", - "```\n", - "\n", - "---\n", - "\n", - "## πŸ”„ Simple End-to-End Workflow\n", - "\n", - "The complete workflow consists of four main steps:\n", - "\n", - "1. **πŸ“₯ Ingest** - Load data from files or other sources\n", - "2. **πŸ“„ Parse** - Extract and structure content from documents\n", - "3. **⛏️ Extract** - Identify entities and relationships\n", - "4. **πŸ•ΈοΈ Build Graph** - Construct the knowledge graph\n", - "\n", - "Each step is demonstrated in the code cells below.\n", - "\n", - "---\n", - "\n", - "## πŸ“‚ Step 1: Ingest a File\n", - "\n", - "In this step, we'll use `FileIngestor` to load a document. The ingestor supports various file formats including PDF, DOCX, TXT, and more.\n" - ] + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Your_First_Knowledge_Graph.ipynb)\n", + "\n", + "# πŸš€ Your First Knowledge Graph\n", + "\n", + "## Overview\n", + "\n", + "This notebook walks you through creating your first knowledge graph from a simple document. You'll learn the complete end-to-end workflow from ingesting a file to visualizing the resulting knowledge graph.\n", + "\n", + "> [!TIP]\n", + "> This is the perfect starting point if you are new to Semantica. No prior knowledge of knowledge graphs is required!\n", + "\n", + "**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/kg/)\n", + "\n", + "### 🎯 Learning Objectives\n", + "\n", + "- **Understand the Workflow**: Learn the `File β†’ Parse β†’ Extract β†’ Graph` pipeline\n", + "- **Ingest Data**: Load documents using `FileIngestor`\n", + "- **Parse Content**: Extract text using `DocumentParser`\n", + "- **Extract Knowledge**: Identify entities using `NERExtractor`\n", + "- **Build Graph**: Construct a graph using `GraphBuilder`\n", + "- **Visualize**: See your graph come to life with `KGVisualizer`\n", + "\n", + "## Installation\n", + "\n", + "Install Semantica from PyPI:\n", + "\n", + "```bash\n", + "pip install semantica\n", + "# Or with all optional dependencies:\n", + "pip install semantica[all]\n", + "```\n", + "\n", + "---\n", + "\n", + "## πŸ”„ Simple End-to-End Workflow\n", + "\n", + "The complete workflow consists of four main steps:\n", + "\n", + "1. **πŸ“₯ Ingest** - Load data from files or other sources\n", + "2. **πŸ“„ Parse** - Extract and structure content from documents\n", + "3. **⛏️ Extract** - Identify entities and relationships\n", + "4. **πŸ•ΈοΈ Build Graph** - Construct the knowledge graph\n", + "\n", + "Each step is demonstrated in the code cells below.\n", + "\n", + "> [!TIP]\n", + "> **Alternative: Using Semantica Framework**\n", + "> \n", + "> For a simpler, high-level approach, you can use the `Semantica` framework class which orchestrates all these steps:\n", + "> \n", + "> ```python\n", + "> from semantica.core import Semantica\n", + "> \n", + "> framework = Semantica()\n", + "> framework.initialize()\n", + "> \n", + "> result = framework.build_knowledge_base(\n", + "> sources=[\"sample_document.txt\"],\n", + "> embeddings=True,\n", + "> graph=True\n", + "> )\n", + "> \n", + "> framework.shutdown()\n", + "> ```\n", + "> \n", + "> This notebook shows the step-by-step approach for learning. See [Core Module Usage Guide](../../../semantica/core/core_usage.md) for more details.\n", + "\n", + "---\n", + "\n", + "## πŸ“‚ Step 1: Ingest a File\n", + "\n", + "In this step, we'll use `FileIngestor` to load a document. The ingestor supports various file formats including PDF, DOCX, TXT, and more.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.ingest import FileIngestor\n", + "from pathlib import Path\n", + "\n", + "# Initialize the ingestor\n", + "ingestor = FileIngestor()\n", + "\n", + "# Create a sample document for demonstration\n", + "sample_text = \"\"\"\n", + "Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n", + "The company is headquartered in Cupertino, California.\n", + "Tim Cook is the current CEO of Apple Inc.\n", + "Apple designs and manufactures consumer electronics, software, and online services.\n", + "\"\"\"\n", + "\n", + "sample_file = Path(\"sample_document.txt\")\n", + "sample_file.write_text(sample_text)\n", + "\n", + "print(f\"File: {sample_file}\")\n", + "print(f\"Content length: {len(sample_text)} characters\")\n", + "\n", + "# Ingest the file\n", + "file_object = ingestor.ingest_file(sample_file, read_content=True)\n", + "print(f\" File name: {file_object.name}\")\n", + "print(f\" File type: {file_object.file_type}\")\n", + "print(f\" Content available: {file_object.content is not None}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## πŸ“„ Step 2: Parse the Document\n", + "\n", + "After ingesting the file, we need to parse it to extract the text content. The `DocumentParser` handles various file formats and extracts structured content.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.parse import DocumentParser\n", + "\n", + "parser = DocumentParser()\n", + "\n", + "# Parse the document to extract text\n", + "parsed_content = parser.parse_document(str(sample_file))\n", + "print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n", + "print(f\" Preview: {parsed_content[:200] if parsed_content else 'N/A'}...\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## ⛏️ Step 3: Extract Entities\n", + "\n", + "Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n", + "\n", + "> [!NOTE]\n", + "> In a real scenario, you would use `NERExtractor` with an LLM or model backend. Here we simulate the output for demonstration purposes.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.semantic_extract import NamedEntityRecognizer, NERExtractor\n", + "\n", + "ner = NamedEntityRecognizer()\n", + "extractor = NERExtractor()\n", + "\n", + "print(f\"\\nText: {parsed_content[:100]}...\")\n", + "\n", + "# Simulated extraction results\n", + "expected_entities = [\n", + " {\"text\": \"Apple Inc.\", \"type\": \"Organization\", \"start\": 0, \"end\": 10},\n", + " {\"text\": \"Steve Jobs\", \"type\": \"Person\", \"start\": 50, \"end\": 60},\n", + " {\"text\": \"Steve Wozniak\", \"type\": \"Person\", \"start\": 62, \"end\": 75},\n", + " {\"text\": \"Ronald Wayne\", \"type\": \"Person\", \"start\": 81, \"end\": 93},\n", + " {\"text\": \"1976\", \"type\": \"Date\", \"start\": 97, \"end\": 101},\n", + " {\"text\": \"Cupertino, California\", \"type\": \"Location\", \"start\": 130, \"end\": 151},\n", + " {\"text\": \"Tim Cook\", \"type\": \"Person\", \"start\": 153, \"end\": 161},\n", + "]\n", + "\n", + "for entity in expected_entities:\n", + " print(f\" - {entity['text']} ({entity['type']})\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## πŸ•ΈοΈ Step 4: Build the Knowledge Graph\n", + "\n", + "Using the extracted entities and relationships, we'll construct a knowledge graph. The graph represents entities as nodes and relationships as edges.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg import GraphBuilder\n", + "import networkx as nx\n", + "\n", + "builder = GraphBuilder()\n", + "\n", + "# Prepare data for graph construction\n", + "entities_data = [\n", + " {\"id\": f\"entity_{i}\", \"name\": entity[\"text\"], \"type\": entity[\"type\"]}\n", + " for i, entity in enumerate(expected_entities)\n", + "]\n", + "\n", + "relationships_data = [\n", + " {\"source\": \"entity_0\", \"target\": \"entity_1\", \"type\": \"founded_by\"},\n", + " {\"source\": \"entity_0\", \"target\": \"entity_2\", \"type\": \"founded_by\"},\n", + " {\"source\": \"entity_0\", \"target\": \"entity_3\", \"type\": \"founded_by\"},\n", + " {\"source\": \"entity_0\", \"target\": \"entity_4\", \"type\": \"founded_in\"},\n", + " {\"source\": \"entity_0\", \"target\": \"entity_5\", \"type\": \"located_in\"},\n", + " {\"source\": \"entity_6\", \"target\": \"entity_0\", \"type\": \"ceo_of\"},\n", + "]\n", + "\n", + "# Build the graph using NetworkX\n", + "kg = nx.DiGraph()\n", + "\n", + "for entity in entities_data:\n", + " kg.add_node(entity[\"id\"], name=entity[\"name\"], type=entity[\"type\"])\n", + "\n", + "for rel in relationships_data:\n", + " source_name = entities_data[int(rel[\"source\"].split(\"_\")[1])][\"name\"]\n", + " target_name = entities_data[int(rel[\"target\"].split(\"_\")[1])][\"name\"]\n", + " kg.add_edge(rel[\"source\"], rel[\"target\"], type=rel[\"type\"])\n", + "\n", + "print(f\" Nodes (entities): {len(kg.nodes)}\")\n", + "print(f\" Edges (relationships): {len(kg.edges)}\")\n", + "\n", + "for node_id in kg.nodes():\n", + " node_data = kg.nodes[node_id]\n", + " print(f\" Node: {node_data['name']} ({node_data['type']})\")\n", + "\n", + "for source, target, data in kg.edges(data=True):\n", + " source_name = kg.nodes[source]['name']\n", + " target_name = kg.nodes[target]['name']\n", + " print(f\" {source_name} --[{data['type']}]--> {target_name}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## πŸ“Š Step 5: Visualize and Analyze\n", + "\n", + "Finally, we'll visualize the knowledge graph and analyze its structure. This helps you understand the relationships and entities in your data.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.visualization import KGVisualizer\n", + "\n", + "visualizer = KGVisualizer()\n", + "\n", + "print(f\" Total entities: {len(kg.nodes)}\")\n", + "print(f\" Total relationships: {len(kg.edges)}\")\n", + "\n", + "entity_types = {}\n", + "for node_id in kg.nodes():\n", + " entity_type = kg.nodes[node_id]['type']\n", + " entity_types[entity_type] = entity_types.get(entity_type, 0) + 1\n", + "\n", + "for etype, count in entity_types.items():\n", + " print(f\" - {etype}: {count}\")\n", + "\n", + "rel_types = {}\n", + "for _, _, data in kg.edges(data=True):\n", + " rel_type = data.get('type', 'unknown')\n", + " rel_types[rel_type] = rel_types.get(rel_type, 0) + 1\n", + "\n", + "for rtype, count in rel_types.items():\n", + " print(f\" - {rtype}: {count}\")\n", + "\n", + "# Cleanup\n", + "if sample_file.exists():\n", + " sample_file.unlink()\n" + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.ingest import FileIngestor\n", - "from pathlib import Path\n", - "\n", - "# Initialize the ingestor\n", - "ingestor = FileIngestor()\n", - "\n", - "# Create a sample document for demonstration\n", - "sample_text = \"\"\"\n", - "Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n", - "The company is headquartered in Cupertino, California.\n", - "Tim Cook is the current CEO of Apple Inc.\n", - "Apple designs and manufactures consumer electronics, software, and online services.\n", - "\"\"\"\n", - "\n", - "sample_file = Path(\"sample_document.txt\")\n", - "sample_file.write_text(sample_text)\n", - "\n", - "print(f\"File: {sample_file}\")\n", - "print(f\"Content length: {len(sample_text)} characters\")\n", - "\n", - "# Ingest the file\n", - "file_object = ingestor.ingest_file(sample_file, read_content=True)\n", - "print(f\" File name: {file_object.name}\")\n", - "print(f\" File type: {file_object.file_type}\")\n", - "print(f\" Content available: {file_object.content is not None}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## πŸ“„ Step 2: Parse the Document\n", - "\n", - "After ingesting the file, we need to parse it to extract the text content. The `DocumentParser` handles various file formats and extracts structured content.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.parse import DocumentParser\n", - "\n", - "parser = DocumentParser()\n", - "\n", - "# Parse the document to extract text\n", - "parsed_content = parser.parse_document(str(sample_file))\n", - "print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n", - "print(f\" Preview: {parsed_content[:200] if parsed_content else 'N/A'}...\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## ⛏️ Step 3: Extract Entities\n", - "\n", - "Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n", - "\n", - "> [!NOTE]\n", - "> In a real scenario, you would use `NERExtractor` with an LLM or model backend. Here we simulate the output for demonstration purposes.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.semantic_extract import NamedEntityRecognizer, NERExtractor\n", - "\n", - "ner = NamedEntityRecognizer()\n", - "extractor = NERExtractor()\n", - "\n", - "print(f\"\\nText: {parsed_content[:100]}...\")\n", - "\n", - "# Simulated extraction results\n", - "expected_entities = [\n", - " {\"text\": \"Apple Inc.\", \"type\": \"Organization\", \"start\": 0, \"end\": 10},\n", - " {\"text\": \"Steve Jobs\", \"type\": \"Person\", \"start\": 50, \"end\": 60},\n", - " {\"text\": \"Steve Wozniak\", \"type\": \"Person\", \"start\": 62, \"end\": 75},\n", - " {\"text\": \"Ronald Wayne\", \"type\": \"Person\", \"start\": 81, \"end\": 93},\n", - " {\"text\": \"1976\", \"type\": \"Date\", \"start\": 97, \"end\": 101},\n", - " {\"text\": \"Cupertino, California\", \"type\": \"Location\", \"start\": 130, \"end\": 151},\n", - " {\"text\": \"Tim Cook\", \"type\": \"Person\", \"start\": 153, \"end\": 161},\n", - "]\n", - "\n", - "for entity in expected_entities:\n", - " print(f\" - {entity['text']} ({entity['type']})\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## πŸ•ΈοΈ Step 4: Build the Knowledge Graph\n", - "\n", - "Using the extracted entities and relationships, we'll construct a knowledge graph. The graph represents entities as nodes and relationships as edges.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.kg import GraphBuilder\n", - "import networkx as nx\n", - "\n", - "builder = GraphBuilder()\n", - "\n", - "# Prepare data for graph construction\n", - "entities_data = [\n", - " {\"id\": f\"entity_{i}\", \"name\": entity[\"text\"], \"type\": entity[\"type\"]}\n", - " for i, entity in enumerate(expected_entities)\n", - "]\n", - "\n", - "relationships_data = [\n", - " {\"source\": \"entity_0\", \"target\": \"entity_1\", \"type\": \"founded_by\"},\n", - " {\"source\": \"entity_0\", \"target\": \"entity_2\", \"type\": \"founded_by\"},\n", - " {\"source\": \"entity_0\", \"target\": \"entity_3\", \"type\": \"founded_by\"},\n", - " {\"source\": \"entity_0\", \"target\": \"entity_4\", \"type\": \"founded_in\"},\n", - " {\"source\": \"entity_0\", \"target\": \"entity_5\", \"type\": \"located_in\"},\n", - " {\"source\": \"entity_6\", \"target\": \"entity_0\", \"type\": \"ceo_of\"},\n", - "]\n", - "\n", - "# Build the graph using NetworkX\n", - "kg = nx.DiGraph()\n", - "\n", - "for entity in entities_data:\n", - " kg.add_node(entity[\"id\"], name=entity[\"name\"], type=entity[\"type\"])\n", - "\n", - "for rel in relationships_data:\n", - " source_name = entities_data[int(rel[\"source\"].split(\"_\")[1])][\"name\"]\n", - " target_name = entities_data[int(rel[\"target\"].split(\"_\")[1])][\"name\"]\n", - " kg.add_edge(rel[\"source\"], rel[\"target\"], type=rel[\"type\"])\n", - "\n", - "print(f\" Nodes (entities): {len(kg.nodes)}\")\n", - "print(f\" Edges (relationships): {len(kg.edges)}\")\n", - "\n", - "for node_id in kg.nodes():\n", - " node_data = kg.nodes[node_id]\n", - " print(f\" Node: {node_data['name']} ({node_data['type']})\")\n", - "\n", - "for source, target, data in kg.edges(data=True):\n", - " source_name = kg.nodes[source]['name']\n", - " target_name = kg.nodes[target]['name']\n", - " print(f\" {source_name} --[{data['type']}]--> {target_name}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## πŸ“Š Step 5: Visualize and Analyze\n", - "\n", - "Finally, we'll visualize the knowledge graph and analyze its structure. This helps you understand the relationships and entities in your data.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.visualization import KGVisualizer\n", - "\n", - "visualizer = KGVisualizer()\n", - "\n", - "print(f\" Total entities: {len(kg.nodes)}\")\n", - "print(f\" Total relationships: {len(kg.edges)}\")\n", - "\n", - "entity_types = {}\n", - "for node_id in kg.nodes():\n", - " entity_type = kg.nodes[node_id]['type']\n", - " entity_types[entity_type] = entity_types.get(entity_type, 0) + 1\n", - "\n", - "for etype, count in entity_types.items():\n", - " print(f\" - {etype}: {count}\")\n", - "\n", - "rel_types = {}\n", - "for _, _, data in kg.edges(data=True):\n", - " rel_type = data.get('type', 'unknown')\n", - " rel_types[rel_type] = rel_types.get(rel_type, 0) + 1\n", - "\n", - "for rtype, count in rel_types.items():\n", - " print(f\" - {rtype}: {count}\")\n", - "\n", - "# Cleanup\n", - "if sample_file.exists():\n", - " sample_file.unlink()\n" - ] - } - ], - "metadata": { - "language_info": { - "name": "python" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 } diff --git a/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb index c8008efe..b748c116 100644 --- a/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb +++ b/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb @@ -1,1686 +1,1697 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)\n", - "\n", - "# GraphRAG Complete - End-to-End Pipeline\n", - "\n", - "## Overview\n", - "\n", - "This notebook demonstrates a **complete end-to-end GraphRAG (Graph-based Retrieval Augmented Generation) system** using Semantica framework. It showcases how to build a production-ready GraphRAG system that combines vector search with knowledge graph traversal for enhanced retrieval and question answering.\n", - "\n", - "**Key Features:**\n", - "\n", - "- **Real-World Data**: Uses actual data sources via MCP servers, web scraping, and RSS feeds (NO mock data)\n", - "- **Complete Pipeline**: From data ingestion to LLM-powered question answering\n", - "- **Hybrid Retrieval**: Combines vector similarity search with knowledge graph traversal\n", - "- **Multi-hop Reasoning**: Follows relationships across the graph for deeper context\n", - "- **20+ Semantica Modules**: Demonstrates comprehensive use of the framework\n", - "\n", - "**Documentation**: [API Reference](https://semantica.readthedocs.io/concepts/) β€’ [GraphRAG Guide](https://semantica.readthedocs.io/concepts/)\n", - "\n", - "### What You'll Learn\n", - "\n", - "- How to ingest real-world data from multiple sources (MCP, web, feeds)\n", - "- How to build knowledge graphs from unstructured text\n", - "- How to implement hybrid search combining vectors and graphs\n", - "- How to use ContextRetriever for intelligent context expansion\n", - "- How to integrate LLMs with GraphRAG for question answering\n", - "- How to visualize and export knowledge graphs\n", - "\n", - "### Pipeline Overview\n", - "\n", - "**Real-World Data Sources (MCP/Web/Feeds) β†’ Parse β†’ Extract Entities & Relationships β†’ Build Knowledge Graph β†’ Generate Embeddings β†’ Vector Store β†’ Hybrid Search β†’ Context Retrieval β†’ GraphRAG Query System β†’ LLM Integration β†’ Answer Generation**\n", - "\n", - "---\n", - "\n", - "## Installation\n", - "\n", - "Install Semantica from PyPI:\n", - "\n", - "```bash\n", - "pip install semantica\n", - "\n", - "# Or with all optional dependencies:\n", - "pip install semantica[all]\n", - "```\n", - "\n", - "### Additional Dependencies\n", - "\n", - "```bash\n", - "pip install openai anthropic # For LLM integration\n", - "pip install jupyter # For running this notebook\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 1: Setup and Import Semantica Modules\n", - "\n", - "Import all necessary Semantica modules for the complete GraphRAG pipeline. This includes modules for ingestion, parsing, extraction, graph building, embeddings, vector storage, context retrieval, and more.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.core import ConfigManager\n", - "\n", - "config = ConfigManager()\n", - "print(\"Configuration initialized\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 2: Ingest Real-World Data from Multiple Sources\n", - "\n", - "1. **MCP Servers** (Primary): Connect to real MCP servers providing news feeds, documentation, APIs\n", - "2. **Web Sources**: Scrape real web content from news sites and documentation\n", - "3. **RSS Feeds**: Ingest real RSS/Atom feeds from news sources\n", - "\n", - "### 2.1: Connect to MCP Servers\n", - "\n", - "Connect to real MCP servers via URL. MCP servers can provide resources (databases, files) and tools (APIs, queries).\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.ingest import MCPIngestor\n", - "\n", - "mcp_ingestor = MCPIngestor()\n", - "\n", - "connected_servers = mcp_ingestor.get_connected_servers()\n", - "print(f\"Connected MCP Servers: {len(connected_servers)}\")\n", - "for server_name in connected_servers:\n", - " print(f\" - {server_name}\")\n", - "\n", - "if connected_servers:\n", - " for server_name in connected_servers:\n", - " try:\n", - " resources = mcp_ingestor.list_available_resources(server_name)\n", - " tools = mcp_ingestor.list_available_tools(server_name)\n", - " print(f\"\\n{server_name}:\")\n", - " print(f\" Resources: {len(resources)}\")\n", - " print(f\" Tools: {len(tools)}\")\n", - " except Exception as e:\n", - " print(f\"Error connecting to {server_name}: {e}\")\n", - "\n", - "print(\"\\nNote: Configure your MCP server URLs above to ingest real data\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.2: Ingest Data from MCP Servers\n", - "\n", - "Ingest real data from MCP server resources and tools.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.ingest import WebIngestor, FeedIngestor\n", - "\n", - "all_documents = []\n", - "\n", - "if connected_servers:\n", - " for server_name in connected_servers:\n", - " try:\n", - " print(f\"\\nIngesting from {server_name}...\")\n", - " mcp_data = mcp_ingestor.ingest_all_resources(server_name)\n", - " \n", - " if isinstance(mcp_data, list):\n", - " all_documents.extend(mcp_data)\n", - " print(f\" Ingested {len(mcp_data)} resources\")\n", - " else:\n", - " all_documents.append(mcp_data)\n", - " print(f\" Ingested 1 resource\")\n", - " \n", - " except Exception as e:\n", - " print(f\" Error ingesting from {server_name}: {e}\")\n", - "\n", - "print(f\"\\nTotal documents from MCP: {len(all_documents)}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.3: Ingest Data from Web Sources\n", - "\n", - "Scrape real web content from news sites, documentation, and articles.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "web_ingestor = WebIngestor()\n", - "\n", - "web_sources = []\n", - "\n", - "web_documents = []\n", - "for url in web_sources:\n", - " try:\n", - " print(f\"Scraping {url}...\")\n", - " docs = web_ingestor.ingest(url)\n", - " if isinstance(docs, list):\n", - " web_documents.extend(docs)\n", - " else:\n", - " web_documents.append(docs)\n", - " print(f\" Scraped {len(docs) if isinstance(docs, list) else 1} document(s)\")\n", - " except Exception as e:\n", - " print(f\" Error scraping {url}: {e}\")\n", - "\n", - "all_documents.extend(web_documents)\n", - "print(f\"\\nTotal documents from web: {len(web_documents)}\")\n", - "print(f\"Total documents so far: {len(all_documents)}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.4: Ingest Data from RSS Feeds\n", - "\n", - "Ingest real RSS/Atom feeds from news sources.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "feed_ingestor = FeedIngestor()\n", - "\n", - "feed_urls = []\n", - "\n", - "feed_documents = []\n", - "for feed_url in feed_urls:\n", - " try:\n", - " print(f\"Fetching feed {feed_url}...\")\n", - " feeds = feed_ingestor.ingest(feed_url)\n", - " if isinstance(feeds, list):\n", - " feed_documents.extend(feeds)\n", - " else:\n", - " feed_documents.append(feeds)\n", - " print(f\" Fetched {len(feeds) if isinstance(feeds, list) else 1} feed item(s)\")\n", - " except Exception as e:\n", - " print(f\" Error fetching feed {feed_url}: {e}\")\n", - "\n", - "all_documents.extend(feed_documents)\n", - "print(f\"\\nTotal documents from feeds: {len(feed_documents)}\")\n", - "print(f\"Total documents collected: {len(all_documents)}\")\n", - "\n", - "if len(all_documents) == 0:\n", - " print(\"\\nNo documents collected. Please configure MCP servers, web URLs, or RSS feeds above.\")\n", - " print(\"For this demonstration, we'll continue with the pipeline structure.\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 3: Document Processing Pipeline\n", - "\n", - "Process the ingested documents: parse, split, and normalize the text for extraction.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.parse import DocumentParser, MCPParser\n", - "\n", - "document_parser = DocumentParser()\n", - "mcp_parser = MCPParser()\n", - "\n", - "parsed_documents = []\n", - "\n", - "for doc in all_documents:\n", - " try:\n", - " if hasattr(doc, 'source') and 'mcp' in doc.source.lower():\n", - " parsed = mcp_parser.parse(doc)\n", - " else:\n", - " parsed = document_parser.parse(doc)\n", - " \n", - " if isinstance(parsed, list):\n", - " parsed_documents.extend(parsed)\n", - " else:\n", - " parsed_documents.append(parsed)\n", - " except Exception as e:\n", - " print(f\"Error parsing document: {e}\")\n", - " continue\n", - "\n", - "print(f\"Parsed {len(parsed_documents)} documents\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.2: Split Documents Using Dual Chunking Strategy\n", - "\n", - "For GraphRAG, we use **two different chunking methods** optimized for different stores:\n", - "\n", - "**For Vector Store** (semantic similarity search):\n", - "- **Semantic Chunking**: Uses embeddings to find natural semantic boundaries\n", - "- Better for vector similarity search and retrieval\n", - "\n", - "**For Graph Store** (knowledge structure preservation):\n", - "- **Entity-Aware Chunking**: Preserves entity boundaries (prevents splitting entities)\n", - "- **Relation-Aware Chunking**: Preserves relationship triples (keeps subject-predicate-object together)\n", - "- **Graph-Based Chunking**: Uses existing graph structure for optimal chunking\n", - "\n", - "We'll create chunks optimized for each store type.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.semantic_extract import NERExtractor, RelationExtractor\n", - "from semantica.split import (\n", - " SemanticChunker, EntityAwareChunker, RelationAwareChunker, GraphBasedChunker\n", - ")\n", - "import numpy as np\n", - "\n", - "print(\"Step 1: Creating chunks for Vector Store (semantic chunking)...\")\n", - "semantic_chunker = SemanticChunker(\n", - " chunk_size=1000,\n", - " chunk_overlap=200\n", - ")\n", - "\n", - "vector_store_chunks = []\n", - "for i, doc in enumerate(parsed_documents):\n", - " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", - " if doc_text.strip():\n", - " chunks = semantic_chunker.chunk(doc_text)\n", - " if isinstance(chunks, list):\n", - " for chunk in chunks:\n", - " if hasattr(chunk, 'metadata'):\n", - " chunk.metadata['chunking_method'] = 'semantic'\n", - " chunk.metadata['store_type'] = 'vector'\n", - " chunk.metadata['source_doc'] = i\n", - " vector_store_chunks.extend(chunks)\n", - " else:\n", - " if hasattr(chunks, 'metadata'):\n", - " chunks.metadata['chunking_method'] = 'semantic'\n", - " chunks.metadata['store_type'] = 'vector'\n", - " chunks.metadata['source_doc'] = i\n", - " vector_store_chunks.append(chunks)\n", - "\n", - "print(f\"Created {len(vector_store_chunks)} semantic chunks for vector store\")\n", - "\n", - "print(\"\\nStep 2: Extracting entities/relationships for graph-aware chunking...\")\n", - "ner_extractor = NERExtractor()\n", - "relation_extractor = RelationExtractor()\n", - "\n", - "doc_entities = {}\n", - "doc_relationships = {}\n", - "\n", - "for i, doc in enumerate(parsed_documents):\n", - " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", - " if doc_text.strip():\n", - " entities = ner_extractor.extract(doc_text)\n", - " if isinstance(entities, list):\n", - " doc_entities[i] = entities\n", - " else:\n", - " doc_entities[i] = [entities] if entities else []\n", - " \n", - " relationships = relation_extractor.extract(doc_text, doc_entities[i])\n", - " if isinstance(relationships, list):\n", - " doc_relationships[i] = relationships\n", - " else:\n", - " doc_relationships[i] = [relationships] if relationships else []\n", - "\n", - "print(f\"Extracted entities from {len(doc_entities)} documents\")\n", - "print(f\"Extracted relationships from {len(doc_relationships)} documents\")\n", - "\n", - "print(\"\\nStep 3: Creating chunks for Graph Store (graph-aware chunking)...\")\n", - "entity_chunker = EntityAwareChunker(\n", - " chunk_size=1000,\n", - " chunk_overlap=200,\n", - " ner_method=\"spacy\",\n", - " preserve_entities=True\n", - ")\n", - "\n", - "relation_chunker = RelationAwareChunker(\n", - " chunk_size=1000,\n", - " chunk_overlap=200,\n", - " preserve_triples=True\n", - ")\n", - "\n", - "graph_store_chunks = []\n", - "\n", - "for i, doc in enumerate(parsed_documents):\n", - " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", - " if not doc_text.strip():\n", - " continue\n", - " \n", - " if i in doc_relationships and len(doc_relationships[i]) > 0:\n", - " chunks = relation_chunker.chunk(\n", - " doc_text,\n", - " relationships=doc_relationships[i]\n", - " )\n", - " elif i in doc_entities and len(doc_entities[i]) > 0:\n", - " chunks = entity_chunker.chunk(\n", - " doc_text,\n", - " entities=doc_entities[i]\n", - " )\n", - " else:\n", - " chunks = entity_chunker.chunk(doc_text)\n", - " \n", - " if isinstance(chunks, list):\n", - " for chunk in chunks:\n", - " if hasattr(chunk, 'metadata'):\n", - " chunk.metadata['chunking_method'] = 'graph_aware'\n", - " chunk.metadata['store_type'] = 'graph'\n", - " chunk.metadata['source_doc'] = i\n", - " if i in doc_entities:\n", - " chunk.metadata['entities'] = doc_entities[i]\n", - " if i in doc_relationships:\n", - " chunk.metadata['relationships'] = doc_relationships[i]\n", - " graph_store_chunks.extend(chunks)\n", - " else:\n", - " if hasattr(chunks, 'metadata'):\n", - " chunks.metadata['chunking_method'] = 'graph_aware'\n", - " chunks.metadata['store_type'] = 'graph'\n", - " chunks.metadata['source_doc'] = i\n", - " graph_store_chunks.append(chunks)\n", - "\n", - "print(f\"Created {len(graph_store_chunks)} graph-aware chunks for graph store\")\n", - "\n", - "chunked_documents = vector_store_chunks + graph_store_chunks\n", - "print(f\"\\nTotal chunks: {len(chunked_documents)}\")\n", - "print(f\" Vector store chunks: {len(vector_store_chunks)}\")\n", - "print(f\" Graph store chunks: {len(graph_store_chunks)}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.2.1: Graph-Based Chunking (Iterative Refinement)\n", - "\n", - "After building the knowledge graph, we can use graph-based chunking to refine chunks based on graph structure. This is useful for re-chunking or optimizing existing chunks.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "graph_chunker = GraphBasedChunker(\n", - " chunk_size=1000,\n", - " chunk_overlap=200,\n", - " strategy=\"community\",\n", - " algorithm=\"louvain\"\n", - ")\n", - "\n", - "print(\"Graph-based chunker initialized\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.3: Normalize Text\n", - "\n", - "Clean and normalize text for better extraction quality.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.normalize import TextNormalizer\n", - "\n", - "text_normalizer = TextNormalizer()\n", - "\n", - "print(\"Normalizing vector store chunks...\")\n", - "normalized_vector_chunks = []\n", - "for chunk in vector_store_chunks:\n", - " normalized = text_normalizer.normalize(chunk)\n", - " if isinstance(normalized, list):\n", - " normalized_vector_chunks.extend(normalized)\n", - " else:\n", - " normalized_vector_chunks.append(normalized)\n", - "\n", - "print(\"Normalizing graph store chunks...\")\n", - "normalized_graph_chunks = []\n", - "for chunk in graph_store_chunks:\n", - " normalized = text_normalizer.normalize(chunk)\n", - " if isinstance(normalized, list):\n", - " normalized_graph_chunks.extend(normalized)\n", - " else:\n", - " normalized_graph_chunks.append(normalized)\n", - "\n", - "normalized_documents = normalized_vector_chunks + normalized_graph_chunks\n", - "print(f\"Normalized {len(normalized_documents)} chunks\")\n", - "print(f\" Vector store chunks: {len(normalized_vector_chunks)}\")\n", - "print(f\" Graph store chunks: {len(normalized_graph_chunks)}\")\n", - "print(\"Document processing complete!\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 4: Semantic Extraction\n", - "\n", - "Extract entities, relationships, and triples from the processed documents. This is the foundation for building the knowledge graph.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.semantic_extract import build as extract_build\n", - "\n", - "print(\"Extracting entities, relationships, and triples...\")\n", - "\n", - "extraction_result = extract_build(\n", - " text=[str(doc.content) if hasattr(doc, 'content') else str(doc) for doc in normalized_documents],\n", - " extract_entities=True,\n", - " extract_relations=True,\n", - " extract_triples=True\n", - ")\n", - "\n", - "flat_entities = extraction_result.get('entities', [])\n", - "flat_relationships = extraction_result.get('relationships', [])\n", - "flat_triples = extraction_result.get('triples', [])\n", - "\n", - "print(f\"Extracted {len(flat_entities)} entities\")\n", - "print(f\"Extracted {len(flat_relationships)} relationships\")\n", - "print(f\"Extracted {len(flat_triples)} triples\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(f\"\\nExtraction Summary:\")\n", - "print(f\"Entities: {len(flat_entities)}\")\n", - "print(f\"Relationships: {len(flat_relationships)}\")\n", - "print(f\"Triples: {len(flat_triples)}\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 5: Knowledge Graph Construction\n", - "\n", - "Build the knowledge graph from extracted entities and relationships. Apply quality assurance measures including deduplication and entity resolution.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.kg.methods import build_kg, resolve_entities, deduplicate_graph\n", - "\n", - "print(\"Deduplicating and resolving entities...\")\n", - "\n", - "deduplicated_result = deduplicate_graph(flat_entities, method=\"default\")\n", - "deduplicated_entities = deduplicated_result.get('entities', flat_entities)\n", - "\n", - "resolved_result = resolve_entities(deduplicated_entities, method=\"fuzzy\")\n", - "resolved_entities = resolved_result.get('entities', deduplicated_entities)\n", - "\n", - "print(f\"Deduplicated: {len(flat_entities)} β†’ {len(deduplicated_entities)} entities\")\n", - "print(f\"Resolved: {len(deduplicated_entities)} β†’ {len(resolved_entities)} entities\")\n", - "\n", - "print(\"Building knowledge graph...\")\n", - "\n", - "kg_result = build_kg(\n", - " sources=[{\n", - " 'entities': resolved_entities,\n", - " 'relationships': flat_relationships,\n", - " 'triples': flat_triples\n", - " }],\n", - " method=\"default\",\n", - " merge_entities=True,\n", - " resolve_conflicts=True\n", - ")\n", - "\n", - "knowledge_graph = kg_result.get('graph')\n", - "\n", - "print(f\"Knowledge graph built!\")\n", - "print(f\"Nodes: {knowledge_graph.number_of_nodes()}\")\n", - "print(f\"Edges: {knowledge_graph.number_of_edges()}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5.2: Analyze Knowledge Graph\n", - "\n", - "Analyze the graph structure to understand its properties and quality.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5.3: Refine Chunks Using Graph-Based Chunking\n", - "\n", - "After building the knowledge graph, we can use graph-based chunking to refine chunks based on graph structure. This creates chunks that align with graph communities or centrality.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "if knowledge_graph and knowledge_graph.number_of_nodes() > 0:\n", - " print(\"Refining chunks using graph-based chunking...\")\n", - " \n", - " refined_chunks = []\n", - " \n", - " for i, doc in enumerate(parsed_documents[:5]):\n", - " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", - " if doc_text.strip():\n", - " try:\n", - " graph_chunks = graph_chunker.chunk(\n", - " doc_text,\n", - " graph=knowledge_graph\n", - " )\n", - " \n", - " if isinstance(graph_chunks, list):\n", - " for chunk in graph_chunks:\n", - " if hasattr(chunk, 'metadata'):\n", - " chunk.metadata['chunking_method'] = 'graph_based'\n", - " chunk.metadata['source_doc'] = i\n", - " refined_chunks.extend(graph_chunks)\n", - " else:\n", - " refined_chunks.append(graph_chunks)\n", - " except Exception as e:\n", - " print(f\"Note: Graph-based chunking not available for doc {i}, using original chunks\")\n", - " continue\n", - " \n", - " if refined_chunks:\n", - " print(f\"Created {len(refined_chunks)} graph-based refined chunks\")\n", - " print(\"These chunks are aligned with graph communities/structure\")\n", - " else:\n", - " print(\"Using original entity/relation-aware chunks\")\n", - "else:\n", - " print(\"Graph is empty, using original entity/relation-aware chunks\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.kg.methods import analyze_graph, calculate_centrality, detect_communities, analyze_connectivity\n", - "\n", - "print(\"Analyzing knowledge graph...\")\n", - "\n", - "graph_metrics = analyze_graph(knowledge_graph, method=\"default\")\n", - "print(f\"\\nGraph Metrics:\")\n", - "print(f\"Nodes: {graph_metrics.get('nodes', 0)}\")\n", - "print(f\"Edges: {graph_metrics.get('edges', 0)}\")\n", - "print(f\"Density: {graph_metrics.get('density', 0):.4f}\")\n", - "\n", - "connectivity = analyze_connectivity(knowledge_graph, method=\"default\")\n", - "print(f\"\\nConnectivity:\")\n", - "print(f\"Connected Components: {connectivity.get('connected_components', 0)}\")\n", - "print(f\"Largest Component Size: {connectivity.get('largest_component_size', 0)}\")\n", - "\n", - "if knowledge_graph.number_of_nodes() > 0:\n", - " centrality = calculate_centrality(knowledge_graph, method='pagerank')\n", - " top_nodes = sorted(centrality.items(), key=lambda x: x[1], reverse=True)[:5]\n", - " print(f\"\\nTop 5 Central Nodes (PageRank):\")\n", - " for node, score in top_nodes:\n", - " print(f\" {node}: {score:.4f}\")\n", - " \n", - " communities = detect_communities(knowledge_graph, method='louvain')\n", - " print(f\"\\nCommunities Detected: {len(communities)}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 5.3: Store Knowledge Graph (Optional)\n", - "\n", - "Optionally persist the knowledge graph to a graph database for long-term storage.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Optional: Store graph in persistent graph database# Uncomment to use KuzuDB (embedded, no server required)# graph_store = GraphStore(backend=\"kuzu\", database_path=\"./graphrag_db\")# graph_store.connect()# # # Store nodes# for node_id, node_data in knowledge_graph.nodes(data=True):# labels = [node_data.get('type', 'Entity')]# properties = {k: v for k, v in node_data.items() if k != 'type'}# graph_store.create_node(labels, properties)# # # Store relationships# for source, target, edge_data in knowledge_graph.edges(data=True):# rel_type = edge_data.get('type', 'RELATED_TO')# properties = {k: v for k, v in edge_data.items() if k != 'type'}# graph_store.create_relationship(source, target, rel_type, properties)# # graph_store.close()# print(\"Knowledge graph stored in database\")print(\"Graph storage is optional. The in-memory graph is ready for GraphRAG.\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 6: Embedding Generation\n", - "\n", - "Generate vector embeddings for documents, entities, and relationships. These embeddings enable semantic search and similarity calculations.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.embeddings import EmbeddingGenerator\n", - "\n", - "embedding_generator = EmbeddingGenerator()\n", - "\n", - "print(\"Generating embeddings for vector store chunks (semantic chunks)...\")\n", - "vector_chunk_embeddings = {}\n", - "\n", - "for i, chunk in enumerate(normalized_vector_chunks):\n", - " text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n", - " if text.strip():\n", - " embedding = embedding_generator.generate(text)\n", - " vector_chunk_embeddings[f\"vector_chunk_{i}\"] = {\n", - " 'embedding': embedding,\n", - " 'text': text,\n", - " 'chunking_method': 'semantic',\n", - " 'store_type': 'vector'\n", - " }\n", - "\n", - "print(f\"Generated {len(vector_chunk_embeddings)} vector store chunk embeddings\")\n", - "\n", - "print(\"\\nGenerating embeddings for graph store chunks (graph-aware chunks)...\")\n", - "graph_chunk_embeddings = {}\n", - "\n", - "for i, chunk in enumerate(normalized_graph_chunks):\n", - " text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n", - " if text.strip():\n", - " embedding = embedding_generator.generate(text)\n", - " graph_chunk_embeddings[f\"graph_chunk_{i}\"] = {\n", - " 'embedding': embedding,\n", - " 'text': text,\n", - " 'chunking_method': 'graph_aware',\n", - " 'store_type': 'graph'\n", - " }\n", - "\n", - "print(f\"Generated {len(graph_chunk_embeddings)} graph store chunk embeddings\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 6.2: Generate Entity Embeddings\n", - "\n", - "Generate embeddings for entities to enable entity-based semantic search.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Generate embeddings for entities\n", - "print(\"Generating embeddings for entities...\")\n", - "entity_embeddings = {}\n", - "\n", - "for entity in resolved_entities[:100]: # Limit to first 100 for demo\n", - " if isinstance(entity, dict):\n", - " entity_text = entity.get('text', entity.get('name', str(entity)))\n", - " else:\n", - " entity_text = str(entity)\n", - " \n", - " if entity_text.strip():\n", - " embedding = embedding_generator.generate(entity_text)\n", - " entity_id = entity.get('id', entity.get('text', str(entity))) if isinstance(entity, dict) else str(entity)\n", - " entity_embeddings[entity_id] = {\n", - " 'embedding': embedding,\n", - " 'text': entity_text\n", - " }\n", - "\n", - "print(f\"Generated {len(entity_embeddings)} entity embeddings\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 7: Vector Store Setup\n", - "\n", - "Store embeddings in a vector store for fast similarity search and retrieval.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.vector_store import VectorStore, HybridSearch\n", - "from semantica.graph_store import GraphStore\n", - "\n", - "vector_store = VectorStore()\n", - "\n", - "vectors = []\n", - "metadata_list = []\n", - "ids = []\n", - "\n", - "print(\"Storing semantic chunks in vector store...\")\n", - "for chunk_id, chunk_data in vector_chunk_embeddings.items():\n", - " vectors.append(chunk_data['embedding'])\n", - " metadata_list.append({\n", - " 'type': 'chunk',\n", - " 'chunking_method': 'semantic',\n", - " 'store_type': 'vector',\n", - " 'text': chunk_data['text'][:200]\n", - " })\n", - " ids.append(chunk_id)\n", - "\n", - "for entity_id, entity_data in entity_embeddings.items():\n", - " vectors.append(entity_data['embedding'])\n", - " metadata_list.append({'type': 'entity', 'text': entity_data['text']})\n", - " ids.append(entity_id)\n", - "\n", - "if vectors:\n", - " vector_store.store(vectors=vectors, metadata=metadata_list, ids=ids)\n", - " print(f\"Stored {len(vectors)} vectors in vector store\")\n", - " print(f\" Semantic chunks: {len(vector_chunk_embeddings)}\")\n", - " print(f\" Entities: {len(entity_embeddings)}\")\n", - "else:\n", - " print(\"No vectors to store\")\n", - "\n", - "print(\"\\nStoring graph-aware chunks in graph store...\")\n", - "graph_store = GraphStore(backend=\"kuzu\", database_path=\"./graphrag_db\")\n", - "graph_store.connect()\n", - "\n", - "for i, chunk in enumerate(graph_store_chunks):\n", - " chunk_text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n", - " if chunk_text.strip():\n", - " chunk_metadata = {\n", - " 'chunking_method': 'graph_aware',\n", - " 'store_type': 'graph',\n", - " 'text': chunk_text[:500]\n", - " }\n", - " if hasattr(chunk, 'metadata'):\n", - " if chunk.metadata.get('entities'):\n", - " chunk_metadata['entities'] = chunk.metadata['entities']\n", - " if chunk.metadata.get('relationships'):\n", - " chunk_metadata['relationships'] = chunk.metadata['relationships']\n", - " \n", - " graph_store.create_node(\n", - " labels=['Chunk'],\n", - " properties={\n", - " 'id': f\"graph_chunk_{i}\",\n", - " **chunk_metadata\n", - " }\n", - " )\n", - "\n", - "print(f\"Stored {len(graph_store_chunks)} graph-aware chunks in graph store\")\n", - "graph_store.close()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 8: Hybrid Search Implementation\n", - "\n", - "Implement hybrid search that combines vector similarity search with knowledge graph traversal for enhanced retrieval.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.vector_store import HybridSearch\n", - "\n", - "hybrid_search = HybridSearch(vector_store=vector_store)\n", - "\n", - "def perform_hybrid_search(query: str, top_k: int = 10):\n", - " query_embedding = embedding_generator.generate(query)\n", - " vector_results = vector_store.search(\n", - " query_vector=query_embedding,\n", - " top_k=top_k * 2\n", - " )\n", - " \n", - " # Graph-based search (if query contains entity mentions)\n", - " graph_results = []\n", - " if knowledge_graph.number_of_nodes() > 0:\n", - " # Extract entities from query\n", - " query_entities = ner_extractor.extract(query)\n", - " if query_entities:\n", - " # Find related nodes in graph\n", - " for entity in query_entities:\n", - " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n", - " # Search for entity in graph\n", - " for node in knowledge_graph.nodes():\n", - " if entity_text.lower() in str(node).lower():\n", - " # Get neighbors\n", - " neighbors = list(knowledge_graph.neighbors(node))\n", - " for neighbor in neighbors[:5]: # Limit neighbors\n", - " graph_results.append({\n", - " 'id': f\"graph_{node}_{neighbor}\",\n", - " 'content': f\"{node} -> {neighbor}\",\n", - " 'score': 0.7, # Graph relevance score\n", - " 'source': 'graph'\n", - " })\n", - " \n", - " # Combine and rank results using hybrid search\n", - " all_results = vector_results + graph_results\n", - " \n", - " # Use hybrid search ranker\n", - " if all_results:\n", - " ranked_results = hybrid_search.ranker.rank([all_results], top_k=top_k)\n", - " return ranked_results[:top_k]\n", - " \n", - " return []\n", - "\n", - "# Test hybrid search\n", - "test_query = \"artificial intelligence and machine learning\"\n", - "print(f\"Testing hybrid search with query: '{test_query}'\")\n", - "search_results = perform_hybrid_search(test_query, top_k=5)\n", - "\n", - "print(f\"Search Results ({len(search_results)}):\")\n", - "for i, result in enumerate(search_results[:5], 1):\n", - " print(f\"\\n{i}. Score: {result.get('score', 0):.4f}\")\n", - " print(f\" Source: {result.get('source', 'unknown')}\")\n", - " content = result.get('content', result.get('text', 'N/A'))\n", - " print(f\" Content: {content[:100]}...\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.context import ContextRetriever, ContextGraphBuilder, AgentMemory\n", - "from semantica.context.methods import retrieve_context, build_context_graph\n", - "\n", - "agent_memory = AgentMemory(\n", - " vector_store=vector_store,\n", - " knowledge_graph=knowledge_graph\n", - ")\n", - "\n", - "context_retriever = ContextRetriever(\n", - " memory_store=agent_memory,\n", - " knowledge_graph=knowledge_graph,\n", - " vector_store=vector_store,\n", - " use_graph_expansion=True,\n", - " max_expansion_hops=2,\n", - " hybrid_alpha=0.5\n", - ")\n", - "\n", - "context_graph_builder = ContextGraphBuilder()\n", - "\n", - "print(\"Context retrieval system initialized\")\n", - "print(f\"Graph expansion: Enabled (max {context_retriever.max_expansion_hops} hops)\")\n", - "print(f\"Hybrid alpha: {context_retriever.hybrid_alpha} (0=vector only, 1=graph only)\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 9.2: Retrieve Context with Graph Expansion\n", - "\n", - "Retrieve context using hybrid approach with graph expansion for multi-hop reasoning.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def retrieve_context_for_query(query: str, max_results: int = 10):\n", - " print(f\"\\nRetrieving context for: '{query}'\")\n", - " \n", - " retrieved_contexts = retrieve_context(\n", - " query=query,\n", - " method=\"hybrid\",\n", - " max_results=max_results,\n", - " knowledge_graph=knowledge_graph,\n", - " vector_store=vector_store,\n", - " use_graph_expansion=True,\n", - " max_hops=2\n", - " )\n", - " \n", - " print(f\"Retrieved {len(retrieved_contexts)} context items\")\n", - " \n", - " for i, ctx in enumerate(retrieved_contexts[:5], 1):\n", - " print(f\"\\n{i}. Relevance: {ctx.score:.4f}\")\n", - " print(f\" Source: {ctx.source}\")\n", - " print(f\" Content: {ctx.content[:150]}...\")\n", - " if hasattr(ctx, 'related_entities') and ctx.related_entities:\n", - " print(f\" Related entities: {len(ctx.related_entities)}\")\n", - " if hasattr(ctx, 'related_relationships') and ctx.related_relationships:\n", - " print(f\" Related relationships: {len(ctx.related_relationships)}\")\n", - " \n", - " return retrieved_contexts\n", - "\n", - "test_query = \"What are the relationships between AI and machine learning?\"\n", - "contexts = retrieve_context_for_query(test_query, max_results=10)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 9.3: Build Context Graph\n", - "\n", - "Build a context graph from retrieved contexts to visualize relationships.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "if contexts:\n", - " context_graph = build_context_graph(\n", - " contexts=contexts,\n", - " method=\"entities_relationships\"\n", - " )\n", - " \n", - " print(f\"Context Graph:\")\n", - " print(f\"Nodes: {context_graph.number_of_nodes()}\")\n", - " print(f\"Edges: {context_graph.number_of_edges()}\")\n", - " \n", - " if context_graph.number_of_nodes() > 0:\n", - " print(f\"\\nSample Context Graph Nodes:\")\n", - " for i, node in enumerate(list(context_graph.nodes())[:5], 1):\n", - " print(f\" {i}. {node}\")\n", - "else:\n", - " print(\"No contexts to build graph from\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 10: GraphRAG Query System\n", - "\n", - "Build a complete GraphRAG query processing pipeline that handles different types of queries and prepares context for LLM integration.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.semantic_extract import NERExtractor\n", - "\n", - "class GraphRAGQuerySystem:\n", - " def __init__(self, context_retriever, knowledge_graph, vector_store):\n", - " self.context_retriever = context_retriever\n", - " self.knowledge_graph = knowledge_graph\n", - " self.vector_store = vector_store\n", - " self.ner_extractor = NERExtractor()\n", - " \n", - " def process_query(self, query: str, max_context: int = 10):\n", - " \"\"\"\n", - " Process a query through the complete GraphRAG pipeline.\n", - " \n", - " Steps:\n", - " 1. Parse user query\n", - " 2. Extract query entities\n", - " 3. Perform hybrid search (vector + graph)\n", - " 4. Retrieve relevant context\n", - " 5. Expand context with graph relationships\n", - " 6. Prepare context for LLM\n", - " \"\"\"\n", - " print(f\"Processing query: '{query}'\")\n", - " \n", - " # Step 1: Extract entities from query\n", - " query_entities = self.ner_extractor.extract(query)\n", - " print(f\"Extracted {len(query_entities)} entities from query\")\n", - " \n", - " # Step 2: Retrieve context\n", - " contexts = self.context_retriever.retrieve(\n", - " query=query,\n", - " max_results=max_context,\n", - " use_graph_expansion=True,\n", - " max_hops=2\n", - " )\n", - " \n", - " # Step 3: Expand context with graph relationships\n", - " expanded_context = self._expand_context_with_graph(contexts, query_entities)\n", - " \n", - " # Step 4: Prepare context for LLM\n", - " llm_context = self._prepare_llm_context(expanded_context, query)\n", - " \n", - " return {\n", - " 'query': query,\n", - " 'query_entities': query_entities,\n", - " 'contexts': contexts,\n", - " 'expanded_context': expanded_context,\n", - " 'llm_context': llm_context\n", - " }\n", - " \n", - " def _expand_context_with_graph(self, contexts, query_entities):\n", - " \"\"\"Expand context by following graph relationships.\"\"\"\n", - " expanded = []\n", - " \n", - " for ctx in contexts:\n", - " expanded.append(ctx)\n", - " \n", - " # Add related entities from graph\n", - " if ctx.related_entities:\n", - " for entity in ctx.related_entities[:3]: # Limit expansion\n", - " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n", - " # Find in graph and get neighbors\n", - " for node in self.knowledge_graph.nodes():\n", - " if entity_text.lower() in str(node).lower():\n", - " neighbors = list(self.knowledge_graph.neighbors(node))[:2]\n", - " for neighbor in neighbors:\n", - " expanded.append({\n", - " 'content': f\"Related: {node} -> {neighbor}\",\n", - " 'score': 0.6,\n", - " 'source': 'graph_expansion'\n", - " })\n", - " \n", - " return expanded\n", - " \n", - " def _prepare_llm_context(self, contexts, query):\n", - " \"\"\"Prepare formatted context for LLM.\"\"\"\n", - " context_text = f\"Query: {query}\\n\\nRelevant Context:\\n\\n\"\n", - " \n", - " for i, ctx in enumerate(contexts[:10], 1):\n", - " content = ctx.content if hasattr(ctx, 'content') else ctx.get('content', str(ctx))\n", - " score = ctx.score if hasattr(ctx, 'score') else ctx.get('score', 0)\n", - " source = ctx.source if hasattr(ctx, 'source') else ctx.get('source', 'unknown')\n", - " \n", - " context_text += f\"{i}. [Relevance: {score:.3f}, Source: {source}]\\n\"\n", - " context_text += f\"{content[:300]}...\\n\\n\"\n", - " \n", - " return context_text\n", - "\n", - "# Initialize GraphRAG query system\n", - "graphrag_system = GraphRAGQuerySystem(\n", - " context_retriever=context_retriever,\n", - " knowledge_graph=knowledge_graph,\n", - " vector_store=vector_store\n", - ")\n", - "\n", - "print(f\"GraphRAG query system initialized\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example queries\n", - "example_queries = [\n", - " \"What is artificial intelligence?\", # Factual question\n", - " \"How are AI and machine learning related?\", # Relationship query\n", - " \"What are the applications of deep learning in healthcare?\", # Complex multi-hop query\n", - "]\n", - "\n", - "# Process each query\n", - "query_results = {}\n", - "for query in example_queries:\n", - " print(f\"\\n{'='*60}\")\n", - " result = graphrag_system.process_query(query, max_context=10)\n", - " query_results[query] = result\n", - " \n", - " print(f\"Prepared LLM Context ({len(result['llm_context'])} chars):\")\n", - " print(result['llm_context'][:500] + \"...\")\n", - "\n", - "print(f\"\\nProcessed {len(query_results)} queries\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 11: LLM Integration\n", - "\n", - "Integrate with LLM (OpenAI, Anthropic, or local) to generate answers using the retrieved GraphRAG context.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# LLM Integration\n", - "# This section demonstrates how to integrate with LLMs using the retrieved context\n", - "\n", - "def generate_answer_with_llm(query: str, llm_context: str, llm_provider: str = \"openai\"):\n", - " \"\"\"\n", - " Generate answer using LLM with GraphRAG context.\n", - " \n", - " Supports OpenAI, Anthropic, or local LLMs.\n", - " \"\"\"\n", - " # Build prompt\n", - " prompt = f\"\"\"You are an AI assistant with access to a knowledge graph and retrieved context.\n", - "\n", - "Context from Knowledge Graph:\n", - "{llm_context}\n", - "\n", - "Question: {query}\n", - "\n", - "Based on the context provided above, please answer the question. If the context doesn't contain enough information, say so. Cite specific entities or relationships from the context when relevant.\n", - "\n", - "Answer:\"\"\"\n", - " \n", - " # Here you would call your LLM\n", - " # Example with OpenAI (uncomment and configure):\n", - " # try:\n", - " # from openai import OpenAI\n", - " # client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))\n", - " # response = client.chat.completions.create(\n", - " # model=\"gpt-4\",\n", - " # messages=[\n", - " # {\"role\": \"system\", \"content\": \"You are a helpful assistant with access to knowledge graphs.\"},\n", - " # {\"role\": \"user\", \"content\": prompt}\n", - " # ],\n", - " # temperature=0.7\n", - " # )\n", - " # return response.choices[0].message.content\n", - " # except Exception as e:\n", - " # return f\"Error calling LLM: {e}\"\n", - " \n", - " # For demonstration, return the prompt structure\n", - " return f\"[LLM Answer would be generated here using the context above]\"\n", - "\n", - "# Example: Generate answer for a query\n", - "if query_results:\n", - " sample_query = list(query_results.keys())[0]\n", - " sample_result = query_results[sample_query]\n", - " \n", - " print(f\"Generating answer for: '{sample_query}'\")\n", - " answer = generate_answer_with_llm(\n", - " query=sample_query,\n", - " llm_context=sample_result['llm_context']\n", - " )\n", - " \n", - " print(f\"\\nAnswer:\")\n", - " print(answer)\n", - " print(f\"\\nContext Statistics:\")\n", - " print(f\" Context items: {len(sample_result['contexts'])}\")\n", - " print(f\" Expanded context: {len(sample_result['expanded_context'])}\")\n", - " print(f\" Query entities: {len(sample_result['query_entities'])}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 11.2: Source Attribution and Explainability\n", - "\n", - "Show which parts of the knowledge graph contributed to the answer for explainability.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def explain_answer_sources(query_result):\n", - " \"\"\"\n", - " Explain which sources contributed to the answer.\n", - " \"\"\"\n", - " print(f\"Answer Sources and Attribution:\")\n", - " print(f\"Query: {query_result['query']}\")\n", - " print(f\"\\nRetrieved Context Sources:\")\n", - " \n", - " sources = {}\n", - " for ctx in query_result['contexts']:\n", - " source = ctx.source if hasattr(ctx, 'source') else ctx.get('source', 'unknown')\n", - " sources[source] = sources.get(source, 0) + 1\n", - " \n", - " for source, count in sources.items():\n", - " print(f\" {source}: {count} context items\")\n", - " \n", - " print(f\"\\nGraph Entities Involved:\")\n", - " for entity in query_result['query_entities'][:5]:\n", - " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n", - " print(f\" - {entity_text}\")\n", - " \n", - " print(f\"\\nContext Expansion:\")\n", - " print(f\" Original contexts: {len(query_result['contexts'])}\")\n", - " print(f\" Expanded contexts: {len(query_result['expanded_context'])}\")\n", - " print(f\" Expansion ratio: {len(query_result['expanded_context']) / max(len(query_result['contexts']), 1):.2f}x\")\n", - "\n", - "# Explain sources for sample query\n", - "if query_results:\n", - " explain_answer_sources(list(query_results.values())[0])\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 12: Advanced Features\n", - "\n", - "Demonstrate advanced features including reasoning, quality assessment, and visualization.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.reasoning import InferenceEngine, RuleManager\n", - "from semantica.kg_qa import KGQualityAssessor\n", - "\n", - "# Advanced Feature 1: Reasoning with Inference Engine\n", - "print(f\"Advanced Feature: Logical Reasoning\")\n", - "inference_engine = InferenceEngine()\n", - "rule_manager = RuleManager()\n", - "\n", - "# Example: Add inference rules\n", - "# rule_manager.add_rule(\"IF entity A works_for entity B AND entity B located_in entity C THEN entity A located_in entity C\")\n", - "# new_facts = inference_engine.forward_chain(knowledge_graph, rule_manager)\n", - "# print(f\"Inferred {len(new_facts)} new facts\")\n", - "\n", - "print(f\"Reasoning can infer new relationships from existing knowledge\")\n", - "\n", - "# Advanced Feature 2: Quality Assessment\n", - "print(\"\\nAdvanced Feature: Knowledge Graph Quality Assessment\")\n", - "kg_quality_assessor = KGQualityAssessor()\n", - "\n", - "if knowledge_graph.number_of_nodes() > 0:\n", - " quality_metrics = kg_quality_assessor.assess(knowledge_graph)\n", - " print(f\"Quality Assessment:\")\n", - " print(f\" Completeness: {quality_metrics.get('completeness', 0):.2%}\")\n", - " print(f\" Consistency: {quality_metrics.get('consistency', 0):.2%}\")\n", - " print(f\" Connectivity: {quality_metrics.get('connectivity', 0):.2%}\")\n", - "else:\n", - " print(f\"Graph is empty, skipping quality assessment\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 12.2: Visualize Knowledge Graph\n", - "\n", - "Visualize the knowledge graph to understand its structure.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.visualization import KGVisualizer, AnalyticsVisualizer\n", - "\n", - "# Initialize visualizer\n", - "kg_visualizer = KGVisualizer()\n", - "analytics_visualizer = AnalyticsVisualizer()\n", - "\n", - "# Visualize knowledge graph\n", - "if knowledge_graph.number_of_nodes() > 0:\n", - " print(f\"Visualizing knowledge graph...\")\n", - " \n", - " # Create visualization\n", - " # Uncomment to generate visualization\n", - " # visualization = kg_visualizer.visualize(\n", - " # knowledge_graph,\n", - " # output_path=\"graphrag_visualization.html\",\n", - " # layout=\"spring\",\n", - " # show_labels=True\n", - " # )\n", - " # print(f\"Visualization saved to graphrag_visualization.html\")\n", - " \n", - " print(f\"Graph Statistics for Visualization:\")\n", - " print(f\" Nodes: {knowledge_graph.number_of_nodes()}\")\n", - " print(f\" Edges: {knowledge_graph.number_of_edges()}\")\n", - " print(f\" Node types: {len(set(n.get('type', 'Unknown') for _, n in knowledge_graph.nodes(data=True)))}\")\n", - " print(f\" Edge types: {len(set(e.get('type', 'Unknown') for _, _, e in knowledge_graph.edges(data=True)))}\")\n", - " \n", - " # Analytics visualization\n", - " # analytics_viz = analytics_visualizer.visualize(\n", - " # knowledge_graph,\n", - " # metrics=['centrality', 'communities'],\n", - " # output_path=\"graphrag_analytics.html\"\n", - " # )\n", - " # print(f\"Analytics visualization saved\")\n", - "else:\n", - " print(f\"Graph is empty, skipping visualization\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 13: Complete End-to-End Example\n", - "\n", - "Demonstrate a complete end-to-end GraphRAG workflow with real-world data, showing the full pipeline from ingestion to answer generation.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def complete_graphrag_workflow(query: str):\n", - " \"\"\"\n", - " Complete GraphRAG workflow from query to answer.\n", - " \"\"\"\n", - " print(f\"\\n{'='*70}\")\n", - " print(f\"Complete GraphRAG Workflow\")\n", - " print(f\"{'='*70}\")\n", - " print(f\"Query: {query}\\n\")\n", - " \n", - " # Step 1: Process query\n", - " print(\"Step 1: Processing query...\")\n", - " result = graphrag_system.process_query(query, max_context=10)\n", - " \n", - " # Step 2: Generate answer\n", - " print(\"\\nStep 2: Generating answer with LLM...\")\n", - " answer = generate_answer_with_llm(query, result['llm_context'])\n", - " \n", - " # Step 3: Explain sources\n", - " print(\"\\nStep 3: Explaining sources...\")\n", - " explain_answer_sources(result)\n", - " \n", - " # Step 4: Show performance metrics\n", - " print(\"\\nStep 4: Performance Metrics:\")\n", - " print(f\" Context retrieval time: <1s (simulated)\")\n", - " print(f\" Context items retrieved: {len(result['contexts'])}\")\n", - " print(f\" Graph expansion hops: 2\")\n", - " print(f\" Total context size: {len(result['llm_context'])} characters\")\n", - " \n", - " return {\n", - " 'query': query,\n", - " 'answer': answer,\n", - " 'contexts': result['contexts'],\n", - " 'metrics': {\n", - " 'context_items': len(result['contexts']),\n", - " 'expanded_items': len(result['expanded_context']),\n", - " 'query_entities': len(result['query_entities'])\n", - " }\n", - " }\n", - "\n", - "# Run complete workflow example\n", - "if len(all_documents) > 0 or knowledge_graph.number_of_nodes() > 0:\n", - " example_query = \"What are the main concepts and their relationships?\"\n", - " workflow_result = complete_graphrag_workflow(example_query)\n", - " \n", - " print(f\"\\nComplete workflow executed successfully!\")\n", - " print(f\"Final Results:\")\n", - " print(f\" Query processed: βœ“\")\n", - " print(f\" Context retrieved: {workflow_result['metrics']['context_items']} items\")\n", - " print(f\" Answer generated: βœ“\")\n", - "else:\n", - " print(\"Configure data sources above to run complete workflow with real data\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(f\"Comparison: Traditional RAG vs GraphRAG\\n\")\n", - "\n", - "comparison = {\n", - " \"Traditional RAG\": {\n", - " \"Retrieval\": \"Vector similarity only\",\n", - " \"Context\": \"Flat document chunks\",\n", - " \"Relationships\": \"Not captured\",\n", - " \"Multi-hop\": \"Not supported\",\n", - " \"Explainability\": \"Limited (source documents only)\"\n", - " },\n", - " \"GraphRAG\": {\n", - " \"Retrieval\": \"Vector + Graph traversal\",\n", - " \"Context\": \"Structured knowledge graph\",\n", - " \"Relationships\": \"Explicitly modeled\",\n", - " \"Multi-hop\": \"Supported (graph expansion)\",\n", - " \"Explainability\": \"High (entities, relationships, paths)\"\n", - " }\n", - "}\n", - "\n", - "print(\"Feature Comparison:\")\n", - "print(f\"{'Feature':<20} {'Traditional RAG':<25} {'GraphRAG':<25}\")\n", - "print(\"-\" * 70)\n", - "\n", - "for feature in comparison[\"Traditional RAG\"].keys():\n", - " trad = comparison[\"Traditional RAG\"][feature]\n", - " graph = comparison[\"GraphRAG\"][feature]\n", - " print(f\"{feature:<20} {trad:<25} {graph:<25}\")\n", - "\n", - "print(\"\\nGraphRAG Advantages:\")\n", - "print(f\" β€’ Better handling of complex queries requiring relationship understanding\")\n", - "print(f\" β€’ Multi-hop reasoning across entities\")\n", - "print(f\" β€’ More accurate answers through structured knowledge\")\n", - "print(f\" β€’ Better explainability with graph paths\")\n", - "print(f\" β€’ Reduced hallucinations through graph validation\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Step 14: Export and Persistence\n", - "\n", - "Export the knowledge graph and save the vector store for reuse and sharing.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from semantica.export import JSONExporter, RDFExporter, CSVExporter\n", - "\n", - "json_exporter = JSONExporter()\n", - "rdf_exporter = RDFExporter()\n", - "csv_exporter = CSVExporter()\n", - "\n", - "# Export knowledge graph to JSON\n", - "if knowledge_graph.number_of_nodes() > 0:\n", - " print(f\"Exporting knowledge graph...\")\n", - " \n", - " # Export to JSON\n", - " json_output = json_exporter.export(knowledge_graph, \"graphrag_knowledge_graph.json\")\n", - " print(f\"Exported to JSON: graphrag_knowledge_graph.json\")\n", - " \n", - " # Export to RDF\n", - " rdf_output = rdf_exporter.export(knowledge_graph, \"graphrag_knowledge_graph.rdf\")\n", - " print(f\"Exported to RDF: graphrag_knowledge_graph.rdf\")\n", - " \n", - " # Export entities to CSV\n", - " entities_data = []\n", - " for entity in resolved_entities[:100]: # Limit for demo\n", - " if isinstance(entity, dict):\n", - " entities_data.append({\n", - " 'id': entity.get('id', ''),\n", - " 'text': entity.get('text', entity.get('name', '')),\n", - " 'type': entity.get('type', 'Unknown')\n", - " })\n", - " \n", - " if entities_data:\n", - " csv_output = csv_exporter.export(entities_data, \"graphrag_entities.csv\")\n", - " print(f\"Exported entities to CSV: graphrag_entities.csv\")\n", - " \n", - " print(f\"\\nExport Summary:\")\n", - " print(f\" Nodes exported: {knowledge_graph.number_of_nodes()}\")\n", - " print(f\" Edges exported: {knowledge_graph.number_of_edges()}\")\n", - " print(f\" Entities exported: {len(entities_data)}\")\n", - "else:\n", - " print(f\"Graph is empty, skipping export\")\n", - "\n", - "# Save vector store (if supported)\n", - "print(\"\\nVector Store:\")\n", - "print(f\" Vectors stored: βœ“\")\n", - "print(f\" Metadata stored: βœ“\")\n", - "print(f\" Ready for reuse: βœ“\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Summary and Next Steps\n", - "\n", - "### What We Built\n", - "\n", - "This notebook demonstrated a **complete end-to-end GraphRAG system** using Semantica:\n", - "\n", - "1. **Real-World Data Ingestion**: MCP servers, web scraping, RSS feeds\n", - "2. **Document Processing**: Parsing, splitting, normalization\n", - "3. **Semantic Extraction**: Entities, relationships, triples\n", - "4. **Knowledge Graph Construction**: With quality assurance\n", - "5. **Embedding Generation**: For documents and entities\n", - "6. **Vector Store**: Fast similarity search\n", - "7. **Hybrid Search**: Combining vectors and graphs\n", - "8. **Context Retrieval**: With graph expansion\n", - "9. **GraphRAG Query System**: Complete query processing\n", - "10. **LLM Integration**: Answer generation with context\n", - "11. **Advanced Features**: Reasoning, quality, visualization\n", - "12. **Export**: Persistence and sharing\n", - "\n", - "### Key Takeaways\n", - "\n", - "- **GraphRAG** combines the best of vector search and knowledge graphs\n", - "- **Multi-hop reasoning** enables deeper understanding\n", - "- **Real-world data** makes the system production-ready\n", - "- **Semantica** provides all modules needed for GraphRAG\n", - "\n", - "### Next Steps\n", - "\n", - "1. **Configure Real Data Sources**: Set up MCP servers, web URLs, or RSS feeds\n", - "2. **Customize Extraction**: Adjust entity and relationship extraction for your domain\n", - "3. **Tune Hybrid Search**: Experiment with `hybrid_alpha` for your use case\n", - "4. **Add More LLMs**: Integrate with Anthropic, local models, or other providers\n", - "5. **Scale Up**: Process larger datasets and optimize performance\n", - "6. **Deploy**: Build production GraphRAG applications\n", - "\n", - "### Resources\n", - "\n", - "- [Semantica Documentation](https://semantica.readthedocs.io/)\n", - "- [GraphRAG Concepts](https://semantica.readthedocs.io/concepts/)\n", - "- [API Reference](https://semantica.readthedocs.io/reference/)\n", - "- [More Examples](https://semantica.readthedocs.io/cookbook/)\n", - "\n", - "---\n", - "\n", - "**Congratulations!** You've built a complete GraphRAG system with Semantica!\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - } - ], - "metadata": { - "language_info": { - "name": "python" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)\n", + "\n", + "# GraphRAG Complete - End-to-End Pipeline\n", + "\n", + "## Overview\n", + "\n", + "This notebook demonstrates a **complete end-to-end GraphRAG (Graph-based Retrieval Augmented Generation) system** using Semantica framework. It showcases how to build a production-ready GraphRAG system that combines vector search with knowledge graph traversal for enhanced retrieval and question answering.\n", + "\n", + "**Key Features:**\n", + "\n", + "- **Real-World Data**: Uses actual data sources via MCP servers, web scraping, and RSS feeds (NO mock data)\n", + "- **Complete Pipeline**: From data ingestion to LLM-powered question answering\n", + "- **Hybrid Retrieval**: Combines vector similarity search with knowledge graph traversal\n", + "- **Multi-hop Reasoning**: Follows relationships across the graph for deeper context\n", + "- **20+ Semantica Modules**: Demonstrates comprehensive use of the framework\n", + "\n", + "**Documentation**: [API Reference](https://semantica.readthedocs.io/concepts/) β€’ [GraphRAG Guide](https://semantica.readthedocs.io/concepts/)\n", + "\n", + "### What You'll Learn\n", + "\n", + "- How to ingest real-world data from multiple sources (MCP, web, feeds)\n", + "- How to build knowledge graphs from unstructured text\n", + "- How to implement hybrid search combining vectors and graphs\n", + "- How to use ContextRetriever for intelligent context expansion\n", + "- How to integrate LLMs with GraphRAG for question answering\n", + "- How to visualize and export knowledge graphs\n", + "\n", + "### Pipeline Overview\n", + "\n", + "**Real-World Data Sources (MCP/Web/Feeds) β†’ Parse β†’ Extract Entities & Relationships β†’ Build Knowledge Graph β†’ Generate Embeddings β†’ Vector Store β†’ Hybrid Search β†’ Context Retrieval β†’ GraphRAG Query System β†’ LLM Integration β†’ Answer Generation**\n", + "\n", + "---\n", + "\n", + "## Installation\n", + "\n", + "Install Semantica from PyPI:\n", + "\n", + "```bash\n", + "pip install semantica\n", + "\n", + "# Or with all optional dependencies:\n", + "pip install semantica[all]\n", + "```\n", + "\n", + "### Additional Dependencies\n", + "\n", + "```bash\n", + "pip install openai anthropic # For LLM integration\n", + "pip install jupyter # For running this notebook\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 1: Setup and Import Semantica Modules\n", + "\n", + "Import all necessary Semantica modules for the complete GraphRAG pipeline. This includes modules for ingestion, parsing, extraction, graph building, embeddings, vector storage, context retrieval, and more.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.core import ConfigManager, Semantica\n", + "\n", + "# Initialize configuration manager\n", + "config_manager = ConfigManager()\n", + "\n", + "# Optionally load from file or dictionary\n", + "# config = config_manager.load_from_file(\"config.yaml\")\n", + "# Or use defaults\n", + "config = config_manager.load_from_dict({})\n", + "\n", + "# Initialize Semantica framework\n", + "framework = Semantica(config=config)\n", + "framework.initialize()\n", + "\n", + "print(\"Configuration and framework initialized\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 2: Ingest Real-World Data from Multiple Sources\n", + "\n", + "1. **MCP Servers** (Primary): Connect to real MCP servers providing news feeds, documentation, APIs\n", + "2. **Web Sources**: Scrape real web content from news sites and documentation\n", + "3. **RSS Feeds**: Ingest real RSS/Atom feeds from news sources\n", + "\n", + "### 2.1: Connect to MCP Servers\n", + "\n", + "Connect to real MCP servers via URL. MCP servers can provide resources (databases, files) and tools (APIs, queries).\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.ingest import MCPIngestor\n", + "\n", + "mcp_ingestor = MCPIngestor()\n", + "\n", + "connected_servers = mcp_ingestor.get_connected_servers()\n", + "print(f\"Connected MCP Servers: {len(connected_servers)}\")\n", + "for server_name in connected_servers:\n", + " print(f\" - {server_name}\")\n", + "\n", + "if connected_servers:\n", + " for server_name in connected_servers:\n", + " try:\n", + " resources = mcp_ingestor.list_available_resources(server_name)\n", + " tools = mcp_ingestor.list_available_tools(server_name)\n", + " print(f\"\\n{server_name}:\")\n", + " print(f\" Resources: {len(resources)}\")\n", + " print(f\" Tools: {len(tools)}\")\n", + " except Exception as e:\n", + " print(f\"Error connecting to {server_name}: {e}\")\n", + "\n", + "print(\"\\nNote: Configure your MCP server URLs above to ingest real data\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.2: Ingest Data from MCP Servers\n", + "\n", + "Ingest real data from MCP server resources and tools.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.ingest import WebIngestor, FeedIngestor\n", + "\n", + "all_documents = []\n", + "\n", + "if connected_servers:\n", + " for server_name in connected_servers:\n", + " try:\n", + " print(f\"\\nIngesting from {server_name}...\")\n", + " mcp_data = mcp_ingestor.ingest_all_resources(server_name)\n", + " \n", + " if isinstance(mcp_data, list):\n", + " all_documents.extend(mcp_data)\n", + " print(f\" Ingested {len(mcp_data)} resources\")\n", + " else:\n", + " all_documents.append(mcp_data)\n", + " print(f\" Ingested 1 resource\")\n", + " \n", + " except Exception as e:\n", + " print(f\" Error ingesting from {server_name}: {e}\")\n", + "\n", + "print(f\"\\nTotal documents from MCP: {len(all_documents)}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.3: Ingest Data from Web Sources\n", + "\n", + "Scrape real web content from news sites, documentation, and articles.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "web_ingestor = WebIngestor()\n", + "\n", + "web_sources = []\n", + "\n", + "web_documents = []\n", + "for url in web_sources:\n", + " try:\n", + " print(f\"Scraping {url}...\")\n", + " docs = web_ingestor.ingest(url)\n", + " if isinstance(docs, list):\n", + " web_documents.extend(docs)\n", + " else:\n", + " web_documents.append(docs)\n", + " print(f\" Scraped {len(docs) if isinstance(docs, list) else 1} document(s)\")\n", + " except Exception as e:\n", + " print(f\" Error scraping {url}: {e}\")\n", + "\n", + "all_documents.extend(web_documents)\n", + "print(f\"\\nTotal documents from web: {len(web_documents)}\")\n", + "print(f\"Total documents so far: {len(all_documents)}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.4: Ingest Data from RSS Feeds\n", + "\n", + "Ingest real RSS/Atom feeds from news sources.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "feed_ingestor = FeedIngestor()\n", + "\n", + "feed_urls = []\n", + "\n", + "feed_documents = []\n", + "for feed_url in feed_urls:\n", + " try:\n", + " print(f\"Fetching feed {feed_url}...\")\n", + " feeds = feed_ingestor.ingest(feed_url)\n", + " if isinstance(feeds, list):\n", + " feed_documents.extend(feeds)\n", + " else:\n", + " feed_documents.append(feeds)\n", + " print(f\" Fetched {len(feeds) if isinstance(feeds, list) else 1} feed item(s)\")\n", + " except Exception as e:\n", + " print(f\" Error fetching feed {feed_url}: {e}\")\n", + "\n", + "all_documents.extend(feed_documents)\n", + "print(f\"\\nTotal documents from feeds: {len(feed_documents)}\")\n", + "print(f\"Total documents collected: {len(all_documents)}\")\n", + "\n", + "if len(all_documents) == 0:\n", + " print(\"\\nNo documents collected. Please configure MCP servers, web URLs, or RSS feeds above.\")\n", + " print(\"For this demonstration, we'll continue with the pipeline structure.\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 3: Document Processing Pipeline\n", + "\n", + "Process the ingested documents: parse, split, and normalize the text for extraction.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.parse import DocumentParser, MCPParser\n", + "\n", + "document_parser = DocumentParser()\n", + "mcp_parser = MCPParser()\n", + "\n", + "parsed_documents = []\n", + "\n", + "for doc in all_documents:\n", + " try:\n", + " if hasattr(doc, 'source') and 'mcp' in doc.source.lower():\n", + " parsed = mcp_parser.parse(doc)\n", + " else:\n", + " parsed = document_parser.parse(doc)\n", + " \n", + " if isinstance(parsed, list):\n", + " parsed_documents.extend(parsed)\n", + " else:\n", + " parsed_documents.append(parsed)\n", + " except Exception as e:\n", + " print(f\"Error parsing document: {e}\")\n", + " continue\n", + "\n", + "print(f\"Parsed {len(parsed_documents)} documents\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.2: Split Documents Using Dual Chunking Strategy\n", + "\n", + "For GraphRAG, we use **two different chunking methods** optimized for different stores:\n", + "\n", + "**For Vector Store** (semantic similarity search):\n", + "- **Semantic Chunking**: Uses embeddings to find natural semantic boundaries\n", + "- Better for vector similarity search and retrieval\n", + "\n", + "**For Graph Store** (knowledge structure preservation):\n", + "- **Entity-Aware Chunking**: Preserves entity boundaries (prevents splitting entities)\n", + "- **Relation-Aware Chunking**: Preserves relationship triples (keeps subject-predicate-object together)\n", + "- **Graph-Based Chunking**: Uses existing graph structure for optimal chunking\n", + "\n", + "We'll create chunks optimized for each store type.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.semantic_extract import NERExtractor, RelationExtractor\n", + "from semantica.split import (\n", + " SemanticChunker, EntityAwareChunker, RelationAwareChunker, GraphBasedChunker\n", + ")\n", + "import numpy as np\n", + "\n", + "print(\"Step 1: Creating chunks for Vector Store (semantic chunking)...\")\n", + "semantic_chunker = SemanticChunker(\n", + " chunk_size=1000,\n", + " chunk_overlap=200\n", + ")\n", + "\n", + "vector_store_chunks = []\n", + "for i, doc in enumerate(parsed_documents):\n", + " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", + " if doc_text.strip():\n", + " chunks = semantic_chunker.chunk(doc_text)\n", + " if isinstance(chunks, list):\n", + " for chunk in chunks:\n", + " if hasattr(chunk, 'metadata'):\n", + " chunk.metadata['chunking_method'] = 'semantic'\n", + " chunk.metadata['store_type'] = 'vector'\n", + " chunk.metadata['source_doc'] = i\n", + " vector_store_chunks.extend(chunks)\n", + " else:\n", + " if hasattr(chunks, 'metadata'):\n", + " chunks.metadata['chunking_method'] = 'semantic'\n", + " chunks.metadata['store_type'] = 'vector'\n", + " chunks.metadata['source_doc'] = i\n", + " vector_store_chunks.append(chunks)\n", + "\n", + "print(f\"Created {len(vector_store_chunks)} semantic chunks for vector store\")\n", + "\n", + "print(\"\\nStep 2: Extracting entities/relationships for graph-aware chunking...\")\n", + "ner_extractor = NERExtractor()\n", + "relation_extractor = RelationExtractor()\n", + "\n", + "doc_entities = {}\n", + "doc_relationships = {}\n", + "\n", + "for i, doc in enumerate(parsed_documents):\n", + " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", + " if doc_text.strip():\n", + " entities = ner_extractor.extract(doc_text)\n", + " if isinstance(entities, list):\n", + " doc_entities[i] = entities\n", + " else:\n", + " doc_entities[i] = [entities] if entities else []\n", + " \n", + " relationships = relation_extractor.extract(doc_text, doc_entities[i])\n", + " if isinstance(relationships, list):\n", + " doc_relationships[i] = relationships\n", + " else:\n", + " doc_relationships[i] = [relationships] if relationships else []\n", + "\n", + "print(f\"Extracted entities from {len(doc_entities)} documents\")\n", + "print(f\"Extracted relationships from {len(doc_relationships)} documents\")\n", + "\n", + "print(\"\\nStep 3: Creating chunks for Graph Store (graph-aware chunking)...\")\n", + "entity_chunker = EntityAwareChunker(\n", + " chunk_size=1000,\n", + " chunk_overlap=200,\n", + " ner_method=\"spacy\",\n", + " preserve_entities=True\n", + ")\n", + "\n", + "relation_chunker = RelationAwareChunker(\n", + " chunk_size=1000,\n", + " chunk_overlap=200,\n", + " preserve_triples=True\n", + ")\n", + "\n", + "graph_store_chunks = []\n", + "\n", + "for i, doc in enumerate(parsed_documents):\n", + " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", + " if not doc_text.strip():\n", + " continue\n", + " \n", + " if i in doc_relationships and len(doc_relationships[i]) > 0:\n", + " chunks = relation_chunker.chunk(\n", + " doc_text,\n", + " relationships=doc_relationships[i]\n", + " )\n", + " elif i in doc_entities and len(doc_entities[i]) > 0:\n", + " chunks = entity_chunker.chunk(\n", + " doc_text,\n", + " entities=doc_entities[i]\n", + " )\n", + " else:\n", + " chunks = entity_chunker.chunk(doc_text)\n", + " \n", + " if isinstance(chunks, list):\n", + " for chunk in chunks:\n", + " if hasattr(chunk, 'metadata'):\n", + " chunk.metadata['chunking_method'] = 'graph_aware'\n", + " chunk.metadata['store_type'] = 'graph'\n", + " chunk.metadata['source_doc'] = i\n", + " if i in doc_entities:\n", + " chunk.metadata['entities'] = doc_entities[i]\n", + " if i in doc_relationships:\n", + " chunk.metadata['relationships'] = doc_relationships[i]\n", + " graph_store_chunks.extend(chunks)\n", + " else:\n", + " if hasattr(chunks, 'metadata'):\n", + " chunks.metadata['chunking_method'] = 'graph_aware'\n", + " chunks.metadata['store_type'] = 'graph'\n", + " chunks.metadata['source_doc'] = i\n", + " graph_store_chunks.append(chunks)\n", + "\n", + "print(f\"Created {len(graph_store_chunks)} graph-aware chunks for graph store\")\n", + "\n", + "chunked_documents = vector_store_chunks + graph_store_chunks\n", + "print(f\"\\nTotal chunks: {len(chunked_documents)}\")\n", + "print(f\" Vector store chunks: {len(vector_store_chunks)}\")\n", + "print(f\" Graph store chunks: {len(graph_store_chunks)}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.2.1: Graph-Based Chunking (Iterative Refinement)\n", + "\n", + "After building the knowledge graph, we can use graph-based chunking to refine chunks based on graph structure. This is useful for re-chunking or optimizing existing chunks.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "graph_chunker = GraphBasedChunker(\n", + " chunk_size=1000,\n", + " chunk_overlap=200,\n", + " strategy=\"community\",\n", + " algorithm=\"louvain\"\n", + ")\n", + "\n", + "print(\"Graph-based chunker initialized\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.3: Normalize Text\n", + "\n", + "Clean and normalize text for better extraction quality.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.normalize import TextNormalizer\n", + "\n", + "text_normalizer = TextNormalizer()\n", + "\n", + "print(\"Normalizing vector store chunks...\")\n", + "normalized_vector_chunks = []\n", + "for chunk in vector_store_chunks:\n", + " normalized = text_normalizer.normalize(chunk)\n", + " if isinstance(normalized, list):\n", + " normalized_vector_chunks.extend(normalized)\n", + " else:\n", + " normalized_vector_chunks.append(normalized)\n", + "\n", + "print(\"Normalizing graph store chunks...\")\n", + "normalized_graph_chunks = []\n", + "for chunk in graph_store_chunks:\n", + " normalized = text_normalizer.normalize(chunk)\n", + " if isinstance(normalized, list):\n", + " normalized_graph_chunks.extend(normalized)\n", + " else:\n", + " normalized_graph_chunks.append(normalized)\n", + "\n", + "normalized_documents = normalized_vector_chunks + normalized_graph_chunks\n", + "print(f\"Normalized {len(normalized_documents)} chunks\")\n", + "print(f\" Vector store chunks: {len(normalized_vector_chunks)}\")\n", + "print(f\" Graph store chunks: {len(normalized_graph_chunks)}\")\n", + "print(\"Document processing complete!\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 4: Semantic Extraction\n", + "\n", + "Extract entities, relationships, and triples from the processed documents. This is the foundation for building the knowledge graph.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.semantic_extract import build as extract_build\n", + "\n", + "print(\"Extracting entities, relationships, and triples...\")\n", + "\n", + "extraction_result = extract_build(\n", + " text=[str(doc.content) if hasattr(doc, 'content') else str(doc) for doc in normalized_documents],\n", + " extract_entities=True,\n", + " extract_relations=True,\n", + " extract_triples=True\n", + ")\n", + "\n", + "flat_entities = extraction_result.get('entities', [])\n", + "flat_relationships = extraction_result.get('relationships', [])\n", + "flat_triples = extraction_result.get('triples', [])\n", + "\n", + "print(f\"Extracted {len(flat_entities)} entities\")\n", + "print(f\"Extracted {len(flat_relationships)} relationships\")\n", + "print(f\"Extracted {len(flat_triples)} triples\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(f\"\\nExtraction Summary:\")\n", + "print(f\"Entities: {len(flat_entities)}\")\n", + "print(f\"Relationships: {len(flat_relationships)}\")\n", + "print(f\"Triples: {len(flat_triples)}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 5: Knowledge Graph Construction\n", + "\n", + "Build the knowledge graph from extracted entities and relationships. Apply quality assurance measures including deduplication and entity resolution.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg.methods import build_kg, resolve_entities, deduplicate_graph\n", + "\n", + "print(\"Deduplicating and resolving entities...\")\n", + "\n", + "deduplicated_result = deduplicate_graph(flat_entities, method=\"default\")\n", + "deduplicated_entities = deduplicated_result.get('entities', flat_entities)\n", + "\n", + "resolved_result = resolve_entities(deduplicated_entities, method=\"fuzzy\")\n", + "resolved_entities = resolved_result.get('entities', deduplicated_entities)\n", + "\n", + "print(f\"Deduplicated: {len(flat_entities)} β†’ {len(deduplicated_entities)} entities\")\n", + "print(f\"Resolved: {len(deduplicated_entities)} β†’ {len(resolved_entities)} entities\")\n", + "\n", + "print(\"Building knowledge graph...\")\n", + "\n", + "kg_result = build_kg(\n", + " sources=[{\n", + " 'entities': resolved_entities,\n", + " 'relationships': flat_relationships,\n", + " 'triples': flat_triples\n", + " }],\n", + " method=\"default\",\n", + " merge_entities=True,\n", + " resolve_conflicts=True\n", + ")\n", + "\n", + "knowledge_graph = kg_result.get('graph')\n", + "\n", + "print(f\"Knowledge graph built!\")\n", + "print(f\"Nodes: {knowledge_graph.number_of_nodes()}\")\n", + "print(f\"Edges: {knowledge_graph.number_of_edges()}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5.2: Analyze Knowledge Graph\n", + "\n", + "Analyze the graph structure to understand its properties and quality.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5.3: Refine Chunks Using Graph-Based Chunking\n", + "\n", + "After building the knowledge graph, we can use graph-based chunking to refine chunks based on graph structure. This creates chunks that align with graph communities or centrality.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "if knowledge_graph and knowledge_graph.number_of_nodes() > 0:\n", + " print(\"Refining chunks using graph-based chunking...\")\n", + " \n", + " refined_chunks = []\n", + " \n", + " for i, doc in enumerate(parsed_documents[:5]):\n", + " doc_text = str(doc.content) if hasattr(doc, 'content') else str(doc)\n", + " if doc_text.strip():\n", + " try:\n", + " graph_chunks = graph_chunker.chunk(\n", + " doc_text,\n", + " graph=knowledge_graph\n", + " )\n", + " \n", + " if isinstance(graph_chunks, list):\n", + " for chunk in graph_chunks:\n", + " if hasattr(chunk, 'metadata'):\n", + " chunk.metadata['chunking_method'] = 'graph_based'\n", + " chunk.metadata['source_doc'] = i\n", + " refined_chunks.extend(graph_chunks)\n", + " else:\n", + " refined_chunks.append(graph_chunks)\n", + " except Exception as e:\n", + " print(f\"Note: Graph-based chunking not available for doc {i}, using original chunks\")\n", + " continue\n", + " \n", + " if refined_chunks:\n", + " print(f\"Created {len(refined_chunks)} graph-based refined chunks\")\n", + " print(\"These chunks are aligned with graph communities/structure\")\n", + " else:\n", + " print(\"Using original entity/relation-aware chunks\")\n", + "else:\n", + " print(\"Graph is empty, using original entity/relation-aware chunks\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.kg.methods import analyze_graph, calculate_centrality, detect_communities, analyze_connectivity\n", + "\n", + "print(\"Analyzing knowledge graph...\")\n", + "\n", + "graph_metrics = analyze_graph(knowledge_graph, method=\"default\")\n", + "print(f\"\\nGraph Metrics:\")\n", + "print(f\"Nodes: {graph_metrics.get('nodes', 0)}\")\n", + "print(f\"Edges: {graph_metrics.get('edges', 0)}\")\n", + "print(f\"Density: {graph_metrics.get('density', 0):.4f}\")\n", + "\n", + "connectivity = analyze_connectivity(knowledge_graph, method=\"default\")\n", + "print(f\"\\nConnectivity:\")\n", + "print(f\"Connected Components: {connectivity.get('connected_components', 0)}\")\n", + "print(f\"Largest Component Size: {connectivity.get('largest_component_size', 0)}\")\n", + "\n", + "if knowledge_graph.number_of_nodes() > 0:\n", + " centrality = calculate_centrality(knowledge_graph, method='pagerank')\n", + " top_nodes = sorted(centrality.items(), key=lambda x: x[1], reverse=True)[:5]\n", + " print(f\"\\nTop 5 Central Nodes (PageRank):\")\n", + " for node, score in top_nodes:\n", + " print(f\" {node}: {score:.4f}\")\n", + " \n", + " communities = detect_communities(knowledge_graph, method='louvain')\n", + " print(f\"\\nCommunities Detected: {len(communities)}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 5.3: Store Knowledge Graph (Optional)\n", + "\n", + "Optionally persist the knowledge graph to a graph database for long-term storage.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Optional: Store graph in persistent graph database# Uncomment to use KuzuDB (embedded, no server required)# graph_store = GraphStore(backend=\"kuzu\", database_path=\"./graphrag_db\")# graph_store.connect()# # # Store nodes# for node_id, node_data in knowledge_graph.nodes(data=True):# labels = [node_data.get('type', 'Entity')]# properties = {k: v for k, v in node_data.items() if k != 'type'}# graph_store.create_node(labels, properties)# # # Store relationships# for source, target, edge_data in knowledge_graph.edges(data=True):# rel_type = edge_data.get('type', 'RELATED_TO')# properties = {k: v for k, v in edge_data.items() if k != 'type'}# graph_store.create_relationship(source, target, rel_type, properties)# # graph_store.close()# print(\"Knowledge graph stored in database\")print(\"Graph storage is optional. The in-memory graph is ready for GraphRAG.\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 6: Embedding Generation\n", + "\n", + "Generate vector embeddings for documents, entities, and relationships. These embeddings enable semantic search and similarity calculations.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.embeddings import EmbeddingGenerator\n", + "\n", + "embedding_generator = EmbeddingGenerator()\n", + "\n", + "print(\"Generating embeddings for vector store chunks (semantic chunks)...\")\n", + "vector_chunk_embeddings = {}\n", + "\n", + "for i, chunk in enumerate(normalized_vector_chunks):\n", + " text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n", + " if text.strip():\n", + " embedding = embedding_generator.generate(text)\n", + " vector_chunk_embeddings[f\"vector_chunk_{i}\"] = {\n", + " 'embedding': embedding,\n", + " 'text': text,\n", + " 'chunking_method': 'semantic',\n", + " 'store_type': 'vector'\n", + " }\n", + "\n", + "print(f\"Generated {len(vector_chunk_embeddings)} vector store chunk embeddings\")\n", + "\n", + "print(\"\\nGenerating embeddings for graph store chunks (graph-aware chunks)...\")\n", + "graph_chunk_embeddings = {}\n", + "\n", + "for i, chunk in enumerate(normalized_graph_chunks):\n", + " text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n", + " if text.strip():\n", + " embedding = embedding_generator.generate(text)\n", + " graph_chunk_embeddings[f\"graph_chunk_{i}\"] = {\n", + " 'embedding': embedding,\n", + " 'text': text,\n", + " 'chunking_method': 'graph_aware',\n", + " 'store_type': 'graph'\n", + " }\n", + "\n", + "print(f\"Generated {len(graph_chunk_embeddings)} graph store chunk embeddings\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 6.2: Generate Entity Embeddings\n", + "\n", + "Generate embeddings for entities to enable entity-based semantic search.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Generate embeddings for entities\n", + "print(\"Generating embeddings for entities...\")\n", + "entity_embeddings = {}\n", + "\n", + "for entity in resolved_entities[:100]: # Limit to first 100 for demo\n", + " if isinstance(entity, dict):\n", + " entity_text = entity.get('text', entity.get('name', str(entity)))\n", + " else:\n", + " entity_text = str(entity)\n", + " \n", + " if entity_text.strip():\n", + " embedding = embedding_generator.generate(entity_text)\n", + " entity_id = entity.get('id', entity.get('text', str(entity))) if isinstance(entity, dict) else str(entity)\n", + " entity_embeddings[entity_id] = {\n", + " 'embedding': embedding,\n", + " 'text': entity_text\n", + " }\n", + "\n", + "print(f\"Generated {len(entity_embeddings)} entity embeddings\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 7: Vector Store Setup\n", + "\n", + "Store embeddings in a vector store for fast similarity search and retrieval.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.vector_store import VectorStore, HybridSearch\n", + "from semantica.graph_store import GraphStore\n", + "\n", + "vector_store = VectorStore()\n", + "\n", + "vectors = []\n", + "metadata_list = []\n", + "ids = []\n", + "\n", + "print(\"Storing semantic chunks in vector store...\")\n", + "for chunk_id, chunk_data in vector_chunk_embeddings.items():\n", + " vectors.append(chunk_data['embedding'])\n", + " metadata_list.append({\n", + " 'type': 'chunk',\n", + " 'chunking_method': 'semantic',\n", + " 'store_type': 'vector',\n", + " 'text': chunk_data['text'][:200]\n", + " })\n", + " ids.append(chunk_id)\n", + "\n", + "for entity_id, entity_data in entity_embeddings.items():\n", + " vectors.append(entity_data['embedding'])\n", + " metadata_list.append({'type': 'entity', 'text': entity_data['text']})\n", + " ids.append(entity_id)\n", + "\n", + "if vectors:\n", + " vector_store.store(vectors=vectors, metadata=metadata_list, ids=ids)\n", + " print(f\"Stored {len(vectors)} vectors in vector store\")\n", + " print(f\" Semantic chunks: {len(vector_chunk_embeddings)}\")\n", + " print(f\" Entities: {len(entity_embeddings)}\")\n", + "else:\n", + " print(\"No vectors to store\")\n", + "\n", + "print(\"\\nStoring graph-aware chunks in graph store...\")\n", + "graph_store = GraphStore(backend=\"kuzu\", database_path=\"./graphrag_db\")\n", + "graph_store.connect()\n", + "\n", + "for i, chunk in enumerate(graph_store_chunks):\n", + " chunk_text = str(chunk.text if hasattr(chunk, 'text') else chunk)\n", + " if chunk_text.strip():\n", + " chunk_metadata = {\n", + " 'chunking_method': 'graph_aware',\n", + " 'store_type': 'graph',\n", + " 'text': chunk_text[:500]\n", + " }\n", + " if hasattr(chunk, 'metadata'):\n", + " if chunk.metadata.get('entities'):\n", + " chunk_metadata['entities'] = chunk.metadata['entities']\n", + " if chunk.metadata.get('relationships'):\n", + " chunk_metadata['relationships'] = chunk.metadata['relationships']\n", + " \n", + " graph_store.create_node(\n", + " labels=['Chunk'],\n", + " properties={\n", + " 'id': f\"graph_chunk_{i}\",\n", + " **chunk_metadata\n", + " }\n", + " )\n", + "\n", + "print(f\"Stored {len(graph_store_chunks)} graph-aware chunks in graph store\")\n", + "graph_store.close()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 8: Hybrid Search Implementation\n", + "\n", + "Implement hybrid search that combines vector similarity search with knowledge graph traversal for enhanced retrieval.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.vector_store import HybridSearch\n", + "\n", + "hybrid_search = HybridSearch(vector_store=vector_store)\n", + "\n", + "def perform_hybrid_search(query: str, top_k: int = 10):\n", + " query_embedding = embedding_generator.generate(query)\n", + " vector_results = vector_store.search(\n", + " query_vector=query_embedding,\n", + " top_k=top_k * 2\n", + " )\n", + " \n", + " # Graph-based search (if query contains entity mentions)\n", + " graph_results = []\n", + " if knowledge_graph.number_of_nodes() > 0:\n", + " # Extract entities from query\n", + " query_entities = ner_extractor.extract(query)\n", + " if query_entities:\n", + " # Find related nodes in graph\n", + " for entity in query_entities:\n", + " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n", + " # Search for entity in graph\n", + " for node in knowledge_graph.nodes():\n", + " if entity_text.lower() in str(node).lower():\n", + " # Get neighbors\n", + " neighbors = list(knowledge_graph.neighbors(node))\n", + " for neighbor in neighbors[:5]: # Limit neighbors\n", + " graph_results.append({\n", + " 'id': f\"graph_{node}_{neighbor}\",\n", + " 'content': f\"{node} -> {neighbor}\",\n", + " 'score': 0.7, # Graph relevance score\n", + " 'source': 'graph'\n", + " })\n", + " \n", + " # Combine and rank results using hybrid search\n", + " all_results = vector_results + graph_results\n", + " \n", + " # Use hybrid search ranker\n", + " if all_results:\n", + " ranked_results = hybrid_search.ranker.rank([all_results], top_k=top_k)\n", + " return ranked_results[:top_k]\n", + " \n", + " return []\n", + "\n", + "# Test hybrid search\n", + "test_query = \"artificial intelligence and machine learning\"\n", + "print(f\"Testing hybrid search with query: '{test_query}'\")\n", + "search_results = perform_hybrid_search(test_query, top_k=5)\n", + "\n", + "print(f\"Search Results ({len(search_results)}):\")\n", + "for i, result in enumerate(search_results[:5], 1):\n", + " print(f\"\\n{i}. Score: {result.get('score', 0):.4f}\")\n", + " print(f\" Source: {result.get('source', 'unknown')}\")\n", + " content = result.get('content', result.get('text', 'N/A'))\n", + " print(f\" Content: {content[:100]}...\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.context import ContextRetriever, ContextGraphBuilder, AgentMemory\n", + "from semantica.context.methods import retrieve_context, build_context_graph\n", + "\n", + "agent_memory = AgentMemory(\n", + " vector_store=vector_store,\n", + " knowledge_graph=knowledge_graph\n", + ")\n", + "\n", + "context_retriever = ContextRetriever(\n", + " memory_store=agent_memory,\n", + " knowledge_graph=knowledge_graph,\n", + " vector_store=vector_store,\n", + " use_graph_expansion=True,\n", + " max_expansion_hops=2,\n", + " hybrid_alpha=0.5\n", + ")\n", + "\n", + "context_graph_builder = ContextGraphBuilder()\n", + "\n", + "print(\"Context retrieval system initialized\")\n", + "print(f\"Graph expansion: Enabled (max {context_retriever.max_expansion_hops} hops)\")\n", + "print(f\"Hybrid alpha: {context_retriever.hybrid_alpha} (0=vector only, 1=graph only)\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 9.2: Retrieve Context with Graph Expansion\n", + "\n", + "Retrieve context using hybrid approach with graph expansion for multi-hop reasoning.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def retrieve_context_for_query(query: str, max_results: int = 10):\n", + " print(f\"\\nRetrieving context for: '{query}'\")\n", + " \n", + " retrieved_contexts = retrieve_context(\n", + " query=query,\n", + " method=\"hybrid\",\n", + " max_results=max_results,\n", + " knowledge_graph=knowledge_graph,\n", + " vector_store=vector_store,\n", + " use_graph_expansion=True,\n", + " max_hops=2\n", + " )\n", + " \n", + " print(f\"Retrieved {len(retrieved_contexts)} context items\")\n", + " \n", + " for i, ctx in enumerate(retrieved_contexts[:5], 1):\n", + " print(f\"\\n{i}. Relevance: {ctx.score:.4f}\")\n", + " print(f\" Source: {ctx.source}\")\n", + " print(f\" Content: {ctx.content[:150]}...\")\n", + " if hasattr(ctx, 'related_entities') and ctx.related_entities:\n", + " print(f\" Related entities: {len(ctx.related_entities)}\")\n", + " if hasattr(ctx, 'related_relationships') and ctx.related_relationships:\n", + " print(f\" Related relationships: {len(ctx.related_relationships)}\")\n", + " \n", + " return retrieved_contexts\n", + "\n", + "test_query = \"What are the relationships between AI and machine learning?\"\n", + "contexts = retrieve_context_for_query(test_query, max_results=10)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 9.3: Build Context Graph\n", + "\n", + "Build a context graph from retrieved contexts to visualize relationships.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "if contexts:\n", + " context_graph = build_context_graph(\n", + " contexts=contexts,\n", + " method=\"entities_relationships\"\n", + " )\n", + " \n", + " print(f\"Context Graph:\")\n", + " print(f\"Nodes: {context_graph.number_of_nodes()}\")\n", + " print(f\"Edges: {context_graph.number_of_edges()}\")\n", + " \n", + " if context_graph.number_of_nodes() > 0:\n", + " print(f\"\\nSample Context Graph Nodes:\")\n", + " for i, node in enumerate(list(context_graph.nodes())[:5], 1):\n", + " print(f\" {i}. {node}\")\n", + "else:\n", + " print(\"No contexts to build graph from\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 10: GraphRAG Query System\n", + "\n", + "Build a complete GraphRAG query processing pipeline that handles different types of queries and prepares context for LLM integration.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.semantic_extract import NERExtractor\n", + "\n", + "class GraphRAGQuerySystem:\n", + " def __init__(self, context_retriever, knowledge_graph, vector_store):\n", + " self.context_retriever = context_retriever\n", + " self.knowledge_graph = knowledge_graph\n", + " self.vector_store = vector_store\n", + " self.ner_extractor = NERExtractor()\n", + " \n", + " def process_query(self, query: str, max_context: int = 10):\n", + " \"\"\"\n", + " Process a query through the complete GraphRAG pipeline.\n", + " \n", + " Steps:\n", + " 1. Parse user query\n", + " 2. Extract query entities\n", + " 3. Perform hybrid search (vector + graph)\n", + " 4. Retrieve relevant context\n", + " 5. Expand context with graph relationships\n", + " 6. Prepare context for LLM\n", + " \"\"\"\n", + " print(f\"Processing query: '{query}'\")\n", + " \n", + " # Step 1: Extract entities from query\n", + " query_entities = self.ner_extractor.extract(query)\n", + " print(f\"Extracted {len(query_entities)} entities from query\")\n", + " \n", + " # Step 2: Retrieve context\n", + " contexts = self.context_retriever.retrieve(\n", + " query=query,\n", + " max_results=max_context,\n", + " use_graph_expansion=True,\n", + " max_hops=2\n", + " )\n", + " \n", + " # Step 3: Expand context with graph relationships\n", + " expanded_context = self._expand_context_with_graph(contexts, query_entities)\n", + " \n", + " # Step 4: Prepare context for LLM\n", + " llm_context = self._prepare_llm_context(expanded_context, query)\n", + " \n", + " return {\n", + " 'query': query,\n", + " 'query_entities': query_entities,\n", + " 'contexts': contexts,\n", + " 'expanded_context': expanded_context,\n", + " 'llm_context': llm_context\n", + " }\n", + " \n", + " def _expand_context_with_graph(self, contexts, query_entities):\n", + " \"\"\"Expand context by following graph relationships.\"\"\"\n", + " expanded = []\n", + " \n", + " for ctx in contexts:\n", + " expanded.append(ctx)\n", + " \n", + " # Add related entities from graph\n", + " if ctx.related_entities:\n", + " for entity in ctx.related_entities[:3]: # Limit expansion\n", + " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n", + " # Find in graph and get neighbors\n", + " for node in self.knowledge_graph.nodes():\n", + " if entity_text.lower() in str(node).lower():\n", + " neighbors = list(self.knowledge_graph.neighbors(node))[:2]\n", + " for neighbor in neighbors:\n", + " expanded.append({\n", + " 'content': f\"Related: {node} -> {neighbor}\",\n", + " 'score': 0.6,\n", + " 'source': 'graph_expansion'\n", + " })\n", + " \n", + " return expanded\n", + " \n", + " def _prepare_llm_context(self, contexts, query):\n", + " \"\"\"Prepare formatted context for LLM.\"\"\"\n", + " context_text = f\"Query: {query}\\n\\nRelevant Context:\\n\\n\"\n", + " \n", + " for i, ctx in enumerate(contexts[:10], 1):\n", + " content = ctx.content if hasattr(ctx, 'content') else ctx.get('content', str(ctx))\n", + " score = ctx.score if hasattr(ctx, 'score') else ctx.get('score', 0)\n", + " source = ctx.source if hasattr(ctx, 'source') else ctx.get('source', 'unknown')\n", + " \n", + " context_text += f\"{i}. [Relevance: {score:.3f}, Source: {source}]\\n\"\n", + " context_text += f\"{content[:300]}...\\n\\n\"\n", + " \n", + " return context_text\n", + "\n", + "# Initialize GraphRAG query system\n", + "graphrag_system = GraphRAGQuerySystem(\n", + " context_retriever=context_retriever,\n", + " knowledge_graph=knowledge_graph,\n", + " vector_store=vector_store\n", + ")\n", + "\n", + "print(f\"GraphRAG query system initialized\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Example queries\n", + "example_queries = [\n", + " \"What is artificial intelligence?\", # Factual question\n", + " \"How are AI and machine learning related?\", # Relationship query\n", + " \"What are the applications of deep learning in healthcare?\", # Complex multi-hop query\n", + "]\n", + "\n", + "# Process each query\n", + "query_results = {}\n", + "for query in example_queries:\n", + " print(f\"\\n{'='*60}\")\n", + " result = graphrag_system.process_query(query, max_context=10)\n", + " query_results[query] = result\n", + " \n", + " print(f\"Prepared LLM Context ({len(result['llm_context'])} chars):\")\n", + " print(result['llm_context'][:500] + \"...\")\n", + "\n", + "print(f\"\\nProcessed {len(query_results)} queries\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 11: LLM Integration\n", + "\n", + "Integrate with LLM (OpenAI, Anthropic, or local) to generate answers using the retrieved GraphRAG context.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# LLM Integration\n", + "# This section demonstrates how to integrate with LLMs using the retrieved context\n", + "\n", + "def generate_answer_with_llm(query: str, llm_context: str, llm_provider: str = \"openai\"):\n", + " \"\"\"\n", + " Generate answer using LLM with GraphRAG context.\n", + " \n", + " Supports OpenAI, Anthropic, or local LLMs.\n", + " \"\"\"\n", + " # Build prompt\n", + " prompt = f\"\"\"You are an AI assistant with access to a knowledge graph and retrieved context.\n", + "\n", + "Context from Knowledge Graph:\n", + "{llm_context}\n", + "\n", + "Question: {query}\n", + "\n", + "Based on the context provided above, please answer the question. If the context doesn't contain enough information, say so. Cite specific entities or relationships from the context when relevant.\n", + "\n", + "Answer:\"\"\"\n", + " \n", + " # Here you would call your LLM\n", + " # Example with OpenAI (uncomment and configure):\n", + " # try:\n", + " # from openai import OpenAI\n", + " # client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))\n", + " # response = client.chat.completions.create(\n", + " # model=\"gpt-4\",\n", + " # messages=[\n", + " # {\"role\": \"system\", \"content\": \"You are a helpful assistant with access to knowledge graphs.\"},\n", + " # {\"role\": \"user\", \"content\": prompt}\n", + " # ],\n", + " # temperature=0.7\n", + " # )\n", + " # return response.choices[0].message.content\n", + " # except Exception as e:\n", + " # return f\"Error calling LLM: {e}\"\n", + " \n", + " # For demonstration, return the prompt structure\n", + " return f\"[LLM Answer would be generated here using the context above]\"\n", + "\n", + "# Example: Generate answer for a query\n", + "if query_results:\n", + " sample_query = list(query_results.keys())[0]\n", + " sample_result = query_results[sample_query]\n", + " \n", + " print(f\"Generating answer for: '{sample_query}'\")\n", + " answer = generate_answer_with_llm(\n", + " query=sample_query,\n", + " llm_context=sample_result['llm_context']\n", + " )\n", + " \n", + " print(f\"\\nAnswer:\")\n", + " print(answer)\n", + " print(f\"\\nContext Statistics:\")\n", + " print(f\" Context items: {len(sample_result['contexts'])}\")\n", + " print(f\" Expanded context: {len(sample_result['expanded_context'])}\")\n", + " print(f\" Query entities: {len(sample_result['query_entities'])}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 11.2: Source Attribution and Explainability\n", + "\n", + "Show which parts of the knowledge graph contributed to the answer for explainability.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def explain_answer_sources(query_result):\n", + " \"\"\"\n", + " Explain which sources contributed to the answer.\n", + " \"\"\"\n", + " print(f\"Answer Sources and Attribution:\")\n", + " print(f\"Query: {query_result['query']}\")\n", + " print(f\"\\nRetrieved Context Sources:\")\n", + " \n", + " sources = {}\n", + " for ctx in query_result['contexts']:\n", + " source = ctx.source if hasattr(ctx, 'source') else ctx.get('source', 'unknown')\n", + " sources[source] = sources.get(source, 0) + 1\n", + " \n", + " for source, count in sources.items():\n", + " print(f\" {source}: {count} context items\")\n", + " \n", + " print(f\"\\nGraph Entities Involved:\")\n", + " for entity in query_result['query_entities'][:5]:\n", + " entity_text = entity.get('text', str(entity)) if isinstance(entity, dict) else str(entity)\n", + " print(f\" - {entity_text}\")\n", + " \n", + " print(f\"\\nContext Expansion:\")\n", + " print(f\" Original contexts: {len(query_result['contexts'])}\")\n", + " print(f\" Expanded contexts: {len(query_result['expanded_context'])}\")\n", + " print(f\" Expansion ratio: {len(query_result['expanded_context']) / max(len(query_result['contexts']), 1):.2f}x\")\n", + "\n", + "# Explain sources for sample query\n", + "if query_results:\n", + " explain_answer_sources(list(query_results.values())[0])\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 12: Advanced Features\n", + "\n", + "Demonstrate advanced features including reasoning, quality assessment, and visualization.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.reasoning import InferenceEngine, RuleManager\n", + "from semantica.kg_qa import KGQualityAssessor\n", + "\n", + "# Advanced Feature 1: Reasoning with Inference Engine\n", + "print(f\"Advanced Feature: Logical Reasoning\")\n", + "inference_engine = InferenceEngine()\n", + "rule_manager = RuleManager()\n", + "\n", + "# Example: Add inference rules\n", + "# rule_manager.add_rule(\"IF entity A works_for entity B AND entity B located_in entity C THEN entity A located_in entity C\")\n", + "# new_facts = inference_engine.forward_chain(knowledge_graph, rule_manager)\n", + "# print(f\"Inferred {len(new_facts)} new facts\")\n", + "\n", + "print(f\"Reasoning can infer new relationships from existing knowledge\")\n", + "\n", + "# Advanced Feature 2: Quality Assessment\n", + "print(\"\\nAdvanced Feature: Knowledge Graph Quality Assessment\")\n", + "kg_quality_assessor = KGQualityAssessor()\n", + "\n", + "if knowledge_graph.number_of_nodes() > 0:\n", + " quality_metrics = kg_quality_assessor.assess(knowledge_graph)\n", + " print(f\"Quality Assessment:\")\n", + " print(f\" Completeness: {quality_metrics.get('completeness', 0):.2%}\")\n", + " print(f\" Consistency: {quality_metrics.get('consistency', 0):.2%}\")\n", + " print(f\" Connectivity: {quality_metrics.get('connectivity', 0):.2%}\")\n", + "else:\n", + " print(f\"Graph is empty, skipping quality assessment\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 12.2: Visualize Knowledge Graph\n", + "\n", + "Visualize the knowledge graph to understand its structure.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.visualization import KGVisualizer, AnalyticsVisualizer\n", + "\n", + "# Initialize visualizer\n", + "kg_visualizer = KGVisualizer()\n", + "analytics_visualizer = AnalyticsVisualizer()\n", + "\n", + "# Visualize knowledge graph\n", + "if knowledge_graph.number_of_nodes() > 0:\n", + " print(f\"Visualizing knowledge graph...\")\n", + " \n", + " # Create visualization\n", + " # Uncomment to generate visualization\n", + " # visualization = kg_visualizer.visualize(\n", + " # knowledge_graph,\n", + " # output_path=\"graphrag_visualization.html\",\n", + " # layout=\"spring\",\n", + " # show_labels=True\n", + " # )\n", + " # print(f\"Visualization saved to graphrag_visualization.html\")\n", + " \n", + " print(f\"Graph Statistics for Visualization:\")\n", + " print(f\" Nodes: {knowledge_graph.number_of_nodes()}\")\n", + " print(f\" Edges: {knowledge_graph.number_of_edges()}\")\n", + " print(f\" Node types: {len(set(n.get('type', 'Unknown') for _, n in knowledge_graph.nodes(data=True)))}\")\n", + " print(f\" Edge types: {len(set(e.get('type', 'Unknown') for _, _, e in knowledge_graph.edges(data=True)))}\")\n", + " \n", + " # Analytics visualization\n", + " # analytics_viz = analytics_visualizer.visualize(\n", + " # knowledge_graph,\n", + " # metrics=['centrality', 'communities'],\n", + " # output_path=\"graphrag_analytics.html\"\n", + " # )\n", + " # print(f\"Analytics visualization saved\")\n", + "else:\n", + " print(f\"Graph is empty, skipping visualization\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 13: Complete End-to-End Example\n", + "\n", + "Demonstrate a complete end-to-end GraphRAG workflow with real-world data, showing the full pipeline from ingestion to answer generation.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def complete_graphrag_workflow(query: str):\n", + " \"\"\"\n", + " Complete GraphRAG workflow from query to answer.\n", + " \"\"\"\n", + " print(f\"\\n{'='*70}\")\n", + " print(f\"Complete GraphRAG Workflow\")\n", + " print(f\"{'='*70}\")\n", + " print(f\"Query: {query}\\n\")\n", + " \n", + " # Step 1: Process query\n", + " print(\"Step 1: Processing query...\")\n", + " result = graphrag_system.process_query(query, max_context=10)\n", + " \n", + " # Step 2: Generate answer\n", + " print(\"\\nStep 2: Generating answer with LLM...\")\n", + " answer = generate_answer_with_llm(query, result['llm_context'])\n", + " \n", + " # Step 3: Explain sources\n", + " print(\"\\nStep 3: Explaining sources...\")\n", + " explain_answer_sources(result)\n", + " \n", + " # Step 4: Show performance metrics\n", + " print(\"\\nStep 4: Performance Metrics:\")\n", + " print(f\" Context retrieval time: <1s (simulated)\")\n", + " print(f\" Context items retrieved: {len(result['contexts'])}\")\n", + " print(f\" Graph expansion hops: 2\")\n", + " print(f\" Total context size: {len(result['llm_context'])} characters\")\n", + " \n", + " return {\n", + " 'query': query,\n", + " 'answer': answer,\n", + " 'contexts': result['contexts'],\n", + " 'metrics': {\n", + " 'context_items': len(result['contexts']),\n", + " 'expanded_items': len(result['expanded_context']),\n", + " 'query_entities': len(result['query_entities'])\n", + " }\n", + " }\n", + "\n", + "# Run complete workflow example\n", + "if len(all_documents) > 0 or knowledge_graph.number_of_nodes() > 0:\n", + " example_query = \"What are the main concepts and their relationships?\"\n", + " workflow_result = complete_graphrag_workflow(example_query)\n", + " \n", + " print(f\"\\nComplete workflow executed successfully!\")\n", + " print(f\"Final Results:\")\n", + " print(f\" Query processed: βœ“\")\n", + " print(f\" Context retrieved: {workflow_result['metrics']['context_items']} items\")\n", + " print(f\" Answer generated: βœ“\")\n", + "else:\n", + " print(\"Configure data sources above to run complete workflow with real data\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(f\"Comparison: Traditional RAG vs GraphRAG\\n\")\n", + "\n", + "comparison = {\n", + " \"Traditional RAG\": {\n", + " \"Retrieval\": \"Vector similarity only\",\n", + " \"Context\": \"Flat document chunks\",\n", + " \"Relationships\": \"Not captured\",\n", + " \"Multi-hop\": \"Not supported\",\n", + " \"Explainability\": \"Limited (source documents only)\"\n", + " },\n", + " \"GraphRAG\": {\n", + " \"Retrieval\": \"Vector + Graph traversal\",\n", + " \"Context\": \"Structured knowledge graph\",\n", + " \"Relationships\": \"Explicitly modeled\",\n", + " \"Multi-hop\": \"Supported (graph expansion)\",\n", + " \"Explainability\": \"High (entities, relationships, paths)\"\n", + " }\n", + "}\n", + "\n", + "print(\"Feature Comparison:\")\n", + "print(f\"{'Feature':<20} {'Traditional RAG':<25} {'GraphRAG':<25}\")\n", + "print(\"-\" * 70)\n", + "\n", + "for feature in comparison[\"Traditional RAG\"].keys():\n", + " trad = comparison[\"Traditional RAG\"][feature]\n", + " graph = comparison[\"GraphRAG\"][feature]\n", + " print(f\"{feature:<20} {trad:<25} {graph:<25}\")\n", + "\n", + "print(\"\\nGraphRAG Advantages:\")\n", + "print(f\" β€’ Better handling of complex queries requiring relationship understanding\")\n", + "print(f\" β€’ Multi-hop reasoning across entities\")\n", + "print(f\" β€’ More accurate answers through structured knowledge\")\n", + "print(f\" β€’ Better explainability with graph paths\")\n", + "print(f\" β€’ Reduced hallucinations through graph validation\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 14: Export and Persistence\n", + "\n", + "Export the knowledge graph and save the vector store for reuse and sharing.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from semantica.export import JSONExporter, RDFExporter, CSVExporter\n", + "\n", + "json_exporter = JSONExporter()\n", + "rdf_exporter = RDFExporter()\n", + "csv_exporter = CSVExporter()\n", + "\n", + "# Export knowledge graph to JSON\n", + "if knowledge_graph.number_of_nodes() > 0:\n", + " print(f\"Exporting knowledge graph...\")\n", + " \n", + " # Export to JSON\n", + " json_output = json_exporter.export(knowledge_graph, \"graphrag_knowledge_graph.json\")\n", + " print(f\"Exported to JSON: graphrag_knowledge_graph.json\")\n", + " \n", + " # Export to RDF\n", + " rdf_output = rdf_exporter.export(knowledge_graph, \"graphrag_knowledge_graph.rdf\")\n", + " print(f\"Exported to RDF: graphrag_knowledge_graph.rdf\")\n", + " \n", + " # Export entities to CSV\n", + " entities_data = []\n", + " for entity in resolved_entities[:100]: # Limit for demo\n", + " if isinstance(entity, dict):\n", + " entities_data.append({\n", + " 'id': entity.get('id', ''),\n", + " 'text': entity.get('text', entity.get('name', '')),\n", + " 'type': entity.get('type', 'Unknown')\n", + " })\n", + " \n", + " if entities_data:\n", + " csv_output = csv_exporter.export(entities_data, \"graphrag_entities.csv\")\n", + " print(f\"Exported entities to CSV: graphrag_entities.csv\")\n", + " \n", + " print(f\"\\nExport Summary:\")\n", + " print(f\" Nodes exported: {knowledge_graph.number_of_nodes()}\")\n", + " print(f\" Edges exported: {knowledge_graph.number_of_edges()}\")\n", + " print(f\" Entities exported: {len(entities_data)}\")\n", + "else:\n", + " print(f\"Graph is empty, skipping export\")\n", + "\n", + "# Save vector store (if supported)\n", + "print(\"\\nVector Store:\")\n", + "print(f\" Vectors stored: βœ“\")\n", + "print(f\" Metadata stored: βœ“\")\n", + "print(f\" Ready for reuse: βœ“\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Summary and Next Steps\n", + "\n", + "### What We Built\n", + "\n", + "This notebook demonstrated a **complete end-to-end GraphRAG system** using Semantica:\n", + "\n", + "1. **Real-World Data Ingestion**: MCP servers, web scraping, RSS feeds\n", + "2. **Document Processing**: Parsing, splitting, normalization\n", + "3. **Semantic Extraction**: Entities, relationships, triples\n", + "4. **Knowledge Graph Construction**: With quality assurance\n", + "5. **Embedding Generation**: For documents and entities\n", + "6. **Vector Store**: Fast similarity search\n", + "7. **Hybrid Search**: Combining vectors and graphs\n", + "8. **Context Retrieval**: With graph expansion\n", + "9. **GraphRAG Query System**: Complete query processing\n", + "10. **LLM Integration**: Answer generation with context\n", + "11. **Advanced Features**: Reasoning, quality, visualization\n", + "12. **Export**: Persistence and sharing\n", + "\n", + "### Key Takeaways\n", + "\n", + "- **GraphRAG** combines the best of vector search and knowledge graphs\n", + "- **Multi-hop reasoning** enables deeper understanding\n", + "- **Real-world data** makes the system production-ready\n", + "- **Semantica** provides all modules needed for GraphRAG\n", + "\n", + "### Next Steps\n", + "\n", + "1. **Configure Real Data Sources**: Set up MCP servers, web URLs, or RSS feeds\n", + "2. **Customize Extraction**: Adjust entity and relationship extraction for your domain\n", + "3. **Tune Hybrid Search**: Experiment with `hybrid_alpha` for your use case\n", + "4. **Add More LLMs**: Integrate with Anthropic, local models, or other providers\n", + "5. **Scale Up**: Process larger datasets and optimize performance\n", + "6. **Deploy**: Build production GraphRAG applications\n", + "\n", + "### Resources\n", + "\n", + "- [Semantica Documentation](https://semantica.readthedocs.io/)\n", + "- [GraphRAG Concepts](https://semantica.readthedocs.io/concepts/)\n", + "- [API Reference](https://semantica.readthedocs.io/reference/)\n", + "- [More Examples](https://semantica.readthedocs.io/cookbook/)\n", + "\n", + "---\n", + "\n", + "**Congratulations!** You've built a complete GraphRAG system with Semantica!\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 } diff --git a/docs/reference/core.md b/docs/reference/core.md index e69585fb..e0e7e214 100644 --- a/docs/reference/core.md +++ b/docs/reference/core.md @@ -8,11 +8,11 @@
-- :material-cogs:{ .lg .middle } **Orchestrator** +- :material-cogs:{ .lg .middle } **Semantica** --- - Central coordinator for all framework components and workflows + Main framework class coordinating all components and workflows - :material-lifecycle:{ .lg .middle } **Lifecycle Management** @@ -32,11 +32,11 @@ Extensible plugin registry for adding custom modules and capabilities -- :material-console:{ .lg .middle } **Logging & Telemetry** +- :material-console:{ .lg .middle } **Method Registry** --- - Centralized logging and metrics collection + Registry for custom orchestration methods and extensibility
@@ -51,69 +51,318 @@ ## βš™οΈ Algorithms Used ### Lifecycle Management -- **State Machine**: `CREATED` -> `INITIALIZED` -> `RUNNING` -> `STOPPED` -- **Dependency Injection**: Resolving and injecting dependencies between modules. -- **Graceful Shutdown**: Ensuring all resources (DB connections, thread pools) are closed properly. +- **State Machine**: `UNINITIALIZED` -> `INITIALIZING` -> `READY` -> `RUNNING` -> `STOPPING` -> `STOPPED` +- **Priority-based Hooks**: Startup and shutdown hooks executed in priority order (lower = earlier) +- **Graceful Shutdown**: Ensuring all resources (DB connections, thread pools) are closed properly ### Configuration -- **Layered Loading**: Defaults -> Config File -> Environment Variables -> CLI Arguments (Priority order). -- **Schema Validation**: Validating config structure against defined schemas. +- **Layered Loading**: Defaults -> Config File -> Environment Variables (Priority order) +- **Schema Validation**: Validating config structure against defined schemas +- **Nested Access**: Dot notation for accessing nested configuration values ### Plugin System -- **Discovery**: Auto-discovery of plugins via entry points or directory scanning. -- **Registration**: Dynamic registration of classes and functions. -- **Hook Execution**: Running plugin hooks at specific lifecycle events. +- **Discovery**: Auto-discovery of plugins via directory scanning +- **Registration**: Dynamic registration of classes and functions +- **Dependency Resolution**: Automatic loading of plugin dependencies --- ## Main Classes -### Orchestrator +### Semantica -The brain of the framework. +The main framework class that coordinates all components. **Methods:** | Method | Description | |--------|-------------| -| `start()` | Initialize and start all components | -| `stop()` | Graceful shutdown | -| `get_component(name)` | Access initialized module | +| `__init__(config=None, **kwargs)` | Initialize framework with optional configuration | +| `initialize()` | Initialize all framework components | +| `build_knowledge_base(sources, **kwargs)` | Build knowledge base from data sources | +| `run_pipeline(pipeline, data)` | Execute a processing pipeline | +| `get_status()` | Get system health and status | +| `shutdown(graceful=True)` | Shutdown the framework gracefully | **Example:** ```python -from semantica.core import Orchestrator +from semantica.core import Semantica -app = Orchestrator() -app.start() +# Initialize framework +framework = Semantica() +framework.initialize() -# Access modules -kg = app.get_component("knowledge_graph") -ingest = app.get_component("ingest") +# Build knowledge base +result = framework.build_knowledge_base( + sources=["doc1.pdf", "doc2.docx"], + embeddings=True, + graph=True +) + +# Check status +status = framework.get_status() +print(f"System state: {status['state']}") + +# Shutdown +framework.shutdown() ``` ### ConfigManager -Manages global configuration. +Manages global configuration loading, validation, and merging. **Methods:** | Method | Description | |--------|-------------| -| `load(path)` | Load config from file | -| `get(key, default)` | Get config value | +| `load_from_file(file_path, validate=True)` | Load config from YAML or JSON file | +| `load_from_dict(config_dict, validate=True)` | Load config from dictionary | +| `merge_configs(*configs, validate=True)` | Merge multiple configurations | +| `get_config()` | Get current configuration | +| `set_config(config, validate=True)` | Set current configuration | +| `reload(file_path=None)` | Reload configuration from file | + +**Example:** + +```python +from semantica.core import ConfigManager + +manager = ConfigManager() +config = manager.load_from_file("config.yaml") + +# Merge configurations +config1 = manager.load_from_file("base_config.yaml") +config2 = manager.load_from_file("override_config.yaml") +merged = manager.merge_configs(config1, config2) +``` + +### Config + +Configuration data class with validation and nested access. + +**Methods:** + +| Method | Description | +|--------|-------------| +| `get(key_path, default=None)` | Get nested configuration value by key path | +| `set(key_path, value)` | Set nested configuration value | +| `update(updates, merge=True)` | Update configuration with new values | +| `validate()` | Validate configuration settings | +| `to_dict()` | Convert configuration to dictionary | + +**Example:** + +```python +from semantica.core import Config, ConfigManager + +manager = ConfigManager() +config = manager.load_from_dict({"processing": {"batch_size": 32}}) + +# Access nested values +batch_size = config.get("processing.batch_size", default=16) + +# Update values +config.set("processing.batch_size", 64) +config.update({"quality": {"min_confidence": 0.9}}) + +# Validate +config.validate() +``` + +### LifecycleManager + +System lifecycle management with hooks and health monitoring. + +**Methods:** + +| Method | Description | +|--------|-------------| +| `startup()` | Execute startup sequence with registered hooks | +| `shutdown(graceful=True)` | Execute shutdown sequence | +| `register_startup_hook(hook_fn, priority=50)` | Register a startup hook | +| `register_shutdown_hook(hook_fn, priority=50)` | Register a shutdown hook | +| `register_component(name, component)` | Register component for health monitoring | +| `health_check()` | Perform comprehensive system health check | +| `get_health_summary()` | Get summary of system health | +| `get_state()` | Get current system state | +| `is_ready()` | Check if system is ready | +| `is_running()` | Check if system is running | + +**Example:** + +```python +from semantica.core import LifecycleManager + +manager = LifecycleManager() + +# Register hooks +def init_db(): + print("Initializing database...") + +manager.register_startup_hook(init_db, priority=10) +manager.startup() + +# Register component for health monitoring +class DatabaseComponent: + def health_check(self): + return {"healthy": True, "message": "Connected"} + +db = DatabaseComponent() +manager.register_component("database", db) + +# Check health +health = manager.health_check() +summary = manager.get_health_summary() + +manager.shutdown(graceful=True) +``` ### PluginRegistry -Manages extensions. +Plugin registry and management system for dynamic plugin discovery and loading. **Methods:** | Method | Description | |--------|-------------| -| `register(plugin)` | Register new plugin | -| `get_plugin(name)` | Retrieve plugin | +| `__init__(plugin_paths=None)` | Initialize with optional plugin paths for auto-discovery | +| `register_plugin(plugin_name, plugin_class, version="1.0.0", **metadata)` | Manually register a plugin | +| `load_plugin(plugin_name, **config)` | Load and initialize a plugin | +| `unload_plugin(plugin_name)` | Unload a plugin | +| `list_plugins()` | List all available plugins | +| `get_plugin_info(plugin_name)` | Get information about a plugin | +| `is_plugin_loaded(plugin_name)` | Check if a plugin is loaded | +| `get_loaded_plugin(plugin_name)` | Get loaded plugin instance | + +**Example:** + +```python +from semantica.core import PluginRegistry + +# Auto-discover plugins +registry = PluginRegistry(plugin_paths=["./plugins"]) + +# Load plugin with configuration +plugin = registry.load_plugin("my_plugin", api_key="xxx") + +# List all plugins +plugins = registry.list_plugins() +for plugin_info in plugins: + print(f"{plugin_info['name']}: {plugin_info['version']}") + +# Get plugin info +info = registry.get_plugin_info("my_plugin") +``` + +### MethodRegistry + +Registry for custom orchestration methods. + +**Methods:** + +| Method | Description | +|--------|-------------| +| `register(task, name, method_func)` | Register a custom orchestration method | +| `get(task, name)` | Get method by task and name | +| `list_all(task=None)` | List all registered methods | +| `unregister(task, name)` | Unregister a method | +| `clear(task=None)` | Clear all registered methods | + +**Example:** + +```python +from semantica.core import method_registry + +def custom_kb_builder(sources, **kwargs): + # Custom logic + return {"knowledge_graph": {}} + +method_registry.register("knowledge_base", "custom", custom_kb_builder) + +# Use custom method +method = method_registry.get("knowledge_base", "custom") +result = method(sources=["doc.pdf"]) +``` + +--- + +## Orchestration Methods + +Convenience functions for common orchestration tasks. + +### build_knowledge_base() + +Build knowledge base from data sources. + +```python +from semantica.core.methods import build_knowledge_base + +result = build_knowledge_base( + sources=["doc1.pdf", "doc2.docx"], + method="default", + embeddings=True, + graph=True +) +``` + +### run_pipeline() + +Execute a processing pipeline. + +```python +from semantica.core.methods import run_pipeline + +result = run_pipeline( + pipeline={"steps": ["parse", "extract"]}, + data="sample text", + method="default" +) +``` + +### initialize_framework() + +Initialize Semantica framework. + +```python +from semantica.core.methods import initialize_framework + +framework = initialize_framework( + config={"llm_provider": {"name": "openai"}}, + method="default" +) +``` + +### get_status() + +Get system status. + +```python +from semantica.core.methods import get_status + +status = get_status(framework=my_framework, method="detailed") +``` + +### get_orchestration_method() + +Get orchestration method by task and name. + +```python +from semantica.core.methods import get_orchestration_method + +method = get_orchestration_method("knowledge_base", "custom") +``` + +### list_available_methods() + +List all available orchestration methods. + +```python +from semantica.core.methods import list_available_methods + +all_methods = list_available_methods() +kb_methods = list_available_methods("knowledge_base") +``` --- @@ -121,68 +370,187 @@ Manages extensions. ### Environment Variables +Configuration can be loaded from environment variables with `SEMANTICA_` prefix: + ```bash -export SEMANTICA_ENV=production -export SEMANTICA_LOG_LEVEL=INFO -export SEMANTICA_CONFIG_PATH=./config.yaml +export SEMANTICA_PROCESSING_BATCH_SIZE=64 +export SEMANTICA_LLM_PROVIDER_MODEL=gpt-4 +export SEMANTICA_QUALITY_MIN_CONFIDENCE=0.8 ``` ### YAML Configuration ```yaml -core: - environment: production - log_level: INFO - plugins: +llm_provider: + name: openai + model: gpt-4 + api_key: ${OPENAI_API_KEY} + +embedding_model: + name: openai + model: text-embedding-ada-002 + +processing: + batch_size: 32 + max_workers: 4 + +quality: + min_confidence: 0.7 + +logging: + level: INFO + +plugins: + my_plugin: enabled: true - directory: ./plugins + config_key: config_value +``` + +### JSON Configuration + +```json +{ + "llm_provider": { + "name": "openai", + "model": "gpt-4" + }, + "processing": { + "batch_size": 32 + } +} ``` --- ## Integration Examples -### Custom Application +### Basic Usage ```python -from semantica.core import Orchestrator, ConfigManager +from semantica.core import Semantica, ConfigManager -# 1. Load Config -config = ConfigManager() -config.load("config.yaml") +# 1. Load configuration +config_manager = ConfigManager() +config = config_manager.load_from_file("config.yaml") -# 2. Initialize Orchestrator -app = Orchestrator(config=config) +# 2. Initialize framework +framework = Semantica(config=config) +framework.initialize() -# 3. Register Custom Plugin -class MyPlugin: - name = "my_plugin" - def initialize(self): - print("My Plugin Started") - -app.plugin_registry.register(MyPlugin()) - -# 4. Start -app.start() - -# 5. Run Workload try: - app.run_pipeline("my_pipeline") + # 3. Build knowledge base + result = framework.build_knowledge_base( + sources=["doc1.pdf", "doc2.docx"], + embeddings=True, + graph=True + ) + + # 4. Check status + status = framework.get_status() + print(f"System state: {status['state']}") + finally: - app.stop() + # 5. Shutdown gracefully + framework.shutdown(graceful=True) +``` + +### Custom Plugin + +```python +from semantica.core import PluginRegistry + +class MyPlugin: + def initialize(self): + print("Plugin initialized") + + def execute(self, data): + return {"processed": True} + +registry = PluginRegistry() +registry.register_plugin( + plugin_name="my_plugin", + plugin_class=MyPlugin, + version="1.0.0" +) + +plugin = registry.load_plugin("my_plugin") +result = plugin.execute("sample data") +``` + +### Lifecycle Hooks + +```python +from semantica.core import LifecycleManager + +manager = LifecycleManager() + +def init_database(): + print("Initializing database...") + +def cleanup_database(): + print("Cleaning up database...") + +manager.register_startup_hook(init_database, priority=10) +manager.register_shutdown_hook(cleanup_database, priority=10) + +manager.startup() +# ... do work ... +manager.shutdown(graceful=True) +``` + +### Custom Orchestration Method + +```python +from semantica.core import method_registry, Semantica + +def fast_kb_builder(sources, **kwargs): + framework = Semantica() + framework.initialize() + try: + return framework.build_knowledge_base( + sources=sources, + embeddings=False, # Skip for speed + graph=True, + **kwargs + ) + finally: + framework.shutdown() + +method_registry.register("knowledge_base", "fast", fast_kb_builder) + +# Use custom method +from semantica.core.methods import build_knowledge_base +result = build_knowledge_base(sources=["doc.pdf"], method="fast") ``` --- ## Best Practices -1. **Use Orchestrator**: Avoid manually instantiating every module; let the Orchestrator handle dependencies. -2. **Graceful Shutdown**: Always ensure `app.stop()` is called (e.g., in a `finally` block) to prevent resource leaks. -3. **Config Layers**: Use `config.yaml` for defaults and Environment Variables for secrets/overrides. +1. **Always Initialize**: Always call `initialize()` after creating a `Semantica` instance before using it. + +2. **Graceful Shutdown**: Always call `shutdown(graceful=True)` in a `finally` block to ensure proper cleanup. + +3. **Configuration Management**: Use `ConfigManager` for loading and managing configurations. Prefer YAML files for complex configurations. + +4. **Error Handling**: Wrap framework operations in try-except blocks to handle `ConfigurationError` and `ProcessingError` appropriately. + +5. **Health Monitoring**: Register components with `LifecycleManager` for health monitoring and use `health_check()` regularly. + +6. **Plugin Development**: Follow the plugin interface (must have `initialize()` and `execute()` methods) when creating custom plugins. + +7. **Method Registration**: Use `MethodRegistry` for extensibility. Register custom methods for knowledge base building, pipeline execution, etc. + +8. **Hook Priorities**: Use appropriate priorities for lifecycle hooks. Lower numbers execute first. + +9. **Configuration Validation**: Always validate configurations using `config.validate()` before using them. + +10. **Resource Cleanup**: Ensure all resources are properly cleaned up in shutdown hooks. --- ## See Also -- [Pipeline Module](pipeline.md) - Executed by the Orchestrator +- [Core Usage Guide](../core/core_usage.md) - Comprehensive usage guide with detailed examples +- [Pipeline Module](pipeline.md) - Executed by the Semantica framework - [Utils Module](utils.md) - Shared utilities used by Core diff --git a/semantica/core/__init__.py b/semantica/core/__init__.py index b4a63210..85315b3b 100644 --- a/semantica/core/__init__.py +++ b/semantica/core/__init__.py @@ -86,9 +86,6 @@ __all__ = [ "get_status", "get_orchestration_method", "list_available_methods", -<<<<<<< HEAD -] -======= # Convenience "build", ] @@ -158,4 +155,3 @@ def build( pipeline=pipeline_config, **{k: v for k, v in options.items() if k not in ["pipeline", "method"]}, ) ->>>>>>> origin/main diff --git a/semantica/core/config_manager.py b/semantica/core/config_manager.py index 069dfb43..8ed70287 100644 --- a/semantica/core/config_manager.py +++ b/semantica/core/config_manager.py @@ -96,33 +96,68 @@ class Config: config_dict: Dictionary of configuration values **kwargs: Additional configuration parameters """ + # Build configuration dictionary from all sources + config_data = self._build_config_dict(config_dict, kwargs) + + # Load from environment variables (overrides file/kwargs) + self._load_from_env(config_data) + + # Initialize configuration sections + self._initialize_sections(config_data) + + def _build_config_dict( + self, config_dict: Optional[Dict[str, Any]], kwargs: Dict[str, Any] + ) -> Dict[str, Any]: + """ + Build configuration dictionary from multiple sources. + + Priority order: defaults -> config_dict -> kwargs + + Args: + config_dict: Optional configuration dictionary + kwargs: Additional configuration parameters + + Returns: + Merged configuration dictionary + """ # Start with defaults - default_dict = DEFAULT_CONFIG.copy() + result = DEFAULT_CONFIG.copy() # Merge with provided config_dict if config_dict: - default_dict = merge_dicts(default_dict, config_dict, deep=True) + result = merge_dicts(result, config_dict, deep=True) # Merge with kwargs if kwargs: - default_dict = merge_dicts(default_dict, kwargs, deep=True) + result = merge_dicts(result, kwargs, deep=True) - # Load from environment variables - self._load_from_env(default_dict) + return result - # Initialize dataclass fields - self.llm_provider = default_dict.get("llm_provider", {}) - self.embedding_model = default_dict.get("embedding_model", {}) - self.vector_store = default_dict.get("vector_store", {}) - self.graph_db = default_dict.get("graph_db", {}) - self.processing = default_dict.get( + def _initialize_sections(self, config_data: Dict[str, Any]) -> None: + """ + Initialize configuration section attributes. + + Args: + config_data: Configuration dictionary + """ + self.llm_provider = config_data.get("llm_provider", {}) + self.embedding_model = config_data.get("embedding_model", {}) + self.vector_store = config_data.get("vector_store", {}) + self.graph_db = config_data.get("graph_db", {}) + self.processing = config_data.get( "processing", DEFAULT_CONFIG.get("processing", {}) ) - self.pipeline = default_dict.get("pipeline", {}) - self.logging = default_dict.get("logging", DEFAULT_CONFIG.get("logging", {})) - self.quality = default_dict.get("quality", DEFAULT_CONFIG.get("quality", {})) - self.security = default_dict.get("security", DEFAULT_CONFIG.get("security", {})) - self.custom = default_dict.get("custom", {}) + self.pipeline = config_data.get("pipeline", {}) + self.logging = config_data.get( + "logging", DEFAULT_CONFIG.get("logging", {}) + ) + self.quality = config_data.get( + "quality", DEFAULT_CONFIG.get("quality", {}) + ) + self.security = config_data.get( + "security", DEFAULT_CONFIG.get("security", {}) + ) + self.custom = config_data.get("custom", {}) def _load_from_env(self, config_dict: Dict[str, Any]) -> None: """ @@ -140,27 +175,35 @@ class Config: config_dict: Configuration dictionary to update with env values """ prefix = "SEMANTICA_" - prefix_length = len(prefix) - + for env_key, env_value in os.environ.items(): if not env_key.startswith(prefix): continue - # Remove prefix and convert to lowercase for consistency - config_key = env_key[prefix_length:].lower() - - # Parse the environment variable value - # Try JSON first (for complex types like lists/dicts) - try: - parsed_value = json.loads(env_value) - except (json.JSONDecodeError, ValueError): - # Not valid JSON, try type conversion - parsed_value = self._parse_env_value(env_value) - + # Extract and normalize key + config_key = self._normalize_env_key(env_key, prefix) + + # Parse value (try JSON first, then type conversion) + parsed_value = self._parse_env_value(env_value) + # Set nested value using dot notation - # e.g., "processing_batch_size" -> "processing.batch_size" - normalized_key = config_key.replace("_", ".") - set_nested_value(config_dict, normalized_key, parsed_value) + set_nested_value(config_dict, config_key, parsed_value) + + def _normalize_env_key(self, env_key: str, prefix: str) -> str: + """ + Normalize environment variable key to configuration key path. + + Args: + env_key: Environment variable key (e.g., "SEMANTICA_PROCESSING_BATCH_SIZE") + prefix: Prefix to remove (e.g., "SEMANTICA_") + + Returns: + Normalized key path (e.g., "processing.batch_size") + """ + # Remove prefix and convert to lowercase + key = env_key[len(prefix):].lower() + # Convert underscores to dots for nested access + return key.replace("_", ".") def _parse_env_value(self, value: str) -> Union[str, int, float, bool]: """ @@ -202,59 +245,11 @@ class Config: ConfigurationError: If configuration is invalid with detailed error messages """ validation_errors = [] - - # Validate processing settings - processing_config = self.processing - if processing_config: - # Validate batch_size - if "batch_size" in processing_config: - batch_size = processing_config["batch_size"] - if not isinstance(batch_size, int): - validation_errors.append( - f"processing.batch_size must be an integer, got {type(batch_size).__name__}" - ) - elif batch_size <= 0: - validation_errors.append( - f"processing.batch_size must be positive, got {batch_size}" - ) - - # Validate max_workers - if "max_workers" in processing_config: - max_workers = processing_config["max_workers"] - if not isinstance(max_workers, int): - validation_errors.append( - f"processing.max_workers must be an integer, got {type(max_workers).__name__}" - ) - elif max_workers <= 0: - validation_errors.append( - f"processing.max_workers must be positive, got {max_workers}" - ) - - # Validate quality settings - quality_config = self.quality - if quality_config: - # Validate min_confidence - if "min_confidence" in quality_config: - confidence = quality_config["min_confidence"] - if not isinstance(confidence, (int, float)): - validation_errors.append( - f"quality.min_confidence must be a number, got {type(confidence).__name__}" - ) - elif not (0.0 <= confidence <= 1.0): - validation_errors.append( - f"quality.min_confidence must be between 0.0 and 1.0, got {confidence}" - ) - - # Validate logging settings - logging_config = self.logging - if logging_config: - valid_levels = ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"] - if "level" in logging_config: - level = logging_config["level"] - if level not in valid_levels: - validation_errors.append( - f"logging.level must be one of {valid_levels}, got {level}" - ) + + # Validate each configuration section + validation_errors.extend(self._validate_processing()) + validation_errors.extend(self._validate_quality()) + validation_errors.extend(self._validate_logging()) # Raise error if any validation failures if validation_errors: @@ -266,6 +261,74 @@ class Config: config_context=self.to_dict(), ) + def _validate_processing(self) -> List[str]: + """Validate processing configuration section.""" + errors = [] + if not self.processing: + return errors + + # Validate batch_size + if "batch_size" in self.processing: + batch_size = self.processing["batch_size"] + if not isinstance(batch_size, int): + errors.append( + f"processing.batch_size must be an integer, got {type(batch_size).__name__}" + ) + elif batch_size <= 0: + errors.append( + f"processing.batch_size must be positive, got {batch_size}" + ) + + # Validate max_workers + if "max_workers" in self.processing: + max_workers = self.processing["max_workers"] + if not isinstance(max_workers, int): + errors.append( + f"processing.max_workers must be an integer, got {type(max_workers).__name__}" + ) + elif max_workers <= 0: + errors.append( + f"processing.max_workers must be positive, got {max_workers}" + ) + + return errors + + def _validate_quality(self) -> List[str]: + """Validate quality configuration section.""" + errors = [] + if not self.quality: + return errors + + # Validate min_confidence + if "min_confidence" in self.quality: + confidence = self.quality["min_confidence"] + if not isinstance(confidence, (int, float)): + errors.append( + f"quality.min_confidence must be a number, got {type(confidence).__name__}" + ) + elif not (0.0 <= confidence <= 1.0): + errors.append( + f"quality.min_confidence must be between 0.0 and 1.0, got {confidence}" + ) + + return errors + + def _validate_logging(self) -> List[str]: + """Validate logging configuration section.""" + errors = [] + if not self.logging: + return errors + + valid_levels = ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"] + if "level" in self.logging: + level = self.logging["level"] + if level not in valid_levels: + errors.append( + f"logging.level must be one of {valid_levels}, got {level}" + ) + + return errors + def to_dict(self) -> Dict[str, Any]: """ Convert configuration to dictionary. @@ -308,12 +371,12 @@ class Config: key_path: Dot-separated key path (e.g., "processing.batch_size") value: Value to set """ + # Update the dictionary representation config_dict = self.to_dict() set_nested_value(config_dict, key_path, value) - # Reinitialize from updated dict - updated = Config(config_dict=config_dict) - self.__dict__.update(updated.__dict__) + # Reinitialize sections from updated dict + self._initialize_sections(config_dict) def update(self, updates: Dict[str, Any], merge: bool = True) -> None: """ @@ -325,14 +388,14 @@ class Config: """ current_dict = self.to_dict() + # Merge or replace based on merge flag if merge: updated_dict = merge_dicts(current_dict, updates, deep=True) else: updated_dict = {**current_dict, **updates} - # Reinitialize from updated dict - updated = Config(config_dict=updated_dict) - self.__dict__.update(updated.__dict__) + # Reinitialize sections from updated dict + self._initialize_sections(updated_dict) class ConfigManager: @@ -396,64 +459,105 @@ class ConfigManager: try: file_path = Path(file_path) + self._validate_file_exists(file_path) - if not file_path.exists(): - raise ConfigurationError( - f"Configuration file not found: {file_path}", - config_context={"file_path": str(file_path)}, - ) + # Load configuration dictionary from file + config_dict = self._load_file_content(file_path) - # Detect format from extension - suffix = file_path.suffix.lower() + # Create and validate config object + config = Config(config_dict=config_dict) + if validate: + config.validate() - try: - if suffix in (".yaml", ".yml"): - with open(file_path, "r", encoding="utf-8") as f: - config_dict = yaml.safe_load(f) + # Store config and file path for potential reload + self._config = config + self._last_file_path = file_path - elif suffix == ".json": - config_dict = read_json_file(file_path) - else: - raise ConfigurationError( - f"Unsupported configuration file format: {suffix}. " - "Supported formats: .yaml, .yml, .json" - ) - - # Create config object from loaded dictionary - config = Config(config_dict=config_dict) - - # Validate configuration if requested - if validate: - config.validate() - - # Store config and file path for potential reload - self._config = config - self._last_file_path = file_path - - self.progress_tracker.stop_tracking( - tracking_id, - status="completed", - message="Configuration loaded successfully", - ) - return config - except Exception as e: - # Re-raise as ConfigurationError if inner try fails - raise ConfigurationError( - f"Failed to parse configuration file: {str(e)}", - config_context={"file_path": str(file_path)}, - ) from e + self.progress_tracker.stop_tracking( + tracking_id, + status="completed", + message="Configuration loaded successfully", + ) + return config + except ConfigurationError: + # Re-raise configuration errors as-is + self.progress_tracker.stop_tracking( + tracking_id, status="failed", message="Configuration error" + ) + raise except Exception as e: + # Wrap other exceptions self.progress_tracker.stop_tracking( tracking_id, status="failed", message=str(e) ) - if isinstance(e, ConfigurationError): - raise raise ConfigurationError( f"Failed to load configuration file: {str(e)}", config_context={"file_path": str(file_path)}, + ) from e + + def _validate_file_exists(self, file_path: Path) -> None: + """ + Validate that configuration file exists. + + Args: + file_path: Path to configuration file + + Raises: + ConfigurationError: If file does not exist + """ + if not file_path.exists(): + raise ConfigurationError( + f"Configuration file not found: {file_path}", + config_context={"file_path": str(file_path)}, ) + def _load_file_content(self, file_path: Path) -> Dict[str, Any]: + """ + Load configuration dictionary from file. + + Args: + file_path: Path to configuration file + + Returns: + Configuration dictionary + + Raises: + ConfigurationError: If file format is unsupported or parsing fails + """ + suffix = file_path.suffix.lower() + + if suffix in (".yaml", ".yml"): + return self._load_yaml_file(file_path) + elif suffix == ".json": + return self._load_json_file(file_path) + else: + raise ConfigurationError( + f"Unsupported configuration file format: {suffix}. " + "Supported formats: .yaml, .yml, .json" + ) + + def _load_yaml_file(self, file_path: Path) -> Dict[str, Any]: + """Load YAML configuration file.""" + try: + with open(file_path, "r", encoding="utf-8") as f: + return yaml.safe_load(f) or {} + except Exception as e: + raise ConfigurationError( + f"Failed to parse YAML file: {str(e)}", + config_context={"file_path": str(file_path)}, + ) from e + + def _load_json_file(self, file_path: Path) -> Dict[str, Any]: + """Load JSON configuration file.""" + try: + return read_json_file(file_path) + except Exception as e: + raise ConfigurationError( + f"Failed to parse JSON file: {str(e)}", + config_context={"file_path": str(file_path)}, + ) from e + def load_from_dict( self, config_dict: Dict[str, Any], validate: bool = True ) -> Config: diff --git a/semantica/core/core_usage.md b/semantica/core/core_usage.md new file mode 100644 index 00000000..020bc5a7 --- /dev/null +++ b/semantica/core/core_usage.md @@ -0,0 +1,1338 @@ +# Core Module Usage Guide + +This comprehensive guide demonstrates how to use the core orchestration module for framework initialization, knowledge base construction, pipeline execution, configuration management, lifecycle management, and plugin system integration. + +## Table of Contents + +1. [Basic Usage](#basic-usage) +2. [Semantica Class](#semantica-class) +3. [ConfigManager](#configmanager) +4. [Config Class](#config-class) +5. [LifecycleManager](#lifecyclemanager) +6. [PluginRegistry](#pluginregistry) +7. [MethodRegistry](#methodregistry) +8. [Orchestration Methods](#orchestration-methods) +9. [Convenience Functions](#convenience-functions) +10. [Configuration](#configuration) +11. [Advanced Examples](#advanced-examples) +12. [Best Practices](#best-practices) + +## Basic Usage + +### Quick Start + +```python +from semantica.core import Semantica + +# Initialize framework +framework = Semantica() +framework.initialize() + +# Build knowledge base from documents +result = framework.build_knowledge_base( + sources=["doc1.pdf", "doc2.docx"], + embeddings=True, + graph=True +) + +# Check system status +status = framework.get_status() +print(f"System state: {status['state']}") + +# Shutdown gracefully +framework.shutdown() +``` + +### Using Convenience Functions + +```python +from semantica.core import build +from semantica.core.methods import build_knowledge_base + +# Using module-level convenience function +result = build( + sources=["doc1.pdf"], + extract_entities=True, + extract_relations=True, + embeddings=True, + graph=True +) + +# Using methods directly +result = build_knowledge_base( + sources=["doc1.pdf", "doc2.docx"], + method="default", + embeddings=True, + graph=True +) +``` + +## Semantica Class + +The main framework class that coordinates all components and provides a unified API. + +### Initialization + +```python +from semantica.core import Semantica, Config + +# Initialize with default configuration +framework = Semantica() + +# Initialize with configuration dictionary +config_dict = { + "llm_provider": {"name": "openai", "model": "gpt-4"}, + "processing": {"batch_size": 32} +} +framework = Semantica(config=config_dict) + +# Initialize with Config object +from semantica.core import ConfigManager +config_manager = ConfigManager() +config = config_manager.load_from_dict(config_dict) +framework = Semantica(config=config) +``` + +### Methods + +#### `initialize()` + +Initialize all framework components, load plugins, and prepare the system for processing. + +```python +framework = Semantica() +framework.initialize() + +# Check if initialization was successful +status = framework.get_status() +if status['state'] == 'ready': + print("Framework initialized successfully") +``` + +**Raises:** +- `ConfigurationError`: If configuration is invalid +- `SemanticaError`: If initialization fails + +#### `build_knowledge_base()` + +Build knowledge base from various data sources. + +```python +# Basic usage +result = framework.build_knowledge_base( + sources=["doc1.pdf", "doc2.docx"] +) + +# With options +result = framework.build_knowledge_base( + sources=["doc1.pdf", "https://example.com/doc.html"], + embeddings=True, + graph=True, + pipeline={"extract": {"entities": True, "relations": True}}, + fail_fast=False +) + +# Access results +kg = result["knowledge_graph"] +embeddings = result["embeddings"] +stats = result["statistics"] +print(f"Processed {stats['sources_processed']} sources") +``` + +**Parameters:** +- `sources`: List of data sources (files, URLs, streams) +- `embeddings`: Whether to generate embeddings (default: True) +- `graph`: Whether to build knowledge graph (default: True) +- `pipeline`: Custom pipeline configuration dictionary +- `fail_fast`: Whether to stop on first error (default: False) + +**Returns:** +Dictionary containing: +- `knowledge_graph`: Knowledge graph data +- `embeddings`: Embedding vectors +- `results`: Processing results +- `statistics`: Processing statistics +- `metadata`: Processing metadata + +**Raises:** +- `ProcessingError`: If processing fails + +#### `run_pipeline()` + +Execute a processing pipeline. + +```python +# Using pipeline configuration dictionary +pipeline_config = { + "steps": ["parse", "extract", "normalize"] +} + +result = framework.run_pipeline( + pipeline=pipeline_config, + data="sample text data" +) + +# Access results +output = result["output"] +metrics = result["metrics"] +print(f"Execution time: {metrics.get('execution_time', 0)}s") +``` + +**Parameters:** +- `pipeline`: Pipeline object or configuration dictionary +- `data`: Input data for pipeline + +**Returns:** +Dictionary containing: +- `success`: Whether execution succeeded +- `output`: Pipeline output data +- `metrics`: Performance metrics +- `metadata`: Processing metadata + +**Raises:** +- `ProcessingError`: If pipeline execution fails + +#### `get_status()` + +Get system health and status. + +```python +status = framework.get_status() + +# Access status information +print(f"System state: {status['state']}") +print(f"Healthy components: {status['health']['healthy_components']}") + +# Check module status +for name, module_status in status['modules'].items(): + print(f"{name}: {module_status['status']}") + +# Check plugin status +for name, plugin_status in status['plugins'].items(): + print(f"{name}: loaded={plugin_status['loaded']}") +``` + +**Returns:** +Dictionary containing: +- `state`: System state (uninitialized, ready, running, stopped, error) +- `health`: Health summary with component statuses +- `modules`: Module status information +- `plugins`: Plugin status information +- `config`: Configuration status + +#### `shutdown()` + +Shutdown the framework gracefully. + +```python +# Graceful shutdown (default) +framework.shutdown(graceful=True) + +# Force shutdown +framework.shutdown(graceful=False) +``` + +**Parameters:** +- `graceful`: Whether to shutdown gracefully (default: True) + +## ConfigManager + +Configuration management system for loading, validating, and merging configurations. + +### Methods + +#### `load_from_file()` + +Load configuration from YAML or JSON file. + +```python +from semantica.core import ConfigManager + +manager = ConfigManager() + +# Load from YAML file +config = manager.load_from_file("config.yaml") + +# Load from JSON file +config = manager.load_from_file("config.json") + +# Load without validation +config = manager.load_from_file("config.yaml", validate=False) +``` + +**Parameters:** +- `file_path`: Path to configuration file (YAML or JSON) +- `validate`: Whether to validate configuration after loading (default: True) + +**Returns:** +- `Config`: Loaded configuration object + +**Raises:** +- `ConfigurationError`: If file cannot be loaded or is invalid + +#### `load_from_dict()` + +Load configuration from dictionary. + +```python +config_dict = { + "llm_provider": {"name": "openai"}, + "processing": {"batch_size": 64} +} + +config = manager.load_from_dict(config_dict) +``` + +**Parameters:** +- `config_dict`: Dictionary of configuration values +- `validate`: Whether to validate configuration after loading (default: True) + +**Returns:** +- `Config`: Configuration object + +**Raises:** +- `ConfigurationError`: If configuration is invalid + +#### `merge_configs()` + +Merge multiple configurations. + +```python +config1 = manager.load_from_file("base_config.yaml") +config2 = manager.load_from_file("override_config.yaml") +config3 = manager.load_from_dict({"processing": {"batch_size": 128}}) + +# Later configurations take priority +merged = manager.merge_configs(config1, config2, config3) +``` + +**Parameters:** +- `*configs`: Configuration objects to merge +- `validate`: Whether to validate merged configuration (default: True) + +**Returns:** +- `Config`: Merged configuration + +**Raises:** +- `ConfigurationError`: If no configurations provided or merged config is invalid + +#### `get_config()` + +Get current configuration. + +```python +current_config = manager.get_config() +if current_config: + batch_size = current_config.get("processing.batch_size") +``` + +**Returns:** +- `Optional[Config]`: Current configuration or None if not loaded + +#### `set_config()` + +Set current configuration. + +```python +config = manager.load_from_file("config.yaml") +manager.set_config(config, validate=True) +``` + +**Parameters:** +- `config`: Configuration object to set +- `validate`: Whether to validate configuration (default: True) + +#### `reload()` + +Reload configuration from file. + +```python +# Reload from last loaded file +config = manager.reload() + +# Reload from specific file +config = manager.reload("config.yaml") +``` + +**Parameters:** +- `file_path`: Optional path to configuration file (uses last loaded file if None) + +**Returns:** +- `Config`: Reloaded configuration + +## Config Class + +Configuration data class with validation and nested access. + +### Methods + +#### `get()` + +Get nested configuration value by key path. + +```python +from semantica.core import Config + +config = Config(config_dict={"processing": {"batch_size": 32}}) + +# Get nested value +batch_size = config.get("processing.batch_size", default=16) + +# Get with default +timeout = config.get("processing.timeout", default=30) +``` + +**Parameters:** +- `key_path`: Dot-separated key path (e.g., "processing.batch_size") +- `default`: Default value if key not found + +**Returns:** +- Configuration value or default + +#### `set()` + +Set nested configuration value by key path. + +```python +config.set("processing.batch_size", 64) +config.set("llm_provider.model", "gpt-4") +``` + +**Parameters:** +- `key_path`: Dot-separated key path +- `value`: Value to set + +#### `update()` + +Update configuration with new values. + +```python +# Merge updates +config.update({ + "processing": {"batch_size": 128}, + "quality": {"min_confidence": 0.9} +}, merge=True) + +# Replace (don't merge) +config.update({ + "processing": {"batch_size": 128} +}, merge=False) +``` + +**Parameters:** +- `updates`: Dictionary of updates +- `merge`: Whether to merge nested dictionaries (default: True) + +#### `validate()` + +Validate configuration settings. + +```python +try: + config.validate() + print("Configuration is valid") +except ConfigurationError as e: + print(f"Validation failed: {e}") +``` + +**Raises:** +- `ConfigurationError`: If configuration is invalid with detailed error messages + +#### `to_dict()` + +Convert configuration to dictionary. + +```python +config_dict = config.to_dict() +print(config_dict["processing"]["batch_size"]) +``` + +**Returns:** +- Dictionary representation of configuration + +## LifecycleManager + +System lifecycle management with hooks and health monitoring. + +### Methods + +#### `startup()` + +Execute startup sequence with registered hooks. + +```python +from semantica.core import LifecycleManager + +manager = LifecycleManager() + +# Register startup hooks +def init_database(): + print("Initializing database...") + +def init_cache(): + print("Initializing cache...") + +manager.register_startup_hook(init_database, priority=10) +manager.register_startup_hook(init_cache, priority=20) + +# Execute startup (hooks run in priority order) +manager.startup() +``` + +**Raises:** +- `SemanticaError`: If startup fails + +#### `shutdown()` + +Execute shutdown sequence with registered hooks. + +```python +def cleanup_database(): + print("Cleaning up database...") + +manager.register_shutdown_hook(cleanup_database, priority=10) + +# Graceful shutdown (continues even if hooks fail) +manager.shutdown(graceful=True) + +# Force shutdown (stops on first error) +manager.shutdown(graceful=False) +``` + +**Parameters:** +- `graceful`: Whether to shutdown gracefully (default: True) + +**Raises:** +- `SemanticaError`: If shutdown fails and graceful=False + +#### `register_startup_hook()` + +Register a startup hook. + +```python +def my_startup_hook(): + # Your initialization code + pass + +# Lower priority = earlier execution +manager.register_startup_hook(my_startup_hook, priority=50) +``` + +**Parameters:** +- `hook_fn`: Function to call during startup (no arguments) +- `priority`: Hook priority (lower = earlier execution, default: 50) + +#### `register_shutdown_hook()` + +Register a shutdown hook. + +```python +def my_shutdown_hook(): + # Your cleanup code + pass + +manager.register_shutdown_hook(my_shutdown_hook, priority=50) +``` + +**Parameters:** +- `hook_fn`: Function to call during shutdown (no arguments) +- `priority`: Hook priority (lower = earlier execution, default: 50) + +#### `register_component()` + +Register a component for health monitoring. + +```python +class DatabaseConnection: + def health_check(self): + return {"healthy": True, "message": "Connected"} + +db = DatabaseConnection() +manager.register_component("database", db) +``` + +**Parameters:** +- `name`: Component name +- `component`: Component instance + +#### `health_check()` + +Perform comprehensive system health check. + +```python +health_results = manager.health_check() + +for component_name, status in health_results.items(): + if status.healthy: + print(f"{component_name}: βœ“ {status.message}") + else: + print(f"{component_name}: βœ— {status.message}") +``` + +**Returns:** +- Dictionary mapping component names to `HealthStatus` objects + +#### `get_health_summary()` + +Get summary of system health. + +```python +summary = manager.get_health_summary() + +print(f"Total components: {summary['total_components']}") +print(f"Healthy: {summary['healthy_components']}") +print(f"Unhealthy: {summary['unhealthy_components']}") +print(f"System healthy: {summary['is_healthy']}") +``` + +**Returns:** +- Dictionary with health summary information + +#### `get_state()` + +Get current system state. + +```python +state = manager.get_state() +print(f"Current state: {state.value}") # uninitialized, ready, running, etc. +``` + +**Returns:** +- `SystemState`: Current system state enum + +#### `is_ready()` / `is_running()` + +Check system state. + +```python +if manager.is_ready(): + print("System is ready") + +if manager.is_running(): + print("System is running") +``` + +## PluginRegistry + +Plugin registry and management system for dynamic plugin discovery and loading. + +### Methods + +#### `register_plugin()` + +Manually register a plugin. + +```python +from semantica.core import PluginRegistry + +registry = PluginRegistry() + +class MyPlugin: + def initialize(self): + print("Plugin initialized") + + def execute(self, data): + return {"result": "processed"} + +registry.register_plugin( + plugin_name="my_plugin", + plugin_class=MyPlugin, + version="1.0.0", + description="My custom plugin", + author="John Doe", + dependencies=["base_plugin"], + capabilities=["processing", "transformation"] +) +``` + +**Parameters:** +- `plugin_name`: Name of the plugin +- `plugin_class`: Plugin class to register +- `version`: Plugin version (default: "1.0.0") +- `**metadata`: Additional plugin metadata + +**Raises:** +- `ValidationError`: If plugin is invalid + +#### `load_plugin()` + +Load and initialize a plugin. + +```python +# Auto-discover and load +registry = PluginRegistry(plugin_paths=["./plugins"]) + +# Load with configuration +plugin = registry.load_plugin( + "my_plugin", + api_key="xxx", + host="localhost", + port=8080 +) + +# Dependencies are automatically loaded first +plugin = registry.load_plugin("dependent_plugin") +``` + +**Parameters:** +- `plugin_name`: Name of the plugin to load +- `**config`: Plugin configuration passed to plugin constructor + +**Returns:** +- Loaded and initialized plugin instance + +**Raises:** +- `ConfigurationError`: If plugin not found, dependencies missing, or initialization fails + +#### `unload_plugin()` + +Unload a plugin. + +```python +registry.unload_plugin("my_plugin") +``` + +**Parameters:** +- `plugin_name`: Name of plugin to unload + +**Raises:** +- `ConfigurationError`: If plugin not loaded + +#### `list_plugins()` + +List all available plugins. + +```python +plugins = registry.list_plugins() + +for plugin_info in plugins: + print(f"Name: {plugin_info['name']}") + print(f"Version: {plugin_info['version']}") + print(f"Loaded: {plugin_info['loaded']}") + print(f"Dependencies: {plugin_info['dependencies']}") +``` + +**Returns:** +- List of plugin information dictionaries + +#### `get_plugin_info()` + +Get information about a specific plugin. + +```python +info = registry.get_plugin_info("my_plugin") +print(f"Description: {info['description']}") +print(f"Author: {info['author']}") +print(f"Capabilities: {info['capabilities']}") +``` + +**Parameters:** +- `plugin_name`: Name of plugin + +**Returns:** +- Dictionary with plugin information + +**Raises:** +- `ConfigurationError`: If plugin not found + +#### `is_plugin_loaded()` + +Check if a plugin is loaded. + +```python +if registry.is_plugin_loaded("my_plugin"): + print("Plugin is loaded") +``` + +**Returns:** +- `bool`: True if plugin is loaded + +#### `get_loaded_plugin()` + +Get loaded plugin instance. + +```python +plugin = registry.get_loaded_plugin("my_plugin") +if plugin: + result = plugin.execute(data) +``` + +**Returns:** +- Plugin instance or None if not loaded + +## MethodRegistry + +Registry for custom orchestration methods. + +### Methods + +#### `register()` + +Register a custom orchestration method. + +```python +from semantica.core import method_registry + +def custom_kb_builder(sources, **kwargs): + # Custom knowledge base building logic + return {"knowledge_graph": {}, "embeddings": []} + +method_registry.register("knowledge_base", "custom", custom_kb_builder) +``` + +**Parameters:** +- `task`: Task type ("pipeline", "knowledge_base", "orchestration", "lifecycle") +- `name`: Method name +- `method_func`: Method function + +#### `get()` + +Get method by task and name. + +```python +method = method_registry.get("knowledge_base", "custom") +if method: + result = method(sources=["doc.pdf"]) +``` + +**Parameters:** +- `task`: Task type +- `name`: Method name + +**Returns:** +- Method function or None + +#### `list_all()` + +List all registered methods. + +```python +# List all methods +all_methods = method_registry.list_all() +# Returns: {"knowledge_base": ["default", "custom"], "pipeline": ["default"]} + +# List methods for specific task +kb_methods = method_registry.list_all("knowledge_base") +# Returns: {"knowledge_base": ["default", "custom"]} +``` + +**Parameters:** +- `task`: Optional task type to filter by + +**Returns:** +- Dictionary mapping task types to method names + +#### `unregister()` + +Unregister a method. + +```python +method_registry.unregister("knowledge_base", "custom") +``` + +**Parameters:** +- `task`: Task type +- `name`: Method name + +#### `clear()` + +Clear all registered methods. + +```python +# Clear all methods +method_registry.clear() + +# Clear methods for specific task +method_registry.clear("knowledge_base") +``` + +**Parameters:** +- `task`: Optional task type to clear (clears all if None) + +## Orchestration Methods + +Convenience functions for common orchestration tasks. + +### `build_knowledge_base()` + +Build knowledge base from data sources. + +```python +from semantica.core.methods import build_knowledge_base + +# Default method +result = build_knowledge_base( + sources=["doc1.pdf", "doc2.docx"], + method="default" +) + +# Minimal method (no embeddings or graph) +result = build_knowledge_base( + sources=["doc.pdf"], + method="minimal" +) + +# Full method (all features enabled) +result = build_knowledge_base( + sources=["doc.pdf"], + method="full", + embeddings=True, + graph=True +) + +# With custom configuration +config = {"llm_provider": {"name": "openai"}} +result = build_knowledge_base( + sources=["doc.pdf"], + config=config, + embeddings=True +) +``` + +**Parameters:** +- `sources`: Single source or list of sources +- `method`: Knowledge base construction method (default: "default") +- `config`: Optional configuration object or dictionary +- `**kwargs`: Additional options + +**Returns:** +- Dictionary with knowledge base data + +### `run_pipeline()` + +Execute a processing pipeline. + +```python +from semantica.core.methods import run_pipeline + +result = run_pipeline( + pipeline={"steps": ["parse", "extract"]}, + data="sample text", + method="default" +) +``` + +**Parameters:** +- `pipeline`: Pipeline object or configuration dictionary +- `data`: Input data for pipeline +- `method`: Pipeline execution method (default: "default") +- `config`: Optional configuration object or dictionary +- `**kwargs`: Additional pipeline options + +**Returns:** +- Dictionary with pipeline results + +### `initialize_framework()` + +Initialize Semantica framework. + +```python +from semantica.core.methods import initialize_framework + +# Default initialization +framework = initialize_framework() + +# Minimal initialization (no plugins) +framework = initialize_framework(method="minimal") + +# Full initialization +framework = initialize_framework(method="full", config=config_dict) +``` + +**Parameters:** +- `config`: Optional configuration object or dictionary +- `method`: Initialization method (default: "default") +- `**kwargs`: Additional initialization options + +**Returns:** +- Initialized `Semantica` framework instance + +### `get_status()` + +Get system status. + +```python +from semantica.core.methods import get_status + +# Default status +status = get_status(framework=my_framework) + +# Summary status +status = get_status(framework=my_framework, method="summary") + +# Detailed status +status = get_status(framework=my_framework, method="detailed") +``` + +**Parameters:** +- `framework`: Optional Semantica framework instance (creates new if None) +- `method`: Status retrieval method (default: "default") +- `**kwargs`: Additional options + +**Returns:** +- Dictionary with system status + +### `get_orchestration_method()` + +Get orchestration method by task and name. + +```python +from semantica.core.methods import get_orchestration_method + +method = get_orchestration_method("knowledge_base", "custom") +if method: + result = method(sources=["doc.pdf"]) +``` + +**Parameters:** +- `task`: Task type +- `name`: Method name + +**Returns:** +- Method function or None + +### `list_available_methods()` + +List all available orchestration methods. + +```python +from semantica.core.methods import list_available_methods + +# List all methods +all_methods = list_available_methods() + +# List methods for specific task +kb_methods = list_available_methods("knowledge_base") +``` + +**Parameters:** +- `task`: Optional task type to filter by + +**Returns:** +- Dictionary mapping task types to method names + +## Convenience Functions + +### Module-level `build()` Function + +Convenience function for building knowledge bases. + +```python +from semantica.core import build + +result = build( + sources=["doc1.pdf", "doc2.docx"], + extract_entities=True, + extract_relations=True, + embeddings=True, + graph=True +) +``` + +**Parameters:** +- `sources`: Input source or list of sources +- `extract_entities`: Whether to extract named entities (default: True) +- `extract_relations`: Whether to extract relationships (default: True) +- `embeddings`: Whether to generate embeddings (default: True) +- `graph`: Whether to build knowledge graph (default: True) +- `**options`: Additional processing options + +**Returns:** +- Dictionary with knowledge base data + +## Configuration + +### Environment Variables + +Configuration can be loaded from environment variables with `SEMANTICA_` prefix: + +```bash +export SEMANTICA_PROCESSING_BATCH_SIZE=64 +export SEMANTICA_LLM_PROVIDER_MODEL=gpt-4 +export SEMANTICA_QUALITY_MIN_CONFIDENCE=0.8 +``` + +### YAML Configuration File + +```yaml +llm_provider: + name: openai + model: gpt-4 + api_key: ${OPENAI_API_KEY} + +embedding_model: + name: openai + model: text-embedding-ada-002 + +processing: + batch_size: 32 + max_workers: 4 + +quality: + min_confidence: 0.7 + +logging: + level: INFO + +plugins: + my_plugin: + enabled: true + config_key: config_value +``` + +### JSON Configuration File + +```json +{ + "llm_provider": { + "name": "openai", + "model": "gpt-4" + }, + "processing": { + "batch_size": 32 + } +} +``` + +### Loading Configuration + +```python +from semantica.core import ConfigManager + +manager = ConfigManager() + +# Load from file +config = manager.load_from_file("config.yaml") + +# Load from dictionary +config = manager.load_from_dict({"processing": {"batch_size": 64}}) + +# Use with framework +from semantica.core import Semantica +framework = Semantica(config=config) +``` + +## Advanced Examples + +### Custom Plugin Development + +```python +from semantica.core import PluginRegistry + +class CustomProcessor: + def initialize(self): + print("Custom processor initialized") + + def execute(self, data): + # Process data + return {"processed": True, "data": data} + +# Register plugin +registry = PluginRegistry() +registry.register_plugin( + plugin_name="custom_processor", + plugin_class=CustomProcessor, + version="1.0.0", + description="Custom data processor", + capabilities=["processing"] +) + +# Load and use +processor = registry.load_plugin("custom_processor") +result = processor.execute("sample data") +``` + +### Custom Orchestration Method + +```python +from semantica.core import method_registry +from semantica.core import Semantica + +def fast_kb_builder(sources, **kwargs): + """Fast knowledge base builder with minimal processing.""" + framework = Semantica() + framework.initialize() + + try: + result = framework.build_knowledge_base( + sources=sources, + embeddings=False, # Skip embeddings for speed + graph=True, + **kwargs + ) + return result + finally: + framework.shutdown() + +# Register custom method +method_registry.register("knowledge_base", "fast", fast_kb_builder) + +# Use custom method +from semantica.core.methods import build_knowledge_base +result = build_knowledge_base(sources=["doc.pdf"], method="fast") +``` + +### Lifecycle Hooks + +```python +from semantica.core import LifecycleManager + +manager = LifecycleManager() + +# Register startup hooks with priorities +def init_logging(): + print("Initializing logging...") + +def init_database(): + print("Initializing database...") + +def init_cache(): + print("Initializing cache...") + +manager.register_startup_hook(init_logging, priority=10) # Runs first +manager.register_startup_hook(init_database, priority=20) # Runs second +manager.register_startup_hook(init_cache, priority=30) # Runs third + +# Register shutdown hooks +def cleanup_cache(): + print("Cleaning up cache...") + +def cleanup_database(): + print("Cleaning up database...") + +manager.register_shutdown_hook(cleanup_cache, priority=10) # Runs first +manager.register_shutdown_hook(cleanup_database, priority=20) # Runs second + +# Execute lifecycle +manager.startup() +# ... do work ... +manager.shutdown(graceful=True) +``` + +### Component Health Monitoring + +```python +from semantica.core import LifecycleManager + +class DatabaseComponent: + def __init__(self): + self.connected = False + + def connect(self): + self.connected = True + + def health_check(self): + return { + "healthy": self.connected, + "message": "Connected" if self.connected else "Not connected" + } + +manager = LifecycleManager() + +# Register component +db = DatabaseComponent() +db.connect() +manager.register_component("database", db) + +# Check health +health = manager.health_check() +for name, status in health.items(): + print(f"{name}: {status.healthy} - {status.message}") + +# Get health summary +summary = manager.get_health_summary() +print(f"System healthy: {summary['is_healthy']}") +``` + +### Configuration Merging + +```python +from semantica.core import ConfigManager + +manager = ConfigManager() + +# Load base configuration +base_config = manager.load_from_file("base_config.yaml") + +# Load environment-specific overrides +dev_config = manager.load_from_file("dev_config.yaml") + +# Load runtime overrides +runtime_config = manager.load_from_dict({ + "processing": {"batch_size": 128} +}) + +# Merge (later configs override earlier ones) +merged = manager.merge_configs(base_config, dev_config, runtime_config) + +# Use merged configuration +from semantica.core import Semantica +framework = Semantica(config=merged) +``` + +## Best Practices + +1. **Always Initialize**: Always call `initialize()` after creating a `Semantica` instance before using it. + +2. **Graceful Shutdown**: Always call `shutdown(graceful=True)` in a `finally` block to ensure proper cleanup. + +3. **Configuration Management**: Use `ConfigManager` for loading and managing configurations. Prefer YAML files for complex configurations. + +4. **Error Handling**: Wrap framework operations in try-except blocks to handle `ConfigurationError` and `ProcessingError` appropriately. + +5. **Health Monitoring**: Register components with `LifecycleManager` for health monitoring and use `health_check()` regularly. + +6. **Plugin Development**: Follow the plugin interface (must have `initialize()` and `execute()` methods) when creating custom plugins. + +7. **Method Registration**: Use `MethodRegistry` for extensibility. Register custom methods for knowledge base building, pipeline execution, etc. + +8. **Hook Priorities**: Use appropriate priorities for lifecycle hooks. Lower numbers execute first. + +9. **Configuration Validation**: Always validate configurations using `config.validate()` before using them. + +10. **Resource Cleanup**: Ensure all resources are properly cleaned up in shutdown hooks. + +### Example: Complete Workflow + +```python +from semantica.core import Semantica, ConfigManager + +# 1. Load configuration +config_manager = ConfigManager() +config = config_manager.load_from_file("config.yaml") + +# 2. Initialize framework +framework = Semantica(config=config) + +try: + # 3. Initialize all components + framework.initialize() + + # 4. Check system health + status = framework.get_status() + if not status['health']['is_healthy']: + print("Warning: Some components are unhealthy") + + # 5. Build knowledge base + result = framework.build_knowledge_base( + sources=["doc1.pdf", "doc2.docx"], + embeddings=True, + graph=True + ) + + # 6. Process results + print(f"Processed {result['statistics']['sources_processed']} sources") + print(f"Knowledge graph has {len(result['knowledge_graph'].get('entities', []))} entities") + + # 7. Run pipeline + pipeline_result = framework.run_pipeline( + pipeline={"steps": ["extract", "normalize"]}, + data="sample text" + ) + +finally: + # 8. Always shutdown gracefully + framework.shutdown(graceful=True) +``` + +This completes the comprehensive usage guide for the core module. All classes, methods, and their usage patterns are documented with examples. diff --git a/semantica/core/lifecycle.py b/semantica/core/lifecycle.py index 14b7e4bd..383d4ac2 100644 --- a/semantica/core/lifecycle.py +++ b/semantica/core/lifecycle.py @@ -135,53 +135,17 @@ class LifecycleManager: try: # Check if already started - if self.state in (SystemState.READY, SystemState.RUNNING): - self.logger.warning( - f"System already in {self.state.value} state, skipping startup" - ) + if self._is_already_started(): self.progress_tracker.stop_tracking( tracking_id, status="completed", message="System already started" ) return - # Transition to initializing state - self.state = SystemState.INITIALIZING - self.logger.info("Starting system lifecycle") + # Execute startup sequence + self._execute_startup_sequence() - # Sort hooks by priority (lower priority = earlier execution) - sorted_hooks = sorted(self.startup_hooks, key=lambda x: x[1]) - - if sorted_hooks: - self.logger.debug(f"Executing {len(sorted_hooks)} startup hook(s)") - - # Execute all startup hooks in priority order - for hook_fn, priority in sorted_hooks: - try: - self.logger.debug( - f"Executing startup hook with priority {priority}" - ) - hook_fn() - except Exception as e: - error_msg = f"Startup hook (priority {priority}) failed: {e}" - self.logger.error(error_msg) - self.state = SystemState.ERROR - raise SemanticaError(error_msg) from e - - # Verify all registered components are properly initialized - self._verify_components() - - # Run initial health checks on all components - health_results = self.health_check() - unhealthy_components = [ - name for name, status in health_results.items() if not status.healthy - ] - - if unhealthy_components: - self.logger.warning( - f"Some components are unhealthy after startup: {unhealthy_components}" - ) - else: - self.logger.debug("All components are healthy") + # Verify and check health + self._verify_and_check_health() # Transition to ready state self.state = SystemState.READY @@ -202,6 +166,41 @@ class LifecycleManager: ) raise + def _is_already_started(self) -> bool: + """Check if system is already in a started state.""" + if self.state in (SystemState.READY, SystemState.RUNNING): + self.logger.warning( + f"System already in {self.state.value} state, skipping startup" + ) + return True + return False + + def _execute_startup_sequence(self) -> None: + """Execute startup hooks in priority order.""" + self.state = SystemState.INITIALIZING + self.logger.info("Starting system lifecycle") + + # Execute hooks + self._execute_hooks(self.startup_hooks, "startup") + + def _verify_and_check_health(self) -> None: + """Verify components and check their health.""" + # Verify all registered components are properly initialized + self._verify_components() + + # Run initial health checks + health_results = self.health_check() + unhealthy_components = [ + name for name, status in health_results.items() if not status.healthy + ] + + if unhealthy_components: + self.logger.warning( + f"Some components are unhealthy after startup: {unhealthy_components}" + ) + else: + self.logger.debug("All components are healthy") + def shutdown(self, graceful: bool = True) -> None: """ Execute shutdown sequence. @@ -232,42 +231,16 @@ class LifecycleManager: try: # Check if already stopped - if self.state == SystemState.STOPPED: - self.logger.warning("System already in STOPPED state") + if self._is_already_stopped(): self.progress_tracker.stop_tracking( tracking_id, status="completed", message="System already stopped" ) return - # Transition to stopping state - self.state = SystemState.STOPPING - self.logger.info(f"Shutting down system (graceful={graceful})") + # Execute shutdown sequence + self._execute_shutdown_sequence(graceful) - # Sort hooks by priority (lower priority = earlier execution) - sorted_hooks = sorted(self.shutdown_hooks, key=lambda x: x[1]) - - if sorted_hooks: - self.logger.debug(f"Executing {len(sorted_hooks)} shutdown hook(s)") - - # Execute all shutdown hooks in priority order - for hook_fn, priority in sorted_hooks: - try: - self.logger.debug( - f"Executing shutdown hook with priority {priority}" - ) - hook_fn() - except Exception as e: - error_msg = f"Shutdown hook (priority {priority}) failed: {e}" - - if graceful: - # In graceful mode, log warning but continue - self.logger.warning(error_msg) - else: - # In non-graceful mode, stop on first error - self.logger.error(error_msg) - raise SemanticaError(error_msg) from e - - # Cleanup all registered components + # Cleanup resources self._cleanup_resources() # Transition to stopped state @@ -290,6 +263,57 @@ class LifecycleManager: if not graceful: raise + def _is_already_stopped(self) -> bool: + """Check if system is already stopped.""" + if self.state == SystemState.STOPPED: + self.logger.warning("System already in STOPPED state") + return True + return False + + def _execute_shutdown_sequence(self, graceful: bool) -> None: + """Execute shutdown hooks in priority order.""" + self.state = SystemState.STOPPING + self.logger.info(f"Shutting down system (graceful={graceful})") + + # Execute hooks with graceful error handling + self._execute_hooks(self.shutdown_hooks, "shutdown", graceful=graceful) + + def _execute_hooks( + self, hooks: List[Tuple[Callable[[], None], int]], hook_type: str, graceful: bool = False + ) -> None: + """ + Execute hooks in priority order. + + Args: + hooks: List of (hook_function, priority) tuples + hook_type: Type of hooks ("startup" or "shutdown") + graceful: Whether to continue on errors (only for shutdown) + + Raises: + SemanticaError: If hook fails and not graceful + """ + # Sort hooks by priority (lower priority = earlier execution) + sorted_hooks = sorted(hooks, key=lambda x: x[1]) + + if sorted_hooks: + self.logger.debug(f"Executing {len(sorted_hooks)} {hook_type} hook(s)") + + # Execute all hooks in priority order + for hook_fn, priority in sorted_hooks: + try: + self.logger.debug(f"Executing {hook_type} hook with priority {priority}") + hook_fn() + except Exception as e: + error_msg = f"{hook_type.capitalize()} hook (priority {priority}) failed: {e}" + + if graceful: + # In graceful mode, log warning but continue + self.logger.warning(error_msg) + else: + # In non-graceful mode, stop on first error + self.logger.error(error_msg) + raise SemanticaError(error_msg) from e + def health_check(self) -> Dict[str, HealthStatus]: """ Perform comprehensive system health check. @@ -310,65 +334,87 @@ class LifecycleManager: # Record health check timestamp self._last_health_check = time.time() - health_results = {} - # Check health of each registered component - for component_name, component in self._component_registry.items(): - try: - # Try to get health status from component - if hasattr(component, "health_check"): - # Component has its own health check method - component_health = component.health_check() + health_results = { + name: self._check_component_health(name, component) + for name, component in self._component_registry.items() + } - # Handle different return types - if isinstance(component_health, dict): - # Dictionary format: {"healthy": bool, "message": str, "details": dict} - healthy = component_health.get("healthy", True) - message = component_health.get("message", "") - details = component_health.get("details", {}) - elif isinstance(component_health, bool): - # Simple boolean - healthy = component_health - message = "" - details = {} - else: - # Other types: convert to boolean - healthy = bool(component_health) - message = "" - details = {} - else: - # No health_check method: assume healthy if component exists - healthy = component is not None - message = "Component exists" if healthy else "Component is None" - details = {} + # Update cached health status + self.health_status.update(health_results) - # Create health status object - status = HealthStatus( - component=component_name, - healthy=healthy, - message=message, - details=details, - ) + # Log summary + self._log_health_summary(health_results) - except Exception as e: - # Health check failed: mark as unhealthy - error_msg = f"Health check failed: {e}" - self.logger.warning( - f"Component {component_name} health check error: {e}" - ) + return health_results - status = HealthStatus( - component=component_name, - healthy=False, - message=error_msg, - details={"error": str(e), "error_type": type(e).__name__}, - ) + def _check_component_health(self, component_name: str, component: Any) -> HealthStatus: + """ + Check health of a single component. - # Store results - health_results[component_name] = status - self.health_status[component_name] = status + Args: + component_name: Name of the component + component: Component instance - # Log summary of unhealthy components + Returns: + HealthStatus object for the component + """ + try: + if hasattr(component, "health_check"): + # Component has its own health check method + component_health = component.health_check() + healthy, message, details = self._parse_health_result(component_health) + else: + # No health_check method: assume healthy if component exists + healthy = component is not None + message = "Component exists" if healthy else "Component is None" + details = {} + + return HealthStatus( + component=component_name, + healthy=healthy, + message=message, + details=details, + ) + + except Exception as e: + # Health check failed: mark as unhealthy + error_msg = f"Health check failed: {e}" + self.logger.warning(f"Component {component_name} health check error: {e}") + + return HealthStatus( + component=component_name, + healthy=False, + message=error_msg, + details={"error": str(e), "error_type": type(e).__name__}, + ) + + def _parse_health_result(self, health_result: Any) -> Tuple[bool, str, Dict[str, Any]]: + """ + Parse component health check result into standardized format. + + Args: + health_result: Health check result (dict, bool, or other) + + Returns: + Tuple of (healthy, message, details) + """ + if isinstance(health_result, dict): + # Dictionary format: {"healthy": bool, "message": str, "details": dict} + return ( + health_result.get("healthy", True), + health_result.get("message", ""), + health_result.get("details", {}), + ) + elif isinstance(health_result, bool): + # Simple boolean + return health_result, "", {} + else: + # Other types: convert to boolean + return bool(health_result), "", {} + + def _log_health_summary(self, health_results: Dict[str, HealthStatus]) -> None: + """Log summary of health check results.""" unhealthy_components = [ name for name, status in health_results.items() if not status.healthy ] @@ -383,8 +429,6 @@ class LifecycleManager: f"Health check passed for all {len(health_results)} component(s)" ) - return health_results - def register_component(self, name: str, component: Any) -> None: """ Register a component for health monitoring.