From 6bcc71f48a57ece623217cc7196e39666be26e92 Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Sat, 29 Nov 2025 19:21:04 +0530 Subject: [PATCH] docs: Add comprehensive Modules & Architecture guide - Add 7 new module sections (Split, Triple Store, Deduplication, Conflicts, KG QA, Context, Seed) - Organize modules into 6 logical layers - Add key features and components in bullet points for all modules - Add quick reference table with all 20 modules - Add 4 integration pattern examples - Include algorithms/strategies tables where applicable --- docs/modules.md | 1388 +++++++++++++++++++++++++---------------------- 1 file changed, 729 insertions(+), 659 deletions(-) diff --git a/docs/modules.md b/docs/modules.md index 096ca4dc..dfb55f2f 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -1,206 +1,204 @@ # Modules & Architecture -Semantica is built with a modular architecture, designed to be flexible, extensible, and scalable. This guide provides a comprehensive overview of the key modules, their responsibilities, and how they interact to build powerful semantic applications. +Semantica is built with a modular architecture, designed to be flexible, extensible, and scalable. This guide provides a comprehensive overview of all modules, their responsibilities, key features, and components. !!! info "About This Guide" - This guide covers all 13 core modules in Semantica, organized by their functional layer. It details their purpose, key features, usage patterns, and configuration options. + This guide covers all 20+ core modules in Semantica, organized by their functional layer. Each module can be used independently or combined into powerful pipelines. --- ## Module Overview -Semantica's modules are organized into five logical layers. You can use these modules independently or combine them into a complete pipeline. +Semantica's modules are organized into six logical layers: | Layer | Modules | Description | | :--- | :--- | :--- | -| **Input Layer** | [Ingest](#1-ingest-module), [Parse](#2-parse-module), [Normalize](#3-normalize-module) | Handles data ingestion, parsing, and cleaning from various sources. | -| **Core Processing** | [Semantic Extract](#4-semantic-extract-module), [Knowledge Graph](#5-knowledge-graph-kg-module), [Ontology](#10-ontology-module), [Reasoning](#9-reasoning-module) | The "brain" of the system. Extracts meaning, builds graphs, and infers new knowledge. | -| **Storage & Embeddings** | [Embeddings](#6-embeddings-module), [Vector Store](#7-vector-store-module), [Graph Store](#8-graph-store-module) | Manages persistent storage and retrieval of vectors and graphs. | -| **Output Layer** | [Export](#11-export-module), [Visualization](#12-visualization-module) | Tools for visualizing data and exporting it to external systems. | -| **Orchestration** | [Pipeline](#13-pipeline-module) | Manages workflows, execution, and resource scheduling. | +| **Input Layer** | [Ingest](#ingest-module), [Parse](#parse-module), [Split](#split-module), [Normalize](#normalize-module) | Data ingestion, parsing, chunking, and cleaning | +| **Core Processing** | [Semantic Extract](#semantic-extract-module), [Knowledge Graph](#knowledge-graph-kg-module), [Ontology](#ontology-module), [Reasoning](#reasoning-module) | Entity extraction, graph construction, inference | +| **Storage** | [Embeddings](#embeddings-module), [Vector Store](#vector-store-module), [Graph Store](#graph-store-module), [Triple Store](#triple-store-module) | Vector and graph persistence | +| **Quality Assurance** | [Deduplication](#deduplication-module), [Conflicts](#conflicts-module), [KG QA](#kg-quality-assurance-module) | Data quality and consistency | +| **Context & Memory** | [Context](#context-module), [Seed](#seed-module) | Agent memory and foundation data | +| **Output & Orchestration** | [Export](#export-module), [Visualization](#visualization-module), [Pipeline](#pipeline-module) | Export, visualization, and workflow management | --- ## Input Layer -These modules are responsible for getting data into the system and preparing it for processing. +These modules handle data ingestion, parsing, chunking, and preparation. -### 1. Ingest Module +--- + +### Ingest Module !!! abstract "Purpose" - The `ingest` module is the entry point for data ingestion. It handles the complexity of connecting to different data sources, from local files to web streams, and supports connecting to your own MCP (Model Context Protocol) servers. + The entry point for data ingestion. Connects to various data sources including files, web, databases, and MCP servers. -**Key Features**: +**Key Features:** -- Support for 50+ file formats (PDF, DOCX, HTML, JSON, CSV, etc.) +- 50+ file format support (PDF, DOCX, HTML, JSON, CSV, etc.) - Web scraping with JavaScript rendering - Database integration (SQL, NoSQL) - Real-time streaming support -- MCP server integration +- MCP (Model Context Protocol) server integration - Batch processing capabilities -- Metadata extraction +- Metadata extraction and preservation -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :--------- | :---------------- | :----------------------------------- | -| `source` | `str`, `Path`, `List` | File path, URL, directory, or list of sources | -| `recursive`| `bool` | Whether to recursively scan directories | -| `filters` | `Dict` | Filters for file types, sizes, etc. | +- `FileIngestor` — Read files (PDF, DOCX, HTML, JSON, CSV, etc.) +- `WebIngestor` — Scrape and ingest web pages +- `FeedIngestor` — Process RSS/Atom feeds +- `StreamIngestor` — Real-time data streaming +- `DBIngestor` — Database queries and ingestion +- `EmailIngestor` — Process email messages +- `RepoIngestor` — Git repository analysis +- `MCPIngestor` — Connect to MCP servers for resource and tool-based ingestion -| Output | Type | Description | -| :--------- | :---------------- | :----------------------------------- | -| `documents`| `List[Document]` | List of Document objects with content and metadata | -| `metadata` | `Dict` | Source metadata (paths, timestamps, formats) | - -**Components**: - -- `FileIngestor`: Read files (PDF, DOCX, HTML, JSON, CSV, etc.) -- `WebIngestor`: Scrape and ingest web pages -- `FeedIngestor`: Process RSS/Atom feeds -- `StreamIngestor`: Real-time data streaming -- `DBIngestor`: Database queries and ingestion -- `EmailIngestor`: Process email messages -- `RepoIngestor`: Git repository analysis -- `MCPIngestor`: Connect to your own Python/FastMCP MCP servers via URL for resource and tool-based data ingestion - -**Complete Code Example**: +**Quick Example:** ```python -from semantica.ingest import FileIngestor, WebIngestor, DBIngestor, MCPIngestor +from semantica.ingest import FileIngestor, WebIngestor -# Example 1: Ingest local files +# Ingest local files file_ingestor = FileIngestor() -documents = file_ingestor.ingest("data/", recursive=True) # (1) -print(f"Ingested {len(documents)} documents") +documents = file_ingestor.ingest("data/", recursive=True) -# Example 2: Ingest web content +# Ingest web content web_ingestor = WebIngestor() -web_docs = web_ingestor.ingest("https://example.com") # (2) -print(f"Web content: {web_docs[0].content[:100]}...") - -# Example 3: Ingest from database -db_ingestor = DBIngestor(connection_string="postgresql://user:pass@localhost/db") -db_docs = db_ingestor.ingest("SELECT * FROM documents") - -# Example 4: Ingest from MCP server -mcp_ingestor = MCPIngestor() -mcp_docs = mcp_ingestor.ingest("https://your-mcp-server.com") +web_docs = web_ingestor.ingest("https://example.com") ``` -1. Recursively scans the directory for supported file types (PDF, DOCX, etc.) and converts them to standard Document objects. -2. Fetches the URL, renders JavaScript if necessary, and extracts the main content while stripping boilerplate. - -**Configuration Options**: - -| Option | Type | Default | Description | -| :--------------| :---------- | :------ | :----------------------------------- | -| `recursive` | `bool` | `False` | Recursively scan directories | -| `file_filters` | `List[str]` | `None` | Filter by file extensions | -| `max_file_size`| `int` | `None` | Maximum file size in bytes | -| `timeout` | `int` | `30` | Request timeout in seconds | -| `user_agent` | `str` | `None` | Custom user agent for web requests | - **API Reference**: [Ingest Module](reference/ingest.md) -### 2. Parse Module +--- + +### Parse Module !!! abstract "Purpose" - Once data is ingested, the `parse` module extracts the raw text and metadata. It supports a wide range of formats and includes OCR capabilities. + Extracts raw text and metadata from ingested documents. Supports OCR, table extraction, and structured data parsing. + +**Key Features:** -**Key Features**: - 50+ file format support - OCR for images and scanned documents - Table extraction from PDFs and spreadsheets - Metadata preservation - Automatic format detection - Structured data parsing (JSON, CSV, XML) +- Code file parsing with syntax awareness -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :------------- | :-------------------- | :----------------------------------- | -| `documents` | `List[Document]` | Documents from ingest module | -| `ocr_enabled` | `bool` | Enable OCR for images | +- `DocumentParser` — Main parser orchestrator +- `PDFParser` — Extract text, tables, images from PDFs +- `DOCXParser` — Parse Word documents +- `HTMLParser` — Extract content from HTML +- `JSONParser` — Parse structured JSON data +- `ExcelParser` — Process spreadsheets +- `ImageParser` — OCR and image analysis +- `CodeParser` — Parse source code files -| Output | Type | Description | -| :------------- | :-------------------- | :----------------------------------- | -| `parsed_docs` | `List[ParsedDocument]`| Documents with extracted text and metadata | -| `tables` | `List[Table]` | Extracted tables (if any) | - -**Components**: -- `DocumentParser`: Main parser orchestrator -- `PDFParser`: Extract text, tables, images from PDFs -- `DOCXParser`: Parse Word documents -- `HTMLParser`: Extract content from HTML -- `JSONParser`: Parse structured JSON data -- `ExcelParser`: Process spreadsheets -- `ImageParser`: OCR and image analysis -- `CodeParser`: Parse source code files - -**Complete Code Example**: +**Quick Example:** ```python from semantica.parse import DocumentParser parser = DocumentParser(ocr_enabled=True) -parsed_docs = parser.parse(documents) # (1) +parsed_docs = parser.parse(documents) -# Access parsed content for doc in parsed_docs: print(f"Content: {doc.content[:100]}...") - print(f"Metadata: {doc.metadata}") - if doc.tables: - print(f"Found {len(doc.tables)} tables") + print(f"Tables found: {len(doc.tables)}") ``` -1. Automatically detects the file type of each document and routes it to the appropriate specialized parser (e.g., PDFParser for .pdf). - -**Configuration Options**: - -| Option | Type | Default | Description | -| :-------------- | :----- | :------ | :------------------------------- | -| `ocr_enabled` | `bool` | `False` | Enable OCR for images | -| `extract_tables`| `bool` | `True` | Extract tables from documents | -| `extract_images`| `bool` | `False` | Extract and process images | - **API Reference**: [Parse Module](reference/parse.md) -### 3. Normalize Module +--- + +### Split Module !!! abstract "Purpose" - Raw text is often noisy. The `normalize` module cleans, standardizes, and prepares text for semantic extraction. + Comprehensive document chunking and splitting for optimal processing. Provides 15+ splitting methods including KG-aware chunking. + +**Key Features:** + +- Multiple standard splitting methods (recursive, token, sentence, paragraph) +- Semantic-based chunking using NLP and embeddings +- Entity-aware chunking for GraphRAG workflows +- Relation-aware chunking for KG preservation +- Graph-based and ontology-aware chunking +- Hierarchical multi-level chunking +- Community detection-based splitting +- Sliding window chunking with overlap +- Table-specific chunking +- Chunk validation and quality assessment +- Provenance tracking for data lineage + +**Components:** + +- `TextSplitter` — Unified text splitter with method parameter +- `SemanticChunker` — Semantic-based chunking coordinator +- `StructuralChunker` — Structure-aware chunking (headings, lists) +- `SlidingWindowChunker` — Fixed-size sliding window chunking +- `TableChunker` — Table-specific chunking +- `EntityAwareChunker` — Entity boundary-preserving chunker +- `RelationAwareChunker` — Triple-preserving chunker +- `GraphBasedChunker` — Graph structure-based chunker +- `OntologyAwareChunker` — Ontology concept-based chunker +- `HierarchicalChunker` — Multi-level hierarchical chunker +- `ChunkValidator` — Chunk quality validation +- `ProvenanceTracker` — Chunk provenance tracking + +**Supported Methods:** + +| Category | Methods | +| :--- | :--- | +| **Standard** | recursive, token, sentence, paragraph, character, word, semantic_transformer, llm | +| **KG/Ontology** | entity_aware, relation_aware, graph_based, ontology_aware, hierarchical, community_detection, centrality_based | + +**Quick Example:** + +```python +from semantica.split import TextSplitter + +# Standard recursive splitting +splitter = TextSplitter(method="recursive", chunk_size=1000, chunk_overlap=200) +chunks = splitter.split(text) + +# Entity-aware for GraphRAG +splitter = TextSplitter(method="entity_aware", ner_method="llm", chunk_size=1000) +chunks = splitter.split(text) +``` + +--- + +### Normalize Module + +!!! abstract "Purpose" + Cleans, standardizes, and prepares text for semantic extraction. Handles encoding, entity names, dates, and numbers. + +**Key Features:** -**Key Features**: - Text cleaning and noise removal -- Encoding normalization +- Encoding normalization (Unicode handling) - Entity name standardization - Date and number formatting - Language detection -- Unicode normalization +- Whitespace normalization +- Special character handling -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :----------------- | :---------------------- | :----------------------------------- | -| `documents` | `List[ParsedDocument]` | Parsed documents | -| `normalize_entities`| `bool` | Normalize entity names | +- `TextNormalizer` — Main normalization orchestrator +- `TextCleaner` — Remove noise, fix encoding +- `DataCleaner` — Clean structured data +- `EntityNormalizer` — Normalize entity names +- `DateNormalizer` — Standardize date formats +- `NumberNormalizer` — Normalize numeric values +- `LanguageDetector` — Detect document language +- `EncodingHandler` — Handle character encoding -| Output | Type | Description | -| :------------- | :---------------------- | :----------------------------------- | -| `normalized_docs`| `List[NormalizedDocument]`| Cleaned and normalized documents | - -**Components**: -- `TextNormalizer`: Main normalization orchestrator -- `TextCleaner`: Remove noise, fix encoding -- `DataCleaner`: Clean structured data -- `EntityNormalizer`: Normalize entity names -- `DateNormalizer`: Standardize date formats -- `NumberNormalizer`: Normalize numeric values -- `LanguageDetector`: Detect document language -- `EncodingHandler`: Handle character encoding - -**Complete Code Example**: +**Quick Example:** ```python from semantica.normalize import TextNormalizer @@ -212,62 +210,45 @@ normalizer = TextNormalizer( ) normalized = normalizer.normalize(parsed_docs) -# Check normalization results for doc in normalized: print(f"Language: {doc.language}") - print(f"Normalized text: {doc.content[:100]}...") ``` -**Configuration Options**: - -| Option | Type | Default | Description | -| :----------------- | :----- | :------ | :--------------------------- | -| `normalize_entities` | `bool` | `True` | Normalize entity names | -| `normalize_dates` | `bool` | `True` | Standardize date formats | -| `normalize_numbers` | `bool` | `True` | Normalize numeric values | -| `detect_language` | `bool` | `False` | Detect document language | - **API Reference**: [Normalize Module](reference/normalize.md) --- ## Core Processing Layer -These modules form the intelligence of Semantica, extracting meaning, building relationships, and inferring new knowledge. +These modules form the intelligence core—extracting meaning, building relationships, and inferring knowledge. -### 4. Semantic Extract Module +--- + +### Semantic Extract Module !!! abstract "Purpose" - This is the brain of the operation. It uses LLMs and NLP techniques to understand the text and extract structured knowledge. + The brain of Semantica. Uses LLMs and NLP to extract entities, relationships, and semantic meaning from text. + +**Key Features:** -**Key Features**: - Multiple NER methods (rule-based, ML, LLM) -- Relationship extraction -- Semantic analysis -- Custom entity types -- Confidence scoring +- Relationship extraction with confidence scoring +- Event extraction +- Custom entity type support - Multi-language support +- Semantic network extraction +- Coreference resolution -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :------------- | :------------------------ | :----------------------------------- | -| `documents` | `List[NormalizedDocument]`| Normalized documents | -| `entity_types` | `List[str]` | Custom entity types to extract | -| `method` | `str` | Extraction method (rule/ml/llm) | +- `NERExtractor` — Named Entity Recognition +- `RelationExtractor` — Extract relationships between entities +- `SemanticAnalyzer` — Deep semantic analysis +- `SemanticNetworkExtractor` — Extract semantic networks +- `EventExtractor` — Extract events from text +- `CoreferenceResolver` — Resolve entity coreferences -| Output | Type | Description | -| :------------- | :------------------ | :----------------------------------- | -| `entities` | `List[Entity]` | Extracted entities with types and confidence | -| `relationships`| `List[Relationship]`| Extracted relationships | - -**Components**: -- `NERExtractor`: Named Entity Recognition -- `RelationExtractor`: Extract relationships between entities -- `SemanticAnalyzer`: Deep semantic analysis -- `SemanticNetworkExtractor`: Extract semantic networks - -**Complete Code Example**: +**Quick Example:** ```python from semantica.semantic_extract import NERExtractor, RelationExtractor @@ -276,204 +257,96 @@ from semantica.semantic_extract import NERExtractor, RelationExtractor extractor = NERExtractor(method="llm", model="gpt-4") entities = extractor.extract(normalized_docs) -print(f"Extracted {len(entities)} entities") -for entity in entities[:5]: - print(f" - {entity.text} ({entity.label}) - Confidence: {entity.confidence:.2f}") - # Extract relationships relation_extractor = RelationExtractor() relationships = relation_extractor.extract(normalized_docs, entities=entities) -print(f"\nExtracted {len(relationships)} relationships") for rel in relationships[:5]: - print(f" {rel.subject.text} --[{rel.predicate}]--> {rel.object.text}") + print(f"{rel.subject.text} --[{rel.predicate}]--> {rel.object.text}") ``` -**Configuration Options**: - -| Option | Type | Default | Description | -| :-------------------- | :---------- | :------ | :------------------------------- | -| `method` | `str` | `"ml"` | Extraction method (rule/ml/llm) | -| `model` | `str` | `None` | Model name for ML/LLM methods | -| `confidence_threshold`| `float` | `0.5` | Minimum confidence score | -| `custom_entity_types`| `List[str]` | `[]` | Custom entity types | - **API Reference**: [Semantic Extract Module](reference/semantic_extract.md) -### 5. Knowledge Graph (KG) Module +--- + +### Knowledge Graph (KG) Module !!! abstract "Purpose" - The `kg` module constructs the graph from extracted entities and relationships, handling complex tasks like resolution and analysis. + Constructs and manages knowledge graphs from extracted entities and relationships. Supports multiple backends and advanced analytics. + +**Key Features:** -**Key Features**: - Graph construction from entities/relationships - Multiple backend support (NetworkX, Neo4j, KuzuDB) - Temporal graph support - Graph analytics and metrics -- Conflict detection and resolution -- Entity deduplication +- Entity resolution and deduplication - Community detection - Centrality calculations +- Path finding algorithms +- Graph validation -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :------------- | :------------------ | :----------------------------------- | -| `entities` | `List[Entity]` | Extracted entities | -| `relationships`| `List[Relationship]`| Extracted relationships | -| `backend` | `str` | Graph backend (networkx/neo4j/kuzu) | +- `GraphBuilder` — Construct knowledge graphs +- `GraphAnalyzer` — Analyze graph structure and properties +- `GraphValidator` — Validate graph quality and consistency +- `EntityResolver` — Resolve entity conflicts and duplicates +- `ConflictDetector` — Detect conflicting information +- `CentralityCalculator` — Calculate node importance metrics +- `CommunityDetector` — Detect communities in graphs +- `ConnectivityAnalyzer` — Analyze graph connectivity +- `TemporalQuery` — Query temporal knowledge graphs +- `Deduplicator` — Remove duplicate entities/relationships -| Output | Type | Description | -| :------------- | :---------------- | :----------------------------------- | -| `knowledge_graph`| `KnowledgeGraph`| Constructed knowledge graph | -| `metrics` | `Dict` | Graph metrics (density, centrality, etc.) | - -**Components**: -- `GraphBuilder`: Construct knowledge graphs from entities/relationships -- `GraphAnalyzer`: Analyze graph structure and properties -- `GraphValidator`: Validate graph quality and consistency -- `EntityResolver`: Resolve entity conflicts and duplicates -- `ConflictDetector`: Detect conflicting information -- `CentralityCalculator`: Calculate node importance metrics -- `CommunityDetector`: Detect communities in graphs -- `ConnectivityAnalyzer`: Analyze graph connectivity -- `TemporalQuery`: Query temporal knowledge graphs -- `Deduplicator`: Remove duplicate entities/relationships - -**Complete Code Example**: +**Quick Example:** ```python from semantica.kg import GraphBuilder, GraphAnalyzer # Build graph builder = GraphBuilder(backend="networkx", temporal=True) -kg = builder.build(entities, relationships) # (1) +kg = builder.build(entities, relationships) # Analyze graph analyzer = GraphAnalyzer() -metrics = analyzer.analyze(kg) # (2) +metrics = analyzer.analyze(kg) -print(f"Nodes: {metrics['nodes']}") -print(f"Edges: {metrics['edges']}") +print(f"Nodes: {metrics['nodes']}, Edges: {metrics['edges']}") print(f"Density: {metrics['density']:.3f}") -print(f"Communities: {metrics['communities']}") ``` -1. Constructs a NetworkX or Neo4j graph from the extracted entities and relationships, handling node merging and edge attributes. -2. Computes graph-theoretic metrics like density, diameter, and centrality to assess the quality and structure of the knowledge graph. - -**Configuration Options**: - -| Option | Type | Default | Description | -| :---------------- | :----- | :------------ | :----------------------------------- | -| `backend` | `str` | `"networkx"` | Graph backend (networkx/neo4j/kuzu) | -| `temporal` | `bool` | `False` | Enable temporal graph support | -| `merge_duplicates`| `bool` | `True` | Automatically merge duplicate entities| - **API Reference**: [Knowledge Graph Module](reference/kg.md) -### 9. Reasoning Module +--- + +### Ontology Module !!! abstract "Purpose" - Goes beyond simple retrieval to infer new facts and validate existing knowledge using logical rules. + Defines schema and structure for your knowledge domain. Generates and validates ontologies with OWL/RDF export. -**Key Features**: -- Forward and backward chaining -- Rule-based inference -- Deductive and abductive reasoning -- Explanation generation -- RETE algorithm support -- Custom rule definition +**Key Features:** -**Input/Output Specification**: - -| Input | Type | Description | -| :-------------- | :------------------ | :----------------------------------- | -| `knowledge_graph`| `KnowledgeGraph` | Input knowledge graph | -| `rules` | `List[Rule]` | Inference rules | -| `method` | `str` | Reasoning method | - -| Output | Type | Description | -| :------------ | :------------------ | :----------------------------------- | -| `new_facts` | `List[Fact]` | Inferred facts | -| `explanations`| `List[Explanation]` | Reasoning explanations | - -**Components**: -- `InferenceEngine`: Main inference orchestrator -- `RuleManager`: Manage inference rules -- `DeductiveReasoner`: Deductive reasoning -- `AbductiveReasoner`: Abductive reasoning -- `ExplanationGenerator`: Generate explanations for inferences -- `RETEEngine`: RETE algorithm for rule matching - -**Complete Code Example**: - -```python -from semantica.reasoning import InferenceEngine, RuleManager - -inference_engine = InferenceEngine() -rule_manager = RuleManager() - -# Define rules -rules = [ - "IF Person worksFor Company AND Company locatedIn City THEN Person livesIn City", - "IF Person hasFriend Person2 AND Person2 hasFriend Person3 THEN Person knows Person3" -] - -rule_manager.add_rules(rules) - -# Perform inference -new_facts = inference_engine.forward_chain(kg, rule_manager) -print(f"Inferred {len(new_facts)} new facts") - -# Get explanations -for fact in new_facts: - explanation = inference_engine.explain(fact) - print(f"{fact}: {explanation}") -``` - -**Configuration Options**: - -| Option | Type | Default | Description | -| :-------------- | :----- | :---------------- | :--------------------------- | -| `method` | `str` | `"forward_chain"` | Reasoning method | -| `max_iterations`| `int` | `100` | Maximum inference iterations | - -**API Reference**: [Reasoning Module](reference/reasoning.md) - -### 10. Ontology Module - -!!! abstract "Purpose" - Defines the schema and structure of your knowledge domain, ensuring consistency and enabling interoperability. - -**Key Features**: - Automatic ontology generation (6-stage pipeline) -- OWL/RDF export +- OWL/RDF/Turtle export - Class and property inference - Ontology validation - Symbolic reasoning (HermiT, Pellet) - Version management +- SHACL constraint support +- Ontology merging and alignment -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :-------------- | :---------------- | :----------------------------------- | -| `knowledge_graph`| `KnowledgeGraph` | Input knowledge graph | -| `base_uri` | `str` | Base URI for ontology | +- `OntologyGenerator` — Generate ontologies from knowledge graphs +- `OntologyValidator` — Validate ontology structure +- `OWLGenerator` — Generate OWL format ontologies +- `PropertyGenerator` — Generate ontology properties +- `ClassInferrer` — Infer ontology classes +- `OntologyMerger` — Merge multiple ontologies +- `ReasonerInterface` — Interface with symbolic reasoners -| Output | Type | Description | -| :---------- | :---------- | :----------------------------------- | -| `ontology` | `Ontology` | Generated ontology | -| `owl_content`| `str` | OWL/Turtle format | - -**Components**: -- `OntologyGenerator`: Generate ontologies from knowledge graphs -- `OntologyValidator`: Validate ontology structure -- `OWLGenerator`: Generate OWL format ontologies -- `PropertyGenerator`: Generate ontology properties -- `ClassInferrer`: Infer ontology classes - -**Complete Code Example**: +**Quick Example:** ```python from semantica.ontology import OntologyGenerator @@ -481,66 +354,94 @@ from semantica.ontology import OntologyGenerator generator = OntologyGenerator(base_uri="https://example.org/ontology/") ontology = generator.generate_from_graph(kg) -# Validate ontology -validator = generator.validate(ontology) -print(f"Valid: {validator.is_valid}") - # Export to OWL owl_content = generator.export_owl(ontology, format="turtle") print(f"Generated {len(owl_content)} lines of OWL") ``` -**Configuration Options**: - -| Option | Type | Default | Description | -| :-------- | :----- | :--------- | :--------------------------- | -| `base_uri`| `str` | `None` | Base URI for ontology | -| `reasoner`| `str` | `"hermit"` | Reasoner (hermit/pellet) | - **API Reference**: [Ontology Module](reference/ontology.md) --- -## Storage & Embeddings Layer - -These modules handle the persistence and retrieval of data, both as vectors and as graphs. - -### 6. Embeddings Module +### Reasoning Module !!! abstract "Purpose" - Embeddings are crucial for semantic search. This module generates vectors for text, images, and graph nodes. + Infers new facts and validates existing knowledge using logical rules. Supports forward/backward chaining and explanation generation. -**Key Features**: -- Multiple provider support (OpenAI, Cohere, HuggingFace) +**Key Features:** + +- Forward and backward chaining +- Rule-based inference +- Deductive and abductive reasoning +- Explanation generation +- RETE algorithm support +- Custom rule definition +- Conflict detection in inferences +- Temporal reasoning + +**Components:** + +- `InferenceEngine` — Main inference orchestrator +- `RuleManager` — Manage inference rules +- `DeductiveReasoner` — Deductive reasoning +- `AbductiveReasoner` — Abductive reasoning +- `ExplanationGenerator` — Generate explanations for inferences +- `RETEEngine` — RETE algorithm for rule matching + +**Quick Example:** + +```python +from semantica.reasoning import InferenceEngine, RuleManager + +inference_engine = InferenceEngine() +rule_manager = RuleManager() + +rules = [ + "IF Person worksFor Company AND Company locatedIn City THEN Person livesIn City", + "IF Person hasFriend Person2 AND Person2 hasFriend Person3 THEN Person knows Person3" +] +rule_manager.add_rules(rules) + +new_facts = inference_engine.forward_chain(kg, rule_manager) +print(f"Inferred {len(new_facts)} new facts") +``` + +**API Reference**: [Reasoning Module](reference/reasoning.md) + +--- + +## Storage Layer + +These modules handle persistence and retrieval of vectors, graphs, and triples. + +--- + +### Embeddings Module + +!!! abstract "Purpose" + Generates vector embeddings for text, images, and audio. Supports multiple providers with caching and batch processing. + +**Key Features:** + +- Multiple provider support (OpenAI, Cohere, HuggingFace, Sentence Transformers) - Text, image, and audio embeddings - Multimodal embeddings - Batch processing - Caching support - Custom models +- Similarity calculations -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :-------- | :------------ | :----------------------------------- | -| `texts` | `List[str]` | Text to embed | -| `provider`| `str` | Embedding provider (openai/cohere/hf)| -| `model` | `str` | Model name | +- `EmbeddingGenerator` — Main embedding orchestrator +- `TextEmbedder` — Generate text embeddings +- `ImageEmbedder` — Generate image embeddings +- `AudioEmbedder` — Generate audio embeddings +- `MultimodalEmbedder` — Combine multiple modalities +- `EmbeddingOptimizer` — Optimize embedding quality +- `ProviderAdapters` — Support for OpenAI, Cohere, etc. -| Output | Type | Description | -| :--------- | :------------ | :----------------------------------- | -| `embeddings`| `np.ndarray` | Array of embedding vectors | -| `metadata` | `Dict` | Embedding metadata | - -**Components**: -- `EmbeddingGenerator`: Main embedding orchestrator -- `TextEmbedder`: Generate text embeddings -- `ImageEmbedder`: Generate image embeddings -- `AudioEmbedder`: Generate audio embeddings -- `MultimodalEmbedder`: Combine multiple modalities -- `EmbeddingOptimizer`: Optimize embedding quality -- `ProviderAdapters`: Support for OpenAI, Cohere, etc. - -**Complete Code Example**: +**Quick Example:** ```python from semantica.embeddings import EmbeddingGenerator @@ -556,49 +457,35 @@ similarity = generator.similarity(embeddings[0], embeddings[1]) print(f"Similarity: {similarity:.3f}") ``` -**Configuration Options**: - -| Option | Type | Default | Description | -| :---------- | :----- | :---------- | :--------------------------- | -| `provider` | `str` | `"openai"` | Embedding provider | -| `model` | `str` | `None` | Model name | -| `batch_size`| `int` | `100` | Batch size for processing | -| `cache` | `bool` | `True` | Enable caching | - **API Reference**: [Embeddings Module](reference/embeddings.md) -### 7. Vector Store Module +--- + +### Vector Store Module !!! abstract "Purpose" - Manages the storage and retrieval of high-dimensional vectors, supporting hybrid search strategies. + Manages storage and retrieval of high-dimensional vectors. Supports hybrid search combining vector and keyword search. -**Key Features**: -- Multiple backend support (FAISS, Pinecone, Weaviate) +**Key Features:** + +- Multiple backend support (FAISS, Pinecone, Weaviate, Qdrant, Milvus) - Hybrid search (vector + keyword) - Metadata filtering - Batch operations -- Similarity search +- Similarity search with scoring - Index management +- Namespace support -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :--------- | :---------------- | :----------------------------------- | -| `embeddings`| `np.ndarray` | Vector embeddings | -| `documents` | `List[Document]` | Associated documents | -| `metadata` | `Dict` | Metadata for filtering | +- `VectorStore` — Main vector store interface +- `FAISSAdapter` — FAISS integration +- `PineconeAdapter` — Pinecone integration +- `WeaviateAdapter` — Weaviate integration +- `HybridSearch` — Combine vector and keyword search +- `VectorRetriever` — Retrieve relevant vectors -| Output | Type | Description | -| :------- | :------------------ | :----------------------------------- | -| `results`| `List[SearchResult]`| Search results with scores | - -**Components**: -- `VectorStore`: Main vector store interface -- `FAISSAdapter`: FAISS integration -- `HybridSearch`: Combine vector and keyword search -- `VectorRetriever`: Retrieve relevant vectors - -**Complete Code Example**: +**Quick Example:** ```python from semantica.vector_store import VectorStore, HybridSearch @@ -613,265 +500,498 @@ results = hybrid_search.search( top_k=10, filters={"category": "AI"} ) - -for result in results: - print(f"Score: {result.score:.3f} - {result.document.content[:50]}...") ``` -**Configuration Options**: - -| Option | Type | Default | Description | -| :------------- | :----- | :-------- | :--------------------------- | -| `backend` | `str` | `"faiss"` | Vector store backend | -| `index_type` | `str` | `"flat"` | FAISS index type | -| `hybrid_search`| `bool` | `True` | Enable hybrid search | - **API Reference**: [Vector Store Module](reference/vector_store.md) -### 8. Graph Store Module +--- + +### Graph Store Module !!! abstract "Purpose" - The `graph_store` module provides integration with property graph databases like Neo4j, KuzuDB, and FalkorDB for storing and querying knowledge graphs. + Integration with property graph databases for storing and querying knowledge graphs. + +**Key Features:** -**Key Features**: - Multiple backend support (Neo4j, KuzuDB, FalkorDB) - Cypher query language - Graph algorithms and analytics - Transaction support - Index management - High-performance queries +- Batch operations -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :------------- | :----- | :----------------------------------- | -| `backend` | `str` | Backend (neo4j/kuzu/falkordb) | -| `connection_uri`| `str` | Database connection URI | -| `cypher_query` | `str` | Cypher query string | +- `GraphStore` — Main graph store interface +- `Neo4jAdapter` — Neo4j database integration +- `KuzuAdapter` — KuzuDB embedded database integration +- `FalkorDBAdapter` — FalkorDB (Redis-based) integration +- `NodeManager` — Node CRUD operations +- `RelationshipManager` — Relationship CRUD operations +- `QueryEngine` — Cypher query execution +- `GraphAnalytics` — Graph algorithms and analytics -| Output | Type | Description | -| :------- | :----- | :----------------------------------- | -| `result` | `Dict` | Query results | -| `node_id`| `str` | Created node ID | - -**Components**: -- `GraphStore`: Main graph store interface -- `Neo4jAdapter`: Neo4j database integration -- `KuzuAdapter`: KuzuDB embedded database integration -- `FalkorDBAdapter`: FalkorDB (Redis-based) integration -- `NodeManager`: Node CRUD operations -- `RelationshipManager`: Relationship CRUD operations -- `QueryEngine`: Cypher query execution -- `GraphAnalytics`: Graph algorithms and analytics - -**Complete Code Example**: +**Quick Example:** ```python -from semantica.graph_store import GraphStore, create_node, create_relationship +from semantica.graph_store import GraphStore -# Using GraphStore class store = GraphStore(backend="neo4j", uri="bolt://localhost:7687") store.connect() -# Create nodes +# Create nodes and relationships alice = store.create_node(["Person"], {"name": "Alice", "age": 30}) bob = store.create_node(["Person"], {"name": "Bob", "age": 25}) - -# Create relationship store.create_relationship(alice["id"], bob["id"], "KNOWS", {"since": 2020}) # Query with Cypher results = store.execute_query("MATCH (p:Person) RETURN p.name") -print(f"Found {len(results)} people") - -# Graph analytics -path = store.shortest_path(alice["id"], bob["id"]) -print(f"Shortest path: {path}") - -# Or use convenience functions -node = create_node(labels=["Entity"], properties={"name": "Test"}) ``` -**Configuration Options**: - -| Option | Type | Default | Description | -| :--------- | :----- | :-------- | :--------------------------- | -| `backend` | `str` | `"neo4j"` | Graph database backend | -| `uri` | `str` | `None` | Database connection URI | -| `database` | `str` | `"neo4j"` | Database name | - **API Reference**: [Graph Store Module](reference/graph_store.md) --- -## Output Layer - -These modules handle the export and visualization of your data. - -### 11. Export Module +### Triple Store Module !!! abstract "Purpose" - Allows you to take your knowledge graph and data out of Semantica for use in other tools. + RDF triple store integration for semantic web applications. Supports SPARQL queries and multiple backends. -**Key Features**: -- Multiple export formats (JSON, RDF, CSV, OWL, GraphML) +**Key Features:** + +- Multi-backend support (Blazegraph, Jena, RDF4J, Virtuoso) +- CRUD operations for RDF triples +- SPARQL query execution and optimization +- Bulk data loading with progress tracking +- Query caching and optimization +- Transaction support +- Store adapter pattern + +**Components:** + +- `TripleManager` — Main triple store management coordinator +- `QueryEngine` — SPARQL query execution and optimization +- `BulkLoader` — High-volume data loading with progress tracking +- `BlazegraphAdapter` — Blazegraph integration +- `JenaAdapter` — Apache Jena integration +- `RDF4JAdapter` — Eclipse RDF4J integration +- `VirtuosoAdapter` — Virtuoso RDF store integration +- `QueryPlan` — Query execution plan dataclass +- `LoadProgress` — Bulk loading progress tracking + +**Algorithms:** + +| Category | Algorithms | +| :--- | :--- | +| **Query Optimization** | Cost estimation, query rewriting, LIMIT injection | +| **Caching** | MD5-based cache keys, LRU eviction | +| **Bulk Loading** | Batch processing, retry with exponential backoff | + +**Quick Example:** + +```python +from semantica.triple_store import TripleManager, execute_query + +manager = TripleManager() +store = manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph") + +# Add triple +result = manager.add_triple({ + "subject": "http://example.org/Alice", + "predicate": "http://example.org/knows", + "object": "http://example.org/Bob" +}, store_id="main") + +# Execute SPARQL +query_result = execute_query("SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10", store) +``` + +**API Reference**: [Triple Store Module](reference/triple_store.md) + +--- + +## Quality Assurance Layer + +These modules ensure data quality, handle duplicates, and resolve conflicts. + +--- + +### Deduplication Module + +!!! abstract "Purpose" + Comprehensive entity deduplication and merging. Detects duplicates using multiple similarity methods and merges them intelligently. + +**Key Features:** + +- Multiple similarity methods (exact, Levenshtein, Jaro-Winkler, cosine, embedding) +- Duplicate detection with confidence scoring +- Entity merging with configurable strategies +- Cluster-based batch deduplication +- Provenance preservation during merges +- Relationship preservation +- Incremental processing support + +**Components:** + +- `DuplicateDetector` — Detects duplicate entities using similarity metrics +- `EntityMerger` — Merges duplicate entities using configurable strategies +- `SimilarityCalculator` — Multi-factor similarity between entities +- `MergeStrategyManager` — Manages merge strategies and conflict resolution +- `ClusterBuilder` — Builds clusters for batch deduplication + +**Merge Strategies:** + +| Strategy | Description | +| :--- | :--- | +| `keep_first` | Preserve first entity, merge others | +| `keep_last` | Preserve last entity, merge others | +| `keep_most_complete` | Preserve entity with most properties | +| `keep_highest_confidence` | Preserve entity with highest confidence | +| `merge_all` | Combine all properties and relationships | + +**Quick Example:** + +```python +from semantica.deduplication import deduplicate, DuplicateDetector + +# Using convenience function +result = deduplicate( + entities, + similarity_threshold=0.8, + merge_strategy="keep_most_complete" +) +print(f"Reduced from {result['statistics']['total_entities']} to {result['statistics']['final_entities']}") + +# Using classes directly +detector = DuplicateDetector(similarity_threshold=0.8) +duplicates = detector.detect_duplicates(entities) +``` + +--- + +### Conflicts Module + +!!! abstract "Purpose" + Detects and resolves conflicts from multiple data sources. Provides investigation guides and source tracking. + +**Key Features:** + +- Multi-source conflict detection (value, type, relationship, temporal, logical) +- Source tracking and provenance management +- Conflict analysis and pattern identification +- Multiple resolution strategies (voting, credibility-weighted, recency) +- Investigation guide generation +- Source credibility scoring +- Conflict reporting and statistics + +**Components:** + +- `ConflictDetector` — Detects conflicts from multiple sources +- `ConflictResolver` — Resolves conflicts using various strategies +- `ConflictAnalyzer` — Analyzes conflict patterns and trends +- `SourceTracker` — Tracks source information and provenance +- `InvestigationGuideGenerator` — Generates investigation guides + +**Resolution Strategies:** + +| Strategy | Algorithm | +| :--- | :--- | +| **Voting** | Majority value selection using frequency counting | +| **Credibility Weighted** | Weighted average using source credibility scores | +| **Temporal Selection** | Newest/oldest value based on timestamps | +| **Confidence Selection** | Maximum confidence value selection | + +**Quick Example:** + +```python +from semantica.conflicts import detect_and_resolve, ConflictDetector + +# Using convenience function +conflicts, results = detect_and_resolve( + entities, + property_name="name", + resolution_strategy="voting" +) + +# Using classes directly +detector = ConflictDetector() +conflicts = detector.detect_value_conflicts(entities, "name") +``` + +--- + +### KG Quality Assurance Module + +!!! abstract "Purpose" + Comprehensive quality assessment, validation, and automated fixes for knowledge graphs. + +**Key Features:** + +- Quality metrics calculation (overall, completeness, consistency) +- Consistency checking (logical, temporal, hierarchical) +- Completeness validation (entity, relationship, property) +- Automated fixes (duplicates, inconsistencies, missing properties) +- Quality reporting with issue tracking +- Validation engine with rules and constraints +- Improvement suggestions + +**Components:** + +- `KGQualityAssessor` — Overall quality assessment coordinator +- `ConsistencyChecker` — Consistency validation engine +- `CompletenessValidator` — Completeness validation engine +- `QualityMetrics` — Quality metrics calculator +- `ValidationEngine` — Rule and constraint validation +- `RuleValidator` — Rule-based validation +- `ConstraintValidator` — Constraint-based validation +- `QualityReporter` — Quality report generation +- `IssueTracker` — Issue tracking and management +- `ImprovementSuggestions` — Improvement suggestions generator +- `AutomatedFixer` — Automated issue fixing +- `AutoMerger` — Automatic merging of duplicates +- `AutoResolver` — Automatic conflict resolution + +**Quality Metrics:** + +| Metric | Calculation | +| :--- | :--- | +| **Overall Score** | `(0.6 × completeness) + (0.4 × consistency)` | +| **Entity Quality** | Required field presence (ID, type) | +| **Relationship Quality** | Required field presence (source, target, type) | + +**Quick Example:** + +```python +from semantica.kg_qa import assess_quality, generate_quality_report, KGQualityAssessor + +# Using convenience functions +score = assess_quality(knowledge_graph) +report = generate_quality_report(knowledge_graph, schema) + +# Using classes directly +assessor = KGQualityAssessor() +score = assessor.assess_overall_quality(knowledge_graph) +``` + +--- + +## Context & Memory Layer + +These modules provide context engineering for agents and foundation data management. + +--- + +### Context Module + +!!! abstract "Purpose" + Context engineering infrastructure for agents. Formalizes context as a graph of connections with RAG-enhanced memory. + +**Key Features:** + +- Context graph construction from entities, relationships, and conversations +- Agent memory management with RAG integration +- Entity linking across sources with URI assignment +- Hybrid context retrieval (vector + graph + memory) +- Conversation history management +- Context accumulation and synthesis +- Graph-based context traversal + +**Components:** + +- `ContextGraphBuilder` — Builds context graphs from various sources +- `ContextNode` — Context graph node data structure +- `ContextEdge` — Context graph edge data structure +- `AgentMemory` — Manages persistent agent memory with RAG +- `MemoryItem` — Memory item data structure +- `EntityLinker` — Links entities across sources with URIs +- `ContextRetriever` — Retrieves relevant context from multiple sources + +**Algorithms:** + +| Category | Algorithms | +| :--- | :--- | +| **Graph Construction** | BFS/DFS traversal, type-based indexing | +| **Memory Management** | Vector embedding, similarity search, retention policies | +| **Context Retrieval** | Vector similarity, multi-hop graph expansion, hybrid scoring | +| **Entity Linking** | Hash-based URI generation, text similarity matching | + +**Quick Example:** + +```python +from semantica.context import build_context, ContextGraphBuilder, AgentMemory + +# Using convenience function +result = build_context( + entities=entities, + relationships=relationships, + vector_store=vs, + knowledge_graph=kg +) + +# Using classes directly +builder = ContextGraphBuilder() +graph = builder.build_from_entities_and_relationships(entities, relationships) + +memory = AgentMemory(vector_store=vs, knowledge_graph=kg) +memory_id = memory.store("User asked about Python", metadata={"type": "conversation"}) +results = memory.retrieve("Python", max_results=5) +``` + +--- + +### Seed Module + +!!! abstract "Purpose" + Seed data management for initial knowledge graph construction. Builds on verified knowledge from multiple sources. + +**Key Features:** + +- Multi-source seed data loading (CSV, JSON, Database, API) +- Foundation graph creation from seed data +- Seed data quality validation +- Integration with extracted data using configurable merge strategies +- Version management for seed sources +- Export capabilities (JSON, CSV) +- Schema template validation + +**Components:** + +- `SeedDataManager` — Main coordinator for seed data operations +- `SeedDataSource` — Seed data source definition +- `SeedData` — Seed data container + +**Merge Strategies:** + +| Strategy | Description | +| :--- | :--- | +| `seed_first` | Seed data takes precedence, extracted fills gaps | +| `extracted_first` | Extracted data takes precedence, seed fills gaps | +| `merge` | Property merging, seed takes precedence for conflicts | + +**Quick Example:** + +```python +from semantica.seed import SeedDataManager + +manager = SeedDataManager() +manager.register_source("entities", "json", "data/entities.json") +foundation = manager.create_foundation_graph() +validation = manager.validate_quality(foundation) +``` + +--- + +## Output & Orchestration Layer + +These modules handle export, visualization, and workflow management. + +--- + +### Export Module + +!!! abstract "Purpose" + Export knowledge graphs and data to various formats for use in external tools. + +**Key Features:** + +- Multiple export formats (JSON, RDF, CSV, OWL, GraphML, GEXF) - Custom export formats - Batch export - Metadata preservation - Streaming export for large graphs +- Vector export support -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :----------- | :---------------- | :----------------------------------- | -| `knowledge_graph`| `KnowledgeGraph`| Graph to export | -| `format` | `str` | Export format | -| `output_path`| `str` | Output file path | +- `JSONExporter` — Export to JSON +- `RDFExporter` — Export to RDF/XML +- `CSVExporter` — Export to CSV +- `GraphExporter` — Export to graph formats (GraphML, GEXF) +- `OWLExporter` — Export to OWL +- `VectorExporter` — Export vectors -| Output | Type | Description | -| :------------ | :----- | :----------------------------------- | -| `exported_file`| `str` | Path to exported file | -| `metadata` | `Dict` | Export metadata | - -**Components**: -- `JSONExporter`: Export to JSON -- `RDFExporter`: Export to RDF/XML -- `CSVExporter`: Export to CSV -- `GraphExporter`: Export to graph formats (GraphML, GEXF) -- `OWLExporter`: Export to OWL -- `VectorExporter`: Export vectors - -**Complete Code Example**: +**Quick Example:** ```python from semantica.export import JSONExporter, RDFExporter, CSVExporter -# Export to JSON -json_exporter = JSONExporter() -json_exporter.export(kg, "output.json") - -# Export to RDF -rdf_exporter = RDFExporter() -rdf_exporter.export(kg, "output.rdf") - -# Export to CSV -csv_exporter = CSVExporter() -csv_exporter.export(kg, "output.csv") +# Export to multiple formats +JSONExporter().export(kg, "output.json") +RDFExporter().export(kg, "output.rdf") +CSVExporter().export(kg, "output.csv") ``` -**Configuration Options**: - -| Option | Type | Default | Description | -| :--------------- | :----- | :--------- | :--------------------------- | -| `format` | `str` | `"json"` | Export format | -| `include_metadata`| `bool`| `True` | Include metadata | -| `pretty_print` | `bool` | `False` | Pretty print JSON | - **API Reference**: [Export Module](reference/export.md) -### 12. Visualization Module +--- + +### Visualization Module !!! abstract "Purpose" - Provides tools to visually explore your data, making it easier to understand complex relationships. + Visual exploration of knowledge graphs, embeddings, and analytics data. + +**Key Features:** -**Key Features**: - Interactive graph visualization - Embedding visualization (t-SNE, PCA, UMAP) - Quality metrics visualization - Temporal data visualization +- Ontology visualization - Multiple output formats (HTML, PNG, SVG) - Custom styling -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :-------------- | :---------------- | :----------------------------------- | -| `knowledge_graph`| `KnowledgeGraph` | Graph to visualize | -| `output_format` | `str` | Output format (html/png/svg) | -| `output_path` | `str` | Output file path | +- `KGVisualizer` — Visualize knowledge graphs +- `EmbeddingVisualizer` — Visualize embeddings (t-SNE, PCA, UMAP) +- `QualityVisualizer` — Visualize quality metrics +- `AnalyticsVisualizer` — Visualize graph analytics +- `TemporalVisualizer` — Visualize temporal data +- `OntologyVisualizer` — Visualize ontology structure +- `SemanticNetworkVisualizer` — Visualize semantic networks -| Output | Type | Description | -| :----------------- | :----- | :----------------------------------- | -| `visualization_file`| `str` | Path to visualization file | - -**Components**: -- `KGVisualizer`: Visualize knowledge graphs -- `EmbeddingVisualizer`: Visualize embeddings (t-SNE, PCA, UMAP) -- `QualityVisualizer`: Visualize quality metrics -- `AnalyticsVisualizer`: Visualize graph analytics -- `TemporalVisualizer`: Visualize temporal data - -**Complete Code Example**: +**Quick Example:** ```python from semantica.visualization import KGVisualizer, EmbeddingVisualizer # Visualize knowledge graph -kg_visualizer = KGVisualizer() -kg_visualizer.visualize( - kg, - output_format="html", - output_path="graph.html" -) +KGVisualizer().visualize(kg, output_format="html", output_path="graph.html") # Visualize embeddings -embed_visualizer = EmbeddingVisualizer() -embed_visualizer.visualize( - embeddings, - method="tsne", - output_path="embeddings.png" -) +EmbeddingVisualizer().visualize(embeddings, method="tsne", output_path="embeddings.png") ``` -**Configuration Options**: - -| Option | Type | Default | Description | -| :------------- | :----- | :--------- | :--------------------------- | -| `output_format`| `str` | `"html"` | Output format | -| `layout` | `str` | `"force"` | Graph layout algorithm | -| `max_nodes` | `int` | `1000` | Maximum nodes to visualize | - **API Reference**: [Visualization Module](reference/visualization.md) --- -## Orchestration - -This layer connects all other modules into a cohesive workflow. - -### 13. Pipeline Module +### Pipeline Module !!! abstract "Purpose" - Orchestrates the entire flow, connecting modules together into robust, executable workflows. + Orchestrates workflows, connecting modules into robust, executable pipelines. + +**Key Features:** -**Key Features**: - Pipeline construction DSL - Parallel execution - Error handling and recovery - Resource scheduling - Pipeline validation - Monitoring and logging +- Checkpoint support -**Input/Output Specification**: +**Components:** -| Input | Type | Description | -| :-------- | :------------------ | :----------------------------------- | -| `steps` | `List[PipelineStep]`| Pipeline steps | -| `parallel`| `bool` | Enable parallel execution | +- `PipelineBuilder` — Build complex pipelines +- `ExecutionEngine` — Execute pipelines +- `FailureHandler` — Handle pipeline failures +- `ParallelismManager` — Enable parallel processing +- `ResourceScheduler` — Schedule resources +- `PipelineValidator` — Validate pipeline configuration -| Output | Type | Description | -| :------- | :---------------- | :----------------------------------- | -| `result` | `PipelineResult` | Execution result | -| `metrics`| `Dict` | Performance metrics | - -**Components**: -- `PipelineBuilder`: Build complex pipelines -- `ExecutionEngine`: Execute pipelines -- `FailureHandler`: Handle pipeline failures -- `ParallelismManager`: Enable parallel processing -- `ResourceScheduler`: Schedule resources - -**Complete Code Example**: +**Quick Example:** ```python from semantica.pipeline import PipelineBuilder @@ -886,33 +1006,20 @@ pipeline = builder \ .add_step("extract", NERExtractor()) \ .build() -# Execute pipeline result = pipeline.execute(sources=["data/"], parallel=True) -print(f"Processed {len(result.documents)} documents") ``` -**Configuration Options**: - -| Option | Type | Default | Description | -| :--------------- | :----- | :------ | :--------------------------- | -| `parallel` | `bool` | `False` | Enable parallel execution | -| `max_workers` | `int` | `4` | Maximum parallel workers | -| `retry_on_failure`| `bool`| `True` | Retry failed steps | - **API Reference**: [Pipeline Module](reference/pipeline.md) --- ## Integration Patterns -This section shows common patterns for integrating multiple modules together. - ### Pattern 1: Complete Knowledge Graph Pipeline ```python from semantica import Semantica -# High-level API - all modules integrated semantica = Semantica() result = semantica.build_knowledge_base( sources=["documents/"], @@ -927,60 +1034,51 @@ result = semantica.build_knowledge_base( ```python from semantica.ingest import FileIngestor from semantica.parse import DocumentParser +from semantica.split import TextSplitter from semantica.normalize import TextNormalizer from semantica.semantic_extract import NERExtractor, RelationExtractor from semantica.kg import GraphBuilder -from semantica.embeddings import EmbeddingGenerator -from semantica.vector_store import VectorStore +from semantica.deduplication import deduplicate -# Step-by-step pipeline -ingestor = FileIngestor() -documents = ingestor.ingest("data/") +# Ingest and parse +documents = FileIngestor().ingest("data/") +parsed = DocumentParser().parse(documents) -parser = DocumentParser() -parsed = parser.parse(documents) +# Split and normalize +chunks = TextSplitter(method="entity_aware").split(parsed) +normalized = TextNormalizer().normalize(chunks) -normalizer = TextNormalizer() -normalized = normalizer.normalize(parsed) +# Extract and build +entities = NERExtractor().extract(normalized) +relationships = RelationExtractor().extract(normalized, entities) +kg = GraphBuilder().build(entities, relationships) -extractor = NERExtractor() -entities = extractor.extract(normalized) - -rel_extractor = RelationExtractor() -relationships = rel_extractor.extract(normalized, entities) - -builder = GraphBuilder() -kg = builder.build(entities, relationships) - -generator = EmbeddingGenerator() -embeddings = generator.generate(normalized) - -vector_store = VectorStore() -vector_store.store(embeddings, normalized) +# Quality assurance +deduplicated = deduplicate(entities) ``` -### Pattern 3: GraphRAG Integration +### Pattern 3: GraphRAG with Hybrid Search ```python from semantica import Semantica -from semantica.kg import GraphBuilder from semantica.vector_store import VectorStore, HybridSearch +from semantica.context import AgentMemory -# Build knowledge base semantica = Semantica() result = semantica.build_knowledge_base(["documents/"]) -# Set up GraphRAG -kg = result["knowledge_graph"] -embeddings = result["embeddings"] vector_store = VectorStore() -vector_store.store(embeddings, result["documents"]) +vector_store.store(result["embeddings"], result["documents"]) -# Query with GraphRAG +# Agent memory with RAG +memory = AgentMemory(vector_store=vector_store, knowledge_graph=result["knowledge_graph"]) +memory.store("User query about AI", metadata={"type": "query"}) + +# Hybrid search hybrid_search = HybridSearch(vector_store) results = hybrid_search.search( query="What is the relationship between X and Y?", - graph=kg, + graph=result["knowledge_graph"], top_k=10 ) ``` @@ -998,74 +1096,46 @@ kg = builder.build(entities, relationships) # Add reasoning inference_engine = InferenceEngine() rule_manager = RuleManager() -rules = ["IF A THEN B"] -rule_manager.add_rules(rules) +rule_manager.add_rules(["IF A THEN B"]) new_facts = inference_engine.forward_chain(kg, rule_manager) ``` --- -## Performance Considerations +## Quick Reference: All Modules -### Module Performance Characteristics - -| Module | Typical Speed | Memory Usage | Scalability | Optimization Tips | -| :--- | :--- | :--- | :--- | :--- | -| **Ingest** | Fast | Low | High | Batch processing, parallel I/O | -| **Parse** | Medium | Medium | Medium | OCR only when needed, cache results | -| **Normalize** | Fast | Low | High | Batch processing | -| **Semantic Extract** | Slow (LLM) | Medium | Medium | Use faster models, batch processing | -| **KG** | Fast | High (large graphs) | Medium | Use graph stores for large graphs | -| **Embeddings** | Slow (API) | Low | High | Batch processing, caching | -| **Vector Store** | Fast | Medium | High | Use FAISS for large datasets | -| **Graph Store** | Fast | Low | High | Index optimization, query tuning | -| **Reasoning** | Slow | Medium | Low | Limit iterations, optimize rules | -| **Ontology** | Medium | Medium | Medium | Cache validation results | -| **Export** | Fast | Low | High | Streaming for large graphs | -| **Visualization** | Slow | High | Low | Limit nodes, use sampling | -| **Pipeline** | Varies | Varies | High | Parallel execution, resource limits | - -### Optimization Strategies - -1. **Batch Processing**: Process multiple documents together - ```python - # Good: Batch processing - documents = ingestor.ingest("data/", batch_size=100) - ``` - -2. **Caching**: Cache expensive operations - ```python - # Cache embeddings - generator = EmbeddingGenerator(cache=True) - ``` - -3. **Parallel Execution**: Use pipeline parallelism - ```python - pipeline = builder.build() - result = pipeline.execute(sources=sources, parallel=True, max_workers=8) - ``` - -4. **Backend Selection**: Choose appropriate backends - ```python - # For large graphs, use Neo4j instead of NetworkX - builder = GraphBuilder(backend="neo4j") - ``` - -5. **Resource Limits**: Set appropriate limits - ```python - # Limit memory usage - parser = DocumentParser(max_file_size=10_000_000) # 10MB - ``` +| Module | Import | Main Class | Purpose | +| :--- | :--- | :--- | :--- | +| **Ingest** | `semantica.ingest` | `FileIngestor` | Data ingestion | +| **Parse** | `semantica.parse` | `DocumentParser` | Document parsing | +| **Split** | `semantica.split` | `TextSplitter` | Text chunking | +| **Normalize** | `semantica.normalize` | `TextNormalizer` | Data cleaning | +| **Semantic Extract** | `semantica.semantic_extract` | `NERExtractor` | Entity extraction | +| **KG** | `semantica.kg` | `GraphBuilder` | Graph construction | +| **Ontology** | `semantica.ontology` | `OntologyGenerator` | Ontology generation | +| **Reasoning** | `semantica.reasoning` | `InferenceEngine` | Logical inference | +| **Embeddings** | `semantica.embeddings` | `EmbeddingGenerator` | Vector generation | +| **Vector Store** | `semantica.vector_store` | `VectorStore` | Vector storage | +| **Graph Store** | `semantica.graph_store` | `GraphStore` | Graph database | +| **Triple Store** | `semantica.triple_store` | `TripleManager` | RDF storage | +| **Deduplication** | `semantica.deduplication` | `DuplicateDetector` | Duplicate removal | +| **Conflicts** | `semantica.conflicts` | `ConflictDetector` | Conflict resolution | +| **KG QA** | `semantica.kg_qa` | `KGQualityAssessor` | Quality assurance | +| **Context** | `semantica.context` | `AgentMemory` | Agent context | +| **Seed** | `semantica.seed` | `SeedDataManager` | Foundation data | +| **Export** | `semantica.export` | `JSONExporter` | Data export | +| **Visualization** | `semantica.visualization` | `KGVisualizer` | Visualization | +| **Pipeline** | `semantica.pipeline` | `PipelineBuilder` | Workflow orchestration | --- ## Next Steps -- **[Core Concepts](concepts.md)** - Understand the fundamental concepts -- **[Use Cases](use-cases.md)** - See real-world applications -- **[Examples](examples.md)** - Practical code examples -- **[API Reference](reference/core.md)** - Detailed API documentation +- **[Core Concepts](concepts.md)** — Understand the fundamental concepts +- **[Use Cases](use-cases.md)** — See real-world applications +- **[Examples](examples.md)** — Practical code examples +- **[API Reference](reference/core.md)** — Detailed API documentation ---