diff --git a/docs/getting-started.md b/docs/getting-started.md index 51f72832..db7e96ed 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -56,7 +56,7 @@ Before installing Semantica, ensure you have: === "Source (Dev)" ```bash - git clone https://github.com/your-org/semantica.git + git clone https://github.com/Hawksight-AI/semantica.git cd semantica pip install -e . ``` diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 00000000..66fc8b93 --- /dev/null +++ b/docs/glossary.md @@ -0,0 +1,213 @@ +# Glossary + +A comprehensive reference of terms and concepts used in Semantica. + +--- + +## A + +**Agent** +: An autonomous AI system that can perceive its environment, reason about information, and take actions to achieve specific goals. In Semantica, agents use knowledge graphs for memory and reasoning. + +**API (Application Programming Interface)** +: A set of functions and protocols that allow different software applications to communicate with each other. + +**Axiom** +: A statement or rule that is accepted as true without proof, used in ontologies to define logical constraints and relationships. + +--- + +## C + +**Centrality** +: A measure of the importance or influence of a node in a graph. Common centrality metrics include PageRank, betweenness centrality, and closeness centrality. + +**Class** +: In ontologies, a category or type of entity (e.g., `Person`, `Organization`, `Location`). + +**Community Detection** +: The process of identifying groups or clusters of densely connected nodes in a graph. + +**Conflict Resolution** +: The process of handling contradictory information from multiple sources in a knowledge graph. + +**Coreference Resolution** +: The task of determining when two or more expressions in text refer to the same entity (e.g., "Apple" and "the company" referring to Apple Inc.). + +**Cypher** +: A declarative query language for graph databases, particularly Neo4j. + +--- + +## E + +**Embedding** +: A dense vector representation of text, images, or other data that captures semantic meaning in a continuous vector space. Used for similarity search and semantic matching. + +**Entity** +: A distinct object or concept in the real world, such as a person, place, organization, or event. + +**Entity Resolution** +: The process of determining when two entity mentions refer to the same real-world entity, also known as entity linking or deduplication. + +**Event Detection** +: The task of identifying and classifying events (e.g., acquisitions, partnerships, announcements) in text. + +--- + +## G + +**Graph** +: A data structure consisting of nodes (vertices) and edges (relationships) connecting them. + +**GraphRAG (Graph-Augmented Retrieval Augmented Generation)** +: An advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs. + +--- + +## H + +**Hybrid Search** +: A search strategy that combines multiple retrieval methods, typically vector search and keyword search, to improve accuracy. + +--- + +## I + +**Inference** +: The process of deriving new facts or conclusions from existing knowledge using logical rules. + +**Ingestion** +: The process of loading data from various sources (files, databases, APIs, streams) into a system for processing. + +--- + +## K + +**Knowledge Graph (KG)** +: A structured representation of entities and their relationships, typically stored as a graph with nodes representing entities and edges representing relationships. + +**Knowledge Graph Quality Assurance (KG QA)** +: The process of ensuring knowledge graph quality through completeness validation, consistency checking, and conflict detection. + +--- + +## L + +**LLM (Large Language Model)** +: A type of artificial intelligence model trained on vast amounts of text data, capable of understanding and generating human-like text. + +--- + +## N + +**Named Entity Recognition (NER)** +: The process of identifying and classifying named entities in text into predefined categories such as persons, organizations, locations, dates, and more. + +**Node** +: A vertex in a graph representing an entity or concept. + +**Normalization** +: The process of standardizing data into a consistent format (e.g., converting dates to ISO format, standardizing entity names). + +--- + +## O + +**OCR (Optical Character Recognition)** +: Technology that converts images of text (e.g., scanned documents, photos) into machine-readable text. + +**Ontology** +: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language). + +**OWL (Web Ontology Language)** +: A W3C standard language for defining and instantiating ontologies on the web. + +--- + +## P + +**PageRank** +: An algorithm used to measure the importance of nodes in a graph based on the structure of incoming links. + +**Pipeline** +: A sequence of data processing steps that transform raw data into a desired output format. + +**Property** +: In ontologies, a relationship or attribute that connects entities or describes their characteristics. + +**Provenance** +: Information about the origin, history, and lineage of data, including sources, timestamps, and transformations. + +--- + +## R + +**RAG (Retrieval Augmented Generation)** +: A technique that enhances LLM responses by retrieving relevant information from a knowledge base before generating an answer. + +**RDF (Resource Description Framework)** +: A W3C standard for representing information about resources in the form of subject-predicate-object triples. + +**Reasoning** +: The process of deriving new knowledge from existing facts using logical rules and inference. + +**Relationship Extraction** +: The task of identifying and extracting semantic relationships between entities in text. + +--- + +## S + +**Semantic** +: Relating to meaning in language or logic. + +**Semantic Layer** +: An abstraction layer that provides a unified, business-friendly view of data by adding context, relationships, and meaning to raw data. + +**Semantic Network** +: A knowledge representation that uses a graph structure to represent concepts and their relationships. + +**SPARQL** +: A query language for RDF data, similar to SQL for relational databases. + +--- + +## T + +**Temporal Graph** +: A knowledge graph that tracks changes over time, allowing queries about the state of the graph at specific time points. + +**Triple** +: A basic unit of knowledge in RDF, consisting of a subject, predicate, and object (e.g., ` `). + +**Triple Store** +: A database designed specifically for storing and querying RDF triples. + +--- + +## V + +**Vector** +: A mathematical representation of data as an array of numbers, used in embeddings to capture semantic meaning. + +**Vector Store** +: A database optimized for storing and searching high-dimensional vectors, used for semantic similarity search. + +**Visualization** +: The graphical representation of data, such as knowledge graphs, embeddings, or analytics. + +--- + +## W + +**Web Scraping** +: The automated process of extracting data from websites. + +--- + +## See Also + +- [Core Concepts](concepts.md) - Deep dive into fundamental concepts +- [Getting Started](getting-started.md) - Begin your journey with Semantica +- [API Reference](reference/core.md) - Technical documentation diff --git a/docs/index.md b/docs/index.md index 4a4cce8c..08457a99 100644 --- a/docs/index.md +++ b/docs/index.md @@ -93,7 +93,7 @@ flowchart TD --- - Vector search alone misses crucial relationships - No graph traversal for context expansion - - **30% lower accuracy** than hybrid approaches + - Significantly lower accuracy than hybrid approaches - Can't answer multi-hop questions - **🔴 AI Agents Hallucinate** @@ -180,7 +180,7 @@ flowchart TD ### Powers Next-Gen AI Applications -- **GraphRAG**: Hybrid retrieval combining vector search + graph traversal for 30% better accuracy +- **GraphRAG**: Hybrid retrieval combining vector search + graph traversal for improved accuracy - **AI Agents**: Ontology-constrained actions with semantic routing and persistent memory - **Multi-Agent Systems**: Shared semantic models for coordinated decision-making - **Knowledge Engineering**: Production-grade knowledge graphs with provenance and validation @@ -262,7 +262,7 @@ $ semantica query "Who founded Apple?" ```bash # Clone the repository - git clone https://github.com/semantica-dev/semantica.git + git clone https://github.com/Hawksight-AI/semantica.git cd semantica # Install in editable mode with dev dependencies @@ -371,7 +371,7 @@ Power GraphRAG applications with: - **Vector Search**: Semantic similarity using embeddings - **Graph Traversal**: Multi-hop reasoning for context expansion -- **Hybrid Retrieval**: Combine vector + graph for 30% better accuracy +- **Hybrid Retrieval**: Combine vector + graph for improved accuracy - **Temporal Queries**: Query knowledge at specific time points --- diff --git a/docs/installation.md b/docs/installation.md index e8238f0a..823b7c8f 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -218,8 +218,8 @@ It's recommended to use a virtual environment: | Component | Minimum | Recommended | |-----------|---------|-------------| | Python | 3.8 | 3.11+ | -| RAM | 4 GB | 8 GB+ | -| Disk Space | 2 GB | 5 GB+ | +| RAM | Moderate | Ample for your dataset | +| Disk Space | Sufficient for data | Generous storage | | OS | Windows/Linux/Mac | Linux/Mac | ## Next Steps diff --git a/docs/modules.md b/docs/modules.md index ed9d3502..b486bac1 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -56,8 +56,6 @@ The `ingest` module is the entry point for data. It handles the complexity of co - `EmailIngestor`: Process email messages - `RepoIngestor`: Git repository analysis - - `RepoIngestor`: Git repository analysis - ```mermaid classDiagram class BaseIngestor { @@ -106,8 +104,6 @@ Once data is ingested, the `parse` module extracts the raw text and metadata. It - `ImageParser`: OCR and image analysis - `CodeParser`: Parse source code files - - `CodeParser`: Parse source code files - ```mermaid classDiagram class DocumentParser { @@ -197,8 +193,6 @@ The `kg` module constructs the graph from extracted entities and relationships, - `TemporalQuery`: Query temporal knowledge graphs - `Deduplicator`: Remove duplicate entities/relationships - - `Deduplicator`: Remove duplicate entities/relationships - ```mermaid classDiagram class GraphBuilder { diff --git a/docs/reference/core.md b/docs/reference/core.md index 9acf3f49..d4bd2ef5 100644 --- a/docs/reference/core.md +++ b/docs/reference/core.md @@ -1,3 +1,427 @@ -# Core +# Core Module -::: semantica.core +> **The heart of Semantica - orchestrating all framework components with configuration, lifecycle, and plugin management.** + +--- + +## 🎯 Overview + +
+ +- :material-cog-outline:{ .lg .middle } **Framework Orchestration** + + --- + + Main `Semantica` class coordinating all operations and modules + +- :material-file-cog:{ .lg .middle } **Configuration Management** + + --- + + Centralized config with YAML/JSON support and validation + +- :material-timeline:{ .lg .middle } **Lifecycle Management** + + --- + + Startup, shutdown, health monitoring with hook system + +- :material-puzzle:{ .lg .middle } **Plugin System** + + --- + + Dynamic plugin discovery, loading, and isolation + +- :material-registry:{ .lg .middle } **Method Registry** + + --- + + Extensible method registration for custom implementations + +- :material-api:{ .lg .middle } **Convenience API** + + --- + + Simple `build()` function for quick knowledge base creation + +
+ +!!! example "Quick Start" + ```python + from semantica import Semantica + + # One-liner to build knowledge base + semantica = Semantica() + result = semantica.build_knowledge_base(["documents/"]) + + print(f"Nodes: {result['knowledge_graph'].node_count}") + ``` + +--- + +## ⚙️ Algorithms Used + +### Configuration Management +- **YAML/JSON Parsing**: Configuration file parsing with schema validation +- **Environment Variable Substitution**: `${VAR_NAME}` pattern replacement +- **Configuration Merging**: Deep merge algorithm for layered configs +- **Validation**: JSON Schema validation for type checking + +### Lifecycle Management +- **Hook System**: Observer pattern for lifecycle events +- **Health Monitoring**: Periodic health checks with exponential backoff +- **Graceful Shutdown**: Resource cleanup with timeout handling + +### Plugin System +- **Discovery**: File system scanning for plugin modules +- **Loading**: Dynamic import with dependency resolution +- **Isolation**: Separate namespace for each plugin + +--- + +## Main Classes + +### Semantica + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `__init__(config)` | Initialize framework | Configuration loading + validation | +| `build_knowledge_base(sources)` | Build knowledge graph | Orchestrate ingest → parse → extract → build | +| `run_pipeline(pipeline_config)` | Execute custom pipeline | DAG execution with error handling | +| `shutdown()` | Graceful shutdown | Resource cleanup with lifecycle hooks | +| `get_health_status()` | Get system health | Component health aggregation | + +**Example:** + +```python +from semantica import Semantica + +# Initialize with default config +semantica = Semantica() + +# Build knowledge base +result = semantica.build_knowledge_base( + sources=["documents/"], + extract_entities=True, + extract_relations=True, + embeddings=True, + graph=True +) + +kg = result["knowledge_graph"] +print(f"Nodes: {kg.node_count}, Edges: {kg.edge_count}") + +# Shutdown +semantica.shutdown() +``` + +--- + +### Config + + +**Configuration Structure:** + +```python +@dataclass +class Config: + llm: Optional[Dict[str, Any]] = None + embeddings: Optional[Dict[str, Any]] = None + knowledge_graph: Optional[Dict[str, Any]] = None + vector_store: Optional[Dict[str, Any]] = None + # ... other configuration fields +``` + +**Example:** + +```python +from semantica.core import Config + +config = Config( + llm={"provider": "openai", "model": "gpt-4"}, + embeddings={"provider": "openai", "model": "text-embedding-3-large"}, + knowledge_graph={"merge_entities": True, "resolve_conflicts": True} +) + +semantica = Semantica(config=config) +``` + +--- + +### ConfigManager + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `load_config(path)` | Load configuration file | YAML/JSON parsing + validation | +| `validate_config(config)` | Validate configuration | JSON Schema validation | +| `get(key, default)` | Get config value | Nested key lookup with dot notation | +| `set(key, value)` | Set config value | Nested key assignment | +| `merge_configs(configs)` | Merge multiple configs | Deep merge algorithm | + +**Example:** + +```python +from semantica.core import ConfigManager + +manager = ConfigManager() + +# Load from file +config = manager.load_config("config.yaml") + +# Validate +is_valid, errors = manager.validate_config(config) + +# Get/set values +llm_model = manager.get("llm.model", default="gpt-4") +manager.set("llm.temperature", 0.7) +``` + +--- + +### LifecycleManager + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `startup()` | Execute startup sequence | Sequential hook execution | +| `shutdown()` | Execute shutdown sequence | Reverse-order hook execution | +| `register_hook(event, callback)` | Register lifecycle hook | Hook registration | +| `get_health_status()` | Get system health | Component health aggregation | + +**Lifecycle Events:** +- `on_startup`: Framework initialization +- `on_shutdown`: Framework cleanup +- `on_error`: Error handling +- `on_health_check`: Health monitoring + +**Example:** + +```python +from semantica.core import LifecycleManager + +manager = LifecycleManager() + +# Register hooks +manager.register_hook("on_startup", lambda: print("Starting...")) +manager.register_hook("on_shutdown", lambda: print("Shutting down...")) + +# Startup +manager.startup() + +# Health check +health = manager.get_health_status() +print(f"Status: {health.status}, Components: {health.components}") + +# Shutdown +manager.shutdown() +``` + +--- + +### PluginRegistry + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `discover_plugins(path)` | Discover available plugins | File system scanning | +| `load_plugin(name)` | Load plugin | Dynamic import + initialization | +| `unload_plugin(name)` | Unload plugin | Cleanup + namespace removal | +| `list_plugins()` | List loaded plugins | Plugin registry enumeration | +| `get_plugin(name)` | Get plugin instance | Registry lookup | + +**Example:** + +```python +from semantica.core import PluginRegistry + +registry = PluginRegistry() + +# Discover plugins +registry.discover_plugins("plugins/") + +# Load plugin +registry.load_plugin("custom_extractor") + +# Use plugin +plugin = registry.get_plugin("custom_extractor") +result = plugin.extract(text) + +# Unload +registry.unload_plugin("custom_extractor") +``` + +--- + +## Convenience Functions + +### build() + + +**Signature:** +```python +def build( + sources: Union[str, List[Union[str, Path]]], + extract_entities: bool = True, + extract_relations: bool = True, + embeddings: bool = True, + graph: bool = True, + **options +) -> Dict[str, Any] +``` + +**Example:** + +```python +from semantica.core import build + +# Simple one-liner +result = build(sources=["documents/"]) + +# With options +result = build( + sources=["documents/"], + extract_entities=True, + extract_relations=True, + embeddings=True, + graph=True, + llm_provider="openai", + llm_model="gpt-4" +) +``` + +--- + +## Configuration Reference + +```yaml +# config.yaml - Complete Configuration Example + +# LLM Configuration +llm: + provider: openai # openai, anthropic, google, groq, ollama + model: gpt-4 + temperature: 0.1 + max_tokens: 4000 + api_key: ${OPENAI_API_KEY} + +# Embeddings Configuration +embeddings: + provider: openai + model: text-embedding-3-large + dimensions: 3072 + batch_size: 100 + +# Knowledge Graph Configuration +knowledge_graph: + merge_entities: true + entity_resolution_strategy: fuzzy + similarity_threshold: 0.85 + resolve_conflicts: true + enable_temporal: true + +# Vector Store Configuration +vector_store: + backend: faiss + index_type: HNSW + metric: cosine + dimension: 3072 + +# Pipeline Configuration +pipeline: + parallel: true + max_workers: 4 + error_handling: retry + max_retries: 3 + +# Logging Configuration +logging: + level: INFO + format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + handlers: + - console + - file +``` + +--- + +## Common Patterns + +### Pattern 1: Basic Usage + +```python +from semantica import Semantica + +semantica = Semantica() +result = semantica.build_knowledge_base(["documents/"]) +kg = result["knowledge_graph"] +``` + +### Pattern 2: Custom Configuration + +```python +from semantica import Semantica, Config + +config = Config( + llm={"provider": "openai", "model": "gpt-4"}, + embeddings={"provider": "openai", "model": "text-embedding-3-large"} +) + +semantica = Semantica(config=config) +result = semantica.build_knowledge_base(["documents/"]) +``` + +### Pattern 3: Pipeline Execution + +```python +from semantica import Semantica + +semantica = Semantica() + +pipeline_config = { + "steps": [ + {"name": "ingest", "type": "FileIngestor"}, + {"name": "parse", "type": "DocumentParser"}, + {"name": "extract", "type": "NERExtractor"}, + {"name": "build_kg", "type": "GraphBuilder"} + ] +} + +result = semantica.run_pipeline(pipeline_config) +``` + +--- + +## Error Handling + +```python +from semantica import Semantica +from semantica.core import ConfigurationError, PipelineError + +try: + semantica = Semantica(config="invalid_config.yaml") +except ConfigurationError as e: + print(f"Configuration error: {e}") + +try: + result = semantica.build_knowledge_base([]) +except PipelineError as e: + print(f"Pipeline error: {e}") +``` + +--- + +## See Also + +- [Pipeline Module](pipeline.md) - Pipeline building and execution +- [Knowledge Graph Module](kg.md) - Graph construction +- [Semantic Extract Module](semantic_extract.md) - Entity extraction diff --git a/docs/reference/embeddings.md b/docs/reference/embeddings.md index 37c51c7d..12bce2ab 100644 --- a/docs/reference/embeddings.md +++ b/docs/reference/embeddings.md @@ -1,3 +1,355 @@ -# Embeddings +# Embeddings Module -::: semantica.embeddings +> **Transform text and images into dense vector representations for semantic search and similarity matching.** + +--- + +## 🎯 Overview + +=== "Text Embeddings" + +
+ + - :material-text:{ .lg .middle } **Transformer Encoders** + + BERT, RoBERTa, DeBERTa with advanced pooling strategies + + - :material-lightning-bolt:{ .lg .middle } **Sentence Transformers** + + Fast, efficient embeddings optimized for semantic similarity + + - :material-api:{ .lg .middle } **API Providers** + + OpenAI, Cohere, Google, HuggingFace integration + +
+ +=== "Image Embeddings" + +
+ + - :material-image:{ .lg .middle } **CLIP** + + Contrastive Language-Image Pre-training for multimodal embeddings + + - :material-eye:{ .lg .middle } **Vision Transformers** + + ViT models for high-quality image representations + + - :material-layers:{ .lg .middle } **CNN Models** + + ResNet, EfficientNet for feature extraction + +
+ +=== "Multimodal" + +
+ + - :material-merge:{ .lg .middle } **Fusion Strategies** + + Concatenation, weighted sum, cross-attention fusion + + - :material-compare:{ .lg .middle } **Similarity** + + Text-image similarity in joint embedding space + +
+ +--- + +## ⚙️ Algorithms Used + +### Text Embedding Algorithms +- **Transformer Encoders**: BERT, RoBERTa, DeBERTa with mean pooling +- **Sentence Transformers**: Siamese networks trained on sentence pairs +- **OpenAI Embeddings**: Proprietary transformer models (ada-002, text-embedding-3) +- **Contrastive Learning**: InfoNCE loss for similarity learning + +### Pooling Strategies +- **Mean Pooling**: Average of all token embeddings +- **Max Pooling**: Maximum value across token dimension +- **CLS Token**: Use [CLS] token embedding (BERT-style) +- **Weighted Mean**: Attention-weighted averaging + +### Normalization +- **L2 Normalization**: `v_norm = v / ||v||₂` for cosine similarity +- **Min-Max Scaling**: Scale to [0, 1] range +- **Standard Scaling**: Zero mean, unit variance + +--- + +## Main Classes + +### EmbeddingGenerator + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `generate(texts)` | Generate embeddings | Provider-specific encoding with pooling | +| `generate_batch(texts, batch_size)` | Batch generation | Chunked processing with parallel execution | +| `encode(text)` | Encode single text | Tokenization + forward pass + pooling | +| `normalize(embeddings)` | Normalize vectors | L2 normalization | +| `set_provider(provider, model)` | Set embedding provider | Provider initialization | + +**Supported Providers:** + +| Provider | Models | Dimensions | Speed | Cost | +|----------|--------|------------|-------|------| +| **OpenAI** | text-embedding-3-small, text-embedding-3-large, ada-002 | 1536, 3072 | Fast | $$ | +| **Cohere** | embed-english-v3.0, embed-multilingual-v3.0 | 1024 | Fast | $$ | +| **HuggingFace** | Custom models | Variable | Medium | Free | +| **Sentence Transformers** | all-MiniLM-L6-v2, all-mpnet-base-v2 | 384, 768 | Fast | Free | +| **Google** | textembedding-gecko | 768 | Fast | $$ | + +**Example:** + +```python +from semantica.embeddings import EmbeddingGenerator + +# OpenAI embeddings +generator = EmbeddingGenerator( + provider="openai", + model="text-embedding-3-large", + dimensions=3072 +) + +texts = ["Machine learning is fascinating", "AI transforms industries"] +embeddings = generator.generate(texts) + +print(f"Shape: {embeddings.shape}") # (2, 3072) +print(f"Normalized: {np.linalg.norm(embeddings[0]):.3f}") # 1.000 + +# Batch processing +large_texts = [...] # 10,000 texts +embeddings = generator.generate_batch( + texts=large_texts, + batch_size=100, + show_progress=True +) +``` + +--- + +### TextEmbedder + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `embed(text)` | Embed single text | Tokenization + encoding + pooling | +| `embed_batch(texts)` | Batch embedding | Parallel processing | +| `embed_query(query)` | Embed query (optimized) | Query-specific encoding | +| `embed_documents(documents)` | Embed documents | Document-specific encoding | +| `tokenize(text)` | Tokenize text | Subword tokenization (BPE/WordPiece) | + +**Tokenization Algorithms:** +- **BPE (Byte Pair Encoding)**: Iterative merging of frequent pairs +- **WordPiece**: Similar to BPE, used in BERT +- **SentencePiece**: Unigram language model tokenization + +**Example:** + +```python +from semantica.embeddings import TextEmbedder + +embedder = TextEmbedder( + model="sentence-transformers/all-mpnet-base-v2", + pooling="mean", # mean, max, cls + normalize=True +) + +# Different embedding strategies +query_embedding = embedder.embed_query("What is machine learning?") +doc_embeddings = embedder.embed_documents([ + "Machine learning is a subset of AI...", + "Deep learning uses neural networks..." +]) + +# Calculate similarity +from sklearn.metrics.pairwise import cosine_similarity +similarity = cosine_similarity([query_embedding], doc_embeddings) +print(f"Similarities: {similarity}") +``` + +--- + +### ImageEmbedder + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `embed(image)` | Embed single image | CNN/ViT encoding | +| `embed_batch(images)` | Batch image embedding | Parallel CNN/ViT processing | +| `preprocess(image)` | Preprocess image | Resize, normalize, augment | +| `extract_features(image)` | Extract visual features | Feature map extraction | + +**Image Embedding Models:** +- **CLIP**: Contrastive Language-Image Pre-training +- **ViT (Vision Transformer)**: Transformer for images +- **ResNet**: Residual networks +- **EfficientNet**: Efficient convolutional networks + +**Example:** + +```python +from semantica.embeddings import ImageEmbedder +from PIL import Image + +embedder = ImageEmbedder( + model="openai/clip-vit-base-patch32", + image_size=224 +) + +image = Image.open("photo.jpg") +embedding = embedder.embed(image) + +print(f"Image embedding shape: {embedding.shape}") +``` + +--- + +### MultimodalEmbedder + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `embed(text, image)` | Embed text + image | Multimodal fusion | +| `embed_text_image(text, image)` | Joint embedding | CLIP-style contrastive learning | +| `fuse_embeddings(text_emb, image_emb)` | Fuse embeddings | Concatenation or weighted sum | +| `calculate_similarity(text, image)` | Text-image similarity | Cosine similarity in joint space | + +**Fusion Strategies:** +- **Concatenation**: `[text_emb; image_emb]` +- **Weighted Sum**: `α*text_emb + (1-α)*image_emb` +- **Cross-Attention**: Attention-based fusion + +**Example:** + +```python +from semantica.embeddings import MultimodalEmbedder + +embedder = MultimodalEmbedder( + model="openai/clip-vit-base-patch32", + fusion_strategy="weighted_sum", + text_weight=0.6 +) + +text = "A cat sitting on a couch" +image = Image.open("cat.jpg") + +embedding = embedder.embed(text, image) +similarity = embedder.calculate_similarity(text, image) +print(f"Text-Image similarity: {similarity:.3f}") +``` + +--- + +### EmbeddingOptimizer + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `optimize(embeddings)` | Optimize embeddings | Dimensionality reduction + quantization | +| `reduce_dimensions(embeddings, target_dim)` | Reduce dimensions | PCA or random projection | +| `quantize(embeddings, bits)` | Quantize embeddings | Scalar or product quantization | +| `compress(embeddings)` | Compress embeddings | Lossy compression | + +**Optimization Techniques:** +- **PCA**: Linear dimensionality reduction +- **Random Projection**: Fast approximate reduction +- **Product Quantization**: Divide into subvectors, quantize separately +- **Scalar Quantization**: Reduce precision (float32 → int8) + +**Example:** + +```python +from semantica.embeddings import EmbeddingOptimizer + +optimizer = EmbeddingOptimizer() + +# Reduce dimensions +reduced = optimizer.reduce_dimensions( + embeddings=embeddings, + target_dim=256, # from 1536 to 256 + method="pca" +) + +# Quantize to int8 +quantized = optimizer.quantize( + embeddings=embeddings, + bits=8 # float32 → int8 (4x compression) +) +``` + +--- + +## Configuration + +```yaml +# config.yaml - Embeddings Configuration + +embeddings: + provider: openai # openai, cohere, huggingface, sentence-transformers + + openai: + model: text-embedding-3-large + dimensions: 3072 + api_key: ${OPENAI_API_KEY} + + sentence_transformers: + model: all-mpnet-base-v2 + pooling: mean # mean, max, cls + normalize: true + device: cuda # cuda, cpu + + batch_processing: + batch_size: 100 + parallel_workers: 4 + show_progress: true + + optimization: + reduce_dimensions: false + target_dimensions: 256 + quantize: false + quantization_bits: 8 +``` + +--- + +## Performance Characteristics + +### Embedding Speed + +| Model | Dimensions | Tokens/sec | Batch Size | Device | +|-------|------------|------------|------------|--------| +| OpenAI ada-002 | 1536 | 10,000 | 100 | API | +| OpenAI text-embedding-3-large | 3072 | 8,000 | 100 | API | +| all-MiniLM-L6-v2 | 384 | 5,000 | 32 | GPU | +| all-mpnet-base-v2 | 768 | 2,000 | 32 | GPU | +| BERT-base | 768 | 1,000 | 16 | GPU | + +### Memory Requirements + +| Model | Dimensions | Memory (GPU) | Memory (CPU) | +|-------|------------|--------------|--------------| +| MiniLM | 384 | 1 GB | 2 GB | +| MPNet | 768 | 2 GB | 4 GB | +| BERT-large | 1024 | 4 GB | 8 GB | + +--- + +## See Also + +- [Vector Store Module](vector_store.md) - Store and search embeddings +- [Semantic Extract Module](semantic_extract.md) - Extract entities +- [Core Module](core.md) - Framework orchestration diff --git a/docs/reference/export.md b/docs/reference/export.md index d1abba81..75020e67 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -1,3 +1,684 @@ -# Export +# Export Module -::: semantica.export +> **Export knowledge graphs and data to multiple formats with W3C-compliant serialization.** + +--- + +## 🎯 Overview + +
+ +- :material-file-export:{ .lg .middle } **Multi-Format Export** + + --- + + Support for 10+ export formats including RDF, JSON, CSV, GraphML + +- :material-database-export:{ .lg .middle } **Graph Databases** + + --- + + Direct export to Neo4j, ArangoDB, Memgraph with Cypher generation + +- :material-code-json:{ .lg .middle } **RDF Serialization** + + --- + + W3C-compliant formats: Turtle, RDF/XML, JSON-LD, N-Triples + +- :material-cog:{ .lg .middle } **Custom Serializers** + + --- + + Extensible serialization framework for custom formats + +- :material-flash:{ .lg .middle } **Batch Export** + + --- + + Efficient large-scale data export with streaming support + +- :material-file-multiple:{ .lg .middle } **Multiple Outputs** + + --- + + Export to Cytoscape.js, D3.js, Gephi, Graphviz formats + +
+ +!!! tip "Choosing Export Format" + - **Development**: Use Turtle for human-readable RDF + - **APIs**: Use JSON-LD for web services + - **Visualization**: Use GraphML or Cytoscape.js + - **Databases**: Use Neo4j Cypher or CSV for bulk import + +--- + +## ⚙️ Algorithms Used + +### Serialization Algorithms +- **RDF/XML Serialization**: W3C RDF/XML specification +- **Turtle Serialization**: Compact RDF format with prefix compression +- **JSON-LD Serialization**: JSON-based linked data with context +- **GraphML Generation**: XML-based graph format +- **Cypher Query Generation**: Neo4j query language generation + +### Export Optimization +- **Streaming Export**: Memory-efficient export for large graphs +- **Batch Processing**: Chunked export with configurable batch sizes +- **Compression**: GZIP compression for large exports +- **Incremental Export**: Export only changed data + +--- + +## Main Classes + +### RDFExporter + +Export knowledge graphs to RDF formats (Turtle, RDF/XML, JSON-LD, N-Triples). + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(graph, filename, format)` | Export to RDF format | RDF serialization with format-specific encoding | +| `serialize(graph, format)` | Serialize to string | In-memory RDF generation | +| `validate(rdf_data)` | Validate RDF syntax | RDF schema validation | +| `add_namespace(prefix, uri)` | Add namespace | Prefix registration | +| `set_base_uri(uri)` | Set base URI | Base URI configuration | + +**Supported RDF Formats:** + +| Format | Extension | Description | Use Case | +|--------|-----------|-------------|----------| +| **Turtle** | .ttl | Compact, human-readable | Development, debugging | +| **N-Triples** | .nt | Line-based, simple | Streaming, processing | +| **RDF/XML** | .rdf | XML-based, verbose | Legacy systems | +| **JSON-LD** | .jsonld | JSON with linked data | Web APIs, JavaScript | +| **N-Quads** | .nq | N-Triples with graphs | Named graphs | + +**Example:** + +```python +from semantica.export import RDFExporter + +exporter = RDFExporter( + base_uri="http://example.org/", + namespaces={ + "ex": "http://example.org/", + "foaf": "http://xmlns.com/foaf/0.1/" + } +) + +# Export to Turtle +exporter.export( + graph=kg, + filename="output.ttl", + format="turtle" +) + +# Export to JSON-LD +exporter.export( + graph=kg, + filename="output.jsonld", + format="json-ld" +) +``` + +--- + +### JSONExporter + +Export knowledge graphs to JSON formats including JSON-LD, Cytoscape.js, and D3.js. + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(graph, filename, format)` | Export to JSON | JSON serialization with schema | +| `export_nodes(graph)` | Export nodes only | Node extraction and serialization | +| `export_edges(graph)` | Export edges only | Edge extraction and serialization | +| `export_cytoscape(graph)` | Export Cytoscape.js format | Cytoscape JSON generation | +| `export_d3(graph)` | Export D3.js format | D3 force-directed graph format | + +**JSON Formats:** + +- **Standard JSON**: Simple node/edge lists +- **JSON-LD**: Linked data with @context +- **Cytoscape.js**: For Cytoscape visualization +- **D3.js**: For D3 force-directed graphs +- **Neo4j JSON**: Neo4j-compatible format + +**Example:** + +```python +from semantica.export import JSONExporter + +exporter = JSONExporter() + +# Standard JSON export +exporter.export(kg, "output.json", format="standard") + +# JSON-LD export +exporter.export(kg, "output.jsonld", format="json-ld") + +# Cytoscape format +exporter.export_cytoscape(kg, "cytoscape.json") +``` + +--- + +### GraphExporter + +Export to graph visualization formats (GraphML, GEXF, DOT, Pajek). + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(graph, filename, format)` | Export to graph format | Format-specific serialization | +| `to_graphml(graph, filename)` | Export to GraphML | XML-based graph format | +| `to_gexf(graph, filename)` | Export to GEXF | Gephi exchange format | +| `to_dot(graph, filename)` | Export to DOT | Graphviz format | +| `to_pajek(graph, filename)` | Export to Pajek | Pajek network format | + +**Graph Formats:** + +| Format | Tool | Use Case | +|--------|------|----------| +| **GraphML** | yEd, Gephi | General graph visualization | +| **GEXF** | Gephi | Network analysis | +| **DOT** | Graphviz | Diagram generation | +| **Pajek** | Pajek | Large network analysis | +| **GML** | Various | Graph Modeling Language | + +**Example:** + +```python +from semantica.export import GraphExporter + +exporter = GraphExporter() + +# Export to GraphML +exporter.to_graphml(kg, "graph.graphml") + +# Export to DOT for Graphviz +exporter.to_dot(kg, "graph.dot") +``` + +--- + +### Neo4jExporter + +Export directly to Neo4j graph database with Cypher query generation. + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(graph, uri, username, password)` | Export to Neo4j | Cypher query execution | +| `generate_cypher(graph)` | Generate Cypher queries | CREATE/MERGE statement generation | +| `batch_import(graph, batch_size)` | Batch import | Chunked Cypher execution | +| `create_indexes(properties)` | Create indexes | Index creation for performance | +| `create_constraints(constraints)` | Create constraints | Uniqueness constraint creation | + +**Cypher Generation:** +```cypher +// Node creation +CREATE (n:Person {name: "Steve Jobs", born: 1955}) + +// Relationship creation +MATCH (a:Person {name: "Steve Jobs"}), (b:Organization {name: "Apple Inc."}) +CREATE (a)-[:FOUNDED]->(b) +``` + +**Example:** + +```python +from semantica.export import Neo4jExporter + +exporter = Neo4jExporter() + +# Direct export to Neo4j +exporter.export( + graph=kg, + uri="bolt://localhost:7687", + username="neo4j", + password="password" +) + +# Generate Cypher queries +cypher_queries = exporter.generate_cypher(kg) +print(cypher_queries) +``` + +--- + +### CSVExporter + +Export to CSV format for spreadsheets and database imports. + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export_nodes(graph, filename)` | Export nodes to CSV | Node flattening and CSV writing | +| `export_edges(graph, filename)` | Export edges to CSV | Edge list CSV generation | +| `export_combined(graph, prefix)` | Export nodes + edges | Separate CSV files | +| `flatten_properties(properties)` | Flatten nested properties | Recursive property flattening | + +**CSV Formats:** +- **Nodes CSV**: id, label, properties (flattened) +- **Edges CSV**: source, target, type, properties +- **Neo4j Import Format**: Neo4j-compatible CSV + +**Example:** + +```python +from semantica.export import CSVExporter + +exporter = CSVExporter() + +# Export nodes and edges separately +exporter.export_nodes(kg, "nodes.csv") +exporter.export_edges(kg, "edges.csv") + +# Or combined export +exporter.export_combined(kg, prefix="graph") +# Creates: graph_nodes.csv, graph_edges.csv +``` + +--- + +### VectorExporter + +Export vector embeddings to various formats. + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(embeddings, filename, format)` | Export vectors | Format-specific vector serialization | +| `export_numpy(embeddings, filename)` | Export to NumPy | .npy format | +| `export_hdf5(embeddings, filename)` | Export to HDF5 | Hierarchical data format | +| `export_parquet(embeddings, filename)` | Export to Parquet | Columnar storage format | + +--- + +## Configuration + +```yaml +# config.yaml - Export Configuration + +export: + rdf: + default_format: turtle + base_uri: "http://example.org/" + include_provenance: true + validate_output: true + + json: + pretty_print: true + indent: 2 + ensure_ascii: false + + graph: + include_metadata: true + export_properties: true + + neo4j: + batch_size: 1000 + create_indexes: true + create_constraints: true + + csv: + delimiter: "," + quote_char: "\"" + encoding: "utf-8" + include_header: true +``` + +--- + +## Performance Characteristics + +### Export Speed + +| Format | Small Graph | Large Graph | Compression | +|--------|-------------|-------------|-------------| +| JSON | Fast | Medium | Good with gzip | +| RDF/XML | Medium | Slow | Poor | +| Turtle | Fast | Fast | Good | +| CSV | Very Fast | Very Fast | Excellent | +| Neo4j | Medium | Medium | N/A | + +### Memory Usage + +- **Streaming Export**: Constant memory usage +- **Batch Export**: Memory proportional to batch size +- **Full Export**: Memory proportional to graph size + +--- + +## See Also + +- [Knowledge Graph Module](kg.md) - Build graphs +- [Visualization Module](visualization.md) - Visualize exported graphs +- [Core Module](core.md) - Framework orchestration + + +## Overview + +- **Multi-Format Export**: Support for 10+ export formats +- **Graph Database Export**: Direct export to Neo4j, ArangoDB, Memgraph +- **RDF Serialization**: W3C-compliant RDF formats +- **Custom Serializers**: Extensible serialization framework +- **Batch Export**: Efficient large-scale data export + +--- + +## Algorithms Used + +### Serialization Algorithms +- **RDF/XML Serialization**: W3C RDF/XML specification +- **Turtle Serialization**: Compact RDF format with prefix compression +- **JSON-LD Serialization**: JSON-based linked data with context +- **GraphML Generation**: XML-based graph format +- **Cypher Query Generation**: Neo4j query language generation + +### Export Optimization +- **Streaming Export**: Memory-efficient export for large graphs +- **Batch Processing**: Chunked export with configurable batch sizes +- **Compression**: GZIP compression for large exports +- **Incremental Export**: Export only changed data + +--- + +## Main Classes + +### RDFExporter + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(graph, filename, format)` | Export to RDF format | RDF serialization with format-specific encoding | +| `serialize(graph, format)` | Serialize to string | In-memory RDF generation | +| `validate(rdf_data)` | Validate RDF syntax | RDF schema validation | +| `add_namespace(prefix, uri)` | Add namespace | Prefix registration | +| `set_base_uri(uri)` | Set base URI | Base URI configuration | + +**Supported RDF Formats:** + +| Format | Extension | Description | Use Case | +|--------|-----------|-------------|----------| +| **Turtle** | .ttl | Compact, human-readable | Development, debugging | +| **N-Triples** | .nt | Line-based, simple | Streaming, processing | +| **RDF/XML** | .rdf | XML-based, verbose | Legacy systems | +| **JSON-LD** | .jsonld | JSON with linked data | Web APIs, JavaScript | +| **N-Quads** | .nq | N-Triples with graphs | Named graphs | + +**Example:** + +```python +from semantica.export import RDFExporter + +exporter = RDFExporter( + base_uri="http://example.org/", + namespaces={ + "ex": "http://example.org/", + "foaf": "http://xmlns.com/foaf/0.1/" + } +) + +# Export to Turtle +exporter.export( + graph=kg, + filename="output.ttl", + format="turtle" +) + +# Export to JSON-LD +exporter.export( + graph=kg, + filename="output.jsonld", + format="json-ld" +) +``` + +--- + +### JSONExporter + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(graph, filename, format)` | Export to JSON | JSON serialization with schema | +| `export_nodes(graph)` | Export nodes only | Node extraction and serialization | +| `export_edges(graph)` | Export edges only | Edge extraction and serialization | +| `export_cytoscape(graph)` | Export Cytoscape.js format | Cytoscape JSON generation | +| `export_d3(graph)` | Export D3.js format | D3 force-directed graph format | + +**JSON Formats:** + +- **Standard JSON**: Simple node/edge lists +- **JSON-LD**: Linked data with @context +- **Cytoscape.js**: For Cytoscape visualization +- **D3.js**: For D3 force-directed graphs +- **Neo4j JSON**: Neo4j-compatible format + +**Example:** + +```python +from semantica.export import JSONExporter + +exporter = JSONExporter() + +# Standard JSON export +exporter.export(kg, "output.json", format="standard") + +# JSON-LD export +exporter.export(kg, "output.jsonld", format="json-ld") + +# Cytoscape format +exporter.export_cytoscape(kg, "cytoscape.json") +``` + +--- + +### GraphExporter + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(graph, filename, format)` | Export to graph format | Format-specific serialization | +| `to_graphml(graph, filename)` | Export to GraphML | XML-based graph format | +| `to_gexf(graph, filename)` | Export to GEXF | Gephi exchange format | +| `to_dot(graph, filename)` | Export to DOT | Graphviz format | +| `to_pajek(graph, filename)` | Export to Pajek | Pajek network format | + +**Graph Formats:** + +| Format | Tool | Use Case | +|--------|------|----------| +| **GraphML** | yEd, Gephi | General graph visualization | +| **GEXF** | Gephi | Network analysis | +| **DOT** | Graphviz | Diagram generation | +| **Pajek** | Pajek | Large network analysis | +| **GML** | Various | Graph Modeling Language | + +**Example:** + +```python +from semantica.export import GraphExporter + +exporter = GraphExporter() + +# Export to GraphML +exporter.to_graphml(kg, "graph.graphml") + +# Export to DOT for Graphviz +exporter.to_dot(kg, "graph.dot") +``` + +--- + +### Neo4jExporter + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(graph, uri, username, password)` | Export to Neo4j | Cypher query execution | +| `generate_cypher(graph)` | Generate Cypher queries | CREATE/MERGE statement generation | +| `batch_import(graph, batch_size)` | Batch import | Chunked Cypher execution | +| `create_indexes(properties)` | Create indexes | Index creation for performance | +| `create_constraints(constraints)` | Create constraints | Uniqueness constraint creation | + +**Cypher Generation:** +```cypher +// Node creation +CREATE (n:Person {name: "Steve Jobs", born: 1955}) + +// Relationship creation +MATCH (a:Person {name: "Steve Jobs"}), (b:Organization {name: "Apple Inc."}) +CREATE (a)-[:FOUNDED]->(b) +``` + +**Example:** + +```python +from semantica.export import Neo4jExporter + +exporter = Neo4jExporter() + +# Direct export to Neo4j +exporter.export( + graph=kg, + uri="bolt://localhost:7687", + username="neo4j", + password="password" +) + +# Generate Cypher queries +cypher_queries = exporter.generate_cypher(kg) +print(cypher_queries) +``` + +--- + +### CSVExporter + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export_nodes(graph, filename)` | Export nodes to CSV | Node flattening and CSV writing | +| `export_edges(graph, filename)` | Export edges to CSV | Edge list CSV generation | +| `export_combined(graph, prefix)` | Export nodes + edges | Separate CSV files | +| `flatten_properties(properties)` | Flatten nested properties | Recursive property flattening | + +**CSV Formats:** +- **Nodes CSV**: id, label, properties (flattened) +- **Edges CSV**: source, target, type, properties +- **Neo4j Import Format**: Neo4j-compatible CSV + +**Example:** + +```python +from semantica.export import CSVExporter + +exporter = CSVExporter() + +# Export nodes and edges separately +exporter.export_nodes(kg, "nodes.csv") +exporter.export_edges(kg, "edges.csv") + +# Or combined export +exporter.export_combined(kg, prefix="graph") +# Creates: graph_nodes.csv, graph_edges.csv +``` + +--- + +### VectorExporter + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `export(embeddings, filename, format)` | Export vectors | Format-specific vector serialization | +| `export_numpy(embeddings, filename)` | Export to NumPy | .npy format | +| `export_hdf5(embeddings, filename)` | Export to HDF5 | Hierarchical data format | +| `export_parquet(embeddings, filename)` | Export to Parquet | Columnar storage format | + +--- + +## Configuration + +```yaml +# config.yaml - Export Configuration + +export: + rdf: + default_format: turtle + base_uri: "http://example.org/" + include_provenance: true + validate_output: true + + json: + pretty_print: true + indent: 2 + ensure_ascii: false + + graph: + include_metadata: true + export_properties: true + + neo4j: + batch_size: 1000 + create_indexes: true + create_constraints: true + + csv: + delimiter: "," + quote_char: "\"" + encoding: "utf-8" + include_header: true +``` + +--- + +## Performance Characteristics + +### Export Speed + +| Format | Small Graph | Large Graph | Compression | +|--------|-------------|-------------|-------------| +| JSON | Fast | Medium | Good with gzip | +| RDF/XML | Medium | Slow | Poor | +| Turtle | Fast | Fast | Good | +| CSV | Very Fast | Very Fast | Excellent | +| Neo4j | Medium | Medium | N/A | + +### Memory Usage + +- **Streaming Export**: Constant memory usage +- **Batch Export**: Memory proportional to batch size +- **Full Export**: Memory proportional to graph size + +--- + +## See Also + +- [Knowledge Graph Module](kg.md) - Build graphs +- [Visualization Module](visualization.md) - Visualize exported graphs +- [Core Module](core.md) - Framework orchestration diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index 5071e5dd..8f49268c 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -1,3 +1,413 @@ -# Ingest +# Ingest Module -::: semantica.ingest +The `ingest` module provides comprehensive data ingestion capabilities for loading data from various sources including files, web pages, feeds, databases, and real-time streams. + +## Overview + +The ingest module supports **50+ file formats** and multiple data sources: + +- **File Ingestion**: PDF, DOCX, XLSX, TXT, MD, JSON, CSV, and more +- **Web Scraping**: HTML pages, sitemaps, with JavaScript rendering +- **Feed Processing**: RSS, Atom feeds with automatic updates +- **Database Connectivity**: SQL and NoSQL databases +- **Stream Processing**: Real-time data streams +- **Email Processing**: EML, MSG, MBOX, PST archives +- **Repository Analysis**: Git repositories and code analysis + +--- + +## Algorithms Used + +### File Discovery +- **Recursive Traversal**: Depth-first search for file discovery +- **Pattern Matching**: Glob patterns with regex support +- **Filtering**: Extension-based and size-based filtering + +### Web Scraping +- **HTML Parsing**: BeautifulSoup/lxml DOM parsing +- **JavaScript Rendering**: Headless browser (Selenium/Playwright) +- **Rate Limiting**: Token bucket algorithm +- **Robots.txt**: Compliance checking + +### Stream Processing +- **Batch Buffering**: Sliding window with configurable size +- **Backpressure Handling**: Flow control mechanisms + +--- + +## Quick Start + +```python +from semantica.ingest import FileIngestor, WebIngestor, FeedIngestor + +# Ingest local files +file_ingestor = FileIngestor(recursive=True) +documents = file_ingestor.ingest("documents/", formats=["pdf", "docx"]) + +# Ingest web content +web_ingestor = WebIngestor(max_depth=2) +web_docs = web_ingestor.ingest("https://example.com/articles") + +# Ingest RSS feeds +feed_ingestor = FeedIngestor(max_items=100) +feed_docs = feed_ingestor.ingest("https://example.com/rss") + +print(f"Total documents: {len(documents) + len(web_docs) + len(feed_docs)}") +``` + +--- + +## Main Classes + +### FileIngestor + + +**Supported Formats:** + +| Category | Formats | +|----------|---------| +| **Documents** | PDF, DOCX, XLSX, PPTX, TXT, RTF, ODT, EPUB, LaTeX, Markdown | +| **Structured** | JSON, YAML, TOML, CSV, TSV, Parquet, Avro, ORC | +| **Web** | HTML, XHTML, XML, JSON-LD, RDFa | +| **Archives** | ZIP, TAR, RAR, 7Z, GZ, BZ2 | +| **Scientific** | BibTeX, EndNote, RIS, JATS XML | +| **Email** | EML, MSG, MBOX, PST | + +**Example Usage:** + +```python +from semantica.ingest import FileIngestor + +# Basic usage +ingestor = FileIngestor() +docs = ingestor.ingest("documents/") + +# Advanced configuration +ingestor = FileIngestor( + recursive=True, + max_file_size=100 * 1024 * 1024, # 100MB + supported_formats=["pdf", "docx", "xlsx"], + extract_archives=True, + ocr_enabled=True, + ocr_language="eng" +) + +# Ingest with filters +docs = ingestor.ingest( + "documents/", + formats=["pdf", "docx"], + exclude_patterns=["*draft*", "*temp*"], + metadata={"source": "company_docs", "version": "1.0"} +) + +# Process results +for doc in docs: + print(f"File: {doc.filename}") + print(f"Format: {doc.format}") + print(f"Size: {doc.size} bytes") + print(f"Pages: {doc.metadata.get('pages', 'N/A')}") +``` + +--- + +### WebIngestor + + +**Example Usage:** + +```python +from semantica.ingest import WebIngestor + +# Basic web scraping +ingestor = WebIngestor() +docs = ingestor.ingest("https://example.com") + +# Advanced configuration +ingestor = WebIngestor( + max_depth=3, + respect_robots_txt=True, + delay_between_requests=1.0, + user_agent="Semantica/1.0", + render_javascript=True, + timeout=30 +) + +# Scrape with patterns +docs = ingestor.ingest( + "https://blog.example.com", + patterns=["*.html", "*/articles/*"], + exclude_patterns=["*/admin/*", "*/login/*"], + follow_links=True, + max_pages=100 +) + +# Extract metadata +for doc in docs: + print(f"URL: {doc.url}") + print(f"Title: {doc.metadata.get('title')}") + print(f"Author: {doc.metadata.get('author')}") + print(f"Published: {doc.metadata.get('published_date')}") +``` + +--- + +### FeedIngestor + + +**Example Usage:** + +```python +from semantica.ingest import FeedIngestor + +# Basic feed ingestion +ingestor = FeedIngestor() +docs = ingestor.ingest("https://example.com/rss") + +# Advanced configuration +ingestor = FeedIngestor( + max_items=1000, + update_interval=3600, # 1 hour + include_content=True, + fetch_full_content=True +) + +# Ingest multiple feeds +feeds = [ + "https://news.ycombinator.com/rss", + "https://example.com/atom", + "https://blog.example.com/feed" +] + +all_docs = [] +for feed_url in feeds: + docs = ingestor.ingest(feed_url) + all_docs.extend(docs) + +print(f"Total feed items: {len(all_docs)}") +``` + +--- + +### DBIngestor + + +**Example Usage:** + +```python +from semantica.ingest import DBIngestor + +# SQL database ingestion +ingestor = DBIngestor( + connection_string="postgresql://user:pass@localhost/db" +) + +docs = ingestor.ingest( + query="SELECT title, content, author, created_at FROM articles WHERE published = true", + metadata={"source": "articles_db", "version": "1.0"} +) + +# NoSQL database ingestion +mongo_ingestor = DBIngestor( + connection_string="mongodb://localhost:27017/mydb" +) + +docs = mongo_ingestor.ingest( + collection="articles", + query={"status": "published"}, + projection={"title": 1, "content": 1, "author": 1} +) +``` + +--- + +### StreamIngestor + + +**Example Usage:** + +```python +from semantica.ingest import StreamIngestor + +# Kafka stream ingestion +ingestor = StreamIngestor( + stream_type="kafka", + bootstrap_servers=["localhost:9092"], + topic="documents", + group_id="semantica-consumer" +) + +# Process stream +for doc in ingestor.stream(): + print(f"Received: {doc.id}") + # Process document + +# RabbitMQ stream ingestion +rabbitmq_ingestor = StreamIngestor( + stream_type="rabbitmq", + host="localhost", + queue="documents" +) +``` + +--- + +### EmailIngestor + + +**Example Usage:** + +```python +from semantica.ingest import EmailIngestor + +# Ingest email files +ingestor = EmailIngestor() +docs = ingestor.ingest("emails/", formats=["eml", "msg"]) + +# Extract attachments +ingestor = EmailIngestor( + extract_attachments=True, + attachment_dir="attachments/" +) + +docs = ingestor.ingest("archive.mbox") + +# Process emails +for doc in docs: + print(f"From: {doc.metadata['from']}") + print(f"Subject: {doc.metadata['subject']}") + print(f"Date: {doc.metadata['date']}") + print(f"Attachments: {len(doc.metadata.get('attachments', []))}") +``` + +--- + +### RepoIngestor + + +**Example Usage:** + +```python +from semantica.ingest import RepoIngestor + +# Ingest Git repository +ingestor = RepoIngestor() +docs = ingestor.ingest( + "https://github.com/user/repo.git", + branch="main", + include_history=True +) + +# Analyze code +ingestor = RepoIngestor( + analyze_code=True, + extract_functions=True, + extract_classes=True, + languages=["python", "javascript"] +) + +docs = ingestor.ingest("path/to/local/repo") +``` + +--- + +## Common Patterns + +### Pattern 1: Multi-Source Ingestion + +```python +from semantica.ingest import FileIngestor, WebIngestor, FeedIngestor + +sources = [] + +# Ingest files +file_ingestor = FileIngestor(recursive=True) +sources.extend(file_ingestor.ingest("documents/")) + +# Ingest web +web_ingestor = WebIngestor() +sources.extend(web_ingestor.ingest("https://example.com")) + +# Ingest feeds +feed_ingestor = FeedIngestor() +sources.extend(feed_ingestor.ingest("https://example.com/rss")) + +print(f"Total sources: {len(sources)}") +``` + +### Pattern 2: Batch Processing with Progress + +```python +from semantica.ingest import FileIngestor +from tqdm import tqdm + +ingestor = FileIngestor() +files = ingestor.list_files("documents/", recursive=True) + +docs = [] +for file_path in tqdm(files, desc="Ingesting"): + doc = ingestor.ingest_file(file_path) + docs.append(doc) +``` + +### Pattern 3: Error Handling + +```python +from semantica.ingest import FileIngestor, IngestionError + +ingestor = FileIngestor() + +successful = [] +failed = [] + +for file_path in file_paths: + try: + doc = ingestor.ingest_file(file_path) + successful.append(doc) + except IngestionError as e: + print(f"Failed to ingest {file_path}: {e}") + failed.append((file_path, str(e))) + +print(f"Successful: {len(successful)}, Failed: {len(failed)}") +``` + +--- + +## Configuration + +```yaml +# config.yaml - Ingest Configuration + +ingest: + file: + recursive: true + max_file_size: 104857600 # 100MB + supported_formats: [pdf, docx, xlsx, txt, md, json, csv] + extract_archives: true + ocr_enabled: true + ocr_language: eng + + web: + max_depth: 3 + respect_robots_txt: true + delay_between_requests: 1.0 + render_javascript: true + timeout: 30 + max_pages: 1000 + + feed: + max_items: 1000 + update_interval: 3600 + fetch_full_content: true + + stream: + batch_size: 100 + max_wait_time: 5 +``` + +--- + +## See Also + +- [Parse Module](parse.md) - Document parsing and content extraction +- [Normalize Module](normalize.md) - Data cleaning and normalization +- [Core Module](core.md) - Framework orchestration diff --git a/docs/reference/kg.md b/docs/reference/kg.md index 506f4d2f..58dc694d 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -1,3 +1,446 @@ -# Kg +# Knowledge Graph Module -::: semantica.kg +> **Build, analyze, and manage knowledge graphs with advanced temporal support and graph analytics.** + +--- + +## 🎯 Overview + +
+ +- :material-graph-outline:{ .lg .middle } **Graph Building** + + --- + + Construct knowledge graphs from entities and relationships with automatic entity resolution + +- :material-merge:{ .lg .middle } **Entity Resolution** + + --- + + Merge duplicate entities using fuzzy and semantic matching algorithms + +- :material-alert-circle:{ .lg .middle } **Conflict Detection** + + --- + + Handle contradictory information with multiple resolution strategies + +- :material-clock-outline:{ .lg .middle } **Temporal Graphs** + + --- + + Track changes over time with versioning and temporal queries + +- :material-chart-line:{ .lg .middle } **Graph Analytics** + + --- + + PageRank, centrality, community detection, and path finding + +- :material-database-search:{ .lg .middle } **Graph Querying** + + --- + + Cypher-like query language with pattern matching + +
+ +--- + +## ⚙️ Algorithms Used + +### Graph Construction Algorithms +- **Incremental Building**: Batch processing for large graphs with memory optimization +- **Entity Merging**: Property aggregation, metadata merging, provenance tracking +- **Relationship Deduplication**: Similarity-based relationship merging + +### Entity Resolution Algorithms +- **Fuzzy Matching**: Levenshtein distance, Jaro-Winkler similarity for string matching +- **Semantic Matching**: Embedding-based similarity (cosine similarity on entity embeddings) +- **Threshold-based Grouping**: Configurable similarity thresholds for entity clustering +- **Conflict Resolution Strategies**: Voting, credibility-weighted, most-recent, highest-confidence + +### Graph Analysis Algorithms +- **Degree Centrality**: Normalized degree calculation: `degree / (n-1)` +- **Betweenness Centrality**: Shortest path counting (BFS-based), normalized by `(n-1)*(n-2)/2` +- **Closeness Centrality**: Average shortest path length calculation +- **PageRank**: Iterative power method with damping factor (default: 0.85) +- **Eigenvector Centrality**: Power iteration method for influence calculation + +### Community Detection Algorithms +- **Louvain Method**: Modularity optimization with greedy agglomeration +- **Leiden Algorithm**: Improved Louvain with guaranteed well-connected communities +- **Label Propagation**: Fast community detection via label spreading +- **Greedy Modularity**: Hierarchical community structure detection + +### Temporal Graph Algorithms +- **Time-Point Queries**: Temporal filtering using valid_from/valid_until comparison +- **Time-Range Queries**: Interval overlap detection, union/intersection aggregation +- **Temporal Patterns**: Sequential pattern mining, temporal motif detection +- **Version Management**: Snapshot creation, diff calculation, rollback support + +### Graph Validation Algorithms +- **Schema Validation**: Type checking against ontology definitions +- **Consistency Checking**: Logical constraint validation, relationship type validation +- **Completeness Validation**: Required property checking, orphaned entity detection +- **Density Calculation**: `E / (n*(n-1)/2)` for undirected graphs + +--- + +## Main Classes + +### GraphBuilder + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `build(entities, relationships)` | Build knowledge graph | Incremental graph construction with entity resolution | +| `add_node(entity)` | Add entity node | Node creation with property validation | +| `add_edge(relationship)` | Add relationship edge | Edge creation with type validation | +| `merge_graphs(graphs)` | Merge multiple graphs | Graph union with entity resolution | +| `resolve_entities(threshold)` | Resolve duplicate entities | Fuzzy + semantic matching | +| `detect_conflicts()` | Detect conflicting information | Multi-source conflict detection | +| `to_neo4j(uri, username, password)` | Export to Neo4j | Cypher query generation | +| `to_rdf(filename, format)` | Export to RDF | RDF serialization (Turtle, N-Triples) | + +**Example:** + +```python +from semantica.kg import GraphBuilder + +builder = GraphBuilder( + merge_entities=True, + entity_resolution_strategy="fuzzy", # "fuzzy", "semantic", "hybrid" + similarity_threshold=0.85, + resolve_conflicts=True, + enable_temporal=True, + temporal_granularity="day" +) + +kg = builder.build(entities, relationships) + +# Export +kg.to_neo4j("bolt://localhost:7687", "neo4j", "password") +kg.to_rdf("output.ttl", format="turtle") + +print(f"Nodes: {kg.node_count}, Edges: {kg.edge_count}") +``` + +--- + +### EntityResolver + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `resolve(graph)` | Resolve all entities | Clustering + merging pipeline | +| `find_duplicates(entities)` | Find duplicate entities | Similarity-based grouping | +| `merge_entities(entity_group)` | Merge entity group | Property aggregation with conflict resolution | +| `calculate_similarity(e1, e2)` | Calculate entity similarity | Weighted combination: string (0.4) + semantic (0.6) | +| `cluster_entities(entities)` | Cluster similar entities | Agglomerative clustering | + +**Similarity Calculation:** +``` +similarity = 0.4 * string_similarity(e1.name, e2.name) + + 0.6 * cosine_similarity(embedding(e1), embedding(e2)) +``` + +**Example:** + +```python +from semantica.kg import EntityResolver + +resolver = EntityResolver( + similarity_threshold=0.85, + merge_strategy="highest_confidence", # "voting", "most_recent", "highest_confidence" + use_embeddings=True +) + +resolved_kg = resolver.resolve(kg) +print(f"Merged {resolver.stats['duplicates_found']} duplicate entities") +``` + +--- + +### GraphAnalyzer + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `compute_metrics()` | Compute graph metrics | Density, diameter, clustering coefficient | +| `analyze_structure()` | Analyze graph structure | Component analysis, connectivity | +| `find_patterns(pattern_type)` | Find graph patterns | Subgraph matching, motif detection | +| `detect_anomalies()` | Detect graph anomalies | Statistical outlier detection | +| `calculate_density()` | Calculate graph density | `E / (n*(n-1)/2)` | + +**Graph Metrics:** +- **Density**: Ratio of actual edges to possible edges +- **Diameter**: Longest shortest path in the graph +- **Average Degree**: Mean number of connections per node +- **Clustering Coefficient**: Measure of local clustering +- **Connected Components**: Number of disconnected subgraphs + +**Example:** + +```python +from semantica.kg import GraphAnalyzer + +analyzer = GraphAnalyzer(kg) + +metrics = analyzer.compute_metrics() +print(f"Density: {metrics['density']:.3f}") +print(f"Diameter: {metrics['diameter']}") +print(f"Avg degree: {metrics['avg_degree']:.2f}") +print(f"Clustering: {metrics['clustering']:.3f}") +``` + +--- + +### CentralityCalculator + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `pagerank(damping=0.85)` | Calculate PageRank | Power iteration method | +| `betweenness_centrality()` | Calculate betweenness | Brandes' algorithm (BFS-based) | +| `closeness_centrality()` | Calculate closeness | Average shortest path length | +| `eigenvector_centrality()` | Calculate eigenvector centrality | Power iteration on adjacency matrix | +| `degree_centrality()` | Calculate degree centrality | Normalized degree: `degree / (n-1)` | + +**PageRank Formula:** +``` +PR(v) = (1-d)/N + d * Σ(PR(u)/L(u)) +where d = damping factor (0.85), N = total nodes, L(u) = outlinks from u +``` + +**Example:** + +```python +from semantica.kg import CentralityCalculator + +centrality = CentralityCalculator(kg) + +# Calculate different centrality measures +pagerank = centrality.pagerank(damping=0.85) +betweenness = centrality.betweenness_centrality() +closeness = centrality.closeness_centrality() + +# Find most influential entities +top_entities = sorted(pagerank.items(), key=lambda x: x[1], reverse=True)[:10] + +for entity, score in top_entities: + print(f"{entity}: {score:.3f}") +``` + +--- + +### CommunityDetector + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `detect(algorithm)` | Detect communities | Specified algorithm (louvain/leiden/label_propagation) | +| `louvain()` | Louvain method | Modularity optimization with greedy agglomeration | +| `leiden()` | Leiden algorithm | Improved Louvain with quality guarantees | +| `label_propagation()` | Label propagation | Fast community detection via label spreading | +| `calculate_modularity(communities)` | Calculate modularity | Q = Σ(eii - ai²) | + +**Modularity Formula:** +``` +Q = 1/(2m) * Σ[Aij - (ki*kj)/(2m)] * δ(ci, cj) +where m = total edges, ki = degree of node i, ci = community of node i +``` + +**Example:** + +```python +from semantica.kg import CommunityDetector + +detector = CommunityDetector(kg) + +# Detect communities using Louvain +communities = detector.detect(algorithm="louvain") + +print(f"Found {len(communities)} communities") +for i, community in enumerate(communities[:5], 1): + print(f"Community {i}: {len(community)} entities") + print(f" Sample: {list(community)[:3]}") +``` + +--- + +### TemporalGraphQuery + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `query_at_time(query, timestamp)` | Query graph at specific time | Temporal filtering with valid_from/valid_until | +| `query_time_range(query, start, end)` | Query over time range | Interval overlap detection | +| `find_temporal_paths(source, target, time_range)` | Find time-aware paths | Temporal Dijkstra's algorithm | +| `analyze_evolution(start, end, interval)` | Analyze graph evolution | Time-series analysis of graph metrics | +| `detect_temporal_patterns(pattern_type)` | Detect temporal patterns | Sequential pattern mining | + +**Temporal Filtering:** +``` +valid_at(edge, t) = (edge.valid_from <= t) AND (t <= edge.valid_until OR edge.valid_until IS NULL) +``` + +**Example:** + +```python +from semantica.kg import TemporalGraphQuery +from datetime import datetime + +temporal_query = TemporalGraphQuery( + enable_temporal_reasoning=True, + temporal_granularity="day" +) + +# Query at specific time +results = temporal_query.query_at_time( + graph=kg, + query="Who founded Apple Inc.?", + at_time="2014-06-15" +) + +# Query time range +evolution = temporal_query.analyze_evolution( + graph=kg, + start_time="2000-01-01", + end_time="2024-12-31", + metrics=["node_count", "edge_count", "density"] +) +``` + +--- + +### ConflictDetector + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `detect_conflicts(graph)` | Detect all conflicts | Multi-source comparison | +| `resolve_conflict(conflict, strategy)` | Resolve single conflict | Strategy-based resolution | +| `validate_consistency(graph)` | Validate logical consistency | Constraint checking | +| `find_contradictions(entity)` | Find contradictory properties | Property value comparison | + +**Conflict Resolution Strategies:** +- **Voting**: Majority value wins +- **Credibility Weighted**: Weight by source credibility scores +- **Most Recent**: Use most recent information +- **Highest Confidence**: Use value with highest confidence score +- **First Seen**: Use first encountered value + +**Example:** + +```python +from semantica.kg import ConflictDetector + +detector = ConflictDetector( + default_strategy="voting", + track_provenance=True +) + +conflicts = detector.detect_conflicts(kg) + +for conflict in conflicts: + print(f"Conflict in {conflict.entity}: {conflict.property}") + print(f" Values: {conflict.values}") + + resolved = detector.resolve_conflict(conflict, strategy="voting") + print(f" Resolved to: {resolved.value}") +``` + +--- + +### ProvenanceTracker + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `track(entity, source, timestamp)` | Track entity provenance | Metadata attachment | +| `get_provenance(entity)` | Get entity provenance | Provenance retrieval | +| `trace_lineage(entity)` | Trace entity lineage | Backward tracing through transformations | +| `validate_provenance(entity)` | Validate provenance chain | Chain integrity checking | + +--- + +## Configuration + +```yaml +# config.yaml - Knowledge Graph Configuration + +kg: + graph_builder: + merge_entities: true + entity_resolution_strategy: fuzzy # fuzzy, semantic, hybrid + similarity_threshold: 0.85 + resolve_conflicts: true + enable_temporal: true + temporal_granularity: day # second, minute, hour, day, week, month, year + + entity_resolution: + use_embeddings: true + fuzzy_threshold: 0.80 + semantic_threshold: 0.85 + merge_strategy: highest_confidence + + conflict_resolution: + default_strategy: voting # voting, credibility_weighted, most_recent, highest_confidence + track_provenance: true + + graph_analytics: + calculate_centrality: true + detect_communities: true + community_algorithm: louvain # louvain, leiden, label_propagation + + temporal: + enable_versioning: true + snapshot_interval: 30 # days + track_history: true +``` + +--- + +## Performance Characteristics + +### Graph Construction +- **Incremental Building**: Efficient for large graphs +- **Batch Processing**: Optimized for bulk operations +- **Memory Management**: Streaming for very large graphs + +### Entity Resolution +- **Fuzzy Matching**: O(n²) complexity, optimized with blocking +- **Semantic Matching**: O(n) with vector index +- **Hybrid**: Best of both approaches + +### Graph Analytics +- **PageRank**: O(k*E) where k = iterations, E = edges +- **Betweenness**: O(V*E) for unweighted graphs +- **Community Detection**: O(E*log²V) for Louvain + +--- + +## See Also + +- [Semantic Extract Module](semantic_extract.md) - Extract entities and relationships +- [Vector Store Module](vector_store.md) - Store graph embeddings +- [Triple Store Module](triple_store.md) - RDF storage +- [Ontology Module](ontology.md) - Define graph schemas diff --git a/docs/reference/normalize.md b/docs/reference/normalize.md index a7cef528..1a3afea0 100644 --- a/docs/reference/normalize.md +++ b/docs/reference/normalize.md @@ -1,3 +1,115 @@ -# Normalize +# Normalize Module -::: semantica.normalize +The `normalize` module provides data cleaning, normalization, and standardization capabilities to prepare text and data for semantic processing. + +## Overview + +- **Text Cleaning**: Remove noise, fix encoding, standardize whitespace +- **Entity Normalization**: Standardize entity names and formats +- **Date Normalization**: Parse and standardize date formats +- **Number Normalization**: Standardize numeric values and units +- **Language Detection**: Identify document language +- **Encoding Handling**: Fix character encoding issues + +--- + +## Algorithms Used + +### Text Normalization +- **Unicode Normalization**: NFC, NFD, NFKC, NFKD forms +- **Whitespace Normalization**: Regex-based cleanup +- **Case Folding**: Locale-aware case normalization +- **Diacritic Removal**: Unicode decomposition + +### Entity Normalization +- **Fuzzy Matching**: Levenshtein distance < threshold +- **Phonetic Matching**: Soundex, Metaphone algorithms +- **Abbreviation Expansion**: Dictionary-based expansion + +### Date/Time Normalization +- **Parsing**: dateutil parser with multiple format support +- **Timezone Handling**: pytz for timezone conversion +- **Standardization**: ISO 8601 format output + +--- + +## Quick Start + +```python +from semantica.normalize import TextNormalizer + +# Initialize normalizer +normalizer = TextNormalizer() + +# Normalize text +normalized_text = normalizer.normalize(raw_text) + +# Normalize documents +normalized_docs = normalizer.normalize_documents(documents) +``` + +--- + +## Main Classes + +### TextNormalizer + + +**Example:** + +```python +from semantica.normalize import TextNormalizer + +normalizer = TextNormalizer( + lowercase=False, + remove_punctuation=False, + remove_numbers=False, + remove_whitespace=True, + fix_encoding=True +) + +text = " Apple Inc. was founded in 1976. " +normalized = normalizer.normalize(text) +# Output: "Apple Inc. was founded in 1976." +``` + +### EntityNormalizer + + +**Example:** + +```python +from semantica.normalize import EntityNormalizer + +normalizer = EntityNormalizer( + fuzzy_matching=True, + similarity_threshold=0.85 +) + +# Normalize entity names +entities = ["Apple Inc.", "Apple", "AAPL", "Apple Computer"] +normalized = normalizer.normalize(entities) +# All mapped to canonical form: "Apple Inc." +``` + +### DateNormalizer + + +**Example:** + +```python +from semantica.normalize import DateNormalizer + +normalizer = DateNormalizer(output_format="ISO8601") + +dates = ["Jan 1, 2024", "01/01/2024", "2024-01-01"] +normalized = [normalizer.normalize(d) for d in dates] +# All output: "2024-01-01T00:00:00Z" +``` + +--- + +## See Also + +- [Parse Module](parse.md) +- [Semantic Extract Module](semantic_extract.md) diff --git a/docs/reference/ontology.md b/docs/reference/ontology.md index 29d2dfee..de45e1f3 100644 --- a/docs/reference/ontology.md +++ b/docs/reference/ontology.md @@ -1,3 +1,349 @@ -# Ontology +# Ontology Module -::: semantica.ontology +> **Generate W3C-compliant OWL ontologies from unstructured content using a 6-stage LLM pipeline.** + +--- + +## 🎯 Overview + +
+ +- :material-auto-fix:{ .lg .middle } **Automatic Generation** + + --- + + LLM-based ontology creation from text and knowledge graphs + +- :material-file-tree:{ .lg .middle } **OWL/RDF Support** + + --- + + W3C-compliant ontology formats with Turtle, RDF/XML, JSON-LD + +- :material-check-circle:{ .lg .middle } **Validation** + + --- + + HermiT and Pellet reasoner integration for consistency checking + +- :material-family-tree:{ .lg .middle } **Class Hierarchy** + + --- + + Automatic taxonomy generation with multiple inheritance + +- :material-link:{ .lg .middle } **Property Definition** + + --- + + Object and data properties with domain/range inference + +- :material-brain:{ .lg .middle } **6-Stage Pipeline** + + --- + + Semantic parsing → Definitions → Types → Hierarchy → TTL → Validation + +
+ +!!! info "6-Stage Ontology Generation Pipeline" + ```mermaid + graph LR + A[1. Semantic Network
Parsing] --> B[2. YAML to
Definition] + B --> C[3. Definition
to Types] + C --> D[4. Hierarchy
Generation] + D --> E[5. TTL
Generation] + E --> F[6. Symbolic
Validation] + + style A fill:#e3f2fd + style F fill:#c8e6c9 + ``` + +--- + +## ⚙️ Algorithms Used + +### 6-Stage Ontology Generation Pipeline + +**Stage 1: Semantic Network Parsing** +- Extract domain concepts from text +- Identify key entities and relationships +- Build initial concept graph + +**Stage 2: YAML-to-Definition** +- Transform semantic network to class definitions +- Define class hierarchies and properties +- Generate human-readable descriptions + +**Stage 3: Definition-to-Types** +- Map to OWL types (Class, ObjectProperty, DataProperty) +- Define domains and ranges +- Specify cardinality constraints + +**Stage 4: Hierarchy Generation** +- Build taxonomic structures (is-a relationships) +- Identify parent-child relationships +- Create multiple inheritance where appropriate + +**Stage 5: TTL Generation** +- Generate OWL/Turtle syntax +- Add namespace declarations +- Format according to W3C standards + +**Stage 6: Symbolic Validation** +- HermiT/Pellet reasoning for consistency +- Detect logical contradictions +- Validate class satisfiability + +### Reasoning Algorithms +- **HermiT**: Hypertableau reasoning algorithm +- **Pellet**: Tableau-based DL reasoner +- **Consistency Checking**: Detect unsatisfiable classes +- **Classification**: Compute inferred class hierarchy + +--- + +## Main Classes + +### OntologyGenerator + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `generate(source)` | Generate ontology | 6-stage pipeline | +| `generate_from_graph(kg)` | Generate from knowledge graph | Graph analysis + LLM generation | +| `generate_from_text(text)` | Generate from text | Text analysis + concept extraction | +| `validate(ontology)` | Validate ontology | Reasoner-based validation | +| `infer_hierarchy(classes)` | Infer class hierarchy | Hierarchical clustering + LLM | + +**Example:** + +```python +from semantica.ontology import OntologyGenerator + +generator = OntologyGenerator( + llm_provider="openai", + llm_model="gpt-4", + validation_reasoner="hermit" # hermit, pellet +) + +# Generate from knowledge graph +ontology = generator.generate_from_graph(kg) + +# Validate +is_valid, errors = generator.validate(ontology) +print(f"Valid: {is_valid}, Errors: {len(errors)}") + +# Save +ontology.save("ontology.owl", format="owl") +``` + +--- + +### OntologyValidator + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `validate(ontology)` | Full validation | Consistency + satisfiability checks | +| `check_consistency()` | Check logical consistency | Tableau reasoning | +| `check_satisfiability(class_name)` | Check class satisfiability | Subsumption testing | +| `find_inconsistencies()` | Find logical errors | Reasoner-based detection | +| `explain_inconsistency(class_name)` | Explain why class is unsatisfiable | Axiom tracing | + +**Validation Checks:** +- **Consistency**: No logical contradictions +- **Satisfiability**: All classes can have instances +- **Coherence**: No unsatisfiable classes +- **Completeness**: All required axioms present + +**Example:** + +```python +from semantica.ontology import OntologyValidator + +validator = OntologyValidator(reasoner="hermit") + +# Validate ontology +result = validator.validate(ontology) + +if not result.is_valid: + print("Inconsistencies found:") + for error in result.errors: + print(f" - {error.class_name}: {error.message}") + explanation = validator.explain_inconsistency(error.class_name) + print(f" Reason: {explanation}") +``` + +--- + +### OWLGenerator + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `generate_owl(ontology)` | Generate OWL file | OWL/XML or Turtle serialization | +| `generate_classes(classes)` | Generate class definitions | OWL Class axioms | +| `generate_properties(properties)` | Generate properties | ObjectProperty/DataProperty axioms | +| `generate_individuals(individuals)` | Generate instances | Individual assertions | +| `add_axiom(axiom)` | Add OWL axiom | Axiom insertion | + +**OWL Constructs:** +- **Classes**: `owl:Class` +- **Object Properties**: `owl:ObjectProperty` +- **Data Properties**: `owl:DatatypeProperty` +- **Individuals**: `owl:NamedIndividual` +- **Restrictions**: `owl:Restriction`, `owl:someValuesFrom`, `owl:allValuesFrom` + +**Example:** + +```python +from semantica.ontology import OWLGenerator + +generator = OWLGenerator( + base_uri="http://example.org/ontology#", + format="turtle" # turtle, owl-xml, rdf-xml +) + +# Generate OWL +owl_content = generator.generate_owl(ontology) + +# Save +with open("ontology.ttl", "w") as f: + f.write(owl_content) +``` + +--- + +### PropertyGenerator + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `generate_properties(relationships)` | Generate all properties | Relationship analysis | +| `infer_domain_range(property)` | Infer domain and range | Type inference from usage | +| `generate_object_property(name, domain, range)` | Create object property | OWL ObjectProperty axiom | +| `generate_data_property(name, domain, datatype)` | Create data property | OWL DatatypeProperty axiom | + +**Property Types:** +- **Object Properties**: Relate individuals to individuals +- **Data Properties**: Relate individuals to data values +- **Annotation Properties**: Metadata properties + +**Example:** + +```python +from semantica.ontology import PropertyGenerator + +generator = PropertyGenerator() + +# Generate properties from relationships +properties = generator.generate_properties(relationships) + +# Create specific property +founder_property = generator.generate_object_property( + name="foundedBy", + domain="Organization", + range="Person" +) +``` + +--- + +### ClassInferrer + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `infer_classes(entities)` | Infer class definitions | Entity type clustering | +| `build_hierarchy(classes)` | Build class hierarchy | Hierarchical clustering + LLM | +| `identify_disjoint_classes(classes)` | Find disjoint classes | Logical analysis | +| `generate_restrictions(class_name)` | Generate class restrictions | Property analysis | + +**Hierarchy Building:** +- **Bottom-up**: Start with specific classes, generalize +- **Top-down**: Start with general classes, specialize +- **Hybrid**: Combine both approaches + +--- + +## Configuration + +```yaml +# config.yaml - Ontology Configuration + +ontology: + generation: + llm_provider: openai + llm_model: gpt-4 + temperature: 0.1 + stages: + - semantic_network_parsing + - yaml_to_definition + - definition_to_types + - hierarchy_generation + - ttl_generation + - symbolic_validation + + validation: + reasoner: hermit # hermit, pellet + check_consistency: true + check_satisfiability: true + explain_errors: true + + owl: + base_uri: "http://example.org/ontology#" + format: turtle # turtle, owl-xml, rdf-xml + include_annotations: true + include_individuals: true +``` + +--- + +## OWL Example + +```turtle +@prefix : . +@prefix owl: . +@prefix rdf: . +@prefix rdfs: . + +# Classes +:Person rdf:type owl:Class . +:Organization rdf:type owl:Class . +:Company rdf:type owl:Class ; + rdfs:subClassOf :Organization . + +# Object Properties +:foundedBy rdf:type owl:ObjectProperty ; + rdfs:domain :Organization ; + rdfs:range :Person . + +# Data Properties +:foundedYear rdf:type owl:DatatypeProperty ; + rdfs:domain :Organization ; + rdfs:range xsd:gYear . + +# Individuals +:AppleInc rdf:type :Company ; + :foundedBy :SteveJobs ; + :foundedYear "1976"^^xsd:gYear . +``` + +--- + +## See Also + +- [Knowledge Graph Module](kg.md) - Build knowledge graphs +- [Triple Store Module](triple_store.md) - Store RDF triples +- [Reasoning Module](reasoning.md) - Logical reasoning diff --git a/docs/reference/parse.md b/docs/reference/parse.md index 91a864d5..16e80dfa 100644 --- a/docs/reference/parse.md +++ b/docs/reference/parse.md @@ -1,3 +1,371 @@ -# Parse +# Parse Module -::: semantica.parse +The `parse` module provides comprehensive document parsing capabilities for extracting content, metadata, and structure from various file formats. + +## Overview + +- **Document Parsing**: PDF, DOCX, XLSX, PPTX, and more +- **OCR Processing**: Extract text from images and scanned documents +- **Table Extraction**: Extract tables with structure preservation +- **Metadata Extraction**: Author, title, dates, and custom metadata +- **Structure Analysis**: Headings, sections, lists, and document hierarchy +- **Multi-Format Support**: 50+ file formats with specialized parsers + +--- + +## Algorithms Used + +### PDF Parsing +- **Text Extraction**: PDFMiner algorithm for text layout analysis +- **OCR**: Tesseract with preprocessing (deskewing, noise reduction) +- **Table Detection**: Hough transform for line detection + clustering +- **Layout Analysis**: XY-cut algorithm for reading order determination + +### OCR Algorithms +- **Preprocessing**: Otsu's thresholding, morphological operations +- **Text Detection**: EAST/CRAFT deep learning models +- **Recognition**: Tesseract LSTM + language models +- **Post-processing**: Spell correction, confidence filtering + +### Table Extraction +- **Rule-based**: Line detection + cell clustering +- **ML-based**: Deep learning table structure recognition +- **Hybrid**: Combine rules + ML for robustness + +--- + +# Parse with options +docs = parser.parse( + sources=["documents/"], + formats=["pdf", "docx", "xlsx"], + metadata={"source": "company_docs"} +) + +# Access parsed data +for doc in docs: + print(f"Title: {doc.metadata.get('title')}") + print(f"Author: {doc.metadata.get('author')}") + print(f"Pages: {doc.metadata.get('pages')}") + print(f"Text length: {len(doc.text)} characters") +``` + +--- + +### PDFParser + + +**Example Usage:** + +```python +from semantica.parse import PDFParser + +# Basic PDF parsing +parser = PDFParser() +doc = parser.parse("document.pdf") + +# Advanced PDF parsing with OCR +parser = PDFParser( + ocr_enabled=True, + ocr_language="eng", + extract_tables=True, + extract_images=True, + dpi=300 +) + +doc = parser.parse("scanned_document.pdf") + +# Access PDF-specific data +print(f"Pages: {doc.metadata['pages']}") +print(f"PDF version: {doc.metadata['pdf_version']}") +print(f"Tables extracted: {len(doc.tables)}") + +# Access tables +for i, table in enumerate(doc.tables): + print(f"Table {i+1}:") + print(table.to_dataframe()) +``` + +--- + +### DOCXParser + + +**Example Usage:** + +```python +from semantica.parse import DOCXParser + +# Parse Word document +parser = DOCXParser() +doc = parser.parse("document.docx") + +# With structure preservation +parser = DOCXParser( + preserve_structure=True, + extract_tables=True, + extract_images=True +) + +doc = parser.parse("report.docx") + +# Access structure +print(f"Headings: {doc.structure['headings']}") +print(f"Sections: {len(doc.structure['sections'])}") +print(f"Tables: {len(doc.tables)}") +``` + +--- + +### ExcelParser + + +**Example Usage:** + +```python +from semantica.parse import ExcelParser + +# Parse Excel file +parser = ExcelParser() +doc = parser.parse("data.xlsx") + +# Parse specific sheets +parser = ExcelParser( + sheets=["Sheet1", "Data"], + header_row=0, + skip_empty_rows=True +) + +doc = parser.parse("workbook.xlsx") + +# Access data +for sheet_name, df in doc.dataframes.items(): + print(f"Sheet: {sheet_name}") + print(f"Rows: {len(df)}, Columns: {len(df.columns)}") + print(df.head()) +``` + +--- + +### HTMLParser + + +**Example Usage:** + +```python +from semantica.parse import HTMLParser + +# Parse HTML +parser = HTMLParser() +doc = parser.parse("page.html") + +# Advanced parsing +parser = HTMLParser( + extract_metadata=True, + extract_links=True, + remove_scripts=True, + remove_styles=True +) + +doc = parser.parse("article.html") + +# Access extracted data +print(f"Title: {doc.metadata['title']}") +print(f"Links: {len(doc.links)}") +print(f"Main content: {doc.main_content}") +``` + +--- + +### JSONParser + + +**Example Usage:** + +```python +from semantica.parse import JSONParser + +# Parse JSON +parser = JSONParser() +doc = parser.parse("data.json") + +# Flatten nested JSON +parser = JSONParser( + flatten=True, + separator=".", + extract_schema=True +) + +doc = parser.parse("nested_data.json") + +# Access schema +print(f"Schema: {doc.schema}") +print(f"Flattened keys: {list(doc.flattened_data.keys())}") +``` + +--- + +### ImageParser + + +**Example Usage:** + +```python +from semantica.parse import ImageParser + +# Parse image with OCR +parser = ImageParser( + ocr_enabled=True, + ocr_language="eng", + detect_orientation=True +) + +doc = parser.parse("scanned_page.jpg") + +# Access OCR results +print(f"Extracted text: {doc.text}") +print(f"Confidence: {doc.metadata['ocr_confidence']}") +print(f"Orientation: {doc.metadata['orientation']}") +``` + +--- + +### CodeParser + + +**Example Usage:** + +```python +from semantica.parse import CodeParser + +# Parse source code +parser = CodeParser( + language="python", + extract_functions=True, + extract_classes=True, + extract_docstrings=True +) + +doc = parser.parse("module.py") + +# Access code structure +print(f"Functions: {len(doc.functions)}") +print(f"Classes: {len(doc.classes)}") + +for func in doc.functions: + print(f"Function: {func.name}") + print(f" Parameters: {func.parameters}") + print(f" Docstring: {func.docstring}") +``` + +--- + +## Common Patterns + +### Pattern 1: Batch Parsing + +```python +from semantica.parse import DocumentParser +from pathlib import Path + +parser = DocumentParser() + +# Get all files +files = list(Path("documents/").rglob("*.*")) + +# Parse in batches +batch_size = 10 +for i in range(0, len(files), batch_size): + batch = files[i:i+batch_size] + docs = parser.parse(batch) + # Process docs +``` + +### Pattern 2: Format-Specific Parsing + +```python +from semantica.parse import PDFParser, DOCXParser, ExcelParser + +# Route to appropriate parser +def parse_document(file_path): + if file_path.endswith('.pdf'): + parser = PDFParser(ocr_enabled=True) + elif file_path.endswith('.docx'): + parser = DOCXParser(preserve_structure=True) + elif file_path.endswith('.xlsx'): + parser = ExcelParser() + else: + raise ValueError(f"Unsupported format: {file_path}") + + return parser.parse(file_path) +``` + +### Pattern 3: Error Handling + +```python +from semantica.parse import DocumentParser, ParseError + +parser = DocumentParser() + +successful = [] +failed = [] + +for file_path in file_paths: + try: + doc = parser.parse(file_path) + successful.append(doc) + except ParseError as e: + print(f"Failed to parse {file_path}: {e}") + failed.append((file_path, str(e))) + +print(f"Parsed: {len(successful)}, Failed: {len(failed)}") +``` + +--- + +## Configuration + +```yaml +# config.yaml - Parse Configuration + +parse: + pdf: + ocr_enabled: true + ocr_language: eng + extract_tables: true + extract_images: true + dpi: 300 + + docx: + preserve_structure: true + extract_tables: true + extract_images: true + + excel: + header_row: 0 + skip_empty_rows: true + infer_types: true + + html: + extract_metadata: true + extract_links: true + remove_scripts: true + remove_styles: true + + image: + ocr_enabled: true + ocr_language: eng + detect_orientation: true + + code: + extract_functions: true + extract_classes: true + extract_docstrings: true +``` + +--- + +## See Also + +- [Ingest Module](ingest.md) - Data ingestion +- [Normalize Module](normalize.md) - Data cleaning and normalization +- [Semantic Extract Module](semantic_extract.md) - Entity and relationship extraction diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index 1c3d970e..17258790 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -1,3 +1,127 @@ -# Pipeline +# Pipeline Module -::: semantica.pipeline +Build and execute complex processing pipelines with support for parallel execution, error handling, and resource management. + +## Overview + +- **Pipeline Building**: Fluent API for pipeline construction +- **Parallel Execution**: Multi-threaded/multi-process execution +- **Error Handling**: Retry logic and failure recovery +- **Resource Management**: Memory and CPU optimization +- **Monitoring**: Progress tracking and logging + +--- + +## Algorithms Used + +### Execution Algorithms +- **DAG (Directed Acyclic Graph) Execution**: Topological sort for step ordering +- **Parallel Execution**: Thread pool with work stealing +- **Dependency Resolution**: Kahn's algorithm for topological ordering +- **Resource Scheduling**: Greedy scheduling with priority queue + +### Error Handling +- **Exponential Backoff**: Retry with increasing delays: `delay = base * 2^attempt` +- **Circuit Breaker**: Fail fast after threshold failures +- **Checkpoint/Resume**: State persistence for long-running pipelines + +--- + +## Main Classes + +### PipelineBuilder + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `add_step(name, processor)` | Add pipeline step | Step registration | +| `add_parallel_steps(steps)` | Add parallel steps | Parallel branch creation | +| `build()` | Build pipeline | DAG construction + validation | +| `execute(input_data)` | Execute pipeline | Topological execution | +| `set_error_handler(handler)` | Set error handler | Handler registration | + +**Example:** + +```python +from semantica.pipeline import PipelineBuilder +from semantica.ingest import FileIngestor +from semantica.parse import DocumentParser +from semantica.semantic_extract import NERExtractor + +pipeline = PipelineBuilder() \\ + .add_step("ingest", FileIngestor(recursive=True)) \\ + .add_step("parse", DocumentParser()) \\ + .add_step("extract", NERExtractor()) \\ + .add_step("build_kg", GraphBuilder()) \\ + .build() + +result = pipeline.execute(sources=["documents/"]) +print(f"Processed {result.stats['documents_processed']} documents") +``` + +--- + +### ExecutionEngine + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `execute(pipeline, data)` | Execute pipeline | DAG traversal | +| `execute_parallel(steps, data)` | Parallel execution | Thread/process pool | +| `execute_sequential(steps, data)` | Sequential execution | Linear execution | +| `schedule_steps(dag)` | Schedule execution | Topological sort | + +--- + +### FailureHandler + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `handle_failure(error, context)` | Handle step failure | Strategy-based handling | +| `retry(step, max_retries)` | Retry failed step | Exponential backoff | +| `skip(step)` | Skip failed step | Continue execution | +| `abort(pipeline)` | Abort pipeline | Graceful shutdown | + +**Retry Strategy:** +``` +delay = base_delay * (2 ** attempt) + random_jitter +max_delay = min(delay, max_delay) +``` + +--- + +## Configuration + +```yaml +# config.yaml - Pipeline Configuration + +pipeline: + execution: + parallel: true + max_workers: 4 + timeout: 3600 # seconds + + error_handling: + strategy: retry # retry, skip, abort + max_retries: 3 + retry_delay: 1 # seconds + exponential_backoff: true + + monitoring: + log_level: INFO + progress_bar: true + checkpoint_interval: 100 # steps +``` + +--- + +## See Also + +- [Core Module](core.md) diff --git a/docs/reference/reasoning.md b/docs/reference/reasoning.md index 23fb4d70..c94e1105 100644 --- a/docs/reference/reasoning.md +++ b/docs/reference/reasoning.md @@ -1,3 +1,154 @@ -# Reasoning +# Reasoning Module -::: semantica.reasoning +Perform logical inference and reasoning on knowledge graphs using rule-based and deductive reasoning engines with support for forward/backward chaining. + +## Overview + +- **Rule-Based Reasoning**: Apply logical rules to derive new facts +- **Deductive Reasoning**: Infer conclusions from premises +- **Forward Chaining**: Data-driven reasoning +- **Backward Chaining**: Goal-driven reasoning +- **SWRL Support**: Semantic Web Rule Language + +--- + +## Algorithms Used + +### Inference Algorithms +- **Forward Chaining (Rete Algorithm)**: Efficient pattern matching, O(RFP) complexity where R=rules, F=facts, P=patterns +- **Backward Chaining**: Goal-directed reasoning with SLD resolution +- **Tableau Algorithm**: Description Logic reasoning +- **Resolution**: First-order logic inference + +### Rule Matching +- **Rete Network**: Compiled rule network for efficient matching +- **Pattern Matching**: Unification algorithm for variable binding +- **Conflict Resolution**: Priority-based rule selection + +--- + +## Main Classes + +### InferenceEngine + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `forward_chain(kg, rules)` | Forward chaining inference | Rete algorithm | +| `backward_chain(kg, goal)` | Backward chaining | SLD resolution | +| `infer(kg, rules)` | General inference | Auto-select forward/backward | +| `apply_rules(facts, rules)` | Apply rule set | Pattern matching + unification | +| `explain_inference(fact)` | Explain derivation | Proof tree generation | + +**Example:** + +```python +from semantica.reasoning import InferenceEngine, RuleManager + +engine = InferenceEngine( + strategy="forward", # forward, backward, hybrid + max_iterations=100, + explain_inferences=True +) + +rule_manager = RuleManager() +rule_manager.add_rule( + "IF ?x foundedBy ?y THEN ?y founder_of ?x" +) + +# Forward chaining +new_facts = engine.forward_chain(kg, rule_manager) +print(f"Inferred {len(new_facts)} new facts") + +# Explain inference +for fact in new_facts[:5]: + explanation = engine.explain_inference(fact) + print(f"{fact}: {explanation}") +``` + +--- + +### RuleManager + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `add_rule(rule)` | Add inference rule | Rule parsing + validation | +| `remove_rule(rule_id)` | Remove rule | Rule deletion | +| `load_rules(filename)` | Load rules from file | SWRL/custom format parsing | +| `validate_rules()` | Validate rule set | Consistency checking | +| `compile_rules()` | Compile to Rete network | Rete compilation | + +**Rule Syntax:** +``` +IF THEN + +Examples: +IF ?x type Person AND ?x worksFor ?y THEN ?y employs ?x +IF ?x foundedBy ?y AND ?y type Person THEN ?x type Organization +``` + +**Example:** + +```python +from semantica.reasoning import RuleManager + +rules = RuleManager() + +# Add rules +rules.add_rule("IF ?x type Company AND ?x foundedBy ?y THEN ?y founder_of ?x") +rules.add_rule("IF ?x founder_of ?y AND ?y type Company THEN ?x type Entrepreneur") + +# Load from file +rules.load_rules("rules.swrl") + +# Validate +is_valid, errors = rules.validate_rules() +``` + +--- + +### DeductiveReasoner + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `reason(kg, axioms)` | Perform deductive reasoning | Tableau algorithm | +| `check_entailment(kg, statement)` | Check if statement is entailed | Subsumption testing | +| `find_inconsistencies(kg)` | Find logical contradictions | Consistency checking | +| `classify(kg)` | Compute class hierarchy | Classification algorithm | + +--- + +## Configuration + +```yaml +# config.yaml - Reasoning Configuration + +reasoning: + inference: + strategy: forward # forward, backward, hybrid + max_iterations: 100 + explain_inferences: true + + rules: + format: swrl # swrl, custom + validate_on_load: true + + deductive: + reasoner: hermit # hermit, pellet + check_consistency: true +``` + +--- + +## See Also + +- [Knowledge Graph Module](kg.md) +- [Ontology Module](ontology.md) diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index 503dbabd..334e454b 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -1,3 +1,429 @@ -# Semantic Extract +# Semantic Extract Module -::: semantica.semantic_extract +> **Extract structured knowledge from unstructured text using state-of-the-art NLP and LLM models.** + +--- + +## 🎯 Overview + +
+ +- :material-account-search:{ .lg .middle } **Named Entity Recognition** + + --- + + Extract people, organizations, locations, dates, and custom entities with high accuracy + +- :material-graph:{ .lg .middle } **Relationship Extraction** + + --- + + Identify semantic, temporal, and causal relationships between entities + +- :material-calendar-clock:{ .lg .middle } **Event Detection** + + --- + + Detect and classify events like acquisitions, partnerships, announcements + +- :material-cube-outline:{ .lg .middle } **Triple Extraction** + + --- + + Generate RDF triples for knowledge graphs in multiple formats + +- :material-link-variant:{ .lg .middle } **Coreference Resolution** + + --- + + Resolve pronouns and entity mentions across documents + +- :material-brain:{ .lg .middle } **Semantic Analysis** + + --- + + Deep semantic understanding with role labeling and clustering + +
+ +--- + +## ⚙️ Algorithms Used + +### Entity Extraction Algorithms +- **Transformer-based NER**: BERT, RoBERTa, DeBERTa models for high-accuracy entity recognition +- **spaCy Statistical Models**: Fast, production-ready NER with en_core_web_sm/lg models +- **Rule-based Matching**: Pattern-based entity detection using regex and linguistic rules +- **LLM-based Extraction**: GPT-4, Claude for complex entity extraction with few-shot learning + +### Relationship Extraction Algorithms +- **Dependency Parsing**: Syntactic dependency trees for relationship identification +- **Pattern Matching**: Rule-based relationship templates +- **Neural Relation Classification**: BERT-based relation classifiers +- **Hybrid Approach**: Combines rule-based + ML + LLM for optimal accuracy + +### Coreference Resolution Algorithms +- **Neural Coreference**: Span-based neural models for pronoun resolution +- **Rule-based Resolution**: Heuristic rules for simple coreference cases +- **Cluster-based Merging**: Entity mention clustering across documents + +### Triple Extraction Algorithms +- **Subject-Predicate-Object Extraction**: Dependency parsing for triple generation +- **RDF Serialization**: Turtle, N-Triples, JSON-LD format generation +- **Triple Validation**: Schema-based validation and quality checking + +--- + +## Main Classes + +### NamedEntityRecognizer + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `extract(text)` | Extract entities from text | Transformer-based NER with confidence scoring | +| `extract_batch(texts)` | Batch entity extraction | Parallel processing with batching | +| `add_custom_entity_type(name, patterns)` | Add custom entity type | Pattern matching with regex | +| `classify_entity(entity)` | Classify entity type | Multi-class classification | +| `score_confidence(entity)` | Calculate confidence score | Softmax probability scoring | + +**Example:** + +```python +from semantica.semantic_extract import NamedEntityRecognizer + +# Initialize with transformer model +ner = NamedEntityRecognizer( + model="transformer", # or "spacy", "stanza", "custom" + lang="en", + entities=["PERSON", "ORG", "LOC", "DATE", "MONEY"], + confidence_threshold=0.7, + use_llm_enhancement=True +) + +text = "Apple Inc. was founded by Steve Jobs in 1976." +entities = ner.extract(text) + +for entity in entities: + print(f"{entity.text} ({entity.type}, confidence={entity.confidence:.2f})") +# Output: +# Apple Inc. (ORG, confidence=0.98) +# Steve Jobs (PERSON, confidence=0.97) +# 1976 (DATE, confidence=1.00) +``` + +--- + +### RelationExtractor + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `extract(text, entities)` | Extract relationships | Hybrid: dependency parsing + ML + LLM | +| `extract_with_context(text, entities, context)` | Context-aware extraction | Contextual embeddings with attention | +| `add_custom_relation(name, pattern)` | Add custom relation type | Pattern-based rule addition | +| `classify_relation(subject, object, context)` | Classify relation type | Neural relation classification | +| `score_confidence(relation)` | Calculate relation confidence | Ensemble scoring from multiple models | + +**Algorithms:** + +- **Rule-based**: Pattern matching using dependency trees +- **ML-based**: BERT fine-tuned on relation extraction datasets +- **Hybrid**: Combines rules + ML for high precision and recall +- **LLM-based**: Few-shot prompting with GPT-4/Claude for complex relations + +**Example:** + +```python +from semantica.semantic_extract import RelationExtractor + +extractor = RelationExtractor( + strategy="hybrid", # "rule-based", "ml-based", "hybrid", "llm-based" + confidence_threshold=0.7, + max_relationships_per_entity=10 +) + +relationships = extractor.extract(text, entities) + +for rel in relationships: + print(f"{rel.subject} --[{rel.predicate}]--> {rel.object}") +# Output: +# Apple Inc. --[founded_by]--> Steve Jobs +``` + +--- + +### EventDetector + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `detect(text, entities)` | Detect events in text | Event trigger detection + argument extraction | +| `detect_batch(texts, entities_list)` | Batch event detection | Parallel processing | +| `classify_event(event)` | Classify event type | Multi-label classification | +| `extract_participants(event, entities)` | Extract event participants | Role labeling with semantic roles | +| `extract_temporal_info(event)` | Extract event timing | Temporal expression extraction | + +**Event Types Supported:** +- ACQUISITION, FOUNDING, PARTNERSHIP, ANNOUNCEMENT +- MERGER, IPO, BANKRUPTCY, LAWSUIT +- PRODUCT_LAUNCH, CONFERENCE, ELECTION +- Custom event types via configuration + +**Example:** + +```python +from semantica.semantic_extract import EventDetector + +detector = EventDetector( + event_types=["ACQUISITION", "FOUNDING", "PARTNERSHIP"], + min_confidence=0.75 +) + +events = detector.detect(text, entities) + +for event in events: + print(f"{event.type}: {event.description}") + print(f"Participants: {[p.name for p in event.participants]}") +``` + +--- + +### TripleExtractor + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `extract(text, entities, relationships)` | Extract RDF triples | Subject-predicate-object extraction from parse trees | +| `extract_from_relations(relationships)` | Convert relations to triples | Direct mapping with URI generation | +| `validate_triples(triples)` | Validate triple quality | Schema validation + consistency checking | +| `serialize(triples, format)` | Serialize to RDF format | Turtle, N-Triples, JSON-LD serialization | +| `generate_uri(entity)` | Generate URI for entity | Namespace + normalized entity name | + +**Supported Formats:** +- **RDF/XML**: W3C standard RDF format +- **Turtle**: Human-readable RDF format +- **N-Triples**: Line-based RDF format +- **JSON-LD**: JSON-based linked data format + +**Example:** + +```python +from semantica.semantic_extract import TripleExtractor + +extractor = TripleExtractor( + format="rdf", # "rdf", "turtle", "n-triples", "json-ld" + validate_triples=True, + namespace="http://example.org/" +) + +triples = extractor.extract(text, entities, relationships) + +for triple in triples: + print(f"{triple.subject} {triple.predicate} {triple.object}") +# Output: +# +# "1976"^^xsd:gYear +``` + +--- + +### CoreferenceResolver + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `resolve(text, entities)` | Resolve coreferences | Neural coreference resolution with span ranking | +| `resolve_batch(texts, entities_list)` | Batch resolution | Parallel coreference resolution | +| `cluster_mentions(mentions)` | Cluster entity mentions | Agglomerative clustering with similarity threshold | +| `merge_entities(entity_clusters)` | Merge coreferent entities | Property aggregation with conflict resolution | +| `resolve_pronouns(text, entities)` | Resolve pronoun references | Rule-based + neural pronoun resolution | + +**Algorithms:** +- **Neural Method**: Span-based neural coreference (e2e-coref, SpanBERT) +- **Rule-based Method**: Heuristic rules for simple cases (gender, number agreement) +- **Hybrid Method**: Combines neural + rules for optimal performance + +**Example:** + +```python +from semantica.semantic_extract import CoreferenceResolver + +resolver = CoreferenceResolver( + method="neural", # "rule-based", "neural", "hybrid" + resolve_pronouns=True +) + +resolved_entities = resolver.resolve(text, entities) +``` + +--- + +### SemanticAnalyzer + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `analyze_semantics(text, entities, relationships)` | Analyze semantic structure | Semantic role labeling + clustering | +| `extract_semantic_roles(text)` | Extract semantic roles | PropBank/FrameNet-based role labeling | +| `cluster_semantically(entities)` | Cluster by semantic similarity | K-means clustering on embeddings | +| `calculate_similarity(entity1, entity2)` | Calculate semantic similarity | Cosine similarity on embeddings | +| `analyze_coherence(text)` | Analyze text coherence | Coherence scoring with discourse analysis | + +**Example:** + +```python +from semantica.semantic_extract import SemanticAnalyzer + +analyzer = SemanticAnalyzer() + +analysis = analyzer.analyze_semantics( + text=text, + entities=entities, + relationships=relationships +) + +print(f"Coherence score: {analysis.coherence_score:.2f}") +print(f"Semantic roles: {analysis.semantic_roles}") +``` + +--- + +### LLMEnhancer + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `enhance_entities(text, entities)` | Enhance entities with LLM | Few-shot prompting for entity enrichment | +| `enhance_relationships(text, relationships)` | Enhance relationships | LLM-based relationship validation and enrichment | +| `generate_descriptions(entity)` | Generate entity descriptions | LLM text generation | +| `validate_extractions(extractions)` | Validate with LLM | LLM-based quality checking | +| `classify_with_llm(text, labels)` | LLM-based classification | Zero/few-shot classification | + +**Supported LLM Providers:** +- OpenAI (GPT-4, GPT-3.5-turbo) +- Anthropic (Claude 3 Opus, Sonnet, Haiku) +- Google (Gemini Pro, Ultra) +- Groq (Mixtral, Llama) +- Ollama (Local models) + +**Example:** + +```python +from semantica.semantic_extract import LLMEnhancer + +enhancer = LLMEnhancer( + provider="openai", + model="gpt-4", + temperature=0.1 +) + +enhanced_entities = enhancer.enhance_entities(text, entities) +enhanced_relationships = enhancer.enhance_relationships(text, relationships) +``` + +--- + +### ExtractionValidator + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `validate(text, entities, relationships, triples)` | Validate all extractions | Multi-level validation pipeline | +| `validate_entities(entities)` | Validate entity quality | Schema validation + consistency checks | +| `validate_relationships(relationships)` | Validate relationships | Type checking + logical validation | +| `validate_triples(triples)` | Validate RDF triples | RDF schema validation | +| `check_consistency(extractions)` | Check extraction consistency | Cross-validation between extraction types | + +**Validation Checks:** +- **Schema Validation**: Type checking against ontology +- **Consistency Checking**: Cross-validation between entities and relationships +- **Quality Scoring**: Confidence-based quality metrics +- **Completeness**: Required property validation + +--- + +## Configuration + +```yaml +# config.yaml - Semantic Extract Configuration + +semantic_extract: + ner: + model: transformer # spacy, stanza, transformer, llm + lang: en + entities: [PERSON, ORG, LOC, DATE, MONEY, PRODUCT] + confidence_threshold: 0.7 + use_llm_enhancement: false + + relation_extraction: + strategy: hybrid # rule-based, ml-based, hybrid, llm-based + confidence_threshold: 0.7 + max_relationships_per_entity: 10 + + event_detection: + event_types: [ACQUISITION, FOUNDING, PARTNERSHIP, ANNOUNCEMENT] + min_confidence: 0.75 + extract_participants: true + extract_temporal: true + + triple_extraction: + format: turtle # rdf, turtle, n-triples, json-ld + validate_triples: true + namespace: "http://example.org/" + + coreference: + method: neural # rule-based, neural, hybrid + resolve_pronouns: true + cluster_threshold: 0.85 + + llm: + provider: openai + model: gpt-4 + temperature: 0.1 + max_tokens: 4000 +``` + +--- + +## Performance Characteristics + +### Entity Extraction +- **Transformer models**: High accuracy, moderate speed +- **spaCy models**: Fast, good accuracy for common entities +- **LLM-based**: Highest accuracy, slower, best for complex domains + +### Relationship Extraction +- **Rule-based**: Fast, high precision, lower recall +- **ML-based**: Balanced precision/recall +- **Hybrid**: Best overall performance +- **LLM-based**: Highest quality, slower + +### Scalability +- Batch processing supported for all extractors +- Parallel processing for multi-document extraction +- GPU acceleration available for transformer models + +--- + +## See Also + +- [Knowledge Graph Module](kg.md) - Build graphs from extracted data +- [Ontology Module](ontology.md) - Define extraction schemas +- [Core Module](core.md) - Framework orchestration diff --git a/docs/reference/triple_store.md b/docs/reference/triple_store.md index bd34b4b1..dad5b708 100644 --- a/docs/reference/triple_store.md +++ b/docs/reference/triple_store.md @@ -1,3 +1,114 @@ -# Triple Store +# Triple Store Module -::: semantica.triple_store +Store and query RDF triples with support for SPARQL queries and semantic reasoning using industry-standard triple stores. + +## Overview + +- **RDF Storage**: Store subject-predicate-object triples +- **SPARQL Queries**: W3C-compliant query language +- **Reasoning**: RDFS and OWL reasoning +- **Multiple Backends**: RDFLib, Jena, Virtuoso, GraphDB +- **Federation**: Query across multiple triple stores + +--- + +## Algorithms Used + +### Query Algorithms +- **SPARQL Query Optimization**: Join reordering with selectivity estimation +- **Triple Pattern Matching**: Index-based lookup with B+ trees +- **Graph Pattern Matching**: Subgraph isomorphism +- **Query Planning**: Cost-based optimization + +### Indexing +- **SPO Index**: Subject-Predicate-Object index +- **POS Index**: Predicate-Object-Subject index +- **OSP Index**: Object-Subject-Predicate index +- **Six-Index Scheme**: All permutations for optimal query performance + +--- + +## Main Classes + +### TripleStore + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `add_triple(subject, predicate, object)` | Add single triple | Index insertion | +| `add_triples(triples)` | Batch add triples | Bulk index insertion | +| `query(sparql_query)` | Execute SPARQL query | Query optimization + execution | +| `delete(pattern)` | Delete matching triples | Pattern matching + deletion | +| `serialize(format)` | Serialize to RDF format | Format-specific serialization | + +**Example:** + +```python +from semantica.triple_store import TripleStore + +store = TripleStore(backend="rdflib") + +# Add triples +store.add_triple( + subject="http://example.org/AppleInc", + predicate="http://example.org/foundedBy", + object="http://example.org/SteveJobs" +) + +# SPARQL query +results = store.query(""" + SELECT ?company ?founder WHERE { + ?company ?founder . + } +""") + +for row in results: + print(f"{row['company']} founded by {row['founder']}") +``` + +--- + +### SPARQLQueryEngine + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `execute(query)` | Execute SPARQL query | Parse + optimize + execute | +| `parse_query(sparql)` | Parse SPARQL syntax | SPARQL parser | +| `optimize_query(query)` | Optimize query plan | Join reordering | +| `explain_query(query)` | Explain query plan | Query plan visualization | + +**SPARQL Query Types:** +- **SELECT**: Retrieve variable bindings +- **CONSTRUCT**: Build RDF graph +- **ASK**: Boolean query +- **DESCRIBE**: Describe resources + +--- + +## Configuration + +```yaml +# config.yaml - Triple Store Configuration + +triple_store: + backend: rdflib # rdflib, jena, virtuoso, graphdb + + indexing: + schemes: [SPO, POS, OSP] # Index permutations + + query: + optimize: true + timeout: 30 # seconds +``` + +--- + +## See Also + +- [Knowledge Graph Module](kg.md) +- [Ontology Module](ontology.md) diff --git a/docs/reference/utils.md b/docs/reference/utils.md index 38aabf2d..027b5632 100644 --- a/docs/reference/utils.md +++ b/docs/reference/utils.md @@ -1,3 +1,180 @@ -# Utils +# Utils Module -::: semantica.utils +Utility functions and helper classes for common operations including file handling, text processing, and data manipulation. + +## Overview + +- **File Operations**: File I/O, path manipulation, compression +- **Text Utilities**: String manipulation, encoding, hashing +- **Data Utilities**: JSON/YAML handling, serialization +- **Logging**: Structured logging with multiple handlers +- **Validation**: Schema validation, type checking + +--- + +## Algorithms Used + +### File Operations +- **Hashing**: MD5, SHA256 for file integrity checking +- **Compression**: GZIP, LZMA algorithms for file compression +- **Chunking**: Fixed-size or sliding window chunking for large files + +### Text Processing +- **Tokenization**: Whitespace and punctuation-based tokenization +- **Similarity**: Jaccard similarity, Cosine similarity +- **Hashing**: MurmurHash3 for fast non-cryptographic hashing + +### Data Serialization +- **JSON**: Fast JSON encoding/decoding +- **YAML**: Human-readable configuration format +- **Pickle**: Python object serialization + +--- + +## Main Classes + +### FileUtils + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `list_files(directory, pattern)` | List files matching pattern | Recursive directory traversal | +| `read_file(path)` | Read file contents | Buffered I/O | +| `write_file(path, content)` | Write to file | Atomic write with temp file | +| `hash_file(path, algorithm)` | Calculate file hash | Streaming hash calculation | +| `compress_file(path, format)` | Compress file | GZIP/LZMA compression | + +**Example:** + +```python +from semantica.utils import FileUtils + +# List files +files = FileUtils.list_files( + directory="documents/", + pattern="*.pdf", + recursive=True +) + +# Hash file +file_hash = FileUtils.hash_file("document.pdf", algorithm="sha256") +print(f"SHA256: {file_hash}") + +# Compress file +FileUtils.compress_file("large_file.json", format="gzip") +``` + +--- + +### TextUtils + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `clean_text(text)` | Clean and normalize text | Regex-based cleaning | +| `tokenize(text, method)` | Tokenize text | Whitespace/punctuation splitting | +| `calculate_similarity(text1, text2)` | Calculate text similarity | Jaccard or Cosine similarity | +| `hash_text(text)` | Hash text string | MurmurHash3 | +| `remove_stopwords(text, language)` | Remove stopwords | Dictionary-based filtering | + +**Example:** + +```python +from semantica.utils import TextUtils + +# Clean text +text = " Hello, World! \n\n " +cleaned = TextUtils.clean_text(text) +print(cleaned) # "Hello, World!" + +# Calculate similarity +similarity = TextUtils.calculate_similarity( + "machine learning", + "deep learning", + method="jaccard" +) +print(f"Similarity: {similarity:.2f}") +``` + +--- + +### DataUtils + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `load_json(path)` | Load JSON file | JSON parsing | +| `save_json(data, path)` | Save to JSON | JSON serialization | +| `load_yaml(path)` | Load YAML file | YAML parsing | +| `save_yaml(data, path)` | Save to YAML | YAML serialization | +| `validate_schema(data, schema)` | Validate against schema | JSON Schema validation | + +**Example:** + +```python +from semantica.utils import DataUtils + +# Load/save JSON +data = DataUtils.load_json("config.json") +DataUtils.save_json(data, "output.json", pretty=True) + +# Load/save YAML +config = DataUtils.load_yaml("config.yaml") +DataUtils.save_yaml(config, "output.yaml") + +# Validate schema +is_valid = DataUtils.validate_schema( + data=data, + schema={"type": "object", "properties": {...}} +) +``` + +--- + +### LoggerUtils + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `get_logger(name)` | Get logger instance | Logger factory | +| `configure_logging(level, format)` | Configure logging | Handler setup | + +--- + +## Configuration + +```yaml +# config.yaml - Utils Configuration + +utils: + logging: + level: INFO # DEBUG, INFO, WARNING, ERROR + format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" + handlers: + - console + - file + + file: + default_encoding: utf-8 + buffer_size: 8192 + + text: + default_language: en + remove_stopwords: true +``` + +--- + +## See Also + +- [Core Module](core.md) - Framework orchestration +- [Ingest Module](ingest.md) - Data ingestion +- [Parse Module](parse.md) - Document parsing diff --git a/docs/reference/vector_store.md b/docs/reference/vector_store.md index 9f982c62..9f9df327 100644 --- a/docs/reference/vector_store.md +++ b/docs/reference/vector_store.md @@ -1,3 +1,280 @@ -# Vector Store +# Vector Store Module -::: semantica.vector_store +> **Store and search vector embeddings with lightning-fast approximate nearest neighbor search.** + +--- + +## 🎯 Overview + +
+ +- :material-database:{ .lg .middle } **Vector Storage** + + --- + + Efficient storage of high-dimensional vectors with multiple backend support + +- :material-magnify:{ .lg .middle } **Similarity Search** + + --- + + Fast ANN search with HNSW, IVF, and PQ algorithms + +- :material-merge:{ .lg .middle } **Hybrid Search** + + --- + + Combine vector search with keyword/metadata filtering using RRF + +- :material-cog:{ .lg .middle } **Index Management** + + --- + + Multiple index types optimized for different use cases + +- :material-flash:{ .lg .middle } **Batch Operations** + + --- + + Efficient bulk insert and search with parallel processing + +- :material-cloud:{ .lg .middle } **Multiple Backends** + + --- + + FAISS, Pinecone, Qdrant, Weaviate, Milvus support + +
+ +!!! tip "Choosing the Right Index" + - **Small datasets (<10K)**: Use Flat for exact search + - **Medium (10K-1M)**: Use IVF or HNSW + - **Large (>1M)**: Use HNSW with GPU + - **Memory constrained**: Use PQ for compression + +--- + +## ⚙️ Algorithms Used + +### Indexing Algorithms +- **Flat (Exact Search)**: Brute-force linear scan, O(n*d) complexity +- **IVF (Inverted File Index)**: Clustering-based search with k-means, O(√n*d) complexity +- **HNSW (Hierarchical Navigable Small World)**: Graph-based ANN, O(log n) search complexity +- **PQ (Product Quantization)**: Compression-based search, reduces memory footprint +- **LSH (Locality Sensitive Hashing)**: Hash-based approximate search + +### Similarity Metrics +- **Cosine Similarity**: `cos(θ) = (A·B) / (||A|| * ||B||)` +- **Euclidean Distance**: `d = √(Σ(ai - bi)²)` +- **Dot Product**: `A·B = Σ(ai * bi)` +- **Manhattan Distance**: `d = Σ|ai - bi|` + +### Hybrid Search Algorithms +- **RRF (Reciprocal Rank Fusion)**: `score = Σ(1/(k + rank_i))` where k=60 +- **Weighted Combination**: `score = α*vector_score + (1-α)*keyword_score` +- **Cascade Filtering**: Vector search → metadata filtering → reranking + +--- + +## Main Classes + +### VectorStore + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `store(embeddings, documents, metadata)` | Store vectors with metadata | Batch insertion with index building | +| `search(query_vector, top_k, filters)` | Search similar vectors | ANN search with optional filtering | +| `delete(ids)` | Delete vectors by ID | Index update with tombstoning | +| `update(id, vector, metadata)` | Update vector/metadata | In-place update or delete+insert | +| `create_index(index_type, params)` | Create search index | Index-specific construction algorithm | +| `rebuild_index()` | Rebuild index from scratch | Full index reconstruction | + +**Supported Backends:** + +| Backend | Index Types | Best For | +|---------|-------------|----------| +| **FAISS** | Flat, IVF, HNSW, PQ | High performance, local deployment | +| **Pinecone** | Proprietary | Managed cloud service | +| **Qdrant** | HNSW | Production-ready, filtering support | +| **Weaviate** | HNSW | GraphQL API, hybrid search | +| **Milvus** | IVF, HNSW | Distributed, large-scale | + +**Example:** + +```python +from semantica.vector_store import VectorStore + +# Initialize with FAISS backend +store = VectorStore( + backend="faiss", + index_type="HNSW", # Flat, IVF, HNSW, PQ + metric="cosine", # cosine, euclidean, dot_product + dimension=1536 +) + +# Store embeddings +store.store( + embeddings=embeddings, + documents=documents, + metadata=[{"source": "doc1.pdf", "page": 1}, ...] +) + +# Search +results = store.search( + query_vector=query_embedding, + top_k=10, + filters={"source": "doc1.pdf"} +) + +for result in results: + print(f"Score: {result.score:.3f}, Doc: {result.document}") +``` + +--- + +### HybridSearch + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `search(query, top_k)` | Hybrid search | Vector + keyword search with RRF fusion | +| `vector_search(query_vector, top_k)` | Vector-only search | ANN search | +| `keyword_search(query_text, top_k)` | Keyword-only search | BM25 or TF-IDF | +| `combine_results(vector_results, keyword_results)` | Merge results | RRF or weighted combination | +| `rerank(results, query)` | Rerank results | Cross-encoder reranking | + +**Reciprocal Rank Fusion (RRF):** +``` +RRF_score(d) = Σ(1/(k + rank_i(d))) +where k = 60 (constant), rank_i(d) = rank of document d in result set i +``` + +**Example:** + +```python +from semantica.vector_store import HybridSearch + +hybrid = HybridSearch( + vector_store=store, + keyword_index=keyword_index, + fusion_method="rrf", # rrf, weighted, cascade + vector_weight=0.7 # for weighted fusion +) + +results = hybrid.search( + query="machine learning applications", + top_k=10 +) +``` + +--- + +### VectorRetriever + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `retrieve(query, top_k)` | Retrieve relevant documents | Vector search + document fetching | +| `retrieve_batch(queries, top_k)` | Batch retrieval | Parallel vector search | +| `retrieve_with_context(query, context_size)` | Retrieve with context | Sliding window context expansion | +| `filter_by_metadata(results, filters)` | Filter by metadata | Post-search filtering | + +--- + +## Index Types Comparison + +### FAISS Index Types + +| Index Type | Build Time | Search Time | Memory | Recall | Use Case | +|------------|------------|-------------|--------|--------|----------| +| **Flat** | O(1) | O(n*d) | High | 1.00 | Small datasets, exact search | +| **IVF** | O(n*d) | O(√n*d) | Medium | 0.95 | Medium datasets, good balance | +| **HNSW** | O(n*log n*d) | O(log n*d) | High | 0.98 | Large datasets, fast search | +| **PQ** | O(n*d) | O(n) | Low | 0.85 | Memory-constrained, compression | + +### Index Parameters + +**IVF Parameters:** +- `nlist`: Number of clusters (√n to 4√n recommended) +- `nprobe`: Number of clusters to search (1-nlist) + +**HNSW Parameters:** +- `M`: Number of connections per layer (4-64, default: 16) +- `efConstruction`: Construction time accuracy (100-500) +- `efSearch`: Search time accuracy (efConstruction to 2*efConstruction) + +**PQ Parameters:** +- `m`: Number of subquantizers (dimension/m should be divisible) +- `nbits`: Bits per subquantizer (8 is standard) + +--- + +## Configuration + +```yaml +# config.yaml - Vector Store Configuration + +vector_store: + backend: faiss # faiss, pinecone, qdrant, weaviate, milvus + + faiss: + index_type: HNSW # Flat, IVF, HNSW, PQ + metric: cosine # cosine, euclidean, dot_product + dimension: 1536 + + # HNSW parameters + hnsw_m: 16 + hnsw_ef_construction: 200 + hnsw_ef_search: 100 + + # IVF parameters + ivf_nlist: 100 + ivf_nprobe: 10 + + hybrid_search: + fusion_method: rrf # rrf, weighted, cascade + vector_weight: 0.7 + keyword_weight: 0.3 + enable_reranking: true + + batch_operations: + batch_size: 1000 + parallel_workers: 4 +``` + +--- + +## Performance Characteristics + +### Search Complexity + +| Index Type | Build | Search | Memory | +|------------|-------|--------|--------| +| Flat | O(1) | O(n*d) | O(n*d) | +| IVF | O(n*d*k) | O(√n*d) | O(n*d) | +| HNSW | O(n*log n*d) | O(log n*d) | O(n*d*M) | +| PQ | O(n*d) | O(n*m) | O(n*m) | + +where n = vectors, d = dimensions, k = clusters, M = HNSW connections, m = subquantizers + +### Scalability + +- **Small (<10K vectors)**: Use Flat for exact search +- **Medium (10K-1M vectors)**: Use IVF or HNSW +- **Large (>1M vectors)**: Use HNSW with GPU or distributed systems +- **Very Large (>10M vectors)**: Use distributed backends (Milvus, Weaviate) + +--- + +## See Also + +- [Embeddings Module](embeddings.md) - Generate vector embeddings +- [Knowledge Graph Module](kg.md) - Graph-based retrieval +- [Core Module](core.md) - Framework orchestration diff --git a/docs/reference/visualization.md b/docs/reference/visualization.md index 872c9937..6bb1f8c9 100644 --- a/docs/reference/visualization.md +++ b/docs/reference/visualization.md @@ -1,3 +1,181 @@ -# Visualization +# Visualization Module -::: semantica.visualization +Visualize knowledge graphs, embeddings, and analytics with interactive and static visualizations using multiple rendering engines. + +## Overview + +- **Graph Visualization**: Interactive and static knowledge graph rendering +- **Embedding Visualization**: t-SNE, UMAP, PCA for high-dimensional data +- **Analytics Visualization**: Charts, plots, and dashboards +- **Temporal Visualization**: Time-series and evolution visualization +- **Export Formats**: HTML, PNG, SVG, PDF + +--- + +## Algorithms Used + +### Dimensionality Reduction +- **t-SNE (t-Distributed Stochastic Neighbor Embedding)**: Non-linear dimensionality reduction, preserves local structure +- **UMAP (Uniform Manifold Approximation and Projection)**: Faster than t-SNE, preserves global + local structure +- **PCA (Principal Component Analysis)**: Linear dimensionality reduction, `X_reduced = X * eigenvectors` + +### Graph Layout Algorithms +- **Force-Directed Layout**: Spring-electrical model with Fruchterman-Reingold algorithm +- **Hierarchical Layout**: Tree-based layout with Sugiyama framework +- **Circular Layout**: Nodes arranged in circle, edges minimize crossings +- **Kamada-Kawai**: Energy-based layout minimizing edge length variance + +--- + +## Main Classes + +### KGVisualizer + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `visualize(graph, output)` | Visualize knowledge graph | Force-directed layout with PyVis/Cytoscape | +| `render_interactive(graph)` | Interactive HTML visualization | D3.js/PyVis rendering | +| `render_static(graph, format)` | Static image rendering | Graphviz/Matplotlib rendering | +| `export(graph, filename, format)` | Export visualization | Format-specific export (HTML/PNG/SVG) | +| `customize_style(node_style, edge_style)` | Customize appearance | Style application | + +**Supported Engines:** +- **PyVis**: Interactive HTML with physics simulation +- **Cytoscape.js**: Web-based graph visualization +- **Graphviz**: Static high-quality diagrams +- **Matplotlib**: Python-native plotting +- **Plotly**: Interactive web visualizations + +**Example:** + +```python +from semantica.visualization import KGVisualizer + +visualizer = KGVisualizer( + engine="pyvis", # pyvis, cytoscape, graphviz, matplotlib + layout="force_directed", # force_directed, hierarchical, circular + width="100%", + height="800px" +) + +# Interactive visualization +visualizer.visualize( + graph=kg, + output="graph.html", + show_physics=True, + node_color_by="type", + edge_width_by="weight" +) + +# Static visualization +visualizer.render_static( + graph=kg, + format="png", + output="graph.png", + dpi=300 +) +``` + +--- + +### EmbeddingVisualizer + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `visualize(embeddings, method)` | Visualize embeddings | Dimensionality reduction + scatter plot | +| `plot_tsne(embeddings, perplexity)` | t-SNE visualization | t-SNE with configurable perplexity | +| `plot_umap(embeddings, n_neighbors)` | UMAP visualization | UMAP with neighbor parameter | +| `plot_pca(embeddings, n_components)` | PCA visualization | PCA to 2D/3D | +| `plot_clusters(embeddings, labels)` | Cluster visualization | Color-coded scatter plot | + +**t-SNE Parameters:** +- `perplexity`: Balance between local and global structure (5-50) +- `learning_rate`: Step size (10-1000) +- `n_iter`: Number of iterations (250-1000) + +**UMAP Parameters:** +- `n_neighbors`: Local neighborhood size (2-100) +- `min_dist`: Minimum distance between points (0.0-0.99) +- `metric`: Distance metric (euclidean, cosine, manhattan) + +**Example:** + +```python +from semantica.visualization import EmbeddingVisualizer + +visualizer = EmbeddingVisualizer() + +# t-SNE visualization +visualizer.plot_tsne( + embeddings=embeddings, + labels=labels, + perplexity=30, + output="tsne.html" +) + +# UMAP visualization +visualizer.plot_umap( + embeddings=embeddings, + n_neighbors=15, + min_dist=0.1, + output="umap.html" +) +``` + +--- + +### AnalyticsVisualizer + + +**Methods:** + +| Method | Description | Algorithm | +|--------|-------------|-----------| +| `plot_metrics(metrics)` | Plot graph metrics | Bar/line charts | +| `plot_distribution(data, bins)` | Plot distributions | Histogram generation | +| `plot_timeline(events, timeline)` | Plot temporal data | Time-series visualization | +| `create_dashboard(components)` | Create dashboard | Multi-panel layout | + +--- + +## Configuration + +```yaml +# config.yaml - Visualization Configuration + +visualization: + kg: + engine: pyvis # pyvis, cytoscape, graphviz, matplotlib + layout: force_directed + width: "100%" + height: "800px" + physics_enabled: true + node_size_by: degree + node_color_by: type + edge_width_by: weight + + embeddings: + method: umap # tsne, umap, pca + n_components: 2 + perplexity: 30 # for t-SNE + n_neighbors: 15 # for UMAP + + export: + format: html # html, png, svg, pdf + dpi: 300 # for raster formats + transparent_background: false +``` + +--- + +## See Also + +- [Knowledge Graph Module](kg.md) +- [Embeddings Module](embeddings.md) +- [Export Module](export.md) diff --git a/remove_mkdocstrings.ps1 b/remove_mkdocstrings.ps1 new file mode 100644 index 00000000..3a629b63 --- /dev/null +++ b/remove_mkdocstrings.ps1 @@ -0,0 +1,19 @@ +# Remove all mkdocstrings directives from reference docs +# These are causing griffe alias resolution errors + +$files = Get-ChildItem "c:\Users\Mohd Kaif\semantica\docs\reference\*.md" + +foreach ($file in $files) { + $content = Get-Content $file.FullName -Raw + + # Remove mkdocstrings directives (:::semantica.module.Class blocks) + $content = $content -replace '(?m)^::: semantica\.[^\r\n]+\r?\n(?: options:\r?\n(?: [^\r\n]+\r?\n)*)?', '' + + # Clean up extra blank lines + $content = $content -replace '(\r?\n){4,}', "`r`n`r`n`r`n" + + Set-Content -Path $file.FullName -Value $content -NoNewline + Write-Host "Processed: $($file.Name)" +} + +Write-Host "`nDone! All mkdocstrings directives removed."