diff --git a/README.md b/README.md index c0928fb2..c48e6561 100644 --- a/README.md +++ b/README.md @@ -480,8 +480,8 @@ is_valid = kg_manager.verify_checksum(snapshot) - ๐Ÿ“Š **Detailed Diffs** โ€” Entity-level and relationship-level change tracking - โœ… **Data Integrity** โ€” SHA-256 checksums with tamper detection - ๐Ÿ“ **Standardized Metadata** โ€” ChangeLogEntry with author, timestamp, description -- โšก **Performance Tested** โ€” Benchmarked with 10k entities -- ๐Ÿงช **Test Coverage** โ€” 104 tests covering core functionality +- โšก **Performance Tested** โ€” Tested with large-scale entity datasets +- ๐Ÿงช **Test Coverage** โ€” Comprehensive test coverage covering core functionality **Compliance Note:** Provides technical infrastructure (audit trails, checksums, temporal tracking) that supports compliance efforts for HIPAA, SOX, FDA 21 CFR Part 11. Organizations must implement additional policies and procedures for full regulatory compliance. @@ -576,7 +576,7 @@ from semantica.llms import Groq context = AgentContext( vector_store=VectorStore(backend="faiss"), knowledge_graph=GraphStore(backend="neo4j"), # Optional: Use persistent graph - hybrid_alpha=0.75 # 75% weight to Knowledge Graph, 25% to Vector + hybrid_alpha=0.75 # Balanced weight between Knowledge Graph and Vector ) # Build Context Graph from entities and relationships diff --git a/docs/LIBS_README.md b/docs/LIBS_README.md index dadf4607..dcf1b3ba 100644 --- a/docs/LIBS_README.md +++ b/docs/LIBS_README.md @@ -1380,11 +1380,11 @@ result = semantica.build_knowledge_base(["document.pdf"]) ## ๐Ÿš€ Performance ### Benchmarks -- **Processing Speed**: 1000+ documents per minute +- **Processing Speed**: Optimized for high-throughput document processing - **Memory Usage**: Optimized for large-scale processing -- **Accuracy**: 95%+ entity extraction accuracy +- **Accuracy**: High accuracy entity extraction - **Scalability**: Horizontal scaling support -- **Latency**: Sub-second query response times +- **Latency**: Fast query response times ### Optimization - **Parallel Processing**: Multi-threaded and multi-process support diff --git a/docs/change_management_usage.md b/docs/change_management_usage.md deleted file mode 100644 index e69de29b..00000000 diff --git a/docs/changelog.md b/docs/changelog.md deleted file mode 100644 index 628614e6..00000000 --- a/docs/changelog.md +++ /dev/null @@ -1,3 +0,0 @@ -# Changelog - ---8<-- "CHANGELOG.md" diff --git a/docs/community.md b/docs/community.md index bdb8d879..5b7290bd 100644 --- a/docs/community.md +++ b/docs/community.md @@ -1,86 +1,73 @@ -# Community +๏ปฟ# Community -Welcome to the Semantica community! - -!!! info "Join Us" - We're building an open, collaborative community around semantic AI and knowledge graphs. +**Connect with the Semantica community for support, collaboration, and learning.** --- -## ๐Ÿ’ฌ Communication Channels +## Get Help & Support -### GitHub - -- **[Issues](https://github.com/Hawksight-AI/semantica/issues)** - Bug reports, feature requests, questions +### GitHub Issues +- **[Report Issues](https://github.com/Hawksight-AI/semantica/issues)** - Bug reports and feature requests - **[Pull Requests](https://github.com/Hawksight-AI/semantica/pulls)** - Code contributions -- **[Releases](https://github.com/Hawksight-AI/semantica/releases)** - Release announcements +- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Questions and ideas -### Contact - -- **GitHub Issues**: [Create an issue](https://github.com/Hawksight-AI/semantica/issues) for all communication -- **GitHub Security Advisories**: [Report security issues](https://github.com/Hawksight-AI/semantica/security/advisories/new) +### Security Issues +- **[Report Security](https://github.com/Hawksight-AI/semantica/security/advisories/new)** - Security vulnerabilities --- -## ๐Ÿค Community Values +## Community Guidelines -- **Respect**: Treat everyone with respect and kindness -- **Inclusion**: Welcome people of all backgrounds -- **Collaboration**: Work together to build something great -- **Learning**: Share knowledge and help others -- **Openness**: Transparent communication +### Our Values +- **Respect** - Treat everyone with kindness +- **Inclusion** - Welcome all backgrounds and experience levels +- **Collaboration** - Work together to build great things +- **Learning** - Share knowledge and help others grow ---- - -## ๐Ÿ“– Code of Conduct - -We have a [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md) that all community members must follow. +### Code of Conduct +We follow the [Contributor Covenant Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md). ### Reporting Issues - -If you experience unacceptable behavior: +If you experience unacceptable behavior, please: 1. Document what happened -2. Contact maintainers through [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with "[CoC]" prefix +2. Create an issue with "[CoC]" prefix 3. We'll investigate and respond appropriately --- -## ๐ŸŽฏ Getting Help +## Contributing -### Before Asking +### Ways to Contribute +- **Code** - Fix bugs, add features, improve documentation +- **Documentation** - Improve guides, fix typos, add examples +- **Testing** - Report issues, write tests, validate fixes +- **Community** - Help others, share knowledge, provide feedback -1. Check the [documentation](index.md) -2. Search [GitHub issues](https://github.com/Hawksight-AI/semantica/issues) -3. Review the [FAQ](faq.md) -4. Check the [cookbook](cookbook.md) - -### Asking Questions - -When asking for help: -- Be specific about your problem -- Include environment details -- Share what you've tried -- Provide code examples -- Be patient +### Getting Started +1. **Fork** the repository +2. **Create** a feature branch +3. **Make** your changes +4. **Test** your changes +5. **Submit** a pull request --- -## ๐Ÿ† Recognition +## Stay Connected -All contributors are recognized in: -- [CONTRIBUTORS.md](https://github.com/Hawksight-AI/semantica/blob/main/CONTRIBUTORS.md) -- GitHub contributors page -- Release notes (for significant contributions) +### Follow the Project +- **[GitHub](https://github.com/Hawksight-AI/semantica)** - Source code and releases +- **[PyPI](https://pypi.org/project/semantica/)** - Package information and downloads + +### Share Your Work +- **Blog Posts** - Write about your Semantica projects +- **Tutorials** - Create guides and examples +- **Projects** - Share what you've built with Semantica --- -## ๐Ÿ“š Resources +## Need Help? +- **[Getting Started](getting-started.md)** - Quick start guide +- **[FAQ](faq.md)** - Frequently asked questions - **[Contributing Guide](contributing.md)** - How to contribute -- **[Governance](governance.md)** - Project governance -- **[Community Projects](community-projects.md)** - Community showcase - ---- - -!!! success "Thank You!" - Thank you for being part of the Semantica community! ๐ŸŽ‰ +- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Ask questions diff --git a/docs/concepts.md b/docs/concepts.md index e95c31cb..b32ec0a3 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -1,2386 +1,372 @@ -# Core Concepts +๏ปฟ# Core Concepts -Understand the fundamental concepts behind Semantica. This guide covers the theoretical foundations, key components, and best practices for building semantic applications. +**Learn the fundamental concepts behind Semantica in simple, practical terms.** -!!! info "About This Guide" - This guide provides a comprehensive overview of the core concepts in Semantica. Each concept includes definitions, visual diagrams, practical examples, and guidance on when to use them. +!!! tip "Quick Start" + New to Semantica? Start with [Getting Started](getting-started.md) for hands-on examples. --- -[TOC] +## What is Semantica? +Semantica transforms unstructured data (documents, web pages, reports) into **knowledge graphs** - structured databases that AI systems can understand and reason about. +**What it does:** +- **Reads** documents, PDFs, web pages, databases +- **Extracts** entities (people, companies, dates) and relationships +- **Builds** connected knowledge graphs +- **Enables** AI to reason with structured knowledge -## Core Concepts +--- -### 1. Knowledge Graphs +## Core Architecture -!!! abstract "Definition" - A **knowledge graph** is a structured representation of entities (nodes) and their relationships (edges) with properties and attributes. It transforms unstructured data into a queryable, interconnected knowledge base. +Semantica uses a **layered architecture** - use only what you need:
-- **Nodes (Entities)** +- **Input Layer** + --- - Represent real-world objects, concepts, or events. - *Examples*: People, Organizations, Locations, Concepts + + Data ingestion and preparation + + **Modules**: Ingest, Parse, Split, Normalize -- **Edges (Relationships)** +- **Semantic Layer** + --- - Represent connections between entities. - *Examples*: `works_for`, `located_in`, `founded_by`, `causes` + + Intelligence and understanding + + **Modules**: Semantic Extract, Knowledge Graph, Ontology, Reasoning -- **Properties** +- **Storage Layer** + --- - Attributes of entities and relationships. - *Examples*: Name, Date, Confidence Score, Source + + Persistent data storage + + **Modules**: Embeddings, Vector Store, Graph Store -- **Metadata** +- **Quality Layer** + --- - Additional information about the data. - *Examples*: Source documents, timestamps, extraction methods + + Data quality and consistency + + **Modules**: Deduplication, Conflicts + +- **Context & Memory** + + --- + + Agent memory and foundation data + + **Modules**: Context, Seed, LLM Providers + +- **Output & Orchestration** + + --- + + Export, visualization, and workflows + + **Modules**: Export, Visualization, Pipeline
-**Visual Example**: +--- -```mermaid -graph LR - A[Apple Inc.
Organization
Founded: 1976] -->|founded_by| B[Steve Jobs
Person
1955-2011] - A -->|located_in| C[Cupertino
Location
City] - C -->|in_state| D[California
Location
State] - A -->|has_ceo| E[Tim Cook
Person
CEO since 2011] - - style A fill:#e3f2fd,stroke:#1565c0,stroke-width:2px - style B fill:#fff3e0,stroke:#ef6c00,stroke-width:2px - style C fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px - style D fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px - style E fill:#fff3e0,stroke:#ef6c00,stroke-width:2px +## Knowledge Graphs + +The foundation of Semantica - turning data into structured knowledge. + +### What is a Knowledge Graph? + +A knowledge graph represents real-world information as: +- **Nodes** (entities): People, companies, locations, dates +- **Edges** (relationships): works_for, located_in, founded_by +- **Properties**: Name, date, confidence score, source + +### Why Knowledge Graphs? + +- **Searchable**: Find information instantly +- **Connectable**: Discover hidden relationships +- **Queryable**: Ask complex questions +- **Explainable**: Trace answers back to sources + +--- + +## Entity Extraction (NER) + +Finding and classifying entities in text. + +### What it does: +- Scans text for people, organizations, locations, dates +- Classifies each entity by type +- Assigns confidence scores +- Tracks source provenance + +### Example Output: +```python +# From: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino." +{ + "entities": [ + {"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98}, + {"text": "Steve Jobs", "type": "PERSON", "confidence": 0.99}, + {"text": "1976", "type": "DATE", "confidence": 0.95}, + {"text": "Cupertino", "type": "LOCATION", "confidence": 0.97} + ] +} ``` - -**Learn by Doing:** - -Knowledge graphs are best understood through hands-on practice. The following cookbooks provide step-by-step tutorials: - -- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build a knowledge graph from a document - - **Topics**: Entity extraction, relationship extraction, graph construction, visualization - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Learning the basics, understanding graph structure - -- **[Building Knowledge Graphs](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Advanced graph construction techniques - - **Topics**: Graph building, entity merging, conflict resolution, temporal graphs - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Production graph construction, multi-source integration - -- **[Multi-Source Data Integration](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Merge knowledge from multiple sources - - **Topics**: Multi-source integration, entity resolution, conflict handling - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Building unified knowledge graphs from diverse sources - parsed = parser.parse_document(doc) if isinstance(doc, str) else doc - text = parsed.get("full_text", "") if isinstance(parsed, dict) else str(parsed) - entities = ner.extract_entities(text) - relationships = rel_extractor.extract_relations(text, entities=entities) - all_entities.extend(entities) - all_relationships.extend(relationships) - return builder.build_graph(entities=all_entities, relationships=all_relationships) - - kg_news = build_kg_from_source("news_articles/") - kg_reports = build_kg_from_source("financial_reports/") - )["knowledge_graph"] - - # Merge into a unified knowledge graph - from semantica.kg import GraphBuilder - builder = GraphBuilder( - merge_entities=True, - entity_resolution_strategy="semantic" # Use embeddings for matching - ) - - unified_kg = builder.merge([kg_news, kg_reports]) - - print(f"Merged graph: {len(unified_kg['entities'])} unique entities") - print(f"Deduplicated from {len(kg_news['entities']) + len(kg_reports['entities'])} total") - ``` -=== "Graph Analytics" - Analyze the structure and properties of your knowledge graph: - - ```python - from semantica.kg import ( - GraphAnalyzer, - CentralityCalculator, - CommunityDetector, - ConnectivityAnalyzer, - analyze_graph, - calculate_centrality, - detect_communities, - analyze_connectivity - ) - - # Method 1: Using convenience functions - analysis = analyze_graph(kg, method="default") - centrality_scores = calculate_centrality(kg, method="degree") - communities = detect_communities(kg, method="louvain") - connectivity = analyze_connectivity(kg, method="default") - - # Method 2: Using classes for more control - analyzer = GraphAnalyzer() - centrality_calc = CentralityCalculator() - community_detector = CommunityDetector() - connectivity_analyzer = ConnectivityAnalyzer() - - # Run comprehensive analysis - analysis = analyzer.analyze(kg) - print(f"Graph density: {analysis['density']:.3f}") - print(f"Average degree: {analysis['avg_degree']:.2f}") - - # Calculate centrality measures - degree_centrality = centrality_calc.calculate(kg, method="degree") - betweenness = centrality_calc.calculate(kg, method="betweenness") - print(f"Top entities by degree: {list(degree_centrality.items())[:3]}") - - # Detect communities - communities = community_detector.detect(kg, method="louvain") - print(f"Found {len(communities)} communities") - - # Analyze connectivity - conn = connectivity_analyzer.analyze(kg) - print(f"Connected components: {conn['num_components']}") - print(f"Largest component: {conn['largest_component_size']} nodes") - ``` - - -**Related Modules**: - -- [`kg` Module](reference/kg.md) - Knowledge graph construction and management -- [`graph_store` Module](reference/graph_store.md) - Persistent graph storage -- [`visualization` Module](reference/visualization.md) - Graph visualization - --- -### 2. Entity Extraction (NER) +## Relationship Extraction -!!! abstract "Definition" - **Named Entity Recognition (NER)** is the process of identifying and classifying named entities in text into predefined categories such as persons, organizations, locations, dates, and more. +Finding connections between entities. -**Entity Types**: +### What it does: +- Identifies how entities relate to each other +- Extracts relationship types and directions +- Provides context and confidence +- Links to source documents -| Entity Type | Description | Example | -| :--------------- | :----------------------------- | :------------------------------- | -| **Person** | Names of people | Steve Jobs, Elon Musk, Marie Curie | -| **Organization** | Companies, institutions | Apple Inc., NASA, MIT | -| **Location** | Places, geographic entities | Cupertino, Mars, Pacific Ocean | -| **Date/Time** | Temporal expressions | 1976, next Monday, Q1 2024 | -| **Money** | Monetary values | $100 million, โ‚ฌ50,000 | -| **Event** | Events and occurrences | WWDC 2024, World War II | -| **Product** | Products and services | iPhone 15, Tesla Model S | -| **Technology** | Technologies and methods | Machine Learning, Python | - -!!! tip "Custom Entities" - Semantica allows you to define custom entity types via the [`Ontology`](reference/ontology.md) module. You aren't limited to the standard set! - -**Extraction Methods:** - -Semantica supports multiple extraction methods: -- **Machine Learning Models**: spaCy, transformers (BERT, RoBERTa) -- **Rule-Based**: Pattern matching for specific formats -- **LLM-Based**: Zero-shot extraction using large language models -- **Hybrid**: Combine multiple methods for better accuracy - -**Learn by Doing:** - -- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn different NER methods and configurations - - **Topics**: Named entity recognition, entity types, confidence scores, extraction methods - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Understanding entity extraction options, choosing the right method - -- **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns and custom entity types - - **Topics**: Custom entity types, domain-specific extraction, hybrid methods - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Domain-specific extraction, custom entity definitions - r"\b(?:diabetes|hypertension|arthritis)\b", - r"\b\w+itis\b" # Inflammation conditions - ] - } - ) - - ner = NamedEntityRecognizer( - custom_detectors=[custom_detector], - include_standard_types=True # Also extract Person, Org, etc. - ) - - medical_text = """ - Patient prescribed ibuprofen 400mg twice daily for arthritis. - Previous treatment with amoxicillin was discontinued. - """ - - entities = ner.extract_entities(medical_text) - - # Group by type - from collections import defaultdict - by_type = defaultdict(list) - for e in entities: - by_type[e['type']].append(e['text']) - - for entity_type, items in by_type.items(): - print(f"{entity_type}: {', '.join(items)}") - - # Output: - # Drug: ibuprofen, amoxicillin - # Dosage: 400mg, twice daily - # Condition: arthritis - ``` -**For advanced extraction patterns including LLM-enhanced extraction and batch processing, see:** - -**Related Modules**: -- [`semantic_extract` Module](reference/semantic_extract.md) - Entity and relationship extraction -- [`ontology` Module](reference/ontology.md) - Custom entity type definitions - ---- - -### 3. Relationship Extraction - -!!! abstract "Definition" - **Relationship Extraction** is the process of identifying and extracting semantic relationships between entities in text. It connects entities to form meaningful knowledge structures. - -**Relationship Types**: - -=== "Semantic Relationships" - Relationships that define meaning and connection between entities. - - - `works_for` - Employment relationships - - `located_in` - Geographic relationships - - `founded_by` - Creation relationships - - `owns` - Ownership relationships - - `part_of` - Hierarchical relationships - - ```python - # Example: "Tim Cook works for Apple Inc." - # Extracted: (Tim Cook) --[works_for]--> (Apple Inc.) - ``` -=== "Temporal Relationships" - Relationships defined by time and sequence. - - - `happened_before` - Temporal precedence - - `happened_after` - Temporal succession - - `during` - Temporal containment - - `overlaps_with` - Temporal overlap - - ```python - # Example: "WWDC 2023 happened before WWDC 2024" - # Extracted: (WWDC 2023) --[happened_before]--> (WWDC 2024) - ``` -=== "Causal Relationships" - Cause and effect relationships. - - - `causes` - Direct causation - - `results_in` - Outcome relationships - - `prevents` - Prevention relationships - - `influences` - Indirect influence - - ```python - # Example: "High inflation causes economic instability" - # Extracted: (High inflation) --[causes]--> (Economic instability) - ``` - -**Visual Example**: - -```mermaid -graph LR - A[Apple Inc.] -->|founded_by| B[Steve Jobs] - A -->|located_in| C[Cupertino] - A -->|has_ceo| D[Tim Cook] - C -->|in_state| E[California] - B -->|co-founded| F[Apple Inc.] - - style A fill:#e3f2fd - style B fill:#fff3e0 - style C fill:#f3e5f5 - style D fill:#fff3e0 - style E fill:#f3e5f5 - style F fill:#e3f2fd +### Example Output: +```python +{ + "relationships": [ + {"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", "confidence": 0.92}, + {"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino", "confidence": 0.89} + ] +} ``` +--- -**Practical Examples**: +## Embeddings -**Learn by Doing:** +Turning text into numerical vectors for AI understanding. -- **[Relation Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)**: Learn to extract relationships between entities - - **Topics**: Relationship extraction, dependency parsing, semantic role labeling, triplet extraction - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Building rich knowledge graphs with relationships +### What are embeddings? +- **Numerical representations** of text, entities, and relationships +- **Similarity calculations** - find related concepts +- **AI-powered search** - semantic understanding +- **Clustering and grouping** - discover patterns -- **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns including event detection and coreference resolution - - **Topics**: Event detection, coreference resolution, temporal relationships, RDF triplets - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Complex extraction scenarios, temporal analysis - -**Related Modules**: -- [`semantic_extract` Module](reference/semantic_extract.md) - Relationship extraction -- [`kg` Module](reference/kg.md) - Building graphs from relationships +### Use Cases: +- **Semantic Search** - find documents by meaning, not keywords +- **Entity Resolution** - match similar entities across sources +- **Recommendations** - suggest related content +- **AI Input** - provide structured context to LLMs --- -### 4. Embeddings +## Temporal Graphs -!!! abstract "Definition" - **Embeddings** are dense vector representations of text, images, or other data that capture semantic meaning in a continuous vector space. They enable machines to understand similarity and meaning. +Knowledge graphs that understand time. -!!! note "The Bridge Between Language and Understanding" - Embeddings are the bridge between human language and machine understanding. They convert text into numerical vectors that preserve semantic relationships. +### What they track: +- **When** events happened +- **How** entities changed over time +- **Temporal relationships** - before, after, during +- **Historical context** - point-in-time snapshots -**How Embeddings Work**: - -Embeddings convert text into numerical vectors that capture semantic meaning. Similar texts have similar vectors, enabling semantic search and similarity calculations. - -**How It Works:** - -Embeddings convert text into numerical vectors that capture semantic meaning. Similar texts have similar vectors, enabling semantic search and similarity calculations. The vectors are typically 384-3072 dimensions depending on the model used. - -**Embedding Providers**: - -| Provider | Model | Dimensions | Speed | Cost | Best For | -| :--- | :--- | :--- | :--- | :--- | :--- | -| **OpenAI** | text-embedding-3-large | 3072 | Fast | Paid | Production, high accuracy | -| **OpenAI** | text-embedding-3-small | 1536 | Fast | Paid | Balanced performance | -| **Cohere** | embed-english-v3.0 | 1024 | Fast | Paid | Multilingual support | -| **HuggingFace** | sentence-transformers | 384-768 | Medium | Free | Development, open source | -| **Local** | Various | Variable | Slow | Free | Privacy, offline use | - - -**Practical Examples**: - -=== "Quick Start" - Generate embeddings in one call: - - ```python - from semantica.embeddings import EmbeddingGenerator, embed_text - - # Using EmbeddingGenerator class - texts = [ - "Machine learning is transforming industries", - "Artificial intelligence powers modern applications", - "Deep learning enables image recognition", - "Cooking recipes are delicious" # Different topic - ] - - generator = EmbeddingGenerator() - - # Generate embeddings for multiple texts - embeddings = [] - for text in texts: - emb = generator.generate_embeddings(text, data_type="text") - embeddings.append(emb) - - print(f"Generated {len(embeddings)} embeddings") - print(f"Dimensions: {embeddings[0].shape}") - - # Or use the convenience function for single text - embedding = embed_text("Hello, world!", method="sentence_transformers") - print(f"Single embedding shape: {embedding.shape}") - ``` -=== "Similarity Search" - Calculate semantic similarity between texts: - - ```python - from semantica.embeddings import ( - EmbeddingGenerator, - calculate_similarity - ) - import numpy as np - - generator = EmbeddingGenerator() - - # Generate embeddings - query = "How does photosynthesis work?" - documents = [ - "Plants convert sunlight into energy through photosynthesis", - "The process of photosynthesis occurs in chloroplasts", - "Dogs are popular household pets", - "Solar panels convert sunlight to electricity", - "Chlorophyll absorbs light for photosynthesis" - ] - - query_emb = generator.generate_embeddings(query, data_type="text") - doc_embs = [generator.generate_embeddings(d, data_type="text") for d in documents] - - print("Semantic Similarity Search:") - print(f"Query: \"{query}\"") - - similarities = [] - for i, doc_emb in enumerate(doc_embs): - sim = calculate_similarity(query_emb, doc_emb, method="cosine") - similarities.append((sim, documents[i])) - - for sim, doc in sorted(similarities, reverse=True): - relevance = "HIGH" if sim > 0.6 else "MED" if sim > 0.3 else "LOW" - print(f" [{relevance}] {sim:.3f}: {doc[:50]}...") - ``` -=== "Multiple Embedding Types" - Work with text, image, and audio embeddings: - - ```python - from semantica.embeddings import ( - TextEmbedder, - ImageEmbedder, - AudioEmbedder, - MultimodalEmbedder - ) - - text_embedder = TextEmbedder(model="all-MiniLM-L6-v2") - text_emb = text_embedder.embed("A beautiful sunset over the ocean") - print(f"Text embedding: {text_emb.shape}") - - image_embedder = ImageEmbedder(model="clip") - image_emb = image_embedder.embed("sunset_photo.jpg") - print(f"Image embedding: {image_emb.shape}") - - audio_embedder = AudioEmbedder() - audio_emb = audio_embedder.embed("ocean_waves.mp3") - print(f"Audio embedding: {audio_emb.shape}") - - multimodal = MultimodalEmbedder() - combined = multimodal.embed( - text="sunset over ocean", - image="sunset_photo.jpg", - strategy="concatenate" - ) - print(f"Combined embedding: {combined.shape}") - - from semantica.embeddings import calculate_similarity - text_query = text_embedder.embed("beach at sunset") - images = ["beach.jpg", "mountain.jpg", "city.jpg"] - - for img in images: - img_emb = image_embedder.embed(img) - sim = calculate_similarity(text_query, img_emb, method="cosine") - print(f" {img}: {sim:.3f}") - ``` -=== "Embedding Optimization" - Optimize embeddings for storage and performance: - - ```python - from semantica.embeddings import EmbeddingOptimizer - import numpy as np - - optimizer = EmbeddingOptimizer() - - # Generate some embeddings - embeddings = np.random.randn(1000, 768).astype(np.float32) - print(f"Original size: {embeddings.nbytes / 1024:.1f} KB") - - # Dimension reduction with PCA - reduced = optimizer.reduce_dimensions( - embeddings, - method="pca", - target_dims=256 - ) - print(f"After PCA (256d): {reduced.nbytes / 1024:.1f} KB") - - # Quantization for storage efficiency - quantized = optimizer.quantize( - embeddings, - bits=8 # 8-bit quantization - ) - print(f"After 8-bit quantization: {quantized.nbytes / 1024:.1f} KB") - - # Normalize for cosine similarity - normalized = optimizer.normalize(embeddings, method="l2") - print(f"Normalized: unit vectors ready for dot product") - ``` -=== "Pooling Strategies" - Aggregate embeddings using different strategies: - - ```python - from semantica.embeddings import ( - MeanPooling, - MaxPooling, - AttentionPooling, - HierarchicalPooling - ) - import numpy as np - - # Simulate token-level embeddings (e.g., from a transformer) - # Shape: (sequence_length, embedding_dim) - token_embeddings = np.random.randn(128, 768) - - # Mean pooling - average all tokens - mean_pool = MeanPooling() - mean_result = mean_pool.pool(token_embeddings) - print(f"Mean pooling: {mean_result.shape}") # (768,) - - # Max pooling - take max across tokens - max_pool = MaxPooling() - max_result = max_pool.pool(token_embeddings) - print(f"Max pooling: {max_result.shape}") # (768,) - - # Attention-based pooling - learned weights - attn_pool = AttentionPooling(hidden_dim=768) - attn_result = attn_pool.pool(token_embeddings) - print(f"Attention pooling: {attn_result.shape}") # (768,) - - # Hierarchical pooling for long documents - hier_pool = HierarchicalPooling(chunk_size=32) - document_embeddings = np.random.randn(10, 128, 768) # 10 paragraphs - hier_result = hier_pool.pool(document_embeddings) - print(f"Hierarchical pooling: {hier_result.shape}") # (768,) - ``` -=== "Context Management" - Handle long texts with context windows: - - ```python - from semantica.embeddings import ContextManager, EmbeddingGenerator - - # Long document that exceeds model context - long_document = "..." * 10000 # Very long text - - # Manage context windows - context_manager = ContextManager( - max_window_size=512, # Tokens per window - overlap=50, # Overlap between windows - preserve_sentences=True - ) - - windows = context_manager.split(long_document) - print(f"Split into {len(windows)} windows") - - # Generate embeddings for each window - generator = EmbeddingGenerator() - window_embeddings = [] - - for i, window in enumerate(windows): - emb = generator.generate_embeddings(window.text, data_type="text") - window_embeddings.append({ - 'embedding': emb, - 'start_char': window.start_char, - 'end_char': window.end_char - }) - - # Merge windows for document-level embedding - from semantica.embeddings import pool_embeddings - import numpy as np - - all_embs = np.array([w['embedding'] for w in window_embeddings]) - doc_embedding = pool_embeddings(all_embs, method="mean") - print(f"Document embedding shape: {doc_embedding.shape}") - ``` - - -**Learn by Doing:** - -- **[Embeddings Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb)**: Learn to generate and use embeddings - - **Topics**: Embedding generation, similarity search, vector operations, pooling strategies - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Understanding embeddings, semantic search setup - -**Related Modules**: -- [`embeddings` Module](reference/embeddings.md) - Embedding generation -- [`vector_store` Module](reference/vector_store.md) - Vector storage and search +### Example Uses: +- **Company History** - track mergers, leadership changes +- **Person Careers** - job changes, relocations +- **Policy Evolution** - law changes over time +- **Research Progress** - scientific discoveries timeline --- -### 5. Temporal Graphs +## GraphRAG -!!! abstract "Definition" - **Temporal Graphs** are knowledge graphs that track changes over time, allowing queries about the state of the graph at specific time points. They enable time-aware reasoning and analysis. +Enhanced AI retrieval using knowledge graphs. -**Key Features**: +### How it works: +1. **Query** user question +2. **Retrieve** relevant graph context +3. **Enhance** with relationships and entities +4. **Generate** AI response with sources -- **Time-stamped Entities**: Entities have creation and modification timestamps -- **Time-stamped Relationships**: Relationships have validity periods -- **Historical Queries**: Query the graph state at any point in time -- **Change Tracking**: Track how entities and relationships evolve +### Benefits: +- **More accurate** answers +- **Source attribution** - trace answers back +- **Context awareness** - understand relationships +- **Reduced hallucination** - grounded in facts -**Visual Timeline**: +--- -```mermaid -timeline - title Temporal Graph Evolution - 2020 : Entity A created - : Relationship A->B established - 2021 : Entity B updated - : Relationship B->C created - 2022 : Entity A deleted - : New Relationship D->C - 2023 : Entity C properties updated - : Relationship A->B expired +## Ontology + +Defining the structure and rules of your knowledge. + +### What it provides: +- **Schema definition** - what types exist +- **Relationship rules** - valid connections +- **Property constraints** - required fields +- **Inheritance hierarchies** - parent-child relationships + +### Example: +```python +# Define ontology structure +ontology = { + "classes": ["Person", "Organization", "Location"], + "properties": ["name", "date", "confidence"], + "relationships": ["works_for", "located_in", "born_in"], + "rules": { + "Person": ["must_have_name", "can_have_birth_date"], + "Organization": ["must_have_name", "can_have_founding_date"] + } +} ``` - -**Practical Examples**: - -=== "Building Temporal Graphs" - Create a knowledge graph with time-aware edges: - - ```python - from semantica.kg import GraphBuilder - from datetime import datetime, timedelta - - # Build a temporal knowledge graph - builder = GraphBuilder( - enable_temporal=True, - temporal_granularity="day", # day, hour, minute, second - track_history=True, - version_snapshots=True - ) - - # Add time-stamped data - sources = [{ - "entities": [ - { - "id": "e1", - "text": "Satya Nadella", - "type": "Person", - "valid_from": "2014-02-04" # Became CEO - }, - { - "id": "e2", - "text": "Microsoft", - "type": "Organization" - }, - { - "id": "e3", - "text": "Steve Ballmer", - "type": "Person", - "valid_from": "2000-01-13", - "valid_until": "2014-02-04" # Was CEO until - } - ], - "relationships": [ - { - "source": "e1", - "target": "e2", - "type": "CEO_OF", - "valid_from": "2014-02-04", - "valid_until": None # Still active - }, - { - "source": "e3", - "target": "e2", - "type": "CEO_OF", - "valid_from": "2000-01-13", - "valid_until": "2014-02-04" - } - ] - }] - - kg = builder.build(sources) - print(f"Built temporal graph with {kg['metadata']['num_entities']} entities") - ``` -=== "Temporal Queries" - Query the graph at specific points in time: - - ```python - from semantica.kg import TemporalGraphQuery - from datetime import datetime - - # Initialize temporal query engine - temporal_query = TemporalGraphQuery(kg) - - # Query 1: Who was CEO in 2010? - state_2010 = temporal_query.query_at_time( - datetime(2010, 6, 1), - query_type="relationships", - relationship_type="CEO_OF" - ) - print(f"CEO in 2010: {state_2010['results']}") - - state_2020 = temporal_query.query_at_time( - datetime(2020, 6, 1), - query_type="relationships", - relationship_type="CEO_OF" - ) - print(f"CEO in 2020: {state_2020['results']}") - - ceo_history = temporal_query.query_time_range( - start_time=datetime(2000, 1, 1), - end_time=datetime(2024, 1, 1), - relationship_type="CEO_OF" - ) - - print("CEO History:") - for rel in ceo_history['results']: - print(f" {rel['source']} -> {rel['valid_from']} to {rel.get('valid_until', 'present')}") - ``` -=== "Pattern Detection" - Detect temporal patterns in your graph: - - ```python - from semantica.kg import TemporalPatternDetector - - # Initialize pattern detector - detector = TemporalPatternDetector(kg) - - # Detect sequences: A happened, then B, then C - sequences = detector.detect_sequences( - event_types=["HIRED", "PROMOTED", "RESIGNED"], - max_gap_days=365 # Events must be within 1 year - ) - - print("Detected Sequences:") - for seq in sequences: - print(f" {' -> '.join(seq['events'])}") - print(f" Entity: {seq['entity']}") - print(f" Duration: {seq['duration_days']} days") - - cycles = detector.detect_cycles( - min_occurrences=2, - relationship_types=["QUARTERLY_REPORT"] - ) - - print("Detected Cycles:") - for cycle in cycles: - print(f" Pattern: {cycle['pattern']}") - print(f" Period: {cycle['period_days']} days") - print(f" Occurrences: {cycle['count']}") - - trends = detector.detect_trends( - metric="employee_count", - entity_type="Organization", - window_days=90 - ) - - for trend in trends: - direction = "UP" if trend['direction'] == 'increasing' else "DOWN" - print(f"[{direction}] {trend['entity']}: {trend['change_percent']:.1f}% over {trend['period']}") - ``` - -=== "Version Management" - Track and manage graph versions over time: - - ```python - from semantica.kg import TemporalVersionManager - from datetime import datetime - - # Initialize version manager - version_mgr = TemporalVersionManager(kg) - - # Create a snapshot of current state - snapshot_id = version_mgr.create_snapshot( - description="Q4 2024 knowledge graph", - timestamp=datetime.now() - ) - print(f"Created snapshot: {snapshot_id}") - - diff = version_mgr.compare_versions( - version_a=snapshot_id, - version_b="current" - ) - - print("Changes since snapshot:") - print(f" Added entities: {len(diff['added_entities'])}") - print(f" Removed entities: {len(diff['removed_entities'])}") - print(f" Modified entities: {len(diff['modified_entities'])}") - print(f" Added relationships: {len(diff['added_relationships'])}") - print(f" Removed relationships: {len(diff['removed_relationships'])}") - - evolution = version_mgr.analyze_evolution( - start_date=datetime(2024, 1, 1), - end_date=datetime(2024, 12, 31) - ) - - print("Graph Evolution (2024):") - print(f" Total snapshots: {evolution['snapshot_count']}") - print(f" Net entity growth: {evolution['entity_growth']}") - print(f" Net relationship growth: {evolution['relationship_growth']}") - ``` - -=== "Graph Evolution Analysis" - Analyze how your knowledge graph changes over time: - - ```python - from semantica.kg import GraphAnalyzer - - analyzer = GraphAnalyzer() - - # Analyze temporal evolution - evolution = analyzer.analyze_temporal_evolution( - kg, - time_periods=[ - ("2022-01-01", "2022-06-30"), - ("2022-07-01", "2022-12-31"), - ("2023-01-01", "2023-06-30"), - ("2023-07-01", "2023-12-31") - ] - ) - - print("Graph Evolution Analysis:") - for period in evolution['periods']: - print(f"Period: {period['start']} to {period['end']}") - print(f" Entities: {period['entity_count']} (+{period['entity_growth']})") - print(f" Relationships: {period['relationship_count']} (+{period['rel_growth']})") - print(f" Density: {period['density']:.4f}") - print(f" New entity types: {period['new_types']}") - print() - - stability = analyzer.calculate_stability(kg, window_days=30) - print(f"Graph Stability Score: {stability['score']:.2f}") - print(f" Most stable entities: {stability['most_stable'][:3]}") - print(f" Most volatile entities: {stability['most_volatile'][:3]}") - ``` - - -**Related Modules**: -- [`kg` Module](reference/kg.md) - Temporal graph support -- [`visualization` Module](reference/visualization.md) - Temporal visualization - --- -### 6. GraphRAG +## Reasoning & Inference -!!! abstract "Definition" - **GraphRAG (Graph-Augmented Retrieval Augmented Generation)** is an advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs. +Making logical deductions from your knowledge. -**How GraphRAG Works**: +### What it can do: +- **Infer missing facts** - derive new knowledge +- **Detect inconsistencies** - find contradictions +- **Apply rules** - automate decision making +- **Explain reasoning** - show how conclusions were reached -```mermaid -flowchart TD - subgraph Query["Query Processing"] - Q[User Query] --> VS[Vector Search] - Q --> KE[Keyword Extraction] - end - - subgraph Retrieval["Hybrid Retrieval"] - VS --> Docs[Relevant Documents] - KE --> Nodes[Start Nodes] - Nodes --> Trav[Graph Traversal] - Trav --> Context[Graph Context] - end - - subgraph Synthesis["Answer Generation"] - Docs --> Prompt[Enhanced Prompt] - Context --> Prompt - Prompt --> LLM[LLM Generation] - LLM --> A[Accurate Answer] - end - - style Q fill:#e1f5fe - style LLM fill:#e8f5e9 - style A fill:#fff9c4 +### Example: +``` +Known: Steve Jobs founded Apple Inc. +Known: Apple Inc. is headquartered in Cupertino +Inferred: Steve Jobs has connection to Cupertino ``` -**Advantages over Traditional RAG**: - -| Feature | Traditional RAG | GraphRAG | -| :--- | :--- | :--- | -| **Query Understanding** | Keyword matching | Semantic + structural | -| **Context Retrieval** | Document chunks | Documents + relationships | -| **Answer Accuracy** | Good | Better (grounded in graph) | -| **Hallucination Risk** | Medium | Lower | -| **Complex Queries** | Limited | Excellent | -| **Relationship Awareness** | No | Yes | - - -**Practical Examples**: - -**Learn by Doing:** - -- **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Build a production-ready GraphRAG system - - **Topics**: GraphRAG, hybrid retrieval, graph traversal, LLM integration - - **Difficulty**: Advanced - - **Time**: 1-2 hours - - **Use Cases**: Production GraphRAG systems, enhanced RAG applications - -- **[RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)**: Side-by-side comparison - - **Topics**: RAG comparison, reasoning gap, inference engines - - **Difficulty**: Intermediate - - **Time**: 45-60 minutes - - **Use Cases**: Understanding GraphRAG advantages, choosing the right approach - - print(f"Vector results: {len(vector_results)}") - print(f"Graph context: {len(graph_context)} relevant relationships") - ``` - -=== "Context Retrieval" - Use the context module for agent memory and retrieval: - - ```python - from semantica.context import ( - ContextRetriever, - AgentMemory, - ContextGraphBuilder, - build_context - ) - from semantica.vector_store import VectorStore - - # Initialize vector store - vector_store = VectorStore(backend="faiss", dimension=768) - - # Build context graph - context_result = build_context( - entities=kg["entities"], - relationships=kg["relationships"], - vector_store=vector_store, - knowledge_graph=kg, - store_initial_memories=True - ) - - print(f"Built context with {context_result['statistics']['graph'].get('node_count', 0)} nodes") - - # Initialize agent memory - memory = AgentMemory( - vector_store=vector_store, - knowledge_graph=kg - ) - - # Store facts in memory - memory_id = memory.store( - "Tim Cook is the CEO of Apple Inc.", - metadata={"type": "fact", "source": "document"} - ) - print(f"Stored memory: {memory_id}") - - # Initialize context retriever - retriever = ContextRetriever( - vector_store=vector_store, - knowledge_graph=kg - ) - - # Retrieve relevant context - results = retriever.retrieve( - query="Who leads Apple?", - max_results=10 - ) - - print("Retrieved context:") - for r in results: - print(f" {r.content} (score: {r.score:.2f})") - ``` - -=== "Hybrid Search" - Combine vector and metadata search: - - ```python - from semantica.vector_store import ( - VectorStore, - HybridSearch, - MetadataFilter, - MetadataStore - ) - from semantica.embeddings import embed_text - - # Initialize stores - vector_store = VectorStore(backend="faiss", dimension=768) - metadata_store = MetadataStore() - hybrid_search = HybridSearch() - - # Generate query embedding - query_embedding = embed_text("Apple CEO responsibilities", method="sentence_transformers") - - # Create metadata filter - metadata_filter = MetadataFilter() - metadata_filter.eq("entity_type", "Person") - metadata_filter.eq("relationship", "CEO_OF") - - # Get stored data - all_vectors = vector_store.get_all_vectors() - all_metadata = metadata_store.get_all() - all_ids = vector_store.get_all_ids() - - # Perform hybrid search - results = hybrid_search.search( - query_vector=query_embedding, - vectors=all_vectors, - metadata=all_metadata, - vector_ids=all_ids, - filter=metadata_filter, - top_k=10 - ) - - print("Hybrid search results:") - for result in results: - print(f" ID: {result['id']}, Score: {result['score']:.3f}") - ``` - -=== "Inference-Enhanced Retrieval" - Apply logical rules to expand retrieval results: - - ```python - from semantica.reasoning import Reasoner, Rule - from semantica.semantic_extract import ExplanationGenerator - - # Initialize Reasoner - reasoner = Reasoner() - - # Add reasoning rules - reasoner.add_rule(Rule( - name="employment_transitivity", - conditions=["?person CEO_OF ?company"], - conclusions=["?person WORKS_FOR ?company"], - priority=1 - )) - - reasoner.add_rule(Rule( - name="part_of_transitivity", - conditions=["?a PART_OF ?b", "?b PART_OF ?c"], - conclusions=["?a PART_OF ?c"], - priority=2 - )) - - # Define facts from knowledge graph - facts = [ - {"subject": "Tim Cook", "predicate": "CEO_OF", "object": "Apple"}, - {"subject": "iPhone", "predicate": "PART_OF", "object": "Apple"}, - {"subject": "A15 Chip", "predicate": "PART_OF", "object": "iPhone"} - ] - - # Run inference to discover hidden relationships - results = reasoner.infer_facts(facts) - - print(f"Original facts: {len(facts)}") - print(f"Inferred {len(results)} new facts") - - for fact in results: - print(f" New fact: {fact}") - - # Generate explanations for why these facts were inferred - explainer = ExplanationGenerator() - # (Assuming we have the full inference results with paths) - # The Reasoner.forward_chain() provides detailed paths if needed - ``` - -=== "LLM-Enhanced Retrieval" - Integrate with LLM providers for answer generation: - - ```python - from semantica.semantic_extract import NERExtractor - from semantica.context import ContextRetriever - from semantica.vector_store import VectorStore - from semantica.llms import LLMProvider - - # Initialize components - vector_store = VectorStore(backend="faiss", dimension=768) - llm_provider = LLMProvider(provider="openai", model="gpt-4o") - - # Create LLM provider - provider = create_provider( - provider_type="openai", - model="gpt-4o" - ) - - # Initialize context retriever - retriever = ContextRetriever( - vector_store=vector_store, - knowledge_graph=kg - ) - - # Define question - question = "What products does Apple manufacture?" - - # Retrieve relevant context - context = retriever.retrieve(query=question, max_results=5) - - # Format context for prompt - context_text = "\n".join([ - f"- {c.content}" for c in context - ]) - - # Build prompt - prompt = f"""Based on this context, answer the question. - -Context: -{context_text} - -Question: {question} - -Answer:""" - - # Generate response - response = provider.generate(prompt) - print(f"Answer: {response}") - ``` - - -**Related Modules**: -- [`kg` Module](reference/kg.md) - Knowledge graph construction -- [`vector_store` Module](reference/vector_store.md) - Vector search -- [`reasoning` Module](reference/reasoning.md) - Graph reasoning - --- -### 7. Ontology +## Deduplication & Entity Resolution -!!! abstract "Definition" - An **Ontology** is a formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language). It defines the schema and structure of your knowledge domain. +Finding and merging duplicate entities. -**Key Components**: +### What it does: +- **Detects duplicates** - same entity, different names +- **Merges information** - combine attributes +- **Resolves conflicts** - handle contradictory data +- **Maintains provenance** - track original sources -- **Classes**: Categories of entities (e.g., `Person`, `Company`, `Location`) -- **Properties**: Relationships and attributes (e.g., `worksFor`, `hasName`) -- **Individuals**: Specific instances (e.g., `John Doe`, `Apple Inc.`) -- **Axioms**: Rules and constraints (e.g., "A Person can only workFor one Company") - -**Ontology Structure**: - -```mermaid -classDiagram - class Person { - +String name - +Date birthDate - +worksFor Company - } - class Company { - +String name - +Date founded - +locatedIn Location - } - class Location { - +String city - +String country - } - - Person --> Company : worksFor - Company --> Location : locatedIn +### Example: +```python +# These refer to the same entity: +"Apple Inc." รขโ€ โ€™ "Apple" รขโ€ โ€™ "Apple Computer Inc." +# Merge into single entity with all attributes ``` +--- -**Practical Examples**: +## Data Normalization -=== "Automatic Ontology Generation" - Generate an ontology from your knowledge graph: - - ```python - from semantica.ontology import OntologyGenerator, generate_ontology - - # Your knowledge graph data - kg_data = { - "entities": [ - {"id": "e1", "text": "Apple Inc.", "type": "Company"}, - {"id": "e2", "text": "Tim Cook", "type": "Person"}, - {"id": "e3", "text": "iPhone", "type": "Product"}, - {"id": "e4", "text": "Cupertino", "type": "Location"} - ], - "relationships": [ - {"source": "e2", "target": "e1", "type": "CEO_OF"}, - {"source": "e1", "target": "e3", "type": "MANUFACTURES"}, - {"source": "e1", "target": "e4", "type": "HEADQUARTERED_IN"} - ] - } - - # Method 1: Using convenience function - ontology = generate_ontology(kg_data, method="default") - print(f"Quick generation: {len(ontology.get('classes', []))} classes") - - # Method 2: Using OntologyGenerator class for more control - generator = OntologyGenerator( - base_uri="https://example.org/ontology/", - version="1.0.0" - ) - - ontology = generator.generate_ontology(kg_data) - - print(f"Generated Ontology:") - print(f" Classes: {len(ontology['classes'])}") - print(f" Properties: {len(ontology['properties'])}") - print(f" Axioms: {len(ontology.get('axioms', []))}") - - # Display class details - print("Classes:") - for cls in ontology['classes']: - print(f" {cls['name']}") - print(f" URI: {cls['uri']}") - if cls.get('parent'): - print(f" Parent: {cls['parent']}") - ``` +Cleaning and standardizing your data. -=== "Class and Property Inference" - Automatically infer classes and properties from data: - - ```python - from semantica.ontology import ( - ClassInferrer, - PropertyGenerator, - infer_classes, - infer_properties - ) - - # Sample entities - entities = [ - {"text": "Apple", "type": "Company"}, - {"text": "Microsoft", "type": "Company"}, - {"text": "Google", "type": "Company"}, - {"text": "Tim Cook", "type": "Person"}, - {"text": "Satya Nadella", "type": "Person"}, - {"text": "iPhone", "type": "Product"}, - {"text": "Windows", "type": "Product"}, - ] - - # Method 1: Using convenience function - classes = infer_classes(entities, method="default") - - # Method 2: Using ClassInferrer for more control - class_inferrer = ClassInferrer( - min_occurrence=2, # Minimum entities to create a class - merge_similar=True, - similarity_threshold=0.8 - ) - - classes = class_inferrer.infer_classes(entities) - - print("Inferred Classes:") - for cls in classes: - print(f" {cls['name']}: {cls['instance_count']} instances") - - # Infer properties from relationships - relationships = [ - {"source_type": "Person", "target_type": "Company", "type": "CEO_OF"}, - {"source_type": "Company", "target_type": "Product", "type": "MANUFACTURES"}, - {"source_type": "Person", "target_type": "Company", "type": "WORKS_FOR"}, - ] - - # Using PropertyGenerator - prop_generator = PropertyGenerator() - properties = prop_generator.infer_properties(entities, relationships, classes) - - print("Inferred Properties:") - for prop in properties: - print(f" {prop['name']}") - print(f" Domain: {prop['domain']} -> Range: {prop['range']}") - print(f" Type: {prop['property_type']}") - ``` +### What it fixes: +- **Format inconsistencies** - dates, names, numbers +- **Canonical forms** - standard representations +- **Data quality** - remove errors and noise +- **Standardization** - consistent naming conventions -=== "OWL/RDF Generation" - Generate OWL ontologies in various formats: - - ```python - from semantica.ontology import OWLGenerator, generate_owl - - # Initialize OWL generator - owl_gen = OWLGenerator( - base_uri="https://example.org/onto/", - output_format="turtle" # turtle, rdf-xml, json-ld, n3 - ) - - # Define ontology classes - owl_gen.add_class("Person", parent="Thing") - owl_gen.add_class("Company", parent="Thing") - owl_gen.add_class("CEO", parent="Person") - - # Add object properties - owl_gen.add_object_property( - name="worksFor", - domain="Person", - range="Company" - ) - - owl_gen.add_object_property( - name="isCEOOf", - domain="CEO", - range="Company", - functional=True # A CEO can only be CEO of one company - ) - - # Add data properties - owl_gen.add_data_property( - name="hasName", - domain="Person", - range="xsd:string" - ) - - owl_gen.add_data_property( - name="foundedYear", - domain="Company", - range="xsd:integer" - ) - - # Add axioms/constraints - owl_gen.add_axiom( - class_name="CEO", - axiom="SubClassOf: worksFor exactly 1 Company" - ) - - # Generate Turtle output - turtle_output = owl_gen.serialize() - print(turtle_output) - - # Example output: - # @prefix : . - # @prefix owl: . - # :Person a owl:Class . - # :Company a owl:Class . - # :CEO a owl:Class ; rdfs:subClassOf :Person . - - # Save to file - owl_gen.save("company_ontology.ttl") - print("Ontology saved to company_ontology.ttl") - ``` - -=== "Competency Questions" - Define and evaluate competency questions: - - ```python - from semantica.ontology import ( - CompetencyQuestionsManager, - CompetencyQuestion, - OntologyEvaluator - ) - - # Initialize competency questions manager - cq_manager = CompetencyQuestionsManager() - - # Add questions your ontology should be able to answer - cq_manager.add_question(CompetencyQuestion( - question="Who is the CEO of a given company?", - category="organizational", - priority=1, # 1=high, 2=medium, 3=low - expected_elements=["Person", "Company", "isCEOOf"] - )) - - cq_manager.add_question(CompetencyQuestion( - question="What products does a company manufacture?", - category="products", - priority=1, - expected_elements=["Company", "Product", "manufactures"] - )) - - cq_manager.add_question(CompetencyQuestion( - question="Where is a company headquartered?", - category="location", - priority=2, - expected_elements=["Company", "Location", "headquarteredIn"] - )) - - # Get all questions - questions = cq_manager.get_questions() - print(f"Defined {len(questions)} competency questions") - - # Evaluate ontology against competency questions - evaluator = OntologyEvaluator() - evaluation = evaluator.evaluate( - ontology="company_ontology.ttl", - competency_questions=questions - ) - - print(f"Ontology Evaluation:") - print(f" Coverage Score: {evaluation.coverage_score:.1%}") - print(f" Completeness Score: {evaluation.completeness_score:.1%}") - - print("Question Coverage:") - for result in evaluation.question_results: - status = "YES" if result['answerable'] else "NO" - print(f" [{status}] {result['question']}") - if not result['answerable'] and result.get('missing_elements'): - print(f" Missing: {', '.join(result['missing_elements'])}") - ``` - -=== "Domain Ontologies" - Use pre-built domain ontologies: - - ```python - from semantica.ontology import DomainOntologies, OWLGenerator - - # Access pre-built domain ontologies - domains = DomainOntologies() - - # List available domains - print("Available Domain Ontologies:") - for domain in domains.list_domains(): - print(f" - {domain['name']}: {domain['description']}") - - # Load healthcare ontology - healthcare_onto = domains.load("healthcare") - - print(f"Healthcare Ontology:") - print(f" Classes: {len(healthcare_onto.get('classes', []))}") - print(f" Properties: {len(healthcare_onto.get('properties', []))}") - - # Extend the ontology - owl_gen = OWLGenerator(base_uri="https://example.org/healthcare/") - owl_gen.add_class("MedicalDevice", parent="Equipment") - - # Load and merge multiple ontologies - finance_onto = domains.load("finance") - merged = domains.merge([healthcare_onto, finance_onto]) - - print(f"Merged ontology: {len(merged.get('classes', []))} classes") - - # Find alignments between ontologies - alignments = domains.find_alignments( - healthcare_onto, - finance_onto, - similarity_threshold=0.7 - ) - - print(f"Found {len(alignments)} concept alignments:") - for align in alignments[:5]: - print(f" {align['source']} ~ {align['target']} ({align['similarity']:.2f})") - ``` - - -**Related Modules**: -- [`ontology` Module](reference/ontology.md) - Ontology generation and management -- [`kg` Module](reference/kg.md) - Knowledge graph construction +### Examples: +- **Dates**: "Jan 1, 2020" รขโ€ โ€™ "2020-01-01" +- **Names**: "Dr. Smith PhD" รขโ€ โ€™ "John Smith" +- **Companies**: "Apple" รขโ€ โ€™ "Apple Inc." +- **Locations**: "NYC" รขโ€ โ€™ "New York City" --- -### 8. Reasoning & Inference +## Conflict Detection -!!! abstract "Definition" - **Reasoning** is the process of deriving new knowledge (conclusions) from existing facts (premises) using logical rules. It allows Semantica to "think" beyond what is explicitly stated in the data. +Finding and resolving contradictory information. -**Reasoning Types**: +### What it identifies: +- **Factual conflicts** - different values for same fact +- **Temporal conflicts** - impossible timelines +- **Logical conflicts** - contradictory relationships +- **Source reliability** - trustworthiness assessment -- **Forward Chaining**: Starting from known facts and applying rules to see what new facts can be derived. (Data-driven) -- **Backward Chaining**: Starting from a hypothesis (goal) and working backward to see if the available facts support it. (Goal-driven) -- **Abductive Reasoning**: Finding the most likely explanation for a set of observations. -- **Deductive Reasoning**: Applying general rules to specific cases to reach certain conclusions. - -**Practical Examples**: - -=== "Reasoner Facade" - The `Reasoner` provides a unified interface for all reasoning tasks: - - ```python - from semantica.reasoning import Reasoner, Rule - - # Initialize the reasoner - reasoner = Reasoner() - - # Define a logical rule: If A is a parent of B, and B is a parent of C, then A is a grandparent of C - reasoner.add_rule(Rule( - name="grandparent_rule", - conditions=["?a PARENT_OF ?b", "?b PARENT_OF ?c"], - conclusions=["?a GRANDPARENT_OF ?c"] - )) - - # Add initial facts - facts = [ - "John PARENT_OF Mary", - "Mary PARENT_OF Alice" - ] - - # Infer new facts - new_facts = reasoner.infer_facts(facts) - - print(f"Inferred: {new_facts}") - # Output: ['John GRANDPARENT_OF Alice'] - ``` - -=== "SPARQL Reasoning" - Expand SPARQL queries with inferred patterns: - - ```python - from semantica.reasoning import SPARQLReasoner - - # Initialize SPARQL reasoner with a triplet store - sparql_reasoner = SPARQLReasoner(triplet_store=my_store) - - # Define a rule - sparql_reasoner.reasoner.add_rule("(?x works_at ?y) -> (?x employee_of ?y)") - - # Original query - query = "SELECT ?x WHERE { ?x employee_of }" - - # Expand query to include 'works_at' patterns - expanded_query = sparql_reasoner.expand_query(query) - - print(f"Expanded Query:\n{expanded_query}") - ``` - -=== "Abductive Reasoning" - Generate hypotheses for observed facts: - - ```python - from semantica.reasoning import AbductiveReasoner - - reasoner = AbductiveReasoner() - - # Rule: Fire causes smoke - reasoner.reasoner.add_rule("Fire(?x) -> Smoke(?x)") - - # Observation: There is smoke in the kitchen - observations = ["Smoke(Kitchen)"] - - # What could have caused this? - hypotheses = reasoner.generate_hypotheses(observations) - - for h in hypotheses: - print(f"Possible cause: {h.conclusion} (Likelihood: {h.score})") - ``` - -**Related Modules**: -- [`reasoning` Module](reference/reasoning.md) - Core reasoning engines -- [`ontology` Module](reference/ontology.md) - Domain rules and axioms +### Resolution Strategies: +- **Most recent** - prefer newer information +- **Most reliable** - prefer trusted sources +- **Majority vote** - go with consensus +- **Manual review** - flag for human review --- -### 9. Deduplication & Entity Resolution +## Getting Started -!!! abstract "Definition" - **Deduplication** and **Entity Resolution** are processes that identify and merge duplicate entities in a knowledge graph, ensuring that the same real-world entity is represented by a single node. +Ready to build your first knowledge graph? -**Why It Matters**: +### Quick Start (5 minutes) +```python +from semantica.semantic_extract import NERExtractor +from semantica.kg import GraphBuilder -- Multiple sources may refer to the same entity differently -- "Apple Inc." vs "Apple" vs "Apple Computer" โ†’ Same entity -- Prevents graph fragmentation and improves query accuracy +# Extract entities +ner = NERExtractor() +entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") -**Resolution Process**: - -Deduplication works by calculating similarity between entities. If similarity exceeds a threshold, entities are merged; otherwise, they remain separate. - - -**Practical Examples**: - -=== "Entity Resolution" - Identify and resolve duplicate entities: - - ```python - from semantica.kg import EntityResolver, Deduplicator, resolve_entities - - # Sample entities with potential duplicates - entities = [ - {"id": "e1", "text": "Apple Inc.", "type": "Organization"}, - {"id": "e2", "text": "Apple", "type": "Organization"}, - {"id": "e3", "text": "Apple Computer, Inc.", "type": "Organization"}, - {"id": "e4", "text": "Microsoft", "type": "Organization"}, - {"id": "e5", "text": "MSFT", "type": "Organization"}, - {"id": "e6", "text": "Microsoft Corporation", "type": "Organization"}, - ] - - # Method 1: Using convenience function - resolved = resolve_entities(entities, method="fuzzy") - - # Method 2: Using EntityResolver class for more control - resolver = EntityResolver( - strategy="fuzzy", # "exact", "fuzzy", or "semantic" - similarity_threshold=0.85, - use_embeddings=True - ) - - # Find duplicate groups - duplicate_groups = resolver.find_duplicates(entities) - - print("Duplicate Groups Found:") - for i, group in enumerate(duplicate_groups, 1): - print(f"Group {i}:") - for entity in group['entities']: - print(f" - {entity['text']} (id: {entity['id']})") - print(f" Confidence: {group['confidence']:.2f}") - ``` - -=== "Semantic Matching" - Use embeddings for semantic similarity matching: - - ```python - from semantica.kg import EntityResolver - from semantica.embeddings import EmbeddingGenerator - - # Initialize with semantic matching - generator = EmbeddingGenerator() - - resolver = EntityResolver( - strategy="semantic", - embedding_generator=generator, - similarity_threshold=0.8 - ) - - # Entities with semantic variations - entities = [ - {"id": "e1", "text": "Chief Executive Officer", "type": "Title"}, - {"id": "e2", "text": "CEO", "type": "Title"}, - {"id": "e3", "text": "Managing Director", "type": "Title"}, - {"id": "e4", "text": "Head of the Company", "type": "Title"}, - {"id": "e5", "text": "Software Developer", "type": "Title"}, - {"id": "e6", "text": "Programmer", "type": "Title"}, - ] - - # Find semantically similar entities - semantic_matches = resolver.find_semantic_matches(entities) - - print("Semantic Matches:") - for match in semantic_matches: - print(f" {match['entity1']['text']} ~ {match['entity2']['text']}") - print(f" Similarity: {match['similarity']:.3f}") - - clusters = resolver.cluster_entities( - entities, - method="semantic", - min_cluster_size=2 - ) - - print(f"Found {len(clusters)} semantic clusters") - ``` - -=== "Graph Deduplication" - Deduplicate an entire knowledge graph: - - ```python - from semantica.kg import Deduplicator, deduplicate_graph - - # Method 1: Using convenience function - dedup_result = deduplicate_graph(kg, method="default") - - # Method 2: Using Deduplicator class for more control - deduplicator = Deduplicator( - entity_similarity_threshold=0.85, - relationship_merge_strategy="union", # "union", "intersection", "first" - preserve_provenance=True - ) - - # Deduplicate the knowledge graph - dedup_result = deduplicator.deduplicate(kg) - - print("Deduplication Results:") - print(f" Original entities: {dedup_result['original_entity_count']}") - print(f" After dedup: {dedup_result['final_entity_count']}") - print(f" Duplicates merged: {dedup_result['merged_count']}") - print(f" Reduction: {dedup_result['reduction_percent']:.1f}%") - - # Get deduplicated graph - clean_kg = dedup_result['graph'] - - print("Merge Details:") - for merge in dedup_result['merges'][:5]: - print(f" Merged {len(merge['source_ids'])} entities -> {merge['canonical_id']}") - print(f" Canonical: {merge['canonical_text']}") - print(f" Aliases: {', '.join(merge['aliases'])}") - ``` - -=== "Custom Matching Rules" - Define custom matching rules for your domain: - - ```python - from semantica.kg import EntityResolver - - # Define custom matching rules - custom_rules = { - "Organization": { - "normalize": [ - ("Inc.", ""), - ("Corp.", ""), - ("Corporation", ""), - ("LLC", ""), - ("Ltd.", ""), - ], - "abbreviations": { - "IBM": "International Business Machines", - "GE": "General Electric", - "HP": "Hewlett-Packard", - }, - "threshold": 0.8 - }, - "Person": { - "match_fields": ["name", "email"], - "normalize_case": True, - "threshold": 0.9 - }, - "Location": { - "normalize": [ - ("St.", "Street"), - ("Ave.", "Avenue"), - ("Blvd.", "Boulevard"), - ], - "threshold": 0.85 - } - } - - resolver = EntityResolver( - strategy="rule_based", - custom_rules=custom_rules - ) - - # Apply custom rules - entities = [ - {"id": "e1", "text": "Apple Inc.", "type": "Organization"}, - {"id": "e2", "text": "Apple Corporation", "type": "Organization"}, - {"id": "e3", "text": "International Business Machines", "type": "Organization"}, - {"id": "e4", "text": "IBM", "type": "Organization"}, - ] - - matches = resolver.find_duplicates(entities) - - print("Rule-Based Matches:") - for match in matches: - print(f" {match['entities'][0]['text']} = {match['entities'][1]['text']}") - print(f" Rule: {match['matched_rule']}") - ``` - -=== "Merge Strategies" - Control how entities are merged: - - ```python - from semantica.kg import EntityResolver - - resolver = EntityResolver(strategy="fuzzy", similarity_threshold=0.85) - - # Define merge strategy - merged_entity = resolver.merge_entities( - entities=[ - {"id": "e1", "text": "Apple", "properties": {"founded": 1976}}, - {"id": "e2", "text": "Apple Inc.", "properties": {"ceo": "Tim Cook"}}, - {"id": "e3", "text": "Apple Computer", "properties": {"founded": 1976, "industry": "Tech"}} - ], - strategy={ - "id": "keep_first", # Use first entity's ID - "text": "most_complete", # Use longest/most complete text - "properties": "merge_all", # Combine all properties - "conflicts": "keep_latest" # For conflicting values, keep latest - } - ) - - print("Merged Entity:") - print(f" ID: {merged_entity['id']}") - print(f" Text: {merged_entity['text']}") - print(f" Properties: {merged_entity['properties']}") - print(f" Source IDs: {merged_entity['source_ids']}") - ``` - - -**Related Modules**: -- [`deduplication` Module](reference/deduplication.md) - Deduplication and merging -- [`embeddings` Module](reference/embeddings.md) - Similarity calculation - ---- - -### 10. Data Normalization - -!!! abstract "Definition" - **Data Normalization** is the process of cleaning and standardizing data into a consistent format, ensuring uniformity across your knowledge graph. - -**Normalization Pipeline**: - -```mermaid -flowchart LR - Raw[Raw Text] --> Clean[Text Cleaning] - Clean --> Entity[Entity
Normalization] - Entity --> Date[Date
Normalization] - Date --> Number[Number
Normalization] - Number --> Normalized[Normalized
Data] - - style Raw fill:#ffcdd2 - style Normalized fill:#c8e6c9 +# Build graph +kg = GraphBuilder().build({"entities": entities, "relationships": []}) ``` +### Learn More +- **Getting Started Guide** - [Getting Started](getting-started.md) +- **Cookbook Examples** - [Cookbook](cookbook.md) +- **Module Documentation** - [Reference](reference/) +- **Community Support** - [Community](community.md) -**Practical Examples**: - -=== "Text Normalization" - Clean and normalize text data: - - ```python - from semantica.normalize import TextNormalizer - - normalizer = TextNormalizer( - lowercase=True, - remove_punctuation=False, - remove_extra_whitespace=True, - unicode_normalize=True, - expand_contractions=True - ) - - # Normalize text - raw_texts = [ - " Apple Inc. was founded in 1976!! ", - "The iPhone's really great, isn't it?", - "Microsoftโ„ข announced Windowsยฎ 11", - "cafรฉ, naรฏve, rรฉsumรฉ โ€” special chars" - ] - - for raw in raw_texts: - normalized = normalizer.normalize(raw) - print(f"Before: {raw!r}") - print(f"After: {normalized!r}\n") - - # Batch normalization - normalized_batch = normalizer.normalize_batch(raw_texts) - ``` - -=== "Entity Normalization" - Standardize entity names and types: - - ```python - from semantica.normalize import EntityNormalizer - - normalizer = EntityNormalizer( - case_style="title", # "title", "upper", "lower", "preserve" - remove_suffixes=["Inc.", "Corp.", "LLC", "Ltd."], - remove_prefixes=["The"], - normalize_whitespace=True - ) - - # Normalize entity names - entities = [ - {"text": " THE apple INC. ", "type": "Organization"}, - {"text": "MICROSOFT CORPORATION", "type": "Organization"}, - {"text": "tim cook", "type": "Person"}, - {"text": " new york city ", "type": "Location"} - ] - - normalized = normalizer.normalize_entities(entities) - - print("Normalized Entities:") - for orig, norm in zip(entities, normalized): - print(f" {orig['text']!r} -> {norm['text']!r}") - ``` - -=== "Date Normalization" - Parse and normalize date formats: - - ```python - from semantica.normalize import DateNormalizer - - normalizer = DateNormalizer( - output_format="ISO8601", # "ISO8601", "US", "EU", "unix" - infer_missing=True, - handle_relative=True - ) - - # Various date formats - dates = [ - "January 15, 2024", - "15/01/2024", - "01-15-2024", - "2024-01-15", - "last Tuesday", - "3 days ago", - "Q1 2024", - "mid-2023", - ] - - print("Date Normalization:") - for date_str in dates: - result = normalizer.normalize(date_str) - print(f" {date_str:20} -> {result['normalized']}") - if result.get('is_range'): - print(f" Range: {result['start']} to {result['end']}") - ``` - -=== "Number Normalization" - Standardize numeric formats: - - ```python - from semantica.normalize import NumberNormalizer - - normalizer = NumberNormalizer( - output_format="numeric", # "numeric", "text", "scientific" - handle_currencies=True, - handle_percentages=True - ) - - # Various number formats - numbers = [ - "1,234,567", - "one million", - "$3.5 billion", - "45%", - "1.5M", - "three hundred twenty-one", - "โ‚ฌ50,000", - "12.5 percent" - ] - - print("Number Normalization:") - for num_str in numbers: - result = normalizer.normalize(num_str) - print(f" {num_str:25} -> {result['value']:>15,.2f}") - if result.get('currency'): - print(f" Currency: {result['currency']}") - if result.get('is_percentage'): - print(f" Percentage: {result['decimal']:.4f}") - ``` - -=== "Full Pipeline" - Run complete normalization pipeline: - - ```python - from semantica.normalize import ( - TextNormalizer, - EntityNormalizer, - DateNormalizer, - NumberNormalizer - ) - - # Initialize normalizers - text_norm = TextNormalizer(remove_extra_whitespace=True) - entity_norm = EntityNormalizer(case_style="title", remove_suffixes=["Inc.", "Corp."]) - date_norm = DateNormalizer(output_format="ISO8601") - number_norm = NumberNormalizer(handle_currencies=True) - - # Raw knowledge graph data - raw_kg = { - "entities": [ - {"id": "e1", "text": " APPLE INC. ", "type": "Organization", - "properties": {"founded": "April 1, 1976", "revenue": "$394.3 billion"}}, - {"id": "e2", "text": "tim cook", "type": "Person"}, - ], - "relationships": [ - {"source": "e2", "target": "e1", "type": "CEO_OF", - "properties": {"since": "August 24, 2011"}} - ] - } - - # Normalize entities - normalized_entities = [] - for entity in raw_kg['entities']: - norm_entity = entity.copy() - - # Normalize text - norm_entity['text'] = text_norm.normalize(entity['text']) - norm_entity['text'] = entity_norm.normalize_entity_name( - norm_entity['text'], - entity_type=entity['type'] - ) - - # Normalize properties - if 'properties' in entity: - props = entity['properties'].copy() - if 'founded' in props: - props['founded'] = date_norm.normalize(props['founded'])['normalized'] - if 'revenue' in props: - props['revenue'] = number_norm.normalize(props['revenue']) - norm_entity['properties'] = props - - normalized_entities.append(norm_entity) - - # Normalize relationship properties - normalized_rels = [] - for rel in raw_kg['relationships']: - norm_rel = rel.copy() - if 'properties' in rel and 'since' in rel['properties']: - norm_rel['properties']['since'] = date_norm.normalize( - rel['properties']['since'] - )['normalized'] - normalized_rels.append(norm_rel) - - normalized_kg = { - "entities": normalized_entities, - "relationships": normalized_rels - } - - print("Normalized Knowledge Graph:") - for entity in normalized_kg['entities']: - print(f" Entity: {entity['text']}") - for key, value in entity.get('properties', {}).items(): - print(f" {key}: {value}") - ``` - - -**Related Modules**: -- [`normalize` Module](reference/normalize.md) - Data normalization -- [`parse` Module](reference/parse.md) - Document parsing - ---- - -### 11. Conflict Detection - -!!! abstract "Definition" - **Conflict Detection** identifies contradictory information in a knowledge graph, such as conflicting facts about the same entity from different sources. - -**Conflict Types**: - -| Type | Description | Example | -| :--- | :--- | :--- | -| **Value Conflict** | Different values for same property | "Founded: 1976" vs "Founded: 1977" | -| **Relationship Conflict** | Conflicting relationships | "CEO: Tim Cook" vs "CEO: Steve Jobs" | -| **Type Conflict** | Different entity types | "Apple: Company" vs "Apple: Product" | -| **Temporal Conflict** | Conflicting time information | "Active: 2020-2023" vs "Active: 2021-2024" | - - -**Practical Examples**: - -=== "Detecting Conflicts" - Find conflicts in your knowledge graph: - - ```python - from semantica.conflicts import ConflictAnalyzer, ConflictDetector - - # Sample knowledge graph with conflicts - kg = { - "entities": [ - { - "id": "e1", - "text": "Apple Inc.", - "type": "Organization", - "founded": "1976", - "employees": "160000", - "source": "wikipedia" - }, - { - "id": "e1", - "text": "Apple Inc.", - "type": "Organization", - "founded": "1977", - "employees": "164000", - "source": "bloomberg" - }, - ], - "relationships": [ - {"source": "p1", "target": "e1", "type": "CEO_OF", - "properties": {"since": "2011"}, "data_source": "sec_filing"}, - {"source": "p2", "target": "e1", "type": "CEO_OF", - "properties": {"since": "2011"}, "data_source": "news"} - ] - } - - detector = ConflictDetector() - all_conflicts = detector.detect_value_conflicts(kg["entities"], "founded") - - print(f"Found {len(all_conflicts)} conflicts:") - for conflict in all_conflicts: - print(f" Type: {conflict.conflict_type.value}") - print(f" Entity: {conflict.entity_id}") - print(f" Property: {conflict.property_name}") - print(f" Values: {conflict.conflicting_values}") - print(f" Sources: {conflict.sources}") - - # Analyze conflicts - analyzer = ConflictAnalyzer() - analysis = analyzer.analyze_conflicts(all_conflicts) - - print(f"Conflict Analysis:") - print(f" Total conflicts: {analysis['total_conflicts']}") - print(f" By type: {analysis.get('by_type', {}).get('counts')}") - print(f" By severity: {analysis.get('by_severity', {}).get('counts')}") - ``` - -=== "Resolution Strategies" - Resolve conflicts using different strategies: - - ```python - from semantica.conflicts import ( - ConflictResolver, - Conflict, - ConflictType - ) - - # Create sample conflicts - conflicts = [ - Conflict( - conflict_id="c1", - entity_id="e1", - conflict_type=ConflictType.VALUE_CONFLICT, - property_name="founded", - conflicting_values=["1976", "1976", "1977"], - sources=[ - {"document": "wikipedia", "confidence": 0.95}, - {"document": "sec_filing", "confidence": 0.99}, - {"document": "news_article", "confidence": 0.70}, - ], - ) - ] - - # Strategy 1: Voting (most common value wins) - resolver = ConflictResolver() - resolved_voting = resolver.resolve_conflicts(conflicts, strategy="voting") - - print("Voting Resolution:") - for r in resolved_voting: - print(f" {r.conflict_id}: {r.resolved_value} (confidence: {r.confidence:.2f})") - - # Strategy 2: Highest confidence - resolved_conf = resolver.resolve_conflicts(conflicts, strategy="highest_confidence") - - print("Highest Confidence Resolution:") - for r in resolved_conf: - print(f" {r.conflict_id}: {r.resolved_value} (confidence: {r.confidence:.2f})") - - # Strategy 3: Credibility weighted - resolved_cred = resolver.resolve_conflicts(conflicts, strategy="credibility_weighted") - - print("Credibility Weighted Resolution:") - for r in resolved_cred: - print(f" {r.conflict_id}: {r.resolved_value} (confidence: {r.confidence:.2f})") - ``` - -=== "Custom Resolution Rules" - Define custom resolution rules per property: - - ```python - from semantica.conflicts import ConflictResolver - - # Define property-specific resolution rules - resolution_config = { - "founded": { - "strategy": "highest_confidence", - }, - "revenue": { - "strategy": "most_recent", - }, - "employees": { - "strategy": "voting", - }, - "ceo": { - "strategy": "highest_confidence", - } - } - - resolver = ConflictResolver() - - # Resolve conflicts for each property - resolved_values = {} - for property_name, config in resolution_config.items(): - # Get conflicts for this property - property_conflicts = [c for c in conflicts if c.property_name == property_name] - - if property_conflicts: - results = resolver.resolve_conflicts(property_conflicts, strategy=config["strategy"]) - for result in results: - resolved_values[property_name] = { - "value": result.resolved_value, - "strategy": config["strategy"], - "confidence": result.confidence - } - - print("Resolved Values:") - for prop, info in resolved_values.items(): - print(f" {prop}: {info['value']} (via {info['strategy']})") - ``` - -=== "Source Tracking" - Track the origin of resolved values: - - ```python - from semantica.conflicts import SourceTracker, SourceReference - from datetime import datetime - - # Initialize source tracker - tracker = SourceTracker() - - tracker.track_property_source( - "e1", - "founded", - "1976", - SourceReference(document="wikipedia", confidence=0.95, timestamp=datetime(2024, 1, 15)), - ) - tracker.track_property_source( - "e1", - "founded", - "1976", - SourceReference(document="sec_filing", confidence=0.99, timestamp=datetime(2024, 3, 1)), - ) - tracker.track_property_source( - "e1", - "founded", - "1977", - SourceReference(document="news_article", confidence=0.70, timestamp=datetime(2024, 2, 20)), - ) - - sources = tracker.get_entity_sources("e1") - - print("Entity Sources:") - for source in sources: - print(f" Source: {source.document}") - print(f" Confidence: {source.confidence}") - - # Get property-specific sources - property_sources = tracker.get_property_sources("e1", "founded") - - print("Property Sources for 'founded':") - if property_sources: - print(f" Value: {property_sources.value}") - for ps in property_sources.sources: - print(f" {ps.document} ({ps.timestamp})") - ``` - -=== "Conflict Investigation" - Deep dive into specific conflicts: - - ```python - from semantica.conflicts import ( - InvestigationGuideGenerator, - Conflict, - ConflictType - ) - - # Initialize investigation guide generator - guide_generator = InvestigationGuideGenerator() - - # Create a conflict object - conflict = Conflict( - conflict_id="c1", - entity_id="e1", - conflict_type=ConflictType.VALUE_CONFLICT, - property_name="founded", - conflicting_values=["1976", "1977"], - sources=[ - {"document": "wikipedia", "confidence": 0.95}, - {"document": "news", "confidence": 0.70}, - ], - severity="medium" - ) - - # Generate investigation guide - guide = guide_generator.generate_guide(conflict) - - print("Conflict Investigation Guide:") - print(f" Conflict ID: {guide.conflict_id}") - print(f" Title: {guide.title}") - print(f" Summary: {guide.conflict_summary}") - - print("Investigation Steps:") - for i, step in enumerate(guide.investigation_steps, 1): - print(f" {i}. {step.description}") - if step.expected_outcome: - print(f" Expected: {step.expected_outcome}") - print(f" Value count: {len(conflict.conflicting_values)}") - - ``` - -**Related Modules**: -- [`conflicts` Module](reference/conflicts.md) - Conflict detection and resolution -- [`kg_qa` Module](reference/evals.md) - Quality assurance - ---- - -## Comparison Tables - -### Embedding Providers Comparison - -| Provider | Model | Dimensions | Speed | Cost | Accuracy | Best For | -| :--- | :--- | :--- | :--- | :--- | :--- | :--- | -| **OpenAI** | text-embedding-3-large | 3072 | Fast | Paid | High | Production, high accuracy | -| **OpenAI** | text-embedding-3-small | 1536 | Fast | Paid | High | Balanced performance | -| **Cohere** | embed-english-v3.0 | 1024 | Fast | Paid | High | Multilingual support | -| **HuggingFace** | all-MiniLM-L6-v2 | 384 | Medium | Free | Medium | Development, open source | -| **Local** | sentence-transformers | 384-768 | Slow | Free | Medium | Privacy, offline use | - -### Graph Backend Comparison - -| Backend | Type | Speed | Scalability | Query Language | Best For | -| :--- | :--- | :--- | :--- | :--- | :--- | -| **NetworkX**| In-memory | Fast | Small-medium | Python API | Development, small graphs | -| **Neo4j** | Database | Medium | Large | Cypher | Production, complex queries | -| **FalkorDB**| Redis-based| Very Fast | Large | Cypher | Real-time, high throughput | +### Common Use Cases +- **Document Analysis** - extract knowledge from reports +- **Research Assistant** - find connections in academic papers +- **Business Intelligence** - analyze company relationships +- **Regulatory Compliance** - track policy changes --- ## Best Practices -Following these practices will help you build high-quality knowledge graphs and avoid common pitfalls. +### Start Small +- Begin with a single document type +- Focus on specific entity types +- Validate results before scaling -### 1. Start Small +### Configure Properly +- Choose appropriate models for your domain +- Set confidence thresholds +- Define clear ontology rules -!!! tip "Iterative Approach" - Don't try to model the entire world at once. Start with a small, well-defined domain and expand incrementally. - -**Example**: -```python -# Start with a single document -kg1 = semantica.build_knowledge_base(["doc1.pdf"]) - -# Validate and refine -quality = assessor.assess(kg1) - -# Then expand -kg2 = semantica.build_knowledge_base(["doc2.pdf", "doc3.pdf"]) -merged = semantica.kg.merge([kg1, kg2]) -``` - -### 2. Configure Properly - -- Use environment variables for sensitive data -- Set up proper logging -- Configure appropriate model sizes -- Use configuration files for complex setups - -```python -# Good: Use environment variables -import os -api_key = os.getenv("OPENAI_API_KEY") - -# Good: Use config files -from semantica.core import Config -config = Config.from_file("config.yaml") -semantica = Semantica(config=config) -``` - -### 3. Validate Data - -!!! warning "Garbage In, Garbage Out" - Always validate extracted entities. A knowledge graph with incorrect facts is worse than no graph at all. - -**Validation Checklist**: -- Check entity extraction accuracy -- Validate relationships make sense -- Verify confidence scores -- Review source attribution -- Test with sample queries - -### 4. Handle Errors +### Validate Data +- Check extraction quality +- Review relationship accuracy +- Test with known examples +### Handle Errors - Implement error handling -- Use retry mechanisms -- Log errors for debugging -- Gracefully handle API failures +- Log processing issues +- Provide feedback mechanisms -```python -from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor, RelationExtractor -from semantica.kg import GraphBuilder -import logging +### Optimize Performance +- Use appropriate storage backends +- Cache frequently accessed data +- Monitor resource usage -logging.basicConfig(level=logging.INFO) - -# Use individual modules -ingestor = FileIngestor() -parser = DocumentParser() -ner = NERExtractor() -rel_extractor = RelationExtractor() -builder = GraphBuilder() - -try: - doc = ingestor.ingest_file("doc.pdf") - parsed = parser.parse_document("doc.pdf") - text = parsed.get("full_text", "") - entities = ner.extract_entities(text) - relationships = rel_extractor.extract_relations(text, entities=entities) - kg = builder.build_graph(entities=entities, relationships=relationships) -except Exception as e: - logging.error(f"Error building KG: {e}") - # Handle error appropriately -``` - -### 5. Optimize Performance - -- Use batch processing for large datasets -- Enable parallel processing where possible -- Cache embeddings and results -- Use appropriate backend for your scale - -```python -# Batch processing -sources = ["doc1.pdf", "doc2.pdf", ..., "doc100.pdf"] -batch_size = 10 - -for i in range(0, len(sources), batch_size): - batch = sources[i:i+batch_size] - result = semantica.build_knowledge_base(batch) - # Process and save results -``` - -### 6. Document Workflows - -- Document data sources -- Track processing steps -- Maintain metadata -- Version your knowledge graphs +### Document Workflows +- Record processing steps +- Track data sources +- Maintain change logs --- +## Need Help? - -## Next Steps - -Now that you understand the core concepts: - -1. **[Getting Started](getting-started.md)** - Set up Semantica and build your first knowledge graph -2. **[Modules Guide](modules.md)** - Learn about the available modules -3. **[Use Cases](use-cases.md)** - Explore real-world applications -4. **[Examples](examples.md)** - See practical code examples -5. **[API Reference](reference/core.md)** - Detailed API documentation - ---- - -!!! info "Contribute" - Found an issue or want to improve this guide? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica) - +- **Documentation**: [Getting Started](getting-started.md) +- **Examples**: [Cookbook](cookbook.md) +- **Community**: [Discord](community.md) +- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) +- **Support**: [Contact Us](community.md) diff --git a/docs/contributing.md b/docs/contributing.md index c2c6317e..b2441e3b 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -1,126 +1,130 @@ -# Contributing to Semantica +๏ปฟ# Contributing -Thank you for your interest in contributing to Semantica! - -!!! tip "Quick Start" - New to contributing? Check out issues labeled [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) +**Help us build Semantica! Every contribution makes the project better.** --- -## ๐Ÿ“š Essential Links +## Getting Started -- **[Contributing Guide](https://github.com/Hawksight-AI/semantica/blob/main/CONTRIBUTING.md)** - Complete contribution guidelines -- **[Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md)** - Community standards -- **[Security Policy](https://github.com/Hawksight-AI/semantica/blob/main/SECURITY.md)** - Report vulnerabilities -- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Bug reports and features +### Quick Start +1. **Fork** the repository +2. **Create** a feature branch +3. **Make** your changes +4. **Test** your changes +5. **Submit** a pull request + +### First Contribution? +Look for issues labeled [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) for beginner-friendly tasks. --- -## ๐ŸŽฏ Ways to Contribute +## Ways to Contribute -### Code Contributions - -1. Fork the repository -2. Create a feature branch -3. Make your changes -4. Submit a pull request - -See the [Contributing Guide](https://github.com/Hawksight-AI/semantica/blob/main/CONTRIBUTING.md) for detailed instructions. +### Code +- **Fix bugs** - Resolve reported issues +- **Add features** - Implement new functionality +- **Improve performance** - Optimize existing code +- **Refactor** - Clean up code structure ### Documentation +- **Fix typos** - Correct spelling and grammar +- **Improve guides** - Make documentation clearer +- **Add examples** - Provide practical code examples +- **Update API docs** - Keep reference current -- Fix typos and improve clarity -- Add examples and tutorials -- Update API documentation -- Translate documentation +### Testing +- **Write tests** - Add test coverage +- **Fix tests** - Resolve test failures +- **Report issues** - Identify bugs through testing + +### Community +- **Help others** - Answer questions in issues +- **Share knowledge** - Write tutorials and guides +- **Provide feedback** - Review pull requests + +--- + +## Reporting Issues ### Bug Reports - -Report bugs on [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with: -- Description of the problem -- Steps to reproduce -- Expected vs actual behavior -- Environment details +When reporting bugs, include: +- **Description** - What happened +- **Steps to reproduce** - How to trigger the issue +- **Expected behavior** - What should happen +- **Environment** - Your setup details ### Feature Requests - -Suggest features on [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with: -- Use case description -- Proposed solution -- Benefits to the community +When suggesting features, include: +- **Use case** - Why you need this feature +- **Proposed solution** - How it should work +- **Benefits** - How it helps the community --- -## โœ๏ธ Documentation Style Guide +## Pull Request Guidelines -### Writing Guidelines +### Before Submitting +- **Test** your changes thoroughly +- **Document** new features with examples +- **Update** relevant documentation +- **Follow** the existing code style -- Use clear, concise language -- Include working code examples -- Test all examples before submitting -- Follow existing documentation structure -- Use proper markdown formatting +### Pull Request Checklist +- [ ] Code follows project style +- [ ] Tests pass locally +- [ ] Documentation is updated +- [ ] Commit messages are clear +- [ ] No merge conflicts -### API Documentation Format +--- -```python -def function_name( - param1: str, - param2: int = 0 -) -> ReturnType: - """Brief description. - - Args: - param1: Description of param1 - param2: Description of param2 (default: 0) - - Returns: - Description of return value - - Raises: - ValueError: When and why this is raised - - Example: - >>> result = function_name("test", 5) - >>> print(result) - expected_output - """ +## Development Setup + +### Local Development +```bash +# Clone your fork +git clone https://github.com/your-username/semantica.git +cd semantica + +# Install in development mode +pip install -e .[dev] + +# Run tests +pytest ``` ---- - -## ๐Ÿ“ Documentation Structure - -``` -docs/ -โ”œโ”€โ”€ index.md # Homepage -โ”œโ”€โ”€ getting-started.md # Getting started -โ”œโ”€โ”€ concepts.md # Core concepts -โ”œโ”€โ”€ modules.md # Module overview -โ”œโ”€โ”€ use-cases.md # Use cases -โ”œโ”€โ”€ examples.md # Examples -โ”œโ”€โ”€ cookbook/ # Tutorials -โ””โ”€โ”€ reference/ # API reference -``` +### Code Style +We use standard Python formatting: +- **Black** for code formatting +- **isort** for import sorting +- **flake8** for linting --- -## ๐Ÿ› ๏ธ Documentation Tools +## Community Guidelines -- **[MkDocs](https://www.mkdocs.org/)** - Documentation generator -- **[Material for MkDocs](https://squidfunk.github.io/mkdocs-material/)** - Theme -- **[mkdocstrings](https://mkdocstrings.github.io/)** - API docs from docstrings -- **[Mermaid](https://mermaid.js.org/)** - Diagrams +### Code of Conduct +Please follow our [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md). + +### Communication +- **Be respectful** - Treat everyone with kindness +- **Be helpful** - Assist others when you can +- **Be patient** - Allow time for reviews +- **Be constructive** - Provide helpful feedback --- -## ๐Ÿค Getting Help +## Recognition + +All contributors are recognized in: +- **GitHub contributors** - Automatic recognition +- **Release notes** - Notable contributions +- **Community highlights** - Outstanding work + +--- + +## Need Help? - **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Ask questions -- **Documentation** - Check existing docs for examples -- **Pull Requests** - Review other contributors' PRs - ---- - -!!! success "Thank You!" - Every contribution helps make Semantica better! ๐ŸŽ‰ +- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Community chat +- **[Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md)** - Community standards diff --git a/docs/css/custom.css b/docs/css/custom.css index 0ef31eb9..079c651b 100644 --- a/docs/css/custom.css +++ b/docs/css/custom.css @@ -34,18 +34,18 @@ html { [data-md-color-scheme="slate"] { /* Dark Mode */ --md-default-bg-color: #0F1115; - /* Very dark grey, almost black */ --md-default-fg-color: #E0E0E0; --md-primary-fg-color: #0F1115; - /* Match bg for seamless look or slightly lighter */ --md-primary-fg-color--light: #212121; --md-primary-fg-color--dark: #000000; - - margin-bottom: 1rem; - color: var(--md-default-fg-color); } +/* + ========================================================================== + Typography + ========================================================================== +*/ .md-typeset h2 { font-weight: 700; letter-spacing: -0.01em; @@ -66,6 +66,12 @@ html { background-color: #F1F8F5; } +/* + ========================================================================== + Admonitions + ========================================================================== +*/ +/* Tip */ .md-typeset .admonition.tip .admonition-title { color: #00C853; } @@ -137,7 +143,11 @@ html { border-color: rgba(255, 255, 255, 0.05); } -/* Scrollbars */ +/* + ========================================================================== + Scrollbars + ========================================================================== +*/ ::-webkit-scrollbar { width: 6px; height: 6px; @@ -188,7 +198,6 @@ html { Active Link Highlighting ========================================================================== */ - /* Left Sidebar (Navigation) - Active Link */ .md-nav__link--active { color: var(--md-accent-fg-color) !important; @@ -199,20 +208,19 @@ html { .md-nav__item--active > .md-nav__link { color: var(--md-accent-fg-color) !important; border-left: 2px solid var(--md-accent-fg-color); - padding-left: 0.5rem; /* Adjust padding to look good with border */ + padding-left: 0.5rem; } /* Ensure nested items in TOC don't inherit the border unless active themselves */ .md-nav__item .md-nav__item--active > .md-nav__link { - border-left: 2px solid var(--md-accent-fg-color); + border-left: 2px solid var(--md-accent-fg-color); } /* ========================================================================== - Home Page Content Alignment - Left Align + Layout Optimization ========================================================================== */ - /* Reduce spacing between sidebars and content for all pages */ .md-content__inner { padding-left: 0.75rem; diff --git a/docs/faq.md b/docs/faq.md index 9978ff78..cf96b220 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,280 +1,148 @@ -# Frequently Asked Questions +๏ปฟ# Frequently Asked Questions -Common questions and answers about Semantica. - -!!! tip "Can't find your question?" - Browse existing questions or [ask a new question on GitHub Issues](https://github.com/Hawksight-AI/semantica/issues/new) +**Common questions about Semantica and how to use it.** --- -## General Questions +## General ### What is Semantica? +Semantica is an open-source framework for building knowledge graphs from unstructured data. It transforms documents, web pages, and databases into structured, queryable knowledge. -Semantica is an open-source framework for building semantic layers and knowledge graphs from unstructured data. It transforms raw data into structured, queryable knowledge that powers AI applications. - -### What can I use Semantica for? - -- Building knowledge graphs from documents -- Creating semantic layers for AI applications -- Extracting entities and relationships -- Powering GraphRAG systems -- Integrating multi-source data -- Building AI agent memory +### What can I do with Semantica? +- **Build knowledge graphs** from documents and data +- **Extract entities and relationships** automatically +- **Power AI applications** with structured knowledge +- **Create semantic search** and GraphRAG systems +- **Integrate multiple data sources** into unified graphs ### Is Semantica free? - -Yes! Semantica is 100% open source and free to use under the MIT License. +Yes! Semantica is open source under the MIT License. ### What makes Semantica different? - -- **Modular**: Use only what you need -- **Extensible**: Plug in custom models -- **Production-ready**: Built for scale -- **Open source**: Fully transparent +- **Modular architecture** - Use only what you need +- **Production-ready** - Built for scale and reliability +- **Extensible** - Add custom models and components +- **Open source** - Transparent and community-driven --- -## Installation & Setup +## Installation ### How do I install Semantica? - ```bash pip install semantica ``` -See the [Installation Guide](installation.md) for details. - ### What Python version do I need? +Python 3.8 or higher. Python 3.11+ is recommended. -Python 3.8 or higher. Python 3.11+ is recommended for best performance. - -### Do I need a GPU? - -No, GPU is optional. Semantica works on CPU, but GPU acceleration is available for faster processing. - -### How do I get started? - -1. Install: `pip install semantica` -2. Follow the [Quick Start Guide](quickstart.md) -3. Try the [Examples](examples.md) +### What are the system requirements? +- Python 3.8+ +- 4GB+ RAM for basic use +- Optional GPU for embeddings and ML models --- -## Knowledge Graphs - -### What is a knowledge graph? - -A structured representation where entities (nodes) are connected by relationships (edges). It captures semantic meaning and relationships in data. - -### How do I build a knowledge graph? - -```python -from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor, RelationExtractor -from semantica.kg import GraphBuilder - -# Use individual modules -ingestor = FileIngestor() -parser = DocumentParser() -ner = NERExtractor() -rel_extractor = RelationExtractor() - -doc = ingestor.ingest_file("document.pdf") -parsed = parser.parse_document("document.pdf") -text = parsed.get("full_text", "") - -entities = ner.extract_entities(text) -relationships = rel_extractor.extract_relations(text, entities=entities) - -builder = GraphBuilder() -kg = builder.build_graph(entities=entities, relationships=relationships) -``` - -### Can I merge multiple knowledge graphs? - -Yes! Use the `merge` method: - -```python -merged = semantica.kg.merge([kg1, kg2, kg3]) -``` - -### How do I visualize a knowledge graph? - -```python -semantica.kg.visualize(kg, output_path="graph.html") -``` - ---- - -## Usage & Features - -### Can I process PDF files? - -Yes! Semantica supports PDF, DOCX, HTML, JSON, CSV, and many other formats. - -### How do I extract entities from text? +## Getting Started +### How do I start using Semantica? ```python from semantica.semantic_extract import NERExtractor +from semantica.kg import GraphBuilder -# Use NER extractor directly +# Extract entities ner = NERExtractor() -entities = ner.extract_entities("Your text") +entities = ner.extract("Apple Inc. was founded by Steve Jobs.") + +# Build knowledge graph +kg = GraphBuilder().build({"entities": entities}) ``` -### Can I use my own models? - -Yes, Semantica is extensible. You can plug in custom models for entity extraction, embeddings, and more. - -### What export formats are supported? - -- RDF/XML -- OWL (Ontology) -- JSON -- CSV -- YAML -- And more +### Where can I find examples? +- **[Getting Started Guide](getting-started.md)** - Quick introduction +- **[Cookbook](cookbook.md)** - Practical examples +- **[GitHub Examples](https://github.com/Hawksight-AI/semantica/tree/main/examples)** - Code samples --- -## Conflict Resolution +## Features -### What is conflict resolution? +### What data sources does Semantica support? +- **Files**: PDF, DOCX, TXT, JSON, CSV +- **Web**: Websites, RSS feeds, APIs +- **Databases**: PostgreSQL, MySQL, Snowflake, MongoDB +- **Streams**: Kafka, RabbitMQ, real-time data -When the same entity appears in multiple sources with different information, conflict resolution determines which information to use. +### Can I use custom models? +Yes! Semantica supports custom: +- **Entity extraction models** +- **Embedding models** +- **Language models** +- **Custom processors** -### What strategies are available? - -- **Voting**: Majority wins -- **Credibility Weighted**: Weight by source credibility -- **Most Recent**: Use latest information -- **Highest Confidence**: Use highest confidence score - -### How do I set a resolution strategy? - -```python -from semantica.conflicts import ConflictResolver - -resolver = ConflictResolver(default_strategy="voting") -``` +### Does Semantica support GPUs? +Yes, Semantica automatically uses GPUs when available for: +- **Embedding generation** +- **ML model inference** +- **Vector operations** --- -## Integration +## Technical -### Can I use Semantica with other tools? +### How does Semantica handle large datasets? +- **Batching** - Process data in chunks +- **Streaming** - Handle real-time data +- **Parallel processing** - Use multiple cores +- **Memory management** - Efficient resource usage -Yes! Semantica exports to standard formats that work with: +### Can I deploy Semantica in production? +Yes! Semantica is production-ready with: +- **Scalable architecture** +- **Error handling** +- **Monitoring support** +- **Container deployment** -- Neo4j -- Graph databases -- RDF stores -- Vector databases -- Any tool that accepts RDF/JSON/CSV - -### Does it work with LangChain? - -Yes, Semantica can be integrated with LangChain for RAG applications. - -### Can I connect to databases? - -Yes, Semantica supports connections to Neo4j, FalkorDB, and other graph databases. - ---- - -## Performance - -### How fast is Semantica? - -Performance depends on: - -- Document size -- Number of documents -- Hardware (CPU/GPU) -- Configuration options - -For typical documents, processing takes seconds to minutes. - -### Can I process large datasets? - -Yes, but consider: - -- Processing in batches -- Using GPU acceleration -- Incremental building -- Optimizing configuration - -### How can I improve performance? - -- Enable GPU if available -- Process in smaller batches -- Use faster models -- Optimize configuration -- Cache embeddings +### How do I customize Semantica? +- **Custom processors** - Add new extraction logic +- **Custom models** - Use your own ML models +- **Plugins** - Extend functionality +- **Configuration** - Adjust behavior --- ## Troubleshooting -### Installation fails +### Installation issues +- **Python version**: Ensure Python 3.8+ +- **Dependencies**: Install with `pip install -e .[dev]` +- **Permissions**: Use virtual environments -- Upgrade pip: `pip install --upgrade pip` -- Use virtual environment -- Check Python version: `python --version` +### Performance issues +- **Memory**: Increase available RAM +- **GPU**: Install CUDA for GPU acceleration +- **Batching**: Use smaller chunk sizes -### No entities extracted - -- Verify document contains text (not just images) -- Check document format is supported -- Review extraction configuration - -### Memory errors - -- Process documents one at a time -- Reduce batch sizes -- Use smaller models -- Increase available RAM - -### Slow processing - -- Enable GPU if available -- Process in smaller batches -- Optimize configuration -- Use faster models +### Common errors +- **Import errors**: Check installation path +- **Model loading**: Verify model availability +- **Memory errors**: Reduce batch sizes --- -## Getting Help +## Support ### Where can I get help? +- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report problems +- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Ask questions +- **[Documentation](index.md)** - Browse guides and references -- **Documentation**: This site -- **GitHub Issues**: [Report bugs or ask questions](https://github.com/Hawksight-AI/semantica/issues) - -### How do I report a bug? - -Open an issue on [GitHub](https://github.com/Hawksight-AI/semantica/issues) with: - -- Description of the problem -- Steps to reproduce -- Expected vs actual behavior -- Environment details +### How do I report bugs? +1. **Search** existing issues first +2. **Create** a new issue with details +3. **Include** reproduction steps +4. **Add** environment information ### Can I contribute? - -Yes! We welcome contributions. See our [Contributing Guide](https://github.com/Hawksight-AI/semantica/blob/main/CONTRIBUTING.md). - -### How do I request a feature? - -Open a feature request on [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with: - -- Use case description -- Proposed solution -- Benefits to the community - ---- - -!!! question "Still have questions?" - Check the [API Reference](reference/core.md), browse the [Cookbook](cookbook.md), or [ask on GitHub Issues](https://github.com/Hawksight-AI/semantica/issues/new) +Yes! See the [Contributing Guide](contributing.md) for details on how to help improve Semantica. diff --git a/docs/getting-started.md b/docs/getting-started.md index 33c7922e..bcf6148f 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,214 +1,101 @@ # Getting Started -## Welcome to Semantica +## Overview -**Semantica** is a comprehensive knowledge graph and semantic processing framework designed for building production-ready semantic AI applications. +**Semantica** is a semantic intelligence layer that bridges the gap between raw data and trustworthy AI. It transforms unstructured data into explainable, auditable knowledge graphs perfect for high-stakes domains. -### ๐ŸŽฏ What You'll Learn -- What Semantica is and why it's useful -- How to install and configure the framework -- Understanding the framework architecture -- Key concepts and terminology -- Next steps for getting started +### What You Can Build +- **GraphRAG Systems** - Enhanced retrieval with semantic reasoning +- **AI Agents** - Trustworthy agents with explainable memory +- **Knowledge Graphs** - Production-ready semantic databases +- **Compliance-Ready AI** - Auditable systems with full provenance --- -## ๐Ÿš€ What is Semantica? +## Installation -Semantica is a powerful, production-ready framework for: +```bash +pip install semantica +``` -- **Building Knowledge Graphs**: Transform unstructured data into structured knowledge graphs. -- **Semantic Processing**: Extract entities, relationships, and meaning from text, images, and audio. -- **GraphRAG**: Next-generation retrieval augmented generation using knowledge graphs. -- **Temporal Analysis**: Time-aware knowledge graphs for tracking changes over time. -- **Multi-Modal Processing**: Handle text, images, audio, and structured data. -- **Enterprise Features**: Quality assurance, conflict resolution, ontology generation, and more. +Or with all features: ---- +```bash +pip install semantica[all] +``` -## ๐Ÿ’ก Use Cases - -| Domain | Application | -| :--- | :--- | -| **Cybersecurity** | Threat intelligence and analysis | -| **Healthcare** | Medical research and patient data analysis | -| **Finance** | Fraud detection and financial analysis | -| **Supply Chain** | Optimization and risk management | -| **Research** | Knowledge management and literature review | -| **AI Systems** | Multi-agent memory and reasoning | - ---- - -## ๐Ÿ“ฆ Installation & Setup - -### Prerequisites -Before installing Semantica, ensure you have: -- **Python 3.8** or higher -- **pip** package manager -- (Optional) Virtual environment for isolation - -### Installation Methods - -=== "PyPI (Stable)" - ```bash - pip install semantica - ``` - -=== "Source (Dev)" - ```bash - git clone https://github.com/Hawksight-AI/semantica.git - cd semantica - pip install -e . - ``` - -=== "Extras" - ```bash - pip install semantica[all] # Install all optional dependencies - pip install semantica[gpu] # Install GPU support - pip install semantica[visualization] # Install visualization tools - ``` - -### Verify Installation +Verify installation: ```python import semantica -print(semantica.__version__) +print(f"Semantica {semantica.__version__} installed!") ``` --- -## ๐Ÿ—๏ธ Understanding Semantica's Architecture +## Quick Start -Semantica uses a **modular architecture** where each module handles a specific aspect of semantic processing. This design gives you flexibility and control over your pipeline. - -### Primary Approach: Individual Modules - -The recommended approach is to use individual modules directly. Each module can be imported and used independently: - -- **`semantica.ingest`**: Data ingestion from files, web, databases -- **`semantica.parse`**: Document parsing and text extraction -- **`semantica.semantic_extract`**: Entity and relationship extraction -- **`semantica.kg`**: Knowledge graph construction -- **`semantica.embeddings`**: Vector embedding generation -- **`semantica.vector_store`**: Vector database operations - -**Benefits of the modular approach:** -- **Full control**: Customize each step of your pipeline -- **Flexibility**: Mix and match modules as needed -- **Transparency**: Clear understanding of what each step does -- **Easy debugging**: Isolate issues to specific modules - -**Quick Example:** ```python -from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor, RelationExtractor +from semantica.semantic_extract import NERExtractor from semantica.kg import GraphBuilder -# Each module is used independently -ingestor = FileIngestor() -parser = DocumentParser() -ner = NERExtractor() -builder = GraphBuilder() +# Extract entities +ner = NERExtractor(method="ml", model="en_core_web_sm") +entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") + +# Build knowledge graph +kg = GraphBuilder().build({"entities": entities, "relationships": []}) +print(f"Built KG with {len(kg.get('entities', []))} entities") ``` -**For detailed examples, see:** -- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules and architecture - - **Topics**: Framework overview, all modules, architecture, configuration - - **Difficulty**: Beginner - - **Time**: 30-45 minutes - - **Use Cases**: First-time users, understanding the framework structure - -### Alternative Approach: Orchestration Class - -For complex workflows, you can use the `` `Semantica` `` class for orchestration. This class coordinates multiple modules and provides lifecycle management. - -**When to use orchestration:** -- Complex multi-step workflows spanning multiple modules -- Need lifecycle management (initialization, shutdown) -- Want centralized configuration -- Building applications with multiple components - -!!! tip "Getting Started" - For beginners, start with individual modules to understand how each component works. As you build more complex applications, consider using the orchestration class for workflow management. See the [Core Module Reference](reference/core.md) for orchestration details. - -## โš™๏ธ Configuration - -Semantica modules can be configured individually or through environment variables. Configuration options vary by module, allowing you to customize behavior for your specific needs. - -### Environment Variables - -Common configuration via environment variables: - -```bash -export OPENAI_API_KEY=your_openai_key -export EMBEDDING_MODEL=all-MiniLM-L6-v2 -export EMBEDDING_DEVICE=cuda -``` - -### Module-Specific Configuration - -Each module accepts configuration parameters when instantiated. For example, the NER extractor can be configured with different methods, providers, and thresholds. - -### Config File (`config.yaml`) - -For centralized configuration, you can use a YAML config file to manage settings across multiple modules: - -```yaml -api_keys: - openai: your_key_here - -embedding: - provider: openai - model: text-embedding-3-large - -knowledge_graph: - backend: networkx - temporal: true -``` - -**For detailed configuration examples, see:** -- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Configuration examples for all modules -- **[Core Module Reference](reference/core.md)**: Complete configuration documentation +**What this does:** +- Extracts entities (people, organizations, dates) from text +- Builds a knowledge graph from extracted entities +- Outputs the number of entities found --- -## โญ๏ธ Next Steps +## Core Architecture -Now that you understand the basics, here are recommended next steps: +Semantica uses a **modular architecture** - use only what you need: -### ๐Ÿณ Interactive Tutorials (Cookbook) +### 1๏ธโƒฃ Input Layer - Data Ingestion +```python +from semantica.ingest import FileIngestor +documents = FileIngestor().ingest_directory("docs/") +``` -Get hands-on experience with these interactive Jupyter notebooks: +### 2๏ธโƒฃ Semantic Layer - Intelligence Engine +```python +from semantica.semantic_extract import NERExtractor, RelationExtractor +entities = NERExtractor().extract(text) +relationships = RelationExtractor().extract(text, entities) +``` -1. **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all Semantica modules - - **Topics**: Framework overview, all modules, architecture, configuration - - **Difficulty**: Beginner - - **Time**: 30-45 minutes - - **Use Cases**: First-time users, understanding the framework structure +### 3๏ธโƒฃ Output Layer - Knowledge Assets +```python +from semantica.kg import GraphBuilder +kg = GraphBuilder().build_graph(entities, relationships) +``` -2. **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph from a document - - **Topics**: Entity extraction, relationship extraction, graph construction, visualization - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Learning the basics, quick start +--- -3. **[Data Ingestion](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources - - **Topics**: File, web, feed, stream, database ingestion - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Loading data from various sources +## Next Steps -4. **[Document Parsing](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)**: Parse various document formats - - **Topics**: PDF, DOCX, HTML, JSON parsing - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Extracting text from different file formats +### ๐Ÿณ Interactive Tutorials +1. **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)** - Complete framework overview +2. **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)** - Hands-on graph building +3. **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)** - Production-ready RAG -### ๐Ÿ“š Documentation +### ๐Ÿ“š Learn More +- **[Core Concepts](concepts.md)** - Deep dive into knowledge graphs & ontologies +- **[Cookbook](cookbook.md)** - 14 domain-specific tutorials +- **[API Reference](reference/core.md)** - Complete technical documentation -- **[Quick Start Guide](quickstart.md)**: Step-by-step tutorial to build your first knowledge graph -- **[Core Concepts](concepts.md)**: Deep dive into knowledge graphs, ontologies, and semantic reasoning -- **[API Reference](reference/core.md)**: Complete technical documentation for all modules -- **[Examples](examples.md)**: Real-world examples and use cases -- **[Cookbook](cookbook.md)**: Full list of interactive Jupyter notebooks +--- + +## Need Help? + +- **[๐Ÿ’ฌ Discord Community](https://discord.gg/ggb7vWeP)** - Get help from the community +- **[๐Ÿ› Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report bugs or request features +- **[๐Ÿ“– Documentation](https://semantica.readthedocs.io/)** - Full documentation site diff --git a/docs/glossary.md b/docs/glossary.md index 6f3d1e3b..e065699d 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -1,213 +1,232 @@ # Glossary -A comprehensive reference of terms and concepts used in Semantica. +**Comprehensive reference of terms and concepts used in Semantica and semantic intelligence.** + +!!! tip "Quick Reference" + Looking for a specific term? Use your browser's search function (Ctrl+F) to find terms quickly. --- -## A +## Core Concepts -**Agent** -: An autonomous AI system that can perceive its environment, reason about information, and take actions to achieve specific goals. In Semantica, agents use knowledge graphs for memory and reasoning. +### **Agent** +An autonomous AI system that can perceive its environment, reason about information, and take actions to achieve specific goals. In Semantica, agents use knowledge graphs for memory and reasoning. -**API (Application Programming Interface)** -: A set of functions and protocols that allow different software applications to communicate with each other. +### **Entity** +A distinct object or concept in the real world, such as a person, place, organization, or event. Entities are the fundamental building blocks of knowledge graphs. -**Axiom** -: A statement or rule that is accepted as true without proof, used in ontologies to define logical constraints and relationships. +### **Knowledge Graph (KG)** +A structured representation of knowledge using entities (nodes) and relationships (edges). KGs enable reasoning, querying, and semantic analysis of data. + +### **Relationship** +A connection between two entities that describes how they relate to each other (e.g., "works_for", "located_in", "founded_by"). + +### **Semantic** +Relating to meaning in language or logic. Semantic understanding goes beyond keywords to comprehend context and intent. --- -## C +## Data Processing -**Centrality** -: A measure of the importance or influence of a node in a graph. Common centrality metrics include PageRank, betweenness centrality, and closeness centrality. +### **Ingestion** +The process of loading data from various sources (files, databases, APIs, streams) into a system for processing. -**Class** -: In ontologies, a category or type of entity (e.g., `Person`, `Organization`, `Location`). +### **Normalization** +The process of standardizing data into a consistent format (e.g., converting dates to ISO format, standardizing entity names). -**Community Detection** -: The process of identifying groups or clusters of densely connected nodes in a graph. +### **Parsing** +Extracting structured information from unstructured or semi-structured documents like PDFs, Word documents, or web pages. -**Conflict Resolution** -: The process of handling contradictory information from multiple sources in a knowledge graph. - -**Coreference Resolution** -: The task of determining when two or more expressions in text refer to the same entity (e.g., "Apple" and "the company" referring to Apple Inc.). - -**Cypher** -: A declarative query language for graph databases, particularly Neo4j. +### **Chunking** +Breaking down large documents into smaller, manageable pieces while preserving context and meaning. --- -## E +## Artificial Intelligence -**Embedding** -: A dense vector representation of text, images, or other data that captures semantic meaning in a continuous vector space. Used for similarity search and semantic matching. +### **LLM (Large Language Model)** +A type of artificial intelligence model trained on vast amounts of text data, capable of understanding and generating human-like text. -**Entity** -: A distinct object or concept in the real world, such as a person, place, organization, or event. +### **RAG (Retrieval Augmented Generation)** +A technique that enhances LLM responses by retrieving relevant information from a knowledge base before generating an answer. -**Entity Resolution** -: The process of determining when two entity mentions refer to the same real-world entity, also known as entity linking or deduplication. +### **GraphRAG (Graph-Augmented Retrieval Augmented Generation)** +An advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs. -**Event Detection** -: The task of identifying and classifying events (e.g., acquisitions, partnerships, announcements) in text. +### **Inference** +The process of deriving new facts or conclusions from existing knowledge using logical rules. --- -## G +## Knowledge Graph Components -**Graph** -: A data structure consisting of nodes (vertices) and edges (relationships) connecting them. +### **Node** +A vertex in a graph representing an entity or concept. -**GraphRAG (Graph-Augmented Retrieval Augmented Generation)** -: An advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs. +### **Edge** +A connection between two nodes representing a relationship. + +### **Property** +An attribute or characteristic of an entity or relationship (e.g., name, date, confidence score). + +### **Triplet** +A basic unit of knowledge in RDF, consisting of a subject, predicate, and object (e.g., ` `). + +### **Temporal Graph** +A knowledge graph that tracks changes over time, allowing queries about the state of the graph at specific time points. --- -## H +## Entity Recognition & Extraction -**Hybrid Search** -: A search strategy that combines multiple retrieval methods, typically vector search and keyword search, to improve accuracy. +### **Named Entity Recognition (NER)** +The process of identifying and classifying named entities in text into predefined categories such as persons, organizations, locations, dates, and more. + +### **Relationship Extraction** +The task of identifying and extracting semantic relationships between entities in text. + +### **Entity Resolution** +The process of determining when two entity mentions refer to the same real-world entity, also known as entity linking or deduplication. + +### **Coreference Resolution** +The task of determining when two or more expressions in text refer to the same entity (e.g., "Apple" and "the company" referring to Apple Inc.). + +### **Event Detection** +The task of identifying and classifying events (e.g., acquisitions, partnerships, announcements) in text. --- -## I +## Ontology & Schema -**Inference** -: The process of deriving new facts or conclusions from existing knowledge using logical rules. +### **Ontology** +A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language). -**Ingestion** -: The process of loading data from various sources (files, databases, APIs, streams) into a system for processing. +### **Class** +In ontologies, a category or type of entity (e.g., `Person`, `Organization`, `Location`). + +### **Axiom** +A statement or rule that is accepted as true without proof, used in ontologies to define logical constraints and relationships. + +### **OWL (Web Ontology Language)** +A W3C standard language for defining and instantiating ontologies on the web. + +### **Property** +In ontologies, a relationship or attribute that connects entities or describes their characteristics. --- -## K +## Data Storage & Retrieval -**Knowledge Graph (KG)** -: A structured representation of knowledge using entities (nodes) and relationships (edges). KGs enable reasoning, querying, and semantic analysis of data. +### **Embedding** +A dense vector representation of text, images, or other data that captures semantic meaning in a continuous vector space. Used for similarity search and semantic matching. -**Knowledge Graph Analytics** -: The application of graph algorithms (e.g., centrality, community detection) to gain insights from the structure of a knowledge graph. +### **Vector Store** +A database optimized for storing and searching high-dimensional vectors, used for semantic similarity search. + +### **Triplet Store** +A database designed specifically for storing and querying RDF triplets. + +### **Graph Database** +A database designed specifically for storing and querying graph-structured data. + +### **Hybrid Search** +A search strategy that combines multiple retrieval methods, typically vector search and keyword search, to improve accuracy. --- -## L +## Graph Analytics -**LLM (Large Language Model)** -: A type of artificial intelligence model trained on vast amounts of text data, capable of understanding and generating human-like text. +### **Centrality** +A measure of the importance or influence of a node in a graph. Common centrality metrics include PageRank, betweenness centrality, and closeness centrality. + +### **PageRank** +An algorithm used to measure the importance of nodes in a graph based on the structure of incoming links. + +### **Community Detection** +The process of identifying groups or clusters of densely connected nodes in a graph. + +### **Graph Analytics** +The application of graph algorithms (e.g., centrality, community detection) to gain insights from the structure of a knowledge graph. --- -## N +## Query Languages -**Named Entity Recognition (NER)** -: The process of identifying and classifying named entities in text into predefined categories such as persons, organizations, locations, dates, and more. +### **Cypher** +A declarative query language for graph databases, particularly Neo4j. -**Node** -: A vertex in a graph representing an entity or concept. +### **SPARQL** +A query language for RDF data, similar to SQL for relational databases. -**Normalization** -: The process of standardizing data into a consistent format (e.g., converting dates to ISO format, standardizing entity names). +### **RDF (Resource Description Framework)** +A W3C standard for representing information about resources in the form of subject-predicate-object triplets. --- -## O +## Data Quality -**OCR (Optical Character Recognition)** -: Technology that converts images of text (e.g., scanned documents, photos) into machine-readable text. +### **Conflict Resolution** +The process of handling contradictory information from multiple sources in a knowledge graph. -**Ontology** -: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language). +### **Deduplication** +The process of identifying and removing duplicate records or entities from a dataset. -**OWL (Web Ontology Language)** -: A W3C standard language for defining and instantiating ontologies on the web. +### **Data Provenance** +Information about the origin, history, and lineage of data, including sources, timestamps, and transformations. --- -## P +## Technical Terms -**PageRank** -: An algorithm used to measure the importance of nodes in a graph based on the structure of incoming links. +### **API (Application Programming Interface)** +A set of functions and protocols that allow different software applications to communicate with each other. -**Pipeline** -: A sequence of data processing steps that transform raw data into a desired output format. +### **OCR (Optical Character Recognition)** +Technology that converts images of text (e.g., scanned documents, photos) into machine-readable text. -**Property** -: In ontologies, a relationship or attribute that connects entities or describes their characteristics. +### **Pipeline** +A sequence of data processing steps that transform raw data into a desired output format. -**Provenance** -: Information about the origin, history, and lineage of data, including sources, timestamps, and transformations. +### **Vector** +A mathematical representation of data as an array of numbers, used in embeddings to capture semantic meaning. + +### **Visualization** +The graphical representation of data, such as knowledge graphs, embeddings, or analytics. + +### **Web Scraping** +The automated process of extracting data from websites. --- -## R +## Semantica-Specific Terms -**RAG (Retrieval Augmented Generation)** -: A technique that enhances LLM responses by retrieving relevant information from a knowledge base before generating an answer. +### **Semantic Layer** +An abstraction layer that provides a unified, business-friendly view of data by adding context, relationships, and meaning to raw data. -**RDF (Resource Description Framework)** -: A W3C standard for representing information about resources in the form of subject-predicate-object triplets. +### **Semantic Network** +A knowledge representation that uses a graph structure to represent concepts and their relationships. -**Reasoning** -: The process of deriving new knowledge from existing facts using logical rules and inference. +### **Change Management** +The process of tracking and managing changes to knowledge graphs over time, including version control and audit trails. -**Relationship Extraction** -: The task of identifying and extracting semantic relationships between entities in text. - ---- - -## S - -**Semantic** -: Relating to meaning in language or logic. - -**Semantic Layer** -: An abstraction layer that provides a unified, business-friendly view of data by adding context, relationships, and meaning to raw data. - -**Semantic Network** -: A knowledge representation that uses a graph structure to represent concepts and their relationships. - -**SPARQL** -: A query language for RDF data, similar to SQL for relational databases. - ---- - -## T - -**Temporal Graph** -: A knowledge graph that tracks changes over time, allowing queries about the state of the graph at specific time points. - -**Triplet** -: A basic unit of knowledge in RDF, consisting of a subject, predicate, and object (e.g., ` `). - -**Triplet Store** -: A database designed specifically for storing and querying RDF triplets. - ---- - -## V - -**Vector** -: A mathematical representation of data as an array of numbers, used in embeddings to capture semantic meaning. - -**Vector Store** -: A database optimized for storing and searching high-dimensional vectors, used for semantic similarity search. - -**Visualization** -: The graphical representation of data, such as knowledge graphs, embeddings, or analytics. - ---- - -## W - -**Web Scraping** -: The automated process of extracting data from websites. +### **Provenance Tracking** +W3C PROV-O compliant tracking of data lineage and source attribution. --- ## See Also -- [Core Concepts](concepts.md) - Deep dive into fundamental concepts -- [Getting Started](getting-started.md) - Begin your journey with Semantica -- [API Reference](reference/core.md) - Technical documentation +- **[Core Concepts](concepts.md)** - Deep dive into fundamental concepts +- **[Getting Started](getting-started.md)** - Begin your journey with Semantica +- **[Modules Guide](modules.md)** - Complete module overview +- **[API Reference](reference/)** - Technical documentation + +--- + +## Need Help? + +- **Documentation**: [Getting Started](getting-started.md) +- **Examples**: [Cookbook](cookbook.md) +- **Community**: [Discord](community.md) +- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) +- **Support**: [Contact Us](community.md) diff --git a/docs/index.md b/docs/index.md index b6627573..4862faaa 100644 --- a/docs/index.md +++ b/docs/index.md @@ -5,19 +5,19 @@ Python 3.8+ License: MIT - PyPI version + PyPI version Monthly Downloads Total Downloads Documentation Discord -

Open Source Framework for Semantic Layer & Knowledge Engineering

+

Open-Source Semantic Layer & Knowledge Engineering Framework

-

Transform chaotic data into intelligent knowledge.

+

Transform Chaos into Intelligence. Build AI systems that are explainable, traceable, and trustworthy โ€” not black boxes.

-

The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge โ€” powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.

+

The semantic intelligence layer that makes your AI agents auditable, explainable, and trustworthy. Perfect for high-stakes domains where mistakes have real consequences.

-

๐Ÿ†“ 100% Open Source โ€ข ๐Ÿ“œ MIT Licensed โ€ข ๐Ÿš€ Latest Version: 0.2.3 โ€ข ๐Ÿš€ Production Ready โ€ข ๐ŸŒ Community Driven

+

๐Ÿ†“ Open Source โ€ข ๐Ÿ“œ MIT Licensed โ€ข ๐Ÿš€ Production Ready โ€ข ๐ŸŒ Community Driven

Get Started @@ -27,260 +27,203 @@ --- -## ๐ŸŒŸ What is Semantica? +## ๐Ÿš€ Why Semantica? -Semantica bridges the gap between raw data chaos and AI-ready knowledge. It's a **semantic intelligence platform** that transforms unstructured data into structured, queryable knowledge graphs powering GraphRAG, AI agents, and multi-agent systems. +**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and trustworthy. -### What Makes Semantica Different? - -Unlike traditional approaches that process isolated documents and extract text into vectors, Semantica understands **semantic relationships across all content**, provides **automated ontology generation**, and builds a **unified semantic layer** with **production-grade QA**. - -| **Traditional Approaches** | **Semantica's Approach** | -|:---------------------------|:-------------------------| -| Process data as isolated documents | **Understands semantic relationships across all content** | -| Extract text and store vectors | **Builds knowledge graphs with meaningful connections** | -| Generic entity recognition | **General-purpose ontology generation and validation** | -| Manual schema definition | **Automatic semantic modeling from content patterns** | -| Disconnected data silos | **Unified semantic layer across all data sources** | -| Basic quality checks | **Production-grade QA with conflict detection & resolution** | +Perfect for **high-stakes domains** where mistakes have real consequences. --- -## ๐ŸŽฏ The Problem We Solve +### โšก Get Started in 30 Seconds -### The Semantic Gap - -Organizations today face a **fundamental mismatch** between how data exists and how AI systems need it. - -#### The Semantic Gap: Problem vs. Solution - -Organizations have **unstructured data** (PDFs, emails, logs), **messy data** (inconsistent formats, duplicates, conflicts), and **disconnected silos** (no shared context, missing relationships). AI systems need **clear rules** (formal ontologies), **structured entities** (validated, consistent), and **relationships** (semantic connections, context-aware reasoning). - -| **What Organizations Have** | **What AI Systems Require** | -|:------------------------------|:------------------------------| -| **Unstructured Data** | **Clear Rules** | -| PDFs, emails, logs | Formal ontologies | -| Mixed schemas | Graphs & Networks | -| Conflicting facts | | -| **Messy, Noisy Data** | **Structured Entities** | -| Inconsistent formats | Validated entities | -| Duplicate records | Domain Knowledge | -| Missing relationships | | -| **Disconnected, Siloed Data** | **Relationships** | -| Data in separate systems | Semantic connections | -| No shared context | Context-Aware Reasoning | -| Isolated knowledge | | - -### What Happens Without Semantics? - -**They Break** โ€” Systems crash due to inconsistent formats and missing structure. - -**They Hallucinate** โ€” AI models generate false information without semantic context to validate outputs. - -**They Fail Silently** โ€” Systems return wrong answers without warnings, leading to bad decisions. - -**Why?** Systems have data โ€” not semantics. They can't connect concepts, understand relationships, validate against domain rules, or detect conflicts. - -### The Semantica Framework - -Semantica operates through three integrated layers that transform raw data into AI-ready knowledge: - -**Input Layer** โ€” Universal ingestion from multiple data formats (PDFs, DOCX, HTML, JSON, CSV, databases, live feeds, APIs, streams, archives, multi-modal content) into a unified pipeline. - -**Semantic Layer** โ€” Core intelligence engine performing entity extraction, relationship mapping, ontology generation, context engineering, and quality assurance. Includes **advanced entity deduplication** (Jaro-Winkler, disjoint property handling) to ensure a clean single source of truth. - -**Output Layer** โ€” Production-ready knowledge graphs, vector embeddings, and validated ontologies that power GraphRAG systems, AI agents, and multi-agent systems. - -**Powers: GraphRAG, AI Agents, Multi-Agent Systems** - -#### Semantica Processing Flow - -```mermaid -flowchart TD - A[Raw Data Sources
PDFs, Emails, Logs, Databases
Multiple Formats] --> B[Input Layer
Universal Data Ingestion] - B --> C[Format Detection
& Parsing] - C --> D[Normalization
& Preprocessing] - D --> E[Semantic Layer
Core Intelligence] - - E --> F[Entity Extraction
NER + LLM Enhancement] - E --> G[Relationship Mapping
Triplet Generation] - E --> H[Ontology Generation
6-Stage Pipeline] - E --> I[Context Engineering
Semantic Enrichment] - E --> J[Quality Assurance
Conflict Detection] - - F --> K[Output Layer] - G --> K - H --> K - I --> K - J --> K - - K --> L[Knowledge Graphs
Production-Ready] - K --> M[Vector Embeddings
Semantic Search] - K --> N[Ontologies
OWL Validated] - - L --> O[Application Layer] - M --> O - N --> O - - O --> P[GraphRAG Engine
91% Accuracy] - O --> Q[AI Agents
Persistent Memory] - O --> R[Multi-Agent Systems
Shared Models] - O --> S[Analytics & BI
Graph Insights] +```bash +pip install semantica ``` ---- +```python +from semantica.semantic_extract import NERExtractor +from semantica.kg import GraphBuilder -## ๐Ÿ’ก The Semantica Solution +# Extract entities and build knowledge graph +ner = NERExtractor(method="ml", model="en_core_web_sm") +entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") +kg = GraphBuilder().build({"entities": entities, "relationships": []}) -**Semantica** is an **open-source framework** that closes the semantic gap between real-world messy data and the structured semantic layers required by advanced AI systems โ€” GraphRAG, agents, multi-agent systems, reasoning models, and more. +print(f"Built KG with {len(kg.get('entities', []))} entities") +``` -### How Semantica Solves These Problems - -

- -- :material-lightning-bolt: **Efficient Embeddings** - --- - Uses **FastEmbed** by default for high-performance, lightweight local embedding generation (faster than sentence-transformers). - -- :material-database-import: **Universal Data Ingestion** - --- - Handles multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams) with unified pipeline, no custom parsers needed. - -- :material-brain: **Automated Semantic Extraction** - --- - NER, relationship extraction, and triplet generation with LLM enhancement discovers entities and relationships automatically. - -- :material-graph: **Knowledge Graph Construction** - --- - Production-ready graphs with entity resolution, temporal support, and graph analytics. Queryable knowledge ready for AI applications. - -- :material-robot: **GraphRAG Engine** - --- - Hybrid vector + graph retrieval achieves **91% accuracy** (30% improvement) via semantic search + graph traversal for multi-hop reasoning. - -- :material-account-cog: **AI Agent Context Engineering** - --- - Persistent memory with RAG + knowledge graphs enables context maintenance, action validation, and structured knowledge access. - -- :material-book-open-variant: **Automated Ontology Generation** - --- - 6-stage LLM pipeline generates validated OWL ontologies with HermiT/Pellet validation, eliminating manual engineering. - -- :material-shield-check: **Production-Grade QA** - --- - Conflict detection, deduplication, quality scoring, and provenance tracking ensure trusted, production-ready knowledge graphs. - -- :material-cog-transfer: **Pipeline Orchestration** - --- - Flexible pipeline builder with parallel execution enables scalable processing via orchestrator-worker pattern. - -
- -### Core Features at a Glance - -| **Feature Category** | **Capabilities** | **Key Benefits** | -|:---------------------|:-----------------|:------------------| -| **Data Ingestion** | Multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams, archives) | Universal ingestion, no custom parsers needed | -| **Semantic Extraction** | NER, relationship extraction, triplet generation, LLM enhancement | Automated discovery of entities and relationships | -| **Knowledge Graphs** | Entity resolution, temporal support, graph analytics, query interface | Production-ready, queryable knowledge structures | -| **Ontology Generation** | 6-stage LLM pipeline, OWL generation, HermiT/Pellet validation | Automated ontology creation from documents | -| **GraphRAG** | Hybrid vector + graph retrieval, multi-hop reasoning | 91% accuracy, 30% improvement over vector-only | -| **Agent Memory** | Persistent memory (Save/Load), Hybrid Retrieval (Vector+Graph), FastEmbed support | Context-aware agents with semantic understanding | -| **Pipeline Orchestration** | Parallel execution, custom steps, orchestrator-worker pattern | Scalable, flexible data processing | -| **Quality Assurance** | Conflict detection, deduplication, quality scoring, provenance | Trusted knowledge graphs ready for production | +**[๐Ÿ“– Full Quick Start](getting-started.md)** โ€ข **[๐Ÿณ Cookbook Examples](cookbook.md)** โ€ข **[๐Ÿ’ฌ Join Discord](https://discord.gg/ggb7vWeP)** โ€ข **[โญ Star Us](https://github.com/Hawksight-AI/semantica)** --- -## โœจ Core Capabilities +## Core Value Proposition -### 1. ๐Ÿ“Š Universal Data Ingestion +| **Trustworthy** | **Explainable** | **Auditable** | +|:------------------:|:------------------:|:-----------------:| +| Conflict detection & validation | Transparent reasoning paths | Complete provenance tracking | +| Rule-based governance | Entity relationships & ontologies | W3C PROV-O compliant lineage | +| Production-grade QA | Multi-hop graph reasoning | Source tracking & integrity verification | -Process **multiple file formats** with intelligent semantic extraction: +--- -
+## Key Features & Benefits -- __๐Ÿ“„ Documents__ - --- - - PDF (with OCR) - - DOCX, XLSX, PPTX - - TXT, RTF, ODT - - EPUB, LaTeX, Markdown +### Not Just Another Agentic Framework -- __๐ŸŒ Web & Feeds__ - --- - - HTML, XHTML, XML - - RSS, Atom feeds - - JSON-LD, RDFa - - Web scraping +**Semantica complements** LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK, Agno, and other frameworks to enhance your agents with: -- __๐Ÿ’พ Structured Data__ - --- - - JSON, YAML, TOML - - CSV, TSV, Excel - - Parquet, Avro, ORC - - SQL/NoSQL databases +| Feature | Benefit | +|:--------|:--------| +| **Auditable** | Complete provenance tracking with W3C PROV-O compliance | +| **Explainable** | Transparent reasoning paths with entity relationships | +| **Provenance-Aware** | End-to-end lineage from documents to responses | +| **Validated** | Built-in conflict detection, deduplication, QA | +| **Governed** | Rule-based validation and semantic consistency | +| **Version Control** | Enterprise-grade change management with integrity verification | -- __๐Ÿ“ง Communication__ - --- - - EML, MSG, MBOX - - PST archives - - Email threads - - Attachment extraction +### Perfect For High-Stakes Use Cases -- __๐Ÿ—œ๏ธ Archives__ - --- - - ZIP, TAR, RAR, 7Z - - Recursive processing - - Multi-level extraction +| ๐Ÿฅ **Healthcare** | ๐Ÿ’ฐ **Finance** | โš–๏ธ **Legal** | +|:-----------------:|:--------------:|:------------:| +| Clinical decisions | Fraud detection | Evidence-backed research | +| Drug interactions | Regulatory support | Contract analysis | +| Patient safety | Risk assessment | Case law reasoning | -- __๐Ÿ”ฌ Scientific__ - --- - - BibTeX, EndNote, RIS - - JATS XML - - PubMed formats - - Citation networks +| ๐Ÿ”’ **Cybersecurity** | ๐Ÿ›๏ธ **Government** | ๐Ÿญ **Infrastructure** | ๐Ÿš— **Autonomous** | +|:-------------------:|:----------------:|:-------------------:|:-----------------:| +| Threat attribution | Policy decisions | Power grids | Decision logs | +| Incident response | Classified info | Transportation | Safety validation | -
+### Powers Your AI Stack -### 2. ๐Ÿง  Semantic Intelligence Engine +- **GraphRAG Systems** โ€” Retrieval with graph reasoning and hybrid search +- **AI Agents** โ€” Trustworthy, accountable multi-agent systems with semantic memory +- **Reasoning Models** โ€” Explainable AI decisions with reasoning paths +- **Enterprise AI** โ€” Governed, auditable platforms that support compliance -Transform raw text into structured semantic knowledge with state-of-the-art NLP and AI models: +### Integrations -- **Named Entity Recognition (NER)**: Extract people, organizations, locations, dates, and custom entities -- **Relationship Extraction**: Identify semantic, temporal, and causal relationships -- **Event Detection**: Detect and classify events (acquisitions, partnerships, announcements) -- **Coreference Resolution**: Resolve pronouns and entity mentions across documents -- **Triplet Extraction**: Generate RDF triplets for knowledge graph construction +- **Docling Support** โ€” Document parsing with table extraction (PDF, DOCX, PPTX, XLSX) +- **AWS Neptune** โ€” Amazon Neptune graph database support with IAM authentication +- **Custom Ontology Import** โ€” Import existing ontologies (OWL, RDF, Turtle, JSON-LD) -### 3. ๐Ÿ•ธ๏ธ Knowledge Graph Construction +> **Built for environments where every answer must be explainable and governed.** -Build production-ready knowledge graphs with: +--- -- **Automatic Entity Resolution**: Merge duplicate entities with fuzzy matching -- **Conflict Detection & Resolution**: Handle contradictory information from multiple sources -- **Temporal Knowledge Graphs**: Track changes over time with version history -- **Graph Analytics**: Centrality, community detection, path finding -- **Multi-Format Export**: Neo4j, RDF, JSON-LD, GraphML +## ๐Ÿšจ The Problem: The Semantic Gap -### 4. ๐Ÿ“š Ontology Generation & Management +### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**. -Generate formal ontologies automatically using a **6-stage LLM-based pipeline**: +### Understanding the Semantic Gap -1. **Semantic Network Parsing** โ†’ Extract domain concepts -2. **YAML-to-Definition** โ†’ Transform into class definitions -3. **Definition-to-Types** โ†’ Map to OWL types -4. **Hierarchy Generation** โ†’ Build taxonomic structures -5. **TTL Generation** โ†’ Generate OWL/Turtle syntax -6. **Symbolic Validation** โ†’ HermiT/Pellet reasoning (F1 up to 0.99) +The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships). -### 5. ๐Ÿ” Hybrid Search & Retrieval +**Traditional AI approaches:** +- Rely on statistical patterns and text similarity +- Cannot understand relationships between entities +- Cannot reason about domain-specific rules +- Cannot explain why decisions were made +- Cannot trace back to original sources with confidence -Power GraphRAG applications with: +**High-stakes AI requires:** +- Semantic understanding of entities and their relationships +- Domain knowledge encoded as formal rules (ontologies) +- Explainable reasoning paths +- Source-level provenance +- Conflict detection and resolution -- **Vector Search**: Semantic similarity using embeddings -- **Graph Traversal**: Multi-hop reasoning for context expansion -- **Hybrid Retrieval**: Combine vector + graph for improved accuracy -- **Temporal Queries**: Query knowledge at specific time points +**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge. + +### What Organizations Have vs What They Need + +| **Current State** | **Required for High-Stakes AI** | +|:---------------------|:-----------------------------------| +| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) | +| APIs, databases, streams | Structured and validated entities | +| Conflicting facts and duplicates | Explicit semantic relationships | +| Siloed systems with no lineage | **Explainable reasoning paths** | +| | **Source-level provenance** | +| | **Audit-ready compliance** | + +### The Cost of Missing Semantics + +- **Decisions cannot be explained** โ€” No transparency in AI reasoning +- **Errors cannot be traced** โ€” No way to debug or improve +- **Conflicts go undetected** โ€” Contradictory information causes failures +- **Compliance becomes impossible** โ€” No audit trails for regulations + +**Trustworthy AI requires semantic accountability.** + +--- + +## ๐Ÿ†š Semantica vs Traditional RAG + +| Feature | Traditional RAG | Semantica | +|:--------|:----------------|:----------| +| **Reasoning** | โŒ Black-box answers | โœ… Explainable reasoning paths | +| **Provenance** | โŒ No provenance | โœ… W3C PROV-O compliant lineage tracking | +| **Search** | โš ๏ธ Vector similarity only | โœ… Semantic + graph reasoning | +| **Quality** | โŒ No conflict handling | โœ… Explicit contradiction detection | +| **Safety** | โš ๏ธ Unsafe for high-stakes | โœ… Designed for governed environments | +| **Compliance** | โŒ No audit trails | โœ… Complete audit trails with integrity verification | + +--- + +## ๐Ÿงฉ Semantica Architecture + +### 1๏ธโƒฃ Input Layer โ€” Governed Ingestion +- ๐Ÿ“„ **Multiple Formats** โ€” PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX +- ๐Ÿ”ง **Docling Support** โ€” Docling parser for table extraction +- ๐Ÿ’พ **Data Sources** โ€” Databases, APIs, streams, archives, web content +- ๐ŸŽจ **Media Support** โ€” Image parsing with OCR, audio/video metadata extraction +- ๏ฟฝ **Single Pipeline** โ€” Unified ingestion with metadata and source tracking + +### 2๏ธโƒฃ Semantic Layer โ€” Trust & Reasoning Engine +- ๐Ÿ” **Entity Extraction** โ€” NER, normalization, classification +- ๐Ÿ”— **Relationship Discovery** โ€” Triplet generation, semantic links +- ๐Ÿ“ **Ontology Induction** โ€” Automated domain rule generation +- ๐Ÿ”„ **Deduplication** โ€” Jaro-Winkler similarity, conflict resolution +- โœ… **Quality Assurance** โ€” Conflict detection, validation +- ๐Ÿ“Š **Provenance Tracking** โ€” W3C PROV-O compliant lineage tracking across all modules +- ๐Ÿง  **Reasoning Traces** โ€” Explainable inference paths +- ๐Ÿ” **Change Management** โ€” Version control with audit trails, checksums, compliance support + +### 3๏ธโƒฃ Output Layer โ€” Auditable Knowledge Assets +- ๏ฟฝ **Knowledge Graphs** โ€” Queryable, temporal, explainable +- ๐Ÿ“ **OWL Ontologies** โ€” HermiT/Pellet validated, custom ontology import support +- ๐Ÿ”ข **Vector Embeddings** โ€” FastEmbed by default +- โ˜๏ธ **AWS Neptune** โ€” Amazon Neptune graph database support +- ๐Ÿ” **Provenance** โ€” Every AI response links back to: + - ๐Ÿ“„ Source documents + - ๐Ÿท๏ธ Extracted entities & relations + - ๐Ÿ“ Ontology rules applied + - ๐Ÿง  Reasoning steps used + +--- + +## ๐Ÿฅ Built for High-Stakes Domains + +Designed for domains where **mistakes have real consequences** and **every decision must be accountable**: + +- **๐Ÿฅ Healthcare & Life Sciences** โ€” Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety tracking +- **๐Ÿ’ฐ Finance & Risk** โ€” Fraud detection, regulatory support (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation +- **โš–๏ธ Legal & Compliance** โ€” Evidence-backed legal research, contract analysis, regulatory change tracking, case law reasoning +- **๐Ÿ”’ Cybersecurity & Intelligence** โ€” Threat attribution, incident response, security audit trails, intelligence analysis +- **๐Ÿ›๏ธ Government & Defense** โ€” Governed AI systems, policy decisions, classified information handling, defense intelligence +- **๐Ÿญ Critical Infrastructure** โ€” Power grid management, transportation safety, water treatment, emergency response +- **๐Ÿš— Autonomous Systems** โ€” Self-driving vehicles, drone navigation, robotics safety, industrial automation + +--- + +## ๏ฟฝ Who Uses Semantica? + +- **๐Ÿค– AI / ML Engineers** โ€” Building explainable GraphRAG & agents +- **โš™๏ธ Data Engineers** โ€” Creating governed semantic pipelines +- **๐Ÿ“Š Knowledge Engineers** โ€” Managing ontologies & KGs at scale +- **๐Ÿข Enterprise Teams** โ€” Requiring trustworthy AI infrastructure +- **๐Ÿ›ก๏ธ Risk & Compliance Teams** โ€” Needing audit-ready systems --- @@ -420,7 +363,7 @@ print(f"Created graph with {len(kg.nodes)} nodes and {len(kg.edges)} edges")
-- **๐Ÿ†“ 100% Open Source** +- **๐Ÿ†“ Open Source** --- MIT licensed. No vendor lock-in. Full transparency. @@ -487,13 +430,3 @@ Get hands-on with interactive Jupyter notebooks: - **Difficulty**: Advanced - **Use Cases**: Building AI applications with knowledge graphs ---- - -
- -**Ready to transform your data into knowledge?** - -[Get Started Now](getting-started.md){ .md-button .md-button--primary } -[Join Discord](https://discord.gg/semantica){ .md-button } - -
diff --git a/docs/integrations/snowflake.md b/docs/integrations/snowflake.md new file mode 100644 index 00000000..8e624262 --- /dev/null +++ b/docs/integrations/snowflake.md @@ -0,0 +1,280 @@ +# Snowflake Integration + +Semantica features a native integration with **Snowflake**, the powerful cloud data warehouse that enables scalable data storage and analytics for enterprise workloads. + +## Overview + +Snowflake is integrated into Semantica's `ingest` module via the `SnowflakeIngestor`. This allows you to seamlessly extract structured data from Snowflake tables and queries into semantic structures that can be indexed, searched, and analyzed within the Semantica framework. + +- ๐Ÿ“– **Semantica Snowflake Integration Docs**: [Reference Guide](../reference/ingest.md) +- ๐Ÿ’ป **Semantica Snowflake Integration GitHub**: [Source Code](https://github.com/Hawksight-AI/semantica/blob/main/semantica/ingest/snowflake_ingestor.py) +- ๐Ÿง‘๐Ÿฝโ€๐Ÿณ **Semantica Snowflake Integration Example**: [Snowflake Clear Code Example](../CodeExamples.md#snowflake-clear-code-example) +- ๐Ÿ“ฆ **Semantica Snowflake Integration PyPI**: [Installation Guide](../installation.md) + +--- + +## ๐Ÿ“– Integration Documentation + +The `SnowflakeIngestor` provides a high-level interface for Snowflake data ingestion. It supports: + +* **Multiple Authentication Methods**: Password, key-pair, OAuth, and SSO authentication. +* **Advanced Querying**: Custom SQL queries with parameterization and batching. +* **Schema Introspection**: Automatic table schema discovery and metadata extraction. +* **Document Export**: Convert Snowflake data to Semantica document format. + +### Basic Usage + +```python +from semantica.ingest import SnowflakeIngestor + +# Initialize with environment variables +ingestor = SnowflakeIngestor() + +# Ingest a table +data = ingestor.ingest_table("CUSTOMERS") + +# Access the structured data +print(f"Retrieved {data.row_count} rows") +print(f"Columns: {data.columns}") +``` + +For more details, see the [Ingest Reference](../reference/ingest.md). + +--- + +## ๐Ÿง‘๐Ÿฝโ€๐Ÿณ Integration Example + +We provide a detailed cookbook and clear code examples to help you get started quickly. + +### Snowflake Clear Code Example + +```python +from semantica.ingest import SnowflakeIngestor +import os +from dotenv import load_dotenv + +# 1. Load environment variables +load_dotenv() + +# 2. Initialize the Snowflake Ingestor +ingestor = SnowflakeIngestor( + account=os.getenv("SNOWFLAKE_ACCOUNT"), + user=os.getenv("SNOWFLAKE_USER"), + password=os.getenv("SNOWFLAKE_PASSWORD"), + warehouse=os.getenv("SNOWFLAKE_WAREHOUSE"), + database=os.getenv("SNOWFLAKE_DATABASE"), + schema=os.getenv("SNOWFLAKE_SCHEMA") +) + +# 3. Ingest a table with filters +data = ingestor.ingest_table( + "CUSTOMERS", + where="COUNTRY = 'USA' AND CREATED_DATE > '2024-01-01'", + order_by="CREATED_DATE DESC", + limit=10000 +) + +# 4. Access the structured data +print(f"--- Customer Data ---") +print(f"Retrieved {data.row_count} customers") +print(f"Columns: {data.columns}") + +# 5. Iterate through rows +for row in data.data[:5]: # Print first 5 rows + print(f"Customer: {row['NAME']} ({row['EMAIL']})") + +# 6. Export as documents for Semantica processing +documents = ingestor.export_as_documents( + data, + id_field="CUSTOMER_ID", + text_fields=["NAME", "EMAIL", "NOTES"] +) + +print(f"Created {len(documents)} documents for processing") +``` + +See more in our [Code Examples](../CodeExamples.md). + +--- + +## ๐Ÿ’ป GitHub Source + +The integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues. + +- [snowflake_ingestor.py](https://github.com/Hawksight-AI/semantica/blob/main/semantica/ingest/snowflake_ingestor.py) - The core implementation of the Snowflake integration. + +--- + +## ๐Ÿ“ฆ PyPI & Installation + +Snowflake connector is an optional dependency for Semantica. You can install it along with Semantica or as a separate requirement. + +### Install via Semantica +```bash +# Install with Snowflake support +pip install semantica[db-snowflake] + +# Or install with all database connectors +pip install semantica[db-all] +``` + +### Install Snowflake connector manually +If you are working in a custom environment: +```bash +pip install snowflake-connector-python +``` + +For full installation details, see the [Installation Guide](../installation.md). + +--- + +## ๐Ÿ” Authentication Methods + +Snowflake integration supports multiple authentication methods for different security requirements: + +### Password Authentication +```python +ingestor = SnowflakeIngestor( + account="myaccount", + user="myuser", + password="mypassword", + warehouse="COMPUTE_WH" +) +``` + +### Key-Pair Authentication (Recommended for Production) +```python +ingestor = SnowflakeIngestor( + account="myaccount", + user="myuser", + private_key_path="/path/to/rsa_key.p8", + warehouse="COMPUTE_WH" +) +``` + +### OAuth Authentication +```python +ingestor = SnowflakeIngestor( + account="myaccount", + user="myuser", + authenticator="oauth", + token="your_oauth_token", + warehouse="COMPUTE_WH" +) +``` + +### SSO Authentication +```python +ingestor = SnowflakeIngestor( + account="myaccount", + user="myuser", + authenticator="externalbrowser", + warehouse="COMPUTE_WH" +) +``` + +--- + +## ๐Ÿš€ Advanced Features + +### Schema Introspection +```python +# Get table schema +schema = ingestor.get_table_schema("CUSTOMERS") +for column in schema["columns"]: + print(f"{column['name']}: {column['type']}") +``` + +### Custom Queries +```python +# Execute custom SQL +data = ingestor.ingest_query(""" + SELECT + CUSTOMER_ID, + SUM(AMOUNT) AS TOTAL_AMOUNT + FROM SALES + WHERE DATE >= '2024-01-01' + GROUP BY CUSTOMER_ID +""") +``` + +### Batch Processing +```python +# Handle large result sets +data = ingestor.ingest_query( + "SELECT * FROM LARGE_TABLE", + batch_size=5000 +) +``` + +--- + +## ๐Ÿ“Š Best Practices + +### Use Environment Variables +```python +import os +from dotenv import load_dotenv + +load_dotenv() +ingestor = SnowflakeIngestor() # Reads from environment +``` + +### Use Key-Pair Authentication for Production +```python +ingestor = SnowflakeIngestor( + account=os.getenv("SNOWFLAKE_ACCOUNT"), + user=os.getenv("SNOWFLAKE_USER"), + private_key_path=os.getenv("SNOWFLAKE_PRIVATE_KEY_PATH"), + warehouse="COMPUTE_WH" +) +``` + +### Paginate Large Results +```python +PAGE_SIZE = 10000 +for page in range(total_pages): + data = ingestor.ingest_table( + "LARGE_TABLE", + limit=PAGE_SIZE, + offset=page * PAGE_SIZE + ) + process_batch(data) +``` + +--- + +## ๐Ÿ” Troubleshooting + +### Connection Issues +```python +# Test connection +connector = SnowflakeConnector( + account="myaccount", + user="myuser", + password="mypassword" +) + +if not connector.test_connection(): + print("Connection failed - check credentials") +``` + +### Performance Optimization +```python +# Use appropriate warehouse size +ingestor = SnowflakeIngestor( + account="myaccount", + user="myuser", + password="mypassword", + warehouse="LARGE_WH" # For heavy workloads +) +``` + +--- + +## ๐Ÿ“š See Also + +- **[Ingest Module Reference](../reference/ingest.md)** - Complete ingestion documentation +- **[Getting Started Guide](../getting-started.md)** - Quick start with Semantica +- **[Code Examples](../CodeExamples.md)** - More integration examples +- **[Installation Guide](../installation.md)** - Installation instructions diff --git a/docs/integrations/snowflake_ingestion.md b/docs/integrations/snowflake_ingestion.md deleted file mode 100644 index a0ef1129..00000000 --- a/docs/integrations/snowflake_ingestion.md +++ /dev/null @@ -1,570 +0,0 @@ -# Snowflake Ingestion Guide - -This guide explains how to use the Snowflake ingestor to extract data from Snowflake data warehouses into Semantica. - -## Table of Contents - -- [Installation](#installation) -- [Authentication Methods](#authentication-methods) -- [Basic Usage](#basic-usage) -- [Advanced Features](#advanced-features) -- [Error Handling](#error-handling) -- [Best Practices](#best-practices) - -## Installation - -Install Semantica with Snowflake support: - -```bash -# Install with Snowflake support only -pip install semantica[db-snowflake] - -# Or install with all database connectors -pip install semantica[db-all] -``` - -This will install: -- `snowflake-connector-python>=3.0.0` -- `cryptography>=3.4.0` (required for key-pair authentication) - -## Authentication Methods - -### 1. Password Authentication (Default) - -The simplest authentication method using username and password: - -```python -from semantica.ingest import SnowflakeIngestor - -ingestor = SnowflakeIngestor( - account="myaccount", # Your Snowflake account identifier - user="myuser", - password="mypassword", - warehouse="COMPUTE_WH", - database="MYDB", - schema="PUBLIC" -) -``` - -**Environment Variables:** - -```bash -export SNOWFLAKE_ACCOUNT="myaccount" -export SNOWFLAKE_USER="myuser" -export SNOWFLAKE_PASSWORD="mypassword" -export SNOWFLAKE_WAREHOUSE="COMPUTE_WH" -export SNOWFLAKE_DATABASE="MYDB" -export SNOWFLAKE_SCHEMA="PUBLIC" -``` - -**Note:** For key-pair authentication, you must provide the `private_key_path` parameter directly in code. Environment variables are supported for account, user, warehouse, database, schema, role, authenticator, and token parameters. - -```python -# Now you can omit supported parameters - they'll be read from environment -ingestor = SnowflakeIngestor() -``` - -### 2. Key-Pair Authentication - -More secure authentication using RSA key pairs: - -**Generate Key Pair:** - -```bash -# Generate private key -openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt - -# Generate public key -openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub -``` - -**Add public key to Snowflake:** - -```sql -ALTER USER myuser SET RSA_PUBLIC_KEY='MIIBIjANBgkqh...'; -``` - -**Use in Python:** - -```python -from semantica.ingest import SnowflakeIngestor - -ingestor = SnowflakeIngestor( - account="myaccount", - user="myuser", - private_key_path="/path/to/rsa_key.p8", - warehouse="COMPUTE_WH", - database="MYDB" -) -``` - -**With encrypted private key:** - -```python -ingestor = SnowflakeIngestor( - account="myaccount", - user="myuser", - private_key_path="/path/to/rsa_key.p8", - private_key_passphrase="my_passphrase", - warehouse="COMPUTE_WH" -) -``` - -### 3. OAuth Authentication - -Use OAuth tokens for authentication: - -```python -ingestor = SnowflakeIngestor( - account="myaccount", - user="myuser", - authenticator="oauth", - token="your_oauth_token", - warehouse="COMPUTE_WH" -) -``` - -### 4. SSO Authentication (External Browser) - -For organizations using SSO: - -```python -ingestor = SnowflakeIngestor( - account="myaccount", - user="myuser", - authenticator="externalbrowser", - warehouse="COMPUTE_WH" -) -# This will open a browser for SSO login -``` - -## Basic Usage - -### Ingest a Table - -```python -from semantica.ingest import SnowflakeIngestor - -ingestor = SnowflakeIngestor( - account="myaccount", - user="myuser", - password="mypassword", - warehouse="COMPUTE_WH", - database="MYDB", - schema="PUBLIC" -) - -# Ingest entire table -data = ingestor.ingest_table("CUSTOMERS") - -print(f"Retrieved {data.row_count} rows") -print(f"Columns: {data.columns}") -print(f"First row: {data.data[0]}") -``` - -### Ingest with Filters - -```python -# With WHERE clause -data = ingestor.ingest_table( - "CUSTOMERS", - where="COUNTRY = 'USA' AND CREATED_DATE > '2024-01-01'" -) - -# With limit and offset for pagination -data = ingestor.ingest_table( - "CUSTOMERS", - limit=10000, - offset=0, - order_by="CREATED_DATE DESC" -) -``` - -### Execute Custom Queries - -```python -# Simple query -data = ingestor.ingest_query(""" - SELECT - CUSTOMER_ID, - SUM(AMOUNT) AS TOTAL_AMOUNT - FROM SALES - WHERE DATE >= '2024-01-01' - GROUP BY CUSTOMER_ID - HAVING SUM(AMOUNT) > 1000 -""") - -# Parameterized query -data = ingestor.ingest_query( - "SELECT * FROM SALES WHERE DATE > %(start_date)s AND REGION = %(region)s", - params={ - "start_date": "2024-01-01", - "region": "WEST" - } -) -``` - -### Large Result Sets with Batching - -```python -# Fetch in batches to manage memory -data = ingestor.ingest_query( - "SELECT * FROM LARGE_TABLE", - batch_size=5000 # Fetch 5000 rows at a time -) - -print(f"Total rows: {data.row_count}") -``` - -## Advanced Features - -### Schema Introspection - -```python -# Get table schema -schema = ingestor.get_table_schema("CUSTOMERS") - -for column in schema["columns"]: - print(f"{column['name']}: {column['type']} (nullable: {column['nullable']})") - -print(f"Primary keys: {schema['primary_keys']}") -``` - -### List All Tables - -```python -# List tables in current schema -tables = ingestor.list_tables() -print(f"Found {len(tables)} tables: {tables}") - -# List tables in specific database/schema -tables = ingestor.list_tables(database="OTHER_DB", schema="OTHER_SCHEMA") -``` - -### Export as Documents - -Convert Snowflake data to Semantica document format: - -```python -# Ingest data -data = ingestor.ingest_table("ARTICLES") - -# Convert to documents -documents = ingestor.export_as_documents( - data, - id_field="ARTICLE_ID", - text_fields=["TITLE", "CONTENT", "SUMMARY"] -) - -# Now use in Semantica pipeline -from semantica.pipeline import Pipeline - -pipeline = Pipeline() -for doc in documents: - pipeline.process_document(doc) -``` - -### Context Manager - -Use as a context manager for automatic connection cleanup: - -```python -with SnowflakeIngestor( - account="myaccount", - user="myuser", - password="mypassword" -) as ingestor: - data = ingestor.ingest_table("CUSTOMERS") - # Connection automatically closed on exit -``` - -### Multiple Schemas/Databases - -```python -# Ingest from different databases and schemas -data1 = ingestor.ingest_table( - "CUSTOMERS", - database="SALES_DB", - schema="PROD" -) - -data2 = ingestor.ingest_table( - "ORDERS", - database="SALES_DB", - schema="STAGING" -) -``` - -## Error Handling - -```python -from semantica.utils.exceptions import ProcessingError, ValidationError - -try: - ingestor = SnowflakeIngestor( - account="myaccount", - user="myuser", - password="wrong_password" - ) - data = ingestor.ingest_table("CUSTOMERS") - -except ValidationError as e: - print(f"Configuration error: {e}") - -except ProcessingError as e: - print(f"Processing failed: {e}") - -except Exception as e: - print(f"Unexpected error: {e}") -``` - -### Test Connection - -```python -# Test connection before using -connector = SnowflakeConnector( - account="myaccount", - user="myuser", - password="mypassword" -) - -if connector.test_connection(): - print("Connection successful!") -else: - print("Connection failed") -``` - -## Best Practices - -### 1. Use Environment Variables for Credentials - -**Don't:** -```python -# Hard-coded credentials (bad!) -ingestor = SnowflakeIngestor( - account="myaccount", - user="myuser", - password="mypassword123" -) -``` - -**Do:** -```python -# Use environment variables -import os -from dotenv import load_dotenv - -load_dotenv() - -ingestor = SnowflakeIngestor() # Reads from environment -``` - -### 2. Use Key-Pair Authentication for Production - -Key-pair authentication is more secure than passwords: - -```python -ingestor = SnowflakeIngestor( - account="myaccount", - user="myuser", - private_key_path=os.getenv("SNOWFLAKE_PRIVATE_KEY_PATH"), - warehouse="COMPUTE_WH" -) -``` - -### 3. Paginate Large Result Sets - -```python -# Don't fetch millions of rows at once -PAGE_SIZE = 10000 - -for page in range(total_pages): - data = ingestor.ingest_table( - "LARGE_TABLE", - limit=PAGE_SIZE, - offset=page * PAGE_SIZE - ) - process_batch(data) -``` - -### 4. Use Query Filters - -Push filtering to Snowflake instead of filtering in Python: - -**Don't:** -```python -# Fetching all rows and filtering in Python (inefficient) -all_data = ingestor.ingest_table("CUSTOMERS") -filtered = [row for row in all_data.data if row["COUNTRY"] == "USA"] -``` - -**Do:** -```python -# Filter in Snowflake (efficient) -data = ingestor.ingest_table("CUSTOMERS", where="COUNTRY = 'USA'") -``` - -### 5. Set Appropriate Warehouse Size - -```python -# Use appropriate warehouse for workload -ingestor = SnowflakeIngestor( - account="myaccount", - user="myuser", - password="mypassword", - warehouse="LARGE_WH" # For heavy workloads -) -``` - -### 6. Close Connections - -```python -# Always close connections when done -try: - ingestor = SnowflakeIngestor(...) - data = ingestor.ingest_table("CUSTOMERS") -finally: - ingestor.close() - -# Or use context manager -with SnowflakeIngestor(...) as ingestor: - data = ingestor.ingest_table("CUSTOMERS") -``` - -## Examples - -### Example 1: ETL Pipeline - -```python -from semantica.ingest import SnowflakeIngestor -from semantica.pipeline import Pipeline - -# Initialize ingestor -ingestor = SnowflakeIngestor() - -# Extract data -sales_data = ingestor.ingest_query(""" - SELECT - s.ORDER_ID, - s.CUSTOMER_ID, - c.CUSTOMER_NAME, - s.PRODUCT_ID, - p.PRODUCT_NAME, - s.AMOUNT, - s.ORDER_DATE - FROM SALES s - JOIN CUSTOMERS c ON s.CUSTOMER_ID = c.ID - JOIN PRODUCTS p ON s.PRODUCT_ID = p.ID - WHERE s.ORDER_DATE >= CURRENT_DATE - 30 -""") - -# Transform to documents -documents = ingestor.export_as_documents( - sales_data, - id_field="ORDER_ID", - text_fields=["CUSTOMER_NAME", "PRODUCT_NAME"] -) - -# Load into Semantica -pipeline = Pipeline() -pipeline.process_documents(documents) -``` - -### Example 2: Multi-Table Ingestion - -```python -from semantica.ingest import SnowflakeIngestor - -ingestor = SnowflakeIngestor() - -# Get all tables -tables = ingestor.list_tables() - -# Ingest each table -for table_name in tables: - print(f"Ingesting {table_name}...") - - # Get schema first - schema = ingestor.get_table_schema(table_name) - - # Ingest with limit - data = ingestor.ingest_table(table_name, limit=1000) - - # Process data - process_table(table_name, data, schema) -``` - -### Example 3: Incremental Loading - -```python -import datetime -from semantica.ingest import SnowflakeIngestor - -ingestor = SnowflakeIngestor() - -# Get last load timestamp -last_load = get_last_load_timestamp() # Your function - -# Ingest only new/updated records -data = ingestor.ingest_table( - "CUSTOMERS", - where=f"UPDATED_AT > '{last_load}'", - order_by="UPDATED_AT ASC" -) - -print(f"Loaded {data.row_count} new/updated records") - -# Update last load timestamp -set_last_load_timestamp(datetime.datetime.now()) -``` - -## Troubleshooting - -### Connection Issues - -```python -# Enable verbose logging -import logging -logging.basicConfig(level=logging.DEBUG) - -# Test connection -connector = SnowflakeConnector( - account="myaccount", - user="myuser", - password="mypassword" -) - -if not connector.test_connection(): - print("Connection failed - check credentials and network") -``` - -### Large Query Timeouts - -```python -# Use batching for large queries -data = ingestor.ingest_query( - "SELECT * FROM VERY_LARGE_TABLE", - batch_size=10000 # Fetch in batches -) -``` - -### Memory Issues with Large Results - -```python -# Process in chunks instead of loading all at once -CHUNK_SIZE = 10000 -offset = 0 - -while True: - chunk = ingestor.ingest_table( - "LARGE_TABLE", - limit=CHUNK_SIZE, - offset=offset - ) - - if chunk.row_count == 0: - break - - process_chunk(chunk) - offset += CHUNK_SIZE -``` diff --git a/docs/license.md b/docs/license.md index f6abe1e1..429a4a32 100644 --- a/docs/license.md +++ b/docs/license.md @@ -1,6 +1,6 @@ # License -Semantica is released under the MIT License. +**Semantica is open source under the MIT License.** --- @@ -34,45 +34,52 @@ SOFTWARE. ## What This Means -### You Can: -- โœ… Use commercially -- โœ… Modify the source code -- โœ… Distribute the software -- โœ… Use in private/proprietary projects -- โœ… Sublicense it +### โœ… You Can +- **Use commercially** - Free for business use +- **Modify** - Change the source code +- **Distribute** - Share with others +- **Sublicense** - Use in your own projects +- **Private use** - Use in proprietary software -### You Must: -- โœ… Include copyright notice -- โœ… Include license text +### โœ… You Must +- **Include copyright** - Keep the copyright notice +- **Include license** - Share the MIT license text -### You Cannot: -- โŒ Hold authors liable -- โŒ Use authors' names for endorsement +### โŒ No Warranty +- **No liability** - Authors not responsible for damages +- **No endorsement** - Can't use authors' names for promotion --- ## Commercial Use -**Semantica is free for commercial use.** No attribution required (though appreciated)! +**Semantica is completely free for commercial use.** No attribution required (though appreciated!). --- -## Third-Party Licenses +## Third-Party Dependencies -Key dependencies: -- Python (PSF), NumPy (BSD), Pandas (BSD) -- spaCy (MIT), Transformers (Apache 2.0), RDFLib (BSD) - -See `LICENSE` file for complete list. +Semantica uses open-source libraries with compatible licenses: +- **Python** (PSF License) +- **NumPy, Pandas** (BSD License) +- **spaCy** (MIT License) +- **Transformers** (Apache 2.0) +- **RDFLib** (BSD License) --- ## Contributing -By contributing, you agree your contributions will be licensed under MIT. +By contributing to Semantica, you agree that your contributions will be licensed under the same MIT License. --- -**Questions?** [Open an issue](https://github.com/Hawksight-AI/semantica/issues) +## Questions? -**Semantica is 100% open source and free!** ๐ŸŽ‰ +- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - License questions +- **[Contributing Guide](contributing.md)** - How to contribute +- **[Community](community.md)** - Get in touch + +--- + +**Semantica is open source and free for everyone!** ๐ŸŽ‰ diff --git a/docs/modules.md b/docs/modules.md index 4f676878..23c36801 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -1,1189 +1,740 @@ -# Modules & Architecture +๏ปฟ# Modules & Architecture -Semantica is built with a modular architecture, designed to be flexible, extensible, and scalable. This guide provides a comprehensive overview of all modules, their responsibilities, key features, and components. +**Complete guide to Semantica's modular architecture and how to use each component.** -!!! info "About This Guide" - This guide covers all 20+ core modules in Semantica, organized by their functional layer. Each module can be used independently or combined into powerful pipelines. +!!! tip "Modular Design" + Each Semantica module works independently. Use only what you need for your specific use case. --- -## Module Overview +## Architecture Overview -Semantica's modules are organized into six logical layers: +Semantica is organized into **six logical layers** - each with specific responsibilities: -| Layer | Modules | Description | -| :--- | :--- | :--- | -| **Input Layer** | [Ingest](#ingest-module), [Parse](#parse-module), [Split](#split-module), [Normalize](#normalize-module) | Data ingestion, parsing, chunking, and cleaning | -| **Core Processing** | [Semantic Extract](#semantic-extract-module), [Knowledge Graph](#knowledge-graph-kg-module), [Ontology](#ontology-module), [Reasoning](#reasoning-module) | Entity extraction, graph construction, inference | -| **Storage** | [Embeddings](#embeddings-module), [Vector Store](#vector-store-module), [Graph Store](#graph-store-module), [Triplet Store](#triplet-store-module) | Vector, graph, and triplet persistence | -| **Quality Assurance** | [Deduplication](#deduplication-module), [Conflicts](#conflicts-module) | Data quality and consistency | -| **Context & Memory** | [Context](#context-module), [Seed](#seed-module), [LLM Providers](#llm-providers-module) | Agent memory, foundation data, and LLM integration | -| **Output & Orchestration** | [Export](#export-module), [Visualization](#visualization-module), [Pipeline](#pipeline-module) | Export, visualization, and workflow management | +
+ +- **Input Layer** + + --- + + Data ingestion and preparation + + **Modules**: Ingest, Parse, Split, Normalize + +- **Core Processing** + + --- + + Intelligence and understanding + + **Modules**: Semantic Extract, Knowledge Graph, Ontology, Reasoning + +- **Storage** + + --- + + Persistent data storage + + **Modules**: Embeddings, Vector Store, Graph Store, Triplet Store + +- **Quality Assurance** + + --- + + Data quality and consistency + + **Modules**: Deduplication, Conflicts + +- **Context & Memory** + + --- + + Agent memory and foundation data + + **Modules**: Context, Seed, LLM Providers + +- **Output & Orchestration** + + --- + + Export, visualization, and workflows + + **Modules**: Export, Visualization, Pipeline + +
--- ## Input Layer -These modules handle data ingestion, parsing, chunking, and preparation. - ---- - ### Ingest Module +**Data ingestion from multiple sources** -!!! abstract "Purpose" - The entry point for data ingestion. Connects to various data sources including files, web, databases, and MCP servers. +```python +from semantica.ingest import FileIngestor, WebIngestor -**Key Features:** +# File ingestion +ingestor = FileIngestor() +documents = ingestor.ingest_directory("data/") -- Multiple file format support (PDF, DOCX, HTML, JSON, CSV, etc.) -- Web scraping with JavaScript rendering -- Database integration (SQL, NoSQL) -- Real-time streaming support -- MCP (Model Context Protocol) server integration -- Batch processing capabilities -- Metadata extraction and preservation +# Web ingestion +web_ingestor = WebIngestor() +pages = web_ingestor.ingest_urls(["https://example.com"]) +``` -**Components:** +**What it does:** +- **File formats** - PDF, DOCX, TXT, JSON, CSV +- **Web scraping** - Extract content from websites +- **Database** - Connect to SQL and NoSQL databases +- **Batch processing** - Handle large datasets efficiently -- `FileIngestor` โ€” Read files (PDF, DOCX, HTML, JSON, CSV, etc.) -- `WebIngestor` โ€” Scrape and ingest web pages -- `FeedIngestor` โ€” Process RSS/Atom feeds -- `StreamIngestor` โ€” Real-time data streaming -- `DBIngestor` โ€” Database queries and ingestion -- `EmailIngestor` โ€” Process email messages -- `RepoIngestor` โ€” Git repository analysis -- `MCPIngestor` โ€” Connect to MCP servers for resource and tool-based ingestion - -**Try It:** - -- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources - - **Topics**: File, web, feed, stream, database ingestion - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Loading data from various sources - -**API Reference**: [Ingest Module](reference/ingest.md) - ---- +**Use Cases:** +- Document processing pipelines +- Web data extraction +- Database integration +- Multi-source data collection ### Parse Module +**Document parsing and text extraction** -!!! abstract "Purpose" - Extracts raw text and metadata from ingested documents. Supports OCR, table extraction, and structured data parsing. +```python +from semantica.parse import DocumentParser -**Key Features:** +parser = DocumentParser() +parsed = parser.parse_document("document.pdf") +text = parsed["full_text"] +metadata = parsed["metadata"] +``` -- Multiple file format support -- OCR for images and scanned documents -- Table extraction from PDFs and spreadsheets -- Metadata preservation -- Automatic format detection -- Structured data parsing (JSON, CSV, XML) -- Code file parsing with syntax awareness +**What it does:** +- **Text extraction** - Extract clean text from documents +- **Metadata parsing** - Extract titles, authors, dates +- **Structure analysis** - Identify sections, headings +- **OCR support** - Handle scanned documents -**Components:** - -- `DocumentParser` โ€” Main parser orchestrator -- `PDFParser` โ€” Extract text, tables, images from PDFs -- `DOCXParser` โ€” Parse Word documents -- `HTMLParser` โ€” Extract content from HTML -- `JSONParser` โ€” Parse structured JSON data -- `ExcelParser` โ€” Process spreadsheets -- `ImageParser` โ€” OCR and image analysis -- `CodeParser` โ€” Parse source code files - -**Try It:** - -- **[Document Parsing Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)**: Learn to parse various document formats - - **Topics**: PDF, DOCX, HTML, JSON parsing, OCR, table extraction - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Extracting text from different file formats - -**API Reference**: [Parse Module](reference/parse.md) +**Use Cases:** +- PDF processing +- Document analysis +- Content extraction +- Metadata harvesting --- ### Split Module +**Text chunking and segmentation** -!!! abstract "Purpose" - Comprehensive document chunking and splitting for optimal processing. Provides 15+ splitting methods including KG-aware chunking. +```python +from semantica.split import TextSplitter -**Key Features:** +splitter = TextSplitter(method="semantic") +chunks = splitter.split(text, chunk_size=1000, overlap=200) +``` -- Multiple standard splitting methods (recursive, token, sentence, paragraph) -- Semantic-based chunking using NLP and embeddings -- Entity-aware chunking for GraphRAG workflows -- Relation-aware chunking for KG preservation -- Graph-based and ontology-aware chunking -- Hierarchical multi-level chunking -- Community detection-based splitting -- Sliding window chunking with overlap -- Table-specific chunking -- Chunk validation and quality assessment -- Provenance tracking for data lineage +**What it does:** +- **Intelligent chunking** - Split text while preserving context +- **Semantic splitting** - Break at natural boundaries +- **Size control** - Manage chunk sizes for processing +- **Overlap handling** - Maintain context between chunks -**Components:** - -- `TextSplitter` โ€” Unified text splitter with method parameter -- `SemanticChunker` โ€” Semantic-based chunking coordinator -- `StructuralChunker` โ€” Structure-aware chunking (headings, lists) -- `SlidingWindowChunker` โ€” Fixed-size sliding window chunking -- `TableChunker` โ€” Table-specific chunking -- `EntityAwareChunker` โ€” Entity boundary-preserving chunker -- `RelationAwareChunker` โ€” Triplet-preserving chunker -- `GraphBasedChunker` โ€” Graph structure-based chunker -- `OntologyAwareChunker` โ€” Ontology concept-based chunker -- `HierarchicalChunker` โ€” Multi-level hierarchical chunker -- `ChunkValidator` โ€” Chunk quality validation -- `ProvenanceTracker` โ€” Chunk provenance tracking - -**Supported Methods:** - -| Category | Methods | -| :--- | :--- | -| **Standard** | recursive, token, sentence, paragraph, character, word, semantic_transformer, llm | -| **KG/Ontology** | entity_aware, relation_aware, graph_based, ontology_aware, hierarchical, community_detection, centrality_based | - -**Try It:** - -- **[Text Splitting Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/04_Data_Normalization.ipynb)**: Learn different splitting methods - - **Topics**: Recursive, token, sentence splitting, entity-aware chunking - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Document chunking for processing +**Use Cases:** +- Document preprocessing +- Embedding preparation +- RAG systems +- Large document processing --- ### Normalize Module +**Data cleaning and standardization** -!!! abstract "Purpose" - Cleans, standardizes, and prepares text for semantic extraction. Handles encoding, entity names, dates, and numbers. +```python +from semantica.normalize import DataNormalizer -**Key Features:** +normalizer = DataNormalizer() +clean_text = normalizer.normalize_text(text) +standardized_date = normalizer.normalize_date("Jan 1st, 2020") +``` -- Text cleaning and noise removal -- Encoding normalization (Unicode handling) -- Entity name standardization -- Date and number formatting -- Language detection -- Whitespace normalization -- Special character handling +**What it does:** +- **Text cleaning** - Remove noise and artifacts +- **Date standardization** - Convert to ISO format +- **Name normalization** - Standardize person names +- **Entity normalization** - Clean up company names -**Components:** - -- `TextNormalizer` โ€” Main normalization orchestrator -- `TextCleaner` โ€” Remove noise, fix encoding -- `DataCleaner` โ€” Clean structured data -- `EntityNormalizer` โ€” Normalize entity names -- `DateNormalizer` โ€” Standardize date formats -- `NumberNormalizer` โ€” Normalize numeric values -- `LanguageDetector` โ€” Detect document language -- `EncodingHandler` โ€” Handle character encoding - -**Try It:** - -- **[Data Normalization Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/04_Data_Normalization.ipynb)**: Learn text normalization - - **Topics**: Text cleaning, encoding normalization, entity standardization - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Preparing text for processing - -**API Reference**: [Normalize Module](reference/normalize.md) +**Use Cases:** +- Data preprocessing +- Quality improvement +- Standardization +- Consistency enforcement --- -## Core Processing Layer - -These modules form the intelligence coreโ€”extracting meaning, building relationships, and inferring knowledge. - ---- +## Core Processing ### Semantic Extract Module +**Entity and relationship extraction** -!!! abstract "Purpose" - The brain of Semantica. Uses LLMs and NLP to extract entities, relationships, and semantic meaning from text. - -**Key Features:** - -- Multiple NER methods (rule-based, ML, LLM) -- Relationship extraction with confidence scoring -- Event extraction -- Custom entity type support -- Multi-language support -- Semantic network extraction -- Coreference resolution -- **Auto-chunking**: Automatic text splitting for long documents -- **Robust Error Handling**: Standardized LLM provider diagnostics and retry logic - -**Components:** - -- `NERExtractor` โ€” Named Entity Recognition -- `RelationExtractor` โ€” Extract relationships between entities -- `SemanticAnalyzer` โ€” Deep semantic analysis -- `SemanticNetworkExtractor` โ€” Extract semantic networks -- `EventExtractor` โ€” Extract events from text -- `CoreferenceResolver` โ€” Resolve entity coreferences - -**Try It:** - -- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn entity extraction - - **Topics**: Named entity recognition, entity types, extraction methods - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Understanding entity extraction - -- **[Relation Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)**: Learn relationship extraction - - **Topics**: Relationship extraction, dependency parsing, semantic role labeling - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Building rich knowledge graphs - -**Quick Example:** ```python from semantica.semantic_extract import NERExtractor, RelationExtractor -# Extract entities -extractor = NERExtractor(method="llm", model="gpt-4") -entities = extractor.extract(normalized_docs) +# Entity extraction +ner = NERExtractor() +entities = ner.extract("Apple Inc. was founded by Steve Jobs.") -# Extract relationships -relation_extractor = RelationExtractor() -relationships = relation_extractor.extract(normalized_docs, entities=entities) - -for rel in relationships[:5]: - print(f"{rel.subject.text} --[{rel.predicate}]--> {rel.object.text}") +# Relationship extraction +rel_extractor = RelationExtractor() +relationships = rel_extractor.extract(text, entities) ``` -**API Reference**: [Semantic Extract Module](reference/semantic_extract.md) +**What it does:** +- **Named Entity Recognition** - Find people, orgs, locations +- **Relationship extraction** - Find connections between entities +- **Custom entities** - Define your own entity types +- **Confidence scoring** - Quality assessment for extractions + +**Use Cases:** +- Knowledge graph construction +- Document analysis +- Information extraction +- Content understanding --- -### Knowledge Graph (KG) Module - -!!! abstract "Purpose" - Constructs and manages knowledge graphs from extracted entities and relationships. Supports multiple backends and advanced analytics. - -**Key Features:** - -- Graph construction from entities/relationships -- Multiple backend support (NetworkX, Neo4j) -- Temporal graph support -- Graph analytics and metrics -- Entity resolution and deduplication -- Community detection -- Centrality calculations -- Path finding algorithms -- Graph validation - -**Components:** - -- `GraphBuilder` โ€” Construct knowledge graphs -- `GraphAnalyzer` โ€” Analyze graph structure and properties -- `EntityResolver` โ€” Resolve entity conflicts and duplicates -- `ConflictDetector` โ€” Detect conflicting information -- `CentralityCalculator` โ€” Calculate node importance metrics -- `CommunityDetector` โ€” Detect community structure -- `ConnectivityAnalyzer` โ€” Analyze graph connectivity -- `SeedManager` โ€” Manage seed data for KG initialization -- `TemporalQuery` โ€” Query temporal knowledge graphs -- `Deduplicator` โ€” Remove duplicate entities/relationships - -**Quick Example:** +### Knowledge Graph Module +**Graph construction and management** ```python from semantica.kg import GraphBuilder, GraphAnalyzer # Build graph -builder = GraphBuilder(backend="networkx", temporal=True) -kg = builder.build(entities, relationships) +builder = GraphBuilder() +kg = builder.build({"entities": entities, "relationships": relationships}) # Analyze graph analyzer = GraphAnalyzer() -metrics = analyzer.analyze(kg) - -print(f"Nodes: {metrics['nodes']}, Edges: {metrics['edges']}") -print(f"Density: {metrics['density']:.3f}") +stats = analyzer.analyze(kg) ``` -**API Reference**: [Knowledge Graph Module](reference/kg.md) +**What it does:** +- **Graph construction** - Build knowledge graphs from data +- **Graph analysis** - Calculate metrics and statistics +- **Graph querying** - Search and retrieve information +- **Graph manipulation** - Merge, split, transform graphs + +**Use Cases:** +- Knowledge base creation +- Graph analytics +- Information retrieval +- Data integration --- ### Ontology Module - -!!! abstract "Purpose" - Defines schema and structure for your knowledge domain. Generates and validates ontologies with OWL/RDF export. - -**Key Features:** - -- Automatic ontology generation (6-stage pipeline) -- OWL/RDF/Turtle export -- Class and property inference -- Ontology validation -- Symbolic reasoning (HermiT, Pellet) -- Version management -- SHACL constraint support -- Ontology merging and alignment - -**Components:** - -- `OntologyGenerator` โ€” Generate ontologies from knowledge graphs -- `OntologyValidator` โ€” Validate ontologies -- `OWLGenerator` โ€” Generate OWL format ontologies -- `PropertyGenerator` โ€” Generate ontology properties -- `ClassInferrer` โ€” Infer ontology classes -- `OntologyMerger` โ€” Merge multiple ontologies -- `ReasonerInterface` โ€” Interface with symbolic reasoners - -**Quick Example:** +**Schema definition and validation** ```python -from semantica.ontology import OntologyEngine +from semantica.ontology import OntologyManager -# Initialize engine -engine = OntologyEngine(base_uri="https://example.org/ontology/") +# Define ontology +ontology = OntologyManager() +ontology.add_class("Person", ["name", "birth_date"]) +ontology.add_relationship("works_for", "Person", "Organization") -# Generate ontology from data -ontology = engine.from_data({ - "entities": [...], - "relationships": [...] -}) - -# Validate ontology -result = engine.validate(ontology) -if result.valid: - print("Ontology is valid!") - -# Export to OWL -owl_content = engine.to_owl(ontology, format="turtle") -print(f"Generated {len(owl_content)} lines of OWL") +# Validate data +is_valid = ontology.validate_graph(kg) ``` -**API Reference**: [Ontology Module](reference/ontology.md) +**What it does:** +- **Schema definition** - Define data structure +- **Data validation** - Ensure data conforms to schema +- **Inheritance** - Create hierarchical relationships +- **Constraints** - Enforce data quality rules + +**Use Cases:** +- Data modeling +- Quality assurance +- Schema management +- Rule enforcement --- ### Reasoning Module - -!!! abstract "Purpose" - Infers new facts and validates existing knowledge using logical rules. Supports forward-chaining, high-performance pattern matching, and explanation generation. - -**Key Features:** - -- Forward-chaining inference engine -- IF-THEN rule support with variable substitution -- High-performance Rete algorithm for large-scale rule matching -- Natural language explanation generation for inferred facts -- SPARQL query expansion for RDF graphs -- Conflict detection in inferences -- Priority-based rule execution - -**Components:** - -- `Reasoner` โ€” High-level facade for all reasoning tasks -- `ReteEngine` โ€” High-performance pattern matching (Rete algorithm) -- `ExplanationGenerator` โ€” Generate justifications for inferred facts -- `SPARQLReasoner` โ€” Query expansion for triplet stores - -**Quick Example:** +**Logical inference and deduction** ```python -from semantica.reasoning import Reasoner +from semantica.reasoning import ReasoningEngine -reasoner = Reasoner() - -# Add rules and facts -reasoner.add_rule("IF Person(?x) AND Parent(?x, ?y) THEN ParentOfPerson(?x, ?y)") -reasoner.add_fact("Person(Alice)") -reasoner.add_fact("Parent(Alice, Bob)") - -# Perform inference -inferred = reasoner.infer_facts(["Person(Alice)", "Parent(Alice, Bob)"]) -# Inferred: ["ParentOfPerson(Alice, Bob)"] +engine = ReasoningEngine() +inferences = engine.infer(kg, rules=["transitivity", "symmetry"]) ``` -**API Reference**: [Reasoning Module](reference/reasoning.md) +**What it does:** +- **Logical inference** - Derive new facts from existing ones +- **Pattern matching** - Find complex patterns in data +- **Consistency checking** - Detect contradictions +- **Decision support** - Automated reasoning + +**Use Cases:** +- Knowledge discovery +- Decision making +- Consistency checking +- Advanced analytics --- ## Storage Layer -These modules handle persistence and retrieval of vectors, graphs, and triplets. - ---- - ### Embeddings Module - -!!! abstract "Purpose" - Generates vector embeddings for text, images, and audio. Supports multiple providers with caching and batch processing. - -**Key Features:** - -- Multiple provider support (OpenAI, Cohere, HuggingFace, Sentence Transformers) -- Text, image, and audio embeddings -- Multimodal embeddings -- Batch processing -- Caching support -- Custom models -- Similarity calculations - -**Components:** - -- `EmbeddingGenerator` โ€” Main embedding orchestrator -- `TextEmbedder` โ€” Generate text embeddings -- `ImageEmbedder` โ€” Generate image embeddings -- `AudioEmbedder` โ€” Generate audio embeddings -- `MultimodalEmbedder` โ€” Combine multiple modalities -- `EmbeddingOptimizer` โ€” Optimize embedding quality -- `ProviderStores` โ€” Support for OpenAI, Cohere, etc. - -**Quick Example:** +**Vector embeddings and similarity** ```python from semantica.embeddings import EmbeddingGenerator -generator = EmbeddingGenerator( - provider="openai", - model="text-embedding-3-small" -) -embeddings = generator.generate(documents) - -# Calculate similarity +generator = EmbeddingGenerator(model="sentence-transformers") +embeddings = generator.generate(["text1", "text2"]) similarity = generator.similarity(embeddings[0], embeddings[1]) -print(f"Similarity: {similarity:.3f}") ``` -**API Reference**: [Embeddings Module](reference/embeddings.md) +**What it does:** +- **Text embeddings** - Convert text to vectors +- **Similarity search** - Find similar content +- **Clustering** - Group related items +- **AI integration** - Provide context to LLMs + +**Use Cases:** +- Semantic search +- Recommendation systems +- Clustering +- AI context --- ### Vector Store Module - -!!! abstract "Purpose" - Manages storage and retrieval of high-dimensional vectors. Supports hybrid search combining vector and keyword search. - -**Key Features:** - -- Multiple backend support (FAISS, Weaviate, Qdrant, Milvus) -- Hybrid search (vector + keyword) -- Metadata filtering -- Batch operations -- Similarity search with scoring -- Index management -- Namespace support - -**Components:** - -- `VectorStore` โ€” Main vector store interface -- `FAISSStore` โ€” FAISS integration -- `WeaviateStore` โ€” Weaviate integration -- `HybridSearch` โ€” Combine vector and keyword search -- `VectorRetriever` โ€” Retrieve relevant vectors - -**Quick Example:** +**Vector database management** ```python -from semantica.vector_store import VectorStore, HybridSearch +from semantica.vector_store import VectorStore -vector_store = VectorStore(backend="faiss") -vector_store.store(embeddings, documents, metadata) - -# Hybrid search -hybrid_search = HybridSearch(vector_store) -results = hybrid_search.search( - query="machine learning", - top_k=10, - filters={"category": "AI"} -) +store = VectorStore(backend="faiss") +store.add_vectors(embeddings, ids) +results = store.search(query_vector, top_k=10) ``` -**API Reference**: [Vector Store Module](reference/vector_store.md) +**What it does:** +- **Vector storage** - Efficient vector database +- **Fast search** - Approximate nearest neighbor search +- **Indexing** - Optimize for performance +- **Batch operations** - Handle large datasets + +**Use Cases:** +- Semantic search +- RAG systems +- Recommendation engines +- Similarity matching --- ### Graph Store Module - -!!! abstract "Purpose" - Integration with property graph databases for storing and querying knowledge graphs. - -**Key Features:** - -- Multiple backend support (Neo4j, FalkorDB) -- Cypher query language -- Graph algorithms and analytics -- Transaction support -- Index management -- High-performance queries -- Batch operations - -**Components:** - -- `GraphStore` โ€” Main graph store interface -- `Neo4jStore` โ€” Neo4j database integration -- `FalkorDBStore` โ€” FalkorDB (Redis-based) integration -- `NodeManager` โ€” Node CRUD operations -- `RelationshipManager` โ€” Relationship CRUD operations -- `QueryEngine` โ€” Cypher query execution -- `GraphAnalytics` โ€” Graph algorithms and analytics - -**Quick Example:** +**Graph database integration** ```python from semantica.graph_store import GraphStore -store = GraphStore(backend="neo4j", uri="bolt://localhost:7687") -store.connect() - -# Create nodes and relationships -alice = store.create_node( - labels=["Person"], - properties={"name": "Alice", "age": 30} -) -bob = store.create_node( - labels=["Person"], - properties={"name": "Bob", "age": 25} -) -store.create_relationship( - start_node_id=alice["id"], - end_node_id=bob["id"], - rel_type="KNOWS", - properties={"since": 2020} -) - -# Query with Cypher -results = store.execute_query("MATCH (p:Person) RETURN p.name") +store = GraphStore(backend="neo4j") +store.add_nodes(entities) +store.add_edges(relationships) +results = store.query("MATCH (n)-[r]->(m) RETURN n, r, m") ``` -**API Reference**: [Graph Store Module](reference/graph_store.md) +**What it does:** +- **Graph persistence** - Store graphs in databases +- **Graph queries** - Cypher and Gremlin support +- **Graph algorithms** - Path finding, centrality +- **Transactions** - ACID compliance + +**Use Cases:** +- Knowledge graph storage +- Graph analytics +- Network analysis +- Relationship queries --- ### Triplet Store Module - -!!! abstract "Purpose" - RDF triplet store integration for semantic web applications. Supports SPARQL queries and multiple backends. - -**Key Features:** - -- Multi-backend support (Blazegraph, Jena, RDF4J) -- CRUD operations for RDF triplets -- SPARQL query execution and optimization -- Bulk data loading with progress tracking -- Query caching and optimization -- Transaction support -- Store backend pattern - -**Components:** - -- `TripletStore` โ€” Main triplet store interface -- `QueryEngine` โ€” SPARQL query execution and optimization -- `BulkLoader` โ€” High-volume data loading with progress tracking -- `BlazegraphStore` โ€” Blazegraph integration -- `JenaStore` โ€” Apache Jena integration -- `RDF4JStore` โ€” Eclipse RDF4J integration -- `QueryPlan` โ€” Query execution plan dataclass -- `LoadProgress` โ€” Bulk loading progress tracking - -**Algorithms:** - -| Category | Algorithms | -| :--- | :--- | -| **Query Optimization** | Cost estimation, query rewriting, LIMIT injection | -| **Caching** | MD5-based cache keys, LRU eviction | -| **Bulk Loading** | Batch processing, retry with exponential backoff | - -**Quick Example:** +**Triple-based storage** ```python from semantica.triplet_store import TripletStore -store = TripletStore(backend="blazegraph", endpoint="http://localhost:9999/blazegraph") - -# Add triplet -result = store.add_triplet({ - "subject": "http://example.org/Alice", - "predicate": "http://example.org/knows", - "object": "http://example.org/Bob" -}) - -# Execute SPARQL -query_result = store.execute_query("SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10") +store = TripletStore() +store.add_triplets(subject, predicate, object) +triplets = store.get_triplets(entity="Apple Inc.") ``` -**API Reference**: [Triplet Store Module](reference/triplet_store.md) +**What it does:** +- **Triple storage** - Store (subject, predicate, object) triples +- **Pattern matching** - Find specific patterns +- **RDF support** - Semantic web standards +- **Bulk operations** - Efficient batch processing + +**Use Cases:** +- Semantic web +- Knowledge representation +- Linked data +- Triple stores --- -## Quality Assurance Layer - -These modules ensure data quality, handle duplicates, and resolve conflicts. - ---- +## Quality Assurance ### Deduplication Module - -!!! abstract "Purpose" - Comprehensive entity deduplication and merging. Detects duplicates using multiple similarity methods and merges them intelligently. - -**Key Features:** - -- Multiple similarity methods (exact, Levenshtein, Jaro-Winkler, cosine, embedding) -- **Advanced String Matching**: Jaro-Winkler by default for better company/person name resolution -- **Smart Property Handling**: Neutral scoring for disjoint properties to prevent false negatives -- Duplicate detection with confidence scoring -- Entity merging with configurable strategies -- Cluster-based batch deduplication -- Provenance preservation during merges -- Relationship preservation -- Incremental processing support - -**Components:** - -- `DuplicateDetector` โ€” Detects duplicate entities using similarity metrics -- `EntityMerger` โ€” Merges duplicate entities using configurable strategies -- `SimilarityCalculator` โ€” Multi-factor similarity between entities -- `MergeStrategyManager` โ€” Manages merge strategies and conflict resolution -- `ClusterBuilder` โ€” Builds clusters for batch deduplication - -**Merge Strategies:** - -| Strategy | Description | -| :--- | :--- | -| `"keep_first"` | Preserve first entity, merge others | -| `"keep_last"` | Preserve last entity, merge others | -| `"keep_most_complete"` | Preserve entity with most properties | -| `"keep_highest_confidence"` | Preserve entity with highest confidence | -| `"merge_all"` | Combine all properties and relationships | - -**Quick Example:** +**Entity deduplication and resolution** ```python -from semantica.deduplication import DuplicateDetector, EntityMerger +from semantica.deduplication import EntityResolver -# Detect duplicates -detector = DuplicateDetector(similarity_threshold=0.8) -duplicate_groups = detector.detect_duplicate_groups(entities) - -# Merge duplicates -merger = EntityMerger(preserve_provenance=True) -merge_operations = merger.merge_duplicates( - entities, - strategy="keep_most_complete" -) - -merged_entities = [op.merged_entity for op in merge_operations] -print(f"Reduced from {len(entities)} to {len(merged_entities)} entities") +resolver = EntityResolver() +merged_entities = resolver.resolve(entities, strategy="semantic") ``` +**What it does:** +- **Duplicate detection** - Find similar entities +- **Entity resolution** - Merge duplicate records +- **Similarity scoring** - Quality assessment +- **Record linkage** - Connect related records + +**Use Cases:** +- Data cleaning +- Master data management +- Record linkage +- Quality improvement + --- ### Conflicts Module - -!!! abstract "Purpose" - Detects and resolves conflicts from multiple data sources. Provides investigation guides and source tracking. - -**Key Features:** - -- Multi-source conflict detection (value, type, relationship, temporal, logical) -- Source tracking and provenance management -- Conflict analysis and pattern identification -- Multiple resolution strategies (voting, credibility-weighted, recency) -- Investigation guide generation -- Source credibility scoring -- Conflict reporting and statistics - -**Components:** - -- `ConflictDetector` โ€” Detects conflicts from multiple sources -- `ConflictResolver` โ€” Resolves conflicts using various strategies -- `ConflictAnalyzer` โ€” Analyzes conflict patterns and trends -- `SourceTracker` โ€” Tracks source information and provenance -- `InvestigationGuideGenerator` โ€” Generates investigation guides - -**Resolution Strategies:** - -| Strategy | Algorithm | -| :--- | :--- | -| **Voting** | Majority value selection using frequency counting | -| **Credibility Weighted** | Weighted average using source credibility scores | -| **Temporal Selection** | Newest/oldest value based on timestamps | -| **Confidence Selection** | Maximum confidence value selection | - -**Quick Example:** +**Conflict detection and resolution** ```python -from semantica.conflicts import ConflictDetector, ConflictResolver +from semantica.conflicts import ConflictDetector detector = ConflictDetector() -conflicts = detector.detect_value_conflicts(entities, "name") - -resolver = ConflictResolver() -results = resolver.resolve_conflicts(conflicts, strategy="voting") +conflicts = detector.detect_conflicts(kg) +resolved = detector.resolve(conflicts, strategy="most_recent") ``` ---- +**What it does:** +- **Conflict detection** - Find contradictory information +- **Resolution strategies** - Automated conflict resolution +- **Source reliability** - Trustworthiness assessment +- **Temporal analysis** - Time-based conflict handling -### KG Quality Assurance Module - -!!! abstract "Purpose" - Comprehensive quality assessment, validation, and automated fixes for knowledge graphs. - -**Key Features:** - -- Quality metrics calculation (overall, completeness, consistency) -- Consistency checking (logical, temporal, hierarchical) -- Completeness validation (entity, relationship, property) -- Automated fixes (duplicates, inconsistencies, missing properties) -- Quality reporting with issue tracking -- Validation engine with rules and constraints -- Improvement suggestions - -**Components:** - -- `KGQualityAssessor` โ€” Overall quality assessment coordinator -- `ConsistencyChecker` โ€” Consistency validation engine -- `CompletenessValidator` โ€” Completeness validation engine -- `QualityMetrics` โ€” Quality metrics calculator -- `ValidationEngine` โ€” Rule and constraint validation -- `RuleValidator` โ€” Rule-based validation -- `ConstraintValidator` โ€” Constraint-based validation -- `QualityReporter` โ€” Quality report generation -- `IssueTracker` โ€” Issue tracking and management -- `ImprovementSuggestions` โ€” Improvement suggestions generator -- `AutomatedFixer` โ€” Automated issue fixing -- `AutoMerger` โ€” Automatic merging of duplicates -- `AutoResolver` โ€” Automatic conflict resolution - -Note: The KG quality assessment module has been temporarily removed and will be reintroduced in a future release. +**Use Cases:** +- Data quality +- Consistency checking +- Trust management +- Conflict resolution --- -## Context & Memory Layer - -These modules provide context engineering for agents and foundation data management. - ---- +## Context & Memory ### Context Module - -!!! abstract "Purpose" - Context engineering infrastructure for agents. Formalizes context as a graph of connections with RAG-enhanced memory. Features GraphRAG with multi-hop reasoning and LLM-generated responses. - -**Key Features:** - -- Context graph construction from entities, relationships, and conversations -- Agent memory management with RAG integration -- Entity linking across sources with URI assignment -- Hybrid context retrieval (vector + graph + memory) -- **Multi-hop reasoning** through knowledge graphs -- **LLM-generated responses** grounded in graph context -- **Reasoning trace** showing entity relationship paths -- Conversation history management -- Context accumulation and synthesis -- Graph-based context traversal - -**Components:** - -- `ContextGraph` โ€” In-memory context graph store and builder methods -- `ContextNode` โ€” Context graph node data structure -- `ContextEdge` โ€” Context graph edge data structure -- `AgentMemory` โ€” Manages persistent agent memory with RAG -- `AgentContext` โ€” High-level context interface with GraphRAG capabilities -- `ContextRetriever` โ€” Retrieves relevant context with multi-hop reasoning -- `MemoryItem` โ€” Memory item data structure -- `EntityLinker` โ€” Links entities across sources with URI assignment - -**Algorithms:** - -| Category | Algorithms | -| :--- | :--- | -| **Graph Construction** | BFS/DFS traversal, type-based indexing | -| **Memory Management** | Vector embedding, similarity search, retention policies | -| **Context Retrieval** | Vector similarity, multi-hop graph expansion, hybrid scoring | -| **Multi-Hop Reasoning** | BFS traversal up to N hops, reasoning path construction | -| **LLM Integration** | Prompt engineering with context and reasoning paths | -| **Entity Linking** | Hash-based URI generation, text similarity matching | - -**Quick Example:** +**Context management for AI agents** ```python -from semantica.context import AgentContext, ContextGraph, AgentMemory -from semantica.llms import Groq -from semantica.vector_store import VectorStore -import os +from semantica.context import ContextManager -# Using AgentContext with GraphRAG reasoning -context = AgentContext( - vector_store=VectorStore(backend="faiss"), - knowledge_graph=kg -) - -# Configure LLM provider -llm_provider = Groq( - model="llama-3.1-8b-instant", - api_key=os.getenv("GROQ_API_KEY") -) - -# Query with multi-hop reasoning and LLM-generated response -result = context.query_with_reasoning( - query="What IPs are associated with security alerts?", - llm_provider=llm_provider, - max_results=10, - max_hops=2 -) - -print(f"Response: {result['response']}") -print(f"Reasoning Path: {result['reasoning_path']}") -print(f"Confidence: {result['confidence']:.3f}") - -# Traditional context graph and memory -graph = ContextGraph() -graph_data = graph.build_from_entities_and_relationships(entities, relationships) - -memory = AgentMemory(vector_store=vs, knowledge_graph=kg) -memory_id = memory.store("User asked about Python", metadata={"type": "conversation"}) -results = memory.retrieve("Python", max_results=5) +manager = ContextManager() +context = manager.get_context(query, history) ``` -**API Reference**: [Context Module](reference/context.md) +**What it does:** +- **Context tracking** - Maintain conversation context +- **Memory management** - Store and retrieve context +- **Relevance scoring** - Find relevant context +- **Session management** - Handle multiple conversations ---- - -### LLM Providers Module - -!!! abstract "Purpose" - Unified interface for LLM providers. Supports Groq, OpenAI, HuggingFace, and LiteLLM (100+ LLMs) with clean imports and consistent API. - -**Key Features:** - -- **Unified Interface**: Same `generate()` and `generate_structured()` methods across all providers -- **Multiple Providers**: Groq, OpenAI, HuggingFace, and LiteLLM (100+ LLMs) -- **Clean Imports**: Simple `from semantica.llms import Groq, OpenAI, HuggingFaceLLM, LiteLLM` -- **Structured Output**: JSON generation support -- **API Key Management**: Environment variable and direct key support -- **Error Handling**: Graceful fallback when providers unavailable - -**Components:** - -- `Groq` โ€” Groq API provider for fast inference -- `OpenAI` โ€” OpenAI API provider (GPT-3.5, GPT-4, etc.) -- `HuggingFaceLLM` โ€” HuggingFace Transformers for local LLM inference -- `LiteLLM` โ€” Unified interface to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, etc.) - -**Supported Providers via LiteLLM:** - -- OpenAI, Anthropic, Groq, Azure, Bedrock, Vertex AI, Cohere, Mistral, and 90+ more - -**Quick Example:** - -```python -from semantica.llms import Groq, OpenAI, HuggingFaceLLM, LiteLLM -import os - -# Groq - Fast inference -groq = Groq( - model="llama-3.1-8b-instant", - api_key=os.getenv("GROQ_API_KEY") -) -response = groq.generate("What is AI?") - -# OpenAI -openai = OpenAI( - model="gpt-4", - api_key=os.getenv("OPENAI_API_KEY") -) -response = openai.generate("What is AI?") - -# HuggingFace - Local models -hf = HuggingFaceLLM(model_name="gpt2") # or model="gpt2" for consistency -response = hf.generate("What is AI?") - -# LiteLLM - Unified interface to 100+ LLMs -litellm = LiteLLM( - model="openai/gpt-4o", # or "anthropic/claude-sonnet-4-20250514", etc. - api_key=os.getenv("OPENAI_API_KEY") -) -response = litellm.generate("What is AI?") - -# Structured output -structured = groq.generate_structured("Extract entities from: Apple Inc. was founded by Steve Jobs.") -``` - -**API Reference**: [LLM Providers Module](reference/llms.md) +**Use Cases:** +- AI agents +- Chatbots +- Conversational AI +- Context-aware systems --- ### Seed Module - -!!! abstract "Purpose" - Seed data management for initial knowledge graph construction. Builds on verified knowledge from multiple sources. - -**Key Features:** - -- Multi-source seed data loading (CSV, JSON, Database, API) -- Foundation graph creation from seed data -- Seed data quality validation -- Integration with extracted data using configurable merge strategies -- Version management for seed sources -- Export capabilities (JSON, CSV) -- Schema template validation - -**Components:** - -- `SeedDataManager` โ€” Main coordinator for seed data operations -- `SeedDataSource` โ€” Seed data source definition -- `SeedData` โ€” Seed data container - -**Merge Strategies:** - -| Strategy | Description | -| :--- | :--- | -| `seed_first` | Seed data takes precedence, extracted fills gaps | -| `extracted_first` | Extracted data takes precedence, seed fills gaps | -| `merge` | Property merging, seed takes precedence for conflicts | - -**Quick Example:** +**Foundation data and knowledge** ```python -from semantica.seed import SeedDataManager +from semantica.seed import SeedData -manager = SeedDataManager() -manager.register_source("entities", "json", "data/entities.json") -foundation = manager.create_foundation_graph() -validation = manager.validate_quality(foundation) +seed = SeedData() +knowledge = seed.get_knowledge("technology", "companies") ``` ---- +**What it does:** +- **Seed knowledge** - Foundation data for domains +- **Knowledge bases** - Pre-built domain knowledge +- **Quick start** - Bootstrap applications +- **Domain models** - Industry-specific data -## Output & Orchestration Layer - -These modules handle export, visualization, and workflow management. +**Use Cases:** +- Domain bootstrapping +- Quick start data +- Industry knowledge +- Foundation models --- +### LLM Providers Module +**Large Language Model integration** + +```python +from semantica.llms import LLMProvider + +provider = LLMProvider(model="gpt-4") +response = provider.generate(prompt, context=kg) +``` + +**What it does:** +- **LLM integration** - Connect to various LLM providers +- **Prompt engineering** - Optimize prompts for results +- **Context injection** - Provide knowledge graph context +- **Response parsing** - Extract structured outputs + +**Use Cases:** +- AI generation +- Question answering +- Text completion +- Knowledge reasoning + +--- + +## Output & Orchestration + ### Export Module - -!!! abstract "Purpose" - Export knowledge graphs and data to various formats for use in external tools. - -**Key Features:** - -- Multiple export formats (JSON, RDF, CSV, OWL, GraphML, GEXF) -- Custom export formats -- Batch export -- Metadata preservation -- Streaming export for large graphs -- Vector export support - -**Components:** - -- `JSONExporter` โ€” Export to JSON -- `RDFExporter` โ€” Export to RDF/XML -- `CSVExporter` โ€” Export to CSV -- `GraphExporter` โ€” Export to graph formats (GraphML, GEXF) -- `OWLExporter` โ€” Export to OWL -- `VectorExporter` โ€” Export vectors - -**Quick Example:** +**Data export and serialization** ```python -from semantica.export import JSONExporter, RDFExporter, CSVExporter +from semantica.export import GraphExporter -# Export to multiple formats -JSONExporter().export(kg, "output.json") -RDFExporter().export(kg, "output.rdf") -CSVExporter().export(kg, "output.csv") +exporter = GraphExporter() +exporter.export(kg, format="json", filename="output.json") ``` -**API Reference**: [Export Module](reference/export.md) +**What it does:** +- **Multiple formats** - JSON, CSV, RDF, GraphML +- **Database export** - Export to various databases +- **Streaming** - Handle large datasets +- **Filtering** - Export specific data subsets + +**Use Cases:** +- Data sharing +- System integration +- Backup and restore +- Format conversion --- ### Visualization Module - -!!! abstract "Purpose" - Visual exploration of knowledge graphs, embeddings, and analytics data. - -**Key Features:** - -- Interactive graph visualization -- Embedding visualization (t-SNE, PCA, UMAP) -- Temporal data visualization -- Ontology visualization -- Multiple output formats (HTML, PNG, SVG) -- Custom styling - -**Components:** - -- `KGVisualizer` โ€” Visualize knowledge graphs -- `EmbeddingVisualizer` โ€” Visualize embeddings (t-SNE, PCA, UMAP) -- `AnalyticsVisualizer` โ€” Visualize graph analytics -- `TemporalVisualizer` โ€” Visualize temporal data -- `OntologyVisualizer` โ€” Visualize ontology structure -- `SemanticNetworkVisualizer` โ€” Visualize semantic networks - -**Quick Example:** +**Graph visualization and analysis** ```python -from semantica.visualization import KGVisualizer, EmbeddingVisualizer +from semantica.visualization import GraphVisualizer -# Visualize knowledge graph -KGVisualizer().visualize(kg, output_format="html", output_path="graph.html") - -# Visualize embeddings -EmbeddingVisualizer().visualize(embeddings, method="tsne", output_path="embeddings.png") +visualizer = GraphVisualizer() +visualizer.plot(kg, layout="force_directed") ``` -**API Reference**: [Visualization Module](reference/visualization.md) +**What it does:** +- **Graph visualization** - Interactive graph plots +- **Custom styling** - Tailored visual appearance +- **Analytics charts** - Statistics and metrics +- **Exploration tools** - Interactive data exploration + +**Use Cases:** +- Data exploration +- Presentation +- Analysis +- Reporting --- ### Pipeline Module - -!!! abstract "Purpose" - Orchestrates workflows, connecting modules into robust, executable pipelines. - -**Key Features:** - -- Pipeline construction DSL -- Parallel execution -- Error handling and recovery -- Resource scheduling -- Pipeline validation -- Monitoring and logging -- Checkpoint support - -**Components:** - -- `PipelineBuilder` โ€” Build complex pipelines -- `ExecutionEngine` โ€” Execute pipelines -- `FailureHandler` โ€” Handle pipeline failures -- `ParallelismManager` โ€” Enable parallel processing -- `ResourceScheduler` โ€” Schedule resources -- `PipelineValidator` โ€” Validate pipeline configuration - -**Quick Example:** +**Workflow orchestration** ```python -from semantica.pipeline import PipelineBuilder -from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor +from semantica.pipeline import Pipeline -builder = PipelineBuilder() -pipeline = builder \ - .add_step("ingest", FileIngestor()) \ - .add_step("parse", DocumentParser()) \ - .add_step("extract", NERExtractor()) \ - .build() - -result = pipeline.execute(sources=["data/"], parallel=True) +pipeline = Pipeline() +pipeline.add_step("ingest", FileIngestor()) +pipeline.add_step("extract", NERExtractor()) +pipeline.add_step("build", GraphBuilder()) +result = pipeline.run("data/") ``` -**API Reference**: [Pipeline Module](reference/pipeline.md) +**What it does:** +- **Workflow orchestration** - Coordinate multiple steps +- **Parallel processing** - Run steps concurrently +- **Progress tracking** - Monitor pipeline execution +- **Error handling** - Robust error management + +**Use Cases:** +- Data processing +- Workflow automation +- Batch processing +- System integration --- -## Integration Patterns +## New Features & Modules -### Pattern 1: Complete Knowledge Graph Pipeline +### Change Management Module +**Version control and audit trails** -Build a complete knowledge graph from documents using the full pipeline. +```python +from semantica.change_management import TemporalVersionManager -**For complete examples, see:** -- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Complete pipeline walkthrough - - **Topics**: Ingestion, parsing, extraction, graph building, embeddings - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Learning the complete workflow +manager = TemporalVersionManager(storage_path="versions.db") +snapshot = manager.create_snapshot(kg, "v1.0", "user@example.com", "Initial version") +``` -### Pattern 2: Custom Pipeline with Module Selection +**What it does:** +- **Version control** - Track changes over time +- **Audit trails** - Complete change history +- **Data integrity** - SHA-256 checksums +- **Change comparison** - Detailed diff analysis -Build custom pipelines with specific module selections and quality assurance. - -**For examples, see:** -- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Advanced graph construction - - **Topics**: Custom pipelines, entity merging, conflict resolution - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Production graph construction - -### Pattern 3: GraphRAG with Hybrid Search - -Build GraphRAG systems with hybrid search combining vector and graph retrieval. - -**For complete examples, see:** -- **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production GraphRAG system - - **Topics**: GraphRAG, hybrid retrieval, graph traversal, LLM integration - - **Difficulty**: Advanced - - **Time**: 1-2 hours - - **Use Cases**: Production RAG applications - -### Pattern 4: Temporal Graph with Reasoning - -Build temporal graphs with logical reasoning capabilities. - -**For examples, see:** -- **[Temporal Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb)**: Temporal graph construction - - **Topics**: Time-stamped entities, temporal relationships, historical queries - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Time-aware knowledge graphs +**Use Cases:** +- Knowledge graph versioning +- Compliance tracking +- Data governance +- Change management --- -## Quick Reference: All Modules +### Provenance Module +**W3C PROV-O compliant tracking** -| Module | Import | Main Class | Purpose | -| :--- | :--- | :--- | :--- | -| **Ingest** | `semantica.ingest` | `FileIngestor` | Data ingestion | -| **Parse** | `semantica.parse` | `DocumentParser` | Document parsing | -| **Split** | `semantica.split` | `TextSplitter` | Text chunking | -| **Normalize** | `semantica.normalize` | `TextNormalizer` | Data cleaning | -| **Semantic Extract** | `semantica.semantic_extract` | `NERExtractor` | Entity extraction | -| **KG** | `semantica.kg` | `GraphBuilder` | Graph construction | -| **Ontology** | `semantica.ontology` | `OntologyGenerator` | Ontology generation | -| **Reasoning** | `semantica.reasoning` | `Reasoner` | Logical inference | -| **Embeddings** | `semantica.embeddings` | `EmbeddingGenerator` | Vector generation | -| **Vector Store** | `semantica.vector_store` | `VectorStore` | Vector storage | -| **Graph Store** | `semantica.graph_store` | `GraphStore` | Graph database | -| **Triplet Store** | `semantica.triplet_store` | `TripletStore` | RDF storage | -| **Deduplication** | `semantica.deduplication` | `DuplicateDetector` | Duplicate removal | -| **Conflicts** | `semantica.conflicts` | `ConflictDetector` | Conflict resolution | -| **Context** | `semantica.context` | `AgentContext` | Agent context & GraphRAG | -| **LLM Providers** | `semantica.llms` | `Groq`, `OpenAI`, `HuggingFaceLLM`, `LiteLLM` | LLM integration | -| **Seed** | `semantica.seed` | `SeedDataManager` | Foundation data | -| **Export** | `semantica.export` | `JSONExporter` | Data export | -| **Visualization** | `semantica.visualization` | `KGVisualizer` | Visualization | -| **Pipeline** | `semantica.pipeline` | `PipelineBuilder` | Workflow orchestration | +```python +from semantica.provenance import ProvenanceManager + +manager = ProvenanceManager() +manager.track_entity("entity_1", "document.pdf", "person") +``` + +**What it does:** +- **W3C PROV-O compliant** - Industry standard tracking +- **Complete lineage** - End-to-end traceability +- **Source attribution** - Track data origins +- **Integrity verification** - Tamper detection + +**Use Cases:** +- Regulatory compliance +- Data provenance +- Audit trails +- Source tracking --- -## Next Steps +### Core Module +**Framework orchestration and configuration** -- **[Core Concepts](concepts.md)** โ€” Understand the fundamental concepts -- **[Use Cases](use-cases.md)** โ€” See real-world applications -- **[Examples](examples.md)** โ€” Practical code examples -- **[Cookbook](cookbook.md)** โ€” Interactive Jupyter notebook tutorials -- **[API Reference](reference/core.md)** โ€” Detailed API documentation +```python +from semantica.core import Semantica, Config -### ๐Ÿณ Recommended Cookbooks +# Initialize framework +semantica = Semantica(config=Config()) +result = semantica.process("data/") +``` -- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules - - **Topics**: Framework overview, all modules, architecture - - **Difficulty**: Beginner - - **Time**: 30-45 minutes - - **Use Cases**: Understanding the complete framework +**What it does:** +- **Framework orchestration** - Central coordination +- **Configuration management** - Settings and preferences +- **Lifecycle management** - Start/stop/restart +- **Plugin system** - Extensible architecture -- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph - - **Topics**: Complete pipeline from ingestion to graph construction - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Hands-on practice with all modules +**Use Cases:** +- Framework initialization +- Configuration management +- Plugin development +- System orchestration --- -!!! info "Contribute" - Found an issue or want to improve this guide? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica) +## Getting Started +### Quick Start Example + +```python +# Complete pipeline example +from semantica.ingest import FileIngestor +from semantica.semantic_extract import NERExtractor, RelationExtractor +from semantica.kg import GraphBuilder +from semantica.pipeline import Pipeline + +# Create pipeline +pipeline = Pipeline() +pipeline.add_step("ingest", FileIngestor()) +pipeline.add_step("ner", NERExtractor()) +pipeline.add_step("relations", RelationExtractor()) +pipeline.add_step("build", GraphBuilder()) + +# Run pipeline +kg = pipeline.run("documents/") +print(f"Built graph with {len(kg['entities'])} entities") +``` + +### Choose Your Modules + +**For Document Processing:** +- Ingest โ†’ Parse โ†’ Split โ†’ Semantic Extract โ†’ Knowledge Graph + +**For Web Scraping:** +- Ingest (Web) โ†’ Normalize โ†’ Semantic Extract โ†’ Graph Store + +**For AI Agents:** +- Context โ†’ LLM Providers โ†’ Reasoning โ†’ Export + +**For Analytics:** +- Knowledge Graph โ†’ Graph Store โ†’ Visualization โ†’ Export + +--- + +## Module Reference + +| Module | Purpose | Key Classes | Use Cases | +|--------|---------|-------------|-----------| +| [Ingest](reference/ingest.md) | Data ingestion | FileIngestor, WebIngestor | File processing, web scraping | +| [Parse](reference/parse.md) | Document parsing | DocumentParser | PDF processing, text extraction | +| [Split](reference/split.md) | Text chunking | TextSplitter | RAG systems, preprocessing | +| [Normalize](reference/normalize.md) | Data cleaning | DataNormalizer | Quality improvement | +| [Semantic Extract](reference/semantic_extract.md) | Information extraction | NERExtractor, RelationExtractor | Knowledge graphs | +| [Knowledge Graph](reference/kg.md) | Graph management | GraphBuilder, GraphAnalyzer | Graph construction | +| [Ontology](reference/ontology.md) | Schema management | OntologyManager | Data modeling | +| [Reasoning](reference/reasoning.md) | Logical inference | ReasoningEngine | Knowledge discovery | +| [Embeddings](reference/embeddings.md) | Vector embeddings | EmbeddingGenerator | Semantic search | +| [Vector Store](reference/vector_store.md) | Vector database | VectorStore | Similarity search | +| [Graph Store](reference/graph_store.md) | Graph database | GraphStore | Graph storage | +| [Triplet Store](reference/triplet_store.md) | Triple storage | TripletStore | Semantic web | +| [Deduplication](reference/deduplication.md) | Entity resolution | EntityResolver | Data quality | +| [Conflicts](reference/conflicts.md) | Conflict resolution | ConflictDetector | Consistency | +| [Context](reference/context.md) | Context management | ContextManager | AI agents | +| [Seed](reference/seed.md) | Foundation data | SeedData | Domain knowledge | +| [LLM Providers](reference/llms.md) | LLM integration | LLMProvider | AI generation | +| [Export](reference/export.md) | Data export | GraphExporter | Data sharing | +| [Visualization](reference/visualization.md) | Graph visualization | GraphVisualizer | Data exploration | +| [Pipeline](reference/pipeline.md) | Workflow orchestration | Pipeline | Process automation | +| [Change Management](reference/change_management.md) | Version control | TemporalVersionManager | Audit trails | +| [Provenance](reference/provenance.md) | Data lineage | ProvenanceManager | Source tracking | +| [Core](reference/core.md) | Framework orchestration | Semantica, Config | System management | + +--- + +## Need Help? + +- **Documentation**: [Getting Started](getting-started.md) +- **Examples**: [Cookbook](cookbook.md) +- **Community**: [Discord](community.md) +- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) +- **Support**: [Contact Us](community.md) diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md index 2fdece50..9773a827 100644 --- a/docs/reference/change_management.md +++ b/docs/reference/change_management.md @@ -1,754 +1,274 @@ -# Change Management API Reference +# Change Management -Comprehensive API documentation for the Enhanced Change Management module in Semantica. +**Enterprise-grade version control and audit trails for knowledge graphs and ontologies with data integrity verification** ## Overview -The `semantica.change_management` module provides enterprise-grade version control, audit trails, and compliance tracking for knowledge graphs and ontologies. It includes persistent storage backends, detailed change tracking, data integrity verification, and standardized metadata structures. +The Semantica change management module provides enterprise-grade version control, audit trails, and compliance tracking for knowledge graphs and ontologies. Designed for high-stakes domains where every change must be tracked, verified, and auditable with complete data integrity guarantees. -## Module Structure +
-``` -semantica.change_management/ -โ”œโ”€โ”€ change_log.py # Standardized metadata structures -โ”œโ”€โ”€ version_storage.py # Storage abstraction and implementations -โ”œโ”€โ”€ managers.py # Enhanced version managers -โ”œโ”€โ”€ ontology_version_manager.py # Ontology version management -โ””โ”€โ”€ change_management_usage.md # Usage guide -``` +- :material-history:{ .lg .middle } **Version Control** -## Quick Import + --- -```python -from semantica.change_management import ( - # Metadata - ChangeLogEntry, - - # Storage - VersionStorage, - InMemoryVersionStorage, - SQLiteVersionStorage, - - # Utilities - compute_checksum, - verify_checksum, - - # Version Managers - BaseVersionManager, - TemporalVersionManager, - OntologyVersionManager, - VersionManager, - OntologyVersion -) -``` + Complete snapshot management with SHA-256 integrity verification + +- :material-database:{ .lg .middle } **Dual Storage** + + --- + + InMemory (development) and SQLite (production) with ACID guarantees + +- :material-account-tree:{ .lg .middle } **Knowledge Graph Versioning** + + --- + + Entity and relationship-level change tracking with detailed diffs + +- :material-schema:{ .lg .middle } **Ontology Versioning** + + --- + + Structural change tracking for classes, properties, and axioms + +- :material-audit:{ .lg .middle } **Audit Trail Compliance** + + --- + + Complete change logs with author attribution and timestamps + +- :material-shield-check:{ .lg .middle } **Data Integrity** + + --- + + SHA-256 checksums for tamper detection and verification + +- :material-compare:{ .lg .middle } **Change Comparison** + + --- + + Detailed diff algorithms for entities, relationships, and ontology structures + +- :material-backup-restore:{ .lg .middle } **Backward Compatibility** + + --- + + Legacy support for existing ontology version management + +
+ +### Key Features + +- โœ… **Enterprise Version Control** โ€” Complete snapshot management with SHA-256 integrity verification +- โœ… **Dual Storage Backends** โ€” InMemory (development) and SQLite (production) with ACID guarantees +- โœ… **Knowledge Graph Versioning** โ€” Entity and relationship-level change tracking with detailed diffs +- โœ… **Ontology Versioning** โ€” Structural change tracking for classes, properties, and axioms +- โœ… **Audit Trail Compliance** โ€” Complete change logs with author attribution and timestamps +- โœ… **Data Integrity** โ€” SHA-256 checksums for tamper detection and verification +- โœ… **Change Comparison** โ€” Detailed diff algorithms for entities, relationships, and ontology structures +- โœ… **Backward Compatibility** โ€” Legacy support for existing ontology version management --- -## Core Classes - -### ChangeLogEntry - -Standardized metadata structure for version changes with validation. - -#### Class Definition +## Quick Start ```python -@dataclass -class ChangeLogEntry: - """ - Standardized change log entry with validation. - - Attributes: - timestamp: ISO 8601 formatted timestamp - author: Email address of the change author - description: Change description (max 500 characters) - change_id: Optional ID linking to external systems - """ - timestamp: str - author: str - description: str - change_id: Optional[str] = None -``` +from semantica.change_management import TemporalVersionManager -#### Methods - -##### `__post_init__()` - -Validates all fields after initialization. - -**Raises:** -- `ValidationError`: If any field validation fails - -**Example:** -```python -entry = ChangeLogEntry( - timestamp="2024-01-30T12:00:00Z", - author="user@example.com", - description="Updated entity relationships", - change_id="TICKET-123" -) -``` - -##### `create_now(author, description, change_id=None)` (classmethod) - -Creates a change log entry with the current timestamp. - -**Parameters:** -- `author` (str): Email address of the change author -- `description` (str): Change description (max 500 characters) -- `change_id` (str, optional): ID linking to external systems - -**Returns:** -- `ChangeLogEntry`: New instance with current timestamp - -**Example:** -```python -entry = ChangeLogEntry.create_now( - author="developer@company.com", - description="Fixed entity resolution bug", - change_id="JIRA-1234" -) -``` - -#### Validation Rules - -- **Timestamp**: Must be valid ISO 8601 format with 'T' separator -- **Author**: Must be valid email format (RFC 5322) -- **Description**: Maximum 500 characters -- **Change ID**: Optional, no validation - ---- - -### VersionStorage - -Abstract base class for storage implementations. - -#### Class Definition - -```python -class VersionStorage(ABC): - """ - Abstract base class for version storage backends. - - Provides interface for saving, retrieving, and managing version snapshots. - """ -``` - -#### Abstract Methods - -##### `save(snapshot)` - -Save a version snapshot. - -**Parameters:** -- `snapshot` (Dict[str, Any]): Version snapshot dictionary with metadata - -**Raises:** -- `ValidationError`: If snapshot data is invalid -- `ProcessingError`: If save operation fails - -**Example:** -```python -snapshot = { - "label": "v1.0", - "timestamp": "2024-01-30T12:00:00Z", - "author": "user@example.com", - "description": "Initial version", - "data": {...} -} -storage.save(snapshot) -``` - -##### `get(label)` - -Retrieve a version snapshot by label. - -**Parameters:** -- `label` (str): Version label to retrieve - -**Returns:** -- `Optional[Dict[str, Any]]`: Snapshot dictionary or None if not found - -**Example:** -```python -snapshot = storage.get("v1.0") -if snapshot: - print(f"Retrieved: {snapshot['label']}") -``` - -##### `list_all()` - -List all version snapshots. - -**Returns:** -- `List[Dict[str, Any]]`: List of snapshot metadata dictionaries - -**Example:** -```python -versions = storage.list_all() -for v in versions: - print(f"{v['label']}: {v['description']}") -``` - -##### `exists(label)` - -Check if a version exists. - -**Parameters:** -- `label` (str): Version label to check - -**Returns:** -- `bool`: True if version exists, False otherwise - -**Example:** -```python -if storage.exists("v1.0"): - print("Version exists") -``` - -##### `delete(label)` - -Delete a version snapshot. - -**Parameters:** -- `label` (str): Version label to delete - -**Returns:** -- `bool`: True if deleted, False if not found - -**Example:** -```python -if storage.delete("v1.0"): - print("Version deleted") -``` - ---- - -### InMemoryVersionStorage - -In-memory version storage implementation. - -#### Class Definition - -```python -class InMemoryVersionStorage(VersionStorage): - """ - In-memory version storage implementation. - - Fast, volatile storage for development and testing. - Data is lost when the process ends. - """ -``` - -#### Constructor - -```python -def __init__(self): - """Initialize in-memory storage.""" -``` - -**Example:** -```python -storage = InMemoryVersionStorage() -``` - -#### Performance Characteristics - -- **Save**: 0.37-16ms (10-1000 entities) -- **Get**: 0.20-16ms (10-1000 entities) -- **List**: <0.03ms -- **Thread-safe**: Yes (uses RLock) - -#### Use Cases - -- Development and testing -- Temporary version tracking -- High-performance scenarios where persistence is not required - ---- - -### SQLiteVersionStorage - -SQLite-based persistent version storage implementation. - -#### Class Definition - -```python -class SQLiteVersionStorage(VersionStorage): - """ - SQLite-based persistent version storage implementation. - - Provides persistence across process restarts with ACID guarantees. - """ -``` - -#### Constructor - -```python -def __init__(self, storage_path: str): - """ - Initialize SQLite storage. - - Args: - storage_path: Path to SQLite database file - """ -``` - -**Parameters:** -- `storage_path` (str): Path to SQLite database file (created if doesn't exist) - -**Example:** -```python -storage = SQLiteVersionStorage("versions.db") -``` - -#### Database Schema - -```sql -CREATE TABLE versions ( - label TEXT PRIMARY KEY, - timestamp TEXT NOT NULL, - author TEXT NOT NULL, - description TEXT, - checksum TEXT, - snapshot_data TEXT NOT NULL, - created_at TEXT NOT NULL -) -``` - -#### Performance Characteristics - -- **Save**: 7-25ms (10-1000 entities) -- **Get**: 2-8ms (10-1000 entities) -- **List**: 0.6-13ms -- **Thread-safe**: Yes (uses RLock) -- **ACID**: Full transaction support - -#### Use Cases - -- Production deployments -- Long-term version storage -- Compliance and audit requirements -- Multi-process environments - ---- - -### BaseVersionManager - -Abstract base class for version managers. - -#### Class Definition - -```python -class BaseVersionManager(ABC): - """ - Abstract base class for version managers. - - Provides common functionality for version management across - different data types (knowledge graphs, ontologies, etc.). - """ -``` - -#### Constructor - -```python -def __init__(self, storage_path: Optional[str] = None): - """ - Initialize base version manager. - - Args: - storage_path: Path to SQLite database file for persistent storage. - If None, uses in-memory storage. - """ -``` - -**Parameters:** -- `storage_path` (str, optional): Path to SQLite database file - -**Example:** -```python -# In-memory storage -manager = BaseVersionManager() - -# Persistent storage -manager = BaseVersionManager(storage_path="versions.db") -``` - -#### Abstract Methods - -##### `create_snapshot(data, version_label, author, description, **options)` - -Create a versioned snapshot of the data. - -**Parameters:** -- `data` (Any): Data to snapshot -- `version_label` (str): Version label -- `author` (str): Email address of the author -- `description` (str): Change description -- `**options`: Additional options - -**Returns:** -- `Dict[str, Any]`: Snapshot with metadata and checksum - -##### `compare_versions(version1, version2, **options)` - -Compare two versions and return detailed differences. - -**Parameters:** -- `version1` (Any): First version (label or snapshot) -- `version2` (Any): Second version (label or snapshot) -- `**options`: Comparison options - -**Returns:** -- `Dict[str, Any]`: Detailed differences - -#### Concrete Methods - -##### `list_versions()` - -List all version snapshots. - -**Returns:** -- `List[Dict[str, Any]]`: List of version metadata - -**Example:** -```python -versions = manager.list_versions() -for v in versions: - print(f"{v['label']}: {v['description']}") -``` - -##### `get_version(label)` - -Retrieve specific version by label. - -**Parameters:** -- `label` (str): Version label - -**Returns:** -- `Optional[Dict[str, Any]]`: Version snapshot or None - -**Example:** -```python -version = manager.get_version("v1.0") -``` - -##### `verify_checksum(snapshot)` - -Verify data integrity using checksum. - -**Parameters:** -- `snapshot` (Dict[str, Any]): Snapshot to verify - -**Returns:** -- `bool`: True if checksum is valid - -**Example:** -```python -is_valid = manager.verify_checksum(snapshot) -``` - ---- - -### TemporalVersionManager - -Enhanced temporal version management engine for knowledge graphs. - -#### Class Definition - -```python -class TemporalVersionManager(BaseVersionManager): - """ - Enhanced temporal version management engine for knowledge graphs. - - Features: - - Persistent snapshot storage (SQLite or in-memory) - - Detailed change tracking with entity-level diffs - - SHA-256 checksums for data integrity - - Standardized metadata with author attribution - - Version comparison with backward compatibility - - Input validation and security features - """ -``` - -#### Constructor - -```python -def __init__(self, storage_path: Optional[str] = None, **config): - """ - Initialize enhanced temporal version manager. - - Args: - storage_path: Path to SQLite database file for persistent storage. - If None, uses in-memory storage - **config: Additional configuration options - """ -``` - -**Parameters:** -- `storage_path` (str, optional): Path to SQLite database file -- `**config`: Additional configuration options - -**Example:** -```python -# In-memory storage -manager = TemporalVersionManager() - -# Persistent storage -manager = TemporalVersionManager(storage_path="kg_versions.db") -``` - -#### Methods - -##### `create_snapshot(graph, version_label, author, description, **options)` - -Create and store snapshot with checksum and metadata. - -**Parameters:** -- `graph` (Dict[str, Any]): Knowledge graph dict with "entities" and "relationships" -- `version_label` (str): Version string (e.g., "v1.0") -- `author` (str): Email address of the change author -- `description` (str): Change description (max 500 chars) -- `**options`: Additional options - -**Returns:** -- `Dict[str, Any]`: Snapshot with metadata and checksum - -**Raises:** -- `ValidationError`: If input validation fails -- `ProcessingError`: If snapshot creation fails - -**Example:** -```python -graph = { - "entities": [ - {"id": "e1", "name": "Entity 1", "type": "Person"}, - {"id": "e2", "name": "Entity 2", "type": "Organization"} - ], - "relationships": [ - {"source": "e1", "target": "e2", "type": "works_for"} - ] -} +# Initialize version manager +manager = TemporalVersionManager(storage_path="versions.db") +# Create versioned snapshot snapshot = manager.create_snapshot( - graph, + graph={"entities": [...], "relationships": [...]}, version_label="v1.0", author="user@example.com", description="Initial knowledge graph" ) -print(f"Created: {snapshot['label']}") -print(f"Checksum: {snapshot['checksum']}") -``` - -##### `compare_versions(version1, version2, **options)` - -Compare two versions with detailed entity and relationship diffs. - -**Parameters:** -- `version1` (Union[str, Dict]): First version (label or snapshot dict) -- `version2` (Union[str, Dict]): Second version (label or snapshot dict) -- `**options`: Comparison options - -**Returns:** -- `Dict[str, Any]`: Detailed differences including: - - `summary`: Aggregate statistics - - `entity_changes`: Entity-level changes - - `relationship_changes`: Relationship-level changes - -**Example:** -```python +# Compare versions diff = manager.compare_versions("v1.0", "v2.0") - -print(f"Entities added: {diff['summary']['entities_added']}") -print(f"Entities modified: {diff['summary']['entities_modified']}") -print(f"Relationships added: {diff['summary']['relationships_added']}") - -# Detailed entity changes -for entity_id, changes in diff['entity_changes'].items(): - print(f"Entity {entity_id}: {changes['status']}") - if changes['status'] == 'modified': - print(f" Before: {changes['before']}") - print(f" After: {changes['after']}") ``` -#### Performance - -- **Snapshot Creation**: 1.40-54ms (50-2000 entities) -- **Version Retrieval**: 0.65-26ms (50-2000 entities) -- **Version Comparison**: 3.46-33ms (100-1000 entities) -- **Concurrent Throughput**: 500+ operations/second +**What this does:** +- Initializes version manager with persistent SQLite storage +- Creates a versioned snapshot of knowledge graph data +- Compares two versions to detect changes +- Provides complete audit trail with author attribution --- +## Core Components + +### ChangeLogEntry + +Standardized metadata for tracking version changes with validation. + +```python +from semantica.change_management import ChangeLogEntry + +@dataclass +class ChangeLogEntry: + timestamp: str # ISO 8601 format + author: str # Email address + description: str # Max 500 characters + change_id: Optional[str] = None +``` + +**Key Method:** +- `create_now(author, description, change_id=None)` - Create entry with current timestamp + +### Storage Backends + +#### InMemoryVersionStorage +Fast, volatile storage for development and testing. +```python +from semantica.change_management import InMemoryVersionStorage + +storage = InMemoryVersionStorage() +``` + +#### SQLiteVersionStorage +Persistent storage with ACID guarantees for production. +```python +from semantica.change_management import SQLiteVersionStorage + +storage = SQLiteVersionStorage("versions.db") +``` + +#### VersionStorage (Abstract) +Base interface for custom storage implementations. + +**Core Methods:** +- `save(snapshot)` - Store version snapshot +- `get(label)` - Retrieve by version label +- `list_all()` - List all versions +- `exists(label)` - Check if version exists +- `delete(label)` - Remove version + +--- + +## Version Managers + +### BaseVersionManager + +Abstract base class providing common version management functionality. + +```python +from semantica.change_management import BaseVersionManager + +manager = BaseVersionManager(storage_path="versions.db") +``` + +**Common Methods:** +- `list_versions()` - Get all version metadata +- `get_version(label)` - Retrieve specific version +- `verify_checksum(snapshot)` - Validate data integrity + +### TemporalVersionManager + +**Knowledge Graph Version Management** + +Perfect for tracking changes in knowledge graphs with entity and relationship diffs. + +```python +from semantica.change_management import TemporalVersionManager + +manager = TemporalVersionManager(storage_path="kg_versions.db") + +# Create snapshot +snapshot = manager.create_snapshot( + graph={ + "entities": [ + {"id": "e1", "name": "Entity 1", "type": "Person"}, + {"id": "e2", "name": "Entity 2", "type": "Organization"} + ], + "relationships": [ + {"source": "e1", "target": "e2", "type": "works_for"} + ] + }, + version_label="v1.0", + author="user@example.com", + description="Initial knowledge graph" +) + +# Compare versions with detailed diffs +diff = manager.compare_versions("v1.0", "v2.0") +print(f"Entities added: {diff['summary']['entities_added']}") +print(f"Relationships modified: {diff['summary']['relationships_modified']}") +``` + +**Key Features:** +- Entity-level change tracking +- Relationship diff analysis +- SHA-256 checksums for integrity +- Detailed change summaries + ### OntologyVersionManager -Enhanced version management for ontologies. +**Ontology Version Management** -#### Class Definition +Designed for structural changes in ontologies with class, property, and axiom tracking. ```python -class OntologyVersionManager(BaseVersionManager): - """ - Enhanced version management for ontologies. - - Features: - - Persistent ontology snapshot storage - - Structural comparison (classes, properties, axioms) - - SHA-256 checksums for data integrity - - Standardized metadata with author attribution - """ -``` +from semantica.change_management import OntologyVersionManager -#### Constructor - -```python -def __init__(self, storage_path: Optional[str] = None, **config): - """ - Initialize enhanced version manager for ontologies. - - Args: - storage_path: Path to SQLite database file for persistent storage. - If None, uses in-memory storage - **config: Additional configuration options - """ -``` - -**Example:** -```python manager = OntologyVersionManager(storage_path="ontology_versions.db") -``` - -#### Methods - -##### `create_snapshot(ontology, version_label, author, description, **options)` - -Create ontology snapshot with metadata. - -**Parameters:** -- `ontology` (Dict[str, Any]): Ontology dict with structure information -- `version_label` (str): Version label -- `author` (str): Email address of the author -- `description` (str): Change description -- `**options`: Additional options - -**Returns:** -- `Dict[str, Any]`: Ontology snapshot with metadata - -**Example:** -```python -ontology = { - "uri": "https://example.com/ontology", - "version_info": {"version": "1.0", "date": "2024-01-30"}, - "structure": { - "classes": ["Person", "Organization", "Location"], - "properties": ["name", "address", "email"], - "individuals": ["JohnDoe", "ACME_Corp"], - "axioms": ["Person hasAddress exactly 1 Location"] - } -} +# Create ontology snapshot snapshot = manager.create_snapshot( - ontology, + ontology={ + "uri": "https://example.com/ontology", + "structure": { + "classes": ["Person", "Organization"], + "properties": ["name", "email"], + "axioms": ["Person hasEmail exactly 1 Email"] + } + }, version_label="ont_v1.0", author="architect@example.com", description="Initial ontology design" ) -``` -##### `compare_versions(version1, version2, **options)` - -Compare ontology versions with structural analysis. - -**Parameters:** -- `version1` (Union[str, Dict]): First version -- `version2` (Union[str, Dict]): Second version -- `**options`: Comparison options - -**Returns:** -- `Dict[str, Any]`: Structural differences including: - - `classes_added`, `classes_removed` - - `properties_added`, `properties_removed` - - `individuals_added`, `individuals_removed` - - `axioms_added`, `axioms_removed`, `axioms_modified` - -**Example:** -```python +# Compare structural changes diff = manager.compare_versions("ont_v1.0", "ont_v2.0") - print(f"Classes added: {diff['classes_added']}") -print(f"Properties added: {diff['properties_added']}") print(f"Axioms modified: {diff['axioms_modified']}") ``` +**Key Features:** +- Class and property tracking +- Axiom change detection +- Structural comparison +- Import/export support + --- -## Utility Functions +## Data Integrity ### compute_checksum -Compute SHA-256 checksum for data integrity. +Generate SHA-256 checksum for data integrity verification. -#### Function Signature - -```python -def compute_checksum(data: Dict[str, Any]) -> str: - """ - Compute SHA-256 checksum for data. - - Args: - data: Dictionary to compute checksum for - - Returns: - SHA-256 checksum as hexadecimal string - """ -``` - -**Parameters:** -- `data` (Dict[str, Any]): Dictionary to compute checksum for - -**Returns:** -- `str`: SHA-256 checksum as hexadecimal string - -**Example:** ```python from semantica.change_management import compute_checksum data = {"entities": [...], "relationships": [...]} checksum = compute_checksum(data) -print(f"Checksum: {checksum}") +print(f"SHA-256: {checksum}") ``` -**Performance:** 1.29-110ms (100-10,000 entities) - ---- +**Use cases:** +- Verify data integrity before storing snapshots +- Detect unauthorized modifications to version data +- Ensure consistency across distributed systems +- Generate unique identifiers for data versions ### verify_checksum -Verify data integrity using stored checksum. +Validate data integrity using stored checksums. -#### Function Signature - -```python -def verify_checksum(snapshot: Dict[str, Any]) -> bool: - """ - Verify data integrity using checksum. - - Args: - snapshot: Snapshot dictionary with 'checksum' field - - Returns: - True if checksum is valid, False otherwise - """ -``` - -**Parameters:** -- `snapshot` (Dict[str, Any]): Snapshot dictionary with 'checksum' field - -**Returns:** -- `bool`: True if checksum is valid, False otherwise - -**Example:** ```python from semantica.change_management import verify_checksum @@ -759,180 +279,72 @@ if not is_valid: print("WARNING: Data integrity compromised!") ``` -**Performance:** 0.82-96ms (100-10,000 entities) +**Use cases:** +- Validate snapshot integrity after retrieval +- Detect data corruption or tampering +- Ensure compliance with data integrity requirements +- Verify backup and restore operations --- -## Legacy Classes +## Legacy Support ### VersionManager -Original ontology version manager (moved from `semantica.ontology`). - -#### Import +Original ontology version manager for backward compatibility. ```python from semantica.change_management import VersionManager, OntologyVersion ``` -**Note:** This class is maintained for backward compatibility. New projects should use `OntologyVersionManager`. +**Note:** Use `OntologyVersionManager` for new projects. --- ## Error Handling -### ValidationError - -Raised when input validation fails. - -**Common Causes:** -- Invalid email format -- Description exceeds 500 characters -- Invalid ISO 8601 timestamp -- Missing required fields - -**Example:** ```python -from semantica.utils.exceptions import ValidationError +from semantica.utils.exceptions import ValidationError, ProcessingError try: - entry = ChangeLogEntry( - timestamp="invalid", - author="not-an-email", - description="x" * 501 - ) + snapshot = manager.create_snapshot(...) except ValidationError as e: - print(f"Validation failed: {e}") -``` - -### ProcessingError - -Raised when operations fail. - -**Common Causes:** -- Database connection issues -- File system errors -- Concurrent modification conflicts - -**Example:** -```python -from semantica.utils.exceptions import ProcessingError - -try: - storage.save(snapshot) + print(f"Invalid input: {e}") except ProcessingError as e: - print(f"Save failed: {e}") + print(f"Operation failed: {e}") ``` ---- - -## Performance Considerations - -### Benchmarks - -Based on comprehensive performance testing: - -| Component | Small (100) | Medium (500) | Large (2000) | -|-----------|-------------|--------------|--------------| -| Snapshot Creation | 2.33ms | 10.70ms | 54.23ms | -| Version Retrieval | 1.88ms | 7.33ms | 26.04ms | -| Version Comparison | 3.46ms | 17.39ms | 32.83ms | -| Checksum Compute | 1.29ms | 5.48ms | 22.15ms | -| SQLite Save | 8.69ms | 13.37ms | 25.33ms | -| InMemory Save | 1.18ms | 10.60ms | 14.11ms | - -### Optimization Tips - -1. **Use appropriate storage backend:** - - Development: `InMemoryVersionStorage` - - Production: `SQLiteVersionStorage` - -2. **Batch operations when possible:** - ```python - for data in batch: - manager.create_snapshot(data, ...) - ``` - -3. **Implement retention policies:** - ```python - # Delete old versions periodically - for version in old_versions: - storage.delete(version['label']) - ``` - -4. **Use concurrent operations:** - - Thread-safe: 500+ operations/second - - No performance degradation under load +**Common Errors:** +- `ValidationError` - Invalid email, missing fields, bad timestamps +- `ProcessingError` - Database issues, file system errors --- -## Compliance Features +## Best Practices -### HIPAA Compliance +### Performance Tips +- Use `InMemoryVersionStorage` for development/testing +- Use `SQLiteVersionStorage` for production +- Implement retention policies for old versions -- Complete audit trails with author attribution -- Timestamp tracking for all changes -- Data integrity verification with checksums -- Secure storage with access controls - -### SOX Compliance - -- Immutable change records -- Detailed change descriptions -- External system linking (change IDs) -- Comprehensive audit reports - -### FDA 21 CFR Part 11 - -- Electronic signatures (author email) -- Data integrity verification -- Audit trail generation -- Tamper detection - ---- - -## Examples - -### Complete Healthcare Example +### Security Considerations +- Validate author emails for audit trails +- Use checksums for data integrity +- Store sensitive data with appropriate permissions +### Usage Patterns ```python from semantica.change_management import TemporalVersionManager -# Initialize with HIPAA-compliant storage -manager = TemporalVersionManager(storage_path="hipaa_records.db") +# Development workflow +dev_manager = TemporalVersionManager() # In-memory -# Patient knowledge graph -patient_kg = { - "entities": [ - {"id": "patient_001", "type": "Patient", "name": "Jane Smith"}, - {"id": "diagnosis_001", "type": "Diagnosis", "code": "I10"} - ], - "relationships": [ - {"source": "patient_001", "target": "diagnosis_001", "type": "has_diagnosis"} - ] -} - -# Create versioned record -snapshot = manager.create_snapshot( - patient_kg, - "patient_001_v1.0", - "dr.williams@hospital.com", - "Initial diagnosis - Essential hypertension" +# Production workflow +prod_manager = TemporalVersionManager( + storage_path="secure/production_versions.db" ) -# Verify integrity -assert manager.verify_checksum(snapshot), "Data integrity check failed" - -# Generate audit report -for version in manager.list_versions(): - print(f"{version['timestamp']}: {version['label']} by {version['author']}") +# Audit trail generation +for version in prod_manager.list_versions(): + print(f"{version['timestamp']}: {version['description']} by {version['author']}") ``` - ---- - -## See Also - -- **Usage Guide**: `semantica/change_management/change_management_usage.md` -- **Performance Tests**: `tests/change_management/test_performance.py` -- **CHANGELOG**: `CHANGELOG.md` -- **GitHub**: https://github.com/Hawksight-AI/semantica diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index 55c1cb98..25fff55a 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -14,7 +14,7 @@ The **Ingest Module** is the entry point for loading data into Semantica. It pro - **File Systems**: Local files, cloud storage (S3, GCS, Azure) - **Web Content**: Websites, RSS feeds, APIs - **Streams**: Real-time data from Kafka, RabbitMQ, etc. -- **Databases**: SQL and NoSQL databases +- **Databases**: SQL, NoSQL, and cloud data warehouses including Snowflake - **Repositories**: Git repositories (GitHub, GitLab) - **Email**: IMAP, POP3 servers - **MCP**: Model Context Protocol servers @@ -72,7 +72,7 @@ The **Ingest Module** is the entry point for loading data into Semantica. It pro --- - Ingest tables and query results from SQL databases + Ingest tables and query results from SQL, NoSQL, and cloud data warehouses including Snowflake
@@ -173,7 +173,7 @@ Handles IMAP and POP3 servers. ### DBIngestor -Handles SQL databases. +Handles SQL and NoSQL databases including Snowflake. **Methods:** @@ -181,6 +181,16 @@ Handles SQL databases. |--------|-------------| | `ingest_database(conn)` | Export tables | | `execute_query(sql)` | Run custom SQL | +| `connect_snowflake(account, user, password, warehouse)` | Connect to Snowflake | +| `ingest_snowflake_table(table_name)` | Ingest Snowflake table | +| `execute_snowflake_query(sql)` | Run Snowflake SQL | + +**Supported Databases:** +- **PostgreSQL**, **MySQL**, **SQLite** +- **Microsoft SQL Server**, **Oracle** +- **Snowflake** (Cloud Data Warehouse) +- **MongoDB**, **Cassandra** (NoSQL) +- **BigQuery**, **Redshift** (Cloud Data Warehouses) ### MCPIngestor @@ -258,6 +268,64 @@ ingestor.monitor( ) ``` +### Snowflake Data Warehouse Integration + +```python +from semantica.ingest import DBIngestor + +# 1. Connect to Snowflake +ingestor = DBIngestor() +ingestor.connect_snowflake( + account="your_account.snowflakecomputing.com", + user="your_username", + password="your_password", + warehouse="ANALYTICS_WH", + database="PRODUCTION_DB", + schema="PUBLIC" +) + +# 2. Ingest entire table +data = ingestor.ingest_snowflake_table("CUSTOMERS") + +# 3. Or run custom query +results = ingestor.execute_snowflake_query(""" + SELECT + CUSTOMER_ID, + NAME, + EMAIL, + CREATED_AT + FROM CUSTOMERS + WHERE CREATED_AT > '2024-01-01' +""") + +# 4. Process with pipeline +for row in results: + pipeline.process(row) +``` + +!!! info "Comprehensive Snowflake Guide" + For detailed Snowflake integration including authentication methods, advanced features, and best practices, see the **[Snowflake Integration Guide](../integrations/snowflake.md)**. + +### Multi-Database Integration + +```python +from semantica.ingest import DBIngestor + +ingestor = DBIngestor() + +# Connect to multiple databases +connections = { + "snowflake": ingestor.connect_snowflake(...), + "postgres": ingestor.connect_database("postgresql://..."), + "mysql": ingestor.connect_database("mysql://...") +} + +# Ingest from all sources +for name, conn in connections.items(): + data = ingestor.ingest_database(conn) + print(f"Ingested {len(data)} records from {name}") +``` + --- ## Best Practices @@ -271,6 +339,7 @@ ingestor.monitor( ## See Also +- **[Snowflake Integration Guide](../integrations/snowflake.md)** - Comprehensive Snowflake integration with authentication, advanced features, and best practices - [Parse Module](parse.md) - Processes the raw data ingested here - [Split Module](split.md) - Chunks the ingested content - [Utils Module](utils.md) - Validation helpers diff --git a/docs/reference/provenance.md b/docs/reference/provenance.md index 5c16e061..044f3020 100644 --- a/docs/reference/provenance.md +++ b/docs/reference/provenance.md @@ -6,6 +6,58 @@ The Semantica provenance module provides W3C PROV-O compliant tracking for knowledge graphs, enabling complete end-to-end lineage from source documents to query responses. Designed for high-stakes domains where every decision must be explainable and auditable. +
+ +- :material-web:{ .lg .middle } **W3C PROV-O Compliant** + + --- + + Implements PROV-O ontology (prov:Entity, prov:Activity, prov:Agent, prov:wasDerivedFrom) + +- :material-all-inclusive:{ .lg .middle } **Complete Coverage** + + --- + + All 17 Semantica modules integrated for comprehensive tracking + +- :material-source-document:{ .lg .middle } **Source Tracking** + + --- + + Document identifiers, page numbers, sections, and direct quotes supported + +- :material-backup-restore:{ .lg .middle } **Backward Compatible** + + --- + + 100% backward compatible, opt-in only with zero breaking changes + +- :material-database:{ .lg .middle } **Multiple Storage** + + --- + + InMemory (fast) and SQLite (persistent) backends available + +- :material-bridge:{ .lg .middle } **Bridge Axiom Support** + + --- + + Translation chain tracking for domain transformations (L1 โ†’ L2 โ†’ L3) + +- :material-shield-check:{ .lg .middle } **Integrity Verification** + + --- + + SHA-256 checksums for tamper detection and verification + +- :material-route:{ .lg .middle } **Complete Lineage** + + --- + + End-to-end tracing from document to AI response + +
+ ### Key Features - โœ… **W3C PROV-O Compliant** โ€” Implements PROV-O ontology (prov:Entity, prov:Activity, prov:Agent, prov:wasDerivedFrom) @@ -45,6 +97,12 @@ manager = ProvenanceManager() manager = ProvenanceManager(storage_path="provenance.db") ``` +**Key capabilities:** +- Track entities and relationships with complete lineage +- Store provenance data in memory or persistent SQLite storage +- Query provenance information for audit and compliance +- Maintain W3C PROV-O compliant records + **Methods:** - `track_entity(entity_id, source, entity_type, **metadata)` โ€” Track entity provenance - `track_relationship(relationship_id, source, subject, predicate, obj, **metadata)` โ€” Track relationship provenance @@ -66,6 +124,12 @@ from semantica.provenance import ProvenanceManager, InMemoryStorage manager = ProvenanceManager(storage=InMemoryStorage()) ``` +**Best for:** +- Development and testing environments +- Temporary provenance tracking +- High-performance scenarios where persistence isn't required +- Rapid prototyping and debugging + #### SQLiteStorage Persistent storage for production use. @@ -77,6 +141,12 @@ storage = SQLiteStorage("provenance.db") manager = ProvenanceManager(storage=storage) ``` +**Best for:** +- Production deployments requiring persistence +- Long-term provenance storage +- Compliance and audit requirements +- Multi-process environments + ### Data Schemas #### ProvenanceEntry @@ -504,9 +574,9 @@ manager = ProvenanceManager(storage=storage) ### Benchmarks -- **Entity tracking:** <5ms per operation -- **Lineage retrieval:** <10ms for chains up to 100 levels -- **Batch operations:** 1000+ entities/second +- **Entity tracking:** Fast per operation +- **Lineage retrieval:** Quick retrieval for long chains +- **Batch operations:** High-throughput batch processing - **Storage:** InMemory (fastest), SQLite (persistent) ### Optimization Tips diff --git a/docs/release-guide.md b/docs/release-guide.md deleted file mode 100644 index 6ee0af20..00000000 --- a/docs/release-guide.md +++ /dev/null @@ -1,3 +0,0 @@ -# Release Guide - ---8<-- "RELEASE.md" diff --git a/mkdocs.yml b/mkdocs.yml index bcaaed14..9369bcb1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -133,25 +133,17 @@ nav: - Visualization: reference/visualization.md - Guides: - concepts.md - - deep-dive.md - modules.md - - glossary.md - use-cases.md - examples.md - - Code Examples: CodeExamples.md - - learning-more.md + - glossary.md - Integrations: - Docling: integrations/docling.md + - Snowflake: integrations/snowflake.md - Cookbook: cookbook.md - Resources: - - community-projects.md - community.md - contributing.md - - architecture.md - - governance.md - - citation.md - - Changelog: changelog.md - - Release Guide: release-guide.md - faq.md - license.md