-
-- **Input Layer**
-
- ---
-
- Data ingestion and preparation
-
- **Modules**: Ingest, Parse, Split, Normalize
-
-- **Semantic Layer**
-
- ---
-
- Intelligence and understanding
-
- **Modules**: Semantic Extract, Knowledge Graph, Ontology, Reasoning
-
-- **Storage Layer**
-
- ---
-
- Persistent data storage
-
- **Modules**: Embeddings, Vector Store, Graph Store
-
-- **Quality Layer**
-
- ---
-
- Data quality and consistency
-
- **Modules**: Deduplication, Conflicts
-
-- **Context & Memory**
-
- ---
-
- Agent memory and foundation data
-
- **Modules**: Context, Seed, LLM Providers
-
-- **Output & Orchestration**
-
- ---
-
- Export, visualization, and workflows
-
- **Modules**: Export, Visualization, Pipeline
-
-
+At its core, Semantica adds a **context and intelligence layer** on top of your existing AI stack: it doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs accountable.
---
## Knowledge Graphs
-The foundation of Semantica - turning data into structured knowledge.
+The foundation of everything in Semantica.
-### What is a Knowledge Graph?
+A knowledge graph stores information as:
-A knowledge graph represents real-world information as:
-- **Nodes** (entities): People, companies, locations, dates
-- **Edges** (relationships): works_for, located_in, founded_by
-- **Properties**: Name, date, confidence score, source
+- **Nodes (entities)** — people, companies, locations, events, concepts
+- **Edges (relationships)** — `works_for`, `located_in`, `founded_by`
+- **Properties** — name, date, confidence score, source URL
-### Why Knowledge Graphs?
-
-- **Searchable**: Find information instantly
-- **Connectable**: Discover hidden relationships
-- **Queryable**: Ask complex questions
-- **Explainable**: Trace answers back to sources
+This structure makes knowledge **searchable**, **connectable**, **queryable**, and — critically — **explainable**: every answer can be traced back to the facts and relationships that produced it.
---
## Entity Extraction (NER)
-Finding and classifying entities in text.
+Scanning text to find and classify real-world entities.
-### What it does:
-- Scans text for people, organizations, locations, dates
-- Classifies each entity by type
-- Assigns confidence scores
-- Tracks source provenance
-
-### Example Output:
```python
-# From: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."
+# Input: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."
{
"entities": [
- {"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98},
- {"text": "Steve Jobs", "type": "PERSON", "confidence": 0.99},
- {"text": "1976", "type": "DATE", "confidence": 0.95},
- {"text": "Cupertino", "type": "LOCATION", "confidence": 0.97}
+ {"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98},
+ {"text": "Steve Jobs", "type": "PERSON", "confidence": 0.99},
+ {"text": "1976", "type": "DATE", "confidence": 0.95},
+ {"text": "Cupertino", "type": "LOCATION", "confidence": 0.97}
]
}
```
+Each entity gets a type, confidence score, and a link to its source document.
+
---
## Relationship Extraction
-Finding connections between entities.
+Finding how entities connect to each other.
-### What it does:
-- Identifies how entities relate to each other
-- Extracts relationship types and directions
-- Provides context and confidence
-- Links to source documents
-
-### Example Output:
```python
{
"relationships": [
- {"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", "confidence": 0.92},
- {"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino", "confidence": 0.89}
+ {"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", "confidence": 0.92},
+ {"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino", "confidence": 0.89}
]
}
```
+Relationships can be extracted via rule-based methods, ML models, or LLMs (with `"llm_typed"` metadata).
+
---
## Embeddings
-Turning text into numerical vectors for AI understanding.
+Embeddings convert text into numerical vectors so that AI systems can measure semantic similarity — finding related concepts even when the exact words differ.
-### What are embeddings?
-- **Numerical representations** of text, entities, and relationships
-- **Similarity calculations** - find related concepts
-- **AI-powered search** - semantic understanding
-- **Clustering and grouping** - discover patterns
+Semantica uses embeddings for:
-### Use Cases:
-- **Semantic Search** - find documents by meaning, not keywords
-- **Entity Resolution** - match similar entities across sources
-- **Recommendations** - suggest related content
-- **AI Input** - provide structured context to LLMs
-
----
-
-## Temporal Graphs
-
-Knowledge graphs that understand time.
-
-### What they track:
-- **When** events happened
-- **How** entities changed over time
-- **Temporal relationships** - before, after, during
-- **Historical context** - point-in-time snapshots
-
-### Example Uses:
-- **Company History** - track mergers, leadership changes
-- **Person Careers** - job changes, relocations
-- **Policy Evolution** - law changes over time
-- **Research Progress** - scientific discoveries timeline
+- **Semantic search** — retrieve by meaning, not just keywords
+- **Entity resolution** — match the same entity across different sources
+- **Precedent search** — find similar past decisions
+- **GraphRAG retrieval** — hybrid vector + graph traversal
---
## GraphRAG
-Enhanced AI retrieval using knowledge graphs.
+GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses by grounding them in a structured knowledge graph rather than raw text chunks alone.
-### How it works:
-1. **Query** user question
-2. **Retrieve** relevant graph context
-3. **Enhance** with relationships and entities
-4. **Generate** AI response with sources
+How it works:
-### Benefits:
-- **More accurate** answers
-- **Source attribution** - trace answers back
-- **Context awareness** - understand relationships
-- **Reduced hallucination** - grounded in facts
+1. User submits a query
+2. Semantica retrieves relevant graph context (entities, relationships, reasoning paths)
+3. The LLM generates a response grounded in that context
+4. Every claim in the response links back to a source node in the graph
+
+This eliminates the hallucination and traceability problems of standard RAG.
---
## Ontology
-Defining the structure and rules of your knowledge.
+An ontology defines the schema and rules for your knowledge — what entity types exist, which relationships are valid, and what constraints apply.
-### What it provides:
-- **Schema definition** - what types exist
-- **Relationship rules** - valid connections
-- **Property constraints** - required fields
-- **Inheritance hierarchies** - parent-child relationships
-
-### Example:
```python
-# Define ontology structure
ontology = {
"classes": ["Person", "Organization", "Location"],
- "properties": ["name", "date", "confidence"],
- "relationships": ["works_for", "located_in", "born_in"],
+ "relationships": ["works_for", "located_in", "founded_by"],
"rules": {
- "Person": ["must_have_name", "can_have_birth_date"],
+ "Person": ["must_have_name"],
"Organization": ["must_have_name", "can_have_founding_date"]
}
}
```
+Semantica can auto-generate ontologies from your knowledge graph, or import existing OWL/RDF/Turtle ontologies.
+
---
## Reasoning & Inference
-Making logical deductions from your knowledge.
+Semantica includes multiple reasoning engines to derive new knowledge from existing facts.
-### What it can do:
-- **Infer missing facts** - derive new knowledge
-- **Detect inconsistencies** - find contradictions
-- **Apply rules** - automate decision making
-- **Explain reasoning** - show how conclusions were reached
+```
+Known: Steve Jobs founded Apple Inc.
+Known: Apple Inc. is headquartered in Cupertino
+Inferred: Steve Jobs has a connection to Cupertino
+```
-### Example:
-```
-Known: Steve Jobs founded Apple Inc.
-Known: Apple Inc. is headquartered in Cupertino
-Inferred: Steve Jobs has connection to Cupertino
-```
+Supported engines: forward chaining, Rete network, deductive, abductive, and SPARQL reasoning — all producing **explainable inference paths**, not black-box conclusions.
+
+---
+
+## Temporal Graphs
+
+Knowledge changes over time. Temporal graphs attach `valid_from` / `valid_until` windows to nodes and edges, enabling point-in-time queries and historical analysis.
+
+Common uses: tracking company leadership changes, policy evolution, research timelines, financial instrument histories.
---
## Deduplication & Entity Resolution
-Finding and merging duplicate entities.
+Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance.
-### What it does:
-- **Detects duplicates** - same entity, different names
-- **Merges information** - combine attributes
-- **Resolves conflicts** - handle contradictory data
-- **Maintains provenance** - track original sources
-
-### Example:
-```python
-# These refer to the same entity:
-"Apple Inc." → "Apple" → "Apple Computer Inc."
-# Merge into single entity with all attributes
-```
+Strategies: Jaro-Winkler similarity (v1), `blocking_v2`, `hybrid_v2`, `semantic_v2` (v2 — up to 7x faster).
---
-## Data Normalization
+## Provenance & Auditability
-Cleaning and standardizing your data.
+Every fact in Semantica links back to:
-### What it fixes:
-- **Format inconsistencies** - dates, names, numbers
-- **Canonical forms** - standard representations
-- **Data quality** - remove errors and noise
-- **Standardization** - consistent naming conventions
+- The source document it came from
+- The extraction method used
+- The ontology rules applied
+- The reasoning steps that produced any inference
-### Examples:
-- **Dates**: "Jan 1, 2020" → "2020-01-01"
-- **Names**: "Dr. Smith PhD" → "John Smith"
-- **Companies**: "Apple" → "Apple Inc."
-- **Locations**: "NYC" → "New York City"
+This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails.
---
## Conflict Detection
-Finding and resolving contradictory information.
+When multiple sources disagree on the same fact, Semantica flags and resolves the conflict rather than silently picking one value.
-### What it identifies:
-- **Factual conflicts** - different values for same fact
-- **Temporal conflicts** - impossible timelines
-- **Logical conflicts** - contradictory relationships
-- **Source reliability** - trustworthiness assessment
-
-### Resolution Strategies:
-- **Most recent** - prefer newer information
-- **Most reliable** - prefer trusted sources
-- **Majority vote** - go with consensus
-- **Manual review** - flag for human review
+Resolution strategies: prefer most recent, prefer most reliable source, majority vote, or flag for manual review.
---
-## Getting Started
+## Next Steps
-Ready to build your first knowledge graph?
-
-### Quick Start (5 minutes)
-```python
-from semantica.semantic_extract import NERExtractor
-from semantica.kg import GraphBuilder
-
-# Extract entities
-ner = NERExtractor()
-entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
-
-# Build graph
-kg = GraphBuilder().build({"entities": entities, "relationships": []})
-```
-
-### Learn More
-- **Getting Started Guide** - [Getting Started](getting-started.md)
-- **Cookbook Examples** - [Cookbook](cookbook.md)
-- **Module Documentation** - [Reference](reference/)
-- **Community Support** - [Community](community.md)
-
-### Common Use Cases
-- **Document Analysis** - extract knowledge from reports
-- **Research Assistant** - find connections in academic papers
-- **Business Intelligence** - analyze company relationships
-- **Regulatory Compliance** - track policy changes
-
----
-
-## Best Practices
-
-### Start Small
-- Begin with a single document type
-- Focus on specific entity types
-- Validate results before scaling
-
-### Configure Properly
-- Choose appropriate models for your domain
-- Set confidence thresholds
-- Define clear ontology rules
-
-### Validate Data
-- Check extraction quality
-- Review relationship accuracy
-- Test with known examples
-
-### Handle Errors
-- Implement error handling
-- Log processing issues
-- Provide feedback mechanisms
-
-### Optimize Performance
-- Use appropriate storage backends
-- Cache frequently accessed data
-- Monitor resource usage
-
-### Document Workflows
-- Record processing steps
-- Track data sources
-- Maintain change logs
-
----
-
-## Need Help?
-
-- **Documentation**: [Getting Started](getting-started.md)
-- **Examples**: [Cookbook](cookbook.md)
-- **Community**: [Discord](community.md)
-- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
-- **Support**: [Contact Us](community.md)
+- [Quickstart Tutorial](quickstart.md) — build a full pipeline with code
+- [Modules Guide](modules.md) — every module explained
+- [Use Cases](use-cases.md) — real-world domain examples
+- [API Reference](reference/core.md) — complete technical reference
diff --git a/docs/contributing.md b/docs/contributing.md
index b2441e3b..4b6f0e67 100644
--- a/docs/contributing.md
+++ b/docs/contributing.md
@@ -1,130 +1,99 @@
-# Contributing
+# Contributing
-**Help us build Semantica! Every contribution makes the project better.**
+Contributions of all kinds are welcome — code, documentation, tests, and community support.
---
-## Getting Started
+## Quick Start
-### Quick Start
-1. **Fork** the repository
-2. **Create** a feature branch
-3. **Make** your changes
-4. **Test** your changes
-5. **Submit** a pull request
+```bash
+# Fork the repo, then:
+git clone https://github.com/your-username/semantica.git
+cd semantica
+pip install -e ".[dev]"
+pytest
+```
-### First Contribution?
-Look for issues labeled [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) for beginner-friendly tasks.
+First time? Look for [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) labels for beginner-friendly tasks.
---
## Ways to Contribute
-### Code
-- **Fix bugs** - Resolve reported issues
-- **Add features** - Implement new functionality
-- **Improve performance** - Optimize existing code
-- **Refactor** - Clean up code structure
+**Code**
+- Fix bugs and resolve open issues
+- Implement new features or integrations
+- Optimize performance or refactor existing code
-### Documentation
-- **Fix typos** - Correct spelling and grammar
-- **Improve guides** - Make documentation clearer
-- **Add examples** - Provide practical code examples
-- **Update API docs** - Keep reference current
+**Documentation**
+- Fix typos, improve clarity, add examples
+- Write tutorials or domain-specific cookbook notebooks
+- Keep API reference up to date
-### Testing
-- **Write tests** - Add test coverage
-- **Fix tests** - Resolve test failures
-- **Report issues** - Identify bugs through testing
+**Testing**
+- Add test coverage for untested modules
+- Reproduce and confirm reported bugs
+- Improve test reliability
-### Community
-- **Help others** - Answer questions in issues
-- **Share knowledge** - Write tutorials and guides
-- **Provide feedback** - Review pull requests
+**Community**
+- Answer questions in issues and discussions
+- Review pull requests
+- Share Semantica in your blog posts or talks
---
## Reporting Issues
### Bug Reports
-When reporting bugs, include:
-- **Description** - What happened
-- **Steps to reproduce** - How to trigger the issue
-- **Expected behavior** - What should happen
-- **Environment** - Your setup details
+
+Include: what happened, steps to reproduce, expected behavior, and your environment (Python version, OS, Semantica version).
### Feature Requests
-When suggesting features, include:
-- **Use case** - Why you need this feature
-- **Proposed solution** - How it should work
-- **Benefits** - How it helps the community
+
+Include: your use case, what you'd like Semantica to do, and how it benefits others.
---
## Pull Request Guidelines
-### Before Submitting
-- **Test** your changes thoroughly
-- **Document** new features with examples
-- **Update** relevant documentation
-- **Follow** the existing code style
+Before submitting:
-### Pull Request Checklist
-- [ ] Code follows project style
-- [ ] Tests pass locally
-- [ ] Documentation is updated
-- [ ] Commit messages are clear
-- [ ] No merge conflicts
+- [ ] Tests pass locally (`pytest`)
+- [ ] New features are documented with examples
+- [ ] Code follows project style (Black, isort, flake8)
+- [ ] Commit messages are clear and descriptive
+- [ ] No unresolved merge conflicts
---
## Development Setup
-### Local Development
```bash
-# Clone your fork
git clone https://github.com/your-username/semantica.git
cd semantica
+pip install -e ".[dev]"
+```
-# Install in development mode
-pip install -e .[dev]
+Code style tools used: **Black** (formatting), **isort** (imports), **flake8** (linting).
-# Run tests
+Run the full test suite:
+
+```bash
pytest
```
-### Code Style
-We use standard Python formatting:
-- **Black** for code formatting
-- **isort** for import sorting
-- **flake8** for linting
+---
+
+## Community
+
+Please follow the [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md). Be respectful, patient, and constructive.
+
+All contributors are recognized in release notes and the GitHub contributors list.
---
-## Community Guidelines
+## Help
-### Code of Conduct
-Please follow our [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md).
-
-### Communication
-- **Be respectful** - Treat everyone with kindness
-- **Be helpful** - Assist others when you can
-- **Be patient** - Allow time for reviews
-- **Be constructive** - Provide helpful feedback
-
----
-
-## Recognition
-
-All contributors are recognized in:
-- **GitHub contributors** - Automatic recognition
-- **Release notes** - Notable contributions
-- **Community highlights** - Outstanding work
-
----
-
-## Need Help?
-
-- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Ask questions
-- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Community chat
-- **[Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md)** - Community standards
+- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
+- [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
+- [Discord](https://discord.gg/sV34vps5hH)
diff --git a/docs/cookbook.md b/docs/cookbook.md
index f561aeee..95defb06 100644
--- a/docs/cookbook.md
+++ b/docs/cookbook.md
@@ -1,108 +1,58 @@
-# 🍳 Semantica Cookbook
+# Semantica Cookbook
-Welcome to the **Semantica Cookbook**!
+Interactive Jupyter notebooks covering everything from your first knowledge graph to production GraphRAG systems.
-This collection of Jupyter notebooks is designed to take you from a beginner to an expert in building semantic AI applications. Whether you're looking for quick recipes or deep-dive tutorials, you'll find it here.
-
-!!! tip "How to use this Cookbook"
- - **Beginners**: Start with the [Core Tutorials](#core-tutorials) to learn the basics.
- - **Developers**: Check out [Advanced Concepts](#advanced-concepts) for deep dives into specific features.
- - **Architects**: Explore [Industry Use Cases](#industry-use-cases) for end-to-end solutions.
+!!! tip "Where to start"
+ - **New to Semantica** — begin with [Core Tutorials](#core-tutorials)
+ - **Building an application** — see [Advanced Concepts](#advanced-concepts) or [Industry Use Cases](#industry-use-cases)
+ - **Need installation help** — see the [Installation Guide](installation.md)
!!! note "Prerequisites"
- Before running these notebooks, ensure you have:
- - Python 3.8+ installed
- - A basic understanding of Python and Jupyter
- - An OpenAI API key (for most examples)
-
-!!! success "Installation"
- Install Semantica from PyPI (recommended):
-
- ```bash
- pip install semantica
- # Or with all optional dependencies:
- pip install semantica[all]
- ```
-
- For more installation options, see the [Installation Guide](installation.md).
+ Python 3.8+, Jupyter, and an OpenAI API key (for most examples).
---
-## � Featured Recipes
-
-Hand-picked tutorials to show you the power of Semantica.
+## Featured Recipes
-- :material-robot: **GraphRAG Complete**
- ---
- Build a production-ready Graph Retrieval Augmented Generation system.
-
- **Topics**: RAG, LLMs, Vector Search, Graph Traversal
-
- **Difficulty**: Advanced
-
- [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
-
-- :material-scale-balance: **RAG vs. GraphRAG Comparison**
- ---
- Side-by-side comparison of Standard RAG vs. GraphRAG using real-world data.
-
- **Topics**: RAG, GraphRAG, Benchmarking, Visualization
-
- **Difficulty**: Intermediate
-
- [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
-
-- :material-robot: **GraphRAG Complete**
- ---
- Build a production-ready Graph Retrieval Augmented Generation system.
-
- **New Features**: Graph Validation, Logical Inference, Hybrid Context.
-
- **Topics**: RAG, LLMs, Vector Search, Graph Traversal
-
- **Difficulty**: Advanced
-
- [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
-
-- :material-scale-balance: **RAG vs. GraphRAG Comparison**
- ---
- Side-by-side comparison of Standard RAG vs. GraphRAG using real-world data.
-
- **New Features**: Inference-Enhanced GraphRAG, Reasoning Gap Analysis.
-
- **Topics**: RAG, GraphRAG, Benchmarking, Visualization
-
- **Difficulty**: Intermediate
-
- [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
-
- :material-graph: **Your First Knowledge Graph**
---
Go from raw text to a queryable knowledge graph in 20 minutes.
-
- **Topics**: Extraction, Graph Construction, Visualization
-
- **Difficulty**: Beginner
-
+
+ **Topics**: Extraction, Graph Construction, Visualization · **Difficulty**: Beginner
+
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)
+- :material-robot: **GraphRAG Complete**
+ ---
+ Build a production-ready Graph Retrieval Augmented Generation system with hybrid retrieval and logical inference.
+
+ **Topics**: RAG, LLMs, Vector Search, Graph Traversal · **Difficulty**: Advanced
+
+ [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
+
+- :material-scale-balance: **RAG vs. GraphRAG Comparison**
+ ---
+ Side-by-side benchmark of standard RAG vs. GraphRAG on real-world data.
+
+ **Topics**: RAG, GraphRAG, Benchmarking, Reasoning Gap · **Difficulty**: Intermediate
+
+ [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
+
- :material-shield-alert: **Real-Time Anomaly Detection**
---
- Detect anomalies in streaming data using dynamic graphs.
-
- **Topics**: Streaming, Security, Dynamic Graphs
-
- **Difficulty**: Advanced
-
+ Detect anomalies in streaming data using dynamic knowledge graphs.
+
+ **Topics**: Streaming, Security, Dynamic Graphs · **Difficulty**: Advanced
+
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)
---
-## 🏁 Core Tutorials {#core-tutorials}
+## Core Tutorials {#core-tutorials}
Essential guides to master the Semantica framework.
@@ -210,7 +160,7 @@ Essential guides to master the Semantica framework.
---
-## 🧠 Advanced Concepts
+## Advanced Concepts
Deep dive into advanced features, customization, and complex workflows.
@@ -329,7 +279,7 @@ Deep dive into advanced features, customization, and complex workflows.
---
-## 🏭 Industry Use Cases {#industry-use-cases}
+## Industry Use Cases {#industry-use-cases}
Real-world examples and end-to-end applications across various industries.
@@ -497,7 +447,7 @@ Real-world examples and end-to-end applications across various industries.
---
-## 🛠️ How to Run
+## How to Run
To run these notebooks locally:
diff --git a/docs/css/custom.css b/docs/css/custom.css
index 079c651b..9b8b59ae 100644
--- a/docs/css/custom.css
+++ b/docs/css/custom.css
@@ -216,76 +216,92 @@ html {
border-left: 2px solid var(--md-accent-fg-color);
}
-/*
+/*
==========================================================================
Layout Optimization
- ==========================================================================
+ ==========================================================================
*/
-/* Reduce spacing between sidebars and content for all pages */
-.md-content__inner {
- padding-left: 0.75rem;
- padding-right: 0.75rem;
- margin-left: 0;
-}
-.md-content {
- margin-left: 0;
+/* Widen the overall grid */
+.md-grid {
+ max-width: 1440px;
+ margin-left: auto;
+ margin-right: auto;
padding-left: 0.5rem;
- padding-right: 0.5rem;
-}
-
-/* Reduce spacing before left sidebar for all pages */
-.md-sidebar {
- padding-left: 0.25rem;
- margin-left: 0;
}
+/* Narrow left sidebar to give content more room */
.md-sidebar--primary {
- padding-right: 0.5rem;
+ width: 11rem;
+ padding-right: 0.25rem;
padding-left: 0.25rem;
+}
+
+/* Right TOC sidebar */
+.md-sidebar--secondary {
+ width: 11rem;
+ padding-left: 0.5rem;
+ padding-right: 0;
margin-left: 0;
}
-/* Reduce spacing after right sidebar (table of contents) and shift it right slightly */
-.md-sidebar--secondary {
- padding-left: 1.25rem;
- padding-right: 0;
- margin-right: 0;
- margin-left: 3.5rem;
+.md-sidebar--secondary .md-nav {
+ width: 11rem;
}
-/* Reduce right edge spacing - similar to left */
-.md-container {
- padding-right: 0;
+/* Tighten TOC list spacing */
+.md-sidebar--secondary .md-nav__list {
+ padding-bottom: 1.5rem;
+ margin: 0;
+}
+
+.md-sidebar--secondary .md-nav__item {
+ padding: 0;
+ margin: 0;
+}
+
+.md-sidebar--secondary .md-nav__link {
+ white-space: normal;
+ word-break: break-word;
+ overflow: visible;
+ text-overflow: unset;
+ padding-top: 0.15rem;
+ padding-bottom: 0.15rem;
+ line-height: 1.4;
+ font-size: 0.7rem;
+ margin: 0;
+}
+
+/* Nested TOC items (h3, h4) */
+.md-sidebar--secondary .md-nav__item .md-nav__item .md-nav__link {
+ padding-left: 0.6rem;
+ font-size: 0.68rem;
+}
+
+/* Remove extra gap between TOC title and first item */
+.md-sidebar--secondary .md-nav__title {
+ margin-bottom: 0.25rem;
+ padding-bottom: 0.25rem;
+}
+
+/* Give the main content area maximum available width */
+.md-content {
+ max-width: none;
+ padding-left: 1rem;
+ padding-right: 1rem;
+}
+
+.md-content__inner {
+ max-width: none;
+ padding-left: 1rem;
+ padding-right: 1rem;
+ margin-left: 0;
margin-right: 0;
}
-.md-main {
- margin-right: 0;
- padding-right: 0;
-}
-
-/* Reduce margins of the main container */
.md-main__inner {
margin-left: 0;
margin-right: 0;
- padding-right: 0;
-}
-
-/* Reduce right edge spacing on body/html */
-body {
- margin-right: 0;
- padding-right: 0;
-}
-
-html {
- margin-right: 0;
- padding-right: 0;
-}
-
-.md-grid {
- margin-left: 0;
- padding-left: 0.5rem;
}
/* Ensure text content is left-aligned by default */
diff --git a/docs/deep-dive.md b/docs/deep-dive.md
index 53ada134..3a6ea484 100644
--- a/docs/deep-dive.md
+++ b/docs/deep-dive.md
@@ -1,10 +1,15 @@
# Deep Dive
-Advanced topics, architecture, and internals of Semantica.
+Internals, advanced concepts, and extension points for contributors and power users.
-## Architecture Overview
+!!! tip "Just getting started?"
+ Read [Architecture](architecture.md) for a higher-level overview first.
-Semantica follows a modular, extensible architecture:
+---
+
+## Pipeline Internals
+
+The full data flow through a Semantica pipeline:
```mermaid
graph TB
@@ -16,65 +21,79 @@ graph TB
F --> G[Knowledge Graph Builder]
G --> H[Embedding Generator]
H --> I[Export Layer]
-
+
D --> D1[Entity Extractor]
D --> D2[Relationship Extractor]
D --> D3[Triplet Extractor]
-
+
G --> G1[Graph Validator]
G --> G2[Graph Analyzer]
-
+
H --> H1[Text Embeddings]
H --> H2[Graph Embeddings]
```
+### Sequence Diagram
+
+```mermaid
+sequenceDiagram
+ participant User
+ participant Semantica
+ participant Ingestor
+ participant Parser
+ participant Extractor
+ participant Resolver
+ participant GraphBuilder
+ participant Exporter
+
+ User->>Semantica: build_knowledge_base(sources)
+ Semantica->>Ingestor: ingest(sources)
+ Ingestor->>Parser: parse(documents)
+ Parser->>Extractor: extract(text)
+ Extractor->>Resolver: resolve_conflicts(entities)
+ Resolver->>GraphBuilder: build_graph(resolved_data)
+ GraphBuilder->>Exporter: export(graph)
+ Exporter->>User: return result
+```
+
+---
+
## System Components
-### 1. Ingestion Layer
+### Ingestion Layer
-Handles data input from various sources:
+Handles input from any source:
-- **File Ingestor**: PDF, DOCX, HTML, JSON, CSV
-- **Web Ingestor**: URLs, web scraping
-- **Database Ingestor**: SQL databases
-- **Stream Ingestor**: Real-time data streams
+- **FileIngestor** — PDF, DOCX, HTML, JSON, CSV, archives
+- **WebIngestor** — URL crawling and scraping
+- **DBIngestor** / **SnowflakeIngestor** — SQL databases
+- **StreamIngestor** — Kafka and real-time feeds
-### 2. Parsing Layer
+### Parsing Layer
-Converts raw data into structured format:
+Converts raw data to structured text:
-- Document parsing (PDF, Word, etc.)
-- Text extraction
-- Metadata extraction
-- Format normalization
+- Text and metadata extraction from documents
+- OCR for scanned content
+- Layout analysis (via Docling for tables and columns)
-### 3. Extraction Layer
+### Extraction Layer
-Core semantic extraction:
+Core semantic processing pipeline:
-```python
-# Entity extraction pipeline
+```
text → Tokenization → NER → Entity Linking → Entity Validation
```
-**Components:**
-- Named Entity Recognition (NER)
-- Relationship Extraction
-- Triplet Extraction
-- Coreference Resolution
+Components: Named Entity Recognition, Relationship Extraction, Triplet Extraction, Coreference Resolution.
-### 4. Normalization Layer
+### Normalization Layer
-Standardizes extracted data:
+Standardizes extracted data: entity names, date formats, numbers, encodings, and language normalization.
-- Entity normalization
-- Date/time normalization
-- Number normalization
-- Text cleaning
+### Conflict Resolution
-### 5. Conflict Resolution
-
-Handles conflicting information:
+Handles contradictory facts from multiple sources:
```mermaid
graph LR
@@ -88,64 +107,28 @@ graph LR
E --> H
F --> H
G --> H
-
- style A fill:#ffebee
- style H fill:#c8e6c9
- style C fill:#fff9c4
```
-### 6. Knowledge Graph Builder
+### Knowledge Graph Builder
-Constructs the knowledge graph:
+- Entity resolution across sources
+- Edge creation (typed relationships)
+- Property assignment with confidence scores
+- Graph validation and quality checks
-- Node creation (entities)
-- Edge creation (relationships)
-- Property assignment
-- Graph validation
-- Quality checks
+### Embedding Generator
-### 7. Embedding Generator
+- Text embeddings (Sentence-Transformers, FastEmbed, OpenAI, BGE)
+- Graph embeddings (Node2Vec, GraphSAGE)
-Generates vector representations:
-
-- Text embeddings (sentence transformers)
-- Graph embeddings (node2vec, GraphSAGE)
-- Multimodal embeddings
-
-## Data Flow
-
-```mermaid
-sequenceDiagram
- participant User
- participant Semantica
- participant Ingestor
- participant Parser
- participant Extractor
- participant Resolver
- participant GraphBuilder
- participant Exporter
-
- User->>Semantica: build_knowledge_base(sources)
- Semantica->>Ingestor: ingest(sources)
- Ingestor->>Parser: parse(documents)
- Parser->>Extractor: extract(text)
- Extractor->>Resolver: resolve_conflicts(entities)
- Resolver->>GraphBuilder: build_graph(resolved_data)
- GraphBuilder->>Exporter: export(graph)
- Exporter->>User: return result
-
- Note over User,Exporter: Complete pipeline execution
-```
+---
## Advanced Concepts
-### Entity Resolution
-
-Matching entities across sources:
+### Entity Resolution Algorithm
```python
-# Entity resolution algorithm
-def resolve_entities(entities):
+def resolve_entities(entities, threshold=0.85):
clusters = []
for entity in entities:
matched = False
@@ -161,123 +144,92 @@ def resolve_entities(entities):
### Relationship Inference
-Inferring implicit relationships:
+Semantica's reasoning engines can derive implicit relationships:
-- Transitive relationships
-- Temporal relationships
-- Causal relationships
-- Hierarchical relationships
+- **Transitive** — if A→B and B→C, infer A→C
+- **Temporal** — before, after, during from timestamped facts
+- **Causal** — IF/THEN rules via `Reasoner`
+- **Hierarchical** — subclass/instance inference via `OntologyReasoner`
-### Graph Optimization
-
-Optimizing knowledge graph structure:
-
-- Node deduplication
-- Edge consolidation
-- Path compression
-- Index optimization
-
-## Performance Considerations
-
-### Scalability
-
-- **Horizontal Scaling**: Process multiple documents in parallel
-- **Vertical Scaling**: Use GPU acceleration
-- **Caching**: Cache embeddings and parsed documents
-- **Lazy Loading**: Load components on demand
-
-### Memory Management
+### Batch Processing for Large Datasets
```python
-# Process large datasets efficiently
def process_large_dataset(sources, batch_size=100):
for i in range(0, len(sources), batch_size):
- batch = sources[i:i+batch_size]
+ batch = sources[i : i + batch_size]
result = semantica.build_knowledge_base(batch)
- # Save and clear memory
save_result(result)
del result
gc.collect()
```
+---
+
## Extension Points
-### Custom Plugins
-
-Create custom plugins:
+### Custom Plugin
```python
from semantica.core import Plugin
class CustomPlugin(Plugin):
def process(self, data):
- # Your custom processing
+ # Your custom processing logic
return processed_data
```
-### Custom Extractors
-
-Implement custom extractors:
+### Custom Extractor
```python
from semantica.semantic_extract import BaseExtractor
class DomainSpecificExtractor(BaseExtractor):
- def extract_entities(self, text):
- # Domain-specific extraction logic
+ def extract(self, text):
+ # Domain-specific entity extraction logic
return entities
```
-## Internal APIs
+### Custom Ingestor
-### Core APIs
+```python
+from semantica.ingest import BaseIngestor
-- `Semantica.build_knowledge_base()` - Main entry point
-- `KGBuilder.build()` - Graph construction
-- `ConflictResolver.resolve()` - Conflict resolution
-- `EmbeddingGenerator.generate()` - Embedding generation
-
-### Extension APIs
-
-- Plugin registration
-- Custom extractor registration
-- Custom exporter registration
-- Event hooks
-
-## Design Decisions
-
-### Why Modular Architecture?
-
-- **Extensibility**: Easy to add new features
-- **Testability**: Components can be tested independently
-- **Maintainability**: Clear separation of concerns
-- **Flexibility**: Swap implementations easily
-
-### Why Conflict Resolution?
-
-- **Data Quality**: Ensures consistent knowledge
-- **Multi-Source**: Handles conflicting information
-- **Flexibility**: Multiple resolution strategies
-- **Transparency**: Track resolution decisions
-
-## Future Enhancements
-
-Planned improvements:
-
-- Distributed processing
-- Real-time streaming
-- Advanced reasoning
-- Multi-modal support expansion
-- Enhanced visualization
-
-## Contributing to Core
-
-Interested in contributing to Semantica's core? See our [Contributing Guide](https://github.com/Hawksight-AI/semantica/blob/main/CONTRIBUTING.md).
+class CustomIngestor(BaseIngestor):
+ def ingest(self, source):
+ # Load and return document dicts
+ return documents
+```
---
-For more information:
-- **[API Reference](reference/core.md) - Detailed API documentation
-- **[Learning More](learning-more.md)** - Additional resources
-- **[GitHub Repository](https://github.com/Hawksight-AI/semantica)** - Source code
+## Internal APIs
+| API | Purpose |
+|-----|---------|
+| `Semantica.build_knowledge_base()` | Main orchestration entry point |
+| `GraphBuilder.build()` | Graph construction |
+| `ConflictResolver.resolve()` | Conflict resolution |
+| `EmbeddingGenerator.generate()` | Embedding generation |
+
+Extension hooks: plugin registration, custom extractor registration, custom exporter registration, event hooks.
+
+---
+
+## Design Decisions
+
+**Why modular architecture?** Each component is independently testable and swappable. You can use `NERExtractor` alone without pulling in graph storage or pipelines.
+
+**Why built-in conflict resolution?** Multi-source data always has contradictions. Ignoring them produces garbage graphs. Explicit resolution strategies give you control over data quality.
+
+**Why W3C PROV-O for provenance?** It's an industry standard with tooling support. Using a custom format would make lineage data non-portable.
+
+**Why multiple reasoning engines?** Different problems need different reasoning: forward chaining for rule application, SPARQL for graph queries, abductive for hypothesis generation. No single engine fits all cases.
+
+---
+
+## Further Reading
+
+- [Architecture](architecture.md) — high-level three-layer overview
+- [Modules](modules.md) — every module with code examples
+- [API Reference](reference/core.md) — complete technical reference
+- [Contributing](contributing.md) — how to extend the framework
diff --git a/docs/examples.md b/docs/examples.md
index 48757e7d..7f2c6bfa 100644
--- a/docs/examples.md
+++ b/docs/examples.md
@@ -1,306 +1,134 @@
# Examples
-Real-world examples and use cases for Semantica.
-
-!!! tip "Interactive Learning"
- For hands-on interactive tutorials, check out our [Cookbook](cookbook.md) with Jupyter notebooks covering everything from basics to advanced use cases.
+Code examples organized by complexity. For interactive notebooks, see the [Cookbook](cookbook.md).
---
-## Example Gallery
+## Beginner
-
+### Basic Knowledge Graph
-- :material-school: **Getting Started**
- ---
- Quick examples to get you up and running in 5 minutes.
-
- [View Examples](#getting-started-5-min-examples)
+```python
+from semantica.ingest import FileIngestor
+from semantica.parse import DocumentParser
+from semantica.semantic_extract import NERExtractor, RelationExtractor
+from semantica.kg import GraphBuilder
-- :material-cogs: **Core Workflows**
- ---
- Common workflows for building production-ready graphs.
-
- [View Examples](#core-workflows-15-min-examples)
-
-- :material-rocket: **Advanced Patterns**
- ---
- Complex use cases and production deployments.
-
- [View Examples](#advanced-patterns-30-min-examples)
-
-- :material-factory: **Production Patterns**
- ---
- Scalable deployment patterns for enterprise use.
-
- [View Examples](#production-patterns)
-
-
-
----
-
-## Getting Started (5 min examples)
-
-### Example 1: Basic Knowledge Graph
-
-**Difficulty**: Beginner
-
-Build a knowledge graph from a single document using Semantica's modular approach. This example demonstrates the complete workflow from document ingestion to graph construction.
-
-**What it demonstrates:**
-- Document ingestion and parsing
-- Entity and relationship extraction
-- Knowledge graph construction
-
-**For complete step-by-step examples, see:**
-- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Complete walkthrough
- - **Topics**: Ingestion, parsing, extraction, graph building
- - **Difficulty**: Beginner
- - **Time**: 20-30 minutes
- - **Use Cases**: Learning the complete workflow
-
-### Example 2: Entity Extraction
-
-**Difficulty**: Beginner
-
-Extract entities from text using Named Entity Recognition. This example shows how to identify and classify named entities in text.
-
-**What it demonstrates:**
-- Named Entity Recognition (NER)
-- Entity type classification
-- Confidence scoring
-
-**For complete examples, see:**
-- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn entity extraction
- - **Topics**: NER methods, entity types, extraction techniques
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Understanding entity extraction
- print(f"{entity.text}: {entity.label}")
-```
-
-**Expected Output:**
-```
-Apple Inc.: ORGANIZATION
-Steve Jobs: PERSON
-```
-
-### Example 3: Multi-Source Integration
-
-**Difficulty**: Beginner
-
-Combine data from multiple sources into a unified knowledge graph. This example demonstrates integrating data from diverse sources.
-
-**What it demonstrates:**
-- Multi-source data ingestion
-- Entity merging and resolution
-- Unified graph construction
-
-**For complete examples, see:**
-- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration patterns
- - **Topics**: Multi-source integration, entity resolution, conflict handling
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Building unified knowledge graphs from diverse sources
-
----
-
-## Core Workflows (15 min examples)
-
-### Example 4: Conflict Resolution
-
-**Difficulty**: Intermediate
-
-Resolve conflicts in data from multiple sources. This example shows how to identify and resolve conflicting information.
-
-**What it demonstrates:**
-- Conflict detection
-- Conflict resolution strategies
-- Data quality assurance
-
-**For complete examples, see:**
-- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Conflict resolution patterns
- - **Topics**: Conflict detection, resolution strategies, data quality
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Data integration, quality assurance
ingestor = FileIngestor()
-parser = DocumentParser()
-ner = NERExtractor()
+parser = DocumentParser()
+ner = NERExtractor()
+rel = RelationExtractor()
-all_entities = []
-for source in ["source1.pdf", "source2.pdf"]:
- doc = ingestor.ingest_file(source)
- parsed = parser.parse_document(source)
- text = parsed.get("full_text", "")
- entities = ner.extract_entities(text)
- all_entities.extend(entities)
+sources = ingestor.ingest("data/sample.pdf")
+parsed = parser.parse(sources[0])
+
+entities = ner.extract(parsed)
+relationships = rel.extract(parsed, entities=entities)
+
+kg = GraphBuilder(merge_entities=True).build(
+ entities=entities, relationships=relationships
+)
+print(f"{len(kg.nodes)} nodes, {len(kg.edges)} edges")
+```
+
+### Entity Extraction from Text
+
+```python
+from semantica.semantic_extract import NERExtractor
+
+ner = NERExtractor()
+entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
+
+for entity in entities:
+ print(f"{entity['text']}: {entity['type']}")
+# Apple Inc.: ORGANIZATION
+# Steve Jobs: PERSON
+# 1976: DATE
+```
+
+### Custom NER Configuration
+
+```python
+from semantica.semantic_extract import NERExtractor
+
+ner = NERExtractor(
+ method="llm",
+ provider="openai",
+ model="gpt-4",
+ confidence_threshold=0.8,
+ temperature=0.0,
+)
+entities = ner.extract("Your document text here...")
+```
+
+---
+
+## Intermediate
+
+### Multi-Source Integration
+
+```python
+from semantica.ingest import FileIngestor
+from semantica.parse import DocumentParser
+from semantica.semantic_extract import NERExtractor, RelationExtractor
+from semantica.kg import GraphBuilder
+
+ingestor = FileIngestor()
+parser = DocumentParser()
+ner = NERExtractor()
+rel = RelationExtractor()
+builder = GraphBuilder(merge_entities=True)
+
+all_entities, all_rels = [], []
+
+for path in ["source1.pdf", "source2.pdf", "source3.pdf"]:
+ sources = ingestor.ingest(path)
+ parsed = parser.parse(sources[0])
+ all_entities.extend(ner.extract(parsed))
+ all_rels.extend(rel.extract(parsed, entities=all_entities))
+
+kg = builder.build(entities=all_entities, relationships=all_rels)
+print(f"Unified graph: {len(kg.nodes)} nodes, {len(kg.edges)} edges")
+```
+
+### Conflict Detection and Resolution
+
+```python
+from semantica.conflicts import ConflictDetector, ConflictResolver
-# Detect and resolve conflicts
detector = ConflictDetector()
conflicts = detector.detect_conflicts(all_entities)
resolver = ConflictResolver(default_strategy="voting")
resolved = resolver.resolve_conflicts(conflicts)
-print(f"Detected {len(conflicts)} conflicts")
-print(f"Resolved {len(resolved)} conflicts")
+print(f"Detected {len(conflicts)} conflicts, resolved {len(resolved)}")
```
-### Example 5: Custom Entity Extraction Configuration
-
-**Difficulty**: Intermediate
-
-Use custom configuration for entity extraction with specific models and thresholds.
-
-```python
-from semantica.semantic_extract import NERExtractor
-from semantica.kg import GraphBuilder
-
-# Use LLM-based extraction with custom configuration
-ner = NERExtractor(
- method="llm",
- provider="openai",
- model="gpt-4",
- confidence_threshold=0.8,
- temperature=0.0
-)
-
-text = "Your document text here..."
-entities = ner.extract_entities(text)
-
-# Build graph with custom merge settings
-builder = GraphBuilder(
- merge_entities=True,
- merge_threshold=0.9
-)
-kg = builder.build_graph(entities=entities, relationships=[])
-```
-
-### Example 6: Incremental Graph Building
-
-**Difficulty**: Intermediate
-
-Build knowledge graph incrementally from multiple sources.
-
-```python
-from semantica.ingest import FileIngestor
-from semantica.parse import DocumentParser
-from semantica.semantic_extract import NERExtractor, RelationExtractor
-from semantica.kg import GraphBuilder, GraphMerger
-
-def build_kg_from_source(source_path):
- """Helper function to build a knowledge graph from a single source."""
- ingestor = FileIngestor()
- parser = DocumentParser()
- ner = NERExtractor()
- rel_extractor = RelationExtractor()
-
- doc = ingestor.ingest_file(source_path)
- parsed = parser.parse_document(source_path)
- text = parsed.get("full_text", "")
-
- entities = ner.extract_entities(text)
- relationships = rel_extractor.extract_relations(text, entities=entities)
-
- builder = GraphBuilder()
- return builder.build_graph(entities=entities, relationships=relationships)
-
-# Build graphs separately
-kg1 = build_kg_from_source("source1.pdf")
-kg2 = build_kg_from_source("source2.pdf")
-
-# Merge into unified graph
-merger = GraphMerger()
-merged_kg = merger.merge([kg1, kg2])
-
-print(f"Merged graph: {len(merged_kg.nodes)} nodes, {len(merged_kg.edges)} edges")
-```
-
----
-
-## Advanced Patterns (30+ min examples)
-
-### Example 7: Graph Visualization
-
-**Difficulty**: Beginner
-
-Visualize your knowledge graph to understand entity relationships.
-
-```python
-from semantica.ingest import FileIngestor
-from semantica.parse import DocumentParser
-from semantica.semantic_extract import NERExtractor, RelationExtractor
-from semantica.kg import GraphBuilder
-from semantica.visualization import KGVisualizer
-
-# Build a small graph
-ingestor = FileIngestor()
-parser = DocumentParser()
-ner = NERExtractor()
-rel_extractor = RelationExtractor()
-
-doc = ingestor.ingest_file("semantica_intro.pdf")
-parsed = parser.parse_document("semantica_intro.pdf")
-text = parsed.get("full_text", "")
-
-entities = ner.extract_entities(text)
-relationships = rel_extractor.extract_relations(text, entities=entities)
-
-builder = GraphBuilder()
-kg = builder.build_graph(entities=entities, relationships=relationships)
-
-# Visualize
-viz = KGVisualizer()
-viz.visualize_network(kg, output="html", file_path="semantica_knowledge_map.html")
-print("Visualization saved to semantica_knowledge_map.html")
-```
-
----
-
-## Advanced Patterns (30+ min examples)
-
-### Example 8: Persistent Storage (Neo4j)
-
-**Difficulty**: Intermediate
-
-Store and query knowledge graphs in a persistent graph database.
+### Persistent Storage (Neo4j)
```python
from semantica.graph_store import GraphStore
-# Initialize with Neo4j
store = GraphStore(
backend="neo4j",
uri="bolt://localhost:7687",
user="neo4j",
- password="password"
+ password="password",
)
store.connect()
-# Create nodes and relationships
-apple = store.create_node(
- labels=["Company"],
- properties={"name": "Apple Inc."}
-)
-tim = store.create_node(
- labels=["Person"],
- properties={"name": "Tim Cook"}
-)
+apple = store.create_node(labels=["Company"], properties={"name": "Apple Inc."})
+tim = store.create_node(labels=["Person"], properties={"name": "Tim Cook"})
store.create_relationship(
start_node_id=tim["id"],
end_node_id=apple["id"],
- rel_type="CEO_OF"
+ rel_type="CEO_OF",
)
-
store.close()
```
-### Example 9: FalkorDB for Real-Time Applications
-
-**Difficulty**: Intermediate
-
-Ultra-fast graph queries for LLM applications using FalkorDB.
+### FalkorDB (High-Speed Queries)
```python
from semantica.graph_store import GraphStore
@@ -309,105 +137,49 @@ store = GraphStore(
backend="falkordb",
host="localhost",
port=6379,
- graph_name="knowledge_graph"
+ graph_name="knowledge_graph",
)
store.connect()
-
-# Fast queries
-results = store.execute_query("MATCH (n)-[r]->(m) WHERE n.name CONTAINS 'AI' RETURN n")
+results = store.execute_query(
+ "MATCH (n)-[r]->(m) WHERE n.name CONTAINS 'AI' RETURN n"
+)
store.close()
```
-### Example 10: GraphRAG (Knowledge-Powered Retrieval)
+---
-**Difficulty**: Advanced
+## Advanced
-Build a production-ready GraphRAG system with logical inference and hybrid retrieval.
+### GraphRAG with Reasoning
```python
from semantica.context import AgentContext
from semantica.reasoning import Reasoner
-# 1. Initialize context with GraphRAG (Hybrid Retrieval)
context = AgentContext(
- vector_store=vs,
+ vector_store=vs,
knowledge_graph=kg,
graph_expansion=True,
- hybrid_alpha=0.7
+ hybrid_alpha=0.7,
)
-# 2. Enrich Knowledge Graph using Logical Reasoning
reasoner = Reasoner()
-
-# Add a rule to categorize technology stack items
reasoner.add_rule("IF Library(?x) AND Language(?y) THEN TechStackItem(?x)")
+inferred = reasoner.infer_facts(kg.get_all_triplets())
-# Infer new facts from the existing graph
-all_facts = kg.get_all_triplets()
-inferred = reasoner.infer_facts(all_facts)
+for fact in inferred:
+ kg.add_fact_from_string(fact)
-# Add inferred knowledge back to the graph
-for fact_str in inferred:
- kg.add_fact_from_string(fact_str)
-
-# 3. Retrieve context for a query (now with enriched knowledge)
results = context.retrieve("What technologies are used in this project?")
```
-[**View Complete GraphRAG Tutorial**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
-
-### Example 11: RAG vs. GraphRAG Comparison
-
-**Difficulty**: Intermediate
-
-Benchmark standard Vector RAG against Graph-enhanced retrieval.
-
-[**View RAG vs. GraphRAG Comparison**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
+[Full GraphRAG tutorial](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) · [RAG vs. GraphRAG comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
---
-## Production Patterns
+## Production
-### Example 12: Streaming Data Processing
-
-**Difficulty**: Advanced
-
-Process data streams in real-time.
-
-```python
-from semantica.ingest import StreamIngestor
-from semantica.parse import DocumentParser
-from semantica.semantic_extract import NERExtractor, RelationExtractor
-from semantica.kg import GraphBuilder
-
-stream_ingestor = StreamIngestor(stream_uri="kafka://localhost:9092/topic")
-parser = DocumentParser()
-ner = NERExtractor()
-rel_extractor = RelationExtractor()
-builder = GraphBuilder()
-
-for batch in stream_ingestor.stream(batch_size=100):
- all_entities = []
- all_relationships = []
-
- for item in batch:
- text = str(item) # Convert stream item to text
- entities = ner.extract_entities(text)
- relationships = rel_extractor.extract_relations(text, entities=entities)
- all_entities.extend(entities)
- all_relationships.extend(relationships)
-
- # Build graph from batch
- kg = builder.build_graph(entities=all_entities, relationships=all_relationships)
- # Process results
- print(f"Processed batch: {len(kg.nodes)} nodes")
-```
-
-### Example 13: Batch Processing Large Datasets
-
-**Difficulty**: Intermediate
-
-Process large datasets efficiently with batching.
+### Batch Processing (Large Datasets)
```python
from semantica.ingest import FileIngestor
@@ -416,75 +188,57 @@ from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
ingestor = FileIngestor()
-parser = DocumentParser()
-ner = NERExtractor()
-rel_extractor = RelationExtractor()
-builder = GraphBuilder()
+parser = DocumentParser()
+ner = NERExtractor()
+rel = RelationExtractor()
+builder = GraphBuilder()
-sources = [f"data/doc_{i}.pdf" for i in range(1000)]
+sources = [f"data/doc_{i}.pdf" for i in range(1000)]
batch_size = 50
for i in range(0, len(sources), batch_size):
- batch = sources[i:i+batch_size]
-
- all_entities = []
- all_relationships = []
-
- for source in batch:
- doc = ingestor.ingest_file(source)
- parsed = parser.parse_document(source)
- text = parsed.get("full_text", "")
-
- entities = ner.extract_entities(text)
- relationships = rel_extractor.extract_relations(text, entities=entities)
-
- all_entities.extend(entities)
- all_relationships.extend(relationships)
-
- # Build graph from batch
- kg = builder.build_graph(entities=all_entities, relationships=all_relationships)
-
- # Save intermediate results
- print(f"Processed batch {i//batch_size + 1}: {len(kg.nodes)} nodes")
+ batch = sources[i : i + batch_size]
+ all_entities, all_rels = [], []
+
+ for path in batch:
+ parsed = parser.parse(ingestor.ingest(path)[0])
+ all_entities.extend(ner.extract(parsed))
+ all_rels.extend(rel.extract(parsed, entities=all_entities))
+
+ kg = builder.build(entities=all_entities, relationships=all_rels)
+ print(f"Batch {i // batch_size + 1}: {len(kg.nodes)} nodes")
+```
+
+### Real-Time Streaming
+
+```python
+from semantica.ingest import StreamIngestor
+from semantica.semantic_extract import NERExtractor, RelationExtractor
+from semantica.kg import GraphBuilder
+
+stream = StreamIngestor(stream_uri="kafka://localhost:9092/topic")
+ner = NERExtractor()
+rel = RelationExtractor()
+builder = GraphBuilder()
+
+for batch in stream.stream(batch_size=100):
+ all_entities, all_rels = [], []
+ for item in batch:
+ text = str(item)
+ all_entities.extend(ner.extract(text))
+ all_rels.extend(rel.extract(text, entities=all_entities))
+ kg = builder.build(entities=all_entities, relationships=all_rels)
+ print(f"Processed batch: {len(kg.nodes)} nodes")
```
---
## More Resources
-- **[Quick Start Guide](quickstart.md)** - Step-by-step tutorial
-- **[API Reference](reference/core.md)** - Complete API documentation
-- **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks
-- **[Use Cases](use-cases.md)** - Real-world applications
-
-### 🍳 Recommended Cookbook Tutorials
-
-- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules
- - **Topics**: Framework overview, all modules, architecture, configuration
- - **Difficulty**: Beginner
- - **Time**: 30-45 minutes
- - **Use Cases**: First-time users, understanding the framework
-
-- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph
- - **Topics**: Entity extraction, relationship extraction, graph construction, visualization
- - **Difficulty**: Beginner
- - **Time**: 20-30 minutes
- - **Use Cases**: Learning the basics, quick start
-
-- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production-ready GraphRAG system
- - **Topics**: GraphRAG, hybrid retrieval, vector search, graph traversal, LLM integration
- - **Difficulty**: Advanced
- - **Time**: 1-2 hours
- - **Use Cases**: Production RAG applications
-
-- **[RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)**: Benchmark standard RAG vs GraphRAG
- - **Topics**: RAG, GraphRAG, benchmarking, visualization, reasoning gap
- - **Difficulty**: Intermediate
- - **Time**: 45-60 minutes
- - **Use Cases**: Understanding GraphRAG advantages, choosing the right approach
-
----
-
-!!! info "Contribute"
- Have an example to share? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica)
+- [Quickstart Tutorial](quickstart.md) — step-by-step first pipeline
+- [Cookbook](cookbook.md) — interactive Jupyter notebooks
+- [Use Cases](use-cases.md) — domain-specific examples
+- [API Reference](reference/core.md) — complete API documentation
+!!! info "Have an example to share?"
+ [Contribute on GitHub](https://github.com/Hawksight-AI/semantica)
diff --git a/docs/faq.md b/docs/faq.md
index cf96b220..ff5bfc31 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -1,148 +1,149 @@
-# Frequently Asked Questions
+# Frequently Asked Questions
-**Common questions about Semantica and how to use it.**
+Common questions about Semantica. Use Ctrl+F to find what you need.
---
## General
### What is Semantica?
-Semantica is an open-source framework for building knowledge graphs from unstructured data. It transforms documents, web pages, and databases into structured, queryable knowledge.
-### What can I do with Semantica?
-- **Build knowledge graphs** from documents and data
-- **Extract entities and relationships** automatically
-- **Power AI applications** with structured knowledge
-- **Create semantic search** and GraphRAG systems
-- **Integrate multiple data sources** into unified graphs
+Semantica is an open-source framework for building context graphs and decision intelligence layers for AI. It transforms unstructured data — documents, APIs, databases — into structured knowledge graphs with full provenance tracking, making AI systems explainable and auditable.
+
+### What can I build with Semantica?
+
+- Knowledge graphs from documents and multi-source data
+- GraphRAG systems with graph-grounded retrieval
+- AI agents with structured decision history and semantic memory
+- Compliance-ready pipelines with W3C PROV-O lineage
+
+### What makes Semantica different from other frameworks?
+
+Most frameworks stop at retrieval or generation. Semantica adds an **accountability layer**: every decision is recorded, every fact links to a source, and every reasoning step is explainable. It's designed for environments where you need to audit why an AI reached a conclusion.
### Is Semantica free?
-Yes! Semantica is open source under the MIT License.
-### What makes Semantica different?
-- **Modular architecture** - Use only what you need
-- **Production-ready** - Built for scale and reliability
-- **Extensible** - Add custom models and components
-- **Open source** - Transparent and community-driven
+Yes — MIT licensed, no vendor lock-in. Some features require third-party API keys (e.g., OpenAI embeddings), but Semantica itself is free.
---
## Installation
### How do I install Semantica?
+
```bash
pip install semantica
```
+See [Installation](installation.md) for virtual environment setup, optional extras, and troubleshooting.
+
### What Python version do I need?
-Python 3.8 or higher. Python 3.11+ is recommended.
+
+Python 3.8 or higher. Python 3.11+ is recommended for best performance.
### What are the system requirements?
+
- Python 3.8+
-- 4GB+ RAM for basic use
-- Optional GPU for embeddings and ML models
+- 4 GB RAM minimum; 16 GB+ recommended for larger graphs
+- Optional GPU for embedding generation and ML inference
---
## Getting Started
-### How do I start using Semantica?
-```python
-from semantica.semantic_extract import NERExtractor
-from semantica.kg import GraphBuilder
+### Where do I start?
-# Extract entities
-ner = NERExtractor()
-entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
+1. [Installation](installation.md) — get set up
+2. [Getting Started](getting-started.md) — core concepts and first example
+3. [Quickstart Tutorial](quickstart.md) — full step-by-step pipeline
+4. [Cookbook](cookbook.md) — interactive Jupyter notebooks
-# Build knowledge graph
-kg = GraphBuilder().build({"entities": entities})
-```
+### What data sources does Semantica support?
-### Where can I find examples?
-- **[Getting Started Guide](getting-started.md)** - Quick introduction
-- **[Cookbook](cookbook.md)** - Practical examples
-- **[GitHub Examples](https://github.com/Hawksight-AI/semantica/tree/main/examples)** - Code samples
+- **Files** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives
+- **Web** — crawl with `WebIngestor`, RSS feeds
+- **Databases** — PostgreSQL, MySQL, Snowflake via `DBIngestor` / `SnowflakeIngestor`
+- **Streams** — Kafka, real-time ingestion
+- **Media** — image OCR, audio/video metadata
---
## Features
-### What data sources does Semantica support?
-- **Files**: PDF, DOCX, TXT, JSON, CSV
-- **Web**: Websites, RSS feeds, APIs
-- **Databases**: PostgreSQL, MySQL, Snowflake, MongoDB
-- **Streams**: Kafka, RabbitMQ, real-time data
+### Can I use my own models?
-### Can I use custom models?
-Yes! Semantica supports custom:
-- **Entity extraction models**
-- **Embedding models**
-- **Language models**
-- **Custom processors**
+Yes. Semantica supports custom entity extraction models, embedding models, LLM providers (via LiteLLM — 100+ models), and custom pipeline processors.
### Does Semantica support GPUs?
-Yes, Semantica automatically uses GPUs when available for:
-- **Embedding generation**
-- **ML model inference**
-- **Vector operations**
+
+Yes. When available, GPUs are used automatically for embedding generation, ML model inference, and vector operations. Install `semantica[gpu]` for CUDA support.
+
+### How does Semantica handle large datasets?
+
+- **Batching** — process documents in configurable chunks
+- **Parallel processing** — `PipelineBuilder` supports configurable worker counts
+- **Delta processing** — update graphs incrementally without full recompute
+- **Graph backends** — swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE at scale
---
## Technical
-### How does Semantica handle large datasets?
-- **Batching** - Process data in chunks
-- **Streaming** - Handle real-time data
-- **Parallel processing** - Use multiple cores
-- **Memory management** - Efficient resource usage
+### What graph databases are supported?
-### Can I deploy Semantica in production?
-Yes! Semantica is production-ready with:
-- **Scalable architecture**
-- **Error handling**
-- **Monitoring support**
-- **Container deployment**
+Neo4j, FalkorDB, Apache AGE (PostgreSQL), Amazon Neptune, and in-memory NetworkX for development.
-### How do I customize Semantica?
-- **Custom processors** - Add new extraction logic
-- **Custom models** - Use your own ML models
-- **Plugins** - Extend functionality
-- **Configuration** - Adjust behavior
+### What export formats are available?
+
+RDF (Turtle, JSON-LD, N-Triples, XML), Apache Parquet, ArangoDB AQL, CSV, YAML, and OWL ontologies.
+
+### Is Semantica production-ready?
+
+Yes. v0.3.0 ships with 886+ passing tests, `PipelineValidator`, `FailureHandler` with exponential backoff, W3C PROV-O provenance, and change management with checksums. See [What's New](index.md#whats-new-in-v030) for details.
---
## Troubleshooting
-### Installation issues
-- **Python version**: Ensure Python 3.8+
-- **Dependencies**: Install with `pip install -e .[dev]`
-- **Permissions**: Use virtual environments
+### Import error: `ModuleNotFoundError: No module named 'semantica'`
-### Performance issues
-- **Memory**: Increase available RAM
-- **GPU**: Install CUDA for GPU acceleration
-- **Batching**: Use smaller chunk sizes
+Ensure you have the correct Python environment active, then:
-### Common errors
-- **Import errors**: Check installation path
-- **Model loading**: Verify model availability
-- **Memory errors**: Reduce batch sizes
+```bash
+pip list | grep semantica
+pip install --upgrade semantica
+```
+
+### Installation fails with dependency errors
+
+```bash
+pip install --upgrade pip wheel
+pip install semantica
+```
+
+### Memory errors during processing
+
+Reduce batch sizes, enable streaming ingestion, or switch to a persistent graph backend (Neo4j, FalkorDB).
+
+### Slow embedding or inference
+
+Install GPU support (`pip install semantica[gpu]`) and ensure CUDA is available on your system.
---
## Support
### Where can I get help?
-- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report problems
-- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Ask questions
-- **[Documentation](index.md)** - Browse guides and references
-### How do I report bugs?
-1. **Search** existing issues first
-2. **Create** a new issue with details
-3. **Include** reproduction steps
-4. **Add** environment information
+- [Discord](https://discord.gg/sV34vps5hH) — community chat and support
+- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) — bug reports and feature requests
+- [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) — questions and ideas
-### Can I contribute?
-Yes! See the [Contributing Guide](contributing.md) for details on how to help improve Semantica.
+### How do I report a bug?
+
+1. Search [existing issues](https://github.com/Hawksight-AI/semantica/issues) first
+2. Open a new issue with: description, reproduction steps, expected vs actual behavior, and your environment (Python version, OS, Semantica version)
+
+### How do I contribute?
+
+See the [Contributing Guide](contributing.md).
diff --git a/docs/getting-started.md b/docs/getting-started.md
index fecbc5ff..dcec03a9 100644
--- a/docs/getting-started.md
+++ b/docs/getting-started.md
@@ -1,14 +1,18 @@
# Getting Started
-## Overview
+**Semantica** is the context and intelligence layer for AI — turning raw data into explainable, auditable knowledge graphs for high-stakes domains.
-**Semantica** is a semantic intelligence layer that bridges the gap between raw data and trustworthy AI. It transforms unstructured data into explainable, auditable knowledge graphs perfect for high-stakes domains.
+!!! tip "Just here for code?"
+ Jump straight to the [Quick Start](#quick-start) or explore the [Cookbook](cookbook.md) for interactive notebooks.
-### What You Can Build
-- **GraphRAG Systems** - Enhanced retrieval with semantic reasoning
-- **AI Agents** - Trustworthy agents with explainable memory
-- **Knowledge Graphs** - Production-ready semantic databases
-- **Compliance-Ready AI** - Auditable systems with full provenance
+---
+
+## What You Can Build
+
+- **GraphRAG Systems** — enhanced retrieval with semantic graph reasoning
+- **AI Agents** — accountable agents with structured decision history and memory
+- **Knowledge Graphs** — production-ready semantic knowledge bases
+- **Compliance-Ready AI** — auditable systems with full W3C PROV-O provenance
---
@@ -18,17 +22,17 @@
pip install semantica
```
-Or with all features:
+With all optional dependencies:
```bash
pip install semantica[all]
```
-Verify installation:
+Verify:
```python
import semantica
-print(f"Semantica {semantica.__version__} installed!")
+print(semantica.__version__)
```
---
@@ -36,66 +40,59 @@ print(f"Semantica {semantica.__version__} installed!")
## Quick Start
```python
-from semantica.semantic_extract import NERExtractor
-from semantica.kg import GraphBuilder
+from semantica.context import AgentContext, ContextGraph
+from semantica.vector_store import VectorStore
-# Extract entities
-ner = NERExtractor(method="ml", model="en_core_web_sm")
-entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
+context = AgentContext(
+ vector_store=VectorStore(backend="faiss", dimension=768),
+ knowledge_graph=ContextGraph(advanced_analytics=True),
+ decision_tracking=True,
+)
-# Build knowledge graph
-kg = GraphBuilder().build({"entities": entities, "relationships": []})
-print(f"Built KG with {len(kg.get('entities', []))} entities")
+# Store a memory
+context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%")
+
+# Record a decision
+decision_id = context.record_decision(
+ category="model_selection",
+ scenario="Choose LLM for production pipeline",
+ reasoning="GPT-4 benchmark advantage justifies cost increase",
+ outcome="selected_gpt4",
+ confidence=0.91,
+)
+
+# Find similar past decisions
+precedents = context.find_precedents("model selection", limit=5)
```
-**What this does:**
-- Extracts entities (people, organizations, dates) from text
-- Builds a knowledge graph from extracted entities
-- Outputs the number of entities found
-
---
## Core Architecture
-Semantica uses a **modular architecture** - use only what you need:
+Semantica uses a modular, layered architecture — import only what you need.
-### 1️⃣ Input Layer - Data Ingestion
-```python
-from semantica.ingest import FileIngestor
-documents = FileIngestor().ingest_directory("docs/")
-```
-
-### 2️⃣ Semantic Layer - Intelligence Engine
-```python
-from semantica.semantic_extract import NERExtractor, RelationExtractor
-entities = NERExtractor().extract(text)
-relationships = RelationExtractor().extract(text, entities)
-```
-
-### 3️⃣ Output Layer - Knowledge Assets
-```python
-from semantica.kg import GraphBuilder
-kg = GraphBuilder().build_graph(entities, relationships)
-```
+| Layer | Modules | Purpose |
+|-------|---------|---------|
+| **Input** | `ingest`, `parse`, `split`, `normalize` | Load and prepare data |
+| **Semantic** | `semantic_extract`, `kg`, `ontology`, `reasoning` | Extract meaning |
+| **Storage** | `embeddings`, `vector_store`, `graph_store` | Persist knowledge |
+| **Quality** | `deduplication`, `conflicts` | Validate and clean |
+| **Context** | `context`, `provenance`, `change_management` | Track decisions and lineage |
+| **Output** | `export`, `visualization`, `pipeline` | Deliver results |
---
## Next Steps
-### 🍳 Interactive Tutorials
-1. **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)** - Complete framework overview
-2. **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)** - Hands-on graph building
-3. **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)** - Production-ready RAG
-
-### 📚 Learn More
-- **[Core Concepts](concepts.md)** - Deep dive into knowledge graphs & ontologies
-- **[Cookbook](cookbook.md)** - 14 domain-specific tutorials
-- **[API Reference](reference/core.md)** - Complete technical documentation
+- [Core Concepts](concepts.md) — knowledge graphs, ontologies, reasoning explained
+- [Quickstart Tutorial](quickstart.md) — build a full pipeline step by step
+- [Cookbook](cookbook.md) — 14 domain-specific Jupyter notebook tutorials
+- [API Reference](reference/core.md) — complete module documentation
---
-## Need Help?
+## Help
-- **[💬 Discord Community](https://discord.gg/sV34vps5hH)** - Get help from the community
-- **[🐛 Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report bugs or request features
-- **[📖 Documentation](https://semantica.readthedocs.io/)** - Full documentation site
+- [Discord Community](https://discord.gg/sV34vps5hH) — ask questions, share projects
+- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) — report bugs or request features
+- [FAQ](faq.md) — common questions answered
diff --git a/docs/glossary.md b/docs/glossary.md
index e065699d..4b42ad31 100644
--- a/docs/glossary.md
+++ b/docs/glossary.md
@@ -1,9 +1,9 @@
# Glossary
-**Comprehensive reference of terms and concepts used in Semantica and semantic intelligence.**
+Reference of terms and concepts used throughout Semantica.
-!!! tip "Quick Reference"
- Looking for a specific term? Use your browser's search function (Ctrl+F) to find terms quickly.
+!!! tip "Finding a term"
+ Use Ctrl+F to search this page.
---
@@ -216,17 +216,7 @@ W3C PROV-O compliant tracking of data lineage and source attribution.
## See Also
-- **[Core Concepts](concepts.md)** - Deep dive into fundamental concepts
-- **[Getting Started](getting-started.md)** - Begin your journey with Semantica
-- **[Modules Guide](modules.md)** - Complete module overview
-- **[API Reference](reference/)** - Technical documentation
-
----
-
-## Need Help?
-
-- **Documentation**: [Getting Started](getting-started.md)
-- **Examples**: [Cookbook](cookbook.md)
-- **Community**: [Discord](community.md)
-- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
-- **Support**: [Contact Us](community.md)
+- [Core Concepts](concepts.md) — deeper explanation of key ideas
+- [Getting Started](getting-started.md) — first steps
+- [Modules Guide](modules.md) — every module explained
+- [API Reference](reference/) — technical reference
diff --git a/docs/index.md b/docs/index.md
index b67a3906..63a0ecf7 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,24 +1,23 @@
-
-
+
+
🧠 Semantica
-
+
-
-
+
+
-
-
-
-
Open-Source Semantic Layer & Knowledge Engineering Framework
-
-
Transform Chaos into Intelligence. Build AI systems that are explainable, traceable, and trustworthy — not black boxes.
-
-
The semantic intelligence layer that makes your AI agents auditable, explainable, and trustworthy. Perfect for high-stakes domains where mistakes have real consequences.
-
-
🆓 Open Source • 📜 MIT Licensed • 🚀 Production Ready • 🌍 Community Driven
-
+
+
+
+
+
A Framework for Building Context Graphs and Decision Intelligence Layers for AI
+
+
⭐ Give us a Star • 🍴 Fork us • 💬 Join our Discord • 🐦 Follow on X
+
+
Transform Chaos into Intelligence. Build AI systems with context graphs, decision tracking, and advanced knowledge engineering that are explainable, traceable, and trustworthy — not black boxes.
+
Get Started
View on GitHub
@@ -27,288 +26,266 @@
---
-## 🚀 Why Semantica?
+## The Problem
-**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and trustworthy.
+AI agents today are capable but not trustworthy:
-Perfect for **high-stakes domains** where mistakes have real consequences.
+- **No memory structure** — agents store embeddings, not meaning. Retrieval is fuzzy; there's no way to ask *why* something was recalled.
+- **No decision trail** — agents make decisions continuously but record nothing. When something goes wrong, there's no history to debug or audit.
+- **No provenance** — outputs cannot be traced back to source facts. In regulated industries, this is a compliance blocker.
+- **No reasoning transparency** — black-box answers with no explanation of how a conclusion was reached.
+- **No conflict detection** — contradictory facts silently coexist in vector stores, producing unpredictable answers.
+
+These aren't edge cases. They are the reason AI cannot be deployed in healthcare, finance, legal, and government without custom guardrails built from scratch.
---
-### ⚡ Get Started in 30 Seconds
+## The Solution
+
+Semantica is the **context and intelligence layer** you add to your AI stack:
+
+- **Context Graphs** — structured graph of entities, relationships, and decisions your agent builds as it works. Queryable, traceable, persistent.
+- **Decision Intelligence** — every decision is a first-class object: recorded, linked causally, searchable by precedent, and analyzable for downstream impact.
+- **Provenance** — every fact links to its source. W3C PROV-O compliant. Full lineage from ingestion to inference.
+- **Reasoning engines** — forward chaining, Rete networks, deductive, abductive, and SPARQL reasoning. Explainable inference paths, not black-box answers.
+- **Deduplication & QA** — conflict detection, entity resolution, and validation built into the pipeline.
+
+Works alongside LangChain, LlamaIndex, AutoGen, CrewAI, and any LLM provider — Semantica is not a replacement, it's the accountability layer on top.
+
+---
+
+### ⚡ Quick Installation
```bash
pip install semantica
```
```python
-from semantica.semantic_extract import NERExtractor
-from semantica.kg import GraphBuilder
+from semantica.context import AgentContext, ContextGraph
+from semantica.vector_store import VectorStore
-# Extract entities and build knowledge graph
-ner = NERExtractor(method="ml", model="en_core_web_sm")
-entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
-kg = GraphBuilder().build({"entities": entities, "relationships": []})
+context = AgentContext(
+ vector_store=VectorStore(backend="faiss", dimension=768),
+ knowledge_graph=ContextGraph(advanced_analytics=True),
+ decision_tracking=True,
+)
-print(f"Built KG with {len(kg.get('entities', []))} entities")
+# Store a memory
+context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%")
+
+# Record a decision
+decision_id = context.record_decision(
+ category="model_selection",
+ scenario="Choose LLM for production reasoning pipeline",
+ reasoning="GPT-4 benchmark advantage justifies 3x cost increase",
+ outcome="selected_gpt4",
+ confidence=0.91,
+)
+
+# Find similar past decisions and analyze downstream impact
+precedents = context.find_precedents("model selection reasoning", limit=5)
+influence = context.analyze_decision_influence(decision_id)
```
-**[📖 Full Quick Start](getting-started.md)** • **[🍳 Cookbook Examples](cookbook.md)** • **[💬 Join Discord](https://discord.gg/sV34vps5hH)** • **[⭐ Star Us](https://github.com/Hawksight-AI/semantica)**
+**[Full Quick Start](getting-started.md)** • **[Cookbook](cookbook.md)** • **[Join Discord](https://discord.gg/sV34vps5hH)**
+
+---
+
+## What's New in v0.3.0
+
+> First stable release (`Production/Stable` on PyPI).
+
+| Area | Highlights |
+|------|------------|
+| **Context Graphs** | Temporal validity windows, weighted BFS, cross-graph navigation with save/load persistence |
+| **Decision Intelligence** | Full lifecycle: record → trace → impact → precedent; `PolicyEngine` with versioned rules |
+| **KG Algorithms** | PageRank, betweenness, Louvain community detection, Node2Vec, link prediction |
+| **Semantic Extraction** | LLM extraction fixed (no silent drops), duplicate relation bug removed, `"llm_typed"` metadata corrected |
+| **Deduplication v2** | `blocking_v2`/`hybrid_v2` — 63.6% faster; semantic v2 — 6.98x faster |
+| **Delta Processing** | SPARQL-based incremental diff, `delta_mode` pipelines, snapshot versioning |
+| **Export** | RDF aliases (`"ttl"`, `"json-ld"`), ArangoDB AQL, Apache Parquet (Spark/BigQuery/Databricks) |
+| **Pipeline** | `FailureHandler` with LINEAR/EXPONENTIAL/FIXED backoff; `PipelineValidator` returning `ValidationResult` |
+| **Graph Backends** | Apache AGE (SQL injection fixed), AWS Neptune, FalkorDB, PgVector (HNSW/IVFFlat) |
+| **Tests** | 886+ passing, 0 failures — 335 context, ~430 KG, 70 semantic extraction, 85 real-world E2E |
---
## Core Value Proposition
| **Trustworthy** | **Explainable** | **Auditable** |
-|:------------------:|:------------------:|:-----------------:|
+|:---:|:---:|:---:|
| Conflict detection & validation | Transparent reasoning paths | Complete provenance tracking |
| Rule-based governance | Entity relationships & ontologies | W3C PROV-O compliant lineage |
| Production-grade QA | Multi-hop graph reasoning | Source tracking & integrity verification |
---
-## Key Features & Benefits
+## Features
-### Not Just Another Agentic Framework
+### Context & Decision Intelligence
+- **Context Graphs** — structured, persistent graph of entities, relationships, and decisions
+- **Decision tracking** — `add_decision()`, `record_decision()` for full lifecycle management
+- **Causal chains** — `add_causal_relationship()`, `trace_decision_chain()`
+- **Precedent search** — hybrid similarity search over past decisions via `find_similar_decisions()`
+- **Influence analysis** — `analyze_decision_impact()`, `analyze_decision_influence()`
+- **Policy engine** — `check_decision_rules()` with versioned, automated compliance rules
+- **Agent memory** — `AgentMemory` with short/long-term storage and conversation history
-**Semantica complements** LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK, Agno, and other frameworks to enhance your agents with:
+### Knowledge Graphs
+- **Graph construction** — entities, relationships, properties, typed edges
+- **Algorithms** — PageRank, betweenness centrality, clustering coefficient, community detection
+- **Node embeddings** — Node2Vec via `NodeEmbedder`; cosine similarity via `SimilarityCalculator`
+- **Link prediction** — score potential edges via `LinkPredictor`
+- **Temporal graphs** — time-aware nodes and edges with validity windows
+- **Delta processing** — incremental updates without full recompute
-| Feature | Benefit |
-|:--------|:--------|
-| **Auditable** | Complete provenance tracking with W3C PROV-O compliance |
-| **Explainable** | Transparent reasoning paths with entity relationships |
-| **Provenance-Aware** | End-to-end lineage from documents to responses |
-| **Validated** | Built-in conflict detection, deduplication, QA |
-| **Governed** | Rule-based validation and semantic consistency |
-| **Version Control** | Enterprise-grade change management with integrity verification |
+### Semantic Extraction
+- **NER** — named entity recognition, normalization, classification
+- **Relation extraction** — triplet generation via LLMs or rule-based methods, with `"llm_typed"` metadata
+- **Deduplication v1/v2** — Jaro-Winkler, `blocking_v2`, `hybrid_v2`, `semantic_v2`; `dedup_triplets()` for triples
-### Perfect For High-Stakes Use Cases
+### Reasoning
+- **Forward chaining** — `Reasoner` with IF/THEN string rules and dict facts
+- **Rete network** — `ReteEngine` for high-throughput production rule matching
+- **Deductive / Abductive** — `DeductiveReasoner`, `AbductiveReasoner`
+- **SPARQL** — `SPARQLReasoner` for query-based inference over RDF graphs
-| 🏥 **Healthcare** | 💰 **Finance** | ⚖️ **Legal** |
-|:-----------------:|:--------------:|:------------:|
-| Clinical decisions | Fraud detection | Evidence-backed research |
-| Drug interactions | Regulatory support | Contract analysis |
-| Patient safety | Risk assessment | Case law reasoning |
+### Provenance & Auditability
+- **Entity provenance** — `ProvenanceTracker.track_entity()`
+- **Algorithm provenance** — `AlgorithmTrackerWithProvenance`
+- **W3C PROV-O compliant** — lineage tracking across all modules
+- **Change management** — version control with checksums, audit trails, compliance support
-| 🔒 **Cybersecurity** | 🏛️ **Government** | 🏭 **Infrastructure** | 🚗 **Autonomous** |
-|:-------------------:|:----------------:|:-------------------:|:-----------------:|
-| Threat attribution | Policy decisions | Power grids | Decision logs |
-| Incident response | Classified info | Transportation | Safety validation |
+### Vector Store
+- **Backends** — FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory
+- **Search modes** — semantic top-k, hybrid (vector + keyword), metadata-filtered
-### Powers Your AI Stack
+### Data Ingestion
+- **Files** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives
+- **Sources** — web crawl, SQL databases, Snowflake, feeds, email, repositories
+- **Docling** — advanced parsing with table and layout extraction
+- **Media** — image OCR, audio/video metadata
-- **GraphRAG Systems** — Retrieval with graph reasoning and hybrid search
-- **AI Agents** — Trustworthy, accountable multi-agent systems with semantic memory
-- **Reasoning Models** — Explainable AI decisions with reasoning paths
-- **Enterprise AI** — Governed, auditable platforms that support compliance
+### Export
+- **RDF** — Turtle, JSON-LD, N-Triples, XML via `RDFExporter`
+- **Parquet** — `ParquetExporter` for Spark/BigQuery/Databricks pipelines
+- **ArangoDB AQL** — ready-to-run INSERT statements
+- **OWL ontologies** — Turtle or RDF/XML
-### Integrations
-
-- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX)
-- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication
-- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD)
-
-> **Built for environments where every answer must be explainable and governed.**
+### Pipeline & Ontology
+- **Pipeline DSL** — `PipelineBuilder` with stage chaining, parallel workers, retry policies
+- **Ontology** — auto-generate OWL from KGs, import OWL/RDF/Turtle/JSON-LD, HermiT/Pellet validation
---
-## 🚨 The Problem: The Semantic Gap
+## Modules
-### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**.
-
-### Understanding the Semantic Gap
-
-The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships).
-
-**Traditional AI approaches:**
-- Rely on statistical patterns and text similarity
-- Cannot understand relationships between entities
-- Cannot reason about domain-specific rules
-- Cannot explain why decisions were made
-- Cannot trace back to original sources with confidence
-
-**High-stakes AI requires:**
-- Semantic understanding of entities and their relationships
-- Domain knowledge encoded as formal rules (ontologies)
-- Explainable reasoning paths
-- Source-level provenance
-- Conflict detection and resolution
-
-**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge.
-
-### What Organizations Have vs What They Need
-
-| **Current State** | **Required for High-Stakes AI** |
-|:---------------------|:-----------------------------------|
-| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) |
-| APIs, databases, streams | Structured and validated entities |
-| Conflicting facts and duplicates | Explicit semantic relationships |
-| Siloed systems with no lineage | **Explainable reasoning paths** |
-| | **Source-level provenance** |
-| | **Audit-ready compliance** |
-
-### The Cost of Missing Semantics
-
-- **Decisions cannot be explained** — No transparency in AI reasoning
-- **Errors cannot be traced** — No way to debug or improve
-- **Conflicts go undetected** — Contradictory information causes failures
-- **Compliance becomes impossible** — No audit trails for regulations
-
-**Trustworthy AI requires semantic accountability.**
+| Module | What it provides |
+|--------|-----------------|
+| `semantica.context` | Context graphs, agent memory, decision tracking, causal analysis, precedent search, policy engine |
+| `semantica.kg` | KG construction, graph algorithms, centrality, community detection, embeddings, link prediction |
+| `semantica.semantic_extract` | NER, relation extraction, event extraction, coreference, triplet generation, LLM extraction |
+| `semantica.reasoning` | Forward chaining, Rete network, deductive, abductive, SPARQL reasoning, explanation generation |
+| `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector; hybrid & filtered search |
+| `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, YAML, OWL, graph formats |
+| `semantica.ingest` | Files, web crawl, feeds, databases, Snowflake, MCP, email, repositories |
+| `semantica.ontology` | Auto-generation, OWL/RDF export, import, validation, versioning |
+| `semantica.pipeline` | Pipeline DSL, parallel workers, validation, retry policies, failure handling |
+| `semantica.graph_store` | Neo4j, FalkorDB, Apache AGE, Amazon Neptune; Cypher queries |
+| `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE; similarity calculation |
+| `semantica.deduplication` | Entity deduplication, similarity scoring, merging, clustering |
+| `semantica.provenance` | W3C PROV-O lineage tracking, source attribution, audit trails |
+| `semantica.parse` | PDF, DOCX, PPTX, HTML, code, email, structured data, OCR |
+| `semantica.split` | Recursive, semantic, entity-aware, relation-aware, graph-based chunking |
+| `semantica.normalize` | Text, entities, dates, numbers, quantities, languages, encodings |
+| `semantica.conflicts` | Multi-source conflict detection (value, type, temporal, logical) with resolution |
+| `semantica.change_management` | Version storage, change tracking, checksums, audit trails |
+| `semantica.triplet_store` | Blazegraph, Jena, RDF4J; SPARQL queries and bulk loading |
+| `semantica.visualization` | Interactive/static KG, ontology, embedding, and temporal graph visualization |
+| `semantica.core` | Framework orchestration, configuration, plugin system |
+| `semantica.llms` | Groq, OpenAI, Novita AI, HuggingFace, LiteLLM integrations |
---
-## 🆚 Semantica vs Traditional RAG
+## Built for High-Stakes Domains
-| Feature | Traditional RAG | Semantica |
-|:--------|:----------------|:----------|
-| **Reasoning** | ❌ Black-box answers | ✅ Explainable reasoning paths |
-| **Provenance** | ❌ No provenance | ✅ W3C PROV-O compliant lineage tracking |
-| **Search** | ⚠️ Vector similarity only | ✅ Semantic + graph reasoning |
-| **Quality** | ❌ No conflict handling | ✅ Explicit contradiction detection |
-| **Safety** | ⚠️ Unsafe for high-stakes | ✅ Designed for governed environments |
-| **Compliance** | ❌ No audit trails | ✅ Complete audit trails with integrity verification |
+Where **every decision must be accountable** and **mistakes have real consequences**:
+
+- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interactions, patient safety
+- **💰 Finance & Risk** — Fraud detection, SOX/GDPR/MiFID II compliance, risk assessment
+- **⚖️ Legal & Compliance** — Evidence-backed research, contract analysis, regulatory tracking
+- **🔒 Cybersecurity** — Threat attribution, incident response, security audit trails
+- **🏛️ Government & Defense** — Policy decisions, classified information handling, defense intelligence
+- **🏭 Critical Infrastructure** — Power grids, transportation safety, emergency response
+- **🚗 Autonomous Systems** — Self-driving, robotics safety, industrial automation
---
-## 🧩 Semantica Architecture
-
-### 1️⃣ Input Layer — Governed Ingestion
-- 📄 **Multiple Formats** — PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX
-- 🔧 **Docling Support** — Docling parser for table extraction
-- 💾 **Data Sources** — Databases, APIs, streams, archives, web content
-- 🎨 **Media Support** — Image parsing with OCR, audio/video metadata extraction
-- � **Single Pipeline** — Unified ingestion with metadata and source tracking
-
-### 2️⃣ Semantic Layer — Trust & Reasoning Engine
-- 🔍 **Entity Extraction** — NER, normalization, classification
-- 🔗 **Relationship Discovery** — Triplet generation, semantic links
-- 📐 **Ontology Induction** — Automated domain rule generation
-- 🔄 **Deduplication** — Jaro-Winkler similarity, conflict resolution
-- ✅ **Quality Assurance** — Conflict detection, validation
-- 📊 **Provenance Tracking** — W3C PROV-O compliant lineage tracking across all modules
-- 🧠 **Reasoning Traces** — Explainable inference paths
-- 🔐 **Change Management** — Version control with audit trails, checksums, compliance support
-
-### 3️⃣ Output Layer — Auditable Knowledge Assets
-- � **Knowledge Graphs** — Queryable, temporal, explainable
-- 📐 **OWL Ontologies** — HermiT/Pellet validated, custom ontology import support
-- 🔢 **Vector Embeddings** — FastEmbed by default
-- ☁️ **AWS Neptune** — Amazon Neptune graph database support
-- 🔍 **Provenance** — Every AI response links back to:
- - 📄 Source documents
- - 🏷️ Extracted entities & relations
- - 📐 Ontology rules applied
- - 🧠 Reasoning steps used
-
----
-
-## 🏥 Built for High-Stakes Domains
-
-Designed for domains where **mistakes have real consequences** and **every decision must be accountable**:
-
-- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety tracking
-- **💰 Finance & Risk** — Fraud detection, regulatory support (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation
-- **⚖️ Legal & Compliance** — Evidence-backed legal research, contract analysis, regulatory change tracking, case law reasoning
-- **🔒 Cybersecurity & Intelligence** — Threat attribution, incident response, security audit trails, intelligence analysis
-- **🏛️ Government & Defense** — Governed AI systems, policy decisions, classified information handling, defense intelligence
-- **🏭 Critical Infrastructure** — Power grid management, transportation safety, water treatment, emergency response
-- **🚗 Autonomous Systems** — Self-driving vehicles, drone navigation, robotics safety, industrial automation
-
----
-
-## � Who Uses Semantica?
-
-- **🤖 AI / ML Engineers** — Building explainable GraphRAG & agents
-- **⚙️ Data Engineers** — Creating governed semantic pipelines
-- **📊 Knowledge Engineers** — Managing ontologies & KGs at scale
-- **🏢 Enterprise Teams** — Requiring trustworthy AI infrastructure
-- **🛡️ Risk & Compliance Teams** — Needing audit-ready systems
-
----
-
-## 🚀 Choose Your Path
+## Choose Your Path
- :material-rocket-launch: **Quick Start**
---
- Get up and running with Semantica in minutes. Learn the basics of ingestion and extraction.
-
+ Up and running in minutes.
+
[:arrow_right: Start Here](getting-started.md)
- :material-book-open-page-variant: **Core Concepts**
---
- Deep dive into Knowledge Graphs, Ontologies, and Semantic Reasoning.
-
+ Knowledge graphs, ontologies, and semantic reasoning explained.
+
[:arrow_right: Learn Concepts](concepts.md)
- :material-code-braces: **API Reference**
---
- Detailed technical documentation for all Semantica modules and classes.
-
+ Full technical documentation for every module and class.
+
[:arrow_right: View API](reference/core.md)
- :material-chef-hat: **Cookbook**
---
- Interactive tutorials, real-world examples, and **14 domain-specific cookbooks**.
-
+ 14 domain-specific cookbooks with real-world examples.
+
[:arrow_right: Explore Cookbook](cookbook.md)
---
-## 📦 Installation
+## Installation
!!! success "Now Available on PyPI!"
- Semantica is officially published on PyPI! Install it with a single command.
+ Install with a single command.
-=== "From PyPI (Recommended)"
-
- Install Semantica directly from PyPI:
+=== "PyPI (Recommended)"
```bash
- # Install the core package
pip install semantica
- # Or install with all optional dependencies
+ # With all optional dependencies
pip install semantica[all]
```
=== "From Source"
- Install from the local source for the latest development version:
-
```bash
- # Clone the repository
git clone https://github.com/Hawksight-AI/semantica.git
cd semantica
-
- # Install in editable mode with core dependencies
- pip install -e .
-
- # Or install with all optional dependencies
- pip install -e ".[all]"
+ pip install -e . # core
+ pip install -e ".[all]" # all extras
```
=== "Development"
- For contributors who want to modify the framework:
-
```bash
- # Clone the repository
git clone https://github.com/Hawksight-AI/semantica.git
cd semantica
-
- # Install in editable mode with dev dependencies
pip install -e ".[dev]"
```
=== "Docker"
- Run Semantica in a containerized environment:
-
```bash
docker pull semantica/semantica:latest
docker run -it semantica/semantica
@@ -316,117 +293,41 @@ Designed for domains where **mistakes have real consequences** and **every decis
---
-## 🚦 Quick Example
-
-Semantica uses a modular architecture. You can use individual modules directly for maximum flexibility:
-
-```python
-from semantica.ingest import FileIngestor
-from semantica.parse import DocumentParser
-from semantica.semantic_extract import NERExtractor, RelationExtractor
-from semantica.kg import GraphBuilder
-
-# 1. Ingest documents
-ingestor = FileIngestor()
-documents = ingestor.ingest_directory("documents/", recursive=True)
-
-# 2. Parse documents
-parser = DocumentParser()
-parsed_docs = [parser.parse_document(doc) for doc in documents]
-
-# 3. Extract entities and relationships
-ner = NERExtractor()
-rel_extractor = RelationExtractor()
-
-entities = []
-relationships = []
-for doc in parsed_docs:
- text = doc.get("full_text", "")
- doc_entities = ner.extract_entities(text)
- doc_rels = rel_extractor.extract_relations(text, entities=doc_entities)
- entities.extend(doc_entities)
- relationships.extend(doc_rels)
-
-# 4. Build knowledge graph
-builder = GraphBuilder(merge_entities=True)
-kg = builder.build_graph(entities=entities, relationships=relationships)
-
-print(f"Created graph with {len(kg.nodes)} nodes and {len(kg.edges)} edges")
-```
-
-!!! tip "Orchestration Option"
- For complex workflows, you can also use the `Semantica` class for orchestration. See the [Core Module](reference/core.md) documentation for details.
-
----
-
-## 🎯 Why Semantica?
+## Why Semantica?
- **🆓 Open Source**
---
- MIT licensed. No vendor lock-in. Full transparency.
+ MIT licensed. No vendor lock-in.
- **🚀 Production Ready**
---
- Battle-tested with quality assurance, conflict resolution, and validation.
+ Battle-tested with QA, conflict resolution, and validation built in.
-- **🧩 Modular Architecture**
+- **🧩 Modular**
---
Use only what you need. Swap components easily.
- **🌍 Community Driven**
---
- Built by developers, for developers. Active Discord community.
+ Built by developers, for developers. Active Discord.
-- **📚 Comprehensive**
+- **📚 End-to-End**
---
- End-to-end solution from ingestion to reasoning. No duct-taping required.
+ From ingestion to reasoning — no duct-taping required.
- **🔬 Research-Backed**
---
- Based on latest research in knowledge graphs, ontologies, and semantic web.
+ Grounded in knowledge graph, ontology, and semantic web research.
---
-## 🏗️ Built For
-
-- **Data Scientists**: Transform messy data into clean knowledge graphs
-- **Data Engineers**: Build scalable data pipelines with semantic enrichment
-- **AI Engineers**: Build GraphRAG, AI agents, and multi-agent systems
-- **Knowledge Engineers**: Generate and manage formal ontologies
-- **Ontologists**: Design and validate domain-specific ontologies and taxonomies
-- **Researchers**: Analyze scientific literature and build citation networks
-- **ML Engineers**: Create semantic features for machine learning models
-- **Enterprises**: Unify data silos into a semantic layer
-
----
-
-## 📚 Learn More
-
-- [Getting Started Guide](getting-started.md) - Your first knowledge graph in 5 minutes
-- [Core Concepts](concepts.md) - Deep dive into knowledge graphs and ontologies
-- [Cookbook](cookbook.md) - Real-world examples and **14 domain-specific cookbooks**
-- [API Reference](reference/core.md) - Complete technical documentation
-
-### 🍳 Recommended Cookbook Tutorials
-
-Get hands-on with interactive Jupyter notebooks:
-
-- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all Semantica modules
- - **Topics**: Framework overview, all modules, architecture
- - **Difficulty**: Beginner
- - **Use Cases**: First-time users, understanding the framework
-
-- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph from scratch
- - **Topics**: Entity extraction, relationship extraction, graph construction
- - **Difficulty**: Beginner
- - **Use Cases**: Learning the basics, quick start
-
-- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production-ready Graph Retrieval Augmented Generation
- - **Topics**: GraphRAG, hybrid retrieval, vector search, graph traversal
- - **Difficulty**: Advanced
- - **Use Cases**: Building AI applications with knowledge graphs
+## Learn More
+- [Getting Started](getting-started.md) — your first knowledge graph in 5 minutes
+- [Core Concepts](concepts.md) — knowledge graphs, ontologies, and semantic reasoning
+- [Cookbook](cookbook.md) — 14 domain-specific cookbooks with Jupyter notebooks
+- [API Reference](reference/core.md) — complete technical documentation
diff --git a/docs/installation.md b/docs/installation.md
index d786c39a..844eaeb0 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -1,293 +1,169 @@
# Installation
-Get Semantica up and running in minutes.
+Get Semantica installed in under a minute.
-!!! success "Now Available on PyPI!"
- Semantica is officially published on PyPI! Install it with a single command: `pip install semantica`
+!!! success "Available on PyPI"
+ `pip install semantica` — that's it.
-!!! note "System Requirements"
- Semantica requires Python 3.8 or higher. For best performance, we recommend Python 3.10+.
+!!! note "Requirements"
+ Python 3.8 or higher. Python 3.11+ recommended.
-## Prerequisites
-
-Before installing Semantica, ensure you have:
-
-- **Python 3.8 or higher** - Check your version:
- ```bash
- python --version
- ```
-- **pip** - Python package installer (usually comes with Python)
+---
## Basic Installation
-Install Semantica from PyPI:
-
```bash
pip install semantica
```
-This installs Semantica with all core dependencies.
-
-### GitHub Workaround
-
-If you encounter issues with the PyPI version, you can install directly from the main branch:
-
-```bash
-pip install git+https://github.com/Hawksight-AI/semantica.git@main
-```
-
-!!! tip "Virtual Environment"
- We recommend installing Semantica in a virtual environment to avoid dependency conflicts. Use `python -m venv venv` to create one, then activate it before installing.
-
-## Verify Installation
-
-Verify that Semantica is installed correctly:
-
-```bash
-python -c "from semantica.parse import DoclingParser; DoclingParser(); print('✓ Semantica ready')"
-```
-
-!!! info "Windows PyTorch Note"
- If you encounter PyTorch DLL errors on Windows, ensure you have the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) installed. This is a common environment-specific issue with PyTorch on Windows and not a bug in Semantica.
-
-Expected output:
-```
-✓ Semantica ready
-```
-
-You can also check the installation:
-
-```bash
-pip show semantica
-```
-
-## Development Installation
-
-To install Semantica in development mode (for contributing):
-
-```bash
-# Clone the repository
-git clone https://github.com/Hawksight-AI/semantica.git
-cd semantica
-
-# Install in editable mode
-pip install -e .
-
-# Or install with development dependencies
-pip install -e ".[dev]"
-```
-
-## Optional Dependencies
-
-Semantica supports optional features that can be installed separately:
-
-### GPU Support
-
-For GPU-accelerated operations:
-
-```bash
-pip install semantica[gpu]
-```
-
-This includes:
-- PyTorch with CUDA support
-- FAISS GPU
-- CuPy
-
-### Visualization
-
-For enhanced visualization capabilities:
-
-```bash
-pip install semantica[viz]
-```
-
-Includes:
-- PyVis for interactive graphs
-- Graphviz for static diagrams
-- UMAP for dimensionality reduction
-
-### LLM Providers
-
-Install all LLM provider integrations:
-
-```bash
-pip install semantica[llm-all]
-```
-
-Or install specific providers:
-
-```bash
-# OpenAI
-pip install semantica[llm-openai]
-
-# Anthropic
-pip install semantica[llm-anthropic]
-
-# Google Gemini
-pip install semantica[llm-gemini]
-
-# Groq
-pip install semantica[llm-groq]
-
-# Ollama
-pip install semantica[llm-ollama]
-```
-
-### Cloud Integrations
-
-For cloud storage and deployment:
-
-```bash
-pip install semantica[cloud]
-```
-
-Includes:
-- AWS S3 (boto3)
-- Azure Blob Storage
-- Google Cloud Storage
-- Kubernetes support
-
-### All Optional Features
-
-Install everything:
+With all optional dependencies:
```bash
pip install semantica[all]
```
-## Virtual Environment (Recommended)
+### Verify
-It's recommended to use a virtual environment:
+```bash
+python -c "import semantica; print(semantica.__version__)"
+```
+
+---
+
+## Virtual Environment (Recommended)
=== "venv"
```bash
- # Create virtual environment
python -m venv venv
-
- # Activate (Windows)
- venv\Scripts\activate
-
- # Activate (Linux/Mac)
- source venv/bin/activate
-
- # Install Semantica
+ source venv/bin/activate # Linux / Mac
+ venv\Scripts\activate # Windows
pip install semantica
```
=== "conda"
```bash
- # Create conda environment
conda create -n semantica python=3.11
conda activate semantica
-
- # Install Semantica
pip install semantica
```
+---
+
+## Optional Dependencies
+
+Install only what you need:
+
+=== "GPU"
+
+ ```bash
+ pip install semantica[gpu]
+ ```
+ Includes PyTorch with CUDA, FAISS GPU, CuPy.
+
+=== "Visualization"
+
+ ```bash
+ pip install semantica[viz]
+ ```
+ Includes PyVis, Graphviz, UMAP.
+
+=== "LLM Providers"
+
+ ```bash
+ pip install semantica[llm-all] # all providers
+
+ pip install semantica[llm-openai] # OpenAI
+ pip install semantica[llm-anthropic] # Anthropic
+ pip install semantica[llm-gemini] # Google Gemini
+ pip install semantica[llm-groq] # Groq
+ pip install semantica[llm-ollama] # Ollama (local)
+ ```
+
+=== "Cloud"
+
+ ```bash
+ pip install semantica[cloud]
+ ```
+ Includes AWS S3, Azure Blob, Google Cloud Storage.
+
+---
+
+## Install from Source
+
+For the latest development version or to contribute:
+
+```bash
+git clone https://github.com/Hawksight-AI/semantica.git
+cd semantica
+
+pip install -e . # core only
+pip install -e ".[all]" # all extras
+pip install -e ".[dev]" # dev tools (pytest, black, etc.)
+```
+
+If you encounter issues with the PyPI release, install directly from the main branch:
+
+```bash
+pip install git+https://github.com/Hawksight-AI/semantica.git@main
+```
+
+---
+
## Troubleshooting
-### Common Issues
+### ModuleNotFoundError
-#### ModuleNotFoundError
+Check you have the right environment active:
-**Error**: `ModuleNotFoundError: No module named 'semantica'`
+```bash
+pip list | grep semantica
+pip install --upgrade semantica
+```
-**Solutions**:
-- Make sure you've activated the correct Python environment
-- Verify installation: `pip list | grep semantica`
-- Reinstall: `pip install --upgrade semantica`
+### Installation fails with dependency errors
-#### Installation Fails
+```bash
+pip install --upgrade pip
+pip install build wheel
+pip install semantica --no-deps # install without optional deps first
+```
-**Error**: Installation fails with dependency errors
+### GPU dependencies fail
-**Solutions**:
-- Upgrade pip: `pip install --upgrade pip`
-- Install build tools: `pip install build wheel`
-- Try installing without optional dependencies first: `pip install semantica --no-deps`
+Install CPU-only first, then add GPU support:
-#### GPU Dependencies Fail
+```bash
+pip install semantica
+pip install semantica[gpu]
+```
-**Error**: GPU dependencies fail to install
+### Permission denied
-**Solutions**:
-- Install CPU-only version first: `pip install semantica`
-- Then add GPU support: `pip install semantica[gpu]`
-- Check CUDA compatibility for your system
+```bash
+pip install --user semantica # or use a virtual environment
+```
-#### Permission Errors
+### Windows PyTorch DLL errors
-**Error**: Permission denied during installation
+Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe). This is a Windows system dependency, not a Semantica bug.
-**Solutions**:
-- Use `--user` flag: `pip install --user semantica`
-- Use virtual environment (recommended)
-- On Linux/Mac, avoid using `sudo` with pip
+---
-### System Requirements
+## System Requirements
| Component | Minimum | Recommended |
|-----------|---------|-------------|
| Python | 3.8 | 3.11+ |
-| RAM | Moderate | Ample for your dataset |
-| Disk Space | Sufficient for data | Generous storage |
-| OS | Windows/Linux/Mac | Linux/Mac |
+| OS | Windows / Linux / Mac | Linux / Mac |
+| RAM | 4 GB | 16 GB+ |
+| Storage | 2 GB | 20 GB+ (for models and data) |
-## After Installation
-
-Once Semantica is installed, verify your setup and get started:
-
-### Verify Your Installation
-
-Test that everything works correctly:
-
-```bash
-python -c "import semantica; print(semantica.__version__)"
-```
-
-**For detailed setup verification and first steps, see:**
-- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Verify installation and explore all modules
- - **Topics**: Framework overview, installation verification, module exploration
- - **Difficulty**: Beginner
- - **Time**: 30-45 minutes
- - **Use Cases**: First-time setup, understanding the framework
+---
## Next Steps
-Now that Semantica is installed:
-
-1. **[Quick Start Guide](quickstart.md)** - Build your first knowledge graph in 5 minutes
-2. **[Getting Started Guide](getting-started.md)** - Learn the fundamentals
-3. **[Examples](examples.md)** - See real-world use cases
-4. **[Cookbook](cookbook.md)** - Interactive Jupyter notebook tutorials
-
-### 🍳 Recommended First Cookbooks
-
-Start with these interactive tutorials:
-
-- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction
- - **Topics**: Framework overview, all modules, architecture, configuration
- - **Difficulty**: Beginner
- - **Time**: 30-45 minutes
- - **Use Cases**: First-time users, understanding the framework
-
-- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first graph
- - **Topics**: Entity extraction, relationship extraction, graph construction
- - **Difficulty**: Beginner
- - **Time**: 20-30 minutes
- - **Use Cases**: Hands-on practice, quick start
-
-## Getting Help
-
-If you encounter issues:
-
-- Check the [troubleshooting section](#troubleshooting) above
-- Review [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
-- Ask questions in [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
-
-**For installation and setup help:**
-- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Includes setup verification steps
-- **[Installation Troubleshooting Guide](getting-started.md#installation--setup)**: Additional troubleshooting tips
+- [Getting Started](getting-started.md) — build your first knowledge graph
+- [Quickstart Tutorial](quickstart.md) — full step-by-step pipeline
+- [Cookbook](cookbook.md) — interactive Jupyter notebook tutorials
diff --git a/docs/integrations/docling.md b/docs/integrations/docling.md
index 1b2f59ed..5a5593cf 100644
--- a/docs/integrations/docling.md
+++ b/docs/integrations/docling.md
@@ -13,7 +13,7 @@ Docling is integrated into Semantica's `parse` module via the `DoclingParser`. T
---
-## 📖 Integration Documentation
+## Integration Documentation
The `DoclingParser` provides a high-level interface for document processing. It supports:
@@ -42,7 +42,7 @@ For more details, see the [Parse Reference](../reference/parse.md).
---
-## 🧑🏽🍳 Integration Example
+## Integration Example
We provide a detailed cookbook and clear code examples to help you get started quickly.
@@ -81,7 +81,7 @@ See more in our [Code Examples](../CodeExamples.md).
---
-## 💻 GitHub Source
+## GitHub Source
The integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues.
@@ -89,7 +89,7 @@ The integration is open-source and available on GitHub. You can explore the impl
---
-## 📦 PyPI & Installation
+## PyPI & Installation
Docling is an optional but highly recommended dependency for Semantica. You can install it along with Semantica or as a separate requirement.
diff --git a/docs/integrations/snowflake.md b/docs/integrations/snowflake.md
index 8e624262..d0b1d092 100644
--- a/docs/integrations/snowflake.md
+++ b/docs/integrations/snowflake.md
@@ -13,7 +13,7 @@ Snowflake is integrated into Semantica's `ingest` module via the `SnowflakeInges
---
-## 📖 Integration Documentation
+## Integration Documentation
The `SnowflakeIngestor` provides a high-level interface for Snowflake data ingestion. It supports:
@@ -42,7 +42,7 @@ For more details, see the [Ingest Reference](../reference/ingest.md).
---
-## 🧑🏽🍳 Integration Example
+## Integration Example
We provide a detailed cookbook and clear code examples to help you get started quickly.
@@ -97,7 +97,7 @@ See more in our [Code Examples](../CodeExamples.md).
---
-## 💻 GitHub Source
+## GitHub Source
The integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues.
@@ -105,7 +105,7 @@ The integration is open-source and available on GitHub. You can explore the impl
---
-## 📦 PyPI & Installation
+## PyPI & Installation
Snowflake connector is an optional dependency for Semantica. You can install it along with Semantica or as a separate requirement.
@@ -128,7 +128,7 @@ For full installation details, see the [Installation Guide](../installation.md).
---
-## 🔐 Authentication Methods
+## Authentication Methods
Snowflake integration supports multiple authentication methods for different security requirements:
@@ -175,7 +175,7 @@ ingestor = SnowflakeIngestor(
---
-## 🚀 Advanced Features
+## Advanced Features
### Schema Introspection
```python
@@ -209,7 +209,7 @@ data = ingestor.ingest_query(
---
-## 📊 Best Practices
+## Best Practices
### Use Environment Variables
```python
@@ -244,7 +244,7 @@ for page in range(total_pages):
---
-## 🔍 Troubleshooting
+## Troubleshooting
### Connection Issues
```python
@@ -272,7 +272,7 @@ ingestor = SnowflakeIngestor(
---
-## 📚 See Also
+## See Also
- **[Ingest Module Reference](../reference/ingest.md)** - Complete ingestion documentation
- **[Getting Started Guide](../getting-started.md)** - Quick start with Semantica
diff --git a/docs/learning-more.md b/docs/learning-more.md
index def221ea..49299455 100644
--- a/docs/learning-more.md
+++ b/docs/learning-more.md
@@ -1,213 +1,104 @@
# Learning More
-Additional resources, tutorials, and advanced learning materials for Semantica.
-
-!!! info "About This Guide"
- This guide provides structured learning paths, quick references, troubleshooting guides, and advanced topics to help you master Semantica.
+Structured learning paths, quick references, and performance guidance for going deeper with Semantica.
---
-## Structured Learning Paths
+## Learning Paths
-- :material-school: **Beginner Path**
+- :material-school: **Beginner** (1–2 hours)
---
- Perfect for those new to Semantica and knowledge graphs.
-
+ New to Semantica and knowledge graphs.
-
- [Start Path](#beginner-path-1-2-hours)
+ [Start here](#beginner-path)
-- :material-compass: **Intermediate Path**
+- :material-compass: **Intermediate** (4–6 hours)
---
- For users comfortable with basics who want to build production applications.
-
+ Comfortable with basics, building production applications.
-
- [Start Path](#intermediate-path-4-6-hours)
+ [Start here](#intermediate-path)
-- :material-rocket: **Advanced Path**
+- :material-rocket: **Advanced** (8+ hours)
---
- For experienced users building enterprise applications.
-
+ Enterprise applications and customization.
-
- [Start Path](#advanced-path-8-hours)
+ [Start here](#advanced-path)
---
-### Beginner Path (1-2 hours)
+### Beginner Path
-1. **Installation & Setup** (15 min)
- - [Installation Guide](installation.md)
- - **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction
- - **Topics**: Framework overview, all modules, architecture, configuration
- - **Difficulty**: Beginner
- - **Time**: 30-45 minutes
- - **Use Cases**: First-time users, understanding the framework
-
-2. **Core Concepts** (30 min)
- - [Core Concepts](concepts.md)
- - [Getting Started Guide](getting-started.md)
- - **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources
- - **Topics**: File, web, feed, stream, database ingestion
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Loading data from various sources
-
-3. **First Knowledge Graph** (30 min)
- - [Quickstart Tutorial](quickstart.md)
- - **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph
- - **Topics**: Entity extraction, relationship extraction, graph construction, visualization
- - **Difficulty**: Beginner
- - **Time**: 20-30 minutes
- - **Use Cases**: Learning the basics, quick start
-
-4. **Basic Operations** (30 min)
- - [Examples](examples.md)
- - **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn entity extraction
- - **Topics**: Named entity recognition, entity types, extraction methods
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Understanding entity extraction
+1. **Installation & Setup** — [Installation Guide](installation.md)
+2. **Core Concepts** — [Core Concepts](concepts.md) + [Getting Started](getting-started.md)
+3. **First Knowledge Graph** — [Quickstart Tutorial](quickstart.md)
+4. **Interactive Introduction** — [Welcome to Semantica notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)
+5. **Hands-On Practice** — [Your First Knowledge Graph notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)
---
-### Intermediate Path (4-6 hours)
+### Intermediate Path
-1. **Advanced Concepts** (1 hour)
- - [Modules Guide](modules.md)
- - **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Advanced graph construction
- - **Topics**: Graph building, entity merging, conflict resolution, temporal graphs
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Production graph construction
- - **[Embeddings Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb)**: Learn embeddings
- - **Topics**: Embedding generation, similarity search, vector operations
- - **Difficulty**: Beginner
- - **Time**: 20-30 minutes
- - **Use Cases**: Understanding embeddings, semantic search
-
-2. **Use Cases** (1 hour)
- - [Use Cases Guide](use-cases.md)
- - **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Build production GraphRAG
- - **Topics**: GraphRAG, hybrid retrieval, graph traversal, LLM integration
- - **Difficulty**: Advanced
- - **Time**: 1-2 hours
- - **Use Cases**: Production GraphRAG systems
-
-3. **Advanced Examples** (1 hour)
- - [Examples](examples.md)
- - **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns
- - **Topics**: Custom entity types, domain-specific extraction, hybrid methods
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Domain-specific extraction
-
-4. **Quality & Optimization** (1 hour)
- - [Quality Assurance](concepts.md#8-quality-assurance)
- - [Performance Optimization](#performance-optimization)
- - **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Integrate multiple sources
- - **Topics**: Multi-source integration, entity resolution, conflict handling
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Building unified knowledge graphs
+1. **All Modules** — [Modules Guide](modules.md)
+2. **Advanced Graph Construction** — [Building Knowledge Graphs notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)
+3. **Embeddings & Search** — [Embeddings notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb)
+4. **GraphRAG** — [GraphRAG Complete notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
+5. **Multi-Source Integration** — [Multi-Source Data Integration notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)
+6. **Use Case Examples** — [Use Cases](use-cases.md)
---
-### Advanced Path (8+ hours)
+### Advanced Path
-1. **Advanced Architecture** (2 hours)
- - [Architecture Guide](architecture.md)
- - **[Temporal Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb)**: Build temporal graphs
- - **Topics**: Time-stamped entities, temporal relationships, historical queries
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Time-aware knowledge graphs
- - **[Ontology Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)**: Generate ontologies
- - **Topics**: Ontology generation, OWL, schema design
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Formal knowledge representation
-
-2. **Production Deployment** (2 hours)
- - [Security Best Practices](#security-best-practices)
- - **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production GraphRAG
- - **Topics**: Production deployment, scalability, optimization
- - **Difficulty**: Advanced
- - **Time**: 1-2 hours
- - **Use Cases**: Production systems
-
-3. **Customization** (2 hours)
- - **[Complete Visualization Suite Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)**: Advanced visualization
- - **Topics**: Custom layouts, filtering, styling, multiple graph types
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Production visualizations
- - **[Multi-Format Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)**: Advanced export patterns
- - **Topics**: Batch export, custom formats, format conversion
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Production exports
+1. **Architecture Deep Dive** — [Architecture Guide](architecture.md)
+2. **Temporal Graphs** — [Temporal Graphs notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb)
+3. **Ontologies** — [Ontology notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)
+4. **Visualization** — [Complete Visualization Suite notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)
+5. **Export Pipelines** — [Multi-Format Export notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)
+6. **Production GraphRAG** — [GraphRAG Complete notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
---
-## Quick Reference
+## Configuration Reference
-### Common Operations
-
-The typical workflow involves these steps:
-
-1. **Ingest** documents using `` `FileIngestor` ``
-2. **Parse** documents using `` `DocumentParser` ``
-3. **Extract** entities and relationships using `` `NERExtractor` `` and `` `RelationExtractor` ``
-4. **Build** knowledge graph using `` `GraphBuilder` ``
-5. **Generate** embeddings using `` `TextEmbedder` ``
-
-**For complete examples, see:**
-- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Complete workflow example
-- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: All modules overview
-
-### Configuration Reference
-
-| Setting | Environment Variable | Config File | Default |
-| :--- | :--- | :--- | :--- |
-| OpenAI API Key | `OPENAI_API_KEY` | `api_keys.openai` | `None` |
-| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `embedding.provider` | `"openai"` |
-| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `knowledge_graph.backend` | `"networkx"` |
+| Setting | Environment Variable | Default |
+|---------|---------------------|---------|
+| OpenAI API Key | `OPENAI_API_KEY` | `None` |
+| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `"openai"` |
+| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `"networkx"` |
---
-## Troubleshooting Guide
+## Troubleshooting
- :material-alert: **Import Errors**
---
`ModuleNotFoundError`
-
- **Solution**: Verify installation (`pip list`) and Python version (3.8+).
+
+ Verify installation: `pip list | grep semantica`. Ensure Python 3.8+.
- :material-key: **API Key Errors**
---
`AuthenticationError`
-
- **Solution**: Set `OPENAI_API_KEY` environment variable.
+
+ Set `OPENAI_API_KEY` (or the relevant provider key) as an environment variable.
- :material-memory: **Memory Errors**
---
- `MemoryError`
-
- **Solution**: Use batch processing and graph stores (Neo4j).
+ `MemoryError` or OOM crashes
+
+ Reduce batch sizes or switch to a persistent graph backend (Neo4j, FalkorDB).
- :material-speedometer: **Slow Processing**
---
- Long processing times
-
- **Solution**: Enable parallel processing and GPU acceleration.
+ Long runtimes on large datasets
+
+ Enable parallel processing (`PipelineBuilder` workers) and GPU acceleration.
@@ -215,80 +106,46 @@ The typical workflow involves these steps:
## Performance Optimization
-### 1. Batch Processing
+### Batch Processing
-Process multiple documents together for better throughput. Use batch processing when working with large document collections.
+Process documents in batches rather than one at a time. Configure chunk sizes based on available RAM.
-**For examples, see:**
-- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Batch ingestion patterns
-- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration
+### Parallel Execution
-### 2. Parallel Execution
+`PipelineBuilder` supports configurable worker counts per stage for independent operations.
-Use parallel processing for independent operations to improve performance on multi-core systems.
+### Backend Selection
-### 3. Backend Selection
+| Operation | NetworkX | Neo4j / FalkorDB |
+|-----------|----------|------------------|
+| Graph construction | Fast | Moderate |
+| Query performance | Moderate | Fast |
+| Scalability | Low (in-memory) | High (persistent) |
-| Operation | NetworkX | Neo4j |
-| :--- | :--- | :--- |
-| **Graph Construction** | ⚡⚡⚡ | ⚡⚡ |
-| **Query Performance** | ⚡⚡ | ⚡⚡⚡ |
-| **Scalability** | Low | High |
+Use NetworkX for development and smaller graphs; switch to a persistent backend for production at scale.
---
## Security Best Practices
-### API Key Management
+**API keys**
+- Store in environment variables or a secrets manager
+- Never hardcode keys or commit them to version control
+- Rotate keys regularly
-- **DO**: Use environment variables, rotate keys regularly.
-- **DON'T**: Hardcode keys, commit to version control.
-
-### Data Privacy
-
-- **DO**: Encrypt sensitive data, use local models.
-- **DON'T**: Send PII to external APIs without protection.
-
----
-
-## FAQ
-
-**Q: What is Semantica?**
-A: A framework for building knowledge graphs and semantic applications.
-
-**Q: Is Semantica free?**
-A: Yes, it is open source. Some features (e.g., OpenAI) require paid APIs.
-
-**Q: Can I use Semantica in production?**
-A: Yes, it is designed for production with proper configuration.
+**Data privacy**
+- Use local embedding models for sensitive data
+- Avoid sending PII to external APIs without appropriate data handling agreements
+- Encrypt sensitive graph exports at rest
---
## Next Steps
-Continue your learning journey:
-
-- **[Cookbook](cookbook.md)** - Interactive Jupyter notebook tutorials
-- **[API Reference](reference/core.md)** - Complete API documentation
-- **[Use Cases](use-cases.md)** - Real-world applications
-- **[Examples](examples.md)** - Code examples and patterns
-
-### 🍳 Recommended Next Cookbooks
-
-- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production GraphRAG system
- - **Topics**: GraphRAG, hybrid retrieval, LLM integration
- - **Difficulty**: Advanced
- - **Time**: 1-2 hours
- - **Use Cases**: Production RAG applications
-
-- **[RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)**: Understand the differences
- - **Topics**: RAG comparison, reasoning gap, inference engines
- - **Difficulty**: Intermediate
- - **Time**: 45-60 minutes
- - **Use Cases**: Choosing the right approach
-
----
-
-!!! info "Contribute"
- Have questions? [Open an issue](https://github.com/Hawksight-AI/semantica/issues) or [start a discussion](https://github.com/Hawksight-AI/semantica/discussions)!
+- [Cookbook](cookbook.md) — interactive Jupyter notebook tutorials
+- [API Reference](reference/core.md) — complete technical documentation
+- [Use Cases](use-cases.md) — real-world domain examples
+- [FAQ](faq.md) — common questions
+!!! info "Questions or feedback?"
+ [Open an issue](https://github.com/Hawksight-AI/semantica/issues) or [start a discussion](https://github.com/Hawksight-AI/semantica/discussions).
diff --git a/docs/modules.md b/docs/modules.md
index 23c36801..d03bc0a3 100644
--- a/docs/modules.md
+++ b/docs/modules.md
@@ -1,9 +1,9 @@
-# Modules & Architecture
+# Modules
-**Complete guide to Semantica's modular architecture and how to use each component.**
+Every Semantica module works independently — use only what you need.
-!!! tip "Modular Design"
- Each Semantica module works independently. Use only what you need for your specific use case.
+!!! tip "Just need a quick reference?"
+ Jump to the [Module Index](#module-index) at the bottom of this page.
---
@@ -82,13 +82,11 @@ web_ingestor = WebIngestor()
pages = web_ingestor.ingest_urls(["https://example.com"])
```
-**What it does:**
- **File formats** - PDF, DOCX, TXT, JSON, CSV
- **Web scraping** - Extract content from websites
- **Database** - Connect to SQL and NoSQL databases
- **Batch processing** - Handle large datasets efficiently
-**Use Cases:**
- Document processing pipelines
- Web data extraction
- Database integration
@@ -106,13 +104,11 @@ text = parsed["full_text"]
metadata = parsed["metadata"]
```
-**What it does:**
- **Text extraction** - Extract clean text from documents
- **Metadata parsing** - Extract titles, authors, dates
- **Structure analysis** - Identify sections, headings
- **OCR support** - Handle scanned documents
-**Use Cases:**
- PDF processing
- Document analysis
- Content extraction
@@ -130,13 +126,11 @@ splitter = TextSplitter(method="semantic")
chunks = splitter.split(text, chunk_size=1000, overlap=200)
```
-**What it does:**
- **Intelligent chunking** - Split text while preserving context
- **Semantic splitting** - Break at natural boundaries
- **Size control** - Manage chunk sizes for processing
- **Overlap handling** - Maintain context between chunks
-**Use Cases:**
- Document preprocessing
- Embedding preparation
- RAG systems
@@ -155,13 +149,11 @@ clean_text = normalizer.normalize_text(text)
standardized_date = normalizer.normalize_date("Jan 1st, 2020")
```
-**What it does:**
- **Text cleaning** - Remove noise and artifacts
- **Date standardization** - Convert to ISO format
- **Name normalization** - Standardize person names
- **Entity normalization** - Clean up company names
-**Use Cases:**
- Data preprocessing
- Quality improvement
- Standardization
@@ -186,13 +178,11 @@ rel_extractor = RelationExtractor()
relationships = rel_extractor.extract(text, entities)
```
-**What it does:**
- **Named Entity Recognition** - Find people, orgs, locations
- **Relationship extraction** - Find connections between entities
- **Custom entities** - Define your own entity types
- **Confidence scoring** - Quality assessment for extractions
-**Use Cases:**
- Knowledge graph construction
- Document analysis
- Information extraction
@@ -215,13 +205,11 @@ analyzer = GraphAnalyzer()
stats = analyzer.analyze(kg)
```
-**What it does:**
- **Graph construction** - Build knowledge graphs from data
- **Graph analysis** - Calculate metrics and statistics
- **Graph querying** - Search and retrieve information
- **Graph manipulation** - Merge, split, transform graphs
-**Use Cases:**
- Knowledge base creation
- Graph analytics
- Information retrieval
@@ -244,13 +232,11 @@ ontology.add_relationship("works_for", "Person", "Organization")
is_valid = ontology.validate_graph(kg)
```
-**What it does:**
- **Schema definition** - Define data structure
- **Data validation** - Ensure data conforms to schema
- **Inheritance** - Create hierarchical relationships
- **Constraints** - Enforce data quality rules
-**Use Cases:**
- Data modeling
- Quality assurance
- Schema management
@@ -268,13 +254,11 @@ engine = ReasoningEngine()
inferences = engine.infer(kg, rules=["transitivity", "symmetry"])
```
-**What it does:**
- **Logical inference** - Derive new facts from existing ones
- **Pattern matching** - Find complex patterns in data
- **Consistency checking** - Detect contradictions
- **Decision support** - Automated reasoning
-**Use Cases:**
- Knowledge discovery
- Decision making
- Consistency checking
@@ -295,13 +279,11 @@ embeddings = generator.generate(["text1", "text2"])
similarity = generator.similarity(embeddings[0], embeddings[1])
```
-**What it does:**
- **Text embeddings** - Convert text to vectors
- **Similarity search** - Find similar content
- **Clustering** - Group related items
- **AI integration** - Provide context to LLMs
-**Use Cases:**
- Semantic search
- Recommendation systems
- Clustering
@@ -320,13 +302,11 @@ store.add_vectors(embeddings, ids)
results = store.search(query_vector, top_k=10)
```
-**What it does:**
- **Vector storage** - Efficient vector database
- **Fast search** - Approximate nearest neighbor search
- **Indexing** - Optimize for performance
- **Batch operations** - Handle large datasets
-**Use Cases:**
- Semantic search
- RAG systems
- Recommendation engines
@@ -346,13 +326,11 @@ store.add_edges(relationships)
results = store.query("MATCH (n)-[r]->(m) RETURN n, r, m")
```
-**What it does:**
- **Graph persistence** - Store graphs in databases
- **Graph queries** - Cypher and Gremlin support
- **Graph algorithms** - Path finding, centrality
- **Transactions** - ACID compliance
-**Use Cases:**
- Knowledge graph storage
- Graph analytics
- Network analysis
@@ -371,13 +349,11 @@ store.add_triplets(subject, predicate, object)
triplets = store.get_triplets(entity="Apple Inc.")
```
-**What it does:**
- **Triple storage** - Store (subject, predicate, object) triples
- **Pattern matching** - Find specific patterns
- **RDF support** - Semantic web standards
- **Bulk operations** - Efficient batch processing
-**Use Cases:**
- Semantic web
- Knowledge representation
- Linked data
@@ -397,13 +373,11 @@ resolver = EntityResolver()
merged_entities = resolver.resolve(entities, strategy="semantic")
```
-**What it does:**
- **Duplicate detection** - Find similar entities
- **Entity resolution** - Merge duplicate records
- **Similarity scoring** - Quality assessment
- **Record linkage** - Connect related records
-**Use Cases:**
- Data cleaning
- Master data management
- Record linkage
@@ -422,13 +396,11 @@ conflicts = detector.detect_conflicts(kg)
resolved = detector.resolve(conflicts, strategy="most_recent")
```
-**What it does:**
- **Conflict detection** - Find contradictory information
- **Resolution strategies** - Automated conflict resolution
- **Source reliability** - Trustworthiness assessment
- **Temporal analysis** - Time-based conflict handling
-**Use Cases:**
- Data quality
- Consistency checking
- Trust management
@@ -448,13 +420,11 @@ manager = ContextManager()
context = manager.get_context(query, history)
```
-**What it does:**
- **Context tracking** - Maintain conversation context
- **Memory management** - Store and retrieve context
- **Relevance scoring** - Find relevant context
- **Session management** - Handle multiple conversations
-**Use Cases:**
- AI agents
- Chatbots
- Conversational AI
@@ -472,13 +442,11 @@ seed = SeedData()
knowledge = seed.get_knowledge("technology", "companies")
```
-**What it does:**
- **Seed knowledge** - Foundation data for domains
- **Knowledge bases** - Pre-built domain knowledge
- **Quick start** - Bootstrap applications
- **Domain models** - Industry-specific data
-**Use Cases:**
- Domain bootstrapping
- Quick start data
- Industry knowledge
@@ -496,13 +464,11 @@ provider = LLMProvider(model="gpt-4")
response = provider.generate(prompt, context=kg)
```
-**What it does:**
- **LLM integration** - Connect to various LLM providers
- **Prompt engineering** - Optimize prompts for results
- **Context injection** - Provide knowledge graph context
- **Response parsing** - Extract structured outputs
-**Use Cases:**
- AI generation
- Question answering
- Text completion
@@ -522,13 +488,11 @@ exporter = GraphExporter()
exporter.export(kg, format="json", filename="output.json")
```
-**What it does:**
- **Multiple formats** - JSON, CSV, RDF, GraphML
- **Database export** - Export to various databases
- **Streaming** - Handle large datasets
- **Filtering** - Export specific data subsets
-**Use Cases:**
- Data sharing
- System integration
- Backup and restore
@@ -546,13 +510,11 @@ visualizer = GraphVisualizer()
visualizer.plot(kg, layout="force_directed")
```
-**What it does:**
- **Graph visualization** - Interactive graph plots
- **Custom styling** - Tailored visual appearance
- **Analytics charts** - Statistics and metrics
- **Exploration tools** - Interactive data exploration
-**Use Cases:**
- Data exploration
- Presentation
- Analysis
@@ -573,13 +535,11 @@ pipeline.add_step("build", GraphBuilder())
result = pipeline.run("data/")
```
-**What it does:**
- **Workflow orchestration** - Coordinate multiple steps
- **Parallel processing** - Run steps concurrently
- **Progress tracking** - Monitor pipeline execution
- **Error handling** - Robust error management
-**Use Cases:**
- Data processing
- Workflow automation
- Batch processing
@@ -587,7 +547,7 @@ result = pipeline.run("data/")
---
-## New Features & Modules
+## Additional Modules
### Change Management Module
**Version control and audit trails**
@@ -599,13 +559,11 @@ manager = TemporalVersionManager(storage_path="versions.db")
snapshot = manager.create_snapshot(kg, "v1.0", "user@example.com", "Initial version")
```
-**What it does:**
- **Version control** - Track changes over time
- **Audit trails** - Complete change history
- **Data integrity** - SHA-256 checksums
- **Change comparison** - Detailed diff analysis
-**Use Cases:**
- Knowledge graph versioning
- Compliance tracking
- Data governance
@@ -623,13 +581,11 @@ manager = ProvenanceManager()
manager.track_entity("entity_1", "document.pdf", "person")
```
-**What it does:**
- **W3C PROV-O compliant** - Industry standard tracking
- **Complete lineage** - End-to-end traceability
- **Source attribution** - Track data origins
- **Integrity verification** - Tamper detection
-**Use Cases:**
- Regulatory compliance
- Data provenance
- Audit trails
@@ -648,13 +604,11 @@ semantica = Semantica(config=Config())
result = semantica.process("data/")
```
-**What it does:**
- **Framework orchestration** - Central coordination
- **Configuration management** - Settings and preferences
- **Lifecycle management** - Start/stop/restart
- **Plugin system** - Extensible architecture
-**Use Cases:**
- Framework initialization
- Configuration management
- Plugin development
@@ -662,46 +616,18 @@ result = semantica.process("data/")
---
-## Getting Started
+## Common Module Chains
-### Quick Start Example
-
-```python
-# Complete pipeline example
-from semantica.ingest import FileIngestor
-from semantica.semantic_extract import NERExtractor, RelationExtractor
-from semantica.kg import GraphBuilder
-from semantica.pipeline import Pipeline
-
-# Create pipeline
-pipeline = Pipeline()
-pipeline.add_step("ingest", FileIngestor())
-pipeline.add_step("ner", NERExtractor())
-pipeline.add_step("relations", RelationExtractor())
-pipeline.add_step("build", GraphBuilder())
-
-# Run pipeline
-kg = pipeline.run("documents/")
-print(f"Built graph with {len(kg['entities'])} entities")
-```
-
-### Choose Your Modules
-
-**For Document Processing:**
-- Ingest → Parse → Split → Semantic Extract → Knowledge Graph
-
-**For Web Scraping:**
-- Ingest (Web) → Normalize → Semantic Extract → Graph Store
-
-**For AI Agents:**
-- Context → LLM Providers → Reasoning → Export
-
-**For Analytics:**
-- Knowledge Graph → Graph Store → Visualization → Export
+| Goal | Modules |
+|------|---------|
+| Document processing | Ingest → Parse → Split → Semantic Extract → KG |
+| Web scraping | Ingest (Web) → Normalize → Semantic Extract → Graph Store |
+| AI agents | Context → LLM Providers → Reasoning → Export |
+| Analytics | KG → Graph Store → Visualization → Export |
---
-## Module Reference
+## Module Index
| Module | Purpose | Key Classes | Use Cases |
|--------|---------|-------------|-----------|
@@ -731,10 +657,9 @@ print(f"Built graph with {len(kg['entities'])} entities")
---
-## Need Help?
+## More
-- **Documentation**: [Getting Started](getting-started.md)
-- **Examples**: [Cookbook](cookbook.md)
-- **Community**: [Discord](community.md)
-- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
-- **Support**: [Contact Us](community.md)
+- [Getting Started](getting-started.md)
+- [Examples](examples.md)
+- [Cookbook](cookbook.md)
+- [API Reference](reference/core.md)
diff --git a/docs/quickstart.md b/docs/quickstart.md
index f3d9b00e..9c6d5392 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -1,246 +1,150 @@
# Quickstart
-Get started with Semantica in 5 minutes. This guide will walk you through building your first knowledge graph.
+Build your first knowledge graph in 5 minutes.
-!!! tip "Before You Start"
- Make sure you have Semantica installed. If not, follow the [Installation Guide](installation.md) first. This quickstart assumes basic Python knowledge.
+!!! tip "Prerequisites"
+ Semantica installed (`pip install semantica`). If not, see the [Installation Guide](installation.md).
-## Overview
+---
+
+## Pipeline Overview
```mermaid
flowchart LR
- A[Install] --> B[Initialize]
- B --> C[Load Data]
- C --> D[Extract]
- D --> E[Build Graph]
- E --> F[Visualize]
-
- style A fill:#e3f2fd
- style F fill:#c8e6c9
+ A[Ingest] --> B[Parse]
+ B --> C[Extract]
+ C --> D[Build Graph]
+ D --> E[Visualize / Export]
```
-## Step 1: Installation
+---
-If you haven't installed Semantica yet:
+## Step 1 — Ingest
-```bash
-pip install semantica
-```
+Load documents from files, directories, or the web.
-See the [Installation Guide](installation.md) for detailed instructions.
-
-!!! note "Installation Options"
- For production use, consider installing with optional dependencies for better performance: `pip install semantica[all]`. See the [Installation Guide](installation.md) for all options.
-
-## Step 2: Your First Knowledge Graph
-
-Building a knowledge graph involves these key steps:
-
-1. **Ingest** your documents using `FileIngestor`
-2. **Parse** documents to extract text using `DocumentParser` or `DoclingParser` (for enhanced layout support)
-3. **Extract** entities and relationships using `NERExtractor` and `RelationExtractor`
-4. **Build** the graph using `GraphBuilder`
-5. **Generate** embeddings (optional) using `TextEmbedder`
-
-**Quick Example:**
```python
from semantica.ingest import FileIngestor
-from semantica.parse import DocumentParser, DoclingParser
-from semantica.semantic_extract import NERExtractor, RelationExtractor
-from semantica.kg import GraphBuilder
-# 1. Ingest document
ingestor = FileIngestor()
sources = ingestor.ingest("data/sample.pdf")
-
-# 2. Parse (choose your parser)
-# Option A: Standard parser
-parser = DocumentParser()
-parsed_content = parser.parse(sources[0])
-
-# Option B: Enhanced Docling parser (recommended for complex tables)
-# docling_parser = DoclingParser()
-# parsed_content = docling_parser.parse(sources[0])
-
-# 3. Extract entities and relations
-ner = NERExtractor()
-entities = ner.extract(parsed_content)
-
-relations = RelationExtractor()
-relationships = relations.extract(parsed_content, entities=entities)
-
-# 4. Build graph
-builder = GraphBuilder()
-graph = builder.build(entities=entities, relationships=relationships)
-
-print(f"Built knowledge graph with {len(graph.nodes)} nodes and {len(graph.edges)} edges")
```
-**For complete step-by-step examples with detailed explanations, see:**
-- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Full tutorial with detailed explanations and expected outputs
- - **Topics**: Entity extraction, relationship extraction, graph construction, visualization
- - **Difficulty**: Beginner
- - **Time**: 20-30 minutes
- - **Use Cases**: Learning the basics, quick start
+Supported formats: PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives. For web content, use `WebIngestor`.
-## Step 3: Extract Entities and Relationships
+---
-The semantic extraction step identifies named entities (people, organizations, locations) and relationships between them from your text.
+## Step 2 — Parse
-**What gets extracted:**
-- **Entities**: People, organizations, locations, dates, and other named entities
-- **Relationships**: Connections between entities (e.g., `founded_by`, `located_in`, `has_ceo`)
+Extract structured text from raw documents.
-**For detailed examples and different extraction methods, see:**
-- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn different NER methods and configurations
- - **Topics**: Named entity recognition, entity types, confidence scores
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Understanding entity extraction options
+```python
+from semantica.parse import DocumentParser
-- **[Relation Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)**: Learn to extract relationships between entities
- - **Topics**: Relationship extraction, dependency parsing, semantic role labeling
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Building rich knowledge graphs with relationships
+parser = DocumentParser()
+parsed = parser.parse(sources[0])
+```
-## Step 4: Build Knowledge Graph from Multiple Sources
+For complex layouts (tables, columns): use `DoclingParser` instead — it handles PDF tables and structured DOCX/PPTX better.
-You can combine data from multiple sources (files, web, databases) to build a unified knowledge graph. The process involves:
+---
-1. **Ingest** from multiple sources using different ingestors
-2. **Parse** all documents to extract text
-3. **Extract** entities and relationships from each source
-4. **Build** a unified graph with entity merging enabled
+## Step 3 — Extract Entities and Relationships
-**For complete examples with multiple sources, see:**
-- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from files, web, feeds, streams, and databases
- - **Topics**: File, web, feed, stream, database ingestion
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Loading data from various sources
+```python
+from semantica.semantic_extract import NERExtractor, RelationExtractor
-- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced patterns for integrating multiple data sources
- - **Topics**: Multi-source integration, entity resolution, conflict handling
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Building knowledge graphs from diverse data sources
+ner = NERExtractor()
+entities = ner.extract(parsed)
-## Step 5: Visualize Your Knowledge Graph
+rel = RelationExtractor()
+relationships = rel.extract(parsed, entities=entities)
+```
-Visualization helps you understand and explore your knowledge graph structure. Semantica supports multiple visualization formats including interactive HTML, static images, and export formats.
+Each entity gets a type, confidence score, and source reference. Relationships are extracted as typed triplets: `(subject, predicate, object)`.
-**For detailed visualization examples, see:**
-- **[Visualization Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/16_Visualization.ipynb)**: Learn to create interactive and static visualizations
- - **Topics**: Network graphs, interactive HTML, static images, export formats
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Exploring graph structure, presentations, analysis
+---
-- **[Complete Visualization Suite Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)**: Advanced visualization techniques
- - **Topics**: Custom layouts, filtering, styling, multiple graph types
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Production visualizations, custom dashboards
+## Step 4 — Build the Knowledge Graph
-## Step 6: Export Your Knowledge Graph
+```python
+from semantica.kg import GraphBuilder
-Export your knowledge graph to various formats for integration with other systems or tools. Semantica supports RDF, JSON, CSV, OWL, GraphML, and more.
+builder = GraphBuilder(merge_entities=True)
+graph = builder.build(entities=entities, relationships=relationships)
-**Supported export formats:**
-- **RDF**: Turtle, RDF/XML, JSON-LD, N-Triples
-- **JSON**: Standard JSON, JSON-LD, Cytoscape.js format
-- **CSV**: Node and edge lists for spreadsheet tools
-- **OWL**: OWL/XML and Turtle for ontologies
-- **Graph Formats**: GraphML, GEXF, DOT for visualization tools
+print(f"{len(graph.nodes)} nodes, {len(graph.edges)} edges")
+```
-**For detailed export examples, see:**
-- **[Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/15_Export.ipynb)**: Learn to export to all supported formats
- - **Topics**: RDF, JSON, CSV, OWL, GraphML export
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Data integration, sharing knowledge graphs
+`merge_entities=True` resolves duplicates across sources automatically.
-- **[Multi-Format Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)**: Advanced export patterns
- - **Topics**: Batch export, custom formats, format conversion
- - **Difficulty**: Intermediate
- - **Time**: 30-45 minutes
- - **Use Cases**: Production exports, format migration
+---
+
+## Step 5 — Visualize
+
+```python
+from semantica.visualization import GraphVisualizer
+
+viz = GraphVisualizer()
+viz.visualize(graph, output="graph.html") # interactive HTML
+```
+
+---
+
+## Step 6 — Export
+
+```python
+from semantica.export import RDFExporter
+
+exporter = RDFExporter()
+rdf = exporter.export_to_rdf(graph, format="turtle")
+```
+
+Other formats: `"json-ld"`, `"nt"`, `"xml"`, Parquet, ArangoDB AQL. See [Export Reference](reference/export.md).
+
+---
## Common Patterns
-### Pattern 1: Process Text Directly
+### Process text directly (no file)
-You can process text directly without file ingestion. This is useful when you already have text content in memory.
+```python
+from semantica.semantic_extract import NERExtractor
-**For examples, see:**
-- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Processing text directly
-- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Graph construction from text
+ner = NERExtractor()
+entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
+```
-### Pattern 2: Custom Entity Extraction
+### Incremental build from multiple sources
-Configure entity extraction with different methods (ML models, LLMs) and parameters for your specific needs.
+```python
+from semantica.kg import GraphBuilder
-**For examples, see:**
-- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Different extraction methods and configurations
-- **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns
+all_entities, all_rels = [], []
+for doc in parsed_docs:
+ all_entities.extend(ner.extract(doc))
+ all_rels.extend(rel.extract(doc, entities=all_entities))
-### Pattern 3: Incremental Building
+graph = GraphBuilder(merge_entities=True).build(
+ entities=all_entities, relationships=all_rels
+)
+```
-Build knowledge graphs incrementally from multiple sources and merge them together.
-
-**For examples, see:**
-- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Graph construction and merging
-- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration patterns
-
-## Next Steps
-
-Now that you've built your first knowledge graph:
-
-1. **[Explore Examples](examples.md)** - See more advanced use cases
-2. **[API Reference](reference/core.md)** - Learn about all available methods
-3. **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks
-4. **[Full Documentation](https://github.com/Hawksight-AI/semantica/blob/main/README.md)** - Comprehensive guide
-
-### 🍳 Recommended Cookbook Tutorials
-
-Continue learning with these interactive tutorials:
-
-- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules
- - **Topics**: Framework overview, all modules, architecture, configuration
- - **Difficulty**: Beginner
- - **Time**: 30-45 minutes
- - **Use Cases**: Understanding the complete framework
-
-- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph
- - **Topics**: Entity extraction, relationship extraction, graph construction, visualization
- - **Difficulty**: Beginner
- - **Time**: 20-30 minutes
- - **Use Cases**: Hands-on practice with complete workflow
-
-- **[Data Ingestion](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources
- - **Topics**: File, web, feed, stream, database ingestion
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Loading data from various sources
-
-- **[Document Parsing](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)**: Parse various document formats
- - **Topics**: PDF, DOCX, HTML, JSON parsing
- - **Difficulty**: Beginner
- - **Time**: 15-20 minutes
- - **Use Cases**: Extracting text from different file formats
+---
## Troubleshooting
-### Common Issues
+| Problem | Fix |
+|---------|-----|
+| No entities extracted | Check the document has machine-readable text (not just scanned images) |
+| Slow processing | Process in chunks; use GPU acceleration (`pip install semantica[gpu]`) |
+| Memory errors | Reduce batch size or switch to a persistent graph backend |
-**Issue**: No entities extracted
-- **Solution**: Check that your document contains text content. PDFs with images only won't work without OCR.
+---
-**Issue**: Slow processing
-- **Solution**: For large documents, consider processing in chunks or using GPU acceleration.
+## Next Steps
-**Issue**: Memory errors
-- **Solution**: Process documents one at a time or reduce batch sizes.
-
-Need help? Check the [Installation Troubleshooting](installation.md#troubleshooting) or [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues).
+- [Core Concepts](concepts.md) — understand how knowledge graphs and reasoning work
+- [Modules Guide](modules.md) — every module explained
+- [Use Cases](use-cases.md) — domain-specific examples
+- [Cookbook](cookbook.md) — interactive Jupyter notebooks for each step
diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md
index 7fb9a1a4..47d40499 100644
--- a/docs/reference/change_management.md
+++ b/docs/reference/change_management.md
@@ -58,16 +58,14 @@ The Semantica change management module provides enterprise-grade version control
+### Biomedical Knowledge Graphs
-- :material-dna: **Biomedical Knowledge Graphs**
- ---
- Accelerate drug discovery and understand disease pathways by connecting genes, proteins, drugs, and diseases.
-
- **Goal**: Connect genes, proteins, drugs, and diseases from scientific literature and databases.
-
- **Difficulty**: Intermediate
-
- [:material-arrow-right: Drug Discovery Pipeline](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)
-
- [:material-arrow-right: Genomic Variant Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb)
+Connect genes, proteins, drugs, and diseases from scientific literature and databases to accelerate drug discovery and understand disease pathways.
-
-
-### Biomedical Knowledge Graphs Implementation
-
-**Prerequisites**:
-- Domain knowledge of biomedical concepts
-- Access to biomedical literature/databases
-
-**Implementation Guides:**
-
-- **[Drug Discovery Pipeline Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)**: Build knowledge graphs from PubMed RSS feeds
- - **Topics**: PubMed RSS ingestion, entity-aware chunking, GraphRAG, vector similarity search
- - **Difficulty**: Intermediate
- - **Time**: 1-2 hours
- - **Use Cases**: Drug discovery, biomedical research
-
-- **[Genomic Variant Analysis Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb)**: Analyze genomic variants using temporal knowledge graphs
- - **Topics**: bioRxiv RSS, temporal KGs, deduplication, pathway analysis
- - **Difficulty**: Intermediate
- - **Time**: 1-2 hours
- - **Use Cases**: Genomic research, variant analysis
+**Cookbooks**:
+- [Drug Discovery Pipeline](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) — PubMed RSS ingestion, entity-aware chunking, GraphRAG, vector similarity search
+- [Genomic Variant Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) — bioRxiv RSS, temporal KGs, deduplication, pathway analysis
---
## Finance & Trading
-
+### Financial Data Integration
-- :material-finance: **Financial Data Integration**
- ---
- Integrate financial data from multiple sources using MCP servers and real-time ingestion.
-
- **Goal**: Connect Alpha Vantage API, MCP servers, seed data, and real-time ingestion for comprehensive financial analysis.
-
- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb)
+Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph.
-- :material-shield-alert: **Fraud Detection**
- ---
- Detect complex fraud rings using temporal knowledge graphs and pattern detection.
-
- **Goal**: Build a graph of Users, Devices, IP Addresses, and Transactions to find cycles and detect fraud patterns.
-
- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb)
+**Cookbook**: [Financial Data Integration (MCP)](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) — Alpha Vantage API, MCP servers, seed data, real-time ingestion
-- :material-bitcoin: **Blockchain Analytics**
- ---
- Analyze DeFi protocols and transaction networks for intelligence and fraud detection.
-
- **Goal**: Map transaction flows between wallets and exchanges, analyze DeFi protocols, and detect illicit activity.
-
- [:material-arrow-right: DeFi Protocol Intelligence](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb)
-
- [:material-arrow-right: Transaction Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb)
+### Fraud Detection
-
+Detect complex fraud rings using temporal graphs and pattern detection over transaction, device, and user data.
----
+**Cookbook**: [Fraud Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) — temporal KGs, cycle detection, fraud pattern analysis
+### Blockchain Analytics
+
+Map transaction flows, analyze DeFi protocols, and detect illicit activity across wallet and exchange networks.
+
+**Cookbooks**:
+- [DeFi Protocol Intelligence](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb)
+- [Transaction Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb)
---
## Security & Intelligence
-
+### Cybersecurity Threat Intelligence
-- :material-shield-lock: **Cybersecurity Threat Intelligence**
- ---
- Proactively identify and mitigate cyber threats using real-time anomaly detection and threat intelligence.
-
- **Goal**: Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs.
-
- [:material-arrow-right: Real-Time Anomaly Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)
-
- [:material-arrow-right: Threat Intelligence Hybrid RAG](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb)
+Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs for proactive defense.
-- :material-account-network: **Criminal Network Analysis**
- ---
- Analyze criminal networks to identify key players, communities, and suspicious patterns using OSINT RSS feeds, deduplication, and network centrality analysis.
-
- **Goal**: Build knowledge graphs from police reports, court records, and surveillance data.
-
- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb)
+**Cookbooks**:
+- [Real-Time Anomaly Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)
+- [Threat Intelligence Hybrid RAG](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb)
-- :material-file-search: **Intelligence Analysis Orchestrator Worker**
- ---
- Comprehensive intelligence analysis using pipeline orchestrator with multiple RSS feeds, conflict detection, and multi-source integration.
-
- **Goal**: Process multiple intelligence sources in parallel using orchestrator-worker pattern.
-
- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb)
+### Criminal Network Analysis
-
+Build knowledge graphs from police reports, court records, and OSINT feeds to identify key players, communities, and suspicious patterns using network centrality analysis.
+
+**Cookbook**: [Criminal Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb)
+
+### Intelligence Analysis Orchestrator
+
+Process multiple intelligence sources in parallel using an orchestrator-worker pipeline pattern with multi-source conflict detection and integration.
+
+**Cookbook**: [Intelligence Analysis Orchestrator-Worker](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb)
---
## Industry & Operations
-
+### Supply Chain Optimization
-- :material-truck-delivery: **Supply Chain Optimization**
- ---
- Visualize and optimize complex global supply chains.
-
- **Goal**: Map suppliers, logistics routes, and inventory levels to identify bottlenecks.
-
- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb)
+Map suppliers, logistics routes, and inventory levels to identify bottlenecks and optimize global supply chains.
-- :material-wind-turbine: **Renewable Energy Management**
- ---
- Optimize grid operations and asset maintenance.
-
- **Goal**: Connect sensor data, weather forecasts, and maintenance logs to predict failures.
-
- [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb)
+**Cookbook**: [Supply Chain Data Integration](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb)
-
+### Renewable Energy Management
+
+Connect sensor data, weather forecasts, and maintenance logs to predict equipment failures and optimize grid operations.
+
+**Cookbook**: [Energy Market Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb)
---
## Advanced AI Patterns
-
+### GraphRAG (Graph-Augmented Generation)
-- :material-robot: **Graph-Augmented Generation (GraphRAG)**
- ---
- Enhance LLM responses with structured ground truth using knowledge graphs.
-
- **Goal**: Use the knowledge graph to retrieve precise context for RAG applications with hybrid retrieval and logical inference.
-
- [:material-arrow-right: GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
-
- [:material-scale-balance: RAG vs GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
+Use knowledge graphs to retrieve precise, structured context for LLM responses — with hybrid retrieval, logical inference, and source attribution.
-
-
----
-
-
----
-
-## Summary
-
-This guide covered use cases across multiple domains with corresponding cookbooks:
-
-- **Research & Science**: Biomedical knowledge graphs (Drug Discovery, Genomic Variant Analysis)
-- **Finance & Trading**: Financial data integration, fraud detection, blockchain analytics
-- **Security & Intelligence**: Cybersecurity threat intelligence, criminal network analysis, intelligence orchestration
-- **Industry**: Supply chain optimization, renewable energy management
-- **AI Applications**: GraphRAG (Complete implementation and comparison)
+**Cookbooks**:
+- [GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — production-ready implementation
+- [RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) — side-by-side comparison
---
## Next Steps
-- **[Examples](examples.md)** - More detailed code examples
-- **[Modules Guide](modules.md)** - Learn about available modules
-- **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks
-- **[API Reference](reference/core.md)** - Complete API documentation
-
----
-
-!!! info "Contribute"
- Have a use case to add? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica)
+- [Cookbook](cookbook.md) — full notebook catalog organized by topic and difficulty
+- [Modules Guide](modules.md) — every module with examples
+- [API Reference](reference/core.md) — complete technical documentation
+!!! info "Have a use case to add?"
+ [Open a PR](https://github.com/Hawksight-AI/semantica) or start a discussion on GitHub.
diff --git a/mkdocs_local.yml b/mkdocs_local.yml
new file mode 100644
index 00000000..26261fc3
--- /dev/null
+++ b/mkdocs_local.yml
@@ -0,0 +1,14 @@
+INHERIT: mkdocs.yml
+plugins:
+ - search:
+ lang: en
+ - minify:
+ minify_html: true
+ - mkdocstrings:
+ handlers:
+ python:
+ options:
+ docstring_style: google
+ show_source: true
+ show_root_heading: true
+ show_category_heading: true