mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: rewrite and polish documentation site

Browse Source 
- Rewrote index.md to match README (tagline, badges, Problem/Solution text)
- Improved getting-started, concepts, quickstart, installation, faq, use-cases, contributing, glossary, learning-more, examples, modules, architecture, cookbook, deep-dive pages: tighter prose, fixed headings/bullets, removed inconsistencies and duplicate sections
- Removed overuse of emojis from headings in integration pages (docling, snowflake)
- Fixed change_management reference page: closed unclosed JSON code block that broke the right TOC, demoted noisy sub-headings to bold text
- CSS layout: widened content area (max-width 1440px grid, left sidebar 11rem, right TOC narrowed to 11rem for broader content), tightened TOC spacing and font size, fixed word-wrap/overflow on TOC links
- Added mkdocs_local.yml for local serving without mkdocs-jupyter plugin

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
KaifAhmad1
2026-03-26 18:31:39 +05:30
co-authored by Claude Sonnet 4.6
parent 21c7933ec1
commit 129edaf05b
20 changed files with 1319 additions and 2558 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+68 -122
View File
@@ -1,180 +1,126 @@
# Architecture
Semantica's modular, extensible framework for semantic intelligence and knowledge engineering.
Semantica is built around a three-layer, modular architecture designed for independent use of components, clean separation of concerns, and extensibility at each layer.
---
## Design Principles
- **Modular**: Independent, reusable components
- **Extensible**: Easy to add new functionality
- **Scalable**: Handle large-scale data processing
- **Maintainable**: Clear separation of concerns
---
## System Architecture
## System Overview
```mermaid
graph TB
A[Data Ingestion Layer] --> B[Semantic Processing Layer]
B --> C[Application Layer]
A1[Files • Web • APIs • Streams] --> A
B1[Parse • Normalize • Extract • Build] --> B
C1[GraphRAG • AI Agents • Analytics] --> C
A1[Files · Web · APIs · Streams] --> A
B1[Parse · Normalize · Extract · Build] --> B
C1[GraphRAG · AI Agents · Analytics] --> C
```
### Three-Layer Architecture
**1. Data Ingestion Layer**
- Multiple file formats (PDF, DOCX, JSON, CSV, etc.)
- Web scraping and APIs
- Real-time streams (Kafka, RabbitMQ)
- Database connectors (SQL, NoSQL)
**2. Semantic Processing Layer**
- Document parsing and normalization
- Entity and relationship extraction
- Embedding generation
- Knowledge graph construction
- Quality assurance and deduplication
**3. Application Layer**
- GraphRAG for enhanced retrieval
- AI agent memory and context
- Multi-agent systems
- Analytics and visualization
---
## Core Modules
## Three-Layer Architecture
### Orchestration
- **`semantica.core`** - Main framework class and coordination
- **`semantica.pipeline`** - Pipeline management and execution
### 1. Data Ingestion Layer
### Data Processing
- **`semantica.ingest`** - Universal data ingestion
- **`semantica.parse`** - Document parsing
- **`semantica.normalize`** - Data cleaning and normalization
Responsible for loading data from any source into the pipeline.
### Semantic Intelligence
- **`semantica.semantic_extract`** - Entity and relationship extraction
- **`semantica.embeddings`** - Vector embedding generation
- **`semantica.ontology`** - Ontology generation and management
- **File formats** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives
- **Web** — crawl via `WebIngestor` with configurable depth
- **Databases** — SQL, NoSQL, Snowflake via `DBIngestor` / `SnowflakeIngestor`
- **Streams** — Kafka, real-time feeds
### Knowledge Graphs
- **`semantica.kg`** - Knowledge graph construction
- **`semantica.vector_store`** - Vector storage (Weaviate, FAISS)
- **`semantica.triplet_store`** - RDF triplet storage (Jena, Blazegraph)
- **`semantica.graph_store`** - Property graphs (Neo4j, FalkorDB)
### 2. Semantic Processing Layer
### Quality Assurance
- **`semantica.deduplication`** - Entity deduplication
- **`semantica.conflicts`** - Conflict detection and resolution
The core intelligence engine — transforms raw data into structured knowledge.
- Document parsing and normalization
- Entity and relationship extraction (NER, LLM-typed, rule-based)
- Embedding generation
- Knowledge graph construction with entity merging
- Deduplication, conflict detection, and validation
### 3. Application Layer
Consumes the knowledge graph for downstream use cases.
- GraphRAG — graph-grounded retrieval for LLMs
- AI agent context and decision tracking
- Multi-agent pipelines
- Analytics, visualization, and export
---
## Data Flow
```
1. Ingestion → Raw data from sources
2. Parsing → Structured content extraction
3. Normalization → Cleaned data
4. Semantic Extraction → Entities, relationships, events
5. Graph Construction → Entity resolution, conflict resolution
6. Quality Assurance → Deduplication, validation
7. Storage → Vector, triplet, and graph stores
8. Application → GraphRAG, agents, analytics
Ingest → raw data from sources
Parse → structured text extraction
Normalize → canonical forms, date/name standardization
Extract → entities, relationships, events
Build → entity resolution, graph construction
QA → deduplication, conflict resolution, validation
Store → vector store, graph store, triplet store
Deliver → GraphRAG, agents, export, visualization
```
---
## Module Map
| Layer | Modules |
|-------|---------|
| **Ingestion** | `ingest`, `parse`, `split`, `normalize` |
| **Semantic** | `semantic_extract`, `kg`, `ontology`, `reasoning` |
| **Storage** | `embeddings`, `vector_store`, `graph_store`, `triplet_store` |
| **Quality** | `deduplication`, `conflicts` |
| **Context** | `context`, `provenance`, `change_management` |
| **Output** | `export`, `visualization`, `pipeline` |
For full module documentation, see the [Modules Guide](modules.md).
---
## Extension Points
### Custom Ingestors
### Custom Ingestor
```python
from semantica.ingest import BaseIngestor
class CustomIngestor(BaseIngestor):
def ingest(self, source):
# Custom ingestion logic
pass
# Return a list of document dicts
...
```
### Custom Extractors
### Custom Extractor
```python
from semantica.semantic_extract import BaseExtractor
class CustomExtractor(BaseExtractor):
def extract(self, text):
# Custom extraction logic
pass
# Return a list of entity dicts
...
```
### Custom Validators
Validators can be implemented within domain-specific modules (e.g., graph or ontology) as needed.
---
## Design Decisions
### Modularity
Independent components that can be used standalone or together. Easy to test, maintain, and extend.
**Modularity** — every component can be used standalone. Import only what you need; the framework never forces a full stack.
### Plugin System
Extensible architecture allowing custom functionality without modifying core code.
**Pluggability** — extend any layer without modifying core code. Custom ingestors, extractors, validators, and exporters all follow the same base class pattern.
### Configuration Management
Centralized configuration with environment variable support for different deployment environments.
**Configuration over convention** — centralized config with environment variable overrides for deployment flexibility.
### Error Handling
Comprehensive error handling with graceful degradation and recovery mechanisms.
**Provenance by default** — lineage tracking is built into graph construction, not bolted on. Every node traces back to a source document.
---
## Performance
## Performance Characteristics
**Scalability**
- Parallel processing support
- Streaming for large datasets
- Efficient memory usage
- Intelligent caching
**Optimization**
- Lazy loading
- Batch processing
- Connection pooling
- Query optimization
---
## Security
**Data Security**
- Secure credential handling
- Input validation and output sanitization
- Audit logging
**Access Control**
- Authentication and authorization
- API key management
- Role-based access control
---
## Future Roadmap
- Distributed processing
- Real-time streaming improvements
- Advanced reasoning capabilities
- Multi-modal expansion
- Enhanced visualization
---
For detailed module documentation, see [Modules Guide](modules.md)
- **Parallel execution** — `PipelineBuilder` supports configurable worker counts per stage
- **Delta processing** — incremental graph updates without full recompute
- **Streaming ingestion** — process large corpora without loading everything into memory
- **Backend flexibility** — swap in-memory NetworkX for Neo4j/FalkorDB at scale with no API changes
docs/concepts.md
+75 -275
View File
@@ -1,372 +1,172 @@
# Core Concepts
# Core Concepts
**Learn the fundamental concepts behind Semantica in simple, practical terms.**
The fundamental ideas behind Semantica — explained plainly.
!!! tip "Quick Start"
New to Semantica? Start with [Getting Started](getting-started.md) for hands-on examples.
!!! tip "New here?"
Start with [Getting Started](getting-started.md) for hands-on examples, then come back to this page for deeper understanding.
---
## What is Semantica?
Semantica transforms unstructured data (documents, web pages, reports) into **knowledge graphs** - structured databases that AI systems can understand and reason about.
Semantica transforms unstructured data (documents, web pages, reports, databases) into **knowledge graphs** — structured representations that AI systems can query, reason about, and trace back to sources.
**What it does:**
- **Reads** documents, PDFs, web pages, databases
- **Extracts** entities (people, companies, dates) and relationships
- **Builds** connected knowledge graphs
- **Enables** AI to reason with structured knowledge
---
## Core Architecture
Semantica uses a **layered architecture** - use only what you need:
<div class="grid cards" markdown>
- **Input Layer**
---
Data ingestion and preparation
**Modules**: Ingest, Parse, Split, Normalize
- **Semantic Layer**
---
Intelligence and understanding
**Modules**: Semantic Extract, Knowledge Graph, Ontology, Reasoning
- **Storage Layer**
---
Persistent data storage
**Modules**: Embeddings, Vector Store, Graph Store
- **Quality Layer**
---
Data quality and consistency
**Modules**: Deduplication, Conflicts
- **Context & Memory**
---
Agent memory and foundation data
**Modules**: Context, Seed, LLM Providers
- **Output & Orchestration**
---
Export, visualization, and workflows
**Modules**: Export, Visualization, Pipeline
</div>
At its core, Semantica adds a **context and intelligence layer** on top of your existing AI stack: it doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs accountable.
---
## Knowledge Graphs
The foundation of Semantica - turning data into structured knowledge.
The foundation of everything in Semantica.
### What is a Knowledge Graph?
A knowledge graph stores information as:
A knowledge graph represents real-world information as:
- **Nodes** (entities): People, companies, locations, dates
- **Edges** (relationships): works_for, located_in, founded_by
- **Properties**: Name, date, confidence score, source
- **Nodes (entities)** — people, companies, locations, events, concepts
- **Edges (relationships)** — `works_for`, `located_in`, `founded_by`
- **Properties** — name, date, confidence score, source URL
### Why Knowledge Graphs?
- **Searchable**: Find information instantly
- **Connectable**: Discover hidden relationships
- **Queryable**: Ask complex questions
- **Explainable**: Trace answers back to sources
This structure makes knowledge **searchable**, **connectable**, **queryable**, and — critically — **explainable**: every answer can be traced back to the facts and relationships that produced it.
---
## Entity Extraction (NER)
Finding and classifying entities in text.
Scanning text to find and classify real-world entities.
### What it does:
- Scans text for people, organizations, locations, dates
- Classifies each entity by type
- Assigns confidence scores
- Tracks source provenance
### Example Output:
```python
# From: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."
# Input: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."
{
"entities": [
{"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98},
{"text": "Steve Jobs", "type": "PERSON", "confidence": 0.99},
{"text": "1976", "type": "DATE", "confidence": 0.95},
{"text": "Cupertino", "type": "LOCATION", "confidence": 0.97}
{"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98},
{"text": "Steve Jobs", "type": "PERSON", "confidence": 0.99},
{"text": "1976", "type": "DATE", "confidence": 0.95},
{"text": "Cupertino", "type": "LOCATION", "confidence": 0.97}
]
}
```
Each entity gets a type, confidence score, and a link to its source document.
---
## Relationship Extraction
Finding connections between entities.
Finding how entities connect to each other.
### What it does:
- Identifies how entities relate to each other
- Extracts relationship types and directions
- Provides context and confidence
- Links to source documents
### Example Output:
```python
{
"relationships": [
{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", "confidence": 0.92},
{"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino", "confidence": 0.89}
{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", "confidence": 0.92},
{"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino", "confidence": 0.89}
]
}
```
Relationships can be extracted via rule-based methods, ML models, or LLMs (with `"llm_typed"` metadata).
---
## Embeddings
Turning text into numerical vectors for AI understanding.
Embeddings convert text into numerical vectors so that AI systems can measure semantic similarity — finding related concepts even when the exact words differ.
### What are embeddings?
- **Numerical representations** of text, entities, and relationships
- **Similarity calculations** - find related concepts
- **AI-powered search** - semantic understanding
- **Clustering and grouping** - discover patterns
Semantica uses embeddings for:
### Use Cases:
- **Semantic Search** - find documents by meaning, not keywords
- **Entity Resolution** - match similar entities across sources
- **Recommendations** - suggest related content
- **AI Input** - provide structured context to LLMs
---
## Temporal Graphs
Knowledge graphs that understand time.
### What they track:
- **When** events happened
- **How** entities changed over time
- **Temporal relationships** - before, after, during
- **Historical context** - point-in-time snapshots
### Example Uses:
- **Company History** - track mergers, leadership changes
- **Person Careers** - job changes, relocations
- **Policy Evolution** - law changes over time
- **Research Progress** - scientific discoveries timeline
- **Semantic search** — retrieve by meaning, not just keywords
- **Entity resolution** — match the same entity across different sources
- **Precedent search** — find similar past decisions
- **GraphRAG retrieval** — hybrid vector + graph traversal
---
## GraphRAG
Enhanced AI retrieval using knowledge graphs.
GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses by grounding them in a structured knowledge graph rather than raw text chunks alone.
### How it works:
1. **Query** user question
2. **Retrieve** relevant graph context
3. **Enhance** with relationships and entities
4. **Generate** AI response with sources
How it works:
### Benefits:
- **More accurate** answers
- **Source attribution** - trace answers back
- **Context awareness** - understand relationships
- **Reduced hallucination** - grounded in facts
1. User submits a query
2. Semantica retrieves relevant graph context (entities, relationships, reasoning paths)
3. The LLM generates a response grounded in that context
4. Every claim in the response links back to a source node in the graph
This eliminates the hallucination and traceability problems of standard RAG.
---
## Ontology
Defining the structure and rules of your knowledge.
An ontology defines the schema and rules for your knowledge — what entity types exist, which relationships are valid, and what constraints apply.
### What it provides:
- **Schema definition** - what types exist
- **Relationship rules** - valid connections
- **Property constraints** - required fields
- **Inheritance hierarchies** - parent-child relationships
### Example:
```python
# Define ontology structure
ontology = {
"classes": ["Person", "Organization", "Location"],
"properties": ["name", "date", "confidence"],
"relationships": ["works_for", "located_in", "born_in"],
"relationships": ["works_for", "located_in", "founded_by"],
"rules": {
"Person": ["must_have_name", "can_have_birth_date"],
"Person": ["must_have_name"],
"Organization": ["must_have_name", "can_have_founding_date"]
}
}
```
Semantica can auto-generate ontologies from your knowledge graph, or import existing OWL/RDF/Turtle ontologies.
---
## Reasoning & Inference
Making logical deductions from your knowledge.
Semantica includes multiple reasoning engines to derive new knowledge from existing facts.
### What it can do:
- **Infer missing facts** - derive new knowledge
- **Detect inconsistencies** - find contradictions
- **Apply rules** - automate decision making
- **Explain reasoning** - show how conclusions were reached
```
Known: Steve Jobs founded Apple Inc.
Known: Apple Inc. is headquartered in Cupertino
Inferred: Steve Jobs has a connection to Cupertino
```
### Example:
```
Known: Steve Jobs founded Apple Inc.
Known: Apple Inc. is headquartered in Cupertino
Inferred: Steve Jobs has connection to Cupertino
```
Supported engines: forward chaining, Rete network, deductive, abductive, and SPARQL reasoning — all producing **explainable inference paths**, not black-box conclusions.
---
## Temporal Graphs
Knowledge changes over time. Temporal graphs attach `valid_from` / `valid_until` windows to nodes and edges, enabling point-in-time queries and historical analysis.
Common uses: tracking company leadership changes, policy evolution, research timelines, financial instrument histories.
---
## Deduplication & Entity Resolution
Finding and merging duplicate entities.
Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance.
### What it does:
- **Detects duplicates** - same entity, different names
- **Merges information** - combine attributes
- **Resolves conflicts** - handle contradictory data
- **Maintains provenance** - track original sources
### Example:
```python
# These refer to the same entity:
"Apple Inc." → "Apple" → "Apple Computer Inc."
# Merge into single entity with all attributes
```
Strategies: Jaro-Winkler similarity (v1), `blocking_v2`, `hybrid_v2`, `semantic_v2` (v2 — up to 7x faster).
---
## Data Normalization
## Provenance & Auditability
Cleaning and standardizing your data.
Every fact in Semantica links back to:
### What it fixes:
- **Format inconsistencies** - dates, names, numbers
- **Canonical forms** - standard representations
- **Data quality** - remove errors and noise
- **Standardization** - consistent naming conventions
- The source document it came from
- The extraction method used
- The ontology rules applied
- The reasoning steps that produced any inference
### Examples:
- **Dates**: "Jan 1, 2020" → "2020-01-01"
- **Names**: "Dr. Smith PhD" → "John Smith"
- **Companies**: "Apple" → "Apple Inc."
- **Locations**: "NYC" → "New York City"
This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails.
---
## Conflict Detection
Finding and resolving contradictory information.
When multiple sources disagree on the same fact, Semantica flags and resolves the conflict rather than silently picking one value.
### What it identifies:
- **Factual conflicts** - different values for same fact
- **Temporal conflicts** - impossible timelines
- **Logical conflicts** - contradictory relationships
- **Source reliability** - trustworthiness assessment
### Resolution Strategies:
- **Most recent** - prefer newer information
- **Most reliable** - prefer trusted sources
- **Majority vote** - go with consensus
- **Manual review** - flag for human review
Resolution strategies: prefer most recent, prefer most reliable source, majority vote, or flag for manual review.
---
## Getting Started
## Next Steps
Ready to build your first knowledge graph?
### Quick Start (5 minutes)
```python
from semantica.semantic_extract import NERExtractor
from semantica.kg import GraphBuilder
# Extract entities
ner = NERExtractor()
entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
# Build graph
kg = GraphBuilder().build({"entities": entities, "relationships": []})
```
### Learn More
- **Getting Started Guide** - [Getting Started](getting-started.md)
- **Cookbook Examples** - [Cookbook](cookbook.md)
- **Module Documentation** - [Reference](reference/)
- **Community Support** - [Community](community.md)
### Common Use Cases
- **Document Analysis** - extract knowledge from reports
- **Research Assistant** - find connections in academic papers
- **Business Intelligence** - analyze company relationships
- **Regulatory Compliance** - track policy changes
---
## Best Practices
### Start Small
- Begin with a single document type
- Focus on specific entity types
- Validate results before scaling
### Configure Properly
- Choose appropriate models for your domain
- Set confidence thresholds
- Define clear ontology rules
### Validate Data
- Check extraction quality
- Review relationship accuracy
- Test with known examples
### Handle Errors
- Implement error handling
- Log processing issues
- Provide feedback mechanisms
### Optimize Performance
- Use appropriate storage backends
- Cache frequently accessed data
- Monitor resource usage
### Document Workflows
- Record processing steps
- Track data sources
- Maintain change logs
---
## Need Help?
- **Documentation**: [Getting Started](getting-started.md)
- **Examples**: [Cookbook](cookbook.md)
- **Community**: [Discord](community.md)
- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
- **Support**: [Contact Us](community.md)
- [Quickstart Tutorial](quickstart.md) — build a full pipeline with code
- [Modules Guide](modules.md) — every module explained
- [Use Cases](use-cases.md) — real-world domain examples
- [API Reference](reference/core.md) — complete technical reference
docs/contributing.md
+54 -85
View File
@@ -1,130 +1,99 @@
# Contributing
# Contributing
**Help us build Semantica! Every contribution makes the project better.**
Contributions of all kinds are welcome — code, documentation, tests, and community support.
---
## Getting Started
## Quick Start
### Quick Start
1. **Fork** the repository
2. **Create** a feature branch
3. **Make** your changes
4. **Test** your changes
5. **Submit** a pull request
```bash
# Fork the repo, then:
git clone https://github.com/your-username/semantica.git
cd semantica
pip install -e ".[dev]"
pytest
```
### First Contribution?
Look for issues labeled [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) for beginner-friendly tasks.
First time? Look for [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) labels for beginner-friendly tasks.
---
## Ways to Contribute
### Code
- **Fix bugs** - Resolve reported issues
- **Add features** - Implement new functionality
- **Improve performance** - Optimize existing code
- **Refactor** - Clean up code structure
**Code**
- Fix bugs and resolve open issues
- Implement new features or integrations
- Optimize performance or refactor existing code
### Documentation
- **Fix typos** - Correct spelling and grammar
- **Improve guides** - Make documentation clearer
- **Add examples** - Provide practical code examples
- **Update API docs** - Keep reference current
**Documentation**
- Fix typos, improve clarity, add examples
- Write tutorials or domain-specific cookbook notebooks
- Keep API reference up to date
### Testing
- **Write tests** - Add test coverage
- **Fix tests** - Resolve test failures
- **Report issues** - Identify bugs through testing
**Testing**
- Add test coverage for untested modules
- Reproduce and confirm reported bugs
- Improve test reliability
### Community
- **Help others** - Answer questions in issues
- **Share knowledge** - Write tutorials and guides
- **Provide feedback** - Review pull requests
**Community**
- Answer questions in issues and discussions
- Review pull requests
- Share Semantica in your blog posts or talks
---
## Reporting Issues
### Bug Reports
When reporting bugs, include:
- **Description** - What happened
- **Steps to reproduce** - How to trigger the issue
- **Expected behavior** - What should happen
- **Environment** - Your setup details
Include: what happened, steps to reproduce, expected behavior, and your environment (Python version, OS, Semantica version).
### Feature Requests
When suggesting features, include:
- **Use case** - Why you need this feature
- **Proposed solution** - How it should work
- **Benefits** - How it helps the community
Include: your use case, what you'd like Semantica to do, and how it benefits others.
---
## Pull Request Guidelines
### Before Submitting
- **Test** your changes thoroughly
- **Document** new features with examples
- **Update** relevant documentation
- **Follow** the existing code style
Before submitting:
### Pull Request Checklist
- [ ] Code follows project style
- [ ] Tests pass locally
- [ ] Documentation is updated
- [ ] Commit messages are clear
- [ ] No merge conflicts
- [ ] Tests pass locally (`pytest`)
- [ ] New features are documented with examples
- [ ] Code follows project style (Black, isort, flake8)
- [ ] Commit messages are clear and descriptive
- [ ] No unresolved merge conflicts
---
## Development Setup
### Local Development
```bash
# Clone your fork
git clone https://github.com/your-username/semantica.git
cd semantica
pip install -e ".[dev]"
```
# Install in development mode
pip install -e .[dev]
Code style tools used: **Black** (formatting), **isort** (imports), **flake8** (linting).
# Run tests
Run the full test suite:
```bash
pytest
```
### Code Style
We use standard Python formatting:
- **Black** for code formatting
- **isort** for import sorting
- **flake8** for linting
---
## Community
Please follow the [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md). Be respectful, patient, and constructive.
All contributors are recognized in release notes and the GitHub contributors list.
---
## Community Guidelines
## Help
### Code of Conduct
Please follow our [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md).
### Communication
- **Be respectful** - Treat everyone with kindness
- **Be helpful** - Assist others when you can
- **Be patient** - Allow time for reviews
- **Be constructive** - Provide helpful feedback
---
## Recognition
All contributors are recognized in:
- **GitHub contributors** - Automatic recognition
- **Release notes** - Notable contributions
- **Community highlights** - Outstanding work
---
## Need Help?
- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Ask questions
- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Community chat
- **[Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md)** - Community standards
- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
- [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
- [Discord](https://discord.gg/sV34vps5hH)
docs/cookbook.md
+35 -85
View File
@@ -1,108 +1,58 @@
# 🍳 Semantica Cookbook
# Semantica Cookbook
Welcome to the **Semantica Cookbook**!
Interactive Jupyter notebooks covering everything from your first knowledge graph to production GraphRAG systems.
This collection of Jupyter notebooks is designed to take you from a beginner to an expert in building semantic AI applications. Whether you're looking for quick recipes or deep-dive tutorials, you'll find it here.
!!! tip "How to use this Cookbook"
- **Beginners**: Start with the [Core Tutorials](#core-tutorials) to learn the basics.
- **Developers**: Check out [Advanced Concepts](#advanced-concepts) for deep dives into specific features.
- **Architects**: Explore [Industry Use Cases](#industry-use-cases) for end-to-end solutions.
!!! tip "Where to start"
- **New to Semantica** — begin with [Core Tutorials](#core-tutorials)
- **Building an application** — see [Advanced Concepts](#advanced-concepts) or [Industry Use Cases](#industry-use-cases)
- **Need installation help** — see the [Installation Guide](installation.md)
!!! note "Prerequisites"
Before running these notebooks, ensure you have:
- Python 3.8+ installed
- A basic understanding of Python and Jupyter
- An OpenAI API key (for most examples)
!!! success "Installation"
Install Semantica from PyPI (recommended):
```bash
pip install semantica
# Or with all optional dependencies:
pip install semantica[all]
```
For more installation options, see the [Installation Guide](installation.md).
Python 3.8+, Jupyter, and an OpenAI API key (for most examples).
---
## � Featured Recipes
Hand-picked tutorials to show you the power of Semantica.
## Featured Recipes
<div class="grid cards" markdown>
- :material-robot: **GraphRAG Complete**
---
Build a production-ready Graph Retrieval Augmented Generation system.
**Topics**: RAG, LLMs, Vector Search, Graph Traversal
**Difficulty**: Advanced
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
- :material-scale-balance: **RAG vs. GraphRAG Comparison**
---
Side-by-side comparison of Standard RAG vs. GraphRAG using real-world data.
**Topics**: RAG, GraphRAG, Benchmarking, Visualization
**Difficulty**: Intermediate
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
- :material-robot: **GraphRAG Complete**
---
Build a production-ready Graph Retrieval Augmented Generation system.
**New Features**: Graph Validation, Logical Inference, Hybrid Context.
**Topics**: RAG, LLMs, Vector Search, Graph Traversal
**Difficulty**: Advanced
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
- :material-scale-balance: **RAG vs. GraphRAG Comparison**
---
Side-by-side comparison of Standard RAG vs. GraphRAG using real-world data.
**New Features**: Inference-Enhanced GraphRAG, Reasoning Gap Analysis.
**Topics**: RAG, GraphRAG, Benchmarking, Visualization
**Difficulty**: Intermediate
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
- :material-graph: **Your First Knowledge Graph**
---
Go from raw text to a queryable knowledge graph in 20 minutes.
**Topics**: Extraction, Graph Construction, Visualization
**Difficulty**: Beginner
**Topics**: Extraction, Graph Construction, Visualization · **Difficulty**: Beginner
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)
- :material-robot: **GraphRAG Complete**
---
Build a production-ready Graph Retrieval Augmented Generation system with hybrid retrieval and logical inference.
**Topics**: RAG, LLMs, Vector Search, Graph Traversal · **Difficulty**: Advanced
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
- :material-scale-balance: **RAG vs. GraphRAG Comparison**
---
Side-by-side benchmark of standard RAG vs. GraphRAG on real-world data.
**Topics**: RAG, GraphRAG, Benchmarking, Reasoning Gap · **Difficulty**: Intermediate
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
- :material-shield-alert: **Real-Time Anomaly Detection**
---
Detect anomalies in streaming data using dynamic graphs.
**Topics**: Streaming, Security, Dynamic Graphs
**Difficulty**: Advanced
Detect anomalies in streaming data using dynamic knowledge graphs.
**Topics**: Streaming, Security, Dynamic Graphs · **Difficulty**: Advanced
[Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)
</div>
---
## 🏁 Core Tutorials {#core-tutorials}
## Core Tutorials {#core-tutorials}
Essential guides to master the Semantica framework.
@@ -210,7 +160,7 @@ Essential guides to master the Semantica framework.
---
## 🧠 Advanced Concepts
## Advanced Concepts
Deep dive into advanced features, customization, and complex workflows.
@@ -329,7 +279,7 @@ Deep dive into advanced features, customization, and complex workflows.
---
## 🏭 Industry Use Cases {#industry-use-cases}
## Industry Use Cases {#industry-use-cases}
Real-world examples and end-to-end applications across various industries.
@@ -497,7 +447,7 @@ Real-world examples and end-to-end applications across various industries.
---
## 🛠️ How to Run
## How to Run
To run these notebooks locally:
docs/css/custom.css
+66 -50
View File
@@ -216,76 +216,92 @@ html {
border-left: 2px solid var(--md-accent-fg-color);
}
/*
/*
==========================================================================
Layout Optimization
==========================================================================
==========================================================================
*/
/* Reduce spacing between sidebars and content for all pages */
.md-content__inner {
padding-left: 0.75rem;
padding-right: 0.75rem;
margin-left: 0;
}
.md-content {
margin-left: 0;
/* Widen the overall grid */
.md-grid {
max-width: 1440px;
margin-left: auto;
margin-right: auto;
padding-left: 0.5rem;
padding-right: 0.5rem;
}
/* Reduce spacing before left sidebar for all pages */
.md-sidebar {
padding-left: 0.25rem;
margin-left: 0;
}
/* Narrow left sidebar to give content more room */
.md-sidebar--primary {
padding-right: 0.5rem;
width: 11rem;
padding-right: 0.25rem;
padding-left: 0.25rem;
}
/* Right TOC sidebar */
.md-sidebar--secondary {
width: 11rem;
padding-left: 0.5rem;
padding-right: 0;
margin-left: 0;
}
/* Reduce spacing after right sidebar (table of contents) and shift it right slightly */
.md-sidebar--secondary {
padding-left: 1.25rem;
padding-right: 0;
margin-right: 0;
margin-left: 3.5rem;
.md-sidebar--secondary .md-nav {
width: 11rem;
}
/* Reduce right edge spacing - similar to left */
.md-container {
padding-right: 0;
/* Tighten TOC list spacing */
.md-sidebar--secondary .md-nav__list {
padding-bottom: 1.5rem;
margin: 0;
}
.md-sidebar--secondary .md-nav__item {
padding: 0;
margin: 0;
}
.md-sidebar--secondary .md-nav__link {
white-space: normal;
word-break: break-word;
overflow: visible;
text-overflow: unset;
padding-top: 0.15rem;
padding-bottom: 0.15rem;
line-height: 1.4;
font-size: 0.7rem;
margin: 0;
}
/* Nested TOC items (h3, h4) */
.md-sidebar--secondary .md-nav__item .md-nav__item .md-nav__link {
padding-left: 0.6rem;
font-size: 0.68rem;
}
/* Remove extra gap between TOC title and first item */
.md-sidebar--secondary .md-nav__title {
margin-bottom: 0.25rem;
padding-bottom: 0.25rem;
}
/* Give the main content area maximum available width */
.md-content {
max-width: none;
padding-left: 1rem;
padding-right: 1rem;
}
.md-content__inner {
max-width: none;
padding-left: 1rem;
padding-right: 1rem;
margin-left: 0;
margin-right: 0;
}
.md-main {
margin-right: 0;
padding-right: 0;
}
/* Reduce margins of the main container */
.md-main__inner {
margin-left: 0;
margin-right: 0;
padding-right: 0;
}
/* Reduce right edge spacing on body/html */
body {
margin-right: 0;
padding-right: 0;
}
html {
margin-right: 0;
padding-right: 0;
}
.md-grid {
margin-left: 0;
padding-left: 0.5rem;
}
/* Ensure text content is left-aligned by default */
docs/deep-dive.md
+118 -166
View File
@@ -1,10 +1,15 @@
# Deep Dive
Advanced topics, architecture, and internals of Semantica.
Internals, advanced concepts, and extension points for contributors and power users.
## Architecture Overview
!!! tip "Just getting started?"
Read [Architecture](architecture.md) for a higher-level overview first.
Semantica follows a modular, extensible architecture:
---
## Pipeline Internals
The full data flow through a Semantica pipeline:
```mermaid
graph TB
@@ -16,65 +21,79 @@ graph TB
F --> G[Knowledge Graph Builder]
G --> H[Embedding Generator]
H --> I[Export Layer]
D --> D1[Entity Extractor]
D --> D2[Relationship Extractor]
D --> D3[Triplet Extractor]
G --> G1[Graph Validator]
G --> G2[Graph Analyzer]
H --> H1[Text Embeddings]
H --> H2[Graph Embeddings]
```
### Sequence Diagram
```mermaid
sequenceDiagram
participant User
participant Semantica
participant Ingestor
participant Parser
participant Extractor
participant Resolver
participant GraphBuilder
participant Exporter
User->>Semantica: build_knowledge_base(sources)
Semantica->>Ingestor: ingest(sources)
Ingestor->>Parser: parse(documents)
Parser->>Extractor: extract(text)
Extractor->>Resolver: resolve_conflicts(entities)
Resolver->>GraphBuilder: build_graph(resolved_data)
GraphBuilder->>Exporter: export(graph)
Exporter->>User: return result
```
---
## System Components
### 1. Ingestion Layer
### Ingestion Layer
Handles data input from various sources:
Handles input from any source:
- **File Ingestor**: PDF, DOCX, HTML, JSON, CSV
- **Web Ingestor**: URLs, web scraping
- **Database Ingestor**: SQL databases
- **Stream Ingestor**: Real-time data streams
- **FileIngestor** — PDF, DOCX, HTML, JSON, CSV, archives
- **WebIngestor** — URL crawling and scraping
- **DBIngestor** / **SnowflakeIngestor** — SQL databases
- **StreamIngestor** — Kafka and real-time feeds
### 2. Parsing Layer
### Parsing Layer
Converts raw data into structured format:
Converts raw data to structured text:
- Document parsing (PDF, Word, etc.)
- Text extraction
- Metadata extraction
- Format normalization
- Text and metadata extraction from documents
- OCR for scanned content
- Layout analysis (via Docling for tables and columns)
### 3. Extraction Layer
### Extraction Layer
Core semantic extraction:
Core semantic processing pipeline:
```python
# Entity extraction pipeline
```
text → Tokenization → NER → Entity Linking → Entity Validation
```
**Components:**
- Named Entity Recognition (NER)
- Relationship Extraction
- Triplet Extraction
- Coreference Resolution
Components: Named Entity Recognition, Relationship Extraction, Triplet Extraction, Coreference Resolution.
### 4. Normalization Layer
### Normalization Layer
Standardizes extracted data:
Standardizes extracted data: entity names, date formats, numbers, encodings, and language normalization.
- Entity normalization
- Date/time normalization
- Number normalization
- Text cleaning
### Conflict Resolution
### 5. Conflict Resolution
Handles conflicting information:
Handles contradictory facts from multiple sources:
```mermaid
graph LR
@@ -88,64 +107,28 @@ graph LR
E --> H
F --> H
G --> H
style A fill:#ffebee
style H fill:#c8e6c9
style C fill:#fff9c4
```
### 6. Knowledge Graph Builder
### Knowledge Graph Builder
Constructs the knowledge graph:
- Entity resolution across sources
- Edge creation (typed relationships)
- Property assignment with confidence scores
- Graph validation and quality checks
- Node creation (entities)
- Edge creation (relationships)
- Property assignment
- Graph validation
- Quality checks
### Embedding Generator
### 7. Embedding Generator
- Text embeddings (Sentence-Transformers, FastEmbed, OpenAI, BGE)
- Graph embeddings (Node2Vec, GraphSAGE)
Generates vector representations:
- Text embeddings (sentence transformers)
- Graph embeddings (node2vec, GraphSAGE)
- Multimodal embeddings
## Data Flow
```mermaid
sequenceDiagram
participant User
participant Semantica
participant Ingestor
participant Parser
participant Extractor
participant Resolver
participant GraphBuilder
participant Exporter
User->>Semantica: build_knowledge_base(sources)
Semantica->>Ingestor: ingest(sources)
Ingestor->>Parser: parse(documents)
Parser->>Extractor: extract(text)
Extractor->>Resolver: resolve_conflicts(entities)
Resolver->>GraphBuilder: build_graph(resolved_data)
GraphBuilder->>Exporter: export(graph)
Exporter->>User: return result
Note over User,Exporter: Complete pipeline execution
```
---
## Advanced Concepts
### Entity Resolution
Matching entities across sources:
### Entity Resolution Algorithm
```python
# Entity resolution algorithm
def resolve_entities(entities):
def resolve_entities(entities, threshold=0.85):
clusters = []
for entity in entities:
matched = False
@@ -161,123 +144,92 @@ def resolve_entities(entities):
### Relationship Inference
Inferring implicit relationships:
Semantica's reasoning engines can derive implicit relationships:
- Transitive relationships
- Temporal relationships
- Causal relationships
- Hierarchical relationships
- **Transitive** — if A→B and B→C, infer A→C
- **Temporal** — before, after, during from timestamped facts
- **Causal** — IF/THEN rules via `Reasoner`
- **Hierarchical** — subclass/instance inference via `OntologyReasoner`
### Graph Optimization
Optimizing knowledge graph structure:
- Node deduplication
- Edge consolidation
- Path compression
- Index optimization
## Performance Considerations
### Scalability
- **Horizontal Scaling**: Process multiple documents in parallel
- **Vertical Scaling**: Use GPU acceleration
- **Caching**: Cache embeddings and parsed documents
- **Lazy Loading**: Load components on demand
### Memory Management
### Batch Processing for Large Datasets
```python
# Process large datasets efficiently
def process_large_dataset(sources, batch_size=100):
for i in range(0, len(sources), batch_size):
batch = sources[i:i+batch_size]
batch = sources[i : i + batch_size]
result = semantica.build_knowledge_base(batch)
# Save and clear memory
save_result(result)
del result
gc.collect()
```
---
## Extension Points
### Custom Plugins
Create custom plugins:
### Custom Plugin
```python
from semantica.core import Plugin
class CustomPlugin(Plugin):
def process(self, data):
# Your custom processing
# Your custom processing logic
return processed_data
```
### Custom Extractors
Implement custom extractors:
### Custom Extractor
```python
from semantica.semantic_extract import BaseExtractor
class DomainSpecificExtractor(BaseExtractor):
def extract_entities(self, text):
# Domain-specific extraction logic
def extract(self, text):
# Domain-specific entity extraction logic
return entities
```
## Internal APIs
### Custom Ingestor
### Core APIs
```python
from semantica.ingest import BaseIngestor
- `Semantica.build_knowledge_base()` - Main entry point
- `KGBuilder.build()` - Graph construction
- `ConflictResolver.resolve()` - Conflict resolution
- `EmbeddingGenerator.generate()` - Embedding generation
### Extension APIs
- Plugin registration
- Custom extractor registration
- Custom exporter registration
- Event hooks
## Design Decisions
### Why Modular Architecture?
- **Extensibility**: Easy to add new features
- **Testability**: Components can be tested independently
- **Maintainability**: Clear separation of concerns
- **Flexibility**: Swap implementations easily
### Why Conflict Resolution?
- **Data Quality**: Ensures consistent knowledge
- **Multi-Source**: Handles conflicting information
- **Flexibility**: Multiple resolution strategies
- **Transparency**: Track resolution decisions
## Future Enhancements
Planned improvements:
- Distributed processing
- Real-time streaming
- Advanced reasoning
- Multi-modal support expansion
- Enhanced visualization
## Contributing to Core
Interested in contributing to Semantica's core? See our [Contributing Guide](https://github.com/Hawksight-AI/semantica/blob/main/CONTRIBUTING.md).
class CustomIngestor(BaseIngestor):
def ingest(self, source):
# Load and return document dicts
return documents
```
---
For more information:
- **[API Reference](reference/core.md) - Detailed API documentation
- **[Learning More](learning-more.md)** - Additional resources
- **[GitHub Repository](https://github.com/Hawksight-AI/semantica)** - Source code
## Internal APIs
| API | Purpose |
|-----|---------|
| `Semantica.build_knowledge_base()` | Main orchestration entry point |
| `GraphBuilder.build()` | Graph construction |
| `ConflictResolver.resolve()` | Conflict resolution |
| `EmbeddingGenerator.generate()` | Embedding generation |
Extension hooks: plugin registration, custom extractor registration, custom exporter registration, event hooks.
---
## Design Decisions
**Why modular architecture?** Each component is independently testable and swappable. You can use `NERExtractor` alone without pulling in graph storage or pipelines.
**Why built-in conflict resolution?** Multi-source data always has contradictions. Ignoring them produces garbage graphs. Explicit resolution strategies give you control over data quality.
**Why W3C PROV-O for provenance?** It's an industry standard with tooling support. Using a custom format would make lineage data non-portable.
**Why multiple reasoning engines?** Different problems need different reasoning: forward chaining for rule application, SPARQL for graph queries, abductive for hypothesis generation. No single engine fits all cases.
---
## Further Reading
- [Architecture](architecture.md) — high-level three-layer overview
- [Modules](modules.md) — every module with code examples
- [API Reference](reference/core.md) — complete technical reference
- [Contributing](contributing.md) — how to extend the framework
docs/examples.md
+152 -398
View File
@@ -1,306 +1,134 @@
# Examples
Real-world examples and use cases for Semantica.
!!! tip "Interactive Learning"
For hands-on interactive tutorials, check out our [Cookbook](cookbook.md) with Jupyter notebooks covering everything from basics to advanced use cases.
Code examples organized by complexity. For interactive notebooks, see the [Cookbook](cookbook.md).
---
## Example Gallery
## Beginner
<div class="grid cards" markdown>
### Basic Knowledge Graph
- :material-school: **Getting Started**
---
Quick examples to get you up and running in 5 minutes.
[View Examples](#getting-started-5-min-examples)
```python
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
- :material-cogs: **Core Workflows**
---
Common workflows for building production-ready graphs.
[View Examples](#core-workflows-15-min-examples)
- :material-rocket: **Advanced Patterns**
---
Complex use cases and production deployments.
[View Examples](#advanced-patterns-30-min-examples)
- :material-factory: **Production Patterns**
---
Scalable deployment patterns for enterprise use.
[View Examples](#production-patterns)
</div>
---
## Getting Started (5 min examples)
### Example 1: Basic Knowledge Graph
**Difficulty**: Beginner
Build a knowledge graph from a single document using Semantica's modular approach. This example demonstrates the complete workflow from document ingestion to graph construction.
**What it demonstrates:**
- Document ingestion and parsing
- Entity and relationship extraction
- Knowledge graph construction
**For complete step-by-step examples, see:**
- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Complete walkthrough
- **Topics**: Ingestion, parsing, extraction, graph building
- **Difficulty**: Beginner
- **Time**: 20-30 minutes
- **Use Cases**: Learning the complete workflow
### Example 2: Entity Extraction
**Difficulty**: Beginner
Extract entities from text using Named Entity Recognition. This example shows how to identify and classify named entities in text.
**What it demonstrates:**
- Named Entity Recognition (NER)
- Entity type classification
- Confidence scoring
**For complete examples, see:**
- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn entity extraction
- **Topics**: NER methods, entity types, extraction techniques
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Understanding entity extraction
print(f"{entity.text}: {entity.label}")
```
**Expected Output:**
```
Apple Inc.: ORGANIZATION
Steve Jobs: PERSON
```
### Example 3: Multi-Source Integration
**Difficulty**: Beginner
Combine data from multiple sources into a unified knowledge graph. This example demonstrates integrating data from diverse sources.
**What it demonstrates:**
- Multi-source data ingestion
- Entity merging and resolution
- Unified graph construction
**For complete examples, see:**
- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration patterns
- **Topics**: Multi-source integration, entity resolution, conflict handling
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Building unified knowledge graphs from diverse sources
---
## Core Workflows (15 min examples)
### Example 4: Conflict Resolution
**Difficulty**: Intermediate
Resolve conflicts in data from multiple sources. This example shows how to identify and resolve conflicting information.
**What it demonstrates:**
- Conflict detection
- Conflict resolution strategies
- Data quality assurance
**For complete examples, see:**
- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Conflict resolution patterns
- **Topics**: Conflict detection, resolution strategies, data quality
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Data integration, quality assurance
ingestor = FileIngestor()
parser = DocumentParser()
ner = NERExtractor()
parser = DocumentParser()
ner = NERExtractor()
rel = RelationExtractor()
all_entities = []
for source in ["source1.pdf", "source2.pdf"]:
doc = ingestor.ingest_file(source)
parsed = parser.parse_document(source)
text = parsed.get("full_text", "")
entities = ner.extract_entities(text)
all_entities.extend(entities)
sources = ingestor.ingest("data/sample.pdf")
parsed = parser.parse(sources[0])
entities = ner.extract(parsed)
relationships = rel.extract(parsed, entities=entities)
kg = GraphBuilder(merge_entities=True).build(
entities=entities, relationships=relationships
)
print(f"{len(kg.nodes)} nodes, {len(kg.edges)} edges")
```
### Entity Extraction from Text
```python
from semantica.semantic_extract import NERExtractor
ner = NERExtractor()
entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
for entity in entities:
print(f"{entity['text']}: {entity['type']}")
# Apple Inc.: ORGANIZATION
# Steve Jobs: PERSON
# 1976: DATE
```
### Custom NER Configuration
```python
from semantica.semantic_extract import NERExtractor
ner = NERExtractor(
method="llm",
provider="openai",
model="gpt-4",
confidence_threshold=0.8,
temperature=0.0,
)
entities = ner.extract("Your document text here...")
```
---
## Intermediate
### Multi-Source Integration
```python
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
ingestor = FileIngestor()
parser = DocumentParser()
ner = NERExtractor()
rel = RelationExtractor()
builder = GraphBuilder(merge_entities=True)
all_entities, all_rels = [], []
for path in ["source1.pdf", "source2.pdf", "source3.pdf"]:
sources = ingestor.ingest(path)
parsed = parser.parse(sources[0])
all_entities.extend(ner.extract(parsed))
all_rels.extend(rel.extract(parsed, entities=all_entities))
kg = builder.build(entities=all_entities, relationships=all_rels)
print(f"Unified graph: {len(kg.nodes)} nodes, {len(kg.edges)} edges")
```
### Conflict Detection and Resolution
```python
from semantica.conflicts import ConflictDetector, ConflictResolver
# Detect and resolve conflicts
detector = ConflictDetector()
conflicts = detector.detect_conflicts(all_entities)
resolver = ConflictResolver(default_strategy="voting")
resolved = resolver.resolve_conflicts(conflicts)
print(f"Detected {len(conflicts)} conflicts")
print(f"Resolved {len(resolved)} conflicts")
print(f"Detected {len(conflicts)} conflicts, resolved {len(resolved)}")
```
### Example 5: Custom Entity Extraction Configuration
**Difficulty**: Intermediate
Use custom configuration for entity extraction with specific models and thresholds.
```python
from semantica.semantic_extract import NERExtractor
from semantica.kg import GraphBuilder
# Use LLM-based extraction with custom configuration
ner = NERExtractor(
method="llm",
provider="openai",
model="gpt-4",
confidence_threshold=0.8,
temperature=0.0
)
text = "Your document text here..."
entities = ner.extract_entities(text)
# Build graph with custom merge settings
builder = GraphBuilder(
merge_entities=True,
merge_threshold=0.9
)
kg = builder.build_graph(entities=entities, relationships=[])
```
### Example 6: Incremental Graph Building
**Difficulty**: Intermediate
Build knowledge graph incrementally from multiple sources.
```python
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder, GraphMerger
def build_kg_from_source(source_path):
"""Helper function to build a knowledge graph from a single source."""
ingestor = FileIngestor()
parser = DocumentParser()
ner = NERExtractor()
rel_extractor = RelationExtractor()
doc = ingestor.ingest_file(source_path)
parsed = parser.parse_document(source_path)
text = parsed.get("full_text", "")
entities = ner.extract_entities(text)
relationships = rel_extractor.extract_relations(text, entities=entities)
builder = GraphBuilder()
return builder.build_graph(entities=entities, relationships=relationships)
# Build graphs separately
kg1 = build_kg_from_source("source1.pdf")
kg2 = build_kg_from_source("source2.pdf")
# Merge into unified graph
merger = GraphMerger()
merged_kg = merger.merge([kg1, kg2])
print(f"Merged graph: {len(merged_kg.nodes)} nodes, {len(merged_kg.edges)} edges")
```
---
## Advanced Patterns (30+ min examples)
### Example 7: Graph Visualization
**Difficulty**: Beginner
Visualize your knowledge graph to understand entity relationships.
```python
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
from semantica.visualization import KGVisualizer
# Build a small graph
ingestor = FileIngestor()
parser = DocumentParser()
ner = NERExtractor()
rel_extractor = RelationExtractor()
doc = ingestor.ingest_file("semantica_intro.pdf")
parsed = parser.parse_document("semantica_intro.pdf")
text = parsed.get("full_text", "")
entities = ner.extract_entities(text)
relationships = rel_extractor.extract_relations(text, entities=entities)
builder = GraphBuilder()
kg = builder.build_graph(entities=entities, relationships=relationships)
# Visualize
viz = KGVisualizer()
viz.visualize_network(kg, output="html", file_path="semantica_knowledge_map.html")
print("Visualization saved to semantica_knowledge_map.html")
```
---
## Advanced Patterns (30+ min examples)
### Example 8: Persistent Storage (Neo4j)
**Difficulty**: Intermediate
Store and query knowledge graphs in a persistent graph database.
### Persistent Storage (Neo4j)
```python
from semantica.graph_store import GraphStore
# Initialize with Neo4j
store = GraphStore(
backend="neo4j",
uri="bolt://localhost:7687",
user="neo4j",
password="password"
password="password",
)
store.connect()
# Create nodes and relationships
apple = store.create_node(
labels=["Company"],
properties={"name": "Apple Inc."}
)
tim = store.create_node(
labels=["Person"],
properties={"name": "Tim Cook"}
)
apple = store.create_node(labels=["Company"], properties={"name": "Apple Inc."})
tim = store.create_node(labels=["Person"], properties={"name": "Tim Cook"})
store.create_relationship(
start_node_id=tim["id"],
end_node_id=apple["id"],
rel_type="CEO_OF"
rel_type="CEO_OF",
)
store.close()
```
### Example 9: FalkorDB for Real-Time Applications
**Difficulty**: Intermediate
Ultra-fast graph queries for LLM applications using FalkorDB.
### FalkorDB (High-Speed Queries)
```python
from semantica.graph_store import GraphStore
@@ -309,105 +137,49 @@ store = GraphStore(
backend="falkordb",
host="localhost",
port=6379,
graph_name="knowledge_graph"
graph_name="knowledge_graph",
)
store.connect()
# Fast queries
results = store.execute_query("MATCH (n)-[r]->(m) WHERE n.name CONTAINS 'AI' RETURN n")
results = store.execute_query(
"MATCH (n)-[r]->(m) WHERE n.name CONTAINS 'AI' RETURN n"
)
store.close()
```
### Example 10: GraphRAG (Knowledge-Powered Retrieval)
---
**Difficulty**: Advanced
## Advanced
Build a production-ready GraphRAG system with logical inference and hybrid retrieval.
### GraphRAG with Reasoning
```python
from semantica.context import AgentContext
from semantica.reasoning import Reasoner
# 1. Initialize context with GraphRAG (Hybrid Retrieval)
context = AgentContext(
vector_store=vs,
vector_store=vs,
knowledge_graph=kg,
graph_expansion=True,
hybrid_alpha=0.7
hybrid_alpha=0.7,
)
# 2. Enrich Knowledge Graph using Logical Reasoning
reasoner = Reasoner()
# Add a rule to categorize technology stack items
reasoner.add_rule("IF Library(?x) AND Language(?y) THEN TechStackItem(?x)")
inferred = reasoner.infer_facts(kg.get_all_triplets())
# Infer new facts from the existing graph
all_facts = kg.get_all_triplets()
inferred = reasoner.infer_facts(all_facts)
for fact in inferred:
kg.add_fact_from_string(fact)
# Add inferred knowledge back to the graph
for fact_str in inferred:
kg.add_fact_from_string(fact_str)
# 3. Retrieve context for a query (now with enriched knowledge)
results = context.retrieve("What technologies are used in this project?")
```
[**View Complete GraphRAG Tutorial**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
### Example 11: RAG vs. GraphRAG Comparison
**Difficulty**: Intermediate
Benchmark standard Vector RAG against Graph-enhanced retrieval.
[**View RAG vs. GraphRAG Comparison**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
[Full GraphRAG tutorial](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) · [RAG vs. GraphRAG comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
---
## Production Patterns
## Production
### Example 12: Streaming Data Processing
**Difficulty**: Advanced
Process data streams in real-time.
```python
from semantica.ingest import StreamIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
stream_ingestor = StreamIngestor(stream_uri="kafka://localhost:9092/topic")
parser = DocumentParser()
ner = NERExtractor()
rel_extractor = RelationExtractor()
builder = GraphBuilder()
for batch in stream_ingestor.stream(batch_size=100):
all_entities = []
all_relationships = []
for item in batch:
text = str(item) # Convert stream item to text
entities = ner.extract_entities(text)
relationships = rel_extractor.extract_relations(text, entities=entities)
all_entities.extend(entities)
all_relationships.extend(relationships)
# Build graph from batch
kg = builder.build_graph(entities=all_entities, relationships=all_relationships)
# Process results
print(f"Processed batch: {len(kg.nodes)} nodes")
```
### Example 13: Batch Processing Large Datasets
**Difficulty**: Intermediate
Process large datasets efficiently with batching.
### Batch Processing (Large Datasets)
```python
from semantica.ingest import FileIngestor
@@ -416,75 +188,57 @@ from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
ingestor = FileIngestor()
parser = DocumentParser()
ner = NERExtractor()
rel_extractor = RelationExtractor()
builder = GraphBuilder()
parser = DocumentParser()
ner = NERExtractor()
rel = RelationExtractor()
builder = GraphBuilder()
sources = [f"data/doc_{i}.pdf" for i in range(1000)]
sources = [f"data/doc_{i}.pdf" for i in range(1000)]
batch_size = 50
for i in range(0, len(sources), batch_size):
batch = sources[i:i+batch_size]
all_entities = []
all_relationships = []
for source in batch:
doc = ingestor.ingest_file(source)
parsed = parser.parse_document(source)
text = parsed.get("full_text", "")
entities = ner.extract_entities(text)
relationships = rel_extractor.extract_relations(text, entities=entities)
all_entities.extend(entities)
all_relationships.extend(relationships)
# Build graph from batch
kg = builder.build_graph(entities=all_entities, relationships=all_relationships)
# Save intermediate results
print(f"Processed batch {i//batch_size + 1}: {len(kg.nodes)} nodes")
batch = sources[i : i + batch_size]
all_entities, all_rels = [], []
for path in batch:
parsed = parser.parse(ingestor.ingest(path)[0])
all_entities.extend(ner.extract(parsed))
all_rels.extend(rel.extract(parsed, entities=all_entities))
kg = builder.build(entities=all_entities, relationships=all_rels)
print(f"Batch {i // batch_size + 1}: {len(kg.nodes)} nodes")
```
### Real-Time Streaming
```python
from semantica.ingest import StreamIngestor
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
stream = StreamIngestor(stream_uri="kafka://localhost:9092/topic")
ner = NERExtractor()
rel = RelationExtractor()
builder = GraphBuilder()
for batch in stream.stream(batch_size=100):
all_entities, all_rels = [], []
for item in batch:
text = str(item)
all_entities.extend(ner.extract(text))
all_rels.extend(rel.extract(text, entities=all_entities))
kg = builder.build(entities=all_entities, relationships=all_rels)
print(f"Processed batch: {len(kg.nodes)} nodes")
```
---
## More Resources
- **[Quick Start Guide](quickstart.md)** - Step-by-step tutorial
- **[API Reference](reference/core.md)** - Complete API documentation
- **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks
- **[Use Cases](use-cases.md)** - Real-world applications
### 🍳 Recommended Cookbook Tutorials
- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules
- **Topics**: Framework overview, all modules, architecture, configuration
- **Difficulty**: Beginner
- **Time**: 30-45 minutes
- **Use Cases**: First-time users, understanding the framework
- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph
- **Topics**: Entity extraction, relationship extraction, graph construction, visualization
- **Difficulty**: Beginner
- **Time**: 20-30 minutes
- **Use Cases**: Learning the basics, quick start
- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production-ready GraphRAG system
- **Topics**: GraphRAG, hybrid retrieval, vector search, graph traversal, LLM integration
- **Difficulty**: Advanced
- **Time**: 1-2 hours
- **Use Cases**: Production RAG applications
- **[RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)**: Benchmark standard RAG vs GraphRAG
- **Topics**: RAG, GraphRAG, benchmarking, visualization, reasoning gap
- **Difficulty**: Intermediate
- **Time**: 45-60 minutes
- **Use Cases**: Understanding GraphRAG advantages, choosing the right approach
---
!!! info "Contribute"
Have an example to share? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica)
- [Quickstart Tutorial](quickstart.md) — step-by-step first pipeline
- [Cookbook](cookbook.md) — interactive Jupyter notebooks
- [Use Cases](use-cases.md) — domain-specific examples
- [API Reference](reference/core.md) — complete API documentation
!!! info "Have an example to share?"
[Contribute on GitHub](https://github.com/Hawksight-AI/semantica)
docs/faq.md
+86 -85
View File
@@ -1,148 +1,149 @@
# Frequently Asked Questions
# Frequently Asked Questions
**Common questions about Semantica and how to use it.**
Common questions about Semantica. Use Ctrl+F to find what you need.
---
## General
### What is Semantica?
Semantica is an open-source framework for building knowledge graphs from unstructured data. It transforms documents, web pages, and databases into structured, queryable knowledge.
### What can I do with Semantica?
- **Build knowledge graphs** from documents and data
- **Extract entities and relationships** automatically
- **Power AI applications** with structured knowledge
- **Create semantic search** and GraphRAG systems
- **Integrate multiple data sources** into unified graphs
Semantica is an open-source framework for building context graphs and decision intelligence layers for AI. It transforms unstructured data — documents, APIs, databases — into structured knowledge graphs with full provenance tracking, making AI systems explainable and auditable.
### What can I build with Semantica?
- Knowledge graphs from documents and multi-source data
- GraphRAG systems with graph-grounded retrieval
- AI agents with structured decision history and semantic memory
- Compliance-ready pipelines with W3C PROV-O lineage
### What makes Semantica different from other frameworks?
Most frameworks stop at retrieval or generation. Semantica adds an **accountability layer**: every decision is recorded, every fact links to a source, and every reasoning step is explainable. It's designed for environments where you need to audit why an AI reached a conclusion.
### Is Semantica free?
Yes! Semantica is open source under the MIT License.
### What makes Semantica different?
- **Modular architecture** - Use only what you need
- **Production-ready** - Built for scale and reliability
- **Extensible** - Add custom models and components
- **Open source** - Transparent and community-driven
Yes — MIT licensed, no vendor lock-in. Some features require third-party API keys (e.g., OpenAI embeddings), but Semantica itself is free.
---
## Installation
### How do I install Semantica?
```bash
pip install semantica
```
See [Installation](installation.md) for virtual environment setup, optional extras, and troubleshooting.
### What Python version do I need?
Python 3.8 or higher. Python 3.11+ is recommended.
Python 3.8 or higher. Python 3.11+ is recommended for best performance.
### What are the system requirements?
- Python 3.8+
- 4GB+ RAM for basic use
- Optional GPU for embeddings and ML models
- 4 GB RAM minimum; 16 GB+ recommended for larger graphs
- Optional GPU for embedding generation and ML inference
---
## Getting Started
### How do I start using Semantica?
```python
from semantica.semantic_extract import NERExtractor
from semantica.kg import GraphBuilder
### Where do I start?
# Extract entities
ner = NERExtractor()
entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
1. [Installation](installation.md) — get set up
2. [Getting Started](getting-started.md) — core concepts and first example
3. [Quickstart Tutorial](quickstart.md) — full step-by-step pipeline
4. [Cookbook](cookbook.md) — interactive Jupyter notebooks
# Build knowledge graph
kg = GraphBuilder().build({"entities": entities})
```
### What data sources does Semantica support?
### Where can I find examples?
- **[Getting Started Guide](getting-started.md)** - Quick introduction
- **[Cookbook](cookbook.md)** - Practical examples
- **[GitHub Examples](https://github.com/Hawksight-AI/semantica/tree/main/examples)** - Code samples
- **Files** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives
- **Web** — crawl with `WebIngestor`, RSS feeds
- **Databases** — PostgreSQL, MySQL, Snowflake via `DBIngestor` / `SnowflakeIngestor`
- **Streams** — Kafka, real-time ingestion
- **Media** — image OCR, audio/video metadata
---
## Features
### What data sources does Semantica support?
- **Files**: PDF, DOCX, TXT, JSON, CSV
- **Web**: Websites, RSS feeds, APIs
- **Databases**: PostgreSQL, MySQL, Snowflake, MongoDB
- **Streams**: Kafka, RabbitMQ, real-time data
### Can I use my own models?
### Can I use custom models?
Yes! Semantica supports custom:
- **Entity extraction models**
- **Embedding models**
- **Language models**
- **Custom processors**
Yes. Semantica supports custom entity extraction models, embedding models, LLM providers (via LiteLLM — 100+ models), and custom pipeline processors.
### Does Semantica support GPUs?
Yes, Semantica automatically uses GPUs when available for:
- **Embedding generation**
- **ML model inference**
- **Vector operations**
Yes. When available, GPUs are used automatically for embedding generation, ML model inference, and vector operations. Install `semantica[gpu]` for CUDA support.
### How does Semantica handle large datasets?
- **Batching** — process documents in configurable chunks
- **Parallel processing** — `PipelineBuilder` supports configurable worker counts
- **Delta processing** — update graphs incrementally without full recompute
- **Graph backends** — swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE at scale
---
## Technical
### How does Semantica handle large datasets?
- **Batching** - Process data in chunks
- **Streaming** - Handle real-time data
- **Parallel processing** - Use multiple cores
- **Memory management** - Efficient resource usage
### What graph databases are supported?
### Can I deploy Semantica in production?
Yes! Semantica is production-ready with:
- **Scalable architecture**
- **Error handling**
- **Monitoring support**
- **Container deployment**
Neo4j, FalkorDB, Apache AGE (PostgreSQL), Amazon Neptune, and in-memory NetworkX for development.
### How do I customize Semantica?
- **Custom processors** - Add new extraction logic
- **Custom models** - Use your own ML models
- **Plugins** - Extend functionality
- **Configuration** - Adjust behavior
### What export formats are available?
RDF (Turtle, JSON-LD, N-Triples, XML), Apache Parquet, ArangoDB AQL, CSV, YAML, and OWL ontologies.
### Is Semantica production-ready?
Yes. v0.3.0 ships with 886+ passing tests, `PipelineValidator`, `FailureHandler` with exponential backoff, W3C PROV-O provenance, and change management with checksums. See [What's New](index.md#whats-new-in-v030) for details.
---
## Troubleshooting
### Installation issues
- **Python version**: Ensure Python 3.8+
- **Dependencies**: Install with `pip install -e .[dev]`
- **Permissions**: Use virtual environments
### Import error: `ModuleNotFoundError: No module named 'semantica'`
### Performance issues
- **Memory**: Increase available RAM
- **GPU**: Install CUDA for GPU acceleration
- **Batching**: Use smaller chunk sizes
Ensure you have the correct Python environment active, then:
### Common errors
- **Import errors**: Check installation path
- **Model loading**: Verify model availability
- **Memory errors**: Reduce batch sizes
```bash
pip list | grep semantica
pip install --upgrade semantica
```
### Installation fails with dependency errors
```bash
pip install --upgrade pip wheel
pip install semantica
```
### Memory errors during processing
Reduce batch sizes, enable streaming ingestion, or switch to a persistent graph backend (Neo4j, FalkorDB).
### Slow embedding or inference
Install GPU support (`pip install semantica[gpu]`) and ensure CUDA is available on your system.
---
## Support
### Where can I get help?
- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report problems
- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Ask questions
- **[Documentation](index.md)** - Browse guides and references
### How do I report bugs?
1. **Search** existing issues first
2. **Create** a new issue with details
3. **Include** reproduction steps
4. **Add** environment information
- [Discord](https://discord.gg/sV34vps5hH) — community chat and support
- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) — bug reports and feature requests
- [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) — questions and ideas
### Can I contribute?
Yes! See the [Contributing Guide](contributing.md) for details on how to help improve Semantica.
### How do I report a bug?
1. Search [existing issues](https://github.com/Hawksight-AI/semantica/issues) first
2. Open a new issue with: description, reproduction steps, expected vs actual behavior, and your environment (Python version, OS, Semantica version)
### How do I contribute?
See the [Contributing Guide](contributing.md).
docs/getting-started.md
+52 -55
View File
@@ -1,14 +1,18 @@
# Getting Started
## Overview
**Semantica** is the context and intelligence layer for AI — turning raw data into explainable, auditable knowledge graphs for high-stakes domains.
**Semantica** is a semantic intelligence layer that bridges the gap between raw data and trustworthy AI. It transforms unstructured data into explainable, auditable knowledge graphs perfect for high-stakes domains.
!!! tip "Just here for code?"
Jump straight to the [Quick Start](#quick-start) or explore the [Cookbook](cookbook.md) for interactive notebooks.
### What You Can Build
- **GraphRAG Systems** - Enhanced retrieval with semantic reasoning
- **AI Agents** - Trustworthy agents with explainable memory
- **Knowledge Graphs** - Production-ready semantic databases
- **Compliance-Ready AI** - Auditable systems with full provenance
---
## What You Can Build
- **GraphRAG Systems** — enhanced retrieval with semantic graph reasoning
- **AI Agents** — accountable agents with structured decision history and memory
- **Knowledge Graphs** — production-ready semantic knowledge bases
- **Compliance-Ready AI** — auditable systems with full W3C PROV-O provenance
---
@@ -18,17 +22,17 @@
pip install semantica
```
Or with all features:
With all optional dependencies:
```bash
pip install semantica[all]
```
Verify installation:
Verify:
```python
import semantica
print(f"Semantica {semantica.__version__} installed!")
print(semantica.__version__)
```
---
@@ -36,66 +40,59 @@ print(f"Semantica {semantica.__version__} installed!")
## Quick Start
```python
from semantica.semantic_extract import NERExtractor
from semantica.kg import GraphBuilder
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
# Extract entities
ner = NERExtractor(method="ml", model="en_core_web_sm")
entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
context = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768),
knowledge_graph=ContextGraph(advanced_analytics=True),
decision_tracking=True,
)
# Build knowledge graph
kg = GraphBuilder().build({"entities": entities, "relationships": []})
print(f"Built KG with {len(kg.get('entities', []))} entities")
# Store a memory
context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%")
# Record a decision
decision_id = context.record_decision(
category="model_selection",
scenario="Choose LLM for production pipeline",
reasoning="GPT-4 benchmark advantage justifies cost increase",
outcome="selected_gpt4",
confidence=0.91,
)
# Find similar past decisions
precedents = context.find_precedents("model selection", limit=5)
```
**What this does:**
- Extracts entities (people, organizations, dates) from text
- Builds a knowledge graph from extracted entities
- Outputs the number of entities found
---
## Core Architecture
Semantica uses a **modular architecture** - use only what you need:
Semantica uses a modular, layered architecture — import only what you need.
### 1️⃣ Input Layer - Data Ingestion
```python
from semantica.ingest import FileIngestor
documents = FileIngestor().ingest_directory("docs/")
```
### 2️⃣ Semantic Layer - Intelligence Engine
```python
from semantica.semantic_extract import NERExtractor, RelationExtractor
entities = NERExtractor().extract(text)
relationships = RelationExtractor().extract(text, entities)
```
### 3️⃣ Output Layer - Knowledge Assets
```python
from semantica.kg import GraphBuilder
kg = GraphBuilder().build_graph(entities, relationships)
```
| Layer | Modules | Purpose |
|-------|---------|---------|
| **Input** | `ingest`, `parse`, `split`, `normalize` | Load and prepare data |
| **Semantic** | `semantic_extract`, `kg`, `ontology`, `reasoning` | Extract meaning |
| **Storage** | `embeddings`, `vector_store`, `graph_store` | Persist knowledge |
| **Quality** | `deduplication`, `conflicts` | Validate and clean |
| **Context** | `context`, `provenance`, `change_management` | Track decisions and lineage |
| **Output** | `export`, `visualization`, `pipeline` | Deliver results |
---
## Next Steps
### 🍳 Interactive Tutorials
1. **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)** - Complete framework overview
2. **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)** - Hands-on graph building
3. **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)** - Production-ready RAG
### 📚 Learn More
- **[Core Concepts](concepts.md)** - Deep dive into knowledge graphs & ontologies
- **[Cookbook](cookbook.md)** - 14 domain-specific tutorials
- **[API Reference](reference/core.md)** - Complete technical documentation
- [Core Concepts](concepts.md) — knowledge graphs, ontologies, reasoning explained
- [Quickstart Tutorial](quickstart.md) — build a full pipeline step by step
- [Cookbook](cookbook.md) — 14 domain-specific Jupyter notebook tutorials
- [API Reference](reference/core.md) — complete module documentation
---
## Need Help?
## Help
- **[💬 Discord Community](https://discord.gg/sV34vps5hH)** - Get help from the community
- **[🐛 Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report bugs or request features
- **[📖 Documentation](https://semantica.readthedocs.io/)** - Full documentation site
- [Discord Community](https://discord.gg/sV34vps5hH) — ask questions, share projects
- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) — report bugs or request features
- [FAQ](faq.md) — common questions answered
docs/glossary.md
+7 -17
View File
@@ -1,9 +1,9 @@
# Glossary
**Comprehensive reference of terms and concepts used in Semantica and semantic intelligence.**
Reference of terms and concepts used throughout Semantica.
!!! tip "Quick Reference"
Looking for a specific term? Use your browser's search function (Ctrl+F) to find terms quickly.
!!! tip "Finding a term"
Use Ctrl+F to search this page.
---
@@ -216,17 +216,7 @@ W3C PROV-O compliant tracking of data lineage and source attribution.
## See Also
- **[Core Concepts](concepts.md)** - Deep dive into fundamental concepts
- **[Getting Started](getting-started.md)** - Begin your journey with Semantica
- **[Modules Guide](modules.md)** - Complete module overview
- **[API Reference](reference/)** - Technical documentation
---
## Need Help?
- **Documentation**: [Getting Started](getting-started.md)
- **Examples**: [Cookbook](cookbook.md)
- **Community**: [Discord](community.md)
- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
- **Support**: [Contact Us](community.md)
- [Core Concepts](concepts.md) — deeper explanation of key ideas
- [Getting Started](getting-started.md) — first steps
- [Modules Guide](modules.md) — every module explained
- [API Reference](reference/) — technical reference
docs/index.md
+191 -290
View File
@@ -1,24 +1,23 @@
<div align="center">
<img src="assets/img/Semantica Logo.png" alt="Semantica Logo" width="450" height="auto">
<img src="assets/img/Semantica Logo.png" alt="Semantica Logo" width="420" height="auto">
<h1>🧠 Semantica</h1>
<a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.8+-blue.svg" alt="Python 3.8+"></a>
<a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
<a href="https://badge.fury.io/py/semantica"><img src="https://img.shields.io/badge/pypi-v0.2.3-blue.svg" alt="PyPI version"></a>
<a href="https://pypi.org/project/semantica/"><img src="https://img.shields.io/pypi/dm/semantica" alt="Monthly Downloads"></a>
<a href="https://pypi.org/project/semantica/"><img src="https://img.shields.io/pypi/v/semantica.svg" alt="PyPI"></a>
<a href="https://github.com/Hawksight-AI/semantica/releases/tag/v0.3.0"><img src="https://img.shields.io/badge/version-0.3.0-brightgreen.svg" alt="Version"></a>
<a href="https://pepy.tech/project/semantica"><img src="https://static.pepy.tech/badge/semantica" alt="Total Downloads"></a>
<a href="https://semantica.readthedocs.io/"><img src="https://img.shields.io/badge/docs-latest-brightgreen.svg" alt="Documentation"></a>
<a href="https://discord.gg/sV34vps5hH"><img src="https://img.shields.io/badge/Discord-Join%20Us-7289da?style=flat&logo=discord&logoColor=white" alt="Discord"></a>
<p><strong>Open-Source Semantic Layer & Knowledge Engineering Framework</strong></p>
<p><strong>Transform Chaos into Intelligence. Build AI systems that are explainable, traceable, and trustworthy — not black boxes.</strong></p>
<p><em>The semantic intelligence layer that makes your AI agents auditable, explainable, and trustworthy. Perfect for high-stakes domains where mistakes have real consequences.</em></p>
<p>🆓 <strong>Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>
<a href="https://github.com/Hawksight-AI/semantica/actions"><img src="https://github.com/Hawksight-AI/semantica/workflows/CI/badge.svg" alt="CI"></a>
<a href="https://discord.gg/sV34vps5hH"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white" alt="Discord"></a>
<a href="https://x.com/BuildSemantica"><img src="https://img.shields.io/badge/X-Follow-black?logo=x&logoColor=white" alt="X"></a>
<p><strong>A Framework for Building Context Graphs and Decision Intelligence Layers for AI</strong></p>
<p>⭐ Give us a Star &nbsp;•&nbsp; 🍴 Fork us &nbsp;•&nbsp; 💬 Join our Discord &nbsp;•&nbsp; 🐦 Follow on X</p>
<p><em>Transform Chaos into Intelligence. Build AI systems with context graphs, decision tracking, and advanced knowledge engineering that are explainable, traceable, and trustworthy — not black boxes.</em></p>
<p>
<a href="getting-started/" class="md-button md-button--primary">Get Started</a>
<a href="https://github.com/Hawksight-AI/semantica" class="md-button">View on GitHub</a>
@@ -27,288 +26,266 @@
---
## 🚀 Why Semantica?
## The Problem
**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and trustworthy.
AI agents today are capable but not trustworthy:
Perfect for **high-stakes domains** where mistakes have real consequences.
- **No memory structure** — agents store embeddings, not meaning. Retrieval is fuzzy; there's no way to ask *why* something was recalled.
- **No decision trail** — agents make decisions continuously but record nothing. When something goes wrong, there's no history to debug or audit.
- **No provenance** — outputs cannot be traced back to source facts. In regulated industries, this is a compliance blocker.
- **No reasoning transparency** — black-box answers with no explanation of how a conclusion was reached.
- **No conflict detection** — contradictory facts silently coexist in vector stores, producing unpredictable answers.
These aren't edge cases. They are the reason AI cannot be deployed in healthcare, finance, legal, and government without custom guardrails built from scratch.
---
### ⚡ Get Started in 30 Seconds
## The Solution
Semantica is the **context and intelligence layer** you add to your AI stack:
- **Context Graphs** — structured graph of entities, relationships, and decisions your agent builds as it works. Queryable, traceable, persistent.
- **Decision Intelligence** — every decision is a first-class object: recorded, linked causally, searchable by precedent, and analyzable for downstream impact.
- **Provenance** — every fact links to its source. W3C PROV-O compliant. Full lineage from ingestion to inference.
- **Reasoning engines** — forward chaining, Rete networks, deductive, abductive, and SPARQL reasoning. Explainable inference paths, not black-box answers.
- **Deduplication & QA** — conflict detection, entity resolution, and validation built into the pipeline.
Works alongside LangChain, LlamaIndex, AutoGen, CrewAI, and any LLM provider — Semantica is not a replacement, it's the accountability layer on top.
---
### ⚡ Quick Installation
```bash
pip install semantica
```
```python
from semantica.semantic_extract import NERExtractor
from semantica.kg import GraphBuilder
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
# Extract entities and build knowledge graph
ner = NERExtractor(method="ml", model="en_core_web_sm")
entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
kg = GraphBuilder().build({"entities": entities, "relationships": []})
context = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768),
knowledge_graph=ContextGraph(advanced_analytics=True),
decision_tracking=True,
)
print(f"Built KG with {len(kg.get('entities', []))} entities")
# Store a memory
context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%")
# Record a decision
decision_id = context.record_decision(
category="model_selection",
scenario="Choose LLM for production reasoning pipeline",
reasoning="GPT-4 benchmark advantage justifies 3x cost increase",
outcome="selected_gpt4",
confidence=0.91,
)
# Find similar past decisions and analyze downstream impact
precedents = context.find_precedents("model selection reasoning", limit=5)
influence = context.analyze_decision_influence(decision_id)
```
**[📖 Full Quick Start](getting-started.md)** • **[🍳 Cookbook Examples](cookbook.md)** • **[💬 Join Discord](https://discord.gg/sV34vps5hH)** • **[⭐ Star Us](https://github.com/Hawksight-AI/semantica)**
**[Full Quick Start](getting-started.md)** &nbsp;•&nbsp; **[Cookbook](cookbook.md)** &nbsp;•&nbsp; **[Join Discord](https://discord.gg/sV34vps5hH)**
---
## What's New in v0.3.0
> First stable release (`Production/Stable` on PyPI).
| Area | Highlights |
|------|------------|
| **Context Graphs** | Temporal validity windows, weighted BFS, cross-graph navigation with save/load persistence |
| **Decision Intelligence** | Full lifecycle: record → trace → impact → precedent; `PolicyEngine` with versioned rules |
| **KG Algorithms** | PageRank, betweenness, Louvain community detection, Node2Vec, link prediction |
| **Semantic Extraction** | LLM extraction fixed (no silent drops), duplicate relation bug removed, `"llm_typed"` metadata corrected |
| **Deduplication v2** | `blocking_v2`/`hybrid_v2` — 63.6% faster; semantic v2 — 6.98x faster |
| **Delta Processing** | SPARQL-based incremental diff, `delta_mode` pipelines, snapshot versioning |
| **Export** | RDF aliases (`"ttl"`, `"json-ld"`), ArangoDB AQL, Apache Parquet (Spark/BigQuery/Databricks) |
| **Pipeline** | `FailureHandler` with LINEAR/EXPONENTIAL/FIXED backoff; `PipelineValidator` returning `ValidationResult` |
| **Graph Backends** | Apache AGE (SQL injection fixed), AWS Neptune, FalkorDB, PgVector (HNSW/IVFFlat) |
| **Tests** | 886+ passing, 0 failures — 335 context, ~430 KG, 70 semantic extraction, 85 real-world E2E |
---
## Core Value Proposition
| **Trustworthy** | **Explainable** | **Auditable** |
|:------------------:|:------------------:|:-----------------:|
|:---:|:---:|:---:|
| Conflict detection & validation | Transparent reasoning paths | Complete provenance tracking |
| Rule-based governance | Entity relationships & ontologies | W3C PROV-O compliant lineage |
| Production-grade QA | Multi-hop graph reasoning | Source tracking & integrity verification |
---
## Key Features & Benefits
## Features
### Not Just Another Agentic Framework
### Context & Decision Intelligence
- **Context Graphs** — structured, persistent graph of entities, relationships, and decisions
- **Decision tracking** — `add_decision()`, `record_decision()` for full lifecycle management
- **Causal chains** — `add_causal_relationship()`, `trace_decision_chain()`
- **Precedent search** — hybrid similarity search over past decisions via `find_similar_decisions()`
- **Influence analysis** — `analyze_decision_impact()`, `analyze_decision_influence()`
- **Policy engine** — `check_decision_rules()` with versioned, automated compliance rules
- **Agent memory** — `AgentMemory` with short/long-term storage and conversation history
**Semantica complements** LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK, Agno, and other frameworks to enhance your agents with:
### Knowledge Graphs
- **Graph construction** — entities, relationships, properties, typed edges
- **Algorithms** — PageRank, betweenness centrality, clustering coefficient, community detection
- **Node embeddings** — Node2Vec via `NodeEmbedder`; cosine similarity via `SimilarityCalculator`
- **Link prediction** — score potential edges via `LinkPredictor`
- **Temporal graphs** — time-aware nodes and edges with validity windows
- **Delta processing** — incremental updates without full recompute
| Feature | Benefit |
|:--------|:--------|
| **Auditable** | Complete provenance tracking with W3C PROV-O compliance |
| **Explainable** | Transparent reasoning paths with entity relationships |
| **Provenance-Aware** | End-to-end lineage from documents to responses |
| **Validated** | Built-in conflict detection, deduplication, QA |
| **Governed** | Rule-based validation and semantic consistency |
| **Version Control** | Enterprise-grade change management with integrity verification |
### Semantic Extraction
- **NER** — named entity recognition, normalization, classification
- **Relation extraction** — triplet generation via LLMs or rule-based methods, with `"llm_typed"` metadata
- **Deduplication v1/v2** — Jaro-Winkler, `blocking_v2`, `hybrid_v2`, `semantic_v2`; `dedup_triplets()` for triples
### Perfect For High-Stakes Use Cases
### Reasoning
- **Forward chaining** — `Reasoner` with IF/THEN string rules and dict facts
- **Rete network** — `ReteEngine` for high-throughput production rule matching
- **Deductive / Abductive** — `DeductiveReasoner`, `AbductiveReasoner`
- **SPARQL** — `SPARQLReasoner` for query-based inference over RDF graphs
| 🏥 **Healthcare** | 💰 **Finance** | ⚖️ **Legal** |
|:-----------------:|:--------------:|:------------:|
| Clinical decisions | Fraud detection | Evidence-backed research |
| Drug interactions | Regulatory support | Contract analysis |
| Patient safety | Risk assessment | Case law reasoning |
### Provenance & Auditability
- **Entity provenance** — `ProvenanceTracker.track_entity()`
- **Algorithm provenance** — `AlgorithmTrackerWithProvenance`
- **W3C PROV-O compliant** — lineage tracking across all modules
- **Change management** — version control with checksums, audit trails, compliance support
| 🔒 **Cybersecurity** | 🏛️ **Government** | 🏭 **Infrastructure** | 🚗 **Autonomous** |
|:-------------------:|:----------------:|:-------------------:|:-----------------:|
| Threat attribution | Policy decisions | Power grids | Decision logs |
| Incident response | Classified info | Transportation | Safety validation |
### Vector Store
- **Backends** — FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory
- **Search modes** — semantic top-k, hybrid (vector + keyword), metadata-filtered
### Powers Your AI Stack
### Data Ingestion
- **Files** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives
- **Sources** — web crawl, SQL databases, Snowflake, feeds, email, repositories
- **Docling** — advanced parsing with table and layout extraction
- **Media** — image OCR, audio/video metadata
- **GraphRAG Systems** — Retrieval with graph reasoning and hybrid search
- **AI Agents** — Trustworthy, accountable multi-agent systems with semantic memory
- **Reasoning Models** — Explainable AI decisions with reasoning paths
- **Enterprise AI** — Governed, auditable platforms that support compliance
### Export
- **RDF** — Turtle, JSON-LD, N-Triples, XML via `RDFExporter`
- **Parquet** — `ParquetExporter` for Spark/BigQuery/Databricks pipelines
- **ArangoDB AQL** — ready-to-run INSERT statements
- **OWL ontologies** — Turtle or RDF/XML
### Integrations
- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX)
- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication
- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD)
> **Built for environments where every answer must be explainable and governed.**
### Pipeline & Ontology
- **Pipeline DSL** — `PipelineBuilder` with stage chaining, parallel workers, retry policies
- **Ontology** — auto-generate OWL from KGs, import OWL/RDF/Turtle/JSON-LD, HermiT/Pellet validation
---
## 🚨 The Problem: The Semantic Gap
## Modules
### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**.
### Understanding the Semantic Gap
The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships).
**Traditional AI approaches:**
- Rely on statistical patterns and text similarity
- Cannot understand relationships between entities
- Cannot reason about domain-specific rules
- Cannot explain why decisions were made
- Cannot trace back to original sources with confidence
**High-stakes AI requires:**
- Semantic understanding of entities and their relationships
- Domain knowledge encoded as formal rules (ontologies)
- Explainable reasoning paths
- Source-level provenance
- Conflict detection and resolution
**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge.
### What Organizations Have vs What They Need
| **Current State** | **Required for High-Stakes AI** |
|:---------------------|:-----------------------------------|
| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) |
| APIs, databases, streams | Structured and validated entities |
| Conflicting facts and duplicates | Explicit semantic relationships |
| Siloed systems with no lineage | **Explainable reasoning paths** |
| | **Source-level provenance** |
| | **Audit-ready compliance** |
### The Cost of Missing Semantics
- **Decisions cannot be explained** — No transparency in AI reasoning
- **Errors cannot be traced** — No way to debug or improve
- **Conflicts go undetected** — Contradictory information causes failures
- **Compliance becomes impossible** — No audit trails for regulations
**Trustworthy AI requires semantic accountability.**
| Module | What it provides |
|--------|-----------------|
| `semantica.context` | Context graphs, agent memory, decision tracking, causal analysis, precedent search, policy engine |
| `semantica.kg` | KG construction, graph algorithms, centrality, community detection, embeddings, link prediction |
| `semantica.semantic_extract` | NER, relation extraction, event extraction, coreference, triplet generation, LLM extraction |
| `semantica.reasoning` | Forward chaining, Rete network, deductive, abductive, SPARQL reasoning, explanation generation |
| `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector; hybrid & filtered search |
| `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, YAML, OWL, graph formats |
| `semantica.ingest` | Files, web crawl, feeds, databases, Snowflake, MCP, email, repositories |
| `semantica.ontology` | Auto-generation, OWL/RDF export, import, validation, versioning |
| `semantica.pipeline` | Pipeline DSL, parallel workers, validation, retry policies, failure handling |
| `semantica.graph_store` | Neo4j, FalkorDB, Apache AGE, Amazon Neptune; Cypher queries |
| `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE; similarity calculation |
| `semantica.deduplication` | Entity deduplication, similarity scoring, merging, clustering |
| `semantica.provenance` | W3C PROV-O lineage tracking, source attribution, audit trails |
| `semantica.parse` | PDF, DOCX, PPTX, HTML, code, email, structured data, OCR |
| `semantica.split` | Recursive, semantic, entity-aware, relation-aware, graph-based chunking |
| `semantica.normalize` | Text, entities, dates, numbers, quantities, languages, encodings |
| `semantica.conflicts` | Multi-source conflict detection (value, type, temporal, logical) with resolution |
| `semantica.change_management` | Version storage, change tracking, checksums, audit trails |
| `semantica.triplet_store` | Blazegraph, Jena, RDF4J; SPARQL queries and bulk loading |
| `semantica.visualization` | Interactive/static KG, ontology, embedding, and temporal graph visualization |
| `semantica.core` | Framework orchestration, configuration, plugin system |
| `semantica.llms` | Groq, OpenAI, Novita AI, HuggingFace, LiteLLM integrations |
---
## 🆚 Semantica vs Traditional RAG
## Built for High-Stakes Domains
| Feature | Traditional RAG | Semantica |
|:--------|:----------------|:----------|
| **Reasoning** | ❌ Black-box answers | ✅ Explainable reasoning paths |
| **Provenance** | ❌ No provenance | ✅ W3C PROV-O compliant lineage tracking |
| **Search** | ⚠️ Vector similarity only | ✅ Semantic + graph reasoning |
| **Quality** | ❌ No conflict handling | ✅ Explicit contradiction detection |
| **Safety** | ⚠️ Unsafe for high-stakes | ✅ Designed for governed environments |
| **Compliance** | ❌ No audit trails | ✅ Complete audit trails with integrity verification |
Where **every decision must be accountable** and **mistakes have real consequences**:
- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interactions, patient safety
- **💰 Finance & Risk** — Fraud detection, SOX/GDPR/MiFID II compliance, risk assessment
- **⚖️ Legal & Compliance** — Evidence-backed research, contract analysis, regulatory tracking
- **🔒 Cybersecurity** — Threat attribution, incident response, security audit trails
- **🏛️ Government & Defense** — Policy decisions, classified information handling, defense intelligence
- **🏭 Critical Infrastructure** — Power grids, transportation safety, emergency response
- **🚗 Autonomous Systems** — Self-driving, robotics safety, industrial automation
---
## 🧩 Semantica Architecture
### 1️⃣ Input Layer — Governed Ingestion
- 📄 **Multiple Formats** — PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX
- 🔧 **Docling Support** — Docling parser for table extraction
- 💾 **Data Sources** — Databases, APIs, streams, archives, web content
- 🎨 **Media Support** — Image parsing with OCR, audio/video metadata extraction
- � **Single Pipeline** — Unified ingestion with metadata and source tracking
### 2️⃣ Semantic Layer — Trust & Reasoning Engine
- 🔍 **Entity Extraction** — NER, normalization, classification
- 🔗 **Relationship Discovery** — Triplet generation, semantic links
- 📐 **Ontology Induction** — Automated domain rule generation
- 🔄 **Deduplication** — Jaro-Winkler similarity, conflict resolution
- ✅ **Quality Assurance** — Conflict detection, validation
- 📊 **Provenance Tracking** — W3C PROV-O compliant lineage tracking across all modules
- 🧠 **Reasoning Traces** — Explainable inference paths
- 🔐 **Change Management** — Version control with audit trails, checksums, compliance support
### 3️⃣ Output Layer — Auditable Knowledge Assets
- � **Knowledge Graphs** — Queryable, temporal, explainable
- 📐 **OWL Ontologies** — HermiT/Pellet validated, custom ontology import support
- 🔢 **Vector Embeddings** — FastEmbed by default
- ☁️ **AWS Neptune** — Amazon Neptune graph database support
- 🔍 **Provenance** — Every AI response links back to:
- 📄 Source documents
- 🏷️ Extracted entities & relations
- 📐 Ontology rules applied
- 🧠 Reasoning steps used
---
## 🏥 Built for High-Stakes Domains
Designed for domains where **mistakes have real consequences** and **every decision must be accountable**:
- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety tracking
- **💰 Finance & Risk** — Fraud detection, regulatory support (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation
- **⚖️ Legal & Compliance** — Evidence-backed legal research, contract analysis, regulatory change tracking, case law reasoning
- **🔒 Cybersecurity & Intelligence** — Threat attribution, incident response, security audit trails, intelligence analysis
- **🏛️ Government & Defense** — Governed AI systems, policy decisions, classified information handling, defense intelligence
- **🏭 Critical Infrastructure** — Power grid management, transportation safety, water treatment, emergency response
- **🚗 Autonomous Systems** — Self-driving vehicles, drone navigation, robotics safety, industrial automation
---
## � Who Uses Semantica?
- **🤖 AI / ML Engineers** — Building explainable GraphRAG & agents
- **⚙️ Data Engineers** — Creating governed semantic pipelines
- **📊 Knowledge Engineers** — Managing ontologies & KGs at scale
- **🏢 Enterprise Teams** — Requiring trustworthy AI infrastructure
- **🛡️ Risk & Compliance Teams** — Needing audit-ready systems
---
## 🚀 Choose Your Path
## Choose Your Path
<div class="grid cards" markdown>
- :material-rocket-launch: **Quick Start**
---
Get up and running with Semantica in minutes. Learn the basics of ingestion and extraction.
Up and running in minutes.
[:arrow_right: Start Here](getting-started.md)
- :material-book-open-page-variant: **Core Concepts**
---
Deep dive into Knowledge Graphs, Ontologies, and Semantic Reasoning.
Knowledge graphs, ontologies, and semantic reasoning explained.
[:arrow_right: Learn Concepts](concepts.md)
- :material-code-braces: **API Reference**
---
Detailed technical documentation for all Semantica modules and classes.
Full technical documentation for every module and class.
[:arrow_right: View API](reference/core.md)
- :material-chef-hat: **Cookbook**
---
Interactive tutorials, real-world examples, and **14 domain-specific cookbooks**.
14 domain-specific cookbooks with real-world examples.
[:arrow_right: Explore Cookbook](cookbook.md)
</div>
---
## 📦 Installation
## Installation
!!! success "Now Available on PyPI!"
Semantica is officially published on PyPI! Install it with a single command.
Install with a single command.
=== "From PyPI (Recommended)"
Install Semantica directly from PyPI:
=== "PyPI (Recommended)"
```bash
# Install the core package
pip install semantica
# Or install with all optional dependencies
# With all optional dependencies
pip install semantica[all]
```
=== "From Source"
Install from the local source for the latest development version:
```bash
# Clone the repository
git clone https://github.com/Hawksight-AI/semantica.git
cd semantica
# Install in editable mode with core dependencies
pip install -e .
# Or install with all optional dependencies
pip install -e ".[all]"
pip install -e . # core
pip install -e ".[all]" # all extras
```
=== "Development"
For contributors who want to modify the framework:
```bash
# Clone the repository
git clone https://github.com/Hawksight-AI/semantica.git
cd semantica
# Install in editable mode with dev dependencies
pip install -e ".[dev]"
```
=== "Docker"
Run Semantica in a containerized environment:
```bash
docker pull semantica/semantica:latest
docker run -it semantica/semantica
@@ -316,117 +293,41 @@ Designed for domains where **mistakes have real consequences** and **every decis
---
## 🚦 Quick Example
Semantica uses a modular architecture. You can use individual modules directly for maximum flexibility:
```python
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
# 1. Ingest documents
ingestor = FileIngestor()
documents = ingestor.ingest_directory("documents/", recursive=True)
# 2. Parse documents
parser = DocumentParser()
parsed_docs = [parser.parse_document(doc) for doc in documents]
# 3. Extract entities and relationships
ner = NERExtractor()
rel_extractor = RelationExtractor()
entities = []
relationships = []
for doc in parsed_docs:
text = doc.get("full_text", "")
doc_entities = ner.extract_entities(text)
doc_rels = rel_extractor.extract_relations(text, entities=doc_entities)
entities.extend(doc_entities)
relationships.extend(doc_rels)
# 4. Build knowledge graph
builder = GraphBuilder(merge_entities=True)
kg = builder.build_graph(entities=entities, relationships=relationships)
print(f"Created graph with {len(kg.nodes)} nodes and {len(kg.edges)} edges")
```
!!! tip "Orchestration Option"
For complex workflows, you can also use the `Semantica` class for orchestration. See the [Core Module](reference/core.md) documentation for details.
---
## 🎯 Why Semantica?
## Why Semantica?
<div class="grid cards" markdown>
- **🆓 Open Source**
---
MIT licensed. No vendor lock-in. Full transparency.
MIT licensed. No vendor lock-in.
- **🚀 Production Ready**
---
Battle-tested with quality assurance, conflict resolution, and validation.
Battle-tested with QA, conflict resolution, and validation built in.
- **🧩 Modular Architecture**
- **🧩 Modular**
---
Use only what you need. Swap components easily.
- **🌍 Community Driven**
---
Built by developers, for developers. Active Discord community.
Built by developers, for developers. Active Discord.
- **📚 Comprehensive**
- **📚 End-to-End**
---
End-to-end solution from ingestion to reasoning. No duct-taping required.
From ingestion to reasoning — no duct-taping required.
- **🔬 Research-Backed**
---
Based on latest research in knowledge graphs, ontologies, and semantic web.
Grounded in knowledge graph, ontology, and semantic web research.
</div>
---
## 🏗️ Built For
- **Data Scientists**: Transform messy data into clean knowledge graphs
- **Data Engineers**: Build scalable data pipelines with semantic enrichment
- **AI Engineers**: Build GraphRAG, AI agents, and multi-agent systems
- **Knowledge Engineers**: Generate and manage formal ontologies
- **Ontologists**: Design and validate domain-specific ontologies and taxonomies
- **Researchers**: Analyze scientific literature and build citation networks
- **ML Engineers**: Create semantic features for machine learning models
- **Enterprises**: Unify data silos into a semantic layer
---
## 📚 Learn More
- [Getting Started Guide](getting-started.md) - Your first knowledge graph in 5 minutes
- [Core Concepts](concepts.md) - Deep dive into knowledge graphs and ontologies
- [Cookbook](cookbook.md) - Real-world examples and **14 domain-specific cookbooks**
- [API Reference](reference/core.md) - Complete technical documentation
### 🍳 Recommended Cookbook Tutorials
Get hands-on with interactive Jupyter notebooks:
- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all Semantica modules
- **Topics**: Framework overview, all modules, architecture
- **Difficulty**: Beginner
- **Use Cases**: First-time users, understanding the framework
- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph from scratch
- **Topics**: Entity extraction, relationship extraction, graph construction
- **Difficulty**: Beginner
- **Use Cases**: Learning the basics, quick start
- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production-ready Graph Retrieval Augmented Generation
- **Topics**: GraphRAG, hybrid retrieval, vector search, graph traversal
- **Difficulty**: Advanced
- **Use Cases**: Building AI applications with knowledge graphs
## Learn More
- [Getting Started](getting-started.md) — your first knowledge graph in 5 minutes
- [Core Concepts](concepts.md) — knowledge graphs, ontologies, and semantic reasoning
- [Cookbook](cookbook.md) — 14 domain-specific cookbooks with Jupyter notebooks
- [API Reference](reference/core.md) — complete technical documentation
docs/installation.md
+112 -236
View File
@@ -1,293 +1,169 @@
# Installation
Get Semantica up and running in minutes.
Get Semantica installed in under a minute.
!!! success "Now Available on PyPI!"
Semantica is officially published on PyPI! Install it with a single command: `pip install semantica`
!!! success "Available on PyPI"
`pip install semantica` — that's it.
!!! note "System Requirements"
Semantica requires Python 3.8 or higher. For best performance, we recommend Python 3.10+.
!!! note "Requirements"
Python 3.8 or higher. Python 3.11+ recommended.
## Prerequisites
Before installing Semantica, ensure you have:
- **Python 3.8 or higher** - Check your version:
```bash
python --version
```
- **pip** - Python package installer (usually comes with Python)
---
## Basic Installation
Install Semantica from PyPI:
```bash
pip install semantica
```
This installs Semantica with all core dependencies.
### GitHub Workaround
If you encounter issues with the PyPI version, you can install directly from the main branch:
```bash
pip install git+https://github.com/Hawksight-AI/semantica.git@main
```
!!! tip "Virtual Environment"
We recommend installing Semantica in a virtual environment to avoid dependency conflicts. Use `python -m venv venv` to create one, then activate it before installing.
## Verify Installation
Verify that Semantica is installed correctly:
```bash
python -c "from semantica.parse import DoclingParser; DoclingParser(); print('✓ Semantica ready')"
```
!!! info "Windows PyTorch Note"
If you encounter PyTorch DLL errors on Windows, ensure you have the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) installed. This is a common environment-specific issue with PyTorch on Windows and not a bug in Semantica.
Expected output:
```
✓ Semantica ready
```
You can also check the installation:
```bash
pip show semantica
```
## Development Installation
To install Semantica in development mode (for contributing):
```bash
# Clone the repository
git clone https://github.com/Hawksight-AI/semantica.git
cd semantica
# Install in editable mode
pip install -e .
# Or install with development dependencies
pip install -e ".[dev]"
```
## Optional Dependencies
Semantica supports optional features that can be installed separately:
### GPU Support
For GPU-accelerated operations:
```bash
pip install semantica[gpu]
```
This includes:
- PyTorch with CUDA support
- FAISS GPU
- CuPy
### Visualization
For enhanced visualization capabilities:
```bash
pip install semantica[viz]
```
Includes:
- PyVis for interactive graphs
- Graphviz for static diagrams
- UMAP for dimensionality reduction
### LLM Providers
Install all LLM provider integrations:
```bash
pip install semantica[llm-all]
```
Or install specific providers:
```bash
# OpenAI
pip install semantica[llm-openai]
# Anthropic
pip install semantica[llm-anthropic]
# Google Gemini
pip install semantica[llm-gemini]
# Groq
pip install semantica[llm-groq]
# Ollama
pip install semantica[llm-ollama]
```
### Cloud Integrations
For cloud storage and deployment:
```bash
pip install semantica[cloud]
```
Includes:
- AWS S3 (boto3)
- Azure Blob Storage
- Google Cloud Storage
- Kubernetes support
### All Optional Features
Install everything:
With all optional dependencies:
```bash
pip install semantica[all]
```
## Virtual Environment (Recommended)
### Verify
It's recommended to use a virtual environment:
```bash
python -c "import semantica; print(semantica.__version__)"
```
---
## Virtual Environment (Recommended)
=== "venv"
```bash
# Create virtual environment
python -m venv venv
# Activate (Windows)
venv\Scripts\activate
# Activate (Linux/Mac)
source venv/bin/activate
# Install Semantica
source venv/bin/activate # Linux / Mac
venv\Scripts\activate # Windows
pip install semantica
```
=== "conda"
```bash
# Create conda environment
conda create -n semantica python=3.11
conda activate semantica
# Install Semantica
pip install semantica
```
---
## Optional Dependencies
Install only what you need:
=== "GPU"
```bash
pip install semantica[gpu]
```
Includes PyTorch with CUDA, FAISS GPU, CuPy.
=== "Visualization"
```bash
pip install semantica[viz]
```
Includes PyVis, Graphviz, UMAP.
=== "LLM Providers"
```bash
pip install semantica[llm-all] # all providers
pip install semantica[llm-openai] # OpenAI
pip install semantica[llm-anthropic] # Anthropic
pip install semantica[llm-gemini] # Google Gemini
pip install semantica[llm-groq] # Groq
pip install semantica[llm-ollama] # Ollama (local)
```
=== "Cloud"
```bash
pip install semantica[cloud]
```
Includes AWS S3, Azure Blob, Google Cloud Storage.
---
## Install from Source
For the latest development version or to contribute:
```bash
git clone https://github.com/Hawksight-AI/semantica.git
cd semantica
pip install -e . # core only
pip install -e ".[all]" # all extras
pip install -e ".[dev]" # dev tools (pytest, black, etc.)
```
If you encounter issues with the PyPI release, install directly from the main branch:
```bash
pip install git+https://github.com/Hawksight-AI/semantica.git@main
```
---
## Troubleshooting
### Common Issues
### ModuleNotFoundError
#### ModuleNotFoundError
Check you have the right environment active:
**Error**: `ModuleNotFoundError: No module named 'semantica'`
```bash
pip list | grep semantica
pip install --upgrade semantica
```
**Solutions**:
- Make sure you've activated the correct Python environment
- Verify installation: `pip list | grep semantica`
- Reinstall: `pip install --upgrade semantica`
### Installation fails with dependency errors
#### Installation Fails
```bash
pip install --upgrade pip
pip install build wheel
pip install semantica --no-deps # install without optional deps first
```
**Error**: Installation fails with dependency errors
### GPU dependencies fail
**Solutions**:
- Upgrade pip: `pip install --upgrade pip`
- Install build tools: `pip install build wheel`
- Try installing without optional dependencies first: `pip install semantica --no-deps`
Install CPU-only first, then add GPU support:
#### GPU Dependencies Fail
```bash
pip install semantica
pip install semantica[gpu]
```
**Error**: GPU dependencies fail to install
### Permission denied
**Solutions**:
- Install CPU-only version first: `pip install semantica`
- Then add GPU support: `pip install semantica[gpu]`
- Check CUDA compatibility for your system
```bash
pip install --user semantica # or use a virtual environment
```
#### Permission Errors
### Windows PyTorch DLL errors
**Error**: Permission denied during installation
Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe). This is a Windows system dependency, not a Semantica bug.
**Solutions**:
- Use `--user` flag: `pip install --user semantica`
- Use virtual environment (recommended)
- On Linux/Mac, avoid using `sudo` with pip
---
### System Requirements
## System Requirements
| Component | Minimum | Recommended |
|-----------|---------|-------------|
| Python | 3.8 | 3.11+ |
| RAM | Moderate | Ample for your dataset |
| Disk Space | Sufficient for data | Generous storage |
| OS | Windows/Linux/Mac | Linux/Mac |
| OS | Windows / Linux / Mac | Linux / Mac |
| RAM | 4 GB | 16 GB+ |
| Storage | 2 GB | 20 GB+ (for models and data) |
## After Installation
Once Semantica is installed, verify your setup and get started:
### Verify Your Installation
Test that everything works correctly:
```bash
python -c "import semantica; print(semantica.__version__)"
```
**For detailed setup verification and first steps, see:**
- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Verify installation and explore all modules
- **Topics**: Framework overview, installation verification, module exploration
- **Difficulty**: Beginner
- **Time**: 30-45 minutes
- **Use Cases**: First-time setup, understanding the framework
---
## Next Steps
Now that Semantica is installed:
1. **[Quick Start Guide](quickstart.md)** - Build your first knowledge graph in 5 minutes
2. **[Getting Started Guide](getting-started.md)** - Learn the fundamentals
3. **[Examples](examples.md)** - See real-world use cases
4. **[Cookbook](cookbook.md)** - Interactive Jupyter notebook tutorials
### 🍳 Recommended First Cookbooks
Start with these interactive tutorials:
- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction
- **Topics**: Framework overview, all modules, architecture, configuration
- **Difficulty**: Beginner
- **Time**: 30-45 minutes
- **Use Cases**: First-time users, understanding the framework
- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first graph
- **Topics**: Entity extraction, relationship extraction, graph construction
- **Difficulty**: Beginner
- **Time**: 20-30 minutes
- **Use Cases**: Hands-on practice, quick start
## Getting Help
If you encounter issues:
- Check the [troubleshooting section](#troubleshooting) above
- Review [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
- Ask questions in [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
**For installation and setup help:**
- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Includes setup verification steps
- **[Installation Troubleshooting Guide](getting-started.md#installation--setup)**: Additional troubleshooting tips
- [Getting Started](getting-started.md) — build your first knowledge graph
- [Quickstart Tutorial](quickstart.md) — full step-by-step pipeline
- [Cookbook](cookbook.md) — interactive Jupyter notebook tutorials
docs/integrations/docling.md
+4 -4
View File
@@ -13,7 +13,7 @@ Docling is integrated into Semantica's `parse` module via the `DoclingParser`. T
---
## 📖 Integration Documentation
## Integration Documentation
The `DoclingParser` provides a high-level interface for document processing. It supports:
@@ -42,7 +42,7 @@ For more details, see the [Parse Reference](../reference/parse.md).
---
## 🧑🏽‍🍳 Integration Example
## Integration Example
We provide a detailed cookbook and clear code examples to help you get started quickly.
@@ -81,7 +81,7 @@ See more in our [Code Examples](../CodeExamples.md).
---
## 💻 GitHub Source
## GitHub Source
The integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues.
@@ -89,7 +89,7 @@ The integration is open-source and available on GitHub. You can explore the impl
---
## 📦 PyPI & Installation
## PyPI & Installation
Docling is an optional but highly recommended dependency for Semantica. You can install it along with Semantica or as a separate requirement.
docs/integrations/snowflake.md
+9 -9
View File
@@ -13,7 +13,7 @@ Snowflake is integrated into Semantica's `ingest` module via the `SnowflakeInges
---
## 📖 Integration Documentation
## Integration Documentation
The `SnowflakeIngestor` provides a high-level interface for Snowflake data ingestion. It supports:
@@ -42,7 +42,7 @@ For more details, see the [Ingest Reference](../reference/ingest.md).
---
## 🧑🏽‍🍳 Integration Example
## Integration Example
We provide a detailed cookbook and clear code examples to help you get started quickly.
@@ -97,7 +97,7 @@ See more in our [Code Examples](../CodeExamples.md).
---
## 💻 GitHub Source
## GitHub Source
The integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues.
@@ -105,7 +105,7 @@ The integration is open-source and available on GitHub. You can explore the impl
---
## 📦 PyPI & Installation
## PyPI & Installation
Snowflake connector is an optional dependency for Semantica. You can install it along with Semantica or as a separate requirement.
@@ -128,7 +128,7 @@ For full installation details, see the [Installation Guide](../installation.md).
---
## 🔐 Authentication Methods
## Authentication Methods
Snowflake integration supports multiple authentication methods for different security requirements:
@@ -175,7 +175,7 @@ ingestor = SnowflakeIngestor(
---
## 🚀 Advanced Features
## Advanced Features
### Schema Introspection
```python
@@ -209,7 +209,7 @@ data = ingestor.ingest_query(
---
## 📊 Best Practices
## Best Practices
### Use Environment Variables
```python
@@ -244,7 +244,7 @@ for page in range(total_pages):
---
## 🔍 Troubleshooting
## Troubleshooting
### Connection Issues
```python
@@ -272,7 +272,7 @@ ingestor = SnowflakeIngestor(
---
## 📚 See Also
## See Also
- **[Ingest Module Reference](../reference/ingest.md)** - Complete ingestion documentation
- **[Getting Started Guide](../getting-started.md)** - Quick start with Semantica
docs/learning-more.md
+73 -216
View File
@@ -1,213 +1,104 @@
# Learning More
Additional resources, tutorials, and advanced learning materials for Semantica.
!!! info "About This Guide"
This guide provides structured learning paths, quick references, troubleshooting guides, and advanced topics to help you master Semantica.
Structured learning paths, quick references, and performance guidance for going deeper with Semantica.
---
## Structured Learning Paths
## Learning Paths
<div class="grid cards" markdown>
- :material-school: **Beginner Path**
- :material-school: **Beginner** (1–2 hours)
---
Perfect for those new to Semantica and knowledge graphs.
New to Semantica and knowledge graphs.
[Start Path](#beginner-path-1-2-hours)
[Start here](#beginner-path)
- :material-compass: **Intermediate Path**
- :material-compass: **Intermediate** (4–6 hours)
---
For users comfortable with basics who want to build production applications.
Comfortable with basics, building production applications.
[Start Path](#intermediate-path-4-6-hours)
[Start here](#intermediate-path)
- :material-rocket: **Advanced Path**
- :material-rocket: **Advanced** (8+ hours)
---
For experienced users building enterprise applications.
Enterprise applications and customization.
[Start Path](#advanced-path-8-hours)
[Start here](#advanced-path)
</div>
---
### Beginner Path (1-2 hours)
### Beginner Path
1. **Installation & Setup** (15 min)
- [Installation Guide](installation.md)
- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction
- **Topics**: Framework overview, all modules, architecture, configuration
- **Difficulty**: Beginner
- **Time**: 30-45 minutes
- **Use Cases**: First-time users, understanding the framework
2. **Core Concepts** (30 min)
- [Core Concepts](concepts.md)
- [Getting Started Guide](getting-started.md)
- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources
- **Topics**: File, web, feed, stream, database ingestion
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Loading data from various sources
3. **First Knowledge Graph** (30 min)
- [Quickstart Tutorial](quickstart.md)
- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph
- **Topics**: Entity extraction, relationship extraction, graph construction, visualization
- **Difficulty**: Beginner
- **Time**: 20-30 minutes
- **Use Cases**: Learning the basics, quick start
4. **Basic Operations** (30 min)
- [Examples](examples.md)
- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn entity extraction
- **Topics**: Named entity recognition, entity types, extraction methods
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Understanding entity extraction
1. **Installation & Setup** — [Installation Guide](installation.md)
2. **Core Concepts** — [Core Concepts](concepts.md) + [Getting Started](getting-started.md)
3. **First Knowledge Graph** — [Quickstart Tutorial](quickstart.md)
4. **Interactive Introduction** — [Welcome to Semantica notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)
5. **Hands-On Practice** — [Your First Knowledge Graph notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)
---
### Intermediate Path (4-6 hours)
### Intermediate Path
1. **Advanced Concepts** (1 hour)
- [Modules Guide](modules.md)
- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Advanced graph construction
- **Topics**: Graph building, entity merging, conflict resolution, temporal graphs
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Production graph construction
- **[Embeddings Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb)**: Learn embeddings
- **Topics**: Embedding generation, similarity search, vector operations
- **Difficulty**: Beginner
- **Time**: 20-30 minutes
- **Use Cases**: Understanding embeddings, semantic search
2. **Use Cases** (1 hour)
- [Use Cases Guide](use-cases.md)
- **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Build production GraphRAG
- **Topics**: GraphRAG, hybrid retrieval, graph traversal, LLM integration
- **Difficulty**: Advanced
- **Time**: 1-2 hours
- **Use Cases**: Production GraphRAG systems
3. **Advanced Examples** (1 hour)
- [Examples](examples.md)
- **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns
- **Topics**: Custom entity types, domain-specific extraction, hybrid methods
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Domain-specific extraction
4. **Quality & Optimization** (1 hour)
- [Quality Assurance](concepts.md#8-quality-assurance)
- [Performance Optimization](#performance-optimization)
- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Integrate multiple sources
- **Topics**: Multi-source integration, entity resolution, conflict handling
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Building unified knowledge graphs
1. **All Modules** — [Modules Guide](modules.md)
2. **Advanced Graph Construction** — [Building Knowledge Graphs notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)
3. **Embeddings & Search** — [Embeddings notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb)
4. **GraphRAG** — [GraphRAG Complete notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
5. **Multi-Source Integration** — [Multi-Source Data Integration notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)
6. **Use Case Examples** — [Use Cases](use-cases.md)
---
### Advanced Path (8+ hours)
### Advanced Path
1. **Advanced Architecture** (2 hours)
- [Architecture Guide](architecture.md)
- **[Temporal Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb)**: Build temporal graphs
- **Topics**: Time-stamped entities, temporal relationships, historical queries
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Time-aware knowledge graphs
- **[Ontology Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)**: Generate ontologies
- **Topics**: Ontology generation, OWL, schema design
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Formal knowledge representation
2. **Production Deployment** (2 hours)
- [Security Best Practices](#security-best-practices)
- **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production GraphRAG
- **Topics**: Production deployment, scalability, optimization
- **Difficulty**: Advanced
- **Time**: 1-2 hours
- **Use Cases**: Production systems
3. **Customization** (2 hours)
- **[Complete Visualization Suite Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)**: Advanced visualization
- **Topics**: Custom layouts, filtering, styling, multiple graph types
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Production visualizations
- **[Multi-Format Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)**: Advanced export patterns
- **Topics**: Batch export, custom formats, format conversion
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Production exports
1. **Architecture Deep Dive** — [Architecture Guide](architecture.md)
2. **Temporal Graphs** — [Temporal Graphs notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb)
3. **Ontologies** — [Ontology notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)
4. **Visualization** — [Complete Visualization Suite notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)
5. **Export Pipelines** — [Multi-Format Export notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)
6. **Production GraphRAG** — [GraphRAG Complete notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
---
## Quick Reference
## Configuration Reference
### Common Operations
The typical workflow involves these steps:
1. **Ingest** documents using `` `FileIngestor` ``
2. **Parse** documents using `` `DocumentParser` ``
3. **Extract** entities and relationships using `` `NERExtractor` `` and `` `RelationExtractor` ``
4. **Build** knowledge graph using `` `GraphBuilder` ``
5. **Generate** embeddings using `` `TextEmbedder` ``
**For complete examples, see:**
- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Complete workflow example
- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: All modules overview
### Configuration Reference
| Setting | Environment Variable | Config File | Default |
| :--- | :--- | :--- | :--- |
| OpenAI API Key | `OPENAI_API_KEY` | `api_keys.openai` | `None` |
| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `embedding.provider` | `"openai"` |
| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `knowledge_graph.backend` | `"networkx"` |
| Setting | Environment Variable | Default |
|---------|---------------------|---------|
| OpenAI API Key | `OPENAI_API_KEY` | `None` |
| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `"openai"` |
| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `"networkx"` |
---
## Troubleshooting Guide
## Troubleshooting
<div class="grid cards" markdown>
- :material-alert: **Import Errors**
---
`ModuleNotFoundError`
**Solution**: Verify installation (`pip list`) and Python version (3.8+).
Verify installation: `pip list | grep semantica`. Ensure Python 3.8+.
- :material-key: **API Key Errors**
---
`AuthenticationError`
**Solution**: Set `OPENAI_API_KEY` environment variable.
Set `OPENAI_API_KEY` (or the relevant provider key) as an environment variable.
- :material-memory: **Memory Errors**
---
`MemoryError`
**Solution**: Use batch processing and graph stores (Neo4j).
`MemoryError` or OOM crashes
Reduce batch sizes or switch to a persistent graph backend (Neo4j, FalkorDB).
- :material-speedometer: **Slow Processing**
---
Long processing times
**Solution**: Enable parallel processing and GPU acceleration.
Long runtimes on large datasets
Enable parallel processing (`PipelineBuilder` workers) and GPU acceleration.
</div>
@@ -215,80 +106,46 @@ The typical workflow involves these steps:
## Performance Optimization
### 1. Batch Processing
### Batch Processing
Process multiple documents together for better throughput. Use batch processing when working with large document collections.
Process documents in batches rather than one at a time. Configure chunk sizes based on available RAM.
**For examples, see:**
- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Batch ingestion patterns
- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration
### Parallel Execution
### 2. Parallel Execution
`PipelineBuilder` supports configurable worker counts per stage for independent operations.
Use parallel processing for independent operations to improve performance on multi-core systems.
### Backend Selection
### 3. Backend Selection
| Operation | NetworkX | Neo4j / FalkorDB |
|-----------|----------|------------------|
| Graph construction | Fast | Moderate |
| Query performance | Moderate | Fast |
| Scalability | Low (in-memory) | High (persistent) |
| Operation | NetworkX | Neo4j |
| :--- | :--- | :--- |
| **Graph Construction** | ⚡⚡⚡ | ⚡⚡ |
| **Query Performance** | ⚡⚡ | ⚡⚡⚡ |
| **Scalability** | Low | High |
Use NetworkX for development and smaller graphs; switch to a persistent backend for production at scale.
---
## Security Best Practices
### API Key Management
**API keys**
- Store in environment variables or a secrets manager
- Never hardcode keys or commit them to version control
- Rotate keys regularly
- **DO**: Use environment variables, rotate keys regularly.
- **DON'T**: Hardcode keys, commit to version control.
### Data Privacy
- **DO**: Encrypt sensitive data, use local models.
- **DON'T**: Send PII to external APIs without protection.
---
## FAQ
**Q: What is Semantica?**
A: A framework for building knowledge graphs and semantic applications.
**Q: Is Semantica free?**
A: Yes, it is open source. Some features (e.g., OpenAI) require paid APIs.
**Q: Can I use Semantica in production?**
A: Yes, it is designed for production with proper configuration.
**Data privacy**
- Use local embedding models for sensitive data
- Avoid sending PII to external APIs without appropriate data handling agreements
- Encrypt sensitive graph exports at rest
---
## Next Steps
Continue your learning journey:
- **[Cookbook](cookbook.md)** - Interactive Jupyter notebook tutorials
- **[API Reference](reference/core.md)** - Complete API documentation
- **[Use Cases](use-cases.md)** - Real-world applications
- **[Examples](examples.md)** - Code examples and patterns
### 🍳 Recommended Next Cookbooks
- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production GraphRAG system
- **Topics**: GraphRAG, hybrid retrieval, LLM integration
- **Difficulty**: Advanced
- **Time**: 1-2 hours
- **Use Cases**: Production RAG applications
- **[RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)**: Understand the differences
- **Topics**: RAG comparison, reasoning gap, inference engines
- **Difficulty**: Intermediate
- **Time**: 45-60 minutes
- **Use Cases**: Choosing the right approach
---
!!! info "Contribute"
Have questions? [Open an issue](https://github.com/Hawksight-AI/semantica/issues) or [start a discussion](https://github.com/Hawksight-AI/semantica/discussions)!
- [Cookbook](cookbook.md) — interactive Jupyter notebook tutorials
- [API Reference](reference/core.md) — complete technical documentation
- [Use Cases](use-cases.md) — real-world domain examples
- [FAQ](faq.md) — common questions
!!! info "Questions or feedback?"
[Open an issue](https://github.com/Hawksight-AI/semantica/issues) or [start a discussion](https://github.com/Hawksight-AI/semantica/discussions).
docs/modules.md
+18 -93
View File
@@ -1,9 +1,9 @@
# Modules & Architecture
# Modules
**Complete guide to Semantica's modular architecture and how to use each component.**
Every Semantica module works independently — use only what you need.
!!! tip "Modular Design"
Each Semantica module works independently. Use only what you need for your specific use case.
!!! tip "Just need a quick reference?"
Jump to the [Module Index](#module-index) at the bottom of this page.
---
@@ -82,13 +82,11 @@ web_ingestor = WebIngestor()
pages = web_ingestor.ingest_urls(["https://example.com"])
```
**What it does:**
- **File formats** - PDF, DOCX, TXT, JSON, CSV
- **Web scraping** - Extract content from websites
- **Database** - Connect to SQL and NoSQL databases
- **Batch processing** - Handle large datasets efficiently
**Use Cases:**
- Document processing pipelines
- Web data extraction
- Database integration
@@ -106,13 +104,11 @@ text = parsed["full_text"]
metadata = parsed["metadata"]
```
**What it does:**
- **Text extraction** - Extract clean text from documents
- **Metadata parsing** - Extract titles, authors, dates
- **Structure analysis** - Identify sections, headings
- **OCR support** - Handle scanned documents
**Use Cases:**
- PDF processing
- Document analysis
- Content extraction
@@ -130,13 +126,11 @@ splitter = TextSplitter(method="semantic")
chunks = splitter.split(text, chunk_size=1000, overlap=200)
```
**What it does:**
- **Intelligent chunking** - Split text while preserving context
- **Semantic splitting** - Break at natural boundaries
- **Size control** - Manage chunk sizes for processing
- **Overlap handling** - Maintain context between chunks
**Use Cases:**
- Document preprocessing
- Embedding preparation
- RAG systems
@@ -155,13 +149,11 @@ clean_text = normalizer.normalize_text(text)
standardized_date = normalizer.normalize_date("Jan 1st, 2020")
```
**What it does:**
- **Text cleaning** - Remove noise and artifacts
- **Date standardization** - Convert to ISO format
- **Name normalization** - Standardize person names
- **Entity normalization** - Clean up company names
**Use Cases:**
- Data preprocessing
- Quality improvement
- Standardization
@@ -186,13 +178,11 @@ rel_extractor = RelationExtractor()
relationships = rel_extractor.extract(text, entities)
```
**What it does:**
- **Named Entity Recognition** - Find people, orgs, locations
- **Relationship extraction** - Find connections between entities
- **Custom entities** - Define your own entity types
- **Confidence scoring** - Quality assessment for extractions
**Use Cases:**
- Knowledge graph construction
- Document analysis
- Information extraction
@@ -215,13 +205,11 @@ analyzer = GraphAnalyzer()
stats = analyzer.analyze(kg)
```
**What it does:**
- **Graph construction** - Build knowledge graphs from data
- **Graph analysis** - Calculate metrics and statistics
- **Graph querying** - Search and retrieve information
- **Graph manipulation** - Merge, split, transform graphs
**Use Cases:**
- Knowledge base creation
- Graph analytics
- Information retrieval
@@ -244,13 +232,11 @@ ontology.add_relationship("works_for", "Person", "Organization")
is_valid = ontology.validate_graph(kg)
```
**What it does:**
- **Schema definition** - Define data structure
- **Data validation** - Ensure data conforms to schema
- **Inheritance** - Create hierarchical relationships
- **Constraints** - Enforce data quality rules
**Use Cases:**
- Data modeling
- Quality assurance
- Schema management
@@ -268,13 +254,11 @@ engine = ReasoningEngine()
inferences = engine.infer(kg, rules=["transitivity", "symmetry"])
```
**What it does:**
- **Logical inference** - Derive new facts from existing ones
- **Pattern matching** - Find complex patterns in data
- **Consistency checking** - Detect contradictions
- **Decision support** - Automated reasoning
**Use Cases:**
- Knowledge discovery
- Decision making
- Consistency checking
@@ -295,13 +279,11 @@ embeddings = generator.generate(["text1", "text2"])
similarity = generator.similarity(embeddings[0], embeddings[1])
```
**What it does:**
- **Text embeddings** - Convert text to vectors
- **Similarity search** - Find similar content
- **Clustering** - Group related items
- **AI integration** - Provide context to LLMs
**Use Cases:**
- Semantic search
- Recommendation systems
- Clustering
@@ -320,13 +302,11 @@ store.add_vectors(embeddings, ids)
results = store.search(query_vector, top_k=10)
```
**What it does:**
- **Vector storage** - Efficient vector database
- **Fast search** - Approximate nearest neighbor search
- **Indexing** - Optimize for performance
- **Batch operations** - Handle large datasets
**Use Cases:**
- Semantic search
- RAG systems
- Recommendation engines
@@ -346,13 +326,11 @@ store.add_edges(relationships)
results = store.query("MATCH (n)-[r]->(m) RETURN n, r, m")
```
**What it does:**
- **Graph persistence** - Store graphs in databases
- **Graph queries** - Cypher and Gremlin support
- **Graph algorithms** - Path finding, centrality
- **Transactions** - ACID compliance
**Use Cases:**
- Knowledge graph storage
- Graph analytics
- Network analysis
@@ -371,13 +349,11 @@ store.add_triplets(subject, predicate, object)
triplets = store.get_triplets(entity="Apple Inc.")
```
**What it does:**
- **Triple storage** - Store (subject, predicate, object) triples
- **Pattern matching** - Find specific patterns
- **RDF support** - Semantic web standards
- **Bulk operations** - Efficient batch processing
**Use Cases:**
- Semantic web
- Knowledge representation
- Linked data
@@ -397,13 +373,11 @@ resolver = EntityResolver()
merged_entities = resolver.resolve(entities, strategy="semantic")
```
**What it does:**
- **Duplicate detection** - Find similar entities
- **Entity resolution** - Merge duplicate records
- **Similarity scoring** - Quality assessment
- **Record linkage** - Connect related records
**Use Cases:**
- Data cleaning
- Master data management
- Record linkage
@@ -422,13 +396,11 @@ conflicts = detector.detect_conflicts(kg)
resolved = detector.resolve(conflicts, strategy="most_recent")
```
**What it does:**
- **Conflict detection** - Find contradictory information
- **Resolution strategies** - Automated conflict resolution
- **Source reliability** - Trustworthiness assessment
- **Temporal analysis** - Time-based conflict handling
**Use Cases:**
- Data quality
- Consistency checking
- Trust management
@@ -448,13 +420,11 @@ manager = ContextManager()
context = manager.get_context(query, history)
```
**What it does:**
- **Context tracking** - Maintain conversation context
- **Memory management** - Store and retrieve context
- **Relevance scoring** - Find relevant context
- **Session management** - Handle multiple conversations
**Use Cases:**
- AI agents
- Chatbots
- Conversational AI
@@ -472,13 +442,11 @@ seed = SeedData()
knowledge = seed.get_knowledge("technology", "companies")
```
**What it does:**
- **Seed knowledge** - Foundation data for domains
- **Knowledge bases** - Pre-built domain knowledge
- **Quick start** - Bootstrap applications
- **Domain models** - Industry-specific data
**Use Cases:**
- Domain bootstrapping
- Quick start data
- Industry knowledge
@@ -496,13 +464,11 @@ provider = LLMProvider(model="gpt-4")
response = provider.generate(prompt, context=kg)
```
**What it does:**
- **LLM integration** - Connect to various LLM providers
- **Prompt engineering** - Optimize prompts for results
- **Context injection** - Provide knowledge graph context
- **Response parsing** - Extract structured outputs
**Use Cases:**
- AI generation
- Question answering
- Text completion
@@ -522,13 +488,11 @@ exporter = GraphExporter()
exporter.export(kg, format="json", filename="output.json")
```
**What it does:**
- **Multiple formats** - JSON, CSV, RDF, GraphML
- **Database export** - Export to various databases
- **Streaming** - Handle large datasets
- **Filtering** - Export specific data subsets
**Use Cases:**
- Data sharing
- System integration
- Backup and restore
@@ -546,13 +510,11 @@ visualizer = GraphVisualizer()
visualizer.plot(kg, layout="force_directed")
```
**What it does:**
- **Graph visualization** - Interactive graph plots
- **Custom styling** - Tailored visual appearance
- **Analytics charts** - Statistics and metrics
- **Exploration tools** - Interactive data exploration
**Use Cases:**
- Data exploration
- Presentation
- Analysis
@@ -573,13 +535,11 @@ pipeline.add_step("build", GraphBuilder())
result = pipeline.run("data/")
```
**What it does:**
- **Workflow orchestration** - Coordinate multiple steps
- **Parallel processing** - Run steps concurrently
- **Progress tracking** - Monitor pipeline execution
- **Error handling** - Robust error management
**Use Cases:**
- Data processing
- Workflow automation
- Batch processing
@@ -587,7 +547,7 @@ result = pipeline.run("data/")
---
## New Features & Modules
## Additional Modules
### Change Management Module
**Version control and audit trails**
@@ -599,13 +559,11 @@ manager = TemporalVersionManager(storage_path="versions.db")
snapshot = manager.create_snapshot(kg, "v1.0", "user@example.com", "Initial version")
```
**What it does:**
- **Version control** - Track changes over time
- **Audit trails** - Complete change history
- **Data integrity** - SHA-256 checksums
- **Change comparison** - Detailed diff analysis
**Use Cases:**
- Knowledge graph versioning
- Compliance tracking
- Data governance
@@ -623,13 +581,11 @@ manager = ProvenanceManager()
manager.track_entity("entity_1", "document.pdf", "person")
```
**What it does:**
- **W3C PROV-O compliant** - Industry standard tracking
- **Complete lineage** - End-to-end traceability
- **Source attribution** - Track data origins
- **Integrity verification** - Tamper detection
**Use Cases:**
- Regulatory compliance
- Data provenance
- Audit trails
@@ -648,13 +604,11 @@ semantica = Semantica(config=Config())
result = semantica.process("data/")
```
**What it does:**
- **Framework orchestration** - Central coordination
- **Configuration management** - Settings and preferences
- **Lifecycle management** - Start/stop/restart
- **Plugin system** - Extensible architecture
**Use Cases:**
- Framework initialization
- Configuration management
- Plugin development
@@ -662,46 +616,18 @@ result = semantica.process("data/")
---
## Getting Started
## Common Module Chains
### Quick Start Example
```python
# Complete pipeline example
from semantica.ingest import FileIngestor
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
from semantica.pipeline import Pipeline
# Create pipeline
pipeline = Pipeline()
pipeline.add_step("ingest", FileIngestor())
pipeline.add_step("ner", NERExtractor())
pipeline.add_step("relations", RelationExtractor())
pipeline.add_step("build", GraphBuilder())
# Run pipeline
kg = pipeline.run("documents/")
print(f"Built graph with {len(kg['entities'])} entities")
```
### Choose Your Modules
**For Document Processing:**
- Ingest → Parse → Split → Semantic Extract → Knowledge Graph
**For Web Scraping:**
- Ingest (Web) → Normalize → Semantic Extract → Graph Store
**For AI Agents:**
- Context → LLM Providers → Reasoning → Export
**For Analytics:**
- Knowledge Graph → Graph Store → Visualization → Export
| Goal | Modules |
|------|---------|
| Document processing | Ingest → Parse → Split → Semantic Extract → KG |
| Web scraping | Ingest (Web) → Normalize → Semantic Extract → Graph Store |
| AI agents | Context → LLM Providers → Reasoning → Export |
| Analytics | KG → Graph Store → Visualization → Export |
---
## Module Reference
## Module Index
| Module | Purpose | Key Classes | Use Cases |
|--------|---------|-------------|-----------|
@@ -731,10 +657,9 @@ print(f"Built graph with {len(kg['entities'])} entities")
---
## Need Help?
## More
- **Documentation**: [Getting Started](getting-started.md)
- **Examples**: [Cookbook](cookbook.md)
- **Community**: [Discord](community.md)
- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)
- **Support**: [Contact Us](community.md)
- [Getting Started](getting-started.md)
- [Examples](examples.md)
- [Cookbook](cookbook.md)
- [API Reference](reference/core.md)
docs/quickstart.md
+96 -192
View File
@@ -1,246 +1,150 @@
# Quickstart
Get started with Semantica in 5 minutes. This guide will walk you through building your first knowledge graph.
Build your first knowledge graph in 5 minutes.
!!! tip "Before You Start"
Make sure you have Semantica installed. If not, follow the [Installation Guide](installation.md) first. This quickstart assumes basic Python knowledge.
!!! tip "Prerequisites"
Semantica installed (`pip install semantica`). If not, see the [Installation Guide](installation.md).
## Overview
---
## Pipeline Overview
```mermaid
flowchart LR
A[Install] --> B[Initialize]
B --> C[Load Data]
C --> D[Extract]
D --> E[Build Graph]
E --> F[Visualize]
style A fill:#e3f2fd
style F fill:#c8e6c9
A[Ingest] --> B[Parse]
B --> C[Extract]
C --> D[Build Graph]
D --> E[Visualize / Export]
```
## Step 1: Installation
---
If you haven't installed Semantica yet:
## Step 1 — Ingest
```bash
pip install semantica
```
Load documents from files, directories, or the web.
See the [Installation Guide](installation.md) for detailed instructions.
!!! note "Installation Options"
For production use, consider installing with optional dependencies for better performance: `pip install semantica[all]`. See the [Installation Guide](installation.md) for all options.
## Step 2: Your First Knowledge Graph
Building a knowledge graph involves these key steps:
1. **Ingest** your documents using `FileIngestor`
2. **Parse** documents to extract text using `DocumentParser` or `DoclingParser` (for enhanced layout support)
3. **Extract** entities and relationships using `NERExtractor` and `RelationExtractor`
4. **Build** the graph using `GraphBuilder`
5. **Generate** embeddings (optional) using `TextEmbedder`
**Quick Example:**
```python
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser, DoclingParser
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
# 1. Ingest document
ingestor = FileIngestor()
sources = ingestor.ingest("data/sample.pdf")
# 2. Parse (choose your parser)
# Option A: Standard parser
parser = DocumentParser()
parsed_content = parser.parse(sources[0])
# Option B: Enhanced Docling parser (recommended for complex tables)
# docling_parser = DoclingParser()
# parsed_content = docling_parser.parse(sources[0])
# 3. Extract entities and relations
ner = NERExtractor()
entities = ner.extract(parsed_content)
relations = RelationExtractor()
relationships = relations.extract(parsed_content, entities=entities)
# 4. Build graph
builder = GraphBuilder()
graph = builder.build(entities=entities, relationships=relationships)
print(f"Built knowledge graph with {len(graph.nodes)} nodes and {len(graph.edges)} edges")
```
**For complete step-by-step examples with detailed explanations, see:**
- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Full tutorial with detailed explanations and expected outputs
- **Topics**: Entity extraction, relationship extraction, graph construction, visualization
- **Difficulty**: Beginner
- **Time**: 20-30 minutes
- **Use Cases**: Learning the basics, quick start
Supported formats: PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives. For web content, use `WebIngestor`.
## Step 3: Extract Entities and Relationships
---
The semantic extraction step identifies named entities (people, organizations, locations) and relationships between them from your text.
## Step 2 — Parse
**What gets extracted:**
- **Entities**: People, organizations, locations, dates, and other named entities
- **Relationships**: Connections between entities (e.g., `founded_by`, `located_in`, `has_ceo`)
Extract structured text from raw documents.
**For detailed examples and different extraction methods, see:**
- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn different NER methods and configurations
- **Topics**: Named entity recognition, entity types, confidence scores
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Understanding entity extraction options
```python
from semantica.parse import DocumentParser
- **[Relation Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)**: Learn to extract relationships between entities
- **Topics**: Relationship extraction, dependency parsing, semantic role labeling
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Building rich knowledge graphs with relationships
parser = DocumentParser()
parsed = parser.parse(sources[0])
```
## Step 4: Build Knowledge Graph from Multiple Sources
For complex layouts (tables, columns): use `DoclingParser` instead — it handles PDF tables and structured DOCX/PPTX better.
You can combine data from multiple sources (files, web, databases) to build a unified knowledge graph. The process involves:
---
1. **Ingest** from multiple sources using different ingestors
2. **Parse** all documents to extract text
3. **Extract** entities and relationships from each source
4. **Build** a unified graph with entity merging enabled
## Step 3 — Extract Entities and Relationships
**For complete examples with multiple sources, see:**
- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from files, web, feeds, streams, and databases
- **Topics**: File, web, feed, stream, database ingestion
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Loading data from various sources
```python
from semantica.semantic_extract import NERExtractor, RelationExtractor
- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced patterns for integrating multiple data sources
- **Topics**: Multi-source integration, entity resolution, conflict handling
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Building knowledge graphs from diverse data sources
ner = NERExtractor()
entities = ner.extract(parsed)
## Step 5: Visualize Your Knowledge Graph
rel = RelationExtractor()
relationships = rel.extract(parsed, entities=entities)
```
Visualization helps you understand and explore your knowledge graph structure. Semantica supports multiple visualization formats including interactive HTML, static images, and export formats.
Each entity gets a type, confidence score, and source reference. Relationships are extracted as typed triplets: `(subject, predicate, object)`.
**For detailed visualization examples, see:**
- **[Visualization Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/16_Visualization.ipynb)**: Learn to create interactive and static visualizations
- **Topics**: Network graphs, interactive HTML, static images, export formats
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Exploring graph structure, presentations, analysis
---
- **[Complete Visualization Suite Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)**: Advanced visualization techniques
- **Topics**: Custom layouts, filtering, styling, multiple graph types
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Production visualizations, custom dashboards
## Step 4 — Build the Knowledge Graph
## Step 6: Export Your Knowledge Graph
```python
from semantica.kg import GraphBuilder
Export your knowledge graph to various formats for integration with other systems or tools. Semantica supports RDF, JSON, CSV, OWL, GraphML, and more.
builder = GraphBuilder(merge_entities=True)
graph = builder.build(entities=entities, relationships=relationships)
**Supported export formats:**
- **RDF**: Turtle, RDF/XML, JSON-LD, N-Triples
- **JSON**: Standard JSON, JSON-LD, Cytoscape.js format
- **CSV**: Node and edge lists for spreadsheet tools
- **OWL**: OWL/XML and Turtle for ontologies
- **Graph Formats**: GraphML, GEXF, DOT for visualization tools
print(f"{len(graph.nodes)} nodes, {len(graph.edges)} edges")
```
**For detailed export examples, see:**
- **[Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/15_Export.ipynb)**: Learn to export to all supported formats
- **Topics**: RDF, JSON, CSV, OWL, GraphML export
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Data integration, sharing knowledge graphs
`merge_entities=True` resolves duplicates across sources automatically.
- **[Multi-Format Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)**: Advanced export patterns
- **Topics**: Batch export, custom formats, format conversion
- **Difficulty**: Intermediate
- **Time**: 30-45 minutes
- **Use Cases**: Production exports, format migration
---
## Step 5 — Visualize
```python
from semantica.visualization import GraphVisualizer
viz = GraphVisualizer()
viz.visualize(graph, output="graph.html") # interactive HTML
```
---
## Step 6 — Export
```python
from semantica.export import RDFExporter
exporter = RDFExporter()
rdf = exporter.export_to_rdf(graph, format="turtle")
```
Other formats: `"json-ld"`, `"nt"`, `"xml"`, Parquet, ArangoDB AQL. See [Export Reference](reference/export.md).
---
## Common Patterns
### Pattern 1: Process Text Directly
### Process text directly (no file)
You can process text directly without file ingestion. This is useful when you already have text content in memory.
```python
from semantica.semantic_extract import NERExtractor
**For examples, see:**
- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Processing text directly
- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Graph construction from text
ner = NERExtractor()
entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
```
### Pattern 2: Custom Entity Extraction
### Incremental build from multiple sources
Configure entity extraction with different methods (ML models, LLMs) and parameters for your specific needs.
```python
from semantica.kg import GraphBuilder
**For examples, see:**
- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Different extraction methods and configurations
- **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns
all_entities, all_rels = [], []
for doc in parsed_docs:
all_entities.extend(ner.extract(doc))
all_rels.extend(rel.extract(doc, entities=all_entities))
### Pattern 3: Incremental Building
graph = GraphBuilder(merge_entities=True).build(
entities=all_entities, relationships=all_rels
)
```
Build knowledge graphs incrementally from multiple sources and merge them together.
**For examples, see:**
- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Graph construction and merging
- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration patterns
## Next Steps
Now that you've built your first knowledge graph:
1. **[Explore Examples](examples.md)** - See more advanced use cases
2. **[API Reference](reference/core.md)** - Learn about all available methods
3. **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks
4. **[Full Documentation](https://github.com/Hawksight-AI/semantica/blob/main/README.md)** - Comprehensive guide
### 🍳 Recommended Cookbook Tutorials
Continue learning with these interactive tutorials:
- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules
- **Topics**: Framework overview, all modules, architecture, configuration
- **Difficulty**: Beginner
- **Time**: 30-45 minutes
- **Use Cases**: Understanding the complete framework
- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph
- **Topics**: Entity extraction, relationship extraction, graph construction, visualization
- **Difficulty**: Beginner
- **Time**: 20-30 minutes
- **Use Cases**: Hands-on practice with complete workflow
- **[Data Ingestion](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources
- **Topics**: File, web, feed, stream, database ingestion
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Loading data from various sources
- **[Document Parsing](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)**: Parse various document formats
- **Topics**: PDF, DOCX, HTML, JSON parsing
- **Difficulty**: Beginner
- **Time**: 15-20 minutes
- **Use Cases**: Extracting text from different file formats
---
## Troubleshooting
### Common Issues
| Problem | Fix |
|---------|-----|
| No entities extracted | Check the document has machine-readable text (not just scanned images) |
| Slow processing | Process in chunks; use GPU acceleration (`pip install semantica[gpu]`) |
| Memory errors | Reduce batch size or switch to a persistent graph backend |
**Issue**: No entities extracted
- **Solution**: Check that your document contains text content. PDFs with images only won't work without OCR.
---
**Issue**: Slow processing
- **Solution**: For large documents, consider processing in chunks or using GPU acceleration.
## Next Steps
**Issue**: Memory errors
- **Solution**: Process documents one at a time or reduce batch sizes.
Need help? Check the [Installation Troubleshooting](installation.md#troubleshooting) or [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues).
- [Core Concepts](concepts.md) — understand how knowledge graphs and reasoning work
- [Modules Guide](modules.md) — every module explained
- [Use Cases](use-cases.md) — domain-specific examples
- [Cookbook](cookbook.md) — interactive Jupyter notebooks for each step
docs/reference/change_management.md
+19 -25
View File
@@ -58,16 +58,14 @@ The Semantica change management module provides enterprise-grade version control
</div>
### Key Features
- ✅ **Enterprise Version Control** — Complete snapshot management with SHA-256 integrity verification
- ✅ **Dual Storage Backends** — InMemory (development) and SQLite (production) with ACID guarantees
- ✅ **Knowledge Graph Versioning** — Entity and relationship-level change tracking with detailed diffs
- ✅ **Ontology Versioning** — Structural change tracking for classes, properties, and axioms
- ✅ **Audit Trail Compliance** — Complete change logs with author attribution and timestamps
- ✅ **Data Integrity** — SHA-256 checksums for tamper detection and verification
- ✅ **Change Comparison** — Detailed diff algorithms for entities, relationships, and ontology structures
- ✅ **Backward Compatibility** — Legacy support for existing ontology version management
- **Enterprise Version Control** — Complete snapshot management with SHA-256 integrity verification
- **Dual Storage Backends** — InMemory (development) and SQLite (production) with ACID guarantees
- **Knowledge Graph Versioning** — Entity and relationship-level change tracking with detailed diffs
- **Ontology Versioning** — Structural change tracking for classes, properties, and axioms
- **Audit Trail Compliance** — Complete change logs with author attribution and timestamps
- **Data Integrity** — SHA-256 checksums for tamper detection and verification
- **Change Comparison** — Detailed diff algorithms for entities, relationships, and ontology structures
- **Backward Compatibility** — Legacy support for existing ontology version management
---
@@ -121,24 +119,21 @@ class ChangeLogEntry:
### Storage Backends
#### InMemoryVersionStorage
Fast, volatile storage for development and testing.
**InMemoryVersionStorage** — Fast, volatile storage for development and testing.
```python
from semantica.change_management import InMemoryVersionStorage
storage = InMemoryVersionStorage()
```
#### SQLiteVersionStorage
Persistent storage with ACID guarantees for production.
**SQLiteVersionStorage** — Persistent storage with ACID guarantees for production.
```python
from semantica.change_management import SQLiteVersionStorage
storage = SQLiteVersionStorage("versions.db")
```
#### VersionStorage (Abstract)
Base interface for custom storage implementations.
**VersionStorage (Abstract)** — Base interface for custom storage implementations.
**Core Methods:**
- `save(snapshot)` - Store version snapshot
@@ -251,7 +246,7 @@ For large-scale knowledge graphs, reprocessing the entire dataset on every updat
Semantica supports **Delta-Aware Pipelines**, allowing you to compute the exact differences (added and removed triples)
between the two graph snapshots and run validation, enrichment, or export jobs *only* on the changes.
### Delta Pipeline Example
**Delta Pipeline Example**
```python
from semantica.change_management import TemporalVersionManager
@@ -286,6 +281,7 @@ result = engine.execute_pipeline(
triplet_store=triplet_store
)
```
---
## Data Integrity
@@ -399,7 +395,7 @@ for version in prod_manager.list_versions():
Semantica allows you to treat ontology schema changes with the same rigor as database migrations. By comparing two versions, you can generate a machine-readable diff and a structured impact report to catch breaking changes before they reach production.
### Comparing Versions
**Comparing Versions**
The `OntologyEngine` provides a high-level API to orchestrate the comparison of two schema versions.
@@ -410,21 +406,18 @@ engine = OntologyEngine()
# Generate a migration impact report between v1.0 and v2.0
report = engine.compare_versions(
base_id="v1.0",
base_id="v1.0",
target_id="v2.0"
)
print(f"Total changes detected: {report['summary']['total_changes']}")
```
---
### Understanding the Report Format
**Report Format**
The `compare_versions` method returns a comprehensive dictionary containing both a machine-readable diff and a human-readable impact analysis.
Here is the exact structure of the returned report:
The `compare_versions` method returns a dictionary with a machine-readable diff and a human-readable impact analysis:
```json
{
@@ -467,3 +460,4 @@ Here is the exact structure of the returned report:
"warnings": []
}
}
```
docs/use-cases.md
+70 -155
View File
@@ -1,210 +1,125 @@
# Use Cases
Semantica is designed to solve complex data challenges across various domains. This guide explores common use cases and how to implement them.
!!! info "About This Guide"
This guide provides detailed implementation guides for real-world use cases, complete with code examples, prerequisites, and step-by-step instructions.
Real-world applications of Semantica across domains, with linked cookbook notebooks for each.
---
## Use Case Comparison
## Overview
| Use Case | Difficulty | Time | Domain | Key Features | Cookbook |
| :-------------------------------- | :------------ | :---------- | :---------- | :---------------------------------------------- | :------------------------------------------ |
| **Biomedical Knowledge Graphs** | Intermediate | 1-2 hours | Healthcare | Gene-protein-disease relationships | Drug Discovery, Genomic Variant Analysis |
| **Financial Data Integration** | Intermediate | 1-2 hours | Finance | MCP integration, real-time data | Financial Data Integration MCP |
| **Fraud Detection** | Advanced | 2-3 hours | Finance | Temporal graphs, pattern detection | Fraud Detection |
| **Blockchain Analytics** | Intermediate | 1-2 hours | Finance | Transaction tracing, DeFi intelligence | DeFi Protocol Intelligence, Transaction Network |
| **Cybersecurity Threat Intelligence**| Advanced | 2-3 hours | Security | Threat mapping, anomaly detection | Real-Time Anomaly Detection, Threat Intelligence |
| **Intelligence Analysis** | Intermediate | 1-2 hours | Security | Criminal networks, OSINT analysis | Criminal Network Analysis, Intelligence Orchestrator |
| **Supply Chain Optimization** | Intermediate | 1-2 hours | Industry | Data integration, route optimization | Supply Chain Data Integration |
| **Renewable Energy Management** | Intermediate | 1-2 hours | Energy | Energy market analysis, optimization | Energy Market Analysis |
| **GraphRAG** | Advanced | 1-2 hours | AI | Enhanced RAG with knowledge graphs | GraphRAG Complete, RAG vs GraphRAG |
| Use Case | Domain | Difficulty | Estimated Time |
|----------|--------|------------|----------------|
| Biomedical Knowledge Graphs | Healthcare | Intermediate | 1–2 hours |
| Financial Data Integration | Finance | Intermediate | 1–2 hours |
| Fraud Detection | Finance | Advanced | 2–3 hours |
| Blockchain Analytics | Finance | Intermediate | 1–2 hours |
| Cybersecurity Threat Intelligence | Security | Advanced | 2–3 hours |
| Criminal Network Analysis | Security / Intelligence | Intermediate | 1–2 hours |
| Intelligence Analysis Orchestrator | Intelligence | Intermediate | 1–2 hours |
| Supply Chain Optimization | Operations | Intermediate | 1–2 hours |
| Renewable Energy Management | Energy | Intermediate | 1–2 hours |
| GraphRAG | AI | Advanced | 1–2 hours |
**Difficulty Levels**:
- **Beginner**: Basic Semantica knowledge required
- **Intermediate**: Some domain knowledge helpful
- **Advanced**: Requires domain expertise and advanced Semantica features
**Difficulty levels**:
- **Beginner** — basic Semantica knowledge only
- **Intermediate** — some domain knowledge helpful
- **Advanced** — domain expertise + advanced Semantica features
---
## Research & Science
<div class="grid cards" markdown>
### Biomedical Knowledge Graphs
- :material-dna: **Biomedical Knowledge Graphs**
---
Accelerate drug discovery and understand disease pathways by connecting genes, proteins, drugs, and diseases.
**Goal**: Connect genes, proteins, drugs, and diseases from scientific literature and databases.
**Difficulty**: Intermediate
[:material-arrow-right: Drug Discovery Pipeline](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)
[:material-arrow-right: Genomic Variant Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb)
Connect genes, proteins, drugs, and diseases from scientific literature and databases to accelerate drug discovery and understand disease pathways.
</div>
### Biomedical Knowledge Graphs Implementation
**Prerequisites**:
- Domain knowledge of biomedical concepts
- Access to biomedical literature/databases
**Implementation Guides:**
- **[Drug Discovery Pipeline Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)**: Build knowledge graphs from PubMed RSS feeds
- **Topics**: PubMed RSS ingestion, entity-aware chunking, GraphRAG, vector similarity search
- **Difficulty**: Intermediate
- **Time**: 1-2 hours
- **Use Cases**: Drug discovery, biomedical research
- **[Genomic Variant Analysis Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb)**: Analyze genomic variants using temporal knowledge graphs
- **Topics**: bioRxiv RSS, temporal KGs, deduplication, pathway analysis
- **Difficulty**: Intermediate
- **Time**: 1-2 hours
- **Use Cases**: Genomic research, variant analysis
**Cookbooks**:
- [Drug Discovery Pipeline](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) — PubMed RSS ingestion, entity-aware chunking, GraphRAG, vector similarity search
- [Genomic Variant Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) — bioRxiv RSS, temporal KGs, deduplication, pathway analysis
---
## Finance & Trading
<div class="grid cards" markdown>
### Financial Data Integration
- :material-finance: **Financial Data Integration**
---
Integrate financial data from multiple sources using MCP servers and real-time ingestion.
**Goal**: Connect Alpha Vantage API, MCP servers, seed data, and real-time ingestion for comprehensive financial analysis.
[:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb)
Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph.
- :material-shield-alert: **Fraud Detection**
---
Detect complex fraud rings using temporal knowledge graphs and pattern detection.
**Goal**: Build a graph of Users, Devices, IP Addresses, and Transactions to find cycles and detect fraud patterns.
[:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb)
**Cookbook**: [Financial Data Integration (MCP)](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) — Alpha Vantage API, MCP servers, seed data, real-time ingestion
- :material-bitcoin: **Blockchain Analytics**
---
Analyze DeFi protocols and transaction networks for intelligence and fraud detection.
**Goal**: Map transaction flows between wallets and exchanges, analyze DeFi protocols, and detect illicit activity.
[:material-arrow-right: DeFi Protocol Intelligence](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb)
[:material-arrow-right: Transaction Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb)
### Fraud Detection
</div>
Detect complex fraud rings using temporal graphs and pattern detection over transaction, device, and user data.
---
**Cookbook**: [Fraud Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) — temporal KGs, cycle detection, fraud pattern analysis
### Blockchain Analytics
Map transaction flows, analyze DeFi protocols, and detect illicit activity across wallet and exchange networks.
**Cookbooks**:
- [DeFi Protocol Intelligence](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb)
- [Transaction Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb)
---
## Security & Intelligence
<div class="grid cards" markdown>
### Cybersecurity Threat Intelligence
- :material-shield-lock: **Cybersecurity Threat Intelligence**
---
Proactively identify and mitigate cyber threats using real-time anomaly detection and threat intelligence.
**Goal**: Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs.
[:material-arrow-right: Real-Time Anomaly Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)
[:material-arrow-right: Threat Intelligence Hybrid RAG](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb)
Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs for proactive defense.
- :material-account-network: **Criminal Network Analysis**
---
Analyze criminal networks to identify key players, communities, and suspicious patterns using OSINT RSS feeds, deduplication, and network centrality analysis.
**Goal**: Build knowledge graphs from police reports, court records, and surveillance data.
[:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb)
**Cookbooks**:
- [Real-Time Anomaly Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)
- [Threat Intelligence Hybrid RAG](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb)
- :material-file-search: **Intelligence Analysis Orchestrator Worker**
---
Comprehensive intelligence analysis using pipeline orchestrator with multiple RSS feeds, conflict detection, and multi-source integration.
**Goal**: Process multiple intelligence sources in parallel using orchestrator-worker pattern.
[:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb)
### Criminal Network Analysis
</div>
Build knowledge graphs from police reports, court records, and OSINT feeds to identify key players, communities, and suspicious patterns using network centrality analysis.
**Cookbook**: [Criminal Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb)
### Intelligence Analysis Orchestrator
Process multiple intelligence sources in parallel using an orchestrator-worker pipeline pattern with multi-source conflict detection and integration.
**Cookbook**: [Intelligence Analysis Orchestrator-Worker](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb)
---
## Industry & Operations
<div class="grid cards" markdown>
### Supply Chain Optimization
- :material-truck-delivery: **Supply Chain Optimization**
---
Visualize and optimize complex global supply chains.
**Goal**: Map suppliers, logistics routes, and inventory levels to identify bottlenecks.
[:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb)
Map suppliers, logistics routes, and inventory levels to identify bottlenecks and optimize global supply chains.
- :material-wind-turbine: **Renewable Energy Management**
---
Optimize grid operations and asset maintenance.
**Goal**: Connect sensor data, weather forecasts, and maintenance logs to predict failures.
[:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb)
**Cookbook**: [Supply Chain Data Integration](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb)
</div>
### Renewable Energy Management
Connect sensor data, weather forecasts, and maintenance logs to predict equipment failures and optimize grid operations.
**Cookbook**: [Energy Market Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb)
---
## Advanced AI Patterns
<div class="grid cards" markdown>
### GraphRAG (Graph-Augmented Generation)
- :material-robot: **Graph-Augmented Generation (GraphRAG)**
---
Enhance LLM responses with structured ground truth using knowledge graphs.
**Goal**: Use the knowledge graph to retrieve precise context for RAG applications with hybrid retrieval and logical inference.
[:material-arrow-right: GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)
[:material-scale-balance: RAG vs GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
Use knowledge graphs to retrieve precise, structured context for LLM responses — with hybrid retrieval, logical inference, and source attribution.
</div>
---
---
## Summary
This guide covered use cases across multiple domains with corresponding cookbooks:
- **Research & Science**: Biomedical knowledge graphs (Drug Discovery, Genomic Variant Analysis)
- **Finance & Trading**: Financial data integration, fraud detection, blockchain analytics
- **Security & Intelligence**: Cybersecurity threat intelligence, criminal network analysis, intelligence orchestration
- **Industry**: Supply chain optimization, renewable energy management
- **AI Applications**: GraphRAG (Complete implementation and comparison)
**Cookbooks**:
- [GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — production-ready implementation
- [RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) — side-by-side comparison
---
## Next Steps
- **[Examples](examples.md)** - More detailed code examples
- **[Modules Guide](modules.md)** - Learn about available modules
- **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks
- **[API Reference](reference/core.md)** - Complete API documentation
---
!!! info "Contribute"
Have a use case to add? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica)
- [Cookbook](cookbook.md) — full notebook catalog organized by topic and difficulty
- [Modules Guide](modules.md) — every module with examples
- [API Reference](reference/core.md) — complete technical documentation
!!! info "Have a use case to add?"
[Open a PR](https://github.com/Hawksight-AI/semantica) or start a discussion on GitHub.
mkdocs_local.yml
+14
View File
@@ -0,0 +1,14 @@
INHERIT: mkdocs.yml
plugins:
- search:
lang: en
- minify:
minify_html: true
- mkdocstrings:
handlers:
python:
options:
docstring_style: google
show_source: true
show_root_heading: true
show_category_heading: true