From 129edaf05bdcbefafdf0223d4c7b8678787983aa Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Thu, 26 Mar 2026 18:31:39 +0530 Subject: [PATCH] docs: rewrite and polish documentation site - Rewrote index.md to match README (tagline, badges, Problem/Solution text) - Improved getting-started, concepts, quickstart, installation, faq, use-cases, contributing, glossary, learning-more, examples, modules, architecture, cookbook, deep-dive pages: tighter prose, fixed headings/bullets, removed inconsistencies and duplicate sections - Removed overuse of emojis from headings in integration pages (docling, snowflake) - Fixed change_management reference page: closed unclosed JSON code block that broke the right TOC, demoted noisy sub-headings to bold text - CSS layout: widened content area (max-width 1440px grid, left sidebar 11rem, right TOC narrowed to 11rem for broader content), tightened TOC spacing and font size, fixed word-wrap/overflow on TOC links - Added mkdocs_local.yml for local serving without mkdocs-jupyter plugin Co-Authored-By: Claude Sonnet 4.6 --- docs/architecture.md | 190 ++++------ docs/concepts.md | 350 ++++-------------- docs/contributing.md | 139 +++---- docs/cookbook.md | 120 ++---- docs/css/custom.css | 116 +++--- docs/deep-dive.md | 284 ++++++-------- docs/examples.md | 550 ++++++++-------------------- docs/faq.md | 171 ++++----- docs/getting-started.md | 107 +++--- docs/glossary.md | 24 +- docs/index.md | 481 ++++++++++-------------- docs/installation.md | 348 ++++++------------ docs/integrations/docling.md | 8 +- docs/integrations/snowflake.md | 18 +- docs/learning-more.md | 289 ++++----------- docs/modules.md | 111 +----- docs/quickstart.md | 288 +++++---------- docs/reference/change_management.md | 44 +-- docs/use-cases.md | 225 ++++-------- mkdocs_local.yml | 14 + 20 files changed, 1319 insertions(+), 2558 deletions(-) create mode 100644 mkdocs_local.yml diff --git a/docs/architecture.md b/docs/architecture.md index f230af11..8b73ab40 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,180 +1,126 @@ # Architecture -Semantica's modular, extensible framework for semantic intelligence and knowledge engineering. +Semantica is built around a three-layer, modular architecture designed for independent use of components, clean separation of concerns, and extensibility at each layer. --- -## Design Principles - -- **Modular**: Independent, reusable components -- **Extensible**: Easy to add new functionality -- **Scalable**: Handle large-scale data processing -- **Maintainable**: Clear separation of concerns - ---- - -## System Architecture +## System Overview ```mermaid graph TB A[Data Ingestion Layer] --> B[Semantic Processing Layer] B --> C[Application Layer] - - A1[Files • Web • APIs • Streams] --> A - B1[Parse • Normalize • Extract • Build] --> B - C1[GraphRAG • AI Agents • Analytics] --> C + + A1[Files · Web · APIs · Streams] --> A + B1[Parse · Normalize · Extract · Build] --> B + C1[GraphRAG · AI Agents · Analytics] --> C ``` -### Three-Layer Architecture - -**1. Data Ingestion Layer** -- Multiple file formats (PDF, DOCX, JSON, CSV, etc.) -- Web scraping and APIs -- Real-time streams (Kafka, RabbitMQ) -- Database connectors (SQL, NoSQL) - -**2. Semantic Processing Layer** -- Document parsing and normalization -- Entity and relationship extraction -- Embedding generation -- Knowledge graph construction -- Quality assurance and deduplication - -**3. Application Layer** -- GraphRAG for enhanced retrieval -- AI agent memory and context -- Multi-agent systems -- Analytics and visualization - --- -## Core Modules +## Three-Layer Architecture -### Orchestration -- **`semantica.core`** - Main framework class and coordination -- **`semantica.pipeline`** - Pipeline management and execution +### 1. Data Ingestion Layer -### Data Processing -- **`semantica.ingest`** - Universal data ingestion -- **`semantica.parse`** - Document parsing -- **`semantica.normalize`** - Data cleaning and normalization +Responsible for loading data from any source into the pipeline. -### Semantic Intelligence -- **`semantica.semantic_extract`** - Entity and relationship extraction -- **`semantica.embeddings`** - Vector embedding generation -- **`semantica.ontology`** - Ontology generation and management +- **File formats** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives +- **Web** — crawl via `WebIngestor` with configurable depth +- **Databases** — SQL, NoSQL, Snowflake via `DBIngestor` / `SnowflakeIngestor` +- **Streams** — Kafka, real-time feeds -### Knowledge Graphs -- **`semantica.kg`** - Knowledge graph construction -- **`semantica.vector_store`** - Vector storage (Weaviate, FAISS) -- **`semantica.triplet_store`** - RDF triplet storage (Jena, Blazegraph) -- **`semantica.graph_store`** - Property graphs (Neo4j, FalkorDB) +### 2. Semantic Processing Layer -### Quality Assurance -- **`semantica.deduplication`** - Entity deduplication -- **`semantica.conflicts`** - Conflict detection and resolution +The core intelligence engine — transforms raw data into structured knowledge. + +- Document parsing and normalization +- Entity and relationship extraction (NER, LLM-typed, rule-based) +- Embedding generation +- Knowledge graph construction with entity merging +- Deduplication, conflict detection, and validation + +### 3. Application Layer + +Consumes the knowledge graph for downstream use cases. + +- GraphRAG — graph-grounded retrieval for LLMs +- AI agent context and decision tracking +- Multi-agent pipelines +- Analytics, visualization, and export --- ## Data Flow ``` -1. Ingestion → Raw data from sources -2. Parsing → Structured content extraction -3. Normalization → Cleaned data -4. Semantic Extraction → Entities, relationships, events -5. Graph Construction → Entity resolution, conflict resolution -6. Quality Assurance → Deduplication, validation -7. Storage → Vector, triplet, and graph stores -8. Application → GraphRAG, agents, analytics +Ingest → raw data from sources +Parse → structured text extraction +Normalize → canonical forms, date/name standardization +Extract → entities, relationships, events +Build → entity resolution, graph construction +QA → deduplication, conflict resolution, validation +Store → vector store, graph store, triplet store +Deliver → GraphRAG, agents, export, visualization ``` --- +## Module Map + +| Layer | Modules | +|-------|---------| +| **Ingestion** | `ingest`, `parse`, `split`, `normalize` | +| **Semantic** | `semantic_extract`, `kg`, `ontology`, `reasoning` | +| **Storage** | `embeddings`, `vector_store`, `graph_store`, `triplet_store` | +| **Quality** | `deduplication`, `conflicts` | +| **Context** | `context`, `provenance`, `change_management` | +| **Output** | `export`, `visualization`, `pipeline` | + +For full module documentation, see the [Modules Guide](modules.md). + +--- + ## Extension Points -### Custom Ingestors +### Custom Ingestor ```python from semantica.ingest import BaseIngestor class CustomIngestor(BaseIngestor): def ingest(self, source): - # Custom ingestion logic - pass + # Return a list of document dicts + ... ``` -### Custom Extractors +### Custom Extractor ```python from semantica.semantic_extract import BaseExtractor class CustomExtractor(BaseExtractor): def extract(self, text): - # Custom extraction logic - pass + # Return a list of entity dicts + ... ``` -### Custom Validators - -Validators can be implemented within domain-specific modules (e.g., graph or ontology) as needed. - --- ## Design Decisions -### Modularity -Independent components that can be used standalone or together. Easy to test, maintain, and extend. +**Modularity** — every component can be used standalone. Import only what you need; the framework never forces a full stack. -### Plugin System -Extensible architecture allowing custom functionality without modifying core code. +**Pluggability** — extend any layer without modifying core code. Custom ingestors, extractors, validators, and exporters all follow the same base class pattern. -### Configuration Management -Centralized configuration with environment variable support for different deployment environments. +**Configuration over convention** — centralized config with environment variable overrides for deployment flexibility. -### Error Handling -Comprehensive error handling with graceful degradation and recovery mechanisms. +**Provenance by default** — lineage tracking is built into graph construction, not bolted on. Every node traces back to a source document. --- -## Performance +## Performance Characteristics -**Scalability** -- Parallel processing support -- Streaming for large datasets -- Efficient memory usage -- Intelligent caching - -**Optimization** -- Lazy loading -- Batch processing -- Connection pooling -- Query optimization - ---- - -## Security - -**Data Security** -- Secure credential handling -- Input validation and output sanitization -- Audit logging - -**Access Control** -- Authentication and authorization -- API key management -- Role-based access control - ---- - -## Future Roadmap - -- Distributed processing -- Real-time streaming improvements -- Advanced reasoning capabilities -- Multi-modal expansion -- Enhanced visualization - ---- - -For detailed module documentation, see [Modules Guide](modules.md) +- **Parallel execution** — `PipelineBuilder` supports configurable worker counts per stage +- **Delta processing** — incremental graph updates without full recompute +- **Streaming ingestion** — process large corpora without loading everything into memory +- **Backend flexibility** — swap in-memory NetworkX for Neo4j/FalkorDB at scale with no API changes diff --git a/docs/concepts.md b/docs/concepts.md index b32ec0a3..e40e868c 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -1,372 +1,172 @@ -# Core Concepts +# Core Concepts -**Learn the fundamental concepts behind Semantica in simple, practical terms.** +The fundamental ideas behind Semantica — explained plainly. -!!! tip "Quick Start" - New to Semantica? Start with [Getting Started](getting-started.md) for hands-on examples. +!!! tip "New here?" + Start with [Getting Started](getting-started.md) for hands-on examples, then come back to this page for deeper understanding. --- ## What is Semantica? -Semantica transforms unstructured data (documents, web pages, reports) into **knowledge graphs** - structured databases that AI systems can understand and reason about. +Semantica transforms unstructured data (documents, web pages, reports, databases) into **knowledge graphs** — structured representations that AI systems can query, reason about, and trace back to sources. -**What it does:** -- **Reads** documents, PDFs, web pages, databases -- **Extracts** entities (people, companies, dates) and relationships -- **Builds** connected knowledge graphs -- **Enables** AI to reason with structured knowledge - ---- - -## Core Architecture - -Semantica uses a **layered architecture** - use only what you need: - -
- -- **Input Layer** - - --- - - Data ingestion and preparation - - **Modules**: Ingest, Parse, Split, Normalize - -- **Semantic Layer** - - --- - - Intelligence and understanding - - **Modules**: Semantic Extract, Knowledge Graph, Ontology, Reasoning - -- **Storage Layer** - - --- - - Persistent data storage - - **Modules**: Embeddings, Vector Store, Graph Store - -- **Quality Layer** - - --- - - Data quality and consistency - - **Modules**: Deduplication, Conflicts - -- **Context & Memory** - - --- - - Agent memory and foundation data - - **Modules**: Context, Seed, LLM Providers - -- **Output & Orchestration** - - --- - - Export, visualization, and workflows - - **Modules**: Export, Visualization, Pipeline - -
+At its core, Semantica adds a **context and intelligence layer** on top of your existing AI stack: it doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs accountable. --- ## Knowledge Graphs -The foundation of Semantica - turning data into structured knowledge. +The foundation of everything in Semantica. -### What is a Knowledge Graph? +A knowledge graph stores information as: -A knowledge graph represents real-world information as: -- **Nodes** (entities): People, companies, locations, dates -- **Edges** (relationships): works_for, located_in, founded_by -- **Properties**: Name, date, confidence score, source +- **Nodes (entities)** — people, companies, locations, events, concepts +- **Edges (relationships)** — `works_for`, `located_in`, `founded_by` +- **Properties** — name, date, confidence score, source URL -### Why Knowledge Graphs? - -- **Searchable**: Find information instantly -- **Connectable**: Discover hidden relationships -- **Queryable**: Ask complex questions -- **Explainable**: Trace answers back to sources +This structure makes knowledge **searchable**, **connectable**, **queryable**, and — critically — **explainable**: every answer can be traced back to the facts and relationships that produced it. --- ## Entity Extraction (NER) -Finding and classifying entities in text. +Scanning text to find and classify real-world entities. -### What it does: -- Scans text for people, organizations, locations, dates -- Classifies each entity by type -- Assigns confidence scores -- Tracks source provenance - -### Example Output: ```python -# From: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino." +# Input: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino." { "entities": [ - {"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98}, - {"text": "Steve Jobs", "type": "PERSON", "confidence": 0.99}, - {"text": "1976", "type": "DATE", "confidence": 0.95}, - {"text": "Cupertino", "type": "LOCATION", "confidence": 0.97} + {"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98}, + {"text": "Steve Jobs", "type": "PERSON", "confidence": 0.99}, + {"text": "1976", "type": "DATE", "confidence": 0.95}, + {"text": "Cupertino", "type": "LOCATION", "confidence": 0.97} ] } ``` +Each entity gets a type, confidence score, and a link to its source document. + --- ## Relationship Extraction -Finding connections between entities. +Finding how entities connect to each other. -### What it does: -- Identifies how entities relate to each other -- Extracts relationship types and directions -- Provides context and confidence -- Links to source documents - -### Example Output: ```python { "relationships": [ - {"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", "confidence": 0.92}, - {"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino", "confidence": 0.89} + {"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc.", "confidence": 0.92}, + {"subject": "Apple Inc.", "predicate": "located_in", "object": "Cupertino", "confidence": 0.89} ] } ``` +Relationships can be extracted via rule-based methods, ML models, or LLMs (with `"llm_typed"` metadata). + --- ## Embeddings -Turning text into numerical vectors for AI understanding. +Embeddings convert text into numerical vectors so that AI systems can measure semantic similarity — finding related concepts even when the exact words differ. -### What are embeddings? -- **Numerical representations** of text, entities, and relationships -- **Similarity calculations** - find related concepts -- **AI-powered search** - semantic understanding -- **Clustering and grouping** - discover patterns +Semantica uses embeddings for: -### Use Cases: -- **Semantic Search** - find documents by meaning, not keywords -- **Entity Resolution** - match similar entities across sources -- **Recommendations** - suggest related content -- **AI Input** - provide structured context to LLMs - ---- - -## Temporal Graphs - -Knowledge graphs that understand time. - -### What they track: -- **When** events happened -- **How** entities changed over time -- **Temporal relationships** - before, after, during -- **Historical context** - point-in-time snapshots - -### Example Uses: -- **Company History** - track mergers, leadership changes -- **Person Careers** - job changes, relocations -- **Policy Evolution** - law changes over time -- **Research Progress** - scientific discoveries timeline +- **Semantic search** — retrieve by meaning, not just keywords +- **Entity resolution** — match the same entity across different sources +- **Precedent search** — find similar past decisions +- **GraphRAG retrieval** — hybrid vector + graph traversal --- ## GraphRAG -Enhanced AI retrieval using knowledge graphs. +GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses by grounding them in a structured knowledge graph rather than raw text chunks alone. -### How it works: -1. **Query** user question -2. **Retrieve** relevant graph context -3. **Enhance** with relationships and entities -4. **Generate** AI response with sources +How it works: -### Benefits: -- **More accurate** answers -- **Source attribution** - trace answers back -- **Context awareness** - understand relationships -- **Reduced hallucination** - grounded in facts +1. User submits a query +2. Semantica retrieves relevant graph context (entities, relationships, reasoning paths) +3. The LLM generates a response grounded in that context +4. Every claim in the response links back to a source node in the graph + +This eliminates the hallucination and traceability problems of standard RAG. --- ## Ontology -Defining the structure and rules of your knowledge. +An ontology defines the schema and rules for your knowledge — what entity types exist, which relationships are valid, and what constraints apply. -### What it provides: -- **Schema definition** - what types exist -- **Relationship rules** - valid connections -- **Property constraints** - required fields -- **Inheritance hierarchies** - parent-child relationships - -### Example: ```python -# Define ontology structure ontology = { "classes": ["Person", "Organization", "Location"], - "properties": ["name", "date", "confidence"], - "relationships": ["works_for", "located_in", "born_in"], + "relationships": ["works_for", "located_in", "founded_by"], "rules": { - "Person": ["must_have_name", "can_have_birth_date"], + "Person": ["must_have_name"], "Organization": ["must_have_name", "can_have_founding_date"] } } ``` +Semantica can auto-generate ontologies from your knowledge graph, or import existing OWL/RDF/Turtle ontologies. + --- ## Reasoning & Inference -Making logical deductions from your knowledge. +Semantica includes multiple reasoning engines to derive new knowledge from existing facts. -### What it can do: -- **Infer missing facts** - derive new knowledge -- **Detect inconsistencies** - find contradictions -- **Apply rules** - automate decision making -- **Explain reasoning** - show how conclusions were reached +``` +Known: Steve Jobs founded Apple Inc. +Known: Apple Inc. is headquartered in Cupertino +Inferred: Steve Jobs has a connection to Cupertino +``` -### Example: -``` -Known: Steve Jobs founded Apple Inc. -Known: Apple Inc. is headquartered in Cupertino -Inferred: Steve Jobs has connection to Cupertino -``` +Supported engines: forward chaining, Rete network, deductive, abductive, and SPARQL reasoning — all producing **explainable inference paths**, not black-box conclusions. + +--- + +## Temporal Graphs + +Knowledge changes over time. Temporal graphs attach `valid_from` / `valid_until` windows to nodes and edges, enabling point-in-time queries and historical analysis. + +Common uses: tracking company leadership changes, policy evolution, research timelines, financial instrument histories. --- ## Deduplication & Entity Resolution -Finding and merging duplicate entities. +Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance. -### What it does: -- **Detects duplicates** - same entity, different names -- **Merges information** - combine attributes -- **Resolves conflicts** - handle contradictory data -- **Maintains provenance** - track original sources - -### Example: -```python -# These refer to the same entity: -"Apple Inc." → "Apple" → "Apple Computer Inc." -# Merge into single entity with all attributes -``` +Strategies: Jaro-Winkler similarity (v1), `blocking_v2`, `hybrid_v2`, `semantic_v2` (v2 — up to 7x faster). --- -## Data Normalization +## Provenance & Auditability -Cleaning and standardizing your data. +Every fact in Semantica links back to: -### What it fixes: -- **Format inconsistencies** - dates, names, numbers -- **Canonical forms** - standard representations -- **Data quality** - remove errors and noise -- **Standardization** - consistent naming conventions +- The source document it came from +- The extraction method used +- The ontology rules applied +- The reasoning steps that produced any inference -### Examples: -- **Dates**: "Jan 1, 2020" → "2020-01-01" -- **Names**: "Dr. Smith PhD" → "John Smith" -- **Companies**: "Apple" → "Apple Inc." -- **Locations**: "NYC" → "New York City" +This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails. --- ## Conflict Detection -Finding and resolving contradictory information. +When multiple sources disagree on the same fact, Semantica flags and resolves the conflict rather than silently picking one value. -### What it identifies: -- **Factual conflicts** - different values for same fact -- **Temporal conflicts** - impossible timelines -- **Logical conflicts** - contradictory relationships -- **Source reliability** - trustworthiness assessment - -### Resolution Strategies: -- **Most recent** - prefer newer information -- **Most reliable** - prefer trusted sources -- **Majority vote** - go with consensus -- **Manual review** - flag for human review +Resolution strategies: prefer most recent, prefer most reliable source, majority vote, or flag for manual review. --- -## Getting Started +## Next Steps -Ready to build your first knowledge graph? - -### Quick Start (5 minutes) -```python -from semantica.semantic_extract import NERExtractor -from semantica.kg import GraphBuilder - -# Extract entities -ner = NERExtractor() -entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") - -# Build graph -kg = GraphBuilder().build({"entities": entities, "relationships": []}) -``` - -### Learn More -- **Getting Started Guide** - [Getting Started](getting-started.md) -- **Cookbook Examples** - [Cookbook](cookbook.md) -- **Module Documentation** - [Reference](reference/) -- **Community Support** - [Community](community.md) - -### Common Use Cases -- **Document Analysis** - extract knowledge from reports -- **Research Assistant** - find connections in academic papers -- **Business Intelligence** - analyze company relationships -- **Regulatory Compliance** - track policy changes - ---- - -## Best Practices - -### Start Small -- Begin with a single document type -- Focus on specific entity types -- Validate results before scaling - -### Configure Properly -- Choose appropriate models for your domain -- Set confidence thresholds -- Define clear ontology rules - -### Validate Data -- Check extraction quality -- Review relationship accuracy -- Test with known examples - -### Handle Errors -- Implement error handling -- Log processing issues -- Provide feedback mechanisms - -### Optimize Performance -- Use appropriate storage backends -- Cache frequently accessed data -- Monitor resource usage - -### Document Workflows -- Record processing steps -- Track data sources -- Maintain change logs - ---- - -## Need Help? - -- **Documentation**: [Getting Started](getting-started.md) -- **Examples**: [Cookbook](cookbook.md) -- **Community**: [Discord](community.md) -- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) -- **Support**: [Contact Us](community.md) +- [Quickstart Tutorial](quickstart.md) — build a full pipeline with code +- [Modules Guide](modules.md) — every module explained +- [Use Cases](use-cases.md) — real-world domain examples +- [API Reference](reference/core.md) — complete technical reference diff --git a/docs/contributing.md b/docs/contributing.md index b2441e3b..4b6f0e67 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -1,130 +1,99 @@ -# Contributing +# Contributing -**Help us build Semantica! Every contribution makes the project better.** +Contributions of all kinds are welcome — code, documentation, tests, and community support. --- -## Getting Started +## Quick Start -### Quick Start -1. **Fork** the repository -2. **Create** a feature branch -3. **Make** your changes -4. **Test** your changes -5. **Submit** a pull request +```bash +# Fork the repo, then: +git clone https://github.com/your-username/semantica.git +cd semantica +pip install -e ".[dev]" +pytest +``` -### First Contribution? -Look for issues labeled [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) for beginner-friendly tasks. +First time? Look for [`good-first-issue`](https://github.com/Hawksight-AI/semantica/labels/good-first-issue) labels for beginner-friendly tasks. --- ## Ways to Contribute -### Code -- **Fix bugs** - Resolve reported issues -- **Add features** - Implement new functionality -- **Improve performance** - Optimize existing code -- **Refactor** - Clean up code structure +**Code** +- Fix bugs and resolve open issues +- Implement new features or integrations +- Optimize performance or refactor existing code -### Documentation -- **Fix typos** - Correct spelling and grammar -- **Improve guides** - Make documentation clearer -- **Add examples** - Provide practical code examples -- **Update API docs** - Keep reference current +**Documentation** +- Fix typos, improve clarity, add examples +- Write tutorials or domain-specific cookbook notebooks +- Keep API reference up to date -### Testing -- **Write tests** - Add test coverage -- **Fix tests** - Resolve test failures -- **Report issues** - Identify bugs through testing +**Testing** +- Add test coverage for untested modules +- Reproduce and confirm reported bugs +- Improve test reliability -### Community -- **Help others** - Answer questions in issues -- **Share knowledge** - Write tutorials and guides -- **Provide feedback** - Review pull requests +**Community** +- Answer questions in issues and discussions +- Review pull requests +- Share Semantica in your blog posts or talks --- ## Reporting Issues ### Bug Reports -When reporting bugs, include: -- **Description** - What happened -- **Steps to reproduce** - How to trigger the issue -- **Expected behavior** - What should happen -- **Environment** - Your setup details + +Include: what happened, steps to reproduce, expected behavior, and your environment (Python version, OS, Semantica version). ### Feature Requests -When suggesting features, include: -- **Use case** - Why you need this feature -- **Proposed solution** - How it should work -- **Benefits** - How it helps the community + +Include: your use case, what you'd like Semantica to do, and how it benefits others. --- ## Pull Request Guidelines -### Before Submitting -- **Test** your changes thoroughly -- **Document** new features with examples -- **Update** relevant documentation -- **Follow** the existing code style +Before submitting: -### Pull Request Checklist -- [ ] Code follows project style -- [ ] Tests pass locally -- [ ] Documentation is updated -- [ ] Commit messages are clear -- [ ] No merge conflicts +- [ ] Tests pass locally (`pytest`) +- [ ] New features are documented with examples +- [ ] Code follows project style (Black, isort, flake8) +- [ ] Commit messages are clear and descriptive +- [ ] No unresolved merge conflicts --- ## Development Setup -### Local Development ```bash -# Clone your fork git clone https://github.com/your-username/semantica.git cd semantica +pip install -e ".[dev]" +``` -# Install in development mode -pip install -e .[dev] +Code style tools used: **Black** (formatting), **isort** (imports), **flake8** (linting). -# Run tests +Run the full test suite: + +```bash pytest ``` -### Code Style -We use standard Python formatting: -- **Black** for code formatting -- **isort** for import sorting -- **flake8** for linting +--- + +## Community + +Please follow the [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md). Be respectful, patient, and constructive. + +All contributors are recognized in release notes and the GitHub contributors list. --- -## Community Guidelines +## Help -### Code of Conduct -Please follow our [Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md). - -### Communication -- **Be respectful** - Treat everyone with kindness -- **Be helpful** - Assist others when you can -- **Be patient** - Allow time for reviews -- **Be constructive** - Provide helpful feedback - ---- - -## Recognition - -All contributors are recognized in: -- **GitHub contributors** - Automatic recognition -- **Release notes** - Notable contributions -- **Community highlights** - Outstanding work - ---- - -## Need Help? - -- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Ask questions -- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Community chat -- **[Code of Conduct](https://github.com/Hawksight-AI/semantica/blob/main/CODE_OF_CONDUCT.md)** - Community standards +- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) +- [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) +- [Discord](https://discord.gg/sV34vps5hH) diff --git a/docs/cookbook.md b/docs/cookbook.md index f561aeee..95defb06 100644 --- a/docs/cookbook.md +++ b/docs/cookbook.md @@ -1,108 +1,58 @@ -# 🍳 Semantica Cookbook +# Semantica Cookbook -Welcome to the **Semantica Cookbook**! +Interactive Jupyter notebooks covering everything from your first knowledge graph to production GraphRAG systems. -This collection of Jupyter notebooks is designed to take you from a beginner to an expert in building semantic AI applications. Whether you're looking for quick recipes or deep-dive tutorials, you'll find it here. - -!!! tip "How to use this Cookbook" - - **Beginners**: Start with the [Core Tutorials](#core-tutorials) to learn the basics. - - **Developers**: Check out [Advanced Concepts](#advanced-concepts) for deep dives into specific features. - - **Architects**: Explore [Industry Use Cases](#industry-use-cases) for end-to-end solutions. +!!! tip "Where to start" + - **New to Semantica** — begin with [Core Tutorials](#core-tutorials) + - **Building an application** — see [Advanced Concepts](#advanced-concepts) or [Industry Use Cases](#industry-use-cases) + - **Need installation help** — see the [Installation Guide](installation.md) !!! note "Prerequisites" - Before running these notebooks, ensure you have: - - Python 3.8+ installed - - A basic understanding of Python and Jupyter - - An OpenAI API key (for most examples) - -!!! success "Installation" - Install Semantica from PyPI (recommended): - - ```bash - pip install semantica - # Or with all optional dependencies: - pip install semantica[all] - ``` - - For more installation options, see the [Installation Guide](installation.md). + Python 3.8+, Jupyter, and an OpenAI API key (for most examples). --- -## � Featured Recipes - -Hand-picked tutorials to show you the power of Semantica. +## Featured Recipes
-- :material-robot: **GraphRAG Complete** - --- - Build a production-ready Graph Retrieval Augmented Generation system. - - **Topics**: RAG, LLMs, Vector Search, Graph Traversal - - **Difficulty**: Advanced - - [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) - -- :material-scale-balance: **RAG vs. GraphRAG Comparison** - --- - Side-by-side comparison of Standard RAG vs. GraphRAG using real-world data. - - **Topics**: RAG, GraphRAG, Benchmarking, Visualization - - **Difficulty**: Intermediate - - [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) - -- :material-robot: **GraphRAG Complete** - --- - Build a production-ready Graph Retrieval Augmented Generation system. - - **New Features**: Graph Validation, Logical Inference, Hybrid Context. - - **Topics**: RAG, LLMs, Vector Search, Graph Traversal - - **Difficulty**: Advanced - - [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) - -- :material-scale-balance: **RAG vs. GraphRAG Comparison** - --- - Side-by-side comparison of Standard RAG vs. GraphRAG using real-world data. - - **New Features**: Inference-Enhanced GraphRAG, Reasoning Gap Analysis. - - **Topics**: RAG, GraphRAG, Benchmarking, Visualization - - **Difficulty**: Intermediate - - [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) - - :material-graph: **Your First Knowledge Graph** --- Go from raw text to a queryable knowledge graph in 20 minutes. - - **Topics**: Extraction, Graph Construction, Visualization - - **Difficulty**: Beginner - + + **Topics**: Extraction, Graph Construction, Visualization · **Difficulty**: Beginner + [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb) +- :material-robot: **GraphRAG Complete** + --- + Build a production-ready Graph Retrieval Augmented Generation system with hybrid retrieval and logical inference. + + **Topics**: RAG, LLMs, Vector Search, Graph Traversal · **Difficulty**: Advanced + + [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) + +- :material-scale-balance: **RAG vs. GraphRAG Comparison** + --- + Side-by-side benchmark of standard RAG vs. GraphRAG on real-world data. + + **Topics**: RAG, GraphRAG, Benchmarking, Reasoning Gap · **Difficulty**: Intermediate + + [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) + - :material-shield-alert: **Real-Time Anomaly Detection** --- - Detect anomalies in streaming data using dynamic graphs. - - **Topics**: Streaming, Security, Dynamic Graphs - - **Difficulty**: Advanced - + Detect anomalies in streaming data using dynamic knowledge graphs. + + **Topics**: Streaming, Security, Dynamic Graphs · **Difficulty**: Advanced + [Open Notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb)
--- -## 🏁 Core Tutorials {#core-tutorials} +## Core Tutorials {#core-tutorials} Essential guides to master the Semantica framework. @@ -210,7 +160,7 @@ Essential guides to master the Semantica framework. --- -## 🧠 Advanced Concepts +## Advanced Concepts Deep dive into advanced features, customization, and complex workflows. @@ -329,7 +279,7 @@ Deep dive into advanced features, customization, and complex workflows. --- -## 🏭 Industry Use Cases {#industry-use-cases} +## Industry Use Cases {#industry-use-cases} Real-world examples and end-to-end applications across various industries. @@ -497,7 +447,7 @@ Real-world examples and end-to-end applications across various industries. --- -## 🛠️ How to Run +## How to Run To run these notebooks locally: diff --git a/docs/css/custom.css b/docs/css/custom.css index 079c651b..9b8b59ae 100644 --- a/docs/css/custom.css +++ b/docs/css/custom.css @@ -216,76 +216,92 @@ html { border-left: 2px solid var(--md-accent-fg-color); } -/* +/* ========================================================================== Layout Optimization - ========================================================================== + ========================================================================== */ -/* Reduce spacing between sidebars and content for all pages */ -.md-content__inner { - padding-left: 0.75rem; - padding-right: 0.75rem; - margin-left: 0; -} -.md-content { - margin-left: 0; +/* Widen the overall grid */ +.md-grid { + max-width: 1440px; + margin-left: auto; + margin-right: auto; padding-left: 0.5rem; - padding-right: 0.5rem; -} - -/* Reduce spacing before left sidebar for all pages */ -.md-sidebar { - padding-left: 0.25rem; - margin-left: 0; } +/* Narrow left sidebar to give content more room */ .md-sidebar--primary { - padding-right: 0.5rem; + width: 11rem; + padding-right: 0.25rem; padding-left: 0.25rem; +} + +/* Right TOC sidebar */ +.md-sidebar--secondary { + width: 11rem; + padding-left: 0.5rem; + padding-right: 0; margin-left: 0; } -/* Reduce spacing after right sidebar (table of contents) and shift it right slightly */ -.md-sidebar--secondary { - padding-left: 1.25rem; - padding-right: 0; - margin-right: 0; - margin-left: 3.5rem; +.md-sidebar--secondary .md-nav { + width: 11rem; } -/* Reduce right edge spacing - similar to left */ -.md-container { - padding-right: 0; +/* Tighten TOC list spacing */ +.md-sidebar--secondary .md-nav__list { + padding-bottom: 1.5rem; + margin: 0; +} + +.md-sidebar--secondary .md-nav__item { + padding: 0; + margin: 0; +} + +.md-sidebar--secondary .md-nav__link { + white-space: normal; + word-break: break-word; + overflow: visible; + text-overflow: unset; + padding-top: 0.15rem; + padding-bottom: 0.15rem; + line-height: 1.4; + font-size: 0.7rem; + margin: 0; +} + +/* Nested TOC items (h3, h4) */ +.md-sidebar--secondary .md-nav__item .md-nav__item .md-nav__link { + padding-left: 0.6rem; + font-size: 0.68rem; +} + +/* Remove extra gap between TOC title and first item */ +.md-sidebar--secondary .md-nav__title { + margin-bottom: 0.25rem; + padding-bottom: 0.25rem; +} + +/* Give the main content area maximum available width */ +.md-content { + max-width: none; + padding-left: 1rem; + padding-right: 1rem; +} + +.md-content__inner { + max-width: none; + padding-left: 1rem; + padding-right: 1rem; + margin-left: 0; margin-right: 0; } -.md-main { - margin-right: 0; - padding-right: 0; -} - -/* Reduce margins of the main container */ .md-main__inner { margin-left: 0; margin-right: 0; - padding-right: 0; -} - -/* Reduce right edge spacing on body/html */ -body { - margin-right: 0; - padding-right: 0; -} - -html { - margin-right: 0; - padding-right: 0; -} - -.md-grid { - margin-left: 0; - padding-left: 0.5rem; } /* Ensure text content is left-aligned by default */ diff --git a/docs/deep-dive.md b/docs/deep-dive.md index 53ada134..3a6ea484 100644 --- a/docs/deep-dive.md +++ b/docs/deep-dive.md @@ -1,10 +1,15 @@ # Deep Dive -Advanced topics, architecture, and internals of Semantica. +Internals, advanced concepts, and extension points for contributors and power users. -## Architecture Overview +!!! tip "Just getting started?" + Read [Architecture](architecture.md) for a higher-level overview first. -Semantica follows a modular, extensible architecture: +--- + +## Pipeline Internals + +The full data flow through a Semantica pipeline: ```mermaid graph TB @@ -16,65 +21,79 @@ graph TB F --> G[Knowledge Graph Builder] G --> H[Embedding Generator] H --> I[Export Layer] - + D --> D1[Entity Extractor] D --> D2[Relationship Extractor] D --> D3[Triplet Extractor] - + G --> G1[Graph Validator] G --> G2[Graph Analyzer] - + H --> H1[Text Embeddings] H --> H2[Graph Embeddings] ``` +### Sequence Diagram + +```mermaid +sequenceDiagram + participant User + participant Semantica + participant Ingestor + participant Parser + participant Extractor + participant Resolver + participant GraphBuilder + participant Exporter + + User->>Semantica: build_knowledge_base(sources) + Semantica->>Ingestor: ingest(sources) + Ingestor->>Parser: parse(documents) + Parser->>Extractor: extract(text) + Extractor->>Resolver: resolve_conflicts(entities) + Resolver->>GraphBuilder: build_graph(resolved_data) + GraphBuilder->>Exporter: export(graph) + Exporter->>User: return result +``` + +--- + ## System Components -### 1. Ingestion Layer +### Ingestion Layer -Handles data input from various sources: +Handles input from any source: -- **File Ingestor**: PDF, DOCX, HTML, JSON, CSV -- **Web Ingestor**: URLs, web scraping -- **Database Ingestor**: SQL databases -- **Stream Ingestor**: Real-time data streams +- **FileIngestor** — PDF, DOCX, HTML, JSON, CSV, archives +- **WebIngestor** — URL crawling and scraping +- **DBIngestor** / **SnowflakeIngestor** — SQL databases +- **StreamIngestor** — Kafka and real-time feeds -### 2. Parsing Layer +### Parsing Layer -Converts raw data into structured format: +Converts raw data to structured text: -- Document parsing (PDF, Word, etc.) -- Text extraction -- Metadata extraction -- Format normalization +- Text and metadata extraction from documents +- OCR for scanned content +- Layout analysis (via Docling for tables and columns) -### 3. Extraction Layer +### Extraction Layer -Core semantic extraction: +Core semantic processing pipeline: -```python -# Entity extraction pipeline +``` text → Tokenization → NER → Entity Linking → Entity Validation ``` -**Components:** -- Named Entity Recognition (NER) -- Relationship Extraction -- Triplet Extraction -- Coreference Resolution +Components: Named Entity Recognition, Relationship Extraction, Triplet Extraction, Coreference Resolution. -### 4. Normalization Layer +### Normalization Layer -Standardizes extracted data: +Standardizes extracted data: entity names, date formats, numbers, encodings, and language normalization. -- Entity normalization -- Date/time normalization -- Number normalization -- Text cleaning +### Conflict Resolution -### 5. Conflict Resolution - -Handles conflicting information: +Handles contradictory facts from multiple sources: ```mermaid graph LR @@ -88,64 +107,28 @@ graph LR E --> H F --> H G --> H - - style A fill:#ffebee - style H fill:#c8e6c9 - style C fill:#fff9c4 ``` -### 6. Knowledge Graph Builder +### Knowledge Graph Builder -Constructs the knowledge graph: +- Entity resolution across sources +- Edge creation (typed relationships) +- Property assignment with confidence scores +- Graph validation and quality checks -- Node creation (entities) -- Edge creation (relationships) -- Property assignment -- Graph validation -- Quality checks +### Embedding Generator -### 7. Embedding Generator +- Text embeddings (Sentence-Transformers, FastEmbed, OpenAI, BGE) +- Graph embeddings (Node2Vec, GraphSAGE) -Generates vector representations: - -- Text embeddings (sentence transformers) -- Graph embeddings (node2vec, GraphSAGE) -- Multimodal embeddings - -## Data Flow - -```mermaid -sequenceDiagram - participant User - participant Semantica - participant Ingestor - participant Parser - participant Extractor - participant Resolver - participant GraphBuilder - participant Exporter - - User->>Semantica: build_knowledge_base(sources) - Semantica->>Ingestor: ingest(sources) - Ingestor->>Parser: parse(documents) - Parser->>Extractor: extract(text) - Extractor->>Resolver: resolve_conflicts(entities) - Resolver->>GraphBuilder: build_graph(resolved_data) - GraphBuilder->>Exporter: export(graph) - Exporter->>User: return result - - Note over User,Exporter: Complete pipeline execution -``` +--- ## Advanced Concepts -### Entity Resolution - -Matching entities across sources: +### Entity Resolution Algorithm ```python -# Entity resolution algorithm -def resolve_entities(entities): +def resolve_entities(entities, threshold=0.85): clusters = [] for entity in entities: matched = False @@ -161,123 +144,92 @@ def resolve_entities(entities): ### Relationship Inference -Inferring implicit relationships: +Semantica's reasoning engines can derive implicit relationships: -- Transitive relationships -- Temporal relationships -- Causal relationships -- Hierarchical relationships +- **Transitive** — if A→B and B→C, infer A→C +- **Temporal** — before, after, during from timestamped facts +- **Causal** — IF/THEN rules via `Reasoner` +- **Hierarchical** — subclass/instance inference via `OntologyReasoner` -### Graph Optimization - -Optimizing knowledge graph structure: - -- Node deduplication -- Edge consolidation -- Path compression -- Index optimization - -## Performance Considerations - -### Scalability - -- **Horizontal Scaling**: Process multiple documents in parallel -- **Vertical Scaling**: Use GPU acceleration -- **Caching**: Cache embeddings and parsed documents -- **Lazy Loading**: Load components on demand - -### Memory Management +### Batch Processing for Large Datasets ```python -# Process large datasets efficiently def process_large_dataset(sources, batch_size=100): for i in range(0, len(sources), batch_size): - batch = sources[i:i+batch_size] + batch = sources[i : i + batch_size] result = semantica.build_knowledge_base(batch) - # Save and clear memory save_result(result) del result gc.collect() ``` +--- + ## Extension Points -### Custom Plugins - -Create custom plugins: +### Custom Plugin ```python from semantica.core import Plugin class CustomPlugin(Plugin): def process(self, data): - # Your custom processing + # Your custom processing logic return processed_data ``` -### Custom Extractors - -Implement custom extractors: +### Custom Extractor ```python from semantica.semantic_extract import BaseExtractor class DomainSpecificExtractor(BaseExtractor): - def extract_entities(self, text): - # Domain-specific extraction logic + def extract(self, text): + # Domain-specific entity extraction logic return entities ``` -## Internal APIs +### Custom Ingestor -### Core APIs +```python +from semantica.ingest import BaseIngestor -- `Semantica.build_knowledge_base()` - Main entry point -- `KGBuilder.build()` - Graph construction -- `ConflictResolver.resolve()` - Conflict resolution -- `EmbeddingGenerator.generate()` - Embedding generation - -### Extension APIs - -- Plugin registration -- Custom extractor registration -- Custom exporter registration -- Event hooks - -## Design Decisions - -### Why Modular Architecture? - -- **Extensibility**: Easy to add new features -- **Testability**: Components can be tested independently -- **Maintainability**: Clear separation of concerns -- **Flexibility**: Swap implementations easily - -### Why Conflict Resolution? - -- **Data Quality**: Ensures consistent knowledge -- **Multi-Source**: Handles conflicting information -- **Flexibility**: Multiple resolution strategies -- **Transparency**: Track resolution decisions - -## Future Enhancements - -Planned improvements: - -- Distributed processing -- Real-time streaming -- Advanced reasoning -- Multi-modal support expansion -- Enhanced visualization - -## Contributing to Core - -Interested in contributing to Semantica's core? See our [Contributing Guide](https://github.com/Hawksight-AI/semantica/blob/main/CONTRIBUTING.md). +class CustomIngestor(BaseIngestor): + def ingest(self, source): + # Load and return document dicts + return documents +``` --- -For more information: -- **[API Reference](reference/core.md) - Detailed API documentation -- **[Learning More](learning-more.md)** - Additional resources -- **[GitHub Repository](https://github.com/Hawksight-AI/semantica)** - Source code +## Internal APIs +| API | Purpose | +|-----|---------| +| `Semantica.build_knowledge_base()` | Main orchestration entry point | +| `GraphBuilder.build()` | Graph construction | +| `ConflictResolver.resolve()` | Conflict resolution | +| `EmbeddingGenerator.generate()` | Embedding generation | + +Extension hooks: plugin registration, custom extractor registration, custom exporter registration, event hooks. + +--- + +## Design Decisions + +**Why modular architecture?** Each component is independently testable and swappable. You can use `NERExtractor` alone without pulling in graph storage or pipelines. + +**Why built-in conflict resolution?** Multi-source data always has contradictions. Ignoring them produces garbage graphs. Explicit resolution strategies give you control over data quality. + +**Why W3C PROV-O for provenance?** It's an industry standard with tooling support. Using a custom format would make lineage data non-portable. + +**Why multiple reasoning engines?** Different problems need different reasoning: forward chaining for rule application, SPARQL for graph queries, abductive for hypothesis generation. No single engine fits all cases. + +--- + +## Further Reading + +- [Architecture](architecture.md) — high-level three-layer overview +- [Modules](modules.md) — every module with code examples +- [API Reference](reference/core.md) — complete technical reference +- [Contributing](contributing.md) — how to extend the framework diff --git a/docs/examples.md b/docs/examples.md index 48757e7d..7f2c6bfa 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -1,306 +1,134 @@ # Examples -Real-world examples and use cases for Semantica. - -!!! tip "Interactive Learning" - For hands-on interactive tutorials, check out our [Cookbook](cookbook.md) with Jupyter notebooks covering everything from basics to advanced use cases. +Code examples organized by complexity. For interactive notebooks, see the [Cookbook](cookbook.md). --- -## Example Gallery +## Beginner -
+### Basic Knowledge Graph -- :material-school: **Getting Started** - --- - Quick examples to get you up and running in 5 minutes. - - [View Examples](#getting-started-5-min-examples) +```python +from semantica.ingest import FileIngestor +from semantica.parse import DocumentParser +from semantica.semantic_extract import NERExtractor, RelationExtractor +from semantica.kg import GraphBuilder -- :material-cogs: **Core Workflows** - --- - Common workflows for building production-ready graphs. - - [View Examples](#core-workflows-15-min-examples) - -- :material-rocket: **Advanced Patterns** - --- - Complex use cases and production deployments. - - [View Examples](#advanced-patterns-30-min-examples) - -- :material-factory: **Production Patterns** - --- - Scalable deployment patterns for enterprise use. - - [View Examples](#production-patterns) - -
- ---- - -## Getting Started (5 min examples) - -### Example 1: Basic Knowledge Graph - -**Difficulty**: Beginner - -Build a knowledge graph from a single document using Semantica's modular approach. This example demonstrates the complete workflow from document ingestion to graph construction. - -**What it demonstrates:** -- Document ingestion and parsing -- Entity and relationship extraction -- Knowledge graph construction - -**For complete step-by-step examples, see:** -- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Complete walkthrough - - **Topics**: Ingestion, parsing, extraction, graph building - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Learning the complete workflow - -### Example 2: Entity Extraction - -**Difficulty**: Beginner - -Extract entities from text using Named Entity Recognition. This example shows how to identify and classify named entities in text. - -**What it demonstrates:** -- Named Entity Recognition (NER) -- Entity type classification -- Confidence scoring - -**For complete examples, see:** -- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn entity extraction - - **Topics**: NER methods, entity types, extraction techniques - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Understanding entity extraction - print(f"{entity.text}: {entity.label}") -``` - -**Expected Output:** -``` -Apple Inc.: ORGANIZATION -Steve Jobs: PERSON -``` - -### Example 3: Multi-Source Integration - -**Difficulty**: Beginner - -Combine data from multiple sources into a unified knowledge graph. This example demonstrates integrating data from diverse sources. - -**What it demonstrates:** -- Multi-source data ingestion -- Entity merging and resolution -- Unified graph construction - -**For complete examples, see:** -- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration patterns - - **Topics**: Multi-source integration, entity resolution, conflict handling - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Building unified knowledge graphs from diverse sources - ---- - -## Core Workflows (15 min examples) - -### Example 4: Conflict Resolution - -**Difficulty**: Intermediate - -Resolve conflicts in data from multiple sources. This example shows how to identify and resolve conflicting information. - -**What it demonstrates:** -- Conflict detection -- Conflict resolution strategies -- Data quality assurance - -**For complete examples, see:** -- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Conflict resolution patterns - - **Topics**: Conflict detection, resolution strategies, data quality - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Data integration, quality assurance ingestor = FileIngestor() -parser = DocumentParser() -ner = NERExtractor() +parser = DocumentParser() +ner = NERExtractor() +rel = RelationExtractor() -all_entities = [] -for source in ["source1.pdf", "source2.pdf"]: - doc = ingestor.ingest_file(source) - parsed = parser.parse_document(source) - text = parsed.get("full_text", "") - entities = ner.extract_entities(text) - all_entities.extend(entities) +sources = ingestor.ingest("data/sample.pdf") +parsed = parser.parse(sources[0]) + +entities = ner.extract(parsed) +relationships = rel.extract(parsed, entities=entities) + +kg = GraphBuilder(merge_entities=True).build( + entities=entities, relationships=relationships +) +print(f"{len(kg.nodes)} nodes, {len(kg.edges)} edges") +``` + +### Entity Extraction from Text + +```python +from semantica.semantic_extract import NERExtractor + +ner = NERExtractor() +entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") + +for entity in entities: + print(f"{entity['text']}: {entity['type']}") +# Apple Inc.: ORGANIZATION +# Steve Jobs: PERSON +# 1976: DATE +``` + +### Custom NER Configuration + +```python +from semantica.semantic_extract import NERExtractor + +ner = NERExtractor( + method="llm", + provider="openai", + model="gpt-4", + confidence_threshold=0.8, + temperature=0.0, +) +entities = ner.extract("Your document text here...") +``` + +--- + +## Intermediate + +### Multi-Source Integration + +```python +from semantica.ingest import FileIngestor +from semantica.parse import DocumentParser +from semantica.semantic_extract import NERExtractor, RelationExtractor +from semantica.kg import GraphBuilder + +ingestor = FileIngestor() +parser = DocumentParser() +ner = NERExtractor() +rel = RelationExtractor() +builder = GraphBuilder(merge_entities=True) + +all_entities, all_rels = [], [] + +for path in ["source1.pdf", "source2.pdf", "source3.pdf"]: + sources = ingestor.ingest(path) + parsed = parser.parse(sources[0]) + all_entities.extend(ner.extract(parsed)) + all_rels.extend(rel.extract(parsed, entities=all_entities)) + +kg = builder.build(entities=all_entities, relationships=all_rels) +print(f"Unified graph: {len(kg.nodes)} nodes, {len(kg.edges)} edges") +``` + +### Conflict Detection and Resolution + +```python +from semantica.conflicts import ConflictDetector, ConflictResolver -# Detect and resolve conflicts detector = ConflictDetector() conflicts = detector.detect_conflicts(all_entities) resolver = ConflictResolver(default_strategy="voting") resolved = resolver.resolve_conflicts(conflicts) -print(f"Detected {len(conflicts)} conflicts") -print(f"Resolved {len(resolved)} conflicts") +print(f"Detected {len(conflicts)} conflicts, resolved {len(resolved)}") ``` -### Example 5: Custom Entity Extraction Configuration - -**Difficulty**: Intermediate - -Use custom configuration for entity extraction with specific models and thresholds. - -```python -from semantica.semantic_extract import NERExtractor -from semantica.kg import GraphBuilder - -# Use LLM-based extraction with custom configuration -ner = NERExtractor( - method="llm", - provider="openai", - model="gpt-4", - confidence_threshold=0.8, - temperature=0.0 -) - -text = "Your document text here..." -entities = ner.extract_entities(text) - -# Build graph with custom merge settings -builder = GraphBuilder( - merge_entities=True, - merge_threshold=0.9 -) -kg = builder.build_graph(entities=entities, relationships=[]) -``` - -### Example 6: Incremental Graph Building - -**Difficulty**: Intermediate - -Build knowledge graph incrementally from multiple sources. - -```python -from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor, RelationExtractor -from semantica.kg import GraphBuilder, GraphMerger - -def build_kg_from_source(source_path): - """Helper function to build a knowledge graph from a single source.""" - ingestor = FileIngestor() - parser = DocumentParser() - ner = NERExtractor() - rel_extractor = RelationExtractor() - - doc = ingestor.ingest_file(source_path) - parsed = parser.parse_document(source_path) - text = parsed.get("full_text", "") - - entities = ner.extract_entities(text) - relationships = rel_extractor.extract_relations(text, entities=entities) - - builder = GraphBuilder() - return builder.build_graph(entities=entities, relationships=relationships) - -# Build graphs separately -kg1 = build_kg_from_source("source1.pdf") -kg2 = build_kg_from_source("source2.pdf") - -# Merge into unified graph -merger = GraphMerger() -merged_kg = merger.merge([kg1, kg2]) - -print(f"Merged graph: {len(merged_kg.nodes)} nodes, {len(merged_kg.edges)} edges") -``` - ---- - -## Advanced Patterns (30+ min examples) - -### Example 7: Graph Visualization - -**Difficulty**: Beginner - -Visualize your knowledge graph to understand entity relationships. - -```python -from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor, RelationExtractor -from semantica.kg import GraphBuilder -from semantica.visualization import KGVisualizer - -# Build a small graph -ingestor = FileIngestor() -parser = DocumentParser() -ner = NERExtractor() -rel_extractor = RelationExtractor() - -doc = ingestor.ingest_file("semantica_intro.pdf") -parsed = parser.parse_document("semantica_intro.pdf") -text = parsed.get("full_text", "") - -entities = ner.extract_entities(text) -relationships = rel_extractor.extract_relations(text, entities=entities) - -builder = GraphBuilder() -kg = builder.build_graph(entities=entities, relationships=relationships) - -# Visualize -viz = KGVisualizer() -viz.visualize_network(kg, output="html", file_path="semantica_knowledge_map.html") -print("Visualization saved to semantica_knowledge_map.html") -``` - ---- - -## Advanced Patterns (30+ min examples) - -### Example 8: Persistent Storage (Neo4j) - -**Difficulty**: Intermediate - -Store and query knowledge graphs in a persistent graph database. +### Persistent Storage (Neo4j) ```python from semantica.graph_store import GraphStore -# Initialize with Neo4j store = GraphStore( backend="neo4j", uri="bolt://localhost:7687", user="neo4j", - password="password" + password="password", ) store.connect() -# Create nodes and relationships -apple = store.create_node( - labels=["Company"], - properties={"name": "Apple Inc."} -) -tim = store.create_node( - labels=["Person"], - properties={"name": "Tim Cook"} -) +apple = store.create_node(labels=["Company"], properties={"name": "Apple Inc."}) +tim = store.create_node(labels=["Person"], properties={"name": "Tim Cook"}) store.create_relationship( start_node_id=tim["id"], end_node_id=apple["id"], - rel_type="CEO_OF" + rel_type="CEO_OF", ) - store.close() ``` -### Example 9: FalkorDB for Real-Time Applications - -**Difficulty**: Intermediate - -Ultra-fast graph queries for LLM applications using FalkorDB. +### FalkorDB (High-Speed Queries) ```python from semantica.graph_store import GraphStore @@ -309,105 +137,49 @@ store = GraphStore( backend="falkordb", host="localhost", port=6379, - graph_name="knowledge_graph" + graph_name="knowledge_graph", ) store.connect() - -# Fast queries -results = store.execute_query("MATCH (n)-[r]->(m) WHERE n.name CONTAINS 'AI' RETURN n") +results = store.execute_query( + "MATCH (n)-[r]->(m) WHERE n.name CONTAINS 'AI' RETURN n" +) store.close() ``` -### Example 10: GraphRAG (Knowledge-Powered Retrieval) +--- -**Difficulty**: Advanced +## Advanced -Build a production-ready GraphRAG system with logical inference and hybrid retrieval. +### GraphRAG with Reasoning ```python from semantica.context import AgentContext from semantica.reasoning import Reasoner -# 1. Initialize context with GraphRAG (Hybrid Retrieval) context = AgentContext( - vector_store=vs, + vector_store=vs, knowledge_graph=kg, graph_expansion=True, - hybrid_alpha=0.7 + hybrid_alpha=0.7, ) -# 2. Enrich Knowledge Graph using Logical Reasoning reasoner = Reasoner() - -# Add a rule to categorize technology stack items reasoner.add_rule("IF Library(?x) AND Language(?y) THEN TechStackItem(?x)") +inferred = reasoner.infer_facts(kg.get_all_triplets()) -# Infer new facts from the existing graph -all_facts = kg.get_all_triplets() -inferred = reasoner.infer_facts(all_facts) +for fact in inferred: + kg.add_fact_from_string(fact) -# Add inferred knowledge back to the graph -for fact_str in inferred: - kg.add_fact_from_string(fact_str) - -# 3. Retrieve context for a query (now with enriched knowledge) results = context.retrieve("What technologies are used in this project?") ``` -[**View Complete GraphRAG Tutorial**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) - -### Example 11: RAG vs. GraphRAG Comparison - -**Difficulty**: Intermediate - -Benchmark standard Vector RAG against Graph-enhanced retrieval. - -[**View RAG vs. GraphRAG Comparison**](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) +[Full GraphRAG tutorial](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) · [RAG vs. GraphRAG comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) --- -## Production Patterns +## Production -### Example 12: Streaming Data Processing - -**Difficulty**: Advanced - -Process data streams in real-time. - -```python -from semantica.ingest import StreamIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor, RelationExtractor -from semantica.kg import GraphBuilder - -stream_ingestor = StreamIngestor(stream_uri="kafka://localhost:9092/topic") -parser = DocumentParser() -ner = NERExtractor() -rel_extractor = RelationExtractor() -builder = GraphBuilder() - -for batch in stream_ingestor.stream(batch_size=100): - all_entities = [] - all_relationships = [] - - for item in batch: - text = str(item) # Convert stream item to text - entities = ner.extract_entities(text) - relationships = rel_extractor.extract_relations(text, entities=entities) - all_entities.extend(entities) - all_relationships.extend(relationships) - - # Build graph from batch - kg = builder.build_graph(entities=all_entities, relationships=all_relationships) - # Process results - print(f"Processed batch: {len(kg.nodes)} nodes") -``` - -### Example 13: Batch Processing Large Datasets - -**Difficulty**: Intermediate - -Process large datasets efficiently with batching. +### Batch Processing (Large Datasets) ```python from semantica.ingest import FileIngestor @@ -416,75 +188,57 @@ from semantica.semantic_extract import NERExtractor, RelationExtractor from semantica.kg import GraphBuilder ingestor = FileIngestor() -parser = DocumentParser() -ner = NERExtractor() -rel_extractor = RelationExtractor() -builder = GraphBuilder() +parser = DocumentParser() +ner = NERExtractor() +rel = RelationExtractor() +builder = GraphBuilder() -sources = [f"data/doc_{i}.pdf" for i in range(1000)] +sources = [f"data/doc_{i}.pdf" for i in range(1000)] batch_size = 50 for i in range(0, len(sources), batch_size): - batch = sources[i:i+batch_size] - - all_entities = [] - all_relationships = [] - - for source in batch: - doc = ingestor.ingest_file(source) - parsed = parser.parse_document(source) - text = parsed.get("full_text", "") - - entities = ner.extract_entities(text) - relationships = rel_extractor.extract_relations(text, entities=entities) - - all_entities.extend(entities) - all_relationships.extend(relationships) - - # Build graph from batch - kg = builder.build_graph(entities=all_entities, relationships=all_relationships) - - # Save intermediate results - print(f"Processed batch {i//batch_size + 1}: {len(kg.nodes)} nodes") + batch = sources[i : i + batch_size] + all_entities, all_rels = [], [] + + for path in batch: + parsed = parser.parse(ingestor.ingest(path)[0]) + all_entities.extend(ner.extract(parsed)) + all_rels.extend(rel.extract(parsed, entities=all_entities)) + + kg = builder.build(entities=all_entities, relationships=all_rels) + print(f"Batch {i // batch_size + 1}: {len(kg.nodes)} nodes") +``` + +### Real-Time Streaming + +```python +from semantica.ingest import StreamIngestor +from semantica.semantic_extract import NERExtractor, RelationExtractor +from semantica.kg import GraphBuilder + +stream = StreamIngestor(stream_uri="kafka://localhost:9092/topic") +ner = NERExtractor() +rel = RelationExtractor() +builder = GraphBuilder() + +for batch in stream.stream(batch_size=100): + all_entities, all_rels = [], [] + for item in batch: + text = str(item) + all_entities.extend(ner.extract(text)) + all_rels.extend(rel.extract(text, entities=all_entities)) + kg = builder.build(entities=all_entities, relationships=all_rels) + print(f"Processed batch: {len(kg.nodes)} nodes") ``` --- ## More Resources -- **[Quick Start Guide](quickstart.md)** - Step-by-step tutorial -- **[API Reference](reference/core.md)** - Complete API documentation -- **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks -- **[Use Cases](use-cases.md)** - Real-world applications - -### 🍳 Recommended Cookbook Tutorials - -- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules - - **Topics**: Framework overview, all modules, architecture, configuration - - **Difficulty**: Beginner - - **Time**: 30-45 minutes - - **Use Cases**: First-time users, understanding the framework - -- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph - - **Topics**: Entity extraction, relationship extraction, graph construction, visualization - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Learning the basics, quick start - -- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production-ready GraphRAG system - - **Topics**: GraphRAG, hybrid retrieval, vector search, graph traversal, LLM integration - - **Difficulty**: Advanced - - **Time**: 1-2 hours - - **Use Cases**: Production RAG applications - -- **[RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)**: Benchmark standard RAG vs GraphRAG - - **Topics**: RAG, GraphRAG, benchmarking, visualization, reasoning gap - - **Difficulty**: Intermediate - - **Time**: 45-60 minutes - - **Use Cases**: Understanding GraphRAG advantages, choosing the right approach - ---- - -!!! info "Contribute" - Have an example to share? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica) +- [Quickstart Tutorial](quickstart.md) — step-by-step first pipeline +- [Cookbook](cookbook.md) — interactive Jupyter notebooks +- [Use Cases](use-cases.md) — domain-specific examples +- [API Reference](reference/core.md) — complete API documentation +!!! info "Have an example to share?" + [Contribute on GitHub](https://github.com/Hawksight-AI/semantica) diff --git a/docs/faq.md b/docs/faq.md index cf96b220..ff5bfc31 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,148 +1,149 @@ -# Frequently Asked Questions +# Frequently Asked Questions -**Common questions about Semantica and how to use it.** +Common questions about Semantica. Use Ctrl+F to find what you need. --- ## General ### What is Semantica? -Semantica is an open-source framework for building knowledge graphs from unstructured data. It transforms documents, web pages, and databases into structured, queryable knowledge. -### What can I do with Semantica? -- **Build knowledge graphs** from documents and data -- **Extract entities and relationships** automatically -- **Power AI applications** with structured knowledge -- **Create semantic search** and GraphRAG systems -- **Integrate multiple data sources** into unified graphs +Semantica is an open-source framework for building context graphs and decision intelligence layers for AI. It transforms unstructured data — documents, APIs, databases — into structured knowledge graphs with full provenance tracking, making AI systems explainable and auditable. + +### What can I build with Semantica? + +- Knowledge graphs from documents and multi-source data +- GraphRAG systems with graph-grounded retrieval +- AI agents with structured decision history and semantic memory +- Compliance-ready pipelines with W3C PROV-O lineage + +### What makes Semantica different from other frameworks? + +Most frameworks stop at retrieval or generation. Semantica adds an **accountability layer**: every decision is recorded, every fact links to a source, and every reasoning step is explainable. It's designed for environments where you need to audit why an AI reached a conclusion. ### Is Semantica free? -Yes! Semantica is open source under the MIT License. -### What makes Semantica different? -- **Modular architecture** - Use only what you need -- **Production-ready** - Built for scale and reliability -- **Extensible** - Add custom models and components -- **Open source** - Transparent and community-driven +Yes — MIT licensed, no vendor lock-in. Some features require third-party API keys (e.g., OpenAI embeddings), but Semantica itself is free. --- ## Installation ### How do I install Semantica? + ```bash pip install semantica ``` +See [Installation](installation.md) for virtual environment setup, optional extras, and troubleshooting. + ### What Python version do I need? -Python 3.8 or higher. Python 3.11+ is recommended. + +Python 3.8 or higher. Python 3.11+ is recommended for best performance. ### What are the system requirements? + - Python 3.8+ -- 4GB+ RAM for basic use -- Optional GPU for embeddings and ML models +- 4 GB RAM minimum; 16 GB+ recommended for larger graphs +- Optional GPU for embedding generation and ML inference --- ## Getting Started -### How do I start using Semantica? -```python -from semantica.semantic_extract import NERExtractor -from semantica.kg import GraphBuilder +### Where do I start? -# Extract entities -ner = NERExtractor() -entities = ner.extract("Apple Inc. was founded by Steve Jobs.") +1. [Installation](installation.md) — get set up +2. [Getting Started](getting-started.md) — core concepts and first example +3. [Quickstart Tutorial](quickstart.md) — full step-by-step pipeline +4. [Cookbook](cookbook.md) — interactive Jupyter notebooks -# Build knowledge graph -kg = GraphBuilder().build({"entities": entities}) -``` +### What data sources does Semantica support? -### Where can I find examples? -- **[Getting Started Guide](getting-started.md)** - Quick introduction -- **[Cookbook](cookbook.md)** - Practical examples -- **[GitHub Examples](https://github.com/Hawksight-AI/semantica/tree/main/examples)** - Code samples +- **Files** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives +- **Web** — crawl with `WebIngestor`, RSS feeds +- **Databases** — PostgreSQL, MySQL, Snowflake via `DBIngestor` / `SnowflakeIngestor` +- **Streams** — Kafka, real-time ingestion +- **Media** — image OCR, audio/video metadata --- ## Features -### What data sources does Semantica support? -- **Files**: PDF, DOCX, TXT, JSON, CSV -- **Web**: Websites, RSS feeds, APIs -- **Databases**: PostgreSQL, MySQL, Snowflake, MongoDB -- **Streams**: Kafka, RabbitMQ, real-time data +### Can I use my own models? -### Can I use custom models? -Yes! Semantica supports custom: -- **Entity extraction models** -- **Embedding models** -- **Language models** -- **Custom processors** +Yes. Semantica supports custom entity extraction models, embedding models, LLM providers (via LiteLLM — 100+ models), and custom pipeline processors. ### Does Semantica support GPUs? -Yes, Semantica automatically uses GPUs when available for: -- **Embedding generation** -- **ML model inference** -- **Vector operations** + +Yes. When available, GPUs are used automatically for embedding generation, ML model inference, and vector operations. Install `semantica[gpu]` for CUDA support. + +### How does Semantica handle large datasets? + +- **Batching** — process documents in configurable chunks +- **Parallel processing** — `PipelineBuilder` supports configurable worker counts +- **Delta processing** — update graphs incrementally without full recompute +- **Graph backends** — swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE at scale --- ## Technical -### How does Semantica handle large datasets? -- **Batching** - Process data in chunks -- **Streaming** - Handle real-time data -- **Parallel processing** - Use multiple cores -- **Memory management** - Efficient resource usage +### What graph databases are supported? -### Can I deploy Semantica in production? -Yes! Semantica is production-ready with: -- **Scalable architecture** -- **Error handling** -- **Monitoring support** -- **Container deployment** +Neo4j, FalkorDB, Apache AGE (PostgreSQL), Amazon Neptune, and in-memory NetworkX for development. -### How do I customize Semantica? -- **Custom processors** - Add new extraction logic -- **Custom models** - Use your own ML models -- **Plugins** - Extend functionality -- **Configuration** - Adjust behavior +### What export formats are available? + +RDF (Turtle, JSON-LD, N-Triples, XML), Apache Parquet, ArangoDB AQL, CSV, YAML, and OWL ontologies. + +### Is Semantica production-ready? + +Yes. v0.3.0 ships with 886+ passing tests, `PipelineValidator`, `FailureHandler` with exponential backoff, W3C PROV-O provenance, and change management with checksums. See [What's New](index.md#whats-new-in-v030) for details. --- ## Troubleshooting -### Installation issues -- **Python version**: Ensure Python 3.8+ -- **Dependencies**: Install with `pip install -e .[dev]` -- **Permissions**: Use virtual environments +### Import error: `ModuleNotFoundError: No module named 'semantica'` -### Performance issues -- **Memory**: Increase available RAM -- **GPU**: Install CUDA for GPU acceleration -- **Batching**: Use smaller chunk sizes +Ensure you have the correct Python environment active, then: -### Common errors -- **Import errors**: Check installation path -- **Model loading**: Verify model availability -- **Memory errors**: Reduce batch sizes +```bash +pip list | grep semantica +pip install --upgrade semantica +``` + +### Installation fails with dependency errors + +```bash +pip install --upgrade pip wheel +pip install semantica +``` + +### Memory errors during processing + +Reduce batch sizes, enable streaming ingestion, or switch to a persistent graph backend (Neo4j, FalkorDB). + +### Slow embedding or inference + +Install GPU support (`pip install semantica[gpu]`) and ensure CUDA is available on your system. --- ## Support ### Where can I get help? -- **[GitHub Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report problems -- **[Discussions](https://github.com/Hawksight-AI/semantica/discussions)** - Ask questions -- **[Documentation](index.md)** - Browse guides and references -### How do I report bugs? -1. **Search** existing issues first -2. **Create** a new issue with details -3. **Include** reproduction steps -4. **Add** environment information +- [Discord](https://discord.gg/sV34vps5hH) — community chat and support +- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) — bug reports and feature requests +- [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) — questions and ideas -### Can I contribute? -Yes! See the [Contributing Guide](contributing.md) for details on how to help improve Semantica. +### How do I report a bug? + +1. Search [existing issues](https://github.com/Hawksight-AI/semantica/issues) first +2. Open a new issue with: description, reproduction steps, expected vs actual behavior, and your environment (Python version, OS, Semantica version) + +### How do I contribute? + +See the [Contributing Guide](contributing.md). diff --git a/docs/getting-started.md b/docs/getting-started.md index fecbc5ff..dcec03a9 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,14 +1,18 @@ # Getting Started -## Overview +**Semantica** is the context and intelligence layer for AI — turning raw data into explainable, auditable knowledge graphs for high-stakes domains. -**Semantica** is a semantic intelligence layer that bridges the gap between raw data and trustworthy AI. It transforms unstructured data into explainable, auditable knowledge graphs perfect for high-stakes domains. +!!! tip "Just here for code?" + Jump straight to the [Quick Start](#quick-start) or explore the [Cookbook](cookbook.md) for interactive notebooks. -### What You Can Build -- **GraphRAG Systems** - Enhanced retrieval with semantic reasoning -- **AI Agents** - Trustworthy agents with explainable memory -- **Knowledge Graphs** - Production-ready semantic databases -- **Compliance-Ready AI** - Auditable systems with full provenance +--- + +## What You Can Build + +- **GraphRAG Systems** — enhanced retrieval with semantic graph reasoning +- **AI Agents** — accountable agents with structured decision history and memory +- **Knowledge Graphs** — production-ready semantic knowledge bases +- **Compliance-Ready AI** — auditable systems with full W3C PROV-O provenance --- @@ -18,17 +22,17 @@ pip install semantica ``` -Or with all features: +With all optional dependencies: ```bash pip install semantica[all] ``` -Verify installation: +Verify: ```python import semantica -print(f"Semantica {semantica.__version__} installed!") +print(semantica.__version__) ``` --- @@ -36,66 +40,59 @@ print(f"Semantica {semantica.__version__} installed!") ## Quick Start ```python -from semantica.semantic_extract import NERExtractor -from semantica.kg import GraphBuilder +from semantica.context import AgentContext, ContextGraph +from semantica.vector_store import VectorStore -# Extract entities -ner = NERExtractor(method="ml", model="en_core_web_sm") -entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") +context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph(advanced_analytics=True), + decision_tracking=True, +) -# Build knowledge graph -kg = GraphBuilder().build({"entities": entities, "relationships": []}) -print(f"Built KG with {len(kg.get('entities', []))} entities") +# Store a memory +context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%") + +# Record a decision +decision_id = context.record_decision( + category="model_selection", + scenario="Choose LLM for production pipeline", + reasoning="GPT-4 benchmark advantage justifies cost increase", + outcome="selected_gpt4", + confidence=0.91, +) + +# Find similar past decisions +precedents = context.find_precedents("model selection", limit=5) ``` -**What this does:** -- Extracts entities (people, organizations, dates) from text -- Builds a knowledge graph from extracted entities -- Outputs the number of entities found - --- ## Core Architecture -Semantica uses a **modular architecture** - use only what you need: +Semantica uses a modular, layered architecture — import only what you need. -### 1️⃣ Input Layer - Data Ingestion -```python -from semantica.ingest import FileIngestor -documents = FileIngestor().ingest_directory("docs/") -``` - -### 2️⃣ Semantic Layer - Intelligence Engine -```python -from semantica.semantic_extract import NERExtractor, RelationExtractor -entities = NERExtractor().extract(text) -relationships = RelationExtractor().extract(text, entities) -``` - -### 3️⃣ Output Layer - Knowledge Assets -```python -from semantica.kg import GraphBuilder -kg = GraphBuilder().build_graph(entities, relationships) -``` +| Layer | Modules | Purpose | +|-------|---------|---------| +| **Input** | `ingest`, `parse`, `split`, `normalize` | Load and prepare data | +| **Semantic** | `semantic_extract`, `kg`, `ontology`, `reasoning` | Extract meaning | +| **Storage** | `embeddings`, `vector_store`, `graph_store` | Persist knowledge | +| **Quality** | `deduplication`, `conflicts` | Validate and clean | +| **Context** | `context`, `provenance`, `change_management` | Track decisions and lineage | +| **Output** | `export`, `visualization`, `pipeline` | Deliver results | --- ## Next Steps -### 🍳 Interactive Tutorials -1. **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)** - Complete framework overview -2. **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)** - Hands-on graph building -3. **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)** - Production-ready RAG - -### 📚 Learn More -- **[Core Concepts](concepts.md)** - Deep dive into knowledge graphs & ontologies -- **[Cookbook](cookbook.md)** - 14 domain-specific tutorials -- **[API Reference](reference/core.md)** - Complete technical documentation +- [Core Concepts](concepts.md) — knowledge graphs, ontologies, reasoning explained +- [Quickstart Tutorial](quickstart.md) — build a full pipeline step by step +- [Cookbook](cookbook.md) — 14 domain-specific Jupyter notebook tutorials +- [API Reference](reference/core.md) — complete module documentation --- -## Need Help? +## Help -- **[💬 Discord Community](https://discord.gg/sV34vps5hH)** - Get help from the community -- **[🐛 Issues](https://github.com/Hawksight-AI/semantica/issues)** - Report bugs or request features -- **[📖 Documentation](https://semantica.readthedocs.io/)** - Full documentation site +- [Discord Community](https://discord.gg/sV34vps5hH) — ask questions, share projects +- [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) — report bugs or request features +- [FAQ](faq.md) — common questions answered diff --git a/docs/glossary.md b/docs/glossary.md index e065699d..4b42ad31 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -1,9 +1,9 @@ # Glossary -**Comprehensive reference of terms and concepts used in Semantica and semantic intelligence.** +Reference of terms and concepts used throughout Semantica. -!!! tip "Quick Reference" - Looking for a specific term? Use your browser's search function (Ctrl+F) to find terms quickly. +!!! tip "Finding a term" + Use Ctrl+F to search this page. --- @@ -216,17 +216,7 @@ W3C PROV-O compliant tracking of data lineage and source attribution. ## See Also -- **[Core Concepts](concepts.md)** - Deep dive into fundamental concepts -- **[Getting Started](getting-started.md)** - Begin your journey with Semantica -- **[Modules Guide](modules.md)** - Complete module overview -- **[API Reference](reference/)** - Technical documentation - ---- - -## Need Help? - -- **Documentation**: [Getting Started](getting-started.md) -- **Examples**: [Cookbook](cookbook.md) -- **Community**: [Discord](community.md) -- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) -- **Support**: [Contact Us](community.md) +- [Core Concepts](concepts.md) — deeper explanation of key ideas +- [Getting Started](getting-started.md) — first steps +- [Modules Guide](modules.md) — every module explained +- [API Reference](reference/) — technical reference diff --git a/docs/index.md b/docs/index.md index b67a3906..63a0ecf7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,24 +1,23 @@
- Semantica Logo - + Semantica Logo +

🧠 Semantica

- + Python 3.8+ License: MIT - PyPI version - Monthly Downloads + PyPI + Version Total Downloads - Documentation - Discord - -

Open-Source Semantic Layer & Knowledge Engineering Framework

- -

Transform Chaos into Intelligence. Build AI systems that are explainable, traceable, and trustworthy — not black boxes.

- -

The semantic intelligence layer that makes your AI agents auditable, explainable, and trustworthy. Perfect for high-stakes domains where mistakes have real consequences.

- -

🆓 Open Source • 📜 MIT Licensed • 🚀 Production Ready • 🌍 Community Driven

- + CI + Discord + X + +

A Framework for Building Context Graphs and Decision Intelligence Layers for AI

+ +

⭐ Give us a Star  •  🍴 Fork us  •  💬 Join our Discord  •  🐦 Follow on X

+ +

Transform Chaos into Intelligence. Build AI systems with context graphs, decision tracking, and advanced knowledge engineering that are explainable, traceable, and trustworthy — not black boxes.

+

Get Started View on GitHub @@ -27,288 +26,266 @@ --- -## 🚀 Why Semantica? +## The Problem -**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and trustworthy. +AI agents today are capable but not trustworthy: -Perfect for **high-stakes domains** where mistakes have real consequences. +- **No memory structure** — agents store embeddings, not meaning. Retrieval is fuzzy; there's no way to ask *why* something was recalled. +- **No decision trail** — agents make decisions continuously but record nothing. When something goes wrong, there's no history to debug or audit. +- **No provenance** — outputs cannot be traced back to source facts. In regulated industries, this is a compliance blocker. +- **No reasoning transparency** — black-box answers with no explanation of how a conclusion was reached. +- **No conflict detection** — contradictory facts silently coexist in vector stores, producing unpredictable answers. + +These aren't edge cases. They are the reason AI cannot be deployed in healthcare, finance, legal, and government without custom guardrails built from scratch. --- -### ⚡ Get Started in 30 Seconds +## The Solution + +Semantica is the **context and intelligence layer** you add to your AI stack: + +- **Context Graphs** — structured graph of entities, relationships, and decisions your agent builds as it works. Queryable, traceable, persistent. +- **Decision Intelligence** — every decision is a first-class object: recorded, linked causally, searchable by precedent, and analyzable for downstream impact. +- **Provenance** — every fact links to its source. W3C PROV-O compliant. Full lineage from ingestion to inference. +- **Reasoning engines** — forward chaining, Rete networks, deductive, abductive, and SPARQL reasoning. Explainable inference paths, not black-box answers. +- **Deduplication & QA** — conflict detection, entity resolution, and validation built into the pipeline. + +Works alongside LangChain, LlamaIndex, AutoGen, CrewAI, and any LLM provider — Semantica is not a replacement, it's the accountability layer on top. + +--- + +### ⚡ Quick Installation ```bash pip install semantica ``` ```python -from semantica.semantic_extract import NERExtractor -from semantica.kg import GraphBuilder +from semantica.context import AgentContext, ContextGraph +from semantica.vector_store import VectorStore -# Extract entities and build knowledge graph -ner = NERExtractor(method="ml", model="en_core_web_sm") -entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") -kg = GraphBuilder().build({"entities": entities, "relationships": []}) +context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph(advanced_analytics=True), + decision_tracking=True, +) -print(f"Built KG with {len(kg.get('entities', []))} entities") +# Store a memory +context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%") + +# Record a decision +decision_id = context.record_decision( + category="model_selection", + scenario="Choose LLM for production reasoning pipeline", + reasoning="GPT-4 benchmark advantage justifies 3x cost increase", + outcome="selected_gpt4", + confidence=0.91, +) + +# Find similar past decisions and analyze downstream impact +precedents = context.find_precedents("model selection reasoning", limit=5) +influence = context.analyze_decision_influence(decision_id) ``` -**[📖 Full Quick Start](getting-started.md)** • **[🍳 Cookbook Examples](cookbook.md)** • **[💬 Join Discord](https://discord.gg/sV34vps5hH)** • **[⭐ Star Us](https://github.com/Hawksight-AI/semantica)** +**[Full Quick Start](getting-started.md)**  •  **[Cookbook](cookbook.md)**  •  **[Join Discord](https://discord.gg/sV34vps5hH)** + +--- + +## What's New in v0.3.0 + +> First stable release (`Production/Stable` on PyPI). + +| Area | Highlights | +|------|------------| +| **Context Graphs** | Temporal validity windows, weighted BFS, cross-graph navigation with save/load persistence | +| **Decision Intelligence** | Full lifecycle: record → trace → impact → precedent; `PolicyEngine` with versioned rules | +| **KG Algorithms** | PageRank, betweenness, Louvain community detection, Node2Vec, link prediction | +| **Semantic Extraction** | LLM extraction fixed (no silent drops), duplicate relation bug removed, `"llm_typed"` metadata corrected | +| **Deduplication v2** | `blocking_v2`/`hybrid_v2` — 63.6% faster; semantic v2 — 6.98x faster | +| **Delta Processing** | SPARQL-based incremental diff, `delta_mode` pipelines, snapshot versioning | +| **Export** | RDF aliases (`"ttl"`, `"json-ld"`), ArangoDB AQL, Apache Parquet (Spark/BigQuery/Databricks) | +| **Pipeline** | `FailureHandler` with LINEAR/EXPONENTIAL/FIXED backoff; `PipelineValidator` returning `ValidationResult` | +| **Graph Backends** | Apache AGE (SQL injection fixed), AWS Neptune, FalkorDB, PgVector (HNSW/IVFFlat) | +| **Tests** | 886+ passing, 0 failures — 335 context, ~430 KG, 70 semantic extraction, 85 real-world E2E | --- ## Core Value Proposition | **Trustworthy** | **Explainable** | **Auditable** | -|:------------------:|:------------------:|:-----------------:| +|:---:|:---:|:---:| | Conflict detection & validation | Transparent reasoning paths | Complete provenance tracking | | Rule-based governance | Entity relationships & ontologies | W3C PROV-O compliant lineage | | Production-grade QA | Multi-hop graph reasoning | Source tracking & integrity verification | --- -## Key Features & Benefits +## Features -### Not Just Another Agentic Framework +### Context & Decision Intelligence +- **Context Graphs** — structured, persistent graph of entities, relationships, and decisions +- **Decision tracking** — `add_decision()`, `record_decision()` for full lifecycle management +- **Causal chains** — `add_causal_relationship()`, `trace_decision_chain()` +- **Precedent search** — hybrid similarity search over past decisions via `find_similar_decisions()` +- **Influence analysis** — `analyze_decision_impact()`, `analyze_decision_influence()` +- **Policy engine** — `check_decision_rules()` with versioned, automated compliance rules +- **Agent memory** — `AgentMemory` with short/long-term storage and conversation history -**Semantica complements** LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK, Agno, and other frameworks to enhance your agents with: +### Knowledge Graphs +- **Graph construction** — entities, relationships, properties, typed edges +- **Algorithms** — PageRank, betweenness centrality, clustering coefficient, community detection +- **Node embeddings** — Node2Vec via `NodeEmbedder`; cosine similarity via `SimilarityCalculator` +- **Link prediction** — score potential edges via `LinkPredictor` +- **Temporal graphs** — time-aware nodes and edges with validity windows +- **Delta processing** — incremental updates without full recompute -| Feature | Benefit | -|:--------|:--------| -| **Auditable** | Complete provenance tracking with W3C PROV-O compliance | -| **Explainable** | Transparent reasoning paths with entity relationships | -| **Provenance-Aware** | End-to-end lineage from documents to responses | -| **Validated** | Built-in conflict detection, deduplication, QA | -| **Governed** | Rule-based validation and semantic consistency | -| **Version Control** | Enterprise-grade change management with integrity verification | +### Semantic Extraction +- **NER** — named entity recognition, normalization, classification +- **Relation extraction** — triplet generation via LLMs or rule-based methods, with `"llm_typed"` metadata +- **Deduplication v1/v2** — Jaro-Winkler, `blocking_v2`, `hybrid_v2`, `semantic_v2`; `dedup_triplets()` for triples -### Perfect For High-Stakes Use Cases +### Reasoning +- **Forward chaining** — `Reasoner` with IF/THEN string rules and dict facts +- **Rete network** — `ReteEngine` for high-throughput production rule matching +- **Deductive / Abductive** — `DeductiveReasoner`, `AbductiveReasoner` +- **SPARQL** — `SPARQLReasoner` for query-based inference over RDF graphs -| 🏥 **Healthcare** | 💰 **Finance** | ⚖️ **Legal** | -|:-----------------:|:--------------:|:------------:| -| Clinical decisions | Fraud detection | Evidence-backed research | -| Drug interactions | Regulatory support | Contract analysis | -| Patient safety | Risk assessment | Case law reasoning | +### Provenance & Auditability +- **Entity provenance** — `ProvenanceTracker.track_entity()` +- **Algorithm provenance** — `AlgorithmTrackerWithProvenance` +- **W3C PROV-O compliant** — lineage tracking across all modules +- **Change management** — version control with checksums, audit trails, compliance support -| 🔒 **Cybersecurity** | 🏛️ **Government** | 🏭 **Infrastructure** | 🚗 **Autonomous** | -|:-------------------:|:----------------:|:-------------------:|:-----------------:| -| Threat attribution | Policy decisions | Power grids | Decision logs | -| Incident response | Classified info | Transportation | Safety validation | +### Vector Store +- **Backends** — FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory +- **Search modes** — semantic top-k, hybrid (vector + keyword), metadata-filtered -### Powers Your AI Stack +### Data Ingestion +- **Files** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives +- **Sources** — web crawl, SQL databases, Snowflake, feeds, email, repositories +- **Docling** — advanced parsing with table and layout extraction +- **Media** — image OCR, audio/video metadata -- **GraphRAG Systems** — Retrieval with graph reasoning and hybrid search -- **AI Agents** — Trustworthy, accountable multi-agent systems with semantic memory -- **Reasoning Models** — Explainable AI decisions with reasoning paths -- **Enterprise AI** — Governed, auditable platforms that support compliance +### Export +- **RDF** — Turtle, JSON-LD, N-Triples, XML via `RDFExporter` +- **Parquet** — `ParquetExporter` for Spark/BigQuery/Databricks pipelines +- **ArangoDB AQL** — ready-to-run INSERT statements +- **OWL ontologies** — Turtle or RDF/XML -### Integrations - -- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX) -- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication -- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD) - -> **Built for environments where every answer must be explainable and governed.** +### Pipeline & Ontology +- **Pipeline DSL** — `PipelineBuilder` with stage chaining, parallel workers, retry policies +- **Ontology** — auto-generate OWL from KGs, import OWL/RDF/Turtle/JSON-LD, HermiT/Pellet validation --- -## 🚨 The Problem: The Semantic Gap +## Modules -### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**. - -### Understanding the Semantic Gap - -The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships). - -**Traditional AI approaches:** -- Rely on statistical patterns and text similarity -- Cannot understand relationships between entities -- Cannot reason about domain-specific rules -- Cannot explain why decisions were made -- Cannot trace back to original sources with confidence - -**High-stakes AI requires:** -- Semantic understanding of entities and their relationships -- Domain knowledge encoded as formal rules (ontologies) -- Explainable reasoning paths -- Source-level provenance -- Conflict detection and resolution - -**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge. - -### What Organizations Have vs What They Need - -| **Current State** | **Required for High-Stakes AI** | -|:---------------------|:-----------------------------------| -| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) | -| APIs, databases, streams | Structured and validated entities | -| Conflicting facts and duplicates | Explicit semantic relationships | -| Siloed systems with no lineage | **Explainable reasoning paths** | -| | **Source-level provenance** | -| | **Audit-ready compliance** | - -### The Cost of Missing Semantics - -- **Decisions cannot be explained** — No transparency in AI reasoning -- **Errors cannot be traced** — No way to debug or improve -- **Conflicts go undetected** — Contradictory information causes failures -- **Compliance becomes impossible** — No audit trails for regulations - -**Trustworthy AI requires semantic accountability.** +| Module | What it provides | +|--------|-----------------| +| `semantica.context` | Context graphs, agent memory, decision tracking, causal analysis, precedent search, policy engine | +| `semantica.kg` | KG construction, graph algorithms, centrality, community detection, embeddings, link prediction | +| `semantica.semantic_extract` | NER, relation extraction, event extraction, coreference, triplet generation, LLM extraction | +| `semantica.reasoning` | Forward chaining, Rete network, deductive, abductive, SPARQL reasoning, explanation generation | +| `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector; hybrid & filtered search | +| `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, YAML, OWL, graph formats | +| `semantica.ingest` | Files, web crawl, feeds, databases, Snowflake, MCP, email, repositories | +| `semantica.ontology` | Auto-generation, OWL/RDF export, import, validation, versioning | +| `semantica.pipeline` | Pipeline DSL, parallel workers, validation, retry policies, failure handling | +| `semantica.graph_store` | Neo4j, FalkorDB, Apache AGE, Amazon Neptune; Cypher queries | +| `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE; similarity calculation | +| `semantica.deduplication` | Entity deduplication, similarity scoring, merging, clustering | +| `semantica.provenance` | W3C PROV-O lineage tracking, source attribution, audit trails | +| `semantica.parse` | PDF, DOCX, PPTX, HTML, code, email, structured data, OCR | +| `semantica.split` | Recursive, semantic, entity-aware, relation-aware, graph-based chunking | +| `semantica.normalize` | Text, entities, dates, numbers, quantities, languages, encodings | +| `semantica.conflicts` | Multi-source conflict detection (value, type, temporal, logical) with resolution | +| `semantica.change_management` | Version storage, change tracking, checksums, audit trails | +| `semantica.triplet_store` | Blazegraph, Jena, RDF4J; SPARQL queries and bulk loading | +| `semantica.visualization` | Interactive/static KG, ontology, embedding, and temporal graph visualization | +| `semantica.core` | Framework orchestration, configuration, plugin system | +| `semantica.llms` | Groq, OpenAI, Novita AI, HuggingFace, LiteLLM integrations | --- -## 🆚 Semantica vs Traditional RAG +## Built for High-Stakes Domains -| Feature | Traditional RAG | Semantica | -|:--------|:----------------|:----------| -| **Reasoning** | ❌ Black-box answers | ✅ Explainable reasoning paths | -| **Provenance** | ❌ No provenance | ✅ W3C PROV-O compliant lineage tracking | -| **Search** | ⚠️ Vector similarity only | ✅ Semantic + graph reasoning | -| **Quality** | ❌ No conflict handling | ✅ Explicit contradiction detection | -| **Safety** | ⚠️ Unsafe for high-stakes | ✅ Designed for governed environments | -| **Compliance** | ❌ No audit trails | ✅ Complete audit trails with integrity verification | +Where **every decision must be accountable** and **mistakes have real consequences**: + +- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interactions, patient safety +- **💰 Finance & Risk** — Fraud detection, SOX/GDPR/MiFID II compliance, risk assessment +- **⚖️ Legal & Compliance** — Evidence-backed research, contract analysis, regulatory tracking +- **🔒 Cybersecurity** — Threat attribution, incident response, security audit trails +- **🏛️ Government & Defense** — Policy decisions, classified information handling, defense intelligence +- **🏭 Critical Infrastructure** — Power grids, transportation safety, emergency response +- **🚗 Autonomous Systems** — Self-driving, robotics safety, industrial automation --- -## 🧩 Semantica Architecture - -### 1️⃣ Input Layer — Governed Ingestion -- 📄 **Multiple Formats** — PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX -- 🔧 **Docling Support** — Docling parser for table extraction -- 💾 **Data Sources** — Databases, APIs, streams, archives, web content -- 🎨 **Media Support** — Image parsing with OCR, audio/video metadata extraction -- � **Single Pipeline** — Unified ingestion with metadata and source tracking - -### 2️⃣ Semantic Layer — Trust & Reasoning Engine -- 🔍 **Entity Extraction** — NER, normalization, classification -- 🔗 **Relationship Discovery** — Triplet generation, semantic links -- 📐 **Ontology Induction** — Automated domain rule generation -- 🔄 **Deduplication** — Jaro-Winkler similarity, conflict resolution -- ✅ **Quality Assurance** — Conflict detection, validation -- 📊 **Provenance Tracking** — W3C PROV-O compliant lineage tracking across all modules -- 🧠 **Reasoning Traces** — Explainable inference paths -- 🔐 **Change Management** — Version control with audit trails, checksums, compliance support - -### 3️⃣ Output Layer — Auditable Knowledge Assets -- � **Knowledge Graphs** — Queryable, temporal, explainable -- 📐 **OWL Ontologies** — HermiT/Pellet validated, custom ontology import support -- 🔢 **Vector Embeddings** — FastEmbed by default -- ☁️ **AWS Neptune** — Amazon Neptune graph database support -- 🔍 **Provenance** — Every AI response links back to: - - 📄 Source documents - - 🏷️ Extracted entities & relations - - 📐 Ontology rules applied - - 🧠 Reasoning steps used - ---- - -## 🏥 Built for High-Stakes Domains - -Designed for domains where **mistakes have real consequences** and **every decision must be accountable**: - -- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety tracking -- **💰 Finance & Risk** — Fraud detection, regulatory support (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation -- **⚖️ Legal & Compliance** — Evidence-backed legal research, contract analysis, regulatory change tracking, case law reasoning -- **🔒 Cybersecurity & Intelligence** — Threat attribution, incident response, security audit trails, intelligence analysis -- **🏛️ Government & Defense** — Governed AI systems, policy decisions, classified information handling, defense intelligence -- **🏭 Critical Infrastructure** — Power grid management, transportation safety, water treatment, emergency response -- **🚗 Autonomous Systems** — Self-driving vehicles, drone navigation, robotics safety, industrial automation - ---- - -## � Who Uses Semantica? - -- **🤖 AI / ML Engineers** — Building explainable GraphRAG & agents -- **⚙️ Data Engineers** — Creating governed semantic pipelines -- **📊 Knowledge Engineers** — Managing ontologies & KGs at scale -- **🏢 Enterprise Teams** — Requiring trustworthy AI infrastructure -- **🛡️ Risk & Compliance Teams** — Needing audit-ready systems - ---- - -## 🚀 Choose Your Path +## Choose Your Path

- :material-rocket-launch: **Quick Start** --- - Get up and running with Semantica in minutes. Learn the basics of ingestion and extraction. - + Up and running in minutes. + [:arrow_right: Start Here](getting-started.md) - :material-book-open-page-variant: **Core Concepts** --- - Deep dive into Knowledge Graphs, Ontologies, and Semantic Reasoning. - + Knowledge graphs, ontologies, and semantic reasoning explained. + [:arrow_right: Learn Concepts](concepts.md) - :material-code-braces: **API Reference** --- - Detailed technical documentation for all Semantica modules and classes. - + Full technical documentation for every module and class. + [:arrow_right: View API](reference/core.md) - :material-chef-hat: **Cookbook** --- - Interactive tutorials, real-world examples, and **14 domain-specific cookbooks**. - + 14 domain-specific cookbooks with real-world examples. + [:arrow_right: Explore Cookbook](cookbook.md)
--- -## 📦 Installation +## Installation !!! success "Now Available on PyPI!" - Semantica is officially published on PyPI! Install it with a single command. + Install with a single command. -=== "From PyPI (Recommended)" - - Install Semantica directly from PyPI: +=== "PyPI (Recommended)" ```bash - # Install the core package pip install semantica - # Or install with all optional dependencies + # With all optional dependencies pip install semantica[all] ``` === "From Source" - Install from the local source for the latest development version: - ```bash - # Clone the repository git clone https://github.com/Hawksight-AI/semantica.git cd semantica - - # Install in editable mode with core dependencies - pip install -e . - - # Or install with all optional dependencies - pip install -e ".[all]" + pip install -e . # core + pip install -e ".[all]" # all extras ``` === "Development" - For contributors who want to modify the framework: - ```bash - # Clone the repository git clone https://github.com/Hawksight-AI/semantica.git cd semantica - - # Install in editable mode with dev dependencies pip install -e ".[dev]" ``` === "Docker" - Run Semantica in a containerized environment: - ```bash docker pull semantica/semantica:latest docker run -it semantica/semantica @@ -316,117 +293,41 @@ Designed for domains where **mistakes have real consequences** and **every decis --- -## 🚦 Quick Example - -Semantica uses a modular architecture. You can use individual modules directly for maximum flexibility: - -```python -from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor, RelationExtractor -from semantica.kg import GraphBuilder - -# 1. Ingest documents -ingestor = FileIngestor() -documents = ingestor.ingest_directory("documents/", recursive=True) - -# 2. Parse documents -parser = DocumentParser() -parsed_docs = [parser.parse_document(doc) for doc in documents] - -# 3. Extract entities and relationships -ner = NERExtractor() -rel_extractor = RelationExtractor() - -entities = [] -relationships = [] -for doc in parsed_docs: - text = doc.get("full_text", "") - doc_entities = ner.extract_entities(text) - doc_rels = rel_extractor.extract_relations(text, entities=doc_entities) - entities.extend(doc_entities) - relationships.extend(doc_rels) - -# 4. Build knowledge graph -builder = GraphBuilder(merge_entities=True) -kg = builder.build_graph(entities=entities, relationships=relationships) - -print(f"Created graph with {len(kg.nodes)} nodes and {len(kg.edges)} edges") -``` - -!!! tip "Orchestration Option" - For complex workflows, you can also use the `Semantica` class for orchestration. See the [Core Module](reference/core.md) documentation for details. - ---- - -## 🎯 Why Semantica? +## Why Semantica?
- **🆓 Open Source** --- - MIT licensed. No vendor lock-in. Full transparency. + MIT licensed. No vendor lock-in. - **🚀 Production Ready** --- - Battle-tested with quality assurance, conflict resolution, and validation. + Battle-tested with QA, conflict resolution, and validation built in. -- **🧩 Modular Architecture** +- **🧩 Modular** --- Use only what you need. Swap components easily. - **🌍 Community Driven** --- - Built by developers, for developers. Active Discord community. + Built by developers, for developers. Active Discord. -- **📚 Comprehensive** +- **📚 End-to-End** --- - End-to-end solution from ingestion to reasoning. No duct-taping required. + From ingestion to reasoning — no duct-taping required. - **🔬 Research-Backed** --- - Based on latest research in knowledge graphs, ontologies, and semantic web. + Grounded in knowledge graph, ontology, and semantic web research.
--- -## 🏗️ Built For - -- **Data Scientists**: Transform messy data into clean knowledge graphs -- **Data Engineers**: Build scalable data pipelines with semantic enrichment -- **AI Engineers**: Build GraphRAG, AI agents, and multi-agent systems -- **Knowledge Engineers**: Generate and manage formal ontologies -- **Ontologists**: Design and validate domain-specific ontologies and taxonomies -- **Researchers**: Analyze scientific literature and build citation networks -- **ML Engineers**: Create semantic features for machine learning models -- **Enterprises**: Unify data silos into a semantic layer - ---- - -## 📚 Learn More - -- [Getting Started Guide](getting-started.md) - Your first knowledge graph in 5 minutes -- [Core Concepts](concepts.md) - Deep dive into knowledge graphs and ontologies -- [Cookbook](cookbook.md) - Real-world examples and **14 domain-specific cookbooks** -- [API Reference](reference/core.md) - Complete technical documentation - -### 🍳 Recommended Cookbook Tutorials - -Get hands-on with interactive Jupyter notebooks: - -- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all Semantica modules - - **Topics**: Framework overview, all modules, architecture - - **Difficulty**: Beginner - - **Use Cases**: First-time users, understanding the framework - -- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph from scratch - - **Topics**: Entity extraction, relationship extraction, graph construction - - **Difficulty**: Beginner - - **Use Cases**: Learning the basics, quick start - -- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production-ready Graph Retrieval Augmented Generation - - **Topics**: GraphRAG, hybrid retrieval, vector search, graph traversal - - **Difficulty**: Advanced - - **Use Cases**: Building AI applications with knowledge graphs +## Learn More +- [Getting Started](getting-started.md) — your first knowledge graph in 5 minutes +- [Core Concepts](concepts.md) — knowledge graphs, ontologies, and semantic reasoning +- [Cookbook](cookbook.md) — 14 domain-specific cookbooks with Jupyter notebooks +- [API Reference](reference/core.md) — complete technical documentation diff --git a/docs/installation.md b/docs/installation.md index d786c39a..844eaeb0 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,293 +1,169 @@ # Installation -Get Semantica up and running in minutes. +Get Semantica installed in under a minute. -!!! success "Now Available on PyPI!" - Semantica is officially published on PyPI! Install it with a single command: `pip install semantica` +!!! success "Available on PyPI" + `pip install semantica` — that's it. -!!! note "System Requirements" - Semantica requires Python 3.8 or higher. For best performance, we recommend Python 3.10+. +!!! note "Requirements" + Python 3.8 or higher. Python 3.11+ recommended. -## Prerequisites - -Before installing Semantica, ensure you have: - -- **Python 3.8 or higher** - Check your version: - ```bash - python --version - ``` -- **pip** - Python package installer (usually comes with Python) +--- ## Basic Installation -Install Semantica from PyPI: - ```bash pip install semantica ``` -This installs Semantica with all core dependencies. - -### GitHub Workaround - -If you encounter issues with the PyPI version, you can install directly from the main branch: - -```bash -pip install git+https://github.com/Hawksight-AI/semantica.git@main -``` - -!!! tip "Virtual Environment" - We recommend installing Semantica in a virtual environment to avoid dependency conflicts. Use `python -m venv venv` to create one, then activate it before installing. - -## Verify Installation - -Verify that Semantica is installed correctly: - -```bash -python -c "from semantica.parse import DoclingParser; DoclingParser(); print('✓ Semantica ready')" -``` - -!!! info "Windows PyTorch Note" - If you encounter PyTorch DLL errors on Windows, ensure you have the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) installed. This is a common environment-specific issue with PyTorch on Windows and not a bug in Semantica. - -Expected output: -``` -✓ Semantica ready -``` - -You can also check the installation: - -```bash -pip show semantica -``` - -## Development Installation - -To install Semantica in development mode (for contributing): - -```bash -# Clone the repository -git clone https://github.com/Hawksight-AI/semantica.git -cd semantica - -# Install in editable mode -pip install -e . - -# Or install with development dependencies -pip install -e ".[dev]" -``` - -## Optional Dependencies - -Semantica supports optional features that can be installed separately: - -### GPU Support - -For GPU-accelerated operations: - -```bash -pip install semantica[gpu] -``` - -This includes: -- PyTorch with CUDA support -- FAISS GPU -- CuPy - -### Visualization - -For enhanced visualization capabilities: - -```bash -pip install semantica[viz] -``` - -Includes: -- PyVis for interactive graphs -- Graphviz for static diagrams -- UMAP for dimensionality reduction - -### LLM Providers - -Install all LLM provider integrations: - -```bash -pip install semantica[llm-all] -``` - -Or install specific providers: - -```bash -# OpenAI -pip install semantica[llm-openai] - -# Anthropic -pip install semantica[llm-anthropic] - -# Google Gemini -pip install semantica[llm-gemini] - -# Groq -pip install semantica[llm-groq] - -# Ollama -pip install semantica[llm-ollama] -``` - -### Cloud Integrations - -For cloud storage and deployment: - -```bash -pip install semantica[cloud] -``` - -Includes: -- AWS S3 (boto3) -- Azure Blob Storage -- Google Cloud Storage -- Kubernetes support - -### All Optional Features - -Install everything: +With all optional dependencies: ```bash pip install semantica[all] ``` -## Virtual Environment (Recommended) +### Verify -It's recommended to use a virtual environment: +```bash +python -c "import semantica; print(semantica.__version__)" +``` + +--- + +## Virtual Environment (Recommended) === "venv" ```bash - # Create virtual environment python -m venv venv - - # Activate (Windows) - venv\Scripts\activate - - # Activate (Linux/Mac) - source venv/bin/activate - - # Install Semantica + source venv/bin/activate # Linux / Mac + venv\Scripts\activate # Windows pip install semantica ``` === "conda" ```bash - # Create conda environment conda create -n semantica python=3.11 conda activate semantica - - # Install Semantica pip install semantica ``` +--- + +## Optional Dependencies + +Install only what you need: + +=== "GPU" + + ```bash + pip install semantica[gpu] + ``` + Includes PyTorch with CUDA, FAISS GPU, CuPy. + +=== "Visualization" + + ```bash + pip install semantica[viz] + ``` + Includes PyVis, Graphviz, UMAP. + +=== "LLM Providers" + + ```bash + pip install semantica[llm-all] # all providers + + pip install semantica[llm-openai] # OpenAI + pip install semantica[llm-anthropic] # Anthropic + pip install semantica[llm-gemini] # Google Gemini + pip install semantica[llm-groq] # Groq + pip install semantica[llm-ollama] # Ollama (local) + ``` + +=== "Cloud" + + ```bash + pip install semantica[cloud] + ``` + Includes AWS S3, Azure Blob, Google Cloud Storage. + +--- + +## Install from Source + +For the latest development version or to contribute: + +```bash +git clone https://github.com/Hawksight-AI/semantica.git +cd semantica + +pip install -e . # core only +pip install -e ".[all]" # all extras +pip install -e ".[dev]" # dev tools (pytest, black, etc.) +``` + +If you encounter issues with the PyPI release, install directly from the main branch: + +```bash +pip install git+https://github.com/Hawksight-AI/semantica.git@main +``` + +--- + ## Troubleshooting -### Common Issues +### ModuleNotFoundError -#### ModuleNotFoundError +Check you have the right environment active: -**Error**: `ModuleNotFoundError: No module named 'semantica'` +```bash +pip list | grep semantica +pip install --upgrade semantica +``` -**Solutions**: -- Make sure you've activated the correct Python environment -- Verify installation: `pip list | grep semantica` -- Reinstall: `pip install --upgrade semantica` +### Installation fails with dependency errors -#### Installation Fails +```bash +pip install --upgrade pip +pip install build wheel +pip install semantica --no-deps # install without optional deps first +``` -**Error**: Installation fails with dependency errors +### GPU dependencies fail -**Solutions**: -- Upgrade pip: `pip install --upgrade pip` -- Install build tools: `pip install build wheel` -- Try installing without optional dependencies first: `pip install semantica --no-deps` +Install CPU-only first, then add GPU support: -#### GPU Dependencies Fail +```bash +pip install semantica +pip install semantica[gpu] +``` -**Error**: GPU dependencies fail to install +### Permission denied -**Solutions**: -- Install CPU-only version first: `pip install semantica` -- Then add GPU support: `pip install semantica[gpu]` -- Check CUDA compatibility for your system +```bash +pip install --user semantica # or use a virtual environment +``` -#### Permission Errors +### Windows PyTorch DLL errors -**Error**: Permission denied during installation +Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe). This is a Windows system dependency, not a Semantica bug. -**Solutions**: -- Use `--user` flag: `pip install --user semantica` -- Use virtual environment (recommended) -- On Linux/Mac, avoid using `sudo` with pip +--- -### System Requirements +## System Requirements | Component | Minimum | Recommended | |-----------|---------|-------------| | Python | 3.8 | 3.11+ | -| RAM | Moderate | Ample for your dataset | -| Disk Space | Sufficient for data | Generous storage | -| OS | Windows/Linux/Mac | Linux/Mac | +| OS | Windows / Linux / Mac | Linux / Mac | +| RAM | 4 GB | 16 GB+ | +| Storage | 2 GB | 20 GB+ (for models and data) | -## After Installation - -Once Semantica is installed, verify your setup and get started: - -### Verify Your Installation - -Test that everything works correctly: - -```bash -python -c "import semantica; print(semantica.__version__)" -``` - -**For detailed setup verification and first steps, see:** -- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Verify installation and explore all modules - - **Topics**: Framework overview, installation verification, module exploration - - **Difficulty**: Beginner - - **Time**: 30-45 minutes - - **Use Cases**: First-time setup, understanding the framework +--- ## Next Steps -Now that Semantica is installed: - -1. **[Quick Start Guide](quickstart.md)** - Build your first knowledge graph in 5 minutes -2. **[Getting Started Guide](getting-started.md)** - Learn the fundamentals -3. **[Examples](examples.md)** - See real-world use cases -4. **[Cookbook](cookbook.md)** - Interactive Jupyter notebook tutorials - -### 🍳 Recommended First Cookbooks - -Start with these interactive tutorials: - -- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction - - **Topics**: Framework overview, all modules, architecture, configuration - - **Difficulty**: Beginner - - **Time**: 30-45 minutes - - **Use Cases**: First-time users, understanding the framework - -- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first graph - - **Topics**: Entity extraction, relationship extraction, graph construction - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Hands-on practice, quick start - -## Getting Help - -If you encounter issues: - -- Check the [troubleshooting section](#troubleshooting) above -- Review [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) -- Ask questions in [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) - -**For installation and setup help:** -- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Includes setup verification steps -- **[Installation Troubleshooting Guide](getting-started.md#installation--setup)**: Additional troubleshooting tips +- [Getting Started](getting-started.md) — build your first knowledge graph +- [Quickstart Tutorial](quickstart.md) — full step-by-step pipeline +- [Cookbook](cookbook.md) — interactive Jupyter notebook tutorials diff --git a/docs/integrations/docling.md b/docs/integrations/docling.md index 1b2f59ed..5a5593cf 100644 --- a/docs/integrations/docling.md +++ b/docs/integrations/docling.md @@ -13,7 +13,7 @@ Docling is integrated into Semantica's `parse` module via the `DoclingParser`. T --- -## 📖 Integration Documentation +## Integration Documentation The `DoclingParser` provides a high-level interface for document processing. It supports: @@ -42,7 +42,7 @@ For more details, see the [Parse Reference](../reference/parse.md). --- -## 🧑🏽‍🍳 Integration Example +## Integration Example We provide a detailed cookbook and clear code examples to help you get started quickly. @@ -81,7 +81,7 @@ See more in our [Code Examples](../CodeExamples.md). --- -## 💻 GitHub Source +## GitHub Source The integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues. @@ -89,7 +89,7 @@ The integration is open-source and available on GitHub. You can explore the impl --- -## 📦 PyPI & Installation +## PyPI & Installation Docling is an optional but highly recommended dependency for Semantica. You can install it along with Semantica or as a separate requirement. diff --git a/docs/integrations/snowflake.md b/docs/integrations/snowflake.md index 8e624262..d0b1d092 100644 --- a/docs/integrations/snowflake.md +++ b/docs/integrations/snowflake.md @@ -13,7 +13,7 @@ Snowflake is integrated into Semantica's `ingest` module via the `SnowflakeInges --- -## 📖 Integration Documentation +## Integration Documentation The `SnowflakeIngestor` provides a high-level interface for Snowflake data ingestion. It supports: @@ -42,7 +42,7 @@ For more details, see the [Ingest Reference](../reference/ingest.md). --- -## 🧑🏽‍🍳 Integration Example +## Integration Example We provide a detailed cookbook and clear code examples to help you get started quickly. @@ -97,7 +97,7 @@ See more in our [Code Examples](../CodeExamples.md). --- -## 💻 GitHub Source +## GitHub Source The integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues. @@ -105,7 +105,7 @@ The integration is open-source and available on GitHub. You can explore the impl --- -## 📦 PyPI & Installation +## PyPI & Installation Snowflake connector is an optional dependency for Semantica. You can install it along with Semantica or as a separate requirement. @@ -128,7 +128,7 @@ For full installation details, see the [Installation Guide](../installation.md). --- -## 🔐 Authentication Methods +## Authentication Methods Snowflake integration supports multiple authentication methods for different security requirements: @@ -175,7 +175,7 @@ ingestor = SnowflakeIngestor( --- -## 🚀 Advanced Features +## Advanced Features ### Schema Introspection ```python @@ -209,7 +209,7 @@ data = ingestor.ingest_query( --- -## 📊 Best Practices +## Best Practices ### Use Environment Variables ```python @@ -244,7 +244,7 @@ for page in range(total_pages): --- -## 🔍 Troubleshooting +## Troubleshooting ### Connection Issues ```python @@ -272,7 +272,7 @@ ingestor = SnowflakeIngestor( --- -## 📚 See Also +## See Also - **[Ingest Module Reference](../reference/ingest.md)** - Complete ingestion documentation - **[Getting Started Guide](../getting-started.md)** - Quick start with Semantica diff --git a/docs/learning-more.md b/docs/learning-more.md index def221ea..49299455 100644 --- a/docs/learning-more.md +++ b/docs/learning-more.md @@ -1,213 +1,104 @@ # Learning More -Additional resources, tutorials, and advanced learning materials for Semantica. - -!!! info "About This Guide" - This guide provides structured learning paths, quick references, troubleshooting guides, and advanced topics to help you master Semantica. +Structured learning paths, quick references, and performance guidance for going deeper with Semantica. --- -## Structured Learning Paths +## Learning Paths
-- :material-school: **Beginner Path** +- :material-school: **Beginner** (1–2 hours) --- - Perfect for those new to Semantica and knowledge graphs. - + New to Semantica and knowledge graphs. - - [Start Path](#beginner-path-1-2-hours) + [Start here](#beginner-path) -- :material-compass: **Intermediate Path** +- :material-compass: **Intermediate** (4–6 hours) --- - For users comfortable with basics who want to build production applications. - + Comfortable with basics, building production applications. - - [Start Path](#intermediate-path-4-6-hours) + [Start here](#intermediate-path) -- :material-rocket: **Advanced Path** +- :material-rocket: **Advanced** (8+ hours) --- - For experienced users building enterprise applications. - + Enterprise applications and customization. - - [Start Path](#advanced-path-8-hours) + [Start here](#advanced-path)
--- -### Beginner Path (1-2 hours) +### Beginner Path -1. **Installation & Setup** (15 min) - - [Installation Guide](installation.md) - - **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction - - **Topics**: Framework overview, all modules, architecture, configuration - - **Difficulty**: Beginner - - **Time**: 30-45 minutes - - **Use Cases**: First-time users, understanding the framework - -2. **Core Concepts** (30 min) - - [Core Concepts](concepts.md) - - [Getting Started Guide](getting-started.md) - - **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources - - **Topics**: File, web, feed, stream, database ingestion - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Loading data from various sources - -3. **First Knowledge Graph** (30 min) - - [Quickstart Tutorial](quickstart.md) - - **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph - - **Topics**: Entity extraction, relationship extraction, graph construction, visualization - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Learning the basics, quick start - -4. **Basic Operations** (30 min) - - [Examples](examples.md) - - **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn entity extraction - - **Topics**: Named entity recognition, entity types, extraction methods - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Understanding entity extraction +1. **Installation & Setup** — [Installation Guide](installation.md) +2. **Core Concepts** — [Core Concepts](concepts.md) + [Getting Started](getting-started.md) +3. **First Knowledge Graph** — [Quickstart Tutorial](quickstart.md) +4. **Interactive Introduction** — [Welcome to Semantica notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb) +5. **Hands-On Practice** — [Your First Knowledge Graph notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb) --- -### Intermediate Path (4-6 hours) +### Intermediate Path -1. **Advanced Concepts** (1 hour) - - [Modules Guide](modules.md) - - **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Advanced graph construction - - **Topics**: Graph building, entity merging, conflict resolution, temporal graphs - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Production graph construction - - **[Embeddings Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb)**: Learn embeddings - - **Topics**: Embedding generation, similarity search, vector operations - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Understanding embeddings, semantic search - -2. **Use Cases** (1 hour) - - [Use Cases Guide](use-cases.md) - - **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Build production GraphRAG - - **Topics**: GraphRAG, hybrid retrieval, graph traversal, LLM integration - - **Difficulty**: Advanced - - **Time**: 1-2 hours - - **Use Cases**: Production GraphRAG systems - -3. **Advanced Examples** (1 hour) - - [Examples](examples.md) - - **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns - - **Topics**: Custom entity types, domain-specific extraction, hybrid methods - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Domain-specific extraction - -4. **Quality & Optimization** (1 hour) - - [Quality Assurance](concepts.md#8-quality-assurance) - - [Performance Optimization](#performance-optimization) - - **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Integrate multiple sources - - **Topics**: Multi-source integration, entity resolution, conflict handling - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Building unified knowledge graphs +1. **All Modules** — [Modules Guide](modules.md) +2. **Advanced Graph Construction** — [Building Knowledge Graphs notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) +3. **Embeddings & Search** — [Embeddings notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb) +4. **GraphRAG** — [GraphRAG Complete notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) +5. **Multi-Source Integration** — [Multi-Source Data Integration notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb) +6. **Use Case Examples** — [Use Cases](use-cases.md) --- -### Advanced Path (8+ hours) +### Advanced Path -1. **Advanced Architecture** (2 hours) - - [Architecture Guide](architecture.md) - - **[Temporal Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb)**: Build temporal graphs - - **Topics**: Time-stamped entities, temporal relationships, historical queries - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Time-aware knowledge graphs - - **[Ontology Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)**: Generate ontologies - - **Topics**: Ontology generation, OWL, schema design - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Formal knowledge representation - -2. **Production Deployment** (2 hours) - - [Security Best Practices](#security-best-practices) - - **[GraphRAG Complete Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production GraphRAG - - **Topics**: Production deployment, scalability, optimization - - **Difficulty**: Advanced - - **Time**: 1-2 hours - - **Use Cases**: Production systems - -3. **Customization** (2 hours) - - **[Complete Visualization Suite Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)**: Advanced visualization - - **Topics**: Custom layouts, filtering, styling, multiple graph types - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Production visualizations - - **[Multi-Format Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)**: Advanced export patterns - - **Topics**: Batch export, custom formats, format conversion - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Production exports +1. **Architecture Deep Dive** — [Architecture Guide](architecture.md) +2. **Temporal Graphs** — [Temporal Graphs notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb) +3. **Ontologies** — [Ontology notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb) +4. **Visualization** — [Complete Visualization Suite notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb) +5. **Export Pipelines** — [Multi-Format Export notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb) +6. **Production GraphRAG** — [GraphRAG Complete notebook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) --- -## Quick Reference +## Configuration Reference -### Common Operations - -The typical workflow involves these steps: - -1. **Ingest** documents using `` `FileIngestor` `` -2. **Parse** documents using `` `DocumentParser` `` -3. **Extract** entities and relationships using `` `NERExtractor` `` and `` `RelationExtractor` `` -4. **Build** knowledge graph using `` `GraphBuilder` `` -5. **Generate** embeddings using `` `TextEmbedder` `` - -**For complete examples, see:** -- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Complete workflow example -- **[Welcome to Semantica Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: All modules overview - -### Configuration Reference - -| Setting | Environment Variable | Config File | Default | -| :--- | :--- | :--- | :--- | -| OpenAI API Key | `OPENAI_API_KEY` | `api_keys.openai` | `None` | -| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `embedding.provider` | `"openai"` | -| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `knowledge_graph.backend` | `"networkx"` | +| Setting | Environment Variable | Default | +|---------|---------------------|---------| +| OpenAI API Key | `OPENAI_API_KEY` | `None` | +| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `"openai"` | +| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `"networkx"` | --- -## Troubleshooting Guide +## Troubleshooting
- :material-alert: **Import Errors** --- `ModuleNotFoundError` - - **Solution**: Verify installation (`pip list`) and Python version (3.8+). + + Verify installation: `pip list | grep semantica`. Ensure Python 3.8+. - :material-key: **API Key Errors** --- `AuthenticationError` - - **Solution**: Set `OPENAI_API_KEY` environment variable. + + Set `OPENAI_API_KEY` (or the relevant provider key) as an environment variable. - :material-memory: **Memory Errors** --- - `MemoryError` - - **Solution**: Use batch processing and graph stores (Neo4j). + `MemoryError` or OOM crashes + + Reduce batch sizes or switch to a persistent graph backend (Neo4j, FalkorDB). - :material-speedometer: **Slow Processing** --- - Long processing times - - **Solution**: Enable parallel processing and GPU acceleration. + Long runtimes on large datasets + + Enable parallel processing (`PipelineBuilder` workers) and GPU acceleration.
@@ -215,80 +106,46 @@ The typical workflow involves these steps: ## Performance Optimization -### 1. Batch Processing +### Batch Processing -Process multiple documents together for better throughput. Use batch processing when working with large document collections. +Process documents in batches rather than one at a time. Configure chunk sizes based on available RAM. -**For examples, see:** -- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Batch ingestion patterns -- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration +### Parallel Execution -### 2. Parallel Execution +`PipelineBuilder` supports configurable worker counts per stage for independent operations. -Use parallel processing for independent operations to improve performance on multi-core systems. +### Backend Selection -### 3. Backend Selection +| Operation | NetworkX | Neo4j / FalkorDB | +|-----------|----------|------------------| +| Graph construction | Fast | Moderate | +| Query performance | Moderate | Fast | +| Scalability | Low (in-memory) | High (persistent) | -| Operation | NetworkX | Neo4j | -| :--- | :--- | :--- | -| **Graph Construction** | ⚡⚡⚡ | ⚡⚡ | -| **Query Performance** | ⚡⚡ | ⚡⚡⚡ | -| **Scalability** | Low | High | +Use NetworkX for development and smaller graphs; switch to a persistent backend for production at scale. --- ## Security Best Practices -### API Key Management +**API keys** +- Store in environment variables or a secrets manager +- Never hardcode keys or commit them to version control +- Rotate keys regularly -- **DO**: Use environment variables, rotate keys regularly. -- **DON'T**: Hardcode keys, commit to version control. - -### Data Privacy - -- **DO**: Encrypt sensitive data, use local models. -- **DON'T**: Send PII to external APIs without protection. - ---- - -## FAQ - -**Q: What is Semantica?** -A: A framework for building knowledge graphs and semantic applications. - -**Q: Is Semantica free?** -A: Yes, it is open source. Some features (e.g., OpenAI) require paid APIs. - -**Q: Can I use Semantica in production?** -A: Yes, it is designed for production with proper configuration. +**Data privacy** +- Use local embedding models for sensitive data +- Avoid sending PII to external APIs without appropriate data handling agreements +- Encrypt sensitive graph exports at rest --- ## Next Steps -Continue your learning journey: - -- **[Cookbook](cookbook.md)** - Interactive Jupyter notebook tutorials -- **[API Reference](reference/core.md)** - Complete API documentation -- **[Use Cases](use-cases.md)** - Real-world applications -- **[Examples](examples.md)** - Code examples and patterns - -### 🍳 Recommended Next Cookbooks - -- **[GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)**: Production GraphRAG system - - **Topics**: GraphRAG, hybrid retrieval, LLM integration - - **Difficulty**: Advanced - - **Time**: 1-2 hours - - **Use Cases**: Production RAG applications - -- **[RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)**: Understand the differences - - **Topics**: RAG comparison, reasoning gap, inference engines - - **Difficulty**: Intermediate - - **Time**: 45-60 minutes - - **Use Cases**: Choosing the right approach - ---- - -!!! info "Contribute" - Have questions? [Open an issue](https://github.com/Hawksight-AI/semantica/issues) or [start a discussion](https://github.com/Hawksight-AI/semantica/discussions)! +- [Cookbook](cookbook.md) — interactive Jupyter notebook tutorials +- [API Reference](reference/core.md) — complete technical documentation +- [Use Cases](use-cases.md) — real-world domain examples +- [FAQ](faq.md) — common questions +!!! info "Questions or feedback?" + [Open an issue](https://github.com/Hawksight-AI/semantica/issues) or [start a discussion](https://github.com/Hawksight-AI/semantica/discussions). diff --git a/docs/modules.md b/docs/modules.md index 23c36801..d03bc0a3 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -1,9 +1,9 @@ -# Modules & Architecture +# Modules -**Complete guide to Semantica's modular architecture and how to use each component.** +Every Semantica module works independently — use only what you need. -!!! tip "Modular Design" - Each Semantica module works independently. Use only what you need for your specific use case. +!!! tip "Just need a quick reference?" + Jump to the [Module Index](#module-index) at the bottom of this page. --- @@ -82,13 +82,11 @@ web_ingestor = WebIngestor() pages = web_ingestor.ingest_urls(["https://example.com"]) ``` -**What it does:** - **File formats** - PDF, DOCX, TXT, JSON, CSV - **Web scraping** - Extract content from websites - **Database** - Connect to SQL and NoSQL databases - **Batch processing** - Handle large datasets efficiently -**Use Cases:** - Document processing pipelines - Web data extraction - Database integration @@ -106,13 +104,11 @@ text = parsed["full_text"] metadata = parsed["metadata"] ``` -**What it does:** - **Text extraction** - Extract clean text from documents - **Metadata parsing** - Extract titles, authors, dates - **Structure analysis** - Identify sections, headings - **OCR support** - Handle scanned documents -**Use Cases:** - PDF processing - Document analysis - Content extraction @@ -130,13 +126,11 @@ splitter = TextSplitter(method="semantic") chunks = splitter.split(text, chunk_size=1000, overlap=200) ``` -**What it does:** - **Intelligent chunking** - Split text while preserving context - **Semantic splitting** - Break at natural boundaries - **Size control** - Manage chunk sizes for processing - **Overlap handling** - Maintain context between chunks -**Use Cases:** - Document preprocessing - Embedding preparation - RAG systems @@ -155,13 +149,11 @@ clean_text = normalizer.normalize_text(text) standardized_date = normalizer.normalize_date("Jan 1st, 2020") ``` -**What it does:** - **Text cleaning** - Remove noise and artifacts - **Date standardization** - Convert to ISO format - **Name normalization** - Standardize person names - **Entity normalization** - Clean up company names -**Use Cases:** - Data preprocessing - Quality improvement - Standardization @@ -186,13 +178,11 @@ rel_extractor = RelationExtractor() relationships = rel_extractor.extract(text, entities) ``` -**What it does:** - **Named Entity Recognition** - Find people, orgs, locations - **Relationship extraction** - Find connections between entities - **Custom entities** - Define your own entity types - **Confidence scoring** - Quality assessment for extractions -**Use Cases:** - Knowledge graph construction - Document analysis - Information extraction @@ -215,13 +205,11 @@ analyzer = GraphAnalyzer() stats = analyzer.analyze(kg) ``` -**What it does:** - **Graph construction** - Build knowledge graphs from data - **Graph analysis** - Calculate metrics and statistics - **Graph querying** - Search and retrieve information - **Graph manipulation** - Merge, split, transform graphs -**Use Cases:** - Knowledge base creation - Graph analytics - Information retrieval @@ -244,13 +232,11 @@ ontology.add_relationship("works_for", "Person", "Organization") is_valid = ontology.validate_graph(kg) ``` -**What it does:** - **Schema definition** - Define data structure - **Data validation** - Ensure data conforms to schema - **Inheritance** - Create hierarchical relationships - **Constraints** - Enforce data quality rules -**Use Cases:** - Data modeling - Quality assurance - Schema management @@ -268,13 +254,11 @@ engine = ReasoningEngine() inferences = engine.infer(kg, rules=["transitivity", "symmetry"]) ``` -**What it does:** - **Logical inference** - Derive new facts from existing ones - **Pattern matching** - Find complex patterns in data - **Consistency checking** - Detect contradictions - **Decision support** - Automated reasoning -**Use Cases:** - Knowledge discovery - Decision making - Consistency checking @@ -295,13 +279,11 @@ embeddings = generator.generate(["text1", "text2"]) similarity = generator.similarity(embeddings[0], embeddings[1]) ``` -**What it does:** - **Text embeddings** - Convert text to vectors - **Similarity search** - Find similar content - **Clustering** - Group related items - **AI integration** - Provide context to LLMs -**Use Cases:** - Semantic search - Recommendation systems - Clustering @@ -320,13 +302,11 @@ store.add_vectors(embeddings, ids) results = store.search(query_vector, top_k=10) ``` -**What it does:** - **Vector storage** - Efficient vector database - **Fast search** - Approximate nearest neighbor search - **Indexing** - Optimize for performance - **Batch operations** - Handle large datasets -**Use Cases:** - Semantic search - RAG systems - Recommendation engines @@ -346,13 +326,11 @@ store.add_edges(relationships) results = store.query("MATCH (n)-[r]->(m) RETURN n, r, m") ``` -**What it does:** - **Graph persistence** - Store graphs in databases - **Graph queries** - Cypher and Gremlin support - **Graph algorithms** - Path finding, centrality - **Transactions** - ACID compliance -**Use Cases:** - Knowledge graph storage - Graph analytics - Network analysis @@ -371,13 +349,11 @@ store.add_triplets(subject, predicate, object) triplets = store.get_triplets(entity="Apple Inc.") ``` -**What it does:** - **Triple storage** - Store (subject, predicate, object) triples - **Pattern matching** - Find specific patterns - **RDF support** - Semantic web standards - **Bulk operations** - Efficient batch processing -**Use Cases:** - Semantic web - Knowledge representation - Linked data @@ -397,13 +373,11 @@ resolver = EntityResolver() merged_entities = resolver.resolve(entities, strategy="semantic") ``` -**What it does:** - **Duplicate detection** - Find similar entities - **Entity resolution** - Merge duplicate records - **Similarity scoring** - Quality assessment - **Record linkage** - Connect related records -**Use Cases:** - Data cleaning - Master data management - Record linkage @@ -422,13 +396,11 @@ conflicts = detector.detect_conflicts(kg) resolved = detector.resolve(conflicts, strategy="most_recent") ``` -**What it does:** - **Conflict detection** - Find contradictory information - **Resolution strategies** - Automated conflict resolution - **Source reliability** - Trustworthiness assessment - **Temporal analysis** - Time-based conflict handling -**Use Cases:** - Data quality - Consistency checking - Trust management @@ -448,13 +420,11 @@ manager = ContextManager() context = manager.get_context(query, history) ``` -**What it does:** - **Context tracking** - Maintain conversation context - **Memory management** - Store and retrieve context - **Relevance scoring** - Find relevant context - **Session management** - Handle multiple conversations -**Use Cases:** - AI agents - Chatbots - Conversational AI @@ -472,13 +442,11 @@ seed = SeedData() knowledge = seed.get_knowledge("technology", "companies") ``` -**What it does:** - **Seed knowledge** - Foundation data for domains - **Knowledge bases** - Pre-built domain knowledge - **Quick start** - Bootstrap applications - **Domain models** - Industry-specific data -**Use Cases:** - Domain bootstrapping - Quick start data - Industry knowledge @@ -496,13 +464,11 @@ provider = LLMProvider(model="gpt-4") response = provider.generate(prompt, context=kg) ``` -**What it does:** - **LLM integration** - Connect to various LLM providers - **Prompt engineering** - Optimize prompts for results - **Context injection** - Provide knowledge graph context - **Response parsing** - Extract structured outputs -**Use Cases:** - AI generation - Question answering - Text completion @@ -522,13 +488,11 @@ exporter = GraphExporter() exporter.export(kg, format="json", filename="output.json") ``` -**What it does:** - **Multiple formats** - JSON, CSV, RDF, GraphML - **Database export** - Export to various databases - **Streaming** - Handle large datasets - **Filtering** - Export specific data subsets -**Use Cases:** - Data sharing - System integration - Backup and restore @@ -546,13 +510,11 @@ visualizer = GraphVisualizer() visualizer.plot(kg, layout="force_directed") ``` -**What it does:** - **Graph visualization** - Interactive graph plots - **Custom styling** - Tailored visual appearance - **Analytics charts** - Statistics and metrics - **Exploration tools** - Interactive data exploration -**Use Cases:** - Data exploration - Presentation - Analysis @@ -573,13 +535,11 @@ pipeline.add_step("build", GraphBuilder()) result = pipeline.run("data/") ``` -**What it does:** - **Workflow orchestration** - Coordinate multiple steps - **Parallel processing** - Run steps concurrently - **Progress tracking** - Monitor pipeline execution - **Error handling** - Robust error management -**Use Cases:** - Data processing - Workflow automation - Batch processing @@ -587,7 +547,7 @@ result = pipeline.run("data/") --- -## New Features & Modules +## Additional Modules ### Change Management Module **Version control and audit trails** @@ -599,13 +559,11 @@ manager = TemporalVersionManager(storage_path="versions.db") snapshot = manager.create_snapshot(kg, "v1.0", "user@example.com", "Initial version") ``` -**What it does:** - **Version control** - Track changes over time - **Audit trails** - Complete change history - **Data integrity** - SHA-256 checksums - **Change comparison** - Detailed diff analysis -**Use Cases:** - Knowledge graph versioning - Compliance tracking - Data governance @@ -623,13 +581,11 @@ manager = ProvenanceManager() manager.track_entity("entity_1", "document.pdf", "person") ``` -**What it does:** - **W3C PROV-O compliant** - Industry standard tracking - **Complete lineage** - End-to-end traceability - **Source attribution** - Track data origins - **Integrity verification** - Tamper detection -**Use Cases:** - Regulatory compliance - Data provenance - Audit trails @@ -648,13 +604,11 @@ semantica = Semantica(config=Config()) result = semantica.process("data/") ``` -**What it does:** - **Framework orchestration** - Central coordination - **Configuration management** - Settings and preferences - **Lifecycle management** - Start/stop/restart - **Plugin system** - Extensible architecture -**Use Cases:** - Framework initialization - Configuration management - Plugin development @@ -662,46 +616,18 @@ result = semantica.process("data/") --- -## Getting Started +## Common Module Chains -### Quick Start Example - -```python -# Complete pipeline example -from semantica.ingest import FileIngestor -from semantica.semantic_extract import NERExtractor, RelationExtractor -from semantica.kg import GraphBuilder -from semantica.pipeline import Pipeline - -# Create pipeline -pipeline = Pipeline() -pipeline.add_step("ingest", FileIngestor()) -pipeline.add_step("ner", NERExtractor()) -pipeline.add_step("relations", RelationExtractor()) -pipeline.add_step("build", GraphBuilder()) - -# Run pipeline -kg = pipeline.run("documents/") -print(f"Built graph with {len(kg['entities'])} entities") -``` - -### Choose Your Modules - -**For Document Processing:** -- Ingest → Parse → Split → Semantic Extract → Knowledge Graph - -**For Web Scraping:** -- Ingest (Web) → Normalize → Semantic Extract → Graph Store - -**For AI Agents:** -- Context → LLM Providers → Reasoning → Export - -**For Analytics:** -- Knowledge Graph → Graph Store → Visualization → Export +| Goal | Modules | +|------|---------| +| Document processing | Ingest → Parse → Split → Semantic Extract → KG | +| Web scraping | Ingest (Web) → Normalize → Semantic Extract → Graph Store | +| AI agents | Context → LLM Providers → Reasoning → Export | +| Analytics | KG → Graph Store → Visualization → Export | --- -## Module Reference +## Module Index | Module | Purpose | Key Classes | Use Cases | |--------|---------|-------------|-----------| @@ -731,10 +657,9 @@ print(f"Built graph with {len(kg['entities'])} entities") --- -## Need Help? +## More -- **Documentation**: [Getting Started](getting-started.md) -- **Examples**: [Cookbook](cookbook.md) -- **Community**: [Discord](community.md) -- **Issues**: [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) -- **Support**: [Contact Us](community.md) +- [Getting Started](getting-started.md) +- [Examples](examples.md) +- [Cookbook](cookbook.md) +- [API Reference](reference/core.md) diff --git a/docs/quickstart.md b/docs/quickstart.md index f3d9b00e..9c6d5392 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -1,246 +1,150 @@ # Quickstart -Get started with Semantica in 5 minutes. This guide will walk you through building your first knowledge graph. +Build your first knowledge graph in 5 minutes. -!!! tip "Before You Start" - Make sure you have Semantica installed. If not, follow the [Installation Guide](installation.md) first. This quickstart assumes basic Python knowledge. +!!! tip "Prerequisites" + Semantica installed (`pip install semantica`). If not, see the [Installation Guide](installation.md). -## Overview +--- + +## Pipeline Overview ```mermaid flowchart LR - A[Install] --> B[Initialize] - B --> C[Load Data] - C --> D[Extract] - D --> E[Build Graph] - E --> F[Visualize] - - style A fill:#e3f2fd - style F fill:#c8e6c9 + A[Ingest] --> B[Parse] + B --> C[Extract] + C --> D[Build Graph] + D --> E[Visualize / Export] ``` -## Step 1: Installation +--- -If you haven't installed Semantica yet: +## Step 1 — Ingest -```bash -pip install semantica -``` +Load documents from files, directories, or the web. -See the [Installation Guide](installation.md) for detailed instructions. - -!!! note "Installation Options" - For production use, consider installing with optional dependencies for better performance: `pip install semantica[all]`. See the [Installation Guide](installation.md) for all options. - -## Step 2: Your First Knowledge Graph - -Building a knowledge graph involves these key steps: - -1. **Ingest** your documents using `FileIngestor` -2. **Parse** documents to extract text using `DocumentParser` or `DoclingParser` (for enhanced layout support) -3. **Extract** entities and relationships using `NERExtractor` and `RelationExtractor` -4. **Build** the graph using `GraphBuilder` -5. **Generate** embeddings (optional) using `TextEmbedder` - -**Quick Example:** ```python from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser, DoclingParser -from semantica.semantic_extract import NERExtractor, RelationExtractor -from semantica.kg import GraphBuilder -# 1. Ingest document ingestor = FileIngestor() sources = ingestor.ingest("data/sample.pdf") - -# 2. Parse (choose your parser) -# Option A: Standard parser -parser = DocumentParser() -parsed_content = parser.parse(sources[0]) - -# Option B: Enhanced Docling parser (recommended for complex tables) -# docling_parser = DoclingParser() -# parsed_content = docling_parser.parse(sources[0]) - -# 3. Extract entities and relations -ner = NERExtractor() -entities = ner.extract(parsed_content) - -relations = RelationExtractor() -relationships = relations.extract(parsed_content, entities=entities) - -# 4. Build graph -builder = GraphBuilder() -graph = builder.build(entities=entities, relationships=relationships) - -print(f"Built knowledge graph with {len(graph.nodes)} nodes and {len(graph.edges)} edges") ``` -**For complete step-by-step examples with detailed explanations, see:** -- **[Your First Knowledge Graph Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Full tutorial with detailed explanations and expected outputs - - **Topics**: Entity extraction, relationship extraction, graph construction, visualization - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Learning the basics, quick start +Supported formats: PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, archives. For web content, use `WebIngestor`. -## Step 3: Extract Entities and Relationships +--- -The semantic extraction step identifies named entities (people, organizations, locations) and relationships between them from your text. +## Step 2 — Parse -**What gets extracted:** -- **Entities**: People, organizations, locations, dates, and other named entities -- **Relationships**: Connections between entities (e.g., `founded_by`, `located_in`, `has_ceo`) +Extract structured text from raw documents. -**For detailed examples and different extraction methods, see:** -- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Learn different NER methods and configurations - - **Topics**: Named entity recognition, entity types, confidence scores - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Understanding entity extraction options +```python +from semantica.parse import DocumentParser -- **[Relation Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)**: Learn to extract relationships between entities - - **Topics**: Relationship extraction, dependency parsing, semantic role labeling - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Building rich knowledge graphs with relationships +parser = DocumentParser() +parsed = parser.parse(sources[0]) +``` -## Step 4: Build Knowledge Graph from Multiple Sources +For complex layouts (tables, columns): use `DoclingParser` instead — it handles PDF tables and structured DOCX/PPTX better. -You can combine data from multiple sources (files, web, databases) to build a unified knowledge graph. The process involves: +--- -1. **Ingest** from multiple sources using different ingestors -2. **Parse** all documents to extract text -3. **Extract** entities and relationships from each source -4. **Build** a unified graph with entity merging enabled +## Step 3 — Extract Entities and Relationships -**For complete examples with multiple sources, see:** -- **[Data Ingestion Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from files, web, feeds, streams, and databases - - **Topics**: File, web, feed, stream, database ingestion - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Loading data from various sources +```python +from semantica.semantic_extract import NERExtractor, RelationExtractor -- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced patterns for integrating multiple data sources - - **Topics**: Multi-source integration, entity resolution, conflict handling - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Building knowledge graphs from diverse data sources +ner = NERExtractor() +entities = ner.extract(parsed) -## Step 5: Visualize Your Knowledge Graph +rel = RelationExtractor() +relationships = rel.extract(parsed, entities=entities) +``` -Visualization helps you understand and explore your knowledge graph structure. Semantica supports multiple visualization formats including interactive HTML, static images, and export formats. +Each entity gets a type, confidence score, and source reference. Relationships are extracted as typed triplets: `(subject, predicate, object)`. -**For detailed visualization examples, see:** -- **[Visualization Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/16_Visualization.ipynb)**: Learn to create interactive and static visualizations - - **Topics**: Network graphs, interactive HTML, static images, export formats - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Exploring graph structure, presentations, analysis +--- -- **[Complete Visualization Suite Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)**: Advanced visualization techniques - - **Topics**: Custom layouts, filtering, styling, multiple graph types - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Production visualizations, custom dashboards +## Step 4 — Build the Knowledge Graph -## Step 6: Export Your Knowledge Graph +```python +from semantica.kg import GraphBuilder -Export your knowledge graph to various formats for integration with other systems or tools. Semantica supports RDF, JSON, CSV, OWL, GraphML, and more. +builder = GraphBuilder(merge_entities=True) +graph = builder.build(entities=entities, relationships=relationships) -**Supported export formats:** -- **RDF**: Turtle, RDF/XML, JSON-LD, N-Triples -- **JSON**: Standard JSON, JSON-LD, Cytoscape.js format -- **CSV**: Node and edge lists for spreadsheet tools -- **OWL**: OWL/XML and Turtle for ontologies -- **Graph Formats**: GraphML, GEXF, DOT for visualization tools +print(f"{len(graph.nodes)} nodes, {len(graph.edges)} edges") +``` -**For detailed export examples, see:** -- **[Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/15_Export.ipynb)**: Learn to export to all supported formats - - **Topics**: RDF, JSON, CSV, OWL, GraphML export - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Data integration, sharing knowledge graphs +`merge_entities=True` resolves duplicates across sources automatically. -- **[Multi-Format Export Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)**: Advanced export patterns - - **Topics**: Batch export, custom formats, format conversion - - **Difficulty**: Intermediate - - **Time**: 30-45 minutes - - **Use Cases**: Production exports, format migration +--- + +## Step 5 — Visualize + +```python +from semantica.visualization import GraphVisualizer + +viz = GraphVisualizer() +viz.visualize(graph, output="graph.html") # interactive HTML +``` + +--- + +## Step 6 — Export + +```python +from semantica.export import RDFExporter + +exporter = RDFExporter() +rdf = exporter.export_to_rdf(graph, format="turtle") +``` + +Other formats: `"json-ld"`, `"nt"`, `"xml"`, Parquet, ArangoDB AQL. See [Export Reference](reference/export.md). + +--- ## Common Patterns -### Pattern 1: Process Text Directly +### Process text directly (no file) -You can process text directly without file ingestion. This is useful when you already have text content in memory. +```python +from semantica.semantic_extract import NERExtractor -**For examples, see:** -- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Processing text directly -- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Graph construction from text +ner = NERExtractor() +entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.") +``` -### Pattern 2: Custom Entity Extraction +### Incremental build from multiple sources -Configure entity extraction with different methods (ML models, LLMs) and parameters for your specific needs. +```python +from semantica.kg import GraphBuilder -**For examples, see:** -- **[Entity Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)**: Different extraction methods and configurations -- **[Advanced Extraction Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)**: Advanced extraction patterns +all_entities, all_rels = [], [] +for doc in parsed_docs: + all_entities.extend(ner.extract(doc)) + all_rels.extend(rel.extract(doc, entities=all_entities)) -### Pattern 3: Incremental Building +graph = GraphBuilder(merge_entities=True).build( + entities=all_entities, relationships=all_rels +) +``` -Build knowledge graphs incrementally from multiple sources and merge them together. - -**For examples, see:** -- **[Building Knowledge Graphs Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb)**: Graph construction and merging -- **[Multi-Source Data Integration Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)**: Advanced integration patterns - -## Next Steps - -Now that you've built your first knowledge graph: - -1. **[Explore Examples](examples.md)** - See more advanced use cases -2. **[API Reference](reference/core.md)** - Learn about all available methods -3. **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks -4. **[Full Documentation](https://github.com/Hawksight-AI/semantica/blob/main/README.md)** - Comprehensive guide - -### 🍳 Recommended Cookbook Tutorials - -Continue learning with these interactive tutorials: - -- **[Welcome to Semantica](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)**: Comprehensive introduction to all modules - - **Topics**: Framework overview, all modules, architecture, configuration - - **Difficulty**: Beginner - - **Time**: 30-45 minutes - - **Use Cases**: Understanding the complete framework - -- **[Your First Knowledge Graph](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph - - **Topics**: Entity extraction, relationship extraction, graph construction, visualization - - **Difficulty**: Beginner - - **Time**: 20-30 minutes - - **Use Cases**: Hands-on practice with complete workflow - -- **[Data Ingestion](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)**: Learn to ingest from multiple sources - - **Topics**: File, web, feed, stream, database ingestion - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Loading data from various sources - -- **[Document Parsing](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)**: Parse various document formats - - **Topics**: PDF, DOCX, HTML, JSON parsing - - **Difficulty**: Beginner - - **Time**: 15-20 minutes - - **Use Cases**: Extracting text from different file formats +--- ## Troubleshooting -### Common Issues +| Problem | Fix | +|---------|-----| +| No entities extracted | Check the document has machine-readable text (not just scanned images) | +| Slow processing | Process in chunks; use GPU acceleration (`pip install semantica[gpu]`) | +| Memory errors | Reduce batch size or switch to a persistent graph backend | -**Issue**: No entities extracted -- **Solution**: Check that your document contains text content. PDFs with images only won't work without OCR. +--- -**Issue**: Slow processing -- **Solution**: For large documents, consider processing in chunks or using GPU acceleration. +## Next Steps -**Issue**: Memory errors -- **Solution**: Process documents one at a time or reduce batch sizes. - -Need help? Check the [Installation Troubleshooting](installation.md#troubleshooting) or [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues). +- [Core Concepts](concepts.md) — understand how knowledge graphs and reasoning work +- [Modules Guide](modules.md) — every module explained +- [Use Cases](use-cases.md) — domain-specific examples +- [Cookbook](cookbook.md) — interactive Jupyter notebooks for each step diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md index 7fb9a1a4..47d40499 100644 --- a/docs/reference/change_management.md +++ b/docs/reference/change_management.md @@ -58,16 +58,14 @@ The Semantica change management module provides enterprise-grade version control
-### Key Features - -- ✅ **Enterprise Version Control** — Complete snapshot management with SHA-256 integrity verification -- ✅ **Dual Storage Backends** — InMemory (development) and SQLite (production) with ACID guarantees -- ✅ **Knowledge Graph Versioning** — Entity and relationship-level change tracking with detailed diffs -- ✅ **Ontology Versioning** — Structural change tracking for classes, properties, and axioms -- ✅ **Audit Trail Compliance** — Complete change logs with author attribution and timestamps -- ✅ **Data Integrity** — SHA-256 checksums for tamper detection and verification -- ✅ **Change Comparison** — Detailed diff algorithms for entities, relationships, and ontology structures -- ✅ **Backward Compatibility** — Legacy support for existing ontology version management +- **Enterprise Version Control** — Complete snapshot management with SHA-256 integrity verification +- **Dual Storage Backends** — InMemory (development) and SQLite (production) with ACID guarantees +- **Knowledge Graph Versioning** — Entity and relationship-level change tracking with detailed diffs +- **Ontology Versioning** — Structural change tracking for classes, properties, and axioms +- **Audit Trail Compliance** — Complete change logs with author attribution and timestamps +- **Data Integrity** — SHA-256 checksums for tamper detection and verification +- **Change Comparison** — Detailed diff algorithms for entities, relationships, and ontology structures +- **Backward Compatibility** — Legacy support for existing ontology version management --- @@ -121,24 +119,21 @@ class ChangeLogEntry: ### Storage Backends -#### InMemoryVersionStorage -Fast, volatile storage for development and testing. +**InMemoryVersionStorage** — Fast, volatile storage for development and testing. ```python from semantica.change_management import InMemoryVersionStorage storage = InMemoryVersionStorage() ``` -#### SQLiteVersionStorage -Persistent storage with ACID guarantees for production. +**SQLiteVersionStorage** — Persistent storage with ACID guarantees for production. ```python from semantica.change_management import SQLiteVersionStorage storage = SQLiteVersionStorage("versions.db") ``` -#### VersionStorage (Abstract) -Base interface for custom storage implementations. +**VersionStorage (Abstract)** — Base interface for custom storage implementations. **Core Methods:** - `save(snapshot)` - Store version snapshot @@ -251,7 +246,7 @@ For large-scale knowledge graphs, reprocessing the entire dataset on every updat Semantica supports **Delta-Aware Pipelines**, allowing you to compute the exact differences (added and removed triples) between the two graph snapshots and run validation, enrichment, or export jobs *only* on the changes. -### Delta Pipeline Example +**Delta Pipeline Example** ```python from semantica.change_management import TemporalVersionManager @@ -286,6 +281,7 @@ result = engine.execute_pipeline( triplet_store=triplet_store ) ``` + --- ## Data Integrity @@ -399,7 +395,7 @@ for version in prod_manager.list_versions(): Semantica allows you to treat ontology schema changes with the same rigor as database migrations. By comparing two versions, you can generate a machine-readable diff and a structured impact report to catch breaking changes before they reach production. -### Comparing Versions +**Comparing Versions** The `OntologyEngine` provides a high-level API to orchestrate the comparison of two schema versions. @@ -410,21 +406,18 @@ engine = OntologyEngine() # Generate a migration impact report between v1.0 and v2.0 report = engine.compare_versions( - base_id="v1.0", + base_id="v1.0", target_id="v2.0" ) print(f"Total changes detected: {report['summary']['total_changes']}") ``` + --- -### Understanding the Report Format +**Report Format** -The `compare_versions` method returns a comprehensive dictionary containing both a machine-readable diff and a human-readable impact analysis. - - - -Here is the exact structure of the returned report: +The `compare_versions` method returns a dictionary with a machine-readable diff and a human-readable impact analysis: ```json { @@ -467,3 +460,4 @@ Here is the exact structure of the returned report: "warnings": [] } } +``` diff --git a/docs/use-cases.md b/docs/use-cases.md index 9ad0013c..2d9c4ba2 100644 --- a/docs/use-cases.md +++ b/docs/use-cases.md @@ -1,210 +1,125 @@ # Use Cases -Semantica is designed to solve complex data challenges across various domains. This guide explores common use cases and how to implement them. - -!!! info "About This Guide" - This guide provides detailed implementation guides for real-world use cases, complete with code examples, prerequisites, and step-by-step instructions. +Real-world applications of Semantica across domains, with linked cookbook notebooks for each. --- -## Use Case Comparison +## Overview -| Use Case | Difficulty | Time | Domain | Key Features | Cookbook | -| :-------------------------------- | :------------ | :---------- | :---------- | :---------------------------------------------- | :------------------------------------------ | -| **Biomedical Knowledge Graphs** | Intermediate | 1-2 hours | Healthcare | Gene-protein-disease relationships | Drug Discovery, Genomic Variant Analysis | -| **Financial Data Integration** | Intermediate | 1-2 hours | Finance | MCP integration, real-time data | Financial Data Integration MCP | -| **Fraud Detection** | Advanced | 2-3 hours | Finance | Temporal graphs, pattern detection | Fraud Detection | -| **Blockchain Analytics** | Intermediate | 1-2 hours | Finance | Transaction tracing, DeFi intelligence | DeFi Protocol Intelligence, Transaction Network | -| **Cybersecurity Threat Intelligence**| Advanced | 2-3 hours | Security | Threat mapping, anomaly detection | Real-Time Anomaly Detection, Threat Intelligence | -| **Intelligence Analysis** | Intermediate | 1-2 hours | Security | Criminal networks, OSINT analysis | Criminal Network Analysis, Intelligence Orchestrator | -| **Supply Chain Optimization** | Intermediate | 1-2 hours | Industry | Data integration, route optimization | Supply Chain Data Integration | -| **Renewable Energy Management** | Intermediate | 1-2 hours | Energy | Energy market analysis, optimization | Energy Market Analysis | -| **GraphRAG** | Advanced | 1-2 hours | AI | Enhanced RAG with knowledge graphs | GraphRAG Complete, RAG vs GraphRAG | +| Use Case | Domain | Difficulty | Estimated Time | +|----------|--------|------------|----------------| +| Biomedical Knowledge Graphs | Healthcare | Intermediate | 1–2 hours | +| Financial Data Integration | Finance | Intermediate | 1–2 hours | +| Fraud Detection | Finance | Advanced | 2–3 hours | +| Blockchain Analytics | Finance | Intermediate | 1–2 hours | +| Cybersecurity Threat Intelligence | Security | Advanced | 2–3 hours | +| Criminal Network Analysis | Security / Intelligence | Intermediate | 1–2 hours | +| Intelligence Analysis Orchestrator | Intelligence | Intermediate | 1–2 hours | +| Supply Chain Optimization | Operations | Intermediate | 1–2 hours | +| Renewable Energy Management | Energy | Intermediate | 1–2 hours | +| GraphRAG | AI | Advanced | 1–2 hours | -**Difficulty Levels**: -- **Beginner**: Basic Semantica knowledge required -- **Intermediate**: Some domain knowledge helpful -- **Advanced**: Requires domain expertise and advanced Semantica features +**Difficulty levels**: + +- **Beginner** — basic Semantica knowledge only +- **Intermediate** — some domain knowledge helpful +- **Advanced** — domain expertise + advanced Semantica features --- ## Research & Science -
+### Biomedical Knowledge Graphs -- :material-dna: **Biomedical Knowledge Graphs** - --- - Accelerate drug discovery and understand disease pathways by connecting genes, proteins, drugs, and diseases. - - **Goal**: Connect genes, proteins, drugs, and diseases from scientific literature and databases. - - **Difficulty**: Intermediate - - [:material-arrow-right: Drug Discovery Pipeline](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) - - [:material-arrow-right: Genomic Variant Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) +Connect genes, proteins, drugs, and diseases from scientific literature and databases to accelerate drug discovery and understand disease pathways. -
- -### Biomedical Knowledge Graphs Implementation - -**Prerequisites**: -- Domain knowledge of biomedical concepts -- Access to biomedical literature/databases - -**Implementation Guides:** - -- **[Drug Discovery Pipeline Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb)**: Build knowledge graphs from PubMed RSS feeds - - **Topics**: PubMed RSS ingestion, entity-aware chunking, GraphRAG, vector similarity search - - **Difficulty**: Intermediate - - **Time**: 1-2 hours - - **Use Cases**: Drug discovery, biomedical research - -- **[Genomic Variant Analysis Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb)**: Analyze genomic variants using temporal knowledge graphs - - **Topics**: bioRxiv RSS, temporal KGs, deduplication, pathway analysis - - **Difficulty**: Intermediate - - **Time**: 1-2 hours - - **Use Cases**: Genomic research, variant analysis +**Cookbooks**: +- [Drug Discovery Pipeline](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) — PubMed RSS ingestion, entity-aware chunking, GraphRAG, vector similarity search +- [Genomic Variant Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) — bioRxiv RSS, temporal KGs, deduplication, pathway analysis --- ## Finance & Trading -
+### Financial Data Integration -- :material-finance: **Financial Data Integration** - --- - Integrate financial data from multiple sources using MCP servers and real-time ingestion. - - **Goal**: Connect Alpha Vantage API, MCP servers, seed data, and real-time ingestion for comprehensive financial analysis. - - [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) +Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph. -- :material-shield-alert: **Fraud Detection** - --- - Detect complex fraud rings using temporal knowledge graphs and pattern detection. - - **Goal**: Build a graph of Users, Devices, IP Addresses, and Transactions to find cycles and detect fraud patterns. - - [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) +**Cookbook**: [Financial Data Integration (MCP)](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) — Alpha Vantage API, MCP servers, seed data, real-time ingestion -- :material-bitcoin: **Blockchain Analytics** - --- - Analyze DeFi protocols and transaction networks for intelligence and fraud detection. - - **Goal**: Map transaction flows between wallets and exchanges, analyze DeFi protocols, and detect illicit activity. - - [:material-arrow-right: DeFi Protocol Intelligence](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb) - - [:material-arrow-right: Transaction Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb) +### Fraud Detection -
+Detect complex fraud rings using temporal graphs and pattern detection over transaction, device, and user data. ---- +**Cookbook**: [Fraud Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) — temporal KGs, cycle detection, fraud pattern analysis +### Blockchain Analytics + +Map transaction flows, analyze DeFi protocols, and detect illicit activity across wallet and exchange networks. + +**Cookbooks**: +- [DeFi Protocol Intelligence](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb) +- [Transaction Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb) --- ## Security & Intelligence -
+### Cybersecurity Threat Intelligence -- :material-shield-lock: **Cybersecurity Threat Intelligence** - --- - Proactively identify and mitigate cyber threats using real-time anomaly detection and threat intelligence. - - **Goal**: Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs. - - [:material-arrow-right: Real-Time Anomaly Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) - - [:material-arrow-right: Threat Intelligence Hybrid RAG](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) +Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs for proactive defense. -- :material-account-network: **Criminal Network Analysis** - --- - Analyze criminal networks to identify key players, communities, and suspicious patterns using OSINT RSS feeds, deduplication, and network centrality analysis. - - **Goal**: Build knowledge graphs from police reports, court records, and surveillance data. - - [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) +**Cookbooks**: +- [Real-Time Anomaly Detection](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) +- [Threat Intelligence Hybrid RAG](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) -- :material-file-search: **Intelligence Analysis Orchestrator Worker** - --- - Comprehensive intelligence analysis using pipeline orchestrator with multiple RSS feeds, conflict detection, and multi-source integration. - - **Goal**: Process multiple intelligence sources in parallel using orchestrator-worker pattern. - - [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb) +### Criminal Network Analysis -
+Build knowledge graphs from police reports, court records, and OSINT feeds to identify key players, communities, and suspicious patterns using network centrality analysis. + +**Cookbook**: [Criminal Network Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) + +### Intelligence Analysis Orchestrator + +Process multiple intelligence sources in parallel using an orchestrator-worker pipeline pattern with multi-source conflict detection and integration. + +**Cookbook**: [Intelligence Analysis Orchestrator-Worker](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb) --- ## Industry & Operations -
+### Supply Chain Optimization -- :material-truck-delivery: **Supply Chain Optimization** - --- - Visualize and optimize complex global supply chains. - - **Goal**: Map suppliers, logistics routes, and inventory levels to identify bottlenecks. - - [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb) +Map suppliers, logistics routes, and inventory levels to identify bottlenecks and optimize global supply chains. -- :material-wind-turbine: **Renewable Energy Management** - --- - Optimize grid operations and asset maintenance. - - **Goal**: Connect sensor data, weather forecasts, and maintenance logs to predict failures. - - [:material-arrow-right: View Cookbook](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb) +**Cookbook**: [Supply Chain Data Integration](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb) -
+### Renewable Energy Management + +Connect sensor data, weather forecasts, and maintenance logs to predict equipment failures and optimize grid operations. + +**Cookbook**: [Energy Market Analysis](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb) --- ## Advanced AI Patterns -
+### GraphRAG (Graph-Augmented Generation) -- :material-robot: **Graph-Augmented Generation (GraphRAG)** - --- - Enhance LLM responses with structured ground truth using knowledge graphs. - - **Goal**: Use the knowledge graph to retrieve precise context for RAG applications with hybrid retrieval and logical inference. - - [:material-arrow-right: GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) - - [:material-scale-balance: RAG vs GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) +Use knowledge graphs to retrieve precise, structured context for LLM responses — with hybrid retrieval, logical inference, and source attribution. -
- ---- - - ---- - -## Summary - -This guide covered use cases across multiple domains with corresponding cookbooks: - -- **Research & Science**: Biomedical knowledge graphs (Drug Discovery, Genomic Variant Analysis) -- **Finance & Trading**: Financial data integration, fraud detection, blockchain analytics -- **Security & Intelligence**: Cybersecurity threat intelligence, criminal network analysis, intelligence orchestration -- **Industry**: Supply chain optimization, renewable energy management -- **AI Applications**: GraphRAG (Complete implementation and comparison) +**Cookbooks**: +- [GraphRAG Complete](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — production-ready implementation +- [RAG vs. GraphRAG Comparison](https://github.com/Hawksight-AI/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) — side-by-side comparison --- ## Next Steps -- **[Examples](examples.md)** - More detailed code examples -- **[Modules Guide](modules.md)** - Learn about available modules -- **[Cookbook](cookbook.md)** - Interactive Jupyter notebooks -- **[API Reference](reference/core.md)** - Complete API documentation - ---- - -!!! info "Contribute" - Have a use case to add? [Contribute on GitHub](https://github.com/Hawksight-AI/semantica) +- [Cookbook](cookbook.md) — full notebook catalog organized by topic and difficulty +- [Modules Guide](modules.md) — every module with examples +- [API Reference](reference/core.md) — complete technical documentation +!!! info "Have a use case to add?" + [Open a PR](https://github.com/Hawksight-AI/semantica) or start a discussion on GitHub. diff --git a/mkdocs_local.yml b/mkdocs_local.yml new file mode 100644 index 00000000..26261fc3 --- /dev/null +++ b/mkdocs_local.yml @@ -0,0 +1,14 @@ +INHERIT: mkdocs.yml +plugins: + - search: + lang: en + - minify: + minify_html: true + - mkdocstrings: + handlers: + python: + options: + docstring_style: google + show_source: true + show_root_heading: true + show_category_heading: true