diff --git a/docs/architecture.md b/docs/architecture.md index aeb31d68..4d7e9faa 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -185,17 +185,7 @@ Centralized `ConfigManager` with environment variable overrides. No magic defaul | **Deduplication v2** | `blocking_v2`, `hybrid_v2`, `semantic_v2`: up to 7x faster than v1 | | **Indexed search** | Explorer search at 0.004ms on 118k nodes (v0.5.0) | - - - Full module documentation with code examples. - - - Configuration reference, performance guide, and troubleshooting. - - - Pipeline orchestration, workers, and retry policies. - - - Framework lifecycle, plugin registry, and configuration. - - +- [Modules](modules) — Full module documentation with code examples. +- [Learning More](learning-more) — Configuration reference, performance guide, and troubleshooting. +- [Pipeline Reference](reference/pipeline) — Pipeline orchestration, workers, and retry policies. +- [Core Reference](reference/core) — Framework lifecycle, plugin registry, and configuration. diff --git a/docs/assets/custom.css b/docs/assets/custom.css index 97b5e3ee..a4717786 100644 --- a/docs/assets/custom.css +++ b/docs/assets/custom.css @@ -109,6 +109,47 @@ nav a, transition: color 0.15s ease !important; } +/* ============================================================ + BULLET POINTS — cursor-animated hover (premium feel) + ============================================================ */ +ul > li, +ol > li { + position: relative; + transition: + background-color 0.18s ease, + box-shadow 0.18s ease, + color 0.15s ease; + border-radius: 4px; + cursor: default; +} + +ul > li:hover, +ol > li:hover { + background-color: rgba(16, 185, 129, 0.06); + box-shadow: inset 3px 0 0 #10B981; + color: rgba(255, 255, 255, 0.95); +} + +/* Animate the bullet marker green on hover */ +ul > li:hover::marker, +ol > li:hover::marker { + color: #10B981; +} + +/* Slide-in left accent bar for nested lists */ +ul > li > ul > li:hover, +ol > li > ul > li:hover { + background-color: rgba(16, 185, 129, 0.04); + box-shadow: inset 2px 0 0 rgba(16, 185, 129, 0.6); +} + +/* Strong text inside list items — subtle green tint on parent hover */ +ul > li:hover > strong, +ol > li:hover > strong { + color: #10B981; + transition: color 0.15s ease; +} + /* ============================================================ HIDE THEME TOGGLE (moon / sun emoji button) ============================================================ */ diff --git a/docs/citation.md b/docs/citation.md index aff60344..dce248d7 100644 --- a/docs/citation.md +++ b/docs/citation.md @@ -49,11 +49,5 @@ Published research using Semantica? [Let us know](https://github.com/semantica-a ## See Also - - - MIT License details. - - - Connect with the Semantica community. - - +- [License](project-license) — MIT License details. +- [Community](community) — Connect with the Semantica community. diff --git a/docs/cli-setup.md b/docs/cli-setup.md index dea48d68..588ef980 100644 --- a/docs/cli-setup.md +++ b/docs/cli-setup.md @@ -49,23 +49,11 @@ python -c "import semantica; print(semantica.__version__)" ## When to Use Each Command - - - The general-purpose CLI. Use it for one-off pipeline runs, entity extraction, and graph operations from a shell script or CI job. - - - Starts the REST API server. Binds to `0.0.0.0:8000`. Use this when another service or application needs programmatic access to Semantica over HTTP. - - - Background task processor. Run alongside `semantica-server` when you need async pipeline execution outside the request cycle. Start the server first, then start one or more workers pointing at the same backend. - - - Launches the browser dashboard. Requires `pip install semantica[explorer]`. Use this to explore a saved knowledge graph interactively. See [Explorer Setup](explorer-setup). - - - Runs the MCP server over stdio. Configure it in your MCP client's settings file to expose all 12 tools and 3 resources to Claude Desktop, Cursor, Windsurf, or any MCP-aware client. See [MCP Server](reference/mcp_server). - - +- **semantica** — The general-purpose CLI. Use it for one-off pipeline runs, entity extraction, and graph operations from a shell script or CI job. +- **semantica-server** — Starts the REST API server. Binds to `0.0.0.0:8000`. Use this when another service or application needs programmatic access to Semantica over HTTP. +- **semantica-worker** — Background task processor. Run alongside `semantica-server` when you need async pipeline execution outside the request cycle. Start the server first, then start one or more workers pointing at the same backend. +- **semantica-explorer** — Launches the browser dashboard. Requires `pip install semantica[explorer]`. Use this to explore a saved knowledge graph interactively. See [Explorer Setup](explorer-setup). +- **semantica-mcp** — Runs the MCP server over stdio. Configure it in your MCP client's settings file to expose all 12 tools and 3 resources to Claude Desktop, Cursor, Windsurf, or any MCP-aware client. See [MCP Server](reference/mcp_server). ## Usage Examples @@ -240,17 +228,7 @@ Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/ ## Next Steps - - - Build a graph, save it, and launch the browser dashboard. - - - All 12 tools and 3 resources exposed over the MCP protocol. - - - Virtual environments, optional extras, and platform-specific notes. - - - End-to-end pipeline walkthrough with working code. - - +- [Explorer Setup](explorer-setup) — Build a graph, save it, and launch the browser dashboard. +- [MCP Server](reference/mcp_server) — All 12 tools and 3 resources exposed over the MCP protocol. +- [Installation](installation) — Virtual environments, optional extras, and platform-specific notes. +- [Quickstart](quickstart) — End-to-end pipeline walkthrough with working code. diff --git a/docs/community-projects.md b/docs/community-projects.md index b37beafd..be6afdfe 100644 --- a/docs/community-projects.md +++ b/docs/community-projects.md @@ -114,17 +114,7 @@ See [Architecture](architecture#extension-points) for the full extension guide. ## How to Contribute - - - Submit code, documentation, tests, or cookbook notebooks. - - - Report bugs, request features, or propose integrations. - - - Share what you're building with the community. - - - Long-form questions, design discussions, and ideas. - - +- [Contributing Guide](contributing-guide) — Submit code, documentation, tests, or cookbook notebooks. +- [GitHub Issues](https://github.com/semantica-agi/semantica/issues) — Report bugs, request features, or propose integrations. +- [Discord](https://discord.gg/sV34vps5hH) — Share what you're building with the community. +- [GitHub Discussions](https://github.com/semantica-agi/semantica/discussions) — Long-form questions, design discussions, and ideas. diff --git a/docs/community.md b/docs/community.md index facd9604..36b1b5de 100644 --- a/docs/community.md +++ b/docs/community.md @@ -9,20 +9,10 @@ Semantica is built in the open, with contributions from researchers, engineers, ## Get Help - - - File bug reports and feature requests with full context. - - - Ask questions, share ideas, and discuss design decisions. - - - Browse open contributions and submit your own. - - - Report vulnerabilities privately: never in public issues. - - +- [GitHub Issues](https://github.com/semantica-agi/semantica/issues) — File bug reports and feature requests with full context. +- [GitHub Discussions](https://github.com/semantica-agi/semantica/discussions) — Ask questions, share ideas, and discuss design decisions. +- [Pull Requests](https://github.com/semantica-agi/semantica/pulls) — Browse open contributions and submit your own. +- [Security Issues](https://github.com/semantica-agi/semantica/security/advisories/new) — Report vulnerabilities privately: never in public issues. ## Community Guidelines @@ -78,17 +68,7 @@ See the [Contributing Guide](contributing-guide) for the full development workfl ## See Also - - - Step-by-step guide for submitting PRs and setting up your dev environment. - - - Projects and integrations built by the community. - - - Common questions answered. - - - How the project is run and decisions are made. - - +- [Contributing Guide](contributing-guide) — Step-by-step guide for submitting PRs and setting up your dev environment. +- [Community Projects](community-projects) — Projects and integrations built by the community. +- [FAQ](faq) — Common questions answered. +- [Governance](governance) — How the project is run and decisions are made. diff --git a/docs/concepts.md b/docs/concepts.md index 05820389..698ef7d9 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -12,17 +12,9 @@ Semantica transforms unstructured data: documents, web pages, reports, databases At its core, Semantica adds a **context and accountability layer** on top of your existing AI stack. It doesn't replace LangChain, LlamaIndex, or your LLM provider: it makes their outputs **grounded**, **traceable**, and **auditable**. - - - Knowledge graphs, GraphRAG retrieval, semantic embeddings, and temporal intelligence ground every LLM response in structured, queryable facts. - - - Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable. - - - `PluginRegistry` and `MethodRegistry` let you replace or augment any component: ingestors, extractors, reasoning engines, backends: without changing framework code. - - +- **Context Layer** — Knowledge graphs, GraphRAG retrieval, semantic embeddings, and temporal intelligence ground every LLM response in structured, queryable facts. +- **Accountability Layer** — Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable. +- **Extension Layer** — `PluginRegistry` and `MethodRegistry` let you replace or augment any component: ingestors, extractors, reasoning engines, backends: without changing framework code. ## Knowledge Graphs @@ -487,14 +479,6 @@ Semantica is designed for extension. Any component: ingestor, extractor, graph b - - - Build a full pipeline with code. - - - Every module explained with examples. - - - Complete technical reference. - - +- [Quickstart Tutorial](quickstart) — Build a full pipeline with code. +- [Modules Guide](modules) — Every module explained with examples. +- [API Reference](reference/context) — Complete technical reference. diff --git a/docs/contributing-guide.md b/docs/contributing-guide.md index 59125a7a..6018d0c8 100644 --- a/docs/contributing-guide.md +++ b/docs/contributing-guide.md @@ -22,20 +22,10 @@ New to the project? Start with [`good-first-issue`](https://github.com/semantica ## Ways to Contribute - - - Fix bugs, implement features, optimize performance, or add new ingestors, parsers, and exporters using the plugin registry. - - - Fix typos, improve clarity, add missing examples, write tutorials, or keep the API reference accurate as modules evolve. - - - Add test coverage for untested modules or edge cases, reproduce reported bugs with minimal repros, or improve cross-platform reliability. - - - Answer questions in GitHub Issues and Discussions, review pull requests with constructive feedback, or share Semantica in blog posts and talks. - - +- **Code** — Fix bugs, implement features, optimize performance, or add new ingestors, parsers, and exporters using the plugin registry. +- **Documentation** — Fix typos, improve clarity, add missing examples, write tutorials, or keep the API reference accurate as modules evolve. +- **Testing** — Add test coverage for untested modules or edge cases, reproduce reported bugs with minimal repros, or improve cross-platform reliability. +- **Community** — Answer questions in GitHub Issues and Discussions, review pull requests with constructive feedback, or share Semantica in blog posts and talks. ## Development Setup @@ -95,11 +85,5 @@ All contributors are expected to follow the [Contributor Covenant Code of Conduc - [GitHub Discussions](https://github.com/semantica-agi/semantica/discussions) - [Discord](https://discord.gg/sV34vps5hH) - - - Community guidelines and values. - - - How decisions are made and the project is run. - - +- [Community](community) — Community guidelines and values. +- [Governance](governance) — How decisions are made and the project is run. diff --git a/docs/cookbook.md b/docs/cookbook.md index aabf48b5..d661a107 100644 --- a/docs/cookbook.md +++ b/docs/cookbook.md @@ -16,131 +16,41 @@ icon: "flask" -## Featured Recipes +## Featured Recipe - - - Go from raw text to a queryable knowledge graph in 20 minutes. - - **Topics:** Extraction, Graph Construction, Visualization · **Difficulty:** Beginner - - +- **[Your First Knowledge Graph](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb)** — Go from raw text to a queryable knowledge graph in 20 minutes. Topics: Extraction, Graph Construction, Visualization · *Beginner* ## Core Tutorials Essential guides to master the Semantica framework. - - - An interactive introduction to the framework's core philosophy and all modules. - - **Topics:** Framework Overview, Architecture · **Difficulty:** Beginner - - - Loading data from files, web, databases, streams, feeds, repositories, email, and MCP. - - **Topics:** FileIngestor, WebIngestor, DBIngestor, Streams · **Difficulty:** Beginner - - - Extracting clean text from complex formats like PDF, DOCX, and HTML. - - **Topics:** OCR, PDF Parsing, Text Extraction · **Difficulty:** Beginner - - - Pipelines for cleaning, normalizing, and preparing text. - - **Topics:** Text Cleaning, Unicode, Formatting · **Difficulty:** Beginner - - - Using NER to identify people, organizations, and custom entities. - - **Topics:** NER, spaCy, LLM Extraction · **Difficulty:** Beginner - - - Discovering and classifying relationships between entities. - - **Topics:** Relation Classification, Dependency Parsing · **Difficulty:** Beginner - - - Creating and managing vector embeddings for semantic search. - - **Topics:** Embeddings, OpenAI, HuggingFace · **Difficulty:** Intermediate - - - Setting up vector stores for similarity search and retrieval. - - **Difficulty:** Intermediate - - - Persisting knowledge graphs in Neo4j or FalkorDB. - - **Topics:** Neo4j, Cypher, Persistence · **Difficulty:** Intermediate - - - Defining domain schemas and ontologies to structure your data. - - **Topics:** OWL, RDF, Schema Design · **Difficulty:** Intermediate - - +- **[Welcome to Semantica](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb)** — Interactive introduction to the framework's core philosophy and all modules. Topics: Framework Overview, Architecture · *Beginner* +- **[Data Ingestion](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/02_Data_Ingestion.ipynb)** — Loading data from files, web, databases, streams, feeds, repositories, email, and MCP. Topics: FileIngestor, WebIngestor, DBIngestor · *Beginner* +- **[Document Parsing](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/03_Document_Parsing.ipynb)** — Extracting clean text from complex formats like PDF, DOCX, and HTML. Topics: OCR, PDF Parsing, Text Extraction · *Beginner* +- **[Data Normalization](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/04_Data_Normalization.ipynb)** — Pipelines for cleaning, normalizing, and preparing text. Topics: Text Cleaning, Unicode, Formatting · *Beginner* +- **[Entity Extraction](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/05_Entity_Extraction.ipynb)** — Using NER to identify people, organizations, and custom entities. Topics: NER, spaCy, LLM Extraction · *Beginner* +- **[Relation Extraction](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/06_Relation_Extraction.ipynb)** — Discovering and classifying relationships between entities. Topics: Relation Classification, Dependency Parsing · *Beginner* +- **[Embedding Generation](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/12_Embedding_Generation.ipynb)** — Creating and managing vector embeddings for semantic search. Topics: Embeddings, OpenAI, HuggingFace · *Intermediate* +- **[Vector Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/13_Vector_Store.ipynb)** — Setting up vector stores for similarity search and retrieval. *Intermediate* +- **[Graph Store](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Graph_Store.ipynb)** — Persisting knowledge graphs in Neo4j or FalkorDB. Topics: Neo4j, Cypher, Persistence · *Intermediate* +- **[Ontology](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb)** — Defining domain schemas and ontologies to structure your data. Topics: OWL, RDF, Schema Design · *Intermediate* ## Advanced Concepts Deep dive into advanced features, customization, and complex workflows. - - - Custom extractors, LLM-based extraction, and complex pattern matching. - - **Topics:** Custom Models, Regex, LLMs · **Difficulty:** Advanced - - - Centrality, community detection, and pathfinding algorithms. - - **Topics:** PageRank, Louvain, Shortest Path · **Difficulty:** Advanced - - - Production-grade memory system for AI agents using FAISS and Neo4j. - - **Topics:** Agent Memory, GraphRAG, Entity Injection · **Difficulty:** Advanced - - - Interactive, publication-ready visualizations of your graphs. - - **Topics:** PyVis, NetworkX, D3.js · **Difficulty:** Intermediate - - - Strategies for handling contradictory information from multiple sources. - - **Topics:** Truth Discovery, Voting, Confidence · **Difficulty:** Advanced - - - Exporting to RDF, OWL, JSON-LD, and NetworkX formats. - - **Topics:** Serialization, Interoperability · **Difficulty:** Intermediate - - - Merging data from disparate sources into a unified graph. - - **Topics:** Entity Resolution, Merging, Fusion · **Difficulty:** Advanced - - - Building robust, automated data processing pipelines. - - **Topics:** Workflows, Automation, Error Handling · **Difficulty:** Advanced - - - Using logical reasoning to infer new knowledge from existing facts. - - **Topics:** Logic Rules, Inference Engines · **Difficulty:** Advanced - - - Modeling and querying data that changes over time. - - **Topics:** Time Series, Temporal Logic, Allen Algebra · **Difficulty:** Advanced - - +- **[Advanced Extraction](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/01_Advanced_Extraction.ipynb)** — Custom extractors, LLM-based extraction, and complex pattern matching. Topics: Custom Models, Regex, LLMs · *Advanced* +- **[Advanced Graph Analytics](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/02_Advanced_Graph_Analytics.ipynb)** — Centrality, community detection, and pathfinding algorithms. Topics: PageRank, Louvain, Shortest Path · *Advanced* +- **[Advanced Context Engineering](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb)** — Production-grade memory system for AI agents using FAISS and Neo4j. Topics: Agent Memory, GraphRAG, Entity Injection · *Advanced* +- **[Complete Visualization Suite](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb)** — Interactive, publication-ready visualizations of your graphs. Topics: PyVis, NetworkX, D3.js · *Intermediate* +- **[Conflict Resolution](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/17_Conflict_Detection_and_Resolution.ipynb)** — Strategies for handling contradictory information from multiple sources. Topics: Truth Discovery, Voting, Confidence · *Advanced* +- **[Multi-Format Export](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb)** — Exporting to RDF, OWL, JSON-LD, and NetworkX formats. Topics: Serialization, Interoperability · *Intermediate* +- **[Multi-Source Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb)** — Merging data from disparate sources into a unified graph. Topics: Entity Resolution, Merging, Fusion · *Advanced* +- **[Pipeline Orchestration](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/07_Pipeline_Orchestration.ipynb)** — Building robust, automated data processing pipelines. Topics: Workflows, Automation, Error Handling · *Advanced* +- **[Reasoning and Inference](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/08_Reasoning_and_Inference.ipynb)** — Using logical reasoning to infer new knowledge from existing facts. Topics: Logic Rules, Inference Engines · *Advanced* +- **[Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb)** — Modeling and querying data that changes over time. Topics: Time Series, Temporal Logic, Allen Algebra · *Advanced* ## How to Run diff --git a/docs/explorer-setup.md b/docs/explorer-setup.md index a9610a57..6714c084 100644 --- a/docs/explorer-setup.md +++ b/docs/explorer-setup.md @@ -264,17 +264,7 @@ Once running, Explorer exposes a REST API and dashboard for: The full endpoint catalogue is documented in the Swagger UI at `/docs` and in the reference page below. - - - Every REST endpoint, WebSocket events, analytics, and all supported flags. - - - All five Semantica executables and when to use each one. - - - Full documentation for ContextGraph: build, query, save, and load. - - - End-to-end pipeline: ingest → extract → build graph → export. - - +- [Explorer Reference](reference/explorer) — Every REST endpoint, WebSocket events, analytics, and all supported flags. +- [CLI Setup](cli-setup) — All five Semantica executables and when to use each one. +- [Context Module](reference/context) — Full documentation for ContextGraph: build, query, save, and load. +- [Quickstart](quickstart) — End-to-end pipeline: ingest → extract → build graph → export. diff --git a/docs/faq.md b/docs/faq.md index 8095a408..afe0c09a 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -338,14 +338,6 @@ set PYTHONIOENCODING=utf-8 ## Support - - - Community chat and live support. - - - Bug reports and feature requests. - - - Help improve Semantica. - - +- [Discord](https://discord.gg/sV34vps5hH) — Community chat and live support. +- [GitHub Issues](https://github.com/semantica-agi/semantica/issues) — Bug reports and feature requests. +- [Contributing](contributing-guide) — Help improve Semantica. diff --git a/docs/getting-started.md b/docs/getting-started.md index 0bc4a266..be02345b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -10,20 +10,10 @@ icon: "rocket" ## What You Can Build - - - Ground LLM responses in traceable, structured knowledge. Every claim links back to a source node. - - - Agents with structured decision history, causal chains, and precedent search. Every choice is recorded and auditable. - - - Build, validate, and maintain enterprise-grade semantic knowledge bases from multi-source data. - - - W3C PROV-O provenance on every fact. HIPAA, SOX, GDPR, FDA 21 CFR Part 11 infrastructure built in. - - +- **GraphRAG Systems** — Ground LLM responses in traceable, structured knowledge. Every claim links back to a source node. +- **Accountable AI Agents** — Agents with structured decision history, causal chains, and precedent search. Every choice is recorded and auditable. +- **Production Knowledge Graphs** — Build, validate, and maintain enterprise-grade semantic knowledge bases from multi-source data. +- **Compliance-Ready AI** — W3C PROV-O provenance on every fact. HIPAA, SOX, GDPR, FDA 21 CFR Part 11 infrastructure built in. ## Setup in 3 Steps @@ -204,32 +194,12 @@ icon: "rocket" Semantica uses a modular, layered architecture: import only what you need. - - - Load and prepare data from any source. - **Modules:** `ingest`, `parse`, `split`, `normalize` - - - Extract meaning from raw text. - **Modules:** `semantic_extract`, `kg`, `ontology`, `reasoning` - - - Persist knowledge for retrieval. - **Modules:** `embeddings`, `vector_store`, `graph_store`, `triplet_store` - - - Validate and deduplicate. - **Modules:** `deduplication`, `conflicts` - - - Track decisions and lineage. - **Modules:** `context`, `provenance`, `change_management` - - - Deliver results downstream. - **Modules:** `export`, `visualization`, `pipeline`, `explorer` - - +- **[Input Layer](reference/ingest)** — Load and prepare data from any source. Modules: `ingest`, `parse`, `split`, `normalize` +- **[Semantic Layer](reference/semantic_extract)** — Extract meaning from raw text. Modules: `semantic_extract`, `kg`, `ontology`, `reasoning` +- **[Storage Layer](reference/vector_store)** — Persist knowledge for retrieval. Modules: `embeddings`, `vector_store`, `graph_store`, `triplet_store` +- **[Quality Layer](reference/deduplication)** — Validate and deduplicate. Modules: `deduplication`, `conflicts` +- **[Context Layer](reference/context)** — Track decisions and lineage. Modules: `context`, `provenance`, `change_management` +- **[Output Layer](reference/export)** — Deliver results downstream. Modules: `export`, `visualization`, `pipeline`, `explorer` ## "Which module do I need?" Quick Reference @@ -252,32 +222,14 @@ Semantica uses a modular, layered architecture: import only what you need. ## Next Steps - - - Knowledge graphs, ontologies, and reasoning explained in depth. - - - Full 6-step pipeline walkthrough with working code. - - - Every module, class, and common chain explained. - - - Complete module documentation for every class and method. - - +- [Core Concepts](concepts) — Knowledge graphs, ontologies, and reasoning explained in depth. +- [Quickstart Tutorial](quickstart) — Full 6-step pipeline walkthrough with working code. +- [Module Reference](modules) — Every module, class, and common chain explained. +- [API Reference](reference/context) — Complete module documentation for every class and method. ## Help - - - Ask questions, share projects, get community support. - - - Report bugs or request features. - - - Common questions answered. - - +- [Discord](https://discord.gg/sV34vps5hH) — Ask questions, share projects, get community support. +- [GitHub Issues](https://github.com/semantica-agi/semantica/issues) — Report bugs or request features. +- [FAQ](faq) — Common questions answered. diff --git a/docs/glossary.md b/docs/glossary.md index d0776e6a..385da52e 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -214,17 +214,7 @@ A vulnerability in XML parsers that allows attackers to read arbitrary files or ## See Also - - - Deeper explanation of key ideas with code examples. - - - First working examples: no prior graph experience required. - - - All 27 modules explained with code and pipeline chains. - - - Complete technical reference for every class and method. - - +- [Core Concepts](concepts) — Deeper explanation of key ideas with code examples. +- [Getting Started](getting-started) — First working examples: no prior graph experience required. +- [Modules Guide](modules) — All 27 modules explained with code and pipeline chains. +- [API Reference](reference/context) — Complete technical reference for every class and method. diff --git a/docs/governance.md b/docs/governance.md index c1233160..77fc18b2 100644 --- a/docs/governance.md +++ b/docs/governance.md @@ -9,17 +9,9 @@ icon: "scale-balanced" ## Roles - - - Hawksight AI team: review and merge PRs, manage releases and code quality, set project direction and community standards. - - - Submit code, documentation, and bug reports. Help with issues and reviews. Recognized in [CONTRIBUTORS.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTORS.md). - - - Use Semantica, provide feedback, share use cases, and participate in GitHub Discussions and Discord. - - +- **Maintainers** — Hawksight AI team: review and merge PRs, manage releases and code quality, set project direction and community standards. +- **Contributors** — Submit code, documentation, and bug reports. Help with issues and reviews. Recognized in [CONTRIBUTORS.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTORS.md). +- **Community Members** — Use Semantica, provide feedback, share use cases, and participate in GitHub Discussions and Discord. ## Decision Process @@ -73,23 +65,11 @@ Semantica follows **Semantic Versioning** (`MAJOR.MINOR.PATCH`): ## Project Goals - - - Easy to use and understand: sensible defaults, clear documentation, minimal ceremony. - - - Production-ready quality: tested across Python versions, platforms, and real-world workloads. - - - Efficient and scalable: from single-machine notebooks to enterprise graph databases. - - - Easy to extend with plugins and custom modules via the `PluginRegistry` pattern. - - - Welcoming and inclusive: all backgrounds and experience levels contribute and are recognized. - - +- **Usability** — Easy to use and understand: sensible defaults, clear documentation, minimal ceremony. +- **Reliability** — Production-ready quality: tested across Python versions, platforms, and real-world workloads. +- **Performance** — Efficient and scalable: from single-machine notebooks to enterprise graph databases. +- **Extensibility** — Easy to extend with plugins and custom modules via the `PluginRegistry` pattern. +- **Community** — Welcoming and inclusive: all backgrounds and experience levels contribute and are recognized. ## License @@ -99,11 +79,5 @@ MIT License: see [LICENSE](https://github.com/semantica-agi/semantica/blob/main/ ## See Also - - - How to submit changes. - - - Community guidelines and channels. - - +- [Contributing](contributing-guide) — How to submit changes. +- [Community](community) — Community guidelines and channels. diff --git a/docs/index.md b/docs/index.md index 11eaddc6..4de8e1a3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,7 +4,7 @@ description: "The Accountability and Context Layer for AI: Context Graphs · Dec --- - **v0.5.0 is live**: Ontology Hub, Distance Intelligence, SHACL Studio, Parquet & XML ingestion, 12 security fixes. [What's new →](#whats-new) + **v0.5.0 is live** — Ontology Hub, Distance Intelligence, SHACL Studio, Parquet & XML ingestion, 12 security fixes. What's new → Your AI agent just made a decision. Now someone needs to explain it. @@ -15,58 +15,42 @@ If your stack can't answer those questions with a traceable record, you have a g **Semantica closes that gap.** It's the context and accountability layer that sits beneath your existing agent framework: not a replacement for LangChain or LlamaIndex, but the infrastructure that makes their outputs trustworthy. - - - Production-hardened with a full regression suite - - - Every capability independently importable - - - OpenAI, Anthropic, Ollama, Groq, and more - - - Open source, no vendor lock-in, fully forkable - - +
+
1,000+
passing tests
+
25+
modules
+
12
LLM providers
+
MIT
open source
+
## The Problem Every Production AI Team Hits Powerful agents aren't automatically trustworthy ones. Five structural blind spots make modern AI systems impossible to deploy in regulated environments: - - - Agents store embeddings, not meaning. - - No way to ask *why* a fact was recalled - - No link from a recalled fact back to its source document - - Context is a black box that resets on every run - - - Agents act continuously but record nothing. - - No history to hand to a regulator or auditor - - No way to replay or reproduce a past decision - - Debugging means re-running, not reviewing - - - Outputs can't be traced to source facts. - - In healthcare, finance, and legal: this is a hard compliance blocker - - No lineage from inference back to the original document - - Impossible to demonstrate what the agent actually relied on - - - Black-box answers with no explanation. - - Impossible to validate the reasoning path - - Impossible to contest a specific conclusion - - No basis for improving or correcting future behavior - - - Contradictory facts silently coexist in vector stores. - - No detection when two sources disagree - - Outputs become inconsistent and unpredictable over time - - Silent failures compound as the knowledge base grows - - +**No memory structure** — agents store embeddings, not meaning +- No way to ask *why* a fact was recalled +- No link from a recalled fact back to its source document +- Context is a black box that resets on every run + +**No decision trail** — agents act continuously but record nothing +- No history to hand to a regulator or auditor +- No way to replay or reproduce a past decision +- Debugging means re-running, not reviewing + +**No provenance** — outputs can't be traced to source facts +- In healthcare, finance, and legal: this is a hard compliance blocker +- No lineage from inference back to the original document +- Impossible to demonstrate what the agent actually relied on + +**No reasoning transparency** — black-box answers with no explanation +- Impossible to validate the reasoning path +- Impossible to contest a specific conclusion +- No basis for improving or correcting future behavior + +**No conflict detection** — contradictory facts silently coexist in vector stores +- No detection when two sources disagree +- Outputs become inconsistent and unpredictable over time +- Silent failures compound as the knowledge base grows These aren't edge cases. They're why enterprise AI pilots stall: and why your compliance team keeps saying *not yet*. @@ -77,50 +61,41 @@ Powerful agents aren't automatically trustworthy ones. Five structural blind spo Semantica gives every agent the infrastructure it needs to be accountable. Drop it into your existing setup in minutes: - - - A structured, queryable graph of everything your agent knows, decides, and reasons about. - - Persistent across agent runs: no context loss between sessions - - Queryable with SPARQL and full graph algorithms - - Temporal model with `valid_from` / `valid_until` on nodes and edges - - Point-in-time snapshots of the full knowledge state - - - Every decision is a first-class object in your system. - - `record_decision()` captures full lifecycle and causal chain - - Hybrid precedent search over past decisions for consistency - - `analyze_decision_impact()` shows downstream consequences - - Causal chain visualization from trigger to outcome - - - Every fact links to its source document and ingestion event. - - W3C PROV-O compliant lineage across all modules - - Full traceability from raw input to final inference - - `recorded_at` stamping with OWL-Time export - - Audit-ready for HIPAA, SOX, GDPR, FDA 21 CFR Part 11 - - - Explainable reasoning paths: not black boxes. - - Forward chaining, Rete, deductive, abductive - - SPARQL query-based inference over RDF graphs - - Datalog with recursive Horn clause rules - - Every conclusion backed by a traceable derivation path - - - Your graph knows not just *what*: but *when*. - - Allen interval algebra: all 13 temporal relations - - Point-in-time queries over historical graph states - - Temporal provenance stamping on every fact - - OWL-Time export for standards-compliant archiving - - - Full ontology lifecycle in the browser. - - Visual editor for schema design and editing - - SHACL Studio for constraint authoring and validation - - Alignment authoring across multiple ontologies - - Health dashboard and version control built in - - +**Context Graphs** — a structured, queryable graph of everything your agent knows, decides, and reasons about +- Persistent across agent runs: no context loss between sessions +- Queryable with SPARQL and full graph algorithms +- Temporal model with `valid_from` / `valid_until` on nodes and edges +- Point-in-time snapshots of the full knowledge state + +**Decision Intelligence** — every decision is a first-class object in your system +- `record_decision()` captures full lifecycle and causal chain +- Hybrid precedent search over past decisions for consistency +- `analyze_decision_impact()` shows downstream consequences +- Causal chain visualization from trigger to outcome + +**Full Provenance** — every fact links to its source document and ingestion event +- W3C PROV-O compliant lineage across all modules +- Full traceability from raw input to final inference +- `recorded_at` stamping with OWL-Time export +- Audit-ready for HIPAA, SOX, GDPR, FDA 21 CFR Part 11 + +**Reasoning Engines** — explainable reasoning paths, not black boxes +- Forward chaining, Rete, deductive, abductive +- SPARQL query-based inference over RDF graphs +- Datalog with recursive Horn clause rules +- Every conclusion backed by a traceable derivation path + +**Temporal Intelligence** — your graph knows not just *what*, but *when* +- Allen interval algebra: all 13 temporal relations +- Point-in-time queries over historical graph states +- Temporal provenance stamping on every fact +- OWL-Time export for standards-compliant archiving + +**Ontology Hub** — full ontology lifecycle in the browser +- Visual editor for schema design and editing +- SHACL Studio for constraint authoring and validation +- Alignment authoring across multiple ontologies +- Health dashboard and version control built in Works alongside any LLM provider and any agent framework: add it to an existing stack without changing your architecture. @@ -217,61 +192,50 @@ decision_id = context.record_decision( - - - Step-by-step pipeline walkthrough - - - 40+ real-world Jupyter notebooks - - - Community chat and support - - +- [Full Quickstart](quickstart) — Step-by-step pipeline walkthrough +- [Cookbook](cookbook) — 40+ real-world Jupyter notebooks +- [Join Discord](https://discord.gg/sV34vps5hH) — Community chat and support ## Built for Where Mistakes Have Consequences Semantica was designed for domains where every decision must be explainable and every fact must be traceable: - - - - Clinical decision support with full audit trails - - Drug interaction and contraindication graphs - - Patient safety event tracking and root-cause analysis - - HIPAA-compliant provenance chains out of the box - - - - Fraud detection knowledge graphs - - Risk assessment trails built to survive an audit - - SOX, GDPR, and MiFID II compliance infrastructure - - Model decision lineage for regulatory reporting - - - - Evidence-backed research with every cited fact provenance-linked - - Contract analysis with traceable clause extraction - - Regulatory change tracking across jurisdictions - - Full reasoning paths ready for court-admissible documentation - - - - Threat attribution graphs linking actors, TTPs, and indicators - - Incident response timelines with full event provenance - - Security audit trails across the complete kill chain - - MITRE ATT&CK-aligned knowledge graph integration - - - - Policy decision trails from brief to outcome - - Classified information handling with provenance chains - - Chain-of-custody scrutiny for intelligence reporting - - Air-gapped deployment with local LLM support - - - - Power grid state tracking with temporal intelligence - - Transportation safety event graphs - - Emergency response coordination with decision audit trails - - Consequence modeling for high-stakes operational decisions - - +**Healthcare & Life Sciences** +- Clinical decision support with full audit trails +- Drug interaction and contraindication graphs +- Patient safety event tracking and root-cause analysis +- HIPAA-compliant provenance chains out of the box + +**Finance & Risk** +- Fraud detection knowledge graphs +- Risk assessment trails built to survive an audit +- SOX, GDPR, and MiFID II compliance infrastructure +- Model decision lineage for regulatory reporting + +**Legal & Compliance** +- Evidence-backed research with every cited fact provenance-linked +- Contract analysis with traceable clause extraction +- Regulatory change tracking across jurisdictions +- Full reasoning paths ready for court-admissible documentation + +**Cybersecurity** +- Threat attribution graphs linking actors, TTPs, and indicators +- Incident response timelines with full event provenance +- Security audit trails across the complete kill chain +- MITRE ATT&CK-aligned knowledge graph integration + +**Government & Defense** +- Policy decision trails from brief to outcome +- Classified information handling with provenance chains +- Chain-of-custody scrutiny for intelligence reporting +- Air-gapped deployment with local LLM support + +**Critical Infrastructure** +- Power grid state tracking with temporal intelligence +- Transportation safety event graphs +- Emergency response coordination with decision audit trails +- Consequence modeling for high-stakes operational decisions ## Start Here @@ -305,23 +269,11 @@ Semantica was designed for domains where every decision must be explainable and - - - Get Semantica installed in under a minute - - - Build a complete knowledge graph pipeline in 5 minutes - - - The mental model behind the API - - - Exact module, class, and method details - - - Domain notebooks for real-world use cases - - +- [Installation](installation) — Get Semantica installed in under a minute +- [Quickstart](quickstart) — Build a complete knowledge graph pipeline in 5 minutes +- [Core Concepts](concepts) — The mental model behind the API +- [API Reference](reference/context) — Exact module, class, and method details +- [Cookbook](cookbook) — Domain notebooks for real-world use cases ## What's New @@ -332,15 +284,13 @@ Semantica was designed for domains where every decision must be explainable and Released **May 11, 2026** -| Area | Highlights | -| :------ | :------------ | -| **Ontology Hub** | Visual editor, SHACL Studio, alignment authoring, health dashboard, version control: full ontology lifecycle in the browser | -| **Distance Intelligence** | Semantic neighborhoods, N×N distance matrices, ego-mode visualization, distance band classification, embedding cache optimization | -| **Parquet Ingestion** | `ParquetIngestor` with PyArrow: single file, partitioned directories, Hive-style discovery, selective column reading | -| **XML Ingestion** | `XMLIngestor` with XXE-safe lxml backend, XSD/DTD validation, namespace handling, directory scanning | -| **Graph Explorer** | Landing page redesign, bidirectional path finding, indexed search (0.004ms on 118k nodes) | -| **Security** | 12 vulnerability fixes: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, path traversal | -| **Bug Fixes** | NER LLM silent fallback on enterprise gateways, ConflictDetector duplicate definition, Windows `[all]` install, cp1252 crash | +- **Ontology Hub** — Visual editor, SHACL Studio, alignment authoring, health dashboard, version control: full ontology lifecycle in the browser +- **Distance Intelligence** — Semantic neighborhoods, N×N distance matrices, ego-mode visualization, distance band classification, embedding cache optimization +- **Parquet Ingestion** — `ParquetIngestor` with PyArrow: single file, partitioned directories, Hive-style discovery, selective column reading +- **XML Ingestion** — `XMLIngestor` with XXE-safe lxml backend, XSD/DTD validation, namespace handling, directory scanning +- **Graph Explorer** — Landing page redesign, bidirectional path finding, indexed search (0.004ms on 118k nodes) +- **Security** — 12 vulnerability fixes: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, path traversal +- **Bug Fixes** — NER LLM silent fallback on enterprise gateways, ConflictDetector duplicate definition, Windows `[all]` install, cp1252 crash ```bash pip install semantica==0.5.0 @@ -350,13 +300,11 @@ pip install semantica==0.5.0 -| Area | Highlights | -| :------ | :------------ | -| **Temporal Intelligence** | 6-PR system: temporal data model, point-in-time queries, Allen interval algebra (all 13 relations), OWL-Time export | -| **Knowledge Explorer API** | Full FastAPI backend: 99 tests, 12 export formats, WebSocket progress, thread-safe sessions, audit trail | -| **Ontology Foundations** | SHACL generation/validation, SKOS vocabulary, ontology alignment API, diff & migration tooling | -| **Datalog Reasoning** | Pure-Python bottom-up semi-naive fixpoint, recursive Horn clause rules, guaranteed termination | -| **Agno Integration** | 5 components: graph-backed memory, multi-hop GraphRAG, decision toolkit, KG toolkit, shared team context; 110 tests | +- **Temporal Intelligence** — 6-PR system: temporal data model, point-in-time queries, Allen interval algebra (all 13 relations), OWL-Time export +- **Knowledge Explorer API** — Full FastAPI backend: 99 tests, 12 export formats, WebSocket progress, thread-safe sessions, audit trail +- **Ontology Foundations** — SHACL generation/validation, SKOS vocabulary, ontology alignment API, diff & migration tooling +- **Datalog Reasoning** — Pure-Python bottom-up semi-naive fixpoint, recursive Horn clause rules, guaranteed termination +- **Agno Integration** — 5 components: graph-backed memory, multi-hop GraphRAG, decision toolkit, KG toolkit, shared team context; 110 tests @@ -483,26 +431,20 @@ pip install semantica==0.5.0 ## Why Semantica? - - - No vendor lock-in. No paywalled features. - - Full source available on GitHub - - Every line auditable by your security team - - Fork, extend, and self-host with no restrictions - - No telemetry, no usage reporting - - - Built for teams that can't afford surprises. - - 1,000+ passing tests with full regression coverage - - `PipelineValidator` catches configuration errors at startup - - `FailureHandler` with exponential backoff and dead-letter queues - - 12 security vulnerabilities fixed in v0.5.0 - - - Import only what you need. - - Use `NERExtractor` without a graph store - - Use `ContextGraph` without vector storage - - Every component independently swappable and testable - - No framework lock-in: works with any agent stack - - +**Open Source, MIT** — No vendor lock-in. No paywalled features. +- Full source available on GitHub +- Every line auditable by your security team +- Fork, extend, and self-host with no restrictions +- No telemetry, no usage reporting + +**Production Ready** — Built for teams that can't afford surprises. +- 1,000+ passing tests with full regression coverage +- `PipelineValidator` catches configuration errors at startup +- `FailureHandler` with exponential backoff and dead-letter queues +- 12 security vulnerabilities fixed in v0.5.0 + +**Modular by Design** — Import only what you need. +- Use `NERExtractor` without a graph store +- Use `ContextGraph` without vector storage +- Every component independently swappable and testable +- No framework lock-in: works with any agent stack diff --git a/docs/installation.md b/docs/installation.md index 7aa1f179..f5adfeae 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -183,14 +183,6 @@ Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/ ## Next Steps - - - Understand what Semantica does before you build. - - - Follow the end-to-end workflow with code. - - - See notebook examples organized by use case. - - +- [Getting Started](getting-started) — Understand what Semantica does before you build. +- [Build the Pipeline](quickstart) — Follow the end-to-end workflow with code. +- [Browse Examples](cookbook) — See notebook examples organized by use case. diff --git a/docs/integrations/agno.md b/docs/integrations/agno.md index 8ca7bffc..e368d86b 100644 --- a/docs/integrations/agno.md +++ b/docs/integrations/agno.md @@ -24,23 +24,11 @@ pip install "semantica[agno,graph-neo4j,vectorstore-pgvector]" ## Components at a Glance - - - `AgentMemory(db=…)`: Replaces Agno's flat storage with hybrid vector + context graph memory. Adds decision tracking and precedent search to any agent. - - - `Agent(knowledge=…)`: Documents flow through the full Semantica extraction pipeline into a queryable `ContextGraph` with multi-hop GraphRAG. - - - `Agent(tools=[…])`: 6 decision intelligence tools: record decisions, find precedents, trace causal chains, analyze impact, check policies, summarize history. - - - `Agent(tools=[…])`: 7 KG construction tools: extract entities, extract relations, add to graph, query graph, find related, infer facts, export subgraph. - - - Team-level: A single `ContextGraph` shared across all agents. Each agent gets a role-scoped view via `bind_agent()`. Writes are tagged by role. - - +- **AgnoContextStore** — `AgentMemory(db=…)`: Replaces Agno's flat storage with hybrid vector + context graph memory. Adds decision tracking and precedent search to any agent. +- **AgnoKnowledgeGraph** — `Agent(knowledge=…)`: Documents flow through the full Semantica extraction pipeline into a queryable `ContextGraph` with multi-hop GraphRAG. +- **AgnoDecisionKit** — `Agent(tools=[…])`: 6 decision intelligence tools: record decisions, find precedents, trace causal chains, analyze impact, check policies, summarize history. +- **AgnoKGToolkit** — `Agent(tools=[…])`: 7 KG construction tools: extract entities, extract relations, add to graph, query graph, find related, infer facts, export subgraph. +- **AgnoSharedContext** — Team-level: A single `ContextGraph` shared across all agents. Each agent gets a role-scoped view via `bind_agent()`. Writes are tagged by role. ## Component Details @@ -233,17 +221,7 @@ All five classes are usable without `agno` installed: they carry the full Semant ## See Also - - - AgentContext and ContextGraph backing the integration. - - - KG construction used by AgnoKnowledgeGraph. - - - Configure LLM providers for Agno agents. - - - Vector backend for AgnoContextStore. - - +- [Context Module](../reference/context) — AgentContext and ContextGraph backing the integration. +- [Knowledge Graph](../reference/kg) — KG construction used by AgnoKnowledgeGraph. +- [LLMs](../reference/llms) — Configure LLM providers for Agno agents. +- [Vector Store](../reference/vector_store) — Vector backend for AgnoContextStore. diff --git a/docs/integrations/docling.md b/docs/integrations/docling.md index 29e3b402..f246be0a 100644 --- a/docs/integrations/docling.md +++ b/docs/integrations/docling.md @@ -11,20 +11,10 @@ icon: "file-lines" Docling is integrated into Semantica's `parse` module via the **`DoclingParser`**. Documents pass through Docling's **layout engine**, then feed directly into Semantica's extraction and KG pipeline. - - - PDF, DOCX, PPTX, HTML, and more. - - - High-fidelity table parsing with header detection. - - - Built-in OCR for scanned documents. - - - Clean Markdown output optimized for LLM consumption. - - +- **Multi-format** — PDF, DOCX, PPTX, HTML, and more. +- **Table Extraction** — High-fidelity table parsing with header detection. +- **OCR Support** — Built-in OCR for scanned documents. +- **Markdown Export** — Clean Markdown output optimized for LLM consumption. ## Installation @@ -100,17 +90,7 @@ print(f"Pages: {result.get('total_pages')}") ## See Also - - - Full DocumentParser and DoclingParser reference. - - - Loading documents before parsing. - - - NER and relation extraction on parsed text. - - - Using DoclingParser in a full pipeline. - - +- [Parse Module](../reference/parse) — Full DocumentParser and DoclingParser reference. +- [Ingest Module](../reference/ingest) — Loading documents before parsing. +- [Semantic Extract](../reference/semantic_extract) — NER and relation extraction on parsed text. +- [Pipeline](../reference/pipeline) — Using DoclingParser in a full pipeline. diff --git a/docs/integrations/snowflake.md b/docs/integrations/snowflake.md index 3bd4725a..b5cac3c8 100644 --- a/docs/integrations/snowflake.md +++ b/docs/integrations/snowflake.md @@ -171,17 +171,7 @@ if not connector.test_connection(): ## See Also - - - Full SnowflakeIngestor and all other ingestors. - - - Use Snowflake ingestion as a pipeline step. - - - All optional dependency extras. - - - Build a KG from ingested Snowflake data. - - +- [Ingest Module](../reference/ingest) — Full SnowflakeIngestor and all other ingestors. +- [Pipeline](../reference/pipeline) — Use Snowflake ingestion as a pipeline step. +- [Installation](../installation) — All optional dependency extras. +- [Knowledge Graph](../reference/kg) — Build a KG from ingested Snowflake data. diff --git a/docs/learning-more.md b/docs/learning-more.md index 55f674f4..5e8ed438 100644 --- a/docs/learning-more.md +++ b/docs/learning-more.md @@ -9,20 +9,9 @@ Whether you're running your first pipeline or deploying Semantica in production, ## Learning Paths - - - New to Semantica and knowledge graphs. - [Start with Installation →](installation) - - - Comfortable with basics, building real applications. - [Start with Modules →](modules) - - - Enterprise deployments, customization, and extension. - [Start with Architecture →](architecture) - - +- **Beginner (1–2 hrs)** — New to Semantica and knowledge graphs. [Start with Installation →](installation) +- **Intermediate (4–6 hrs)** — Comfortable with basics, building real applications. [Start with Modules →](modules) +- **Advanced (8+ hrs)** — Enterprise deployments, customization, and extension. [Start with Architecture →](architecture) @@ -247,14 +236,6 @@ The `blocking_v2`, `hybrid_v2`, and `semantic_v2` strategies reduce O(n²) compa - **Graph exports**: encrypt sensitive exports at rest; use the v0.5.0 SSRF-safe `base_url` validation when configuring custom LLM gateways - **XML ingestion**: always use `XMLIngestor` (v0.5.0), which uses the XXE-safe lxml backend; never parse untrusted XML with the standard library parser - - - Interactive Jupyter notebooks from beginner to advanced. - - - Common questions answered. - - - Complete technical documentation. - - +- [Cookbook](cookbook) — Interactive Jupyter notebooks from beginner to advanced. +- [FAQ](faq) — Common questions answered. +- [API Reference](reference/core) — Complete technical documentation. diff --git a/docs/modules.md b/docs/modules.md index 9f45e334..cbe50b4c 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -12,26 +12,12 @@ Semantica is organized into **27 modules** across six logical layers. Each modul ## Architecture Overview - - - Data ingestion and preparation. **Modules:** Ingest, Parse, Split, Normalize - - - Intelligence and understanding. **Modules:** Semantic Extract, KG, Ontology, Reasoning - - - Persistent data storage. **Modules:** Embeddings, Vector Store, Graph Store, Triplet Store - - - Data quality and consistency. **Modules:** Deduplication, Conflicts - - - Agent memory and decision tracking. **Modules:** Context, Provenance, Change Management - - - Export, visualization, and workflows. **Modules:** Export, Visualization, Pipeline, Explorer - - +- **Input Layer** — Data ingestion and preparation. Modules: `ingest`, `parse`, `split`, `normalize` +- **Core Processing** — Intelligence and understanding. Modules: `semantic_extract`, `kg`, `ontology`, `reasoning` +- **Storage** — Persistent data storage. Modules: `embeddings`, `vector_store`, `graph_store`, `triplet_store` +- **Quality Assurance** — Data quality and consistency. Modules: `deduplication`, `conflicts` +- **Context & Memory** — Agent memory and decision tracking. Modules: `context`, `provenance`, `change_management` +- **Output & Orchestration** — Export, visualization, and workflows. Modules: `export`, `visualization`, `pipeline`, `explorer` ## Input Layer @@ -712,14 +698,6 @@ versioner.create_snapshot(kg, "2024-Q1", author="user@example.com", description= | [core](reference/core) | Base classes & registry | `Semantica`, `ConfigManager`, `PluginRegistry`, `LifecycleManager` | | [utils](reference/utils) | Shared utilities | `helpers`, `validators` | - - - Your first knowledge graph in 5 minutes. - - - 40+ domain notebooks with real-world examples. - - - Full technical documentation. - - +- [Getting Started](getting-started) — Your first knowledge graph in 5 minutes. +- [Cookbook](cookbook) — 40+ domain notebooks with real-world examples. +- [API Reference](reference/context) — Full technical documentation. diff --git a/docs/project-license.md b/docs/project-license.md index 2eaf2efe..b1fb31ef 100644 --- a/docs/project-license.md +++ b/docs/project-license.md @@ -76,11 +76,5 @@ By contributing to Semantica, you agree that your contributions will be licensed ## See Also - - - How to contribute to the project. - - - How to cite Semantica in research. - - +- [Contributing](contributing-guide) — How to contribute to the project. +- [Citation](citation) — How to cite Semantica in research. diff --git a/docs/quickstart.md b/docs/quickstart.md index eab082f6..0584fe23 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -5,7 +5,7 @@ icon: "rocket" --- - **v0.5.0**: Ontology Hub, Distance Intelligence, Parquet & XML ingestion, 12 security fixes. [What's new →](index#whats-new) + **v0.5.0** — Ontology Hub, Distance Intelligence, Parquet & XML ingestion, 12 security fixes. What's new → This guide walks you through the end-to-end pipeline for building your first knowledge graph. Start here after installation. An LLM API key is optional: pattern-based extraction works out of the box. @@ -420,17 +420,7 @@ pip install --upgrade semantica ## Next Steps - - - Knowledge graphs, ontologies, reasoning engines: the mental model behind Semantica. - - - Every module explained with key classes and common chains. - - - Complete documentation for every module, class, and parameter. - - - 40+ interactive Jupyter notebooks with real-world datasets. - - +- [Core Concepts](concepts) — Knowledge graphs, ontologies, reasoning engines: the mental model behind Semantica. +- [Module Reference](modules) — Every module explained with key classes and common chains. +- [API Reference](reference/context) — Complete documentation for every module, class, and parameter. +- [Cookbook](cookbook) — 40+ interactive Jupyter notebooks with real-world datasets. diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md index 18f98754..61b11824 100644 --- a/docs/reference/change_management.md +++ b/docs/reference/change_management.md @@ -30,26 +30,12 @@ icon: "clock-rotate-left" ## What You Get - - - Snapshot, diff, rollback, and per-entity audit trail for knowledge graphs. - - - Version control for OWL ontologies with diff and schema migration support. - - - Pluggable backends: `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production. - - - SHA-256 checksums on every snapshot to detect any unauthorised modification. - - - Internal metadata validated on every snapshot: ISO 8601 timestamp, email author, and description (max 500 chars). - - - Full tamper-evident version history via `list_versions()` and `diff()` for regulatory review. - - +- **TemporalVersionManager** — Snapshot, diff, rollback, and per-entity audit trail for knowledge graphs. +- **OntologyVersionManager** — Version control for OWL ontologies with diff and schema migration support. +- **VersionStorage** — Pluggable backends: `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production. +- **Integrity Verification** — SHA-256 checksums on every snapshot to detect any unauthorised modification. +- **ChangeLogEntry** — Internal metadata validated on every snapshot: ISO 8601 timestamp, email author, and description (max 500 chars). +- **Version History** — Full tamper-evident version history via `list_versions()` and `diff()` for regulatory review. ## Typical Workflow @@ -364,17 +350,7 @@ for record in history: - - - W3C PROV-O lineage tracking. - - - The graph being versioned. - - - Export versioned snapshots. - - - Detect conflicts introduced between versions. - - +- [Provenance](provenance) — W3C PROV-O lineage tracking. +- [Knowledge Graph](kg) — The graph being versioned. +- [Export](export) — Export versioned snapshots. +- [Conflicts](conflicts) — Detect conflicts introduced between versions. diff --git a/docs/reference/conflicts.md b/docs/reference/conflicts.md index da2ea361..977b1ded 100644 --- a/docs/reference/conflicts.md +++ b/docs/reference/conflicts.md @@ -45,26 +45,12 @@ Semantica's conflict detection makes disagreements explicit and actionable: ## What You Get - - - Value, type, and relationship conflict detection across entity and relationship lists. - - - 7 resolution strategies including voting, credibility-weighted, and temporal preference. - - - Track which source each conflicting fact came from, with per-source credibility scores. - - - Pattern analysis, severity grouping, source-level statistics, and trend identification. - - - Auto-generate step-by-step investigation checklists for human and expert review. - - - `detect_conflicts()` and `resolve_conflicts()` for one-call workflows. - - +- **ConflictDetector** — Value, type, and relationship conflict detection across entity and relationship lists. +- **ConflictResolver** — 7 resolution strategies including voting, credibility-weighted, and temporal preference. +- **SourceTracker** — Track which source each conflicting fact came from, with per-source credibility scores. +- **ConflictAnalyzer** — Pattern analysis, severity grouping, source-level statistics, and trend identification. +- **InvestigationGuideGenerator** — Auto-generate step-by-step investigation checklists for human and expert review. +- **Convenience Functions** — `detect_conflicts()` and `resolve_conflicts()` for one-call workflows. ## Quick Start @@ -464,17 +450,7 @@ class InvestigationStep: - - - Resolve duplicate entities before conflict detection. - - - Logical conflicts use SHACL shapes and ontology axioms. - - - Track which source each conflicting fact came from. - - - The graph being checked for conflicts. - - +- [Deduplication](deduplication) — Resolve duplicate entities before conflict detection. +- [Ontology](ontology) — Logical conflicts use SHACL shapes and ontology axioms. +- [Provenance](provenance) — Track which source each conflicting fact came from. +- [Knowledge Graph](kg) — The graph being checked for conflicts. diff --git a/docs/reference/context.md b/docs/reference/context.md index ed5203de..e013667e 100644 --- a/docs/reference/context.md +++ b/docs/reference/context.md @@ -29,48 +29,30 @@ icon: "brain" ## What You Get - - - - Memory, decision tracking, and graph-backed retrieval behind one API - - Conversation history and checkpoint diffing - - Persist and restore full context state to disk - - - - Thread-safe in-memory knowledge graph - - PageRank, centrality, community detection, temporal validity - - Cross-graph navigation and link traversal - - - - Embedding-backed memory with retention policy - - LRU eviction at configurable `max_memory_size` - - Per-conversation history isolation - - - - Records decisions with causal chains and confidence scores - - Temporal validity windows (`valid_from` / `valid_until`) - - Cross-system context capture on every decision - - - - Versioned policy storage in the knowledge graph - - Compliance checking against recorded decisions - - Policy exception tracking with approver audit trail - - - - Maps entity text to stable URIs - - Creates typed links between entity IDs - - Prevents "Apple", "Apple Inc.", "AAPL" becoming separate nodes - - - - Fuses vector similarity, graph traversal, and agent memory - - Richer context than pure vector search - - Configurable `hybrid_alpha` and expansion hops - - - - Traces upstream causes and downstream effects of any decision - - Explainability paths with relationship types - - Configurable depth and direction - - +- **AgentContext** — Memory, decision tracking, and graph-backed retrieval behind one API + - Conversation history and checkpoint diffing + - Persist and restore full context state to disk +- **ContextGraph** — Thread-safe in-memory knowledge graph + - PageRank, centrality, community detection, temporal validity + - Cross-graph navigation and link traversal +- **AgentMemory** — Embedding-backed memory with retention policy + - LRU eviction at configurable `max_memory_size` + - Per-conversation history isolation +- **DecisionRecorder** — Records decisions with causal chains and confidence scores + - Temporal validity windows (`valid_from` / `valid_until`) + - Cross-system context capture on every decision +- **PolicyEngine** — Versioned policy storage in the knowledge graph + - Compliance checking against recorded decisions + - Policy exception tracking with approver audit trail +- **EntityLinker** — Maps entity text to stable URIs + - Creates typed links between entity IDs + - Prevents "Apple", "Apple Inc.", "AAPL" becoming separate nodes +- **ContextRetriever** — Fuses vector similarity, graph traversal, and agent memory + - Richer context than pure vector search + - Configurable `hybrid_alpha` and expansion hops +- **CausalChainAnalyzer** — Traces upstream causes and downstream effects of any decision + - Explainability paths with relationship types + - Configurable depth and direction ## Quick Start @@ -889,26 +871,10 @@ class EntityLink: - - - Embedding storage backend for memory retrieval. - - - Graph algorithms and analytics used inside ContextGraph. - - - Logical inference layered on top of context. - - - W3C PROV-O lineage for every stored fact. - - +- [Vector Store](vector_store) — Embedding storage backend for memory retrieval. +- [Knowledge Graph](kg) — Graph algorithms and analytics used inside ContextGraph. +- [Reasoning](reasoning) — Logical inference layered on top of context. +- [Provenance](provenance) — W3C PROV-O lineage for every stored fact. - - - Memory and decision tracking · Intermediate - - - Production FAISS + Neo4j setup · Advanced - - +- [Context Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb) — Memory and decision tracking · Intermediate +- [Advanced Context Engineering](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb) — Production FAISS + Neo4j setup · Advanced diff --git a/docs/reference/core.md b/docs/reference/core.md index 86259399..f2cbd864 100644 --- a/docs/reference/core.md +++ b/docs/reference/core.md @@ -18,20 +18,10 @@ icon: "gear" ## What You Get - - - High-level orchestrator: coordinates the full KG construction pipeline from a single `config.yaml`. Entry point for application-level deployments. - - - YAML config with deep-merge, `SEMANTICA_` env var overrides, and dot-notation nested key access. Keeps secrets out of source files. - - - Ordered startup/shutdown hooks, health monitoring, and a 6-state machine. Essential for long-running services like FastAPI apps. - - - Register custom ingestors, parsers, exporters, or any component. Load them by name at runtime: no imports required. - - +- **Semantica** — High-level orchestrator: coordinates the full KG construction pipeline from a single `config.yaml`. Entry point for application-level deployments. +- **ConfigManager** — YAML config with deep-merge, `SEMANTICA_` env var overrides, and dot-notation nested key access. Keeps secrets out of source files. +- **LifecycleManager** — Ordered startup/shutdown hooks, health monitoring, and a 6-state machine. Essential for long-running services like FastAPI apps. +- **PluginRegistry** — Register custom ingestors, parsers, exporters, or any component. Load them by name at runtime: no imports required. ## Exported Classes @@ -236,17 +226,7 @@ result = build_knowledge_base(sources=["doc.pdf"], method="fast") Use `Semantica` and `LifecycleManager` only when building a long-running application (e.g. a FastAPI service) that needs ordered startup, health checks, and graceful shutdown. For scripts and notebooks, use individual modules directly. - - - Pipeline execution and step orchestration. - - - Shared utilities used by Core internally. - - - Learn the basics before using Core. - - - Configure LLM providers via ConfigManager. - - +- [Pipeline](pipeline) — Pipeline execution and step orchestration. +- [Utils](utils) — Shared utilities used by Core internally. +- [Getting Started](../getting-started) — Learn the basics before using Core. +- [LLMs](llms) — Configure LLM providers via ConfigManager. diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.md index 4fde0158..bb514abd 100644 --- a/docs/reference/deduplication.md +++ b/docs/reference/deduplication.md @@ -30,26 +30,12 @@ icon: "copy" ## What You Get - - - Pairwise, batch, incremental, and group detection modes. Returns scored candidates with reasons. - - - Five merge strategies: keep first, last, most complete, highest confidence, or merge all fields. - - - Multi-factor scoring across string edit distance, property overlap, relationship overlap, and embeddings. - - - Union-Find and hierarchical clustering for batch deduplication at scale: handles 100k+ entity sets. - - - Per-property merge rules with conflict resolution priorities. Apply different strategies to different fields. - - - `blocking_v2`, `hybrid_v2`, `semantic_v2`: up to 7× faster than v1 for large entity sets. - - +- **DuplicateDetector** — Pairwise, batch, incremental, and group detection modes. Returns scored candidates with reasons. +- **EntityMerger** — Five merge strategies: keep first, last, most complete, highest confidence, or merge all fields. +- **SimilarityCalculator** — Multi-factor scoring across string edit distance, property overlap, relationship overlap, and embeddings. +- **ClusterBuilder** — Union-Find and hierarchical clustering for batch deduplication at scale: handles 100k+ entity sets. +- **MergeStrategyManager** — Per-property merge rules with conflict resolution priorities. Apply different strategies to different fields. +- **v2 Strategies** — `blocking_v2`, `hybrid_v2`, `semantic_v2`: up to 7× faster than v1 for large entity sets. ## Getting Started @@ -451,17 +437,7 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name") - - - Detect value conflicts between non-duplicate entities. - - - GraphBuilder uses deduplication during construction. - - - Normalize entity names before deduplication. - - - Track merged entity lineage. - - +- [Conflicts](conflicts) — Detect value conflicts between non-duplicate entities. +- [Knowledge Graph](kg) — GraphBuilder uses deduplication during construction. +- [Normalize](normalize) — Normalize entity names before deduplication. +- [Provenance](provenance) — Track merged entity lineage. diff --git a/docs/reference/embeddings.md b/docs/reference/embeddings.md index e1719285..f62dc6e3 100644 --- a/docs/reference/embeddings.md +++ b/docs/reference/embeddings.md @@ -41,26 +41,12 @@ Semantica uses embeddings for: ## What You Get - - - Main entry point: provider-agnostic, handles batching automatically across all backends. - - - Text-specific with automatic batching and progress tracking. Default method is FastEmbed. - - - Node and edge embeddings for graph databases: Neo4j, NetworkX, FalkorDB. - - - Prepare, normalize, and format embeddings for FAISS, Weaviate, Qdrant, and Milvus. - - - `OpenAIStore`, `BGEStore`, `FastEmbedStore`, and `ProviderStoreFactory`. - - - Mean, Max, CLS, Attention, and Hierarchical: control token-to-vector aggregation. - - +- **EmbeddingGenerator** — Main entry point: provider-agnostic, handles batching automatically across all backends. +- **TextEmbedder** — Text-specific with automatic batching and progress tracking. Default method is FastEmbed. +- **GraphEmbeddingManager** — Node and edge embeddings for graph databases: Neo4j, NetworkX, FalkorDB. +- **VectorEmbeddingManager** — Prepare, normalize, and format embeddings for FAISS, Weaviate, Qdrant, and Milvus. +- **Provider Stores** — `OpenAIStore`, `BGEStore`, `FastEmbedStore`, and `ProviderStoreFactory`. +- **Pooling Strategies** — Mean, Max, CLS, Attention, and Hierarchical: control token-to-vector aggregation. ## Provider Setup @@ -633,17 +619,7 @@ providers = check_available_providers() # → {"sentence_transformers": True, "fastembed": True, "openai": False} ``` - - - Store and search the generated embeddings. - - - Chunk text before embedding for better retrieval quality. - - - Distance Intelligence uses graph embeddings for semantic neighbourhoods. - - - Semantic deduplication uses embedding distance for entity resolution. - - +- [Vector Store](vector_store) — Store and search the generated embeddings. +- [Split](split) — Chunk text before embedding for better retrieval quality. +- [KG Module](kg) — Distance Intelligence uses graph embeddings for semantic neighbourhoods. +- [Deduplication](deduplication) — Semantic deduplication uses embedding distance for entity resolution. diff --git a/docs/reference/evals.md b/docs/reference/evals.md index 0f6303a2..e39f06b3 100644 --- a/docs/reference/evals.md +++ b/docs/reference/evals.md @@ -58,17 +58,7 @@ print("Relation coverage: ", report["relation_completeness"]["relation_coverage" | `suggestions` | `List[str]` | Improvement suggestions | | `metrics` | `dict` | Detailed sub-metrics | - - - Extraction module. - - - Graph quality assessment. - - - Pipeline performance metrics. - - - Available now for ontology quality metrics. - - +- [Semantic Extract](semantic_extract) — Extraction module. +- [Knowledge Graph](kg) — Graph quality assessment. +- [Pipeline](pipeline) — Pipeline performance metrics. +- [Ontology Evaluator](ontology) — Available now for ontology quality metrics. diff --git a/docs/reference/explorer.md b/docs/reference/explorer.md index 3495eef4..2bccb30e 100644 --- a/docs/reference/explorer.md +++ b/docs/reference/explorer.md @@ -115,26 +115,12 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \ ## What You Get - - - Interactive node/edge search, path finding, and neighborhood expansion. Indexed search at 0.004ms on 118k-node graphs. - - - SKOS vocabulary management, SHACL shape generation and validation, ontology alignment, health dashboard, and versioning. - - - Degree centrality, community detection, connectivity analysis, graph validation, and distance matrices. - - - All features available as a REST API: fully documented at `/docs`. - - - Real-time graph mutation events streamed over WebSocket at `/ws/graph-updates`. - - - `semantica-explorer --graph my_graph.json` for instant local startup. - - +- **Graph Explorer** — Interactive node/edge search, path finding, and neighborhood expansion. Indexed search at 0.004ms on 118k-node graphs. +- **Ontology Hub** — SKOS vocabulary management, SHACL shape generation and validation, ontology alignment, health dashboard, and versioning. +- **Analytics** — Degree centrality, community detection, connectivity analysis, graph validation, and distance matrices. +- **REST API** — All features available as a REST API: fully documented at `/docs`. +- **WebSocket Updates** — Real-time graph mutation events streamed over WebSocket at `/ws/graph-updates`. +- **CLI Launcher** — `semantica-explorer --graph my_graph.json` for instant local startup. ## Features @@ -412,17 +398,7 @@ Semantic neighborhood requires node embeddings stored in node properties (keys ` **Session state lost after restart** Session state is in-memory only. Use `POST /api/export` to save a JSON snapshot before shutting down. - - - Build and save the ContextGraph that Explorer loads. - - - Programmatic ontology management and SHACL generation. - - - Programmatic graph rendering without the Explorer server. - - - Export to RDF, Parquet, and other formats without launching a server. - - +- [Context](context) — Build and save the ContextGraph that Explorer loads. +- [Ontology](ontology) — Programmatic ontology management and SHACL generation. +- [Visualization](visualization) — Programmatic graph rendering without the Explorer server. +- [Export](export) — Export to RDF, Parquet, and other formats without launching a server. diff --git a/docs/reference/export.md b/docs/reference/export.md index 39ec7d42..49df715e 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -381,17 +381,7 @@ The `export_csv` convenience function delegates to `CSVExporter.export()`. For p **Match your export format to your consumer.** Neo4j → `cypher`; ArangoDB → `aql`; Gephi/yEd → `graphml` or `gexf`; semantic web tools → `turtle` or `json-ld`; analytics pipelines → `parquet`; zero-copy IPC → `arrow`. - - - Store RDF exports in a SPARQL-queryable backend. - - - Export OWL ontologies. - - - Include provenance metadata in RDF exports. - - - Add export as a final pipeline step. - - +- [Triplet Store](triplet_store) — Store RDF exports in a SPARQL-queryable backend. +- [Ontology](ontology) — Export OWL ontologies. +- [Provenance](provenance) — Include provenance metadata in RDF exports. +- [Pipeline](pipeline) — Add export as a final pipeline step. diff --git a/docs/reference/graph_store.md b/docs/reference/graph_store.md index 709c25aa..e14da6f6 100644 --- a/docs/reference/graph_store.md +++ b/docs/reference/graph_store.md @@ -28,38 +28,24 @@ icon: "server" ## What You Get - - - - Unified API across Neo4j, FalkorDB, Apache AGE, Amazon Neptune - - Context manager support for automatic connection cleanup - - `create_nodes()` for bulk loading: faster than individual calls - - - - Parameterized Cypher construction prevents injection attacks - - Optional in-process result caching with `use_cache=True` - - `clear_cache()` on writes, toggle with `enable_cache()` / `disable_cache()` - - - - Degree centrality ordered by degree DESC - - Connected component assignment - - Shortest path between nodes, neighbor traversal up to N hops - - - - `create_nodes(list)`: one round-trip for many nodes - - `create_relationship()` with typed properties - - `delete_node(detach=True)` removes all connected relationships - - - - `create_index(label, property_name=)`: makes MATCH queries orders-of-magnitude faster - - `get_stats()`: node counts, edge counts, type breakdown - - Create indexes before bulk loading for best performance - - - - `shortest_path()` returns `length`, `nodes`, `relationships` - - `get_neighbors()` with direction and depth control - - Cross-backend path traversal via the unified API - - +- **GraphStore** — Unified API across Neo4j, FalkorDB, Apache AGE, Amazon Neptune + - Context manager support for automatic connection cleanup + - `create_nodes()` for bulk loading: faster than individual calls +- **QueryEngine** — Parameterized Cypher construction prevents injection attacks + - Optional in-process result caching with `use_cache=True` + - `clear_cache()` on writes, toggle with `enable_cache()` / `disable_cache()` +- **GraphAnalytics** — Degree centrality ordered by degree DESC + - Connected component assignment + - Shortest path between nodes, neighbor traversal up to N hops +- **Bulk Operations** — `create_nodes(list)`: one round-trip for many nodes + - `create_relationship()` with typed properties + - `delete_node(detach=True)` removes all connected relationships +- **Schema Management** — `create_index(label, property_name=)`: makes MATCH queries orders-of-magnitude faster + - `get_stats()`: node counts, edge counts, type breakdown + - Create indexes before bulk loading for best performance +- **Path Traversal** — `shortest_path()` returns `length`, `nodes`, `relationships` + - `get_neighbors()` with direction and depth control + - Cross-backend path traversal via the unified API ## Getting Started @@ -517,17 +503,7 @@ stats = store.get_stats() - - - Build the graph before persisting it. - - - RDF triple store for semantic web and SPARQL queries. - - - Visualize graphs stored in any backend. - - - AgentContext uses GraphStore for memory retrieval. - - +- [KG Module](kg) — Build the graph before persisting it. +- [Triplet Store](triplet_store) — RDF triple store for semantic web and SPARQL queries. +- [Visualization](visualization) — Visualize graphs stored in any backend. +- [Context](context) — AgentContext uses GraphStore for memory retrieval. diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index 072a318f..fdc3d092 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -625,17 +625,7 @@ from semantica.ingest import ingest_file result = ingest_file("source_path", method="my_format") ``` - - - Parse raw sources into structured text and tables. - - - Orchestrate ingest as the first pipeline step. - - - Snowflake-specific setup and authentication guide. - - - Track lineage from ingest through to inference. - - +- [Parse](parse) — Parse raw sources into structured text and tables. +- [Pipeline](pipeline) — Orchestrate ingest as the first pipeline step. +- [Snowflake Integration](../integrations/snowflake) — Snowflake-specific setup and authentication guide. +- [Provenance](provenance) — Track lineage from ingest through to inference. diff --git a/docs/reference/kg.md b/docs/reference/kg.md index c1532dfd..2fb7e66c 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -316,20 +316,10 @@ kg: default_validity: infinite ``` - - - Persist graphs in Neo4j, FalkorDB, or Apache AGE. - - - Source of entities and relationships fed to GraphBuilder. - - - Visualize knowledge graphs interactively. - - - Conflict detection and resolution. - - +- [Graph Store](graph_store) — Persist graphs in Neo4j, FalkorDB, or Apache AGE. +- [Semantic Extract](semantic_extract) — Source of entities and relationships fed to GraphBuilder. +- [Visualization](visualization) — Visualize knowledge graphs interactively. +- [Conflicts](conflicts) — Conflict detection and resolution. ### Cookbooks diff --git a/docs/reference/llms.md b/docs/reference/llms.md index 0101d43e..4a0e5f2b 100644 --- a/docs/reference/llms.md +++ b/docs/reference/llms.md @@ -439,17 +439,7 @@ extractor = NERExtractor( ) ``` - - - Use LLMs for NER and relation extraction. - - - LLM providers in Agno multi-agent teams. - - - LLM-backed deductive and abductive reasoning. - - - GraphRAG uses LLMs for reasoning over knowledge graphs. - - +- [Semantic Extract](semantic_extract) — Use LLMs for NER and relation extraction. +- [Agno Integration](../integrations/agno) — LLM providers in Agno multi-agent teams. +- [Reasoning](reasoning) — LLM-backed deductive and abductive reasoning. +- [Context](context) — GraphRAG uses LLMs for reasoning over knowledge graphs. diff --git a/docs/reference/mcp_server.md b/docs/reference/mcp_server.md index 1f213347..972995b4 100644 --- a/docs/reference/mcp_server.md +++ b/docs/reference/mcp_server.md @@ -40,26 +40,12 @@ python -m semantica.mcp_server ## What You Get - - - Extract entities, extract relations, record decisions, query decisions, find precedents, trace causal chains, add entities, add relationships, run analytics, summarise graph, run reasoning, export graph. - - - Live graph JSON (`semantica://graph/summary`), decision list, and schema/version info: readable by any MCP client. - - - Runs over stdio: no server, no port, no Docker required. One config block to activate in any MCP client. - - - Point `SEMANTICA_KG_PATH` at a saved graph file to reload it automatically on every server startup. - - - Record decisions, find precedents via hybrid similarity search, and trace causal chains across agent runs. - - - The [Explorer](explorer) module offers a full HTTP API and browser dashboard if you prefer programmatic access. - - +- **12 MCP Tools** — Extract entities, extract relations, record decisions, query decisions, find precedents, trace causal chains, add entities, add relationships, run analytics, summarise graph, run reasoning, export graph. +- **3 Readable Resources** — Live graph JSON (`semantica://graph/summary`), decision list, and schema/version info: readable by any MCP client. +- **Zero Infrastructure** — Runs over stdio: no server, no port, no Docker required. One config block to activate in any MCP client. +- **Persistent Graphs** — Point `SEMANTICA_KG_PATH` at a saved graph file to reload it automatically on every server startup. +- **Decision Intelligence** — Record decisions, find precedents via hybrid similarity search, and trace causal chains across agent runs. +- **REST Alternative** — The [Explorer](explorer) module offers a full HTTP API and browser dashboard if you prefer programmatic access. ## Installation @@ -459,17 +445,7 @@ The MCP server exposes three readable resources: | `semantica://decisions/list` | All recorded decisions (up to 50) | | `semantica://schema/info` | Server version and available tools | - - - The ContextGraph that the MCP server operates on. - - - NER and relation extraction powering the MCP tools. - - - Forward-chaining engine behind run_reasoning. - - - Use Semantica inside Agno multi-agent teams. - - +- [Context](context) — The ContextGraph that the MCP server operates on. +- [Semantic Extract](semantic_extract) — NER and relation extraction powering the MCP tools. +- [Reasoning](reasoning) — Forward-chaining engine behind run_reasoning. +- [Agno Integration](../integrations/agno) — Use Semantica inside Agno multi-agent teams. diff --git a/docs/reference/normalize.md b/docs/reference/normalize.md index c8d39403..965875e4 100644 --- a/docs/reference/normalize.md +++ b/docs/reference/normalize.md @@ -584,17 +584,7 @@ normalized = normalize_text("Apple Inc.", method="expand_suffixes") # → "Apple Incorporated" ``` - - - Parse documents before normalization. - - - Chunk normalized text for embedding. - - - Resolve duplicate entities after normalization. - - - Include normalization as a named pipeline step. - - +- [Parse](parse) — Parse documents before normalization. +- [Split](split) — Chunk normalized text for embedding. +- [Deduplication](deduplication) — Resolve duplicate entities after normalization. +- [Pipeline](pipeline) — Include normalization as a named pipeline step. diff --git a/docs/reference/ontology.md b/docs/reference/ontology.md index a8763b6a..0aa87061 100644 --- a/docs/reference/ontology.md +++ b/docs/reference/ontology.md @@ -286,17 +286,7 @@ ontology_data = ingest_ontology("schema.jsonld") # JSON-LD Ontology versioning (`VersionManager`, `OntologyVersion`) has moved to `semantica.change_management`. Import from there: `from semantica.change_management import VersionManager`. - - - Apply inference rules over ontology axioms. - - - The graph being modeled by the ontology. - - - Export ontologies as RDF, OWL, or JSON-LD. - - - Detect ontology constraint violations. - - +- [Reasoning](reasoning) — Apply inference rules over ontology axioms. +- [Knowledge Graph](kg) — The graph being modeled by the ontology. +- [Export](export) — Export ontologies as RDF, OWL, or JSON-LD. +- [Conflicts](conflicts) — Detect ontology constraint violations. diff --git a/docs/reference/parse.md b/docs/reference/parse.md index 4c7fced6..55ecf3b8 100644 --- a/docs/reference/parse.md +++ b/docs/reference/parse.md @@ -297,17 +297,7 @@ for source in sources: Docling is an optional dependency. If `docling` is not installed, `DoclingParser` raises an `ImportError` with installation instructions: `pip install docling`. `DocumentParser` is always available and requires no extras. - - - Load files before parsing. - - - Chunk parsed text for embedding and extraction. - - - Full Docling integration setup guide. - - - Extract entities and relations from parsed text. - - +- [Ingest](ingest) — Load files before parsing. +- [Split](split) — Chunk parsed text for embedding and extraction. +- [Docling Integration](../integrations/docling) — Full Docling integration setup guide. +- [Semantic Extract](semantic_extract) — Extract entities and relations from parsed text. diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index 82d2c7e3..56218391 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -29,26 +29,12 @@ icon: "gear" You could wire Semantica modules together with plain Python code. Pipelines add: - - - A single bad document doesn't crash a 10,000-document run. - - - Run extraction across multiple workers with one parameter. - - - tqdm console bar or WebSocket streaming to Explorer. - - - Save the exact pipeline configuration to YAML and replay on any machine. - - - On re-runs, only process documents that changed since the last run. - - - Catch misconfigured steps and dependency cycles before they fail mid-run. - - +- **Retry and failure handling** — A single bad document doesn't crash a 10,000-document run. +- **Parallelism** — Run extraction across multiple workers with one parameter. +- **Progress tracking** — tqdm console bar or WebSocket streaming to Explorer. +- **Reproducibility** — Save the exact pipeline configuration to YAML and replay on any machine. +- **Delta mode** — On re-runs, only process documents that changed since the last run. +- **Validation** — Catch misconfigured steps and dependency cycles before they fail mid-run. Use plain module calls for quick scripts and notebooks. Use pipelines for anything you run repeatedly, at scale, or in production. @@ -322,48 +308,33 @@ manager = PipelineTemplateManager() The `create_pipeline_from_template(name)` method returns a configured `PipelineBuilder`. Call `.build(pipeline_name)` on it to produce a runnable `Pipeline`. - - - **Ingest → Parse → Normalize → Extract → Embed → Build KG** +- **document_processing** — **Ingest → Parse → Normalize → Extract → Embed → Build KG** — Complete document processing from ingestion to knowledge graph. - Complete document processing from ingestion to knowledge graph. + ```python + builder = manager.create_pipeline_from_template("document_processing") + pipeline = builder.build("doc_pipeline") + ``` - ```python - builder = manager.create_pipeline_from_template("document_processing") - pipeline = builder.build("doc_pipeline") - ``` - - - **Ingest → Chunk → Embed → Store Vectors** +- **rag_pipeline** — **Ingest → Chunk → Embed → Store Vectors** — RAG pipeline for question answering: builds a vector-indexed store. - RAG pipeline for question answering: builds a vector-indexed store. + ```python + builder = manager.create_pipeline_from_template("rag_pipeline") + pipeline = builder.build("rag_pipeline") + ``` - ```python - builder = manager.create_pipeline_from_template("rag_pipeline") - pipeline = builder.build("rag_pipeline") - ``` - - - **Ingest → Extract Entities → Extract Relations → Dedup → Resolve → Build Graph** +- **kg_construction** — **Ingest → Extract Entities → Extract Relations → Dedup → Resolve → Build Graph** — Knowledge graph construction from multiple sources. - Knowledge graph construction from multiple sources. + ```python + builder = manager.create_pipeline_from_template("kg_construction") + pipeline = builder.build("kg_pipeline") + ``` - ```python - builder = manager.create_pipeline_from_template("kg_construction") - pipeline = builder.build("kg_pipeline") - ``` - - - **Extract Concepts → Infer Classes → Infer Properties → Generate OWL → Validate** +- **ontology_generation** — **Extract Concepts → Infer Classes → Infer Properties → Generate OWL → Validate** — Ontology generation from extracted data. - Ontology generation from extracted data. - - ```python - builder = manager.create_pipeline_from_template("ontology_generation") - pipeline = builder.build("ontology_pipeline") - ``` - - + ```python + builder = manager.create_pipeline_from_template("ontology_generation") + pipeline = builder.build("ontology_pipeline") + ``` **Use templates from `PipelineTemplateManager` for common patterns.** `create_pipeline_from_template("kg_construction")` wires normalization, deduplication, conflict detection, and graph construction in the correct order: saving you from common mistakes like deduplicating before normalizing. @@ -583,17 +554,7 @@ StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy - - - First step in most pipelines. - - - Core extraction step. - - - Graph construction step. - - - Final output step. - - +- [Ingest](ingest) — First step in most pipelines. +- [Semantic Extract](semantic_extract) — Core extraction step. +- [Knowledge Graph](kg) — Graph construction step. +- [Export](export) — Final output step. diff --git a/docs/reference/provenance.md b/docs/reference/provenance.md index bc414e92..84300bfc 100644 --- a/docs/reference/provenance.md +++ b/docs/reference/provenance.md @@ -517,17 +517,7 @@ Provenance tracking in Semantica produces the following audit artifacts: `ProvenanceManager` does not include built-in Turtle or JSON-LD serialization. Use `entry.to_dict()` and `get_lineage()` to retrieve provenance data, then serialize with your preferred RDF library if W3C PROV-O RDF output is required. - - - Version control and snapshot audit trails. - - - Provenance begins at the ingestion stage. - - - Include provenance metadata in RDF exports. - - - Decision provenance via AgentContext. - - +- [Change Management](change_management) — Version control and snapshot audit trails. +- [Ingest](ingest) — Provenance begins at the ingestion stage. +- [Export](export) — Include provenance metadata in RDF exports. +- [Context](context) — Decision provenance via AgentContext. diff --git a/docs/reference/reasoning.md b/docs/reference/reasoning.md index 29f64fbc..41a05190 100644 --- a/docs/reference/reasoning.md +++ b/docs/reference/reasoning.md @@ -31,26 +31,12 @@ icon: "microchip" ## Which Engine Should I Use? - - - IF/THEN rules, forward and backward chaining. **Start here**: covers 90% of use cases. No query language required. - - - Natural language queries over a knowledge graph via LLM. No SPARQL or rules: just ask a question. - - - Recursive Horn clause rules with guaranteed termination. Use for complex multi-hop transitive rules. - - - Rete pattern matching for high-frequency inference. Use when you need to match many facts against many rules simultaneously. - - - SPARQL query expansion and rule-based inference. Use when you're working with RDF/OWL data. - - - All 13 Allen interval algebra relations. Use for time-aware reasoning: overlaps, before/after, during, contains. - - +- [Reasoner](#reasoner-forwardbackward-chaining) — IF/THEN rules, forward and backward chaining. **Start here**: covers 90% of use cases. No query language required. +- [GraphReasoner](#graphreasoner) — Natural language queries over a knowledge graph via LLM. No SPARQL or rules: just ask a question. +- [DatalogReasoner](#datalogreasoner) — Recursive Horn clause rules with guaranteed termination. Use for complex multi-hop transitive rules. +- [ReteEngine](#reteengine) — Rete pattern matching for high-frequency inference. Use when you need to match many facts against many rules simultaneously. +- [SPARQLReasoner](#sparqlreasoner) — SPARQL query expansion and rule-based inference. Use when you're working with RDF/OWL data. +- [TemporalReasoningEngine](#temporalreasoningengine) — All 13 Allen interval algebra relations. Use for time-aware reasoning: overlaps, before/after, during, contains. ## Getting Started @@ -479,17 +465,7 @@ step.confidence # float `GraphReasoner` requires a configured LLM provider. If the provider fails to initialize, `reason()` returns an error string instead of raising. Check `reasoner.provider is not None` before calling if you need to surface failures explicitly. - - - The knowledge graph being reasoned over. - - - Ontology axioms and SHACL constraints for logical reasoning. - - - RDF backend for SPARQL-based reasoning. - - - Reasoning integrated into agent decision intelligence. - - +- [Knowledge Graph](kg) — The knowledge graph being reasoned over. +- [Ontology](ontology) — Ontology axioms and SHACL constraints for logical reasoning. +- [Triplet Store](triplet_store) — RDF backend for SPARQL-based reasoning. +- [Context](context) — Reasoning integrated into agent decision intelligence. diff --git a/docs/reference/seed.md b/docs/reference/seed.md index 25b4026a..2e65b21e 100644 --- a/docs/reference/seed.md +++ b/docs/reference/seed.md @@ -23,26 +23,12 @@ icon: "database" ## What You Get - - - Register sources, build a foundation graph, validate quality, and merge with extracted data. - - - Typed source definition supporting CSV, JSON, SQL, and API with format-specific config. - - - Build a foundation graph from all registered sources in one pass, ready to merge with extracted data. - - - `seed_first`, `extracted_first`, and `merge` with property-level conflict detection. - - - Required field checks, ID uniqueness, type consistency, reference integrity, and encoding validation before loading. - - - Track seed data versions across pipeline runs and diff changes between versions. - - +- **SeedDataManager** — Register sources, build a foundation graph, validate quality, and merge with extracted data. +- **SeedDataSource** — Typed source definition supporting CSV, JSON, SQL, and API with format-specific config. +- **Foundation Graph** — Build a foundation graph from all registered sources in one pass, ready to merge with extracted data. +- **Merge Strategies** — `seed_first`, `extracted_first`, and `merge` with property-level conflict detection. +- **Validation** — Required field checks, ID uniqueness, type consistency, reference integrity, and encoding validation before loading. +- **Versioning** — Track seed data versions across pipeline runs and diff changes between versions. **When to use the Seed Module:** Bootstrapping with structured reference data (taxonomies, user lists, product catalogs), loading immutable facts (ISO country codes, standard ontology terms) that extracted data should not override, ensuring test reproducibility with deterministic datasets, and anchoring entity disambiguation with canonical forms. @@ -335,17 +321,7 @@ export SEMANTICA_SEED_MERGE_STRATEGY=seed_first **Use YAML configuration for production deployments.** Hard-coding source paths in Python scripts makes environment-switching (dev → staging → prod) fragile. Declare sources in `config.yaml` under the `seed:` key and override paths with `SEMANTICA_SEED_DATA_DIR`. This way, the same code runs in every environment. - - - Load unstructured data alongside seed data. - - - The target graph that seed data populates. - - - Handle duplicates during seed-extracted merge. - - - Incorporate seed loading as a named pipeline step. - - +- [Ingest](ingest) — Load unstructured data alongside seed data. +- [Knowledge Graph](kg) — The target graph that seed data populates. +- [Deduplication](deduplication) — Handle duplicates during seed-extracted merge. +- [Pipeline](pipeline) — Incorporate seed loading as a named pipeline step. diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index b81e093e..8f5a58df 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -410,17 +410,7 @@ triplets = trip.extract(text) | `ml` | Fast | Free | High | Limited | | `llm` | Medium | API cost | Highest | Yes (schema) | - - - Configure which LLM is used for extraction. - - - Build graphs from extracted entities and relationships. - - - Parse documents before extraction. - - - Resolve duplicate entities after extraction. - - +- [LLM Providers](llms) — Configure which LLM is used for extraction. +- [Knowledge Graph](kg) — Build graphs from extracted entities and relationships. +- [Parse Module](parse) — Parse documents before extraction. +- [Deduplication](deduplication) — Resolve duplicate entities after extraction. diff --git a/docs/reference/split.md b/docs/reference/split.md index d64471da..2a648450 100644 --- a/docs/reference/split.md +++ b/docs/reference/split.md @@ -51,23 +51,11 @@ Semantica's chunking methods are designed to avoid these failure modes. ## What You Get - - - Unified interface for 11 chunking strategies: swap methods without changing downstream code. - - - Embedding-based topic shift detection: splits only when the topic actually changes. - - - Entity spans never cross chunk boundaries: guaranteed by boundary adjustment. - - - Subject–predicate–object triplets kept within a single chunk for KG pipelines. - - - Output dataclass with text, character offsets, optional id, and method-specific metadata. - - +- **TextSplitter** — Unified interface for 11 chunking strategies: swap methods without changing downstream code. +- **Semantic Chunking** — Embedding-based topic shift detection: splits only when the topic actually changes. +- **Entity-Aware Chunking** — Entity spans never cross chunk boundaries: guaranteed by boundary adjustment. +- **Relation-Aware Chunking** — Subject–predicate–object triplets kept within a single chunk for KG pipelines. +- **Chunk Object** — Output dataclass with text, character offsets, optional id, and method-specific metadata. ## Quick Start @@ -385,17 +373,7 @@ for chunk in chunks: For the full pipeline orchestration API, see the [Pipeline reference](pipeline). - - - Parse documents before chunking: produces sections and metadata. - - - Embed chunks for vector search and semantic chunking. - - - Extract entities and relations from individual chunks. - - - Integrate splitting as a named pipeline step. - - +- [Parse](parse) — Parse documents before chunking: produces sections and metadata. +- [Embeddings](embeddings) — Embed chunks for vector search and semantic chunking. +- [Semantic Extract](semantic_extract) — Extract entities and relations from individual chunks. +- [Pipeline](pipeline) — Integrate splitting as a named pipeline step. diff --git a/docs/reference/triplet_store.md b/docs/reference/triplet_store.md index d9a368e1..cb74bda0 100644 --- a/docs/reference/triplet_store.md +++ b/docs/reference/triplet_store.md @@ -19,26 +19,12 @@ icon: "table" ## What You Get - - - Unified interface across Blazegraph, Apache Jena, and RDF4J: swap backends with one parameter. - - - Full SPARQL SELECT, ASK, CONSTRUCT, and UPDATE query support via `execute_query()`. - - - `add_triplets()` batches writes with configurable batch size, retry logic, and progress tracking. - - - Built-in helpers: `add_skos_concept()` and `get_skos_concepts()` for controlled vocabulary management. - - - Blazegraph and RDF4J support named graph scoping via `graph=` on `execute_query()`. - - - `compute_delta(old_graph_uri, new_graph_uri)` returns added and removed triples between two named graph snapshots. - - +- **TripletStore** — Unified interface across Blazegraph, Apache Jena, and RDF4J: swap backends with one parameter. +- **SPARQL** — Full SPARQL SELECT, ASK, CONSTRUCT, and UPDATE query support via `execute_query()`. +- **Bulk Loading** — `add_triplets()` batches writes with configurable batch size, retry logic, and progress tracking. +- **SKOS Vocabulary** — Built-in helpers: `add_skos_concept()` and `get_skos_concepts()` for controlled vocabulary management. +- **Named Graphs** — Blazegraph and RDF4J support named graph scoping via `graph=` on `execute_query()`. +- **Delta Computation** — `compute_delta(old_graph_uri, new_graph_uri)` returns added and removed triples between two named graph snapshots. ## Getting Started @@ -490,17 +476,7 @@ for row in result.bindings: print(row) ``` - - - Export knowledge graphs to RDF formats. - - - Load OWL ontologies and store as RDF triples. - - - SPARQL-based property chain inference. - - - Property graph alternative for Cypher queries. - - +- [Export](export) — Export knowledge graphs to RDF formats. +- [Ontology](ontology) — Load OWL ontologies and store as RDF triples. +- [Reasoning](reasoning) — SPARQL-based property chain inference. +- [Graph Store](graph_store) — Property graph alternative for Cypher queries. diff --git a/docs/reference/utils.md b/docs/reference/utils.md index 14ae2106..83f9cc97 100644 --- a/docs/reference/utils.md +++ b/docs/reference/utils.md @@ -36,26 +36,12 @@ Most users won't call utils directly: it's the **shared foundation** for all mod ## What You Get - - - Structured logging with `@log_execution_time` decorator and quality metrics via environment variables. - - - `validate_entity` and `validate_config` with a typed `ValidationError` carrying field and value context. - - - `track_progress` wraps any iterable: auto-detects console vs Jupyter for the right renderer. - - - `clean_text`, `hash_data`, `safe_filename`, and nested dict utilities used throughout the framework. - - - `SemanticaError` → `ValidationError`, `ProcessingError`: typed exceptions for targeted recovery. - - - `read_json_file` raises `FileNotFoundError` or `json.JSONDecodeError` on failure: no boilerplate try/except around JSON I/O. - - +- **Logging** — Structured logging with `@log_execution_time` decorator and quality metrics via environment variables. +- **Validation** — `validate_entity` and `validate_config` with a typed `ValidationError` carrying field and value context. +- **Progress Tracking** — `track_progress` wraps any iterable: auto-detects console vs Jupyter for the right renderer. +- **Helper Functions** — `clean_text`, `hash_data`, `safe_filename`, and nested dict utilities used throughout the framework. +- **Exception Hierarchy** — `SemanticaError` → `ValidationError`, `ProcessingError`: typed exceptions for targeted recovery. +- **File Utilities** — `read_json_file` raises `FileNotFoundError` or `json.JSONDecodeError` on failure: no boilerplate try/except around JSON I/O. ## Logging @@ -226,11 +212,5 @@ from semantica.utils import read_json_file config = read_json_file("config.json") ``` - - - Framework orchestration that uses Utils internally. - - - Uses ProgressTracker for per-step tracking. - - +- [Core](core) — Framework orchestration that uses Utils internally. +- [Pipeline](pipeline) — Uses ProgressTracker for per-step tracking. diff --git a/docs/reference/vector_store.md b/docs/reference/vector_store.md index 8b0e0e60..ea264909 100644 --- a/docs/reference/vector_store.md +++ b/docs/reference/vector_store.md @@ -32,38 +32,24 @@ icon: "database" ## What You Get - - - - Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector - - One-line backend swap: no application code changes - - `add_documents()` auto-embeds; `store_vectors()` for pre-computed embeddings - - - - Dense vector similarity with metadata filtering - - RRF or weighted-average fusion strategies - - Multi-source fusion across separate collections - - - - Rich metadata indexing by field values - - Update metadata fields without re-embedding - - OR and AND query operators - - - - Structural per-tenant namespace isolation - - Faster queries (smaller search space per tenant) - - Safer than metadata-filter-only separation - - - - Bulk add, delete, and metadata updates - - Parallel embedding with configurable `batch_size` and `workers` - - In-place vector updates without full re-indexing - - - - flat, ivf, hnsw, and pq index types - - Full configuration control via `FAISSStore.create_index()` - - `save()` / `load()` for disk persistence - - +- **VectorStore** — Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector + - One-line backend swap: no application code changes + - `add_documents()` auto-embeds; `store_vectors()` for pre-computed embeddings +- **HybridSearch** — Dense vector similarity with metadata filtering + - RRF or weighted-average fusion strategies + - Multi-source fusion across separate collections +- **MetadataStore** — Rich metadata indexing by field values + - Update metadata fields without re-embedding + - OR and AND query operators +- **NamespaceManager** — Structural per-tenant namespace isolation + - Faster queries (smaller search space per tenant) + - Safer than metadata-filter-only separation +- **Batch Operations** — Bulk add, delete, and metadata updates + - Parallel embedding with configurable `batch_size` and `workers` + - In-place vector updates without full re-indexing +- **FAISS Index Types** — flat, ivf, hnsw, and pq index types + - Full configuration control via `FAISSStore.create_index()` + - `save()` / `load()` for disk persistence ## Getting Started @@ -602,17 +588,7 @@ store.create_index(index_type="pq", metric="L2", m=8) - - - Generate the vectors stored here. - - - AgentContext uses VectorStore for memory retrieval. - - - Chunk documents before embedding and storing. - - - Ingest documents before embedding and storing. - - +- [Embeddings](embeddings) — Generate the vectors stored here. +- [Context](context) — AgentContext uses VectorStore for memory retrieval. +- [Split](split) — Chunk documents before embedding and storing. +- [Ingest](ingest) — Ingest documents before embedding and storing. diff --git a/docs/reference/visualization.md b/docs/reference/visualization.md index 5cc23816..a5d2f2c8 100644 --- a/docs/reference/visualization.md +++ b/docs/reference/visualization.md @@ -290,17 +290,7 @@ semantica-explorer --graph my_graph.json See the [Explorer reference](explorer) for the full feature set and REST API. - - - The graph being visualized. - - - Visualize ontology class structure. - - - Generate the embeddings visualized here. - - - Full interactive Knowledge Explorer UI. - - +- [Knowledge Graph](kg) — The graph being visualized. +- [Ontology](ontology) — Visualize ontology class structure. +- [Embeddings](embeddings) — Generate the embeddings visualized here. +- [Explorer](explorer) — Full interactive Knowledge Explorer UI.