From 1f3cea5f0a4d1006df8c50a389e85550ee3ce03a Mon Sep 17 00:00:00 2001 From: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Date: Wed, 17 Jun 2026 12:58:44 +0530 Subject: [PATCH] docs: upgrade all docs pages with Mintlify premium components (#642) Replace plain markdown lists, tables, and numbered steps with interactive Mintlify v3 MDX components across all 50+ documentation files: - Tabs: provider/parser/method selection guides, citation formats, component details - Steps: setup flows, pipeline stages, connection initialization - CardGroup/Card: feature overviews, "what you get" sections, navigation footers - AccordionGroup: FAQ entries - Check/Warning/Tip/Note/Info: callouts replacing plain bold text and inline notes Files improved span the full docs surface: reference modules (context, llms, kg, reasoning, embeddings, deduplication, provenance, parse, ontology, core, semantic_extract), integrations (agno, docling, snowflake), graph/vector store backends (apache_age, pgvector), and top-level guides (contributing, governance, glossary, citation, community-projects, learning-more). --- docs/architecture.md | 18 +- docs/citation.md | 66 ++--- docs/cli-setup.md | 13 +- docs/community-projects.md | 78 +++--- docs/community.md | 5 + docs/concepts.md | 93 ++++++- docs/contributing-guide.md | 53 ++-- docs/cookbook.md | 5 - docs/docs.json | 12 +- docs/explorer-setup.md | 17 +- docs/faq.md | 26 +- docs/getting-started.md | 278 ++++++++++++++------- docs/glossary.md | 30 ++- docs/governance.md | 56 +++-- docs/graph_stores/apache_age.md | 110 ++++----- docs/index.md | 371 +++++++++++++++++++--------- docs/installation.md | 8 +- docs/integrations/agno.md | 354 ++++++++++++-------------- docs/integrations/docling.md | 11 +- docs/integrations/snowflake.md | 8 - docs/learning-more.md | 90 +++++-- docs/modules.md | 175 ++++++++++++- docs/project-license.md | 18 +- docs/quickstart.md | 6 + docs/reference/change_management.md | 15 +- docs/reference/conflicts.md | 23 +- docs/reference/context.md | 212 ++++++++++++++-- docs/reference/core.md | 40 ++- docs/reference/deduplication.md | 49 +++- docs/reference/embeddings.md | 131 ++++++++-- docs/reference/evals.md | 6 +- docs/reference/explorer.md | 39 +-- docs/reference/export.md | 13 +- docs/reference/graph_store.md | 77 ++++-- docs/reference/ingest.md | 17 +- docs/reference/kg.md | 208 ++++++++++------ docs/reference/llms.md | 152 ++++++++++-- docs/reference/mcp_server.md | 15 +- docs/reference/normalize.md | 19 +- docs/reference/ontology.md | 46 ++-- docs/reference/parse.md | 132 +++++----- docs/reference/pipeline.md | 15 +- docs/reference/provenance.md | 91 ++++--- docs/reference/reasoning.md | 62 ++++- docs/reference/seed.md | 13 +- docs/reference/semantic_extract.md | 146 +++++++++-- docs/reference/split.md | 21 +- docs/reference/triplet_store.md | 28 +-- docs/reference/utils.md | 20 +- docs/reference/vector_store.md | 46 +++- docs/reference/visualization.md | 18 +- docs/use-cases.md | 204 +++++++++------ docs/vector_stores/pgvector.md | 117 +++++---- 53 files changed, 2660 insertions(+), 1216 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 1ecb88b8..2dfefaab 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,6 +6,7 @@ icon: "building" Semantica is built around a four-layer modular architecture. Import only what you need — the framework never forces a full stack. Every component is independently swappable, and every layer communicates through clean interfaces with no hidden coupling. + ## Four-Layer Architecture Semantica four-layer architecture @@ -17,7 +18,7 @@ Semantica is built around a four-layer modular architecture. Import only what yo Loads data from any source into the pipeline as a unified `SourceDocument`. | Source | Module | Notes | -| ------ | ------ | ----- | +| :------ | :------ | :----- | | PDF, DOCX, PPTX, HTML, JSON, CSV | `ingest.FileIngestor` | Supports archives, recursive directory scan | | Parquet | `ingest.ParquetIngestor` | PyArrow, Hive-style partitions (v0.5.0) | | XML | `ingest.XMLIngestor` | XXE-safe lxml, XSD/DTD validation (v0.5.0) | @@ -35,7 +36,7 @@ Loads data from any source into the pipeline as a unified `SourceDocument`. Transforms raw text into structured, enriched documents ready for knowledge store ingestion. | Step | Module | What it does | -| ---- | ------ | ------------ | +| :---- | :------ | :------------ | | Parse | `parse.DocumentParser` / `parse.DoclingParser` | Text + layout extraction, table detection | | Normalize | `normalize` | Canonical forms, date/name standardization, encoding fix | | Extract | `semantic_extract` | NER, relation extraction, event detection, triplets | @@ -49,7 +50,7 @@ Transforms raw text into structured, enriched documents ready for knowledge stor Persistent knowledge stores and embedding infrastructure that power retrieval and reasoning. | Component | Module | Description | -| --------- | ------ | ----------- | +| :--------- | :------ | :----------- | | Knowledge Graph | `kg` | Graph construction, temporal models, analytics, Distance Intelligence | | Vector Store | `vector_store` | pgvector, Qdrant, Weaviate, Pinecone — semantic similarity search | | Ontology | `ontology` | OWL/RDFS modeling, SHACL validation, ontology alignment | @@ -64,7 +65,7 @@ Persistent knowledge stores and embedding infrastructure that power retrieval an Consumes the knowledge graph and vector stores for downstream use cases. | Use Case | Module | Description | -| -------- | ------ | ----------- | +| :-------- | :------ | :----------- | | GraphRAG | `context.AgentContext` | Graph-grounded retrieval for LLMs | | Agent memory | `context.ContextGraph` | Persistent semantic memory across agent runs | | Decision tracking | `context.AgentContext` | Record, trace, and audit every agent decision | @@ -78,22 +79,25 @@ Consumes the knowledge graph and vector stores for downstream use cases. + ## Data Flow Every pipeline follows the same linear path from raw source to delivered output: Semantica 8-step pipeline: Ingest → Parse → Normalize → Extract → Build KG → QA → Store → Deliver + ## Module Map | Layer | Category | Modules | -| ----- | -------- | ------- | +| :----- | :-------- | :------- | | **Layer 1 — Ingestion** | Sources | `ingest`, `split` | | **Layer 2 — Processing** | Transform | `parse`, `normalize`, `semantic_extract`, `deduplication`, `conflicts` | | **Layer 3 — Intelligence** | Stores | `kg`, `vector_store`, `graph_store`, `triplet_store`, `embeddings`, `ontology` | | **Layer 4 — Application** | Delivery | `context`, `reasoning`, `export`, `visualization`, `explorer`, `pipeline` | | — | Cross-cutting | `provenance`, `change_management`, `llms`, `mcp_server`, `seed`, `evals`, `core`, `utils` | + ## Extension Points Every layer exposes a registry-based extension point. Register custom implementations and they participate in the full pipeline with zero changes to core code. @@ -138,6 +142,7 @@ registry.register_plugin("my_plugin", MyPlugin, version="1.0.0") + ## Design Decisions @@ -168,10 +173,11 @@ Centralized `ConfigManager` with environment variable overrides. No magic defaul + ## Performance Characteristics | Characteristic | Mechanism | -| -------------- | --------- | +| :-------------- | :--------- | | **Parallel execution** | `Pipeline(workers=N)` with configurable workers per stage | | **Delta processing** | Incremental graph updates — no full recompute on new data | | **Streaming ingestion** | Process large corpora without loading everything into memory | diff --git a/docs/citation.md b/docs/citation.md index fd4a6728..74bfe50d 100644 --- a/docs/citation.md +++ b/docs/citation.md @@ -6,58 +6,46 @@ icon: "quote-left" > Use Semantica in your research? Here's how to cite it. ---- -## BibTeX +## Citation Formats -```bibtex -@software{semantica2026, - title = {Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering}, - author = {Hawksight AI}, - year = {2026}, - url = {https://github.com/semantica-agi/semantica}, - version = {0.5.0}, - doi = {10.5281/zenodo.XXXXXXX} -} -``` + + + ```bibtex + @software{semantica2026, + title = {Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering}, + author = {Hawksight AI}, + year = {2026}, + url = {https://github.com/semantica-agi/semantica}, + version = {0.5.0}, + doi = {10.5281/zenodo.XXXXXXX} + } + ``` + + + Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.5.0) \[Computer software\]. https://github.com/semantica-agi/semantica + + + Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.5.0, GitHub, 2026, https://github.com/semantica-agi/semantica. + + + Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.5.0. GitHub, 2026. https://github.com/semantica-agi/semantica. + + + Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.5.0, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica + + ---- - -## APA - -Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.5.0) \[Computer software\]. https://github.com/semantica-agi/semantica - ---- - -## MLA - -Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.5.0, GitHub, 2026, https://github.com/semantica-agi/semantica. - ---- - -## Chicago - -Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.5.0. GitHub, 2026. https://github.com/semantica-agi/semantica. - ---- - -## IEEE - -Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.5.0, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica - ---- ## Acknowledgment Text > "This work uses Semantica (Hawksight AI, 2026), an open-source framework for semantic layer construction and knowledge engineering." ---- ## Share Your Research Published research using Semantica? [Let us know](https://github.com/semantica-agi/semantica/issues) — we may feature your work. ---- ## See Also diff --git a/docs/cli-setup.md b/docs/cli-setup.md index b139b1d7..abb9aea9 100644 --- a/docs/cli-setup.md +++ b/docs/cli-setup.md @@ -6,6 +6,7 @@ icon: "terminal" Installing the base package registers five executables on your `PATH`. Each serves a distinct purpose. This page explains what they are, how to verify they are available, and which one to reach for in each situation. + ## Installed Commands ```bash @@ -15,7 +16,7 @@ pip install semantica After installation the following commands are available: | Command | Entry point | What it does | -| ------- | ----------- | ------------ | +| :------- | :----------- | :------------ | | `semantica` | `semantica.cli:main` | General-purpose CLI for pipeline runs, extraction, and graph operations | | `semantica-server` | `semantica.server:main` | FastAPI/uvicorn REST API server bound to `0.0.0.0:8000` | | `semantica-worker` | `semantica.worker:main` | Background worker process entry point for Semantica deployments | @@ -26,6 +27,7 @@ After installation the following commands are available: `semantica-explorer` requires `pip install semantica[explorer]`. Running it without that extra will immediately print an error and exit. See [Explorer Setup](explorer-setup) for the full walkthrough. + ## Verify the Installation Confirm each command is reachable and prints its usage: @@ -44,6 +46,7 @@ Confirm the package version: python -c "import semantica; print(semantica.__version__)" ``` + ## When to Use Each Command @@ -64,6 +67,7 @@ python -c "import semantica; print(semantica.__version__)" + ## Usage Examples @@ -144,23 +148,25 @@ python -c "import semantica; print(semantica.__version__)" + ## Environment Variables `semantica-mcp` reads two environment variables: | Variable | Default | Description | -| -------- | ------- | ----------- | +| :-------- | :------- | :----------- | | `SEMANTICA_KG_PATH` | *(none)* | Path to a saved graph file to load on startup | | `SEMANTICA_LOG_LEVEL` | `WARNING` | Log verbosity: `DEBUG`, `INFO`, `WARNING` | `semantica-server` reads one: | Variable | Default | Description | -| -------- | ------- | ----------- | +| :-------- | :------- | :----------- | | `SEMANTICA_CORS_ORIGINS` | `http://localhost:5173,http://127.0.0.1:5173` | Comma-separated list of allowed CORS origins | No other environment variables are read by these commands. + ## Troubleshooting ### `command not found` @@ -217,6 +223,7 @@ A response of `{"jsonrpc":"2.0","id":1,"result":{}}` confirms the server is work Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe). This is a Windows system dependency required by PyTorch and related packages, not a Semantica bug. + ## Next Steps diff --git a/docs/community-projects.md b/docs/community-projects.md index afd99467..faf7375f 100644 --- a/docs/community-projects.md +++ b/docs/community-projects.md @@ -10,6 +10,7 @@ icon: "people-group" Semantica is used across academia, enterprise, and independent research. Below is a snapshot of the ecosystem being built by the community. + ## Projects Using Semantica ### Research & Academia @@ -38,42 +39,51 @@ Production deployments span regulated and high-stakes industries where AI accoun - **Domain-specific extractors** — NER and relation extractors for clinical, legal, and scientific text - **Temporal graph dashboards** — visual timelines built with Semantica's `TemporalKnowledgeGraph` + custom visualization adapters + ## Supported Integrations -### Vector Databases + + + | Store | Notes | + | :---- | :---- | + | **FAISS** | In-process, CPU/GPU | + | **Pinecone** | Managed vector cloud | + | **Weaviate** | Schema-first hybrid search | + | **Qdrant** | High-performance Rust-native | + | **Milvus** | Enterprise-scale distributed | + | **PgVector** | Postgres-native for SQL stacks | + + + | Store | Notes | + | :---- | :---- | + | **Neo4j** | Industry standard, Cypher query | + | **FalkorDB** | Redis-protocol, low-latency | + | **Apache AGE** | PostgreSQL extension, OpenCypher | + | **Amazon Neptune** | Managed AWS, SPARQL + Gremlin | + + + | Provider | Notes | + | :-------- | :---- | + | **OpenAI** | GPT-4o, GPT-4, GPT-3.5 | + | **Anthropic** | Claude Opus, Sonnet, Haiku | + | **Google Gemini** | — | + | **Groq** | LLaMA, Mixtral — fast inference | + | **Ollama** | Fully local, air-gapped | + | **HuggingFace** | — | + | **DeepSeek** | — | + | **Novita AI** | — | + | **LiteLLM** | 100+ model gateway | + + + | Library | Notes | + | :------- | :---- | + | **spaCy** | Production NER and dependency parsing | + | **NLTK** | Tokenization and feature extraction | + | **Sentence Transformers** | Semantic embeddings | + | **FastEmbed** | Lightweight, fast inference | + + -- FAISS — in-process, CPU/GPU -- Pinecone — managed vector cloud -- Weaviate — schema-first hybrid search -- Qdrant — high-performance Rust-native -- Milvus — enterprise-scale distributed -- PgVector — Postgres-native for SQL stacks - -### Graph Databases - -- Neo4j — industry standard, Cypher query -- FalkorDB — Redis-protocol, low-latency -- Apache AGE — PostgreSQL extension, OpenCypher -- Amazon Neptune — managed AWS, SPARQL + Gremlin - -### LLM Providers - -- OpenAI (GPT-4o, GPT-4, GPT-3.5) -- Anthropic (Claude Opus, Sonnet, Haiku) -- Google Gemini -- Groq (LLaMA, Mixtral — fast inference) -- Ollama (fully local, air-gapped) -- HuggingFace -- DeepSeek -- Novita AI -- LiteLLM (100+ model gateway) - -### NLP Libraries - -- spaCy — production NER and dependency parsing -- NLTK — tokenization and feature extraction -- Sentence Transformers — semantic embeddings -- FastEmbed — lightweight, fast inference ## Community Extensions @@ -85,6 +95,7 @@ The plugin system (`PluginRegistry`) makes it easy to add new capabilities witho - **Visualization plugins** — enhanced dashboards with Plotly, D3.js, and custom graph renderers - **Evaluation harnesses** — domain-specific precision/recall benchmarks using `semantica.evals` + ## Build Your Own Extension Any Semantica component can be extended via the registry pattern: @@ -100,6 +111,7 @@ method_registry.register("file", "my_format", my_ingestor) See [Architecture](architecture#extension-points) for the full extension guide. + ## How to Contribute diff --git a/docs/community.md b/docs/community.md index e49caa48..5f8512b3 100644 --- a/docs/community.md +++ b/docs/community.md @@ -6,6 +6,7 @@ icon: "users" Semantica is built in the open, with contributions from researchers, engineers, and practitioners across many domains. Whether you're filing a bug, sharing a project, or reviewing a PR — you're part of the ecosystem. + ## Get Help @@ -23,6 +24,7 @@ Semantica is built in the open, with contributions from researchers, engineers, + ## Community Guidelines We follow the [Contributor Covenant Code of Conduct](https://github.com/semantica-agi/semantica/blob/main/CODE_OF_CONDUCT.md). The community is built on four principles: @@ -34,6 +36,7 @@ We follow the [Contributor Covenant Code of Conduct](https://github.com/semantic To report unacceptable behavior, open a GitHub issue with the `[CoC]` prefix. Every report is investigated. + ## Ways to Contribute There's no single right way to contribute. Pick the path that fits your skills and time: @@ -64,6 +67,7 @@ There's no single right way to contribute. Pick the path that fits your skills a See the [Contributing Guide](contributing-guide) for the full development workflow. + ## Stay Connected - **[GitHub](https://github.com/semantica-agi/semantica)** — source code, releases, and the public roadmap @@ -71,6 +75,7 @@ See the [Contributing Guide](contributing-guide) for the full development workfl - **[Discord](https://discord.gg/sV34vps5hH)** — real-time community chat - **[X / Twitter](https://x.com/BuildSemantica)** — announcements and release highlights + ## See Also diff --git a/docs/concepts.md b/docs/concepts.md index b21a0688..98c048fc 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -24,7 +24,6 @@ At its core, Semantica adds a **context and accountability layer** on top of you ---- ## Knowledge Graphs @@ -38,6 +37,7 @@ The foundation of everything in Semantica. A knowledge graph stores information This structure makes knowledge **searchable**, **connectable**, **queryable**, and — critically — **explainable**: every answer can be traced back to the facts and relationships that produced it. + ## Entity Extraction (NER) Scanning text to find and classify real-world entities: @@ -57,7 +57,7 @@ Scanning text to find and classify real-world entities: Each entity gets a type, confidence score, and a link to its source document. Three extraction methods are available: | Method | Speed | Accuracy | Requirements | -| ------ | ----- | -------- | ------------ | +| :------ | :----- | :-------- | :------------ | | `"pattern"` | ⚡ Very fast | Moderate | No API key — regex-based | | `"ml"` | Fast | High | Local ML model | | `"llm"` | Medium | Highest | LLM provider — all 9 supported | @@ -77,6 +77,81 @@ Finding how entities connect to each other: Relationships can be extracted via rule-based methods, ML models, or LLMs — each producing typed triplets with confidence scores and source attribution. + +## Knowledge Graph vs. Vector Store + +Both store information for AI retrieval — but they're built for different jobs. + + + + Stores **structured facts** as typed nodes and labeled edges. Answers questions that require understanding relationships between entities. + + | Strength | Why it matters | + | :-------- | :------------- | + | **Traversal** | Multi-hop queries: "Who founded companies that Apple alumni later joined?" | + | **Explainability** | Every answer traces back to specific nodes and edges — no black-box retrieval | + | **Temporal reasoning** | Point-in-time queries, `valid_from`/`valid_until` windows, historical snapshots | + | **Conflict detection** | Two sources disagreeing on the same fact is surfaced and resolvable | + | **Schema enforcement** | SHACL validation catches constraint violations before they corrupt results | + + **Use when:** you need structured reasoning, provenance, compliance, or explainability. + + ```python + from semantica.kg import GraphBuilder, PathFinder + + graph = GraphBuilder(merge_entities=True).build(entities=entities, relationships=rels) + finder = PathFinder() + path = finder.dijkstra_shortest_path(graph, "Steve Jobs", "Tim Cook") + ``` + + + + Stores **dense embeddings** of text chunks. Answers questions by finding semantically similar passages — useful when the structure of the answer isn't known in advance. + + | Strength | Why it matters | + | :-------- | :------------- | + | **Fuzzy similarity** | Finds relevant content even when exact words don't match | + | **Speed** | Sub-millisecond approximate nearest-neighbor search at scale | + | **Unstructured text** | Works directly on paragraphs, sentences, and raw documents | + | **Simplicity** | No schema design required — embed and index | + + **Use when:** you need fast semantic search over large text corpora. + + ```python + from semantica.vector_store import VectorStore + + store = VectorStore(backend="faiss", dimension=768) + store.add_documents(["Apple was founded in 1976.", "Google was founded in 1998."]) + results = store.search("tech company founding dates", limit=5) + ``` + + + + Semantica combines both — vector search seeds the graph traversal, and the graph provides structure and provenance the vector store cannot. + + | Step | What happens | + | :---- | :----------- | + | **Query embedding** | User query is embedded and used to find anchor nodes via vector similarity | + | **Graph traversal** | Multi-hop traversal from anchor nodes retrieves related entities and relationships | + | **Context assembly** | Facts + relationships are assembled with source attribution for each claim | + | **LLM generation** | LLM generates an answer grounded in the retrieved structured context | + + **Result:** every claim in the response links back to a specific graph node — no hallucination from training data, full audit trail. + + ```python + from semantica.context import AgentContext, ContextGraph + from semantica.vector_store import VectorStore + + context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph(advanced_analytics=True), + ) + result = context.query("Who founded Apple?", mode="graphrag") + ``` + + + + ## Embeddings Embeddings convert text into numerical vectors so AI systems can measure semantic similarity — finding related concepts even when the exact words differ. @@ -91,6 +166,7 @@ Semantica uses embeddings for: **Supported models:** Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama local embeddings. + ## GraphRAG GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses by grounding them in a structured knowledge graph rather than raw text chunks alone. @@ -116,6 +192,7 @@ GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses **GraphRAG eliminates the hallucination and traceability problems of standard RAG.** Standard RAG retrieves text chunks; GraphRAG retrieves structured facts with typed relationships. The LLM cannot confabulate structure that was never in the graph. + ## Ontology An ontology defines the schema and rules for your knowledge — what entity types exist, which relationships are valid, and what constraints apply. @@ -133,6 +210,7 @@ ontology = { Semantica can auto-generate ontologies from your knowledge graph or import existing OWL/RDF/Turtle ontologies. The **Ontology Hub** (v0.5.0) adds a visual editor, SHACL Studio, alignment authoring, and a live health dashboard. See the [Ontology reference](reference/ontology) for the full 6-stage generation pipeline. + ## Reasoning & Inference Semantica includes multiple reasoning engines to derive new knowledge from existing facts. @@ -200,7 +278,7 @@ Inferred: Steve Jobs has a connection to Cupertino | Engine | Description | Best For | - | ------ | ----------- | -------- | + | :------ | :----------- | :-------- | | Forward chaining | Applies rules until fixpoint | Alert systems, compliance checks | | Rete network | Efficient pattern matching | Large rule sets, high fact throughput | | Deductive | Classical syllogistic reasoning | Mathematical and logical inference | @@ -213,6 +291,7 @@ Inferred: Steve Jobs has a connection to Cupertino All engines produce **explainable inference paths** — not black-box conclusions. Every derived fact includes the rules and premises that produced it. + ## Temporal Intelligence Knowledge changes over time. Temporal graphs attach `valid_from` / `valid_until` windows to nodes and edges, enabling point-in-time queries and historical analysis. @@ -231,6 +310,7 @@ snapshot = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15 **Common uses:** tracking company leadership changes, policy evolution, research timelines, financial instrument histories, regulatory compliance windows. + ## Distance Intelligence Explore the semantic neighborhood of any entity in your graph — useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology. @@ -246,6 +326,7 @@ scores = calc.calculate_similarity(entity_a, entity_b) The [Visualization module](reference/visualization) renders distance matrices as interactive heatmaps and ego-mode neighborhood graphs. The [Explorer](reference/explorer) embeds distance intelligence directly in the browser dashboard. + ## Deduplication & Entity Resolution Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance. @@ -254,7 +335,7 @@ Real-world data contains the same entity under many names — "Apple", "Apple In | Strategy | Algorithm | Best For | - | -------- | --------- | -------- | + | :-------- | :--------- | :-------- | | `v1` | Jaro-Winkler string similarity | Small datasets, fast baseline | | `blocking_v2` | Candidate blocking + similarity | Large corpora — reduces O(n²) comparisons | | `hybrid_v2` | Blocking + semantic embedding match | Mixed structured/unstructured entity names | @@ -274,6 +355,7 @@ Real-world data contains the same entity under many names — "Apple", "Apple In + ## Provenance & Auditability Every fact in Semantica links back to: @@ -299,6 +381,7 @@ print(f"Extracted: {lineage.timestamp}") print(f"Checksum: {lineage.checksum}") ``` + ## Decision Intelligence Every agent decision is a first-class object in Semantica — recorded, causally linked, and searchable by precedent. This is the **accountability layer** for AI pipelines: decisions are no longer ephemeral log messages, they are queryable knowledge graph nodes. @@ -323,6 +406,7 @@ influence = context.analyze_decision_influence(decision_id) **Use `find_precedents()` before every high-stakes decision.** Hybrid similarity search over all recorded decisions surfaces past reasoning that may apply — reducing inconsistency across agent runs and enabling genuine organisational learning from AI decision history. + ## Conflict Detection When multiple sources disagree on the same fact, Semantica flags and resolves the conflict rather than silently picking one value. @@ -336,6 +420,7 @@ When multiple sources disagree on the same fact, Semantica flags and resolves th See the [Conflicts reference](reference/conflicts) for `ConflictResolver`, `SourceTracker`, and `InvestigationGuideGenerator`. + ## Custom Plugin Development Semantica is designed for extension. Any component — ingestor, extractor, graph builder, reasoning engine — can be replaced or augmented with a custom implementation registered at runtime. diff --git a/docs/contributing-guide.md b/docs/contributing-guide.md index d7421d7a..6b16979b 100644 --- a/docs/contributing-guide.md +++ b/docs/contributing-guide.md @@ -6,6 +6,7 @@ icon: "code-pull-request" Contributions of all kinds are welcome — code, documentation, tests, and community support. Every contribution is recognized in release notes and the GitHub contributors list. + ## Quick Start ```bash @@ -18,32 +19,24 @@ pytest New to the project? Start with [`good-first-issue`](https://github.com/semantica-agi/semantica/labels/good-first-issue) labeled tickets — they're scoped to be completable in a few hours without deep codebase knowledge. + ## Ways to Contribute -### Code + + + Fix bugs, implement features, optimize performance, or add new ingestors, parsers, and exporters using the plugin registry. + + + Fix typos, improve clarity, add missing examples, write tutorials, or keep the API reference accurate as modules evolve. + + + Add test coverage for untested modules or edge cases, reproduce reported bugs with minimal repros, or improve cross-platform reliability. + + + Answer questions in GitHub Issues and Discussions, review pull requests with constructive feedback, or share Semantica in blog posts and talks. + + -- Fix bugs and resolve open issues -- Implement new features or integrations -- Optimize performance or refactor existing modules -- Add new ingestors, parsers, or exporters using the plugin registry - -### Documentation - -- Fix typos, improve clarity, and add missing examples -- Write tutorials or domain-specific cookbook notebooks -- Keep the API reference accurate as modules evolve - -### Testing - -- Add test coverage for untested modules or edge cases -- Reproduce and confirm reported bugs with a minimal repro -- Improve test reliability across Python versions and platforms - -### Community - -- Answer questions in GitHub Issues and Discussions -- Review open pull requests with constructive feedback -- Share Semantica in blog posts, talks, or conference demos ## Development Setup @@ -64,6 +57,7 @@ flake8 semantica/ # lint Style conventions: **Black** for formatting, **isort** for imports, **flake8** for linting. All three run in CI. + ## Reporting Issues **Bug reports** should include: @@ -78,20 +72,23 @@ Style conventions: **Black** for formatting, **isort** for imports, **flake8** f - What you'd like Semantica to do - Why it benefits a broad set of users, not just your specific workflow + ## Pull Request Checklist Before submitting a PR, confirm: -- [ ] Tests pass locally — `pytest` -- [ ] New features include documentation with working code examples -- [ ] Code follows project style — Black, isort, flake8 -- [ ] Commit messages are clear and describe the *why*, not just the *what* -- [ ] No unresolved merge conflicts +Tests pass locally — `pytest` +New features include documentation with working code examples +Code follows project style — Black, isort, flake8 +Commit messages are clear and describe the *why*, not just the *what* +No unresolved merge conflicts + ## Code of Conduct All contributors are expected to follow the [Contributor Covenant Code of Conduct](https://github.com/semantica-agi/semantica/blob/main/CODE_OF_CONDUCT.md). Be respectful, patient, and constructive — especially toward newcomers. Report violations by opening an issue with the `[CoC]` prefix. + ## Help - [GitHub Issues](https://github.com/semantica-agi/semantica/issues) diff --git a/docs/cookbook.md b/docs/cookbook.md index f2bd903b..9faf9dda 100644 --- a/docs/cookbook.md +++ b/docs/cookbook.md @@ -15,7 +15,6 @@ icon: "flask" Prerequisites: Python 3.8+, Jupyter, and an API key for your preferred LLM provider. ---- ## Featured Recipes @@ -42,7 +41,6 @@ icon: "flask" ---- ## Core Tutorials @@ -101,7 +99,6 @@ Essential guides to master the Semantica framework. ---- ## Advanced Concepts @@ -160,7 +157,6 @@ Deep dive into advanced features, customization, and complex workflows. ---- ## Industry Use Cases @@ -254,7 +250,6 @@ Deep dive into advanced features, customization, and complex workflows. ---- ## How to Run diff --git a/docs/docs.json b/docs/docs.json index 2ba492b4..e4c5af30 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1,12 +1,12 @@ { "$schema": "https://mintlify.com/docs.json", - "theme": "maple", + "theme": "venus", "name": "Semantica", "description": "The Accountability and Context Layer for AI — Context Graphs · Decision Intelligence · Full Provenance", "colors": { - "primary": "#059669", + "primary": "#10B981", "light": "#10B981", - "dark": "#047857" + "dark": "#10B981" }, "favicon": "/assets/img/semantica-logo.png", "logo": { @@ -28,8 +28,8 @@ }, "background": { "color": { - "dark": "#0A0A0A", - "light": "#FAF7F0" + "dark": "#080C10", + "light": "#080C10" } }, "chat": { @@ -50,7 +50,7 @@ "navigation": { "tabs": [ { - "tab": "Documentation", + "tab": "Overview", "groups": [ { "group": "Overview", diff --git a/docs/explorer-setup.md b/docs/explorer-setup.md index 47195ac4..ccbf4634 100644 --- a/docs/explorer-setup.md +++ b/docs/explorer-setup.md @@ -4,10 +4,11 @@ description: "Install the Explorer extras, save a ContextGraph to JSON, and laun icon: "map" --- -`semantica-explorer` is an interactive browser dashboard for knowledge graph exploration. You give it a graph file, it starts a local server, and opens a browser tab where you can search nodes, find paths, inspect provenance, and run analytics — no code required after launch. +**`semantica-explorer`** is an **interactive browser dashboard** for knowledge graph exploration. You give it a graph file, it starts a local server, and opens a browser tab where you can search nodes, find paths, inspect provenance, and run analytics — no code required after launch. This page covers everything needed to go from zero to a running Explorer. For the full REST API reference and endpoint catalogue, see [Explorer Reference](reference/explorer). + ## Prerequisites The Explorer depends on FastAPI and uvicorn, which are not included in the base install: @@ -28,6 +29,7 @@ semantica-explorer --help You should see the usage message with the four available flags. If you see `command not found`, activate your virtual environment first. See [CLI Setup](cli-setup#troubleshooting) for PATH help. + ## Minimal End-to-End Example The following four steps are everything needed to get Explorer running: @@ -57,6 +59,7 @@ curl http://127.0.0.1:8000/api/health # {"status": "healthy"} ``` + ## Step 1 — Build and Save a ContextGraph Explorer loads a graph from a JSON file on disk. You need to create that file first. @@ -109,6 +112,7 @@ Explorer loads a graph from a JSON file on disk. You need to create that file fi Already have a graph from a pipeline run? Skip straight to Step 2. The only requirement is that the file was saved with `ContextGraph.save_to_file()`. + ## Step 2 — Launch Explorer ```bash @@ -127,13 +131,14 @@ The startup sequence prints: The browser opens automatically at `http://127.0.0.1:8000` shortly after the server starts. + ## CLI Flags `semantica-explorer` accepts exactly four flags: | Flag | Short | Default | Description | -| ---- | ----- | ------- | ----------- | -| `--graph` | `-g` | *(required)* | Path to a ContextGraph JSON file | +| :---- | :----- | :------- | :----------- | +| `--graph` | `-g` | *(**required**)* | Path to a ContextGraph JSON file | | `--port` | `-p` | `8000` | Port to bind the server | | `--host` | — | `127.0.0.1` | Host to bind the server | | `--no-browser` | — | off | Do not open a browser tab automatically | @@ -160,18 +165,20 @@ semantica-explorer --graph my_graph.json --no-browser `--host 0.0.0.0` makes Explorer reachable on every network interface. The server has no built-in authentication. Only use this on a trusted private network. + ## Browser Access Once the server is running: | URL | What you get | -| --- | ------------ | +| :--- | :------------ | | `http://127.0.0.1:8000` | Interactive dashboard | | `http://127.0.0.1:8000/docs` | Swagger UI — every REST endpoint, interactive | | `http://127.0.0.1:8000/api/health` | Health check — `{"status": "healthy"}` | The browser tab opens shortly after startup. If it does not open, navigate to the URL manually or pass `--no-browser` and open it yourself. + ## Running as a Python Module If `semantica-explorer` is not on `PATH`, use the module form: @@ -180,6 +187,7 @@ If `semantica-explorer` is not on `PATH`, use the module form: python -m semantica.explorer --graph my_graph.json --port 8080 ``` + ## Common Startup Errors **`Error: graph file not found: my_graph.json`** @@ -228,6 +236,7 @@ semantica-explorer --graph my_graph.json --host 0.0.0.0 This is expected in headless, SSH, and container environments. Add `--no-browser` to suppress the warning and open `http://127.0.0.1:8000` in a browser that has network access to the server. + ## What Explorer Gives You Once running, Explorer exposes a REST API and dashboard for: diff --git a/docs/faq.md b/docs/faq.md index 3f1fa7b6..1f8c500e 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -4,6 +4,23 @@ description: "Common questions about Semantica — installation, features, integ icon: "circle-question" --- + + Use **Ctrl+F** / **Cmd+F** to search this page. Common jumps: [Installation](#installation) · [Data & Features](#data--features) · [Troubleshooting](#troubleshooting) + + +## Quick Answers + +| Question | Answer | +| :-------- | :------ | +| License? | MIT — free forever, no paywalled features | +| Python version? | 3.8+ (3.11+ recommended) | +| API key required? | Optional — pattern extraction works with no keys | +| Works with LangChain / LlamaIndex? | Yes — Semantica is a layer on top, not a replacement | +| Production-ready? | Yes — 1,000+ tests, v0.5.0 ships with 12 security fixes | +| Latest version? | **v0.5.0** (May 2026) | +| Local LLMs? | Yes — Ollama via LiteLLM, HuggingFaceLLM for air-gapped | + + ## General @@ -55,6 +72,7 @@ pip install --upgrade semantica + ## Installation @@ -90,7 +108,7 @@ If you're on an older version, install extras individually: `pip install "semant | Requirement | Minimum | Recommended | -| ----------- | ------- | ----------- | +| :----------- | :------- | :----------- | | Python | 3.8 | 3.11+ | | RAM | 4 GB | 16 GB+ | | Storage | 2 GB | 20 GB+ | @@ -100,6 +118,7 @@ If you're on an older version, install extras individually: `pip install "semant + ## Data & Features @@ -107,7 +126,7 @@ If you're on an older version, install extras individually: `pip install "semant | Category | Sources | -| -------- | ------- | +| :-------- | :------- | | **Files** | PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, Parquet (v0.5.0), XML (v0.5.0), archives | | **Web** | `WebIngestor` crawl, RSS feeds, sitemaps | | **Databases** | PostgreSQL, MySQL, Snowflake via `DBIngestor` / `SnowflakeIngestor` | @@ -205,6 +224,7 @@ pip install --upgrade semantica + ## Technical @@ -251,6 +271,7 @@ Yes. v0.5.0 ships with: + ## Troubleshooting @@ -314,6 +335,7 @@ set PYTHONIOENCODING=utf-8 + ## Support diff --git a/docs/getting-started.md b/docs/getting-started.md index 5b51fd73..8883c692 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -25,119 +25,230 @@ icon: "rocket" -## Installation -```bash -pip install semantica -``` +## Setup in 3 Steps -With all optional dependencies: + + + -```bash -pip install semantica[all] -``` + ```bash pip (recommended) + pip install semantica + ``` -For virtual environments and platform-specific setup, see the full [Installation](installation) guide. + ```bash With all extras + pip install semantica[all] + ``` -**Verify:** + ```bash From source + git clone https://github.com/semantica-agi/semantica.git + cd semantica + pip install -e ".[dev]" + ``` -```python -import semantica -print(semantica.__version__) # 0.5.0 -``` + -## Quick Start + + Verify installation: + ```python + import semantica + print(semantica.__version__) # 0.5.0 + ``` + + - + + Pick the track that matches what you're building — each starts with a focused 5-minute example. -```python Knowledge Graph -from semantica.ingest import FileIngestor -from semantica.parse import DocumentParser -from semantica.semantic_extract import NERExtractor, RelationExtractor -from semantica.kg import GraphBuilder + | Track | You want to... | Start with | + | :----- | :-------------- | :--------- | + | **Knowledge Graph** | Turn documents into structured, queryable graphs | [Quickstart → Step 1](quickstart) | + | **Agent Context** | Give your AI agent persistent memory and decision tracking | [Context reference](reference/context) | + | **GraphRAG** | Ground LLM answers in structured knowledge | [Concepts → GraphRAG](concepts#graphrag) | + | **MCP Integration** | Use Semantica from Claude Desktop or VS Code | [MCP Server](reference/mcp_server) | -ingestor = FileIngestor() -sources = ingestor.ingest("data/sample.pdf") + -parser = DocumentParser() -parsed = parser.parse(sources[0]) + + The full 6-step pipeline — ingest, parse, extract, build, visualize, export — is in the [Quickstart](quickstart). Takes under 5 minutes with pattern-based extraction (no API key required). -ner = NERExtractor() -entities = ner.extract(parsed) + + An LLM API key is **optional** for the quickstart. Pattern-based extraction works out of the box — upgrade to LLM extraction for higher accuracy when you're ready. + + + -rel = RelationExtractor() -relationships = rel.extract(parsed, entities=entities) -graph = GraphBuilder(merge_entities=True).build( - entities=entities, relationships=relationships -) -print(f"{len(graph.nodes)} nodes, {len(graph.edges)} edges") -``` +## Choose Your Path -```python Agent Context -from semantica.context import AgentContext, ContextGraph -from semantica.vector_store import VectorStore + + + Build a structured knowledge graph from any document or data source. -context = AgentContext( - vector_store=VectorStore(backend="faiss", dimension=768), - knowledge_graph=ContextGraph(advanced_analytics=True), - decision_tracking=True, -) + ```python + from semantica.ingest import FileIngestor + from semantica.parse import DocumentParser + from semantica.semantic_extract import NERExtractor, RelationExtractor + from semantica.kg import GraphBuilder -# Store a fact with provenance -context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%") + # 1. Ingest + sources = FileIngestor().ingest("data/report.pdf") -# Record a decision with causal chain -decision_id = context.record_decision( - category="model_selection", - scenario="Choose LLM for production pipeline", - reasoning="GPT-4 benchmark advantage justifies cost increase", - outcome="selected_gpt4", - confidence=0.91, -) + # 2. Parse + parsed = DocumentParser().parse(sources[0]) -# Find similar past decisions -precedents = context.find_precedents("model selection", limit=5) -``` + # 3. Extract + ner = NERExtractor(method="pattern") # no API key needed + entities = ner.extract(parsed) + relationships = RelationExtractor().extract(parsed, entities=entities) -```python GraphRAG -from semantica.context import AgentContext, ContextGraph -from semantica.vector_store import VectorStore -from semantica.reasoning import ReasoningEngine + # 4. Build + graph = GraphBuilder(merge_entities=True).build( + entities=entities, relationships=relationships + ) + print(f"{len(graph['nodes'])} nodes, {len(graph['relationships'])} edges") + ``` -context = AgentContext( - vector_store=VectorStore(backend="faiss", dimension=768), - knowledge_graph=ContextGraph(advanced_analytics=True), -) + **Next:** [Full pipeline walkthrough →](quickstart) + -context.load_graph("company_kg.json") + + Give your agent persistent memory, decision tracking, and precedent search. -# Multi-hop GraphRAG query -result = context.query( - "What companies were founded by people who worked at Apple?", - mode="graphrag", - reasoning=True, -) + ```python + from semantica.context import AgentContext, ContextGraph + from semantica.vector_store import VectorStore -# Every claim links back to a source node -for claim in result.claims: - print(f"{claim.text} → source: {claim.source_node}") -``` + context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph(advanced_analytics=True), + decision_tracking=True, + ) + + # Store a fact with provenance + context.store("GPT-4 outperforms GPT-3.5 on reasoning by 40%") + + # Record a decision with full causal chain + decision_id = context.record_decision( + category="model_selection", + scenario="Choose LLM for production pipeline", + reasoning="GPT-4 benchmark advantage justifies cost", + outcome="selected_gpt4", + confidence=0.91, + ) + + # Search past decisions before making a new one + precedents = context.find_precedents("model selection", limit=5) + ``` + + **Next:** [Context module reference →](reference/context) + + + + Ground every LLM response in your knowledge graph — no floating assertions. + + ```python + from semantica.context import AgentContext, ContextGraph + from semantica.vector_store import VectorStore + + context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph(advanced_analytics=True), + ) + + # Load your knowledge graph + context.load_graph("company_kg.json") + + # Multi-hop GraphRAG query + result = context.query( + "What companies were founded by people who worked at Apple?", + mode="graphrag", + reasoning=True, + ) + + # Every claim links back to a source node + for claim in result.claims: + print(f"{claim.text} → source: {claim.source_node}") + ``` + + **Next:** [GraphRAG concepts →](concepts#graphrag) + + + + Use Semantica from Claude Desktop, VS Code, Cursor, or any MCP client — no Python code required after setup. + + ```bash + pip install semantica + ``` + + Add to your MCP client config: + + ```json + { + "mcpServers": { + "semantica": { + "command": "semantica-mcp" + } + } + } + ``` + + 12 tools available instantly: extract entities, query graph, record decisions, run reasoning, export results. + + **Next:** [MCP Server reference →](reference/mcp_server) + + - ## Core Architecture Semantica uses a modular, layered architecture — import only what you need. -| Layer | Modules | Purpose | -| ----- | ------- | ------- | -| **Input** | `ingest`, `parse`, `split`, `normalize` | Load and prepare data | -| **Semantic** | `semantic_extract`, `kg`, `ontology`, `reasoning` | Extract meaning | -| **Storage** | `embeddings`, `vector_store`, `graph_store`, `triplet_store` | Persist knowledge | -| **Quality** | `deduplication`, `conflicts` | Validate and clean | -| **Context** | `context`, `provenance`, `change_management` | Track decisions and lineage | -| **Output** | `export`, `visualization`, `pipeline`, `explorer` | Deliver results | + + + Load and prepare data from any source. + **Modules:** `ingest`, `parse`, `split`, `normalize` + + + Extract meaning from raw text. + **Modules:** `semantic_extract`, `kg`, `ontology`, `reasoning` + + + Persist knowledge for retrieval. + **Modules:** `embeddings`, `vector_store`, `graph_store`, `triplet_store` + + + Validate and deduplicate. + **Modules:** `deduplication`, `conflicts` + + + Track decisions and lineage. + **Modules:** `context`, `provenance`, `change_management` + + + Deliver results downstream. + **Modules:** `export`, `visualization`, `pipeline`, `explorer` + + + + +## "Which module do I need?" Quick Reference + +| I want to... | Module | Key class | +| :------------ | :------ | :--------- | +| Load a PDF / web page / database | `ingest` | `FileIngestor`, `WebIngestor` | +| Extract text and tables from a PDF | `parse` | `DocumentParser`, `DoclingParser` | +| Find entities in text | `semantic_extract` | `NERExtractor` | +| Build a knowledge graph | `kg` | `GraphBuilder` | +| Store and search vectors | `vector_store` | `VectorStore` | +| Give my agent persistent memory | `context` | `AgentContext` | +| Record AI decisions with audit trail | `context` | `AgentContext.record_decision()` | +| Query my graph with natural language | `reasoning` | `GraphReasoner` | +| Export to RDF / Neo4j / Parquet | `export` | `RDFExporter`, `LPGExporter` | +| Visualize a knowledge graph | `visualization` | `KGVisualizer` | +| Run a reproducible pipeline | `pipeline` | `PipelineBuilder` | +| Use Semantica from Claude Desktop | `mcp_server` | `semantica-mcp` | + ## Next Steps @@ -156,6 +267,7 @@ Semantica uses a modular, layered architecture — import only what you need. + ## Help diff --git a/docs/glossary.md b/docs/glossary.md index 3240d29c..70da7d95 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -10,6 +10,7 @@ icon: "book" A quick-reference dictionary of every concept, data structure, algorithm, and standard referenced in Semantica's documentation and codebase. + ## Core Concepts **Agent** @@ -33,6 +34,7 @@ A directed, typed connection between two entities: e.g., `works_for`, `located_i **Semantic** Relating to meaning in language or logic. Semantic understanding captures context and intent — going beyond keyword matching to understand what text *means*. + ## Data Processing **Chunking** @@ -47,6 +49,7 @@ Standardizing data into a consistent canonical form: converting dates to ISO for **Parsing** Extracting structured text, layout, and metadata from unstructured or semi-structured documents — PDFs, Word files, HTML, PPTX. `DoclingParser` additionally handles multi-column layouts, merged-cell tables, and OCR. + ## Artificial Intelligence **Abductive Reasoning** @@ -67,6 +70,7 @@ An AI model trained on large text corpora, capable of understanding and generati **RAG (Retrieval Augmented Generation)** A technique that enhances LLM outputs by retrieving relevant context from a knowledge base before generating a response. GraphRAG extends this with graph traversal for more precise, structured retrieval. + ## Knowledge Graph Components **Allen Interval Algebra** @@ -90,6 +94,7 @@ A knowledge graph where nodes and edges carry `valid_from` / `valid_until` time **Triplet** The atomic unit of knowledge: a `(subject, predicate, object)` triple — e.g., `(Apple_Inc, founded_by, Steve_Jobs)`. The building block of RDF and SPARQL-based storage. + ## Entity Recognition & Extraction **Coreference Resolution** @@ -107,6 +112,7 @@ Identifying and classifying named entities in text into predefined categories: p **Relationship Extraction** Identifying and extracting typed semantic relationships between entities — e.g., `(Google, acquired, DeepMind)` — from raw text. + ## Ontology & Schema **Axiom** @@ -130,6 +136,7 @@ The W3C standard for validating RDF graphs against a set of shape constraints. S **SKOS (Simple Knowledge Organization System)** A W3C standard for representing controlled vocabularies, taxonomies, and thesauri. Used in Semantica for domain vocabulary management. + ## Storage & Retrieval **Embedding** @@ -147,6 +154,7 @@ A database designed specifically for storing and querying RDF `(subject, predica **Vector Store** A database optimized for storing and searching high-dimensional embedding vectors by similarity. Semantica supports FAISS, Pinecone, Weaviate, Qdrant, Milvus, and PgVector. + ## Graph Analytics **Centrality** @@ -164,6 +172,7 @@ Semantica's v0.5.0 feature for semantic neighborhood exploration: N×N distance **PageRank** An algorithm measuring node importance based on the structure of incoming relationships — originally designed for web pages, applicable to any directed graph. + ## Query Languages & Standards **Cypher** @@ -178,6 +187,7 @@ The W3C standard for representing information as subject-predicate-object triple **SPARQL** The W3C query language for RDF data. Semantica's `SparqlReasoner` uses SPARQL for query-based inference over RDF graphs. + ## Data Quality **Conflict Resolution** @@ -192,6 +202,7 @@ Identifying and merging duplicate entity records. Semantica v2 strategies (`bloc **W3C PROV-O** The W3C provenance ontology standard. Semantica tracks lineage across all modules in PROV-O compliant format — suitable for HIPAA, SOX, GDPR, and FDA 21 CFR Part 11 compliance. + ## Security Terms **SSRF (Server-Side Request Forgery)** @@ -200,9 +211,20 @@ A vulnerability where a server is induced to make requests to unintended destina **XXE (XML External Entity)** A vulnerability in XML parsers that allows attackers to read arbitrary files or trigger SSRF. Semantica's `XMLIngestor` (v0.5.0) uses an XXE-safe lxml backend. + ## See Also -- [Core Concepts](concepts) — deeper explanation of key ideas with code examples -- [Getting Started](getting-started) — first working examples -- [Modules Guide](modules) — every module explained -- [API Reference](reference/context) — complete technical reference + + + Deeper explanation of key ideas with code examples. + + + First working examples — no prior graph experience required. + + + All 27 modules explained with code and pipeline chains. + + + Complete technical reference for every class and method. + + diff --git a/docs/governance.md b/docs/governance.md index 791279d8..19fc19a8 100644 --- a/docs/governance.md +++ b/docs/governance.md @@ -6,25 +6,21 @@ icon: "scale-balanced" > Semantica is maintained by Hawksight AI with community contributions under an open governance model. ---- ## Roles -**Maintainers** (Hawksight AI team) -- Review and merge pull requests -- Manage releases and code quality -- Set project direction and community standards + + + Hawksight AI team — review and merge PRs, manage releases and code quality, set project direction and community standards. + + + Submit code, documentation, and bug reports. Help with issues and reviews. Recognized in [CONTRIBUTORS.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTORS.md). + + + Use Semantica, provide feedback, share use cases, and participate in GitHub Discussions and Discord. + + -**Contributors** -- Submit code, documentation, and bug reports -- Help with issues and reviews -- Recognized in [CONTRIBUTORS.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTORS.md) - -**Community Members** -- Use Semantica and provide feedback -- Share use cases and participate in discussions - ---- ## Decision Process @@ -44,19 +40,17 @@ icon: "scale-balanced" - Minimum 1-week community discussion period - Maintainers decide based on community feedback and technical feasibility ---- ## Releases Semantica follows **Semantic Versioning** (`MAJOR.MINOR.PATCH`): | Level | Trigger | Cadence | -|-------|---------|---------| +| :------- | :--------- | :--------- | | MAJOR | Breaking changes | Quarterly or as needed | | MINOR | New features (backward compatible) | Monthly or when ready | | PATCH | Bug fixes (backward compatible) | As bugs are fixed | ---- ## Code Review @@ -68,7 +62,6 @@ Semantica follows **Semantic Versioning** (`MAJOR.MINOR.PATCH`): **Guidelines for contributors:** address comments promptly, ask questions when unclear, be open to feedback. ---- ## Communication @@ -77,23 +70,32 @@ Semantica follows **Semantic Versioning** (`MAJOR.MINOR.PATCH`): - **GitHub Discussions** — community conversation - **Security Advisories** — [report security issues privately](https://github.com/semantica-agi/semantica/security/advisories/new) ---- ## Project Goals -1. **Usability** — Easy to use and understand -2. **Reliability** — Production-ready quality -3. **Performance** — Efficient and scalable -4. **Extensibility** — Easy to extend with plugins and custom modules -5. **Community** — Welcoming and inclusive + + + Easy to use and understand — sensible defaults, clear documentation, minimal ceremony. + + + Production-ready quality — tested across Python versions, platforms, and real-world workloads. + + + Efficient and scalable — from single-machine notebooks to enterprise graph databases. + + + Easy to extend with plugins and custom modules via the `PluginRegistry` pattern. + + + Welcoming and inclusive — all backgrounds and experience levels contribute and are recognized. + + ---- ## License MIT License — see [LICENSE](https://github.com/semantica-agi/semantica/blob/main/LICENSE) and the [License page](license). ---- ## See Also diff --git a/docs/graph_stores/apache_age.md b/docs/graph_stores/apache_age.md index 82341a64..778a75e1 100644 --- a/docs/graph_stores/apache_age.md +++ b/docs/graph_stores/apache_age.md @@ -9,12 +9,11 @@ icon: "database" Apache AGE is a PostgreSQL extension that adds graph database functionality, enabling you to run openCypher queries alongside traditional SQL. This backend lets Semantica use AGE as a property graph store with the same interface as Neo4j and FalkorDB. ---- ## Prerequisites | Component | Version | -|-----------|---------| +| :----------- | :--------- | | PostgreSQL | 12+ | | Apache AGE | 1.4+ (compiled and installed) | | psycopg2 | 2.9+ | @@ -25,61 +24,61 @@ pip install psycopg2-binary Apache AGE must be compiled and installed into your PostgreSQL instance. See the [AGE installation guide](https://age.apache.org/age-manual/master/intro/setup.html). ---- ## Quick Start -```python -from semantica.graph_store import GraphStore + + + Use `GraphStore(backend="age", …)` — the same interface as Neo4j and FalkorDB: -# Using the unified GraphStore facade -store = GraphStore( - backend="age", - connection_string="host=localhost dbname=agedb user=postgres password=secret", - graph_name="semantica", -) -store.connect() + ```python + from semantica.graph_store import GraphStore -# Create nodes -alice = store.create_node(labels=["Person"], properties={"name": "Alice", "age": 30}) -bob = store.create_node(labels=["Person"], properties={"name": "Bob", "age": 25}) + store = GraphStore( + backend="age", + connection_string="host=localhost dbname=agedb user=postgres password=secret", + graph_name="semantica", + ) + store.connect() -# Create relationship -rel = store.create_relationship(alice["id"], bob["id"], "KNOWS", {"since": 2023}) + alice = store.create_node(labels=["Person"], properties={"name": "Alice", "age": 30}) + bob = store.create_node(labels=["Person"], properties={"name": "Bob", "age": 25}) + store.create_relationship(alice["id"], bob["id"], "KNOWS", {"since": 2023}) -# Query -result = store.execute_query("MATCH (p:Person) RETURN p", cols="p agtype") -print(result["records"]) + result = store.execute_query("MATCH (p:Person) RETURN p", cols="p agtype") + print(result["records"]) + store.close() + ``` + + + Import `ApacheAgeStore` directly when you need AGE-specific behaviour: -store.close() -``` + ```python + from semantica.graph_store.age_store import ApacheAgeStore -### Direct Usage (without facade) + store = ApacheAgeStore( + connection_string="host=localhost dbname=agedb user=postgres password=secret", + graph_name="my_graph", + ) + store.connect() -```python -from semantica.graph_store.age_store import ApacheAgeStore + node = store.create_node(["Entity"], {"semantica_id": "ent-001", "value": "test"}) + print(node) + # {"id": 844424930131969, "labels": ["Entity"], + # "properties": {"semantica_id": "ent-001", "value": "test"}} -store = ApacheAgeStore( - connection_string="host=localhost dbname=agedb user=postgres password=secret", - graph_name="my_graph", -) -store.connect() + store.close() + ``` + + -node = store.create_node(["Entity"], {"semantica_id": "ent-001", "value": "test"}) -print(node) -# {"id": 844424930131969, "labels": ["Entity"], "properties": {"semantica_id": "ent-001", "value": "test"}} - -store.close() -``` - ---- ## Configuration ### Environment Variables | Variable | Description | Default | -|----------|-------------|---------| +| :---------- | :------------- | :--------- | | `GRAPH_STORE_AGE_CONNECTION_STRING` | PostgreSQL connection string | `host=localhost dbname=agedb user=postgres password=postgres` | | `GRAPH_STORE_AGE_GRAPH_NAME` | AGE graph name | `semantica` | @@ -92,27 +91,33 @@ graph_store_config.set("age_connection_string", "host=db.example.com dbname=prod graph_store_config.set("age_graph_name", "production") ``` ---- ## Connection & Initialization -On `connect()`, the store performs idempotent setup: +On `connect()`, the store performs idempotent setup — safe to call repeatedly: -1. `CREATE EXTENSION IF NOT EXISTS age;` -2. `LOAD 'age';` -3. `SET search_path = ag_catalog, "$user", public;` -4. Creates the named graph if it does not already exist. + + + `CREATE EXTENSION IF NOT EXISTS age;` + + + `LOAD 'age';` + + + `SET search_path = ag_catalog, "$user", public;` + + + Creates the named graph if it does not already exist. + + -This is safe to call repeatedly. - ---- ## ID Handling Apache AGE auto-generates internal vertex/edge IDs (large integers). These are **not** the same as any semantic or application-level ID you may want to assign. | Concept | Description | -|---------|-------------| +| :--------- | :------------- | | **AGE internal ID** | Auto-generated by AGE. Exposed as `"id"` in all returned dicts. Used in `delete_node()`, `get_node()`, etc. | | **Semantic ID** | Application-level identifier. Store it in the `semantica_id` property. | @@ -127,7 +132,6 @@ node = store.create_node( Never mix AGE internal IDs with semantic IDs. Use `node["id"]` for graph operations (delete, update, traverse) and `node["properties"]["semantica_id"]` for application-level lookups. ---- ## Label Handling @@ -147,7 +151,6 @@ node = store.create_node( # Returned: {"id": ..., "labels": ["Person", "Employee", "Admin"], "properties": {"name": "Alice"}} ``` ---- ## Cypher Query Execution @@ -182,7 +185,6 @@ result = store.execute_query( If omitted, the store attempts to infer columns from the `RETURN` clause. ---- ## Transactions @@ -192,14 +194,13 @@ The store uses explicit PostgreSQL transactions: - **Exception** → `ROLLBACK`, then re-raise as `ProcessingError` - No silent failures ---- ## API Reference All methods match the standard Semantica graph store backend interface: | Method | Description | -|--------|-------------| +| :-------- | :------------- | | `connect(**options)` | Connect and initialize AGE | | `close()` | Close the connection | | `create_node(labels, properties)` | Create a vertex | @@ -217,7 +218,6 @@ All methods match the standard Semantica graph store backend interface: | `create_index(label, property_name, index_type)` | Create a PostgreSQL index | | `get_stats()` | Graph statistics | ---- ## Docker Setup diff --git a/docs/index.md b/docs/index.md index ce3eb122..349cbf87 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,62 +7,131 @@ description: "The Accountability and Context Layer for AI — Context Graphs · **v0.5.0 is live** — Ontology Hub, Distance Intelligence, SHACL Studio, Parquet & XML ingestion, 12 security fixes. [What's new →](#whats-new) -> Most AI agents act without a trail. Semantica adds the layer your stack is missing: structured context graphs, auditable decision records, and full provenance from every output back to its source — so your AI isn't just powerful, it's **accountable**. +Your AI agent just made a decision. Now someone needs to explain it. -## The Problem +*What did it know at the time? Which facts shaped the outcome? Where did those facts come from? Has it made the same call before — and did that go well?* -AI agents today are powerful but not trustworthy. Five structural gaps make them impossible to deploy in regulated environments: +If your stack can't answer those questions with a traceable record, you have a gap. Not a capability gap — an **accountability gap**. It's the reason AI hasn't landed at scale in healthcare, finance, legal, and government. And it's why teams building for those markets keep rebuilding the same guardrails from scratch. + +**Semantica closes that gap.** It's the context and accountability layer that sits beneath your existing agent framework — not a replacement for LangChain or LlamaIndex, but the infrastructure that makes their outputs trustworthy. + + + + Production-hardened with a full regression suite + + + Every capability independently importable + + + OpenAI, Anthropic, Ollama, Groq, and more + + + Open source, no vendor lock-in, fully forkable + + + + +## The Problem Every Production AI Team Hits + +Powerful agents aren't automatically trustworthy ones. Five structural blind spots make modern AI systems impossible to deploy in regulated environments: - Agents store embeddings, not meaning. There's no way to ask *why* something was recalled or trace a fact to its source. + Agents store embeddings, not meaning. + - No way to ask *why* a fact was recalled + - No link from a recalled fact back to its source document + - Context is a black box that resets on every run - Agents act continuously but record nothing. When something breaks, there's no history to debug or audit. + Agents act continuously but record nothing. + - No history to hand to a regulator or auditor + - No way to replay or reproduce a past decision + - Debugging means re-running, not reviewing - Outputs can't be traced back to source facts. In healthcare, finance, and legal, this is a hard compliance blocker. + Outputs can't be traced to source facts. + - In healthcare, finance, and legal — this is a hard compliance blocker + - No lineage from inference back to the original document + - Impossible to demonstrate what the agent actually relied on - Black-box answers with zero explanation of how a conclusion was reached. + Black-box answers with no explanation. + - Impossible to validate the reasoning path + - Impossible to contest a specific conclusion + - No basis for improving or correcting future behavior - Contradictory facts silently coexist in vector stores, producing unpredictable and inconsistent outputs. + Contradictory facts silently coexist in vector stores. + - No detection when two sources disagree + - Outputs become inconsistent and unpredictable over time + - Silent failures compound as the knowledge base grows -These aren't edge cases. They're why AI cannot be deployed in healthcare, finance, legal, and government without custom guardrails built from scratch. + + These aren't edge cases. They're why enterprise AI pilots stall — and why your compliance team keeps saying *not yet*. + -## The Solution -Semantica is the **accountability and context layer** you add on top of your existing AI stack. Not a replacement for LangChain or LlamaIndex — the infrastructure that makes their outputs trustworthy. +## What Semantica Adds to Your Stack + +Semantica gives every agent the infrastructure it needs to be accountable. Drop it into your existing setup in minutes: - A structured, queryable graph of everything your agent knows, decides, and reasons about. Persistent across runs. + A structured, queryable graph of everything your agent knows, decides, and reasons about. + - Persistent across agent runs — no context loss between sessions + - Queryable with SPARQL and full graph algorithms + - Temporal model with `valid_from` / `valid_until` on nodes and edges + - Point-in-time snapshots of the full knowledge state - Every decision is a first-class object: recorded, causally linked, searchable by precedent, and analyzable for downstream impact. + Every decision is a first-class object in your system. + - `record_decision()` captures full lifecycle and causal chain + - Hybrid precedent search over past decisions for consistency + - `analyze_decision_impact()` shows downstream consequences + - Causal chain visualization from trigger to outcome - Every fact links back to its source. W3C PROV-O compliant. Full lineage from ingestion to inference. + Every fact links to its source document and ingestion event. + - W3C PROV-O compliant lineage across all modules + - Full traceability from raw input to final inference + - `recorded_at` stamping with OWL-Time export + - Audit-ready for HIPAA, SOX, GDPR, FDA 21 CFR Part 11 - Forward chaining, Rete, deductive, abductive, SPARQL, Datalog. Explainable paths, not black boxes. + Explainable reasoning paths — not black boxes. + - Forward chaining, Rete, deductive, abductive + - SPARQL query-based inference over RDF graphs + - Datalog with recursive Horn clause rules + - Every conclusion backed by a traceable derivation path - Point-in-time queries, Allen interval algebra, temporal provenance, OWL-Time export. + Your graph knows not just *what* — but *when*. + - Allen interval algebra — all 13 temporal relations + - Point-in-time queries over historical graph states + - Temporal provenance stamping on every fact + - OWL-Time export for standards-compliant archiving - Visual editor, SHACL Studio, alignment authoring, health dashboard. Full ontology lifecycle in the browser. + Full ontology lifecycle in the browser. + - Visual editor for schema design and editing + - SHACL Studio for constraint authoring and validation + - Alignment authoring across multiple ontologies + - Health dashboard and version control built in -Works alongside any LLM provider and any agent framework. + + Works alongside any LLM provider and any agent framework — add it to an existing stack without changing your architecture. + Semantica four-layer architecture: Ingestion → Processing → Intelligence → Application -## Quick Start + +## See It In Action + +One pip install. A few lines to connect your agent. Everything else becomes traceable. ```bash pip install semantica @@ -150,16 +219,111 @@ decision_id = context.record_decision( - Step-by-step pipeline walkthrough. + Step-by-step pipeline walkthrough - 40+ real-world Jupyter notebooks. + 40+ real-world Jupyter notebooks - Community chat and support. + Community chat and support + +## Built for Where Mistakes Have Consequences + +Semantica was designed for domains where every decision must be explainable and every fact must be traceable: + + + + - Clinical decision support with full audit trails + - Drug interaction and contraindication graphs + - Patient safety event tracking and root-cause analysis + - HIPAA-compliant provenance chains out of the box + + + - Fraud detection knowledge graphs + - Risk assessment trails built to survive an audit + - SOX, GDPR, and MiFID II compliance infrastructure + - Model decision lineage for regulatory reporting + + + - Evidence-backed research with every cited fact provenance-linked + - Contract analysis with traceable clause extraction + - Regulatory change tracking across jurisdictions + - Full reasoning paths ready for court-admissible documentation + + + - Threat attribution graphs linking actors, TTPs, and indicators + - Incident response timelines with full event provenance + - Security audit trails across the complete kill chain + - MITRE ATT&CK-aligned knowledge graph integration + + + - Policy decision trails from brief to outcome + - Classified information handling with provenance chains + - Chain-of-custody scrutiny for intelligence reporting + - Air-gapped deployment with local LLM support + + + - Power grid state tracking with temporal intelligence + - Transportation safety event graphs + - Emergency response coordination with decision audit trails + - Consequence modeling for high-stakes operational decisions + + + + +## Start Here + + + + ```bash + pip install semantica + ``` + See [Installation](installation) for optional extras (`[all]`, `[neo4j]`, `[pinecone]`) and environment setup. + + + Build a complete knowledge graph pipeline in [5 minutes](quickstart): + - Ingest documents from any source + - Extract entities and relationships + - Build and query the graph + - Record and trace a decision + + + [Core Concepts](concepts) covers: + - Knowledge graphs vs. vector stores — when to use each + - What GraphRAG is and how Semantica implements it + - How provenance and decision tracking work together + - The accountability layer architecture + + + Every module has a dedicated [reference page](reference/context) with: + - Full class and method documentation + - Parameter tables with types and defaults + - Runnable code examples for each feature + + + + + + Get Semantica installed in under a minute + + + Build a complete knowledge graph pipeline in 5 minutes + + + The mental model behind the API + + + Exact module, class, and method details + + + Domain notebooks for real-world use cases + + + + ## What's New @@ -169,7 +333,7 @@ decision_id = context.record_decision( Released **May 11, 2026** | Area | Highlights | -|------|------------| +| :------ | :------------ | | **Ontology Hub** | Visual editor, SHACL Studio, alignment authoring, health dashboard, version control — full ontology lifecycle in the browser | | **Distance Intelligence** | Semantic neighborhoods, N×N distance matrices, ego-mode visualization, distance band classification, embedding cache optimization | | **Parquet Ingestion** | `ParquetIngestor` with PyArrow — single file, partitioned directories, Hive-style discovery, selective column reading | @@ -187,7 +351,7 @@ pip install semantica==0.5.0 | Area | Highlights | -|------|------------| +| :------ | :------------ | | **Temporal Intelligence** | 6-PR system: temporal data model, point-in-time queries, Allen interval algebra (all 13 relations), OWL-Time export | | **Knowledge Explorer API** | Full FastAPI backend — 99 tests, 12 export formats, WebSocket progress, thread-safe sessions, audit trail | | **Ontology Foundations** | SHACL generation/validation, SKOS vocabulary, ontology alignment API, diff & migration tooling | @@ -198,97 +362,96 @@ pip install semantica==0.5.0 -## Start Here - - - ```bash - pip install semantica - ``` - See [Installation](installation) for optional extras and environment setup. - - - Build a complete knowledge graph pipeline — ingest, extract, build, query — in [5 minutes](quickstart). - - - [Core Concepts](concepts) explains knowledge graphs, GraphRAG, provenance, and decision intelligence. Read this before the API reference. - - - Every module has a dedicated [reference page](reference/context) with class docs, parameter tables, and runnable examples. - - - - - - Get Semantica installed in under a minute. - - - Build a complete knowledge graph pipeline in 5 minutes. - - - The mental model behind the API. - - - Jump here for exact module, class, and method details. - - - Explore domain notebooks once you have the basics working. - - - -## Capabilities +## Full Capabilities -- **Context Graphs** — structured, persistent graph of entities, relationships, and decisions -- **Decision tracking** — `record_decision()` with full lifecycle management and causal chains -- **Precedent search** — hybrid similarity search over past decisions for consistency -- **Influence analysis** — `analyze_decision_impact()`, `analyze_decision_influence()` -- **Temporal graphs** — `valid_from` / `valid_until` on nodes and edges, point-in-time queries -- **Distance Intelligence** — semantic neighborhoods, N×N distance matrices, ego-mode exploration +### Context Graphs + +- Structured, persistent graph of entities, relationships, and decisions +- Temporal model with `valid_from` / `valid_until` on every node and edge +- Point-in-time queries across historical graph states +- Distance Intelligence — semantic neighborhoods and N×N distance matrices + +### Decision Tracking + +- `record_decision()` with full lifecycle management and causal chains +- Hybrid similarity search over past decisions for consistency enforcement +- `analyze_decision_impact()` and `analyze_decision_influence()` for consequence modeling +- Ego-mode exploration for targeted neighborhood investigation -- **NER** — named entity recognition with pattern, ML, or LLM methods -- **Relation extraction** — typed triplets via LLM or rule-based methods -- **Deduplication v2** — `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster -- **Ontology Hub** — visual editor, SHACL Studio, alignments, health dashboard -- **Datalog reasoning** — recursive Horn clause rules with fixpoint semantics -- **SPARQL reasoning** — query-based inference over RDF graphs +### Entity & Relation Extraction + +- Named entity recognition — pattern, ML, or LLM methods +- Typed triplet extraction via LLM or rule-based pipelines +- Event extraction with temporal and causal linking + +### Ontology & Schema + +- Ontology Hub — visual editor, SHACL Studio, alignments, health dashboard +- Deduplication v2 — `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster +- Datalog reasoning — recursive Horn clause rules with fixpoint semantics +- SPARQL reasoning — query-based inference over RDF graphs -- **W3C PROV-O** — lineage tracking across all modules -- **Change management** — version control with SHA-256 checksums and audit trails -- **Temporal provenance** — `recorded_at` stamping, OWL-Time export -- **Compliance** — HIPAA, SOX, GDPR, FDA 21 CFR Part 11 infrastructure +### Lineage Tracking + +- W3C PROV-O lineage across all modules — every fact has a source +- `recorded_at` stamping with full OWL-Time export +- Change management with SHA-256 checksums and version control +- Full audit trails from ingestion event to final inference + +### Compliance Infrastructure + +- HIPAA — patient data handling with audit-ready provenance chains +- SOX / MiFID II — financial decision records with full traceability +- GDPR — data lineage for subject access and right-to-erasure workflows +- FDA 21 CFR Part 11 — electronic records and signature compliance -**Ingestion:** PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, Parquet, XML, archives, web crawl, SQL, Snowflake, feeds, email, repositories, MCP +### Ingestion Formats -**Vector Stores:** FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory +- Documents: PDF, DOCX, HTML, PPTX, Docling layout analysis +- Structured data: JSON, CSV, Excel, Parquet, XML +- Sources: web crawl, SQL, Snowflake, feeds, email, code repositories, MCP -**Graph Stores:** Neo4j, FalkorDB, Apache AGE, Amazon Neptune +### Vector Stores -**Export:** RDF (Turtle, JSON-LD, N-Triples, XML), Parquet, ArangoDB AQL, OWL ontologies +- FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory + +### Graph Stores + +- Neo4j, FalkorDB, Apache AGE, Amazon Neptune + +### Export Formats + +- RDF: Turtle, JSON-LD, N-Triples, RDF/XML +- Tabular: Parquet, CSV, Arrow +- Graph: GraphML, GEXF, DOT, ArangoDB AQL +- Ontology: OWL, SKOS, SHACL + ## Module Reference | Module | What it provides | -|--------|-----------------| +| :-------- | :----------------- | | `semantica.context` | Context graphs, agent memory, decision tracking, causal analysis, precedent search | | `semantica.kg` | KG construction, graph algorithms, temporal model, Allen interval algebra | | `semantica.semantic_extract` | NER, relation extraction, event extraction, triplet generation | @@ -317,41 +480,29 @@ pip install semantica==0.5.0 | `semantica.core` | Orchestration, ConfigManager, LifecycleManager, PluginRegistry, MethodRegistry | | `semantica.utils` | Logging, validation, progress tracking, hash utilities, nested dict helpers | -## Built for High-Stakes Domains - -Where every decision must be accountable and mistakes have real consequences: - - - - Clinical decision support, drug interaction graphs, patient safety audit trails, HIPAA compliance. - - - Fraud detection graphs, SOX/GDPR/MiFID II compliance, risk assessment trails. - - - Evidence-backed research, contract analysis, regulatory change tracking. - - - Threat attribution graphs, incident response timelines, security audit trails. - - - Policy decision trails, classified information handling, provenance chains. - - - Power grids, transportation safety, emergency response coordination. - - ## Why Semantica? - No vendor lock-in, no paywalled features. Every line of code is available and forkable. + No vendor lock-in. No paywalled features. + - Full source available on GitHub + - Every line auditable by your security team + - Fork, extend, and self-host with no restrictions + - No telemetry, no usage reporting - 1,000+ passing tests, `PipelineValidator`, `FailureHandler` with exponential backoff, 12 security fixes in v0.5.0. + Built for teams that can't afford surprises. + - 1,000+ passing tests with full regression coverage + - `PipelineValidator` catches configuration errors at startup + - `FailureHandler` with exponential backoff and dead-letter queues + - 12 security vulnerabilities fixed in v0.5.0 - Import only what you need. Use `NERExtractor` without a graph store. Every component is independently swappable. + Import only what you need. + - Use `NERExtractor` without a graph store + - Use `ContextGraph` without vector storage + - Every component independently swappable and testable + - No framework lock-in — works with any agent stack diff --git a/docs/installation.md b/docs/installation.md index ae12904e..af7fc75c 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -15,12 +15,13 @@ icon: "download" ## System Requirements | Component | Minimum | Recommended | -| --------- | ------- | ----------- | +| :--------- | :------- | :----------- | | Python | 3.8 | 3.11+ | | OS | Windows / Linux / Mac | Linux / Mac | | RAM | 4 GB | 16 GB+ | | Storage | 2 GB | 20 GB+ (models and data) | + ## Basic Installation ```bash @@ -39,6 +40,7 @@ pip install semantica[all] python -c "import semantica; print(semantica.__version__)" ``` + ## Virtual Environment (Recommended) @@ -59,6 +61,7 @@ python -c "import semantica; print(semantica.__version__)" + ## Optional Dependencies Install only what you need: @@ -94,6 +97,7 @@ Install only what you need: + ## Install from Source For the latest development version or to contribute: @@ -113,6 +117,7 @@ Install directly from the main branch if the PyPI release has issues: pip install git+https://github.com/semantica-agi/semantica.git@main ``` + ## Troubleshooting ### ModuleNotFoundError @@ -159,6 +164,7 @@ pip install --upgrade semantica Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe). This is a Windows system dependency, not a Semantica bug. + ## Next Steps diff --git a/docs/integrations/agno.md b/docs/integrations/agno.md index 370434cc..59a86999 100644 --- a/docs/integrations/agno.md +++ b/docs/integrations/agno.md @@ -6,7 +6,6 @@ icon: "robot" > Five drop-in components that bring Semantica's KG, vector memory, and decision intelligence into any Agno agent or team. ---- ## Installation @@ -22,223 +21,199 @@ pip install "semantica[agno,graph-falkordb]" pip install "semantica[agno,graph-neo4j,vectorstore-pgvector]" ``` ---- ## Components at a Glance -| Class | Agno Primitive | Semantica Backing | -|-------|---------------|-------------------| -| `AgnoContextStore` | `AgentMemory(db=…)` | `AgentContext` + `VectorStore` | -| `AgnoKnowledgeGraph` | `Agent(knowledge=…)` | `ContextGraph` + KG pipeline | -| `AgnoDecisionKit` | `Agent(tools=[…])` | `DecisionQuery`, `CausalChainAnalyzer`, `PolicyEngine` | -| `AgnoKGToolkit` | `Agent(tools=[…])` | `NERExtractor`, `RelationExtractor`, `Reasoner` | -| `AgnoSharedContext` | Team-level | Shared `ContextGraph` across agents | + + + `AgentMemory(db=…)` — Replaces Agno's flat storage with hybrid vector + context graph memory. Adds decision tracking and precedent search to any agent. + + + `Agent(knowledge=…)` — Documents flow through the full Semantica extraction pipeline into a queryable `ContextGraph` with multi-hop GraphRAG. + + + `Agent(tools=[…])` — 6 decision intelligence tools: record decisions, find precedents, trace causal chains, analyze impact, check policies, summarize history. + + + `Agent(tools=[…])` — 7 KG construction tools: extract entities, extract relations, add to graph, query graph, find related, infer facts, export subgraph. + + + Team-level — A single `ContextGraph` shared across all agents. Each agent gets a role-scoped view via `bind_agent()`. Writes are tagged by role. + + ---- -## 1. AgnoContextStore +## Component Details -Replaces Agno's flat conversation storage with a hybrid **vector + context graph** memory store. Implements `agno.memory.db.base.MemoryDb`. + + + Replaces Agno's flat conversation storage with a hybrid **vector + context graph** memory store. Implements `agno.memory.db.base.MemoryDb`. -```python -from agno.agent import Agent -from agno.memory import AgentMemory -from agno.models.openai import OpenAIChat + ```python + from agno.agent import Agent + from agno.memory import AgentMemory + from agno.models.openai import OpenAIChat + from semantica.context import ContextGraph + from semantica.vector_store import VectorStore + from integrations.agno import AgnoContextStore -from semantica.context import ContextGraph -from semantica.vector_store import VectorStore -from integrations.agno import AgnoContextStore + store = AgnoContextStore( + vector_store=VectorStore(backend="faiss"), + knowledge_graph=ContextGraph(advanced_analytics=True), + decision_tracking=True, + graph_expansion=True, + session_id="user_session_42", + ) -store = AgnoContextStore( - vector_store=VectorStore(backend="faiss"), - knowledge_graph=ContextGraph(advanced_analytics=True), - decision_tracking=True, - graph_expansion=True, - session_id="user_session_42", -) + agent = Agent( + model=OpenAIChat(id="gpt-4o"), + memory=AgentMemory(db=store), + description="A financially aware assistant with persistent decision intelligence.", + ) + ``` -agent = Agent( - model=OpenAIChat(id="gpt-4o"), - memory=AgentMemory(db=store), - description="A financially aware assistant with persistent decision intelligence.", -) + | Method | Description | + | :-------- | :------------- | + | `upsert_memory()` | Store text in `AgentContext` (vector index + graph node) | + | `read_memories()` | Hybrid retrieval: vector similarity + graph hop expansion | + | `record_decision()` | Record a structured decision with reasoning and outcome | + | `find_precedents()` | Return semantically similar historical decisions | + + + Gives Agno agents a queryable `ContextGraph` instead of a flat document store. Ingested documents pass through the full Semantica extraction pipeline. -agent.print_response("Recommend a portfolio allocation for a risk-averse investor.") -``` + ```python + from agno.agent import Agent + from agno.models.openai import OpenAIChat + from semantica.kg import GraphBuilder + from semantica.semantic_extract import NERExtractor, RelationExtractor + from integrations.agno import AgnoKnowledgeGraph -| Method | Description | -|--------|-------------| -| `upsert_memory()` | Store text in `AgentContext` (vector index + graph node) | -| `read_memories()` | Hybrid retrieval: vector similarity + graph hop expansion | -| `record_decision()` | Record a structured decision with reasoning and outcome | -| `find_precedents()` | Return semantically similar historical decisions | + kg = AgnoKnowledgeGraph( + graph_builder=GraphBuilder(), + ner_extractor=NERExtractor(), + relation_extractor=RelationExtractor(), + ) ---- + kg.load("regulatory_docs/", recursive=True) + kg.load(texts=["Basel IV capital requirements apply from January 2026."]) -## 2. AgnoKnowledgeGraph + agent = Agent(model=OpenAIChat(id="gpt-4o"), knowledge=kg, search_knowledge=True) + ``` -Gives Agno agents a queryable `ContextGraph` instead of a flat document store. Ingested documents pass through the full Semantica extraction pipeline. + **Ingestion:** `parse → NER → relation extract → graph build → vector index` -```python -from agno.agent import Agent -from agno.models.openai import OpenAIChat + **Search:** `vector retrieval → entity lookup → graph hop expansion → context injection` -from semantica.kg import GraphBuilder -from semantica.semantic_extract import NERExtractor, RelationExtractor -from integrations.agno import AgnoKnowledgeGraph + ```python + ctx = kg.get_graph_context("Basel IV") + # Returns a text summary of the entity's immediate neighbourhood + ``` + + + Exposes Semantica's decision intelligence as native Agno tools. -kg = AgnoKnowledgeGraph( - graph_builder=GraphBuilder(), - ner_extractor=NERExtractor(), - relation_extractor=RelationExtractor(), -) + ```python + from agno.agent import Agent + from agno.models.openai import OpenAIChat + from semantica.context import AgentContext + from integrations.agno import AgnoDecisionKit -kg.load("regulatory_docs/", recursive=True) -kg.load(texts=["Basel IV capital requirements apply from January 2026."]) + ctx = AgentContext(decision_tracking=True) + agent = Agent( + model=OpenAIChat(id="gpt-4o"), + tools=[AgnoDecisionKit(context=ctx)], + show_tool_calls=True, + ) + agent.print_response("Should we approve this mortgage application?") + ``` -agent = Agent( - model=OpenAIChat(id="gpt-4o"), - knowledge=kg, - search_knowledge=True, -) -``` + | Tool | Description | + | :------ | :------------- | + | `record_decision` | Record a decision with reasoning, outcome, and confidence | + | `find_precedents` | Search for similar past decisions | + | `trace_causal_chain` | Trace causal chain of a decision | + | `analyze_impact` | Assess downstream influence of a decision | + | `check_policy` | Validate decision against policy rules | + | `get_decision_summary` | Summarise decision history by category | + + + Lets agents actively build and query the context graph during reasoning. -**Ingestion pipeline:** -``` -parse → NER → relation extract → graph build → vector index -``` + ```python + from agno.agent import Agent + from agno.models.openai import OpenAIChat + from integrations.agno import AgnoKGToolkit -**Search (multi-hop GraphRAG):** -``` -vector retrieval → entity lookup → graph hop expansion → context injection -``` + agent = Agent( + model=OpenAIChat(id="gpt-4o"), + tools=[AgnoKGToolkit()], + show_tool_calls=True, + ) + ``` -```python -ctx = kg.get_graph_context("Basel IV") -# Returns a text summary of the entity's immediate neighbourhood -``` + | Tool | Description | + | :------ | :------------- | + | `extract_entities` | Extract named entities from text | + | `extract_relations` | Extract relationships between entities | + | `add_to_graph` | Add entities / relations to the context graph | + | `query_graph` | Query the graph (natural-language or Cypher) | + | `find_related` | Find concepts related to a given entity | + | `infer_facts` | Apply rules to infer new facts from the graph | + | `export_subgraph` | Export a subgraph as RDF / JSON-LD | + + + A single `ContextGraph` shared across an Agno `Team`. Each agent gets a role-scoped view via `bind_agent()`. Writes are tagged by role. ---- + ```python + from agno.agent import Agent + from agno.team import Team + from agno.models.openai import OpenAIChat + from semantica.context import ContextGraph + from semantica.vector_store import VectorStore + from integrations.agno import AgnoSharedContext, AgnoDecisionKit, AgnoKGToolkit -## 3. AgnoDecisionKit + shared = AgnoSharedContext( + vector_store=VectorStore(backend="faiss"), + knowledge_graph=ContextGraph(advanced_analytics=True), + decision_tracking=True, + ) -Exposes Semantica's decision intelligence as native Agno tools. + research_agent = Agent( + name="Researcher", + model=OpenAIChat(id="gpt-4o"), + memory=shared.bind_agent("researcher"), + tools=[AgnoKGToolkit(context=shared)], + ) + decision_agent = Agent( + name="Analyst", + model=OpenAIChat(id="gpt-4o"), + memory=shared.bind_agent("analyst"), + tools=[AgnoDecisionKit(context=shared)], + ) -```python -from agno.agent import Agent -from agno.models.openai import OpenAIChat + team = Team( + name="Research & Decision Team", + agents=[research_agent, decision_agent], + mode="coordinate", + ) + ``` -from semantica.context import AgentContext -from integrations.agno import AgnoDecisionKit + ```python + decision_id = shared.record_decision( + category="strategy", + scenario="Expand to EU market", + reasoning="Strong demand signals from Q1 survey", + outcome="approved", + confidence=0.87, + agent_role="cfo", + ) + precedents = shared.find_precedents("market expansion") + insights = shared.get_shared_insights() + ``` + + -ctx = AgentContext(decision_tracking=True) - -agent = Agent( - model=OpenAIChat(id="gpt-4o"), - tools=[AgnoDecisionKit(context=ctx)], - show_tool_calls=True, -) - -agent.print_response("Should we approve this mortgage application?") -``` - -| Tool | Description | -|------|-------------| -| `record_decision` | Record a decision with reasoning, outcome, and confidence | -| `find_precedents` | Search for similar past decisions | -| `trace_causal_chain` | Trace causal chain of a decision | -| `analyze_impact` | Assess downstream influence of a decision | -| `check_policy` | Validate decision against policy rules | -| `get_decision_summary` | Summarise decision history by category | - ---- - -## 4. AgnoKGToolkit - -Lets agents actively build and query the context graph during reasoning. - -```python -from agno.agent import Agent -from agno.models.openai import OpenAIChat -from integrations.agno import AgnoKGToolkit - -agent = Agent( - model=OpenAIChat(id="gpt-4o"), - tools=[AgnoKGToolkit()], - show_tool_calls=True, -) -``` - -| Tool | Description | -|------|-------------| -| `extract_entities` | Extract named entities from text | -| `extract_relations` | Extract relationships between entities | -| `add_to_graph` | Add entities / relations to the context graph | -| `query_graph` | Query the graph (natural-language or Cypher) | -| `find_related` | Find concepts related to a given entity | -| `infer_facts` | Apply rules to infer new facts from the graph | -| `export_subgraph` | Export a subgraph as RDF / JSON-LD | - ---- - -## 5. AgnoSharedContext - -A single `ContextGraph` shared across an Agno `Team`. Each agent gets a role-scoped view via `bind_agent()`. - -```python -from agno.agent import Agent -from agno.team import Team -from agno.models.openai import OpenAIChat - -from semantica.context import ContextGraph -from semantica.vector_store import VectorStore -from integrations.agno import AgnoSharedContext, AgnoDecisionKit, AgnoKGToolkit - -shared = AgnoSharedContext( - vector_store=VectorStore(backend="faiss"), - knowledge_graph=ContextGraph(advanced_analytics=True), - decision_tracking=True, -) - -research_agent = Agent( - name="Researcher", - model=OpenAIChat(id="gpt-4o"), - memory=shared.bind_agent("researcher"), - tools=[AgnoKGToolkit(context=shared)], -) - -decision_agent = Agent( - name="Analyst", - model=OpenAIChat(id="gpt-4o"), - memory=shared.bind_agent("analyst"), - tools=[AgnoDecisionKit(context=shared)], -) - -team = Team( - name="Research & Decision Team", - agents=[research_agent, decision_agent], - mode="coordinate", -) -``` - -```python -# Record a team-level decision -decision_id = shared.record_decision( - category="strategy", - scenario="Expand to EU market", - reasoning="Strong demand signals from Q1 survey", - outcome="approved", - confidence=0.87, - agent_role="cfo", -) - -precedents = shared.find_precedents("market expansion") -insights = shared.get_shared_insights() -``` - -Memories written by one agent are immediately visible to all other agents in the team. Each agent's writes are tagged with their role for independent filtering. - ---- ## API Reference @@ -255,7 +230,6 @@ from integrations.agno import ( All five classes are usable without `agno` installed — they carry the full Semantica API and degrade gracefully. ---- ## See Also diff --git a/docs/integrations/docling.md b/docs/integrations/docling.md index b51e78aa..fb28e14d 100644 --- a/docs/integrations/docling.md +++ b/docs/integrations/docling.md @@ -6,11 +6,10 @@ icon: "file-lines" > Parse complex documents — PDFs, DOCX, PPTX, HTML — with high-fidelity table extraction and built-in OCR. ---- ## Overview -Docling is integrated into Semantica's `parse` module via the `DoclingParser`. Documents pass through Docling's layout engine, then feed directly into Semantica's extraction and KG pipeline. +Docling is integrated into Semantica's `parse` module via the **`DoclingParser`**. Documents pass through Docling's **layout engine**, then feed directly into Semantica's extraction and KG pipeline. @@ -27,7 +26,6 @@ Docling is integrated into Semantica's `parse` module via the `DoclingParser`. D ---- ## Installation @@ -38,7 +36,6 @@ pip install semantica pip install docling ``` ---- ## Basic Usage @@ -52,7 +49,6 @@ print(result["full_text"][:200]) print(f"Found {len(result['tables'])} tables") ``` ---- ## Full Example @@ -81,16 +77,14 @@ print(f"Title: {metadata.get('title')}") print(f"Pages: {result.get('total_pages')}") ``` ---- ## DoclingParser Parameters | Parameter | Default | Description | -|-----------|---------|-------------| +| :----------- | :--------- | :------------- | | `enable_ocr` | `False` | Enable OCR for scanned pages | | `export_format` | `"markdown"` | Output format: `"markdown"` or `"text"` | ---- ## Parsed Result Structure @@ -103,7 +97,6 @@ print(f"Pages: {result.get('total_pages')}") } ``` ---- ## See Also diff --git a/docs/integrations/snowflake.md b/docs/integrations/snowflake.md index 0a135427..153dd604 100644 --- a/docs/integrations/snowflake.md +++ b/docs/integrations/snowflake.md @@ -6,7 +6,6 @@ icon: "snowflake" > Extract data from Snowflake into Semantica with password, key-pair, OAuth, and SSO authentication. ---- ## Installation @@ -18,7 +17,6 @@ pip install "semantica[db-snowflake]" pip install snowflake-connector-python ``` ---- ## Basic Usage @@ -43,7 +41,6 @@ print(f"Retrieved {data.row_count} rows — columns: {data.columns}") Use environment variables (or a `.env` file with `python-dotenv`) to keep credentials out of source code. `SnowflakeIngestor()` with no arguments reads from `SNOWFLAKE_*` environment variables automatically. ---- ## Authentication Methods @@ -92,7 +89,6 @@ Use environment variables (or a `.env` file with `python-dotenv`) to keep creden ---- ## Querying @@ -126,7 +122,6 @@ for column in schema["columns"]: print(f"{column['name']}: {column['type']}") ``` ---- ## Export as Semantica Documents @@ -139,7 +134,6 @@ documents = ingestor.export_as_documents( print(f"Created {len(documents)} documents for processing") ``` ---- ## Batch Processing Large Tables @@ -163,7 +157,6 @@ data = ingestor.ingest_query( ) ``` ---- ## Troubleshooting @@ -175,7 +168,6 @@ if not connector.test_connection(): print("Connection failed — check credentials and account identifier") ``` ---- ## See Also diff --git a/docs/learning-more.md b/docs/learning-more.md index b884bbb3..1eb34d65 100644 --- a/docs/learning-more.md +++ b/docs/learning-more.md @@ -6,6 +6,7 @@ icon: "graduation-cap" Whether you're running your first pipeline or deploying Semantica in production, this page gives you a structured path forward — from beginner to enterprise-grade usage. + ## Learning Paths @@ -23,37 +24,79 @@ Whether you're running your first pipeline or deploying Semantica in production, -### Beginner Path + + + New to Semantica and knowledge graphs. No prior graph database experience required. -1. [Installation Guide](installation) — set up your environment -2. [Core Concepts](concepts) — understand KGs, embeddings, and extraction -3. [Getting Started](getting-started) — first working example -4. [Quickstart Tutorial](quickstart) — build your first knowledge graph -5. [Welcome to Semantica notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb) — interactive introduction to all modules + + + [Installation Guide](installation) — virtual environments, optional extras, platform-specific fixes. + + + [Core Concepts](concepts) — what knowledge graphs are, how embeddings work, what extraction does. + + + [Getting Started](getting-started) — 5-minute code walkthrough with pattern-based extraction (no API key needed). + + + [Quickstart Tutorial](quickstart) — full 6-step pipeline from ingestion to visualization. + + + [Welcome to Semantica notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb) — Jupyter walkthrough of every module. + + + + + Comfortable with the basics, building real applications. Assumes you've completed the Beginner path. -### Intermediate Path + + + [Modules Guide](modules) — all 27 modules with code examples and common pipeline chains. + + + [Building Knowledge Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) — multi-source, deduplication, conflict resolution. + + + [Embeddings notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb) — providers, pooling strategies, vector stores. + + + [GraphRAG Complete notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — hybrid retrieval, reasoning, source attribution. + + + [Multi-Source Data Integration notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb) and [Use Cases](use-cases) for domain-specific patterns. + + + + + Enterprise deployments, customization, and extension. Assumes production usage experience. -1. [Modules Guide](modules) — every module with code examples -2. [Building Knowledge Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) -3. [Embeddings notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb) -4. [GraphRAG Complete notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) -5. [Multi-Source Data Integration notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb) -6. [Use Cases](use-cases) — domain-specific examples with notebooks + + + [Architecture Guide](architecture) — four-layer design, extension points, and design decisions. + + + [Temporal Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb) — `valid_from`/`valid_until`, Allen interval algebra, point-in-time queries. + + + [Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb) — auto-generation, SHACL validation, Ontology Hub (v0.5.0). + + + [Complete Visualization Suite notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb) — UMAP, t-SNE, community layouts, embedding projections. + + + [Multi-Format Export notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb) — RDF with PROV-O, Parquet, Neo4j Cypher, Arrow, OWL. + + + + -### Advanced Path - -1. [Architecture Guide](architecture) — three-layer system, extension points, design decisions -2. [Temporal Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb) — v0.4.0 temporal intelligence -3. [Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb) — v0.5.0 Ontology Hub -4. [Complete Visualization Suite notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb) -5. [Multi-Format Export notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb) ## Configuration Reference All settings can be overridden with environment variables — no code changes needed. | Setting | Environment Variable | Default | -| ------- | -------------------- | ------- | +| :------- | :-------------------- | :------- | | OpenAI API Key | `OPENAI_API_KEY` | `None` | | Groq API Key | `GROQ_API_KEY` | `None` | | Anthropic API Key | `ANTHROPIC_API_KEY` | `None` | @@ -62,6 +105,7 @@ All settings can be overridden with environment variables — no code changes ne | Log Level | `SEMANTICA_LOG_LEVEL` | `"INFO"` | | Log Format | `SEMANTICA_LOG_FORMAT` | `"text"` | + ## Troubleshooting ### `ModuleNotFoundError: No module named 'semantica'` @@ -136,12 +180,13 @@ Fixed in **v0.5.0**. For earlier versions, pass encoding explicitly or set the e set PYTHONIOENCODING=utf-8 ``` + ## Performance Optimization ### Backend Selection | Operation | NetworkX (default) | Neo4j / FalkorDB | -| --------- | ------------------ | ---------------- | +| :--------- | :------------------ | :---------------- | | Graph construction | Fast | Moderate | | Query performance | Moderate | Fast | | Scalability | Low — in-memory only | High — persistent | @@ -162,6 +207,7 @@ resolver = EntityResolver() merged = resolver.resolve(entities, strategy="semantic_v2") # up to 7x faster ``` + ## Security Best Practices - **API keys** — store in environment variables or a secrets manager; never commit them to version control; rotate on a schedule diff --git a/docs/modules.md b/docs/modules.md index 9c9d064c..54eb0d66 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -33,6 +33,7 @@ Semantica is organized into **27 modules** across six logical layers. Each modul + ## Input Layer ### Ingest @@ -106,6 +107,7 @@ standardized_date = normalize_date("Jan 1st, 2020") **Normalizers available:** text cleaning, entity canonicalization, date normalization, number normalization, encoding handling, language detection + ## Core Processing ### Semantic Extract @@ -189,6 +191,7 @@ results = datalog.query("ancestor(alice, ?)") **Engines:** forward chaining, Rete network, deductive, abductive, SPARQL, Datalog — all produce explainable inference paths + ## Storage ### Embeddings @@ -252,6 +255,7 @@ results = store.sparql("SELECT ?s ?p ?o WHERE { ?s ?p ?o }") **Backends:** Blazegraph, Apache Jena, RDF4J + ## Quality Assurance ### Deduplication @@ -287,6 +291,7 @@ resolved = detector.resolve(conflicts, strategy="most_recent") **Resolution strategies:** prefer most recent, prefer most reliable source, majority vote, flag for manual review + ## Context & Memory ### Context @@ -345,6 +350,7 @@ diff = manager.diff("v1.0", "v1.1") **Components:** `TemporalVersionManager`, `ChangeLog`, `OntologyVersionManager`, `VersionStorage` + ## Output & Orchestration ### Export @@ -409,6 +415,7 @@ FastAPI Knowledge Explorer with Ontology Hub, WebSocket progress, bidirectional **Routes:** graph, ontology, provenance, decisions, analytics, SPARQL, temporal, annotations, export/import, vocabulary + ## Utilities ### LLM Providers @@ -513,22 +520,170 @@ from semantica.utils import helpers, validators, logging **Components:** `helpers`, `validators`, `constants`, `types`, `exceptions`, `logging`, `ProgressTracker` + ## Common Module Chains -| Goal | Pipeline | -| ---- | -------- | -| Document processing | Ingest → Parse → Split → Semantic Extract → KG | -| Web scraping | Ingest (Web) → Normalize → Semantic Extract → Graph Store | -| GraphRAG | KG + Vector Store → Context → Reasoning → Export | -| AI agents | Context → LLM Providers → Reasoning → Export | -| Temporal analysis | KG (Temporal) → Context → Change Management → Export | -| Compliance pipeline | Ingest → Semantic Extract → KG → Provenance → Export | -| Evaluation workflow | Ingest → Parse → Semantic Extract → Evals | + + + Load documents from any source and turn them into a queryable knowledge graph. + + **Pipeline:** `Ingest` → `Parse` → `Normalize` → `Semantic Extract` → `GraphBuilder` → `KG` + +```python +from semantica.ingest import FileIngestor +from semantica.parse import DocumentParser +from semantica.semantic_extract import NERExtractor, RelationExtractor +from semantica.kg import GraphBuilder + +sources = FileIngestor().ingest("data/") +parsed = DocumentParser().parse(sources[0]) +entities = NERExtractor(method="llm", llm_provider=llm).extract(parsed) +relationships = RelationExtractor(method="llm", llm_provider=llm).extract(parsed, entities=entities) +graph = GraphBuilder(merge_entities=True).build( + entities=entities, relationships=relationships + ) +``` + + **Best for:** research pipelines, enterprise data extraction, document intelligence + + + + Ground every LLM response in a knowledge graph — structured retrieval with source attribution. + + **Pipeline:** `KG` + `VectorStore` → `AgentContext` → GraphRAG query → grounded answer + +```python +from semantica.context import AgentContext, ContextGraph +from semantica.vector_store import VectorStore + +context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph(advanced_analytics=True), +) +context.load_graph("company_kg.json") + +result = context.query( + "What companies did Apple alumni found?", + mode="graphrag", + reasoning=True, +) +for claim in result.claims: + print(f"{claim.text} → {claim.source_node}") +``` + + **Best for:** question-answering systems, RAG with source attribution, research assistants + + + + Give your agent persistent memory, decision tracking, and policy enforcement. + + **Pipeline:** `AgentContext` → decision recording → precedent search → policy check → causal analysis + +```python +from semantica.context import AgentContext, ContextGraph +from semantica.vector_store import VectorStore + +context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph(advanced_analytics=True), + decision_tracking=True, +) +context.store("GPT-4 outperforms GPT-3.5 on reasoning by 40%") + +decision_id = context.record_decision( + category="model_selection", + scenario="Choose LLM for production", + reasoning="Benchmark advantage justifies cost", + outcome="selected_gpt4", + confidence=0.91, +) +precedents = context.find_precedents("model selection", limit=5) +``` + + **Best for:** autonomous agents, AI copilots, decision-support systems + + + + Full provenance from raw data to final inference — W3C PROV-O, SHA-256 checksums, audit trail. + + **Pipeline:** `Ingest` → `Parse` → `Extract` → `KG` → `Provenance` → `ChangeManagement` → `Export` + +```python +from semantica.ingest import FileIngestor +from semantica.semantic_extract import NERExtractor +from semantica.kg import GraphBuilder +from semantica.provenance import ProvenanceManager +from semantica.export import RDFExporter + +sources = FileIngestor().ingest("records/") +entities = NERExtractor(method="llm", llm_provider=llm).extract(sources) +graph = GraphBuilder(merge_entities=True).build(entities=entities, relationships=[]) +prov = ProvenanceManager() +lineage = prov.get_entity_lineage("entity_id") + +RDFExporter(include_provenance=True).export_to_rdf(graph, format="turtle", output="audit.ttl") +``` + + **Best for:** HIPAA, SOX, GDPR, FDA 21 CFR Part 11 deployments + + + + Crawl websites, normalize text, and extract knowledge directly from the web. + + **Pipeline:** `WebIngestor` → `Normalize` → `Semantic Extract` → `GraphStore` + +```python +from semantica.ingest import WebIngestor +from semantica.normalize import TextNormalizer +from semantica.semantic_extract import NERExtractor, RelationExtractor +from semantica.graph_store import Neo4jStore + +pages = WebIngestor(max_depth=2).ingest("https://example.com") +normalizer = TextNormalizer() +store = Neo4jStore(uri="bolt://localhost:7687", user="neo4j", password="password") + +for page in pages: + text = normalizer.normalize_text(page.text) + entities = NERExtractor().extract(text) + relationships = RelationExtractor().extract(text, entities=entities) + store.add_nodes(entities) + store.add_edges(relationships) +``` + + **Best for:** competitive intelligence, news monitoring, research aggregation + + + + Track how facts change over time — point-in-time queries, snapshots, and versioning. + + **Pipeline:** `KG (Temporal)` → `TemporalGraphQuery` → `VersionManager` → `ChangeManagement` + +```python +from semantica.kg import GraphBuilder, TemporalGraphQuery, TemporalVersionManager + +builder = GraphBuilder() +kg = builder.build(sources=[{ + "entities": [{"id": "alice", "type": "Person"}], + "relationships": [{"source": "alice", "target": "acme", "type": "ceo_of", + "valid_from": "2020-01-01", "valid_until": "2023-06-01"}] +}]) + +query = TemporalGraphQuery() +snapshot_2021 = query.reconstruct_at_time(kg, "2021-06-15") + +versioner = TemporalVersionManager() +versioner.create_snapshot(kg, "2024-Q1", author="user@example.com", description="Q1 snapshot") +``` + + **Best for:** financial history, regulatory timelines, organizational change tracking + + + ## Module Index | Module | Purpose | Key Classes | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | [ingest](reference/ingest) | Data ingestion | `FileIngestor`, `WebIngestor`, `ParquetIngestor`, `XMLIngestor` | | [parse](reference/parse) | Document parsing | `DocumentParser`, `DoclingParser` | | [split](reference/split) | Text chunking | `TextSplitter` | diff --git a/docs/project-license.md b/docs/project-license.md index bad87bef..12d83c54 100644 --- a/docs/project-license.md +++ b/docs/project-license.md @@ -6,7 +6,6 @@ icon: "file-contract" > Semantica is open source and free for everyone under the MIT License. ---- ## MIT License @@ -34,51 +33,46 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` ---- ## What This Means **You can:** -- Use Semantica commercially — free for business use -- Modify the source code -- Distribute copies or derivatives -- Use in proprietary software +Use Semantica commercially — free for business use +Modify the source code +Distribute copies or derivatives +Use in proprietary software **You must:** - Keep the copyright notice in copies - Include the MIT license text in distributions -**No warranty** — the authors are not responsible for damages or liable for how you use the software. +**No warranty** — the authors are not responsible for damages or liable for how you use the software. ---- ## Commercial Use Semantica is completely free for commercial use. No attribution is required (though it's appreciated). ---- ## Third-Party Dependencies Semantica uses open-source libraries with compatible licenses: | Library | License | -|---------|---------| +| :--------- | :--------- | | Python | PSF License | | NumPy, Pandas | BSD | | spaCy | MIT | | Transformers | Apache 2.0 | | RDFLib | BSD | ---- ## Contributing By contributing to Semantica, you agree that your contributions will be licensed under the same MIT License. ---- ## See Also diff --git a/docs/quickstart.md b/docs/quickstart.md index dfb11686..94baaad0 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -10,6 +10,7 @@ icon: "rocket" This guide walks you through the end-to-end pipeline for building your first knowledge graph. Start here after installation. An LLM API key is optional — pattern-based extraction works out of the box. + ## Install @@ -37,6 +38,7 @@ python -c "import semantica; print(semantica.__version__)" # 0.5.0 ``` + ## Full Pipeline Semantica end-to-end pipeline: Ingest → Parse → Normalize → Extract → Build KG → QA → Store → Deliver @@ -220,6 +222,7 @@ aql = exporter.export(graph) + ## Add Decision Intelligence Track every agent decision with full causal chains and provenance — one extra import: @@ -251,6 +254,7 @@ precedents = context.find_precedents("model selection reasoning", limit=5) influence = context.analyze_decision_influence(decision_id) ``` + ## Common Patterns @@ -353,6 +357,7 @@ print(node.provenance) + ## Troubleshooting @@ -412,6 +417,7 @@ pip install --upgrade semantica + ## Next Steps diff --git a/docs/reference/change_management.md b/docs/reference/change_management.md index fa6cd8c1..86f470ab 100644 --- a/docs/reference/change_management.md +++ b/docs/reference/change_management.md @@ -4,16 +4,23 @@ description: "Version control, SHA-256 checksums, diff analysis, rollback, and a icon: "clock-rotate-left" --- -`semantica.change_management` provides enterprise-grade versioning and audit trails for knowledge graphs and ontologies. Every snapshot carries a SHA-256 checksum, every modification is logged, and every state can be diffed or rolled back — giving you a complete, tamper-evident record suitable for regulated industries. +**`semantica.change_management`** provides **enterprise-grade versioning and audit trails** for knowledge graphs and ontologies: + +- SHA-256 checksums on every snapshot — tamper detection without external infrastructure +- Structural diff between any two versions: nodes added, removed, or modified +- Full rollback to any named snapshot +- Per-entity mutation history for audit trail queries +- Compliance frameworks supported: HIPAA, SOX, GDPR, FDA 21 CFR Part 11 Compliance frameworks supported out of the box: **HIPAA**, **SOX**, **GDPR**, and **FDA 21 CFR Part 11**. + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `TemporalVersionManager` | Snapshot, diff, rollback, and per-node mutation history for KGs | | `OntologyVersionManager` | Schema versioning with structural diff support | | `InMemoryVersionStorage` | Fast in-memory storage for dev and testing — no persistence | @@ -97,7 +104,7 @@ Version control for knowledge graphs — snapshot, diff, and rollback. ### Constructor Parameters | Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | +| :--------- | :---- | :------- | :----------- | | `storage_path` | `str` | `None` | Path to SQLite database; uses in-memory if omitted | ### List and Retrieve @@ -115,7 +122,7 @@ snapshot = manager.get_version("v1.0") ### TemporalVersionManager Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `create_snapshot(graph, version_label, author, description)` | `Dict[str, Any]` | Create a version snapshot; returns the full snapshot dict including `checksum` | | `get_version(label)` | `Optional[Dict[str, Any]]` | Retrieve a snapshot dict for a specific version label | | `list_versions()` | `List[Dict[str, Any]]` | List all version metadata dicts | diff --git a/docs/reference/conflicts.md b/docs/reference/conflicts.md index f7ecd11e..3fef582f 100644 --- a/docs/reference/conflicts.md +++ b/docs/reference/conflicts.md @@ -4,7 +4,14 @@ description: "Multi-source conflict detection and resolution — value, type, te icon: "triangle-exclamation" --- -`semantica.conflicts` detects and resolves contradictions when multiple sources disagree on the same fact. It surfaces five conflict types, seven resolution strategies, and generates investigation guides for manual review — so conflicts never silently corrupt your knowledge graph. +**`semantica.conflicts`** detects and resolves **contradictions when multiple sources disagree** on the same fact: + +- Five conflict types: value, type, temporal, logical, and relationship +- Seven resolution strategies: voting, credibility-weighted, most-recent, first-seen, highest-confidence, manual review, expert review +- `InvestigationGuideGenerator` produces step-by-step investigation instructions for manual resolution +- `SourceTracker` maps each property value to its contributing source for full attribution +- Conflicts are surfaced explicitly — never silently corrupting the knowledge graph + ## Why Detect Conflicts? @@ -21,7 +28,7 @@ Semantica's conflict detection makes disagreements explicit and actionable: ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `ConflictDetector` | Detects value, type, and relationship conflicts across entity lists | | `ConflictResolver` | Resolves conflicts with configurable strategy: `voting`, `credibility_weighted`, `most_recent`, `first_seen`, `highest_confidence`, `manual_review`, `expert_review` | | `ConflictType` | Enum: `VALUE_CONFLICT`, `TYPE_CONFLICT`, `TEMPORAL_CONFLICT`, `LOGICAL_CONFLICT`, `RELATIONSHIP_CONFLICT` | @@ -146,7 +153,7 @@ conflicts = detector.detect_value_conflicts(entities, "revenue") ### Detection Types | Type | What It Detects | Example | -| ---- | --------------- | ------- | +| :---- | :--------------- | :------- | | `VALUE` | Same entity, same property, different values across sources | Revenue $391B vs $383B | | `TYPE` | Same entity classified as different types | "Python" as Language vs Snake | | `TEMPORAL` | Conflicting timestamps or validity windows | CEO at two companies simultaneously | @@ -172,7 +179,7 @@ all_conflicts = detector.detect_entity_conflicts(entities) ### ConflictDetector Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `detect_value_conflicts(entities, property_name, entity_type=None)` | `List[Conflict]` | Detect value disagreements on a specific property across entity instances | | `detect_type_conflicts(entities)` | `List[Conflict]` | Detect type classification conflicts | | `detect_relationship_conflicts(relationships)` | `List[Conflict]` | Detect relationship property conflicts (takes a list of relationship dicts) | @@ -213,7 +220,7 @@ for result in results: ) ``` - Best for: sources with known reliability rankings (SEC > blog). + **Best for:** sources with known reliability rankings (SEC > blog). Majority vote — most common value across sources wins: @@ -222,7 +229,7 @@ for result in results: results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING) ``` - Best for: 3+ sources with roughly equal credibility. When all sources have identical credibility scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`. + **Best for:** 3+ sources with roughly equal credibility. When all sources have identical credibility scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`. ```python @@ -246,12 +253,12 @@ for result in results: print(" [%d] %s" % (step.step_number, step.description)) ``` - Best for: high-stakes decisions (`severity == "critical"`), regulated data (HIPAA/SOX), and domain-specific ambiguity. + **Best for:** high-stakes decisions (`severity == "critical"`), regulated data (HIPAA/SOX), and domain-specific ambiguity. | Strategy | Enum | When to Use | - | -------- | ---- | ----------- | + | :-------- | :---- | :----------- | | Majority vote | `VOTING` | 3+ sources with roughly equal credibility | | Credibility-weighted | `CREDIBILITY_WEIGHTED` | Sources have different authority levels | | Most recent | `MOST_RECENT` | Fast-changing facts: stock price, headcount, status | diff --git a/docs/reference/context.md b/docs/reference/context.md index d2be4cd7..4d6413e5 100644 --- a/docs/reference/context.md +++ b/docs/reference/context.md @@ -4,12 +4,19 @@ description: "Agent context graphs, decision tracking, causal chains, precedent icon: "brain" --- -`semantica.context` is the memory and decision layer for AI agents. It stores facts with provenance, records decisions as first-class objects with full causal chains, lets agents search their own history to stay consistent across runs, and answers complex queries by traversing the knowledge graph. +`semantica.context` is the memory and decision layer for AI agents: + +- Stores facts with provenance and embedding-backed retrieval +- Records decisions as first-class graph objects with full causal chains +- Lets agents search their own history to stay consistent across runs +- Answers complex queries via multi-hop GraphRAG traversal +- Enforces versioned policies and tracks compliance exceptions + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `AgentContext` | Primary entry point — memory, retrieval, decisions, graph traversal, checkpoints | | `ContextGraph` | In-memory knowledge graph with centrality, community detection, and decision tracking | | `AgentMemory` | Vector-backed persistent memory: `store(text)`, `retrieve(query, max_results)` | @@ -19,35 +26,53 @@ icon: "brain" | `PolicyEngine` | Policy management: `add_policy()`, `check_compliance()`, `get_applicable_policies()` | | `CausalChainAnalyzer` | Trace how decisions influenced each other: `get_causal_chain(decision_id)` | + ## What You Get - Unified interface for memory, decision tracking, graph-backed retrieval, conversation history, checkpoints, and persistence. + - Memory, decision tracking, and graph-backed retrieval behind one API + - Conversation history and checkpoint diffing + - Persist and restore full context state to disk - Thread-safe in-memory knowledge graph with centrality analysis, community detection, temporal validity, cross-graph links, and decision management. + - Thread-safe in-memory knowledge graph + - PageRank, centrality, community detection, temporal validity + - Cross-graph navigation and link traversal - Embedding-backed memory with retention policy and LRU eviction. + - Embedding-backed memory with retention policy + - LRU eviction at configurable `max_memory_size` + - Per-conversation history isolation - Records decisions with causal chains, confidence scores, temporal validity windows, and cross-system context capture. + - Records decisions with causal chains and confidence scores + - Temporal validity windows (`valid_from` / `valid_until`) + - Cross-system context capture on every decision - Manages policy versions, checks compliance for recorded decisions, and tracks policy exceptions in the graph. + - Versioned policy storage in the knowledge graph + - Compliance checking against recorded decisions + - Policy exception tracking with approver audit trail - Maps entity text to URIs and creates typed links between entity IDs — prevents "Apple", "Apple Inc.", and "AAPL" from becoming three separate nodes. + - Maps entity text to stable URIs + - Creates typed links between entity IDs + - Prevents "Apple", "Apple Inc.", "AAPL" becoming separate nodes - Hybrid retrieval fusing vector similarity, graph traversal, and agent memory for richer context than pure vector search. + - Fuses vector similarity, graph traversal, and agent memory + - Richer context than pure vector search + - Configurable `hybrid_alpha` and expansion hops - Traces upstream causes and downstream effects of any decision through the knowledge graph. + - Traces upstream causes and downstream effects of any decision + - Explainability paths with relationship types + - Configurable depth and direction + ## Getting Started ```python @@ -144,14 +169,149 @@ decision_id = context.record_decision( + +## Usage Patterns + + + + Fastest setup — no knowledge graph. Best for agents that need semantic search over facts without graph traversal overhead. + + ```python + from semantica.context import AgentContext + from semantica.vector_store import VectorStore + + # Zero-graph setup — vector memory only + context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + ) + + context.store("User prefers concise responses with code examples") + context.store("Project uses Python 3.11 with FastAPI and PostgreSQL") + + results = context.retrieve("user coding preferences", max_results=5) + for r in results: + print("{:.3f} {}".format(r["score"], r["content"])) + ``` + + + Swap `backend="faiss"` to `backend="inmemory"` for zero-dependency local development. + + + + Production setup — graph + decisions + analytics. Use when you need explainability and contradiction-free decision history. + + ```python + from semantica.context import AgentContext, ContextGraph + from semantica.vector_store import VectorStore + + context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph( + advanced_analytics=True, # PageRank, centrality, community detection + kg_algorithms=True, # path-finding, link prediction + ), + decision_tracking=True, # requires knowledge_graph + retention_days=90, + max_memories=50000, + ) + + decision_id = context.record_decision( + category="model_selection", + scenario="Choose LLM for production reasoning pipeline", + reasoning="GPT-4 benchmark advantage justifies 3x cost", + outcome="selected_gpt4", + confidence=0.91, + entities=["gpt-4", "gpt-3.5"], + ) + + # Prevent contradictions across runs + precedents = context.find_precedents("model selection", limit=5) + ``` + + + `decision_tracking=True` silently no-ops unless `knowledge_graph` is also provided at construction time. + + + + Load a pre-built knowledge graph and answer complex questions with multi-hop graph traversal. + + ```python + from semantica.context import AgentContext, ContextGraph + from semantica.vector_store import VectorStore + + context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph(advanced_analytics=True), + hybrid_alpha=0.4, # 0.0 = pure vector → 1.0 = pure graph + max_expansion_hops=3, + ) + + # Load a pre-built knowledge graph + context.load_graph("company_kg.json") + + # Multi-hop GraphRAG retrieval + results = context.retrieve( + "companies founded by Apple alumni", + use_graph=True, + max_results=10, + ) + for r in results: + print("[{:.3f}] {}".format(r["score"], r["content"])) + ``` + + + Increase `max_expansion_hops` for deeper traversal at the cost of latency. Start at 2 and tune upward. + + + + Add versioned compliance policies and gate every decision against them before recording. + + ```python + from semantica.context import AgentContext, ContextGraph, PolicyEngine + from semantica.vector_store import VectorStore + + context = AgentContext( + vector_store=VectorStore(backend="faiss", dimension=768), + knowledge_graph=ContextGraph(), + decision_tracking=True, + ) + + engine = PolicyEngine(knowledge_graph=context.knowledge_graph) + + engine.add_policy( + name="data_privacy", + description="No PII stored without user consent flag", + version="1.2", + effective_date="2024-01-01", + category="privacy", + rules={"requires_consent": True, "max_retention_days": 90}, + ) + + decision_data = {"action": "store_user_email", "user_consent": True} + result = engine.check_compliance(decision_data, policy_names=["data_privacy"]) + + if result["compliant"]: + context.record_decision( + category="data_storage", + scenario="Store user profile", + outcome="stored", + confidence=1.0, + ) + else: + print("Blocked by policy:", result["violations"]) + ``` + + + + ## AgentContext -The main entry point. Wraps memory, graph, and decision tracking behind a single API. +**`AgentContext`** is the main entry point. Wraps memory, graph, and decision tracking behind a **single unified API**. ### Constructor Parameters | Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | +| :--------- | :---- | :------- | :----------- | | `vector_store` | `VectorStore` | **required** | Backend for embedding-based memory retrieval | | `knowledge_graph` | `ContextGraph` | `None` | Enables graph-backed relationships and GraphRAG | | `decision_tracking` | `bool` | `False` | Activates `DecisionRecorder` — requires `knowledge_graph` to also be set | @@ -170,7 +330,7 @@ The main entry point. Wraps memory, graph, and decision tracking behind a single ### Memory Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `store(content, metadata, conversation_id, user_id)` | `str` or `Dict` | Store a fact (str → memory ID) or list of documents (list → stats dict) | | `batch_store(items)` | `List[str]` | Store multiple items at once — returns list of memory IDs | | `retrieve(query, max_results, min_score, use_graph, conversation_id)` | `List[Dict]` | Semantic retrieval; auto-selects GraphRAG if `knowledge_graph` is set | @@ -206,7 +366,7 @@ results = context.retrieve( ### Multi-Hop GraphRAG -Requires `knowledge_graph` to be set at construction: +**Requires `knowledge_graph`** to be set at construction — enables `query_with_reasoning()` for LLM-grounded multi-hop traversal: ```python import os @@ -228,7 +388,7 @@ print("Sources used: {}".format(result["num_sources"])) ### Decision Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `record_decision(category, scenario, reasoning, outcome, confidence, entities, decision_maker, valid_from, valid_until)` | `str` | Record a decision; raises `RuntimeError` if `decision_tracking=False` or no `knowledge_graph` | | `find_precedents(scenario, category, limit, use_hybrid_search, max_hops, as_of)` | `List[Decision]` | Find similar past decisions by semantic + structural similarity | | `query_decisions(query, max_hops, use_hybrid_search)` | `List[Decision]` | Broad context-aware decision search | @@ -238,7 +398,7 @@ print("Sources used: {}".format(result["num_sources"])) ### Checkpoint Methods -Useful for detecting what changed across reasoning runs: +**Ideal for auditing reasoning loops** — take a snapshot before and after a pass to see exactly what changed: ```python # Take a named snapshot of the current graph state @@ -257,9 +417,10 @@ print("Relationships added: {}".format(len(diff["relationships_added"]))) context.flush_checkpoint("after_inference") ``` + ## ContextGraph -The knowledge graph backing `AgentContext`. Can also be used standalone for relationship modelling. +**`ContextGraph`** is the knowledge graph backing `AgentContext`. Can also be used **standalone** for relationship modelling without the full context layer. ```python from semantica.context import ContextGraph @@ -289,7 +450,7 @@ print("Nodes: {}, Edges: {}".format(stats["node_count"], stats["edge_count"])) ### Constructor Options | Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | +| :--------- | :---- | :------- | :----------- | | `advanced_analytics` | `bool` | `True` | PageRank, betweenness centrality | | `centrality_analysis` | `bool` | `True` | Full centrality suite | | `community_detection` | `bool` | `True` | Louvain community clustering | @@ -298,7 +459,7 @@ print("Nodes: {}, Edges: {}".format(stats["node_count"], stats["edge_count"])) ### ContextGraph — Full Method Reference | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `add_node(node_id, node_type, properties, valid_from, valid_until)` | `None` | Add a node; supports temporal validity windows | | `add_edge(source_id, target_id, edge_type, weight, properties)` | `None` | Add a directed edge with optional weight | | `add_nodes(nodes)` | `int` | Bulk-add from a list of dicts; returns count added | @@ -353,6 +514,7 @@ path = domain_graph.cross_graph_path( print("Reachable: {}, hops: {}".format(path["reachable"], path["hop_count"])) ``` + ## AgentMemory (Low-Level) For fine-grained control over memory storage and retrieval: @@ -385,11 +547,12 @@ history = memory.get_conversation_history(conversation_id="conv_001", max_items= ``` | Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | +| :--------- | :---- | :------- | :----------- | | `vector_store` | `VectorStore` | **required** | Embedding backend for semantic retrieval | | `max_memory_size` | `int` | `10000` | Max items before LRU eviction | | `retention_policy` | `str` | `"unlimited"` | `"N_days"` (e.g. `"30_days"`) or `"unlimited"` | + ## PolicyEngine `PolicyEngine` manages versioned policies stored in the knowledge graph. Policies are stored as nodes and can be linked to decisions: @@ -436,6 +599,7 @@ for p in policies: print("{} v{}".format(p.name, p.version)) ``` + ## EntityLinker Maps entity text to URIs and creates typed links between entity IDs: @@ -458,7 +622,7 @@ linked = linker.link(text="Apple Inc. was founded by Steve Jobs.", entities=enti for e in linked: print("{} → {} (confidence: {:.2f})".format(e.text, e.uri, e.confidence)) -# Explicitly link two entity IDs +# Explicitly link two entity IDs (not a list — takes two IDs) linker.link_entities( entity1_id="apple_inc", entity2_id="aapl", @@ -475,7 +639,7 @@ print("Links: ", web["statistics"]["total_links"]) `LinkedEntity` fields returned by `link()`: | Field | Type | Description | -| ----- | ---- | ----------- | +| :----- | :---- | :----------- | | `entity_id` | `str` | Entity identifier | | `uri` | `str` | Generated URI (e.g. `"https://semantica.dev/entity/apple_inc."`) | | `text` | `str` | Surface form text | @@ -484,6 +648,7 @@ print("Links: ", web["statistics"]["total_links"]) | `context` | `Dict` | Entity metadata | | `confidence` | `float` | Overall confidence score | + ## ContextRetriever Hybrid retrieval combining vector similarity, graph traversal, and memory: @@ -511,6 +676,7 @@ for r in results: print("[{}] score={:.3f}: {}".format(r.source, r.score, r.content[:80])) ``` + ## Data Structures @@ -622,6 +788,7 @@ class EntityLink: + ## Real-World Patterns @@ -740,6 +907,7 @@ class EntityLink: + ## Tips and Common Pitfalls diff --git a/docs/reference/core.md b/docs/reference/core.md index e4c4481f..9660c1b5 100644 --- a/docs/reference/core.md +++ b/docs/reference/core.md @@ -4,25 +4,49 @@ description: "Framework orchestration, lifecycle management, configuration, and icon: "gear" --- -`semantica.core` is the coordination layer for the framework. For most tasks you should use individual modules directly (`semantica.ingest`, `semantica.kg`, etc.). Reach for Core when you need application-level lifecycle management, centralized configuration, or a plugin registry. +**`semantica.core`** is the **coordination layer** for the framework: + +- `Semantica` orchestrator coordinates the full KG construction pipeline from a YAML config +- `ConfigManager` loads YAML config with deep-merge, validation, and environment variable overrides +- `PluginRegistry` enables dynamic component registration and loading at runtime +- `LifecycleManager` manages startup/shutdown with health monitoring and lifecycle hooks + + + Use individual modules directly for the vast majority of use cases. Reach for `semantica.core` only when you need application-level lifecycle management, centralized config, or a plugin system. + + + +## What You Get + + + + High-level orchestrator — coordinates the full KG construction pipeline from a single `config.yaml`. Entry point for application-level deployments. + + + YAML config with deep-merge, `SEMANTICA_` env var overrides, and dot-notation nested key access. Keeps secrets out of source files. + + + Ordered startup/shutdown hooks, health monitoring, and a 6-state machine. Essential for long-running services like FastAPI apps. + + + Register custom ingestors, parsers, exporters, or any component. Load them by name at runtime — no imports required. + + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `Semantica` | Orchestration entry point — coordinates the full KG construction pipeline | | `ConfigManager` | YAML config loading, deep-merge, validation, and env var overrides | | `LifecycleManager` | Startup/shutdown state machine with health monitoring and lifecycle hooks | | `PluginRegistry` | Dynamic plugin discovery, registration, and loading | | `method_registry` | Global `MethodRegistry` instance — register and dispatch custom orchestration methods | - - **Use individual modules directly** for the vast majority of use cases. Use the `Semantica` orchestration class only when you need application-level lifecycle management or a plugin system. - ## Semantica (Orchestration) -High-level entry point that coordinates the full KG construction pipeline: +**`Semantica`** is the high-level entry point that coordinates the **full KG construction pipeline**: ```python from semantica.core import Semantica, ConfigManager @@ -48,7 +72,7 @@ finally: ### Core Methods | Method | Description | -| ------ | ----------- | +| :------ | :----------- | | `initialize()` | Initialize all framework components | | `build_knowledge_base(sources, **kwargs)` | Orchestrate full KG construction pipeline | | `run_pipeline(pipeline, data)` | Execute an existing `Pipeline` instance | @@ -201,7 +225,7 @@ result = build_knowledge_base(sources=["doc.pdf"], method="fast") ## When to Use Core vs. Individual Modules | Scenario | Recommended Approach | -| -------- | -------------------- | +| :-------- | :-------------------- | | Single extraction task | `from semantica.semantic_extract import NERExtractor` | | Build a knowledge graph | `from semantica.kg import GraphBuilder` | | Multi-step pipeline | `from semantica.pipeline import Pipeline` | diff --git a/docs/reference/deduplication.md b/docs/reference/deduplication.md index 4d47d464..b9a86b02 100644 --- a/docs/reference/deduplication.md +++ b/docs/reference/deduplication.md @@ -4,12 +4,19 @@ description: "Entity deduplication — similarity scoring, blocking, merging, an icon: "copy" --- -`semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. All deduplication workflows operate on plain Python dicts with `id`, `name`, `type`, `properties`, and `relationships` keys. +**`semantica.deduplication`** detects and merges duplicate entities across sources to produce a **clean, single-source-of-truth** knowledge graph: + +- Four v2 strategies up to 7× faster than v1: `blocking_v2`, `hybrid_v2`, `semantic_v2` +- `ClusterBuilder` uses Union-Find and hierarchical clustering for batch deduplication at scale +- `EntityMerger` preserves original source provenance on every merged entity +- `MergeStrategyManager` supports per-property rules and conflict resolution +- All workflows operate on plain Python dicts — no ORM or schema required + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `DuplicateDetector` | Pairwise and batch detection — returns `DuplicateCandidate` or `DuplicateGroup` lists | | `EntityMerger` | Merge duplicate groups — returns `List[MergeOperation]` | | `SimilarityCalculator` | Multi-factor similarity: string, property, relationship, and embedding | @@ -21,6 +28,30 @@ icon: "copy" | `merge_entities()` | Convenience function — `merge_entities(entities, method="keep_most_complete")` | | `calculate_similarity()` | Convenience function — `calculate_similarity(entity_a, entity_b, method="multi_factor")` | +## What You Get + + + + Pairwise, batch, incremental, and group detection modes. Returns scored candidates with reasons. + + + Five merge strategies — keep first, last, most complete, highest confidence, or merge all fields. + + + Multi-factor scoring across string edit distance, property overlap, relationship overlap, and embeddings. + + + Union-Find and hierarchical clustering for batch deduplication at scale — handles 100k+ entity sets. + + + Per-property merge rules with conflict resolution priorities. Apply different strategies to different fields. + + + `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7× faster than v1 for large entity sets. + + + + ## Getting Started ```python @@ -100,7 +131,7 @@ the comparison is performed. These are independent of the `SimilarityCalculator` method used internally: | `method=` | Algorithm | Returns | -| --------- | --------- | ------- | +| :--------- | :--------- | :------- | | `"pairwise"` (default) | O(n²) all-pairs comparison | `List[DuplicateCandidate]` | | `"batch"` | Batch similarity calculation | `List[DuplicateCandidate]` | | `"incremental"` | New vs existing entities | `List[DuplicateCandidate]` | @@ -109,7 +140,7 @@ method used internally: ### DuplicateCandidate fields | Field | Type | Description | -| ----- | ---- | ----------- | +| :----- | :---- | :----------- | | `entity1` | `Dict` | First entity | | `entity2` | `Dict` | Second entity | | `similarity_score` | `float` | Similarity score (0–1) | @@ -120,7 +151,7 @@ method used internally: ### DuplicateGroup fields | Field | Type | Description | -| ----- | ---- | ----------- | +| :----- | :---- | :----------- | | `entities` | `List[Dict]` | All entities in the group | | `similarity_scores` | `Dict` | Pair → score mapping | | `representative` | `Optional[Dict]` | Most complete entity in group | @@ -162,7 +193,7 @@ print("Total merges performed:", len(history)) Pass as a string to `strategy=` on `merge_duplicates()` or `merge_entity_group()`: | Strategy | Behavior | -| -------- | -------- | +| :-------- | :-------- | | `"keep_first"` | Keep the first entity in each duplicate group | | `"keep_last"` | Keep the most recently seen entity | | `"keep_most_complete"` | Keep the entity with the most non-null properties + relationships | @@ -201,7 +232,7 @@ operations = merger.merge_duplicates(entities) ### MergeOperation fields | Field | Type | Description | -| ----- | ---- | ----------- | +| :----- | :---- | :----------- | | `source_entities` | `List[Dict]` | Original entities that were merged | | `merged_entity` | `Dict` | Resulting merged entity | | `merge_result` | `MergeResult` | Detailed result with conflicts | @@ -247,7 +278,7 @@ rel_score = calc.calculate_relationship_similarity(entity_a, entity_b) ### SimilarityResult fields | Field | Type | Description | -| ----- | ---- | ----------- | +| :----- | :---- | :----------- | | `score` | `float` | Overall weighted similarity score (0–1) | | `method` | `str` | Method used (e.g. `"multi_factor"`, `"levenshtein"`) | | `components` | `Dict[str, float]` | Per-component scores: `"string"`, `"property"`, `"relationship"`, `"embedding"` | @@ -284,7 +315,7 @@ print("Quality metrics:", result.quality_metrics) ### Cluster fields | Field | Type | Description | -| ----- | ---- | ----------- | +| :----- | :---- | :----------- | | `cluster_id` | `str` | Unique cluster identifier | | `entities` | `List[Dict]` | Entities in the cluster | | `centroid` | `Optional[Dict]` | Representative entity (optional) | diff --git a/docs/reference/embeddings.md b/docs/reference/embeddings.md index a19e2bf6..7d085cd5 100644 --- a/docs/reference/embeddings.md +++ b/docs/reference/embeddings.md @@ -4,7 +4,14 @@ description: "Text and graph embedding generation — FastEmbed, Sentence-Transf icon: "vector-square" --- -`semantica.embeddings` converts text and graph structures into dense vectors. These vectors power semantic search, entity resolution, GraphRAG retrieval, and deduplication across every Semantica module. A single provider-agnostic API abstracts FastEmbed, Sentence-Transformers, OpenAI, and BGE behind one interface. +**`semantica.embeddings`** converts text and graph structures into **dense vector representations**: + +- Provider-agnostic API: FastEmbed (default, ONNX, no GPU), Sentence-Transformers, OpenAI, BGE +- Powers semantic search, entity resolution, GraphRAG retrieval, and deduplication +- `GraphEmbeddingManager` embeds KG nodes and edges for graph database backends +- Five pooling strategies: Mean (default), Max, CLS, Attention, Hierarchical +- `check_available_providers()` shows which backends are installed in your environment + ## Why Embeddings Matter @@ -21,14 +28,14 @@ Semantica uses embeddings for: ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `EmbeddingGenerator` | Provider-agnostic entry point — handles batching and provider selection | | `TextEmbedder` | Text embedding with automatic batch processing; default uses FastEmbed | | `GraphEmbeddingManager` | Embed KG nodes and edges for GraphRAG and graph databases | | `VectorEmbeddingManager` | Prepare and format embeddings for vector database backends | | `OpenAIStore` | OpenAI `text-embedding-3-small` / `text-embedding-3-large` provider | | `BGEStore` | BAAI/bge models via `sentence-transformers` | -| `FastEmbedStore` | ONNX-accelerated local embeddings — no CUDA required | +| `FastEmbedStore` | ONNX-accelerated local embeddings — no CUDA **required** | | `LlamaStore` | Placeholder store — not production-ready; do not use for embeddings | | `MeanPooling` | Default pooling strategy — best for retrieval and clustering | @@ -55,17 +62,93 @@ Semantica uses embeddings for: -## Installation +## Provider Setup -| Provider | Install Command | API Key Required | -| -------- | --------------- | ---------------- | -| FastEmbed (default) | `pip install "semantica[fastembed]"` | No | -| Sentence-Transformers | `pip install semantica` | No | -| BGE | `pip install semantica` | No (uses sentence-transformers) | -| OpenAI | `pip install "semantica[llm-openai]"` | Yes — `OPENAI_API_KEY` | -| All providers | `pip install "semantica[all]"` | Varies | + + + ONNX-accelerated local embeddings. No GPU required, no API key. Best starting point. -Check which providers are available in your environment: + ```bash + pip install "semantica[fastembed]" + ``` + + ```python + from semantica.embeddings import EmbeddingGenerator + + # FastEmbed is the default — no config needed + generator = EmbeddingGenerator() + embedding = generator.generate_embeddings("Text about AI") + ``` + + + Default model is `BAAI/bge-small-en-v1.5`. Zero cost, zero GPU, works on any machine. + + + + Broad model selection via HuggingFace. Runs locally, no API key. + + ```bash + pip install semantica # sentence-transformers included + ``` + + ```python + from semantica.embeddings import EmbeddingGenerator + + generator = EmbeddingGenerator(config={ + "text": { + "method": "sentence_transformers", + "model_name": "all-MiniLM-L6-v2", + } + }) + ``` + + Popular models: `all-MiniLM-L6-v2` (fast, small), `all-mpnet-base-v2` (balanced), `BAAI/bge-large-en-v1.5` (high accuracy). + + + BAAI/bge models via sentence-transformers. State-of-the-art retrieval performance, runs locally. + + ```bash + pip install semantica + ``` + + ```python + from semantica.embeddings import BGEStore, EmbeddingGenerator + + store = BGEStore(model="BAAI/bge-large-en-v1.5") + embedding = store.embed("Text about AI") + + # Or switch model on an existing EmbeddingGenerator + generator = EmbeddingGenerator() + generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") + ``` + + + Cloud embeddings via OpenAI API. Highest quality, requires API key. + + ```bash + pip install "semantica[llm-openai]" + export OPENAI_API_KEY="sk-..." + ``` + + ```python + import os + from semantica.embeddings import OpenAIStore + + store = OpenAIStore( + api_key=os.getenv("OPENAI_API_KEY"), + model="text-embedding-3-small", # or text-embedding-3-large + ) + embedding = store.embed("Text about AI") + ``` + + | Model | Dimensions | Best for | + | :---- | :--------- | :-------- | + | `text-embedding-3-small` | 1536 | Cost-efficient retrieval | + | `text-embedding-3-large` | 3072 | Highest accuracy workloads | + + + +Check which providers are installed in your environment: ```python from semantica.embeddings import check_available_providers @@ -156,8 +239,8 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") ## Supported Models | Provider | Model | Dimension | Speed | Best For | -| -------- | ----- | --------- | ----- | -------- | -| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Very fast | **Default** — CPU-optimised, no GPU required | +| :-------- | :----- | :--------- | :----- | :-------- | +| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Very fast | **Default** — CPU-optimised, no GPU **required** | | `sentence_transformers` | `all-MiniLM-L6-v2` | 384 | Fast | Good balance of speed and quality | | `sentence_transformers` | `all-mpnet-base-v2` | 768 | Medium | Higher retrieval quality | | `sentence_transformers` | `BAAI/bge-large-en-v1.5` | 1024 | Medium | State-of-the-art retrieval accuracy | @@ -177,7 +260,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") similarity = generator.compare_embeddings(embeddings[0], embeddings[1]) ``` - Best for: CPU-only production, lowest latency without GPU. Default — works out of the box. + **Best for:** CPU-only production, lowest latency without GPU. Default — works out of the box. ```python @@ -188,7 +271,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") embeddings = generator.generate_embeddings(texts) ``` - Best for: higher-quality retrieval when GPU is available, or when fine-tuned models are needed. + **Best for:** higher-quality retrieval when GPU is available, or when fine-tuned models are needed. ```python @@ -199,7 +282,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") embedding = store.embed("Hello world") ``` - Best for: highest quality (`text-embedding-3-large`), or matching an existing OpenAI pipeline. + **Best for:** highest quality (`text-embedding-3-large`), or matching an existing OpenAI pipeline. ```python @@ -219,7 +302,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5") ### Constructor Parameters | Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | +| :--------- | :---- | :------- | :----------- | | `config` | `dict` | `None` | Config dict; `config["text"]` is passed to `TextEmbedder` | | `**kwargs` | | | Additional key/value config merged into `config` | @@ -251,7 +334,7 @@ dim = embedder.get_embedding_dimension() ### TextEmbedder Constructor Parameters | Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | +| :--------- | :---- | :------- | :----------- | | `model_name` | `str` | `"BAAI/bge-small-en-v1.5"` | Model name to load | | `method` | `str` | `"fastembed"` | Embedding method: `"fastembed"` or `"sentence_transformers"` | | `device` | `str` | `"cpu"` | Device for sentence-transformers: `"cpu"`, `"cuda"`, `"mps"`. Ignored for FastEmbed. | @@ -308,7 +391,7 @@ Pooling aggregates a set of embeddings into a single vector — useful when you pooled = pooler.pool(token_embeddings) # shape: (hidden_dim,) ``` - Best for: retrieval, semantic search, and clustering — averages all contributions. + **Best for:** retrieval, semantic search, and clustering — averages all contributions. ```python @@ -318,7 +401,7 @@ Pooling aggregates a set of embeddings into a single vector — useful when you pooled = pooler.pool(token_embeddings) ``` - Best for: capturing the presence of any feature — takes the max activation per dimension. + **Best for:** capturing the presence of any feature — takes the max activation per dimension. ```python @@ -328,7 +411,7 @@ Pooling aggregates a set of embeddings into a single vector — useful when you pooled = pooler.pool(token_embeddings) ``` - Best for: classification-style tasks; models explicitly trained with CLS pooling (BERT). + **Best for:** classification-style tasks; models explicitly trained with CLS pooling (BERT). ```python @@ -339,12 +422,12 @@ Pooling aggregates a set of embeddings into a single vector — useful when you pooled = pooler.pool(token_embeddings, chunk_size=10) ``` - Best for: long documents — chunk-level mean pooling, then global mean pooling across chunks. + **Best for:** long documents — chunk-level mean pooling, then global mean pooling across chunks. | Strategy | When to Use | - | -------- | ----------- | + | :-------- | :----------- | | `mean` | Default for retrieval, semantic search, and clustering | | `max` | When you want to capture the presence of any feature, not average presence | | `cls` | Classification-style tasks; models explicitly trained with CLS pooling (BERT) | diff --git a/docs/reference/evals.md b/docs/reference/evals.md index ac14c58b..dc4c756a 100644 --- a/docs/reference/evals.md +++ b/docs/reference/evals.md @@ -4,7 +4,7 @@ description: "Evaluation framework for measuring Knowledge Graph quality, extrac icon: "chart-line" --- -`semantica.evals` is planned as a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance. +**`semantica.evals`** is planned as a comprehensive evaluation framework for measuring **extraction accuracy, graph quality, and pipeline performance**. **`semantica.evals` is not yet implemented.** The module is a placeholder with `__all__ = []`. No classes or functions are available for import. This page describes the planned API only. @@ -15,7 +15,7 @@ icon: "chart-line" When released, `semantica.evals` will provide: | Planned Class | Role | -| --- | --- | +| :--- | :--- | | `KGEvaluator` | Completeness, consistency, schema compliance, coverage, and orphan node detection | | `ExtractionEvaluator` | NER precision / recall / F1 and relation extraction metrics against gold datasets | | `PipelineBenchmark` | Throughput (docs/sec), per-step latency, peak memory, and error rate | @@ -51,7 +51,7 @@ print("Relation coverage: ", report["relation_completeness"]["relation_coverage" `EvaluationResult` fields returned by `evaluate_ontology()`: | Field | Type | Description | -| ----- | ---- | ----------- | +| :----- | :---- | :----------- | | `coverage_score` | `float` | Fraction of competency questions answerable by the ontology | | `completeness_score` | `float` | Average of class and property completeness scores | | `gaps` | `List[str]` | Identified gaps in coverage | diff --git a/docs/reference/explorer.md b/docs/reference/explorer.md index 6475b3f2..cd60f773 100644 --- a/docs/reference/explorer.md +++ b/docs/reference/explorer.md @@ -4,7 +4,14 @@ description: "Interactive FastAPI dashboard for knowledge graph exploration, ont icon: "map" --- -`semantica.explorer` is a browser-based dashboard for exploring knowledge graphs, managing ontologies, and running visual analyses — no code required after launch. +**`semantica.explorer`** is a **browser-based dashboard** for exploring knowledge graphs, managing ontologies, and running visual analyses: + +- Indexed search: 0.004ms on 118k nodes — no full scans +- Ontology Hub: visual editor, SHACL Studio, alignment authoring, and health dashboard +- Bidirectional path finding between any two nodes +- WebSocket progress streaming for live pipeline monitoring +- No code required after launch — full graph exploration in the browser + ## Getting Started @@ -86,8 +93,8 @@ The browser opens automatically at `http://127.0.0.1:8000`. The interactive API The `semantica-explorer` command accepts exactly four flags: | Flag | Short | Default | Description | -| ---- | ----- | ------- | ----------- | -| `--graph` | `-g` | *(required)* | Path to a ContextGraph JSON file to load | +| :---- | :----- | :------- | :----------- | +| `--graph` | `-g` | *(**required**)* | Path to a ContextGraph JSON file to load | | `--port` | `-p` | `8000` | Port to bind the server | | `--host` | — | `127.0.0.1` | Host to bind the server — use `0.0.0.0` to expose on the network | | `--no-browser` | — | off | Skip auto-opening the browser tab | @@ -178,7 +185,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/graph/stats` | `GET` | Node count, edge count, entity type distribution | | `/api/graph/nodes` | `GET` | List nodes — `?type=&search=&skip=&limit=&cursor=&bbox=` | | `/api/graph/node/{id}` | `GET` | Fetch a single node with all properties | @@ -195,14 +202,14 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and **Analytics:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/analytics` | `GET` | Graph metrics — `?metrics=centrality,community,connectivity` | | `/api/analytics/validation` | `GET` | Graph validation report | **Enrich:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/enrich/extract` | `POST` | Entity extraction from text | | `/api/enrich/links` | `POST` | Link prediction for nodes | | `/api/enrich/dedup` | `POST` | Duplicate detection | @@ -212,7 +219,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and **Temporal:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/temporal/snapshot` | `GET` | Graph snapshot at `?at=ISO8601` (defaults to now) | | `/api/temporal/diff` | `GET` | Diff between two times — `?from_time=&to_time=` | | `/api/temporal/patterns` | `GET` | Temporal activity patterns | @@ -225,7 +232,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and **Ontology:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/ontology/registry` | `GET` | List loaded ontologies | | `/api/ontology/load` | `POST` | Load an ontology from URL or content | | `/api/ontology/create` | `POST` | Create a new ontology | @@ -243,7 +250,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and **Vocabulary:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/vocabulary/schemes` | `GET` | SKOS schemes via TripletStore | | `/api/vocabulary/concepts` | `GET` | Concepts in a scheme — `?scheme=URI` | | `/api/vocabulary/hierarchy` | `GET` | Concept hierarchy tree | @@ -252,7 +259,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and **SPARQL:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/sparql` | `POST` | Execute a SPARQL SELECT or ASK query | @@ -261,7 +268,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and **Decisions:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/decisions` | `GET` | Paginated list of recorded decisions | | `/api/decisions/{id}` | `GET` | Single decision details | | `/api/decisions/{id}/chain` | `GET` | Causal chain for a decision | @@ -272,14 +279,14 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and **Provenance:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/provenance` | `GET` | Entity provenance lineage — `?node_id=` | | `/api/provenance/report` | `GET` | Provenance export report — `?node_id=` | **Annotations:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/annotations` | `GET` | List annotations — `?node_id=` (optional) | | `/api/annotations` | `POST` | Create annotation (returns 201) | | `/api/annotations/{id}` | `DELETE` | Delete annotation (returns 204) | @@ -287,7 +294,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and **Export / Import:** | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/export` | `POST` | Export graph as JSON or CSV — body: `{format, node_ids}` | | `/api/export/distance-enriched` | `POST` | Export pairwise distances as CSV or JSONL | | `/api/import` | `POST` | Import nodes/edges from `.json` or `.csv` file (max 50 MB) | @@ -296,7 +303,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and | Endpoint | Method | Description | - | -------- | ------ | ----------- | + | :-------- | :------ | :----------- | | `/api/health` | `GET` | Returns `{"status": "healthy"}` | | `/api/info` | `GET` | Server name, version, status | | `/docs` | `GET` | Interactive Swagger UI — all endpoints | @@ -348,7 +355,7 @@ Event types broadcast over the WebSocket include: `connection_ack`, `pong`, and ## Performance | Scenario | Latency | -| -------- | ------- | +| :-------- | :------- | | Node search (118k nodes, indexed) | 0.004ms | | Neighbor expansion (depth 2) | < 5ms | | BFS path (118k nodes) | < 50ms | diff --git a/docs/reference/export.md b/docs/reference/export.md index 730b209f..e78e081c 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -4,12 +4,19 @@ description: "Export knowledge graphs to RDF, Parquet, LPG, ArangoDB AQL, CSV, G icon: "file-export" --- -`semantica.export` serializes knowledge graphs to every downstream format — semantic web standards, analytics pipelines, graph databases, and vector stores. +**`semantica.export`** serializes knowledge graphs to **every downstream format**: + +- RDF: Turtle, JSON-LD, N-Triples, RDF/XML — with optional W3C PROV-O provenance inline +- Analytics: Apache Parquet and Arrow for Spark, BigQuery, Databricks +- Graph databases: Cypher `CREATE` statements for Neo4j; AQL `INSERT` for ArangoDB +- Standard formats: GraphML, GEXF, Graphviz DOT, CSV, OWL 2.0 +- Vector export: NumPy `.npz`, FAISS index, binary for embedding pipelines + ## Exported Classes | Class | Output formats | Notes | -| --- | --- | --- | +| :--- | :--- | :--- | | `RDFExporter` | Turtle, JSON-LD, N-Triples, RDF/XML | `export_to_rdf()` → string; `export()` → file | | `ParquetExporter` | `.parquet` | Requires `pyarrow`; explicit typed schema | | `LPGExporter` | Cypher `CREATE` | Neo4j and Memgraph compatible | @@ -321,7 +328,7 @@ The `export_csv` convenience function delegates to `CSVExporter.export()`. For p ## Format Reference | Format string | Canonical name | Exporter | File ext | Best for | -| --- | --- | --- | --- | --- | +| :--- | :--- | :--- | :--- | :--- | | `"turtle"` / `"ttl"` | `turtle` | `RDFExporter` | `.ttl` | Readable RDF, ontology sharing | | `"jsonld"` / `"json-ld"` | `jsonld` | `RDFExporter` | `.jsonld` | APIs, Linked Data, JSON pipelines | | `"ntriples"` / `"nt"` | `ntriples` | `RDFExporter` | `.nt` | Streaming RDF, line-by-line processing | diff --git a/docs/reference/graph_store.md b/docs/reference/graph_store.md index b192aaf0..b998f942 100644 --- a/docs/reference/graph_store.md +++ b/docs/reference/graph_store.md @@ -4,12 +4,19 @@ description: "Unified interface for Neo4j, FalkorDB, Apache AGE, and Amazon Nept icon: "server" --- -`semantica.graph_store` provides a single API for persisting and querying knowledge graphs in production graph databases. Swap backends with a one-line change — no application code changes needed. +**`semantica.graph_store`** provides a **single unified API** for persisting and querying knowledge graphs in production graph databases: + +- Swap backends with a one-line change — Neo4j, FalkorDB, Apache AGE, Amazon Neptune +- Parameterized Cypher execution with optional result caching via `QueryEngine` +- Batch node and edge loading — faster than individual writes +- `GraphAnalytics` for degree centrality, connected components, shortest path, neighbor traversal +- Context manager support: `with GraphStore(...) as store:` closes connection automatically + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `GraphStore` | Unified interface: `create_node`, `create_relationship`, `query`, `get_neighbors`, `shortest_path` | | `QueryEngine` | Parameterized Cypher execution with result caching | | `GraphAnalytics` | `degree_centrality`, `connected_components`, `shortest_path`, `get_neighbors` | @@ -18,29 +25,43 @@ icon: "server" | `AmazonNeptuneStore` | AWS Neptune — OpenCypher via Bolt protocol | | `FalkorDBStore` | Redis-based — sub-millisecond latency for real-time applications | + ## What You Get - Unified interface across Neo4j, FalkorDB, Apache AGE, and Amazon Neptune. + - Unified API across Neo4j, FalkorDB, Apache AGE, Amazon Neptune + - Context manager support for automatic connection cleanup + - `create_nodes()` for bulk loading — faster than individual calls - Parameterized Cypher construction and optional result caching. + - Parameterized Cypher construction prevents injection attacks + - Optional in-process result caching with `use_cache=True` + - `clear_cache()` on writes, toggle with `enable_cache()` / `disable_cache()` - Degree centrality, connected components, shortest path, and neighbor traversal. + - Degree centrality ordered by degree DESC + - Connected component assignment + - Shortest path between nodes, neighbor traversal up to N hops - Batched node and edge loading — faster than individual writes. + - `create_nodes(list)` — one round-trip for many nodes + - `create_relationship()` with typed properties + - `delete_node(detach=True)` removes all connected relationships - Create indexes to optimize query performance. + - `create_index(label, property_name=)` — makes MATCH queries orders-of-magnitude faster + - `get_stats()` — node counts, edge counts, type breakdown + - Create indexes before bulk loading for best performance - Find shortest paths between nodes and walk neighbors by depth. + - `shortest_path()` returns `length`, `nodes`, `relationships` + - `get_neighbors()` with direction and depth control + - Cross-backend path traversal via the unified API + ## Getting Started `GraphStore` wraps the backend of your choice behind a single API. Call `connect()` (or use it as a context manager) before running any queries: @@ -128,10 +149,11 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass + ## GraphStore Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `create_node(labels, properties)` | `dict` | Create a single node, returns node dict with `"id"` | | `create_nodes(nodes)` | `List[dict]` | Batch-create nodes from list of `{"labels", "properties"}` dicts | | `get_node(node_id)` | `dict \| None` | Retrieve a node by its backend ID | @@ -148,6 +170,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass | `shortest_path(start_node_id, end_node_id, rel_type, max_depth)` | `dict \| None` | Find shortest path between two nodes | | `get_stats()` | `dict` | Get graph statistics from the backend | + ## Backends @@ -169,7 +192,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass store.connect() ``` - Best for: production workloads, complex Cypher queries, Bloom visualization. + **Best for:** production workloads, complex Cypher queries, Bloom visualization. ```bash @@ -186,7 +209,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass store.connect() ``` - Best for: ultra-low latency queries over Redis protocol, edge deployments. + **Best for:** ultra-low latency queries over Redis protocol, edge deployments. ```bash @@ -202,7 +225,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass store.connect() ``` - Best for: teams already running PostgreSQL who want graph queries without a separate service. + **Best for:** teams already running PostgreSQL who want graph queries without a separate service. ```bash @@ -225,12 +248,12 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass ) ``` - Best for: managed AWS deployments. Neptune uses the Bolt protocol for OpenCypher queries — the same query API used for Neo4j. + **Best for:** managed AWS deployments. Neptune uses the Bolt protocol for OpenCypher queries — the same query API used for Neo4j. | Backend | Query Language | Deployment | IAM Auth | Best For | - | ------- | -------------- | ---------- | -------- | -------- | + | :------- | :-------------- | :---------- | :-------- | :-------- | | Neo4j | Cypher | Self-hosted / Aura | No | Production, complex traversals, Bloom UI | | FalkorDB | OpenCypher | Redis-based | No | Ultra-low latency, edge deployments | | Apache AGE | OpenCypher | PostgreSQL extension | No | Teams already on Postgres | @@ -324,12 +347,13 @@ engine.enable_cache() ### QueryEngine Methods -| Method | Description | -| ------ | ----------- | -| `execute(query, parameters, use_cache)` | Execute Cypher, optionally using in-process cache | -| `clear_cache()` | Flush all cached query results | -| `enable_cache()` | Turn on caching (on by default) | -| `disable_cache()` | Turn off caching | +| Method | Returns | Description | +| :------ | :------- | :----------- | +| `execute(query, parameters, use_cache)` | `dict` | Execute Cypher, returns `{success, records, keys, metadata}` | +| `clear_cache()` | `None` | Flush all cached query results | +| `enable_cache()` | `None` | Turn on caching (on by default) | +| `disable_cache()` | `None` | Turn off caching | + ## GraphAnalytics @@ -375,17 +399,18 @@ neighbors = analytics.get_neighbors( ### GraphAnalytics Methods -| Method | Description | -| ------ | ----------- | -| `degree_centrality(labels, rel_type, direction)` | Degree-based node importance — returns list of records ordered by degree | -| `connected_components(labels)` | Connected component assignment (requires GDS on Neo4j) | -| `shortest_path(start_node_id, end_node_id, rel_type, max_depth)` | Returns path dict or None | -| `get_neighbors(node_id, rel_type, direction, depth)` | Neighbor nodes up to `depth` hops | +| Method | Returns | Description | +| :------ | :------- | :----------- | +| `degree_centrality(labels, rel_type, direction)` | `List[dict]` | Degree-based node importance — records ordered by degree DESC | +| `connected_components(labels)` | `List[dict]` | Connected component assignment (requires GDS on Neo4j) | +| `shortest_path(start_node_id, end_node_id, rel_type, max_depth)` | `dict \| None` | Path with `length`, `nodes`, `relationships` or `None` | +| `get_neighbors(node_id, rel_type, direction, depth)` | `List[dict]` | Neighbor nodes up to `depth` hops | `betweenness_centrality()`, `pagerank()`, `detect_communities()`, and `all_paths()` are not implemented. Use Neo4j GDS procedures directly via `store.execute_query()` for those algorithms. + ## Schema Management ```python diff --git a/docs/reference/ingest.md b/docs/reference/ingest.md index af1133bc..5d124e28 100644 --- a/docs/reference/ingest.md +++ b/docs/reference/ingest.md @@ -4,12 +4,19 @@ description: "Universal data ingestion from files, Parquet, XML, web, public API icon: "database" --- -`semantica.ingest` is the entry point for loading data into Semantica. Each ingestor returns its own typed object (`FileObject`, `WebContent`, `TableData`, etc.) with normalized content and metadata for its source type. +**`semantica.ingest`** is the **universal entry point** for loading data into Semantica: + +- 15+ ingestion adapters: files, web, SQL, Snowflake, Kafka, MCP, Git repos, email +- PyArrow Parquet with column selection and partitioned dataset support +- XXE-safe lxml XML with optional XSD schema validation +- `ingest()` unified dispatcher — auto-detects source type from path or URL +- Each ingestor returns its own typed object (`FileObject`, `WebContent`, `TableData`, etc.) + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `FileIngestor` | PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, ZIP/TAR — type auto-detected from extension | | `CloudStorageIngestor` | Unified client for AWS S3, Google Cloud Storage, and Azure Blob Storage | | `WebIngestor` | Web scraping and crawling with `ingest_url`, `crawl_sitemap`, `crawl_domain` | @@ -29,7 +36,7 @@ icon: "database" ## Getting Started -Use `FileIngestor` for local files — it auto-detects format from the file extension and handles archives: +Use **`FileIngestor`** for local files — it **auto-detects format** from the file extension and handles archives: ```python from semantica.ingest import FileIngestor @@ -502,8 +509,8 @@ result = ingest( ### `ingest()` Parameters | Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | -| `sources` | `str`, `Path`, or `List` | required | File path, URL, directory, or connection string | +| :--------- | :---- | :------- | :----------- | +| `sources` | `str`, `Path`, or `List` | **required** | File path, URL, directory, or connection string | | `source_type` | `str` | `None` (auto-detected) | `"file"`, `"web"`, `"public_api"`, `"feed"`, `"stream"`, `"repo"`, `"email"`, `"db"`, `"parquet"`, `"xml"`, `"ontology"`, `"mcp"` | | `method` | `str` | `None` | Optional method override passed to the underlying ingestor | | `**kwargs` | | | Extra options forwarded to the underlying ingestor method | diff --git a/docs/reference/kg.md b/docs/reference/kg.md index 10f2790d..64945d16 100644 --- a/docs/reference/kg.md +++ b/docs/reference/kg.md @@ -4,12 +4,19 @@ description: "Graph construction, temporal models, analytics, similarity scoring icon: "diagram-project" --- -`semantica.kg` transforms extracted entities and relationships into structured, queryable knowledge graphs. It includes temporal support, a full suite of graph analytics algorithms, node embeddings, and structural similarity scoring. +`semantica.kg` transforms extracted entities and relationships into structured, queryable knowledge graphs: + +- Temporal nodes and edges with `valid_from` / `valid_until` windows and all 13 Allen interval relations +- Full graph analytics suite: centrality, community detection, path finding, link prediction +- Node2Vec structural embeddings for downstream ML and similarity scoring +- OWL-Time export and versioned snapshots via `TemporalVersionManager` +- Schema and constraint validation before persistence + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `KnowledgeGraph` | Core graph data structure — nodes, edges, properties, temporal validity | | `GraphBuilder` | Construct from entities + relationships; pass `merge_entities=True` to enable deduplication | | `GraphBuilderWithProvenance` | Wraps `GraphBuilder` with optional provenance tracking; pass `provenance=True` to enable | @@ -25,6 +32,7 @@ icon: "diagram-project" | `SimilarityCalculator` | Cosine, Euclidean, Manhattan, and correlation similarity scoring | | `GraphValidator` | Schema and constraint validation before persistence | + For conflict detection and advanced entity resolution, use `semantica.conflicts` and `semantica.deduplication` alongside this module. @@ -33,7 +41,7 @@ icon: "diagram-project" ## GraphBuilder -Constructs knowledge graphs from extracted entities and relationships. `merge_entities` defaults to `False` — pass `True` to enable entity deduplication during construction: +**`GraphBuilder`** constructs knowledge graphs from extracted entities and relationships. `merge_entities` defaults to `False` — pass **`True`** to enable entity deduplication during construction: ```python from semantica.kg import GraphBuilder @@ -43,14 +51,15 @@ builder = GraphBuilder(merge_entities=True) kg = builder.build({"entities": entities, "relationships": relationships}) ``` -| Method | Description | -| ------ | ----------- | -| `build(sources)` | Build graph from a dict, list of dicts, or list of entity/relation objects | -| `build_single_source(data)` | Build graph from a single data source dict | +| Method | Returns | Description | +| :------ | :------- | :----------- | +| `build(sources)` | `dict` | Build graph from a dict, list of dicts, or list of entity/relation objects | +| `build_single_source(data)` | `dict` | Build graph from a single data source dict | + ## Temporal Knowledge Graphs (v0.4.0) -Use `TemporalGraphQuery` to attach `valid_from`/`valid_until` windows and query time-aware graphs: +Use **`TemporalGraphQuery`** to attach `valid_from`/`valid_until` windows and query **point-in-time snapshots** of any graph: ```python from semantica.kg import GraphBuilder, TemporalGraphQuery, TemporalVersionManager @@ -91,7 +100,14 @@ versioner.create_snapshot(kg, version_label="2024-Q1", description="Q1 2024 snapshot") ``` -Supports all 13 Allen interval algebra relations (before, after, meets, overlaps, during, starts, finishes, equals, and their inverses). OWL-Time export available. +Supports all 13 Allen interval algebra relations: + +- before, after, meets, met_by +- overlaps, overlapped_by +- during, contains, starts, started_by, finishes, finished_by, equals + +OWL-Time export available. + ## Similarity Scoring @@ -115,97 +131,150 @@ for node_id in similar: print(node_id) ``` + ## Graph Analytics -### Centrality Analysis + + + Measure node importance across five algorithms. Use `calculate_all_centrality()` to run them all at once. -```python -from semantica.kg import CentralityCalculator + ```python + from semantica.kg import CentralityCalculator -calculator = CentralityCalculator() + calculator = CentralityCalculator() -centrality = calculator.calculate_degree_centrality(graph) -pagerank = calculator.calculate_pagerank(graph, damping_factor=0.85) -betweenness = calculator.calculate_betweenness_centrality(graph) -closeness = calculator.calculate_closeness_centrality(graph) -eigenvector = calculator.calculate_eigenvector_centrality(graph) -all_metrics = calculator.calculate_all_centrality(graph) + # Run all centrality measures at once + all_metrics = calculator.calculate_all_centrality(graph) -top_nodes = calculator.get_top_nodes(centrality, top_k=10) -``` + # Or run individually + pagerank = calculator.calculate_pagerank(graph, damping_factor=0.85) + betweenness = calculator.calculate_betweenness_centrality(graph) + closeness = calculator.calculate_closeness_centrality(graph) -| Method | Algorithm | -| ------ | --------- | -| `calculate_degree_centrality()` | Degree-based importance | -| `calculate_betweenness_centrality()` | Bridge-based importance (bottleneck nodes) | -| `calculate_closeness_centrality()` | Distance-based importance | -| `calculate_eigenvector_centrality()` | Influence-based importance | -| `calculate_pagerank()` | Link-based importance (PageRank) | -| `calculate_all_centrality()` | All measures at once | + # Get the top 10 most important nodes + top_nodes = calculator.get_top_nodes(pagerank, top_k=10) + ``` -### Community Detection + | Method | Best for | + | :------ | :-------- | + | `calculate_degree_centrality()` | Most-connected nodes | + | `calculate_pagerank()` | Link-based influence (like Google PageRank) | + | `calculate_betweenness_centrality()` | Bottleneck / bridge nodes | + | `calculate_closeness_centrality()` | Nodes closest to all others | + | `calculate_eigenvector_centrality()` | Nodes connected to other high-influence nodes | + + + Discover clusters and communities within the graph. Louvain is the fastest; Leiden produces higher-quality partitions. -```python -from semantica.kg import CommunityDetector + ```python + from semantica.kg import CommunityDetector -detector = CommunityDetector() + detector = CommunityDetector() -# Louvain (default — fast, high quality) -communities = detector.detect_communities(graph, algorithm="louvain") + # Louvain — fast, high quality (default) + communities = detector.detect_communities(graph, algorithm="louvain") -# Leiden (higher quality, slower) -leiden_communities = detector.detect_communities_leiden(graph, resolution=1.2) + # Leiden — higher quality, slower + communities = detector.detect_communities_leiden(graph, resolution=1.2) -metrics = detector.calculate_community_metrics(graph, communities) -``` + # Evaluate community quality + metrics = detector.calculate_community_metrics(graph, communities) + print(f"Modularity: {metrics['modularity']:.3f}") + print(f"Communities found: {metrics['num_communities']}") + ``` -Algorithms: Louvain, Leiden, Label Propagation, K-Clique Communities. + | Algorithm | Strength | + | :--------- | :-------- | + | Louvain | Fast, good modularity — use for large graphs | + | Leiden | Best modularity — use when quality matters more than speed | + | Label Propagation | Near-linear time — use for very large graphs | + | K-Clique | Overlapping communities — nodes can belong to multiple groups | + + + Find shortest paths and route alternatives between any two nodes. -### Path Finding + ```python + from semantica.kg import PathFinder -```python -from semantica.kg import PathFinder + finder = PathFinder() -finder = PathFinder() + # Dijkstra shortest path + path = finder.dijkstra_shortest_path(graph, "Alice", "Bob") + print(" → ".join(path["path"])) -path = finder.dijkstra_shortest_path(graph, "node_a", "node_b") -paths = finder.all_shortest_paths(graph, "source", "target") -k_paths = finder.find_k_shortest_paths(graph, "source", "target", k=3) -``` + # All shortest paths between two nodes + paths = finder.all_shortest_paths(graph, "source", "target") -Algorithms: Dijkstra, A\*, BFS, All Shortest Paths, K-Shortest Paths. + # K-Shortest paths (alternative routes) + k_paths = finder.find_k_shortest_paths(graph, "source", "target", k=3) + ``` -### Link Prediction + | Algorithm | Use case | + | :--------- | :-------- | + | Dijkstra | Weighted shortest path — standard routing | + | A\* | Heuristic-guided search — faster on large sparse graphs | + | BFS | Unweighted shortest path — hop count only | + | K-Shortest | Multiple alternative routes | + + + Predict missing or future edges. Use to complete knowledge graphs or find implicit relationships. -```python -from semantica.kg import LinkPredictor + ```python + from semantica.kg import LinkPredictor -predictor = LinkPredictor(method="preferential_attachment") -links = predictor.predict_links(graph, top_k=20) -score = predictor.score_link(graph, "node_a", "node_b") -``` + predictor = LinkPredictor(method="preferential_attachment") -Algorithms: Preferential Attachment, Common Neighbors, Jaccard, Adamic-Adar, Resource Allocation. + # Predict the top 20 most likely missing edges + predicted = predictor.predict_links(graph, top_k=20) + for link in predicted: + print(f"{link['source']} → {link['target']} (score: {link['score']:.3f})") -### Node Embeddings + # Score a specific pair + score = predictor.score_link(graph, "Alice", "CompanyX") + ``` -```python -from semantica.kg import NodeEmbedder + | Algorithm | Best for | + | :--------- | :-------- | + | Preferential Attachment | High-degree node connection prediction | + | Common Neighbors | Nodes with shared connections | + | Jaccard | Normalized common-neighbor overlap | + | Adamic-Adar | Weighted common neighbors (penalizes hubs) | + | Resource Allocation | Conservative — ignores high-degree intermediaries | + + + Compute structural embeddings with Node2Vec, then find similar nodes or feed into downstream ML. -embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) -embeddings = embedder.compute_embeddings(graph_store, ["Entity"], ["RELATED_TO"]) -similar_nodes = embedder.find_similar_nodes(graph_store, "entity_123", top_k=10) -# find_similar_nodes returns List[str] — a list of similar node IDs -for node_id in similar_nodes: - print(node_id) -``` + ```python + from semantica.kg import NodeEmbedder, SimilarityCalculator + + # Compute Node2Vec embeddings + embedder = NodeEmbedder(method="node2vec", embedding_dimension=128) + embeddings = embedder.compute_embeddings( + graph, ["Person", "Organization"], ["RELATED_TO"] + ) + + # Find structurally similar nodes + similar = embedder.find_similar_nodes(graph, "Apple Inc.", top_k=5) + for node_id in similar: + print(node_id) + + # Compare two specific nodes by embedding similarity + calc = SimilarityCalculator() + score = calc.cosine_similarity(embeddings["Apple Inc."], embeddings["Google"]) + print(f"Structural similarity: {score:.3f}") + ``` + + + `find_similar_nodes` returns `List[str]` — a list of node IDs, not node objects. Look up full node data via `graph["nodes"]`. + + + -Supported algorithm: `node2vec`. ## Algorithm Summary | Category | Algorithms | Use Cases | -| -------- | ---------- | --------- | +| :-------- | :---------- | :--------- | | Node Embeddings | Node2Vec | Structural similarity, node representation | | Similarity | Cosine, Euclidean, Manhattan, Correlation | Node matching, recommendation | | Path Finding | Dijkstra, A\*, BFS, K-Shortest | Route planning, network analysis | @@ -214,6 +283,7 @@ Supported algorithm: `node2vec`. | Community Detection | Louvain, Leiden, Label Propagation | Social clustering | | Connectivity | Components, Bridges, Density | Network robustness | + ## GraphValidator Validates graph structure — checks required fields, duplicate IDs, dangling edges, and optionally detects cycles and orphan nodes: diff --git a/docs/reference/llms.md b/docs/reference/llms.md index 8caaf25e..ce67b3d9 100644 --- a/docs/reference/llms.md +++ b/docs/reference/llms.md @@ -4,7 +4,14 @@ description: "Unified interface for Groq, OpenAI, LiteLLM (Anthropic, Gemini, Ol icon: "microchip" --- -`semantica.llms` provides a single consistent API across every major LLM provider. Every provider is a drop-in replacement for the `llm_provider=` parameter in extractors, reasoning engines, and agents. +**`semantica.llms`** provides a **single consistent API** across every major LLM provider: + +- Every provider is a drop-in replacement for the `llm_provider=` parameter in extractors, reasoners, and agents +- `LiteLLM` routes to 100+ providers with a single class and model-string prefixes +- `HuggingFaceLLM` runs fully on-premise — no API key, no network calls +- Structured output via `generate_with_schema()` for JSON extraction from any provider +- Streaming, tool use, and `generate_batch()` for bulk inference + ## Exported Classes @@ -13,7 +20,7 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM ``` | Class | Provider | API Key Required | -| ----- | -------- | ---------------- | +| :----- | :-------- | :---------------- | | `Groq` | Groq Cloud | `GROQ_API_KEY` | | `OpenAI` | OpenAI / any OpenAI-compatible gateway | `OPENAI_API_KEY` | | `LiteLLM` | 100+ providers via LiteLLM routing | Depends on model | @@ -33,32 +40,123 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM ## Choosing a Provider -Use this decision matrix to select the right LLM provider for your use case: + + + Free tier, fastest inference, zero setup friction. Best for development and high-throughput extraction pipelines. -| Priority | Recommended Provider | Model | Why | -|----------|---------------------|-------|-----| -| **Getting Started** | Groq | `llama-3.1-8b-instant` | Free tier, fast inference, no complex setup | -| **Production Quality** | OpenAI | `gpt-4o` | Highest capability, function calling, JSON mode | -| **Cost Optimization** | LiteLLM + DeepSeek | `deepseek/deepseek-chat` | Lowest cost per token for high-volume workloads | -| **Privacy/On-Premise** | Ollama (via LiteLLM) | `ollama/llama3.2:3b` | Fully local, no data leaves your infrastructure | -| **Advanced Reasoning** | Anthropic Claude (via LiteLLM) | `anthropic/claude-sonnet-4-20250514` | Highest quality for complex analysis | + | | | + | :-- | :-- | + | **Speed** | Very fast — 100+ tok/s | + | **Cost** | Free tier available | + | **Context** | 128k | + | **Best for** | Development, high-throughput extraction | -### Quick Start Recommendation + ```python + import os + from semantica.llms import Groq -For new users, start with Groq: + llm = Groq( + model="llama-3.1-8b-instant", + api_key=os.getenv("GROQ_API_KEY"), + temperature=0.0, + ) + ``` -```python -from semantica.llms import Groq -import os + Get your free key at [console.groq.com](https://console.groq.com). + + + Highest accuracy, best JSON mode and function calling. Use for production pipelines where extraction quality matters. -# Fastest path to working extraction -llm = Groq( - model="llama-3.1-8b-instant", # Default model - api_key=os.getenv("GROQ_API_KEY") -) -``` + | | | + | :-- | :-- | + | **Speed** | Fast | + | **Cost** | Medium | + | **Context** | 128k | + | **Best for** | Production quality, JSON extraction, function calling | -Get your free API key at [console.groq.com](https://console.groq.com). + ```python + import os + from semantica.llms import OpenAI + + llm = OpenAI( + model="gpt-4o", + api_key=os.getenv("OPENAI_API_KEY"), + temperature=0.0, + max_tokens=4096, + ) + ``` + + + Fully on-premise — no API key, no data leaves your infrastructure. Required for air-gapped deployments. + + | | | + | :-- | :-- | + | **Speed** | Medium (hardware-dependent) | + | **Cost** | Free (local compute only) | + | **Context** | Varies by model | + | **Best for** | Privacy, air-gapped, custom fine-tunes | + + ```bash + # Install Ollama and pull a model first + ollama pull llama3.2:3b + ``` + + ```python + from semantica.llms import LiteLLM + + llm = LiteLLM( + model="ollama/llama3.2:3b", + api_base="http://localhost:11434", # Ollama default port + ) + ``` + + + No API key required. Ensure the Ollama server is running (`ollama serve`) before creating the `LiteLLM` instance. + + + + Largest context window, best multi-hop reasoning, highest safety bar. Use for complex analysis and long-document extraction. + + | | | + | :-- | :-- | + | **Speed** | Fast | + | **Cost** | Medium | + | **Context** | 200k | + | **Best for** | Complex reasoning, long documents, safety-critical outputs | + + ```python + import os + from semantica.llms import LiteLLM + + llm = LiteLLM( + model="anthropic/claude-sonnet-4-20250514", + api_key=os.getenv("ANTHROPIC_API_KEY"), + temperature=0.0, + ) + ``` + + + Lowest cost per token for high-volume workloads. Strong on coding and structured data extraction. + + | | | + | :-- | :-- | + | **Speed** | Fast | + | **Cost** | Very low | + | **Context** | 64k | + | **Best for** | High-volume pipelines, coding tasks, budget-sensitive workloads | + + ```python + import os + from semantica.llms import LiteLLM + + llm = LiteLLM( + model="deepseek/deepseek-chat", + api_key=os.getenv("DEEPSEEK_API_KEY"), + temperature=0.0, + ) + ``` + + ## API Key Setup @@ -132,7 +230,7 @@ llm = Groq( max_tokens=64000, temperature=0.0, ) -# Best for: high-throughput extraction, fast inference at low cost +# **Best for:** high-throughput extraction, fast inference at low cost ``` ```python OpenAI @@ -144,7 +242,7 @@ llm = OpenAI( api_key=os.getenv("OPENAI_API_KEY"), temperature=0.0, ) -# Best for: general purpose, function calling, JSON mode +# **Best for:** general purpose, function calling, JSON mode ``` ```python LiteLLM (100+ providers) @@ -255,7 +353,7 @@ trip = TripletExtractor(method="llm", llm_provider=llm) ## Provider Comparison | Provider | Import | Speed | Cost | Local | Context | Best For | -| -------- | ------ | ----- | ---- | ----- | ------- | -------- | +| :-------- | :------ | :----- | :---- | :----- | :------- | :-------- | | Groq | `Groq` | Very fast | Low | No | 128k | High-throughput extraction | | OpenAI | `OpenAI` | Fast | Medium | No | 128k | General purpose, function calling | | Anthropic | `LiteLLM(model="anthropic/...")` | Fast | Medium | No | 200k | Complex reasoning, safety | @@ -277,7 +375,7 @@ Documentation examples may showcase stronger models for better developer experie **Verified Implementation Defaults:** | Provider | Default Model | Notes | -|----------|---------------|-------| +| :---------- | :--------------- | :------- | | `Groq` | `llama-3.1-8b-instant` | Implementation default; examples use `llama-3.3-70b-versatile` for showcase | | `OpenAI` | `gpt-3.5-turbo` | Implementation default; examples use `gpt-4o` for showcase | | `HuggingFaceLLM` | `gpt2` | Lightweight, widely compatible | @@ -315,7 +413,7 @@ for text in texts: ### Model Selection by Use Case | Use Case | Recommended Provider/Model | Reasoning | -|----------|---------------------------|-----------| +| :---------- | :--------------------------- | :----------- | | **Entity Extraction** | `Groq("llama-3.3-70b-versatile")` | Fast, good accuracy for structured tasks | | **Relation Extraction** | `OpenAI("gpt-4o")` | Best at complex relationship reasoning | | **Complex Analysis** | `LiteLLM("anthropic/claude-sonnet-4-20250514")` | Highest reasoning capability | diff --git a/docs/reference/mcp_server.md b/docs/reference/mcp_server.md index 63ec640b..ff84af0a 100644 --- a/docs/reference/mcp_server.md +++ b/docs/reference/mcp_server.md @@ -4,11 +4,12 @@ description: "Model Context Protocol server — expose Semantica's full capabili icon: "plug" --- -`semantica.mcp_server` exposes Semantica's knowledge graph, decision intelligence, semantic extraction, and reasoning capabilities as an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server over stdio. +**`semantica.mcp_server`** exposes Semantica's knowledge graph, decision intelligence, semantic extraction, and reasoning capabilities as an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) **server over stdio**: -Once configured, any connected AI assistant can extract entities, record decisions, query the graph, run reasoning, and export results — without writing a single line of Python. +- 12 MCP tools exposed: extract entities, query graph, record decisions, run reasoning, export results +- No Python code required after launch — configure once, use from any MCP-aware client +- Compatible with Claude Desktop, Windsurf, Cline, Continue, VS Code, Roo Code, Cursor -Compatible with **Claude Desktop**, **Windsurf**, **Cline**, **Continue**, **VS Code**, **Roo Code**, **Cursor**, and any MCP-aware client. ## Server Interface @@ -71,7 +72,7 @@ The MCP server is included in the base install — no extras required. | Client | Settings file | - | ------ | ------------- | + | :------ | :------------- | | Claude Desktop (macOS) | `~/Library/Application Support/Claude/claude_desktop_config.json` | | Claude Desktop (Windows) | `%APPDATA%\Claude\claude_desktop_config.json` | | Cursor | `.cursor/mcp.json` in your project, or `~/.cursor/mcp.json` globally | @@ -151,7 +152,7 @@ The MCP server is included in the base install — no extras required. ## Environment Variables | Variable | Default | Description | -| -------- | ------- | ----------- | +| :-------- | :------- | :----------- | | `SEMANTICA_KG_PATH` | *(none — in-memory graph)* | Path to a persisted graph file to load on startup | | `SEMANTICA_LOG_LEVEL` | `WARNING` | Log verbosity: `DEBUG`, `INFO`, `WARNING` | @@ -160,7 +161,7 @@ The MCP server is included in the base install — no extras required. The MCP server exposes 12 tools that any connected AI assistant can call: | Tool | Category | Description | -| ---- | -------- | ----------- | +| :---- | :-------- | :----------- | | `extract_entities` | Extraction | NER — find people, places, organisations, concepts | | `extract_relations` | Extraction | Typed relation and triplet extraction | | `record_decision` | Decision Intelligence | Save a decision with reasoning and outcome | @@ -434,7 +435,7 @@ Supported formats: `turtle`, `ttl`, `nt`, `xml`, `json-ld`, `json`. Default is ` The MCP server exposes three readable resources: | URI | Description | -| --- | ----------- | +| :--- | :----------- | | `semantica://graph/summary` | High-level graph statistics | | `semantica://decisions/list` | All recorded decisions (up to 50) | | `semantica://schema/info` | Server version and available tools | diff --git a/docs/reference/normalize.md b/docs/reference/normalize.md index ac713078..dc224e3f 100644 --- a/docs/reference/normalize.md +++ b/docs/reference/normalize.md @@ -4,7 +4,16 @@ description: "Text cleaning, entity canonicalization, date normalization, number icon: "broom" --- -`semantica.normalize` standardizes raw data before extraction and graph construction. All normalizers expose both convenience functions (one-liners) and stateful class instances (full control over configuration and reuse). +**`semantica.normalize`** standardizes raw data **before extraction and graph construction**: + +- Text cleaning: Unicode NFC/NFKC, whitespace collapse, smart-quote and dash normalization +- Entity canonicalization: alias resolution and disambiguation via configurable alias maps +- Date normalization: any format → ISO 8601, including relative dates +- Number conversion: `"$1.2B"` → `1200000000.0` with unit and currency handling +- Language detection and encoding repair for inconsistent source data + +All normalizers expose convenience functions (one-liners) and stateful class instances (full control). + ## Why Normalize Before Extraction @@ -20,7 +29,7 @@ Normalization collapses these variants before any extractor, deduplicator, or gr ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `TextNormalizer` | Unicode forms (NFC/NFKC), whitespace collapse, smart-quote and dash normalization | | `EntityNormalizer` | Alias resolution and entity disambiguation using configurable alias maps | | `DateNormalizer` | Parses any date string format to ISO 8601; handles relative dates | @@ -212,7 +221,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") ``` | `normalize_text()` parameter | Type | Default | Description | - | ----------------------------- | ---- | ------- | ----------- | + | :----------------------------- | :---- | :------- | :----------- | | `unicode_form` | `str` | `"NFC"` | Unicode form: `"NFC"` / `"NFD"` / `"NFKC"` / `"NFKD"` | | `case` | `str` | `"preserve"` | `"preserve"` / `"lower"` / `"upper"` / `"title"` | | `normalize_diacritics` | `bool` | `False` | Strip diacritical marks | @@ -221,7 +230,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert") **Unicode form guide:** | Form | Use When | - | ---- | -------- | + | :---- | :-------- | | `NFC` | Default — best for storage and display | | `NFKC` | Search indexing — normalises ligatures, fullwidth chars, and fractions | | `NFD` | Stripping diacritics — split é → e + combining accent, then strip accents | @@ -496,7 +505,7 @@ print(f"Warnings: {len(result.warnings)}") ### DataCleaner Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `clean_data(dataset, remove_duplicates, validate, handle_missing, **options)` | `List[Dict]` | Combined cleaning pipeline | | `detect_duplicates(dataset, threshold, key_fields)` | `List[DuplicateGroup]` | Return duplicate groups above similarity threshold | | `validate_data(dataset, schema)` | `ValidationResult` | Validate records against a schema dict | diff --git a/docs/reference/ontology.md b/docs/reference/ontology.md index 6c709985..fcd44f76 100644 --- a/docs/reference/ontology.md +++ b/docs/reference/ontology.md @@ -4,12 +4,19 @@ description: "Automated ontology generation, SHACL validation, OWL/RDF export, n icon: "sitemap" --- -`semantica.ontology` provides the full lifecycle for knowledge graph schemas — from auto-generation and SHACL validation to OWL/RDF export. Use it for schema design, data modeling, semantic web interoperability, and SHACL-based data quality validation. +`semantica.ontology` provides the full lifecycle for knowledge graph schemas: + +- Auto-generate ontologies from KG data via a 5-stage pipeline (Semantic Network → YAML → Types → Hierarchy → TTL) +- LLM-powered ontology generation for complex domains via `LLMOntologyGenerator` +- SHACL validation: generate shapes, validate graphs, and get violation reports +- OWL/RDF export in Turtle, RDF/XML, and JSON-LD formats +- Ontology Hub visual editor available in `semantica.explorer` (v0.5.0) + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `OntologyEngine` | Unified facade orchestrating the full ontology lifecycle | | `OntologyGenerator` | Auto-generate ontologies from KG data (5-stage pipeline) | | `LLMOntologyGenerator` | LLM-powered ontology generation for complex domains | @@ -22,9 +29,10 @@ icon: "sitemap" | `PropertyGenerator` | Generate properties from entity attributes and relationships | | `AssociativeClassBuilder` | Model N-ary relationships as intermediate OWL classes | + ## Getting Started -`OntologyEngine` is your main entry point for the complete ontology workflow: +**`OntologyEngine`** is your main entry point for the complete ontology workflow: ```python from semantica.ontology import OntologyEngine @@ -47,7 +55,7 @@ engine.export_owl(ontology, "ontology.ttl", format="turtle") ## OntologyEngine (Unified Facade) -The `OntologyEngine` orchestrates the full ontology lifecycle — generation, validation, export, and evaluation: +**`OntologyEngine`** orchestrates the full ontology lifecycle — **generation, validation, export, and evaluation**: ```python from semantica.ontology import OntologyEngine @@ -70,7 +78,7 @@ engine.export_owl(ontology, "ontology.ttl", format="turtle") ### OntologyEngine Methods | Method | Description | -| ------ | ----------- | +| :------ | :----------- | | `from_data(data)` | Run the 5-stage pipeline on entity/relationship data | | `validate_graph(kg, ontology=...)` | Check a knowledge graph against generated SHACL shapes | | `export_owl(ontology, path, format)` | Serialize to `"turtle"`, `"xml"`, or `"json-ld"` | @@ -78,7 +86,7 @@ engine.export_owl(ontology, "ontology.ttl", format="turtle") ## OntologyGenerator (5-Stage Pipeline) -Generate a formal ontology automatically from your knowledge graph entities and relationships: +**`OntologyGenerator`** auto-generates a formal ontology from your knowledge graph entities and relationships: ```python from semantica.ontology import OntologyGenerator @@ -90,13 +98,23 @@ ontology = generator.generate_ontology({ }) ``` -The pipeline runs through these stages in order: - -1. **Semantic Network Parsing** — extract concepts and patterns from entity/relationship data -2. **YAML-to-Definition** — transform patterns into intermediate class definitions -3. **Definition-to-Types** — map definitions to OWL types (`owl:Class`, `owl:ObjectProperty`) -4. **Hierarchy Generation** — build taxonomy trees using transitive closure and cycle detection -5. **TTL Generation** — serialize to Turtle format using `rdflib` + + + Extract concepts and patterns from entity types and relationship structures in the source data. + + + Transform the extracted patterns into intermediate class and property definitions. + + + Map definitions to OWL constructs: `owl:Class`, `owl:ObjectProperty`, `owl:DatatypeProperty`. + + + Build taxonomy trees using transitive closure and cycle detection — produces `rdfs:subClassOf` chains. + + + Serialize the final ontology to Turtle format using `rdflib`. Also available: RDF/XML and JSON-LD. + + ## SHACL Validation @@ -125,7 +143,7 @@ if not report.conforms: ### Validation Report Fields | Field | Type | Description | -| ----- | ---- | ----------- | +| :----- | :---- | :----------- | | `conforms` | `bool` | `True` if the graph passes all SHACL constraints | | `violations` | `List[SHACLViolation]` | Detailed failure records | | `focus_node` | `str` | IRI of the violating graph node | diff --git a/docs/reference/parse.md b/docs/reference/parse.md index b075b2a7..73813750 100644 --- a/docs/reference/parse.md +++ b/docs/reference/parse.md @@ -4,7 +4,13 @@ description: "Document parsing and text extraction — DocumentParser for standa icon: "file-lines" --- -`semantica.parse` extracts structured text, layout, tables, and metadata from unstructured documents. `DocumentParser` handles clean machine-readable files; `DoclingParser` handles complex layouts, scanned PDFs, and multi-column documents. +**`semantica.parse`** extracts **structured text, layout, tables, and metadata** from unstructured documents: + +- `DocumentParser` — broad format support (PDF, DOCX, HTML, JSON, CSV, PPTX, XLSX), no extra dependencies +- `DoclingParser` — complex layouts, merged-cell tables, multi-column PDFs, OCR (`pip install docling`) +- Both return a consistent `dict` with `full_text`, `metadata`, `pages`, and `tables` keys +- `parse_batch()` processes multiple files in parallel with configurable error handling + ## Getting Started @@ -53,80 +59,90 @@ print(f"Extracted {len(text)} characters from {metadata.get('page_count', 0)} pa ## Parser Selection Guide -### DocumentParser -- **Best for**: Clean PDFs, Word docs, HTML, plain text -- **Formats**: PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX -- **Strengths**: Fast processing, broad format support, no dependencies + + + Zero extra dependencies. Use for clean PDFs, Word docs, HTML, and structured formats. -### DoclingParser -- **Best for**: Complex layouts, merged-cell tables, scanned documents -- **Formats**: PDF, DOCX, PPTX, XLSX, HTML, images -- **Strengths**: Superior table extraction, OCR support, multi-column handling -- **Requirements**: `pip install docling` + | | | + | :-- | :-- | + | **Formats** | PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX | + | **Speed** | Fast | + | **Setup** | None — included in base install | + | **Best for** | Clean documents, broad format support, production pipelines | -**Simple rule**: Start with `DocumentParser`. Use `DoclingParser` when you need better table extraction or handle complex document layouts. + ```python + from semantica.parse import DocumentParser -## Common Workflows + parser = DocumentParser() + result = parser.parse("contract.pdf") -### Single Document Parsing + print(result["full_text"]) # Extracted text + print(result["metadata"]) # Title, author, page count, ... + print(len(result.get("pages", []))) # Per-page breakdown + ``` + + + Superior table extraction, OCR, multi-column PDFs. Requires `pip install docling`. -```python -from semantica.parse import DocumentParser + | | | + | :-- | :-- | + | **Formats** | PDF, DOCX, PPTX, XLSX, HTML, images | + | **Speed** | Slower (deep layout analysis) | + | **Setup** | `pip install docling` | + | **Best for** | Merged-cell tables, scanned documents, multi-column layouts | -parser = DocumentParser() -result = parser.parse("contract.pdf") + ```python + from semantica.parse import DoclingParser -# Check what was extracted -print(f"Text length: {len(result['full_text'])}") -print(f"Metadata: {result['metadata']}") -if "tables" in result: - print(f"Tables found: {len(result['tables'])}") -``` + parser = DoclingParser(export_format="markdown") + result = parser.parse( + "financial_report.pdf", + extract_tables=True, + extract_text=True, + ) -### Batch Document Processing + for i, table in enumerate(result["tables"]): + print(f"Table {i+1}: {table['row_count']} rows × {table['col_count']} columns") + print(f" Page: {table['page_number']}") + for row in table["rows"][:3]: + print(" | ".join(row)) + ``` -```python -from semantica.parse import DocumentParser + + Start with `DocumentParser`. Switch to `DoclingParser` only when you need better table extraction or encounter complex PDF layouts. + + + + Process multiple files in parallel with per-file error isolation. -parser = DocumentParser() -files = ["doc1.pdf", "doc2.docx", "doc3.html"] + ```python + from semantica.parse import DocumentParser -# Process multiple files -results = parser.parse_batch(files, continue_on_error=True) + parser = DocumentParser() + results = parser.parse_batch( + ["doc1.pdf", "doc2.docx", "doc3.html"], + continue_on_error=True, # skip failed files instead of raising + ) -print(f"Successfully parsed: {results['success_count']}/{results['total']}") -for item in results["successful"]: - file_path = item["file_path"] - content = item["result"]["full_text"] - print(f"{file_path}: {len(content)} characters") -``` + print(f"Parsed: {results['success_count']}/{results['total']}") -### Enhanced Table Extraction + for item in results["successful"]: + print(f"{item['file_path']}: {len(item['result']['full_text'])} chars") -```python -from semantica.parse import DoclingParser + for item in results["failed"]: + print(f"FAILED: {item['file_path']} — {item['error']}") + ``` -parser = DoclingParser(export_format="markdown") -result = parser.parse( - "financial_report.pdf", - extract_tables=True, - extract_text=True -) - -# Access structured table data -for i, table in enumerate(result["tables"]): - print(f"Table {i+1}: {table['row_count']} rows, {table['col_count']} columns") - print(f"Page: {table['page_number']}") - - # Table data is in rows format - for row in table["rows"][:3]: # First 3 rows - print(" | ".join(row)) -``` + + `continue_on_error=True` is recommended for production batch jobs where individual files may be corrupted or unsupported. + + + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `DocumentParser` | Auto-detects format — delegates to format-specific parser (PDF, DOCX, HTML, JSON, CSV, ...) | | `DoclingParser` | Complex layouts, merged-cell tables, multi-column PDFs, and OCR (`pip install docling`) | | `DoclingMetadata` | Document metadata from Docling parsing | @@ -206,7 +222,7 @@ print(result["full_text"]) # OCR-extracted text ## Supported Formats | Format | Extension | Parser Used | Notes | -| ------ | --------- | ----------- | ----- | +| :------ | :--------- | :----------- | :----- | | PDF | `.pdf` | `PDFParser` / `DoclingParser` | Text, tables, metadata; Docling adds OCR | | Word | `.docx` | Built-in | Text, headings, tables, metadata | | HTML | `.html`, `.htm` | `HTMLParser` / `WebParser` | `WebParser` fetches remote URLs | @@ -251,7 +267,7 @@ metadata = { ## DocumentParser Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `parse(source)` | `dict` | Auto-detect format and extract text, metadata, tables | | `parse_batch(sources)` | `dict` | Process multiple sources in parallel | | `extract_text(path)` | `str` | Extract only text content from document | diff --git a/docs/reference/pipeline.md b/docs/reference/pipeline.md index fb567304..f2b0bfd9 100644 --- a/docs/reference/pipeline.md +++ b/docs/reference/pipeline.md @@ -4,12 +4,19 @@ description: "Pipeline DSL with parallel workers, retry policies, failure handli icon: "gear" --- -`semantica.pipeline` lets you chain Semantica components into reproducible, fault-tolerant workflows with parallel execution and configurable error handling. Pipelines are serializable — save them to YAML and reload in any environment. +**`semantica.pipeline`** lets you chain Semantica components into **reproducible, fault-tolerant workflows**: + +- Per-step failure strategies: `skip`, `retry`, `abort`, or `fallback` +- Parallel workers via `ParallelismManager` — thread or process pool +- `PipelineValidator` catches cycles, missing handlers, and config errors before running +- Pre-built templates: `"document_processing"`, `"rag_pipeline"`, `"kg_construction"`, `"ontology_generation"` +- Pipelines are serializable to YAML — save and reload in any environment + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `PipelineBuilder` | DSL for wiring steps: `add_step`, `connect_steps`, `set_parallel`, `build` | | `ExecutionEngine` | Runs a built pipeline: `execute_pipeline(pipeline, data)` → `ExecutionResult` | | `ExecutionResult` | `{success, output, metadata, metrics, errors}` — full run summary | @@ -180,7 +187,7 @@ result = engine.execute_pipeline(pipeline, data="data/") ### Failure Strategies | Strategy | Behaviour | When to Use | -| -------- | --------- | ----------- | +| :-------- | :--------- | :----------- | | `"skip"` | Log failure, continue to next document | Production — one bad doc shouldn't stop 10k | | `"stop"` | Raise exception immediately | Development — surface errors fast | | `"retry"` | Retry via `RetryPolicy`, then skip | When failures are likely transient | @@ -368,7 +375,7 @@ engine.stop_pipeline(pipeline_id) ``` | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `execute_pipeline(pipeline, data)` | `ExecutionResult` | Execute pipeline from start to finish | | `get_pipeline_status(pipeline_id)` | `PipelineStatus` | Current state (RUNNING, PAUSED, STOPPED) | | `get_progress(pipeline_id)` | `Dict` | `completed_steps`, `total_steps`, `progress_percentage`, `status` | diff --git a/docs/reference/provenance.md b/docs/reference/provenance.md index aebda7e6..311bee6d 100644 --- a/docs/reference/provenance.md +++ b/docs/reference/provenance.md @@ -4,12 +4,19 @@ description: "W3C PROV-O lineage tracking, source attribution, tamper-evident ch icon: "link" --- -`semantica.provenance` tracks the full lineage of every fact — from raw ingestion through extraction, chunking, and relationship building. Designed for high-stakes domains requiring complete traceability. +`semantica.provenance` tracks the full lineage of every fact — from raw ingestion through extraction, chunking, and relationship building: + +- W3C PROV-O compliant — suitable for HIPAA, SOX, GDPR, FDA 21 CFR Part 11 audit trails +- SHA-256 checksums for tamper detection on every stored `ProvenanceEntry` +- `SQLiteStorage` for persistence across restarts; `InMemoryStorage` for development +- `ProvenanceManager` provides `track_entity`, `track_relationship`, `track_chunk`, and `get_lineage` +- Bridges to W3C PROV-O ontology via `BridgeAxiom` for semantic web export + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `ProvenanceManager` | Central tracker: `track_entity`, `track_relationship`, `track_chunk`, `get_lineage`, `get_statistics` | | `ProvenanceEntry` | Single lineage record: `{entity_id, entity_type, activity_id, source_document, confidence, checksum, ...}` | | `SourceReference` | Rich source pointer: `{document, page, section, line, confidence, metadata}` | @@ -21,47 +28,65 @@ icon: "link" ## Getting Started -```python -from semantica.provenance import ProvenanceManager, SQLiteStorage + + + Zero configuration — fast, no disk writes. Use for notebooks, testing, and single-run scripts. -# In-memory (default) — no arguments required -manager = ProvenanceManager() + ```python + from semantica.provenance import ProvenanceManager, compute_checksum, verify_checksum -# SQLite — persists to disk -manager = ProvenanceManager(storage=SQLiteStorage("provenance.db")) + manager = ProvenanceManager() # InMemoryStorage by default -# Shorthand for SQLite — equivalent to the line above -manager = ProvenanceManager(storage_path="provenance.db") -``` + entry = manager.track_entity( + entity_id="apple_inc", + source="annual_report_2023.pdf", + source_location="Page 12, Section 3.1", + source_quote="Apple Inc. was incorporated on January 3, 1977.", + confidence=0.98, + ) -Track an entity with a source document, retrieve its lineage, then verify integrity: + print(entry.checksum) # SHA-256 hex auto-computed + print(verify_checksum(entry)) # True — tamper detection + ``` -```python -from semantica.provenance import ProvenanceManager, compute_checksum, verify_checksum + + In-memory storage is lost when the process exits. Use `SQLiteStorage` for anything that needs to survive restarts. + + + + Persists provenance to a local SQLite file. Use for production pipelines and audit trails. -manager = ProvenanceManager() + ```python + from semantica.provenance import ProvenanceManager, SQLiteStorage -# track_entity returns a ProvenanceEntry -entry = manager.track_entity( - entity_id="apple_inc", - source="annual_report_2023.pdf", - source_location="Page 12, Section 3.1", - source_quote="Apple Inc. was incorporated on January 3, 1977.", - confidence=0.98, -) + # Option 1: explicit storage instance + manager = ProvenanceManager(storage=SQLiteStorage("provenance.db")) -print(entry.entity_id) # "apple_inc" -print(entry.source_document) # "annual_report_2023.pdf" -print(entry.confidence) # 0.98 -print(entry.checksum) # SHA-256 hex string (set automatically) + # Option 2: shorthand — equivalent to above + manager = ProvenanceManager(storage_path="provenance.db") -# Verify the entry has not been tampered with -is_valid = verify_checksum(entry) -print(is_valid) # True -``` + entry = manager.track_entity( + entity_id="apple_inc", + source="annual_report_2023.pdf", + source_location="Page 12, Section 3.1", + confidence=0.98, + ) + + # Retrieve lineage after restart — entries persist in provenance.db + lineage = manager.get_lineage("apple_inc") + print(f"{len(lineage)} provenance entries for apple_inc") + ``` + + + The SQLite file is created automatically on first write. No schema setup required. + + + ## ProvenanceManager +**`ProvenanceManager`** is the central tracker for all lineage data. Every call to `track_entity`, `track_relationship`, or `track_chunk` automatically computes and stores a **SHA-256 checksum** for tamper detection. + ### Constructor ```python @@ -185,7 +210,7 @@ cleared = manager.clear() ### ProvenanceManager Methods Reference | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `track_entity(entity_id, source, metadata, **kwargs)` | `ProvenanceEntry` | Record entity provenance; checksum set automatically | | `track_relationship(relationship_id, source, metadata, **kwargs)` | `ProvenanceEntry` | Record relationship provenance | | `track_chunk(chunk_id, source_document, ...)` | `ProvenanceEntry` | Record chunk provenance with char offsets | @@ -465,7 +490,7 @@ print(lineage["source_documents"]) Provenance tracking in Semantica produces the following audit artifacts: | Standard | Available | -| -------- | --------- | +| :-------- | :--------- | | **W3C PROV-O** | Compliant data model; `to_dict()` and `from_dict()` for serialization | | **HIPAA** | Audit trail: entity → source document → timestamp → confidence | | **SOX** | Tamper-evident checksums; timestamps on every entry | diff --git a/docs/reference/reasoning.md b/docs/reference/reasoning.md index 96b304af..779093fe 100644 --- a/docs/reference/reasoning.md +++ b/docs/reference/reasoning.md @@ -4,12 +4,19 @@ description: "Forward chaining, Rete, deductive, abductive, SPARQL, Datalog, and icon: "microchip" --- -`semantica.reasoning` derives new knowledge from existing facts using logical rules. Every engine produces explainable inference paths — traceable chains of rules and facts, not black-box conclusions. +`semantica.reasoning` derives new knowledge from existing facts using logical rules: + +- Six reasoning engines: forward chaining, Rete, SPARQL, Datalog, temporal, and LLM-powered GraphReasoner +- Every engine produces explainable inference paths — traceable chains of rules and facts +- `DatalogReasoner` guarantees termination via semi-naive fixpoint evaluation +- `TemporalReasoningEngine` implements all 13 Allen interval algebra relations +- `ExplanationGenerator` produces step-by-step natural-language justifications + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `Reasoner` | Forward-chaining inference: `add_fact`, `add_rule`, `forward_chain`, `backward_chain`, `infer_facts` | | `GraphReasoner` | LLM-powered reasoning over a KG dict — answers natural language queries via `reason(graph, query)` | | `ReteEngine` | Rete pattern matching: `build_network`, `add_fact`, `match_patterns`, `execute_matches` | @@ -21,6 +28,31 @@ icon: "microchip" | `Fact` | Working-memory fact: `{fact_id, predicate, arguments}` | | `InferenceResult` | Single derived conclusion: `{conclusion, rule_used, premises, confidence}` | + +## Which Engine Should I Use? + + + + IF/THEN rules, forward and backward chaining. **Start here** — covers 90% of use cases. No query language required. + + + Natural language queries over a knowledge graph via LLM. No SPARQL or rules — just ask a question. + + + Recursive Horn clause rules with guaranteed termination. Use for complex multi-hop transitive rules. + + + Rete pattern matching for high-frequency inference. Use when you need to match many facts against many rules simultaneously. + + + SPARQL query expansion and rule-based inference. Use when you're working with RDF/OWL data. + + + All 13 Allen interval algebra relations. Use for time-aware reasoning: overlaps, before/after, during, contains. + + + + ## Getting Started The most common pattern is the `Reasoner` for IF/THEN forward-chaining: @@ -62,9 +94,10 @@ rule = Rule( reasoner.add_rule(rule) ``` + ## Reasoner (Forward/Backward Chaining) -The unified entry point for rule-based inference: +**`Reasoner`** is the unified entry point for rule-based inference — iterates facts and rules to a **fixpoint**, then optionally proves a specific goal via backward chaining: ```python from semantica.reasoning import Reasoner, Rule, RuleType, InferenceResult @@ -102,7 +135,7 @@ conclusions = reasoner.infer_facts( ### Reasoner Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `add_fact(fact)` | `None` | Add a string, entity dict, or relationship dict to working memory | | `add_rule(rule)` | `Rule` | Add a `Rule` object or IF-THEN string; rules are sorted by `priority` descending | | `forward_chain()` | `List[InferenceResult]` | Derive all possible conclusions iteratively until fixpoint | @@ -137,9 +170,10 @@ fact = Fact( ) ``` + ## GraphReasoner -LLM-powered reasoning that answers natural language queries over a knowledge graph dict: +**`GraphReasoner`** uses an LLM to answer **natural language queries** over a knowledge graph dict — no SPARQL or rule authoring required: ```python from semantica.reasoning import GraphReasoner @@ -166,6 +200,7 @@ print(answer) `reason()` converts the graph to a text context and calls the LLM with a structured prompt. Returns a plain string answer. + ## ReteEngine High-performance Rete pattern matching for large rule sets: @@ -204,7 +239,7 @@ engine.reset() ### ReteEngine Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `build_network(rules)` | `None` | Build the Rete network from a list of `Rule` objects | | `add_fact(fact)` | `None` | Add a `Fact` to working memory and propagate through the network | | `match_patterns(facts)` | `List[Match]` | Match all patterns; optionally add facts before matching | @@ -212,9 +247,10 @@ engine.reset() | `reset()` | `None` | Clear facts and all node activation state | | `get_network_stats()` | `dict` | Return counts of alpha, beta, terminal nodes and facts | + ## SPARQLReasoner -Query-based inference with rule expansion for triplet stores: +**`SPARQLReasoner`** extends SPARQL with **inference rule expansion** — add IF-THEN rules and they are automatically woven into queries before execution: ```python from semantica.reasoning import SPARQLReasoner @@ -257,6 +293,7 @@ SPARQLReasoner( `execute_query()` returns empty bindings when no `triplet_store` is configured. Pass a `TripletStore` instance via the `triplet_store=` kwarg to execute queries against a live backend. + ## DatalogReasoner Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed** — the engine detects fixpoint convergence and stops: @@ -304,7 +341,7 @@ fact = DatalogFact(predicate="parent", args=("alice", "bob")) ### DatalogReasoner Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `add_fact(fact)` | `None` | Add string, dict, or `DatalogFact`; constants must be lowercase-starting | | `add_rule(rule_str)` | `None` | Parse and add a Horn clause string like `"ancestor(X,Y) :- parent(X,Y)."` | | `derive_all()` | `List[str]` | Run semi-naive fixpoint evaluation; returns all facts as strings | @@ -312,6 +349,7 @@ fact = DatalogFact(predicate="parent", args=("alice", "bob")) | `load_from_graph(graph)` | `int` | Load a ContextGraph's nodes/edges as Datalog facts; returns count added | | `clear()` | `None` | Clear all facts and rules | + ## TemporalReasoningEngine Pure-Python Allen interval algebra — all 13 relations, no LLM calls: @@ -340,7 +378,7 @@ engine.active_at(ceo_tenure, datetime(2005, 6, 1)) # True All 13 Allen interval algebra relations: | Relation | Meaning | -| -------- | ------- | +| :-------- | :------- | | `BEFORE` | A ends before B starts | | `MEETS` | A ends exactly when B starts | | `OVERLAPS` | A starts before B, ends inside B | @@ -359,6 +397,7 @@ All 13 Allen interval algebra relations: `TemporalInterval.start` expects a `datetime` object, not a string. Import `datetime` from the standard library and construct intervals with `datetime(year, month, day)`. + ## ExplanationGenerator Generate structured explanations for any `InferenceResult`: @@ -396,7 +435,7 @@ print(justification.explanation_text) ### ExplanationGenerator Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `generate_explanation(reasoning)` | `Explanation` | Generate structured explanation for an `InferenceResult`, `Proof`, or abductive result | | `show_reasoning_path(reasoning)` | `ReasoningPath` | Extract and return the reasoning path from any result | | `justify_conclusion(conclusion, path)` | `Justification` | Build a `Justification` with evidence and NL text for a conclusion | @@ -420,10 +459,11 @@ step.output_fact # Any step.confidence # float ``` + ## Engine Selection Guide | Engine | Best For | Termination | Key Method | -| ------ | -------- | ----------- | ---------- | +| :------ | :-------- | :----------- | :---------- | | `Reasoner` | Simple IF/THEN rules | Always (with `max_iterations` cap) | `forward_chain()` | | `GraphReasoner` | NL queries over a KG via LLM | Always | `reason(graph, query)` | | `ReteEngine` | Large rule sets with many facts | Always | `match_patterns()` | diff --git a/docs/reference/seed.md b/docs/reference/seed.md index 87367aa6..844750b0 100644 --- a/docs/reference/seed.md +++ b/docs/reference/seed.md @@ -4,12 +4,19 @@ description: "Bootstrap Knowledge Graphs from verified, structured sources — t icon: "database" --- -`semantica.seed` gives your knowledge graph a reliable starting point. Rather than building from an empty graph and hoping extraction produces consistent reference data, you load verified, structured sources first — ISO codes, employee rosters, product catalogs, domain taxonomies — then merge freshly extracted data on top. +**`semantica.seed`** gives your knowledge graph a **reliable, verified starting point**: + +- Load verified reference data first — ISO codes, employee rosters, product catalogs, domain taxonomies +- `SeedDataManager` merges freshly extracted data onto foundation nodes without creating duplicates +- Supports JSON, CSV, and programmatic registration of seed sources +- Deterministic test graph generation from structured seed data +- Anchors entity extraction to known entities, reducing hallucination and duplicate nodes + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `SeedDataManager` | Coordinator: `register_source`, `load_source`, `create_foundation_graph`, `integrate_with_extracted` | | `SeedDataSource` | Config dataclass: `{name, format, location, entity_type, verified, version, metadata}` | | `SeedData` | Container dataclass: `{entities, relationships, properties, metadata}` | @@ -171,7 +178,7 @@ icon: "database" ## SeedDataManager Reference | Method | Description | -| ------ | ----------- | +| :------ | :----------- | | `register_source(name, format, location, **config)` | Add a new data source to the manager | | `load_source(source_name)` | Load and return raw data from a registered source | | `create_foundation_graph()` | Build the initial graph from all registered sources | diff --git a/docs/reference/semantic_extract.md b/docs/reference/semantic_extract.md index 70c7239d..3a1d27ca 100644 --- a/docs/reference/semantic_extract.md +++ b/docs/reference/semantic_extract.md @@ -4,7 +4,14 @@ description: "Named entity recognition, relation extraction, event detection, an icon: "magnifying-glass-chart" --- -`semantica.semantic_extract` extracts structured information from unstructured text — the foundation of every knowledge graph in Semantica. All extractors support three modes: pattern-based (no API key), ML-based, and LLM-based. +`semantica.semantic_extract` extracts structured information from unstructured text — the foundation of every knowledge graph in Semantica: + +- `NERExtractor` — named entity recognition with confidence scores and source attribution +- `RelationExtractor` — typed relationship extraction (`founded_by`, `located_in`, and custom types) +- `TripletExtractor` — direct `(subject, predicate, object)` triplet generation for RDF output +- `EventDetector` — event detection with participants, temporal context, and confidence +- Three extraction modes on every extractor: `"pattern"` (no API key), `"huggingface"`, `"llm"` + ## Getting Started @@ -50,14 +57,15 @@ ner = NERExtractor(method="llm", llm_provider=llm) entities = ner.extract("Apple Inc. was founded by Steve Jobs.") ``` + ## Exported Classes - `NamedEntityRecognizer` is the high-level coordinator with confidence thresholding and overlap merging. `NERExtractor` is the lower-level implementation. For most use cases, start with `NERExtractor` for simplicity or `NamedEntityRecognizer` for fine-grained control. + **`NamedEntityRecognizer`** is the high-level coordinator with confidence thresholding and overlap merging. **`NERExtractor`** is the lower-level implementation. For most use cases, start with `NERExtractor` for simplicity or `NamedEntityRecognizer` for fine-grained control. | Class | Role | -| --- | --- | +| :--- | :--- | | `NamedEntityRecognizer` | High-level NER with confidence thresholding and overlap merging | | `NERExtractor` | Core NER implementation — use directly for simplicity | | `RelationExtractor` | Typed relationship extraction (`founded_by`, `located_in`, ...) | @@ -70,35 +78,100 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.") ## Method Selection Guide -Choose the right extraction method based on your requirements: + + + Zero dependencies, no API key required. Uses spaCy rules and regex to match standard entity types. -| Priority | Method | Setup Required | API Cost | Accuracy | Use Cases | -|----------|--------|----------------|----------|----------|-----------| -| **Speed** | `pattern` | None | Free | Good | Quick prototyping, known entity types | -| **Custom Models** | `huggingface` | Model download | Free | Varies | Domain-specific models, fine-tuned NER | -| **Best Accuracy** | `llm` | API key | $$ | Highest | Complex schemas, custom entity types | + | | | + | :-- | :-- | + | **Setup** | None — works out of the box | + | **Cost** | Free | + | **Accuracy** | Good for standard entity types | + | **Best for** | Quick prototyping, batch processing, air-gapped systems | -### Quick Recommendations + ```python + from semantica.semantic_extract import NERExtractor, RelationExtractor -```python -# No setup required -ner = NERExtractor(method="pattern") + ner = NERExtractor(method="pattern") + entities = ner.extract("Apple Inc. was founded by Steve Jobs in Cupertino.") -# Best accuracy - requires API key -from semantica.llms import Groq -import os -llm = Groq(api_key=os.getenv("GROQ_API_KEY")) -ner = NERExtractor(method="llm", llm_provider=llm) + rel = RelationExtractor(method="pattern") + relationships = rel.extract(text, entities=entities) + ``` + + + Use any pre-trained or fine-tuned transformer model. Free inference, runs locally. -# Custom models - domain-specific -ner = NERExtractor(method="huggingface") -entities = ner.extract(text, model="dslim/bert-base-NER", device="cpu") -``` + | | | + | :-- | :-- | + | **Setup** | `pip install semantica[models-huggingface]` | + | **Cost** | Free (local compute) | + | **Accuracy** | Excellent for domain-specific NER | + | **Best for** | Medical NER, custom fine-tunes, no API cost | + + ```python + from semantica.semantic_extract import NERExtractor + + ner = NERExtractor(method="huggingface") + + # Pass model per-call + entities = ner.extract(text, model="dslim/bert-base-NER", device="cpu") + + # Biomedical NER + entities = ner.extract(text, model="d4data/biomedical-ner-all") + ``` + + + Highest accuracy for complex schemas and custom entity types. Requires an LLM API key. + + | | | + | :-- | :-- | + | **Setup** | `pip install semantica[llm-groq]` + API key | + | **Cost** | Depends on provider | + | **Accuracy** | Highest — handles complex types and context | + | **Best for** | Production, custom entity types, complex relation schemas | + + ```python + import os + from semantica.llms import Groq + from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor + + llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY")) + + ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3) + rel = RelationExtractor(method="llm", llm_provider=llm) + trip = TripletExtractor(method="llm", llm_provider=llm) + + entities = ner.extract(text) + relationships = rel.extract(text, entities=entities) + triplets = trip.extract(text) + ``` + + + Try methods in priority order — guarantees non-empty results even when the preferred method is unavailable. + + ```python + from semantica.semantic_extract import NERExtractor, RelationExtractor + + # Try LLM first, fall back to pattern on error + ner = NERExtractor(method=["llm", "pattern"]) + rel = RelationExtractor(method=["llm", "pattern"]) + + # Always returns results — safe for production pipelines + entities = ner.extract(text) + relationships = rel.extract(text, entities=entities) + ``` + + + Use fallback chains in pipelines where API availability isn't guaranteed (rate limits, network issues). The first method in the list is always tried first. + + + ### Method Availability by Extractor | Extractor | `pattern` | `huggingface` | `llm` | Notes | -|-----------|-----------|---------------|-------|-------| +| :----------- | :----------- | :--------------- | :------- | :------- | | `NERExtractor` | ✅ | ✅ | ✅ | Full method support | | `RelationExtractor` | ✅ | ✅ | ✅ | Also supports `dependency`, `cooccurrence` | | `TripletExtractor` | ✅ | ✅ | ✅ | Also supports `rules` method | @@ -118,6 +191,7 @@ trip = TripletExtractor(method=["llm", "pattern"]) entities = ner.extract(text) ``` + ## Quick Start ```python @@ -135,13 +209,15 @@ triplets = TripletExtractor(method="llm", llm_provider=llm).extract(text) Semantic extraction pipeline: raw text fans into NER, Relation, and Coreference extractors, then merges into a Triplet Generator + ## Extractor Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `extract(text)` | `List[Entity]` / `List[Relation]` / `List[Triplet]` / `List[Event]` | Extract from single text input | | `extract(texts)` | `List[List[...]]` | Process multiple texts (batch detected automatically) | + ## NERExtractor ```python @@ -207,7 +283,14 @@ Output format: ] ``` -Available methods: `"pattern"` (pattern-based), `"dependency"` (spaCy parsing), `"cooccurrence"` (proximity-based), `"huggingface"` (custom models), `"llm"`. +Available methods: + +- `"pattern"` — rule-based pattern matching +- `"dependency"` — spaCy dependency parsing +- `"cooccurrence"` — proximity-based co-occurrence +- `"huggingface"` — custom models +- `"llm"` — highest accuracy, requires API key + ## TripletExtractor @@ -223,6 +306,7 @@ triplets = trip.extract(text) Triplets are suitable for loading directly into a triplet store or knowledge graph. + ## EventDetector Detect events with participants and temporal context: @@ -241,7 +325,14 @@ for event in events: print(f"Confidence: {event.confidence:.2f}") ``` -Output fields per event: `type`, `participants` (with roles), `temporal`, `location`, and `confidence`. +Output fields per event: + +- `type` — event category (e.g. `"founding"`, `"acquisition"`) +- `participants` — list of entities with roles +- `temporal` — date or time reference +- `location` — location entity (when present) +- `confidence` — extraction confidence score + ## CoreferenceResolver @@ -257,6 +348,7 @@ resolved_text = resolver.resolve( # "Apple Inc." replaces "The company" for consistent downstream extraction ``` + ## Batch Processing All extractors automatically detect batch input and process multiple texts efficiently: @@ -313,7 +405,7 @@ triplets = trip.extract(text) ## Extraction Method Comparison | Method | Speed | Cost | Accuracy | Custom Types | -| ------ | ----- | ---- | -------- | ------------ | +| :------ | :----- | :---- | :-------- | :------------ | | `pattern` | Very fast | Free | Medium | Yes (dictionary) | | `ml` | Fast | Free | High | Limited | | `llm` | Medium | API cost | Highest | Yes (schema) | diff --git a/docs/reference/split.md b/docs/reference/split.md index 71e70f6a..6b380cec 100644 --- a/docs/reference/split.md +++ b/docs/reference/split.md @@ -4,7 +4,14 @@ description: "Text chunking with recursive, semantic, entity-aware, relation-awa icon: "scissors" --- -`semantica.split` breaks documents into chunks that preserve semantic context. Chunking quality directly determines downstream accuracy — a poorly chunked document produces bad embeddings, missed entities, and broken relation triplets. Use the right strategy for your content type and pipeline goal. +**`semantica.split`** breaks documents into chunks that **preserve semantic context**: + +- Six chunking strategies: recursive, semantic, entity-aware, relation-aware, sliding window, structural +- `SemanticChunker` uses embedding-based topic-shift detection to split only when content changes +- `EntityAwareChunker` keeps entity mentions intact across chunk boundaries +- `RelationAwareChunker` keeps subject-predicate-object triplets within a single chunk +- Chunking quality directly determines downstream embedding accuracy and entity extraction quality + ## Why Chunking Matters @@ -19,7 +26,7 @@ Semantica's chunking methods are designed to avoid these failure modes. ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `TextSplitter` | Unified entry point — swap `method=` without changing downstream code | | `Chunk` | `{text, start_index, end_index, metadata, id}` | | `SemanticChunker` | Embedding-based topic-shift detection — splits only when content actually changes | @@ -31,7 +38,7 @@ Semantica's chunking methods are designed to avoid these failure modes. **Available `method=` values for `TextSplitter`:** | Method | Best for | -| --- | --- | +| :--- | :--- | | `recursive` | General text — splits on paragraphs, sentences, words in order | | `sentence` | Conversational text, QA | | `paragraph` | Long-form text where paragraph integrity matters | @@ -118,7 +125,7 @@ Semantica's chunking methods are designed to avoid these failure modes. ## Splitting Methods | Method | How It Splits | Best For | -| ------ | ------------- | -------- | +| :------ | :------------- | :-------- | | `recursive` | Paragraph → sentence → word (cascading fallback) | General-purpose default | | `semantic_transformer` | Embeds sentences, splits at cosine similarity drops | RAG — topic coherence matters | | `entity_aware` | Adjusts boundaries so entity spans are never cut | NER pipelines | @@ -162,7 +169,7 @@ splitter = TextSplitter( ``` | Parameter | Type | Default | Description | -| --------- | ---- | ------- | ----------- | +| :--------- | :---- | :------- | :----------- | | `method` | `str \| list[str]` | `"recursive"` | Chunking strategy, or list of methods as fallback chain | | `chunk_size` | `int` | `1000` | Target size in **characters** (not tokens — if you were using token-based sizing before, multiply by ~4 to approximate the same boundary) | | `chunk_overlap` | `int` | `200` | Character overlap between adjacent chunks | @@ -309,7 +316,7 @@ class Chunk: Metadata keys vary by method. Only keys that are actually set by the implementation are listed. | Field | Type | Set by | Description | -| ----- | ---- | ------ | ----------- | +| :----- | :---- | :------ | :----------- | | `method` | `str` | all methods | Splitting method that produced this chunk | | `chunk_size` | `int` | most methods | Character length of this chunk | | `sentence_count` | `int` | `sentence`, `semantic_transformer`, spaCy path | Number of sentences in this chunk | @@ -331,7 +338,7 @@ Metadata keys vary by method. Only keys that are actually set by the implementat The `token` method accepts a `tokenizer=` kwarg that is passed to `tiktoken.encoding_for_model()`. The value should be a tiktoken model name. Unrecognised names fall back to `cl100k_base` automatically. | Value | Encoding used | -| ----- | ------------- | +| :----- | :------------- | | `"gpt-4"` (default) | `cl100k_base` | | `"gpt-3.5-turbo"` | `cl100k_base` | | `"text-embedding-ada-002"` | `cl100k_base` | diff --git a/docs/reference/triplet_store.md b/docs/reference/triplet_store.md index a7d06a60..937613d9 100644 --- a/docs/reference/triplet_store.md +++ b/docs/reference/triplet_store.md @@ -4,14 +4,14 @@ description: "RDF triple storage with SPARQL queries and bulk loading — Blazeg icon: "table" --- -`semantica.triplet_store` provides W3C-standard RDF storage with SPARQL query support. Use it when you need semantic web compatibility, OWL-style reasoning, SPARQL-based queries, or standards-compliant RDF serialization. +`semantica.triplet_store` provides **W3C-standard RDF storage** with **SPARQL 1.1** query support. Use it when you need **semantic web compatibility**, OWL-style reasoning, SPARQL-based queries, or standards-compliant RDF serialization. ## Exported Classes | Class | Role | -| --- | --- | +| :---- | :--- | | `TripletStore` | Unified interface: `add_triplet`, `add_triplets`, `get_triplets`, `delete_triplet`, `execute_query` | -| `QueryEngine` | SPARQL 1.1 execution with query optimization and result caching | +| `QueryEngine` | **SPARQL 1.1** execution with query optimization and result caching | | `BulkLoader` | High-volume RDF loading with batching, retries, and progress tracking | | `BlazegraphStore` | Blazegraph REST API — SPARQL 1.1 Update, namespace management | | `JenaStore` | Apache Jena — rdflib-backed, SPARQL read support via remote endpoint | @@ -42,7 +42,7 @@ icon: "table" ## Getting Started -`TripletStore` wraps the backend of your choice. Construct a `Triplet` object (from `semantica.semantic_extract.types`) and call `add_triplet()`: +**`TripletStore`** wraps the backend of your choice. Construct a `Triplet` object (from `semantica.semantic_extract.types`) and call `add_triplet()`: ```python from semantica.triplet_store import TripletStore @@ -147,7 +147,7 @@ for row in result.bindings: ) ``` - Best for: Wikidata-style workloads, high triple counts, named graph support, SPARQL 1.1 Update. + **Best for:** Wikidata-style workloads, high triple counts, named graph support, SPARQL 1.1 Update. ```bash @@ -161,7 +161,7 @@ for row in result.bindings: ) ``` - Best for: local development with rdflib, SPARQL read queries against a Fuseki endpoint. + **Best for:** local development with rdflib, SPARQL read queries against a Fuseki endpoint. **Note on inference:** `JenaStore` accepts `enable_inference=True` in config but OWL reasoning is a placeholder and does not produce inferred triples in the current implementation. @@ -178,12 +178,12 @@ for row in result.bindings: ) ``` - Best for: Eclipse Foundation deployments, transaction-based loading via REST API. + **Best for:** Eclipse Foundation deployments, transaction-based loading via REST API. | Backend | License | Named Graphs | Write via | Best For | - | ------- | ------- | ------------ | --------- | -------- | + | :------- | :------- | :------------ | :--------- | :-------- | | Blazegraph | Open source | Yes | SPARQL Update REST | High triple count, SPARQL 1.1 | | Apache Jena | Apache 2.0 | No (rdflib backend) | rdflib in-process | Local dev, read queries | | RDF4J | Eclipse 1.0 | Yes | REST API N-Triples | Enterprise Java, transactions | @@ -208,17 +208,17 @@ t = Triplet( ``` | Field | Type | Default | Description | -| ----- | ---- | ------- | ----------- | -| `subject` | `str` | required | Subject URI | -| `predicate` | `str` | required | Predicate URI | -| `object` | `str` | required | Object URI or literal | +| :----- | :---- | :------- | :----------- | +| `subject` | `str` | **required** | Subject URI | +| `predicate` | `str` | **required** | Predicate URI | +| `object` | `str` | **required** | Object URI or literal | | `confidence` | `float` | `1.0` | Confidence score (0–1) | | `metadata` | `dict` | `{}` | Arbitrary metadata | ## TripletStore Methods | Method | Returns | Description | -| ------ | ------- | ----------- | +| :------ | :------- | :----------- | | `add_triplet(triplet)` | `dict` | Add a single `Triplet` object | | `add_triplets(triplets, batch_size)` | `dict` | Bulk-add a list of `Triplet` objects; returns `{"success", "total", "processed", "failed", "batches"}` | | `get_triplets(subject, predicate, object)` | `List[Triplet]` | Retrieve triplets matching subject/predicate/object filters | @@ -271,7 +271,7 @@ store.execute_query(""" ### QueryResult fields | Field | Type | Description | -| ----- | ---- | ----------- | +| :----- | :---- | :----------- | | `bindings` | `List[dict]` | Each dict maps variable name → `{"value": ..., "type": ...}` | | `variables` | `List[str]` | SPARQL result variable names | | `execution_time` | `float` | Seconds elapsed | diff --git a/docs/reference/utils.md b/docs/reference/utils.md index 7147622b..b0b871ef 100644 --- a/docs/reference/utils.md +++ b/docs/reference/utils.md @@ -4,12 +4,20 @@ description: "Shared utilities for logging, validation, error handling, progress icon: "wrench" --- -`semantica.utils` provides shared infrastructure used throughout Semantica. Most users won't call it directly, but its APIs are available when you need fine-grained control over logging, validation, progress tracking, or error handling. +**`semantica.utils`** provides **shared infrastructure** used throughout Semantica: + +- Structured logging: `setup_logging()`, `get_logger()`, `log_execution_time` decorator +- Validation helpers: `validate_entity()` and `validate_config()` return `(bool, Optional[str])` without raising +- Progress tracking: `ProgressTracker` class and `track_progress()` iterable wrapper with ETA +- Typed exceptions: `SemanticaError`, `ValidationError`, `ProcessingError`, `ConfigurationError`, `QualityError` + +Most users won't call utils directly — it's the **shared foundation** for all modules. + ## Exported Classes | Name | Type | Role | -| --- | --- | --- | +| :--- | :--- | :--- | | `setup_logging` | function | Configure the `semantica` root logger — accepts `level`, `file`, `console`, `rotation` kwargs | | `get_logger` | function | Get a named logger instance (`semantica.`) | | `log_execution_time` | decorator | Wraps a function — logs name, execution time, and success/failure | @@ -96,9 +104,9 @@ if not is_valid: ``` | Function | Description | Returns | -| -------- | ----------- | ------- | -| `validate_entity(data)` | Check entity dict has required fields (`id`, `text`, `type`) and correct types | `Tuple[bool, Optional[str]]` | -| `validate_config(cfg, required_keys=None)` | Check configuration dict; optionally enforce required keys | `Tuple[bool, Optional[str]]` | +| :-------- | :----------- | :------- | +| `validate_entity(data)` | Check entity dict has **required** fields (`id`, `text`, `type`) and correct types | `Tuple[bool, Optional[str]]` | +| `validate_config(cfg, required_keys=None)` | Check configuration dict; optionally enforce **required** keys | `Tuple[bool, Optional[str]]` | ## Progress Tracking @@ -178,7 +186,7 @@ except SemanticaError as e: ``` | Exception | When Raised | Key Attributes | -| --------- | ----------- | -------------- | +| :--------- | :----------- | :-------------- | | `SemanticaError` | Base class — all framework errors inherit from this | `.message`, `.context`, `.error_code` | | `ValidationError` | Input data failed schema or type validation | `.field`, `.value`, `.constraint` | | `ProcessingError` | Failure during extraction, graph build, or pipeline step | `.stage`, `.input_data`, `.output_data` | diff --git a/docs/reference/vector_store.md b/docs/reference/vector_store.md index 9f67a101..004ed114 100644 --- a/docs/reference/vector_store.md +++ b/docs/reference/vector_store.md @@ -4,12 +4,19 @@ description: "Unified interface for FAISS, Pinecone, Weaviate, Qdrant, Milvus, a icon: "database" --- -`semantica.vector_store` provides a unified API for storing and searching vector embeddings across all major backends. Swap backends with a one-line change — no application code changes needed. +`semantica.vector_store` provides a unified API for storing and searching vector embeddings across all major backends: + +- Swap backends with a one-line change — no application code changes needed +- `HybridSearch` fuses dense vector similarity with metadata filtering via RRF or weighted average +- `NamespaceManager` for multi-tenant structural isolation +- `FAISSStore` with flat, ivf, hnsw, and pq index types +- Batch embed and store with parallel workers; metadata update without re-embedding + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `VectorStore` | Unified interface: `store_vectors`, `search_vectors`, `update_vectors`, `delete_vectors` | | `HybridSearch` | Fuses dense vector similarity with metadata filtering via RRF or weighted average | | `MetadataFilter` | Chainable filter builder: `.eq("type", "person").gt("year", 2020).in_list("tag", [...])` | @@ -27,28 +34,41 @@ icon: "database" - Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, and PgVector. + - Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector + - One-line backend swap — no application code changes + - `add_documents()` auto-embeds; `store_vectors()` for pre-computed embeddings - Combine dense vector similarity with metadata filtering and configurable fusion strategies. + - Dense vector similarity with metadata filtering + - RRF or weighted-average fusion strategies + - Multi-source fusion across separate collections - Rich metadata indexing — query by field values, update fields without re-embedding. + - Rich metadata indexing by field values + - Update metadata fields without re-embedding + - OR and AND query operators - Multi-tenant namespace isolation — structural separation, not just metadata filters. + - Structural per-tenant namespace isolation + - Faster queries (smaller search space per tenant) + - Safer than metadata-filter-only separation - Bulk add, delete, and metadata updates — parallel embedding with configurable workers. + - Bulk add, delete, and metadata updates + - Parallel embedding with configurable `batch_size` and `workers` + - In-place vector updates without full re-indexing - flat, ivf, hnsw, and pq index types with full configuration control. + - flat, ivf, hnsw, and pq index types + - Full configuration control via `FAISSStore.create_index()` + - `save()` / `load()` for disk persistence + ## Getting Started -`VectorStore` is the main entry point. Use `"inmemory"` for development and `"faiss"` for local production: +**`VectorStore`** is the main entry point. Use `"inmemory"` for development and `"faiss"` for **local production**: ```python from semantica.vector_store import VectorStore @@ -233,7 +253,7 @@ store = VectorStore( ## Backend Selection Guide | Backend | Deployment | API Key | Persistence | Best For | -| ------- | ---------- | ------- | ----------- | -------- | +| :------- | :---------- | :------- | :----------- | :-------- | | `inmemory` | Process | No | No | Development, unit tests | | `faiss` | Local | No | Via `save()`/`load()` | On-premise, offline production | | `pinecone` | Cloud | Yes | Managed | Managed cloud, serverless | @@ -312,7 +332,7 @@ mf = ( ### MetadataFilter Methods | Method | Operator | Description | -| ------ | -------- | ----------- | +| :------ | :-------- | :----------- | | `.eq(field, value)` | `==` | Exact equality | | `.ne(field, value)` | `!=` | Not equal | | `.gt(field, value)` | `>` | Greater than | @@ -339,7 +359,7 @@ fused = ranker.rank([results_list_1, results_list_2], weights=[0.7, 0.3]) ``` | Fusion strategy | Description | -| --------------- | ----------- | +| :--------------- | :----------- | | `reciprocal_rank_fusion` | Rank-based combination via RRF — robust to score scale differences (default) | | `weighted_average` | Weighted sum of scores — pass `weights=[...]` to `rank()` | @@ -470,7 +490,7 @@ store.create_index(index_type="pq", metric="L2", m=8) ``` | Index | Memory | Speed | Accuracy | When to Use | -| ----- | ------ | ----- | -------- | ----------- | +| :----- | :------ | :----- | :-------- | :----------- | | `flat` | High | Slow | Exact (100%) | < 100K vectors, correctness critical | | `ivf` | Medium | Fast | ~95–98% | 100K–10M vectors, good balance | | `hnsw` | Medium-High | Very fast | ~97–99% | Low latency, production retrieval | diff --git a/docs/reference/visualization.md b/docs/reference/visualization.md index 7d1dada5..35d59320 100644 --- a/docs/reference/visualization.md +++ b/docs/reference/visualization.md @@ -4,14 +4,20 @@ description: "Interactive and static knowledge graph, ontology, embedding, and t icon: "chart-bar" --- -`semantica.visualization` renders knowledge graphs, ontologies, embedding spaces, and temporal data as interactive HTML or static images — without launching the full Explorer server. +**`semantica.visualization`** renders knowledge graphs, ontologies, embedding spaces, and temporal data as **interactive HTML or static images** — without launching the full Explorer server: + +- `KGVisualizer` — interactive network with force, hierarchical, and circular layouts +- `EmbeddingVisualizer` — 2D/3D UMAP or t-SNE projections with cluster labels +- `TemporalVisualizer` — timeline views and graph evolution across snapshots +- `AnalyticsVisualizer` — centrality scores, community structure, degree distribution charts Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` or `graphviz`. + ## Exported Classes | Class | Role | -| --- | --- | +| :--- | :--- | | `KGVisualizer` | Interactive network, community, and subgraph rendering with force/hierarchical/circular layouts | | `OntologyVisualizer` | Class hierarchy and property relationship diagrams from any ontology | | `EmbeddingVisualizer` | 2D/3D UMAP or t-SNE projection of embedding spaces with cluster labels | @@ -91,7 +97,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o **Layout options (`layout=`):** | Layout | Description | Best For | - | ------ | ----------- | -------- | + | :------ | :----------- | :-------- | | `force` | Physics simulation — clusters emerge naturally | General graphs | | `hierarchical` | Top-down tree layout | Taxonomies, org charts | | `circular` | Nodes on a circle, edges as chords | Small dense graphs | @@ -138,7 +144,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o ``` | Method | Speed | Preserves | Best For | - | ------ | ----- | --------- | -------- | + | :------ | :----- | :--------- | :-------- | | `umap` | Fast | Global + local structure | Large datasets, cluster discovery | | `tsne` | Medium | Local structure | Tight cluster separation | | `pca` | Very fast | Variance | Quick overview, linear structure | @@ -217,7 +223,7 @@ viz = KGVisualizer(color_scheme="vibrant") ``` | Scheme | Description | Best For | -| ------ | ----------- | -------- | +| :------ | :----------- | :-------- | | `default` | Blue-grey palette | General use | | `vibrant` | High-contrast, saturated colours | Presentations | | `pastel` | Soft, muted tones | Light backgrounds | @@ -228,7 +234,7 @@ viz = KGVisualizer(color_scheme="vibrant") ## Export Formats | Format | Interactive | Scalable | Best For | -| ------ | ----------- | -------- | -------- | +| :------ | :----------- | :-------- | :-------- | | `.html` | Yes | N/A | Web dashboards, exploratory analysis | | `.png` | No | No | Reports, Jupyter notebooks | | `.svg` | No | Yes | Publications, slide decks | diff --git a/docs/use-cases.md b/docs/use-cases.md index da3b0092..8e224bf5 100644 --- a/docs/use-cases.md +++ b/docs/use-cases.md @@ -4,128 +4,192 @@ description: "Real-world applications of Semantica across domains, with linked c icon: "briefcase" --- -Semantica is purpose-built for environments where AI outputs must be explainable, auditable, and traceable. The use cases below span regulated industries, advanced research, and high-stakes operational domains — each with linked Jupyter notebooks you can run today. +Semantica is purpose-built for environments where AI outputs must be explainable, auditable, and traceable. Every use case below includes linked Jupyter notebooks you can run today. -## At a Glance +## Browse by Sector -| Use Case | Domain | Difficulty | Estimated Time | -| -------- | ------ | ---------- | -------------- | -| Biomedical Knowledge Graphs | Healthcare | Intermediate | 1–2 hours | -| Financial Data Integration | Finance | Intermediate | 1–2 hours | -| Fraud Detection | Finance | Advanced | 2–3 hours | -| Blockchain Analytics | Finance | Intermediate | 1–2 hours | -| Cybersecurity Threat Intelligence | Security | Advanced | 2–3 hours | -| Criminal Network Analysis | Security / Intelligence | Intermediate | 1–2 hours | -| Intelligence Analysis Orchestrator | Intelligence | Intermediate | 1–2 hours | -| Supply Chain Optimization | Operations | Intermediate | 1–2 hours | -| Renewable Energy Management | Energy | Intermediate | 1–2 hours | -| GraphRAG | AI / LLM | Advanced | 1–2 hours | + + -**Difficulty levels:** + + + Connect genes, proteins, drugs, and diseases from scientific literature to accelerate drug discovery and understand disease mechanisms. -- **Beginner** — basic Semantica knowledge only, no domain expertise needed -- **Intermediate** — some domain knowledge helpful, uses 2–4 Semantica modules -- **Advanced** — domain expertise expected, uses advanced features (temporal graphs, multi-source pipelines, reasoning) + **Key modules:** `ingest` (PubMed RSS), `semantic_extract`, `kg`, `deduplication`, `context` -## Research & Science + **Notebooks:** + - [Drug Discovery Pipeline](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) — Intermediate + - [Genomic Variant Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) — Intermediate + + + Ground LLM answers in structured scientific literature with hybrid retrieval, logical inference, and source attribution on every claim. -### Biomedical Knowledge Graphs + **Key modules:** `context`, `vector_store`, `kg`, `reasoning`, `llms` -Connect genes, proteins, drugs, and diseases from scientific literature and databases to accelerate drug discovery and understand disease mechanisms. Semantica's temporal graphs and provenance tracking make every fact in the knowledge base traceable to its source publication. + **Notebooks:** + - [GraphRAG Complete](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — Advanced + - [RAG vs. GraphRAG Comparison](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) — Advanced + + -**Key modules:** `ingest` (PubMed RSS), `semantic_extract`, `kg`, `deduplication`, `context` + -**Cookbooks:** + -- [Drug Discovery Pipeline](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) — PubMed RSS ingestion, entity-aware chunking, GraphRAG, vector similarity search -- [Genomic Variant Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) — bioRxiv RSS, temporal KGs, deduplication, pathway analysis + + + Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph — with conflict detection when sources disagree. -## Finance & Trading + **Key modules:** `ingest` (API, MCP, stream), `normalize`, `kg`, `conflicts`, `provenance` -### Financial Data Integration + **Notebooks:** + - [Financial Data Integration (MCP)](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) — Intermediate + + + Detect complex fraud rings using temporal graphs and pattern detection over transaction, device, and user data. Temporal edges let you query: "what connections existed during this window?" -Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph — with conflict detection when sources disagree and full provenance back to each data feed. + **Key modules:** `kg` (temporal), `conflicts`, `reasoning`, `visualization` -**Key modules:** `ingest` (API, MCP, stream), `normalize`, `kg`, `conflicts`, `provenance` + **Notebooks:** + - [Fraud Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) — Advanced + + + Map transaction flows, analyze DeFi protocols, and detect illicit activity. Graph algorithms (centrality, community detection) surface high-risk actors that linear transaction analysis misses. -**Cookbook:** [Financial Data Integration (MCP)](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) — Alpha Vantage API, MCP servers, seed data, real-time ingestion + **Key modules:** `kg`, `reasoning`, `visualization` -### Fraud Detection + **Notebooks:** + - [DeFi Protocol Intelligence](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb) — Intermediate + - [Transaction Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb) — Intermediate + + -Detect complex fraud rings using temporal graphs and pattern detection over transaction, device, and user data. Temporal edges let you query: "what connections existed during this window?" — critical for reconstructing fraud timelines. + -**Key modules:** `kg` (temporal), `conflicts`, `reasoning`, `visualization` + -**Cookbook:** [Fraud Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) — temporal KGs, cycle detection, fraud pattern analysis + + + Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs for proactive defense. -### Blockchain Analytics + **Key modules:** `ingest` (stream, feed), `kg` (temporal), `context`, `reasoning`, `export` -Map transaction flows, analyze DeFi protocols, and detect illicit activity across wallet and exchange networks. Graph algorithms (centrality, community detection) surface high-risk actors that linear transaction analysis misses. + **Notebooks:** + - [Real-Time Anomaly Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) — Advanced + - [Threat Intelligence Hybrid RAG](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) — Advanced + + + Build knowledge graphs from police reports, court records, and OSINT feeds to identify key players, communities, and suspicious patterns. Network centrality surfaces actors text search alone would miss. -**Cookbooks:** + **Key modules:** `ingest`, `semantic_extract`, `kg`, `visualization` (community detection) -- [DeFi Protocol Intelligence](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb) -- [Transaction Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb) + **Notebooks:** + - [Criminal Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) — Intermediate + + + Process multiple intelligence sources in parallel with an orchestrator-worker pipeline. Multi-source conflict detection flags disagreements rather than silently discarding minority reports. -## Security & Intelligence + **Key modules:** `pipeline`, `ingest`, `conflicts`, `provenance`, `export` -### Cybersecurity Threat Intelligence + **Notebooks:** + - [Intelligence Analysis Orchestrator-Worker](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb) — Intermediate + + -Ingest threat feeds (CVE databases, security RSS), detect anomalies in streaming data, and build threat intelligence knowledge graphs for proactive defense. Real-time streaming ingestion with temporal provenance means every threat event is timestamped and traceable. + -**Key modules:** `ingest` (stream, feed), `kg` (temporal), `context`, `reasoning`, `export` + -**Cookbooks:** + + + Map suppliers, logistics routes, inventory levels, and delivery relationships to identify bottlenecks and optimize global supply chains. Graph path-finding reveals indirect dependencies spreadsheets miss. -- [Real-Time Anomaly Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) -- [Threat Intelligence Hybrid RAG](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) + **Key modules:** `ingest`, `kg`, `reasoning`, `visualization`, `export` (Parquet for analytics) -### Criminal Network Analysis + **Notebooks:** + - [Supply Chain Data Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb) — Intermediate + + + Connect sensor data, weather forecasts, and maintenance logs to predict equipment failures and optimize grid operations. Temporal graphs track asset states over time and correlate maintenance events with performance degradation. -Build knowledge graphs from police reports, court records, and OSINT feeds to identify key players, communities, and suspicious patterns. Network centrality analysis (PageRank, betweenness) surfaces actors that text search alone would miss. + **Key modules:** `ingest` (stream, API), `kg` (temporal), `reasoning`, `visualization` -**Key modules:** `ingest`, `semantic_extract`, `kg`, `visualization` (community detection) + **Notebooks:** + - [Energy Market Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb) — Intermediate + + -**Cookbook:** [Criminal Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) + + -### Intelligence Analysis Orchestrator -Process multiple intelligence sources in parallel using an orchestrator-worker pipeline pattern with multi-source conflict detection and resolution. When sources disagree on the same fact, Semantica flags and resolves rather than silently discarding. +## Compliance Footprint by Domain -**Key modules:** `pipeline`, `ingest`, `conflicts`, `provenance`, `export` + -**Cookbook:** [Intelligence Analysis Orchestrator-Worker](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb) + -## Industry & Operations +**Regulatory requirements:** HIPAA, FDA 21 CFR Part 11 -### Supply Chain Optimization +| Semantica capability | Compliance role | +| :-------------------- | :-------------- | +| W3C PROV-O provenance | Full lineage from raw data to inference — required for FDA audit trails | +| SHA-256 checksums | Tamper detection on every snapshot — supports electronic record integrity | +| Decision tracking | Every AI-assisted recommendation is recorded with causal chain and confidence | +| Temporal graphs | Point-in-time queries for retrospective safety analysis | +| SHACL validation | Schema enforcement before data enters the knowledge graph | -Map suppliers, logistics routes, inventory levels, and delivery relationships to identify bottlenecks and optimize global supply chains. Graph path-finding reveals indirect dependencies that spreadsheet analysis cannot. + -**Key modules:** `ingest`, `kg`, `reasoning`, `visualization`, `export` (Parquet for analytics) + -**Cookbook:** [Supply Chain Data Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb) +**Regulatory requirements:** SOX, MiFID II, GDPR, Basel III -### Renewable Energy Management +| Semantica capability | Compliance role | +| :-------------------- | :-------------- | +| Decision audit trail | Full record of model decisions with reasoning — required for model risk management | +| Conflict detection | Flags when two sources disagree on a valuation or risk figure | +| Version control | SHA-256 snapshot history — supports point-in-time reconstruction for audits | +| Provenance export | RDF with PROV-O inline — submittable to regulatory bodies as structured evidence | -Connect sensor data, weather forecasts, and maintenance logs to predict equipment failures and optimize grid operations. Temporal graphs let you track asset states over time and correlate maintenance events with performance degradation. + -**Key modules:** `ingest` (stream, API), `kg` (temporal), `reasoning`, `visualization` + -**Cookbook:** [Energy Market Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb) +**Operational requirements:** Air-gap capability, chain-of-custody, information provenance -## Advanced AI Patterns +| Semantica capability | Operational role | +| :-------------------- | :--------------- | +| Local LLM support | `HuggingFaceLLM` and Ollama via LiteLLM — fully air-gapped deployments | +| Provenance chains | Every intelligence claim traceable to source document and extraction event | +| Conflict resolution | Multiple-source disagreement resolved with auditable strategy | +| Temporal intelligence | Historical queries over evolving intelligence graphs | -### GraphRAG (Graph-Augmented Generation) + -Use knowledge graphs to retrieve precise, structured context for LLM responses — with hybrid retrieval (vector + graph traversal), logical inference, and source attribution on every claim. Every answer links back to a node in the graph, making hallucination auditable rather than invisible. + -**Key modules:** `context`, `vector_store`, `kg`, `reasoning`, `llms` +**Requirements:** Evidence integrity, chain of custody, regulatory change tracking -**Cookbooks:** +| Semantica capability | Legal role | +| :-------------------- | :--------- | +| Source attribution | Every extracted fact links to document, page, and section | +| PROV-O export | Structured provenance acceptable as supporting evidence | +| Change management | Version-controlled knowledge bases with diff and rollback | +| Reasoning paths | Explainable inference chains for contested conclusions | -- [GraphRAG Complete](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — production-ready implementation with hybrid retrieval -- [RAG vs. GraphRAG Comparison](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) — side-by-side benchmark on real-world data + + + + + +## Difficulty Reference + +| Level | What it means | Typical time | +| :---- | :------------ | :----------- | +| **Beginner** | Basic Semantica knowledge only, no domain expertise needed | 30–60 min | +| **Intermediate** | Some domain knowledge helpful, uses 2–4 Semantica modules | 1–2 hours | +| **Advanced** | Domain expertise expected, uses temporal graphs, multi-source pipelines, or reasoning | 2–3 hours | diff --git a/docs/vector_stores/pgvector.md b/docs/vector_stores/pgvector.md index 22ea25c0..942e3b41 100644 --- a/docs/vector_stores/pgvector.md +++ b/docs/vector_stores/pgvector.md @@ -4,20 +4,22 @@ description: "PostgreSQL with pgvector extension — cosine, L2, and inner produ icon: "database" --- -PostgreSQL with pgvector extension support for Semantica vector storage and similarity search. +**`PgVectorStore`** adds PostgreSQL-native vector storage and similarity search to Semantica — no dedicated vector database required. + ## Overview -The `PgVectorStore` provides native PostgreSQL vector storage using the [pgvector](https://github.com/pgvector/pgvector) extension. It supports multiple distance metrics (cosine similarity, L2/Euclidean, inner product), index types (IVFFlat, HNSW), and JSONB metadata storage with filtering. +**`PgVectorStore`** provides native PostgreSQL vector storage using the [pgvector](https://github.com/pgvector/pgvector) extension. It supports **multiple distance metrics** (cosine similarity, L2/Euclidean, inner product), index types (IVFFlat, HNSW), and JSONB metadata storage with filtering. ## Features -- **Distance Metrics**: Cosine, L2 (Euclidean), Inner Product -- **Index Types**: IVFFlat, HNSW (for approximate nearest neighbor search) -- **Metadata Storage**: JSONB for flexible metadata with filtering support -- **Connection Pooling**: Efficient connection management with psycopg3/psycopg2 -- **Batch Operations**: Bulk insert, update, delete support -- **Idempotent Index Creation**: Safe to call multiple times +Distance metrics: cosine, L2 (Euclidean), inner product +Index types: IVFFlat and HNSW for approximate nearest-neighbor search +JSONB metadata storage with filtering support +Connection pooling via psycopg3/psycopg2 +Batch insert, update, and delete +Idempotent index creation — safe to call multiple times + ## Setup @@ -40,29 +42,29 @@ pip install psycopg2-binary pgvector ### PostgreSQL Setup -1. Install pgvector extension (if not already installed): + + + ```sql + -- Debian/Ubuntu + sudo apt-get install postgresql-16-pgvector -```sql --- Using apt (Debian/Ubuntu) -sudo apt-get install postgresql-16-pgvector + -- macOS (Homebrew) + brew install pgvector --- Using homebrew (macOS) -brew install pgvector - --- Or build from source -``` - -2. Create the extension in your database: - -```sql -CREATE EXTENSION vector; -``` - -3. Verify installation: - -```sql -SELECT * FROM pg_extension WHERE extname = 'vector'; -``` + -- Or build from source: https://github.com/pgvector/pgvector + ``` + + + ```sql + CREATE EXTENSION vector; + ``` + + + ```sql + SELECT * FROM pg_extension WHERE extname = 'vector'; + ``` + + ### Docker Quickstart @@ -74,6 +76,7 @@ docker run -d \ ankane/pgvector:latest ``` + ## Connection String Format Standard PostgreSQL connection string: @@ -95,6 +98,7 @@ Examples: "postgresql://user:pass@localhost/db?connect_timeout=10&application_name=semantica" ``` + ## Usage ### Basic Usage @@ -192,7 +196,7 @@ store.create_index( ) ``` -Index creation is idempotent - calling multiple times is safe. +Index creation is idempotent — calling multiple times is safe. ### Statistics @@ -211,7 +215,7 @@ stats = store.get_stats() ## Distance Metrics | Metric | Operator | Description | Use Case | -|--------|----------|-------------|----------| +| :-------- | :---------- | :------------- | :---------- | | `cosine` | `<=>` | Cosine distance (1 - cosine similarity) | Semantic similarity, text embeddings | | `l2` | `<->` | Euclidean distance | Geometric distance, clustering | | `inner_product` | `<#>` | Negative inner product | Maximum inner product search | @@ -220,32 +224,39 @@ stats = store.get_stats() ## Index Types -### HNSW (Hierarchical Navigable Small World) + + + **Hierarchical Navigable Small World** — best for high-dimensional vectors with high recall requirements. -- **Best for**: High-dimensional vectors, high recall requirements -- **Pros**: Fast search, good recall, incremental build -- **Cons**: Higher memory usage, slower build + | | | + | :-- | :-- | + | **Pros** | Fast search, good recall, incremental build | + | **Cons** | Higher memory usage, slower build time | -```python -store.create_index(index_type="hnsw", params={ - "m": 16, # Number of connections per layer (default: 16) - "ef_construction": 64 # Build-time accuracy/speed tradeoff (default: 64) -}) -``` + ```python + store.create_index(index_type="hnsw", params={ + "m": 16, # connections per layer (default: 16) + "ef_construction": 64 # build-time accuracy/speed tradeoff (default: 64) + }) + ``` + + + **Inverted File with Flat Index** — best for large datasets in memory-constrained environments. -### IVFFlat (Inverted File with Flat Index) + | | | + | :-- | :-- | + | **Pros** | Lower memory usage, tunable speed/accuracy | + | **Cons** | Requires training data, slower incremental updates | -- **Best for**: Large datasets, memory-constrained environments -- **Pros**: Lower memory usage, tunable speed/accuracy -- **Cons**: Requires training, slower incremental updates + ```python + store.create_index(index_type="ivfflat", params={ + "lists": 100 # number of inverted lists (default: 100) + }) + ``` -```python -store.create_index(index_type="ivfflat", params={ - "lists": 100 # Number of inverted lists (default: 100) -}) -``` - -**Note**: IVFFlat requires at least as many vectors as lists for training. + IVFFlat requires at least as many vectors as `lists` before training can run. + + ## Schema @@ -338,7 +349,7 @@ store = PgVectorStore( Common errors and solutions: | Error | Cause | Solution | -|-------|-------|----------| +| :------- | :------- | :---------- | | `ProcessingError: pgvector extension is not installed` | pgvector not in PostgreSQL | Run `CREATE EXTENSION vector;` | | `ValidationError: Unsupported distance metric` | Invalid metric | Use: `cosine`, `l2`, `inner_product` | | `ValidationError: dimension mismatch` | Vector dim != store dim | Ensure consistent dimensions |