mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Fix/mintlify theme (#645)
* fix: replace invalid Mintlify theme 'venus' with 'mint' * docs: replace em dashes with colons across all docs files * fix: strip UTF-8 BOM from all docs files (broke frontmatter detection)
This commit is contained in:
Mohd Kaif
+19
-19
@@ -4,7 +4,7 @@ description: "Four-layer, modular architecture designed for independent componen
|
||||
icon: "building"
|
||||
---
|
||||
|
||||
Semantica is built around a four-layer modular architecture. Import only what you need — the framework never forces a full stack. Every component is independently swappable, and every layer communicates through clean interfaces with no hidden coupling.
|
||||
Semantica is built around a four-layer modular architecture. Import only what you need: the framework never forces a full stack. Every component is independently swappable, and every layer communicates through clean interfaces with no hidden coupling.
|
||||
|
||||
|
||||
## Four-Layer Architecture
|
||||
@@ -13,7 +13,7 @@ Semantica is built around a four-layer modular architecture. Import only what yo
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Layer 1 — Ingestion">
|
||||
<Tab title="Layer 1: Ingestion">
|
||||
|
||||
Loads data from any source into the pipeline as a unified `SourceDocument`.
|
||||
|
||||
@@ -31,7 +31,7 @@ Loads data from any source into the pipeline as a unified `SourceDocument`.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Layer 2 — Processing">
|
||||
<Tab title="Layer 2: Processing">
|
||||
|
||||
Transforms raw text into structured, enriched documents ready for knowledge store ingestion.
|
||||
|
||||
@@ -45,14 +45,14 @@ Transforms raw text into structured, enriched documents ready for knowledge stor
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Layer 3 — Intelligence">
|
||||
<Tab title="Layer 3: Intelligence">
|
||||
|
||||
Persistent knowledge stores and embedding infrastructure that power retrieval and reasoning.
|
||||
|
||||
| Component | Module | Description |
|
||||
| :--------- | :------ | :----------- |
|
||||
| Knowledge Graph | `kg` | Graph construction, temporal models, analytics, Distance Intelligence |
|
||||
| Vector Store | `vector_store` | pgvector, Qdrant, Weaviate, Pinecone — semantic similarity search |
|
||||
| Vector Store | `vector_store` | pgvector, Qdrant, Weaviate, Pinecone: semantic similarity search |
|
||||
| Ontology | `ontology` | OWL/RDFS modeling, SHACL validation, ontology alignment |
|
||||
| Triplet Store | `triplet_store` | RDF triple storage and SPARQL querying |
|
||||
| Embeddings | `embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE |
|
||||
@@ -60,7 +60,7 @@ Persistent knowledge stores and embedding infrastructure that power retrieval an
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Layer 4 — Application">
|
||||
<Tab title="Layer 4: Application">
|
||||
|
||||
Consumes the knowledge graph and vector stores for downstream use cases.
|
||||
|
||||
@@ -91,11 +91,11 @@ Every pipeline follows the same linear path from raw source to delivered output:
|
||||
|
||||
| Layer | Category | Modules |
|
||||
| :----- | :-------- | :------- |
|
||||
| **Layer 1 — Ingestion** | Sources | `ingest`, `split` |
|
||||
| **Layer 2 — Processing** | Transform | `parse`, `normalize`, `semantic_extract`, `deduplication`, `conflicts` |
|
||||
| **Layer 3 — Intelligence** | Stores | `kg`, `vector_store`, `graph_store`, `triplet_store`, `embeddings`, `ontology` |
|
||||
| **Layer 4 — Application** | Delivery | `context`, `reasoning`, `export`, `visualization`, `explorer`, `pipeline` |
|
||||
| — | Cross-cutting | `provenance`, `change_management`, `llms`, `mcp_server`, `seed`, `evals`, `core`, `utils` |
|
||||
| **Layer 1: Ingestion** | Sources | `ingest`, `split` |
|
||||
| **Layer 2: Processing** | Transform | `parse`, `normalize`, `semantic_extract`, `deduplication`, `conflicts` |
|
||||
| **Layer 3: Intelligence** | Stores | `kg`, `vector_store`, `graph_store`, `triplet_store`, `embeddings`, `ontology` |
|
||||
| **Layer 4: Application** | Delivery | `context`, `reasoning`, `export`, `visualization`, `explorer`, `pipeline` |
|
||||
|: | Cross-cutting | `provenance`, `change_management`, `llms`, `mcp_server`, `seed`, `evals`, `core`, `utils` |
|
||||
|
||||
|
||||
## Extension Points
|
||||
@@ -147,27 +147,27 @@ registry.register_plugin("my_plugin", MyPlugin, version="1.0.0")
|
||||
|
||||
<AccordionGroup>
|
||||
|
||||
<Accordion title="Modularity — use only what you need" icon="puzzle-piece">
|
||||
<Accordion title="Modularity: use only what you need" icon="puzzle-piece">
|
||||
|
||||
Every component works standalone. `NERExtractor` runs without a graph store. `VectorStore` runs without decision tracking. The framework never forces a full stack instantiation — you pay only for what you import.
|
||||
Every component works standalone. `NERExtractor` runs without a graph store. `VectorStore` runs without decision tracking. The framework never forces a full stack instantiation: you pay only for what you import.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Pluggability — extend without modifying core" icon="plug">
|
||||
<Accordion title="Pluggability: extend without modifying core" icon="plug">
|
||||
|
||||
Custom ingestors, extractors, validators, and exporters follow the same base class pattern. Register them via `PluginRegistry` and they participate in the full pipeline — provenance tracking, retry policies, and parallel execution included — with no changes to core code.
|
||||
Custom ingestors, extractors, validators, and exporters follow the same base class pattern. Register them via `PluginRegistry` and they participate in the full pipeline: provenance tracking, retry policies, and parallel execution included: with no changes to core code.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Provenance by default" icon="link">
|
||||
|
||||
Lineage tracking is built into graph construction at the lowest level. Every node and edge carries a `source_id` pointing back to the originating document, extraction method, and timestamp. There's no opt-in required — provenance is always on.
|
||||
Lineage tracking is built into graph construction at the lowest level. Every node and edge carries a `source_id` pointing back to the originating document, extraction method, and timestamp. There's no opt-in required: provenance is always on.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Configuration over convention" icon="sliders">
|
||||
|
||||
Centralized `ConfigManager` with environment variable overrides. No magic defaults — all behavior is explicit and overridable. Suitable for multi-environment deployments where dev, staging, and production need different backends.
|
||||
Centralized `ConfigManager` with environment variable overrides. No magic defaults: all behavior is explicit and overridable. Suitable for multi-environment deployments where dev, staging, and production need different backends.
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -179,10 +179,10 @@ Centralized `ConfigManager` with environment variable overrides. No magic defaul
|
||||
| Characteristic | Mechanism |
|
||||
| :-------------- | :--------- |
|
||||
| **Parallel execution** | `Pipeline(workers=N)` with configurable workers per stage |
|
||||
| **Delta processing** | Incremental graph updates — no full recompute on new data |
|
||||
| **Delta processing** | Incremental graph updates: no full recompute on new data |
|
||||
| **Streaming ingestion** | Process large corpora without loading everything into memory |
|
||||
| **Backend flexibility** | Swap in-memory NetworkX for Neo4j / FalkorDB with no API changes |
|
||||
| **Deduplication v2** | `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster than v1 |
|
||||
| **Deduplication v2** | `blocking_v2`, `hybrid_v2`, `semantic_v2`: up to 7x faster than v1 |
|
||||
| **Indexed search** | Explorer search at 0.004ms on 118k nodes (v0.5.0) |
|
||||
|
||||
<CardGroup cols={2}>
|
||||
|
||||
+1
-1
@@ -44,7 +44,7 @@ icon: "quote-left"
|
||||
|
||||
## Share Your Research
|
||||
|
||||
Published research using Semantica? [Let us know](https://github.com/semantica-agi/semantica/issues) — we may feature your work.
|
||||
Published research using Semantica? [Let us know](https://github.com/semantica-agi/semantica/issues): we may feature your work.
|
||||
|
||||
|
||||
## See Also
|
||||
|
||||
+3
-3
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "CLI Setup"
|
||||
description: "The five Semantica executables — what each one does, when to use it, and how to confirm it is working."
|
||||
description: "The five Semantica executables: what each one does, when to use it, and how to confirm it is working."
|
||||
icon: "terminal"
|
||||
---
|
||||
|
||||
@@ -139,7 +139,7 @@ python -c "import semantica; print(semantica.__version__)"
|
||||
See [Explorer Setup](explorer-setup) for the full walkthrough including how to build and save a graph file.
|
||||
</Tab>
|
||||
<Tab title="Python module form">
|
||||
Every command also runs as a Python module — useful when the script directory is not on `PATH`:
|
||||
Every command also runs as a Python module: useful when the script directory is not on `PATH`:
|
||||
|
||||
```bash
|
||||
python -m semantica.mcp_server
|
||||
@@ -201,7 +201,7 @@ If you have multiple Python environments, make sure you are installing into the
|
||||
python -m pip install semantica
|
||||
```
|
||||
|
||||
### `semantica-explorer` — "uvicorn is required"
|
||||
### `semantica-explorer`: "uvicorn is required"
|
||||
|
||||
The Explorer extras are not included in the base install:
|
||||
|
||||
|
||||
+24
-24
@@ -5,7 +5,7 @@ icon: "people-group"
|
||||
---
|
||||
|
||||
<Tip>
|
||||
Building something with Semantica? [Submit it on GitHub](https://github.com/semantica-agi/semantica/issues/new?template=community_project.md) — we'd love to feature it here.
|
||||
Building something with Semantica? [Submit it on GitHub](https://github.com/semantica-agi/semantica/issues/new?template=community_project.md): we'd love to feature it here.
|
||||
</Tip>
|
||||
|
||||
Semantica is used across academia, enterprise, and independent research. Below is a snapshot of the ecosystem being built by the community.
|
||||
@@ -17,27 +17,27 @@ Semantica is used across academia, enterprise, and independent research. Below i
|
||||
|
||||
Teams in academia are using Semantica to build structured, auditable knowledge from unstructured scientific literature.
|
||||
|
||||
- **Academic literature mapping** — citation graph construction across multi-year corpora with temporal provenance
|
||||
- **Biomedical knowledge graphs** — connecting genes, proteins, drugs, and diseases from PubMed and preprint feeds
|
||||
- **Social network analysis** — community detection and influence analysis over entity-linked interaction graphs
|
||||
- **Computational linguistics** — coreference resolution pipelines with entity-linked output for downstream NLP tasks
|
||||
- **Academic literature mapping**: citation graph construction across multi-year corpora with temporal provenance
|
||||
- **Biomedical knowledge graphs**: connecting genes, proteins, drugs, and diseases from PubMed and preprint feeds
|
||||
- **Social network analysis**: community detection and influence analysis over entity-linked interaction graphs
|
||||
- **Computational linguistics**: coreference resolution pipelines with entity-linked output for downstream NLP tasks
|
||||
|
||||
### Enterprise & Industry
|
||||
|
||||
Production deployments span regulated and high-stakes industries where AI accountability is not optional.
|
||||
|
||||
- **Business intelligence** — corporate knowledge bases built from filings, reports, and internal documentation
|
||||
- **Cybersecurity & threat intelligence** — adversary attribution graphs, CVE-linked threat feeds, incident timelines
|
||||
- **Healthcare & clinical AI** — patient safety graphs, drug interaction knowledge bases, HIPAA-compliant audit trails
|
||||
- **Financial services** — fraud detection graphs, regulatory compliance pipelines (SOX/GDPR/MiFID II), risk lineage
|
||||
- **Legal & compliance** — contract analysis pipelines, regulatory change tracking, evidence-backed research graphs
|
||||
- **Critical infrastructure** — supply chain risk graphs, energy grid event graphs, logistics provenance
|
||||
- **Business intelligence**: corporate knowledge bases built from filings, reports, and internal documentation
|
||||
- **Cybersecurity & threat intelligence**: adversary attribution graphs, CVE-linked threat feeds, incident timelines
|
||||
- **Healthcare & clinical AI**: patient safety graphs, drug interaction knowledge bases, HIPAA-compliant audit trails
|
||||
- **Financial services**: fraud detection graphs, regulatory compliance pipelines (SOX/GDPR/MiFID II), risk lineage
|
||||
- **Legal & compliance**: contract analysis pipelines, regulatory change tracking, evidence-backed research graphs
|
||||
- **Critical infrastructure**: supply chain risk graphs, energy grid event graphs, logistics provenance
|
||||
|
||||
### Independent & Open Source
|
||||
|
||||
- **GraphRAG toolkits** — custom retrieval layers built on top of Semantica's `context` + `vector_store` modules
|
||||
- **Domain-specific extractors** — NER and relation extractors for clinical, legal, and scientific text
|
||||
- **Temporal graph dashboards** — visual timelines built with Semantica's `TemporalKnowledgeGraph` + custom visualization adapters
|
||||
- **GraphRAG toolkits**: custom retrieval layers built on top of Semantica's `context` + `vector_store` modules
|
||||
- **Domain-specific extractors**: NER and relation extractors for clinical, legal, and scientific text
|
||||
- **Temporal graph dashboards**: visual timelines built with Semantica's `TemporalKnowledgeGraph` + custom visualization adapters
|
||||
|
||||
|
||||
## Supported Integrations
|
||||
@@ -66,12 +66,12 @@ Production deployments span regulated and high-stakes industries where AI accoun
|
||||
| :-------- | :---- |
|
||||
| **OpenAI** | GPT-4o, GPT-4, GPT-3.5 |
|
||||
| **Anthropic** | Claude Opus, Sonnet, Haiku |
|
||||
| **Google Gemini** | — |
|
||||
| **Groq** | LLaMA, Mixtral — fast inference |
|
||||
| **Google Gemini** |: |
|
||||
| **Groq** | LLaMA, Mixtral: fast inference |
|
||||
| **Ollama** | Fully local, air-gapped |
|
||||
| **HuggingFace** | — |
|
||||
| **DeepSeek** | — |
|
||||
| **Novita AI** | — |
|
||||
| **HuggingFace** |: |
|
||||
| **DeepSeek** |: |
|
||||
| **Novita AI** |: |
|
||||
| **LiteLLM** | 100+ model gateway |
|
||||
</Tab>
|
||||
<Tab title="NLP Libraries">
|
||||
@@ -89,11 +89,11 @@ Production deployments span regulated and high-stakes industries where AI accoun
|
||||
|
||||
The plugin system (`PluginRegistry`) makes it easy to add new capabilities without touching core code. The community has built:
|
||||
|
||||
- **Custom entity extractors** — domain-specific NER for clinical entities, legal clause types, and financial instruments
|
||||
- **Export adapters** — specialized serialization formats for proprietary industry systems
|
||||
- **Ingestor plugins** — adapters for SharePoint, Notion, Confluence, and custom databases
|
||||
- **Visualization plugins** — enhanced dashboards with Plotly, D3.js, and custom graph renderers
|
||||
- **Evaluation harnesses** — domain-specific precision/recall benchmarks using `semantica.evals`
|
||||
- **Custom entity extractors**: domain-specific NER for clinical entities, legal clause types, and financial instruments
|
||||
- **Export adapters**: specialized serialization formats for proprietary industry systems
|
||||
- **Ingestor plugins**: adapters for SharePoint, Notion, Confluence, and custom databases
|
||||
- **Visualization plugins**: enhanced dashboards with Plotly, D3.js, and custom graph renderers
|
||||
- **Evaluation harnesses**: domain-specific precision/recall benchmarks using `semantica.evals`
|
||||
|
||||
|
||||
## Build Your Own Extension
|
||||
|
||||
+10
-10
@@ -4,7 +4,7 @@ description: "Get help, connect with contributors, and share what you build with
|
||||
icon: "users"
|
||||
---
|
||||
|
||||
Semantica is built in the open, with contributions from researchers, engineers, and practitioners across many domains. Whether you're filing a bug, sharing a project, or reviewing a PR — you're part of the ecosystem.
|
||||
Semantica is built in the open, with contributions from researchers, engineers, and practitioners across many domains. Whether you're filing a bug, sharing a project, or reviewing a PR: you're part of the ecosystem.
|
||||
|
||||
|
||||
## Get Help
|
||||
@@ -20,7 +20,7 @@ Semantica is built in the open, with contributions from researchers, engineers,
|
||||
Browse open contributions and submit your own.
|
||||
</Card>
|
||||
<Card title="Security Issues" icon="shield" href="https://github.com/semantica-agi/semantica/security/advisories/new">
|
||||
Report vulnerabilities privately — never in public issues.
|
||||
Report vulnerabilities privately: never in public issues.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -29,10 +29,10 @@ Semantica is built in the open, with contributions from researchers, engineers,
|
||||
|
||||
We follow the [Contributor Covenant Code of Conduct](https://github.com/semantica-agi/semantica/blob/main/CODE_OF_CONDUCT.md). The community is built on four principles:
|
||||
|
||||
- **Respect** — treat everyone with kindness and patience, regardless of experience level
|
||||
- **Inclusion** — welcome all backgrounds, domains, and approaches
|
||||
- **Collaboration** — work toward better solutions together rather than competing on approaches
|
||||
- **Learning** — share knowledge openly, ask questions without hesitation, and help others grow
|
||||
- **Respect**: treat everyone with kindness and patience, regardless of experience level
|
||||
- **Inclusion**: welcome all backgrounds, domains, and approaches
|
||||
- **Collaboration**: work toward better solutions together rather than competing on approaches
|
||||
- **Learning**: share knowledge openly, ask questions without hesitation, and help others grow
|
||||
|
||||
To report unacceptable behavior, open a GitHub issue with the `[CoC]` prefix. Every report is investigated.
|
||||
|
||||
@@ -70,10 +70,10 @@ See the [Contributing Guide](contributing-guide) for the full development workfl
|
||||
|
||||
## Stay Connected
|
||||
|
||||
- **[GitHub](https://github.com/semantica-agi/semantica)** — source code, releases, and the public roadmap
|
||||
- **[PyPI](https://pypi.org/project/semantica/)** — package releases and download stats
|
||||
- **[Discord](https://discord.gg/sV34vps5hH)** — real-time community chat
|
||||
- **[X / Twitter](https://x.com/BuildSemantica)** — announcements and release highlights
|
||||
- **[GitHub](https://github.com/semantica-agi/semantica)**: source code, releases, and the public roadmap
|
||||
- **[PyPI](https://pypi.org/project/semantica/)**: package releases and download stats
|
||||
- **[Discord](https://discord.gg/sV34vps5hH)**: real-time community chat
|
||||
- **[X / Twitter](https://x.com/BuildSemantica)**: announcements and release highlights
|
||||
|
||||
|
||||
## See Also
|
||||
|
||||
+46
-46
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Core Concepts"
|
||||
description: "The fundamental ideas behind Semantica — knowledge graphs, reasoning, provenance, and temporal intelligence explained."
|
||||
description: "The fundamental ideas behind Semantica: knowledge graphs, reasoning, provenance, and temporal intelligence explained."
|
||||
icon: "book-open"
|
||||
---
|
||||
|
||||
@@ -8,9 +8,9 @@ icon: "book-open"
|
||||
New here? Start with [Getting Started](getting-started) for hands-on examples, then return here for deeper understanding.
|
||||
</Tip>
|
||||
|
||||
Semantica transforms unstructured data — documents, web pages, reports, databases — into **knowledge graphs**: structured representations that AI systems can query, reason about, and trace back to sources.
|
||||
Semantica transforms unstructured data: documents, web pages, reports, databases: into **knowledge graphs**: structured representations that AI systems can query, reason about, and trace back to sources.
|
||||
|
||||
At its core, Semantica adds a **context and accountability layer** on top of your existing AI stack. It doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs **grounded**, **traceable**, and **auditable**.
|
||||
At its core, Semantica adds a **context and accountability layer** on top of your existing AI stack. It doesn't replace LangChain, LlamaIndex, or your LLM provider: it makes their outputs **grounded**, **traceable**, and **auditable**.
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Context Layer" icon="diagram-project">
|
||||
@@ -20,7 +20,7 @@ At its core, Semantica adds a **context and accountability layer** on top of you
|
||||
Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable.
|
||||
</Card>
|
||||
<Card title="Extension Layer" icon="plug">
|
||||
`PluginRegistry` and `MethodRegistry` let you replace or augment any component — ingestors, extractors, reasoning engines, backends — without changing framework code.
|
||||
`PluginRegistry` and `MethodRegistry` let you replace or augment any component: ingestors, extractors, reasoning engines, backends: without changing framework code.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -31,11 +31,11 @@ At its core, Semantica adds a **context and accountability layer** on top of you
|
||||
|
||||
The foundation of everything in Semantica. A knowledge graph stores information as three building blocks:
|
||||
|
||||
- **Nodes (entities)** — people, companies, locations, events, concepts
|
||||
- **Edges (relationships)** — `works_for`, `located_in`, `founded_by`
|
||||
- **Properties** — name, date, confidence score, source URL
|
||||
- **Nodes (entities)**: people, companies, locations, events, concepts
|
||||
- **Edges (relationships)**: `works_for`, `located_in`, `founded_by`
|
||||
- **Properties**: name, date, confidence score, source URL
|
||||
|
||||
This structure makes knowledge **searchable**, **connectable**, **queryable**, and — critically — **explainable**: every answer can be traced back to the facts and relationships that produced it.
|
||||
This structure makes knowledge **searchable**, **connectable**, **queryable**, and: critically: **explainable**: every answer can be traced back to the facts and relationships that produced it.
|
||||
|
||||
|
||||
## Entity Extraction (NER)
|
||||
@@ -58,9 +58,9 @@ Each entity gets a type, confidence score, and a link to its source document. Th
|
||||
|
||||
| Method | Speed | Accuracy | Requirements |
|
||||
| :------ | :----- | :-------- | :------------ |
|
||||
| `"pattern"` | ⚡ Very fast | Moderate | No API key — regex-based |
|
||||
| `"pattern"` | ⚡ Very fast | Moderate | No API key: regex-based |
|
||||
| `"ml"` | Fast | High | Local ML model |
|
||||
| `"llm"` | Medium | Highest | LLM provider — all 9 supported |
|
||||
| `"llm"` | Medium | Highest | LLM provider: all 9 supported |
|
||||
|
||||
## Relationship Extraction
|
||||
|
||||
@@ -75,12 +75,12 @@ Finding how entities connect to each other:
|
||||
}
|
||||
```
|
||||
|
||||
Relationships can be extracted via rule-based methods, ML models, or LLMs — each producing typed triplets with confidence scores and source attribution.
|
||||
Relationships can be extracted via rule-based methods, ML models, or LLMs: each producing typed triplets with confidence scores and source attribution.
|
||||
|
||||
|
||||
## Knowledge Graph vs. Vector Store
|
||||
|
||||
Both store information for AI retrieval — but they're built for different jobs.
|
||||
Both store information for AI retrieval: but they're built for different jobs.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Knowledge Graph">
|
||||
@@ -89,7 +89,7 @@ Both store information for AI retrieval — but they're built for different jobs
|
||||
| Strength | Why it matters |
|
||||
| :-------- | :------------- |
|
||||
| **Traversal** | Multi-hop queries: "Who founded companies that Apple alumni later joined?" |
|
||||
| **Explainability** | Every answer traces back to specific nodes and edges — no black-box retrieval |
|
||||
| **Explainability** | Every answer traces back to specific nodes and edges: no black-box retrieval |
|
||||
| **Temporal reasoning** | Point-in-time queries, `valid_from`/`valid_until` windows, historical snapshots |
|
||||
| **Conflict detection** | Two sources disagreeing on the same fact is surfaced and resolvable |
|
||||
| **Schema enforcement** | SHACL validation catches constraint violations before they corrupt results |
|
||||
@@ -106,14 +106,14 @@ Both store information for AI retrieval — but they're built for different jobs
|
||||
</Tab>
|
||||
|
||||
<Tab title="Vector Store">
|
||||
Stores **dense embeddings** of text chunks. Answers questions by finding semantically similar passages — useful when the structure of the answer isn't known in advance.
|
||||
Stores **dense embeddings** of text chunks. Answers questions by finding semantically similar passages: useful when the structure of the answer isn't known in advance.
|
||||
|
||||
| Strength | Why it matters |
|
||||
| :-------- | :------------- |
|
||||
| **Fuzzy similarity** | Finds relevant content even when exact words don't match |
|
||||
| **Speed** | Sub-millisecond approximate nearest-neighbor search at scale |
|
||||
| **Unstructured text** | Works directly on paragraphs, sentences, and raw documents |
|
||||
| **Simplicity** | No schema design required — embed and index |
|
||||
| **Simplicity** | No schema design required: embed and index |
|
||||
|
||||
**Use when:** you need fast semantic search over large text corpora.
|
||||
|
||||
@@ -127,7 +127,7 @@ Both store information for AI retrieval — but they're built for different jobs
|
||||
</Tab>
|
||||
|
||||
<Tab title="GraphRAG (Both)">
|
||||
Semantica combines both — vector search seeds the graph traversal, and the graph provides structure and provenance the vector store cannot.
|
||||
Semantica combines both: vector search seeds the graph traversal, and the graph provides structure and provenance the vector store cannot.
|
||||
|
||||
| Step | What happens |
|
||||
| :---- | :----------- |
|
||||
@@ -136,7 +136,7 @@ Both store information for AI retrieval — but they're built for different jobs
|
||||
| **Context assembly** | Facts + relationships are assembled with source attribution for each claim |
|
||||
| **LLM generation** | LLM generates an answer grounded in the retrieved structured context |
|
||||
|
||||
**Result:** every claim in the response links back to a specific graph node — no hallucination from training data, full audit trail.
|
||||
**Result:** every claim in the response links back to a specific graph node: no hallucination from training data, full audit trail.
|
||||
|
||||
```python
|
||||
from semantica.context import AgentContext, ContextGraph
|
||||
@@ -154,15 +154,15 @@ Both store information for AI retrieval — but they're built for different jobs
|
||||
|
||||
## Embeddings
|
||||
|
||||
Embeddings convert text into numerical vectors so AI systems can measure semantic similarity — finding related concepts even when the exact words differ.
|
||||
Embeddings convert text into numerical vectors so AI systems can measure semantic similarity: finding related concepts even when the exact words differ.
|
||||
|
||||
Semantica uses embeddings for:
|
||||
|
||||
- **Semantic search** — retrieve by meaning, not just keywords
|
||||
- **Entity resolution** — match the same entity across different sources
|
||||
- **Precedent search** — find similar past decisions
|
||||
- **GraphRAG retrieval** — hybrid vector + graph traversal
|
||||
- **Distance Intelligence** — N×N semantic distance matrices between any node set
|
||||
- **Semantic search**: retrieve by meaning, not just keywords
|
||||
- **Entity resolution**: match the same entity across different sources
|
||||
- **Precedent search**: find similar past decisions
|
||||
- **GraphRAG retrieval**: hybrid vector + graph traversal
|
||||
- **Distance Intelligence**: N×N semantic distance matrices between any node set
|
||||
|
||||
**Supported models:** Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama local embeddings.
|
||||
|
||||
@@ -178,13 +178,13 @@ GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses
|
||||
The query is embedded and used to seed both vector search and graph traversal simultaneously.
|
||||
</Step>
|
||||
<Step title="Hybrid context retrieval">
|
||||
Semantica retrieves relevant graph context — entities, typed relationships, and multi-hop reasoning paths — alongside vector-similar text chunks.
|
||||
Semantica retrieves relevant graph context: entities, typed relationships, and multi-hop reasoning paths: alongside vector-similar text chunks.
|
||||
</Step>
|
||||
<Step title="Context building">
|
||||
Retrieved facts and reasoning paths are assembled into a structured prompt context, each fact tagged with its source node and confidence.
|
||||
</Step>
|
||||
<Step title="LLM generates a grounded response">
|
||||
The LLM produces an answer where every claim links back to a source node in the graph — no floating assertions, no hallucinations from training data.
|
||||
The LLM produces an answer where every claim links back to a source node in the graph: no floating assertions, no hallucinations from training data.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
@@ -195,7 +195,7 @@ GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses
|
||||
|
||||
## Ontology
|
||||
|
||||
An ontology defines the schema and rules for your knowledge — what entity types exist, which relationships are valid, and what constraints apply.
|
||||
An ontology defines the schema and rules for your knowledge: what entity types exist, which relationships are valid, and what constraints apply.
|
||||
|
||||
```python
|
||||
ontology = {
|
||||
@@ -239,7 +239,7 @@ Inferred: Steve Jobs has a connection to Cupertino
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Rete Network">
|
||||
Efficient pattern matching for large rule sets — the Rete algorithm avoids re-evaluating rules whose preconditions haven't changed. Best for thousands of rules over millions of facts.
|
||||
Efficient pattern matching for large rule sets: the Rete algorithm avoids re-evaluating rules whose preconditions haven't changed. Best for thousands of rules over millions of facts.
|
||||
|
||||
```python
|
||||
from semantica.reasoning import ReteEngine
|
||||
@@ -250,9 +250,9 @@ Inferred: Steve Jobs has a connection to Cupertino
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Deductive & Abductive">
|
||||
**Deductive** — classical syllogistic reasoning from premises to guaranteed conclusions.
|
||||
**Deductive**: classical syllogistic reasoning from premises to guaranteed conclusions.
|
||||
|
||||
**Abductive** — infers the most likely explanation for observed evidence. Best for diagnostic and investigative use cases.
|
||||
**Abductive**: infers the most likely explanation for observed evidence. Best for diagnostic and investigative use cases.
|
||||
|
||||
```python
|
||||
from semantica.reasoning import GraphReasoner
|
||||
@@ -263,7 +263,7 @@ Inferred: Steve Jobs has a connection to Cupertino
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Datalog (v0.4.0)">
|
||||
Recursive Horn clause rules with fixpoint semantics — handles transitive closure and recursive relationships that forward chaining cannot express.
|
||||
Recursive Horn clause rules with fixpoint semantics: handles transitive closure and recursive relationships that forward chaining cannot express.
|
||||
|
||||
```python
|
||||
from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule
|
||||
@@ -289,7 +289,7 @@ Inferred: Steve Jobs has a connection to Cupertino
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
All engines produce **explainable inference paths** — not black-box conclusions. Every derived fact includes the rules and premises that produced it.
|
||||
All engines produce **explainable inference paths**: not black-box conclusions. Every derived fact includes the rules and premises that produced it.
|
||||
|
||||
|
||||
## Temporal Intelligence
|
||||
@@ -313,7 +313,7 @@ snapshot = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15
|
||||
|
||||
## Distance Intelligence
|
||||
|
||||
Explore the semantic neighborhood of any entity in your graph — useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology.
|
||||
Explore the semantic neighborhood of any entity in your graph: useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology.
|
||||
|
||||
```python
|
||||
from semantica.kg import SimilarityCalculator
|
||||
@@ -329,7 +329,7 @@ The [Visualization module](reference/visualization) renders distance matrices as
|
||||
|
||||
## Deduplication & Entity Resolution
|
||||
|
||||
Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance.
|
||||
Real-world data contains the same entity under many names: "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Strategies">
|
||||
@@ -337,7 +337,7 @@ Real-world data contains the same entity under many names — "Apple", "Apple In
|
||||
| Strategy | Algorithm | Best For |
|
||||
| :-------- | :--------- | :-------- |
|
||||
| `v1` | Jaro-Winkler string similarity | Small datasets, fast baseline |
|
||||
| `blocking_v2` | Candidate blocking + similarity | Large corpora — reduces O(n²) comparisons |
|
||||
| `blocking_v2` | Candidate blocking + similarity | Large corpora: reduces O(n²) comparisons |
|
||||
| `hybrid_v2` | Blocking + semantic embedding match | Mixed structured/unstructured entity names |
|
||||
| `semantic_v2` | Pure embedding-based resolution | Up to 7× faster than v1; handles abbreviations and aliases |
|
||||
|
||||
@@ -366,7 +366,7 @@ Every fact in Semantica links back to:
|
||||
- The **reasoning steps** that produced any inferred fact
|
||||
|
||||
<Note>
|
||||
This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11). Use `RDFExporter(include_provenance=True)` to embed provenance inline in any RDF export.
|
||||
This is W3C PROV-O compliant lineage: suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11). Use `RDFExporter(include_provenance=True)` to embed provenance inline in any RDF export.
|
||||
</Note>
|
||||
|
||||
```python
|
||||
@@ -384,7 +384,7 @@ print(f"Checksum: {lineage.checksum}")
|
||||
|
||||
## Decision Intelligence
|
||||
|
||||
Every agent decision is a first-class object in Semantica — recorded, causally linked, and searchable by precedent. This is the **accountability layer** for AI pipelines: decisions are no longer ephemeral log messages, they are queryable knowledge graph nodes.
|
||||
Every agent decision is a first-class object in Semantica: recorded, causally linked, and searchable by precedent. This is the **accountability layer** for AI pipelines: decisions are no longer ephemeral log messages, they are queryable knowledge graph nodes.
|
||||
|
||||
```python
|
||||
decision_id = context.record_decision(
|
||||
@@ -403,7 +403,7 @@ influence = context.analyze_decision_influence(decision_id)
|
||||
```
|
||||
|
||||
<Tip>
|
||||
**Use `find_precedents()` before every high-stakes decision.** Hybrid similarity search over all recorded decisions surfaces past reasoning that may apply — reducing inconsistency across agent runs and enabling genuine organisational learning from AI decision history.
|
||||
**Use `find_precedents()` before every high-stakes decision.** Hybrid similarity search over all recorded decisions surfaces past reasoning that may apply: reducing inconsistency across agent runs and enabling genuine organisational learning from AI decision history.
|
||||
</Tip>
|
||||
|
||||
|
||||
@@ -413,20 +413,20 @@ When multiple sources disagree on the same fact, Semantica flags and resolves th
|
||||
|
||||
**Resolution strategies:**
|
||||
|
||||
- **Recency** — prefer the most recent source
|
||||
- **Source credibility** — prefer the most reliable source (configurable credibility scores)
|
||||
- **Majority vote** — aggregate across all sources with ≥ 2 agreeing
|
||||
- **Manual review** — flag for human arbitration; continue pipeline without blocking
|
||||
- **Recency**: prefer the most recent source
|
||||
- **Source credibility**: prefer the most reliable source (configurable credibility scores)
|
||||
- **Majority vote**: aggregate across all sources with ≥ 2 agreeing
|
||||
- **Manual review**: flag for human arbitration; continue pipeline without blocking
|
||||
|
||||
See the [Conflicts reference](reference/conflicts) for `ConflictResolver`, `SourceTracker`, and `InvestigationGuideGenerator`.
|
||||
|
||||
|
||||
## Custom Plugin Development
|
||||
|
||||
Semantica is designed for extension. Any component — ingestor, extractor, graph builder, reasoning engine — can be replaced or augmented with a custom implementation registered at runtime.
|
||||
Semantica is designed for extension. Any component: ingestor, extractor, graph builder, reasoning engine: can be replaced or augmented with a custom implementation registered at runtime.
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="PluginRegistry — replace any component by name">
|
||||
<Accordion title="PluginRegistry: replace any component by name">
|
||||
|
||||
`PluginRegistry` provides dynamic plugin discovery, registration, and loading across all modules. Register your own class under a string key; Semantica will use it wherever that key is referenced in config or pipeline steps.
|
||||
|
||||
@@ -447,7 +447,7 @@ Semantica is designed for extension. Any component — ingestor, extractor, grap
|
||||
plugin = registry.load_plugin("my_sql_ingestor", connection_string="postgresql://...")
|
||||
result = plugin.execute("SELECT * FROM documents")
|
||||
|
||||
# Reference by name in pipeline YAML — no code changes needed
|
||||
# Reference by name in pipeline YAML: no code changes needed
|
||||
```
|
||||
|
||||
```yaml
|
||||
@@ -461,9 +461,9 @@ Semantica is designed for extension. Any component — ingestor, extractor, grap
|
||||
**Extension points available:** ingestors, parsers, normalizers, extractors, reasoning engines, export formats, vector store backends, graph store backends, visualization renderers.
|
||||
|
||||
</Accordion>
|
||||
<Accordion title="MethodRegistry — add domain-specific graph operations">
|
||||
<Accordion title="MethodRegistry: add domain-specific graph operations">
|
||||
|
||||
`MethodRegistry` lets you register custom methods on knowledge graph objects by name — useful for adding domain-specific graph operations without subclassing.
|
||||
`MethodRegistry` lets you register custom methods on knowledge graph objects by name: useful for adding domain-specific graph operations without subclassing.
|
||||
|
||||
```python
|
||||
from semantica.kg import MethodRegistry
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "How to contribute code, documentation, tests, and community suppor
|
||||
icon: "code-pull-request"
|
||||
---
|
||||
|
||||
Contributions of all kinds are welcome — code, documentation, tests, and community support. Every contribution is recognized in release notes and the GitHub contributors list.
|
||||
Contributions of all kinds are welcome: code, documentation, tests, and community support. Every contribution is recognized in release notes and the GitHub contributors list.
|
||||
|
||||
|
||||
## Quick Start
|
||||
@@ -17,7 +17,7 @@ pip install -e ".[dev]"
|
||||
pytest
|
||||
```
|
||||
|
||||
New to the project? Start with [`good-first-issue`](https://github.com/semantica-agi/semantica/labels/good-first-issue) labeled tickets — they're scoped to be completable in a few hours without deep codebase knowledge.
|
||||
New to the project? Start with [`good-first-issue`](https://github.com/semantica-agi/semantica/labels/good-first-issue) labeled tickets: they're scoped to be completable in a few hours without deep codebase knowledge.
|
||||
|
||||
|
||||
## Ways to Contribute
|
||||
@@ -77,16 +77,16 @@ Style conventions: **Black** for formatting, **isort** for imports, **flake8** f
|
||||
|
||||
Before submitting a PR, confirm:
|
||||
|
||||
<Check>Tests pass locally — `pytest`</Check>
|
||||
<Check>Tests pass locally: `pytest`</Check>
|
||||
<Check>New features include documentation with working code examples</Check>
|
||||
<Check>Code follows project style — Black, isort, flake8</Check>
|
||||
<Check>Code follows project style: Black, isort, flake8</Check>
|
||||
<Check>Commit messages are clear and describe the *why*, not just the *what*</Check>
|
||||
<Check>No unresolved merge conflicts</Check>
|
||||
|
||||
|
||||
## Code of Conduct
|
||||
|
||||
All contributors are expected to follow the [Contributor Covenant Code of Conduct](https://github.com/semantica-agi/semantica/blob/main/CODE_OF_CONDUCT.md). Be respectful, patient, and constructive — especially toward newcomers. Report violations by opening an issue with the `[CoC]` prefix.
|
||||
All contributors are expected to follow the [Contributor Covenant Code of Conduct](https://github.com/semantica-agi/semantica/blob/main/CODE_OF_CONDUCT.md). Be respectful, patient, and constructive: especially toward newcomers. Report violations by opening an issue with the `[CoC]` prefix.
|
||||
|
||||
|
||||
## Help
|
||||
|
||||
+3
-3
@@ -6,9 +6,9 @@ icon: "flask"
|
||||
|
||||
<Tip>
|
||||
**Where to start:**
|
||||
- **New to Semantica** — begin with [Core Tutorials](#core-tutorials)
|
||||
- **Building an application** — see [Advanced Concepts](#advanced-concepts) or [Industry Use Cases](#industry-use-cases)
|
||||
- **Need installation help** — see the [Installation Guide](installation)
|
||||
- **New to Semantica**: begin with [Core Tutorials](#core-tutorials)
|
||||
- **Building an application**: see [Advanced Concepts](#advanced-concepts) or [Industry Use Cases](#industry-use-cases)
|
||||
- **Need installation help**: see the [Installation Guide](installation)
|
||||
</Tip>
|
||||
|
||||
<Note>
|
||||
|
||||
+17
-17
@@ -4,7 +4,7 @@ description: "Install the Explorer extras, save a ContextGraph to JSON, and laun
|
||||
icon: "map"
|
||||
---
|
||||
|
||||
**`semantica-explorer`** is an **interactive browser dashboard** for knowledge graph exploration. You give it a graph file, it starts a local server, and opens a browser tab where you can search nodes, find paths, inspect provenance, and run analytics — no code required after launch.
|
||||
**`semantica-explorer`** is an **interactive browser dashboard** for knowledge graph exploration. You give it a graph file, it starts a local server, and opens a browser tab where you can search nodes, find paths, inspect provenance, and run analytics: no code required after launch.
|
||||
|
||||
This page covers everything needed to go from zero to a running Explorer. For the full REST API reference and endpoint catalogue, see [Explorer Reference](reference/explorer).
|
||||
|
||||
@@ -60,7 +60,7 @@ curl http://127.0.0.1:8000/api/health
|
||||
```
|
||||
|
||||
|
||||
## Step 1 — Build and Save a ContextGraph
|
||||
## Step 1: Build and Save a ContextGraph
|
||||
|
||||
Explorer loads a graph from a JSON file on disk. You need to create that file first.
|
||||
|
||||
@@ -113,7 +113,7 @@ Explorer loads a graph from a JSON file on disk. You need to create that file fi
|
||||
</Tip>
|
||||
|
||||
|
||||
## Step 2 — Launch Explorer
|
||||
## Step 2: Launch Explorer
|
||||
|
||||
```bash
|
||||
semantica-explorer --graph my_graph.json
|
||||
@@ -122,7 +122,7 @@ semantica-explorer --graph my_graph.json
|
||||
The startup sequence prints:
|
||||
|
||||
```
|
||||
✓ Graph loaded — 3 nodes, 2 edges
|
||||
✓ Graph loaded: 3 nodes, 2 edges
|
||||
╭─ Semantica Explorer · http://127.0.0.1:8000 ─╮
|
||||
│ API docs http://127.0.0.1:8000/docs │
|
||||
│ Health http://127.0.0.1:8000/api/health │
|
||||
@@ -140,15 +140,15 @@ The browser opens automatically at `http://127.0.0.1:8000` shortly after the ser
|
||||
| :---- | :----- | :------- | :----------- |
|
||||
| `--graph` | `-g` | *(**required**)* | Path to a ContextGraph JSON file |
|
||||
| `--port` | `-p` | `8000` | Port to bind the server |
|
||||
| `--host` | — | `127.0.0.1` | Host to bind the server |
|
||||
| `--no-browser` | — | off | Do not open a browser tab automatically |
|
||||
| `--host` |: | `127.0.0.1` | Host to bind the server |
|
||||
| `--no-browser` |: | off | Do not open a browser tab automatically |
|
||||
|
||||
There are no flags for authentication, log level, or TLS. Those are not implemented in the CLI.
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Minimal — local only, port 8000, browser opens automatically
|
||||
# Minimal: local only, port 8000, browser opens automatically
|
||||
semantica-explorer --graph my_graph.json
|
||||
|
||||
# Short flags
|
||||
@@ -157,7 +157,7 @@ semantica-explorer -g my_graph.json -p 8080
|
||||
# Expose on the network so other machines can connect
|
||||
semantica-explorer --graph my_graph.json --host 0.0.0.0 --port 8080
|
||||
|
||||
# Headless — skip the auto-open and navigate manually
|
||||
# Headless: skip the auto-open and navigate manually
|
||||
semantica-explorer --graph my_graph.json --no-browser
|
||||
```
|
||||
|
||||
@@ -173,8 +173,8 @@ Once the server is running:
|
||||
| URL | What you get |
|
||||
| :--- | :------------ |
|
||||
| `http://127.0.0.1:8000` | Interactive dashboard |
|
||||
| `http://127.0.0.1:8000/docs` | Swagger UI — every REST endpoint, interactive |
|
||||
| `http://127.0.0.1:8000/api/health` | Health check — `{"status": "healthy"}` |
|
||||
| `http://127.0.0.1:8000/docs` | Swagger UI: every REST endpoint, interactive |
|
||||
| `http://127.0.0.1:8000/api/health` | Health check: `{"status": "healthy"}` |
|
||||
|
||||
The browser tab opens shortly after startup. If it does not open, navigate to the URL manually or pass `--no-browser` and open it yourself.
|
||||
|
||||
@@ -241,12 +241,12 @@ This is expected in headless, SSH, and container environments. Add `--no-browser
|
||||
|
||||
Once running, Explorer exposes a REST API and dashboard for:
|
||||
|
||||
- **Node and edge search** — indexed search across all nodes by ID, type, and content
|
||||
- **Neighborhood expansion** — inspect neighbors up to configurable hop depth
|
||||
- **Path finding** — BFS shortest path between any two nodes
|
||||
- **Graph analytics** — centrality, community detection, connectivity
|
||||
- **Decisions and provenance** — query recorded decisions and their causal chains
|
||||
- **Import / export** — upload JSON or CSV to extend the graph; download the current state
|
||||
- **Node and edge search**: indexed search across all nodes by ID, type, and content
|
||||
- **Neighborhood expansion**: inspect neighbors up to configurable hop depth
|
||||
- **Path finding**: BFS shortest path between any two nodes
|
||||
- **Graph analytics**: centrality, community detection, connectivity
|
||||
- **Decisions and provenance**: query recorded decisions and their causal chains
|
||||
- **Import / export**: upload JSON or CSV to extend the graph; download the current state
|
||||
|
||||
The full endpoint catalogue is documented in the Swagger UI at `/docs` and in the reference page below.
|
||||
|
||||
@@ -258,7 +258,7 @@ The full endpoint catalogue is documented in the Swagger UI at `/docs` and in th
|
||||
All five Semantica executables and when to use each one.
|
||||
</Card>
|
||||
<Card title="Context Module" icon="brain" href="reference/context">
|
||||
Full documentation for ContextGraph — build, query, save, and load.
|
||||
Full documentation for ContextGraph: build, query, save, and load.
|
||||
</Card>
|
||||
<Card title="Quickstart" icon="rocket" href="quickstart">
|
||||
End-to-end pipeline: ingest → extract → build graph → export.
|
||||
|
||||
+32
-32
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "FAQ"
|
||||
description: "Common questions about Semantica — installation, features, integrations, and troubleshooting."
|
||||
description: "Common questions about Semantica: installation, features, integrations, and troubleshooting."
|
||||
icon: "circle-question"
|
||||
---
|
||||
|
||||
@@ -12,13 +12,13 @@ icon: "circle-question"
|
||||
|
||||
| Question | Answer |
|
||||
| :-------- | :------ |
|
||||
| License? | MIT — free forever, no paywalled features |
|
||||
| License? | MIT: free forever, no paywalled features |
|
||||
| Python version? | 3.8+ (3.11+ recommended) |
|
||||
| API key required? | Optional — pattern extraction works with no keys |
|
||||
| Works with LangChain / LlamaIndex? | Yes — Semantica is a layer on top, not a replacement |
|
||||
| Production-ready? | Yes — 1,000+ tests, v0.5.0 ships with 12 security fixes |
|
||||
| API key required? | Optional: pattern extraction works with no keys |
|
||||
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
|
||||
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
|
||||
| Latest version? | **v0.5.0** (May 2026) |
|
||||
| Local LLMs? | Yes — Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
|
||||
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
|
||||
|
||||
|
||||
## General
|
||||
@@ -27,9 +27,9 @@ icon: "circle-question"
|
||||
|
||||
<Accordion title="What is Semantica?" icon="info-circle">
|
||||
|
||||
Semantica is an open-source framework for building context graphs and decision intelligence layers for AI. It transforms unstructured data — documents, APIs, databases — into structured knowledge graphs with full provenance tracking, making AI systems explainable and auditable.
|
||||
Semantica is an open-source framework for building context graphs and decision intelligence layers for AI. It transforms unstructured data: documents, APIs, databases: into structured knowledge graphs with full provenance tracking, making AI systems explainable and auditable.
|
||||
|
||||
It's not a replacement for LangChain or LlamaIndex. It's the **accountability layer** that goes on top — recording decisions, tracing facts to sources, and making reasoning transparent.
|
||||
It's not a replacement for LangChain or LlamaIndex. It's the **accountability layer** that goes on top: recording decisions, tracing facts to sources, and making reasoning transparent.
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -46,7 +46,7 @@ It's not a replacement for LangChain or LlamaIndex. It's the **accountability la
|
||||
|
||||
<Accordion title="What makes Semantica different from LangChain or LlamaIndex?" icon="scale-balanced">
|
||||
|
||||
Most frameworks stop at retrieval or generation. Semantica adds an **accountability layer**: every decision is recorded, every fact links to a source, and every reasoning step is explainable. It's designed for environments where you need to audit *why* an AI reached a conclusion — not just what it said.
|
||||
Most frameworks stop at retrieval or generation. Semantica adds an **accountability layer**: every decision is recorded, every fact links to a source, and every reasoning step is explainable. It's designed for environments where you need to audit *why* an AI reached a conclusion: not just what it said.
|
||||
|
||||
Semantica works alongside these frameworks, not against them.
|
||||
|
||||
@@ -54,13 +54,13 @@ Semantica works alongside these frameworks, not against them.
|
||||
|
||||
<Accordion title="Is Semantica free?" icon="tag">
|
||||
|
||||
Yes — MIT licensed, no vendor lock-in, no paywalled features. Some capabilities require third-party API keys (e.g., OpenAI embeddings, Groq inference), but Semantica itself is always free and open source.
|
||||
Yes: MIT licensed, no vendor lock-in, no paywalled features. Some capabilities require third-party API keys (e.g., OpenAI embeddings, Groq inference), but Semantica itself is always free and open source.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="What's the latest version?" icon="star">
|
||||
|
||||
**v0.5.0** — released May 2026.
|
||||
**v0.5.0**: released May 2026.
|
||||
|
||||
Highlights: Ontology Hub, Distance Intelligence, Parquet/XML ingestion, 12 security fixes, Graph Explorer redesign, NER gateway fix.
|
||||
|
||||
@@ -95,7 +95,7 @@ Python **3.8 or higher**. Python 3.11+ is recommended for best performance and c
|
||||
|
||||
<Accordion title="The [all] extra fails on Windows" icon="windows">
|
||||
|
||||
This was a known bug — fixed in **v0.5.0**. Upgrade:
|
||||
This was a known bug: fixed in **v0.5.0**. Upgrade:
|
||||
|
||||
```bash
|
||||
pip install --upgrade semantica
|
||||
@@ -141,10 +141,10 @@ If you're on an older version, install extras individually: `pip install "semant
|
||||
|
||||
Yes. Semantica supports:
|
||||
|
||||
- **Custom NER and extraction models** — register via `method_registry`
|
||||
- **Custom embedding models** — any model with a `.encode()` interface
|
||||
- **Custom LLM providers** — via LiteLLM (100+ models) or direct provider integration
|
||||
- **Custom pipeline processors** — register via `PluginRegistry`
|
||||
- **Custom NER and extraction models**: register via `method_registry`
|
||||
- **Custom embedding models**: any model with a `.encode()` interface
|
||||
- **Custom LLM providers**: via LiteLLM (100+ models) or direct provider integration
|
||||
- **Custom pipeline processors**: register via `PluginRegistry`
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -162,10 +162,10 @@ This includes PyTorch with CUDA, FAISS GPU, and CuPy.
|
||||
|
||||
<Accordion title="How does Semantica handle large datasets?" icon="layer-group">
|
||||
|
||||
- **Batching** — process documents in configurable chunks to control memory usage
|
||||
- **Parallel processing** — `Pipeline(workers=N)` runs extraction steps concurrently
|
||||
- **Delta processing** — update graphs incrementally without full recompute on new data
|
||||
- **Persistent backends** — swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE for large-scale production graphs
|
||||
- **Batching**: process documents in configurable chunks to control memory usage
|
||||
- **Parallel processing**: `Pipeline(workers=N)` runs extraction steps concurrently
|
||||
- **Delta processing**: update graphs incrementally without full recompute on new data
|
||||
- **Persistent backends**: swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE for large-scale production graphs
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -187,13 +187,13 @@ Available since v0.4.0.
|
||||
|
||||
<Accordion title="What is the Ontology Hub?" icon="sitemap">
|
||||
|
||||
A visual browser UI for the full ontology lifecycle — launched via `semantica.explorer`. Includes:
|
||||
A visual browser UI for the full ontology lifecycle: launched via `semantica.explorer`. Includes:
|
||||
|
||||
- **Visual editor** — create and edit classes, properties, and relationships
|
||||
- **SHACL Studio** — author, validate, and export SHACL shapes
|
||||
- **Alignment authoring** — map concepts across ontologies
|
||||
- **Health dashboard** — coverage, consistency, and constraint violation metrics
|
||||
- **Version control** — diff and history for ontology changes
|
||||
- **Visual editor**: create and edit classes, properties, and relationships
|
||||
- **SHACL Studio**: author, validate, and export SHACL shapes
|
||||
- **Alignment authoring**: map concepts across ontologies
|
||||
- **Health dashboard**: coverage, consistency, and constraint violation metrics
|
||||
- **Version control**: diff and history for ontology changes
|
||||
|
||||
Available since v0.5.0.
|
||||
|
||||
@@ -231,11 +231,11 @@ pip install --upgrade semantica
|
||||
|
||||
<Accordion title="What graph databases are supported?" icon="diagram-project">
|
||||
|
||||
- **Neo4j** — industry standard, Cypher query language
|
||||
- **FalkorDB** — Redis-protocol, ultra-low latency
|
||||
- **Apache AGE** — PostgreSQL extension, OpenCypher
|
||||
- **Amazon Neptune** — managed AWS, SPARQL and Gremlin
|
||||
- **NetworkX** — in-memory, for development and small graphs
|
||||
- **Neo4j**: industry standard, Cypher query language
|
||||
- **FalkorDB**: Redis-protocol, ultra-low latency
|
||||
- **Apache AGE**: PostgreSQL extension, OpenCypher
|
||||
- **Amazon Neptune**: managed AWS, SPARQL and Gremlin
|
||||
- **NetworkX**: in-memory, for development and small graphs
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -247,7 +247,7 @@ RDF (Turtle, JSON-LD, N-Triples, XML), Apache Parquet, ArangoDB AQL, Apache Arro
|
||||
|
||||
<Accordion title="What vector stores are supported?" icon="server">
|
||||
|
||||
FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, and in-memory. All backends share the same `VectorStore` API — swap with one line change.
|
||||
FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, and in-memory. All backends share the same `VectorStore` API: swap with one line change.
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Getting Started"
|
||||
description: "The context and intelligence layer for AI — turning raw data into explainable, auditable knowledge graphs."
|
||||
description: "The context and intelligence layer for AI: turning raw data into explainable, auditable knowledge graphs."
|
||||
icon: "rocket"
|
||||
---
|
||||
|
||||
@@ -58,7 +58,7 @@ icon: "rocket"
|
||||
</Step>
|
||||
|
||||
<Step title="Choose your path">
|
||||
Pick the track that matches what you're building — each starts with a focused 5-minute example.
|
||||
Pick the track that matches what you're building: each starts with a focused 5-minute example.
|
||||
|
||||
| Track | You want to... | Start with |
|
||||
| :----- | :-------------- | :--------- |
|
||||
@@ -70,10 +70,10 @@ icon: "rocket"
|
||||
</Step>
|
||||
|
||||
<Step title="Run the pipeline">
|
||||
The full 6-step pipeline — ingest, parse, extract, build, visualize, export — is in the [Quickstart](quickstart). Takes under 5 minutes with pattern-based extraction (no API key required).
|
||||
The full 6-step pipeline: ingest, parse, extract, build, visualize, export: is in the [Quickstart](quickstart). Takes under 5 minutes with pattern-based extraction (no API key required).
|
||||
|
||||
<Note>
|
||||
An LLM API key is **optional** for the quickstart. Pattern-based extraction works out of the box — upgrade to LLM extraction for higher accuracy when you're ready.
|
||||
An LLM API key is **optional** for the quickstart. Pattern-based extraction works out of the box: upgrade to LLM extraction for higher accuracy when you're ready.
|
||||
</Note>
|
||||
</Step>
|
||||
</Steps>
|
||||
@@ -145,7 +145,7 @@ icon: "rocket"
|
||||
</Tab>
|
||||
|
||||
<Tab title="GraphRAG">
|
||||
Ground every LLM response in your knowledge graph — no floating assertions.
|
||||
Ground every LLM response in your knowledge graph: no floating assertions.
|
||||
|
||||
```python
|
||||
from semantica.context import AgentContext, ContextGraph
|
||||
@@ -175,7 +175,7 @@ icon: "rocket"
|
||||
</Tab>
|
||||
|
||||
<Tab title="MCP Integration">
|
||||
Use Semantica from Claude Desktop, VS Code, Cursor, or any MCP client — no Python code required after setup.
|
||||
Use Semantica from Claude Desktop, VS Code, Cursor, or any MCP client: no Python code required after setup.
|
||||
|
||||
```bash
|
||||
pip install semantica
|
||||
@@ -202,7 +202,7 @@ icon: "rocket"
|
||||
|
||||
## Core Architecture
|
||||
|
||||
Semantica uses a modular, layered architecture — import only what you need.
|
||||
Semantica uses a modular, layered architecture: import only what you need.
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Input Layer" icon="database" href="reference/ingest">
|
||||
|
||||
+24
-24
@@ -17,22 +17,22 @@ A quick-reference dictionary of every concept, data structure, algorithm, and st
|
||||
An autonomous AI system that perceives its environment, reasons about information, and takes actions to achieve goals. In Semantica, agents use knowledge graphs for structured memory and context, with every decision recorded as a first-class object.
|
||||
|
||||
**Context Graph**
|
||||
A persistent, queryable graph of everything an agent knows, decides, and reasons about — entities, relationships, decisions, and their causal links. The core data structure of `semantica.context`.
|
||||
A persistent, queryable graph of everything an agent knows, decides, and reasons about: entities, relationships, decisions, and their causal links. The core data structure of `semantica.context`.
|
||||
|
||||
**Decision**
|
||||
A first-class object in Semantica: a recorded agent choice with category, scenario, reasoning, outcome, confidence score, causal chain, and source provenance. Stored and searchable via `context.record_decision()`.
|
||||
|
||||
**Entity**
|
||||
A distinct object or concept in the real world — a person, organization, location, event, or abstract concept. Entities are nodes in a knowledge graph, each with typed properties and a source provenance record.
|
||||
A distinct object or concept in the real world: a person, organization, location, event, or abstract concept. Entities are nodes in a knowledge graph, each with typed properties and a source provenance record.
|
||||
|
||||
**Knowledge Graph (KG)**
|
||||
A structured representation of knowledge using entities (nodes) and relationships (edges). Knowledge graphs enable reasoning, querying, semantic search, and traceable inference — unlike flat vector stores.
|
||||
A structured representation of knowledge using entities (nodes) and relationships (edges). Knowledge graphs enable reasoning, querying, semantic search, and traceable inference: unlike flat vector stores.
|
||||
|
||||
**Relationship**
|
||||
A directed, typed connection between two entities: e.g., `works_for`, `located_in`, `founded_by`. Relationships carry confidence scores and provenance back to the source document.
|
||||
|
||||
**Semantic**
|
||||
Relating to meaning in language or logic. Semantic understanding captures context and intent — going beyond keyword matching to understand what text *means*.
|
||||
Relating to meaning in language or logic. Semantic understanding captures context and intent: going beyond keyword matching to understand what text *means*.
|
||||
|
||||
|
||||
## Data Processing
|
||||
@@ -41,19 +41,19 @@ Relating to meaning in language or logic. Semantic understanding captures contex
|
||||
Splitting large documents into smaller pieces while preserving semantic context. Semantica supports recursive, semantic boundary, entity-aware, relation-aware, sliding window, structural, and table-aware chunking strategies.
|
||||
|
||||
**Ingestion**
|
||||
Loading data from external sources — files, databases, APIs, streams — into the pipeline as a unified `SourceDocument`. The first stage in every Semantica pipeline.
|
||||
Loading data from external sources: files, databases, APIs, streams: into the pipeline as a unified `SourceDocument`. The first stage in every Semantica pipeline.
|
||||
|
||||
**Normalization**
|
||||
Standardizing data into a consistent canonical form: converting dates to ISO format, canonicalizing entity names, fixing encoding issues, stripping noise. Ensures downstream extraction works on clean, consistent text.
|
||||
|
||||
**Parsing**
|
||||
Extracting structured text, layout, and metadata from unstructured or semi-structured documents — PDFs, Word files, HTML, PPTX. `DoclingParser` additionally handles multi-column layouts, merged-cell tables, and OCR.
|
||||
Extracting structured text, layout, and metadata from unstructured or semi-structured documents: PDFs, Word files, HTML, PPTX. `DoclingParser` additionally handles multi-column layouts, merged-cell tables, and OCR.
|
||||
|
||||
|
||||
## Artificial Intelligence
|
||||
|
||||
**Abductive Reasoning**
|
||||
Inference to the most plausible explanation for observed facts. One of six reasoning engines in `semantica.reasoning` — returns the most likely hypothesis given available evidence.
|
||||
Inference to the most plausible explanation for observed facts. One of six reasoning engines in `semantica.reasoning`: returns the most likely hypothesis given available evidence.
|
||||
|
||||
**Datalog**
|
||||
A declarative logic programming language for knowledge base queries. Semantica's `DatalogEngine` supports recursive Horn clause rules with bottom-up semi-naive fixpoint semantics. Added in v0.4.0.
|
||||
@@ -62,7 +62,7 @@ A declarative logic programming language for knowledge base queries. Semantica's
|
||||
An advanced RAG approach that combines vector similarity search with knowledge graph traversal. Every LLM response is grounded in structured graph context, with each claim traceable to a source node. Eliminates hallucination without source attribution.
|
||||
|
||||
**Inference**
|
||||
Deriving new facts or conclusions from existing knowledge using logical rules — without the derived facts being explicitly present in the source data.
|
||||
Deriving new facts or conclusions from existing knowledge using logical rules: without the derived facts being explicitly present in the source data.
|
||||
|
||||
**LLM (Large Language Model)**
|
||||
An AI model trained on large text corpora, capable of understanding and generating natural language. Semantica integrates with 8+ LLM providers for entity extraction, relation extraction, and reasoning.
|
||||
@@ -74,7 +74,7 @@ A technique that enhances LLM outputs by retrieving relevant context from a know
|
||||
## Knowledge Graph Components
|
||||
|
||||
**Allen Interval Algebra**
|
||||
A system of 13 relations for describing how two time intervals relate — before, after, meets, overlaps, during, starts, finishes, equals, and their inverses. Supported in `TemporalKnowledgeGraph` since v0.4.0.
|
||||
A system of 13 relations for describing how two time intervals relate: before, after, meets, overlaps, during, starts, finishes, equals, and their inverses. Supported in `TemporalKnowledgeGraph` since v0.4.0.
|
||||
|
||||
**BiTemporalFact**
|
||||
A fact with two independent time dimensions: *valid time* (when it was true in the world) and *transaction time* (when it was recorded in the system). Enables full audit trails for slowly changing data.
|
||||
@@ -86,43 +86,43 @@ A directed connection between two nodes in a graph, representing a typed relatio
|
||||
A vertex in a knowledge graph representing an entity or concept. Nodes carry typed properties, a confidence score, and provenance linking back to the source document.
|
||||
|
||||
**Property**
|
||||
An attribute or characteristic of an entity or relationship — name, date, URI, confidence score, source URL.
|
||||
An attribute or characteristic of an entity or relationship: name, date, URI, confidence score, source URL.
|
||||
|
||||
**Temporal Graph**
|
||||
A knowledge graph where nodes and edges carry `valid_from` / `valid_until` time windows, enabling point-in-time queries and historical state reconstruction.
|
||||
|
||||
**Triplet**
|
||||
The atomic unit of knowledge: a `(subject, predicate, object)` triple — e.g., `(Apple_Inc, founded_by, Steve_Jobs)`. The building block of RDF and SPARQL-based storage.
|
||||
The atomic unit of knowledge: a `(subject, predicate, object)` triple: e.g., `(Apple_Inc, founded_by, Steve_Jobs)`. The building block of RDF and SPARQL-based storage.
|
||||
|
||||
|
||||
## Entity Recognition & Extraction
|
||||
|
||||
**Coreference Resolution**
|
||||
Determining when multiple expressions in text refer to the same entity — e.g., "Apple" and "the company" both referring to Apple Inc. Handled by `CoreferenceResolver` in `semantica.semantic_extract`.
|
||||
Determining when multiple expressions in text refer to the same entity: e.g., "Apple" and "the company" both referring to Apple Inc. Handled by `CoreferenceResolver` in `semantica.semantic_extract`.
|
||||
|
||||
**Entity Resolution**
|
||||
Determining when two entity mentions across different documents refer to the same real-world entity. Also called entity linking or deduplication. Uses similarity scoring, blocking, and semantic embeddings.
|
||||
|
||||
**Event Detection**
|
||||
Identifying and classifying events in text — acquisitions, partnerships, product launches, regulatory decisions. Handled by `EventDetector` in `semantica.semantic_extract`.
|
||||
Identifying and classifying events in text: acquisitions, partnerships, product launches, regulatory decisions. Handled by `EventDetector` in `semantica.semantic_extract`.
|
||||
|
||||
**Named Entity Recognition (NER)**
|
||||
Identifying and classifying named entities in text into predefined categories: persons, organizations, locations, dates, products, and custom types. Three modes: pattern-based, ML-based, and LLM-based.
|
||||
|
||||
**Relationship Extraction**
|
||||
Identifying and extracting typed semantic relationships between entities — e.g., `(Google, acquired, DeepMind)` — from raw text.
|
||||
Identifying and extracting typed semantic relationships between entities: e.g., `(Google, acquired, DeepMind)`: from raw text.
|
||||
|
||||
|
||||
## Ontology & Schema
|
||||
|
||||
**Axiom**
|
||||
A statement accepted as true in an ontology, used to define logical constraints — e.g., "every Person must have a name", "Organization can have at most one CEO at a time".
|
||||
A statement accepted as true in an ontology, used to define logical constraints: e.g., "every Person must have a name", "Organization can have at most one CEO at a time".
|
||||
|
||||
**Class**
|
||||
A category or type of entity in an ontology — `Person`, `Organization`, `Location`. Classes form a hierarchy and carry constraints validated by SHACL.
|
||||
A category or type of entity in an ontology: `Person`, `Organization`, `Location`. Classes form a hierarchy and carry constraints validated by SHACL.
|
||||
|
||||
**Ontology**
|
||||
A formal specification of domain concepts, relationships, and constraints — typically expressed in OWL. Semantica can auto-generate ontologies from knowledge graphs or import existing OWL/RDF/Turtle files.
|
||||
A formal specification of domain concepts, relationships, and constraints: typically expressed in OWL. Semantica can auto-generate ontologies from knowledge graphs or import existing OWL/RDF/Turtle files.
|
||||
|
||||
**Ontology Hub**
|
||||
Semantica's v0.5.0 visual browser UI for the full ontology lifecycle: visual class editor, SHACL Studio, alignment authoring, health dashboard, and version-controlled diffs.
|
||||
@@ -140,13 +140,13 @@ A W3C standard for representing controlled vocabularies, taxonomies, and thesaur
|
||||
## Storage & Retrieval
|
||||
|
||||
**Embedding**
|
||||
A dense numerical vector that represents text, images, or other data in a continuous semantic space. Entities with similar meaning produce vectors that are close together — enabling similarity search and semantic matching.
|
||||
A dense numerical vector that represents text, images, or other data in a continuous semantic space. Entities with similar meaning produce vectors that are close together: enabling similarity search and semantic matching.
|
||||
|
||||
**Graph Database**
|
||||
A database optimized for storing and querying graph-structured data using node and edge primitives. Semantica supports Neo4j, FalkorDB, Apache AGE, and Amazon Neptune.
|
||||
|
||||
**Hybrid Search**
|
||||
A retrieval strategy combining vector similarity search with keyword or metadata filtering — higher accuracy than either approach alone.
|
||||
A retrieval strategy combining vector similarity search with keyword or metadata filtering: higher accuracy than either approach alone.
|
||||
|
||||
**Triplet Store**
|
||||
A database designed specifically for storing and querying RDF `(subject, predicate, object)` triples. Semantica supports Blazegraph, Apache Jena, and RDF4J.
|
||||
@@ -161,7 +161,7 @@ A database optimized for storing and searching high-dimensional embedding vector
|
||||
A measure of a node's importance in the graph. Common metrics: PageRank (link-based importance), betweenness centrality (bridge nodes), closeness centrality (average distance to all others).
|
||||
|
||||
**Community Detection**
|
||||
Identifying groups of densely connected nodes — clusters that share more internal links than external ones. Used for finding subject communities, fraud rings, and organizational clusters.
|
||||
Identifying groups of densely connected nodes: clusters that share more internal links than external ones. Used for finding subject communities, fraud rings, and organizational clusters.
|
||||
|
||||
**Distance Band**
|
||||
A classification of a node's semantic proximity to a target: `near`, `mid`, or `far`, based on embedding distance thresholds. Part of Distance Intelligence (v0.5.0).
|
||||
@@ -170,7 +170,7 @@ A classification of a node's semantic proximity to a target: `near`, `mid`, or `
|
||||
Semantica's v0.5.0 feature for semantic neighborhood exploration: N×N distance matrices, ego-mode visualization centered on a single entity, and distance band classification across the graph.
|
||||
|
||||
**PageRank**
|
||||
An algorithm measuring node importance based on the structure of incoming relationships — originally designed for web pages, applicable to any directed graph.
|
||||
An algorithm measuring node importance based on the structure of incoming relationships: originally designed for web pages, applicable to any directed graph.
|
||||
|
||||
|
||||
## Query Languages & Standards
|
||||
@@ -194,13 +194,13 @@ The W3C query language for RDF data. Semantica's `SparqlReasoner` uses SPARQL fo
|
||||
Handling contradictory facts from multiple sources in the same knowledge graph. Semantica's `ConflictDetector` surfaces conflicts; resolution strategies include prefer-most-recent, prefer-most-reliable, majority-vote, and flag-for-review.
|
||||
|
||||
**Data Provenance**
|
||||
Complete information about the origin, history, and lineage of every fact — source document, extraction method, timestamp, confidence score. W3C PROV-O compliant in Semantica.
|
||||
Complete information about the origin, history, and lineage of every fact: source document, extraction method, timestamp, confidence score. W3C PROV-O compliant in Semantica.
|
||||
|
||||
**Deduplication**
|
||||
Identifying and merging duplicate entity records. Semantica v2 strategies (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to 7x faster than v1.
|
||||
|
||||
**W3C PROV-O**
|
||||
The W3C provenance ontology standard. Semantica tracks lineage across all modules in PROV-O compliant format — suitable for HIPAA, SOX, GDPR, and FDA 21 CFR Part 11 compliance.
|
||||
The W3C provenance ontology standard. Semantica tracks lineage across all modules in PROV-O compliant format: suitable for HIPAA, SOX, GDPR, and FDA 21 CFR Part 11 compliance.
|
||||
|
||||
|
||||
## Security Terms
|
||||
@@ -219,7 +219,7 @@ A vulnerability in XML parsers that allows attackers to read arbitrary files or
|
||||
Deeper explanation of key ideas with code examples.
|
||||
</Card>
|
||||
<Card title="Getting Started" icon="play" href="getting-started">
|
||||
First working examples — no prior graph experience required.
|
||||
First working examples: no prior graph experience required.
|
||||
</Card>
|
||||
<Card title="Modules Guide" icon="cubes" href="modules">
|
||||
All 27 modules explained with code and pipeline chains.
|
||||
|
||||
+11
-11
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Governance"
|
||||
description: "Project governance model — roles, decision process, release cadence, and code review guidelines."
|
||||
description: "Project governance model: roles, decision process, release cadence, and code review guidelines."
|
||||
icon: "scale-balanced"
|
||||
---
|
||||
|
||||
@@ -11,7 +11,7 @@ icon: "scale-balanced"
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Maintainers" icon="shield-halved">
|
||||
Hawksight AI team — review and merge PRs, manage releases and code quality, set project direction and community standards.
|
||||
Hawksight AI team: review and merge PRs, manage releases and code quality, set project direction and community standards.
|
||||
</Card>
|
||||
<Card title="Contributors" icon="code-pull-request">
|
||||
Submit code, documentation, and bug reports. Help with issues and reviews. Recognized in [CONTRIBUTORS.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTORS.md).
|
||||
@@ -65,36 +65,36 @@ Semantica follows **Semantic Versioning** (`MAJOR.MINOR.PATCH`):
|
||||
|
||||
## Communication
|
||||
|
||||
- **GitHub Issues** — bug reports, feature requests, questions
|
||||
- **GitHub PRs** — code contributions
|
||||
- **GitHub Discussions** — community conversation
|
||||
- **Security Advisories** — [report security issues privately](https://github.com/semantica-agi/semantica/security/advisories/new)
|
||||
- **GitHub Issues**: bug reports, feature requests, questions
|
||||
- **GitHub PRs**: code contributions
|
||||
- **GitHub Discussions**: community conversation
|
||||
- **Security Advisories**: [report security issues privately](https://github.com/semantica-agi/semantica/security/advisories/new)
|
||||
|
||||
|
||||
## Project Goals
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Usability" icon="hand-pointer">
|
||||
Easy to use and understand — sensible defaults, clear documentation, minimal ceremony.
|
||||
Easy to use and understand: sensible defaults, clear documentation, minimal ceremony.
|
||||
</Card>
|
||||
<Card title="Reliability" icon="circle-check">
|
||||
Production-ready quality — tested across Python versions, platforms, and real-world workloads.
|
||||
Production-ready quality: tested across Python versions, platforms, and real-world workloads.
|
||||
</Card>
|
||||
<Card title="Performance" icon="bolt">
|
||||
Efficient and scalable — from single-machine notebooks to enterprise graph databases.
|
||||
Efficient and scalable: from single-machine notebooks to enterprise graph databases.
|
||||
</Card>
|
||||
<Card title="Extensibility" icon="puzzle-piece">
|
||||
Easy to extend with plugins and custom modules via the `PluginRegistry` pattern.
|
||||
</Card>
|
||||
<Card title="Community" icon="heart">
|
||||
Welcoming and inclusive — all backgrounds and experience levels contribute and are recognized.
|
||||
Welcoming and inclusive: all backgrounds and experience levels contribute and are recognized.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
|
||||
## License
|
||||
|
||||
MIT License — see [LICENSE](https://github.com/semantica-agi/semantica/blob/main/LICENSE) and the [License page](license).
|
||||
MIT License: see [LICENSE](https://github.com/semantica-agi/semantica/blob/main/LICENSE) and the [License page](license).
|
||||
|
||||
|
||||
## See Also
|
||||
|
||||
@@ -29,7 +29,7 @@ pip install psycopg2-binary
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Unified Facade">
|
||||
Use `GraphStore(backend="age", …)` — the same interface as Neo4j and FalkorDB:
|
||||
Use `GraphStore(backend="age", …)`: the same interface as Neo4j and FalkorDB:
|
||||
|
||||
```python
|
||||
from semantica.graph_store import GraphStore
|
||||
@@ -94,7 +94,7 @@ graph_store_config.set("age_graph_name", "production")
|
||||
|
||||
## Connection & Initialization
|
||||
|
||||
On `connect()`, the store performs idempotent setup — safe to call repeatedly:
|
||||
On `connect()`, the store performs idempotent setup: safe to call repeatedly:
|
||||
|
||||
<Steps>
|
||||
<Step title="Load the extension">
|
||||
|
||||
+36
-36
@@ -1,19 +1,19 @@
|
||||
---
|
||||
title: "Semantica"
|
||||
description: "The Accountability and Context Layer for AI — Context Graphs · Decision Intelligence · Full Provenance"
|
||||
description: "The Accountability and Context Layer for AI: Context Graphs · Decision Intelligence · Full Provenance"
|
||||
---
|
||||
|
||||
<Info>
|
||||
**v0.5.0 is live** — Ontology Hub, Distance Intelligence, SHACL Studio, Parquet & XML ingestion, 12 security fixes. [What's new →](#whats-new)
|
||||
**v0.5.0 is live**: Ontology Hub, Distance Intelligence, SHACL Studio, Parquet & XML ingestion, 12 security fixes. [What's new →](#whats-new)
|
||||
</Info>
|
||||
|
||||
Your AI agent just made a decision. Now someone needs to explain it.
|
||||
|
||||
*What did it know at the time? Which facts shaped the outcome? Where did those facts come from? Has it made the same call before — and did that go well?*
|
||||
*What did it know at the time? Which facts shaped the outcome? Where did those facts come from? Has it made the same call before: and did that go well?*
|
||||
|
||||
If your stack can't answer those questions with a traceable record, you have a gap. Not a capability gap — an **accountability gap**. It's the reason AI hasn't landed at scale in healthcare, finance, legal, and government. And it's why teams building for those markets keep rebuilding the same guardrails from scratch.
|
||||
If your stack can't answer those questions with a traceable record, you have a gap. Not a capability gap: an **accountability gap**. It's the reason AI hasn't landed at scale in healthcare, finance, legal, and government. And it's why teams building for those markets keep rebuilding the same guardrails from scratch.
|
||||
|
||||
**Semantica closes that gap.** It's the context and accountability layer that sits beneath your existing agent framework — not a replacement for LangChain or LlamaIndex, but the infrastructure that makes their outputs trustworthy.
|
||||
**Semantica closes that gap.** It's the context and accountability layer that sits beneath your existing agent framework: not a replacement for LangChain or LlamaIndex, but the infrastructure that makes their outputs trustworthy.
|
||||
|
||||
<CardGroup cols={4}>
|
||||
<Card title="1,000+ Tests" icon="circle-check">
|
||||
@@ -50,7 +50,7 @@ Powerful agents aren't automatically trustworthy ones. Five structural blind spo
|
||||
</Card>
|
||||
<Card title="No provenance" icon="link-slash">
|
||||
Outputs can't be traced to source facts.
|
||||
- In healthcare, finance, and legal — this is a hard compliance blocker
|
||||
- In healthcare, finance, and legal: this is a hard compliance blocker
|
||||
- No lineage from inference back to the original document
|
||||
- Impossible to demonstrate what the agent actually relied on
|
||||
</Card>
|
||||
@@ -69,7 +69,7 @@ Powerful agents aren't automatically trustworthy ones. Five structural blind spo
|
||||
</CardGroup>
|
||||
|
||||
<Note>
|
||||
These aren't edge cases. They're why enterprise AI pilots stall — and why your compliance team keeps saying *not yet*.
|
||||
These aren't edge cases. They're why enterprise AI pilots stall: and why your compliance team keeps saying *not yet*.
|
||||
</Note>
|
||||
|
||||
|
||||
@@ -80,7 +80,7 @@ Semantica gives every agent the infrastructure it needs to be accountable. Drop
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Context Graphs" icon="diagram-project">
|
||||
A structured, queryable graph of everything your agent knows, decides, and reasons about.
|
||||
- Persistent across agent runs — no context loss between sessions
|
||||
- Persistent across agent runs: no context loss between sessions
|
||||
- Queryable with SPARQL and full graph algorithms
|
||||
- Temporal model with `valid_from` / `valid_until` on nodes and edges
|
||||
- Point-in-time snapshots of the full knowledge state
|
||||
@@ -100,15 +100,15 @@ Semantica gives every agent the infrastructure it needs to be accountable. Drop
|
||||
- Audit-ready for HIPAA, SOX, GDPR, FDA 21 CFR Part 11
|
||||
</Card>
|
||||
<Card title="Reasoning Engines" icon="microchip">
|
||||
Explainable reasoning paths — not black boxes.
|
||||
Explainable reasoning paths: not black boxes.
|
||||
- Forward chaining, Rete, deductive, abductive
|
||||
- SPARQL query-based inference over RDF graphs
|
||||
- Datalog with recursive Horn clause rules
|
||||
- Every conclusion backed by a traceable derivation path
|
||||
</Card>
|
||||
<Card title="Temporal Intelligence" icon="clock">
|
||||
Your graph knows not just *what* — but *when*.
|
||||
- Allen interval algebra — all 13 temporal relations
|
||||
Your graph knows not just *what*: but *when*.
|
||||
- Allen interval algebra: all 13 temporal relations
|
||||
- Point-in-time queries over historical graph states
|
||||
- Temporal provenance stamping on every fact
|
||||
- OWL-Time export for standards-compliant archiving
|
||||
@@ -123,7 +123,7 @@ Semantica gives every agent the infrastructure it needs to be accountable. Drop
|
||||
</CardGroup>
|
||||
|
||||
<Tip>
|
||||
Works alongside any LLM provider and any agent framework — add it to an existing stack without changing your architecture.
|
||||
Works alongside any LLM provider and any agent framework: add it to an existing stack without changing your architecture.
|
||||
</Tip>
|
||||
|
||||
<img src="/assets/img/diagrams/architecture-overview.svg" alt="Semantica four-layer architecture: Ingestion → Processing → Intelligence → Application" style={{ width: '100%', borderRadius: '12px', margin: '24px 0' }} />
|
||||
@@ -203,7 +203,7 @@ context = AgentContext(
|
||||
llm=LiteLLM(model="ollama/llama3.2", base_url="http://localhost:11434"),
|
||||
)
|
||||
|
||||
# Fully local — no data leaves your infrastructure
|
||||
# Fully local: no data leaves your infrastructure
|
||||
context.store("Local LLMs enable air-gapped compliance deployments")
|
||||
|
||||
decision_id = context.record_decision(
|
||||
@@ -292,7 +292,7 @@ Semantica was designed for domains where every decision must be explainable and
|
||||
</Step>
|
||||
<Step title="Learn the mental model">
|
||||
[Core Concepts](concepts) covers:
|
||||
- Knowledge graphs vs. vector stores — when to use each
|
||||
- Knowledge graphs vs. vector stores: when to use each
|
||||
- What GraphRAG is and how Semantica implements it
|
||||
- How provenance and decision tracking work together
|
||||
- The accountability layer architecture
|
||||
@@ -328,15 +328,15 @@ Semantica was designed for domains where every decision must be explainable and
|
||||
|
||||
<AccordionGroup>
|
||||
|
||||
<Accordion title="v0.5.0 — Ontology Hub & Distance Intelligence" icon="star" defaultOpen={true}>
|
||||
<Accordion title="v0.5.0: Ontology Hub & Distance Intelligence" icon="star" defaultOpen={true}>
|
||||
|
||||
Released **May 11, 2026**
|
||||
|
||||
| Area | Highlights |
|
||||
| :------ | :------------ |
|
||||
| **Ontology Hub** | Visual editor, SHACL Studio, alignment authoring, health dashboard, version control — full ontology lifecycle in the browser |
|
||||
| **Ontology Hub** | Visual editor, SHACL Studio, alignment authoring, health dashboard, version control: full ontology lifecycle in the browser |
|
||||
| **Distance Intelligence** | Semantic neighborhoods, N×N distance matrices, ego-mode visualization, distance band classification, embedding cache optimization |
|
||||
| **Parquet Ingestion** | `ParquetIngestor` with PyArrow — single file, partitioned directories, Hive-style discovery, selective column reading |
|
||||
| **Parquet Ingestion** | `ParquetIngestor` with PyArrow: single file, partitioned directories, Hive-style discovery, selective column reading |
|
||||
| **XML Ingestion** | `XMLIngestor` with XXE-safe lxml backend, XSD/DTD validation, namespace handling, directory scanning |
|
||||
| **Graph Explorer** | Landing page redesign, bidirectional path finding, indexed search (0.004ms on 118k nodes) |
|
||||
| **Security** | 12 vulnerability fixes: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, path traversal |
|
||||
@@ -348,12 +348,12 @@ pip install semantica==0.5.0
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="v0.4.0 — Temporal Intelligence & Knowledge Explorer" icon="clock">
|
||||
<Accordion title="v0.4.0: Temporal Intelligence & Knowledge Explorer" icon="clock">
|
||||
|
||||
| Area | Highlights |
|
||||
| :------ | :------------ |
|
||||
| **Temporal Intelligence** | 6-PR system: temporal data model, point-in-time queries, Allen interval algebra (all 13 relations), OWL-Time export |
|
||||
| **Knowledge Explorer API** | Full FastAPI backend — 99 tests, 12 export formats, WebSocket progress, thread-safe sessions, audit trail |
|
||||
| **Knowledge Explorer API** | Full FastAPI backend: 99 tests, 12 export formats, WebSocket progress, thread-safe sessions, audit trail |
|
||||
| **Ontology Foundations** | SHACL generation/validation, SKOS vocabulary, ontology alignment API, diff & migration tooling |
|
||||
| **Datalog Reasoning** | Pure-Python bottom-up semi-naive fixpoint, recursive Horn clause rules, guaranteed termination |
|
||||
| **Agno Integration** | 5 components: graph-backed memory, multi-hop GraphRAG, decision toolkit, KG toolkit, shared team context; 110 tests |
|
||||
@@ -374,7 +374,7 @@ pip install semantica==0.5.0
|
||||
- Structured, persistent graph of entities, relationships, and decisions
|
||||
- Temporal model with `valid_from` / `valid_until` on every node and edge
|
||||
- Point-in-time queries across historical graph states
|
||||
- Distance Intelligence — semantic neighborhoods and N×N distance matrices
|
||||
- Distance Intelligence: semantic neighborhoods and N×N distance matrices
|
||||
|
||||
### Decision Tracking
|
||||
|
||||
@@ -389,16 +389,16 @@ pip install semantica==0.5.0
|
||||
|
||||
### Entity & Relation Extraction
|
||||
|
||||
- Named entity recognition — pattern, ML, or LLM methods
|
||||
- Named entity recognition: pattern, ML, or LLM methods
|
||||
- Typed triplet extraction via LLM or rule-based pipelines
|
||||
- Event extraction with temporal and causal linking
|
||||
|
||||
### Ontology & Schema
|
||||
|
||||
- Ontology Hub — visual editor, SHACL Studio, alignments, health dashboard
|
||||
- Deduplication v2 — `blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7x faster
|
||||
- Datalog reasoning — recursive Horn clause rules with fixpoint semantics
|
||||
- SPARQL reasoning — query-based inference over RDF graphs
|
||||
- Ontology Hub: visual editor, SHACL Studio, alignments, health dashboard
|
||||
- Deduplication v2: `blocking_v2`, `hybrid_v2`, `semantic_v2`: up to 7x faster
|
||||
- Datalog reasoning: recursive Horn clause rules with fixpoint semantics
|
||||
- SPARQL reasoning: query-based inference over RDF graphs
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -406,17 +406,17 @@ pip install semantica==0.5.0
|
||||
|
||||
### Lineage Tracking
|
||||
|
||||
- W3C PROV-O lineage across all modules — every fact has a source
|
||||
- W3C PROV-O lineage across all modules: every fact has a source
|
||||
- `recorded_at` stamping with full OWL-Time export
|
||||
- Change management with SHA-256 checksums and version control
|
||||
- Full audit trails from ingestion event to final inference
|
||||
|
||||
### Compliance Infrastructure
|
||||
|
||||
- HIPAA — patient data handling with audit-ready provenance chains
|
||||
- SOX / MiFID II — financial decision records with full traceability
|
||||
- GDPR — data lineage for subject access and right-to-erasure workflows
|
||||
- FDA 21 CFR Part 11 — electronic records and signature compliance
|
||||
- HIPAA: patient data handling with audit-ready provenance chains
|
||||
- SOX / MiFID II: financial decision records with full traceability
|
||||
- GDPR: data lineage for subject access and right-to-erasure workflows
|
||||
- FDA 21 CFR Part 11: electronic records and signature compliance
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -458,25 +458,25 @@ pip install semantica==0.5.0
|
||||
| `semantica.reasoning` | Forward chaining, Rete, deductive, abductive, SPARQL, Datalog |
|
||||
| `semantica.ontology` | SHACL, SKOS, alignments, diff/migration, auto-generation, OWL/RDF |
|
||||
| `semantica.explorer` | FastAPI Knowledge Explorer, Ontology Hub, Distance Intelligence, SHACL Studio |
|
||||
| `semantica.mcp_server` | MCP stdio server — 12 tools for Claude Desktop, VS Code, Cursor, Windsurf, Cline |
|
||||
| `semantica.mcp_server` | MCP stdio server: 12 tools for Claude Desktop, VS Code, Cursor, Windsurf, Cline |
|
||||
| `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector |
|
||||
| `semantica.graph_store` | Neo4j, FalkorDB, Apache AGE, Amazon Neptune |
|
||||
| `semantica.triplet_store` | In-memory and persistent RDF triple store with SPARQL |
|
||||
| `semantica.ingest` | Files, web, feeds, databases, Snowflake, Parquet, XML, MCP |
|
||||
| `semantica.parse` | Document parsing — PDF, DOCX, HTML, PPTX, Docling layout analysis |
|
||||
| `semantica.split` | Text chunking — sentence, paragraph, token, semantic boundary strategies |
|
||||
| `semantica.parse` | Document parsing: PDF, DOCX, HTML, PPTX, Docling layout analysis |
|
||||
| `semantica.split` | Text chunking: sentence, paragraph, token, semantic boundary strategies |
|
||||
| `semantica.normalize` | Text normalization, entity canonicalization, whitespace and encoding cleanup |
|
||||
| `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama local embeddings |
|
||||
| `semantica.pipeline` | Pipeline DSL, parallel workers, retry policies, failure handling |
|
||||
| `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, OWL, Arrow, GraphML, GEXF, DOT |
|
||||
| `semantica.visualization` | Programmatic graph rendering — force, hierarchical, circular, spring layouts |
|
||||
| `semantica.visualization` | Programmatic graph rendering: force, hierarchical, circular, spring layouts |
|
||||
| `semantica.deduplication` | Entity deduplication v1/v2, similarity scoring, blocking, merging |
|
||||
| `semantica.conflicts` | Conflict detection and resolution across overlapping knowledge sources |
|
||||
| `semantica.provenance` | W3C PROV-O lineage tracking, source attribution, audit trails |
|
||||
| `semantica.change_management` | Version control with SHA-256 checksums, diff, rollback |
|
||||
| `semantica.llms` | Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, HuggingFace |
|
||||
| `semantica.seed` | Foundation graph seeding from CSV, JSON, SQL, API, and RDF sources |
|
||||
| `semantica.evals` | Evaluation harness — KG quality, extraction F1, pipeline benchmarking, regression tracking |
|
||||
| `semantica.evals` | Evaluation harness: KG quality, extraction F1, pipeline benchmarking, regression tracking |
|
||||
| `semantica.core` | Orchestration, ConfigManager, LifecycleManager, PluginRegistry, MethodRegistry |
|
||||
| `semantica.utils` | Logging, validation, progress tracking, hash utilities, nested dict helpers |
|
||||
|
||||
@@ -503,6 +503,6 @@ pip install semantica==0.5.0
|
||||
- Use `NERExtractor` without a graph store
|
||||
- Use `ContextGraph` without vector storage
|
||||
- Every component independently swappable and testable
|
||||
- No framework lock-in — works with any agent stack
|
||||
- No framework lock-in: works with any agent stack
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -5,7 +5,7 @@ icon: "download"
|
||||
---
|
||||
|
||||
<Check>
|
||||
**Available on PyPI** — `pip install semantica` and you're ready.
|
||||
**Available on PyPI**: `pip install semantica` and you're ready.
|
||||
</Check>
|
||||
|
||||
<Note>
|
||||
|
||||
@@ -26,19 +26,19 @@ pip install "semantica[agno,graph-neo4j,vectorstore-pgvector]"
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="AgnoContextStore" icon="database">
|
||||
`AgentMemory(db=…)` — Replaces Agno's flat storage with hybrid vector + context graph memory. Adds decision tracking and precedent search to any agent.
|
||||
`AgentMemory(db=…)`: Replaces Agno's flat storage with hybrid vector + context graph memory. Adds decision tracking and precedent search to any agent.
|
||||
</Card>
|
||||
<Card title="AgnoKnowledgeGraph" icon="diagram-project">
|
||||
`Agent(knowledge=…)` — Documents flow through the full Semantica extraction pipeline into a queryable `ContextGraph` with multi-hop GraphRAG.
|
||||
`Agent(knowledge=…)`: Documents flow through the full Semantica extraction pipeline into a queryable `ContextGraph` with multi-hop GraphRAG.
|
||||
</Card>
|
||||
<Card title="AgnoDecisionKit" icon="list-check">
|
||||
`Agent(tools=[…])` — 6 decision intelligence tools: record decisions, find precedents, trace causal chains, analyze impact, check policies, summarize history.
|
||||
`Agent(tools=[…])`: 6 decision intelligence tools: record decisions, find precedents, trace causal chains, analyze impact, check policies, summarize history.
|
||||
</Card>
|
||||
<Card title="AgnoKGToolkit" icon="wrench">
|
||||
`Agent(tools=[…])` — 7 KG construction tools: extract entities, extract relations, add to graph, query graph, find related, infer facts, export subgraph.
|
||||
`Agent(tools=[…])`: 7 KG construction tools: extract entities, extract relations, add to graph, query graph, find related, infer facts, export subgraph.
|
||||
</Card>
|
||||
<Card title="AgnoSharedContext" icon="users">
|
||||
Team-level — A single `ContextGraph` shared across all agents. Each agent gets a role-scoped view via `bind_agent()`. Writes are tagged by role.
|
||||
Team-level: A single `ContextGraph` shared across all agents. Each agent gets a role-scoped view via `bind_agent()`. Writes are tagged by role.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -224,11 +224,11 @@ from integrations.agno import (
|
||||
AgnoDecisionKit, # Decision intelligence Toolkit
|
||||
AgnoKGToolkit, # Knowledge graph Toolkit
|
||||
AgnoSharedContext, # Team-level shared context
|
||||
AGNO_AVAILABLE, # bool — True if agno is installed
|
||||
AGNO_AVAILABLE, # bool: True if agno is installed
|
||||
)
|
||||
```
|
||||
|
||||
All five classes are usable without `agno` installed — they carry the full Semantica API and degrade gracefully.
|
||||
All five classes are usable without `agno` installed: they carry the full Semantica API and degrade gracefully.
|
||||
|
||||
|
||||
## See Also
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Native Docling integration for high-fidelity PDF, DOCX, and PPTX p
|
||||
icon: "file-lines"
|
||||
---
|
||||
|
||||
> Parse complex documents — PDFs, DOCX, PPTX, HTML — with high-fidelity table extraction and built-in OCR.
|
||||
> Parse complex documents: PDFs, DOCX, PPTX, HTML: with high-fidelity table extraction and built-in OCR.
|
||||
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -34,7 +34,7 @@ ingestor = SnowflakeIngestor(
|
||||
)
|
||||
|
||||
data = ingestor.ingest_table("CUSTOMERS")
|
||||
print(f"Retrieved {data.row_count} rows — columns: {data.columns}")
|
||||
print(f"Retrieved {data.row_count} rows: columns: {data.columns}")
|
||||
```
|
||||
|
||||
<Tip>
|
||||
@@ -64,7 +64,7 @@ Use environment variables (or a `.env` file with `python-dotenv`) to keep creden
|
||||
warehouse="COMPUTE_WH",
|
||||
)
|
||||
```
|
||||
Preferred for production — no password stored in config.
|
||||
Preferred for production: no password stored in config.
|
||||
</Tab>
|
||||
<Tab title="OAuth">
|
||||
```python
|
||||
@@ -165,7 +165,7 @@ from semantica.ingest import SnowflakeConnector
|
||||
|
||||
connector = SnowflakeConnector(account="myaccount", user="myuser", password="mypassword")
|
||||
if not connector.test_connection():
|
||||
print("Connection failed — check credentials and account identifier")
|
||||
print("Connection failed: check credentials and account identifier")
|
||||
```
|
||||
|
||||
|
||||
|
||||
+23
-23
@@ -4,7 +4,7 @@ description: "Structured learning paths, configuration reference, troubleshootin
|
||||
icon: "graduation-cap"
|
||||
---
|
||||
|
||||
Whether you're running your first pipeline or deploying Semantica in production, this page gives you a structured path forward — from beginner to enterprise-grade usage.
|
||||
Whether you're running your first pipeline or deploying Semantica in production, this page gives you a structured path forward: from beginner to enterprise-grade usage.
|
||||
|
||||
|
||||
## Learning Paths
|
||||
@@ -30,19 +30,19 @@ Whether you're running your first pipeline or deploying Semantica in production,
|
||||
|
||||
<Steps>
|
||||
<Step title="Set up your environment">
|
||||
[Installation Guide](installation) — virtual environments, optional extras, platform-specific fixes.
|
||||
[Installation Guide](installation): virtual environments, optional extras, platform-specific fixes.
|
||||
</Step>
|
||||
<Step title="Understand the core ideas">
|
||||
[Core Concepts](concepts) — what knowledge graphs are, how embeddings work, what extraction does.
|
||||
[Core Concepts](concepts): what knowledge graphs are, how embeddings work, what extraction does.
|
||||
</Step>
|
||||
<Step title="Run your first example">
|
||||
[Getting Started](getting-started) — 5-minute code walkthrough with pattern-based extraction (no API key needed).
|
||||
[Getting Started](getting-started): 5-minute code walkthrough with pattern-based extraction (no API key needed).
|
||||
</Step>
|
||||
<Step title="Build your first knowledge graph">
|
||||
[Quickstart Tutorial](quickstart) — full 6-step pipeline from ingestion to visualization.
|
||||
[Quickstart Tutorial](quickstart): full 6-step pipeline from ingestion to visualization.
|
||||
</Step>
|
||||
<Step title="Explore interactively">
|
||||
[Welcome to Semantica notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb) — Jupyter walkthrough of every module.
|
||||
[Welcome to Semantica notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/01_Welcome_to_Semantica.ipynb): Jupyter walkthrough of every module.
|
||||
</Step>
|
||||
</Steps>
|
||||
</Tab>
|
||||
@@ -51,16 +51,16 @@ Whether you're running your first pipeline or deploying Semantica in production,
|
||||
|
||||
<Steps>
|
||||
<Step title="Learn every module">
|
||||
[Modules Guide](modules) — all 27 modules with code examples and common pipeline chains.
|
||||
[Modules Guide](modules): all 27 modules with code examples and common pipeline chains.
|
||||
</Step>
|
||||
<Step title="Build production knowledge graphs">
|
||||
[Building Knowledge Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) — multi-source, deduplication, conflict resolution.
|
||||
[Building Knowledge Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb): multi-source, deduplication, conflict resolution.
|
||||
</Step>
|
||||
<Step title="Add semantic search">
|
||||
[Embeddings notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb) — providers, pooling strategies, vector stores.
|
||||
[Embeddings notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/09_Embeddings.ipynb): providers, pooling strategies, vector stores.
|
||||
</Step>
|
||||
<Step title="Build a GraphRAG system">
|
||||
[GraphRAG Complete notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — hybrid retrieval, reasoning, source attribution.
|
||||
[GraphRAG Complete notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb): hybrid retrieval, reasoning, source attribution.
|
||||
</Step>
|
||||
<Step title="Multi-source integration">
|
||||
[Multi-Source Data Integration notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/06_Multi_Source_Data_Integration.ipynb) and [Use Cases](use-cases) for domain-specific patterns.
|
||||
@@ -72,19 +72,19 @@ Whether you're running your first pipeline or deploying Semantica in production,
|
||||
|
||||
<Steps>
|
||||
<Step title="Understand the architecture">
|
||||
[Architecture Guide](architecture) — four-layer design, extension points, and design decisions.
|
||||
[Architecture Guide](architecture): four-layer design, extension points, and design decisions.
|
||||
</Step>
|
||||
<Step title="Temporal intelligence">
|
||||
[Temporal Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb) — `valid_from`/`valid_until`, Allen interval algebra, point-in-time queries.
|
||||
[Temporal Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/04_Temporal_Graphs.ipynb): `valid_from`/`valid_until`, Allen interval algebra, point-in-time queries.
|
||||
</Step>
|
||||
<Step title="Ontology-driven knowledge bases">
|
||||
[Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb) — auto-generation, SHACL validation, Ontology Hub (v0.5.0).
|
||||
[Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb): auto-generation, SHACL validation, Ontology Hub (v0.5.0).
|
||||
</Step>
|
||||
<Step title="Advanced visualization">
|
||||
[Complete Visualization Suite notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb) — UMAP, t-SNE, community layouts, embedding projections.
|
||||
[Complete Visualization Suite notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb): UMAP, t-SNE, community layouts, embedding projections.
|
||||
</Step>
|
||||
<Step title="Enterprise export">
|
||||
[Multi-Format Export notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb) — RDF with PROV-O, Parquet, Neo4j Cypher, Arrow, OWL.
|
||||
[Multi-Format Export notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/05_Multi_Format_Export.ipynb): RDF with PROV-O, Parquet, Neo4j Cypher, Arrow, OWL.
|
||||
</Step>
|
||||
</Steps>
|
||||
</Tab>
|
||||
@@ -93,7 +93,7 @@ Whether you're running your first pipeline or deploying Semantica in production,
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
All settings can be overridden with environment variables — no code changes needed.
|
||||
All settings can be overridden with environment variables: no code changes needed.
|
||||
|
||||
| Setting | Environment Variable | Default |
|
||||
| :------- | :-------------------- | :------- |
|
||||
@@ -126,7 +126,7 @@ pip install "semantica[gpu]" # GPU acceleration
|
||||
|
||||
### `AuthenticationError`
|
||||
|
||||
Set your API key as an environment variable — never hardcode keys in source files:
|
||||
Set your API key as an environment variable: never hardcode keys in source files:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY="sk-..."
|
||||
@@ -189,14 +189,14 @@ set PYTHONIOENCODING=utf-8
|
||||
| :--------- | :------------------ | :---------------- |
|
||||
| Graph construction | Fast | Moderate |
|
||||
| Query performance | Moderate | Fast |
|
||||
| Scalability | Low — in-memory only | High — persistent |
|
||||
| Scalability | Low: in-memory only | High: persistent |
|
||||
| Recommended for | Development, small graphs | Production, large corpora |
|
||||
|
||||
Use NetworkX for local development and prototyping. Switch to a persistent backend before deploying to production.
|
||||
|
||||
### Batch Processing
|
||||
|
||||
Process documents in batches rather than one at a time. Configure `chunk_size` based on available RAM — a good starting point is 1,000 documents per batch on a 16 GB machine.
|
||||
Process documents in batches rather than one at a time. Configure `chunk_size` based on available RAM: a good starting point is 1,000 documents per batch on a 16 GB machine.
|
||||
|
||||
### Deduplication v2
|
||||
|
||||
@@ -210,10 +210,10 @@ merged = resolver.resolve(entities, strategy="semantic_v2") # up to 7x faster
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
- **API keys** — store in environment variables or a secrets manager; never commit them to version control; rotate on a schedule
|
||||
- **Sensitive data** — use local embedding models (Ollama, HuggingFace) for PII or classified content; avoid sending sensitive data to external APIs without data handling agreements
|
||||
- **Graph exports** — encrypt sensitive exports at rest; use the v0.5.0 SSRF-safe `base_url` validation when configuring custom LLM gateways
|
||||
- **XML ingestion** — always use `XMLIngestor` (v0.5.0), which uses the XXE-safe lxml backend; never parse untrusted XML with the standard library parser
|
||||
- **API keys**: store in environment variables or a secrets manager; never commit them to version control; rotate on a schedule
|
||||
- **Sensitive data**: use local embedding models (Ollama, HuggingFace) for PII or classified content; avoid sending sensitive data to external APIs without data handling agreements
|
||||
- **Graph exports**: encrypt sensitive exports at rest; use the v0.5.0 SSRF-safe `base_url` validation when configuring custom LLM gateways
|
||||
- **XML ingestion**: always use `XMLIngestor` (v0.5.0), which uses the XXE-safe lxml backend; never parse untrusted XML with the standard library parser
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Cookbook" icon="flask" href="cookbook">
|
||||
|
||||
+12
-12
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Modules"
|
||||
description: "Every Semantica module works independently — use only what you need."
|
||||
description: "Every Semantica module works independently: use only what you need."
|
||||
icon: "puzzle-piece"
|
||||
---
|
||||
|
||||
@@ -8,7 +8,7 @@ icon: "puzzle-piece"
|
||||
Looking for a quick reference? Jump to the [Module Index](#module-index) at the bottom.
|
||||
</Tip>
|
||||
|
||||
Semantica is organized into **27 modules** across six logical layers. Each module is independently importable — you never pay for what you don't use.
|
||||
Semantica is organized into **27 modules** across six logical layers. Each module is independently importable: you never pay for what you don't use.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
@@ -51,7 +51,7 @@ documents = ingestor.ingest_directory("data/")
|
||||
web_ingestor = WebIngestor()
|
||||
pages = web_ingestor.ingest_urls(["https://example.com"])
|
||||
|
||||
# Parquet — single file, partitioned directory, Hive-style (v0.5.0)
|
||||
# Parquet: single file, partitioned directory, Hive-style (v0.5.0)
|
||||
parquet = ParquetIngestor()
|
||||
sources = parquet.ingest("data/events.parquet")
|
||||
|
||||
@@ -69,7 +69,7 @@ Extracts structured text and layout metadata from raw documents.
|
||||
```python
|
||||
from semantica.parse import DocumentParser, DoclingParser
|
||||
|
||||
# Standard parser — all common formats
|
||||
# Standard parser: all common formats
|
||||
parser = DocumentParser()
|
||||
parsed = parser.parse_document("document.pdf")
|
||||
|
||||
@@ -183,13 +183,13 @@ engine.apply_transitivity("located_in")
|
||||
engine.apply_symmetry("knows")
|
||||
result = engine.infer()
|
||||
|
||||
# Datalog — recursive Horn clause rules (v0.4.0)
|
||||
# Datalog: recursive Horn clause rules (v0.4.0)
|
||||
datalog = DatalogEngine()
|
||||
datalog.add_rule("ancestor(X, Z) :- parent(X, Y), ancestor(Y, Z).")
|
||||
results = datalog.query("ancestor(alice, ?)")
|
||||
```
|
||||
|
||||
**Engines:** forward chaining, Rete network, deductive, abductive, SPARQL, Datalog — all produce explainable inference paths
|
||||
**Engines:** forward chaining, Rete network, deductive, abductive, SPARQL, Datalog: all produce explainable inference paths
|
||||
|
||||
|
||||
## Storage
|
||||
@@ -442,11 +442,11 @@ Exposes Semantica as an MCP stdio server for IDE and agent integrations.
|
||||
python -m semantica.mcp_server
|
||||
```
|
||||
|
||||
**Integrations:** Claude Desktop, VS Code, Cursor, Windsurf, Cline — 12 MCP tools exposed
|
||||
**Integrations:** Claude Desktop, VS Code, Cursor, Windsurf, Cline: 12 MCP tools exposed
|
||||
|
||||
### Seed
|
||||
|
||||
Bootstrap knowledge graphs from verified structured sources — fixed-point reference data, controlled vocabularies, and domain anchors.
|
||||
Bootstrap knowledge graphs from verified structured sources: fixed-point reference data, controlled vocabularies, and domain anchors.
|
||||
|
||||
```python
|
||||
from semantica.seed import SeedManager
|
||||
@@ -499,7 +499,7 @@ from semantica.core import Semantica, PluginRegistry, ConfigManager
|
||||
sem = Semantica(config_path="config.yaml")
|
||||
sem.initialize()
|
||||
|
||||
# Plugin registry — register custom components
|
||||
# Plugin registry: register custom components
|
||||
registry = PluginRegistry()
|
||||
registry.register("my_ingestor", MyCustomIngestor)
|
||||
|
||||
@@ -548,7 +548,7 @@ graph = GraphBuilder(merge_entities=True).build(
|
||||
</Tab>
|
||||
|
||||
<Tab title="GraphRAG">
|
||||
Ground every LLM response in a knowledge graph — structured retrieval with source attribution.
|
||||
Ground every LLM response in a knowledge graph: structured retrieval with source attribution.
|
||||
|
||||
**Pipeline:** `KG` + `VectorStore` → `AgentContext` → GraphRAG query → grounded answer
|
||||
|
||||
@@ -604,7 +604,7 @@ precedents = context.find_precedents("model selection", limit=5)
|
||||
</Tab>
|
||||
|
||||
<Tab title="Compliance Pipeline">
|
||||
Full provenance from raw data to final inference — W3C PROV-O, SHA-256 checksums, audit trail.
|
||||
Full provenance from raw data to final inference: W3C PROV-O, SHA-256 checksums, audit trail.
|
||||
|
||||
**Pipeline:** `Ingest` → `Parse` → `Extract` → `KG` → `Provenance` → `ChangeManagement` → `Export`
|
||||
|
||||
@@ -654,7 +654,7 @@ for page in pages:
|
||||
</Tab>
|
||||
|
||||
<Tab title="Temporal Analysis">
|
||||
Track how facts change over time — point-in-time queries, snapshots, and versioning.
|
||||
Track how facts change over time: point-in-time queries, snapshots, and versioning.
|
||||
|
||||
**Pipeline:** `KG (Temporal)` → `TemporalGraphQuery` → `VersionManager` → `ChangeManagement`
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "License"
|
||||
description: "Semantica is open source under the MIT License — free for personal and commercial use."
|
||||
description: "Semantica is open source under the MIT License: free for personal and commercial use."
|
||||
icon: "file-contract"
|
||||
---
|
||||
|
||||
@@ -38,7 +38,7 @@ SOFTWARE.
|
||||
|
||||
**You can:**
|
||||
|
||||
<Check>Use Semantica commercially — free for business use</Check>
|
||||
<Check>Use Semantica commercially: free for business use</Check>
|
||||
<Check>Modify the source code</Check>
|
||||
<Check>Distribute copies or derivatives</Check>
|
||||
<Check>Use in proprietary software</Check>
|
||||
@@ -48,7 +48,7 @@ SOFTWARE.
|
||||
- Keep the copyright notice in copies
|
||||
- Include the MIT license text in distributions
|
||||
|
||||
<Warning>**No warranty** — the authors are not responsible for damages or liable for how you use the software.</Warning>
|
||||
<Warning>**No warranty**: the authors are not responsible for damages or liable for how you use the software.</Warning>
|
||||
|
||||
|
||||
## Commercial Use
|
||||
|
||||
+13
-13
@@ -5,10 +5,10 @@ icon: "rocket"
|
||||
---
|
||||
|
||||
<Info>
|
||||
**v0.5.0** — Ontology Hub, Distance Intelligence, Parquet & XML ingestion, 12 security fixes. [What's new →](index#whats-new)
|
||||
**v0.5.0**: Ontology Hub, Distance Intelligence, Parquet & XML ingestion, 12 security fixes. [What's new →](index#whats-new)
|
||||
</Info>
|
||||
|
||||
This guide walks you through the end-to-end pipeline for building your first knowledge graph. Start here after installation. An LLM API key is optional — pattern-based extraction works out of the box.
|
||||
This guide walks you through the end-to-end pipeline for building your first knowledge graph. Start here after installation. An LLM API key is optional: pattern-based extraction works out of the box.
|
||||
|
||||
|
||||
## Install
|
||||
@@ -95,7 +95,7 @@ print(parsed.metadata) # title, author, date, source
|
||||
```
|
||||
|
||||
<Tip>
|
||||
For PDFs with tables, charts, or multi-column layouts, use `DoclingParser` — it applies advanced layout analysis and returns structured table data alongside text.
|
||||
For PDFs with tables, charts, or multi-column layouts, use `DoclingParser`: it applies advanced layout analysis and returns structured table data alongside text.
|
||||
</Tip>
|
||||
|
||||
```python
|
||||
@@ -161,7 +161,7 @@ founders = graph.get_neighbors("Apple Inc.", predicate="founded_by")
|
||||
```
|
||||
|
||||
<Note>
|
||||
`merge_entities=True` automatically resolves duplicate entity references — "Apple", "Apple Inc.", "AAPL" — using semantic similarity. No manual deduplication needed.
|
||||
`merge_entities=True` automatically resolves duplicate entity references: "Apple", "Apple Inc.", "AAPL": using semantic similarity. No manual deduplication needed.
|
||||
</Note>
|
||||
|
||||
</Step>
|
||||
@@ -181,7 +181,7 @@ viz = GraphVisualizer(
|
||||
viz.visualize(graph, output="graph.html")
|
||||
```
|
||||
|
||||
Open `graph.html` in any browser — pan, zoom, click nodes for details, filter by entity type.
|
||||
Open `graph.html` in any browser: pan, zoom, click nodes for details, filter by entity type.
|
||||
|
||||
</Step>
|
||||
|
||||
@@ -205,7 +205,7 @@ from semantica.export import ParquetExporter
|
||||
|
||||
exporter = ParquetExporter()
|
||||
exporter.export(graph, output_dir="output/")
|
||||
# Writes nodes.parquet + edges.parquet — ready for Spark, BigQuery, Databricks
|
||||
# Writes nodes.parquet + edges.parquet: ready for Spark, BigQuery, Databricks
|
||||
```
|
||||
|
||||
```python ArangoDB
|
||||
@@ -225,7 +225,7 @@ aql = exporter.export(graph)
|
||||
|
||||
## Add Decision Intelligence
|
||||
|
||||
Track every agent decision with full causal chains and provenance — one extra import:
|
||||
Track every agent decision with full causal chains and provenance: one extra import:
|
||||
|
||||
```python
|
||||
from semantica.context import AgentContext, ContextGraph
|
||||
@@ -249,7 +249,7 @@ decision_id = context.record_decision(
|
||||
confidence=0.91,
|
||||
)
|
||||
|
||||
# Retrieve similar past decisions — prevents inconsistent choices
|
||||
# Retrieve similar past decisions: prevents inconsistent choices
|
||||
precedents = context.find_precedents("model selection reasoning", limit=5)
|
||||
influence = context.analyze_decision_influence(decision_id)
|
||||
```
|
||||
@@ -259,7 +259,7 @@ influence = context.analyze_decision_influence(decision_id)
|
||||
|
||||
<AccordionGroup>
|
||||
|
||||
<Accordion title="Process raw text directly — no file needed" icon="text">
|
||||
<Accordion title="Process raw text directly: no file needed" icon="text">
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
@@ -313,7 +313,7 @@ print(snapshot.get_node("Steve Jobs")) # role: CEO
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Persistent graph store — Neo4j, FalkorDB, Apache AGE" icon="database">
|
||||
<Accordion title="Persistent graph store: Neo4j, FalkorDB, Apache AGE" icon="database">
|
||||
|
||||
```python
|
||||
from semantica.graph_store import Neo4jStore
|
||||
@@ -327,12 +327,12 @@ store = Neo4jStore(
|
||||
|
||||
builder = GraphBuilder(merge_entities=True, graph_store=store)
|
||||
graph = builder.build(entities=entities, relationships=relationships)
|
||||
# Graph persisted to Neo4j — survives process restarts
|
||||
# Graph persisted to Neo4j: survives process restarts
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Full provenance pipeline — W3C PROV-O" icon="link">
|
||||
<Accordion title="Full provenance pipeline: W3C PROV-O" icon="link">
|
||||
|
||||
```python
|
||||
from semantica.provenance import ProvenanceTracker
|
||||
@@ -422,7 +422,7 @@ pip install --upgrade semantica
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Core Concepts" icon="book-open" href="concepts">
|
||||
Knowledge graphs, ontologies, reasoning engines — the mental model behind Semantica.
|
||||
Knowledge graphs, ontologies, reasoning engines: the mental model behind Semantica.
|
||||
</Card>
|
||||
<Card title="Module Reference" icon="puzzle-piece" href="modules">
|
||||
Every module explained with key classes and common chains.
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: "clock-rotate-left"
|
||||
|
||||
**`semantica.change_management`** provides **enterprise-grade versioning and audit trails** for knowledge graphs and ontologies:
|
||||
|
||||
- SHA-256 checksums on every snapshot — tamper detection without external infrastructure
|
||||
- SHA-256 checksums on every snapshot: tamper detection without external infrastructure
|
||||
- Structural diff between any two versions: nodes added, removed, or modified
|
||||
- Full rollback to any named snapshot
|
||||
- Per-entity mutation history for audit trail queries
|
||||
@@ -23,8 +23,8 @@ icon: "clock-rotate-left"
|
||||
| :--- | :--- |
|
||||
| `TemporalVersionManager` | Snapshot, diff, rollback, and per-node mutation history for KGs |
|
||||
| `OntologyVersionManager` | Schema versioning with structural diff support |
|
||||
| `InMemoryVersionStorage` | Fast in-memory storage for dev and testing — no persistence |
|
||||
| `SQLiteVersionStorage` | Production storage — persists to a local SQLite file |
|
||||
| `InMemoryVersionStorage` | Fast in-memory storage for dev and testing: no persistence |
|
||||
| `SQLiteVersionStorage` | Production storage: persists to a local SQLite file |
|
||||
| `compute_checksum()` | Returns SHA-256 fingerprint of any dict (graph snapshot, ontology snapshot) |
|
||||
| `verify_checksum()` | Detects tampering by recomputing and comparing the stored checksum inside a snapshot dict |
|
||||
|
||||
@@ -38,7 +38,7 @@ icon: "clock-rotate-left"
|
||||
Version control for OWL ontologies with diff and schema migration support.
|
||||
</Card>
|
||||
<Card title="VersionStorage" icon="database">
|
||||
Pluggable backends — `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production.
|
||||
Pluggable backends: `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production.
|
||||
</Card>
|
||||
<Card title="Integrity Verification" icon="shield-check">
|
||||
SHA-256 checksums on every snapshot to detect any unauthorised modification.
|
||||
@@ -74,7 +74,7 @@ icon: "clock-rotate-left"
|
||||
```
|
||||
</Step>
|
||||
<Step title="Make your changes">
|
||||
Run deduplication, conflict resolution, merges, or any graph modification. The version manager tracks nothing automatically — you control when snapshots are taken.
|
||||
Run deduplication, conflict resolution, merges, or any graph modification. The version manager tracks nothing automatically: you control when snapshots are taken.
|
||||
</Step>
|
||||
<Step title="Snapshot the result">
|
||||
```python
|
||||
@@ -82,7 +82,7 @@ icon: "clock-rotate-left"
|
||||
graph=kg,
|
||||
version_label="v2.0",
|
||||
author="user@example.com",
|
||||
description="After deduplication — 1 342 duplicates merged"
|
||||
description="After deduplication: 1 342 duplicates merged"
|
||||
)
|
||||
```
|
||||
</Step>
|
||||
@@ -99,7 +99,7 @@ icon: "clock-rotate-left"
|
||||
|
||||
## TemporalVersionManager
|
||||
|
||||
Version control for knowledge graphs — snapshot, diff, and rollback.
|
||||
Version control for knowledge graphs: snapshot, diff, and rollback.
|
||||
|
||||
### Constructor Parameters
|
||||
|
||||
@@ -110,7 +110,7 @@ Version control for knowledge graphs — snapshot, diff, and rollback.
|
||||
### List and Retrieve
|
||||
|
||||
```python
|
||||
# List all versions — returns List[Dict] with label, author, timestamp, checksum, entity_count
|
||||
# List all versions: returns List[Dict] with label, author, timestamp, checksum, entity_count
|
||||
versions = manager.list_versions()
|
||||
for v in versions:
|
||||
print(v["label"], "|", v["author"], "|", v["timestamp"], "|", v["checksum"][:8], "...")
|
||||
@@ -137,7 +137,7 @@ snapshot = manager.get_version("v1.0")
|
||||
|
||||
## Diff Analysis
|
||||
|
||||
Compare any two snapshots to see exactly what changed — useful for code review, incident investigation, and regulatory audit:
|
||||
Compare any two snapshots to see exactly what changed: useful for code review, incident investigation, and regulatory audit:
|
||||
|
||||
```python
|
||||
diff = manager.diff("v1.0", "v2.0")
|
||||
@@ -186,7 +186,7 @@ for item in diff["entities_modified"]:
|
||||
|
||||
## OntologyVersionManager
|
||||
|
||||
Version control for ontologies — save, diff, and track schema changes:
|
||||
Version control for ontologies: save, diff, and track schema changes:
|
||||
|
||||
```python
|
||||
from semantica.change_management import OntologyVersionManager
|
||||
@@ -201,7 +201,7 @@ snapshot = manager.create_snapshot(
|
||||
description="Added FHIR alignment mappings"
|
||||
)
|
||||
|
||||
# Diff two ontology versions — returns a plain dict
|
||||
# Diff two ontology versions: returns a plain dict
|
||||
diff = manager.compare_versions("1.1.0", "1.2.0")
|
||||
print("Classes added: ", diff["classes_added"])
|
||||
print("Classes removed: ", diff["classes_removed"])
|
||||
@@ -229,12 +229,12 @@ print("Properties added: ", diff["properties_added"])
|
||||
manager = TemporalVersionManager()
|
||||
```
|
||||
|
||||
Fast and zero-setup. Data is **not persisted** — all version history is lost when the process exits. Use this for unit tests and development only.
|
||||
Fast and zero-setup. Data is **not persisted**: all version history is lost when the process exits. Use this for unit tests and development only.
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
<Warning>
|
||||
The default `TemporalVersionManager()` with no arguments uses in-memory storage. Always pass `storage_path="versions.db"` or an explicit `SQLiteVersionStorage` in production — otherwise your entire version history disappears on restart.
|
||||
The default `TemporalVersionManager()` with no arguments uses in-memory storage. Always pass `storage_path="versions.db"` or an explicit `SQLiteVersionStorage` in production: otherwise your entire version history disappears on restart.
|
||||
</Warning>
|
||||
|
||||
## Integrity Verification
|
||||
@@ -257,7 +257,7 @@ if not is_valid:
|
||||
```
|
||||
|
||||
<Tip>
|
||||
`verify_checksum` takes the full snapshot dict (which contains the stored `"checksum"` key). Pass the dict returned by `create_snapshot` or `get_version` directly — no separate `expected_checksum` argument is needed.
|
||||
`verify_checksum` takes the full snapshot dict (which contains the stored `"checksum"` key). Pass the dict returned by `create_snapshot` or `get_version` directly: no separate `expected_checksum` argument is needed.
|
||||
</Tip>
|
||||
|
||||
## ChangeLogEntry
|
||||
@@ -334,17 +334,17 @@ for record in history:
|
||||
### Compliance Coverage
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="HIPAA — subject-access requests">
|
||||
<Accordion title="HIPAA: subject-access requests">
|
||||
Use `manager.get_node_history("patient_001")` to retrieve every recorded mutation on a patient entity. Each `MutationRecord` includes `timestamp`, `operation`, `entity_id`, `payload`, and `version_label`. The SHA-256 checksum on each snapshot proves the record has not been altered.
|
||||
</Accordion>
|
||||
<Accordion title="SOX — quarterly reviews">
|
||||
<Accordion title="SOX: quarterly reviews">
|
||||
Use `manager.list_versions()` to enumerate all snapshots and `manager.diff(v1, v2)` to scope the change report to the relevant quarter. The immutable snapshot chain provides the chain of custody required by SOX Section 404.
|
||||
</Accordion>
|
||||
<Accordion title="GDPR — right to erasure verification">
|
||||
<Accordion title="GDPR: right to erasure verification">
|
||||
After deleting a data subject's entities, snapshot the graph and diff against the pre-deletion snapshot. `diff["entities_removed"]` provides a machine-readable record of exactly what was deleted and when, satisfying Article 17 documentation requirements.
|
||||
</Accordion>
|
||||
<Accordion title="FDA 21 CFR Part 11 — electronic records">
|
||||
Every snapshot dict includes `author`, `timestamp`, and `checksum` — the three fields required for a compliant electronic record. `verify_checksum(snapshot)` provides the tamper-evidence required by 21 CFR § 11.10(e).
|
||||
<Accordion title="FDA 21 CFR Part 11: electronic records">
|
||||
Every snapshot dict includes `author`, `timestamp`, and `checksum`: the three fields required for a compliant electronic record. `verify_checksum(snapshot)` provides the tamper-evidence required by 21 CFR § 11.10(e).
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
@@ -359,7 +359,7 @@ for record in history:
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` returns a plain dict with `"summary"`, `"entities_added"`, `"entities_removed"`, and `"entities_modified"` — use the `"summary"` sub-dict to get counts and `"entities_modified"` to inspect property-level changes.
|
||||
**Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` returns a plain dict with `"summary"`, `"entities_added"`, `"entities_removed"`, and `"entities_modified"`: use the `"summary"` sub-dict to get counts and `"entities_modified"` to inspect property-level changes.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
|
||||
+20
-20
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Conflicts Module"
|
||||
description: "Multi-source conflict detection and resolution — value, type, temporal, and logical conflicts with investigation guides."
|
||||
description: "Multi-source conflict detection and resolution: value, type, temporal, and logical conflicts with investigation guides."
|
||||
icon: "triangle-exclamation"
|
||||
---
|
||||
|
||||
@@ -10,7 +10,7 @@ icon: "triangle-exclamation"
|
||||
- Seven resolution strategies: voting, credibility-weighted, most-recent, first-seen, highest-confidence, manual review, expert review
|
||||
- `InvestigationGuideGenerator` produces step-by-step investigation instructions for manual resolution
|
||||
- `SourceTracker` maps each property value to its contributing source for full attribution
|
||||
- Conflicts are surfaced explicitly — never silently corrupting the knowledge graph
|
||||
- Conflicts are surfaced explicitly: never silently corrupting the knowledge graph
|
||||
|
||||
|
||||
## Why Detect Conflicts?
|
||||
@@ -19,11 +19,11 @@ When you ingest data from multiple sources, contradictions are inevitable. One a
|
||||
|
||||
Semantica's conflict detection makes disagreements explicit and actionable:
|
||||
|
||||
- **Value conflicts** — SEC says revenue is $391B; Reuters says $383B
|
||||
- **Type conflicts** — "Python" is a `ProgrammingLanguage` in one source, a `Snake` species in another
|
||||
- **Temporal conflicts** — a CEO had two different employers during overlapping date ranges
|
||||
- **Logical conflicts** — an entity simultaneously holds two mutually exclusive properties
|
||||
- **Relationship conflicts** — the same relationship has inconsistent cardinality or properties across sources
|
||||
- **Value conflicts**: SEC says revenue is $391B; Reuters says $383B
|
||||
- **Type conflicts**: "Python" is a `ProgrammingLanguage` in one source, a `Snake` species in another
|
||||
- **Temporal conflicts**: a CEO had two different employers during overlapping date ranges
|
||||
- **Logical conflicts**: an entity simultaneously holds two mutually exclusive properties
|
||||
- **Relationship conflicts**: the same relationship has inconsistent cardinality or properties across sources
|
||||
|
||||
## Exported Classes
|
||||
|
||||
@@ -118,7 +118,7 @@ Semantica's conflict detection makes disagreements explicit and actionable:
|
||||
|
||||
# Auto-resolve low-severity conflicts
|
||||
low_conflicts = severity_details.get("low", [])
|
||||
# Re-fetch full Conflict objects if needed — severity_details contains dicts
|
||||
# Re-fetch full Conflict objects if needed: severity_details contains dicts
|
||||
auto_resolved = resolver.resolve_conflicts(
|
||||
conflicts,
|
||||
strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED,
|
||||
@@ -203,7 +203,7 @@ for result in results:
|
||||
|
||||
<Tabs>
|
||||
<Tab title="CREDIBILITY_WEIGHTED (recommended)">
|
||||
Weights each source's value by its assigned credibility score — favors authoritative sources automatically:
|
||||
Weights each source's value by its assigned credibility score: favors authoritative sources automatically:
|
||||
|
||||
```python
|
||||
from semantica.conflicts import ConflictResolver, SourceTracker, ResolutionStrategy
|
||||
@@ -223,7 +223,7 @@ for result in results:
|
||||
**Best for:** sources with known reliability rankings (SEC > blog).
|
||||
</Tab>
|
||||
<Tab title="VOTING">
|
||||
Majority vote — most common value across sources wins:
|
||||
Majority vote: most common value across sources wins:
|
||||
|
||||
```python
|
||||
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING)
|
||||
@@ -233,16 +233,16 @@ for result in results:
|
||||
</Tab>
|
||||
<Tab title="MOST_RECENT / FIRST_SEEN">
|
||||
```python
|
||||
# Most recent source wins — for fast-changing facts
|
||||
# Most recent source wins: for fast-changing facts
|
||||
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.MOST_RECENT)
|
||||
|
||||
# First seen wins — for stable facts (founding date, original name)
|
||||
# First seen wins: for stable facts (founding date, original name)
|
||||
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.FIRST_SEEN)
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="MANUAL_REVIEW / EXPERT_REVIEW">
|
||||
```python
|
||||
# Flag for human review — use with InvestigationGuideGenerator
|
||||
# Flag for human review: use with InvestigationGuideGenerator
|
||||
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.MANUAL_REVIEW)
|
||||
generator = InvestigationGuideGenerator()
|
||||
|
||||
@@ -265,7 +265,7 @@ for result in results:
|
||||
| First seen | `FIRST_SEEN` | Stable facts: founding date, original name |
|
||||
| Highest confidence | `HIGHEST_CONFIDENCE` | Extraction pipeline outputs confidence scores |
|
||||
| Manual review | `MANUAL_REVIEW` | High-stakes decisions, regulated data |
|
||||
| Expert review | `EXPERT_REVIEW` | Domain-specific ambiguity — escalate to a specialist |
|
||||
| Expert review | `EXPERT_REVIEW` | Domain-specific ambiguity: escalate to a specialist |
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
@@ -312,7 +312,7 @@ chain = tracker.get_traceability_chain("apple_inc")
|
||||
|
||||
**Key behaviours:**
|
||||
- Credibility scores default to 0.50 for any source not explicitly set
|
||||
- `SourceTracker` stores property-level provenance — so you can trace exactly which source contributed each value
|
||||
- `SourceTracker` stores property-level provenance: so you can trace exactly which source contributed each value
|
||||
|
||||
## ConflictAnalyzer
|
||||
|
||||
@@ -334,9 +334,9 @@ for t in trends:
|
||||
```
|
||||
|
||||
**Key behaviours:**
|
||||
- `analyze_conflicts()["patterns"]` returns a list of `ConflictPattern` objects — use `pattern.pattern_type` and `pattern.frequency` to find systemic data quality issues
|
||||
- `analyze_conflicts()["by_source"]` includes `counts` and `top_sources` — sources appearing in many conflicts may have upstream data quality problems
|
||||
- `analyze_trends()` returns a list of per-period dicts (`period`, `conflict_count`, `trend`, `trend_direction`) — `trend` is `"increasing"`, `"decreasing"`, or `"stable"`
|
||||
- `analyze_conflicts()["patterns"]` returns a list of `ConflictPattern` objects: use `pattern.pattern_type` and `pattern.frequency` to find systemic data quality issues
|
||||
- `analyze_conflicts()["by_source"]` includes `counts` and `top_sources`: sources appearing in many conflicts may have upstream data quality problems
|
||||
- `analyze_trends()` returns a list of per-period dicts (`period`, `conflict_count`, `trend`, `trend_direction`): `trend` is `"increasing"`, `"decreasing"`, or `"stable"`
|
||||
|
||||
## InvestigationGuideGenerator
|
||||
|
||||
@@ -439,7 +439,7 @@ class InvestigationStep:
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder — you lose the original source attribution.
|
||||
**Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder: you lose the original source attribution.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
@@ -447,7 +447,7 @@ class InvestigationStep:
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with `severity == "critical"` or `severity == "high"` — high severity means the disagreement is large and the stakes of getting it wrong are high.
|
||||
**Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with `severity == "critical"` or `severity == "high"`: high severity means the disagreement is large and the stakes of getting it wrong are high.
|
||||
</Tip>
|
||||
|
||||
<Warning>
|
||||
|
||||
+22
-22
@@ -17,7 +17,7 @@ icon: "brain"
|
||||
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `AgentContext` | Primary entry point — memory, retrieval, decisions, graph traversal, checkpoints |
|
||||
| `AgentContext` | Primary entry point: memory, retrieval, decisions, graph traversal, checkpoints |
|
||||
| `ContextGraph` | In-memory knowledge graph with centrality, community detection, and decision tracking |
|
||||
| `AgentMemory` | Vector-backed persistent memory: `store(text)`, `retrieve(query, max_results)` |
|
||||
| `EntityLinker` | Link entity mentions to URIs; create typed edges between entity IDs |
|
||||
@@ -152,7 +152,7 @@ decision_id = context.record_decision(
|
||||
</Step>
|
||||
<Step title="Find precedents and trace causal chains">
|
||||
```python
|
||||
# Search past decisions — prevents contradictory choices across runs
|
||||
# Search past decisions: prevents contradictory choices across runs
|
||||
precedents = context.find_precedents("model selection reasoning", limit=5)
|
||||
for p in precedents:
|
||||
print("[{}] {} (confidence: {:.2f})".format(p.category, p.outcome, p.confidence))
|
||||
@@ -174,13 +174,13 @@ decision_id = context.record_decision(
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Vector Memory Only">
|
||||
Fastest setup — no knowledge graph. Best for agents that need semantic search over facts without graph traversal overhead.
|
||||
Fastest setup: no knowledge graph. Best for agents that need semantic search over facts without graph traversal overhead.
|
||||
|
||||
```python
|
||||
from semantica.context import AgentContext
|
||||
from semantica.vector_store import VectorStore
|
||||
|
||||
# Zero-graph setup — vector memory only
|
||||
# Zero-graph setup: vector memory only
|
||||
context = AgentContext(
|
||||
vector_store=VectorStore(backend="faiss", dimension=768),
|
||||
)
|
||||
@@ -198,7 +198,7 @@ decision_id = context.record_decision(
|
||||
</Check>
|
||||
</Tab>
|
||||
<Tab title="Full Agent Context">
|
||||
Production setup — graph + decisions + analytics. Use when you need explainability and contradiction-free decision history.
|
||||
Production setup: graph + decisions + analytics. Use when you need explainability and contradiction-free decision history.
|
||||
|
||||
```python
|
||||
from semantica.context import AgentContext, ContextGraph
|
||||
@@ -314,7 +314,7 @@ decision_id = context.record_decision(
|
||||
| :--------- | :---- | :------- | :----------- |
|
||||
| `vector_store` | `VectorStore` | **required** | Backend for embedding-based memory retrieval |
|
||||
| `knowledge_graph` | `ContextGraph` | `None` | Enables graph-backed relationships and GraphRAG |
|
||||
| `decision_tracking` | `bool` | `False` | Activates `DecisionRecorder` — requires `knowledge_graph` to also be set |
|
||||
| `decision_tracking` | `bool` | `False` | Activates `DecisionRecorder`: requires `knowledge_graph` to also be set |
|
||||
| `retention_days` | `Optional[int]` | `30` | Auto-expire memories older than N days; `None` = keep forever |
|
||||
| `max_memories` | `int` | `10000` | Hard cap before LRU eviction |
|
||||
| `graph_expansion` | `bool` | `True` | Auto-expands graph from stored memories |
|
||||
@@ -332,13 +332,13 @@ decision_id = context.record_decision(
|
||||
| Method | Returns | Description |
|
||||
| :------ | :------- | :----------- |
|
||||
| `store(content, metadata, conversation_id, user_id)` | `str` or `Dict` | Store a fact (str → memory ID) or list of documents (list → stats dict) |
|
||||
| `batch_store(items)` | `List[str]` | Store multiple items at once — returns list of memory IDs |
|
||||
| `batch_store(items)` | `List[str]` | Store multiple items at once: returns list of memory IDs |
|
||||
| `retrieve(query, max_results, min_score, use_graph, conversation_id)` | `List[Dict]` | Semantic retrieval; auto-selects GraphRAG if `knowledge_graph` is set |
|
||||
| `forget(memory_id, conversation_id, days_old)` | `int` | Delete memories by ID, conversation, or age |
|
||||
| `update(memory_id, content, metadata)` | `bool` | Update content or metadata of a stored memory |
|
||||
| `get_memory(memory_id)` | `Optional[Dict]` | Fetch a specific memory by ID |
|
||||
| `stats()` | `Dict` | Memory counts, vector store status, graph stats |
|
||||
| `health()` | `Dict` | System health — all backends, status flags |
|
||||
| `health()` | `Dict` | System health: all backends, status flags |
|
||||
| `save(path)` | `None` | Persist full context state (memory + graph) to disk |
|
||||
| `load(path)` | `None` | Restore context state from disk |
|
||||
| `export(conversation_id, format)` | `str \| Dict` | Export memories as JSON or dict |
|
||||
@@ -366,7 +366,7 @@ results = context.retrieve(
|
||||
|
||||
### Multi-Hop GraphRAG
|
||||
|
||||
**Requires `knowledge_graph`** to be set at construction — enables `query_with_reasoning()` for LLM-grounded multi-hop traversal:
|
||||
**Requires `knowledge_graph`** to be set at construction: enables `query_with_reasoning()` for LLM-grounded multi-hop traversal:
|
||||
|
||||
```python
|
||||
import os
|
||||
@@ -393,12 +393,12 @@ print("Sources used: {}".format(result["num_sources"]))
|
||||
| `find_precedents(scenario, category, limit, use_hybrid_search, max_hops, as_of)` | `List[Decision]` | Find similar past decisions by semantic + structural similarity |
|
||||
| `query_decisions(query, max_hops, use_hybrid_search)` | `List[Decision]` | Broad context-aware decision search |
|
||||
| `get_causal_chain(decision_id, direction, max_depth)` | `List[Decision]` | Trace `"upstream"` causes or `"downstream"` effects |
|
||||
| `trace_decision_explainability(decision_id)` | `Dict` | Full explainability — causes, effects, relationship paths |
|
||||
| `trace_decision_explainability(decision_id)` | `Dict` | Full explainability: causes, effects, relationship paths |
|
||||
| `get_policy_engine()` | `PolicyEngine` | Access the active `PolicyEngine` instance |
|
||||
|
||||
### Checkpoint Methods
|
||||
|
||||
**Ideal for auditing reasoning loops** — take a snapshot before and after a pass to see exactly what changed:
|
||||
**Ideal for auditing reasoning loops**: take a snapshot before and after a pass to see exactly what changed:
|
||||
|
||||
```python
|
||||
# Take a named snapshot of the current graph state
|
||||
@@ -456,7 +456,7 @@ print("Nodes: {}, Edges: {}".format(stats["node_count"], stats["edge_count"]))
|
||||
| `community_detection` | `bool` | `True` | Louvain community clustering |
|
||||
| `node_embeddings` | `bool` | `True` | Node2Vec embeddings for structural similarity |
|
||||
|
||||
### ContextGraph — Full Method Reference
|
||||
### ContextGraph: Full Method Reference
|
||||
|
||||
| Method | Returns | Description |
|
||||
| :------ | :------- | :----------- |
|
||||
@@ -622,7 +622,7 @@ linked = linker.link(text="Apple Inc. was founded by Steve Jobs.", entities=enti
|
||||
for e in linked:
|
||||
print("{} → {} (confidence: {:.2f})".format(e.text, e.uri, e.confidence))
|
||||
|
||||
# Explicitly link two entity IDs (not a list — takes two IDs)
|
||||
# Explicitly link two entity IDs (not a list: takes two IDs)
|
||||
linker.link_entities(
|
||||
entity1_id="apple_inc",
|
||||
entity2_id="aapl",
|
||||
@@ -792,7 +792,7 @@ class EntityLink:
|
||||
## Real-World Patterns
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Healthcare — Treatment Decisions">
|
||||
<Tab title="Healthcare: Treatment Decisions">
|
||||
```python
|
||||
from semantica.context import AgentContext, ContextGraph
|
||||
from semantica.vector_store import VectorStore
|
||||
@@ -804,7 +804,7 @@ class EntityLink:
|
||||
)
|
||||
|
||||
health_agent.store("Patient has hypertension, type 2 diabetes")
|
||||
health_agent.store("Patient allergic to penicillin — verified 2024-01")
|
||||
health_agent.store("Patient allergic to penicillin: verified 2024-01")
|
||||
|
||||
decision_id = health_agent.record_decision(
|
||||
category="treatment_plan",
|
||||
@@ -822,7 +822,7 @@ class EntityLink:
|
||||
print("Follow-up decisions triggered: {}".format(len(chain)))
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Finance — Loan Decisions">
|
||||
<Tab title="Finance: Loan Decisions">
|
||||
```python
|
||||
from semantica.context import AgentContext, ContextGraph, PolicyEngine
|
||||
from semantica.context.decision_models import Policy, Decision
|
||||
@@ -856,7 +856,7 @@ class EntityLink:
|
||||
d = Decision(
|
||||
decision_id="dec_loan_001",
|
||||
category="loan_approval",
|
||||
scenario="First-time homebuyer — 30yr fixed, 20% down",
|
||||
scenario="First-time homebuyer: 30yr fixed, 20% down",
|
||||
reasoning="Credit score above threshold, DTI within limits",
|
||||
outcome="approved_300k",
|
||||
confidence=0.94,
|
||||
@@ -894,7 +894,7 @@ class EntityLink:
|
||||
# Persist everything
|
||||
context.save("agent_state/")
|
||||
|
||||
# Later — restore and continue
|
||||
# Later: restore and continue
|
||||
restored = AgentContext(
|
||||
vector_store=VectorStore(backend="faiss", dimension=768),
|
||||
knowledge_graph=ContextGraph(),
|
||||
@@ -911,11 +911,11 @@ class EntityLink:
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**`decision_tracking=True` silently does nothing without `knowledge_graph`.** Both must be set at construction. Passing only `decision_tracking=True` without a `knowledge_graph` instance leaves the decision backend uninitialised — `record_decision()` will raise `RuntimeError`.
|
||||
**`decision_tracking=True` silently does nothing without `knowledge_graph`.** Both must be set at construction. Passing only `decision_tracking=True` without a `knowledge_graph` instance leaves the decision backend uninitialised: `record_decision()` will raise `RuntimeError`.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
**Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore` — without it the FAISS index lives only in memory and is lost on shutdown.
|
||||
**Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore`: without it the FAISS index lives only in memory and is lost on shutdown.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
@@ -955,5 +955,5 @@ class EntityLink:
|
||||
|
||||
### Cookbooks
|
||||
|
||||
- [Context Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb) — memory and decision tracking · Intermediate
|
||||
- [Advanced Context Engineering](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb) — production FAISS + Neo4j setup · Advanced
|
||||
- [Context Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb): memory and decision tracking · Intermediate
|
||||
- [Advanced Context Engineering](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb): production FAISS + Neo4j setup · Advanced
|
||||
|
||||
@@ -20,7 +20,7 @@ icon: "gear"
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Semantica" icon="arrows-turn-to-dots">
|
||||
High-level orchestrator — coordinates the full KG construction pipeline from a single `config.yaml`. Entry point for application-level deployments.
|
||||
High-level orchestrator: coordinates the full KG construction pipeline from a single `config.yaml`. Entry point for application-level deployments.
|
||||
</Card>
|
||||
<Card title="ConfigManager" icon="sliders">
|
||||
YAML config with deep-merge, `SEMANTICA_` env var overrides, and dot-notation nested key access. Keeps secrets out of source files.
|
||||
@@ -29,7 +29,7 @@ icon: "gear"
|
||||
Ordered startup/shutdown hooks, health monitoring, and a 6-state machine. Essential for long-running services like FastAPI apps.
|
||||
</Card>
|
||||
<Card title="PluginRegistry" icon="plug">
|
||||
Register custom ingestors, parsers, exporters, or any component. Load them by name at runtime — no imports required.
|
||||
Register custom ingestors, parsers, exporters, or any component. Load them by name at runtime: no imports required.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -37,11 +37,11 @@ icon: "gear"
|
||||
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `Semantica` | Orchestration entry point — coordinates the full KG construction pipeline |
|
||||
| `Semantica` | Orchestration entry point: coordinates the full KG construction pipeline |
|
||||
| `ConfigManager` | YAML config loading, deep-merge, validation, and env var overrides |
|
||||
| `LifecycleManager` | Startup/shutdown state machine with health monitoring and lifecycle hooks |
|
||||
| `PluginRegistry` | Dynamic plugin discovery, registration, and loading |
|
||||
| `method_registry` | Global `MethodRegistry` instance — register and dispatch custom orchestration methods |
|
||||
| `method_registry` | Global `MethodRegistry` instance: register and dispatch custom orchestration methods |
|
||||
|
||||
|
||||
## Semantica (Orchestration)
|
||||
@@ -77,7 +77,7 @@ finally:
|
||||
| `build_knowledge_base(sources, **kwargs)` | Orchestrate full KG construction pipeline |
|
||||
| `run_pipeline(pipeline, data)` | Execute an existing `Pipeline` instance |
|
||||
| `get_status()` | Return system health and current state |
|
||||
| `shutdown(graceful=True)` | Graceful shutdown — waits for in-flight operations |
|
||||
| `shutdown(graceful=True)` | Graceful shutdown: waits for in-flight operations |
|
||||
|
||||
## ConfigManager
|
||||
|
||||
@@ -107,7 +107,7 @@ config.validate()
|
||||
llm_provider:
|
||||
name: openai
|
||||
model: gpt-4o
|
||||
# Do not put API keys in YAML — use environment variables instead.
|
||||
# Do not put API keys in YAML: use environment variables instead.
|
||||
# ConfigManager loads YAML with yaml.safe_load(), which does not
|
||||
# interpolate ${...} expressions. Set secrets via env vars (see below).
|
||||
|
||||
@@ -183,7 +183,7 @@ manager.shutdown(graceful=True)
|
||||
|
||||
## PluginRegistry
|
||||
|
||||
Register custom components that participate in the full pipeline — provenance tracking, retry policies, and parallel execution included:
|
||||
Register custom components that participate in the full pipeline: provenance tracking, retry policies, and parallel execution included:
|
||||
|
||||
```python
|
||||
from semantica.core import PluginRegistry
|
||||
@@ -214,7 +214,7 @@ from semantica.core import method_registry
|
||||
from semantica.core.methods import build_knowledge_base
|
||||
|
||||
def fast_kb_builder(sources, **kwargs):
|
||||
# Custom logic — skip embeddings for speed
|
||||
# Custom logic: skip embeddings for speed
|
||||
...
|
||||
|
||||
method_registry.register("knowledge_base", "fast", fast_kb_builder)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Deduplication Module"
|
||||
description: "Entity deduplication — similarity scoring, blocking, merging, and cluster-based batch processing."
|
||||
description: "Entity deduplication: similarity scoring, blocking, merging, and cluster-based batch processing."
|
||||
icon: "copy"
|
||||
---
|
||||
|
||||
@@ -10,23 +10,23 @@ icon: "copy"
|
||||
- `ClusterBuilder` uses Union-Find and hierarchical clustering for batch deduplication at scale
|
||||
- `EntityMerger` preserves original source provenance on every merged entity
|
||||
- `MergeStrategyManager` supports per-property rules and conflict resolution
|
||||
- All workflows operate on plain Python dicts — no ORM or schema required
|
||||
- All workflows operate on plain Python dicts: no ORM or schema required
|
||||
|
||||
|
||||
## Exported Classes
|
||||
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `DuplicateDetector` | Pairwise and batch detection — returns `DuplicateCandidate` or `DuplicateGroup` lists |
|
||||
| `EntityMerger` | Merge duplicate groups — returns `List[MergeOperation]` |
|
||||
| `DuplicateDetector` | Pairwise and batch detection: returns `DuplicateCandidate` or `DuplicateGroup` lists |
|
||||
| `EntityMerger` | Merge duplicate groups: returns `List[MergeOperation]` |
|
||||
| `SimilarityCalculator` | Multi-factor similarity: string, property, relationship, and embedding |
|
||||
| `ClusterBuilder` | Union-Find and hierarchical clustering for large-scale batch deduplication |
|
||||
| `MergeStrategy` | Enum of merge strategies: `KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL` |
|
||||
| `PropertyMergeRule` | Dataclass holding per-property merge rule: `{property_name, strategy, conflict_resolution, priority}` |
|
||||
| `MergeStrategyManager` | Manage and apply named merge strategies; accepts per-property rules |
|
||||
| `detect_duplicates()` | Convenience function — `detect_duplicates(entities, method="pairwise", similarity_threshold=0.7)` |
|
||||
| `merge_entities()` | Convenience function — `merge_entities(entities, method="keep_most_complete")` |
|
||||
| `calculate_similarity()` | Convenience function — `calculate_similarity(entity_a, entity_b, method="multi_factor")` |
|
||||
| `detect_duplicates()` | Convenience function: `detect_duplicates(entities, method="pairwise", similarity_threshold=0.7)` |
|
||||
| `merge_entities()` | Convenience function: `merge_entities(entities, method="keep_most_complete")` |
|
||||
| `calculate_similarity()` | Convenience function: `calculate_similarity(entity_a, entity_b, method="multi_factor")` |
|
||||
|
||||
## What You Get
|
||||
|
||||
@@ -35,19 +35,19 @@ icon: "copy"
|
||||
Pairwise, batch, incremental, and group detection modes. Returns scored candidates with reasons.
|
||||
</Card>
|
||||
<Card title="EntityMerger" icon="code-merge">
|
||||
Five merge strategies — keep first, last, most complete, highest confidence, or merge all fields.
|
||||
Five merge strategies: keep first, last, most complete, highest confidence, or merge all fields.
|
||||
</Card>
|
||||
<Card title="SimilarityCalculator" icon="equals">
|
||||
Multi-factor scoring across string edit distance, property overlap, relationship overlap, and embeddings.
|
||||
</Card>
|
||||
<Card title="ClusterBuilder" icon="diagram-project">
|
||||
Union-Find and hierarchical clustering for batch deduplication at scale — handles 100k+ entity sets.
|
||||
Union-Find and hierarchical clustering for batch deduplication at scale: handles 100k+ entity sets.
|
||||
</Card>
|
||||
<Card title="MergeStrategyManager" icon="sliders">
|
||||
Per-property merge rules with conflict resolution priorities. Apply different strategies to different fields.
|
||||
</Card>
|
||||
<Card title="v2 Strategies" icon="bolt">
|
||||
`blocking_v2`, `hybrid_v2`, `semantic_v2` — up to 7× faster than v1 for large entity sets.
|
||||
`blocking_v2`, `hybrid_v2`, `semantic_v2`: up to 7× faster than v1 for large entity sets.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -63,13 +63,13 @@ entities = [
|
||||
{"id": "3", "name": "Microsoft", "type": "Company"},
|
||||
]
|
||||
|
||||
# 1. Detect duplicates — returns List[DuplicateCandidate]
|
||||
# 1. Detect duplicates: returns List[DuplicateCandidate]
|
||||
detector = DuplicateDetector(similarity_threshold=0.7)
|
||||
candidates = detector.detect_duplicates(entities)
|
||||
|
||||
for dup in candidates:
|
||||
print(
|
||||
"{} vs {} — sim: {:.2f}, confidence: {:.2f}".format(
|
||||
"{} vs {}: sim: {:.2f}, confidence: {:.2f}".format(
|
||||
dup.entity1.get("name"),
|
||||
dup.entity2.get("name"),
|
||||
dup.similarity_score,
|
||||
@@ -77,7 +77,7 @@ for dup in candidates:
|
||||
)
|
||||
)
|
||||
|
||||
# 2. Merge duplicates — returns List[MergeOperation]
|
||||
# 2. Merge duplicates: returns List[MergeOperation]
|
||||
merger = EntityMerger()
|
||||
operations = merger.merge_duplicates(entities, strategy="keep_most_complete")
|
||||
|
||||
@@ -95,11 +95,11 @@ Find duplicate entity pairs:
|
||||
from semantica.deduplication import DuplicateDetector
|
||||
|
||||
detector = DuplicateDetector(
|
||||
similarity_threshold=0.7, # default 0.7 — minimum score to include a candidate
|
||||
confidence_threshold=0.6, # default 0.6 — minimum confidence to include a candidate
|
||||
max_results=100, # optional — hard cap on total candidates returned
|
||||
top_k_per_entity=3, # optional — max candidates per entity
|
||||
min_similarity=0.75, # optional — additional floor applied after sorting
|
||||
similarity_threshold=0.7, # default 0.7: minimum score to include a candidate
|
||||
confidence_threshold=0.6, # default 0.6: minimum confidence to include a candidate
|
||||
max_results=100, # optional: hard cap on total candidates returned
|
||||
top_k_per_entity=3, # optional: max candidates per entity
|
||||
min_similarity=0.75, # optional: additional floor applied after sorting
|
||||
sort_by="confidence", # "confidence" (default) | "similarity_score"
|
||||
)
|
||||
|
||||
@@ -115,11 +115,11 @@ for c in candidates:
|
||||
# Detect duplicate groups with union-find (returns List[DuplicateGroup])
|
||||
groups = detector.detect_duplicate_groups(entities)
|
||||
for g in groups:
|
||||
print("Group of {} — confidence: {:.2f} — representative: {}".format(
|
||||
print("Group of {}: confidence: {:.2f}: representative: {}".format(
|
||||
len(g.entities), g.confidence, g.representative and g.representative.get("name")
|
||||
))
|
||||
|
||||
# Incremental — compare new entities against existing (returns List[DuplicateCandidate])
|
||||
# Incremental: compare new entities against existing (returns List[DuplicateCandidate])
|
||||
new_entities = [{"id": "4", "name": "Apple Corp.", "type": "Company"}]
|
||||
candidates = detector.incremental_detect(new_entities, entities)
|
||||
```
|
||||
@@ -198,7 +198,7 @@ Pass as a string to `strategy=` on `merge_duplicates()` or `merge_entity_group()
|
||||
| `"keep_last"` | Keep the most recently seen entity |
|
||||
| `"keep_most_complete"` | Keep the entity with the most non-null properties + relationships |
|
||||
| `"keep_highest_confidence"` | Keep the entity with the highest `.confidence` value |
|
||||
| `"merge_all"` | Combine all properties — conflicts resolved to lists |
|
||||
| `"merge_all"` | Combine all properties: conflicts resolved to lists |
|
||||
|
||||
### Per-property merge rules
|
||||
|
||||
@@ -262,7 +262,7 @@ print(result.components["property"]) # property overlap component
|
||||
print(result.components["relationship"]) # relationship jaccard component
|
||||
# result.components["embedding"] is present only when embeddings are supplied
|
||||
|
||||
# String similarity methods — method= accepts "levenshtein", "jaro_winkler", "cosine"
|
||||
# String similarity methods: method= accepts "levenshtein", "jaro_winkler", "cosine"
|
||||
lev = calc.calculate_string_similarity("Apple Inc.", "Apple Inc", method="levenshtein")
|
||||
jaro = calc.calculate_string_similarity("Steve Jobs", "Steven Jobs", method="jaro_winkler")
|
||||
cos = calc.calculate_string_similarity("apple", "apples", method="cosine")
|
||||
@@ -301,7 +301,7 @@ result = builder.build_clusters(entities)
|
||||
|
||||
print("Clusters found:", len(result.clusters))
|
||||
for cluster in result.clusters:
|
||||
print(" [{}] {} entities — quality: {:.2f}".format(
|
||||
print(" [{}] {} entities: quality: {:.2f}".format(
|
||||
cluster.cluster_id,
|
||||
len(cluster.entities),
|
||||
cluster.quality_score,
|
||||
@@ -327,7 +327,7 @@ print("Quality metrics:", result.quality_metrics)
|
||||
```python
|
||||
from semantica.deduplication import detect_duplicates, merge_entities, calculate_similarity
|
||||
|
||||
# Detect — method= accepts "pairwise" (default), "batch", "incremental", "group"
|
||||
# Detect: method= accepts "pairwise" (default), "batch", "incremental", "group"
|
||||
candidates = detect_duplicates(
|
||||
entities,
|
||||
method="pairwise",
|
||||
@@ -335,11 +335,11 @@ candidates = detect_duplicates(
|
||||
confidence_threshold=0.6,
|
||||
)
|
||||
|
||||
# Merge — method= accepts the strategy strings, same as EntityMerger
|
||||
# Merge: method= accepts the strategy strings, same as EntityMerger
|
||||
operations = merge_entities(entities, method="keep_most_complete", preserve_provenance=True)
|
||||
# Returns List[MergeOperation]; access .merged_entity on each
|
||||
|
||||
# Similarity — method= accepts "exact", "levenshtein", "jaro_winkler", "cosine",
|
||||
# Similarity: method= accepts "exact", "levenshtein", "jaro_winkler", "cosine",
|
||||
# "property", "relationship", "embedding", "multi_factor" (default)
|
||||
result = calculate_similarity(entity_a, entity_b, method="multi_factor")
|
||||
print(result.score)
|
||||
@@ -402,7 +402,7 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name")
|
||||
```python
|
||||
from semantica.deduplication import ClusterBuilder, EntityMerger
|
||||
|
||||
# Build clusters first — more efficient for large entity sets
|
||||
# Build clusters first: more efficient for large entity sets
|
||||
builder = ClusterBuilder(similarity_threshold=0.8, min_cluster_size=2)
|
||||
result = builder.build_clusters(entities)
|
||||
|
||||
@@ -430,7 +430,7 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name")
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score` — not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`.
|
||||
**`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score`: not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
@@ -446,7 +446,7 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name")
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
**Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters — if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity.
|
||||
**Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters: if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Embeddings Module"
|
||||
description: "Text and graph embedding generation — FastEmbed, Sentence-Transformers, OpenAI, BGE — with pooling strategies and provider-agnostic API."
|
||||
description: "Text and graph embedding generation: FastEmbed, Sentence-Transformers, OpenAI, BGE: with pooling strategies and provider-agnostic API."
|
||||
icon: "vector-square"
|
||||
---
|
||||
|
||||
@@ -15,41 +15,41 @@ icon: "vector-square"
|
||||
|
||||
## Why Embeddings Matter
|
||||
|
||||
Raw text can't be compared mathematically. Embeddings translate meaning into geometry — two semantically similar sentences produce vectors that are close together in high-dimensional space, even when they share no words.
|
||||
Raw text can't be compared mathematically. Embeddings translate meaning into geometry: two semantically similar sentences produce vectors that are close together in high-dimensional space, even when they share no words.
|
||||
|
||||
Semantica uses embeddings for:
|
||||
|
||||
- **Semantic search** — find knowledge graph nodes by meaning, not just keywords
|
||||
- **Entity resolution** — detect that "Apple Inc." and "Apple Computer" refer to the same entity
|
||||
- **Deduplication** — `semantic_v2` strategy measures entity similarity via embedding distance
|
||||
- **GraphRAG retrieval** — hybrid vector + graph traversal for grounded LLM answers
|
||||
- **Semantic chunking** — detect topic shift boundaries in `TextSplitter(method="semantic_transformer")`
|
||||
- **Semantic search**: find knowledge graph nodes by meaning, not just keywords
|
||||
- **Entity resolution**: detect that "Apple Inc." and "Apple Computer" refer to the same entity
|
||||
- **Deduplication**: `semantic_v2` strategy measures entity similarity via embedding distance
|
||||
- **GraphRAG retrieval**: hybrid vector + graph traversal for grounded LLM answers
|
||||
- **Semantic chunking**: detect topic shift boundaries in `TextSplitter(method="semantic_transformer")`
|
||||
|
||||
## Exported Classes
|
||||
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `EmbeddingGenerator` | Provider-agnostic entry point — handles batching and provider selection |
|
||||
| `EmbeddingGenerator` | Provider-agnostic entry point: handles batching and provider selection |
|
||||
| `TextEmbedder` | Text embedding with automatic batch processing; default uses FastEmbed |
|
||||
| `GraphEmbeddingManager` | Embed KG nodes and edges for GraphRAG and graph databases |
|
||||
| `VectorEmbeddingManager` | Prepare and format embeddings for vector database backends |
|
||||
| `OpenAIStore` | OpenAI `text-embedding-3-small` / `text-embedding-3-large` provider |
|
||||
| `BGEStore` | BAAI/bge models via `sentence-transformers` |
|
||||
| `FastEmbedStore` | ONNX-accelerated local embeddings — no CUDA **required** |
|
||||
| `LlamaStore` | Placeholder store — not production-ready; do not use for embeddings |
|
||||
| `MeanPooling` | Default pooling strategy — best for retrieval and clustering |
|
||||
| `FastEmbedStore` | ONNX-accelerated local embeddings: no CUDA **required** |
|
||||
| `LlamaStore` | Placeholder store: not production-ready; do not use for embeddings |
|
||||
| `MeanPooling` | Default pooling strategy: best for retrieval and clustering |
|
||||
|
||||
## What You Get
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="EmbeddingGenerator" icon="vector-square">
|
||||
Main entry point — provider-agnostic, handles batching automatically across all backends.
|
||||
Main entry point: provider-agnostic, handles batching automatically across all backends.
|
||||
</Card>
|
||||
<Card title="TextEmbedder" icon="text-size">
|
||||
Text-specific with automatic batching and progress tracking. Default method is FastEmbed.
|
||||
</Card>
|
||||
<Card title="GraphEmbeddingManager" icon="diagram-project">
|
||||
Node and edge embeddings for graph databases — Neo4j, NetworkX, FalkorDB.
|
||||
Node and edge embeddings for graph databases: Neo4j, NetworkX, FalkorDB.
|
||||
</Card>
|
||||
<Card title="VectorEmbeddingManager" icon="database">
|
||||
Prepare, normalize, and format embeddings for FAISS, Weaviate, Qdrant, and Milvus.
|
||||
@@ -58,7 +58,7 @@ Semantica uses embeddings for:
|
||||
`OpenAIStore`, `BGEStore`, `FastEmbedStore`, and `ProviderStoreFactory`.
|
||||
</Card>
|
||||
<Card title="Pooling Strategies" icon="layer-group">
|
||||
Mean, Max, CLS, Attention, and Hierarchical — control token-to-vector aggregation.
|
||||
Mean, Max, CLS, Attention, and Hierarchical: control token-to-vector aggregation.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -75,7 +75,7 @@ Semantica uses embeddings for:
|
||||
```python
|
||||
from semantica.embeddings import EmbeddingGenerator
|
||||
|
||||
# FastEmbed is the default — no config needed
|
||||
# FastEmbed is the default: no config needed
|
||||
generator = EmbeddingGenerator()
|
||||
embedding = generator.generate_embeddings("Text about AI")
|
||||
```
|
||||
@@ -159,12 +159,12 @@ providers = check_available_providers()
|
||||
|
||||
## Getting Started
|
||||
|
||||
`EmbeddingGenerator` is the fastest path to embeddings — the default method is FastEmbed (ONNX, no GPU needed):
|
||||
`EmbeddingGenerator` is the fastest path to embeddings: the default method is FastEmbed (ONNX, no GPU needed):
|
||||
|
||||
```python
|
||||
from semantica.embeddings import EmbeddingGenerator
|
||||
|
||||
# Default — FastEmbed with BAAI/bge-small-en-v1.5
|
||||
# Default: FastEmbed with BAAI/bge-small-en-v1.5
|
||||
generator = EmbeddingGenerator()
|
||||
|
||||
# Embed a single text
|
||||
@@ -173,7 +173,7 @@ embedding = generator.generate_embeddings("Text about AI")
|
||||
# Embed a batch
|
||||
embeddings = generator.generate_embeddings(["Text about AI", "Machine learning concepts"])
|
||||
|
||||
# Compare two embeddings (cosine similarity — 0.0 to 1.0)
|
||||
# Compare two embeddings (cosine similarity: 0.0 to 1.0)
|
||||
score = generator.compare_embeddings(embeddings[0], embeddings[1], method="cosine")
|
||||
print(f"Similarity: {score:.3f}")
|
||||
```
|
||||
@@ -195,7 +195,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5")
|
||||
```python
|
||||
from semantica.embeddings import EmbeddingGenerator
|
||||
|
||||
# Default — FastEmbed, free, runs locally with no GPU
|
||||
# Default: FastEmbed, free, runs locally with no GPU
|
||||
generator = EmbeddingGenerator()
|
||||
|
||||
# Use sentence-transformers instead
|
||||
@@ -213,7 +213,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5")
|
||||
</Step>
|
||||
<Step title="Compute similarity">
|
||||
```python
|
||||
# Cosine similarity — 0.0 (unrelated) to 1.0 (identical meaning)
|
||||
# Cosine similarity: 0.0 (unrelated) to 1.0 (identical meaning)
|
||||
score = generator.compare_embeddings(embeddings[0], embeddings[1], method="cosine")
|
||||
print(f"Similarity: {score:.3f}")
|
||||
```
|
||||
@@ -240,7 +240,7 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5")
|
||||
|
||||
| Provider | Model | Dimension | Speed | Best For |
|
||||
| :-------- | :----- | :--------- | :----- | :-------- |
|
||||
| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Very fast | **Default** — CPU-optimised, no GPU **required** |
|
||||
| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Very fast | **Default**: CPU-optimised, no GPU **required** |
|
||||
| `sentence_transformers` | `all-MiniLM-L6-v2` | 384 | Fast | Good balance of speed and quality |
|
||||
| `sentence_transformers` | `all-mpnet-base-v2` | 768 | Medium | Higher retrieval quality |
|
||||
| `sentence_transformers` | `BAAI/bge-large-en-v1.5` | 1024 | Medium | State-of-the-art retrieval accuracy |
|
||||
@@ -254,13 +254,13 @@ generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5")
|
||||
```python
|
||||
from semantica.embeddings import EmbeddingGenerator
|
||||
|
||||
# Default — FastEmbed with BAAI/bge-small-en-v1.5
|
||||
# Default: FastEmbed with BAAI/bge-small-en-v1.5
|
||||
generator = EmbeddingGenerator()
|
||||
embeddings = generator.generate_embeddings(texts)
|
||||
similarity = generator.compare_embeddings(embeddings[0], embeddings[1])
|
||||
```
|
||||
|
||||
**Best for:** CPU-only production, lowest latency without GPU. Default — works out of the box.
|
||||
**Best for:** CPU-only production, lowest latency without GPU. Default: works out of the box.
|
||||
</Tab>
|
||||
<Tab title="Sentence-Transformers">
|
||||
```python
|
||||
@@ -315,7 +315,7 @@ Direct text embedding with batch processing:
|
||||
```python
|
||||
from semantica.embeddings import TextEmbedder
|
||||
|
||||
# Default — FastEmbed with BAAI/bge-small-en-v1.5
|
||||
# Default: FastEmbed with BAAI/bge-small-en-v1.5
|
||||
embedder = TextEmbedder()
|
||||
|
||||
# Single text → 1D array
|
||||
@@ -341,7 +341,7 @@ dim = embedder.get_embedding_dimension()
|
||||
| `normalize` | `bool` | `True` | L2-normalize output vectors |
|
||||
|
||||
**Key behaviours:**
|
||||
- If FastEmbed or sentence-transformers is unavailable, falls back to a 128-dimensional hash-based embedding. Hash embeddings are deterministic but not semantic — do not use in production.
|
||||
- If FastEmbed or sentence-transformers is unavailable, falls back to a 128-dimensional hash-based embedding. Hash embeddings are deterministic but not semantic: do not use in production.
|
||||
- Large batches are chunked internally by the underlying library to avoid OOM.
|
||||
|
||||
## Provider Stores
|
||||
@@ -359,28 +359,28 @@ import os
|
||||
store = OpenAIStore(api_key=os.getenv("OPENAI_API_KEY"), model="text-embedding-3-small")
|
||||
embedding = store.embed("Hello world")
|
||||
|
||||
# BGE (Sentence-Transformers wrapper) — pass model_name= not model=
|
||||
# BGE (Sentence-Transformers wrapper): pass model_name= not model=
|
||||
store = BGEStore(model_name="BAAI/bge-large-en-v1.5")
|
||||
embedding = store.embed("Hello world")
|
||||
|
||||
# FastEmbed — ONNX runtime, no CUDA required
|
||||
# FastEmbed: ONNX runtime, no CUDA required
|
||||
store = FastEmbedStore(model_name="BAAI/bge-small-en-v1.5")
|
||||
embedding = store.embed("Hello world")
|
||||
# FastEmbedStore also has an efficient batch method
|
||||
embeddings = store.embed_batch(["text1", "text2", "text3"])
|
||||
|
||||
# Auto-select from a name string — useful in config-driven pipelines
|
||||
# Auto-select from a name string: useful in config-driven pipelines
|
||||
# Supported providers: "openai", "bge", "fastembed"
|
||||
store = ProviderStoreFactory.create(provider="bge", model_name="BAAI/bge-large-en-v1.5")
|
||||
```
|
||||
|
||||
<Note>
|
||||
`LlamaStore` exists in the module but is a placeholder — it does not connect to Ollama and always raises `ProcessingError` at embed time. Do not use it in production.
|
||||
`LlamaStore` exists in the module but is a placeholder: it does not connect to Ollama and always raises `ProcessingError` at embed time. Do not use it in production.
|
||||
</Note>
|
||||
|
||||
## Pooling Strategies
|
||||
|
||||
Pooling aggregates a set of embeddings into a single vector — useful when you have multiple chunk embeddings to combine:
|
||||
Pooling aggregates a set of embeddings into a single vector: useful when you have multiple chunk embeddings to combine:
|
||||
|
||||
<Tabs>
|
||||
<Tab title="MeanPooling (default)">
|
||||
@@ -391,7 +391,7 @@ Pooling aggregates a set of embeddings into a single vector — useful when you
|
||||
pooled = pooler.pool(token_embeddings) # shape: (hidden_dim,)
|
||||
```
|
||||
|
||||
**Best for:** retrieval, semantic search, and clustering — averages all contributions.
|
||||
**Best for:** retrieval, semantic search, and clustering: averages all contributions.
|
||||
</Tab>
|
||||
<Tab title="MaxPooling">
|
||||
```python
|
||||
@@ -401,7 +401,7 @@ Pooling aggregates a set of embeddings into a single vector — useful when you
|
||||
pooled = pooler.pool(token_embeddings)
|
||||
```
|
||||
|
||||
**Best for:** capturing the presence of any feature — takes the max activation per dimension.
|
||||
**Best for:** capturing the presence of any feature: takes the max activation per dimension.
|
||||
</Tab>
|
||||
<Tab title="CLSPooling">
|
||||
```python
|
||||
@@ -422,7 +422,7 @@ Pooling aggregates a set of embeddings into a single vector — useful when you
|
||||
pooled = pooler.pool(token_embeddings, chunk_size=10)
|
||||
```
|
||||
|
||||
**Best for:** long documents — chunk-level mean pooling, then global mean pooling across chunks.
|
||||
**Best for:** long documents: chunk-level mean pooling, then global mean pooling across chunks.
|
||||
</Tab>
|
||||
<Tab title="Strategy Comparison">
|
||||
|
||||
@@ -518,7 +518,7 @@ combined = manager.batch_prepare([embeddings_a, embeddings_b], backend="qdrant")
|
||||
"Amazon was started by Jeff Bezos.",
|
||||
]
|
||||
|
||||
# All at once — more efficient than calling embed_text() per item
|
||||
# All at once: more efficient than calling embed_text() per item
|
||||
embeddings = embedder.embed_batch(texts)
|
||||
print(f"Shape: {embeddings.shape}") # (3, 384)
|
||||
```
|
||||
@@ -579,7 +579,7 @@ combined = manager.batch_prepare([embeddings_a, embeddings_b], backend="qdrant")
|
||||
```python
|
||||
from semantica.embeddings import calculate_similarity
|
||||
|
||||
# Cosine similarity — direction only, not magnitude; most common for text
|
||||
# Cosine similarity: direction only, not magnitude; most common for text
|
||||
score = calculate_similarity(embedding_a, embedding_b, method="cosine")
|
||||
# → 0.0 (orthogonal / unrelated) to 1.0 (identical direction)
|
||||
|
||||
@@ -595,7 +595,7 @@ from semantica.embeddings import (
|
||||
pool_embeddings, check_available_providers,
|
||||
)
|
||||
|
||||
# Single text — fastest path
|
||||
# Single text: fastest path
|
||||
emb = embed_text("Hello world", method="sentence_transformers")
|
||||
|
||||
# Batch
|
||||
@@ -624,15 +624,15 @@ providers = check_available_providers()
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
**FastEmbed ignores the `device` parameter.** FastEmbed uses ONNX Runtime and manages its own execution providers — passing `device="cuda"` has no effect. Switch to `method="sentence_transformers"` if you need explicit GPU control.
|
||||
**FastEmbed ignores the `device` parameter.** FastEmbed uses ONNX Runtime and manages its own execution providers: passing `device="cuda"` has no effect. Switch to `method="sentence_transformers"` if you need explicit GPU control.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Always use the same model for indexing and querying.** Vectors from different models are not comparable — they live in different vector spaces. Switching models requires re-embedding your entire corpus.
|
||||
**Always use the same model for indexing and querying.** Vectors from different models are not comparable: they live in different vector spaces. Switching models requires re-embedding your entire corpus.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
**Fallback embeddings are not semantic.** If neither FastEmbed nor sentence-transformers loads successfully, TextEmbedder silently falls back to 128-dimensional SHA-256 hash embeddings. These are deterministic but carry no semantic meaning. Check `embedder.get_method()` — if it returns `"fallback"`, install your intended provider.
|
||||
**Fallback embeddings are not semantic.** If neither FastEmbed nor sentence-transformers loads successfully, TextEmbedder silently falls back to 128-dimensional SHA-256 hash embeddings. These are deterministic but carry no semantic meaning. Check `embedder.get_method()`: if it returns `"fallback"`, install your intended provider.
|
||||
</Tip>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Evals Module"
|
||||
description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance — coming soon."
|
||||
description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance: coming soon."
|
||||
icon: "chart-line"
|
||||
---
|
||||
|
||||
|
||||
+50
-50
@@ -6,11 +6,11 @@ icon: "map"
|
||||
|
||||
**`semantica.explorer`** is a **browser-based dashboard** for exploring knowledge graphs, managing ontologies, and running visual analyses:
|
||||
|
||||
- Indexed search: 0.004ms on 118k nodes — no full scans
|
||||
- Indexed search: 0.004ms on 118k nodes: no full scans
|
||||
- Ontology Hub: visual editor, SHACL Studio, alignment authoring, and health dashboard
|
||||
- Bidirectional path finding between any two nodes
|
||||
- WebSocket progress streaming for live pipeline monitoring
|
||||
- No code required after launch — full graph exploration in the browser
|
||||
- No code required after launch: full graph exploration in the browser
|
||||
|
||||
|
||||
## Getting Started
|
||||
@@ -37,7 +37,7 @@ graph.save_to_file("my_graph.json")
|
||||
# 2. Launch the Explorer
|
||||
semantica-explorer --graph my_graph.json
|
||||
# → Loading graph...
|
||||
# → Graph loaded — 2 nodes, 1 edges
|
||||
# → Graph loaded: 2 nodes, 1 edges
|
||||
# → Semantica Explorer · http://127.0.0.1:8000
|
||||
# API docs http://127.0.0.1:8000/docs
|
||||
# Health http://127.0.0.1:8000/api/health
|
||||
@@ -96,8 +96,8 @@ The `semantica-explorer` command accepts exactly four flags:
|
||||
| :---- | :----- | :------- | :----------- |
|
||||
| `--graph` | `-g` | *(**required**)* | Path to a ContextGraph JSON file to load |
|
||||
| `--port` | `-p` | `8000` | Port to bind the server |
|
||||
| `--host` | — | `127.0.0.1` | Host to bind the server — use `0.0.0.0` to expose on the network |
|
||||
| `--no-browser` | — | off | Skip auto-opening the browser tab |
|
||||
| `--host` |: | `127.0.0.1` | Host to bind the server: use `0.0.0.0` to expose on the network |
|
||||
| `--no-browser` |: | off | Skip auto-opening the browser tab |
|
||||
|
||||
<Note>
|
||||
There are no flags for authentication, CORS, or log level in the CLI. CORS allowed origins are configured via the `EXPLORER_CORS_ORIGINS` environment variable (comma-separated, default: `http://localhost:5173,http://127.0.0.1:5173`).
|
||||
@@ -122,7 +122,7 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \
|
||||
Degree centrality, community detection, connectivity analysis, graph validation, and distance matrices.
|
||||
</Card>
|
||||
<Card title="REST API" icon="code">
|
||||
All features available as a REST API — fully documented at `/docs`.
|
||||
All features available as a REST API: fully documented at `/docs`.
|
||||
</Card>
|
||||
<Card title="WebSocket Updates" icon="bolt">
|
||||
Real-time graph mutation events streamed over WebSocket at `/ws/graph-updates`.
|
||||
@@ -138,42 +138,42 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \
|
||||
<Tab title="Graph Explorer">
|
||||
Core dashboard for navigating knowledge graphs:
|
||||
|
||||
- **Indexed search** — POST to `/api/graph/search` with a query; 0.004ms on 118k-node graphs
|
||||
- **Path finding** — BFS or Dijkstra between any two nodes via `GET /api/graph/path?source=&target=`
|
||||
- **Neighbor expansion** — `GET /api/graph/node/{id}/neighbors?depth=2`
|
||||
- **Filter by entity type** — `GET /api/graph/nodes?type=Person`
|
||||
- **Semantic neighborhood** — `GET /api/graph/semantic-neighborhood?node_id=&top_k=20`
|
||||
- **Distance matrix** — `POST /api/graph/distance-matrix`
|
||||
- **Indexed search**: POST to `/api/graph/search` with a query; 0.004ms on 118k-node graphs
|
||||
- **Path finding**: BFS or Dijkstra between any two nodes via `GET /api/graph/path?source=&target=`
|
||||
- **Neighbor expansion**: `GET /api/graph/node/{id}/neighbors?depth=2`
|
||||
- **Filter by entity type**: `GET /api/graph/nodes?type=Person`
|
||||
- **Semantic neighborhood**: `GET /api/graph/semantic-neighborhood?node_id=&top_k=20`
|
||||
- **Distance matrix**: `POST /api/graph/distance-matrix`
|
||||
</Tab>
|
||||
<Tab title="Ontology Hub">
|
||||
Ontology lifecycle management in the browser:
|
||||
|
||||
- **Registry** — `GET /api/ontology/registry` — list loaded ontologies
|
||||
- **SKOS vocabularies** — `GET /api/ontology/skos/schemes`, `GET /api/ontology/skos/concept/{uri}`
|
||||
- **SHACL** — `POST /api/ontology/shacl/generate`, `POST /api/ontology/shacl/validate`
|
||||
- **Alignments** — `GET/POST /api/ontology/alignments`, `POST /api/ontology/suggest-alignments`
|
||||
- **Proposals & versioning** — `POST /api/ontology/propose`, `GET /api/ontology/versions/{uri}`
|
||||
- **Health** — `GET /api/ontology/health`
|
||||
- **Registry**: `GET /api/ontology/registry`: list loaded ontologies
|
||||
- **SKOS vocabularies**: `GET /api/ontology/skos/schemes`, `GET /api/ontology/skos/concept/{uri}`
|
||||
- **SHACL**: `POST /api/ontology/shacl/generate`, `POST /api/ontology/shacl/validate`
|
||||
- **Alignments**: `GET/POST /api/ontology/alignments`, `POST /api/ontology/suggest-alignments`
|
||||
- **Proposals & versioning**: `POST /api/ontology/propose`, `GET /api/ontology/versions/{uri}`
|
||||
- **Health**: `GET /api/ontology/health`
|
||||
</Tab>
|
||||
<Tab title="Analytics">
|
||||
Graph metrics running against the loaded graph:
|
||||
|
||||
- **Combined metrics** — `GET /api/analytics?metrics=centrality,community,connectivity`
|
||||
- **Graph validation** — `GET /api/analytics/validation`
|
||||
- **Enrich: link prediction** — `POST /api/enrich/links`
|
||||
- **Enrich: deduplication** — `POST /api/enrich/dedup`
|
||||
- **Enrich: entity extraction** — `POST /api/enrich/extract`
|
||||
- **Temporal** — `GET /api/temporal/snapshot`, `GET /api/temporal/diff`, `GET /api/temporal/bounds`
|
||||
- **Combined metrics**: `GET /api/analytics?metrics=centrality,community,connectivity`
|
||||
- **Graph validation**: `GET /api/analytics/validation`
|
||||
- **Enrich: link prediction**: `POST /api/enrich/links`
|
||||
- **Enrich: deduplication**: `POST /api/enrich/dedup`
|
||||
- **Enrich: entity extraction**: `POST /api/enrich/extract`
|
||||
- **Temporal**: `GET /api/temporal/snapshot`, `GET /api/temporal/diff`, `GET /api/temporal/bounds`
|
||||
</Tab>
|
||||
<Tab title="Decisions & Provenance">
|
||||
Decision tracking and provenance queries:
|
||||
|
||||
- **Decisions** — `GET /api/decisions`, `GET /api/decisions/{id}`, `GET /api/decisions/{id}/chain`
|
||||
- **Precedents** — `GET /api/decisions/{id}/precedents`
|
||||
- **Causal distance** — `GET /api/decisions/causal-distance?source=&target=`
|
||||
- **Compliance** — `GET /api/decisions/{id}/compliance`
|
||||
- **Provenance** — `GET /api/provenance?node_id=`, `GET /api/provenance/report?node_id=`
|
||||
- **Annotations** — `GET/POST /api/annotations`, `DELETE /api/annotations/{id}`
|
||||
- **Decisions**: `GET /api/decisions`, `GET /api/decisions/{id}`, `GET /api/decisions/{id}/chain`
|
||||
- **Precedents**: `GET /api/decisions/{id}/precedents`
|
||||
- **Causal distance**: `GET /api/decisions/causal-distance?source=&target=`
|
||||
- **Compliance**: `GET /api/decisions/{id}/compliance`
|
||||
- **Provenance**: `GET /api/provenance?node_id=`, `GET /api/provenance/report?node_id=`
|
||||
- **Annotations**: `GET/POST /api/annotations`, `DELETE /api/annotations/{id}`
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
@@ -187,14 +187,14 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and
|
||||
| Endpoint | Method | Description |
|
||||
| :-------- | :------ | :----------- |
|
||||
| `/api/graph/stats` | `GET` | Node count, edge count, entity type distribution |
|
||||
| `/api/graph/nodes` | `GET` | List nodes — `?type=&search=&skip=&limit=&cursor=&bbox=` |
|
||||
| `/api/graph/nodes` | `GET` | List nodes: `?type=&search=&skip=&limit=&cursor=&bbox=` |
|
||||
| `/api/graph/node/{id}` | `GET` | Fetch a single node with all properties |
|
||||
| `/api/graph/node/{id}/neighbors` | `GET` | Neighbors of a node — `?depth=1` (1–5) |
|
||||
| `/api/graph/edges` | `GET` | List edges — `?type=&source=&target=&skip=&limit=&cursor=` |
|
||||
| `/api/graph/path` | `GET` | Shortest path — `?source=&target=&algorithm=bfs&directed=true` |
|
||||
| `/api/graph/search` | `POST` | Indexed search — body: `{query, limit, filters, anchor_node}` |
|
||||
| `/api/graph/distance-matrix` | `POST` | Pairwise distances — body: `{node_ids, metric}` (max 50 nodes) |
|
||||
| `/api/graph/semantic-neighborhood` | `GET` | Semantic neighbors — `?node_id=&top_k=20&min_similarity=0.0` |
|
||||
| `/api/graph/node/{id}/neighbors` | `GET` | Neighbors of a node: `?depth=1` (1–5) |
|
||||
| `/api/graph/edges` | `GET` | List edges: `?type=&source=&target=&skip=&limit=&cursor=` |
|
||||
| `/api/graph/path` | `GET` | Shortest path: `?source=&target=&algorithm=bfs&directed=true` |
|
||||
| `/api/graph/search` | `POST` | Indexed search: body: `{query, limit, filters, anchor_node}` |
|
||||
| `/api/graph/distance-matrix` | `POST` | Pairwise distances: body: `{node_ids, metric}` (max 50 nodes) |
|
||||
| `/api/graph/semantic-neighborhood` | `GET` | Semantic neighbors: `?node_id=&top_k=20&min_similarity=0.0` |
|
||||
|
||||
</Accordion>
|
||||
<Accordion title="Analytics, Enrich & Temporal">
|
||||
@@ -203,7 +203,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| :-------- | :------ | :----------- |
|
||||
| `/api/analytics` | `GET` | Graph metrics — `?metrics=centrality,community,connectivity` |
|
||||
| `/api/analytics` | `GET` | Graph metrics: `?metrics=centrality,community,connectivity` |
|
||||
| `/api/analytics/validation` | `GET` | Graph validation report |
|
||||
|
||||
**Enrich:**
|
||||
@@ -221,10 +221,10 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and
|
||||
| Endpoint | Method | Description |
|
||||
| :-------- | :------ | :----------- |
|
||||
| `/api/temporal/snapshot` | `GET` | Graph snapshot at `?at=ISO8601` (defaults to now) |
|
||||
| `/api/temporal/diff` | `GET` | Diff between two times — `?from_time=&to_time=` |
|
||||
| `/api/temporal/diff` | `GET` | Diff between two times: `?from_time=&to_time=` |
|
||||
| `/api/temporal/patterns` | `GET` | Temporal activity patterns |
|
||||
| `/api/temporal/bounds` | `GET` | Earliest and latest temporal bounds in graph |
|
||||
| `/api/temporal/distance-history` | `GET` | Distance history — `?source=&target=` |
|
||||
| `/api/temporal/distance-history` | `GET` | Distance history: `?source=&target=` |
|
||||
|
||||
</Accordion>
|
||||
<Accordion title="Ontology, Vocabulary & SPARQL">
|
||||
@@ -236,7 +236,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and
|
||||
| `/api/ontology/registry` | `GET` | List loaded ontologies |
|
||||
| `/api/ontology/load` | `POST` | Load an ontology from URL or content |
|
||||
| `/api/ontology/create` | `POST` | Create a new ontology |
|
||||
| `/api/ontology/search` | `GET` | Search ontology entities — `?q=term` |
|
||||
| `/api/ontology/search` | `GET` | Search ontology entities: `?q=term` |
|
||||
| `/api/ontology/health` | `GET` | Ontology health and coverage metrics |
|
||||
| `/api/ontology/alignments` | `GET/POST` | List or create ontology alignments |
|
||||
| `/api/ontology/suggest-alignments` | `POST` | AI-suggested alignments |
|
||||
@@ -252,7 +252,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and
|
||||
| Endpoint | Method | Description |
|
||||
| :-------- | :------ | :----------- |
|
||||
| `/api/vocabulary/schemes` | `GET` | SKOS schemes via TripletStore |
|
||||
| `/api/vocabulary/concepts` | `GET` | Concepts in a scheme — `?scheme=URI` |
|
||||
| `/api/vocabulary/concepts` | `GET` | Concepts in a scheme: `?scheme=URI` |
|
||||
| `/api/vocabulary/hierarchy` | `GET` | Concept hierarchy tree |
|
||||
| `/api/vocabulary/import` | `POST` | Import SKOS/RDF vocabulary file |
|
||||
|
||||
@@ -274,20 +274,20 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and
|
||||
| `/api/decisions/{id}/chain` | `GET` | Causal chain for a decision |
|
||||
| `/api/decisions/{id}/precedents` | `GET` | Similar past decisions |
|
||||
| `/api/decisions/{id}/compliance` | `GET` | Policy compliance check |
|
||||
| `/api/decisions/causal-distance` | `GET` | Causal distance — `?source=&target=` |
|
||||
| `/api/decisions/causal-distance` | `GET` | Causal distance: `?source=&target=` |
|
||||
|
||||
**Provenance:**
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| :-------- | :------ | :----------- |
|
||||
| `/api/provenance` | `GET` | Entity provenance lineage — `?node_id=` |
|
||||
| `/api/provenance/report` | `GET` | Provenance export report — `?node_id=` |
|
||||
| `/api/provenance` | `GET` | Entity provenance lineage: `?node_id=` |
|
||||
| `/api/provenance/report` | `GET` | Provenance export report: `?node_id=` |
|
||||
|
||||
**Annotations:**
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| :-------- | :------ | :----------- |
|
||||
| `/api/annotations` | `GET` | List annotations — `?node_id=` (optional) |
|
||||
| `/api/annotations` | `GET` | List annotations: `?node_id=` (optional) |
|
||||
| `/api/annotations` | `POST` | Create annotation (returns 201) |
|
||||
| `/api/annotations/{id}` | `DELETE` | Delete annotation (returns 204) |
|
||||
|
||||
@@ -295,7 +295,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| :-------- | :------ | :----------- |
|
||||
| `/api/export` | `POST` | Export graph as JSON or CSV — body: `{format, node_ids}` |
|
||||
| `/api/export` | `POST` | Export graph as JSON or CSV: body: `{format, node_ids}` |
|
||||
| `/api/export/distance-enriched` | `POST` | Export pairwise distances as CSV or JSONL |
|
||||
| `/api/import` | `POST` | Import nodes/edges from `.json` or `.csv` file (max 50 MB) |
|
||||
|
||||
@@ -306,7 +306,7 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and
|
||||
| :-------- | :------ | :----------- |
|
||||
| `/api/health` | `GET` | Returns `{"status": "healthy"}` |
|
||||
| `/api/info` | `GET` | Server name, version, status |
|
||||
| `/docs` | `GET` | Interactive Swagger UI — all endpoints |
|
||||
| `/docs` | `GET` | Interactive Swagger UI: all endpoints |
|
||||
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
@@ -395,7 +395,7 @@ Session state is in-memory only. Use `POST /api/export` to save a JSON snapshot
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph before exporting — the force-directed layout becomes unusable on very large graphs.
|
||||
**Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph before exporting: the force-directed layout becomes unusable on very large graphs.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
@@ -403,7 +403,7 @@ Session state is in-memory only. Use `POST /api/export` to save a JSON snapshot
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API — pipe them into scripts to automate batch annotation, SPARQL querying, or exports.
|
||||
**Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API: pipe them into scripts to automate batch annotation, SPARQL querying, or exports.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: "file-export"
|
||||
|
||||
**`semantica.export`** serializes knowledge graphs to **every downstream format**:
|
||||
|
||||
- RDF: Turtle, JSON-LD, N-Triples, RDF/XML — with optional W3C PROV-O provenance inline
|
||||
- RDF: Turtle, JSON-LD, N-Triples, RDF/XML: with optional W3C PROV-O provenance inline
|
||||
- Analytics: Apache Parquet and Arrow for Spark, BigQuery, Databricks
|
||||
- Graph databases: Cypher `CREATE` statements for Neo4j; AQL `INSERT` for ArangoDB
|
||||
- Standard formats: GraphML, GEXF, Graphviz DOT, CSV, OWL 2.0
|
||||
@@ -28,7 +28,7 @@ icon: "file-export"
|
||||
| `ArrowExporter` | Apache Arrow IPC | Requires `pyarrow`; zero-copy transfer |
|
||||
| `DistanceExporter` | CSV, JSONL | Pairwise distance metrics; takes a `graph` arg |
|
||||
| `ReportGenerator` | HTML, Markdown, JSON, plain text | Analytics reports |
|
||||
| `NamespaceManager` | — | RDF namespace extraction and declaration generation |
|
||||
| `NamespaceManager` |: | RDF namespace extraction and declaration generation |
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -91,7 +91,7 @@ export_lpg(graph, "import.cypher", method="cypher")
|
||||
|
||||
<Tabs>
|
||||
<Tab title="RDF">
|
||||
Export to W3C RDF formats — Turtle, JSON-LD, N-Triples, and RDF/XML.
|
||||
Export to W3C RDF formats: Turtle, JSON-LD, N-Triples, and RDF/XML.
|
||||
|
||||
**`export_to_rdf()` returns a string; `export()` writes to a file:**
|
||||
|
||||
@@ -235,7 +235,7 @@ export_lpg(graph, "import.cypher", method="cypher")
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Vectors, Arrow & Reports">
|
||||
**VectorExporter** — takes `(vectors, file_path, format=)`:
|
||||
**VectorExporter**: takes `(vectors, file_path, format=)`:
|
||||
|
||||
```python
|
||||
from semantica.export import VectorExporter
|
||||
@@ -248,7 +248,7 @@ export_lpg(graph, "import.cypher", method="cypher")
|
||||
exporter.export(vectors, "vectors.faiss", format="faiss")
|
||||
```
|
||||
|
||||
**ArrowExporter** — requires `pyarrow`:
|
||||
**ArrowExporter**: requires `pyarrow`:
|
||||
|
||||
```python
|
||||
from semantica.export import ArrowExporter
|
||||
@@ -257,7 +257,7 @@ export_lpg(graph, "import.cypher", method="cypher")
|
||||
exporter.export(graph, "graph.arrow")
|
||||
```
|
||||
|
||||
**DistanceExporter** — takes a `graph` argument at construction:
|
||||
**DistanceExporter**: takes a `graph` argument at construction:
|
||||
|
||||
```python
|
||||
from semantica.export import DistanceExporter
|
||||
@@ -347,12 +347,12 @@ The `export_csv` convenience function delegates to `CSVExporter.export()`. For p
|
||||
| `"numpy"` | `numpy` | `VectorExporter` | `.npz` | NumPy arrays from embeddings |
|
||||
| `"binary"` | `binary` | `VectorExporter` | `.bin` | Raw float32 binary |
|
||||
| `"faiss"` | `faiss` | `VectorExporter` | `.faiss` | Direct FAISS index files |
|
||||
| `"html"` / `"markdown"` / `"json"` / `"text"` | — | `ReportGenerator` | `.html` / `.md` / `.json` / `.txt` | Analytics reports |
|
||||
| `"html"` / `"markdown"` / `"json"` / `"text"` |: | `ReportGenerator` | `.html` / `.md` / `.json` / `.txt` | Analytics reports |
|
||||
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**`export_to_rdf()` returns a string — it does not write a file.** Call `export()` or `export_knowledge_graph()` to write directly to disk.
|
||||
**`export_to_rdf()` returns a string: it does not write a file.** Call `export()` or `export_knowledge_graph()` to write directly to disk.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
@@ -372,7 +372,7 @@ The `export_csv` convenience function delegates to `CSVExporter.export()`. For p
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
**Use `turtle` for human readability, `ntriples` for streaming.** Turtle is compact and readable for debugging and sharing. N-Triples (`.nt`) is line-oriented — one triple per line — making it safe to stream, concatenate, and process with standard Unix tools.
|
||||
**Use `turtle` for human readability, `ntriples` for streaming.** Turtle is compact and readable for debugging and sharing. N-Triples (`.nt`) is line-oriented: one triple per line: making it safe to stream, concatenate, and process with standard Unix tools.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
|
||||
@@ -6,9 +6,9 @@ icon: "server"
|
||||
|
||||
**`semantica.graph_store`** provides a **single unified API** for persisting and querying knowledge graphs in production graph databases:
|
||||
|
||||
- Swap backends with a one-line change — Neo4j, FalkorDB, Apache AGE, Amazon Neptune
|
||||
- Swap backends with a one-line change: Neo4j, FalkorDB, Apache AGE, Amazon Neptune
|
||||
- Parameterized Cypher execution with optional result caching via `QueryEngine`
|
||||
- Batch node and edge loading — faster than individual writes
|
||||
- Batch node and edge loading: faster than individual writes
|
||||
- `GraphAnalytics` for degree centrality, connected components, shortest path, neighbor traversal
|
||||
- Context manager support: `with GraphStore(...) as store:` closes connection automatically
|
||||
|
||||
@@ -20,10 +20,10 @@ icon: "server"
|
||||
| `GraphStore` | Unified interface: `create_node`, `create_relationship`, `query`, `get_neighbors`, `shortest_path` |
|
||||
| `QueryEngine` | Parameterized Cypher execution with result caching |
|
||||
| `GraphAnalytics` | `degree_centrality`, `connected_components`, `shortest_path`, `get_neighbors` |
|
||||
| `Neo4jStore` | Production workloads via Bolt — supports APOC and GDS plugins |
|
||||
| `ApacheAgeStore` | PostgreSQL + AGE extension — no separate graph server needed |
|
||||
| `AmazonNeptuneStore` | AWS Neptune — OpenCypher via Bolt protocol |
|
||||
| `FalkorDBStore` | Redis-based — sub-millisecond latency for real-time applications |
|
||||
| `Neo4jStore` | Production workloads via Bolt: supports APOC and GDS plugins |
|
||||
| `ApacheAgeStore` | PostgreSQL + AGE extension: no separate graph server needed |
|
||||
| `AmazonNeptuneStore` | AWS Neptune: OpenCypher via Bolt protocol |
|
||||
| `FalkorDBStore` | Redis-based: sub-millisecond latency for real-time applications |
|
||||
|
||||
|
||||
## What You Get
|
||||
@@ -32,7 +32,7 @@ icon: "server"
|
||||
<Card title="GraphStore" icon="server">
|
||||
- Unified API across Neo4j, FalkorDB, Apache AGE, Amazon Neptune
|
||||
- Context manager support for automatic connection cleanup
|
||||
- `create_nodes()` for bulk loading — faster than individual calls
|
||||
- `create_nodes()` for bulk loading: faster than individual calls
|
||||
</Card>
|
||||
<Card title="QueryEngine" icon="magnifying-glass">
|
||||
- Parameterized Cypher construction prevents injection attacks
|
||||
@@ -45,13 +45,13 @@ icon: "server"
|
||||
- Shortest path between nodes, neighbor traversal up to N hops
|
||||
</Card>
|
||||
<Card title="Bulk Operations" icon="layer-group">
|
||||
- `create_nodes(list)` — one round-trip for many nodes
|
||||
- `create_nodes(list)`: one round-trip for many nodes
|
||||
- `create_relationship()` with typed properties
|
||||
- `delete_node(detach=True)` removes all connected relationships
|
||||
</Card>
|
||||
<Card title="Schema Management" icon="table">
|
||||
- `create_index(label, property_name=)` — makes MATCH queries orders-of-magnitude faster
|
||||
- `get_stats()` — node counts, edge counts, type breakdown
|
||||
- `create_index(label, property_name=)`: makes MATCH queries orders-of-magnitude faster
|
||||
- `get_stats()`: node counts, edge counts, type breakdown
|
||||
- Create indexes before bulk loading for best performance
|
||||
</Card>
|
||||
<Card title="Path Traversal" icon="route">
|
||||
@@ -132,7 +132,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass
|
||||
</Step>
|
||||
<Step title="Load nodes and relationships">
|
||||
```python
|
||||
# Batch creation — list of dicts with "labels" and "properties" keys
|
||||
# Batch creation: list of dicts with "labels" and "properties" keys
|
||||
store.create_nodes([
|
||||
{"labels": ["Person"], "properties": {"name": "Alice"}},
|
||||
{"labels": ["Organization"], "properties": {"name": "Acme Corp"}},
|
||||
@@ -187,7 +187,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass
|
||||
uri="bolt://localhost:7687",
|
||||
user="neo4j",
|
||||
password="password",
|
||||
database="neo4j", # optional — targets default database
|
||||
database="neo4j", # optional: targets default database
|
||||
)
|
||||
store.connect()
|
||||
```
|
||||
@@ -248,7 +248,7 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass
|
||||
)
|
||||
```
|
||||
|
||||
**Best for:** managed AWS deployments. Neptune uses the Bolt protocol for OpenCypher queries — the same query API used for Neo4j.
|
||||
**Best for:** managed AWS deployments. Neptune uses the Bolt protocol for OpenCypher queries: the same query API used for Neo4j.
|
||||
</Tab>
|
||||
<Tab title="Backend Comparison">
|
||||
|
||||
@@ -300,7 +300,7 @@ neighbors = store.get_neighbors(
|
||||
depth=1,
|
||||
)
|
||||
|
||||
# Shortest path — returns {"length", "nodes", "relationships"} or None
|
||||
# Shortest path: returns {"length", "nodes", "relationships"} or None
|
||||
path = store.shortest_path(
|
||||
start_node_id=jobs_id,
|
||||
end_node_id=cook_id,
|
||||
@@ -331,7 +331,7 @@ result = engine.execute(
|
||||
)
|
||||
# result is {"success": True, "records": [...], "keys": [...], "metadata": {...}}
|
||||
|
||||
# Execute with caching — repeated identical calls return cached result
|
||||
# Execute with caching: repeated identical calls return cached result
|
||||
result = engine.execute(
|
||||
"MATCH (p:Person) RETURN count(p) as total",
|
||||
use_cache=True,
|
||||
@@ -368,7 +368,7 @@ store.connect()
|
||||
# GraphAnalytics takes the backend store, not the GraphStore facade
|
||||
analytics = GraphAnalytics(store._store_backend)
|
||||
|
||||
# Degree centrality — returns list of {"id", "degree"} dicts ordered by degree DESC
|
||||
# Degree centrality: returns list of {"id", "degree"} dicts ordered by degree DESC
|
||||
scores = analytics.degree_centrality(
|
||||
labels=["Person"],
|
||||
rel_type="KNOWS",
|
||||
@@ -380,7 +380,7 @@ for entry in scores[:5]:
|
||||
# Connected components (requires GDS for Neo4j, NetworkX for in-process)
|
||||
components = analytics.connected_components(labels=["Person"])
|
||||
|
||||
# Shortest path — returns {"length", "nodes", "relationships"} or None
|
||||
# Shortest path: returns {"length", "nodes", "relationships"} or None
|
||||
path = analytics.shortest_path(
|
||||
start_node_id=alice_id,
|
||||
end_node_id=charlie_id,
|
||||
@@ -401,7 +401,7 @@ neighbors = analytics.get_neighbors(
|
||||
|
||||
| Method | Returns | Description |
|
||||
| :------ | :------- | :----------- |
|
||||
| `degree_centrality(labels, rel_type, direction)` | `List[dict]` | Degree-based node importance — records ordered by degree DESC |
|
||||
| `degree_centrality(labels, rel_type, direction)` | `List[dict]` | Degree-based node importance: records ordered by degree DESC |
|
||||
| `connected_components(labels)` | `List[dict]` | Connected component assignment (requires GDS on Neo4j) |
|
||||
| `shortest_path(start_node_id, end_node_id, rel_type, max_depth)` | `dict \| None` | Path with `length`, `nodes`, `relationships` or `None` |
|
||||
| `get_neighbors(node_id, rel_type, direction, depth)` | `List[dict]` | Neighbor nodes up to `depth` hops |
|
||||
@@ -414,7 +414,7 @@ neighbors = analytics.get_neighbors(
|
||||
## Schema Management
|
||||
|
||||
```python
|
||||
# Index for fast label-property lookups — use property_name= not property=
|
||||
# Index for fast label-property lookups: use property_name= not property=
|
||||
store.create_index(label="Person", property_name="name")
|
||||
store.create_index(label="Organization", property_name="id")
|
||||
|
||||
@@ -481,7 +481,7 @@ stats = store.get_stats()
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Apache AGE notes">
|
||||
AGE supports one primary label per vertex. If you pass multiple labels, the first is used as the AGE label and the rest are stored in a `labels` property array. Parameterized queries use literal inlining internally (AGE does not support `$param` binding inside `cypher()` calls) — the store handles escaping automatically.
|
||||
AGE supports one primary label per vertex. If you pass multiple labels, the first is used as the AGE label and the rest are stored in a `labels` property array. Parameterized queries use literal inlining internally (AGE does not support `$param` binding inside `cypher()` calls): the store handles escaping automatically.
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
@@ -504,7 +504,7 @@ stats = store.get_stats()
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
**`create_index` parameter is `property_name=`, not `property=`.** `store.create_index(label="Person", property_name="name")` — using `property=` will be silently ignored.
|
||||
**`create_index` parameter is `property_name=`, not `property=`.** `store.create_index(label="Person", property_name="name")`: using `property=` will be silently ignored.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
|
||||
+18
-18
@@ -9,7 +9,7 @@ icon: "database"
|
||||
- 15+ ingestion adapters: files, web, SQL, Snowflake, Kafka, MCP, Git repos, email
|
||||
- PyArrow Parquet with column selection and partitioned dataset support
|
||||
- XXE-safe lxml XML with optional XSD schema validation
|
||||
- `ingest()` unified dispatcher — auto-detects source type from path or URL
|
||||
- `ingest()` unified dispatcher: auto-detects source type from path or URL
|
||||
- Each ingestor returns its own typed object (`FileObject`, `WebContent`, `TableData`, etc.)
|
||||
|
||||
|
||||
@@ -17,26 +17,26 @@ icon: "database"
|
||||
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `FileIngestor` | PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, ZIP/TAR — type auto-detected from extension |
|
||||
| `FileIngestor` | PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, ZIP/TAR: type auto-detected from extension |
|
||||
| `CloudStorageIngestor` | Unified client for AWS S3, Google Cloud Storage, and Azure Blob Storage |
|
||||
| `WebIngestor` | Web scraping and crawling with `ingest_url`, `crawl_sitemap`, `crawl_domain` |
|
||||
| `RESTIngestor` | Generic REST API ingestion with headers, params, retries, and pagination |
|
||||
| `PublicAPIIngestor` | No-auth public API ingestion with pre-configured examples and rate limiting |
|
||||
| `FeedIngestor` | RSS/Atom feed ingestion with live monitoring via `FeedMonitor` |
|
||||
| `StreamIngestor` | Real-time ingestion from Kafka, RabbitMQ, AWS Kinesis, and Apache Pulsar |
|
||||
| `RepoIngestor` | Git repositories — source files, commit history, and metadata |
|
||||
| `DBIngestor` | SQL databases via SQLAlchemy — tables, views, and custom queries |
|
||||
| `RepoIngestor` | Git repositories: source files, commit history, and metadata |
|
||||
| `DBIngestor` | SQL databases via SQLAlchemy: tables, views, and custom queries |
|
||||
| `SnowflakeIngestor` | Snowflake data warehouse queries and table exports |
|
||||
| `ParquetIngestor` | Apache Parquet files and partitioned datasets with column selection |
|
||||
| `XMLIngestor` | XXE-safe XML parsing with optional XSD schema validation |
|
||||
| `EmailIngestor` | IMAP/POP3 email ingestion with attachment extraction |
|
||||
| `OntologyIngestor` | OWL/RDF/Turtle ontology file ingestion |
|
||||
| `MCPIngestor` | Model Context Protocol (MCP) resource ingestion |
|
||||
| `ingest()` | Unified dispatcher — detects source type automatically from path or URL |
|
||||
| `ingest()` | Unified dispatcher: detects source type automatically from path or URL |
|
||||
|
||||
## Getting Started
|
||||
|
||||
Use **`FileIngestor`** for local files — it **auto-detects format** from the file extension and handles archives:
|
||||
Use **`FileIngestor`** for local files: it **auto-detects format** from the file extension and handles archives:
|
||||
|
||||
```python
|
||||
from semantica.ingest import FileIngestor
|
||||
@@ -64,13 +64,13 @@ from semantica.ingest import WebIngestor
|
||||
wc = WebIngestor(delay=1.0, respect_robots=True).ingest_url("https://example.com")
|
||||
print(wc.title, wc.text)
|
||||
|
||||
# Database — constructor takes no required args; pass connection_string to methods
|
||||
# Database: constructor takes no required args; pass connection_string to methods
|
||||
from semantica.ingest import DBIngestor
|
||||
db = DBIngestor()
|
||||
result = db.ingest_database("postgresql://user:pass@localhost/db")
|
||||
# result["tables"]["documents"]["rows"] contains the rows
|
||||
|
||||
# Unified dispatcher — auto-detects source type
|
||||
# Unified dispatcher: auto-detects source type
|
||||
from semantica.ingest import ingest
|
||||
result = ingest("data/report.pdf") # -> {"files": [FileObject]}
|
||||
result = ingest("https://example.com") # -> {"content": WebContent}
|
||||
@@ -87,7 +87,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
|
||||
|
||||
ingestor = FileIngestor()
|
||||
|
||||
# Single file — type auto-detected from extension
|
||||
# Single file: type auto-detected from extension
|
||||
file_obj = ingestor.ingest_file("data/report.pdf")
|
||||
|
||||
# Recursive directory scan
|
||||
@@ -157,7 +157,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
|
||||
# Single file
|
||||
file_obj = ingestor.ingest_file("data/report.pdf")
|
||||
|
||||
# Directory — returns List[FileObject]
|
||||
# Directory: returns List[FileObject]
|
||||
files = ingestor.ingest_directory("data/", recursive=True)
|
||||
|
||||
# ingest() dispatches to ingest_file or ingest_directory automatically
|
||||
@@ -185,7 +185,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
|
||||
# Partitioned directory (year=2024/month=01/...)
|
||||
data = ingestor.ingest_directory("data/partitioned/")
|
||||
|
||||
# Load only specific columns — pass as kwarg
|
||||
# Load only specific columns: pass as kwarg
|
||||
from semantica.ingest import ingest_parquet
|
||||
data = ingest_parquet("data/events.parquet", columns=["id", "text", "timestamp"])
|
||||
|
||||
@@ -206,7 +206,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
|
||||
ingestor = XMLIngestor()
|
||||
data = ingestor.ingest_file("data/records.xml")
|
||||
|
||||
# With XSD validation — pass schema_path as kwarg
|
||||
# With XSD validation: pass schema_path as kwarg
|
||||
from semantica.ingest import ingest_xml
|
||||
data = ingest_xml("data/records.xml", schema_path="schema.xsd")
|
||||
|
||||
@@ -295,7 +295,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
|
||||
|
||||
### RepoIngestor
|
||||
|
||||
Ingest Git repositories — source code, commit history, and dependency graphs:
|
||||
Ingest Git repositories: source code, commit history, and dependency graphs:
|
||||
|
||||
```python
|
||||
from semantica.ingest import RepoIngestor
|
||||
@@ -347,7 +347,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
|
||||
from semantica.ingest import CloudStorageIngestor
|
||||
import os
|
||||
|
||||
# AWS S3 — list and download objects
|
||||
# AWS S3: list and download objects
|
||||
ingestor = CloudStorageIngestor(
|
||||
provider="s3",
|
||||
access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
|
||||
@@ -424,7 +424,7 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
|
||||
<Tab title="Stream">
|
||||
### StreamIngestor
|
||||
|
||||
Real-time ingestion from message brokers — each method returns a typed processor:
|
||||
Real-time ingestion from message brokers: each method returns a typed processor:
|
||||
|
||||
```python
|
||||
from semantica.ingest import StreamIngestor
|
||||
@@ -604,7 +604,7 @@ result = ingest_file("source_path", method="my_format")
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**`DBIngestor()` takes no connection string in its constructor.** Pass the connection string to `ingest_database()`, `execute_query()`, or `export_table()` as the first positional argument — not to `DBIngestor()` itself.
|
||||
**`DBIngestor()` takes no connection string in its constructor.** Pass the connection string to `ingest_database()`, `execute_query()`, or `export_table()` as the first positional argument: not to `DBIngestor()` itself.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
@@ -612,11 +612,11 @@ result = ingest_file("source_path", method="my_format")
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
**Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns — critical for wide tables with hundreds of columns.
|
||||
**Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns: critical for wide tables with hundreds of columns.
|
||||
</Tip>
|
||||
|
||||
<Warning>
|
||||
**`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica — it does not block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML.
|
||||
**`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica: it does not block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
|
||||
+23
-23
@@ -17,11 +17,11 @@ icon: "diagram-project"
|
||||
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `KnowledgeGraph` | Core graph data structure — nodes, edges, properties, temporal validity |
|
||||
| `KnowledgeGraph` | Core graph data structure: nodes, edges, properties, temporal validity |
|
||||
| `GraphBuilder` | Construct from entities + relationships; pass `merge_entities=True` to enable deduplication |
|
||||
| `GraphBuilderWithProvenance` | Wraps `GraphBuilder` with optional provenance tracking; pass `provenance=True` to enable |
|
||||
| `EntityResolver` | Entity deduplication and merging during graph construction |
|
||||
| `GraphAnalyzer` | Unified analytics wrapper — runs centrality, community detection, and connectivity in one call |
|
||||
| `GraphAnalyzer` | Unified analytics wrapper: runs centrality, community detection, and connectivity in one call |
|
||||
| `ConnectivityAnalyzer` | Connected component detection, bridge identification, density, and degree statistics |
|
||||
| `TemporalGraphQuery` | Point-in-time snapshots, temporal diffs, and all 13 Allen interval queries |
|
||||
| `CentralityCalculator` | PageRank, degree, betweenness, closeness, eigenvector centrality |
|
||||
@@ -41,7 +41,7 @@ icon: "diagram-project"
|
||||
|
||||
## GraphBuilder
|
||||
|
||||
**`GraphBuilder`** constructs knowledge graphs from extracted entities and relationships. `merge_entities` defaults to `False` — pass **`True`** to enable entity deduplication during construction:
|
||||
**`GraphBuilder`** constructs knowledge graphs from extracted entities and relationships. `merge_entities` defaults to `False`: pass **`True`** to enable entity deduplication during construction:
|
||||
|
||||
```python
|
||||
from semantica.kg import GraphBuilder
|
||||
@@ -83,7 +83,7 @@ kg = builder.build(sources=[
|
||||
}
|
||||
])
|
||||
|
||||
# Point-in-time snapshot — TemporalGraphQuery takes no positional graph arg;
|
||||
# Point-in-time snapshot: TemporalGraphQuery takes no positional graph arg;
|
||||
# pass the graph into each query method instead.
|
||||
query = TemporalGraphQuery()
|
||||
snapshot_2021 = query.reconstruct_at_time(kg, "2021-06-15")
|
||||
@@ -93,7 +93,7 @@ snapshot_2023 = query.reconstruct_at_time(kg, "2023-01-01")
|
||||
range_result = query.query_time_range(kg, "", "2020-01-01", "2023-01-01")
|
||||
print(f"Relationships in range: {range_result['num_relationships']}")
|
||||
|
||||
# Versioned snapshots — author and description are required
|
||||
# Versioned snapshots: author and description are required
|
||||
versioner = TemporalVersionManager()
|
||||
versioner.create_snapshot(kg, version_label="2024-Q1",
|
||||
author="user@example.com",
|
||||
@@ -125,7 +125,7 @@ calc = SimilarityCalculator()
|
||||
score = calc.cosine_similarity(embeddings["Apple Inc."], embeddings["Google"])
|
||||
print(f"Apple–Google structural similarity: {score:.3f}")
|
||||
|
||||
# Find structurally similar nodes — returns List[str] of node IDs
|
||||
# Find structurally similar nodes: returns List[str] of node IDs
|
||||
similar = embedder.find_similar_nodes(kg, "Apple Inc.", top_k=5)
|
||||
for node_id in similar:
|
||||
print(node_id)
|
||||
@@ -171,10 +171,10 @@ for node_id in similar:
|
||||
|
||||
detector = CommunityDetector()
|
||||
|
||||
# Louvain — fast, high quality (default)
|
||||
# Louvain: fast, high quality (default)
|
||||
communities = detector.detect_communities(graph, algorithm="louvain")
|
||||
|
||||
# Leiden — higher quality, slower
|
||||
# Leiden: higher quality, slower
|
||||
communities = detector.detect_communities_leiden(graph, resolution=1.2)
|
||||
|
||||
# Evaluate community quality
|
||||
@@ -185,10 +185,10 @@ for node_id in similar:
|
||||
|
||||
| Algorithm | Strength |
|
||||
| :--------- | :-------- |
|
||||
| Louvain | Fast, good modularity — use for large graphs |
|
||||
| Leiden | Best modularity — use when quality matters more than speed |
|
||||
| Label Propagation | Near-linear time — use for very large graphs |
|
||||
| K-Clique | Overlapping communities — nodes can belong to multiple groups |
|
||||
| Louvain | Fast, good modularity: use for large graphs |
|
||||
| Leiden | Best modularity: use when quality matters more than speed |
|
||||
| Label Propagation | Near-linear time: use for very large graphs |
|
||||
| K-Clique | Overlapping communities: nodes can belong to multiple groups |
|
||||
</Tab>
|
||||
<Tab title="Path Finding">
|
||||
Find shortest paths and route alternatives between any two nodes.
|
||||
@@ -211,9 +211,9 @@ for node_id in similar:
|
||||
|
||||
| Algorithm | Use case |
|
||||
| :--------- | :-------- |
|
||||
| Dijkstra | Weighted shortest path — standard routing |
|
||||
| A\* | Heuristic-guided search — faster on large sparse graphs |
|
||||
| BFS | Unweighted shortest path — hop count only |
|
||||
| Dijkstra | Weighted shortest path: standard routing |
|
||||
| A\* | Heuristic-guided search: faster on large sparse graphs |
|
||||
| BFS | Unweighted shortest path: hop count only |
|
||||
| K-Shortest | Multiple alternative routes |
|
||||
</Tab>
|
||||
<Tab title="Link Prediction">
|
||||
@@ -239,7 +239,7 @@ for node_id in similar:
|
||||
| Common Neighbors | Nodes with shared connections |
|
||||
| Jaccard | Normalized common-neighbor overlap |
|
||||
| Adamic-Adar | Weighted common neighbors (penalizes hubs) |
|
||||
| Resource Allocation | Conservative — ignores high-degree intermediaries |
|
||||
| Resource Allocation | Conservative: ignores high-degree intermediaries |
|
||||
</Tab>
|
||||
<Tab title="Node Embeddings">
|
||||
Compute structural embeddings with Node2Vec, then find similar nodes or feed into downstream ML.
|
||||
@@ -265,7 +265,7 @@ for node_id in similar:
|
||||
```
|
||||
|
||||
<Note>
|
||||
`find_similar_nodes` returns `List[str]` — a list of node IDs, not node objects. Look up full node data via `graph["nodes"]`.
|
||||
`find_similar_nodes` returns `List[str]`: a list of node IDs, not node objects. Look up full node data via `graph["nodes"]`.
|
||||
</Note>
|
||||
</Tab>
|
||||
</Tabs>
|
||||
@@ -286,7 +286,7 @@ for node_id in similar:
|
||||
|
||||
## GraphValidator
|
||||
|
||||
Validates graph structure — checks required fields, duplicate IDs, dangling edges, and optionally detects cycles and orphan nodes:
|
||||
Validates graph structure: checks required fields, duplicate IDs, dangling edges, and optionally detects cycles and orphan nodes:
|
||||
|
||||
```python
|
||||
from semantica.kg import GraphValidator
|
||||
@@ -333,8 +333,8 @@ kg:
|
||||
|
||||
### Cookbooks
|
||||
|
||||
- [Building Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) — fundamentals of KG construction · Beginner
|
||||
- [Your First Knowledge Graph](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb) — entity extraction to visualization · Beginner
|
||||
- [Graph Analytics](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/10_Graph_Analytics.ipynb) — centrality and community detection · Intermediate
|
||||
- [Advanced Graph Analytics](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/02_Advanced_Graph_Analytics.ipynb) — PageRank, Louvain, shortest path · Advanced
|
||||
- [Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb) — temporal logic and graph evolution · Advanced
|
||||
- [Building Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb): fundamentals of KG construction · Beginner
|
||||
- [Your First Knowledge Graph](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/08_Your_First_Knowledge_Graph.ipynb): entity extraction to visualization · Beginner
|
||||
- [Graph Analytics](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/10_Graph_Analytics.ipynb): centrality and community detection · Intermediate
|
||||
- [Advanced Graph Analytics](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/02_Advanced_Graph_Analytics.ipynb): PageRank, Louvain, shortest path · Advanced
|
||||
- [Temporal Knowledge Graphs](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb): temporal logic and graph evolution · Advanced
|
||||
|
||||
+17
-17
@@ -8,7 +8,7 @@ icon: "microchip"
|
||||
|
||||
- Every provider is a drop-in replacement for the `llm_provider=` parameter in extractors, reasoners, and agents
|
||||
- `LiteLLM` routes to 100+ providers with a single class and model-string prefixes
|
||||
- `HuggingFaceLLM` runs fully on-premise — no API key, no network calls
|
||||
- `HuggingFaceLLM` runs fully on-premise: no API key, no network calls
|
||||
- Structured output via `generate_with_schema()` for JSON extraction from any provider
|
||||
- Streaming, tool use, and `generate_batch()` for bulk inference
|
||||
|
||||
@@ -32,21 +32,21 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM
|
||||
|
||||
## What You Get
|
||||
|
||||
- **Unified `LLMProvider` interface** — swap providers with a one-line change, no application code changes
|
||||
- **`LiteLLM`** — single class for 100+ providers using model-string routing
|
||||
- **Local models** — `HuggingFaceLLM` runs fully on-premise, no API key
|
||||
- **Streaming** — token-by-token output for low-latency UX
|
||||
- **Custom gateways** — point `OpenAI` at any OpenAI-compatible endpoint via `base_url`
|
||||
- **Unified `LLMProvider` interface**: swap providers with a one-line change, no application code changes
|
||||
- **`LiteLLM`**: single class for 100+ providers using model-string routing
|
||||
- **Local models**: `HuggingFaceLLM` runs fully on-premise, no API key
|
||||
- **Streaming**: token-by-token output for low-latency UX
|
||||
- **Custom gateways**: point `OpenAI` at any OpenAI-compatible endpoint via `base_url`
|
||||
|
||||
## Choosing a Provider
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Groq — Getting Started">
|
||||
<Tab title="Groq: Getting Started">
|
||||
Free tier, fastest inference, zero setup friction. Best for development and high-throughput extraction pipelines.
|
||||
|
||||
| | |
|
||||
| :-- | :-- |
|
||||
| **Speed** | Very fast — 100+ tok/s |
|
||||
| **Speed** | Very fast: 100+ tok/s |
|
||||
| **Cost** | Free tier available |
|
||||
| **Context** | 128k |
|
||||
| **Best for** | Development, high-throughput extraction |
|
||||
@@ -64,7 +64,7 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM
|
||||
|
||||
Get your free key at [console.groq.com](https://console.groq.com).
|
||||
</Tab>
|
||||
<Tab title="OpenAI — Production">
|
||||
<Tab title="OpenAI: Production">
|
||||
Highest accuracy, best JSON mode and function calling. Use for production pipelines where extraction quality matters.
|
||||
|
||||
| | |
|
||||
@@ -86,8 +86,8 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM
|
||||
)
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Ollama — Local / Air-gapped">
|
||||
Fully on-premise — no API key, no data leaves your infrastructure. Required for air-gapped deployments.
|
||||
<Tab title="Ollama: Local / Air-gapped">
|
||||
Fully on-premise: no API key, no data leaves your infrastructure. Required for air-gapped deployments.
|
||||
|
||||
| | |
|
||||
| :-- | :-- |
|
||||
@@ -114,7 +114,7 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM
|
||||
No API key required. Ensure the Ollama server is running (`ollama serve`) before creating the `LiteLLM` instance.
|
||||
</Note>
|
||||
</Tab>
|
||||
<Tab title="Claude — Reasoning">
|
||||
<Tab title="Claude: Reasoning">
|
||||
Largest context window, best multi-hop reasoning, highest safety bar. Use for complex analysis and long-document extraction.
|
||||
|
||||
| | |
|
||||
@@ -135,7 +135,7 @@ from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM
|
||||
)
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="DeepSeek — Cost Optimization">
|
||||
<Tab title="DeepSeek: Cost Optimization">
|
||||
Lowest cost per token for high-volume workloads. Strong on coding and structured data extraction.
|
||||
|
||||
| | |
|
||||
@@ -257,7 +257,7 @@ llm = LiteLLM(model="anthropic/claude-opus-4-5", api_key=os.getenv("ANTH
|
||||
# Google Gemini
|
||||
llm = LiteLLM(model="gemini/gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY"))
|
||||
|
||||
# Ollama (local — no API key)
|
||||
# Ollama (local: no API key)
|
||||
llm = LiteLLM(model="ollama/llama3.2:3b", api_base="http://localhost:11434")
|
||||
|
||||
# DeepSeek
|
||||
@@ -282,12 +282,12 @@ llm = HuggingFaceLLM(
|
||||
max_new_tokens=512,
|
||||
temperature=0.1,
|
||||
)
|
||||
# Bring your own model — full local control, no API key
|
||||
# Bring your own model: full local control, no API key
|
||||
```
|
||||
|
||||
</CodeGroup>
|
||||
|
||||
## LiteLLM — 100+ Providers
|
||||
## LiteLLM: 100+ Providers
|
||||
|
||||
`LiteLLM` is the recommended way to access any provider not directly exported by `semantica.llms`. Use the `provider/model` string format:
|
||||
|
||||
@@ -317,7 +317,7 @@ response = providers["Anthropic"].generate("Explain GraphRAG in one paragraph.")
|
||||
|
||||
## Custom / Enterprise Gateways
|
||||
|
||||
Any OpenAI-compatible endpoint — internal routing layers, Qwen proxies, or private LLaMA deployments:
|
||||
Any OpenAI-compatible endpoint: internal routing layers, Qwen proxies, or private LLaMA deployments:
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
---
|
||||
title: "MCP Server"
|
||||
description: "Model Context Protocol server — expose Semantica's full capability set to Claude Desktop, VS Code, Cursor, and any MCP-aware tool."
|
||||
description: "Model Context Protocol server: expose Semantica's full capability set to Claude Desktop, VS Code, Cursor, and any MCP-aware tool."
|
||||
icon: "plug"
|
||||
---
|
||||
|
||||
**`semantica.mcp_server`** exposes Semantica's knowledge graph, decision intelligence, semantic extraction, and reasoning capabilities as an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) **server over stdio**:
|
||||
|
||||
- 12 MCP tools exposed: extract entities, query graph, record decisions, run reasoning, export results
|
||||
- No Python code required after launch — configure once, use from any MCP-aware client
|
||||
- No Python code required after launch: configure once, use from any MCP-aware client
|
||||
- Compatible with Claude Desktop, Windsurf, Cline, Continue, VS Code, Roo Code, Cursor
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ python -m semantica.mcp_server
|
||||
```
|
||||
|
||||
<Tip>
|
||||
`semantica.mcp_server` is a **stdio server process**, not a Python library. It exposes no importable classes — all interaction happens through MCP tool calls from a connected AI client.
|
||||
`semantica.mcp_server` is a **stdio server process**, not a Python library. It exposes no importable classes: all interaction happens through MCP tool calls from a connected AI client.
|
||||
</Tip>
|
||||
|
||||
## What You Get
|
||||
@@ -42,10 +42,10 @@ python -m semantica.mcp_server
|
||||
Extract entities, extract relations, record decisions, query decisions, find precedents, trace causal chains, add entities, add relationships, run analytics, summarise graph, run reasoning, export graph.
|
||||
</Card>
|
||||
<Card title="3 Readable Resources" icon="book-open">
|
||||
Live graph JSON (`semantica://graph/summary`), decision list, and schema/version info — readable by any MCP client.
|
||||
Live graph JSON (`semantica://graph/summary`), decision list, and schema/version info: readable by any MCP client.
|
||||
</Card>
|
||||
<Card title="Zero Infrastructure" icon="bolt">
|
||||
Runs over stdio — no server, no port, no Docker required. One config block to activate in any MCP client.
|
||||
Runs over stdio: no server, no port, no Docker required. One config block to activate in any MCP client.
|
||||
</Card>
|
||||
<Card title="Persistent Graphs" icon="database">
|
||||
Point `SEMANTICA_KG_PATH` at a saved graph file to reload it automatically on every server startup.
|
||||
@@ -64,7 +64,7 @@ python -m semantica.mcp_server
|
||||
pip install semantica
|
||||
```
|
||||
|
||||
The MCP server is included in the base install — no extras required.
|
||||
The MCP server is included in the base install: no extras required.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -153,7 +153,7 @@ The MCP server is included in the base install — no extras required.
|
||||
|
||||
| Variable | Default | Description |
|
||||
| :-------- | :------- | :----------- |
|
||||
| `SEMANTICA_KG_PATH` | *(none — in-memory graph)* | Path to a persisted graph file to load on startup |
|
||||
| `SEMANTICA_KG_PATH` | *(none: in-memory graph)* | Path to a persisted graph file to load on startup |
|
||||
| `SEMANTICA_LOG_LEVEL` | `WARNING` | Log verbosity: `DEBUG`, `INFO`, `WARNING` |
|
||||
|
||||
## Tools
|
||||
@@ -162,7 +162,7 @@ The MCP server exposes 12 tools that any connected AI assistant can call:
|
||||
|
||||
| Tool | Category | Description |
|
||||
| :---- | :-------- | :----------- |
|
||||
| `extract_entities` | Extraction | NER — find people, places, organisations, concepts |
|
||||
| `extract_entities` | Extraction | NER: find people, places, organisations, concepts |
|
||||
| `extract_relations` | Extraction | Typed relation and triplet extraction |
|
||||
| `record_decision` | Decision Intelligence | Save a decision with reasoning and outcome |
|
||||
| `query_decisions` | Decision Intelligence | Search recorded decisions by natural language or category |
|
||||
@@ -173,7 +173,7 @@ The MCP server exposes 12 tools that any connected AI assistant can call:
|
||||
| `get_graph_summary` | Graph Operations | Node count, decision count, graph status |
|
||||
| `get_graph_analytics` | Graph Operations | PageRank centrality and community detection |
|
||||
| `run_reasoning` | Reasoning | Forward-chain IF/THEN rules over facts |
|
||||
| `export_graph` | Reasoning & Export | Serialise the graph (`turtle`/`ttl` — RDF Turtle aliases, `nt`, `xml`, `json-ld`, `json`) |
|
||||
| `export_graph` | Reasoning & Export | Serialise the graph (`turtle`/`ttl`: RDF Turtle aliases, `nt`, `xml`, `json-ld`, `json`) |
|
||||
|
||||
### Knowledge Extraction
|
||||
|
||||
@@ -447,15 +447,15 @@ The MCP server exposes three readable resources:
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path — it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs.
|
||||
**Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path: it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs.
|
||||
</Tip>
|
||||
|
||||
<Warning>
|
||||
**Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently — the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works.
|
||||
**Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently: the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
**The server communicates over stdio — don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. All logging is written to `stderr` only. Configure log verbosity with the `SEMANTICA_LOG_LEVEL` environment variable.
|
||||
**The server communicates over stdio: don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. All logging is written to `stderr` only. Configure log verbosity with the `SEMANTICA_LOG_LEVEL` environment variable.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
|
||||
+28
-28
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Normalize Module"
|
||||
description: "Text cleaning, entity canonicalization, date normalization, number conversion, language detection, and encoding repair — before extraction runs."
|
||||
description: "Text cleaning, entity canonicalization, date normalization, number conversion, language detection, and encoding repair: before extraction runs."
|
||||
icon: "broom"
|
||||
---
|
||||
|
||||
@@ -19,10 +19,10 @@ All normalizers expose convenience functions (one-liners) and stateful class ins
|
||||
|
||||
Unstructured data is inconsistent by nature. Without normalization, the same real-world entity appears as dozens of variants in your graph:
|
||||
|
||||
- `"Apple Inc."`, `"Apple Computer Inc."`, `"APPLE INC."` — multiple nodes, one company
|
||||
- `"Jan 1st, 2020"`, `"01/01/2020"`, `"2020-01-01"` — three formats, one date
|
||||
- `"$1.2B"`, `"1,200,000,000"`, `"1.2 billion USD"` — three strings, one number
|
||||
- `"Hello World"` vs `"Hello\u00a0World"` — a non-breaking space that breaks string matching
|
||||
- `"Apple Inc."`, `"Apple Computer Inc."`, `"APPLE INC."`: multiple nodes, one company
|
||||
- `"Jan 1st, 2020"`, `"01/01/2020"`, `"2020-01-01"`: three formats, one date
|
||||
- `"$1.2B"`, `"1,200,000,000"`, `"1.2 billion USD"`: three strings, one number
|
||||
- `"Hello World"` vs `"Hello\u00a0World"`: a non-breaking space that breaks string matching
|
||||
|
||||
Normalization collapses these variants before any extractor, deduplicator, or graph builder sees the data.
|
||||
|
||||
@@ -49,7 +49,7 @@ from semantica.normalize import (
|
||||
EncodingHandler,
|
||||
)
|
||||
|
||||
# Text — normalize unicode, collapse whitespace, replace smart quotes
|
||||
# Text: normalize unicode, collapse whitespace, replace smart quotes
|
||||
normalizer = TextNormalizer()
|
||||
clean = normalizer.normalize_text(" Hello,\u00a0 World\u2026 ")
|
||||
# → "Hello, World..."
|
||||
@@ -64,12 +64,12 @@ num_norm = NumberNormalizer()
|
||||
num = num_norm.normalize_number("$1.2B")
|
||||
# → 1200000000.0
|
||||
|
||||
# Language — returns a language code string
|
||||
# Language: returns a language code string
|
||||
detector = LanguageDetector()
|
||||
lang = detector.detect("Bonjour le monde")
|
||||
# → "fr"
|
||||
|
||||
# Encoding — returns (encoding_name, confidence) tuple
|
||||
# Encoding: returns (encoding_name, confidence) tuple
|
||||
handler = EncodingHandler()
|
||||
encoding, confidence = handler.detect(raw_bytes)
|
||||
utf8_text = handler.convert_to_utf8(raw_bytes)
|
||||
@@ -78,7 +78,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes)
|
||||
## Recommended Processing Order
|
||||
|
||||
<Steps>
|
||||
<Step title="EncodingHandler — fix encoding first">
|
||||
<Step title="EncodingHandler: fix encoding first">
|
||||
Broken bytes corrupt everything downstream. Always run this before anything else.
|
||||
|
||||
```python
|
||||
@@ -91,7 +91,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes)
|
||||
utf8_text = handler.convert_to_utf8(raw_bytes)
|
||||
```
|
||||
</Step>
|
||||
<Step title="TextNormalizer — unicode, whitespace, special chars">
|
||||
<Step title="TextNormalizer: unicode, whitespace, special chars">
|
||||
```python
|
||||
from semantica.normalize import TextNormalizer
|
||||
|
||||
@@ -104,7 +104,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes)
|
||||
)
|
||||
```
|
||||
</Step>
|
||||
<Step title="EntityNormalizer — canonicalize entity names">
|
||||
<Step title="EntityNormalizer: canonicalize entity names">
|
||||
```python
|
||||
from semantica.normalize import EntityNormalizer
|
||||
|
||||
@@ -119,7 +119,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes)
|
||||
# → "Apple Inc." (if the alias_map contains it, else title-cased input)
|
||||
```
|
||||
</Step>
|
||||
<Step title="DateNormalizer and NumberNormalizer — parse structured values">
|
||||
<Step title="DateNormalizer and NumberNormalizer: parse structured values">
|
||||
```python
|
||||
from semantica.normalize import DateNormalizer, NumberNormalizer
|
||||
|
||||
@@ -134,7 +134,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes)
|
||||
# → 1200000000.0
|
||||
```
|
||||
</Step>
|
||||
<Step title="LanguageDetector — detect language on clean text">
|
||||
<Step title="LanguageDetector: detect language on clean text">
|
||||
```python
|
||||
from semantica.normalize import LanguageDetector
|
||||
|
||||
@@ -153,7 +153,7 @@ utf8_text = handler.convert_to_utf8(raw_bytes)
|
||||
|
||||
## Convenience Functions
|
||||
|
||||
The fastest path — one import, one call:
|
||||
The fastest path: one import, one call:
|
||||
|
||||
```python
|
||||
from semantica.normalize import (
|
||||
@@ -205,7 +205,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
|
||||
line_break_type="unix", # "unix" | "windows"
|
||||
)
|
||||
|
||||
# HTML stripping and text cleaning — separate clean_text() method
|
||||
# HTML stripping and text cleaning: separate clean_text() method
|
||||
cleaned = normalizer.clean_text(html_text, remove_html=True)
|
||||
|
||||
# Batch normalization
|
||||
@@ -231,9 +231,9 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
|
||||
|
||||
| Form | Use When |
|
||||
| :---- | :-------- |
|
||||
| `NFC` | Default — best for storage and display |
|
||||
| `NFKC` | Search indexing — normalises ligatures, fullwidth chars, and fractions |
|
||||
| `NFD` | Stripping diacritics — split é → e + combining accent, then strip accents |
|
||||
| `NFC` | Default: best for storage and display |
|
||||
| `NFKC` | Search indexing: normalises ligatures, fullwidth chars, and fractions |
|
||||
| `NFD` | Stripping diacritics: split é → e + combining accent, then strip accents |
|
||||
| `NFKD` | Same as NFD but also decomposes compatibility characters |
|
||||
|
||||
**Sub-normalizers for fine-grained control:**
|
||||
@@ -262,7 +262,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
|
||||
```python
|
||||
from semantica.normalize import EntityNormalizer
|
||||
|
||||
# With alias map — resolves exact matches (lowercase key lookup)
|
||||
# With alias map: resolves exact matches (lowercase key lookup)
|
||||
normalizer = EntityNormalizer(alias_map={
|
||||
"apple computer inc.": "Apple Inc.",
|
||||
"ms": "Microsoft",
|
||||
@@ -272,21 +272,21 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
|
||||
normalizer.normalize_entity("Apple Computer Inc.", entity_type="Organization")
|
||||
# → "Apple Inc."
|
||||
|
||||
# Without alias map — only whitespace/format cleanup
|
||||
# Without alias map: only whitespace/format cleanup
|
||||
normalizer2 = EntityNormalizer()
|
||||
normalizer2.normalize_entity("apple inc", entity_type="Organization")
|
||||
# → "apple inc" (no built-in suffix expansion)
|
||||
|
||||
# Person — title-cased
|
||||
# Person: title-cased
|
||||
normalizer2.normalize_entity("john doe", entity_type="Person")
|
||||
# → "John Doe"
|
||||
```
|
||||
|
||||
**Key behaviours:**
|
||||
- Alias map uses **lowercase key lookup** — register aliases in lowercase
|
||||
- Alias map uses **lowercase key lookup**: register aliases in lowercase
|
||||
- `entity_type="Person"` activates `title()` casing on the name
|
||||
- There is no built-in corporate suffix normalization (Inc → Incorporated etc.)
|
||||
— add these mappings to `alias_map` manually if needed
|
||||
: add these mappings to `alias_map` manually if needed
|
||||
|
||||
**Sub-normalizers:**
|
||||
|
||||
@@ -350,7 +350,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
|
||||
utc_dt = tz_norm.convert_to_utc(dt_naive)
|
||||
tz_dt = tz_norm.normalize_timezone(dt_naive, target_timezone="America/New_York")
|
||||
|
||||
# RelativeDateProcessor — reference_date is passed to process_relative_expression(),
|
||||
# RelativeDateProcessor: reference_date is passed to process_relative_expression(),
|
||||
# not to the constructor
|
||||
processor = RelativeDateProcessor()
|
||||
ref = datetime(2025, 1, 15)
|
||||
@@ -416,7 +416,7 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
|
||||
results = detector.detect_multiple("This might be mixed", top_n=3)
|
||||
# → [("en", 0.85), ...]
|
||||
|
||||
# Batch — returns List[str]
|
||||
# Batch: returns List[str]
|
||||
codes = detector.detect_batch(["Hello", "Hola", "Bonjour", "Ciao"])
|
||||
|
||||
# Check specific language
|
||||
@@ -452,17 +452,17 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
|
||||
```
|
||||
|
||||
**Key behaviours:**
|
||||
- `detect()` uses `chardet` internally — accuracy improves with longer input
|
||||
- `detect()` uses `chardet` internally: accuracy improves with longer input
|
||||
- `convert_to_utf8()` auto-detects encoding if `source_encoding` is not provided,
|
||||
then falls back through `latin-1`, `cp1252`, `iso-8859-1`
|
||||
- Always run `EncodingHandler` first — broken bytes cause cascading failures
|
||||
- Always run `EncodingHandler` first: broken bytes cause cascading failures
|
||||
in every downstream normalizer
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## DataCleaner
|
||||
|
||||
Cleans structured record sets — useful before loading into a vector store or graph:
|
||||
Cleans structured record sets: useful before loading into a vector store or graph:
|
||||
|
||||
```python
|
||||
from semantica.normalize import DataCleaner, DataValidator, DuplicateDetector
|
||||
|
||||
@@ -21,7 +21,7 @@ icon: "sitemap"
|
||||
| `OntologyGenerator` | Auto-generate ontologies from KG data (5-stage pipeline) |
|
||||
| `LLMOntologyGenerator` | LLM-powered ontology generation for complex domains |
|
||||
| `SHACLGenerator` | Generate SHACL shapes from an ontology or KG schema |
|
||||
| `OntologyValidator` | Validate any graph against SHACL shapes — returns `SHACLValidationReport` |
|
||||
| `OntologyValidator` | Validate any graph against SHACL shapes: returns `SHACLValidationReport` |
|
||||
| `OWLGenerator` | Serialize ontologies to Turtle, RDF/XML, JSON-LD |
|
||||
| `NamespaceManager` | IRI generation, prefix management, and namespace binding |
|
||||
| `OntologyEvaluator` | Coverage, completeness, and granularity quality metrics |
|
||||
@@ -55,7 +55,7 @@ engine.export_owl(ontology, "ontology.ttl", format="turtle")
|
||||
|
||||
## OntologyEngine (Unified Facade)
|
||||
|
||||
**`OntologyEngine`** orchestrates the full ontology lifecycle — **generation, validation, export, and evaluation**:
|
||||
**`OntologyEngine`** orchestrates the full ontology lifecycle: **generation, validation, export, and evaluation**:
|
||||
|
||||
```python
|
||||
from semantica.ontology import OntologyEngine
|
||||
@@ -109,7 +109,7 @@ ontology = generator.generate_ontology({
|
||||
Map definitions to OWL constructs: `owl:Class`, `owl:ObjectProperty`, `owl:DatatypeProperty`.
|
||||
</Step>
|
||||
<Step title="Hierarchy Generation">
|
||||
Build taxonomy trees using transitive closure and cycle detection — produces `rdfs:subClassOf` chains.
|
||||
Build taxonomy trees using transitive closure and cycle detection: produces `rdfs:subClassOf` chains.
|
||||
</Step>
|
||||
<Step title="TTL Generation">
|
||||
Serialize the final ontology to Turtle format using `rdflib`. Also available: RDF/XML and JSON-LD.
|
||||
|
||||
+10
-10
@@ -1,13 +1,13 @@
|
||||
---
|
||||
title: "Parse Module"
|
||||
description: "Document parsing and text extraction — DocumentParser for standard formats and DoclingParser for complex layouts."
|
||||
description: "Document parsing and text extraction: DocumentParser for standard formats and DoclingParser for complex layouts."
|
||||
icon: "file-lines"
|
||||
---
|
||||
|
||||
**`semantica.parse`** extracts **structured text, layout, tables, and metadata** from unstructured documents:
|
||||
|
||||
- `DocumentParser` — broad format support (PDF, DOCX, HTML, JSON, CSV, PPTX, XLSX), no extra dependencies
|
||||
- `DoclingParser` — complex layouts, merged-cell tables, multi-column PDFs, OCR (`pip install docling`)
|
||||
- `DocumentParser`: broad format support (PDF, DOCX, HTML, JSON, CSV, PPTX, XLSX), no extra dependencies
|
||||
- `DoclingParser`: complex layouts, merged-cell tables, multi-column PDFs, OCR (`pip install docling`)
|
||||
- Both return a consistent `dict` with `full_text`, `metadata`, `pages`, and `tables` keys
|
||||
- `parse_batch()` processes multiple files in parallel with configurable error handling
|
||||
|
||||
@@ -60,14 +60,14 @@ print(f"Extracted {len(text)} characters from {metadata.get('page_count', 0)} pa
|
||||
## Parser Selection Guide
|
||||
|
||||
<Tabs>
|
||||
<Tab title="DocumentParser — Standard">
|
||||
<Tab title="DocumentParser: Standard">
|
||||
Zero extra dependencies. Use for clean PDFs, Word docs, HTML, and structured formats.
|
||||
|
||||
| | |
|
||||
| :-- | :-- |
|
||||
| **Formats** | PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX |
|
||||
| **Speed** | Fast |
|
||||
| **Setup** | None — included in base install |
|
||||
| **Setup** | None: included in base install |
|
||||
| **Best for** | Clean documents, broad format support, production pipelines |
|
||||
|
||||
```python
|
||||
@@ -81,7 +81,7 @@ print(f"Extracted {len(text)} characters from {metadata.get('page_count', 0)} pa
|
||||
print(len(result.get("pages", []))) # Per-page breakdown
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="DoclingParser — Complex Layouts">
|
||||
<Tab title="DoclingParser: Complex Layouts">
|
||||
Superior table extraction, OCR, multi-column PDFs. Requires `pip install docling`.
|
||||
|
||||
| | |
|
||||
@@ -130,7 +130,7 @@ print(f"Extracted {len(text)} characters from {metadata.get('page_count', 0)} pa
|
||||
print(f"{item['file_path']}: {len(item['result']['full_text'])} chars")
|
||||
|
||||
for item in results["failed"]:
|
||||
print(f"FAILED: {item['file_path']} — {item['error']}")
|
||||
print(f"FAILED: {item['file_path']}: {item['error']}")
|
||||
```
|
||||
|
||||
<Note>
|
||||
@@ -143,7 +143,7 @@ print(f"Extracted {len(text)} characters from {metadata.get('page_count', 0)} pa
|
||||
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `DocumentParser` | Auto-detects format — delegates to format-specific parser (PDF, DOCX, HTML, JSON, CSV, ...) |
|
||||
| `DocumentParser` | Auto-detects format: delegates to format-specific parser (PDF, DOCX, HTML, JSON, CSV, ...) |
|
||||
| `DoclingParser` | Complex layouts, merged-cell tables, multi-column PDFs, and OCR (`pip install docling`) |
|
||||
| `DoclingMetadata` | Document metadata from Docling parsing |
|
||||
| `PDFParser` | PDF text and metadata extraction |
|
||||
@@ -171,7 +171,7 @@ Supported formats: PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX.
|
||||
|
||||
## DoclingParser
|
||||
|
||||
Advanced parser using the Docling backend — handles layouts that `DocumentParser` cannot:
|
||||
Advanced parser using the Docling backend: handles layouts that `DocumentParser` cannot:
|
||||
|
||||
```bash
|
||||
pip install docling
|
||||
@@ -275,7 +275,7 @@ metadata = {
|
||||
|
||||
## Integration with FileIngestor
|
||||
|
||||
The most common pattern — ingest a directory then parse each source:
|
||||
The most common pattern: ingest a directory then parse each source:
|
||||
|
||||
```python
|
||||
from semantica.ingest import FileIngestor
|
||||
|
||||
+20
-20
@@ -7,10 +7,10 @@ icon: "gear"
|
||||
**`semantica.pipeline`** lets you chain Semantica components into **reproducible, fault-tolerant workflows**:
|
||||
|
||||
- Per-step failure strategies: `skip`, `retry`, `abort`, or `fallback`
|
||||
- Parallel workers via `ParallelismManager` — thread or process pool
|
||||
- Parallel workers via `ParallelismManager`: thread or process pool
|
||||
- `PipelineValidator` catches cycles, missing handlers, and config errors before running
|
||||
- Pre-built templates: `"document_processing"`, `"rag_pipeline"`, `"kg_construction"`, `"ontology_generation"`
|
||||
- Pipelines are serializable to YAML — save and reload in any environment
|
||||
- Pipelines are serializable to YAML: save and reload in any environment
|
||||
|
||||
|
||||
## Exported Classes
|
||||
@@ -19,7 +19,7 @@ icon: "gear"
|
||||
| :--- | :--- |
|
||||
| `PipelineBuilder` | DSL for wiring steps: `add_step`, `connect_steps`, `set_parallel`, `build` |
|
||||
| `ExecutionEngine` | Runs a built pipeline: `execute_pipeline(pipeline, data)` → `ExecutionResult` |
|
||||
| `ExecutionResult` | `{success, output, metadata, metrics, errors}` — full run summary |
|
||||
| `ExecutionResult` | `{success, output, metadata, metrics, errors}`: full run summary |
|
||||
| `FailureHandler` | Per-step strategy: `skip`, `retry`, `abort`, or `fallback` on failure |
|
||||
| `ParallelismManager` | Thread or process pool for concurrent step execution with configurable workers |
|
||||
| `PipelineValidator` | Catches dependency cycles, missing handlers, and config errors before running |
|
||||
@@ -154,7 +154,7 @@ result = engine.execute_pipeline(pipeline, data="data/")
|
||||
result = engine.execute_pipeline(pipeline, data="data/")
|
||||
```
|
||||
|
||||
Best for transient API errors and rate limits — waits longer with each retry, giving upstream services time to recover.
|
||||
Best for transient API errors and rate limits: waits longer with each retry, giving upstream services time to recover.
|
||||
</Tab>
|
||||
<Tab title="Linear backoff">
|
||||
```python
|
||||
@@ -167,7 +167,7 @@ result = engine.execute_pipeline(pipeline, data="data/")
|
||||
)
|
||||
```
|
||||
|
||||
Use when the delay between retries should grow predictably — e.g., waiting for a database lock to release.
|
||||
Use when the delay between retries should grow predictably: e.g., waiting for a database lock to release.
|
||||
</Tab>
|
||||
<Tab title="Fixed backoff">
|
||||
```python
|
||||
@@ -188,8 +188,8 @@ result = engine.execute_pipeline(pipeline, data="data/")
|
||||
|
||||
| Strategy | Behaviour | When to Use |
|
||||
| :-------- | :--------- | :----------- |
|
||||
| `"skip"` | Log failure, continue to next document | Production — one bad doc shouldn't stop 10k |
|
||||
| `"stop"` | Raise exception immediately | Development — surface errors fast |
|
||||
| `"skip"` | Log failure, continue to next document | Production: one bad doc shouldn't stop 10k |
|
||||
| `"stop"` | Raise exception immediately | Development: surface errors fast |
|
||||
| `"retry"` | Retry via `RetryPolicy`, then skip | When failures are likely transient |
|
||||
|
||||
<Warning>
|
||||
@@ -227,7 +227,7 @@ result = engine.execute_pipeline(pipeline, data="data/")
|
||||
while t.is_alive():
|
||||
progress = engine.get_progress(pipeline.name)
|
||||
if progress:
|
||||
print(f" {progress['completed_steps']}/{progress['total_steps']} steps — {progress['status']}")
|
||||
print(f" {progress['completed_steps']}/{progress['total_steps']} steps: {progress['status']}")
|
||||
time.sleep(2)
|
||||
```
|
||||
|
||||
@@ -244,7 +244,7 @@ from semantica.pipeline import PipelineBuilder, ExecutionEngine
|
||||
|
||||
builder = PipelineBuilder()
|
||||
|
||||
# Add steps — step_type is a string label, handler is the callable invoked at runtime
|
||||
# Add steps: step_type is a string label, handler is the callable invoked at runtime
|
||||
builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file)
|
||||
builder.add_step("parse", "document_parse", handler=parser.parse)
|
||||
builder.add_step("normalize", "text_normalize", handler=normalizer.normalize)
|
||||
@@ -291,12 +291,12 @@ result = ExecutionEngine().execute_pipeline(restored, data="data/")
|
||||
```
|
||||
|
||||
<Tip>
|
||||
Serialized pipelines capture step names, types, and config — but not handler functions (callables can't be serialized). Re-register handlers on the restored steps before executing.
|
||||
Serialized pipelines capture step names, types, and config: but not handler functions (callables can't be serialized). Re-register handlers on the restored steps before executing.
|
||||
</Tip>
|
||||
|
||||
## Pre-Built Templates
|
||||
|
||||
`PipelineTemplateManager` wires common workflows with the correct step order — no manual wiring required:
|
||||
`PipelineTemplateManager` wires common workflows with the correct step order: no manual wiring required:
|
||||
|
||||
```python
|
||||
from semantica.pipeline import PipelineTemplateManager
|
||||
@@ -320,7 +320,7 @@ The `create_pipeline_from_template(name)` method returns a configured `PipelineB
|
||||
<Card title="rag_pipeline" icon="magnifying-glass">
|
||||
**Ingest → Chunk → Embed → Store Vectors**
|
||||
|
||||
RAG pipeline for question answering — builds a vector-indexed store.
|
||||
RAG pipeline for question answering: builds a vector-indexed store.
|
||||
|
||||
```python
|
||||
builder = manager.create_pipeline_from_template("rag_pipeline")
|
||||
@@ -351,7 +351,7 @@ The `create_pipeline_from_template(name)` method returns a configured `PipelineB
|
||||
|
||||
## ExecutionEngine
|
||||
|
||||
Fine-grained control over pipeline execution — pause, resume, cancel, and inspect live progress:
|
||||
Fine-grained control over pipeline execution: pause, resume, cancel, and inspect live progress:
|
||||
|
||||
```python
|
||||
from semantica.pipeline import ExecutionEngine
|
||||
@@ -394,7 +394,7 @@ validator = PipelineValidator()
|
||||
result = validator.validate_pipeline(pipeline)
|
||||
|
||||
if result.valid:
|
||||
print("Pipeline is valid — safe to run")
|
||||
print("Pipeline is valid: safe to run")
|
||||
else:
|
||||
for error in result.errors: # errors is List[str]
|
||||
print(f"Error: {error}")
|
||||
@@ -403,10 +403,10 @@ else:
|
||||
```
|
||||
|
||||
Checks performed:
|
||||
- **Dependency cycle detection** — A depends on B, B depends on A
|
||||
- **Step type validation** — each step type must be registered
|
||||
- **Connection integrity** — referenced step names must exist
|
||||
- **Configuration completeness** — required parameters must be present
|
||||
- **Dependency cycle detection**: A depends on B, B depends on A
|
||||
- **Step type validation**: each step type must be registered
|
||||
- **Connection integrity**: referenced step names must exist
|
||||
- **Configuration completeness**: required parameters must be present
|
||||
|
||||
## ParallelismManager
|
||||
|
||||
@@ -556,7 +556,7 @@ from semantica.pipeline import StepStatus
|
||||
StepStatus.PENDING # Not yet started
|
||||
StepStatus.RUNNING # Currently executing
|
||||
StepStatus.COMPLETED # Finished successfully
|
||||
StepStatus.FAILED # Error occurred — check step.error
|
||||
StepStatus.FAILED # Error occurred: check step.error
|
||||
StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy
|
||||
```
|
||||
|
||||
@@ -578,7 +578,7 @@ StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Use templates from `PipelineTemplateManager` for common patterns.** `create_pipeline_from_template("kg_construction")` wires normalization, deduplication, conflict detection, and graph construction in the correct order — saving you from common mistakes like deduplicating before normalizing.
|
||||
**Use templates from `PipelineTemplateManager` for common patterns.** `create_pipeline_from_template("kg_construction")` wires normalization, deduplication, conflict detection, and graph construction in the correct order: saving you from common mistakes like deduplicating before normalizing.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
|
||||
@@ -4,9 +4,9 @@ description: "W3C PROV-O lineage tracking, source attribution, tamper-evident ch
|
||||
icon: "link"
|
||||
---
|
||||
|
||||
`semantica.provenance` tracks the full lineage of every fact — from raw ingestion through extraction, chunking, and relationship building:
|
||||
`semantica.provenance` tracks the full lineage of every fact: from raw ingestion through extraction, chunking, and relationship building:
|
||||
|
||||
- W3C PROV-O compliant — suitable for HIPAA, SOX, GDPR, FDA 21 CFR Part 11 audit trails
|
||||
- W3C PROV-O compliant: suitable for HIPAA, SOX, GDPR, FDA 21 CFR Part 11 audit trails
|
||||
- SHA-256 checksums for tamper detection on every stored `ProvenanceEntry`
|
||||
- `SQLiteStorage` for persistence across restarts; `InMemoryStorage` for development
|
||||
- `ProvenanceManager` provides `track_entity`, `track_relationship`, `track_chunk`, and `get_lineage`
|
||||
@@ -21,8 +21,8 @@ icon: "link"
|
||||
| `ProvenanceEntry` | Single lineage record: `{entity_id, entity_type, activity_id, source_document, confidence, checksum, ...}` |
|
||||
| `SourceReference` | Rich source pointer: `{document, page, section, line, confidence, metadata}` |
|
||||
| `ProvenanceStorage` | Abstract storage interface |
|
||||
| `InMemoryStorage` | Default backend — fast, not persisted across restarts |
|
||||
| `SQLiteStorage` | Persistent backend — persists to a local SQLite file |
|
||||
| `InMemoryStorage` | Default backend: fast, not persisted across restarts |
|
||||
| `SQLiteStorage` | Persistent backend: persists to a local SQLite file |
|
||||
| `compute_checksum` | Returns SHA-256 fingerprint of a `ProvenanceEntry` |
|
||||
| `verify_checksum` | Detects tampering by comparing stored vs recomputed hash |
|
||||
|
||||
@@ -30,7 +30,7 @@ icon: "link"
|
||||
|
||||
<Tabs>
|
||||
<Tab title="In-Memory (default)">
|
||||
Zero configuration — fast, no disk writes. Use for notebooks, testing, and single-run scripts.
|
||||
Zero configuration: fast, no disk writes. Use for notebooks, testing, and single-run scripts.
|
||||
|
||||
```python
|
||||
from semantica.provenance import ProvenanceManager, compute_checksum, verify_checksum
|
||||
@@ -46,7 +46,7 @@ icon: "link"
|
||||
)
|
||||
|
||||
print(entry.checksum) # SHA-256 hex auto-computed
|
||||
print(verify_checksum(entry)) # True — tamper detection
|
||||
print(verify_checksum(entry)) # True: tamper detection
|
||||
```
|
||||
|
||||
<Note>
|
||||
@@ -62,7 +62,7 @@ icon: "link"
|
||||
# Option 1: explicit storage instance
|
||||
manager = ProvenanceManager(storage=SQLiteStorage("provenance.db"))
|
||||
|
||||
# Option 2: shorthand — equivalent to above
|
||||
# Option 2: shorthand: equivalent to above
|
||||
manager = ProvenanceManager(storage_path="provenance.db")
|
||||
|
||||
entry = manager.track_entity(
|
||||
@@ -72,7 +72,7 @@ icon: "link"
|
||||
confidence=0.98,
|
||||
)
|
||||
|
||||
# Retrieve lineage after restart — entries persist in provenance.db
|
||||
# Retrieve lineage after restart: entries persist in provenance.db
|
||||
lineage = manager.get_lineage("apple_inc")
|
||||
print(f"{len(lineage)} provenance entries for apple_inc")
|
||||
```
|
||||
@@ -92,7 +92,7 @@ icon: "link"
|
||||
```python
|
||||
ProvenanceManager(
|
||||
storage=None, # ProvenanceStorage instance; defaults to InMemoryStorage
|
||||
storage_path=None, # str path — creates SQLiteStorage if provided
|
||||
storage_path=None, # str path: creates SQLiteStorage if provided
|
||||
)
|
||||
```
|
||||
|
||||
@@ -169,7 +169,7 @@ count = manager.track_chunks_batch(chunks, source_document="doc_1")
|
||||
### Retrieving Lineage
|
||||
|
||||
```python
|
||||
# get_lineage returns a dict — not a ProvenanceEntry
|
||||
# get_lineage returns a dict: not a ProvenanceEntry
|
||||
lineage = manager.get_lineage("apple_inc")
|
||||
|
||||
print(lineage["entity_id"]) # "apple_inc"
|
||||
@@ -233,23 +233,23 @@ from semantica.provenance import ProvenanceEntry
|
||||
|
||||
# All fields with their types and defaults
|
||||
entry = ProvenanceEntry(
|
||||
entity_id="entity_001", # str — required
|
||||
entity_type="entity", # str — required (entity, chunk, relationship, property)
|
||||
activity_id="ner_extraction", # str — required
|
||||
agent_id="semantica", # str — default "semantica"
|
||||
source_document="report.pdf", # str — default ""
|
||||
source_location="Page 4", # Optional[str] — default None
|
||||
source_quote="Relevant text...", # Optional[str] — default None
|
||||
timestamp="2024-01-01T12:00:00", # str — auto-set to utcnow()
|
||||
first_seen=None, # Optional[str] — ISO timestamp
|
||||
last_updated=None, # Optional[str] — ISO timestamp
|
||||
confidence=0.9, # float — default 1.0
|
||||
checksum=None, # Optional[str] — set by compute_checksum()
|
||||
parent_entity_id=None, # Optional[str] — prov:wasDerivedFrom
|
||||
used_entities=[], # List[str] — prov:used
|
||||
start_index=None, # Optional[int] — for chunks
|
||||
end_index=None, # Optional[int] — for chunks
|
||||
credibility=None, # Optional[float] — source credibility
|
||||
entity_id="entity_001", # str: required
|
||||
entity_type="entity", # str: required (entity, chunk, relationship, property)
|
||||
activity_id="ner_extraction", # str: required
|
||||
agent_id="semantica", # str: default "semantica"
|
||||
source_document="report.pdf", # str: default ""
|
||||
source_location="Page 4", # Optional[str]: default None
|
||||
source_quote="Relevant text...", # Optional[str]: default None
|
||||
timestamp="2024-01-01T12:00:00", # str: auto-set to utcnow()
|
||||
first_seen=None, # Optional[str]: ISO timestamp
|
||||
last_updated=None, # Optional[str]: ISO timestamp
|
||||
confidence=0.9, # float: default 1.0
|
||||
checksum=None, # Optional[str]: set by compute_checksum()
|
||||
parent_entity_id=None, # Optional[str]: prov:wasDerivedFrom
|
||||
used_entities=[], # List[str]: prov:used
|
||||
start_index=None, # Optional[int]: for chunks
|
||||
end_index=None, # Optional[int]: for chunks
|
||||
credibility=None, # Optional[float]: source credibility
|
||||
metadata={}, # Dict[str, Any]
|
||||
version="1.0", # str
|
||||
)
|
||||
@@ -269,12 +269,12 @@ entry2 = ProvenanceEntry.from_dict(d)
|
||||
from semantica.provenance import SourceReference
|
||||
|
||||
ref = SourceReference(
|
||||
document="DOI:10.1038/s41586-021-03371-z", # str — required (DOI, URL, file path)
|
||||
document="DOI:10.1038/s41586-021-03371-z", # str: required (DOI, URL, file path)
|
||||
page=4, # Optional[int]
|
||||
section="Table S4", # Optional[str]
|
||||
line=None, # Optional[int]
|
||||
timestamp=None, # Optional[datetime]
|
||||
confidence=0.92, # float — default 1.0
|
||||
confidence=0.92, # float: default 1.0
|
||||
metadata={"credibility": "peer-reviewed"}, # Dict[str, Any]
|
||||
)
|
||||
|
||||
@@ -387,7 +387,7 @@ prov_manager = ProvenanceManager(storage=SQLiteStorage("provenance.db"))
|
||||
builder = GraphBuilderWithProvenance(provenance_manager=prov_manager)
|
||||
kg = builder.build_single_source(graph_data)
|
||||
|
||||
# Retrieve lineage — get_lineage returns a dict
|
||||
# Retrieve lineage: get_lineage returns a dict
|
||||
lineage = prov_manager.get_lineage("apple_inc")
|
||||
print(lineage["source_documents"]) # list of source document IDs
|
||||
print(lineage["first_seen"]) # ISO timestamp
|
||||
|
||||
+28
-28
@@ -7,7 +7,7 @@ icon: "microchip"
|
||||
`semantica.reasoning` derives new knowledge from existing facts using logical rules:
|
||||
|
||||
- Six reasoning engines: forward chaining, Rete, SPARQL, Datalog, temporal, and LLM-powered GraphReasoner
|
||||
- Every engine produces explainable inference paths — traceable chains of rules and facts
|
||||
- Every engine produces explainable inference paths: traceable chains of rules and facts
|
||||
- `DatalogReasoner` guarantees termination via semi-naive fixpoint evaluation
|
||||
- `TemporalReasoningEngine` implements all 13 Allen interval algebra relations
|
||||
- `ExplanationGenerator` produces step-by-step natural-language justifications
|
||||
@@ -18,11 +18,11 @@ icon: "microchip"
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `Reasoner` | Forward-chaining inference: `add_fact`, `add_rule`, `forward_chain`, `backward_chain`, `infer_facts` |
|
||||
| `GraphReasoner` | LLM-powered reasoning over a KG dict — answers natural language queries via `reason(graph, query)` |
|
||||
| `GraphReasoner` | LLM-powered reasoning over a KG dict: answers natural language queries via `reason(graph, query)` |
|
||||
| `ReteEngine` | Rete pattern matching: `build_network`, `add_fact`, `match_patterns`, `execute_matches` |
|
||||
| `SPARQLReasoner` | Rule-based SPARQL query expansion via `execute_query`, `expand_query`, `infer_results` |
|
||||
| `DatalogReasoner` | Recursive Horn clause rules with semi-naive fixpoint — `add_fact`, `add_rule`, `derive_all`, `query` |
|
||||
| `TemporalReasoningEngine` | All 13 Allen interval algebra relations — `relation(a, b)`, `overlaps`, `contains`, `active_at` |
|
||||
| `DatalogReasoner` | Recursive Horn clause rules with semi-naive fixpoint: `add_fact`, `add_rule`, `derive_all`, `query` |
|
||||
| `TemporalReasoningEngine` | All 13 Allen interval algebra relations: `relation(a, b)`, `overlaps`, `contains`, `active_at` |
|
||||
| `ExplanationGenerator` | Step-by-step explanations via `generate_explanation(inference_result)` |
|
||||
| `Rule` | IF/THEN rule: `{rule_id, name, conditions, conclusion, rule_type, confidence, priority}` |
|
||||
| `Fact` | Working-memory fact: `{fact_id, predicate, arguments}` |
|
||||
@@ -33,10 +33,10 @@ icon: "microchip"
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Reasoner" icon="arrow-right-arrow-left" href="#reasoner-forwardbackward-chaining">
|
||||
IF/THEN rules, forward and backward chaining. **Start here** — covers 90% of use cases. No query language required.
|
||||
IF/THEN rules, forward and backward chaining. **Start here**: covers 90% of use cases. No query language required.
|
||||
</Card>
|
||||
<Card title="GraphReasoner" icon="robot" href="#graphreasoner">
|
||||
Natural language queries over a knowledge graph via LLM. No SPARQL or rules — just ask a question.
|
||||
Natural language queries over a knowledge graph via LLM. No SPARQL or rules: just ask a question.
|
||||
</Card>
|
||||
<Card title="DatalogReasoner" icon="code" href="#datalogreasoner">
|
||||
Recursive Horn clause rules with guaranteed termination. Use for complex multi-hop transitive rules.
|
||||
@@ -69,7 +69,7 @@ reasoner.add_fact("Employee(Alice)")
|
||||
# Add an IF-THEN rule using the string form
|
||||
reasoner.add_rule("IF Manager(?x) THEN HasAuthority(?x)")
|
||||
|
||||
# Run forward chaining — returns List[InferenceResult]
|
||||
# Run forward chaining: returns List[InferenceResult]
|
||||
results = reasoner.forward_chain()
|
||||
for r in results:
|
||||
print(r.conclusion) # "HasAuthority(Alice)"
|
||||
@@ -97,7 +97,7 @@ reasoner.add_rule(rule)
|
||||
|
||||
## Reasoner (Forward/Backward Chaining)
|
||||
|
||||
**`Reasoner`** is the unified entry point for rule-based inference — iterates facts and rules to a **fixpoint**, then optionally proves a specific goal via backward chaining:
|
||||
**`Reasoner`** is the unified entry point for rule-based inference: iterates facts and rules to a **fixpoint**, then optionally proves a specific goal via backward chaining:
|
||||
|
||||
```python
|
||||
from semantica.reasoning import Reasoner, Rule, RuleType, InferenceResult
|
||||
@@ -111,14 +111,14 @@ reasoner.add_fact("Employee(John)")
|
||||
# IF-THEN string form
|
||||
reasoner.add_rule("IF Manager(?x) AND Employee(?x) THEN SeniorStaff(?x)")
|
||||
|
||||
# Forward chaining — iterates until fixpoint
|
||||
# Forward chaining: iterates until fixpoint
|
||||
results = reasoner.forward_chain()
|
||||
for r in results:
|
||||
print(r.conclusion) # e.g. "SeniorStaff(John)"
|
||||
print(r.premises) # list of premise strings matched
|
||||
print(r.confidence) # float
|
||||
|
||||
# Backward chaining — prove a specific goal
|
||||
# Backward chaining: prove a specific goal
|
||||
result = reasoner.backward_chain("SeniorStaff(John)", max_depth=10)
|
||||
if result:
|
||||
print(f"Proven: {result.conclusion}")
|
||||
@@ -149,7 +149,7 @@ conclusions = reasoner.infer_facts(
|
||||
```python
|
||||
from semantica.reasoning import Rule, Fact, RuleType
|
||||
|
||||
# Rule — all fields
|
||||
# Rule: all fields
|
||||
rule = Rule(
|
||||
rule_id="rule_001", # required: unique identifier
|
||||
name="manager_authority", # required: display name
|
||||
@@ -160,7 +160,7 @@ rule = Rule(
|
||||
priority=0, # higher priority rules run first
|
||||
)
|
||||
|
||||
# Fact — for working with the Rete engine directly
|
||||
# Fact: for working with the Rete engine directly
|
||||
from semantica.reasoning import Fact
|
||||
fact = Fact(
|
||||
fact_id="f001", # required: unique identifier
|
||||
@@ -173,12 +173,12 @@ fact = Fact(
|
||||
|
||||
## GraphReasoner
|
||||
|
||||
**`GraphReasoner`** uses an LLM to answer **natural language queries** over a knowledge graph dict — no SPARQL or rule authoring required:
|
||||
**`GraphReasoner`** uses an LLM to answer **natural language queries** over a knowledge graph dict: no SPARQL or rule authoring required:
|
||||
|
||||
```python
|
||||
from semantica.reasoning import GraphReasoner
|
||||
|
||||
# Initialize — uses openai by default; override via kwargs
|
||||
# Initialize: uses openai by default; override via kwargs
|
||||
reasoner = GraphReasoner(provider="openai", model="gpt-4o-mini")
|
||||
|
||||
kg = {
|
||||
@@ -250,7 +250,7 @@ engine.reset()
|
||||
|
||||
## SPARQLReasoner
|
||||
|
||||
**`SPARQLReasoner`** extends SPARQL with **inference rule expansion** — add IF-THEN rules and they are automatically woven into queries before execution:
|
||||
**`SPARQLReasoner`** extends SPARQL with **inference rule expansion**: add IF-THEN rules and they are automatically woven into queries before execution:
|
||||
|
||||
```python
|
||||
from semantica.reasoning import SPARQLReasoner
|
||||
@@ -260,7 +260,7 @@ reasoner = SPARQLReasoner()
|
||||
# Add an inference rule (IF-THEN string form)
|
||||
reasoner.add_inference_rule("IF is_a(?x, Manager) THEN has_authority(?x)")
|
||||
|
||||
# Execute a query — returns SPARQLQueryResult
|
||||
# Execute a query: returns SPARQLQueryResult
|
||||
result = reasoner.execute_query("""
|
||||
PREFIX ex: <http://example.org/>
|
||||
SELECT ?person ?company WHERE {
|
||||
@@ -296,14 +296,14 @@ SPARQLReasoner(
|
||||
|
||||
## DatalogReasoner
|
||||
|
||||
Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed** — the engine detects fixpoint convergence and stops:
|
||||
Pure-Python bottom-up semi-naive fixpoint evaluation for recursive Horn clause rules. Termination is **guaranteed**: the engine detects fixpoint convergence and stops:
|
||||
|
||||
```python
|
||||
from semantica.reasoning import DatalogReasoner, DatalogFact
|
||||
|
||||
datalog = DatalogReasoner()
|
||||
|
||||
# Add base facts — string form is the simplest
|
||||
# Add base facts: string form is the simplest
|
||||
datalog.add_fact("parent(alice, bob)")
|
||||
datalog.add_fact("parent(bob, charlie)")
|
||||
|
||||
@@ -314,11 +314,11 @@ datalog.add_fact(DatalogFact(predicate="parent", args=("charlie", "dave")))
|
||||
datalog.add_rule("ancestor(X, Y) :- parent(X, Y).")
|
||||
datalog.add_rule("ancestor(X, Z) :- parent(X, Y), ancestor(Y, Z).")
|
||||
|
||||
# Evaluate to fixpoint — returns all derived fact strings
|
||||
# Evaluate to fixpoint: returns all derived fact strings
|
||||
all_facts = datalog.derive_all()
|
||||
# e.g. ["parent(alice, bob)", "parent(bob, charlie)", ..., "ancestor(alice, bob)", ...]
|
||||
|
||||
# Query with variable pattern — variables start with uppercase or ?
|
||||
# Query with variable pattern: variables start with uppercase or ?
|
||||
results = datalog.query("ancestor(alice, ?Z)")
|
||||
# → [{"Z": "bob"}, {"Z": "charlie"}, {"Z": "dave"}]
|
||||
|
||||
@@ -331,11 +331,11 @@ datalog.clear()
|
||||
```python
|
||||
from semantica.reasoning import DatalogFact, DatalogRule
|
||||
|
||||
# DatalogFact — ground fact; args must all be constants (lowercase start)
|
||||
# DatalogFact: ground fact; args must all be constants (lowercase start)
|
||||
fact = DatalogFact(predicate="parent", args=("alice", "bob"))
|
||||
|
||||
# DatalogRule — parsed from string; head and body are set by the parser
|
||||
# Use add_rule("head(X, Y) :- body(X, Z), body2(Z, Y).") — do not construct directly
|
||||
# DatalogRule: parsed from string; head and body are set by the parser
|
||||
# Use add_rule("head(X, Y) :- body(X, Z), body2(Z, Y)."): do not construct directly
|
||||
```
|
||||
|
||||
### DatalogReasoner Methods
|
||||
@@ -345,14 +345,14 @@ fact = DatalogFact(predicate="parent", args=("alice", "bob"))
|
||||
| `add_fact(fact)` | `None` | Add string, dict, or `DatalogFact`; constants must be lowercase-starting |
|
||||
| `add_rule(rule_str)` | `None` | Parse and add a Horn clause string like `"ancestor(X,Y) :- parent(X,Y)."` |
|
||||
| `derive_all()` | `List[str]` | Run semi-naive fixpoint evaluation; returns all facts as strings |
|
||||
| `query(pattern)` | `List[dict]` | Query derived facts — auto-runs `derive_all()` if needed |
|
||||
| `query(pattern)` | `List[dict]` | Query derived facts: auto-runs `derive_all()` if needed |
|
||||
| `load_from_graph(graph)` | `int` | Load a ContextGraph's nodes/edges as Datalog facts; returns count added |
|
||||
| `clear()` | `None` | Clear all facts and rules |
|
||||
|
||||
|
||||
## TemporalReasoningEngine
|
||||
|
||||
Pure-Python Allen interval algebra — all 13 relations, no LLM calls:
|
||||
Pure-Python Allen interval algebra: all 13 relations, no LLM calls:
|
||||
|
||||
```python
|
||||
from datetime import datetime
|
||||
@@ -363,7 +363,7 @@ engine = TemporalReasoningEngine()
|
||||
ceo_tenure = TemporalInterval(start=datetime(1997, 9, 16), end=datetime(2011, 8, 24))
|
||||
board_member = TemporalInterval(start=datetime(2000, 1, 1), end=datetime(2012, 6, 1))
|
||||
|
||||
# Compute Allen relation — method is relation(), not get_relation()
|
||||
# Compute Allen relation: method is relation(), not get_relation()
|
||||
rel = engine.relation(ceo_tenure, board_member)
|
||||
# → IntervalRelation.DURING (ceo_tenure is fully inside board_member)
|
||||
|
||||
@@ -413,7 +413,7 @@ results = reasoner.forward_chain()
|
||||
# ExplanationGenerator takes no positional args
|
||||
generator = ExplanationGenerator()
|
||||
|
||||
# Pass an InferenceResult object — not a dict
|
||||
# Pass an InferenceResult object: not a dict
|
||||
explanation = generator.generate_explanation(results[0])
|
||||
|
||||
print(f"Type: {explanation.explanation_type}") # "inference"
|
||||
@@ -472,7 +472,7 @@ step.confidence # float
|
||||
| `TemporalReasoningEngine` | Time interval relationships | Always | `relation(a, b)` |
|
||||
|
||||
<Tip>
|
||||
For recursive rules (e.g. ancestor, reachability, transitivity), use `DatalogReasoner` — it guarantees termination via semi-naive bottom-up fixpoint evaluation. `Reasoner.forward_chain()` has a `max_iterations` cap (default 50) and will silently stop early with deep recursion.
|
||||
For recursive rules (e.g. ancestor, reachability, transitivity), use `DatalogReasoner`: it guarantees termination via semi-naive bottom-up fixpoint evaluation. `Reasoner.forward_chain()` has a `max_iterations` cap (default 50) and will silently stop early with deep recursion.
|
||||
</Tip>
|
||||
|
||||
<Warning>
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
---
|
||||
title: "Seed Module"
|
||||
description: "Bootstrap Knowledge Graphs from verified, structured sources — taxonomies, reference tables, product catalogs, and domain anchors."
|
||||
description: "Bootstrap Knowledge Graphs from verified, structured sources: taxonomies, reference tables, product catalogs, and domain anchors."
|
||||
icon: "database"
|
||||
---
|
||||
|
||||
**`semantica.seed`** gives your knowledge graph a **reliable, verified starting point**:
|
||||
|
||||
- Load verified reference data first — ISO codes, employee rosters, product catalogs, domain taxonomies
|
||||
- Load verified reference data first: ISO codes, employee rosters, product catalogs, domain taxonomies
|
||||
- `SeedDataManager` merges freshly extracted data onto foundation nodes without creating duplicates
|
||||
- Supports JSON, CSV, and programmatic registration of seed sources
|
||||
- Deterministic test graph generation from structured seed data
|
||||
@@ -80,7 +80,7 @@ icon: "database"
|
||||
for warning in report["warnings"]:
|
||||
print(f"Warning: {warning}")
|
||||
else:
|
||||
print(f"Validated {report['metrics']['entity_count']} entities — no issues found")
|
||||
print(f"Validated {report['metrics']['entity_count']} entities: no issues found")
|
||||
```
|
||||
</Step>
|
||||
<Step title="Merge with extracted data">
|
||||
@@ -190,7 +190,7 @@ Different strategies for resolving conflicts during `integrate_with_extracted()`
|
||||
|
||||
<Tabs>
|
||||
<Tab title="seed_first">
|
||||
**Seed data wins conflicts** — preserves curated relationships over extracted ones.
|
||||
**Seed data wins conflicts**: preserves curated relationships over extracted ones.
|
||||
|
||||
```python
|
||||
final_kg = manager.integrate_with_extracted(
|
||||
@@ -203,7 +203,7 @@ Different strategies for resolving conflicts during `integrate_with_extracted()`
|
||||
Use when seed data is high-confidence and extraction is exploratory.
|
||||
</Tab>
|
||||
<Tab title="extracted_first">
|
||||
**Extracted data wins conflicts** — overwrites seed with fresh information.
|
||||
**Extracted data wins conflicts**: overwrites seed with fresh information.
|
||||
|
||||
```python
|
||||
final_kg = manager.integrate_with_extracted(
|
||||
@@ -216,7 +216,7 @@ Different strategies for resolving conflicts during `integrate_with_extracted()`
|
||||
Use for rapid prototyping when extraction quality is known to be good.
|
||||
</Tab>
|
||||
<Tab title="merge">
|
||||
**Intelligent conflict resolution** — merges complementary attributes, deduplicates entities.
|
||||
**Intelligent conflict resolution**: merges complementary attributes, deduplicates entities.
|
||||
|
||||
```python
|
||||
final_kg = manager.integrate_with_extracted(
|
||||
@@ -282,7 +282,7 @@ manager.export_seed_data("output/enriched_kg.json", format="json")
|
||||
|
||||
## YAML Configuration
|
||||
|
||||
Define sources in YAML for production deployments — no code changes needed to switch environments:
|
||||
Define sources in YAML for production deployments: no code changes needed to switch environments:
|
||||
|
||||
```yaml
|
||||
seed:
|
||||
@@ -318,7 +318,7 @@ export SEMANTICA_SEED_MERGE_STRATEGY=seed_first
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**Load seed data before extracted data.** Seed data is your ground truth — normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values.
|
||||
**Load seed data before extracted data.** Seed data is your ground truth: normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
@@ -326,7 +326,7 @@ export SEMANTICA_SEED_MERGE_STRATEGY=seed_first
|
||||
</Tip>
|
||||
|
||||
<Warning>
|
||||
**Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast — always run it first.
|
||||
**Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast: always run it first.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
|
||||
@@ -4,12 +4,12 @@ description: "Named entity recognition, relation extraction, event detection, an
|
||||
icon: "magnifying-glass-chart"
|
||||
---
|
||||
|
||||
`semantica.semantic_extract` extracts structured information from unstructured text — the foundation of every knowledge graph in Semantica:
|
||||
`semantica.semantic_extract` extracts structured information from unstructured text: the foundation of every knowledge graph in Semantica:
|
||||
|
||||
- `NERExtractor` — named entity recognition with confidence scores and source attribution
|
||||
- `RelationExtractor` — typed relationship extraction (`founded_by`, `located_in`, and custom types)
|
||||
- `TripletExtractor` — direct `(subject, predicate, object)` triplet generation for RDF output
|
||||
- `EventDetector` — event detection with participants, temporal context, and confidence
|
||||
- `NERExtractor`: named entity recognition with confidence scores and source attribution
|
||||
- `RelationExtractor`: typed relationship extraction (`founded_by`, `located_in`, and custom types)
|
||||
- `TripletExtractor`: direct `(subject, predicate, object)` triplet generation for RDF output
|
||||
- `EventDetector`: event detection with participants, temporal context, and confidence
|
||||
- Three extraction modes on every extractor: `"pattern"` (no API key), `"huggingface"`, `"llm"`
|
||||
|
||||
|
||||
@@ -67,7 +67,7 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `NamedEntityRecognizer` | High-level NER with confidence thresholding and overlap merging |
|
||||
| `NERExtractor` | Core NER implementation — use directly for simplicity |
|
||||
| `NERExtractor` | Core NER implementation: use directly for simplicity |
|
||||
| `RelationExtractor` | Typed relationship extraction (`founded_by`, `located_in`, ...) |
|
||||
| `TripletExtractor` | Direct `(subject, predicate, object)` triplet generation for RDF output |
|
||||
| `EventDetector` | Event detection with participants, temporal context, and confidence scores |
|
||||
@@ -79,12 +79,12 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
|
||||
## Method Selection Guide
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Pattern — No Setup">
|
||||
<Tab title="Pattern: No Setup">
|
||||
Zero dependencies, no API key required. Uses spaCy rules and regex to match standard entity types.
|
||||
|
||||
| | |
|
||||
| :-- | :-- |
|
||||
| **Setup** | None — works out of the box |
|
||||
| **Setup** | None: works out of the box |
|
||||
| **Cost** | Free |
|
||||
| **Accuracy** | Good for standard entity types |
|
||||
| **Best for** | Quick prototyping, batch processing, air-gapped systems |
|
||||
@@ -99,7 +99,7 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
|
||||
relationships = rel.extract(text, entities=entities)
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="HuggingFace — Custom Models">
|
||||
<Tab title="HuggingFace: Custom Models">
|
||||
Use any pre-trained or fine-tuned transformer model. Free inference, runs locally.
|
||||
|
||||
| | |
|
||||
@@ -121,14 +121,14 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
|
||||
entities = ner.extract(text, model="d4data/biomedical-ner-all")
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="LLM — Best Accuracy">
|
||||
<Tab title="LLM: Best Accuracy">
|
||||
Highest accuracy for complex schemas and custom entity types. Requires an LLM API key.
|
||||
|
||||
| | |
|
||||
| :-- | :-- |
|
||||
| **Setup** | `pip install semantica[llm-groq]` + API key |
|
||||
| **Cost** | Depends on provider |
|
||||
| **Accuracy** | Highest — handles complex types and context |
|
||||
| **Accuracy** | Highest: handles complex types and context |
|
||||
| **Best for** | Production, custom entity types, complex relation schemas |
|
||||
|
||||
```python
|
||||
@@ -148,7 +148,7 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Fallback Chain">
|
||||
Try methods in priority order — guarantees non-empty results even when the preferred method is unavailable.
|
||||
Try methods in priority order: guarantees non-empty results even when the preferred method is unavailable.
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
@@ -157,7 +157,7 @@ entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
|
||||
ner = NERExtractor(method=["llm", "pattern"])
|
||||
rel = RelationExtractor(method=["llm", "pattern"])
|
||||
|
||||
# Always returns results — safe for production pipelines
|
||||
# Always returns results: safe for production pipelines
|
||||
entities = ner.extract(text)
|
||||
relationships = rel.extract(text, entities=entities)
|
||||
```
|
||||
@@ -225,15 +225,15 @@ from semantica.semantic_extract import NERExtractor
|
||||
from semantica.llms import Groq
|
||||
import os
|
||||
|
||||
# Pattern-based — fast, no API key, good for standard entity types
|
||||
# Pattern-based: fast, no API key, good for standard entity types
|
||||
ner = NERExtractor(method="pattern")
|
||||
entities = ner.extract("Apple Inc. was founded by Steve Jobs in Cupertino.")
|
||||
|
||||
# HuggingFace-based — custom models, no API cost
|
||||
# HuggingFace-based: custom models, no API cost
|
||||
ner = NERExtractor(method="huggingface")
|
||||
entities = ner.extract(text, model="dslim/bert-base-NER", device="cpu")
|
||||
|
||||
# LLM-based — best accuracy, handles complex schemas and custom types
|
||||
# LLM-based: best accuracy, handles complex schemas and custom types
|
||||
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
|
||||
ner = NERExtractor(method="llm", llm_provider=llm, max_retries=3)
|
||||
entities = ner.extract(text)
|
||||
@@ -285,11 +285,11 @@ Output format:
|
||||
|
||||
Available methods:
|
||||
|
||||
- `"pattern"` — rule-based pattern matching
|
||||
- `"dependency"` — spaCy dependency parsing
|
||||
- `"cooccurrence"` — proximity-based co-occurrence
|
||||
- `"huggingface"` — custom models
|
||||
- `"llm"` — highest accuracy, requires API key
|
||||
- `"pattern"`: rule-based pattern matching
|
||||
- `"dependency"`: spaCy dependency parsing
|
||||
- `"cooccurrence"`: proximity-based co-occurrence
|
||||
- `"huggingface"`: custom models
|
||||
- `"llm"`: highest accuracy, requires API key
|
||||
|
||||
|
||||
## TripletExtractor
|
||||
@@ -327,11 +327,11 @@ for event in events:
|
||||
|
||||
Output fields per event:
|
||||
|
||||
- `type` — event category (e.g. `"founding"`, `"acquisition"`)
|
||||
- `participants` — list of entities with roles
|
||||
- `temporal` — date or time reference
|
||||
- `location` — location entity (when present)
|
||||
- `confidence` — extraction confidence score
|
||||
- `type`: event category (e.g. `"founding"`, `"acquisition"`)
|
||||
- `participants`: list of entities with roles
|
||||
- `temporal`: date or time reference
|
||||
- `location`: location entity (when present)
|
||||
- `confidence`: extraction confidence score
|
||||
|
||||
|
||||
## CoreferenceResolver
|
||||
@@ -384,7 +384,7 @@ results = ner.extract(documents) # Entities include document_id in metadata
|
||||
|
||||
## Using All Extractors Together
|
||||
|
||||
The standard extraction pipeline — entities → relationships → triplets:
|
||||
The standard extraction pipeline: entities → relationships → triplets:
|
||||
|
||||
```python
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor
|
||||
|
||||
+29
-29
@@ -27,9 +27,9 @@ Semantica's chunking methods are designed to avoid these failure modes.
|
||||
|
||||
| Class | Role |
|
||||
| :--- | :--- |
|
||||
| `TextSplitter` | Unified entry point — swap `method=` without changing downstream code |
|
||||
| `TextSplitter` | Unified entry point: swap `method=` without changing downstream code |
|
||||
| `Chunk` | `{text, start_index, end_index, metadata, id}` |
|
||||
| `SemanticChunker` | Embedding-based topic-shift detection — splits only when content actually changes |
|
||||
| `SemanticChunker` | Embedding-based topic-shift detection: splits only when content actually changes |
|
||||
| `StructuralChunker` | Heading/section-based splits using structural text analysis |
|
||||
| `EntityAwareChunker` | Prevents named entity mentions from being split across chunk boundaries |
|
||||
| `RelationAwareChunker` | Keeps subject-predicate-object triplets intact within a single chunk |
|
||||
@@ -39,7 +39,7 @@ Semantica's chunking methods are designed to avoid these failure modes.
|
||||
|
||||
| Method | Best for |
|
||||
| :--- | :--- |
|
||||
| `recursive` | General text — splits on paragraphs, sentences, words in order |
|
||||
| `recursive` | General text: splits on paragraphs, sentences, words in order |
|
||||
| `sentence` | Conversational text, QA |
|
||||
| `paragraph` | Long-form text where paragraph integrity matters |
|
||||
| `token` | LLM context window enforcement |
|
||||
@@ -53,13 +53,13 @@ Semantica's chunking methods are designed to avoid these failure modes.
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="TextSplitter" icon="scissors">
|
||||
Unified interface for 11 chunking strategies — swap methods without changing downstream code.
|
||||
Unified interface for 11 chunking strategies: swap methods without changing downstream code.
|
||||
</Card>
|
||||
<Card title="Semantic Chunking" icon="brain">
|
||||
Embedding-based topic shift detection — splits only when the topic actually changes.
|
||||
Embedding-based topic shift detection: splits only when the topic actually changes.
|
||||
</Card>
|
||||
<Card title="Entity-Aware Chunking" icon="user">
|
||||
Entity spans never cross chunk boundaries — guaranteed by boundary adjustment.
|
||||
Entity spans never cross chunk boundaries: guaranteed by boundary adjustment.
|
||||
</Card>
|
||||
<Card title="Relation-Aware Chunking" icon="arrows-left-right">
|
||||
Subject–predicate–object triplets kept within a single chunk for KG pipelines.
|
||||
@@ -96,7 +96,7 @@ Semantica's chunking methods are designed to avoid these failure modes.
|
||||
<Step title="Or split a document object">
|
||||
```python
|
||||
# split_documents() accepts any object with a .text attribute,
|
||||
# or a plain string — no specific document class required.
|
||||
# or a plain string: no specific document class required.
|
||||
class Doc:
|
||||
def __init__(self, text, metadata=None):
|
||||
self.text = text
|
||||
@@ -127,7 +127,7 @@ Semantica's chunking methods are designed to avoid these failure modes.
|
||||
| Method | How It Splits | Best For |
|
||||
| :------ | :------------- | :-------- |
|
||||
| `recursive` | Paragraph → sentence → word (cascading fallback) | General-purpose default |
|
||||
| `semantic_transformer` | Embeds sentences, splits at cosine similarity drops | RAG — topic coherence matters |
|
||||
| `semantic_transformer` | Embeds sentences, splits at cosine similarity drops | RAG: topic coherence matters |
|
||||
| `entity_aware` | Adjusts boundaries so entity spans are never cut | NER pipelines |
|
||||
| `relation_aware` | Keeps subject–predicate–object triplets within one chunk | KG construction |
|
||||
| `sentence` | Sentence boundary detection (regex, NLTK, spaCy) | Short documents, Q&A |
|
||||
@@ -158,7 +158,7 @@ Use this decision tree before picking a method:
|
||||
from semantica.split import TextSplitter
|
||||
|
||||
splitter = TextSplitter(
|
||||
method="semantic_transformer", # chunking strategy — see Splitting Methods table
|
||||
method="semantic_transformer", # chunking strategy: see Splitting Methods table
|
||||
chunk_size=1000, # target size in characters
|
||||
chunk_overlap=200, # character overlap between adjacent chunks
|
||||
similarity_threshold=0.7, # cosine similarity cutoff (semantic_transformer only)
|
||||
@@ -171,19 +171,19 @@ splitter = TextSplitter(
|
||||
| Parameter | Type | Default | Description |
|
||||
| :--------- | :---- | :------- | :----------- |
|
||||
| `method` | `str \| list[str]` | `"recursive"` | Chunking strategy, or list of methods as fallback chain |
|
||||
| `chunk_size` | `int` | `1000` | Target size in **characters** (not tokens — if you were using token-based sizing before, multiply by ~4 to approximate the same boundary) |
|
||||
| `chunk_size` | `int` | `1000` | Target size in **characters** (not tokens: if you were using token-based sizing before, multiply by ~4 to approximate the same boundary) |
|
||||
| `chunk_overlap` | `int` | `200` | Character overlap between adjacent chunks |
|
||||
| `similarity_threshold` | `float` | `0.7` | Cosine similarity cutoff for `semantic_transformer` — lower = more splits |
|
||||
| `similarity_threshold` | `float` | `0.7` | Cosine similarity cutoff for `semantic_transformer`: lower = more splits |
|
||||
| `model` | `str` | `"all-MiniLM-L6-v2"` | Sentence-transformers model name for `semantic_transformer` |
|
||||
| `ner_method` | `str` | `"ml"` | NER method for `entity_aware`: `"pattern"` \| `"regex"` \| `"ml"` \| `"huggingface"` \| `"llm"` |
|
||||
| `relation_method` | `str` | `"ml"` | Relation extraction method for `relation_aware`: `"ml"` \| `"llm"` \| `"huggingface"` |
|
||||
| `tokenizer` | `str` | `"gpt-4"` | tiktoken model name for `token` method — unrecognised names fall back to `cl100k_base` |
|
||||
| `tokenizer` | `str` | `"gpt-4"` | tiktoken model name for `token` method: unrecognised names fall back to `cl100k_base` |
|
||||
|
||||
## Splitting Method Details
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Recursive (default)">
|
||||
Tries paragraph breaks first, then sentence boundaries, then word boundaries — falling back only when the chunk exceeds `chunk_size`:
|
||||
Tries paragraph breaks first, then sentence boundaries, then word boundaries: falling back only when the chunk exceeds `chunk_size`:
|
||||
|
||||
```python
|
||||
splitter = TextSplitter(method="recursive", chunk_size=1000, chunk_overlap=200)
|
||||
@@ -192,7 +192,7 @@ splitter = TextSplitter(
|
||||
|
||||
**Key behaviours:**
|
||||
- Preserves paragraph and sentence structure wherever possible
|
||||
- Falls back gracefully — never produces chunks larger than `chunk_size`
|
||||
- Falls back gracefully: never produces chunks larger than `chunk_size`
|
||||
- Overlap ensures context continuity across chunk boundaries
|
||||
- Good starting point when you're unsure which method to use
|
||||
</Tab>
|
||||
@@ -207,14 +207,14 @@ splitter = TextSplitter(
|
||||
model="all-MiniLM-L6-v2", # any sentence-transformers model name
|
||||
similarity_threshold=0.7, # 0.6 = more splits, 0.8 = fewer splits
|
||||
chunk_size=800,
|
||||
chunk_overlap=0, # not needed — chunks are already coherent
|
||||
chunk_overlap=0, # not needed: chunks are already coherent
|
||||
)
|
||||
chunks = splitter.split(text)
|
||||
```
|
||||
|
||||
**Key behaviours:**
|
||||
- Requires the `sentence-transformers` package — uses `all-MiniLM-L6-v2` by default, configurable via `model=`
|
||||
- Produces variable-length chunks — some topics are short, others long
|
||||
- Requires the `sentence-transformers` package: uses `all-MiniLM-L6-v2` by default, configurable via `model=`
|
||||
- Produces variable-length chunks: some topics are short, others long
|
||||
- Falls back to sentence splitting if `sentence-transformers` is not installed
|
||||
- Slower than `recursive` due to embedding computation; cache embeddings for repeated splits
|
||||
</Tab>
|
||||
@@ -241,13 +241,13 @@ splitter = TextSplitter(
|
||||
```
|
||||
|
||||
**Key behaviours:**
|
||||
- NER is run internally — entity extraction happens automatically inside the splitter
|
||||
- NER is run internally: entity extraction happens automatically inside the splitter
|
||||
- Entity objects are available in `chunk.metadata["entities"]` for each chunk
|
||||
- Chunk sizes vary slightly from `chunk_size` — boundary adjustments are ≤ one sentence
|
||||
- Chunk sizes vary slightly from `chunk_size`: boundary adjustments are ≤ one sentence
|
||||
- Works with all entity types: PERSON, ORGANIZATION, LOCATION, DATE, custom types
|
||||
</Tab>
|
||||
<Tab title="Relation-Aware">
|
||||
Keeps subject–predicate–object triplets within the same chunk — critical for KG pipelines:
|
||||
Keeps subject–predicate–object triplets within the same chunk: critical for KG pipelines:
|
||||
|
||||
```python
|
||||
from semantica.split import TextSplitter
|
||||
@@ -268,9 +268,9 @@ splitter = TextSplitter(
|
||||
```
|
||||
|
||||
**Key behaviours:**
|
||||
- Relation extraction is run internally — no pre-computed entities or triplets needed
|
||||
- Relation extraction is run internally: no pre-computed entities or triplets needed
|
||||
- Relation objects are available in `chunk.metadata["relationships"]` for each chunk
|
||||
- Implies entity-aware behaviour — both entities in a triplet are kept whole too
|
||||
- Implies entity-aware behaviour: both entities in a triplet are kept whole too
|
||||
- Best used as the split step in a `Parse → Split → Extract → Build KG` pipeline
|
||||
</Tab>
|
||||
<Tab title="Structural">
|
||||
@@ -288,7 +288,7 @@ splitter = TextSplitter(
|
||||
```
|
||||
|
||||
**Key behaviours:**
|
||||
- Operates on plain text — no structural document format required
|
||||
- Operates on plain text: no structural document format required
|
||||
- Respects heading hierarchy (lines starting with `#` or all-caps headings) and paragraph breaks
|
||||
- Uses `max_chunk_size=` parameter instead of the standard `chunk_size=` for maximum size control
|
||||
- Falls back to `recursive` if `StructuralChunker` is unavailable
|
||||
@@ -306,7 +306,7 @@ class Chunk:
|
||||
text: str # the chunk's text content
|
||||
start_index: int # character offset of start in source text
|
||||
end_index: int # character offset of end in source text
|
||||
metadata: Dict[str, Any] # method-specific fields — see table below
|
||||
metadata: Dict[str, Any] # method-specific fields: see table below
|
||||
id: Optional[str] = None # optional chunk identifier
|
||||
```
|
||||
|
||||
@@ -322,7 +322,7 @@ Metadata keys vary by method. Only keys that are actually set by the implementat
|
||||
| `sentence_count` | `int` | `sentence`, `semantic_transformer`, spaCy path | Number of sentences in this chunk |
|
||||
| `paragraph_count` | `int` | `paragraph` | Number of paragraphs in this chunk |
|
||||
| `word_count` | `int` | `word` | Number of words in this chunk |
|
||||
| `token_count` | `int` | `token`; `sentence`/`semantic_transformer` when spaCy is available | Token count — not always present |
|
||||
| `token_count` | `int` | `token`; `sentence`/`semantic_transformer` when spaCy is available | Token count: not always present |
|
||||
| `entity_count` | `int` | `entity_aware` | Number of entities whose boundaries fall in this chunk |
|
||||
| `entities` | `list` | `entity_aware` | Entity objects whose boundaries fall in this chunk |
|
||||
| `relation_count` | `int` | `relation_aware` | Number of relation triplets in this chunk |
|
||||
@@ -348,7 +348,7 @@ If `tiktoken` is not installed, the `token` method falls back to splitting by wh
|
||||
|
||||
## Pipeline Integration
|
||||
|
||||
`TextSplitter` can be used standalone or composed manually with other Semantica modules. The example below shows a sequential pattern — parse a file, split the text, then extract entities from each chunk:
|
||||
`TextSplitter` can be used standalone or composed manually with other Semantica modules. The example below shows a sequential pattern: parse a file, split the text, then extract entities from each chunk:
|
||||
|
||||
```python
|
||||
from semantica.parse import DocumentParser
|
||||
@@ -376,7 +376,7 @@ For the full pipeline orchestration API, see the [Pipeline reference](pipeline).
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum — for `chunk_size=1000`, set `chunk_overlap=100` to `200`.
|
||||
**`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum: for `chunk_size=1000`, set `chunk_overlap=100` to `200`.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
@@ -384,12 +384,12 @@ For the full pipeline orchestration API, see the [Pipeline reference](pipeline).
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting — use `recursive` instead.
|
||||
**Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting: use `recursive` instead.
|
||||
</Tip>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Parse" icon="file-lines" href="parse">
|
||||
Parse documents before chunking — produces sections and metadata.
|
||||
Parse documents before chunking: produces sections and metadata.
|
||||
</Card>
|
||||
<Card title="Embeddings" icon="vector-square" href="embeddings">
|
||||
Embed chunks for vector search and semantic chunking.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Triplet Store Module"
|
||||
description: "RDF triple storage with SPARQL queries and bulk loading — Blazegraph, Apache Jena, and RDF4J."
|
||||
description: "RDF triple storage with SPARQL queries and bulk loading: Blazegraph, Apache Jena, and RDF4J."
|
||||
icon: "table"
|
||||
---
|
||||
|
||||
@@ -13,15 +13,15 @@ icon: "table"
|
||||
| `TripletStore` | Unified interface: `add_triplet`, `add_triplets`, `get_triplets`, `delete_triplet`, `execute_query` |
|
||||
| `QueryEngine` | **SPARQL 1.1** execution with query optimization and result caching |
|
||||
| `BulkLoader` | High-volume RDF loading with batching, retries, and progress tracking |
|
||||
| `BlazegraphStore` | Blazegraph REST API — SPARQL 1.1 Update, namespace management |
|
||||
| `JenaStore` | Apache Jena — rdflib-backed, SPARQL read support via remote endpoint |
|
||||
| `RDF4JStore` | Eclipse RDF4J — REST API, transaction support |
|
||||
| `BlazegraphStore` | Blazegraph REST API: SPARQL 1.1 Update, namespace management |
|
||||
| `JenaStore` | Apache Jena: rdflib-backed, SPARQL read support via remote endpoint |
|
||||
| `RDF4JStore` | Eclipse RDF4J: REST API, transaction support |
|
||||
|
||||
## What You Get
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="TripletStore" icon="server">
|
||||
Unified interface across Blazegraph, Apache Jena, and RDF4J — swap backends with one parameter.
|
||||
Unified interface across Blazegraph, Apache Jena, and RDF4J: swap backends with one parameter.
|
||||
</Card>
|
||||
<Card title="SPARQL" icon="magnifying-glass">
|
||||
Full SPARQL SELECT, ASK, CONSTRUCT, and UPDATE query support via `execute_query()`.
|
||||
@@ -61,7 +61,7 @@ t = Triplet(
|
||||
)
|
||||
store.add_triplet(t)
|
||||
|
||||
# Query with SPARQL — returns a QueryResult with a .bindings list
|
||||
# Query with SPARQL: returns a QueryResult with a .bindings list
|
||||
result = store.execute_query("""
|
||||
PREFIX ex: <http://example.org/>
|
||||
SELECT ?person ?company WHERE {
|
||||
@@ -105,7 +105,7 @@ for row in result.bindings:
|
||||
</Step>
|
||||
<Step title="Query with SPARQL">
|
||||
```python
|
||||
# execute_query returns a QueryResult — iterate result.bindings
|
||||
# execute_query returns a QueryResult: iterate result.bindings
|
||||
result = store.execute_query("""
|
||||
PREFIX ex: <http://example.org/>
|
||||
SELECT ?person ?company WHERE {
|
||||
@@ -224,7 +224,7 @@ t = Triplet(
|
||||
| `get_triplets(subject, predicate, object)` | `List[Triplet]` | Retrieve triplets matching subject/predicate/object filters |
|
||||
| `delete_triplet(triplet)` | `dict` | Delete a `Triplet` from the store |
|
||||
| `update_triplet(old_triplet, new_triplet)` | `dict` | Atomic delete + add |
|
||||
| `execute_query(query, parameters, graph, graphs)` | `QueryResult` | Execute a SPARQL query — returns `QueryResult` with `.bindings`, `.variables`, `.execution_time` |
|
||||
| `execute_query(query, parameters, graph, graphs)` | `QueryResult` | Execute a SPARQL query: returns `QueryResult` with `.bindings`, `.variables`, `.execution_time` |
|
||||
| `store(knowledge_graph, ontology)` | `dict` | Convert a KG + ontology dict to RDF triples and bulk-load them |
|
||||
| `add_skos_concept(concept_uri, scheme_uri, pref_label, ...)` | `dict` | Add a SKOS concept with optional alt labels, broader/narrower/related |
|
||||
| `get_skos_concepts(scheme_uri)` | `List[dict]` | Retrieve all SKOS concepts, optionally filtered by scheme URI |
|
||||
@@ -233,14 +233,14 @@ t = Triplet(
|
||||
|
||||
## SPARQL Queries
|
||||
|
||||
`execute_query()` is the single entry point for all SPARQL operations. It returns a `QueryResult` — access results via `.bindings`:
|
||||
`execute_query()` is the single entry point for all SPARQL operations. It returns a `QueryResult`: access results via `.bindings`:
|
||||
|
||||
```python
|
||||
from semantica.triplet_store import TripletStore
|
||||
|
||||
store = TripletStore(backend="blazegraph", endpoint="http://localhost:9999/blazegraph/sparql")
|
||||
|
||||
# SELECT — iterate result.bindings
|
||||
# SELECT: iterate result.bindings
|
||||
result = store.execute_query("""
|
||||
PREFIX ex: <http://example.org/>
|
||||
SELECT ?person ?company WHERE {
|
||||
@@ -252,7 +252,7 @@ for row in result.bindings:
|
||||
print(row.get("person", {}).get("value"))
|
||||
print(row.get("company", {}).get("value"))
|
||||
|
||||
# ASK, CONSTRUCT, UPDATE — same method, different SPARQL form
|
||||
# ASK, CONSTRUCT, UPDATE: same method, different SPARQL form
|
||||
result = store.execute_query("""
|
||||
PREFIX ex: <http://example.org/>
|
||||
ASK { ex:apple_inc ex:founded_by ex:steve_jobs . }
|
||||
@@ -304,7 +304,7 @@ while True:
|
||||
Blazegraph and RDF4J support named graphs. Scope `execute_query()` to a named graph with the `graph=` parameter:
|
||||
|
||||
```python
|
||||
# Add a triplet — named graph stored in metadata or backend-specific API
|
||||
# Add a triplet: named graph stored in metadata or backend-specific API
|
||||
from semantica.semantic_extract.types import Triplet
|
||||
|
||||
t = Triplet(
|
||||
@@ -471,7 +471,7 @@ for row in result.bindings:
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Tip>
|
||||
**Use Apache Jena for development, Blazegraph for production.** Jena initializes with rdflib in-memory — no server required for local testing. Switch to Blazegraph for high-throughput persistent workloads by changing `backend=`.
|
||||
**Use Apache Jena for development, Blazegraph for production.** Jena initializes with rdflib in-memory: no server required for local testing. Switch to Blazegraph for high-throughput persistent workloads by changing `backend=`.
|
||||
</Tip>
|
||||
|
||||
<Warning>
|
||||
@@ -479,7 +479,7 @@ for row in result.bindings:
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
**`add_triplet()` takes a `Triplet` object, not keyword arguments.** Use `Triplet(subject=..., predicate=..., object=...)` from `semantica.semantic_extract.types` and pass the object — not `subject=`, `predicate=`, `obj=` to `add_triplet`.
|
||||
**`add_triplet()` takes a `Triplet` object, not keyword arguments.** Use `Triplet(subject=..., predicate=..., object=...)` from `semantica.semantic_extract.types` and pass the object: not `subject=`, `predicate=`, `obj=` to `add_triplet`.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
|
||||
+20
-20
@@ -11,19 +11,19 @@ icon: "wrench"
|
||||
- Progress tracking: `ProgressTracker` class and `track_progress()` iterable wrapper with ETA
|
||||
- Typed exceptions: `SemanticaError`, `ValidationError`, `ProcessingError`, `ConfigurationError`, `QualityError`
|
||||
|
||||
Most users won't call utils directly — it's the **shared foundation** for all modules.
|
||||
Most users won't call utils directly: it's the **shared foundation** for all modules.
|
||||
|
||||
|
||||
## Exported Classes
|
||||
|
||||
| Name | Type | Role |
|
||||
| :--- | :--- | :--- |
|
||||
| `setup_logging` | function | Configure the `semantica` root logger — accepts `level`, `file`, `console`, `rotation` kwargs |
|
||||
| `setup_logging` | function | Configure the `semantica` root logger: accepts `level`, `file`, `console`, `rotation` kwargs |
|
||||
| `get_logger` | function | Get a named logger instance (`semantica.<name>`) |
|
||||
| `log_execution_time` | decorator | Wraps a function — logs name, execution time, and success/failure |
|
||||
| `log_execution_time` | decorator | Wraps a function: logs name, execution time, and success/failure |
|
||||
| `log_performance` | function | Log pre-collected performance metrics: `log_performance(func_name, execution_time, **metrics)` |
|
||||
| `validate_entity` | function | Validate entity dict — returns `(bool, Optional[str])`; does not raise |
|
||||
| `validate_config` | function | Validate config dict — returns `(bool, Optional[str])`; does not raise |
|
||||
| `validate_entity` | function | Validate entity dict: returns `(bool, Optional[str])`; does not raise |
|
||||
| `validate_config` | function | Validate config dict: returns `(bool, Optional[str])`; does not raise |
|
||||
| `ProgressTracker` | class | Class-based progress tracker with ETA and step callbacks |
|
||||
| `track_progress` | function | Wrap any iterable with a live progress bar |
|
||||
| `clean_text` | function | Normalize whitespace and strip zero-width control characters |
|
||||
@@ -44,16 +44,16 @@ Most users won't call utils directly — it's the **shared foundation** for all
|
||||
`validate_entity` and `validate_config` with a typed `ValidationError` carrying field and value context.
|
||||
</Card>
|
||||
<Card title="Progress Tracking" icon="bars-progress">
|
||||
`track_progress` wraps any iterable — auto-detects console vs Jupyter for the right renderer.
|
||||
`track_progress` wraps any iterable: auto-detects console vs Jupyter for the right renderer.
|
||||
</Card>
|
||||
<Card title="Helper Functions" icon="wrench">
|
||||
`clean_text`, `hash_data`, `safe_filename`, and nested dict utilities used throughout the framework.
|
||||
</Card>
|
||||
<Card title="Exception Hierarchy" icon="triangle-exclamation">
|
||||
`SemanticaError` → `ValidationError`, `ProcessingError` — typed exceptions for targeted recovery.
|
||||
`SemanticaError` → `ValidationError`, `ProcessingError`: typed exceptions for targeted recovery.
|
||||
</Card>
|
||||
<Card title="File Utilities" icon="file">
|
||||
`read_json_file` raises `FileNotFoundError` or `json.JSONDecodeError` on failure — no boilerplate try/except around JSON I/O.
|
||||
`read_json_file` raises `FileNotFoundError` or `json.JSONDecodeError` on failure: no boilerplate try/except around JSON I/O.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -113,15 +113,15 @@ if not is_valid:
|
||||
```python
|
||||
from semantica.utils import track_progress
|
||||
|
||||
# Wraps any iterable — auto-detects console vs Jupyter
|
||||
# Wraps any iterable: auto-detects console vs Jupyter
|
||||
for item in track_progress(items, desc="Processing documents"):
|
||||
process(item)
|
||||
```
|
||||
|
||||
Supports:
|
||||
- **Console** — tqdm progress bar with ETA
|
||||
- **Jupyter** — notebook-compatible widget (auto-detected)
|
||||
- **File** — write progress to a log file
|
||||
- **Console**: tqdm progress bar with ETA
|
||||
- **Jupyter**: notebook-compatible widget (auto-detected)
|
||||
- **File**: write progress to a log file
|
||||
|
||||
## Helper Functions
|
||||
|
||||
@@ -140,7 +140,7 @@ fname = safe_filename("My File?.txt") # -> "My_File.txt"
|
||||
|
||||
## Nested Dict Utilities
|
||||
|
||||
Helper functions for deep configuration access — used extensively inside `Config` and `ConfigManager`:
|
||||
Helper functions for deep configuration access: used extensively inside `Config` and `ConfigManager`:
|
||||
|
||||
```python
|
||||
from semantica.utils import get_nested_value, set_nested_value, merge_dicts
|
||||
@@ -150,14 +150,14 @@ config = {
|
||||
"llm": {"provider": "groq", "model": "llama-3.3-70b-versatile"},
|
||||
}
|
||||
|
||||
# Dot-notation read — returns default if key path is absent
|
||||
# Dot-notation read: returns default if key path is absent
|
||||
batch = get_nested_value(config, "processing.batch_size", default=16)
|
||||
# -> 32
|
||||
|
||||
# Dot-notation write
|
||||
set_nested_value(config, "processing.batch_size", 64)
|
||||
|
||||
# Deep merge — nested keys are merged recursively (deep=True by default)
|
||||
# Deep merge: nested keys are merged recursively (deep=True by default)
|
||||
base = {"a": {"x": 1, "y": 2}, "b": 3}
|
||||
overrides = {"a": {"y": 99, "z": 4}, "c": 5}
|
||||
merged = merge_dicts(base, overrides)
|
||||
@@ -187,7 +187,7 @@ except SemanticaError as e:
|
||||
|
||||
| Exception | When Raised | Key Attributes |
|
||||
| :--------- | :----------- | :-------------- |
|
||||
| `SemanticaError` | Base class — all framework errors inherit from this | `.message`, `.context`, `.error_code` |
|
||||
| `SemanticaError` | Base class: all framework errors inherit from this | `.message`, `.context`, `.error_code` |
|
||||
| `ValidationError` | Input data failed schema or type validation | `.field`, `.value`, `.constraint` |
|
||||
| `ProcessingError` | Failure during extraction, graph build, or pipeline step | `.stage`, `.input_data`, `.output_data` |
|
||||
| `ConfigurationError` | Configuration key missing or wrong type | `.config_key`, `.config_value`, `.expected_type` |
|
||||
@@ -201,7 +201,7 @@ except SemanticaError as e:
|
||||
```python
|
||||
from semantica.utils import read_json_file
|
||||
|
||||
# Read and parse a JSON file — raises FileNotFoundError or json.JSONDecodeError on failure
|
||||
# Read and parse a JSON file: raises FileNotFoundError or json.JSONDecodeError on failure
|
||||
config = read_json_file("config.json")
|
||||
```
|
||||
|
||||
@@ -212,11 +212,11 @@ config = read_json_file("config.json")
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**`@log_execution_time` is the performance decorator.** Apply it to any function to automatically log its name, execution time, and success/failure. `log_performance` is a lower-level function for logging metrics you've already collected — it is not a decorator.
|
||||
**`@log_execution_time` is the performance decorator.** Apply it to any function to automatically log its name, execution time, and success/failure. `log_performance` is a lower-level function for logging metrics you've already collected: it is not a decorator.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
**`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string — suitable as a cache key or idempotency token in pipeline steps.
|
||||
**`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string: suitable as a cache key or idempotency token in pipeline steps.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
@@ -224,7 +224,7 @@ config = read_json_file("config.json")
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
**`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment — the same call works in both.
|
||||
**`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment: the same call works in both.
|
||||
</Tip>
|
||||
|
||||
<CardGroup cols={2}>
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: "database"
|
||||
|
||||
`semantica.vector_store` provides a unified API for storing and searching vector embeddings across all major backends:
|
||||
|
||||
- Swap backends with a one-line change — no application code changes needed
|
||||
- Swap backends with a one-line change: no application code changes needed
|
||||
- `HybridSearch` fuses dense vector similarity with metadata filtering via RRF or weighted average
|
||||
- `NamespaceManager` for multi-tenant structural isolation
|
||||
- `FAISSStore` with flat, ivf, hnsw, and pq index types
|
||||
@@ -20,13 +20,13 @@ icon: "database"
|
||||
| `VectorStore` | Unified interface: `store_vectors`, `search_vectors`, `update_vectors`, `delete_vectors` |
|
||||
| `HybridSearch` | Fuses dense vector similarity with metadata filtering via RRF or weighted average |
|
||||
| `MetadataFilter` | Chainable filter builder: `.eq("type", "person").gt("year", 2020).in_list("tag", [...])` |
|
||||
| `NamespaceManager` | Multi-tenant isolation — separate index namespaces per project or user |
|
||||
| `FAISSStore` | Local disk or in-memory — flat, ivf, hnsw, and pq index types |
|
||||
| `NamespaceManager` | Multi-tenant isolation: separate index namespaces per project or user |
|
||||
| `FAISSStore` | Local disk or in-memory: flat, ivf, hnsw, and pq index types |
|
||||
| `WeaviateStore` | Cloud or self-hosted, schema-aware |
|
||||
| `QdrantStore` | Cloud or self-hosted with payload-based filtering |
|
||||
| `PineconeStore` | Managed cloud vector database with serverless and pod modes |
|
||||
| `MilvusStore` | Scalable self-hosted vector database |
|
||||
| `PgVectorStore` | PostgreSQL with `pgvector` extension — no extra infrastructure |
|
||||
| `PgVectorStore` | PostgreSQL with `pgvector` extension: no extra infrastructure |
|
||||
| `MetadataStore` | Standalone metadata indexing and querying |
|
||||
| `SearchRanker` | RRF and weighted-average result fusion |
|
||||
|
||||
@@ -35,7 +35,7 @@ icon: "database"
|
||||
<CardGroup cols={2}>
|
||||
<Card title="VectorStore" icon="database">
|
||||
- Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector
|
||||
- One-line backend swap — no application code changes
|
||||
- One-line backend swap: no application code changes
|
||||
- `add_documents()` auto-embeds; `store_vectors()` for pre-computed embeddings
|
||||
</Card>
|
||||
<Card title="HybridSearch" icon="magnifying-glass">
|
||||
@@ -73,7 +73,7 @@ icon: "database"
|
||||
```python
|
||||
from semantica.vector_store import VectorStore
|
||||
|
||||
# In-memory (development / testing — no persistence)
|
||||
# In-memory (development / testing: no persistence)
|
||||
store = VectorStore(backend="inmemory", dimension=384)
|
||||
|
||||
# FAISS (local, persists to disk via save/load)
|
||||
@@ -88,7 +88,7 @@ ids = store.add_documents(
|
||||
# Search by text query (auto-embedded)
|
||||
results = store.search("technology company founders", limit=5)
|
||||
for r in results:
|
||||
print(f"{r['id']} — score: {r['score']:.3f}")
|
||||
print(f"{r['id']}: score: {r['score']:.3f}")
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
@@ -129,7 +129,7 @@ for r in results:
|
||||
results = store.search_vectors(query_vector, k=10)
|
||||
|
||||
for r in results:
|
||||
print(f"{r['id']} — score: {r['score']:.3f}")
|
||||
print(f"{r['id']}: score: {r['score']:.3f}")
|
||||
```
|
||||
</Step>
|
||||
<Step title="Filter results by metadata">
|
||||
@@ -138,7 +138,7 @@ for r in results:
|
||||
|
||||
mf = MetadataFilter().eq("category", "research").gt("year", 2022)
|
||||
|
||||
# Pass vector_store to the constructor — search() resolves vectors automatically
|
||||
# Pass vector_store to the constructor: search() resolves vectors automatically
|
||||
search = HybridSearch(vector_store=store)
|
||||
results = search.search(query=query_vector, k=10, metadata_filter=mf)
|
||||
```
|
||||
@@ -151,10 +151,10 @@ for r in results:
|
||||
<Tab title="In-memory / FAISS">
|
||||
|
||||
```python
|
||||
# In-memory — no persistence, for development and testing
|
||||
# In-memory: no persistence, for development and testing
|
||||
store = VectorStore(backend="inmemory", dimension=384)
|
||||
|
||||
# FAISS — local disk persistence via save() / load()
|
||||
# FAISS: local disk persistence via save() / load()
|
||||
store = VectorStore(backend="faiss", dimension=384)
|
||||
store.save("./my_store") # save to directory
|
||||
store.load("./my_store") # restore from directory
|
||||
@@ -228,7 +228,7 @@ store = VectorStore(
|
||||
)
|
||||
```
|
||||
|
||||
`connection_string` is required — the store raises `ValueError` at construction if it is absent.
|
||||
`connection_string` is required: the store raises `ValueError` at construction if it is absent.
|
||||
|
||||
</Tab>
|
||||
<Tab title="Milvus">
|
||||
@@ -269,7 +269,7 @@ store = VectorStore(
|
||||
```python
|
||||
from semantica.vector_store import HybridSearch, MetadataFilter
|
||||
|
||||
# With vector_store — search() pulls vectors from the store automatically
|
||||
# With vector_store: search() pulls vectors from the store automatically
|
||||
search = HybridSearch(vector_store=store)
|
||||
mf = MetadataFilter().eq("category", "research").gt("year", 2022)
|
||||
|
||||
@@ -280,7 +280,7 @@ results = search.search(
|
||||
)
|
||||
|
||||
for r in results:
|
||||
print(f"{r['id']} — score: {r['score']:.3f} metadata: {r['metadata']}")
|
||||
print(f"{r['id']}: score: {r['score']:.3f} metadata: {r['metadata']}")
|
||||
```
|
||||
|
||||
Without a `vector_store`, pass vectors explicitly:
|
||||
@@ -309,7 +309,7 @@ fused = search.multi_source_search(query_vector, sources, k=10)
|
||||
|
||||
## Metadata Filtering
|
||||
|
||||
`MetadataFilter` supports chained conditions — all conditions are ANDed:
|
||||
`MetadataFilter` supports chained conditions: all conditions are ANDed:
|
||||
|
||||
```python
|
||||
from semantica.vector_store import MetadataFilter
|
||||
@@ -320,7 +320,7 @@ mf = MetadataFilter().gt("year", 2022).lte("year", 2024) # range
|
||||
mf = MetadataFilter().in_list("tag", ["ai", "ml"]) # set membership
|
||||
mf = MetadataFilter().contains("title", "neural") # substring / list contains
|
||||
|
||||
# Multiple conditions — all must match (AND)
|
||||
# Multiple conditions: all must match (AND)
|
||||
mf = (
|
||||
MetadataFilter()
|
||||
.eq("category", "research")
|
||||
@@ -349,19 +349,19 @@ mf = (
|
||||
```python
|
||||
from semantica.vector_store import SearchRanker
|
||||
|
||||
# Reciprocal Rank Fusion — robust to score scale differences
|
||||
# Reciprocal Rank Fusion: robust to score scale differences
|
||||
ranker = SearchRanker(strategy="reciprocal_rank_fusion")
|
||||
fused = ranker.rank([results_list_1, results_list_2])
|
||||
|
||||
# Weighted average — requires normalised scores on the same scale
|
||||
# Weighted average: requires normalised scores on the same scale
|
||||
ranker = SearchRanker(strategy="weighted_average")
|
||||
fused = ranker.rank([results_list_1, results_list_2], weights=[0.7, 0.3])
|
||||
```
|
||||
|
||||
| Fusion strategy | Description |
|
||||
| :--------------- | :----------- |
|
||||
| `reciprocal_rank_fusion` | Rank-based combination via RRF — robust to score scale differences (default) |
|
||||
| `weighted_average` | Weighted sum of scores — pass `weights=[...]` to `rank()` |
|
||||
| `reciprocal_rank_fusion` | Rank-based combination via RRF: robust to score scale differences (default) |
|
||||
| `weighted_average` | Weighted sum of scores: pass `weights=[...]` to `rank()` |
|
||||
|
||||
## Namespace Isolation
|
||||
|
||||
@@ -397,7 +397,7 @@ ns_manager.delete_namespace("tenant_a")
|
||||
## Batch Operations
|
||||
|
||||
```python
|
||||
# Batch add text documents — parallel embedding with configurable workers
|
||||
# Batch add text documents: parallel embedding with configurable workers
|
||||
ids = store.add_documents(
|
||||
documents=large_doc_list,
|
||||
metadata=large_meta_list,
|
||||
@@ -424,7 +424,7 @@ store.update_vectors(
|
||||
store = VectorStore(backend="faiss", dimension=384)
|
||||
store.add_documents(documents=docs, metadata=meta)
|
||||
|
||||
# Save to a directory — creates index.bin and store_data.pkl
|
||||
# Save to a directory: creates index.bin and store_data.pkl
|
||||
store.save("./vector_store_backup")
|
||||
|
||||
# Restore in a new process
|
||||
@@ -449,7 +449,7 @@ meta_store = MetadataStore()
|
||||
meta_store.store_metadata("doc1", {"author": "Alice", "year": 2024, "category": "research"})
|
||||
meta_store.store_metadata("doc2", {"author": "Bob", "year": 2023, "category": "review"})
|
||||
|
||||
# Query — returns List[str] of matching vector IDs
|
||||
# Query: returns List[str] of matching vector IDs
|
||||
ids = meta_store.query_metadata({"category": "research", "year": 2024})
|
||||
|
||||
# OR query
|
||||
@@ -476,16 +476,16 @@ from semantica.vector_store import FAISSStore
|
||||
|
||||
store = FAISSStore(dimension=384)
|
||||
|
||||
# flat — brute-force exact search
|
||||
# flat: brute-force exact search
|
||||
store.create_index(index_type="flat", metric="L2")
|
||||
|
||||
# ivf — inverted file index
|
||||
# ivf: inverted file index
|
||||
store.create_index(index_type="ivf", metric="L2", nlist=100)
|
||||
|
||||
# hnsw — hierarchical navigable small world graph
|
||||
# hnsw: hierarchical navigable small world graph
|
||||
store.create_index(index_type="hnsw", metric="L2", M=32)
|
||||
|
||||
# pq — product quantization for memory efficiency
|
||||
# pq: product quantization for memory efficiency
|
||||
store.create_index(index_type="pq", metric="L2", m=8)
|
||||
```
|
||||
|
||||
@@ -577,11 +577,11 @@ store.create_index(index_type="pq", metric="L2", m=8)
|
||||
## Tips and Common Pitfalls
|
||||
|
||||
<Warning>
|
||||
**Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size — `BAAI/bge-small-en-v1.5` = 384, `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time.
|
||||
**Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size: `BAAI/bge-small-en-v1.5` = 384, `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
**FAISS index type names are lowercase.** The `FAISSStore.create_index()` method expects `"flat"`, `"ivf"`, `"hnsw"`, `"pq"` — not `"Flat"`, `"IVF"`, `"HNSW"`, `"PQ"`. Uppercase values raise `ValidationError`.
|
||||
**FAISS index type names are lowercase.** The `FAISSStore.create_index()` method expects `"flat"`, `"ivf"`, `"hnsw"`, `"pq"`: not `"Flat"`, `"IVF"`, `"HNSW"`, `"PQ"`. Uppercase values raise `ValidationError`.
|
||||
</Warning>
|
||||
|
||||
<Warning>
|
||||
@@ -589,7 +589,7 @@ store.create_index(index_type="pq", metric="L2", m=8)
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**Use `HybridSearch(vector_store=store)` to avoid passing raw vectors on every call.** When `vector_store` is set, `search()` pulls vectors and metadata from the store automatically — you only need to pass the query and filter.
|
||||
**Use `HybridSearch(vector_store=store)` to avoid passing raw vectors on every call.** When `vector_store` is set, `search()` pulls vectors and metadata from the store automatically: you only need to pass the query and filter.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
|
||||
@@ -4,12 +4,12 @@ description: "Interactive and static knowledge graph, ontology, embedding, and t
|
||||
icon: "chart-bar"
|
||||
---
|
||||
|
||||
**`semantica.visualization`** renders knowledge graphs, ontologies, embedding spaces, and temporal data as **interactive HTML or static images** — without launching the full Explorer server:
|
||||
**`semantica.visualization`** renders knowledge graphs, ontologies, embedding spaces, and temporal data as **interactive HTML or static images**: without launching the full Explorer server:
|
||||
|
||||
- `KGVisualizer` — interactive network with force, hierarchical, and circular layouts
|
||||
- `EmbeddingVisualizer` — 2D/3D UMAP or t-SNE projections with cluster labels
|
||||
- `TemporalVisualizer` — timeline views and graph evolution across snapshots
|
||||
- `AnalyticsVisualizer` — centrality scores, community structure, degree distribution charts
|
||||
- `KGVisualizer`: interactive network with force, hierarchical, and circular layouts
|
||||
- `EmbeddingVisualizer`: 2D/3D UMAP or t-SNE projections with cluster labels
|
||||
- `TemporalVisualizer`: timeline views and graph evolution across snapshots
|
||||
- `AnalyticsVisualizer`: centrality scores, community structure, degree distribution charts
|
||||
|
||||
Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` or `graphviz`.
|
||||
|
||||
@@ -34,7 +34,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
|
||||
|
||||
viz = KGVisualizer(layout="force", color_scheme="default")
|
||||
|
||||
# Interactive — opens in browser, supports hover and click
|
||||
# Interactive: opens in browser, supports hover and click
|
||||
viz.visualize_network(graph, output="interactive")
|
||||
```
|
||||
</Step>
|
||||
@@ -52,10 +52,10 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
|
||||
</Step>
|
||||
<Step title="Export to static formats">
|
||||
```python
|
||||
# Static PNG — for reports and embedding in documents
|
||||
# Static PNG: for reports and embedding in documents
|
||||
viz.visualize_network(graph, output="png", file_path="graph.png")
|
||||
|
||||
# Vector SVG — for publications and scalable diagrams
|
||||
# Vector SVG: for publications and scalable diagrams
|
||||
viz.visualize_network(graph, output="svg", file_path="graph.svg")
|
||||
```
|
||||
</Step>
|
||||
@@ -72,7 +72,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
|
||||
|
||||
viz = KGVisualizer(layout="force", color_scheme="default")
|
||||
|
||||
# Interactive — opens in browser
|
||||
# Interactive: opens in browser
|
||||
viz.visualize_network(graph, output="interactive")
|
||||
|
||||
# Save as HTML file
|
||||
@@ -98,7 +98,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
|
||||
|
||||
| Layout | Description | Best For |
|
||||
| :------ | :----------- | :-------- |
|
||||
| `force` | Physics simulation — clusters emerge naturally | General graphs |
|
||||
| `force` | Physics simulation: clusters emerge naturally | General graphs |
|
||||
| `hierarchical` | Top-down tree layout | Taxonomies, org charts |
|
||||
| `circular` | Nodes on a circle, edges as chords | Small dense graphs |
|
||||
</Tab>
|
||||
@@ -160,7 +160,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
|
||||
# Timeline of entity/relationship changes
|
||||
viz.visualize_timeline(temporal_data, output="interactive")
|
||||
|
||||
# Animated network evolution — one frame per time step
|
||||
# Animated network evolution: one frame per time step
|
||||
viz.visualize_network_evolution(temporal_kg, output="html", file_path="evolution.html")
|
||||
|
||||
# Side-by-side snapshot comparison
|
||||
@@ -171,7 +171,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
|
||||
}
|
||||
viz.visualize_snapshot_comparison(snapshots, output="html", file_path="diff.html")
|
||||
|
||||
# Temporal patterns — pass a list of pattern dicts
|
||||
# Temporal patterns: pass a list of pattern dicts
|
||||
viz.visualize_temporal_patterns(patterns, output="html", file_path="patterns.html")
|
||||
|
||||
# Metrics evolution over time
|
||||
@@ -179,7 +179,7 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="AnalyticsVisualizer">
|
||||
Visualize graph analytics results — centrality, communities, and degree distribution:
|
||||
Visualize graph analytics results: centrality, communities, and degree distribution:
|
||||
|
||||
```python
|
||||
from semantica.visualization import AnalyticsVisualizer
|
||||
|
||||
+21
-21
@@ -18,8 +18,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
**Key modules:** `ingest` (PubMed RSS), `semantic_extract`, `kg`, `deduplication`, `context`
|
||||
|
||||
**Notebooks:**
|
||||
- [Drug Discovery Pipeline](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb) — Intermediate
|
||||
- [Genomic Variant Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb) — Intermediate
|
||||
- [Drug Discovery Pipeline](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb): Intermediate
|
||||
- [Genomic Variant Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb): Intermediate
|
||||
</Card>
|
||||
<Card title="GraphRAG for Research" icon="magnifying-glass" href="https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb">
|
||||
Ground LLM answers in structured scientific literature with hybrid retrieval, logical inference, and source attribution on every claim.
|
||||
@@ -27,8 +27,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
**Key modules:** `context`, `vector_store`, `kg`, `reasoning`, `llms`
|
||||
|
||||
**Notebooks:**
|
||||
- [GraphRAG Complete](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb) — Advanced
|
||||
- [RAG vs. GraphRAG Comparison](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb) — Advanced
|
||||
- [GraphRAG Complete](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb): Advanced
|
||||
- [RAG vs. GraphRAG Comparison](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb): Advanced
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -38,12 +38,12 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Financial Data Integration" icon="chart-line" href="https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb">
|
||||
Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph — with conflict detection when sources disagree.
|
||||
Unify financial data from APIs, MCP servers, and real-time streams into a single queryable knowledge graph: with conflict detection when sources disagree.
|
||||
|
||||
**Key modules:** `ingest` (API, MCP, stream), `normalize`, `kg`, `conflicts`, `provenance`
|
||||
|
||||
**Notebooks:**
|
||||
- [Financial Data Integration (MCP)](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb) — Intermediate
|
||||
- [Financial Data Integration (MCP)](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb): Intermediate
|
||||
</Card>
|
||||
<Card title="Fraud Detection" icon="shield-halved" href="https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb">
|
||||
Detect complex fraud rings using temporal graphs and pattern detection over transaction, device, and user data. Temporal edges let you query: "what connections existed during this window?"
|
||||
@@ -51,7 +51,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
**Key modules:** `kg` (temporal), `conflicts`, `reasoning`, `visualization`
|
||||
|
||||
**Notebooks:**
|
||||
- [Fraud Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb) — Advanced
|
||||
- [Fraud Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/finance/02_Fraud_Detection.ipynb): Advanced
|
||||
</Card>
|
||||
<Card title="Blockchain Analytics" icon="link" href="https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb">
|
||||
Map transaction flows, analyze DeFi protocols, and detect illicit activity. Graph algorithms (centrality, community detection) surface high-risk actors that linear transaction analysis misses.
|
||||
@@ -59,8 +59,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
**Key modules:** `kg`, `reasoning`, `visualization`
|
||||
|
||||
**Notebooks:**
|
||||
- [DeFi Protocol Intelligence](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb) — Intermediate
|
||||
- [Transaction Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb) — Intermediate
|
||||
- [DeFi Protocol Intelligence](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb): Intermediate
|
||||
- [Transaction Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/blockchain/02_Transaction_Network_Analysis.ipynb): Intermediate
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -75,8 +75,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
**Key modules:** `ingest` (stream, feed), `kg` (temporal), `context`, `reasoning`, `export`
|
||||
|
||||
**Notebooks:**
|
||||
- [Real-Time Anomaly Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) — Advanced
|
||||
- [Threat Intelligence Hybrid RAG](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) — Advanced
|
||||
- [Real-Time Anomaly Detection](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb): Advanced
|
||||
- [Threat Intelligence Hybrid RAG](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb): Advanced
|
||||
</Card>
|
||||
<Card title="Criminal Network Analysis" icon="users" href="https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb">
|
||||
Build knowledge graphs from police reports, court records, and OSINT feeds to identify key players, communities, and suspicious patterns. Network centrality surfaces actors text search alone would miss.
|
||||
@@ -84,7 +84,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
**Key modules:** `ingest`, `semantic_extract`, `kg`, `visualization` (community detection)
|
||||
|
||||
**Notebooks:**
|
||||
- [Criminal Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) — Intermediate
|
||||
- [Criminal Network Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb): Intermediate
|
||||
</Card>
|
||||
<Card title="Intelligence Analysis Orchestrator" icon="network-wired" href="https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb">
|
||||
Process multiple intelligence sources in parallel with an orchestrator-worker pipeline. Multi-source conflict detection flags disagreements rather than silently discarding minority reports.
|
||||
@@ -92,7 +92,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
**Key modules:** `pipeline`, `ingest`, `conflicts`, `provenance`, `export`
|
||||
|
||||
**Notebooks:**
|
||||
- [Intelligence Analysis Orchestrator-Worker](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb) — Intermediate
|
||||
- [Intelligence Analysis Orchestrator-Worker](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb): Intermediate
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -107,7 +107,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
**Key modules:** `ingest`, `kg`, `reasoning`, `visualization`, `export` (Parquet for analytics)
|
||||
|
||||
**Notebooks:**
|
||||
- [Supply Chain Data Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb) — Intermediate
|
||||
- [Supply Chain Data Integration](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/supply_chain/01_Supply_Chain_Data_Integration.ipynb): Intermediate
|
||||
</Card>
|
||||
<Card title="Renewable Energy Management" icon="bolt" href="https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb">
|
||||
Connect sensor data, weather forecasts, and maintenance logs to predict equipment failures and optimize grid operations. Temporal graphs track asset states over time and correlate maintenance events with performance degradation.
|
||||
@@ -115,7 +115,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
**Key modules:** `ingest` (stream, API), `kg` (temporal), `reasoning`, `visualization`
|
||||
|
||||
**Notebooks:**
|
||||
- [Energy Market Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb) — Intermediate
|
||||
- [Energy Market Analysis](https://github.com/semantica-agi/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb): Intermediate
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
@@ -133,8 +133,8 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
|
||||
| Semantica capability | Compliance role |
|
||||
| :-------------------- | :-------------- |
|
||||
| W3C PROV-O provenance | Full lineage from raw data to inference — required for FDA audit trails |
|
||||
| SHA-256 checksums | Tamper detection on every snapshot — supports electronic record integrity |
|
||||
| W3C PROV-O provenance | Full lineage from raw data to inference: required for FDA audit trails |
|
||||
| SHA-256 checksums | Tamper detection on every snapshot: supports electronic record integrity |
|
||||
| Decision tracking | Every AI-assisted recommendation is recorded with causal chain and confidence |
|
||||
| Temporal graphs | Point-in-time queries for retrospective safety analysis |
|
||||
| SHACL validation | Schema enforcement before data enters the knowledge graph |
|
||||
@@ -147,10 +147,10 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
|
||||
| Semantica capability | Compliance role |
|
||||
| :-------------------- | :-------------- |
|
||||
| Decision audit trail | Full record of model decisions with reasoning — required for model risk management |
|
||||
| Decision audit trail | Full record of model decisions with reasoning: required for model risk management |
|
||||
| Conflict detection | Flags when two sources disagree on a valuation or risk figure |
|
||||
| Version control | SHA-256 snapshot history — supports point-in-time reconstruction for audits |
|
||||
| Provenance export | RDF with PROV-O inline — submittable to regulatory bodies as structured evidence |
|
||||
| Version control | SHA-256 snapshot history: supports point-in-time reconstruction for audits |
|
||||
| Provenance export | RDF with PROV-O inline: submittable to regulatory bodies as structured evidence |
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -160,7 +160,7 @@ Semantica is purpose-built for environments where AI outputs must be explainable
|
||||
|
||||
| Semantica capability | Operational role |
|
||||
| :-------------------- | :--------------- |
|
||||
| Local LLM support | `HuggingFaceLLM` and Ollama via LiteLLM — fully air-gapped deployments |
|
||||
| Local LLM support | `HuggingFaceLLM` and Ollama via LiteLLM: fully air-gapped deployments |
|
||||
| Provenance chains | Every intelligence claim traceable to source document and extraction event |
|
||||
| Conflict resolution | Multiple-source disagreement resolved with auditable strategy |
|
||||
| Temporal intelligence | Historical queries over evolving intelligence graphs |
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: "pgvector Store"
|
||||
description: "PostgreSQL with pgvector extension — cosine, L2, and inner product similarity search with IVFFlat and HNSW indexing."
|
||||
description: "PostgreSQL with pgvector extension: cosine, L2, and inner product similarity search with IVFFlat and HNSW indexing."
|
||||
icon: "database"
|
||||
---
|
||||
|
||||
**`PgVectorStore`** adds PostgreSQL-native vector storage and similarity search to Semantica — no dedicated vector database required.
|
||||
**`PgVectorStore`** adds PostgreSQL-native vector storage and similarity search to Semantica: no dedicated vector database required.
|
||||
|
||||
|
||||
## Overview
|
||||
@@ -18,7 +18,7 @@ icon: "database"
|
||||
<Check>JSONB metadata storage with filtering support</Check>
|
||||
<Check>Connection pooling via psycopg3/psycopg2</Check>
|
||||
<Check>Batch insert, update, and delete</Check>
|
||||
<Check>Idempotent index creation — safe to call multiple times</Check>
|
||||
<Check>Idempotent index creation: safe to call multiple times</Check>
|
||||
|
||||
|
||||
## Setup
|
||||
@@ -196,7 +196,7 @@ store.create_index(
|
||||
)
|
||||
```
|
||||
|
||||
Index creation is idempotent — calling multiple times is safe.
|
||||
Index creation is idempotent: calling multiple times is safe.
|
||||
|
||||
### Statistics
|
||||
|
||||
@@ -226,7 +226,7 @@ stats = store.get_stats()
|
||||
|
||||
<Tabs>
|
||||
<Tab title="HNSW">
|
||||
**Hierarchical Navigable Small World** — best for high-dimensional vectors with high recall requirements.
|
||||
**Hierarchical Navigable Small World**: best for high-dimensional vectors with high recall requirements.
|
||||
|
||||
| | |
|
||||
| :-- | :-- |
|
||||
@@ -241,7 +241,7 @@ stats = store.get_stats()
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="IVFFlat">
|
||||
**Inverted File with Flat Index** — best for large datasets in memory-constrained environments.
|
||||
**Inverted File with Flat Index**: best for large datasets in memory-constrained environments.
|
||||
|
||||
| | |
|
||||
| :-- | :-- |
|
||||
|
||||
Reference in New Issue
Block a user