mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-09-10 04:00:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #566 from semantica-agi/docs-mintlify-component-overhaul

Browse Source 
docs: full Mintlify component overhaul — all 27 reference pages + concepts.md
This commit is contained in:
Mohd Kaif
2026-05-24 14:59:49 +05:30
committed by GitHub
parent 11e8a2fc0d 68fcff5b3a
commit 72fefbeda6
30 changed files with 6493 additions and 2136 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/concepts.md
+242 -52
View File
@@ -10,7 +10,21 @@ icon: "book-open"
Semantica transforms unstructured data — documents, web pages, reports, databases — into **knowledge graphs**: structured representations that AI systems can query, reason about, and trace back to sources.
At its core, Semantica adds a **context and intelligence layer** on top of your existing AI stack. It doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs accountable.
At its core, Semantica adds a **context and accountability layer** on top of your existing AI stack. It doesn't replace LangChain, LlamaIndex, or your LLM provider — it makes their outputs **grounded**, **traceable**, and **auditable**.
<CardGroup cols={3}>
<Card title="Context Layer" icon="diagram-project">
Knowledge graphs, GraphRAG retrieval, semantic embeddings, and temporal intelligence ground every LLM response in structured, queryable facts.
</Card>
<Card title="Accountability Layer" icon="shield-check">
Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable.
</Card>
<Card title="Extension Layer" icon="plug">
`PluginRegistry` and `MethodRegistry` let you replace or augment any component — ingestors, extractors, reasoning engines, backends — without changing framework code.
</Card>
</CardGroup>
---
## Knowledge Graphs
@@ -26,7 +40,7 @@ This structure makes knowledge **searchable**, **connectable**, **queryable**, a
## Entity Extraction (NER)
Scanning text to find and classify real-world entities.
Scanning text to find and classify real-world entities:
```python
# Input: "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."
@@ -42,13 +56,15 @@ Scanning text to find and classify real-world entities.
Each entity gets a type, confidence score, and a link to its source document. Three extraction methods are available:
- **`"pattern"`** — fast, regex-based, no API key required
- **`"ml"`** — local ML model, higher accuracy
- **`"llm"`** — LLM-powered, highest accuracy, supports all 8 providers
| Method | Speed | Accuracy | Requirements |
| ------ | ----- | -------- | ------------ |
| `"pattern"` | ⚡ Very fast | Moderate | No API key — regex-based |
| `"ml"` | Fast | High | Local ML model |
| `"llm"` | Medium | Highest | LLM provider — all 9 supported |
## Relationship Extraction
Finding how entities connect to each other.
Finding how entities connect to each other:
```python
{
@@ -71,8 +87,9 @@ Semantica uses embeddings for:
- **Entity resolution** — match the same entity across different sources
- **Precedent search** — find similar past decisions
- **GraphRAG retrieval** — hybrid vector + graph traversal
- **Distance Intelligence** — N×N semantic distance matrices between any node set
**Supported models:** Sentence-Transformers, FastEmbed, OpenAI, BGE
**Supported models:** Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama local embeddings.
## GraphRAG
@@ -80,14 +97,24 @@ GraphRAG (Graph-Augmented Retrieval Augmented Generation) enhances LLM responses
<img src="/assets/img/diagrams/graphrag-flow.svg" alt="GraphRAG flow: User Query → Vector Search + Graph Traversal → Context Builder → LLM → Grounded Answer" style={{ width: '100%', borderRadius: '12px', margin: '16px 0 20px' }} />
**How it works:**
<Steps>
<Step title="User submits a query">
The query is embedded and used to seed both vector search and graph traversal simultaneously.
</Step>
<Step title="Hybrid context retrieval">
Semantica retrieves relevant graph context — entities, typed relationships, and multi-hop reasoning paths — alongside vector-similar text chunks.
</Step>
<Step title="Context building">
Retrieved facts and reasoning paths are assembled into a structured prompt context, each fact tagged with its source node and confidence.
</Step>
<Step title="LLM generates a grounded response">
The LLM produces an answer where every claim links back to a source node in the graph — no floating assertions, no hallucinations from training data.
</Step>
</Steps>
1. User submits a query
2. Semantica retrieves relevant graph context — entities, relationships, reasoning paths
3. The LLM generates a response grounded in that context
4. Every claim in the response links back to a source node in the graph
This eliminates the hallucination and traceability problems of standard RAG.
<Tip>
**GraphRAG eliminates the hallucination and traceability problems of standard RAG.** Standard RAG retrieves text chunks; GraphRAG retrieves structured facts with typed relationships. The LLM cannot confabulate structure that was never in the graph.
</Tip>
## Ontology
@@ -104,7 +131,7 @@ ontology = {
}
```
Semantica can auto-generate ontologies from your knowledge graph or import existing OWL/RDF/Turtle ontologies. The **Ontology Hub** (v0.5.0) adds a visual editor, SHACL Studio, alignment authoring, and a live health dashboard.
Semantica can auto-generate ontologies from your knowledge graph or import existing OWL/RDF/Turtle ontologies. The **Ontology Hub** (v0.5.0) adds a visual editor, SHACL Studio, alignment authoring, and a live health dashboard. See the [Ontology reference](reference/ontology) for the full 6-stage generation pipeline.
## Reasoning & Inference
@@ -116,75 +143,165 @@ Known: Apple Inc. is headquartered in Cupertino
Inferred: Steve Jobs has a connection to Cupertino
```
| Engine | Description |
| ------ | ----------- |
| Forward chaining | Applies rules repeatedly until no new facts can be derived |
| Rete network | Efficient pattern matching for large rule sets |
| Deductive | Classical deductive reasoning |
| Abductive | Infers the most likely explanation |
| SPARQL | Query-based inference over RDF graphs |
| Datalog | Recursive Horn clause rules with fixpoint semantics (v0.4.0) |
<Tabs>
<Tab title="Forward Chaining">
Applies IF/THEN rules repeatedly until no new facts can be derived. Best for alert systems, compliance checks, and trigger-based workflows.
All engines produce **explainable inference paths**, not black-box conclusions.
```python
from semantica.reasoning import Reasoner, Rule, Fact, RuleType
engine = Reasoner()
engine.add_fact(Fact(subject="Alice", predicate="is_a", obj="Manager"))
engine.add_rule(Rule(
rule_type=RuleType.FORWARD_CHAIN,
conditions=[{"subject": "?x", "predicate": "is_a", "object": "Manager"}],
conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"}
))
result = engine.infer()
```
</Tab>
<Tab title="Rete Network">
Efficient pattern matching for large rule sets — the Rete algorithm avoids re-evaluating rules whose preconditions haven't changed. Best for thousands of rules over millions of facts.
```python
from semantica.reasoning import ReteEngine
engine = ReteEngine()
engine.load_rules("rules/domain_rules.json")
results = engine.run(kg)
```
</Tab>
<Tab title="Deductive & Abductive">
**Deductive** — classical syllogistic reasoning from premises to guaranteed conclusions.
**Abductive** — infers the most likely explanation for observed evidence. Best for diagnostic and investigative use cases.
```python
from semantica.reasoning import GraphReasoner
graph_reasoner = GraphReasoner(kg)
graph_reasoner.add_rule({"if": [{"subject": "?a", "predicate": "parent_of", "object": "?b"}], "then": {"subject": "?a", "predicate": "ancestor_of", "object": "?b"}})
inferences = graph_reasoner.infer(kg)
```
</Tab>
<Tab title="Datalog (v0.4.0)">
Recursive Horn clause rules with fixpoint semantics — handles transitive closure and recursive relationships that forward chaining cannot express.
```python
from semantica.reasoning import DatalogReasoner, DatalogFact, DatalogRule
reasoner = DatalogReasoner()
reasoner.add_fact(DatalogFact("parent", ("alice", "bob")))
reasoner.add_rule(DatalogRule("ancestor(?X, ?Y) :- parent(?X, ?Y)."))
reasoner.evaluate()
results = reasoner.query("ancestor(alice, ?Z)")
```
</Tab>
<Tab title="Engine Comparison">
| Engine | Description | Best For |
| ------ | ----------- | -------- |
| Forward chaining | Applies rules until fixpoint | Alert systems, compliance checks |
| Rete network | Efficient pattern matching | Large rule sets, high fact throughput |
| Deductive | Classical syllogistic reasoning | Mathematical and logical inference |
| Abductive | Most likely explanation | Diagnostics, investigation |
| SPARQL | Query-based inference over RDF | Semantic web, ontology reasoning |
| Datalog (v0.4.0) | Recursive Horn clause rules | Transitive closure, graph reachability |
</Tab>
</Tabs>
All engines produce **explainable inference paths** — not black-box conclusions. Every derived fact includes the rules and premises that produced it.
## Temporal Intelligence
Knowledge changes over time. Temporal graphs attach `valid_from` / `valid_until` windows to nodes and edges, enabling point-in-time queries and historical analysis.
```python
from semantica.kg import TemporalKnowledgeGraph
from semantica.kg import TemporalGraphQuery
from datetime import datetime
tkg = TemporalKnowledgeGraph()
tkg.add_node("ceo_role", valid_from=datetime(2020, 1, 1), valid_until=datetime(2023, 6, 1))
query_engine = TemporalGraphQuery(enable_temporal_reasoning=True)
# Query the graph as it existed on a specific date
snapshot = tkg.at(datetime(2021, 6, 15))
snapshot = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15))
```
**Features:** Allen interval algebra (all 13 relations), OWL-Time export, `recorded_at` stamping, temporal provenance.
**Supported features:** Allen interval algebra (all 13 temporal relations), OWL-Time export, `recorded_at` stamping, temporal provenance.
**Common uses:** tracking company leadership changes, policy evolution, research timelines, financial instrument histories.
**Common uses:** tracking company leadership changes, policy evolution, research timelines, financial instrument histories, regulatory compliance windows.
## Distance Intelligence
Explore the semantic neighborhood of any entity in your graph. Useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology.
Explore the semantic neighborhood of any entity in your graph — useful for understanding what's conceptually close, detecting clusters, and visualizing knowledge topology.
```python
from semantica.kg import DistanceCalculator
from semantica.kg import SimilarityCalculator
calc = DistanceCalculator(graph)
neighborhood = calc.semantic_neighborhood("Apple Inc.", radius=0.4)
matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"])
calc = SimilarityCalculator()
scores = calc.calculate_similarity(entity_a, entity_b)
```
**Features:** N×N distance matrices, ego-mode visualization, distance band classification (`near` / `mid` / `far`), embedding cache optimization.
**Features:** N×N semantic distance matrices, ego-mode visualization, distance band classification (`near` / `mid` / `far`), embedding cache optimization for large graphs.
The [Visualization module](reference/visualization) renders distance matrices as interactive heatmaps and ego-mode neighborhood graphs. The [Explorer](reference/explorer) embeds distance intelligence directly in the browser dashboard.
## Deduplication & Entity Resolution
Real-world data contains the same entity under many names — "Apple", "Apple Inc.", "Apple Computer Inc." Semantica's deduplication pipeline detects these, merges attributes, resolves conflicts, and preserves the original source provenance.
**Strategies:**
<Tabs>
<Tab title="Strategies">
- **v1** — Jaro-Winkler similarity, suitable for small datasets
- **`blocking_v2`** — candidate blocking for large corpora
- **`hybrid_v2`** — combines blocking with semantic matching
- **`semantic_v2`** — pure embedding-based resolution, up to 7x faster than v1
| Strategy | Algorithm | Best For |
| -------- | --------- | -------- |
| `v1` | Jaro-Winkler string similarity | Small datasets, fast baseline |
| `blocking_v2` | Candidate blocking + similarity | Large corpora — reduces O(n²) comparisons |
| `hybrid_v2` | Blocking + semantic embedding match | Mixed structured/unstructured entity names |
| `semantic_v2` | Pure embedding-based resolution | Up to 7× faster than v1; handles abbreviations and aliases |
</Tab>
<Tab title="Configuration">
```python
from semantica.deduplication import DuplicateDetector, EntityMerger
detector = DuplicateDetector(similarity_threshold=0.85)
duplicates = detector.detect_duplicates(entities)
merger = EntityMerger()
deduplicated_entities = merger.merge_duplicates(entities)
```
</Tab>
</Tabs>
## Provenance & Auditability
Every fact in Semantica links back to:
- The source document it came from
- The extraction method used
- The ontology rules applied
- The reasoning steps that produced any inference
- The **source document** it came from
- The **extraction method** used (pattern / ML / LLM)
- The **ontology rules** applied during graph construction
- The **reasoning steps** that produced any inferred fact
This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11).
<Note>
This is W3C PROV-O compliant lineage — suitable for regulated industries that require audit trails (HIPAA, SOX, GDPR, FDA 21 CFR Part 11). Use `RDFExporter(include_provenance=True)` to embed provenance inline in any RDF export.
</Note>
```python
from semantica.provenance import ProvenanceManager
prov = ProvenanceManager()
lineage = prov.get_entity_lineage("apple_inc")
print(f"Source: {lineage.source_document}")
print(f"Method: {lineage.extraction_method}")
print(f"Extracted: {lineage.timestamp}")
print(f"Checksum: {lineage.checksum}")
```
## Decision Intelligence
Every agent decision is a first-class object in Semantica — recorded, causally linked, and searchable by precedent.
Every agent decision is a first-class object in Semantica — recorded, causally linked, and searchable by precedent. This is the **accountability layer** for AI pipelines: decisions are no longer ephemeral log messages, they are queryable knowledge graph nodes.
```python
decision_id = context.record_decision(
@@ -195,11 +312,16 @@ decision_id = context.record_decision(
confidence=0.91,
)
# Find similar past decisions before making a new one
precedents = context.find_precedents("model selection reasoning", limit=5)
# Trace downstream impact of a past decision
influence = context.analyze_decision_influence(decision_id)
```
This prevents inconsistent decisions, enables audits, and lets agents learn from their own history.
<Tip>
**Use `find_precedents()` before every high-stakes decision.** Hybrid similarity search over all recorded decisions surfaces past reasoning that may apply — reducing inconsistency across agent runs and enabling genuine organisational learning from AI decision history.
</Tip>
## Conflict Detection
@@ -207,10 +329,78 @@ When multiple sources disagree on the same fact, Semantica flags and resolves th
**Resolution strategies:**
- Prefer the most recent source
- Prefer the most reliable source
- Majority vote across sources
- Flag for manual review
- **Recency** — prefer the most recent source
- **Source credibility** — prefer the most reliable source (configurable credibility scores)
- **Majority vote** — aggregate across all sources with ≥ 2 agreeing
- **Manual review** — flag for human arbitration; continue pipeline without blocking
See the [Conflicts reference](reference/conflicts) for `ConflictResolver`, `SourceTracker`, and `InvestigationGuideGenerator`.
## Custom Plugin Development
Semantica is designed for extension. Any component — ingestor, extractor, graph builder, reasoning engine — can be replaced or augmented with a custom implementation registered at runtime.
<AccordionGroup>
<Accordion title="PluginRegistry — replace any component by name">
`PluginRegistry` provides dynamic plugin discovery, registration, and loading across all modules. Register your own class under a string key; Semantica will use it wherever that key is referenced in config or pipeline steps.
```python
from semantica.core import PluginRegistry
registry = PluginRegistry()
# Register a custom ingestor
registry.register_plugin(
"my_sql_ingestor", MySQLIngestor,
version="1.0.0",
description="PostgreSQL ingestor for internal warehouse",
capabilities=["ingest"],
)
# Load and use
plugin = registry.load_plugin("my_sql_ingestor", connection_string="postgresql://...")
result = plugin.execute("SELECT * FROM documents")
# Reference by name in pipeline YAML — no code changes needed
```
```yaml
steps:
- name: ingest
plugin: my_sql_ingestor
config:
connection_string: "${DB_URL}"
```
**Extension points available:** ingestors, parsers, normalizers, extractors, reasoning engines, export formats, vector store backends, graph store backends, visualization renderers.
</Accordion>
<Accordion title="MethodRegistry — add domain-specific graph operations">
`MethodRegistry` lets you register custom methods on knowledge graph objects by name — useful for adding domain-specific graph operations without subclassing.
```python
from semantica.kg import MethodRegistry
registry = MethodRegistry()
def find_supply_chain_hops(graph, source_node, max_hops=3):
"""Custom BFS traversal for supply chain graphs."""
...
# Register under a string key
registry.register("supply_chain_hops", find_supply_chain_hops)
# Call by name on any graph object
result = registry.call("supply_chain_hops", kg, source_node="Supplier_A", max_hops=5)
# List all registered methods
print(registry.list_methods()) # ["supply_chain_hops", ...]
```
</Accordion>
</AccordionGroup>
<CardGroup cols={2}>
<Card title="Quickstart Tutorial" icon="play" href="quickstart">
docs/index.md
+101 -43
View File
@@ -13,11 +13,23 @@ description: "The Accountability and Context Layer for AI — Context Graphs ·
AI agents today are powerful but not trustworthy. Five structural gaps make them impossible to deploy in regulated environments:
- **No memory structure.** Agents store embeddings, not meaning. There's no way to ask *why* something was recalled or trace a fact to its source.
- **No decision trail.** Agents act continuously but record nothing. When something breaks, there's no history to debug or audit.
- **No provenance.** Outputs can't be traced back to source facts. In healthcare, finance, and legal, this is a hard compliance blocker.
- **No reasoning transparency.** Black-box answers with zero explanation of how a conclusion was reached.
- **No conflict detection.** Contradictory facts silently coexist in vector stores, producing unpredictable and inconsistent outputs.
<CardGroup cols={2}>
<Card title="No memory structure" icon="brain">
Agents store embeddings, not meaning. There's no way to ask *why* something was recalled or trace a fact to its source.
</Card>
<Card title="No decision trail" icon="clock-rotate-left">
Agents act continuously but record nothing. When something breaks, there's no history to debug or audit.
</Card>
<Card title="No provenance" icon="link-slash">
Outputs can't be traced back to source facts. In healthcare, finance, and legal, this is a hard compliance blocker.
</Card>
<Card title="No reasoning transparency" icon="eye-slash">
Black-box answers with zero explanation of how a conclusion was reached.
</Card>
<Card title="No conflict detection" icon="triangle-exclamation">
Contradictory facts silently coexist in vector stores, producing unpredictable and inconsistent outputs.
</Card>
</CardGroup>
These aren't edge cases. They're why AI cannot be deployed in healthcare, finance, legal, and government without custom guardrails built from scratch.
@@ -25,12 +37,26 @@ These aren't edge cases. They're why AI cannot be deployed in healthcare, financ
Semantica is the **accountability and context layer** you add on top of your existing AI stack. Not a replacement for LangChain or LlamaIndex — the infrastructure that makes their outputs trustworthy.
- **Context Graphs** — a structured, queryable graph of everything your agent knows, decides, and reasons about. Persistent across runs.
- **Decision Intelligence** — every decision is a first-class object: recorded, causally linked, searchable by precedent, and analyzable for downstream impact.
- **Full Provenance** — every fact links back to its source. W3C PROV-O compliant. Full lineage from ingestion to inference.
- **Reasoning Engines** — forward chaining, Rete, deductive, abductive, SPARQL, Datalog. Explainable paths, not black boxes.
- **Temporal Intelligence** — point-in-time queries, Allen interval algebra, temporal provenance, OWL-Time export.
- **Ontology Hub** — visual editor, SHACL Studio, alignment authoring, health dashboard. Full ontology lifecycle in the browser.
<CardGroup cols={2}>
<Card title="Context Graphs" icon="diagram-project">
A structured, queryable graph of everything your agent knows, decides, and reasons about. Persistent across runs.
</Card>
<Card title="Decision Intelligence" icon="check-circle">
Every decision is a first-class object: recorded, causally linked, searchable by precedent, and analyzable for downstream impact.
</Card>
<Card title="Full Provenance" icon="shield-check">
Every fact links back to its source. W3C PROV-O compliant. Full lineage from ingestion to inference.
</Card>
<Card title="Reasoning Engines" icon="microchip">
Forward chaining, Rete, deductive, abductive, SPARQL, Datalog. Explainable paths, not black boxes.
</Card>
<Card title="Temporal Intelligence" icon="clock">
Point-in-time queries, Allen interval algebra, temporal provenance, OWL-Time export.
</Card>
<Card title="Ontology Hub" icon="sitemap">
Visual editor, SHACL Studio, alignment authoring, health dashboard. Full ontology lifecycle in the browser.
</Card>
</CardGroup>
Works alongside any LLM provider and any agent framework.
@@ -47,13 +73,13 @@ pip install semantica
```python OpenAI
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
from semantica.llms import OpenAIProvider
from semantica.llms import OpenAI
context = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=1536),
knowledge_graph=ContextGraph(advanced_analytics=True),
decision_tracking=True,
llm=OpenAIProvider(model="gpt-4o"),
llm=OpenAI(model="gpt-4o"),
)
context.store("GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%")
@@ -73,13 +99,14 @@ influence = context.analyze_decision_influence(decision_id)
```python Anthropic
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
from semantica.llms import AnthropicProvider
from semantica.llms import LiteLLM
import os
context = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=1024),
knowledge_graph=ContextGraph(advanced_analytics=True),
decision_tracking=True,
llm=AnthropicProvider(model="claude-opus-4-7"),
llm=LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY")),
)
context.store("Claude excels at long-context reasoning and code generation")
@@ -98,13 +125,13 @@ precedents = context.find_precedents("document analysis model", limit=5)
```python Ollama (Local)
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
from semantica.llms import OllamaProvider
from semantica.llms import LiteLLM
context = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768),
knowledge_graph=ContextGraph(advanced_analytics=True),
decision_tracking=True,
llm=OllamaProvider(model="llama3.2", base_url="http://localhost:11434"),
llm=LiteLLM(model="ollama/llama3.2", base_url="http://localhost:11434"),
)
# Fully local — no data leaves your infrastructure
@@ -173,7 +200,23 @@ pip install semantica==0.5.0
## Start Here
If you're new to Semantica, install first and then open [Quickstart](quickstart). Use [Core Concepts](concepts) for the mental model, or jump to [API Reference](reference/context) when you need exact details.
<Steps>
<Step title="Install">
```bash
pip install semantica
```
See [Installation](installation) for optional extras and environment setup.
</Step>
<Step title="Follow the Quickstart">
Build a complete knowledge graph pipeline — ingest, extract, build, query — in [5 minutes](quickstart).
</Step>
<Step title="Learn the mental model">
[Core Concepts](concepts) explains knowledge graphs, GraphRAG, provenance, and decision intelligence. Read this before the API reference.
</Step>
<Step title="Go deep on any module">
Every module has a dedicated [reference page](reference/context) with class docs, parameter tables, and runnable examples.
</Step>
</Steps>
<CardGroup cols={2}>
<Card title="Installation" icon="download" href="installation">
@@ -255,45 +298,60 @@ If you're new to Semantica, install first and then open [Quickstart](quickstart)
| `semantica.mcp_server` | MCP stdio server — 12 tools for Claude Desktop, VS Code, Cursor, Windsurf, Cline |
| `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector |
| `semantica.graph_store` | Neo4j, FalkorDB, Apache AGE, Amazon Neptune |
| `semantica.triplet_store` | In-memory and persistent RDF triple store |
| `semantica.triplet_store` | In-memory and persistent RDF triple store with SPARQL |
| `semantica.ingest` | Files, web, feeds, databases, Snowflake, Parquet, XML, MCP |
| `semantica.parse` | Document parsing — PDF, DOCX, HTML, PPTX, Docling layout analysis |
| `semantica.split` | Text chunking — sentence, paragraph, token, semantic boundary strategies |
| `semantica.normalize` | Text normalization, entity canonicalization, whitespace and encoding cleanup |
| `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE |
| `semantica.embeddings` | Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama local embeddings |
| `semantica.pipeline` | Pipeline DSL, parallel workers, retry policies, failure handling |
| `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, OWL, graph formats |
| `semantica.visualization` | Programmatic graph rendering — force, hierarchical, circular layouts |
| `semantica.export` | RDF, Parquet, ArangoDB AQL, CSV, OWL, Arrow, GraphML, GEXF, DOT |
| `semantica.visualization` | Programmatic graph rendering — force, hierarchical, circular, spring layouts |
| `semantica.deduplication` | Entity deduplication v1/v2, similarity scoring, blocking, merging |
| `semantica.conflicts` | Conflict detection and resolution across overlapping knowledge sources |
| `semantica.provenance` | W3C PROV-O lineage tracking, source attribution, audit trails |
| `semantica.change_management` | Version control with SHA-256 checksums, diff, rollback |
| `semantica.llms` | Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM |
| `semantica.seed` | Deterministic data seeding and synthetic graph generation for tests |
| `semantica.evals` | Evaluation harness — precision, recall, F1 for extraction and reasoning |
| `semantica.core` | Core data models, base classes, shared type definitions |
| `semantica.utils` | Shared utilities — ID generation, date parsing, schema helpers |
| `semantica.llms` | Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, HuggingFace |
| `semantica.seed` | Foundation graph seeding from CSV, JSON, SQL, API, and RDF sources |
| `semantica.evals` | Evaluation harness — KG quality, extraction F1, pipeline benchmarking, regression tracking |
| `semantica.core` | Orchestration, ConfigManager, LifecycleManager, PluginRegistry, MethodRegistry |
| `semantica.utils` | Logging, validation, progress tracking, hash utilities, nested dict helpers |
## Built for High-Stakes Domains
Where every decision must be accountable and mistakes have real consequences:
**Healthcare & Life Sciences** — clinical decision support, drug interaction graphs, patient safety audit trails, HIPAA compliance.
**Finance & Risk** — fraud detection graphs, SOX/GDPR/MiFID II compliance, risk assessment trails.
**Legal & Compliance** — evidence-backed research, contract analysis, regulatory change tracking.
**Cybersecurity** — threat attribution graphs, incident response timelines, security audit trails.
**Government & Defense** — policy decision trails, classified information handling, provenance chains.
**Critical Infrastructure** — power grids, transportation safety, emergency response coordination.
<CardGroup cols={2}>
<Card title="Healthcare & Life Sciences" icon="heart-pulse">
Clinical decision support, drug interaction graphs, patient safety audit trails, HIPAA compliance.
</Card>
<Card title="Finance & Risk" icon="chart-line">
Fraud detection graphs, SOX/GDPR/MiFID II compliance, risk assessment trails.
</Card>
<Card title="Legal & Compliance" icon="scale-balanced">
Evidence-backed research, contract analysis, regulatory change tracking.
</Card>
<Card title="Cybersecurity" icon="shield">
Threat attribution graphs, incident response timelines, security audit trails.
</Card>
<Card title="Government & Defense" icon="building-columns">
Policy decision trails, classified information handling, provenance chains.
</Card>
<Card title="Critical Infrastructure" icon="bolt">
Power grids, transportation safety, emergency response coordination.
</Card>
</CardGroup>
## Why Semantica?
**Open source, MIT licensed.** No vendor lock-in, no paywalled features. Every line of code is available and forkable.
**Production ready.** 1,000+ passing tests, `PipelineValidator`, `FailureHandler` with exponential backoff, conflict resolution, and 12 security fixes in v0.5.0.
**Modular by design.** Import only what you need. Use `NERExtractor` without a graph store. Use `VectorStore` without decision tracking. Every component is independently swappable.
<CardGroup cols={3}>
<Card title="Open Source, MIT" icon="code-branch">
No vendor lock-in, no paywalled features. Every line of code is available and forkable.
</Card>
<Card title="Production Ready" icon="circle-check">
1,000+ passing tests, `PipelineValidator`, `FailureHandler` with exponential backoff, 12 security fixes in v0.5.0.
</Card>
<Card title="Modular by Design" icon="puzzle-piece">
Import only what you need. Use `NERExtractor` without a graph store. Every component is independently swappable.
</Card>
</CardGroup>
docs/modules.md
+70 -42
View File
@@ -134,21 +134,20 @@ triplets = trip.extract(text)
Graph construction, graph algorithms, temporal model, and distance intelligence.
```python
from semantica.kg import GraphBuilder, GraphAnalyzer, TemporalKnowledgeGraph, DistanceCalculator
from semantica.kg import GraphBuilder, GraphAnalyzer, TemporalGraphQuery, SimilarityCalculator
from datetime import datetime
# Build
builder = GraphBuilder(merge_entities=True)
kg = builder.build(entities=entities, relationships=relationships)
# Temporal graphs (v0.4.0)
tkg = TemporalKnowledgeGraph()
tkg.add_node("ceo_role", valid_from=datetime(2020, 1, 1), valid_until=datetime(2023, 6, 1))
snapshot = tkg.at(datetime(2021, 6, 15))
query_engine = TemporalGraphQuery(enable_temporal_reasoning=True)
snapshot = query_engine.query_at_time(kg, query="", at_time=datetime(2021, 6, 15))
# Distance Intelligence (v0.5.0)
calc = DistanceCalculator(kg)
neighborhood = calc.semantic_neighborhood("Apple Inc.", radius=0.4)
matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"])
# Semantic similarity (v0.5.0)
calc = SimilarityCalculator()
scores = calc.calculate_similarity(entity_a, entity_b)
```
**Graph algorithms available:** centrality calculation, community detection, connectivity analysis, entity resolution, link prediction, path finding, similarity calculation
@@ -158,29 +157,29 @@ matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"])
Schema management including SHACL, SKOS, alignments, diff/migration, auto-generation, and the visual Ontology Hub (v0.5.0).
```python
from semantica.ontology import OntologyManager, SHACLGenerator
from semantica.ontology import OntologyGenerator, SHACLGenerator
ontology = OntologyManager()
ontology.add_class("Person", ["name", "birth_date"])
ontology.add_relationship("works_for", "Person", "Organization")
is_valid = ontology.validate_graph(kg)
generator = OntologyGenerator()
ontology = generator.generate_from_graph(kg)
shacl = SHACLGenerator()
shapes = shacl.generate(ontology)
```
**Components:** `OntologyManager`, `SHACLGenerator`, `OntologyGenerator`, `OntologyValidator`, `OntologyEvaluator`, `LLMGenerator`, `OWLGenerator`, `PropertyGenerator`, `DomainOntologies`, `NamespaceManager`
**Components:** `OntologyGenerator`, `SHACLGenerator`, `OntologyValidator`, `OntologyEvaluator`, `LLMOntologyGenerator`, `OWLGenerator`, `PropertyGenerator`, `DomainOntologies`, `NamespaceManager`
### Reasoning
Derives new facts from existing knowledge using multiple inference strategies.
```python
from semantica.reasoning import ReasoningEngine, DatalogEngine
from semantica.reasoning import Reasoner, DatalogReasoner
# Rule-based reasoning
engine = ReasoningEngine()
inferences = engine.infer(kg, rules=["transitivity", "symmetry"])
engine = Reasoner()
engine.apply_transitivity("located_in")
engine.apply_symmetry("knows")
result = engine.infer()
# Datalog — recursive Horn clause rules (v0.4.0)
datalog = DatalogEngine()
@@ -403,9 +402,8 @@ result = pipeline.run("data/")
FastAPI Knowledge Explorer with Ontology Hub, WebSocket progress, bidirectional path finding, and indexed search (0.004ms on 118k nodes).
```python
from semantica.explorer import start_explorer
start_explorer(graph=kg, port=8080)
# Launch via CLI
# semantica explore --port 8080
# Opens at http://localhost:8080
```
@@ -418,11 +416,13 @@ start_explorer(graph=kg, port=8080)
Unified interface to all supported LLM providers.
```python
from semantica.llms import Groq, OpenAI, create_provider
from semantica.llms import Groq, OpenAI, LiteLLM
import os
llm = Groq(model="llama-3.3-70b-versatile")
llm = OpenAI(model="gpt-4o")
llm = create_provider("anthropic", model="claude-opus-4-7")
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
llm = OpenAI(model="gpt-4o", api_key=os.getenv("OPENAI_API_KEY"))
# Anthropic, Gemini, Ollama, DeepSeek via LiteLLM:
llm = LiteLLM(model="anthropic/claude-opus-4-7", api_key=os.getenv("ANTHROPIC_API_KEY"))
```
**Supported providers:** OpenAI, Anthropic, Google Gemini, Groq, Ollama, DeepSeek, Novita AI, LiteLLM (20+ models via one interface)
@@ -439,41 +439,69 @@ python -m semantica.mcp_server
### Seed
Deterministic data seeding for testing and development.
Bootstrap knowledge graphs from verified structured sources — fixed-point reference data, controlled vocabularies, and domain anchors.
```python
from semantica.seed import SeedManager
seed = SeedManager()
seed.populate(kg, dataset="companies", count=100)
# Load domain seeds from file or built-in datasets
seed.load_from_file("seed_data/industries.json")
seed.inject(kg) # merges seed nodes without duplicating existing entities
```
**Use cases:** anchoring extraction with known entities, pre-populating ontology classes, deterministic test graph generation.
### Evals
Evaluation harness for extraction and reasoning quality.
Evaluation framework for measuring KG quality, extraction accuracy, and pipeline performance.
```python
from semantica.evals import Evaluator
from semantica.evals import KGEvaluator, ExtractionEvaluator, PipelineEvaluator, RegressionTracker
evaluator = Evaluator()
scores = evaluator.evaluate(predicted_entities, ground_truth)
# Returns: {"precision": 0.91, "recall": 0.87, "f1": 0.89}
# KG quality
report = KGEvaluator().evaluate(kg, ontology=ontology)
print(f"Completeness: {report.completeness:.2%} Consistency: {report.consistency:.2%}")
# Extraction accuracy
report = ExtractionEvaluator().evaluate_ner(predictions=extracted, gold_standard=annotated)
print(f"Precision: {report.precision:.3f} Recall: {report.recall:.3f} F1: {report.f1:.3f}")
# Pipeline throughput and latency
metrics = PipelineEvaluator().benchmark(pipeline, data="data/", bench_runs=5)
print(f"Throughput: {metrics.docs_per_second:.1f} docs/sec")
# Regression tracking across runs
tracker = RegressionTracker(db_path="eval_history.db")
run_id = tracker.record_run(pipeline_version="v1.2.0", metrics=metrics)
diff = tracker.compare(run_id, baseline_run_id="run_abc123")
```
**Metrics:** precision, recall, F1 for NER, relation extraction, and reasoning
**Components:** `KGEvaluator`, `ExtractionEvaluator`, `PipelineEvaluator`, `RegressionTracker`
### Core
Base classes, shared data models, and the plugin registry used across all modules.
```python
from semantica.core import Orchestrator, PluginRegistry
from semantica.core import Semantica, PluginRegistry, ConfigManager
# Top-level orchestrator
sem = Semantica(config_path="config.yaml")
sem.initialize()
# Plugin registry — register custom components
registry = PluginRegistry()
registry.register("my_ingestor", MyCustomIngestor)
# Config management
config = ConfigManager(config_path="config.yaml")
batch = config.get("processing.batch_size", default=32)
```
**Components:** `Orchestrator`, `PluginRegistry`, `ConfigManager`, `Lifecycle`
**Components:** `Semantica`, `PluginRegistry`, `ConfigManager`, `LifecycleManager`, `HealthMonitor`, `Config`
### Utils
@@ -504,16 +532,16 @@ from semantica.utils import helpers, validators, logging
| [ingest](reference/ingest) | Data ingestion | `FileIngestor`, `WebIngestor`, `ParquetIngestor`, `XMLIngestor` |
| [parse](reference/parse) | Document parsing | `DocumentParser`, `DoclingParser` |
| [split](reference/split) | Text chunking | `TextSplitter` |
| [normalize](reference/normalize) | Data cleaning | `DataNormalizer` |
| [semantic_extract](reference/semantic_extract) | NER & relation extraction | `NERExtractor`, `RelationExtractor`, `TripletExtractor` |
| [kg](reference/kg) | Graph construction | `GraphBuilder`, `TemporalKnowledgeGraph`, `DistanceCalculator` |
| [ontology](reference/ontology) | Schema management | `OntologyManager`, `SHACLGenerator` |
| [reasoning](reference/reasoning) | Logical inference | `ReasoningEngine`, `DatalogEngine` |
| [normalize](reference/normalize) | Data cleaning | `TextNormalizer`, `EntityNormalizer`, `LanguageDetector` |
| [semantic_extract](reference/semantic_extract) | NER & relation extraction | `NERExtractor`, `RelationExtractor`, `TripletExtractor`, `SemanticAnalyzer`, `SemanticNetworkExtractor`, `ExtractionValidator` |
| [kg](reference/kg) | Graph construction | `GraphBuilder`, `TemporalGraphQuery`, `SimilarityCalculator` |
| [ontology](reference/ontology) | Schema management | `OntologyGenerator`, `SHACLGenerator` |
| [reasoning](reference/reasoning) | Logical inference | `Reasoner`, `DatalogReasoner` |
| [embeddings](reference/embeddings) | Vector embeddings | `EmbeddingGenerator` |
| [vector_store](reference/vector_store) | Vector database | `VectorStore` |
| [graph_store](reference/graph_store) | Graph database | `GraphStore` |
| [triplet_store](reference/triplet_store) | RDF triple store | `TripletStore` |
| [deduplication](reference/deduplication) | Entity resolution | `EntityResolver`, `DuplicateDetector` |
| [deduplication](reference/deduplication) | Entity resolution | `EntityResolver`, `DuplicateDetector`, `ClusterBuilder`, `MergeStrategyManager` |
| [conflicts](reference/conflicts) | Conflict resolution | `ConflictDetector` |
| [context](reference/context) | Agent context & decisions | `AgentContext`, `ContextGraph` |
| [provenance](reference/provenance) | W3C PROV-O lineage | `ProvenanceManager` |
@@ -524,9 +552,9 @@ from semantica.utils import helpers, validators, logging
| [explorer](reference/explorer) | Knowledge Explorer UI | `start_explorer` |
| [llms](reference/llms) | LLM providers | `Groq`, `OpenAI`, `create_provider` |
| [mcp_server](reference/mcp_server) | MCP stdio server | `python -m semantica.mcp_server` |
| [seed](reference/seed) | Test data seeding | `SeedManager` |
| [evals](reference/evals) | Quality evaluation | `Evaluator` |
| [core](reference/core) | Base classes & registry | `Orchestrator`, `PluginRegistry` |
| [seed](reference/seed) | KG bootstrapping from structured sources | `SeedManager` |
| [evals](reference/evals) | Quality evaluation | `KGEvaluator`, `ExtractionEvaluator`, `PipelineEvaluator`, `RegressionTracker` |
| [core](reference/core) | Base classes & registry | `Semantica`, `ConfigManager`, `PluginRegistry`, `LifecycleManager` |
| [utils](reference/utils) | Shared utilities | `helpers`, `validators` |
<CardGroup cols={2}>
docs/reference/change_management.md
+248 -67
View File
@@ -4,38 +4,116 @@ description: "Version control, SHA-256 checksums, diff analysis, rollback, and a
icon: "clock-rotate-left"
---
`semantica.change_management` provides enterprise-grade versioning and audit trails for knowledge graphs and ontologies — SHA-256 checksums, snapshot history, diff analysis, rollback protection, and compliance-ready audit export (HIPAA, SOX, FDA 21 CFR Part 11).
`semantica.change_management` provides enterprise-grade versioning and audit trails for knowledge graphs and ontologies. Every snapshot carries a SHA-256 checksum, every modification is logged, and every state can be diffed or rolled back — giving you a complete, tamper-evident record suitable for regulated industries.
<Note>
Compliance frameworks supported out of the box: **HIPAA**, **SOX**, **GDPR**, and **FDA 21 CFR Part 11**.
</Note>
## Exported Classes
```python
from semantica.change_management import (
# Change metadata
ChangeLogEntry, # snapshot record: version, author, message, checksum, changes
# Storage backends
VersionStorage, # abstract storage interface
InMemoryVersionStorage, # fast in-memory backend (dev/test only)
SQLiteVersionStorage, # persistent SQLite backend (production)
# Integrity utilities
compute_checksum, # SHA-256 checksum of a graph state
verify_checksum, # verify graph against a stored checksum
# Version managers
TemporalVersionManager, # KG version management: snapshot, diff, rollback
OntologyVersionManager, # ontology version management
BaseVersionManager, # base class for custom version managers
# Ontology versioning (moved from ontology module)
VersionManager, # OWL ontology version control
OntologyVersion, # ontology version metadata dataclass
)
```
## What You Get
- **`TemporalVersionManager`** — snapshot, diff, rollback, and audit trail for knowledge graphs
- **`OntologyVersionManager`** — version control for OWL ontologies with diff and migration support
- **`VersionStorage`** — pluggable storage: `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production
- **`compute_checksum` / `verify_checksum`** — SHA-256 integrity verification
- **`ChangeLogEntry`** — structured record of every change in a snapshot
<CardGroup cols={2}>
<Card title="TemporalVersionManager" icon="code-branch">
Snapshot, diff, rollback, and per-entity audit trail for knowledge graphs.
</Card>
<Card title="OntologyVersionManager" icon="sitemap">
Version control for OWL ontologies with diff and schema migration support.
</Card>
<Card title="VersionStorage" icon="database">
Pluggable backends — `InMemoryVersionStorage` for tests, `SQLiteVersionStorage` for production.
</Card>
<Card title="Integrity Verification" icon="shield-check">
SHA-256 / SHA-512 checksums to detect any unauthorised graph modification.
</Card>
<Card title="ChangeLogEntry" icon="list-check">
Structured record of every change: author, timestamp, checksum, and change list.
</Card>
<Card title="Version History" icon="file-shield">
Full tamper-evident version history via `list_versions()` and `diff()` for regulatory review.
</Card>
</CardGroup>
## Typical Workflow
<Steps>
<Step title="Initialise the version manager">
```python
from semantica.change_management import TemporalVersionManager
manager = TemporalVersionManager(storage_path="versions.db")
```
</Step>
<Step title="Snapshot before every destructive operation">
```python
snapshot_id = manager.create_snapshot(
graph=kg,
version="v1.0",
author="user@example.com",
message="Before deduplication run"
)
print(f"Snapshot: {snapshot_id}")
print(f"Checksum: {manager.get_checksum(snapshot_id)}")
```
</Step>
<Step title="Make your changes">
Run deduplication, conflict resolution, merges, or any graph modification. The version manager tracks nothing automatically — you control when snapshots are taken.
</Step>
<Step title="Snapshot the result">
```python
snapshot_v2 = manager.create_snapshot(
graph=kg,
version="v2.0",
author="user@example.com",
message="After deduplication — 1 342 duplicates merged"
)
```
</Step>
<Step title="Diff to review what changed">
```python
diff = manager.diff("v1.0", "v2.0")
print(diff.summary)
for change in diff.changes:
print(f" [{change.type}] {change.element}: {change.description}")
```
</Step>
</Steps>
## TemporalVersionManager
Version control for knowledge graphs — snapshot, diff, and rollback:
Version control for knowledge graphs — snapshot, diff, and rollback.
```python
from semantica.change_management import TemporalVersionManager
### Constructor Parameters
manager = TemporalVersionManager(storage_path="versions.db")
| Parameter | Type | Default | Description |
| --------- | ---- | ------- | ----------- |
| `storage_path` | `str` | `None` | Path to SQLite database; uses in-memory if omitted |
| `storage` | `VersionStorage` | `None` | Explicit storage backend instance — overrides `storage_path` |
# Create a snapshot
snapshot_id = manager.create_snapshot(
graph=kg,
version="v1.0",
author="user@example.com",
message="Initial knowledge graph"
)
print(f"Snapshot: {snapshot_id}")
print(f"Checksum: {manager.get_checksum(snapshot_id)}")
```
### List, Retrieve, and Rollback
### List and Retrieve
```python
# List all versions
@@ -45,33 +123,45 @@ for v in versions:
# Retrieve a specific version
kg_v1 = manager.get_version("v1.0")
# Rollback to a previous version (safe by default — fails if data would be lost)
manager.rollback(target_version="v1.0", allow_data_loss=False)
```
### Constructor Parameters
| Parameter | Type | Default | Description |
| --------- | ---- | ------- | ----------- |
| `storage_path` | `str` | `None` | Path to SQLite database; uses in-memory if omitted |
| `storage` | `VersionStorage` | `None` | Explicit storage backend instance |
## Diff Analysis
Compare any two snapshots to see exactly what changed:
Compare any two snapshots to see exactly what changed — useful for code review, incident investigation, and regulatory audit:
```python
diff = manager.diff("v1.0", "v2.0")
print(f"Added nodes: {len(diff.added_nodes)}")
print(f"Removed nodes: {len(diff.removed_nodes)}")
print(f"Modified nodes: {len(diff.modified_nodes)}")
print(f"Added edges: {len(diff.added_edges)}")
print(f"Removed edges: {len(diff.removed_edges)}")
print(f"Modified edges: {len(diff.modified_edges)}")
for change in diff.changes:
print(f" [{change.type}] {change.element}: {change.description}")
```
<Accordion title="DiffResult schema">
```python
@dataclass
class DiffResult:
from_version: str # source snapshot ID
to_version: str # target snapshot ID
added_nodes: List[str] # IDs of newly added entities
removed_nodes: List[str] # IDs of deleted entities
modified_nodes: List[str] # IDs of entities with changed properties
added_edges: List[str] # IDs of newly added relationships
removed_edges: List[str] # IDs of deleted relationships
modified_edges: List[str] # IDs of relationships with changed properties
changes: List[ChangeRecord] # ordered list of all individual changes
summary: str # human-readable summary line
```
</Accordion>
## OntologyVersionManager
Version control for OWL ontologies — save, diff, and track schema migrations:
@@ -97,21 +187,38 @@ for change in diff.changes:
## VersionStorage Backends
```python
from semantica.change_management import (
InMemoryVersionStorage,
SQLiteVersionStorage,
)
<Tabs>
<Tab title="SQLite (production)">
```python
from semantica.change_management import SQLiteVersionStorage, TemporalVersionManager
# In-memory — for tests and development (data not persisted)
storage = InMemoryVersionStorage()
storage = SQLiteVersionStorage(db_path="versions.db")
manager = TemporalVersionManager(storage=storage)
```
# SQLite — for production (persistent across restarts)
storage = SQLiteVersionStorage(db_path="versions.db")
Persists all version history to disk. Survives process restarts. Recommended for any environment where you need to retain the audit trail.
# Pass to a version manager
manager = TemporalVersionManager(storage=storage)
```
You can also pass the path directly to `TemporalVersionManager`:
```python
manager = TemporalVersionManager(storage_path="versions.db")
```
</Tab>
<Tab title="In-Memory (tests)">
```python
from semantica.change_management import InMemoryVersionStorage, TemporalVersionManager
storage = InMemoryVersionStorage()
manager = TemporalVersionManager(storage=storage)
```
Fast and zero-setup. Data is **not persisted** — all version history is lost when the process exits. Use this for unit tests and development only.
</Tab>
</Tabs>
<Warning>
The default `TemporalVersionManager()` with no arguments uses in-memory storage. Always pass `storage_path="versions.db"` or an explicit `SQLiteVersionStorage` in production — otherwise your entire version history disappears on restart.
</Warning>
## Integrity Verification
@@ -130,38 +237,112 @@ if not is_valid:
raise RuntimeError("Graph has been modified since the checksum was recorded")
```
## Audit Trail
Full per-entity audit trail with CSV and JSON export for compliance reporting:
```python
# Get all changes for a specific entity
trail = manager.get_audit_trail(entity_id="apple_inc")
for entry in trail:
print(f"{entry.timestamp} — {entry.author}: {entry.action} — {entry.description}")
# Export audit trail for compliance
manager.export_audit_trail("audit.csv", format="csv")
manager.export_audit_trail("audit.json", format="json")
```
<Tip>
`verify_checksum` is deterministic — the same graph always produces the same digest. Use it as a pre-flight check before any compliance export to confirm the graph hasn't been tampered with since the last snapshot.
</Tip>
## ChangeLogEntry
Every version snapshot includes a structured `ChangeLogEntry`:
Every version snapshot includes a structured `ChangeLogEntry` that records the full context of a change:
```python
from semantica.change_management import ChangeLogEntry
entry: ChangeLogEntry = manager.get_log_entry(snapshot_id)
# Retrieve a version entry
entry = manager.get_version("v1.0")
print(entry.version) # "v1.0"
print(entry.author) # "user@example.com"
print(entry.message) # "Initial knowledge graph"
print(entry.checksum) # SHA-256 hex digest
print(entry.created_at) # datetime
print(entry.changes) # list of individual change records
print(entry.checksum) # SHA-256 hex digest of the full graph state
print(entry.created_at) # datetime of snapshot creation
print(entry.node_count) # total nodes at this snapshot
print(entry.edge_count) # total edges at this snapshot
print(entry.changes) # list[ChangeRecord] — individual property-level changes
```
<Accordion title="ChangeLogEntry schema">
```python
@dataclass
class ChangeLogEntry:
snapshot_id: str # unique snapshot identifier
version: str # human-assigned version tag, e.g. "v1.0"
author: str # identity of the user or process that created it
message: str # commit-style description of what changed
checksum: str # SHA-256 hex digest — changes if graph is tampered
created_at: datetime # UTC timestamp of snapshot creation
node_count: int # total entity count at this point in time
edge_count: int # total relationship count at this point in time
changes: List[ChangeRecord] # granular per-property change records
metadata: Dict # arbitrary key-value pairs for custom tagging
```
</Accordion>
## Compliance and Version History
All version snapshots form a tamper-evident audit trail. Use `list_versions()` and `diff()` to reconstruct and review changes for regulatory purposes:
```python
from semantica.change_management import TemporalVersionManager
manager = TemporalVersionManager(storage_path="versions.db")
# Enumerate the full version history
for entry in manager.list_versions():
print(f"{entry.created_at.isoformat()} | {entry.author} | {entry.version} | {entry.message}")
# Diff any two snapshots for a change report
diff = manager.diff("v1.0", "v2.0")
print(f"Added: {len(diff.added_nodes)} | Removed: {len(diff.removed_nodes)} | Modified: {len(diff.modified_nodes)}")
for change in diff.changes:
print(f" [{change.type}] {change.element}: {change.description}")
```
Use `verify_checksum()` before any compliance export to confirm graph integrity:
```python
from semantica.change_management import verify_checksum
is_valid = verify_checksum(kg, expected_checksum=entry.checksum)
if not is_valid:
raise RuntimeError("Graph has been modified since the snapshot was taken")
```
### Compliance Coverage
<AccordionGroup>
<Accordion title="HIPAA — subject-access requests">
Use `get_audit_trail(entity_id="patient_001")` to retrieve every change ever made to a patient entity, then export to JSON for the access request response. The SHA-256 checksum on each entry proves the record has not been altered.
</Accordion>
<Accordion title="SOX — quarterly reviews">
Use `get_audit_trail(from_date=..., to_date=...)` to scope the export to the relevant quarter. Export to CSV for upload to your audit management system. The immutable snapshot chain provides the chain of custody required by SOX Section 404.
</Accordion>
<Accordion title="GDPR — right to erasure verification">
After deleting a data subject's entities, snapshot the graph and diff against the pre-deletion snapshot. `diff.removed_nodes` provides a machine-readable record of exactly what was deleted and when, satisfying Article 17 documentation requirements.
</Accordion>
<Accordion title="FDA 21 CFR Part 11 — electronic records">
Every `ChangeLogEntry` includes `author`, `timestamp`, and `checksum` — the three fields required for a compliant electronic record. `verify_checksum()` provides the tamper-evidence required by 21 CFR § 11.10(e).
</Accordion>
</AccordionGroup>
## Tips and Common Pitfalls
<Tip>
**Use `SQLiteVersionStorage` in production.** The default in-memory storage loses all version history when the process exits. Pass `storage_path="versions.db"` to `TemporalVersionManager` or create `SQLiteVersionStorage(db_path="versions.db")` explicitly.
</Tip>
<Warning>
**Snapshot before every destructive operation.** Call `manager.create_snapshot()` before running deduplication, conflict resolution, or merge operations. `rollback()` is only possible if a snapshot exists before the change.
</Warning>
<Tip>
**Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` produces a human-readable change summary in seconds — faster than comparing raw graph exports. Use it to review what changed before approving a version for production.
</Tip>
<Tip>
**Use `list_versions()` and `diff()` for compliance reviews.** `manager.list_versions()` enumerates the full version history and `manager.diff(v1, v2)` produces a machine-readable change report. Run `verify_checksum()` first to confirm the graph hasn't been modified since the snapshot was taken.
</Tip>
<CardGroup cols={2}>
<Card title="Provenance" icon="link" href="provenance">
W3C PROV-O lineage tracking.
docs/reference/conflicts.md
+334 -83
View File
@@ -6,74 +6,258 @@ icon: "triangle-exclamation"
`semantica.conflicts` detects and resolves contradictions when multiple sources disagree on the same fact. It surfaces five conflict types, seven resolution strategies, and generates investigation guides for manual review — so conflicts never silently corrupt your knowledge graph.
## Why Detect Conflicts?
When you ingest data from multiple sources, contradictions are inevitable. One annual report says Apple's revenue was $391B; a financial newswire says $383B. Without conflict detection, both values land in your graph and queries silently return inconsistent answers.
Semantica's conflict detection makes disagreements explicit and actionable:
- **Value conflicts** — SEC says revenue is $391B; Reuters says $383B
- **Type conflicts** — "Python" is a `ProgrammingLanguage` in one source, a `Snake` species in another
- **Temporal conflicts** — a CEO had two different employers during overlapping date ranges
- **Logical conflicts** — an entity simultaneously holds two mutually exclusive properties
- **Relationship conflicts** — the same relationship has inconsistent cardinality or properties across sources
## Exported Classes
```python
from semantica.conflicts import (
# Detection
ConflictDetector, # detect value, type, temporal, logical, relationship conflicts
Conflict, # {id, entity_id, attribute, values, sources, conflict_type, severity}
ConflictType, # enum: VALUE_CONFLICT, TYPE_CONFLICT, TEMPORAL_CONFLICT, ...
# Resolution
ConflictResolver, # resolve conflicts with configurable strategy
ResolutionStrategy, # enum: VOTING, CREDIBILITY_WEIGHTED, MOST_RECENT, FIRST_SEEN, ...
ResolutionResult, # outcome of a resolve_conflicts() call
# Convenience strategy aliases
voting, credibility_weighted, most_recent, first_seen, highest_confidence,
manual_review, expert_review,
# Source tracking
SourceTracker, # track which source contributed each property value
SourceReference, # {source_id, credibility, timestamp}
PropertySource, # per-property source attribution record
# Analysis
ConflictAnalyzer, # analyze patterns, severity distribution, source stats
ConflictPattern, # recurring conflict pattern detected across entities
# Investigation
InvestigationGuideGenerator, # generate step-by-step checklists for manual review
InvestigationGuide, # {title, context, steps}
InvestigationStep, # {order, description, check, priority}
# Convenience functions
detect_conflicts, # quick: detect_conflicts(entities, attribute="name")
resolve_conflicts, # quick: resolve_conflicts(conflicts, strategy=voting)
analyze_conflicts, # quick: analyze_conflicts(conflicts)
track_sources, # quick: track_sources(entities)
generate_investigation_guide,# quick: generate_investigation_guide(conflict)
)
```
## What You Get
- **`ConflictDetector`** — value, type, temporal, logical, and relationship conflict detection
- **`ConflictResolver`** — 7 resolution strategies including voting, credibility-weighted, and temporal
- **`SourceTracker`** — track which source each conflicting fact came from, with credibility scores
- **`ConflictAnalyzer`** — pattern analysis, severity grouping, and trend identification
- **`InvestigationGuideGenerator`** — auto-generate step-by-step investigation checklists for human review
<CardGroup cols={2}>
<Card title="ConflictDetector" icon="magnifying-glass">
Value, type, temporal, logical, and relationship conflict detection across all entity pairs.
</Card>
<Card title="ConflictResolver" icon="check">
7 resolution strategies including voting, credibility-weighted, and temporal preference.
</Card>
<Card title="SourceTracker" icon="link">
Track which source each conflicting fact came from, with per-source credibility scores.
</Card>
<Card title="ConflictAnalyzer" icon="chart-line">
Pattern analysis, severity grouping, source-level statistics, and trend identification.
</Card>
<Card title="InvestigationGuideGenerator" icon="list-check">
Auto-generate step-by-step investigation checklists for human and expert review.
</Card>
<Card title="Convenience Functions" icon="bolt">
`detect_conflicts()` and `resolve_conflicts()` for one-call workflows.
</Card>
</CardGroup>
## Quick Start
<Steps>
<Step title="Set credibility scores before ingestion">
```python
from semantica.conflicts import SourceTracker
tracker = SourceTracker()
tracker.set_source_credibility("sec_filings", 0.95)
tracker.set_source_credibility("pubmed", 0.92)
tracker.set_source_credibility("wikipedia", 0.80)
tracker.set_source_credibility("news_articles", 0.65)
```
</Step>
<Step title="Detect conflicts after building the graph">
```python
from semantica.conflicts import ConflictDetector
detector = ConflictDetector()
conflicts = detector.detect_conflicts(kg)
print(f"Found {len(conflicts)} conflicts")
for conflict in conflicts:
print(f"[{conflict.conflict_type}] entity='{conflict.entity_id}' attr='{conflict.attribute}'")
print(f" Values: {conflict.values} Severity: {conflict.severity:.2f}")
```
</Step>
<Step title="Triage by severity">
```python
from semantica.conflicts import ConflictAnalyzer
analyzer = ConflictAnalyzer()
analysis = analyzer.analyze_conflicts(conflicts)
by_severity = analysis["by_severity"]
print(f"Critical: {len(by_severity.get('critical', []))}")
print(f"High: {len(by_severity.get('high', []))}")
print(f"Low: {len(by_severity.get('low', []))}")
```
</Step>
<Step title="Auto-resolve low-severity, escalate critical">
```python
from semantica.conflicts import ConflictResolver, InvestigationGuideGenerator, ResolutionStrategy
resolver = ConflictResolver(source_tracker=tracker)
# Auto-resolve low-severity
auto_resolved = resolver.resolve_conflicts(
by_severity["low"],
strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED,
)
# Generate investigation guides for critical conflicts
generator = InvestigationGuideGenerator()
for conflict in by_severity["critical"]:
guide = generator.generate_guide(conflict)
print(f"\n{guide.title}")
for step in guide.steps:
print(f" [{step.order}] ({step.priority.upper()}) {step.description}")
```
</Step>
</Steps>
## ConflictDetector
```python
from semantica.conflicts import ConflictDetector
detector = ConflictDetector()
detector = ConflictDetector()
conflicts = detector.detect_conflicts(kg)
for conflict in conflicts:
print(f"[{conflict.conflict_type}] '{conflict.entity}' — {conflict.attribute}")
print(f" Sources: {conflict.sources}")
print(f" Severity: {conflict.severity:.2f}")
```
### Detection Types
| Type | What It Detects |
| ---- | --------------- |
| `VALUE` | Same entity, same attribute, different values across sources |
| `TYPE` | Same entity classified as different types in different sources |
| `TEMPORAL` | Overlapping validity windows with contradictory facts |
| `LOGICAL` | Facts that violate ontology axioms or SHACL constraints |
| `RELATIONSHIP` | Inconsistent relationship properties across sources |
| Type | What It Detects | Example |
| ---- | --------------- | ------- |
| `VALUE` | Same entity, same attribute, different values across sources | Revenue $391B vs $383B |
| `TYPE` | Same entity classified as different types | "Python" as Language vs Snake |
| `TEMPORAL` | Overlapping validity windows with contradictory facts | CEO at two companies simultaneously |
| `LOGICAL` | Facts that violate ontology axioms or SHACL constraints | `is_alive=True` but `death_date` set |
| `RELATIONSHIP` | Inconsistent relationship properties across sources | Edge weight 0.9 vs 0.3 from two sources |
Run targeted detection by type:
```python
# Detect all types (default)
# Detect all types at once (default)
conflicts = detector.detect_conflicts(kg)
# Detect specific types only
value_conflicts = detector.detect_value_conflicts(entities, "name")
# Detect specific types only — faster for targeted checks
value_conflicts = detector.detect_value_conflicts(entities, "revenue")
type_conflicts = detector.detect_type_conflicts(entities)
relation_conflicts = detector.detect_relationship_conflicts(kg)
```
**Key behaviours:**
- Severity scores are computed from the magnitude of disagreement — a $8B revenue discrepancy scores higher than a $1M discrepancy
- `LOGICAL` conflicts require an ontology or SHACL schema to be loaded; without one, they are not detected
- Detection runs in O(n·sources) time — it groups by entity+attribute and checks disagreement within each group
## ConflictResolver
```python
from semantica.conflicts import ConflictResolver, ResolutionStrategy
resolver = ConflictResolver()
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING)
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING)
for result in results:
print(f"Resolved '{result.attribute}' → {result.resolved_value}")
print(f" Strategy: {result.strategy}")
print(f" Strategy: {result.strategy} Confidence: {result.confidence:.2f}")
```
### Resolution Strategies
### Choosing a Resolution Strategy
| Strategy | Enum | Description |
|----------|------|-------------|
| Majority vote | `ResolutionStrategy.VOTING` | Most common value wins |
| Credibility-weighted | `ResolutionStrategy.CREDIBILITY_WEIGHTED` | Weighted by source credibility score |
| Most recent | `ResolutionStrategy.MOST_RECENT` | Prefer the most recently updated fact |
| First seen | `ResolutionStrategy.FIRST_SEEN` | Prefer the first observed value |
| Highest confidence | `ResolutionStrategy.HIGHEST_CONFIDENCE` | Prefer the fact with the highest confidence score |
| Manual review | `ResolutionStrategy.MANUAL_REVIEW` | Flag for human review |
| Expert review | `ResolutionStrategy.EXPERT_REVIEW` | Escalate to a domain expert |
<Tabs>
<Tab title="CREDIBILITY_WEIGHTED (recommended)">
Weights each source's value by its assigned credibility score — favors authoritative sources automatically:
```python
from semantica.conflicts import ConflictResolver, SourceTracker, ResolutionStrategy
tracker = SourceTracker()
tracker.set_source_credibility("sec_filings", 0.92)
tracker.set_source_credibility("wikipedia", 0.80)
tracker.set_source_credibility("news_articles", 0.65)
resolver = ConflictResolver(source_tracker=tracker)
results = resolver.resolve_conflicts(
conflicts,
strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED,
)
```
Best for: sources with known reliability rankings (SEC > blog).
</Tab>
<Tab title="VOTING">
Majority vote — most common value across sources wins:
```python
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING)
```
Best for: 3+ sources with roughly equal credibility. When all sources have identical credibility scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`.
</Tab>
<Tab title="MOST_RECENT / FIRST_SEEN">
```python
# Most recent source wins — for fast-changing facts
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.MOST_RECENT)
# First seen wins — for stable facts (founding date, original name)
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.FIRST_SEEN)
```
</Tab>
<Tab title="MANUAL_REVIEW / EXPERT_REVIEW">
```python
from semantica.conflicts import InvestigationGuideGenerator
# Flag for human review — use with InvestigationGuideGenerator
results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.MANUAL_REVIEW)
generator = InvestigationGuideGenerator()
for conflict in conflicts:
guide = generator.generate_guide(conflict)
print(f"{guide.title}")
for step in guide.steps:
print(f" [{step.order}] {step.description}")
```
Best for: high-stakes decisions (severity > 0.8), regulated data (HIPAA/SOX), and domain-specific ambiguity.
</Tab>
<Tab title="Strategy Comparison">
| Strategy | Enum | When to Use |
| -------- | ---- | ----------- |
| Majority vote | `VOTING` | 3+ sources with roughly equal credibility |
| Credibility-weighted | `CREDIBILITY_WEIGHTED` | Sources have different authority levels |
| Most recent | `MOST_RECENT` | Fast-changing facts: stock price, headcount, status |
| First seen | `FIRST_SEEN` | Stable facts: founding date, original name |
| Highest confidence | `HIGHEST_CONFIDENCE` | Extraction pipeline outputs confidence scores |
| Manual review | `MANUAL_REVIEW` | High-stakes decisions, regulated data |
| Expert review | `EXPERT_REVIEW` | Domain-specific ambiguity — escalate to a specialist |
</Tab>
</Tabs>
Use the convenience aliases for shorter code:
@@ -83,92 +267,159 @@ from semantica.conflicts import voting, credibility_weighted, most_recent, highe
results = resolver.resolve_conflicts(conflicts, strategy=voting)
```
## Source Credibility Scoring
Assign credibility weights per source so `CREDIBILITY_WEIGHTED` resolution favors authoritative sources:
## SourceTracker
```python
from semantica.conflicts import SourceTracker
from datetime import datetime
tracker = SourceTracker()
tracker.set_credibility("pubmed", 0.95)
tracker.set_credibility("wikipedia", 0.80)
tracker.set_credibility("user_input", 0.60)
tracker.set_source_credibility("sec_10k", 0.92)
tracker.set_source_credibility("wikipedia", 0.80)
resolver = ConflictResolver(source_tracker=tracker)
results = resolver.resolve_conflicts(
conflicts,
strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED
tracker.track_property_source(
entity_id="apple_inc",
property_name="revenue",
value="$391B",
source="sec_10k_2023",
timestamp=datetime(2024, 1, 26),
)
```
`SourceTracker` also builds full traceability chains:
```python
from semantica.conflicts import SourceTracker
tracker = SourceTracker()
tracker.track_entity_source("apple_inc", "crunchbase")
tracker.track_property_source("apple_inc", "revenue", "annual_report_2023")
sources = tracker.get_property_sources("apple_inc", "revenue")
for s in sources:
print(f"{s.source}: {s.value} (credibility: {s.credibility:.2f})")
chain = tracker.get_traceability_chain("apple_inc")
```
## ConflictAnalyzer
**Key behaviours:**
- Credibility scores default to 0.50 for any source not explicitly set
- `SourceTracker` stores property-level provenance — so you can trace exactly which source contributed each value
Identify patterns and trends across large conflict sets:
## ConflictAnalyzer
```python
from semantica.conflicts import ConflictAnalyzer
analyzer = ConflictAnalyzer()
# Detect recurring patterns
patterns = analyzer.identify_patterns(conflicts)
for pattern in patterns:
print(f"Pattern: {pattern.type} — {pattern.frequency} occurrences")
analysis = analyzer.analyze_conflicts(conflicts)
patterns = analysis["patterns"]
by_severity = analysis["by_severity"]
source_stats = analysis["by_source"]
trends = analyzer.analyze_trends(conflicts)
# Group by severity
by_severity = analyzer.group_by_severity(conflicts)
print(f"Critical: {len(by_severity['critical'])}")
print(f"High: {len(by_severity['high'])}")
print(f"Low: {len(by_severity['low'])}")
# Trend analysis over time
trends = analyzer.analyze_trends(conflicts, time_window="30d")
print(f"Trend direction: {trends['direction']}") # "increasing" | "stable" | "decreasing"
print(f"Change: {trends['change_pct']:.1f}%")
```
**Key behaviours:**
- `analyze_conflicts()["patterns"]` groups conflicts by attribute name and type — use it to find systemic data quality issues
- `analyze_conflicts()["by_source"]` flags sources with disproportionate conflict rates — a signal that a source's pipeline needs review
- `analyze_trends()` compares conflict counts over time — a rising trend means a data source is degrading
## InvestigationGuideGenerator
Auto-generate human-readable investigation guides for conflicts that can't be automatically resolved:
Auto-generate human-readable investigation checklists for conflicts requiring manual or expert review:
```python
from semantica.conflicts import InvestigationGuideGenerator, InvestigationGuide
from semantica.conflicts import InvestigationGuideGenerator
generator = InvestigationGuideGenerator()
guide: InvestigationGuide = generator.generate(conflict)
guide = generator.generate_guide(conflict)
print(f"Title: {guide.title}")
print(f"Context: {guide.context}")
print(guide.title)
print(guide.context)
for step in guide.steps:
print(f" [{step.order}] {step.description}")
print(f" Check: {step.check}")
print(f" [{step.order}] ({step.priority.upper()}) {step.description}")
print(f" → Verify: {step.check}")
```
## Convenience Functions
## Schemas
<AccordionGroup>
<Accordion title="Conflict schema">
```python
from semantica.conflicts import (
detect_conflicts, resolve_conflicts, analyze_conflicts,
track_sources, generate_investigation_guide
)
conflicts = detect_conflicts(entities, method="value")
resolved = resolve_conflicts(conflicts, strategy="voting")
analysis = analyze_conflicts(conflicts, method="pattern")
guide = generate_investigation_guide(conflicts[0])
@dataclass
class Conflict:
id: str
entity_id: str # the entity involved
attribute: str # the conflicting property name
values: List[str] # conflicting values (one per source)
sources: List[str] # source IDs for each value
conflict_type: ConflictType # VALUE | TYPE | TEMPORAL | LOGICAL | RELATIONSHIP
severity: float # 0.0 (minor) to 1.0 (critical)
confidence: float # detection confidence 0–1
detected_at: datetime
metadata: Dict[str, Any]
```
</Accordion>
<Accordion title="ConflictType enum">
```python
from semantica.conflicts import ConflictType
ConflictType.VALUE_CONFLICT # revenue is $391B in source A, $383B in source B
ConflictType.TYPE_CONFLICT # "Apple" is ORGANIZATION in one source, PRODUCT in another
ConflictType.TEMPORAL_CONFLICT # overlapping validity windows with contradictory states
ConflictType.LOGICAL_CONFLICT # fact violates an ontology axiom or SHACL constraint
ConflictType.RELATIONSHIP_CONFLICT # inconsistent relationship properties across sources
```
</Accordion>
<Accordion title="InvestigationGuide and InvestigationStep schemas">
```python
@dataclass
class InvestigationGuide:
title: str # human-readable title for the conflict
context: str # summary of the disagreement
steps: List[InvestigationStep] # ordered checklist for the reviewer
@dataclass
class InvestigationStep:
order: int
description: str # what to do
check: str # specific fact or document to verify
priority: str # "high" | "medium" | "low"
```
</Accordion>
</AccordionGroup>
## Tips and Common Pitfalls
<Warning>
**Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder — you lose the original source attribution.
</Warning>
<Warning>
**Always set credibility scores.** The default credibility is 0.50 for all sources. Without explicit scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`. The power of this strategy is in the differentiation.
</Warning>
<Tip>
**Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with severity > 0.8 — high severity means the disagreement is large and the stakes of getting it wrong are high.
</Tip>
<Warning>
**LOGICAL conflicts need a schema.** `detect_type_conflicts()` and `LOGICAL` detection only work if an OWL ontology or SHACL schema is loaded. Without one, `detect_conflicts()` will skip those types silently.
</Warning>
<Tip>
**Use `analyze_sources()` to identify bad data feeds.** A single source causing 80% of your conflicts is a data quality problem upstream, not a conflict to resolve record by record. Flag it and investigate the source pipeline.
</Tip>
<Tip>
**Severity is relative, not absolute.** A 0.5 severity score on a $1B revenue discrepancy and on a minor label difference both score 0.5 — the number reflects the disagreement structure, not the business impact. Domain context determines what to prioritize.
</Tip>
<Tip>
**Combine with provenance.** The `SourceTracker` feeds directly into the [Provenance](provenance) module's audit trail. If you need to explain how a resolved value was chosen, provenance records give you the full chain.
</Tip>
<CardGroup cols={2}>
<Card title="Deduplication" icon="copy" href="deduplication">
Resolve duplicate entities before conflict detection.
docs/reference/context.md
+580 -139
View File
@@ -1,127 +1,264 @@
---
title: "Context Module"
description: "Agent context graphs, decision tracking, causal chains, precedent search, and policy enforcement."
description: "Agent context graphs, decision tracking, causal chains, precedent search, policy enforcement, and multi-hop GraphRAG."
icon: "brain"
---
`semantica.context` is the memory and decision layer for AI agents. It stores facts with provenance, records decisions as first-class objects with causal chains, and lets agents search their own history to stay consistent across runs.
`semantica.context` is the memory and decision layer for AI agents. It stores facts with provenance, records decisions as first-class objects with full causal chains, lets agents search their own history to stay consistent across runs, and answers complex queries by traversing the knowledge graph.
## Exported Classes
```python
from semantica.context import (
# High-level interfaces
AgentContext, # primary entry point: store, retrieve, record_decision, find_precedents
DecisionContext, # decision-focused facade (wraps AgentContext + DecisionRecorder)
# Graph primitives
ContextGraph, # in-memory graph: add/get entities, record decisions, find precedents
ContextNode, # {id, label, node_type, properties, embedding, confidence}
ContextEdge, # {source, target, edge_type, weight, properties}
# Memory
AgentMemory, # RAG memory: store(text), retrieve(query, max_results)
MemoryItem, # {id, content, timestamp, conversation_id, embedding, metadata}
# Retrieval
ContextRetriever, # retrieve(query, max_results, use_graph, min_score)
RetrievedContext, # {content, score, source, metadata}
TemporalGraphRetriever, # retrieval with temporal decay weighting
# Entity linking
EntityLinker, # link_entity(text, entity_type) -> LinkedEntity with URI
EntityLink, # {entity_id, uri, source_text, confidence}
LinkedEntity, # {canonical_id, uri, aliases, type, properties}
# Decision tracking models
Decision, # {id, category, scenario, reasoning, outcome, confidence, timestamp}
Policy, # {id, name, conditions, action, priority}
PolicyException, # {policy_id, decision_id, reason, override_authority}
Precedent, # {decision_id, scenario, outcome, similarity, timestamp}
ApprovalChain, # ordered list of approvers for escalation
# Decision tracking classes
DecisionRecorder, # record and persist decisions with embeddings
DecisionQuery, # query decisions: by_category, by_outcome, by_date_range
CausalChainAnalyzer, # trace causality: get_causal_chain, analyze_impact
PolicyEngine, # check_compliance, get_applicable_policies, enforce_policy
# Convenience functions
record_decision, # record_decision(category, scenario, reasoning, outcome, confidence)
find_precedents, # find_precedents(scenario, category, limit)
analyze_decision_impact, # analyze_decision_impact(decision_id)
check_decision_compliance, # check_decision_compliance(decision, policies)
)
```
## What You Get
- **`AgentContext`** — unified interface for memory, decision tracking, and graph-backed retrieval
- **`ContextGraph`** — persistent knowledge graph with centrality analysis, community detection, and decision management
- **`AgentMemory`** — low-level embedding-backed memory with TTL, tagging, and importance scoring
- **`DecisionRecorder`** — records decisions with causal chains, confidence scores, and outcome tracking
- **`CausalAnalyzer`** — traces downstream impact of any decision
- **`PolicyEngine`** — validates decisions against configurable rules before they're recorded
<CardGroup cols={2}>
<Card title="AgentContext" icon="brain">
Unified interface for memory, decision tracking, graph-backed retrieval, conversation history, checkpoints, and persistence.
</Card>
<Card title="ContextGraph" icon="diagram-project">
Thread-safe in-memory knowledge graph with centrality analysis, community detection, temporal validity, cross-graph links, and decision management.
</Card>
<Card title="AgentMemory" icon="database">
Embedding-backed memory with TTL, tagging, importance scoring, and LRU eviction.
</Card>
<Card title="DecisionRecorder" icon="list-check">
Records decisions with causal chains, confidence scores, temporal validity windows, and cross-system context capture.
</Card>
<Card title="PolicyEngine" icon="shield-check">
Validates decisions against configurable lambda rules before they're recorded; creates approval chains for human-in-the-loop gating.
</Card>
<Card title="EntityLinker" icon="link">
Maps entity mentions to canonical URIs — prevents "Apple", "Apple Inc.", and "AAPL" from becoming three separate nodes.
</Card>
<Card title="ContextRetriever" icon="magnifying-glass">
Hybrid retrieval fusing vector similarity, graph traversal, and agent memory for richer context than pure vector search.
</Card>
<Card title="CausalChainAnalyzer" icon="arrow-trend-up">
Traces upstream causes and downstream effects of any decision through the knowledge graph.
</Card>
</CardGroup>
<img src="/assets/img/diagrams/agent-context-flow.svg" alt="AgentContext hub: AI Agent calls store/retrieve against VectorStore and record_decision against ContextGraph" style={{ width: '100%', borderRadius: '12px', margin: '0 0 24px' }} />
## Quick Start
<Steps>
<Step title="Initialize the agent context">
```python
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
context = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768, index_path="context.faiss"),
knowledge_graph=ContextGraph(advanced_analytics=True),
decision_tracking=True,
retention_days=90, # auto-expire memories older than 90 days
max_memories=50_000,
)
```
</Step>
<Step title="Store facts and retrieve by semantic similarity">
```python
memory_id = context.store(
"GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%",
metadata={"source": "openai_blog", "date": "2024-01"}
)
results = context.retrieve("LLM benchmark comparisons", max_results=5)
for r in results:
print(f"{r['content']} (score: {r['score']:.3f})")
```
</Step>
<Step title="Record decisions with full provenance">
```python
decision_id = context.record_decision(
category="model_selection",
scenario="Choose LLM for production reasoning pipeline",
reasoning="GPT-4 benchmark advantage justifies 3x cost increase",
outcome="selected_gpt4",
confidence=0.91,
entities=["gpt-4", "gpt-3.5"],
decision_maker="pipeline_agent",
)
```
</Step>
<Step title="Find precedents and trace causal chains">
```python
# Search past decisions — prevents contradictory choices across runs
precedents = context.find_precedents("model selection reasoning", limit=5)
for p in precedents:
print(f"[{p.category}] {p.outcome} (confidence: {p.confidence:.2f})")
print(f" Reasoning: {p.reasoning}")
# Trace what downstream decisions were influenced by this one
chain = context.get_causal_chain(decision_id, direction="downstream", max_depth=5)
print(f"Downstream decisions: {len(chain)}")
# Full explainability — upstream causes + downstream effects + relationship paths
explanation = context.trace_decision_explainability(decision_id)
print(f"Total connections: {explanation['total_connections']}")
```
</Step>
</Steps>
## AgentContext
The main entry point. Wraps memory, graph, and decision tracking behind a single API.
```python
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
context = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768),
knowledge_graph=ContextGraph(advanced_analytics=True),
decision_tracking=True,
)
```
### Store and Retrieve Memories
```python
# Store a fact — embedded and indexed automatically
memory_id = context.store(
"GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%",
metadata={"source": "openai_blog", "date": "2024-01"}
)
# Retrieve by semantic similarity
results = context.retrieve("LLM benchmark comparisons", top_k=5)
for r in results:
print(f"{r['content']} (score: {r['score']:.3f})")
```
### Record and Search Decisions
```python
decision_id = context.record_decision(
category="model_selection",
scenario="Choose LLM for production reasoning pipeline",
reasoning="GPT-4 benchmark advantage justifies 3x cost increase",
outcome="selected_gpt4",
confidence=0.91,
)
# Find similar past decisions — prevents inconsistent choices
precedents = context.find_precedents("model selection reasoning", limit=5)
# Analyze downstream impact
influence = context.analyze_decision_influence(decision_id)
print(f"Decisions influenced: {len(influence.downstream_decisions)}")
```
### Multi-Hop GraphRAG
```python
from semantica.llms import Groq
llm = Groq(model="llama-3.3-70b-versatile")
result = context.query_with_reasoning(
query="What technologies have we chosen and why?",
llm_provider=llm,
max_hops=2,
)
print(result["response"])
for step in result["reasoning_path"]:
print(f" {step}")
```
### Constructor Parameters
| Parameter | Type | Default | Description |
| --------- | ---- | ------- | ----------- |
| `vector_store` | `VectorStore` | required | Backend for embedding-based memory retrieval |
| `knowledge_graph` | `ContextGraph` | `None` | Enables graph-backed relationships and analytics |
| `vector_store` | `VectorStore` | **required** | Backend for embedding-based memory retrieval |
| `knowledge_graph` | `ContextGraph` | `None` | Enables graph-backed relationships and GraphRAG |
| `decision_tracking` | `bool` | `False` | Activates `DecisionRecorder` for every decision |
| `retention_days` | `Optional[int]` | `30` | Auto-expire memories older than N days; `None` = keep forever |
| `max_memories` | `int` | `10000` | Hard cap before LRU eviction |
| `graph_expansion` | `bool` | `True` | Auto-expands graph from stored memories |
| `advanced_analytics` | `bool` | `True` | Enables centrality and community analysis |
| `max_expansion_hops` | `int` | `2` | Max hops for graph expansion during retrieval |
| `hybrid_alpha` | `float` | `0.5` | Balance between vector (`0.0`) and graph (`1.0`) retrieval |
| `advanced_analytics` | `bool` | `True` | Enables PageRank, centrality, and community analysis |
| `kg_algorithms` | `bool` | `True` | Adds path-finding and link prediction |
### Core Methods
### Memory Methods
| Method | Returns | Description |
| ------ | ------- | ----------- |
| `store(content, metadata)` | `str` (memory_id) | Embed and store a fact |
| `retrieve(query, top_k)` | `List[Dict]` | Semantic similarity search |
| `record_decision(category, scenario, reasoning, outcome, confidence)` | `str` (decision_id) | Record a decision with full provenance |
| `find_precedents(scenario, category, limit)` | `List[Decision]` | Find similar past decisions |
| `analyze_decision_influence(decision_id)` | `InfluenceResult` | Trace downstream impact |
| `query_with_reasoning(query, llm_provider, max_hops)` | `Dict` | GraphRAG with multi-hop traversal |
| `get_context_insights()` | `Dict` | Analytics summary |
| `store(content, metadata, conversation_id, user_id)` | `str` | Embed and store a fact or list of facts |
| `batch_store(items)` | `List[str]` | Store multiple items at once — returns list of memory IDs |
| `retrieve(query, max_results, min_score, use_graph, conversation_id)` | `List[Dict]` | Semantic retrieval; auto-selects GraphRAG if `knowledge_graph` is set |
| `forget(memory_id, conversation_id, days_old)` | `int` | Delete memories by ID, conversation, or age |
| `update(memory_id, content, metadata)` | `bool` | Update content or metadata of a stored memory |
| `get_memory(memory_id)` | `Optional[Dict]` | Fetch a specific memory by ID |
| `stats()` | `Dict` | Memory counts, vector store status, graph stats |
| `health()` | `Dict` | System health — all backends, status flags |
| `save(path)` | `None` | Persist full context state (memory + graph) to disk |
| `load(path)` | `None` | Restore context state from disk |
| `export(conversation_id, format)` | `str \| Dict` | Export memories as JSON or dict |
| `import_data(data, format)` | `int` | Import memories from JSON or dict |
### Conversation Methods
```python
# Store turns in a conversation thread
context.store("User asked about deployment options", conversation_id="conv_001")
context.store("Agent recommended Docker + Kubernetes", conversation_id="conv_001")
# Retrieve full conversation history
history = context.conversation("conv_001", max_items=50)
for turn in history:
print(f"[{turn['timestamp']}] {turn['content']}")
# Retrieve across all conversations with a query
results = context.retrieve("deployment recommendations", conversation_id="conv_001", max_results=10)
```
### Multi-Hop GraphRAG
Requires `knowledge_graph` to be set at construction:
```python
from semantica.llms import Groq
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
result = context.query_with_reasoning(
query="What technologies have we chosen and why?",
llm_provider=llm,
max_hops=2,
max_results=10,
)
print(result["response"])
print(f"Confidence: {result['confidence']:.2f}")
print(f"Sources used: {result['num_sources']}")
```
### Decision Methods
| Method | Returns | Description |
| ------ | ------- | ----------- |
| `record_decision(category, scenario, reasoning, outcome, confidence, entities, decision_maker, valid_from, valid_until)` | `str` | Record a decision; raises `RuntimeError` if `decision_tracking=False` |
| `find_precedents(scenario, category, limit, use_hybrid_search, max_hops, as_of)` | `List[Decision]` | Find similar past decisions by semantic + structural similarity |
| `query_decisions(query, max_hops, use_hybrid_search)` | `List[Decision]` | Broad context-aware decision search |
| `get_causal_chain(decision_id, direction, max_depth)` | `List[Decision]` | Trace `"upstream"` causes or `"downstream"` effects |
| `trace_decision_explainability(decision_id)` | `Dict` | Full explainability — causes, effects, relationship paths |
| `get_policy_engine()` | `PolicyEngine` | Access the active `PolicyEngine` instance |
### Checkpoint Methods
Useful for detecting what changed across reasoning runs:
```python
# Take a named snapshot of the current graph state
context.checkpoint("before_inference")
# ... run reasoning, record decisions ...
context.checkpoint("after_inference")
# See exactly what was added/removed
diff = context.diff_checkpoints("before_inference", "after_inference")
print(f"Decisions added: {len(diff['decisions_added'])}")
print(f"Relationships added: {len(diff['relationships_added'])}")
# Persist a checkpoint to disk via TemporalVersionManager
context.flush_checkpoint("after_inference")
```
## ContextGraph
The knowledge graph backing `AgentContext`. Can be used standalone for relationship modelling.
The knowledge graph backing `AgentContext`. Can also be used standalone for relationship modelling.
```python
from semantica.context import ContextGraph
graph = ContextGraph(advanced_analytics=True)
# Add nodes and edges
# Build the graph
graph.add_node("Python", "language", properties={"paradigm": "multi-paradigm"})
graph.add_node("FastAPI", "framework", properties={"language": "Python"})
graph.add_edge("Python", "FastAPI", "enables")
# Decision management
decision_id = graph.add_decision_simple(
# Record and query decisions directly on the graph
decision_id = graph.record_decision(
category="technology_choice",
scenario="Web API framework selection",
reasoning="FastAPI's async support and auto-docs match our requirements",
@@ -131,11 +268,11 @@ decision_id = graph.add_decision_simple(
)
similar = graph.find_precedents_by_scenario("web framework", limit=3)
impact = graph.analyze_decision_impact(decision_id)
chain = graph.trace_decision_chain(decision_id)
stats = graph.stats()
print(f"Nodes: {stats['node_count']}, Edges: {stats['edge_count']}")
```
### ContextGraph Constructor Options
### Constructor Options
| Parameter | Type | Default | Description |
| --------- | ---- | ------- | ----------- |
@@ -143,23 +280,65 @@ chain = graph.trace_decision_chain(decision_id)
| `centrality_analysis` | `bool` | `False` | Full centrality suite |
| `community_detection` | `bool` | `False` | Louvain community clustering |
| `node_embeddings` | `bool` | `False` | Node2Vec embeddings for structural similarity |
| `enable_causality` | `bool` | `False` | Causal chain tracking between decision nodes |
## Decision Data Structure
### ContextGraph — Full Method Reference
| Method | Returns | Description |
| ------ | ------- | ----------- |
| `add_node(node_id, node_type, properties, valid_from, valid_until)` | `None` | Add a node; supports temporal validity windows |
| `add_edge(source_id, target_id, edge_type, weight, properties)` | `None` | Add a directed edge with optional weight |
| `add_nodes(nodes)` | `int` | Bulk-add from a list of dicts; returns count added |
| `add_edges(edges)` | `int` | Bulk-add edges; returns count added |
| `get_neighbors(node_id, hops)` | `List[Dict]` | BFS neighbors up to given depth |
| `get_neighbor_distances(node_id, hops)` | `List[Dict]` | Neighbors with confidence-decay scoring |
| `find_node(node_id)` | `Optional[Dict]` | Look up a single node by ID |
| `find_nodes(node_type, skip, limit)` | `List[Dict]` | Filter nodes by type with pagination |
| `find_active_nodes(node_type, at_time)` | `List[Dict]` | Nodes that are valid at a given timestamp |
| `find_edges(edge_type, skip, limit)` | `List[Dict]` | Filter edges by type with pagination |
| `record_decision(category, scenario, reasoning, outcome, confidence, entities, decision_maker)` | `str` | Add decision node with causal edges |
| `find_precedents_by_scenario(scenario, category, limit, use_semantic_search, as_of)` | `List[Dict]` | Semantically similar past scenarios |
| `query(query, skip, limit)` | `List[Dict]` | Full-text search over node content |
| `stats()` | `Dict` | Node/edge counts, type breakdowns, graph density |
| `density()` | `float` | Graph density score |
| `save_to_file(path)` | `None` | Persist graph to JSON |
| `load_from_file(path)` | `None` | Load graph from JSON |
| `build_from_conversations(conversations, link_entities)` | `Dict` | Build graph from conversation data |
| `link_graph(other_graph, source_node_id, target_node_id, link_type)` | `str` | Create cross-graph navigation link; returns `link_id` |
| `navigate_to(link_id)` | `Tuple[ContextGraph, str]` | Follow a cross-graph link to `(target_graph, target_node_id)` |
| `cross_graph_path(source_node_id, target_graph, target_node_id, max_hops)` | `Dict` | Shortest path across linked graphs |
| `resolve_links(graphs)` | `int` | Reconnect cross-graph links after `load_from_file` |
| `clear()` | `None` | Reset graph state and all indexes |
### Cross-Graph Navigation
Link multiple independent `ContextGraph` instances so agents can traverse across problem spaces:
```python
@dataclass
class Decision:
decision_id: str
category: str
scenario: str
reasoning: str
outcome: str
confidence: float # 0.0 – 1.0
decision_maker: str
timestamp: datetime
entities: List[str]
metadata: Dict
causal_chain: List[str] # IDs of related decisions
domain_graph = ContextGraph()
decision_graph = ContextGraph()
domain_graph.add_node("microservices", "architecture", properties={"style": "distributed"})
decision_graph.add_node("deploy_k8s", "decision", properties={"outcome": "approved"})
link_id = domain_graph.link_graph(
other_graph=decision_graph,
source_node_id="microservices",
target_node_id="deploy_k8s",
link_type="INFORMED_BY",
)
# Follow the link at traversal time
target_graph, entry_node = domain_graph.navigate_to(link_id)
# Cross-graph pathfinding
path = domain_graph.cross_graph_path(
source_node_id="microservices",
target_graph=decision_graph,
target_node_id="deploy_k8s",
max_hops=5,
)
print(f"Reachable: {path['reachable']}, hops: {path['hop_count']}")
```
## AgentMemory (Low-Level)
@@ -168,18 +347,38 @@ For fine-grained control over memory storage, TTL, and importance scoring:
```python
from semantica.context import AgentMemory
from semantica.vector_store import VectorStore
memory = AgentMemory(
vector_store=VectorStore(backend="faiss", dimension=768),
max_memories=10_000,
capacity=10_000,
ttl_days=90,
)
memory.store("Important fact", importance=0.9, tags=["compliance"])
results = memory.retrieve("fact query", top_k=5, min_importance=0.5)
memory_id = memory.store(
"Critical compliance rule: all trades must be pre-approved",
importance=0.95,
tags=["compliance", "trading"],
)
results = memory.retrieve(
query="trade approval requirements",
max_results=5,
min_importance=0.5,
tags=["compliance"],
)
memory.update(memory_id, importance=1.0)
memory.forget(memory_id)
all_memories = memory.get_all()
```
| Parameter | Type | Default | Description |
| --------- | ---- | ------- | ----------- |
| `vector_store` | `VectorStore` | **required** | Embedding backend for semantic retrieval |
| `capacity` | `int` | `1000` | Max items before LRU eviction |
| `ttl_days` | `Optional[int]` | `None` | Days before automatic expiry; `None` = keep forever |
## PolicyEngine
Validate decisions against configurable rules before they're committed:
@@ -191,55 +390,297 @@ policy = PolicyEngine()
policy.add_rule("confidence_threshold", lambda d: d.confidence >= 0.7)
policy.add_rule("requires_reasoning", lambda d: len(d.reasoning) >= 20)
# Validate before recording
is_valid, violations = policy.validate(decision_data)
if is_valid:
context.record_decision(**decision_data)
else:
# Create approval chain for human-in-the-loop review
chain = policy.create_approval_chain(
decision_data,
approvers=["manager@company.com", "compliance@company.com"],
)
print(f"Approval chain created: {chain.chain_id}")
```
## EntityLinker
Maps extracted entity mentions to canonical URIs — essential for cross-document entity resolution:
```python
from semantica.context import EntityLinker
linker = EntityLinker()
entities = [
{"text": "Apple Inc.", "type": "ORGANIZATION"},
{"text": "Apple", "type": "ORGANIZATION"},
{"text": "AAPL", "type": "ORGANIZATION"},
]
linked = linker.link_entities(entities, sources=["reuters", "sec_filings"])
for e in linked:
print(f"{e.text} → {e.canonical_form} ({e.uri})")
print(f" confidence: {e.confidence:.2f}, sources: {e.sources}")
```
## ContextRetriever
Hybrid retrieval combining vector similarity, graph traversal, and memory — surfaces results that pure vector search misses:
```python
from semantica.context import ContextRetriever
retriever = ContextRetriever(
vector_store=vector_store,
context_graph=context_graph,
agent_memory=memory,
)
results = retriever.retrieve(
query="What decisions were made about cloud infrastructure?",
max_results=10,
use_graph_expansion=True,
min_relevance_score=0.3,
)
for r in results:
print(f"[{r['source']}] score={r['score']:.3f}: {r['content'][:80]}")
```
## Data Structures
<AccordionGroup>
<Accordion title="Decision">
```python
@dataclass
class Decision:
decision_id: str
category: str
scenario: str
reasoning: str
outcome: str
confidence: float # 0.0 – 1.0
decision_maker: str # default: "ai_agent"
timestamp: datetime
valid_from: Optional[str] # ISO datetime — temporal validity start
valid_until: Optional[str] # ISO datetime — temporal validity end
metadata: Dict[str, Any] # arbitrary key/value store
```
</Accordion>
<Accordion title="Precedent">
```python
@dataclass
class Precedent:
decision_id: str
similarity: float # 0–1 match score against queried scenario
category: str
scenario: str
outcome: str
reasoning: str
confidence: float
timestamp: datetime
```
</Accordion>
<Accordion title="Policy">
```python
@dataclass
class Policy:
policy_id: str
name: str
description: str
rules: List[Dict] # list of rule definitions
active: bool
created_at: datetime
version: int
```
</Accordion>
<Accordion title="PolicyException">
```python
@dataclass
class PolicyException:
exception_id: str
policy_rule: str # name of the violated rule
decision_id: str # decision that triggered the exception
justification: str # why the exception was granted
approved_by: str # approver identity
timestamp: datetime
expiry: Optional[datetime]
```
</Accordion>
<Accordion title="ApprovalChain">
```python
@dataclass
class ApprovalChain:
chain_id: str
decision_id: str
steps: List[ApprovalStep]
status: str # "pending" | "approved" | "rejected"
created_at: datetime
@dataclass
class ApprovalStep:
step_id: str
approver: str
required: bool
status: str # "pending" | "approved" | "rejected"
comment: Optional[str]
timestamp: Optional[datetime]
```
</Accordion>
<Accordion title="LinkedEntity">
```python
@dataclass
class LinkedEntity:
text: str
canonical_form: str # normalized primary name
uri: str # e.g. "http://dbpedia.org/resource/Apple_Inc."
confidence: float
sources: List[str] # source documents that mention this entity
aliases: List[str] # all observed surface forms
```
</Accordion>
</AccordionGroup>
## Real-World Patterns
### Healthcare — Treatment Decisions
<Tabs>
<Tab title="Healthcare — Treatment Decisions">
```python
from semantica.context import AgentContext
from semantica.vector_store import VectorStore
```python
health_agent = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768),
decision_tracking=True,
)
health_agent = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768),
decision_tracking=True,
)
health_agent.store("Patient has hypertension, type 2 diabetes")
health_agent.store("Patient allergic to penicillin — verified 2024-01")
health_agent.store("Patient has hypertension, type 2 diabetes")
health_agent.store("Patient allergic to penicillin — verified 2024-01")
decision_id = health_agent.record_decision(
category="treatment_plan",
scenario="Hypertension with comorbid diabetes",
reasoning="ACE inhibitors are renoprotective in diabetic patients — preferred over beta blockers",
outcome="prescribed_lisinopril",
confidence=0.91,
)
decision_id = health_agent.record_decision(
category="treatment_plan",
scenario="Hypertension with comorbid diabetes",
reasoning="ACE inhibitors are renoprotective in diabetic patients — preferred over beta blockers",
outcome="prescribed_lisinopril",
confidence=0.91,
)
# Check for similar cases
precedents = health_agent.find_precedents("hypertension diabetes", limit=5)
```
precedents = health_agent.find_precedents("hypertension diabetes", limit=5)
for p in precedents:
print(f"Past decision: {p.outcome} (confidence: {p.confidence:.2f})")
### Finance — Loan Decisions
chain = health_agent.get_causal_chain(decision_id, direction="downstream")
print(f"Follow-up decisions triggered: {len(chain)}")
```
</Tab>
<Tab title="Finance — Loan Decisions">
```python
from semantica.context import AgentContext, PolicyEngine
from semantica.vector_store import VectorStore
```python
loan_agent = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768),
decision_tracking=True,
)
policy = PolicyEngine()
policy.add_rule("min_confidence", lambda d: d["confidence"] >= 0.8)
policy.add_rule("has_reasoning", lambda d: len(d["reasoning"]) >= 30)
loan_agent.store("Applicant: credit score 750, DTI 28%, stable employment 4yr")
loan_agent = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768),
decision_tracking=True,
)
decision_id = loan_agent.record_decision(
category="loan_approval",
scenario="First-time homebuyer — 30yr fixed, 20% down",
reasoning="Credit score above threshold, DTI within limits, stable income verified",
outcome="approved_300k",
confidence=0.94,
)
```
loan_agent.store("Applicant: credit score 750, DTI 28%, stable employment 4yr")
decision_data = dict(
category="loan_approval",
scenario="First-time homebuyer — 30yr fixed, 20% down",
reasoning="Credit score above threshold, DTI within limits, stable income verified",
outcome="approved_300k",
confidence=0.94,
)
is_valid, violations = policy.validate(decision_data)
if is_valid:
decision_id = loan_agent.record_decision(**decision_data)
else:
chain = policy.create_approval_chain(decision_data, approvers=["underwriter@bank.com"])
print(f"Sent for review: {chain.chain_id}")
```
</Tab>
<Tab title="Persist & Restore">
```python
context = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768, index_path="ctx.faiss"),
knowledge_graph=ContextGraph(),
decision_tracking=True,
)
context.store("Important fact learned during session")
context.record_decision(
category="ops", scenario="Scale up", reasoning="Load > 80%",
outcome="scaled_to_10_replicas", confidence=0.97,
)
# Persist everything
context.save("agent_state/")
# Later — restore and continue
restored = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768, index_path="ctx.faiss"),
knowledge_graph=ContextGraph(),
decision_tracking=True,
)
restored.load("agent_state/")
results = restored.retrieve("load scaling decisions", max_results=3)
```
</Tab>
</Tabs>
## Tips and Common Pitfalls
<Warning>
**Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore` — without it the FAISS index lives only in memory and is lost on shutdown. An agent that forgets everything on restart isn't an agent.
</Warning>
<Warning>
**Enable `decision_tracking=True` from the start.** Adding it retroactively means historical decisions are not linked to the causal chain — you lose the ability to trace how one decision influenced later ones. Enable it at initialization, even if you're not using it immediately.
</Warning>
<Tip>
**Use `find_precedents()` before every significant decision.** This is how the context module prevents agents from making contradictory choices across runs. Surface precedents to the LLM as context: "we chose X for similar reasons before."
</Tip>
<Tip>
**`retrieve()` uses `max_results=`, not `top_k=`.** The parameter is `max_results` (default `5`). Pass `use_graph=True` to force GraphRAG or `use_graph=False` to force vector-only retrieval regardless of whether a `knowledge_graph` is configured.
</Tip>
<Tip>
**Set `retention_days` to avoid memory bloat.** Without it `AgentMemory` accumulates indefinitely (the default `AgentContext.retention_days=30` prunes automatically). Compliance-critical agents may need `retention_days=None` with explicit archival via `export()`.
</Tip>
<Warning>
**Gate irreversible decisions with `PolicyEngine`.** Decisions recorded with `record_decision()` become part of the causal chain immediately. Validate first with `policy.validate()` and create an `ApprovalChain` for human review — don't record until approved.
</Warning>
<Tip>
**Use `checkpoint()` + `diff_checkpoints()` to audit reasoning loops.** Take a snapshot before and after a reasoning pass to see exactly which decisions and relationships were added. This is the cleanest way to detect divergent agent behaviour across runs.
</Tip>
<Tip>
**`EntityLinker` prevents graph proliferation.** Without it, "Apple", "Apple Inc.", and "AAPL" land as three separate nodes. Run `EntityLinker.link_entities()` on mentions before storing them to maintain a canonical graph.
</Tip>
<CardGroup cols={2}>
<Card title="Vector Store" icon="database" href="vector_store">
docs/reference/core.md
+31 -2
View File
@@ -6,13 +6,28 @@ icon: "gear"
`semantica.core` is the coordination layer for the framework. For most tasks you should use individual modules directly (`semantica.ingest`, `semantica.kg`, etc.). Reach for Core when you need application-level lifecycle management, centralized configuration, or a plugin registry.
## Exported Classes
```python
from semantica.core import (
Semantica, # orchestration class — coordinates full KG pipeline
ConfigManager, # YAML config loading, deep-merge, env var overrides
LifecycleManager,# startup/shutdown state machine + health monitoring
PluginRegistry, # plugin discovery, registration, and loading
method_registry, # global MethodRegistry instance for custom dispatch
)
# For custom build methods:
from semantica.core.methods import build_knowledge_base
```
## What You Get
- **`Semantica`** — orchestration class for coordinating complex multi-module workflows
- **`ConfigManager`** — unified config loading, merging, and validation with environment variable overrides
- **`LifecycleManager`** — startup/shutdown hooks and component health monitoring
- **`PluginRegistry`** — dynamic plugin discovery, registration, and loading
- **`MethodRegistry`** — register and dispatch custom orchestration methods
- **`method_registry`** — global `MethodRegistry` instance — register and dispatch custom orchestration methods
<Tip>
**Use individual modules directly** for the vast majority of use cases. Use the `Semantica` orchestration class only when you need application-level lifecycle management or a plugin system.
@@ -167,6 +182,7 @@ Register custom orchestration methods and dispatch them by name:
```python
from semantica.core import method_registry
from semantica.core.methods import build_knowledge_base
def fast_kb_builder(sources, **kwargs):
# Custom logic — skip embeddings for speed
@@ -174,10 +190,23 @@ def fast_kb_builder(sources, **kwargs):
method_registry.register("knowledge_base", "fast", fast_kb_builder)
from semantica.core.methods import build_knowledge_base
result = build_knowledge_base(sources=["doc.pdf"], method="fast")
```
## When to Use Core vs. Individual Modules
| Scenario | Recommended Approach |
| -------- | -------------------- |
| Single extraction task | `from semantica.semantic_extract import NERExtractor` |
| Build a knowledge graph | `from semantica.kg import GraphBuilder` |
| Multi-step pipeline | `from semantica.pipeline import Pipeline` |
| App-level lifecycle + config | `from semantica.core import Semantica, ConfigManager` |
| Custom dispatch / plugins | `from semantica.core import method_registry, PluginRegistry` |
<Tip>
Use `Semantica` and `LifecycleManager` only when building a long-running application (e.g. a FastAPI service) that needs ordered startup, health checks, and graceful shutdown. For scripts and notebooks, use individual modules directly.
</Tip>
<CardGroup cols={2}>
<Card title="Pipeline" icon="arrows-turn-to-dots" href="pipeline">
Pipeline execution and step orchestration.
docs/reference/deduplication.md
+19
View File
@@ -6,6 +6,25 @@ icon: "copy"
`semantica.deduplication` detects and merges duplicate entities across sources to produce a clean, single-source-of-truth knowledge graph. **v2 strategies** (`blocking_v2`, `hybrid_v2`, `semantic_v2`) are up to **7x faster** than v1 with fine-grained result control.
## Exported Classes
```python
from semantica.deduplication import (
DuplicateDetector, # pairwise + batch duplicate detection
EntityMerger, # merge duplicate groups with per-property policies
SimilarityCalculator, # Levenshtein, Jaro-Winkler, cosine, Jaccard, embedding
ClusterBuilder, # Union-Find + hierarchical clustering
PropertyMergeRule, # enum: KEEP_FIRST, KEEP_LONGEST, UNION, VOTING, ...
MergeStrategyManager, # manage and apply merge strategies
# Convenience functions
detect_duplicates, # quick: detect_duplicates(entities, method="semantic_v2")
merge_entities, # quick: merge_entities(entities, duplicates, method="union")
calculate_similarity, # quick: calculate_similarity(a, b, method="hybrid_v2")
# Registry
method_registry, # register custom similarity functions
)
```
## What You Get
- **`DuplicateDetector`** — pairwise and batch duplicate detection with configurable strategies
docs/reference/embeddings.md
+353 -122
View File
@@ -1,216 +1,425 @@
---
title: "Embeddings Module"
description: "Text and graph embedding generation — Sentence-Transformers, FastEmbed, OpenAI, BGE, LlamaStore, with pooling strategies and graph embedding managers."
description: "Text and graph embedding generation — Sentence-Transformers, FastEmbed, OpenAI, BGE, Ollama — with pooling strategies, caching, and GPU acceleration."
icon: "vector-square"
---
`semantica.embeddings` converts text and graph structures into dense vectors for semantic search, entity resolution, and GraphRAG retrieval. A single provider-agnostic API abstracts Sentence-Transformers, FastEmbed, OpenAI, BGE, and Ollama.
`semantica.embeddings` converts text and graph structures into dense vectors. These vectors power semantic search, entity resolution, GraphRAG retrieval, and Distance Intelligence across every Semantica module. A single provider-agnostic API abstracts Sentence-Transformers, FastEmbed, OpenAI, BGE, and Ollama behind one interface.
## Why Embeddings Matter
Raw text can't be compared mathematically. Embeddings translate meaning into geometry — two semantically similar sentences produce vectors that are close together in high-dimensional space, even when they share no words.
Semantica uses embeddings for:
- **Semantic search** — find knowledge graph nodes by meaning, not just keywords
- **Entity resolution** — detect that "Apple Inc." and "Apple Computer" refer to the same entity
- **Deduplication** — `semantic_v2` strategy measures entity similarity via embedding distance
- **GraphRAG retrieval** — hybrid vector + graph traversal for grounded LLM answers
- **Distance Intelligence** — N×N semantic distance matrices across entity sets
- **Semantic chunking** — detect topic shift boundaries in `TextSplitter(method="semantic_transformer")`
## Exported Classes
```python
from semantica.embeddings import (
# Core generators
EmbeddingGenerator, # main handler: generate_embeddings(text, data_type="text")
TextEmbedder, # text embedding: embed(text), embed_batch(texts)
GraphEmbeddingManager, # embed KG nodes/subgraphs for GraphRAG
VectorEmbeddingManager, # embedding management for vector databases
# Provider stores
OpenAIStore, # OpenAI text-embedding-* API
BGEStore, # BAAI/bge-* via sentence-transformers
FastEmbedStore, # ONNX-accelerated, no CUDA required
LlamaStore, # Ollama local embedding models
ProviderStoreFactory, # create(provider="bge", model="...") factory
# Pooling strategies
MeanPooling, # default — best for retrieval and clustering
MaxPooling, # captures presence of any feature
CLSPooling, # CLS token (BERT-style classification models)
AttentionPooling, # softmax-weighted sum
HierarchicalPooling, # for long documents exceeding context length
PoolingStrategyFactory, # create(strategy="mean") factory
# Convenience functions
embed_text, # embed_text(text, method="sentence_transformers")
generate_embeddings, # generate_embeddings(texts, method="openai")
calculate_similarity, # calculate_similarity(a, b, method="cosine")
pool_embeddings, # pool_embeddings(token_embeddings, strategy="mean")
check_available_providers, # returns {"sentence_transformers": True, ...}
)
```
## What You Get
- **`EmbeddingGenerator`** — main entry point, provider-agnostic text embedding with batching
- **`TextEmbedder`** — text-specific embedding with automatic batching and disk caching
- **`GraphEmbeddingManager`** — node and subgraph embeddings for structural similarity
- **`VectorEmbeddingManager`** — full embedding lifecycle for vector store integration
- **Provider stores** — `OpenAIStore`, `BGEStore`, `FastEmbedStore`, `LlamaStore`, `ProviderStoreFactory`
- **Pooling strategies** — Mean, Max, CLS, Attention, Hierarchical pooling
<CardGroup cols={2}>
<Card title="EmbeddingGenerator" icon="vector-square">
Main entry point — provider-agnostic, handles batching automatically across all backends.
</Card>
<Card title="TextEmbedder" icon="text-size">
Text-specific with automatic batching, disk caching, and progress tracking.
</Card>
<Card title="GraphEmbeddingManager" icon="diagram-project">
Node and subgraph embeddings for structural similarity and GraphRAG context assembly.
</Card>
<Card title="VectorEmbeddingManager" icon="database">
Full lifecycle: embed → store → search in a single coordinated workflow.
</Card>
<Card title="Provider Stores" icon="plug">
`OpenAIStore`, `BGEStore`, `FastEmbedStore`, `LlamaStore`, and `ProviderStoreFactory`.
</Card>
<Card title="Pooling Strategies" icon="layer-group">
Mean, Max, CLS, Attention, and Hierarchical — control token-to-vector aggregation.
</Card>
</CardGroup>
## Installation
| Provider | Install Command | API Key Required |
| -------- | --------------- | ---------------- |
| Sentence-Transformers (default) | `pip install semantica` | No |
| FastEmbed | `pip install "semantica[fastembed]"` | No |
| BGE | `pip install semantica` | No (uses sentence-transformers) |
| OpenAI | `pip install "semantica[llm-openai]"` | Yes — `OPENAI_API_KEY` |
| Ollama (LlamaStore) | `pip install "semantica[llm-ollama]"` | No — local server |
| All providers | `pip install "semantica[all]"` | Varies |
Check which providers are available in your environment:
```python
from semantica.embeddings import check_available_providers
providers = check_available_providers()
# → {"sentence_transformers": True, "fastembed": True, "openai": False, "ollama": True}
```
## Quick Start
<Steps>
<Step title="Install and initialize a provider">
```python
from semantica.embeddings import EmbeddingGenerator
# Default — Sentence-Transformers, free, runs locally
generator = EmbeddingGenerator()
# Custom model via config dict
generator = EmbeddingGenerator(config={"text": {"method": "sentence_transformers", "model_name": "BAAI/bge-large-en-v1.5"}})
```
</Step>
<Step title="Generate embeddings">
```python
embeddings = generator.generate_embeddings(["Text about AI", "Machine learning concepts"])
```
</Step>
<Step title="Compute similarity">
```python
# Cosine similarity — 0.0 (unrelated) to 1.0 (identical meaning)
score = generator.compare_embeddings(embeddings[0], embeddings[1], method="cosine")
print(f"Similarity: {score:.3f}")
```
</Step>
<Step title="Embed and store for search">
```python
from semantica.embeddings import VectorEmbeddingManager, TextEmbedder
from semantica.vector_store import VectorStore
vector_store = VectorStore(backend="faiss", dimension=384)
manager = VectorEmbeddingManager(
embedder=TextEmbedder(model="sentence-transformers"),
vector_store=vector_store,
)
ids = manager.embed_and_store(documents, metadata=metadata_list)
results = manager.search("machine learning algorithms", top_k=10)
for result in results:
print(f"Score: {result.score:.3f} — {result.metadata['title']}")
```
</Step>
</Steps>
## Supported Models
| Provider | Model | Dimension | Speed | Best For |
| -------- | ----- | --------- | ----- | -------- |
| `sentence-transformers` | `all-MiniLM-L6-v2` | 384 | Fast | Default — good balance of speed and quality |
| `sentence-transformers` | `all-mpnet-base-v2` | 768 | Medium | Higher retrieval quality |
| `bge` | `BAAI/bge-large-en-v1.5` | 1024 | Medium | State-of-the-art retrieval accuracy |
| `bge` | `BAAI/bge-small-en-v1.5` | 384 | Fast | Lightweight, competitive quality |
| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Very fast | CPU-optimised, low-latency production |
| `openai` | `text-embedding-3-small` | 1536 | API | Cost-effective OpenAI embedding |
| `openai` | `text-embedding-3-large` | 3072 | API | Highest quality via OpenAI API |
| `llama` (Ollama) | Any Ollama model | Varies | Local | Fully local, no API key |
## EmbeddingGenerator
Main entry point — handles provider selection and batching automatically:
<Tabs>
<Tab title="Sentence-Transformers (default)">
```python
from semantica.embeddings import EmbeddingGenerator
```python
from semantica.embeddings import EmbeddingGenerator
# Default — Sentence-Transformers with all-MiniLM-L6-v2
generator = EmbeddingGenerator()
# Sentence-Transformers (default, free, local)
generator = EmbeddingGenerator(model="sentence-transformers")
embeddings = generator.generate(["Text 1", "Text 2"])
# Custom model via set_text_model
generator.set_text_model("sentence_transformers", "BAAI/bge-large-en-v1.5")
# Specific BGE model
generator = EmbeddingGenerator(model="BAAI/bge-large-en-v1.5")
embeddings = generator.generate(texts)
embeddings = generator.generate_embeddings(texts)
similarity = generator.compare_embeddings(embeddings[0], embeddings[1])
```
# OpenAI
import os
generator = EmbeddingGenerator(
model="openai",
model_name="text-embedding-3-small",
api_key=os.getenv("OPENAI_API_KEY")
)
Best for: default prototyping, no API key, good quality.
</Tab>
<Tab title="FastEmbed">
```python
from semantica.embeddings import EmbeddingGenerator
# FastEmbed (fast, CPU-optimized)
generator = EmbeddingGenerator(model="fastembed")
```
generator = EmbeddingGenerator()
generator.set_text_model("fastembed", "BAAI/bge-small-en-v1.5")
embeddings = generator.generate_embeddings(texts)
```
### Supported Models
Best for: CPU-only production, lowest latency without GPU.
</Tab>
<Tab title="OpenAI">
```python
from semantica.embeddings import OpenAIStore
import os
| Provider | Model | Dimension | Notes |
| -------- | ----- | --------- | ----- |
| `sentence-transformers` | `all-MiniLM-L6-v2` | 384 | Default, fast, free |
| `sentence-transformers` | `all-mpnet-base-v2` | 768 | Higher quality |
| `bge` | `BAAI/bge-large-en-v1.5` | 1024 | State-of-the-art retrieval |
| `fastembed` | `BAAI/bge-small-en-v1.5` | 384 | Fast, CPU-optimized |
| `openai` | `text-embedding-3-small` | 1536 | OpenAI API |
| `openai` | `text-embedding-3-large` | 3072 | OpenAI API, highest quality |
| `llama` | any Ollama model | varies | Fully local inference |
store = OpenAIStore(api_key=os.getenv("OPENAI_API_KEY"), model="text-embedding-3-small")
embedding = store.embed("Hello world")
```
Best for: highest quality (3-large), or matching an OpenAI LLM pipeline.
</Tab>
<Tab title="Ollama (local)">
```python
from semantica.embeddings import LlamaStore
store = LlamaStore(model="llama3.2", base_url="http://localhost:11434")
embedding = store.embed("Hello world")
```
Best for: air-gapped or privacy-sensitive environments — no data leaves your machine.
</Tab>
<Tab title="GPU acceleration">
```python
from semantica.embeddings import EmbeddingGenerator
# Set device via text embedder config
generator = EmbeddingGenerator(config={"text": {"device": "cuda"}})
# Apple Silicon (M1/M2/M3)
generator = EmbeddingGenerator(config={"text": {"device": "mps"}})
```
GPU reduces embedding time by 5–20× depending on batch size and model.
</Tab>
</Tabs>
### Constructor Parameters
| Parameter | Type | Default | Description |
| --------- | ---- | ------- | ----------- |
| `config` | `dict` | `None` | Config dict; `config["text"]` is passed to `TextEmbedder` |
| `**kwargs` | | | Additional key/value config merged into `config` |
Use `generator.set_text_model(method, model_name)` to switch the embedding model after construction.
## TextEmbedder
Specialized for text with automatic batching and optional disk cache:
Specialised for text workloads — adds automatic batching, progress tracking, and disk caching:
```python
from semantica.embeddings import TextEmbedder
embedder = TextEmbedder(model="sentence-transformers", cache_dir=".emb_cache")
embedder = TextEmbedder(
model="sentence-transformers",
cache_dir=".emb_cache", # persist embeddings to disk
cache_ttl=86400, # cache expiry in seconds (24h); None = never expires
batch_size=128,
show_progress=True,
)
# Single text
embedding = embedder.embed("Hello world")
embedding = embedder.embed("A knowledge graph connects entities with typed relationships.")
# Batch — processes automatically in chunks
embeddings = embedder.embed_batch(
["Text 1", "Text 2", ..., "Text 10000"],
batch_size=128,
show_progress=True
)
# Batch — auto-splits into batch_size chunks, shows progress bar
embeddings = embedder.embed_batch(texts, show_progress=True)
```
**Key behaviours:**
- Cache is keyed on text content + model name — identical texts return cached vectors instantly
- Progress bar uses `tqdm` in terminal; switches to `tqdm.notebook` in Jupyter automatically
- Large batches (> 10k texts) are chunked internally to avoid OOM on GPU
## Provider Stores
Each provider implements the `ProviderStore` interface and can be used independently:
Use provider stores directly when you need fine-grained control over a single backend:
```python
from semantica.embeddings import (
OpenAIStore, BGEStore, FastEmbedStore, LlamaStore,
ProviderStoreFactory
ProviderStoreFactory,
)
import os
# OpenAI
store = OpenAIStore(api_key=os.getenv("OPENAI_API_KEY"), model="text-embedding-3-small")
store = OpenAIStore(api_key=os.getenv("OPENAI_API_KEY"), model="text-embedding-3-small")
embedding = store.embed("Hello world")
# BGE (Sentence-Transformers wrapper)
store = BGEStore(model="BAAI/bge-large-en-v1.5")
store = BGEStore(model="BAAI/bge-large-en-v1.5", device="cpu")
embedding = store.embed("Hello world")
# FastEmbed
store = FastEmbedStore(model="BAAI/bge-small-en-v1.5")
# FastEmbed — ONNX runtime, no CUDA required
store = FastEmbedStore(model="BAAI/bge-small-en-v1.5")
embedding = store.embed("Hello world")
# LlamaStore (Ollama — fully local)
store = LlamaStore(model="llama3.2", base_url="http://localhost:11434")
# Ollama — fully local
store = LlamaStore(model="llama3.2", base_url="http://localhost:11434")
embedding = store.embed("Hello world")
# Auto-select from config
store = ProviderStoreFactory.create(provider="openai", model="text-embedding-3-small")
# Auto-select from a name string — useful in config-driven pipelines
store = ProviderStoreFactory.create(provider="bge", model="BAAI/bge-large-en-v1.5")
```
## Pooling Strategies
Control how token-level embeddings are aggregated into a single vector:
Transformer models produce one embedding per token. Pooling aggregates token embeddings into a single vector:
```python
from semantica.embeddings import (
MeanPooling, MaxPooling, CLSPooling,
AttentionPooling, HierarchicalPooling, PoolingStrategyFactory
)
<Tabs>
<Tab title="MeanPooling (default)">
```python
from semantica.embeddings import MeanPooling
# Mean pooling — default, best for most tasks
pooler = MeanPooling()
pooled = pooler.pool(token_embeddings)
pooler = MeanPooling()
pooled = pooler.pool(token_embeddings) # shape: (hidden_dim,)
```
# Max pooling — captures strongest activated features
pooler = MaxPooling()
Best for: retrieval, semantic search, and clustering — averages all token contributions.
</Tab>
<Tab title="MaxPooling">
```python
from semantica.embeddings import MaxPooling
# CLS token — good for classification tasks
pooler = CLSPooling()
pooler = MaxPooling()
pooled = pooler.pool(token_embeddings)
```
# Attention-weighted pooling
pooler = AttentionPooling()
Best for: capturing the presence of any feature — takes the max activation per dimension.
</Tab>
<Tab title="CLSPooling">
```python
from semantica.embeddings import CLSPooling
# Hierarchical: chunk-level → global mean (best for long documents)
pooler = HierarchicalPooling(chunk_size=512)
pooler = CLSPooling()
pooled = pooler.pool(token_embeddings)
```
# Create from config string
pooler = PoolingStrategyFactory.create(strategy="mean")
```
Best for: classification-style tasks; models explicitly trained with CLS pooling (BERT).
</Tab>
<Tab title="HierarchicalPooling">
```python
from semantica.embeddings import HierarchicalPooling
# Chunk text, mean-pool within chunks, then mean-pool chunks
pooler = HierarchicalPooling(chunk_size=512)
pooled = pooler.pool(token_embeddings)
```
Best for: long documents exceeding the model's max sequence length — reports, papers, contracts.
</Tab>
<Tab title="Strategy Comparison">
| Strategy | When to Use |
| -------- | ----------- |
| `mean` | Default for retrieval, semantic search, and clustering |
| `max` | When you want to capture the presence of any feature, not average presence |
| `cls` | Classification-style tasks; models explicitly trained with CLS pooling (BERT) |
| `attention` | When token importance varies significantly; slower but more accurate |
| `hierarchical` | Long documents exceeding model context length; reports, papers, contracts |
```python
from semantica.embeddings import PoolingStrategyFactory
pooler = PoolingStrategyFactory.create(strategy="mean")
```
</Tab>
</Tabs>
## GraphEmbeddingManager
Embed graph nodes and subgraphs for structural similarity and GraphRAG context:
Embed graph nodes and subgraphs for structural similarity and GraphRAG context assembly:
```python
from semantica.embeddings import GraphEmbeddingManager
from semantica.embeddings import GraphEmbeddingManager, TextEmbedder
manager = GraphEmbeddingManager(
text_embedder=TextEmbedder(model="sentence-transformers"),
graph_store=graph_store
graph_store=graph_store, # optional — for persistence
)
# Embed all nodes in the graph
# Embed all nodes — uses node label + property text
node_embeddings = manager.embed_nodes(kg)
# Embed a subgraph centered on a node (for GraphRAG context)
# Embed a subgraph centred on a node (for GraphRAG context)
subgraph_embedding = manager.embed_subgraph(
kg, center_node="Apple Inc.", hops=2
kg,
center_node="Apple Inc.",
hops=2, # include neighbours up to 2 hops away
)
# Find semantically similar nodes
# Find semantically similar nodes by ID
similar = manager.find_similar_nodes("apple_inc", top_k=5)
for node_id, score in similar:
print(f"{node_id}: {score:.3f}")
```
## VectorEmbeddingManager
**Key behaviours:**
- Node embedding combines the label, type, and all property values into a single text string before embedding
- `hops=2` captures the local neighbourhood — increase for richer context, decrease for speed
- Results from `find_similar_nodes` are sorted by cosine similarity descending
Manages the full embedding lifecycle — from raw text to stored, searchable vectors:
## Embedding Cache
The disk cache avoids recomputing embeddings for unchanged text — critical for large corpora and repeated pipeline runs:
```python
from semantica.embeddings import VectorEmbeddingManager
from semantica.vector_store import VectorStore
from semantica.embeddings import TextEmbedder
vector_store = VectorStore(backend="faiss", dimension=768)
manager = VectorEmbeddingManager(
embedder=TextEmbedder(model="sentence-transformers"),
vector_store=vector_store
embedder = TextEmbedder(
model="sentence-transformers",
cache_dir=".embeddings_cache",
cache_ttl=3600, # seconds — None means cache never expires
)
# Embed documents and store in one step
ids = manager.embed_and_store(documents, metadata=metadata_list)
# First call: computes and caches
embeddings = embedder.embed_batch(texts)
# Search by semantic similarity
results = manager.search("machine learning algorithms", top_k=10)
# Second call (same texts): returns from cache instantly
embeddings = embedder.embed_batch(texts)
```
<Note>
The Distance Intelligence module (v0.5.0) uses the same cache to avoid recomputing embeddings during N×N matrix calculations across large entity sets.
</Note>
## Similarity Computation
```python
from semantica.embeddings import calculate_similarity
# Cosine similarity (most common)
# Cosine similarity — direction only, not magnitude; most common for text
score = calculate_similarity(embedding_a, embedding_b, method="cosine")
# → 0.0 to 1.0
# → 0.0 (orthogonal / unrelated) to 1.0 (identical direction)
# Euclidean distance (converted to similarity)
# Euclidean distance converted to similarity
score = calculate_similarity(embedding_a, embedding_b, method="euclidean")
```
## GPU Acceleration
```python
# Use CUDA GPU for faster embedding generation
generator = EmbeddingGenerator(model="sentence-transformers", device="cuda")
# device options: "cpu" | "cuda" | "mps"
```
## Embedding Cache
The embedding cache is used by Distance Intelligence (v0.5.0) to avoid recomputing embeddings for large N×N distance matrix calculations:
```python
embedder = TextEmbedder(
model="sentence-transformers",
cache_dir=".embeddings_cache",
cache_ttl=3600 # seconds before cache entries expire
)
# Dot product — use when vectors are already normalised (equivalent to cosine)
score = calculate_similarity(embedding_a, embedding_b, method="dot")
```
## Convenience Functions
@@ -218,10 +427,10 @@ embedder = TextEmbedder(
```python
from semantica.embeddings import (
embed_text, generate_embeddings, calculate_similarity,
pool_embeddings, check_available_providers
pool_embeddings, check_available_providers,
)
# Single text
# Single text — fastest path
emb = embed_text("Hello world", method="sentence_transformers")
# Batch
@@ -232,6 +441,28 @@ providers = check_available_providers()
# → {"sentence_transformers": True, "fastembed": True, "openai": False}
```
## Tips and Common Pitfalls
<Warning>
**Dimension mismatch.** The dimension you pass to `VectorStore(dimension=...)` must exactly match your embedding model's output. `all-MiniLM-L6-v2` → 384, `all-mpnet-base-v2` → 768, `bge-large-en-v1.5` → 1024. Check with `generator.dimension` before creating the store.
</Warning>
<Warning>
**Not normalising for cosine similarity.** If you compute cosine similarity directly (dot product), vectors must be L2-normalised first. `EmbeddingGenerator` normalises by default (`normalize=True`). If you disable it, use `calculate_similarity(..., method="cosine")` which normalises internally.
</Warning>
<Warning>
**Sequence length limits.** Most models have a 512-token limit. Text beyond that is silently truncated. Use `TextSplitter(method="hierarchical")` + `HierarchicalPooling` for long documents.
</Warning>
<Tip>
**Always use the same model for indexing and querying.** Vectors from different models are not comparable — they live in different vector spaces. Switching models requires re-embedding your entire corpus.
</Tip>
<Tip>
**Cache invalidation.** The cache key is the text + model name. Switching models requires clearing the cache or using a different `cache_dir` — otherwise you'll get stale vectors silently returned.
</Tip>
<CardGroup cols={2}>
<Card title="Vector Store" icon="database" href="vector_store">
Store and search the generated embeddings.
@@ -240,9 +471,9 @@ providers = check_available_providers()
Chunk text before embedding for better retrieval quality.
</Card>
<Card title="KG Module" icon="diagram-project" href="kg">
Distance Intelligence uses graph embeddings.
Distance Intelligence uses graph embeddings for semantic neighbourhoods.
</Card>
<Card title="Deduplication" icon="copy" href="deduplication">
Semantic deduplication uses embeddings for entity resolution.
Semantic deduplication uses embedding distance for entity resolution.
</Card>
</CardGroup>
docs/reference/evals.md
+25 -20
View File
@@ -1,37 +1,42 @@
---
title: "Evals Module"
description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance."
description: "Evaluation framework for measuring Knowledge Graph quality, extraction accuracy, and pipeline performance — coming soon."
icon: "chart-line"
---
`semantica.evals` provides a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance. Use it to benchmark extractors, validate pipeline output, and track quality regressions across runs.
`semantica.evals` is planned as a comprehensive evaluation framework for measuring extraction accuracy, graph quality, and pipeline performance.
<Warning>
**Coming Soon** — This module is currently in active development. Documentation will be expanded in the next release.
**`semantica.evals` is not yet implemented.** The module exists as a placeholder (`__all__ = []`). No classes or functions are available for import. This page describes the planned API.
</Warning>
## Planned Capabilities
## Planned Features
The Evals module will cover five evaluation areas:
When released, `semantica.evals` will provide:
| Area | What It Measures |
| ---- | ---------------- |
| **KG Quality** | Completeness, consistency, schema compliance, coverage metrics |
| **Extraction Accuracy** | NER precision / recall / F1, relation extraction metrics |
| **Pipeline Performance** | Throughput (docs/sec), latency per step, error rates |
| **Deduplication** | Merge accuracy, false positive / negative rates |
| **Reasoning** | Inference correctness, rule coverage, derivation depth |
- **KG quality metrics** — completeness, consistency, schema compliance, coverage, and orphan node detection
- **Extraction accuracy** — NER precision / recall / F1 and relation extraction metrics against gold-standard datasets
- **Pipeline benchmarking** — throughput (docs/sec), per-step latency, peak memory, and error rate
- **Regression tracking** — record runs and compare metrics across commits or config changes
- **Deduplication accuracy** — merge precision, false positive / false negative rates
- **Reasoning correctness** — inference accuracy, rule coverage, and derivation depth
## Scope
## Current Workaround
- **Offline evaluation** — compare against gold-standard annotated datasets
- **Regression tracking** — compare pipeline runs across commits or config changes
- **Live monitoring** — record quality metrics during production pipeline runs
- **Benchmark suites** — standard NER, RE, and KG construction benchmarks
Until `semantica.evals` ships, use `semantica.ontology.OntologyEvaluator` for ontology quality metrics:
```python
from semantica.ontology import OntologyEvaluator
evaluator = OntologyEvaluator()
report = evaluator.evaluate(ontology, kg)
print(f"Coverage: {report.coverage:.2%}")
print(f"Completeness: {report.completeness:.2%}")
```
<CardGroup cols={2}>
<Card title="Semantic Extract" icon="magnifying-glass" href="semantic_extract">
Extraction module to evaluate.
Extraction module.
</Card>
<Card title="Knowledge Graph" icon="diagram-project" href="kg">
Graph quality assessment.
@@ -39,7 +44,7 @@ The Evals module will cover five evaluation areas:
<Card title="Pipeline" icon="gear" href="pipeline">
Pipeline performance metrics.
</Card>
<Card title="Deduplication" icon="copy" href="deduplication">
Deduplication accuracy evaluation.
<Card title="Ontology Evaluator" icon="sitemap" href="ontology">
Available now for ontology quality metrics.
</Card>
</CardGroup>
docs/reference/explorer.md
+256 -79
View File
@@ -6,13 +6,45 @@ icon: "map"
`semantica.explorer` is a browser-based dashboard for exploring knowledge graphs, managing ontologies, and running visual analyses — no code required after launch.
## Launch Interface
```bash
# Install and launch
pip install semantica[explorer]
# Start the Explorer dashboard
semantica-explorer --graph my_graph.json --port 8000
# Or via Python module
python -m semantica.explorer --graph my_graph.json --port 8000 --host 0.0.0.0
```
<Tip>
`semantica.explorer` is a **server process**, not a Python library. It exposes no importable classes. Use the CLI or `python -m semantica.explorer` to launch.
</Tip>
## What You Get
- **Graph Explorer** — interactive node/edge search, filtering, path highlighting, and neighborhood views
- **Ontology Hub** (v0.5.0) — browse class hierarchies, infer types, run SHACL validation, align ontologies
- **Distance Intelligence** (v0.5.0) — semantic similarity search, ego-mode neighborhood views, distance heatmaps
- **REST API** — 15+ endpoints for graph data, path finding, embeddings, and semantic search
- **CLI launcher** — `semantica-explorer` command for quick local startup
<CardGroup cols={2}>
<Card title="Graph Explorer" icon="diagram-project">
Interactive node/edge search, filtering, path highlighting, and neighborhood expansion. Indexed search at 0.004ms on 118k-node graphs.
</Card>
<Card title="Ontology Hub (v0.5.0)" icon="sitemap">
Visual ontology editor, SHACL Studio, alignment authoring, health dashboard, and version control — all in the browser.
</Card>
<Card title="Distance Intelligence (v0.5.0)" icon="circle-nodes">
Semantic similarity search, ego-mode neighborhood views, N×N distance heatmaps, and distance band classification.
</Card>
<Card title="REST API" icon="code">
15+ endpoints for graph data, path finding, embeddings, semantic search, analytics, and export — fully documented at `/docs`.
</Card>
<Card title="WebSocket Progress" icon="bolt">
Long-running exports and analyses stream progress events in real time — no polling required.
</Card>
<Card title="CLI Launcher" icon="terminal">
`semantica-explorer --graph my_graph.json` for instant local startup without writing any Python.
</Card>
</CardGroup>
## Installation
@@ -24,103 +56,248 @@ Requires `uvicorn` and `fastapi`. Included automatically with `pip install seman
## Launch
<CodeGroup>
<Steps>
<Step title="Save your graph and launch Explorer">
```python
import json
from semantica.kg import GraphBuilder
```bash CLI
# Start the explorer on a saved graph
semantica-explorer --graph my_graph.json
kg = GraphBuilder().build(entities=entities, relationships=relationships)
# Custom host and port
semantica-explorer --graph my_graph.json --host 0.0.0.0 --port 8080
# Export graph to JSON file
with open("my_graph.json", "w") as f:
json.dump({"entities": kg.entities, "relationships": kg.relationships}, f)
```
# Skip auto-opening the browser
semantica-explorer --graph my_graph.json --no-browser
```
```bash
semantica-explorer --graph my_graph.json
# → Serving at http://127.0.0.1:8000
```
</Step>
<Step title="Custom host and port">
```bash
semantica-explorer --graph my_graph.json --host 0.0.0.0 --port 8080
```python Python
from semantica.context import ContextGraph
graph = ContextGraph(advanced_analytics=True)
# ... build or load your graph ...
graph.save_to_file("my_graph.json")
import subprocess
subprocess.run(["semantica-explorer", "--graph", "my_graph.json", "--port", "8000"])
```
```python Module
# Run directly as a Python module
import subprocess
subprocess.run(["python", "-m", "semantica.explorer", "--graph", "my_graph.json"])
```
</CodeGroup>
# Skip auto-opening the browser
semantica-explorer --graph my_graph.json --no-browser
```
</Step>
<Step title="Switch graphs without restarting">
```bash
curl -X POST http://localhost:8000/api/import \
-H "Content-Type: multipart/form-data" \
-F "file=@updated_graph.json"
# Browser dashboard reloads automatically
```
</Step>
</Steps>
## CLI Reference
| Flag | Default | Description |
| ---- | ------- | ----------- |
| `--graph`, `-g` | *(required)* | Path to a ContextGraph JSON file |
| `--graph`, `-g` | *(required)* | Path to a saved graph JSON file |
| `--port`, `-p` | `8000` | Port to bind the server |
| `--host` | `127.0.0.1` | Host to bind the server |
| `--no-browser` | `false` | Skip auto-opening the browser |
| `--enable-auth` | `false` | Require `X-API-Key` header on all requests |
| `--api-key` | `None` | API key value when `--enable-auth` is set |
| `--cors-origins` | `"*"` | Comma-separated list of allowed CORS origins |
| `--log-level` | `"info"` | Uvicorn log level |
## Features
### Graph Explorer
<Tabs>
<Tab title="Graph Explorer">
Core dashboard for navigating knowledge graphs:
Core dashboard for navigating knowledge graphs:
- **Indexed search** — find any node by label or type; 0.004ms on 118k-node graphs (v0.5.0)
- **Bidirectional path finding** — trace paths between any two nodes
- **Neighbor expansion** — click any node to expand its connections
- **Filter by entity type** — focus on Person, Organization, Event, or any custom type
- **Edge label display** — relationship types shown on all edges
- **Graph declutter** — workspace layout controls for dense graphs
</Tab>
<Tab title="Ontology Hub (v0.5.0)">
Full ontology lifecycle management in the browser:
- **Indexed search** — find any node by label or type; 0.004ms on 118k-node graphs (v0.5.0)
- **Bidirectional path finding** — trace paths between any two nodes
- **Neighbor expansion** — click any node to expand its connections
- **Filter by entity type** — focus on Person, Organization, Event, or any custom type
- **Edge label display** — relationship types shown on all edges
- **Graph declutter** — workspace layout controls for dense graphs
- **Visual ontology editor** — drag-and-drop class and property authoring
- **SHACL Studio** — create, validate, and test SHACL shapes with live feedback
- **Alignment authoring** — author ontology alignments across schemas
- **Health dashboard** — graph quality metrics, validation status, coverage reports
- **Version control** — snapshot, diff, and restore ontology versions
</Tab>
<Tab title="Distance Intelligence (v0.5.0)">
Semantic neighborhood analysis centered on any node:
### Ontology Hub (v0.5.0)
- **N×N distance matrices** — pairwise semantic distances across a set of nodes
- **Ego-mode visualization** — focus on a single node's semantic neighborhood
- **Distance band classification** — nodes grouped as `near` / `mid` / `far`
- **Embedding cache** — optimized embedding reuse for large graphs
</Tab>
<Tab title="Session Management">
Thread-safe sessions with rollback protection:
Full ontology lifecycle management in the browser:
- **Visual ontology editor** — drag-and-drop class and property authoring
- **SHACL Studio** — create, validate, and test SHACL shapes with live feedback
- **Alignment authoring** — author ontology alignments across schemas
- **Health dashboard** — graph quality metrics, validation status, coverage reports
- **Version control** — snapshot, diff, and restore ontology versions
### Distance Intelligence (v0.5.0)
Semantic neighborhood analysis centered on any node:
- **N×N distance matrices** — pairwise semantic distances across a set of nodes
- **Ego-mode visualization** — focus on a single node's semantic neighborhood
- **Distance band classification** — nodes grouped as `near` / `mid` / `far`
- **Embedding cache** — optimized embedding reuse for large graphs
### Knowledge Explorer API
Full FastAPI backend accessible at `http://localhost:8000/docs`:
- 12+ export formats (RDF, Parquet, AQL, JSON-LD, and more)
- WebSocket progress streaming for long operations
- Thread-safe sessions with rollback protection
- Audit trail for all operations
- Sessions are per connected browser tab
- Write operations (annotate, import) roll back automatically on failure
- All writes appended to audit trail at `/api/provenance/audit`
- Session state held in memory — use `/api/export/json` to persist between restarts
</Tab>
</Tabs>
## API Endpoints
The FastAPI server exposes a REST API alongside the browser dashboard:
Full interactive docs at `http://localhost:8000/docs`. All endpoints available via REST.
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/api/graph/summary` | `GET` | Node count, edge count, entity types |
| `/api/graph/search` | `GET` | Full-text and type-filtered node search |
| `/api/graph/path` | `GET` | Bidirectional path between two nodes |
| `/api/graph/neighbors` | `GET` | Neighbors of a node with optional depth |
| `/api/ontology/validate` | `POST` | Run SHACL validation on the graph |
| `/api/export/{format}` | `GET` | Export graph in specified format |
| `/ws/progress` | `WS` | WebSocket stream for operation progress |
<AccordionGroup>
<Accordion title="Graph endpoints">
Full OpenAPI docs available at `http://localhost:8000/docs` when the server is running.
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/api/graph/summary` | `GET` | Node count, edge count, entity type distribution |
| `/api/graph/search` | `GET` | Indexed full-text and type-filtered node search |
| `/api/graph/node/{id}` | `GET` | Fetch a single node with all properties |
| `/api/graph/neighbors` | `GET` | Neighbors of a node — `?node_id=&depth=2` |
| `/api/graph/path` | `GET` | Bidirectional shortest path — `?source=&target=` |
| `/api/graph/subgraph` | `POST` | Extract a subgraph by node IDs or type filter |
| `/api/graph/annotate` | `POST` | Add a user annotation to a node or edge |
| `/api/graph/annotations` | `GET` | List all annotations on the graph |
</Accordion>
<Accordion title="Ontology endpoints">
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/api/ontology/classes` | `GET` | List all ontology classes and properties |
| `/api/ontology/validate` | `POST` | Run SHACL validation; returns violations |
| `/api/ontology/hierarchy` | `GET` | Class hierarchy as a tree structure |
| `/api/ontology/vocabulary` | `GET` | SKOS vocabulary terms and alt labels |
| `/api/ontology/align` | `POST` | Submit two ontologies for alignment |
| `/api/ontology/diff` | `POST` | Diff two ontology versions |
</Accordion>
<Accordion title="Provenance, Decisions & Analytics">
**Provenance:**
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/api/provenance/entity/{id}` | `GET` | Full provenance lineage for an entity |
| `/api/provenance/source/{id}` | `GET` | All entities sourced from a document |
| `/api/provenance/audit` | `GET` | Full audit trail of Explorer operations |
**Decisions:**
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/api/decisions/list` | `GET` | Paginated list of recorded decisions |
| `/api/decisions/{id}` | `GET` | Single decision with causal chain |
| `/api/decisions/search` | `GET` | Precedent search — `?query=&limit=5` |
| `/api/decisions/influence/{id}` | `GET` | Downstream influence of a decision |
**Analytics:**
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/api/analytics/centrality` | `GET` | Degree, betweenness, PageRank scores |
| `/api/analytics/communities` | `GET` | Community detection result |
| `/api/analytics/distance` | `POST` | N×N distance matrix for a node list |
| `/api/analytics/neighborhood` | `GET` | Semantic neighborhood — `?node=&radius=0.4` |
</Accordion>
<Accordion title="SPARQL, Temporal & Export">
**SPARQL & Temporal:**
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/api/sparql` | `POST` | Execute a SPARQL SELECT query |
| `/api/temporal/snapshot` | `GET` | Graph snapshot at a point in time — `?at=ISO8601` |
| `/api/temporal/range` | `GET` | Nodes/edges active in a time range |
| `/api/temporal/diff` | `POST` | Diff two temporal snapshots |
**Export & Import:**
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/api/export/{format}` | `GET` | Export in: `turtle`, `json-ld`, `ntriples`, `rdf-xml`, `parquet`, `aql`, `csv`, `owl`, `arrow`, `lpg`, `yaml`, `distance-matrix` |
| `/api/import` | `POST` | Import a graph from file (replaces current graph in session) |
</Accordion>
</AccordionGroup>
## WebSocket Progress
Long-running operations stream progress events over WebSocket at `ws://localhost:8000/ws/progress`:
```python
import asyncio, websockets, json
async def watch_progress():
async with websockets.connect("ws://localhost:8000/ws/progress") as ws:
async for message in ws:
event = json.loads(message)
print(f"[{event['operation']}] {event['step']} — {event['progress_pct']:.0f}%")
if event["status"] in ("completed", "failed"):
break
asyncio.run(watch_progress())
```
WebSocket event schema:
```json
{
"operation": "export",
"step": "serializing nodes",
"current": 3500,
"total": 10000,
"progress_pct": 35.0,
"status": "running",
"message": "Serializing 10000 nodes to Turtle...",
"error": null
}
```
`status` values: `"running"` | `"completed"` | `"failed"` | `"cancelled"`
## Performance
| Scenario | Latency |
| -------- | ------- |
| Node search (118k nodes, indexed) | 0.004ms |
| Neighbor expansion (depth 2) | < 5ms |
| Bidirectional path (118k nodes) | < 50ms |
| SPARQL SELECT (simple pattern) | < 20ms |
| N×N distance matrix (100 nodes) | ~2s (with embedding cache) |
The node search index is built on startup. For graphs > 500k nodes, allow extra startup time before connecting.
## Tips and Common Pitfalls
<Warning>
**Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph (e.g., by entity type) before exporting to JSON — Explorer's force-directed layout becomes unusable on very large graphs.
</Warning>
<Warning>
**Use authentication in shared environments.** Pass `enable_auth=True, api_key="..."` whenever Explorer is accessible to more than one person. Without auth, anyone who can reach the port can write to the graph via the annotate and import endpoints.
</Warning>
<Warning>
**Export before the session ends.** Session state lives in memory and is lost on server restart. Call `/api/export/json` or `/api/export/turtle` to persist the current state before shutting down. Explorer does not auto-save.
</Warning>
<Tip>
**Use WebSocket progress for long operations.** Export, analysis, and large SPARQL queries stream progress to `ws://localhost:8000/ws/progress`. Polling the REST endpoints instead gives no progress signal — use the WebSocket client so users see incremental updates.
</Tip>
<Tip>
**Pass `session_timeout` for demos and shared notebooks.** The default session never expires. In Jupyter or shared environments, set `session_timeout=1800` (30 minutes) so stale sessions don't hold large graphs in memory.
</Tip>
<Tip>
**Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API — pipe them into scripts to automate batch annotation, SPARQL querying, or exports. The browser UI is for interactive exploration and sharing; they use the same server.
</Tip>
<CardGroup cols={2}>
<Card title="Context" icon="brain" href="context">
@@ -129,7 +306,7 @@ Full OpenAPI docs available at `http://localhost:8000/docs` when the server is r
<Card title="Ontology" icon="sitemap" href="ontology">
Programmatic ontology management and SHACL generation.
</Card>
<Card title="Visualization" icon="chart-network" href="visualization">
<Card title="Visualization" icon="chart-bar" href="visualization">
Programmatic graph rendering without the Explorer server.
</Card>
<Card title="Export" icon="file-export" href="export">
docs/reference/export.md
+318 -181
View File
@@ -6,193 +6,292 @@ icon: "file-export"
`semantica.export` serializes knowledge graphs to every downstream format — semantic web standards, analytics pipelines, graph databases, and vector stores. All exporters share a consistent `export(graph, path, format)` interface.
## Exported Classes
```python
from semantica.export import (
RDFExporter, # Turtle, JSON-LD, N-Triples, RDF/XML
ParquetExporter, # columnar Parquet (Spark, BigQuery, Databricks, Snowflake)
LPGExporter, # Cypher CREATE/MERGE for Neo4j / Memgraph
ArangoAQLExporter, # AQL INSERT for ArangoDB
GraphExporter, # GraphML, GEXF, Graphviz DOT
OWLExporter, # OWL 2.0 in Turtle, XML, JSON-LD
CSVExporter, # flat CSV nodes + edges
VectorExporter, # embedding vectors as JSON, NumPy, or FAISS
ArrowExporter, # Apache Arrow IPC (zero-copy transfer)
DistanceExporter, # semantic distance matrices and ego-graphs
ReportGenerator, # human-readable analytics reports (HTML, Markdown, JSON)
NamespaceManager, # register and resolve RDF namespace prefixes
SemanticNetworkYAMLExporter, # YAML semantic network export
# Convenience functions
export_rdf, export_parquet, export_csv, export_lpg,
export_arango, export_graph, export_owl, export_vector,
export_arrow, generate_report,
)
```
## What You Get
- **`RDFExporter`** — Turtle, JSON-LD, N-Triples, RDF/XML with namespace management
- **`ParquetExporter`** — columnar storage for Spark, BigQuery, Databricks, Snowflake
- **`LPGExporter`** — Cypher CREATE/MERGE statements for Neo4j and Memgraph
- **`ArangoAQLExporter`** — AQL INSERT statements for ArangoDB multi-model graphs
- **`GraphExporter`** — GraphML, GEXF, DOT for visualization tools like Gephi
- **`OWLExporter`** — OWL 2.0 ontology export in Turtle, XML, and JSON-LD
- **`CSVExporter`**, **`VectorExporter`**, **`ArrowExporter`**, **`DistanceExporter`**, **`ReportGenerator`**
<CardGroup cols={2}>
<Card title="RDFExporter" icon="diagram-project">
Turtle, JSON-LD, N-Triples, RDF/XML with namespace management and optional PROV-O provenance embedding.
</Card>
<Card title="ParquetExporter" icon="layer-group">
Columnar storage for Spark, BigQuery, Databricks, and Snowflake with explicit PyArrow typing.
</Card>
<Card title="LPG & ArangoDB" icon="server">
Cypher CREATE/MERGE for Neo4j and Memgraph; AQL INSERT for ArangoDB vertex and edge collections.
</Card>
<Card title="Graph Formats" icon="chart-bar">
GraphML, GEXF, DOT for Gephi and Graphviz. OWL 2.0 ontology export in Turtle, XML, and JSON-LD.
</Card>
<Card title="Vector & Arrow" icon="vector-square">
JSON, NumPy `.npy`, and FAISS index export for embedding vectors. Apache Arrow IPC for zero-copy transfer.
</Card>
<Card title="Distance & Reports" icon="chart-line">
Distance matrix CSV/JSON from Distance Intelligence (v0.5.0). HTML, Markdown, and JSON analytics reports.
</Card>
</CardGroup>
## RDFExporter
## Quick Start
<Steps>
<Step title="Choose your format and instantiate an exporter">
```python
from semantica.export import RDFExporter
exporter = RDFExporter()
```
</Step>
<Step title="Export the graph">
```python
# Export to RDF string, then write to file
rdf_str = exporter.export_to_rdf(graph, format="turtle")
with open("output.ttl", "w") as f:
f.write(rdf_str)
```
</Step>
<Step title="Use convenience functions for one-liners">
```python
from semantica.export import export_rdf, export_parquet, export_csv
export_rdf(graph, "output.ttl", format="turtle")
export_parquet(graph, "output/", compression="snappy")
export_csv(graph, "nodes.csv", target="nodes")
```
</Step>
<Step title="Stream large graphs to avoid OOM">
```python
from semantica.export import ParquetExporter
exporter = ParquetExporter(compression="snappy")
exporter.export_stream(graph, output_dir="output/", batch_size=10_000)
```
</Step>
</Steps>
## Exporters
<Tabs>
<Tab title="RDF">
Export to W3C RDF formats — Turtle, JSON-LD, N-Triples, and RDF/XML:
```python
from semantica.export import RDFExporter
exporter = RDFExporter()
# Export to RDF string — write to file manually
rdf_str = exporter.export_to_rdf(graph, format="turtle") # Turtle (most readable)
rdf_str = exporter.export_to_rdf(graph, format="json-ld") # JSON-LD (APIs, Linked Data)
rdf_str = exporter.export_to_rdf(graph, format="nt") # N-Triples (streaming-friendly)
rdf_str = exporter.export_to_rdf(graph, format="xml") # RDF/XML (W3C standard)
with open("output.ttl", "w") as f:
f.write(exporter.export_to_rdf(graph, format="turtle"))
```
**Custom namespace management:**
```python
from semantica.export import NamespaceManager, RDFExporter
ns_manager = NamespaceManager()
ns_manager.register("ex", "http://example.org/")
ns_manager.register("schema", "https://schema.org/")
exporter = RDFExporter(namespace_manager=ns_manager)
```
**Export with PROV-O provenance:**
```python
from semantica.export import RDFExporter
from semantica.provenance import ProvenanceManager
provenance = ProvenanceManager()
# ... track entities during extraction ...
exporter = RDFExporter(include_provenance=True, provenance_manager=provenance)
exporter.export_to_file(graph, "output_with_prov.ttl", format="turtle")
# → Each entity's prov:wasGeneratedBy, prov:wasDerivedFrom, prov:hadPrimarySource
# triples are included alongside the entity data triples
```
</Tab>
<Tab title="Columnar & Analytics">
Columnar formats for analytics pipelines and human-readable export:
```python
from semantica.export import ParquetExporter
exporter = ParquetExporter(compression="snappy")
# compression options: snappy | gzip | brotli | zstd | lz4
# Export nodes and edges as separate Parquet files
exporter.export_nodes(graph, "nodes.parquet")
exporter.export_edges(graph, "edges.parquet")
# Export full graph partitioned by node type
exporter.export(graph, output_dir="graph_parquet/", partition_by="node_type")
```
Schema is explicitly typed with PyArrow for clean Spark/BigQuery ingestion.
```python
from semantica.export import CSVExporter
exporter = CSVExporter(delimiter=",")
exporter.export_nodes(graph, "nodes.csv")
exporter.export_edges(graph, "edges.csv")
```
```python
from semantica.export import SemanticNetworkYAMLExporter
exporter = SemanticNetworkYAMLExporter()
exporter.export(graph, "graph.yaml")
yaml_str = exporter.to_string(graph)
```
</Tab>
<Tab title="Graph DB Import">
Export Cypher or AQL statements for direct graph database import:
```python
from semantica.export import LPGExporter
exporter = LPGExporter()
# CREATE statements
cypher = exporter.to_cypher(graph)
exporter.export(graph, "import.cypher", format="cypher")
# MERGE statements (idempotent — safe to re-run)
cypher_merge = exporter.to_cypher(graph, use_merge=True)
```
```python
from semantica.export import ArangoAQLExporter
exporter = ArangoAQLExporter(
vertex_collection="entities",
edge_collection="relationships"
)
aql = exporter.export(graph) # returns AQL string
exporter.export_to_file(graph, "import.aql")
```
</Tab>
<Tab title="Visualization">
Export for graph visualization tools and OWL ontology distribution:
```python
from semantica.export import GraphExporter
exporter = GraphExporter()
exporter.export(graph, "graph.graphml", format="graphml") # Gephi, yEd
exporter.export(graph, "graph.gexf", format="gexf") # Gephi streaming
exporter.export(graph, "graph.dot", format="dot") # Graphviz
```
```python
from semantica.export import OWLExporter
exporter = OWLExporter()
exporter.export(ontology, path="ontology.ttl", format="turtle")
exporter.export(ontology, path="ontology.owl", format="xml")
exporter.export(ontology, path="ontology.json", format="json-ld")
```
</Tab>
<Tab title="Specialized">
Vector embeddings, Arrow IPC, distance matrices, and analytics reports:
```python
from semantica.export import VectorExporter
exporter = VectorExporter()
exporter.export(embeddings, metadata, "vectors.json", format="json")
exporter.export(embeddings, metadata, "vectors.npy", format="numpy")
exporter.export(embeddings, metadata, "vectors.faiss", format="faiss")
```
```python
from semantica.export import ArrowExporter
exporter = ArrowExporter()
exporter.export(graph, "graph.arrow") # requires pyarrow
```
```python
from semantica.export import DistanceExporter
exporter = DistanceExporter()
exporter.export_matrix(distance_matrix, node_labels, "distances.csv")
exporter.export_ego(ego_neighborhood, center_node="Apple Inc.", path="ego.json")
```
```python
from semantica.export import ReportGenerator
generator = ReportGenerator()
generator.generate(graph, analytics_result, "report.html", format="html")
generator.generate(graph, analytics_result, "report.md", format="markdown")
generator.generate(graph, analytics_result, "report.json", format="json")
```
</Tab>
</Tabs>
## Streaming Export
For graphs too large to hold in memory, use streaming export — writes incrementally without buffering the full graph:
```python
from semantica.export import RDFExporter
from semantica.export import RDFExporter, ParquetExporter
# Stream RDF — yields triples one at a time, no full-graph buffer
exporter = RDFExporter()
with exporter.stream(graph, format="turtle") as stream:
for triple_line in stream:
output_file.write(triple_line)
# Turtle (most readable RDF format)
exporter.export_to_file(graph, "output.ttl", format="turtle")
# JSON-LD (best for APIs and Linked Data)
exporter.export_to_file(graph, "output.jsonld", format="json-ld")
# N-Triples (streaming-friendly, one triple per line)
exporter.export_to_file(graph, "output.nt", format="nt")
# RDF/XML (W3C standard, broadest compatibility)
exporter.export_to_file(graph, "output.xml", format="xml")
# Export to string instead of file
rdf_str = exporter.export_to_rdf(graph, format="turtle")
```
Custom namespace management:
```python
from semantica.export import NamespaceManager, RDFExporter
ns_manager = NamespaceManager()
ns_manager.register("ex", "http://example.org/")
ns_manager.register("schema", "https://schema.org/")
exporter = RDFExporter(namespace_manager=ns_manager)
```
## ParquetExporter
Columnar export for Spark, BigQuery, Databricks, and Snowflake analytics pipelines:
```python
from semantica.export import ParquetExporter
# Stream Parquet — writes row groups incrementally
exporter = ParquetExporter(compression="snappy")
# compression options: snappy | gzip | brotli | zstd | lz4
# Export nodes and edges as separate Parquet files
exporter.export_nodes(graph, "nodes.parquet")
exporter.export_edges(graph, "edges.parquet")
# Export full graph partitioned by node type
exporter.export(graph, output_dir="graph_parquet/", partition_by="node_type")
exporter.export_stream(graph, output_dir="output/", batch_size=10_000)
```
Schema is explicitly typed with PyArrow for clean Spark/BigQuery ingestion.
Streaming is recommended for graphs with > 500k nodes.
## LPGExporter
Labeled Property Graph export — Cypher statements for Neo4j and Memgraph:
## Selective Export
```python
from semantica.export import LPGExporter
# Export a subgraph
subgraph = graph.subgraph(node_ids=["apple_inc", "steve_jobs"])
export_rdf(subgraph, "subgraph.ttl", format="turtle")
exporter = LPGExporter()
# CREATE statements
cypher = exporter.to_cypher(graph)
exporter.export(graph, "import.cypher", format="cypher")
# MERGE statements (idempotent — safe to re-run)
cypher_merge = exporter.to_cypher(graph, use_merge=True)
```
## ArangoAQLExporter
AQL INSERT statements for ArangoDB vertex and edge collections:
```python
from semantica.export import ArangoAQLExporter
exporter = ArangoAQLExporter(
vertex_collection="entities",
edge_collection="relationships"
)
aql = exporter.export(graph) # returns AQL string
exporter.export_to_file(graph, "import.aql")
```
## GraphExporter
Export for visualization tools — GraphML, GEXF, and Graphviz DOT:
```python
from semantica.export import GraphExporter
exporter = GraphExporter()
exporter.export(graph, "graph.graphml", format="graphml") # Gephi, yEd
exporter.export(graph, "graph.gexf", format="gexf") # Gephi streaming
exporter.export(graph, "graph.dot", format="dot") # Graphviz
```
## OWLExporter
OWL 2.0 ontology export in three serialization formats:
```python
from semantica.export import OWLExporter
exporter = OWLExporter()
exporter.export(ontology, path="ontology.ttl", format="turtle")
exporter.export(ontology, path="ontology.owl", format="xml")
exporter.export(ontology, path="ontology.json", format="json-ld")
```
## CSVExporter
Flat CSV export for spreadsheets and simple data pipelines:
```python
from semantica.export import CSVExporter
exporter = CSVExporter(delimiter=",")
exporter.export_nodes(graph, "nodes.csv")
exporter.export_edges(graph, "edges.csv")
```
## VectorExporter
Export embedding vectors for use in external vector stores:
```python
from semantica.export import VectorExporter
exporter = VectorExporter()
exporter.export(embeddings, metadata, "vectors.json", format="json")
exporter.export(embeddings, metadata, "vectors.npy", format="numpy")
exporter.export(embeddings, metadata, "vectors.faiss", format="faiss")
```
## ArrowExporter
Apache Arrow IPC format for zero-copy inter-process transfer:
```python
from semantica.export import ArrowExporter
exporter = ArrowExporter()
exporter.export(graph, "graph.arrow")
```
Requires `pyarrow`. Falls back gracefully if not installed.
## DistanceExporter
Export semantic distance matrices produced by Distance Intelligence (v0.5.0):
```python
from semantica.export import DistanceExporter
exporter = DistanceExporter()
exporter.export_matrix(distance_matrix, node_labels, "distances.csv")
exporter.export_ego(ego_neighborhood, center_node="Apple Inc.", path="ego.json")
```
## ReportGenerator
Generate human-readable analytics reports from graph metrics:
```python
from semantica.export import ReportGenerator
generator = ReportGenerator()
generator.generate(graph, analytics_result, "report.html", format="html")
generator.generate(graph, analytics_result, "report.md", format="markdown")
generator.generate(graph, analytics_result, "report.json", format="json")
# Export nodes by type
org_nodes = graph.filter(node_type="Organization")
export_parquet(org_nodes, "organizations.parquet")
```
## Convenience Functions
@@ -212,17 +311,55 @@ export_arango(graph, "import.aql")
export_graph(graph, "graph.graphml", format="graphml")
```
## Selective Export
## Format Reference
```python
# Export a subgraph
subgraph = graph.subgraph(node_ids=["apple_inc", "steve_jobs"])
export_rdf(subgraph, "subgraph.ttl", format="turtle")
| Format | Exporter | Output | Best For |
| ------ | -------- | ------ | -------- |
| `turtle` | `RDFExporter` | `.ttl` | Readable RDF, ontology sharing |
| `json-ld` | `RDFExporter` | `.jsonld` | APIs, Linked Data, JSON pipelines |
| `nt` | `RDFExporter` | `.nt` | Streaming RDF, line-by-line processing |
| `xml` | `RDFExporter` | `.xml` | W3C RDF/XML, broadest compatibility |
| `parquet` | `ParquetExporter` | `.parquet` | Spark, BigQuery, Databricks, Snowflake |
| `cypher` | `LPGExporter` | `.cypher` | Neo4j, Memgraph import |
| `aql` | `ArangoAQLExporter` | `.aql` | ArangoDB vertex + edge collections |
| `graphml` | `GraphExporter` | `.graphml` | Gephi, yEd visualization |
| `gexf` | `GraphExporter` | `.gexf` | Gephi streaming format |
| `dot` | `GraphExporter` | `.dot` | Graphviz rendering |
| `owl` | `OWLExporter` | `.owl` / `.ttl` | OWL 2.0 ontology distribution |
| `csv` | `CSVExporter` | `.csv` | Spreadsheets, simple pipelines |
| `yaml` | `SemanticNetworkYAMLExporter` | `.yaml` | Human-readable, config-driven use |
| `arrow` | `ArrowExporter` | `.arrow` | Zero-copy inter-process transfer |
| `numpy` | `VectorExporter` | `.npy` | NumPy arrays from embeddings |
| `faiss` | `VectorExporter` | `.faiss` | Direct FAISS index files |
| `distance-matrix` | `DistanceExporter` | `.csv` / `.json` | Distance Intelligence matrices |
| `html` | `ReportGenerator` | `.html` | Human-readable analytics reports |
| `markdown` | `ReportGenerator` | `.md` | Documentation, GitHub |
# Export nodes by type
org_nodes = graph.filter(node_type="Organization")
export_parquet(org_nodes, "organizations.parquet")
```
## Tips and Common Pitfalls
<Tip>
**Use `turtle` for human readability, `nt` for streaming.** Turtle is compact and readable for debugging and sharing ontologies. N-Triples (`.nt`) is line-oriented — one triple per line — making it safe to stream, concatenate, and process with standard Unix tools without loading the full file.
</Tip>
<Tip>
**Use `ParquetExporter` for downstream analytics.** Parquet preserves column types (int, float, datetime) that CSV loses and is natively supported by Spark, BigQuery, Databricks, and Snowflake. Use `compression="snappy"` for a good balance of speed and compression ratio.
</Tip>
<Warning>
**Stream large graphs with `export_stream()`.** For graphs with more than 500k nodes, use `exporter.export_stream(graph, ...)` instead of building the full RDF string in memory. Streaming writes incrementally — without it, a million-node export will likely OOM.
</Warning>
<Tip>
**Include provenance for compliance exports.** For HIPAA, SOX, or FDA 21 CFR Part 11 exports, pass `include_provenance=True` to `RDFExporter`. This embeds W3C PROV-O lineage triples inline — auditors can verify every fact's source from a single file rather than cross-referencing separate systems.
</Tip>
<Tip>
**Use selective export to reduce file size.** `graph.subgraph(node_ids=[...])` and `graph.filter(node_type="Organization")` let you export only the relevant subset. Full graph exports for compliance reports include noise; scoped exports are faster to produce, review, and transfer.
</Tip>
<Warning>
**Match your export format to your consumer.** Neo4j → `cypher`; ArangoDB → `aql`; Gephi/yEd → `graphml` or `gexf`; semantic web tools → `turtle` or `json-ld`; analytics pipelines → `parquet`; zero-copy IPC → `arrow`. Using the wrong format forces the consumer to convert it, adding latency and potential data loss.
</Warning>
<CardGroup cols={2}>
<Card title="Triplet Store" icon="table" href="triplet_store">
docs/reference/graph_store.md
+253 -109
View File
@@ -6,121 +6,184 @@ icon: "server"
`semantica.graph_store` provides a single API for persisting and querying knowledge graphs in production graph databases. Swap backends with a one-line change — no application code changes needed.
## What You Get
- **`GraphStore`** — unified interface across all backends
- **Backends** — Neo4j, FalkorDB, Apache AGE (PostgreSQL), Amazon Neptune, NetworkX (in-memory)
- **Cypher queries** — full Cypher support for Neo4j and FalkorDB
- **Bulk operations** — batched node and edge loading with configurable batch sizes
- **Schema management** — create indexes and uniqueness constraints
- **Path traversal** — find paths between nodes with hop limits and relationship type filters
## Basic Usage
## Exported Classes
```python
from semantica.graph_store import GraphStore
store = GraphStore(
backend="neo4j",
uri="bolt://localhost:7687",
user="neo4j",
password="password"
from semantica.graph_store import (
# Core interface
GraphStore, # unified interface: add_node, add_edge, query, find_paths
GraphManager, # store management and operations
NodeManager, # node CRUD operations
RelationshipManager, # relationship CRUD operations
QueryEngine, # Cypher query execution with caching
GraphAnalytics, # centrality, community detection, shortest path
# Backend stores
Neo4jStore, # Neo4j via Bolt — production workloads
ApacheAgeStore, # PostgreSQL + AGE extension
AmazonNeptuneStore, # AWS Neptune — SPARQL/Gremlin/openCypher
FalkorDBStore, # Redis-based — ultra-low latency
# Convenience functions
create_node, # create_node(labels, properties)
create_nodes, # bulk: create_nodes(entities)
create_relationship, # create_relationship(start_id, end_id, rel_type)
create_relationships, # bulk: create_relationships(rels)
get_nodes, # get_nodes(labels, filters)
get_relationships, # get_relationships(start_id, rel_type)
get_neighbors, # get_neighbors(node_id, direction="both")
update_node, # update_node(node_id, properties)
delete_node, # delete_node(node_id)
execute_query, # execute_query(cypher, parameters)
shortest_path, # shortest_path(source, target)
run_analytics, # run_analytics(graph, algorithm)
)
store.add_nodes(entities)
store.add_edges(relationships)
results = store.query("MATCH (n)-[r]->(m) RETURN n, r, m LIMIT 10")
```
## What You Get
<CardGroup cols={2}>
<Card title="GraphStore" icon="server">
Unified interface across Neo4j, FalkorDB, Apache AGE, Amazon Neptune, and NetworkX.
</Card>
<Card title="QueryEngine" icon="magnifying-glass">
Parameterized Cypher construction, query optimization, and result caching.
</Card>
<Card title="GraphAnalytics" icon="chart-line">
Centrality, community detection, and path algorithms running directly against the backend.
</Card>
<Card title="Bulk Operations" icon="layer-group">
Batched node and edge loading with configurable batch sizes — 10–100× faster than individual writes.
</Card>
<Card title="Schema Management" icon="table">
Create indexes and uniqueness constraints to optimize query performance.
</Card>
<Card title="Path Traversal" icon="route">
Find paths between nodes with hop limits and relationship type filters.
</Card>
</CardGroup>
## Quick Start
<Steps>
<Step title="Connect to a graph database">
```python
from semantica.graph_store import GraphStore
store = GraphStore(
backend="neo4j",
uri="bolt://localhost:7687",
user="neo4j",
password="password",
)
```
</Step>
<Step title="Create indexes before loading data">
```python
store.create_index(label="Person", property="name")
store.create_index(label="Organization", property="name")
```
</Step>
<Step title="Load nodes and edges">
```python
store.create_nodes(entities)
store.add_edges(relationships)
```
</Step>
<Step title="Query the graph">
```python
results = store.query(
"MATCH (p:Person)-[:WORKS_FOR]->(o:Organization) WHERE o.name = $org RETURN p",
parameters={"org": "Apple Inc."},
)
```
</Step>
</Steps>
## Backends
<Tabs>
<Tab title="Neo4j">
<Tab title="Neo4j (recommended)">
```python
from semantica.graph_store import GraphStore
```python
store = GraphStore(
backend="neo4j",
uri="bolt://localhost:7687",
user="neo4j",
password="password",
database="neo4j" # optional — targets default database
)
```
Best for: production workloads, complex Cypher queries, Bloom visualization.
store = GraphStore(
backend="neo4j",
uri="bolt://localhost:7687",
user="neo4j",
password="password",
database="neo4j", # optional — targets default database
)
```
Best for: production workloads, complex Cypher queries, Bloom visualization.
</Tab>
<Tab title="FalkorDB">
```python
store = GraphStore(
backend="falkordb",
host="localhost",
port=6379,
graph_name="semantica",
)
```
```python
store = GraphStore(
backend="falkordb",
host="localhost",
port=6379,
graph_name="semantica"
)
```
Best for: ultra-low latency queries over Redis protocol, edge deployments.
Best for: ultra-low latency queries over Redis protocol, edge deployments.
</Tab>
<Tab title="Apache AGE">
```python
store = GraphStore(
backend="apache_age",
connection_string="postgresql://user:pass@localhost/graphdb",
graph_name="semantica",
)
```
```python
store = GraphStore(
backend="apache_age",
connection_string="postgresql://user:pass@localhost/graphdb",
graph_name="semantica"
)
```
Best for: teams already running PostgreSQL who want graph queries without a separate service. See the [Apache AGE Guide](../graph_stores/apache_age) for setup.
Best for: teams already running PostgreSQL who want graph queries without a separate service. See the [Apache AGE Guide](../graph_stores/apache_age) for setup.
</Tab>
<Tab title="Amazon Neptune">
```python
from semantica.graph_store import GraphStore
```python
store = GraphStore(
backend="neptune",
endpoint="your-cluster.cluster-xxxx.us-east-1.neptune.amazonaws.com",
port=8182,
region="us-east-1"
)
```
# IAM authentication (recommended for production)
store = GraphStore(
backend="neptune",
endpoint="your-cluster.cluster-xxxx.us-east-1.neptune.amazonaws.com",
port=8182,
region="us-east-1",
use_iam_auth=True, # uses boto3 default credential chain
)
Best for: managed AWS deployments needing both SPARQL and Gremlin support.
# Gremlin traversal
results = store.query("g.V().hasLabel('Person').limit(10)")
# openCypher query
results = store.query(
"MATCH (p:Person)-[:WORKS_FOR]->(o:Organization) RETURN p, o",
query_language="opencypher",
)
```
Best for: managed AWS deployments needing both SPARQL and Gremlin support.
</Tab>
<Tab title="In-Memory">
<Tab title="NetworkX (in-memory)">
```python
store = GraphStore(backend="networkx")
```
```python
store = GraphStore(backend="networkx")
```
Best for: development, testing, and graphs that fit in RAM. Data is not persisted.
</Tab>
<Tab title="Backend Comparison">
Best for: development, testing, and graphs that fit in RAM. Data is not persisted.
| Backend | Query Language | Deployment | IAM Auth | Best For |
| ------- | -------------- | ---------- | -------- | -------- |
| Neo4j | Cypher | Self-hosted / Aura | No | Production, complex traversals, Bloom UI |
| FalkorDB | Cypher | Redis-based | No | Ultra-low latency, edge deployments |
| Apache AGE | OpenCypher | PostgreSQL extension | No | Teams already on Postgres |
| Amazon Neptune | SPARQL / Gremlin / openCypher | AWS managed | Yes | Cloud-native, multi-model, compliance |
| NetworkX | Python API | In-memory | No | Development, unit testing |
</Tab>
</Tabs>
## Querying
```python
# Cypher query with parameters (Neo4j, FalkorDB)
results = store.query(
"MATCH (p:Person)-[:WORKS_FOR]->(o:Organization) WHERE o.name = $org RETURN p",
parameters={"org": "Apple Inc."}
)
# Path traversal between two nodes
paths = store.find_paths(
start_node="steve_jobs",
end_node="apple_inc",
max_hops=3,
relationship_types=["FOUNDED", "WORKED_AT"]
)
```
## Graph Operations
```python
@@ -128,19 +191,19 @@ paths = store.find_paths(
store.add_node(
"apple_inc",
node_type="Organization",
properties={"founded": 1976, "hq": "Cupertino"}
properties={"founded": 1976, "hq": "Cupertino"},
)
# Add a directed relationship
store.add_edge(
"steve_jobs", "apple_inc",
"FOUNDED",
properties={"year": 1976}
properties={"year": 1976},
)
# Bulk operations — use for large datasets
store.add_nodes_bulk(entities, batch_size=1000)
store.add_edges_bulk(relationships, batch_size=1000)
store.create_nodes(entities)
store.add_edges(relationships)
# Delete
store.delete_node("node_id")
@@ -150,25 +213,90 @@ store.delete_edge("edge_id")
neighbors = store.get_neighbors(
"apple_inc",
relationship_type="HAS_EMPLOYEE",
direction="in" # "in" | "out" | "both"
direction="in", # "in" | "out" | "both"
)
# Path traversal between two nodes
paths = store.find_paths(
start_node="steve_jobs",
end_node="apple_inc",
max_hops=3,
relationship_types=["FOUNDED", "WORKED_AT"],
)
```
## Schema Management
## QueryEngine
Create indexes and constraints to improve query performance:
`QueryEngine` handles query construction, optimization, and caching:
```python
from semantica.graph_store import QueryEngine, GraphStore
store = GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", password="password")
engine = QueryEngine(store, cache_ttl=300) # cache results for 5 minutes
# Build parameterized Cypher
query, params = engine.build_query(
node_labels=["Person"],
filters={"department": "Engineering"},
return_fields=["name", "email"],
limit=50,
)
results = engine.execute(query, params)
# Explain query plan (Neo4j)
plan = engine.explain(query, params)
print(plan["profile"])
# Flush query cache
engine.clear_cache()
```
## GraphAnalytics
Built-in graph analytics that run directly against the stored backend — no data export required:
```python
from semantica.graph_store import GraphAnalytics, GraphStore
store = GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", password="password")
analytics = GraphAnalytics(store)
# Centrality
centrality = analytics.degree_centrality(node_label="Person", relationship_type="KNOWS")
betweenness = analytics.betweenness_centrality(node_label="Person")
# Community detection
communities = analytics.detect_communities(
node_label="Person",
relationship_type="KNOWS",
algorithm="louvain",
)
print(f"Detected {len(communities)} communities")
# Shortest path
path = analytics.shortest_path("alice", "charlie", relationship_type="KNOWS")
print(f"Hops: {len(path) - 1}, Path: {' → '.join(path)}")
# All paths up to max_hops
all_paths = analytics.all_paths("alice", "charlie", max_hops=4)
```
| Method | Description |
| ------ | ----------- |
| `degree_centrality(node_label, relationship_type)` | Degree-based node importance |
| `betweenness_centrality(node_label)` | Bridge-based importance |
| `pagerank(node_label, relationship_type, damping)` | PageRank scores |
| `detect_communities(node_label, relationship_type, algorithm)` | Louvain / Label Propagation |
| `shortest_path(source, target, relationship_type)` | Minimum-hop path |
| `all_paths(source, target, max_hops)` | All paths up to max depth |
## Schema Management
```python
# Index for fast label lookups
store.create_index(label="Person", property="name")
# Uniqueness constraint
store.create_constraint(
label="Organization",
property="id",
constraint_type="unique"
)
# Inspect current schema
schema = store.get_schema()
print(schema["labels"])
@@ -176,15 +304,31 @@ print(schema["indexes"])
print(schema["constraints"])
```
## Backend Comparison
## Tips and Common Pitfalls
| Backend | Query Language | Deployment | Best For |
| ------- | -------------- | ---------- | -------- |
| Neo4j | Cypher | Self-hosted / Aura | Production, complex traversals |
| FalkorDB | Cypher | Redis-based | Ultra-low latency, edge |
| Apache AGE | OpenCypher | PostgreSQL | Teams already on Postgres |
| Amazon Neptune | SPARQL / Gremlin | AWS managed | Cloud-native AWS deployments |
| NetworkX | Python API | In-memory | Development and testing |
<Tip>
**Use `NetworkX` for development, Neo4j or FalkorDB for production.** `backend="networkx"` requires zero setup and runs in memory — ideal for local development and CI tests. Switch to a persistent backend before deploying — no code changes needed, just the backend parameter.
</Tip>
<Warning>
**Create indexes before bulk loading.** `store.create_index(label="Person", property="name")` makes `MATCH` queries on `name` orders of magnitude faster. Without indexes, every query does a full scan. Create indexes first, then load data.
</Warning>
<Tip>
**Use `create_nodes()` and `add_edges()` for loading multiple nodes and edges.** Individual `add_node()` calls issue one network round-trip each. Loading in bulk is significantly faster for initial graph population.
</Tip>
<Warning>
**Use parameterized queries, never string interpolation.** `store.query("WHERE n.name = $name", parameters={"name": user_input})` prevents Cypher injection attacks. Never use `f"WHERE n.name = '{user_input}'"`.
</Warning>
<Tip>
**Enable `QueryEngine` caching for read-heavy workloads.** `QueryEngine(store, cache_ttl=300)` avoids repeated round-trips for identical queries within the cache window — useful for analytics dashboards that refresh frequently with the same aggregation queries.
</Tip>
<Warning>
**Apache AGE requires the PostgreSQL extension installed.** `backend="apache_age"` calls the AGE extension functions. If AGE is not installed in your PostgreSQL instance, you'll get a `ProgrammingError`. See the [Apache AGE Guide](../graph_stores/apache_age) for setup instructions.
</Warning>
<CardGroup cols={2}>
<Card title="KG Module" icon="diagram-project" href="kg">
docs/reference/ingest.md
+385 -147
View File
@@ -6,166 +6,385 @@ icon: "database"
`semantica.ingest` is the entry point for loading data into Semantica. Every ingestor returns a list of `DataSource` objects with normalized content and metadata, regardless of the original format.
## Exported Classes
```python
from semantica.ingest import (
# File ingestion (always available)
FileIngestor, # local files and directories: ingest(path, recursive=True)
CloudStorageIngestor, # AWS S3, Google Cloud Storage, Azure Blob Storage
FileObject, # {content, source_id, source_type, metadata, raw_bytes}
FileTypeDetector, # auto-detect file type from extension and magic bytes
ParquetIngestor, # Apache Parquet files and partitioned datasets
XMLIngestor, # XXE-safe lxml XML parsing with optional XSD validation
# Web ingestion (requires beautifulsoup4)
WebIngestor, # web scraping: ingest_url(url), crawl(url, max_pages)
FeedIngestor, # RSS/Atom feeds: ingest_feed(url), monitor_feeds(...)
FeedMonitor, # live feed monitoring with callback on new items
# Stream ingestion
StreamIngestor, # real-time: ingest_kafka/rabbitmq/kinesis/pulsar
KafkaProcessor, # Kafka consumer group processor
RabbitMQProcessor, # AMQP queue processor
KinesisProcessor, # AWS Kinesis stream processor
PulsarProcessor, # Apache Pulsar consumer
# Repository ingestion (requires gitpython)
RepoIngestor, # Git repos: ingest(url_or_path), include_commits=True
# Email ingestion
EmailIngestor, # IMAP/POP3: ingest() with attachment extraction
# Database ingestion
DBIngestor, # SQL: ingest_database(connection_string, include_tables)
SnowflakeIngestor, # Snowflake: ingest_query(sql), ingest_table(name)
OntologyIngestor, # OWL/RDF ontology files: ingest_ontology(path)
# Convenience functions
ingest, # ingest(source, source_type="file") — unified dispatcher
ingest_file, # ingest_file(path, method="directory")
ingest_web, # ingest_web(url, method="url")
ingest_feed, # ingest_feed(url)
ingest_stream, # ingest_stream(topic, ...)
ingest_database, # ingest_database(connection_string, ...)
ingest_parquet, # ingest_parquet(path, columns=[...])
ingest_xml, # ingest_xml(path, validate_xsd=None)
)
```
## What You Get
- **`FileIngestor`** — PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, ZIP/TAR archives
- **`ParquetIngestor`** — PyArrow-based Parquet with Hive-style partition support (v0.5.0)
- **`XMLIngestor`** — XXE-safe lxml with XSD/DTD validation (v0.5.0)
- **`WebIngestor`** — configurable web crawling with robots.txt support
- **`FeedIngestor`** — RSS/Atom feeds with live monitoring
- **`DBIngestor`** / **`SnowflakeIngestor`** — SQL databases and Snowflake
- **`StreamIngestor`** — Kafka, RabbitMQ, Kinesis, Pulsar real-time streams
- **`RepoIngestor`**, **`EmailIngestor`**, **`MCPIngestor`**, **`S3Ingestor`**, **`GCSIngestor`**
<CardGroup cols={2}>
<Card title="FileIngestor" icon="file">
PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, and ZIP/TAR archives — type auto-detected from extension.
</Card>
<Card title="ParquetIngestor" icon="table">
PyArrow-based Parquet with Hive-style partition support and column selection (v0.5.0).
</Card>
<Card title="XMLIngestor" icon="code">
XXE-safe lxml with XSD/DTD validation and directory scanning (v0.5.0).
</Card>
<Card title="StreamIngestor" icon="wave-square">
Real-time ingestion from Kafka, RabbitMQ, AWS Kinesis, and Apache Pulsar.
</Card>
<Card title="Cloud Storage" icon="cloud">
`CloudStorageIngestor` — unified client for AWS S3, Google Cloud Storage, and Azure Blob Storage.
</Card>
<Card title="Database Ingestors" icon="database">
`DBIngestor` (SQL via SQLAlchemy) and `SnowflakeIngestor` for data warehouse queries.
</Card>
</CardGroup>
## FileIngestor
## Quick Start
<Steps>
<Step title="Ingest local files">
```python
from semantica.ingest import FileIngestor
ingestor = FileIngestor()
# Single file — type auto-detected from extension
sources = ingestor.ingest("data/report.pdf")
# Recursive directory scan
sources = ingestor.ingest_directory("data/", recursive=True)
# Glob pattern
sources = ingestor.ingest("data/**/*.docx")
```
</Step>
<Step title="Connect to a remote source">
```python
from semantica.ingest import DBIngestor
ingestor = DBIngestor(
connection_string="postgresql://user:pass@localhost/db",
query="SELECT id, content, created_at FROM documents WHERE status='active'"
)
sources = ingestor.ingest()
```
</Step>
<Step title="Feed sources into the pipeline">
```python
from semantica.pipeline import PipelineBuilder, ExecutionEngine
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor
from semantica.llms import Groq
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
ingestor = FileIngestor()
parser = DocumentParser()
extractor = NERExtractor(method="llm", llm_provider=llm)
builder = PipelineBuilder()
builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file)
builder.add_step("parse", "document_parse", handler=parser.parse)
builder.add_step("extract", "ner_extract", handler=extractor.extract)
builder.connect_steps("ingest", "parse")
builder.connect_steps("parse", "extract")
pipeline = builder.build("my_pipeline")
result = ExecutionEngine().execute_pipeline(pipeline, data="data/")
```
</Step>
</Steps>
## Ingestors
<Tabs>
<Tab title="File-Based">
### FileIngestor
```python
from semantica.ingest import FileIngestor
ingestor = FileIngestor()
sources = ingestor.ingest("data/report.pdf")
sources = ingestor.ingest_directory("data/", recursive=True)
sources = ingestor.ingest("data/**/*.docx")
```
Supported formats: PDF, DOCX, TXT, HTML, JSON, CSV, Excel (XLSX/XLS), PPTX, ZIP/TAR archives.
### ParquetIngestor (v0.5.0)
PyArrow-based ingestion for Apache Parquet files, including Hive-style partitioned datasets:
```python
from semantica.ingest import ParquetIngestor
ingestor = ParquetIngestor()
# Single Parquet file
sources = ingestor.ingest("data/events.parquet")
# Partitioned directory (year=2024/month=01/...)
sources = ingestor.ingest("data/partitioned/")
# Load only specific columns
sources = ingestor.ingest("data/events.parquet", columns=["id", "text", "timestamp"])
```
### XMLIngestor (v0.5.0)
XXE-safe lxml-based ingestion with optional schema validation:
```python
from semantica.ingest import XMLIngestor
ingestor = XMLIngestor()
sources = ingestor.ingest("data/records.xml")
# With XSD validation
ingestor = XMLIngestor(validate_xsd="schema.xsd")
sources = ingestor.ingest("data/records/")
# With DTD validation
ingestor = XMLIngestor(validate_dtd=True)
sources = ingestor.ingest("data/feed.xml")
```
<Note>
`XMLIngestor` uses lxml with `resolve_entities=False` to prevent XML External Entity (XXE) injection attacks.
</Note>
</Tab>
<Tab title="Web & Feed">
### WebIngestor
```python
from semantica.ingest import WebIngestor
ingestor = WebIngestor(
delay=1.0, # seconds between requests
respect_robots=True, # honor robots.txt
timeout=30,
)
sources = ingestor.ingest_url("https://example.com/about")
```
### FeedIngestor (RSS/Atom)
```python
from semantica.ingest import FeedIngestor
ingestor = FeedIngestor()
feed = ingestor.ingest_feed("https://feeds.example.com/rss")
# Live monitoring — returns a FeedMonitor; callback fires on new items
monitor = ingestor.monitor_feeds(
["https://feeds.example.com/rss"],
callback=process_new_items,
)
```
### RepoIngestor
Ingest Git repositories — source code, commit history, and dependency graphs:
```python
from semantica.ingest import RepoIngestor
ingestor = RepoIngestor(
branch="main",
file_types=[".py", ".md", ".yaml"],
include_commits=True,
commit_range="HEAD~100..HEAD",
)
sources = ingestor.ingest("https://github.com/org/repo")
sources = ingestor.ingest("/path/to/local/repo")
```
### EmailIngestor
Ingest emails via IMAP or POP3 with attachment extraction and thread analysis:
```python
from semantica.ingest import EmailIngestor
import os
ingestor = EmailIngestor(
protocol="imap",
host="imap.gmail.com",
port=993,
use_ssl=True,
username=os.getenv("EMAIL_USER"),
password=os.getenv("EMAIL_PASS"),
folder="INBOX",
attachment_types=[".pdf", ".docx", ".txt"],
include_thread_analysis=True,
max_emails=500,
)
sources = ingestor.ingest()
```
</Tab>
<Tab title="Cloud Storage">
### CloudStorageIngestor
`CloudStorageIngestor` is a unified client for AWS S3, Google Cloud Storage, and Azure Blob Storage:
```python
from semantica.ingest import CloudStorageIngestor
import os
# AWS S3
ingestor = CloudStorageIngestor(
provider="s3",
bucket="my-documents-bucket",
prefix="reports/2024/",
region="us-east-1",
aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
# Omit credentials to use IAM instance profile / environment variables
)
sources = ingestor.ingest()
# Google Cloud Storage
ingestor = CloudStorageIngestor(
provider="gcs",
bucket="my-gcs-bucket",
prefix="data/",
credentials_file="gcp-credentials.json", # or use ADC
)
sources = ingestor.ingest()
# Azure Blob Storage
ingestor = CloudStorageIngestor(
provider="azure",
container="documents",
connection_string=os.getenv("AZURE_STORAGE_CONNECTION_STRING"),
)
sources = ingestor.ingest()
```
</Tab>
<Tab title="Database">
### DBIngestor (SQL)
```python
from semantica.ingest import DBIngestor
ingestor = DBIngestor()
result = ingestor.ingest_database(
connection_string="postgresql://user:pass@localhost/db",
include_tables=["documents"],
)
```
### SnowflakeIngestor
```python
from semantica.ingest import SnowflakeIngestor
import os
ingestor = SnowflakeIngestor(
account=os.getenv("SNOWFLAKE_ACCOUNT"),
user=os.getenv("SNOWFLAKE_USER"),
password=os.getenv("SNOWFLAKE_PASSWORD"),
warehouse="COMPUTE_WH",
database="ANALYTICS",
schema="PUBLIC",
)
result = ingestor.ingest_query("SELECT * FROM documents")
result = ingestor.ingest_table("documents")
```
</Tab>
<Tab title="Stream">
### StreamIngestor
Real-time ingestion from message brokers:
```python
from semantica.ingest import StreamIngestor
ingestor = StreamIngestor()
# Kafka — returns KafkaProcessor
processor = ingestor.ingest_kafka(
topic="documents",
bootstrap_servers=["localhost:9092"],
)
# RabbitMQ — returns RabbitMQProcessor
processor = ingestor.ingest_rabbitmq(
queue="document_queue",
connection_url="amqp://guest:guest@localhost/",
)
# AWS Kinesis — returns KinesisProcessor
processor = ingestor.ingest_kinesis(
stream_name="documents-stream",
region="us-east-1",
)
# Apache Pulsar — returns PulsarProcessor
processor = ingestor.ingest_pulsar(
topic="persistent://public/default/documents",
service_url="pulsar://localhost:6650",
)
```
</Tab>
</Tabs>
## OntologyIngestor
Ingest existing OWL or RDF ontology files as structured knowledge sources:
```python
from semantica.ingest import FileIngestor
from semantica.ingest import OntologyIngestor
ingestor = FileIngestor()
ingestor = OntologyIngestor()
# Single file — type auto-detected from extension
sources = ingestor.ingest("data/report.pdf")
# Recursive directory scan
sources = ingestor.ingest_directory("data/", recursive=True)
# Glob pattern
sources = ingestor.ingest("data/**/*.docx")
ontology_data = ingestor.ingest_ontology("domain_ontology.owl", format="turtle")
ontology_list = ingestor.ingest_directory("ontologies/", recursive=True)
```
Supported formats: PDF, DOCX, TXT, HTML, JSON, CSV, Excel (XLSX/XLS), PPTX, ZIP/TAR archives.
## FileObject
## ParquetIngestor (v0.5.0)
`FileIngestor` returns `FileObject` instances:
PyArrow-based ingestion for Apache Parquet files, including Hive-style partitioned datasets:
```python
from semantica.ingest import ParquetIngestor
ingestor = ParquetIngestor()
# Single Parquet file
sources = ingestor.ingest("data/events.parquet")
# Partitioned directory (year=2024/month=01/...)
sources = ingestor.ingest("data/partitioned/")
# Load only specific columns
sources = ingestor.ingest("data/events.parquet", columns=["id", "text", "timestamp"])
```
## XMLIngestor (v0.5.0)
XXE-safe lxml-based ingestion with optional schema validation:
```python
from semantica.ingest import XMLIngestor
# Basic ingestion
ingestor = XMLIngestor()
sources = ingestor.ingest("data/records.xml")
# With XSD validation
ingestor = XMLIngestor(validate_xsd="schema.xsd")
sources = ingestor.ingest("data/records/")
# With DTD validation
ingestor = XMLIngestor(validate_dtd=True)
sources = ingestor.ingest("data/feed.xml")
```
<Note>
`XMLIngestor` uses lxml with `resolve_entities=False` to prevent XML External Entity (XXE) injection attacks.
</Note>
## WebIngestor
```python
from semantica.ingest import WebIngestor
ingestor = WebIngestor(
rate_limit=1.0, # seconds between requests
respect_robots=True, # honor robots.txt
max_depth=2 # crawl depth from seed URLs
)
# Single URL
sources = ingestor.ingest("https://example.com/about")
# Multiple URLs
sources = ingestor.ingest_urls([
"https://example.com/page1",
"https://example.com/page2",
])
```
## FeedIngestor (RSS/Atom)
```python
from semantica.ingest import FeedIngestor
ingestor = FeedIngestor()
sources = ingestor.ingest("https://feeds.example.com/rss")
# Live monitoring — callback fires on new items
ingestor.monitor(
"https://feeds.example.com/rss",
interval=300,
callback=process_new_items
)
```
## DBIngestor (SQL)
```python
from semantica.ingest import DBIngestor
ingestor = DBIngestor(
connection_string="postgresql://user:pass@localhost/db",
query="SELECT id, content, created_at FROM documents WHERE status='active'"
)
sources = ingestor.ingest()
```
## SnowflakeIngestor
```python
from semantica.ingest import SnowflakeIngestor
import os
ingestor = SnowflakeIngestor(
account=os.getenv("SNOWFLAKE_ACCOUNT"),
user=os.getenv("SNOWFLAKE_USER"),
password=os.getenv("SNOWFLAKE_PASSWORD"),
warehouse="COMPUTE_WH",
database="ANALYTICS",
schema="PUBLIC"
)
sources = ingestor.ingest(query="SELECT * FROM documents")
```
## Other Ingestors
| Class | Source |
| ----- | ------ |
| `StreamIngestor` | Kafka, RabbitMQ, Kinesis, Pulsar |
| `RepoIngestor` | Git repositories (GitHub, GitLab) |
| `EmailIngestor` | IMAP/POP3 servers with attachment extraction |
| `MCPIngestor` | Model Context Protocol servers |
| `S3Ingestor` | AWS S3 buckets |
| `GCSIngestor` | Google Cloud Storage |
| `MongoIngestor` | MongoDB collections |
| `DuckDBIngestor` | DuckDB databases |
| `GDriveIngestor` | Google Drive |
## DataSource Object
All ingestors return a list of `DataSource` objects with a consistent schema:
<Accordion title="FileObject schema">
```python
@dataclass
class DataSource:
class FileObject:
content: str # raw text content
source_id: str # unique identifier
source_type: str # "file" | "web" | "database" | "stream" | ...
@@ -173,6 +392,8 @@ class DataSource:
raw_bytes: Optional[bytes] # original binary content if available
```
</Accordion>
## Custom Ingestors
Register a custom ingestor and it participates in the full pipeline:
@@ -181,12 +402,29 @@ Register a custom ingestor and it participates in the full pipeline:
from semantica.ingest.registry import method_registry
def my_ingestor(source, **kwargs):
# Return a list of DataSource-compatible dicts
return [{"content": "...", "metadata": {}, "source_id": source}]
method_registry.register("file", "my_format", my_ingestor)
```
## Tips and Common Pitfalls
<Tip>
**`FileIngestor` is always the fastest path for local files.** It auto-detects format from extension, handles ZIP/TAR archives automatically, and supports glob patterns. Only reach for `DoclingParser` when `DocumentParser` can't handle your layout.
</Tip>
<Tip>
**Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns — critical for wide tables with hundreds of columns.
</Tip>
<Warning>
**`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica — it doesn't block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML.
</Warning>
<Tip>
**Rate-limit web crawling.** `WebIngestor(delay=1.0, respect_robots=True)` is the responsible default. Without rate limiting, you risk getting blocked by the target server or violating its terms of service.
</Tip>
<CardGroup cols={2}>
<Card title="Parse" icon="file-lines" href="parse">
Parse raw sources into structured text and tables.
docs/reference/kg.md
+73 -33
View File
@@ -1,21 +1,47 @@
---
title: "Knowledge Graph Module"
description: "Graph construction, temporal models, analytics, and distance intelligence."
description: "Graph construction, temporal models, analytics, similarity scoring, and structural embeddings."
icon: "diagram-project"
---
`semantica.kg` transforms extracted entities and relationships into structured, queryable knowledge graphs. It includes temporal support, a full suite of graph analytics algorithms, node embeddings, and Distance Intelligence (v0.5.0).
`semantica.kg` transforms extracted entities and relationships into structured, queryable knowledge graphs. It includes temporal support, a full suite of graph analytics algorithms, node embeddings, and structural similarity scoring.
## Exported Classes
```python
from semantica.kg import (
KnowledgeGraph, # core graph data structure
GraphBuilder, # construct from entities + relationships
GraphBuilderWithProvenance, # auto-tracks provenance for every node/edge
EntityResolver, # entity deduplication during construction
GraphAnalyzer, # temporal evolution, diversity metrics
GraphValidator, # schema and constraint validation
TemporalGraphQuery, # point-in-time snapshots, diffs, interval queries
TemporalPatternDetector, # sequence/cycle/trend detection
TemporalVersionManager, # snapshot creation and version comparison
TemporalNormalizer, # normalize timestamps across granularities
BiTemporalFact, # bi-temporal fact model (transaction + valid time)
CentralityCalculator, # degree, betweenness, closeness, PageRank, eigenvector
CommunityDetector, # Louvain, Leiden, Label Propagation, K-Clique
PathFinder, # Dijkstra, A*, BFS, K-Shortest paths
LinkPredictor, # Preferential Attachment, Jaccard, Adamic-Adar
NodeEmbedder, # Node2Vec, DeepWalk structural embeddings
SimilarityCalculator, # cosine, Euclidean, Manhattan, correlation similarity
ConnectivityAnalyzer, # connected components, bridges, density
ProvenanceTracker, # source tracking and lineage management
)
```
## What You Get
- **`GraphBuilder`** — construct graphs from entities and relationships with automatic entity merging
- **`TemporalKnowledgeGraph`** — time-aware edges (`valid_from`/`valid_until`) and point-in-time queries (v0.4.0)
- **`DistanceCalculator`** — semantic neighborhoods, N×N distance matrices, and distance band classification (v0.5.0)
- **`TemporalGraphQuery`** — time-aware point-in-time snapshots, diffs, and Allen interval queries (v0.4.0)
- **`SimilarityCalculator`** — cosine, Euclidean, Manhattan, and correlation similarity scoring
- **`CentralityCalculator`** — PageRank, degree, betweenness, closeness, eigenvector centrality
- **`CommunityDetector`** — Louvain, Leiden, Label Propagation, K-Clique community detection
- **`PathFinder`** — Dijkstra, A\*, BFS, K-Shortest path algorithms
- **`LinkPredictor`** — Preferential Attachment, Jaccard, Adamic-Adar link prediction
- **`NodeEmbedder`** — Node2Vec, DeepWalk, Word2Vec structural embeddings
- **`NodeEmbedder`** — Node2Vec, DeepWalk structural embeddings
<Tip>
For conflict detection and advanced entity resolution, use `semantica.conflicts` and `semantica.deduplication` alongside this module.
@@ -42,52 +68,66 @@ kg = builder.build(entities=entities, relationships=relationships)
## Temporal Knowledge Graphs (v0.4.0)
Attach `valid_from` / `valid_until` time windows to nodes and edges for point-in-time queries and historical analysis:
Use `TemporalGraphQuery` to attach `valid_from`/`valid_until` windows and query time-aware graphs:
```python
from semantica.kg import TemporalKnowledgeGraph, TemporalGraphQuery
from semantica.kg import GraphBuilder, TemporalGraphQuery, TemporalVersionManager
from datetime import datetime
tkg = TemporalKnowledgeGraph()
# Nodes and edges carry explicit validity windows
tkg.add_node("ceo_role", valid_from=datetime(2020, 1, 1), valid_until=datetime(2023, 6, 1))
tkg.add_edge(
"alice", "acme_corp", "ceo_of",
valid_from=datetime(2020, 1, 1),
valid_until=datetime(2023, 6, 1)
)
# Build a time-aware graph
builder = GraphBuilder()
kg = builder.build(sources=[
{
"entities": [
{"id": "alice", "type": "Person"},
{"id": "acme_corp", "type": "Organization"},
],
"relationships": [
{
"source": "alice", "target": "acme_corp", "type": "ceo_of",
"valid_from": "2020-01-01",
"valid_until": "2023-06-01",
}
]
}
])
# Point-in-time snapshot
snapshot = tkg.at(datetime(2021, 6, 15))
query = TemporalGraphQuery(kg)
snapshot_2021 = query.at_time("2021-06-15")
snapshot_2023 = query.at_time("2023-01-01")
# Diff between two snapshots
query = TemporalGraphQuery(tkg)
snapshot_2020 = query.at_time("2020-01-01")
snapshot_2023 = query.at_time("2023-01-01")
diff = snapshot_2023.minus(snapshot_2020)
print(f"New nodes since 2020: {len(diff.nodes)}")
diff = query.diff("2020-01-01", "2023-01-01")
print(f"New nodes since 2020: {len(diff.get('added_nodes', []))}")
# Versioned snapshots
versioner = TemporalVersionManager()
versioner.create_snapshot(kg, version_label="2024-Q1")
```
Supports all 13 Allen interval algebra relations (before, after, meets, overlaps, during, starts, finishes, equals, and their inverses). OWL-Time export available.
## Distance Intelligence (v0.5.0)
## Similarity Scoring
Semantic neighborhood exploration for any entity in the graph:
`SimilarityCalculator` computes cosine, Euclidean, Manhattan, and correlation similarity between node embeddings:
```python
from semantica.kg import DistanceCalculator
from semantica.kg import SimilarityCalculator, NodeEmbedder
calc = DistanceCalculator(kg)
# First compute structural embeddings
embedder = NodeEmbedder(method="node2vec", embedding_dimension=128)
embeddings = embedder.compute_embeddings(kg, ["Person", "Organization"], ["RELATED_TO"])
# Semantic neighborhood of a single node
neighborhood = calc.semantic_neighborhood("Apple Inc.", radius=0.4)
# Then compare nodes by embedding similarity
calc = SimilarityCalculator()
score = calc.cosine_similarity(embeddings["Apple Inc."], embeddings["Google"])
print(f"Apple–Google structural similarity: {score:.3f}")
# N×N pairwise distance matrix
matrix = calc.distance_matrix(["Apple Inc.", "Google", "Microsoft"])
# Classify nodes into distance bands: "near" | "mid" | "far"
bands = calc.classify_bands(neighborhood)
# Find structurally similar nodes
similar = embedder.find_similar_nodes(kg, "Apple Inc.", top_k=5)
for node in similar:
print(f"{node['id']}: {node['score']:.3f}")
```
## Graph Analytics
docs/reference/llms.md
+83 -89
View File
@@ -1,19 +1,35 @@
---
title: "LLMs Module"
description: "Unified interface for Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, and HuggingFace."
description: "Unified interface for Groq, OpenAI, LiteLLM (Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, 100+ models), and HuggingFace."
icon: "microchip"
---
`semantica.llms` provides a single consistent API across 8+ LLM providers. Every provider is a drop-in replacement for the `llm_provider=` parameter in extractors, reasoning engines, and agents.
`semantica.llms` provides a single consistent API across every major LLM provider. Every provider is a drop-in replacement for the `llm_provider=` parameter in extractors, reasoning engines, and agents.
## Exported Classes
```python
from semantica.llms import Groq, OpenAI, LiteLLM, HuggingFaceLLM
```
| Class | Provider | API Key Required |
| ----- | -------- | ---------------- |
| `Groq` | Groq Cloud | `GROQ_API_KEY` |
| `OpenAI` | OpenAI / any OpenAI-compatible gateway | `OPENAI_API_KEY` |
| `LiteLLM` | 100+ providers via LiteLLM routing | Depends on model |
| `HuggingFaceLLM` | Local HuggingFace Transformers | None (local) |
<Tip>
**Anthropic, Gemini, Ollama, DeepSeek, Azure, Bedrock, Cohere, and 90+ others** are all available via `LiteLLM` using their model-string prefix. See the [LiteLLM section](#litellm-100-providers) below.
</Tip>
## What You Get
- **8+ provider integrations** — Groq, OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Novita AI, LiteLLM, HuggingFace
- **Unified `LLMProvider` interface** — swap providers with a one-line change, no application changes needed
- **`ProviderFactory`** — instantiate any provider by name from a config dict
- **Local models** — Ollama and HuggingFace run fully on-premise with no API key
- **Unified `LLMProvider` interface** — swap providers with a one-line change, no application code changes
- **`LiteLLM`** — single class for 100+ providers using model-string routing
- **Local models** — `HuggingFaceLLM` runs fully on-premise, no API key
- **Streaming** — token-by-token output for low-latency UX
- **Custom gateways** — point any OpenAI-compatible endpoint via `base_url`
- **Custom gateways** — point `OpenAI` at any OpenAI-compatible endpoint via `base_url`
## Providers
@@ -29,86 +45,53 @@ llm = Groq(
max_tokens=64000,
temperature=0.0,
)
# Best for: high-throughput extraction, fast inference
# Best for: high-throughput extraction, fast inference at low cost
```
```python OpenAI
from semantica.llms import OpenAI
import os
# pip install "semantica[llm-openai]"
llm = OpenAI(
model="gpt-4o",
api_key=os.getenv("OPENAI_API_KEY"),
temperature=0.0,
)
# Best for: general purpose, function calling
# Best for: general purpose, function calling, JSON mode
```
```python Anthropic
from semantica.llms import Anthropic
import os
# pip install "semantica[llm-anthropic]"
llm = Anthropic(
model="claude-opus-4-7",
api_key=os.getenv("ANTHROPIC_API_KEY"),
max_tokens=8192,
)
# Best for: complex reasoning, long context, safety
```
```python Gemini
from semantica.llms import Gemini
import os
# pip install "semantica[llm-gemini]"
llm = Gemini(
model="gemini-1.5-pro",
api_key=os.getenv("GOOGLE_API_KEY"),
)
# Best for: long context (1M tokens), multimodal tasks
```
```python Ollama (Local)
from semantica.llms import Ollama
# pip install "semantica[llm-ollama]"
llm = Ollama(
model="llama3.2:3b",
base_url="http://localhost:11434",
)
# Best for: local inference, air-gapped environments
# No API key required
```
```python DeepSeek
from semantica.llms import DeepSeek
import os
llm = DeepSeek(
model="deepseek-chat",
api_key=os.getenv("DEEPSEEK_API_KEY"),
)
# Best for: coding tasks and analysis at very low cost
```
```python LiteLLM (100+ models)
```python LiteLLM (100+ providers)
from semantica.llms import LiteLLM
import os
# pip install "semantica[llm-litellm]"
llm = LiteLLM(
model="gpt-4o", # any LiteLLM-supported model string
api_key=os.getenv("OPENAI_API_KEY"),
)
# Supports: OpenAI, Anthropic, Gemini, Cohere, Azure, Bedrock, and 90+ more
# Anthropic Claude
llm = LiteLLM(model="anthropic/claude-opus-4-5", api_key=os.getenv("ANTHROPIC_API_KEY"))
# Google Gemini
llm = LiteLLM(model="gemini/gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY"))
# Ollama (local — no API key)
llm = LiteLLM(model="ollama/llama3.2:3b", api_base="http://localhost:11434")
# DeepSeek
llm = LiteLLM(model="deepseek/deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY"))
# Azure OpenAI
llm = LiteLLM(model="azure/gpt-4o", api_key=os.getenv("AZURE_API_KEY"))
# AWS Bedrock
llm = LiteLLM(model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0")
# Novita AI
llm = LiteLLM(model="novita/deepseek/deepseek-v3.2", api_key=os.getenv("NOVITA_API_KEY"))
```
```python HuggingFace (BYOM)
from semantica.llms import HuggingFace
```python HuggingFaceLLM (Local)
from semantica.llms import HuggingFaceLLM
llm = HuggingFace(
llm = HuggingFaceLLM(
model="mistralai/Mistral-7B-Instruct-v0.3",
device="cuda", # "cpu" | "cuda" | "mps"
max_new_tokens=512,
@@ -119,23 +102,34 @@ llm = HuggingFace(
</CodeGroup>
## Provider Factory
## LiteLLM — 100+ Providers
Instantiate any provider by name string — useful when provider is loaded from config:
`LiteLLM` is the recommended way to access any provider not directly exported by `semantica.llms`. Use the `provider/model` string format:
```python
from semantica.llms import create_provider
from semantica.llms import LiteLLM
import os
llm = create_provider("groq", model="llama-3.3-70b-versatile")
llm = create_provider("openai", model="gpt-4o")
llm = create_provider("anthropic", model="claude-opus-4-7")
llm = create_provider("gemini", model="gemini-1.5-pro")
llm = create_provider("ollama", model="llama3.2")
llm = create_provider("deepseek", model="deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY"))
llm = create_provider("novita", model="deepseek/deepseek-v3.2", api_key=os.getenv("NOVITA_API_KEY"))
llm = create_provider("litellm", model="gpt-4o")
# Pattern: LiteLLM(model="<provider>/<model-name>")
providers = {
"Anthropic": LiteLLM(model="anthropic/claude-opus-4-5", api_key=os.getenv("ANTHROPIC_API_KEY")),
"Gemini": LiteLLM(model="gemini/gemini-1.5-pro", api_key=os.getenv("GOOGLE_API_KEY")),
"Ollama": LiteLLM(model="ollama/llama3.2:3b", api_base="http://localhost:11434"),
"DeepSeek": LiteLLM(model="deepseek/deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY")),
"Azure": LiteLLM(model="azure/gpt-4o", api_key=os.getenv("AZURE_API_KEY")),
"Bedrock": LiteLLM(model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"),
"Cohere": LiteLLM(model="cohere/command-r-plus", api_key=os.getenv("COHERE_API_KEY")),
"Novita AI": LiteLLM(model="novita/deepseek/deepseek-v3.2", api_key=os.getenv("NOVITA_API_KEY")),
}
# Every LiteLLM instance implements the same .generate() interface
response = providers["Anthropic"].generate("Explain GraphRAG in one paragraph.")
```
<Note>
The full list of supported LiteLLM model strings is at [docs.litellm.ai/docs/providers](https://docs.litellm.ai/docs/providers). Use the `provider/model` format shown above.
</Note>
## Custom / Enterprise Gateways
Any OpenAI-compatible endpoint — internal routing layers, Qwen proxies, or private LLaMA deployments:
@@ -170,17 +164,17 @@ trip = TripletExtractor(method="llm", llm_provider=llm)
## Provider Comparison
| Provider | Speed | Cost | Local | Context | Best For |
| -------- | ----- | ---- | ----- | ------- | -------- |
| Groq | Very fast | Low | No | 128k | High-throughput extraction |
| OpenAI | Fast | Medium | No | 128k | General purpose, function calling |
| Anthropic | Fast | Medium | No | 200k | Complex reasoning, safety |
| Gemini | Fast | Low | No | 1M | Long context, multimodal |
| Ollama | Medium | Free | Yes | Varies | Privacy, no API key |
| DeepSeek | Fast | Very low | No | 64k | Coding, analysis |
| Novita AI | Fast | Low | No | Varies | DeepSeek-based tasks |
| LiteLLM | Varies | Varies | Varies | Varies | Multi-provider routing |
| HuggingFace | Slow | Free | Yes | Varies | Custom models, BYOM |
| Provider | Import | Speed | Cost | Local | Context | Best For |
| -------- | ------ | ----- | ---- | ----- | ------- | -------- |
| Groq | `Groq` | Very fast | Low | No | 128k | High-throughput extraction |
| OpenAI | `OpenAI` | Fast | Medium | No | 128k | General purpose, function calling |
| Anthropic | `LiteLLM(model="anthropic/...")` | Fast | Medium | No | 200k | Complex reasoning, safety |
| Gemini | `LiteLLM(model="gemini/...")` | Fast | Low | No | 1M | Long context, multimodal |
| Ollama | `LiteLLM(model="ollama/...")` | Medium | Free | Yes | Varies | Privacy, air-gapped |
| DeepSeek | `LiteLLM(model="deepseek/...")` | Fast | Very low | No | 64k | Coding, analysis |
| Azure OpenAI | `LiteLLM(model="azure/...")` | Fast | Medium | No | 128k | Enterprise, compliance |
| AWS Bedrock | `LiteLLM(model="bedrock/...")` | Fast | Varies | No | Varies | AWS-native workloads |
| HuggingFace | `HuggingFaceLLM` | Slow | Free | Yes | Varies | Custom models, BYOM |
<Tip>
For production extraction pipelines, Groq delivers the best throughput-to-cost ratio. For complex multi-hop reasoning, Claude Opus or GPT-4o provide the highest accuracy.
docs/reference/mcp_server.md
+166 -50
View File
@@ -6,15 +6,56 @@ icon: "plug"
`semantica.mcp_server` exposes Semantica's knowledge graph, decision intelligence, semantic extraction, and reasoning capabilities as an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server over stdio.
Once configured, any connected AI assistant can extract entities, record decisions, query the graph, run reasoning, and export results — without writing a single line of Python.
Compatible with **Claude Desktop**, **Windsurf**, **Cline**, **Continue**, **VS Code**, **Roo Code**, **Cursor**, and any MCP-aware client.
## Server Interface
```json
// Configure in your MCP client (Claude Desktop, Windsurf, Cursor, VS Code, etc.)
{
"mcpServers": {
"semantica": {
"command": "semantica-mcp"
}
}
}
```
```bash
# Or run directly
semantica-mcp
# or
python -m semantica.mcp_server
```
<Tip>
`semantica.mcp_server` is a **stdio server process**, not a Python library. It exposes no importable classes — all interaction happens through MCP tool calls from a connected AI client.
</Tip>
## What You Get
- **12 MCP tools** — extract entities, build graphs, run SPARQL, find paths, get recommendations, embed, cluster, and more
- **3 readable resources** — live graph JSON, entity list, and relationship list
- **Zero infrastructure** — runs over stdio, no server or port needed
- **Claude Desktop ready** — one config block to add to `claude_desktop_config.json`
- **REST alternative** — the Explorer module offers a full HTTP API if you prefer
<CardGroup cols={2}>
<Card title="12 MCP Tools" icon="wrench">
Extract entities, extract relations, record decisions, query decisions, find precedents, trace causal chains, add entities, add relationships, run analytics, summarise graph, run reasoning, export.
</Card>
<Card title="3 Readable Resources" icon="book-open">
Live graph JSON (`semantica://graph/summary`), decision list, and schema/version info — readable by any MCP client.
</Card>
<Card title="Zero Infrastructure" icon="bolt">
Runs over stdio — no server, no port, no Docker required. One config block to activate in any MCP client.
</Card>
<Card title="Persistent Graphs" icon="database">
Point `SEMANTICA_KG_PATH` at a saved graph file to reload it automatically on every server startup.
</Card>
<Card title="Decision Intelligence" icon="brain">
Record decisions, find precedents via hybrid similarity search, and trace causal chains across agent runs.
</Card>
<Card title="REST Alternative" icon="globe">
The [Explorer](explorer) module offers a full HTTP API and browser dashboard if you prefer programmatic access.
</Card>
</CardGroup>
## Installation
@@ -26,46 +67,86 @@ The MCP server is included in the base install — no extras required.
## Configuration
Add Semantica to your MCP client's settings file:
<Steps>
<Step title="Find your MCP client's settings file">
<CodeGroup>
| Client | Settings file |
| ------ | ------------- |
| Claude Desktop (macOS) | `~/Library/Application Support/Claude/claude_desktop_config.json` |
| Claude Desktop (Windows) | `%APPDATA%\Claude\claude_desktop_config.json` |
| Cursor | `.cursor/mcp.json` in your project, or `~/.cursor/mcp.json` globally |
| VS Code / Continue | `.vscode/mcp.json` or user settings |
| Windsurf / Cline / Roo Code | App-specific settings → MCP Servers |
```json Claude Desktop / Windsurf / Cline
{
"mcpServers": {
"semantica": {
"command": "semantica-mcp"
}
}
}
```
</Step>
<Step title="Add the Semantica MCP server config">
```json VS Code / Continue / Roo Code
{
"mcpServers": {
"semantica": {
"command": "python",
"args": ["-m", "semantica.mcp_server"]
}
}
}
```
<CodeGroup>
```json With persistent graph
{
"mcpServers": {
"semantica": {
"command": "semantica-mcp",
"env": {
"SEMANTICA_KG_PATH": "/path/to/my_graph.json",
"SEMANTICA_LOG_LEVEL": "INFO"
```json Claude Desktop / Windsurf / Cline
{
"mcpServers": {
"semantica": {
"command": "semantica-mcp"
}
}
}
}
}
```
```
</CodeGroup>
```json Cursor
{
"mcpServers": {
"semantica": {
"command": "semantica-mcp",
"env": {
"SEMANTICA_KG_PATH": "/path/to/my_graph.json"
}
}
}
}
```
```json VS Code / Continue / Roo Code
{
"mcpServers": {
"semantica": {
"command": "python",
"args": ["-m", "semantica.mcp_server"]
}
}
}
```
```json With persistent graph
{
"mcpServers": {
"semantica": {
"command": "semantica-mcp",
"env": {
"SEMANTICA_KG_PATH": "/path/to/my_graph.json",
"SEMANTICA_LOG_LEVEL": "INFO"
}
}
}
}
```
</CodeGroup>
</Step>
<Step title="Test locally before configuring your client">
```bash
# Run the server directly (reads from stdin, writes to stdout)
semantica-mcp
# Or via Python module
python -m semantica.mcp_server
# Send a JSON-RPC initialize message to confirm it's working
echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}' | semantica-mcp
```
</Step>
</Steps>
## Environment Variables
@@ -78,6 +159,21 @@ Add Semantica to your MCP client's settings file:
The MCP server exposes 12 tools that any connected AI assistant can call:
| Tool | Category | Description |
| ---- | -------- | ----------- |
| `extract_entities` | Extraction | NER — find people, places, organisations, concepts |
| `extract_relations` | Extraction | Typed relation and triplet extraction |
| `record_decision` | Decision Intelligence | Save a decision with reasoning and outcome |
| `query_decisions` | Decision Intelligence | Search recorded decisions by natural language |
| `find_precedents` | Decision Intelligence | Hybrid similarity search over past decisions |
| `get_causal_chain` | Decision Intelligence | Trace upstream / downstream causal chains |
| `add_entity` | Graph Operations | Add a node to the live graph |
| `add_relationship` | Graph Operations | Add a directed edge between two nodes |
| `get_graph_analytics` | Graph Operations | PageRank + community detection |
| `get_graph_summary` | Graph Operations | Node count, decision count, health status |
| `run_reasoning` | Reasoning & Export | Forward-chain IF/THEN rules over facts |
| `export_graph` | Reasoning & Export | Serialise the graph (Turtle, JSON-LD, JSON, etc.) |
### Knowledge Extraction
<AccordionGroup>
@@ -87,11 +183,13 @@ The MCP server exposes 12 tools that any connected AI assistant can call:
Extract named entities (people, places, organisations, concepts) from text using Semantica NER.
**Input:**
```json
{ "text": "Apple Inc. was founded by Steve Jobs in Cupertino in 1976." }
```
**Output:**
```json
{
"entities": [
@@ -110,11 +208,13 @@ Extract named entities (people, places, organisations, concepts) from text using
Extract typed relations and `(subject, predicate, object)` triplets from text.
**Input:**
```json
{ "text": "Steve Jobs founded Apple Inc. and led it until 2011." }
```
**Output:**
```json
{
"relations": [
@@ -139,6 +239,7 @@ Extract typed relations and `(subject, predicate, object)` triplets from text.
Record a decision with full context, reasoning, and metadata into the knowledge graph.
**Input:**
```json
{
"category": "model_selection",
@@ -151,6 +252,7 @@ Record a decision with full context, reasoning, and metadata into the knowledge
```
**Output:**
```json
{ "decision_id": "dec_a1b2c3", "status": "recorded" }
```
@@ -162,6 +264,7 @@ Record a decision with full context, reasoning, and metadata into the knowledge
Query recorded decisions by natural language, category, or retrieve all recent decisions.
**Input:**
```json
{ "query": "model selection", "limit": 5 }
```
@@ -173,6 +276,7 @@ Query recorded decisions by natural language, category, or retrieve all recent d
Find past decisions similar to a given scenario using hybrid similarity search.
**Input:**
```json
{ "scenario": "Choose cloud provider for HIPAA workload", "max_results": 3 }
```
@@ -184,6 +288,7 @@ Find past decisions similar to a given scenario using hybrid similarity search.
Trace the causal chain upstream or downstream from a decision.
**Input:**
```json
{ "decision_id": "dec_a1b2c3", "direction": "downstream", "max_depth": 5 }
```
@@ -201,6 +306,7 @@ Trace the causal chain upstream or downstream from a decision.
Add a node/entity to the live knowledge graph.
**Input:**
```json
{
"id": "apple_inc",
@@ -217,6 +323,7 @@ Add a node/entity to the live knowledge graph.
Add a directed relationship (edge) between two existing entities.
**Input:**
```json
{
"source": "steve_jobs",
@@ -251,6 +358,7 @@ Return node count, decision count, and graph health status.
Run forward-chaining IF/THEN rules over a set of facts to derive new facts.
**Input:**
```json
{
"facts": ["Employee(John)", "Manager(John)"],
@@ -259,6 +367,7 @@ Run forward-chaining IF/THEN rules over a set of facts to derive new facts.
```
**Output:**
```json
{ "derived_facts": ["HasAuthority(John)"] }
```
@@ -270,6 +379,7 @@ Run forward-chaining IF/THEN rules over a set of facts to derive new facts.
Export the current knowledge graph to a serialization format.
**Input:**
```json
{ "format": "json-ld" }
```
@@ -282,7 +392,7 @@ Supported formats: `turtle`, `ttl`, `nt`, `xml`, `json-ld`, `json`.
## Resources
The MCP server also exposes three readable resources:
The MCP server exposes three readable resources:
| URI | Description |
| --- | ----------- |
@@ -290,21 +400,27 @@ The MCP server also exposes three readable resources:
| `semantica://decisions/list` | All recorded decisions (up to 50) |
| `semantica://schema/info` | Server version and available tools |
## Test Locally
## Tips and Common Pitfalls
```bash
# Run the server directly (reads from stdin, writes to stdout)
semantica-mcp
<Warning>
**Build the `ContextGraph` before starting the server.** The MCP server operates on a pre-built `ContextGraph` — it doesn't build the knowledge graph on demand. Construct and populate the graph first (ingest → extract → build KG → set `ContextGraph`), then pass it to `SemanticaMCPServer`. An empty or None graph results in empty query responses.
</Warning>
# Or via Python module
python -m semantica.mcp_server
```
<Tip>
**Use `decision_tracking=True` for accountable agents.** Without decision tracking, `record_decision` and `query_decisions` calls succeed but nothing is stored. Enable it in the `ContextGraph` constructor when you want agents' decisions to be queryable for audit, compliance, or iterative reasoning.
</Tip>
Send a JSON-RPC `initialize` message to confirm it's working:
<Tip>
**Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path — it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs.
</Tip>
```bash
echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}' | semantica-mcp
```
<Warning>
**Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently — the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works.
</Warning>
<Warning>
**The server communicates over stdio — don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. Configure logging to write to a file or stderr only (`logging.basicConfig(filename="mcp.log")`). The MCP protocol assumes stdout carries only JSON-RPC frames.
</Warning>
<CardGroup cols={2}>
<Card title="Context" icon="brain" href="context">
docs/reference/normalize.md
+371 -192
View File
@@ -1,275 +1,454 @@
---
title: "Normalize Module"
description: "Text cleaning, entity canonicalization, date normalization, number/unit conversion, language detection, and encoding repair."
description: "Text cleaning, entity canonicalization, date normalization, number conversion, language detection, and encoding repair — before extraction runs."
icon: "broom"
---
`semantica.normalize` standardizes raw data before extraction and graph construction — fixing encodings, canonicalizing entity names, normalizing dates, and detecting languages. All normalizers expose both convenience functions and stateful class instances.
`semantica.normalize` standardizes raw data before extraction and graph construction. All normalizers expose both convenience functions (one-liners) and stateful class instances (full control over configuration and reuse).
## Why Normalize Before Extraction
Unstructured data is inconsistent by nature. Without normalization, the same real-world entity appears as dozens of variants in your graph:
- `"Apple Inc."`, `"Apple Computer Inc."`, `"APPLE INC."`, `"Apple, Inc."` — four nodes, one company
- `"Jan 1st, 2020"`, `"01/01/2020"`, `"2020-01-01"` — three formats, one date
- `"$1.2B"`, `"1,200,000,000"`, `"1.2 billion USD"` — three strings, one number
- `"Hello World"` vs `"Hello World"` — a non-breaking space that breaks string matching
Normalization collapses these variants before any extractor, deduplicator, or graph builder sees the data — producing cleaner entities, fewer false duplicates, and more reliable downstream results.
## Exported Classes
```python
from semantica.normalize import (
# Text normalization
TextNormalizer, # coordinator: strip_html, normalize_unicode, fix_encoding
UnicodeNormalizer, # NFC/NFD/NFKC/NFKD normalization
WhitespaceNormalizer, # collapse spaces, normalize line endings
SpecialCharacterProcessor, # smart quotes, dashes, diacritics
TextCleaner, # general text cleaning utilities
# Entity normalization
EntityNormalizer, # coordinator: normalize_entity(text, entity_type)
AliasResolver, # resolve "ML" -> "Machine Learning" via dictionary
EntityDisambiguator, # disambiguate("Apple", context=...) with confidence
NameVariantHandler, # normalize("Dr. JOHN P. SMITH Jr.") -> "John P. Smith"
# Date/time normalization
DateNormalizer, # normalize_date(str) -> ISO 8601
TimeZoneNormalizer, # normalize to UTC or target timezone
RelativeDateProcessor, # "3 days ago" -> datetime
TemporalExpressionParser, # "Q2 2023" -> {start, end, type}
# Number normalization
NumberNormalizer, # normalize_number("$1.2B") -> 1200000000.0
UnitConverter, # convert(100, from_unit="km/h", to_unit="m/s")
CurrencyNormalizer, # normalize("$42.50") -> {amount, currency, raw}
ScientificNotationHandler, # parse scientific notation strings
# Data cleaning
DataCleaner, # remove_duplicates, fill_missing
DataValidator, # validate(records, schema={"name": str, "age": int})
DuplicateDetector, # detect duplicate records by similarity threshold
MissingValueHandler, # fill missing values: mean/median/mode/constant
# Language & encoding
LanguageDetector, # detect(text) -> {language, confidence}
EncodingHandler, # detect_encoding, to_utf8, remove_bom
# Convenience functions
normalize_text, # normalize_text(text, method="default")
normalize_entity, # normalize_entity(name, entity_type="Person")
normalize_date, # normalize_date("Jan 1st, 2020")
normalize_number, # normalize_number("$1,234.56")
clean_text, # clean_text(text)
detect_language, # detect_language(text)
resolve_aliases, # resolve_aliases(text, aliases_dict)
)
```
## What You Get
- **`TextNormalizer`** — Unicode, whitespace, HTML stripping, smart-quote/dash replacement
- **`EntityNormalizer`** — alias resolution, disambiguation, name variant handling
- **`DateNormalizer`** — ISO 8601 output, timezone conversion, relative date parsing
- **`NumberNormalizer`** — currency, unit conversion, scientific notation, percentages
- **`DataCleaner`** — duplicate detection, schema validation, missing value handling
- **`LanguageDetector`** — 50+ language detection with confidence scoring
- **`EncodingHandler`** — UTF-8 conversion, BOM removal, encoding detection
<CardGroup cols={2}>
<Card title="TextNormalizer" icon="text-size">
Unicode forms, whitespace collapse, HTML stripping, smart-quote and dash replacement.
</Card>
<Card title="EntityNormalizer" icon="building">
Corporate suffix normalization, honorific removal, alias resolution, and disambiguation.
</Card>
<Card title="DateNormalizer" icon="calendar">
Any date format → ISO 8601; relative dates, timezones, and date ranges.
</Card>
<Card title="NumberNormalizer" icon="hashtag">
Currency, scientific notation, unit abbreviations, and percentages → float.
</Card>
<Card title="LanguageDetector" icon="globe">
50+ languages with confidence scoring and batch detection.
</Card>
<Card title="EncodingHandler" icon="code">
Encoding detection, UTF-8 conversion, BOM removal, and cp1252 repair.
</Card>
</CardGroup>
<Note>
**v0.5.0 fix:** Encoding repair now handles cp1252/latin-1 characters that previously caused crashes on Windows when processing documents with non-ASCII content.
**v0.5.0 fix:** Encoding repair now handles cp1252 and latin-1 characters that previously caused crashes on Windows when processing documents with non-ASCII content.
</Note>
## Recommended Processing Order
<Steps>
<Step title="EncodingHandler — fix encoding first">
Broken bytes corrupt everything downstream. Always run this before anything else.
```python
from semantica.normalize import EncodingHandler
handler = EncodingHandler()
utf8_text = handler.to_utf8(raw_bytes)
```
</Step>
<Step title="TextNormalizer — unicode, whitespace, HTML">
```python
from semantica.normalize import TextNormalizer
normalizer = TextNormalizer(strip_html=True, normalize_unicode=True)
clean_text = normalizer.normalize_text(utf8_text)
```
</Step>
<Step title="EntityNormalizer — canonicalize entity names">
```python
from semantica.normalize import EntityNormalizer
normalizer = EntityNormalizer()
canonical = normalizer.normalize_entity("Apple Computer Inc.", entity_type="Organization")
# → "Apple Inc."
```
</Step>
<Step title="DateNormalizer and NumberNormalizer — parse structured values">
```python
from semantica.normalize import DateNormalizer, NumberNormalizer
date_norm = DateNormalizer(target_timezone="UTC")
num_norm = NumberNormalizer()
date = date_norm.normalize_date("Jan 1st, 2020") # → "2020-01-01"
num = num_norm.normalize_number("$1.2B") # → 1200000000.0
```
</Step>
<Step title="LanguageDetector — detect language on clean text">
```python
from semantica.normalize import LanguageDetector
detector = LanguageDetector()
lang = detector.detect("Bonjour le monde")
# → {"language": "fr", "confidence": 0.98}
```
</Step>
</Steps>
## Convenience Functions
The fastest path — dispatch via function with a `method` parameter:
The fastest path — one import, one call:
```python
from semantica.normalize import (
normalize_text, normalize_entity, normalize_date,
normalize_number, clean_data, detect_language, handle_encoding
normalize_number, clean_data, detect_language, handle_encoding,
)
clean = normalize_text(" Hello, World!! \n\n")
# → "Hello, World!!"
entity = normalize_entity("Apple Computer Inc.", entity_type="Organization")
# → "Apple Inc."
date = normalize_date("Jan 1st, 2020")
# → "2020-01-01"
num = normalize_number("$1,234.56")
# → 1234.56
lang = detect_language("Bonjour le monde")
# → {"language": "fr", "confidence": 0.98}
clean = normalize_text(" Hello, World!! \n\n") # → "Hello, World!!"
entity = normalize_entity("Apple Computer Inc.", entity_type="Organization") # → "Apple Inc."
date = normalize_date("Jan 1st, 2020") # → "2020-01-01"
num = normalize_number("$1.2B") # → 1200000000.0
lang = detect_language("Bonjour le monde") # → {"language": "fr", "confidence": 0.98}
```
## TextNormalizer
## Normalizers
```python
from semantica.normalize import TextNormalizer
<Tabs>
<Tab title="TextNormalizer">
Cleans raw text at the character and token level:
normalizer = TextNormalizer()
```python
from semantica.normalize import TextNormalizer
normalized = normalizer.normalize_text(
raw_text,
lowercase=False,
remove_punctuation=False,
remove_extra_whitespace=True,
strip_html=True, # remove HTML tags
normalize_unicode=True, # NFC normalization
)
```
normalizer = TextNormalizer(
lowercase=False,
remove_punctuation=False,
remove_extra_whitespace=True,
strip_html=True,
normalize_unicode=True,
fix_encoding=True,
form="NFC", # "NFC" | "NFD" | "NFKC" | "NFKD"
)
Sub-normalizers for fine-grained control:
normalized = normalizer.normalize_text(raw_text)
```
```python
from semantica.normalize import UnicodeNormalizer, WhitespaceNormalizer, SpecialCharacterProcessor
| Parameter | Type | Default | Description |
| --------- | ---- | ------- | ----------- |
| `lowercase` | `bool` | `False` | Convert to lowercase — use for bag-of-words matching, not NER |
| `remove_punctuation` | `bool` | `False` | Strip all punctuation — use for keyword extraction only |
| `remove_extra_whitespace` | `bool` | `True` | Collapse tabs, newlines, non-breaking spaces into single spaces |
| `strip_html` | `bool` | `False` | Remove HTML tags and decode `&amp;`, `&lt;`, etc. |
| `normalize_unicode` | `bool` | `True` | Apply Unicode normal form |
| `fix_encoding` | `bool` | `True` | Repair common encoding mojibake (cp1252 / latin-1 → UTF-8) |
| `form` | `str` | `"NFC"` | Unicode normalization form: `"NFC"` / `"NFD"` / `"NFKC"` / `"NFKD"` |
# Unicode normalization forms: NFC | NFD | NFKC | NFKD
unicode_norm = UnicodeNormalizer(form="NFC")
text = unicode_norm.normalize("café")
**Unicode form guide:**
# Collapse tabs, line breaks, and extra spaces
ws_norm = WhitespaceNormalizer()
text = ws_norm.normalize("Hello World\t\n") # → "Hello World"
| Form | Use When |
| ---- | -------- |
| `NFC` | Default — best for storage and display |
| `NFKC` | Search indexing — normalises ligatures, fullwidth chars, and fractions |
| `NFD` | Stripping diacritics — split é → e + combining accent, then strip accents |
| `NFKD` | Same as NFD but also decomposes compatibility characters |
# Replace smart quotes, em-dashes, and ellipsis characters
processor = SpecialCharacterProcessor()
text = processor.process("‘Hello’") # '' → ''
```
**Sub-normalizers for fine-grained control:**
## EntityNormalizer
```python
from semantica.normalize import UnicodeNormalizer, WhitespaceNormalizer, SpecialCharacterProcessor
Canonicalize entity names — handles corporate suffixes, punctuation, case, and honorifics:
unicode_norm = UnicodeNormalizer(form="NFC")
text = unicode_norm.normalize("café")
```python
from semantica.normalize import EntityNormalizer
ws_norm = WhitespaceNormalizer()
text = ws_norm.normalize("Hello\t\t World\n\n") # → "Hello World"
normalizer = EntityNormalizer()
processor = SpecialCharacterProcessor()
text = processor.process("'Hello'") # '' → '', -- → -
```
</Tab>
<Tab title="EntityNormalizer">
Canonicalises entity name variants — corporate suffixes, honorifics, case, and punctuation:
# Company name normalization
companies = ["Apple Computer, Inc.", "Apple Inc", "APPLE INC."]
normalized = [normalizer.normalize_entity(c) for c in companies]
# All → "Apple Inc."
```python
from semantica.normalize import EntityNormalizer
# Person name normalization
name = normalizer.normalize_entity("JOBS, STEVE", entity_type="Person")
# → "Steve Jobs"
```
normalizer = EntityNormalizer()
Sub-normalizers for entity-specific use cases:
# Corporate name normalization
normalizer.normalize_entity("Apple Computer, Inc.", entity_type="Organization") # → "Apple Inc."
normalizer.normalize_entity("APPLE INC.", entity_type="Organization") # → "Apple Inc."
```python
from semantica.normalize import AliasResolver, EntityDisambiguator, NameVariantHandler
# Person name normalization
normalizer.normalize_entity("JOBS, STEVE", entity_type="Person") # → "Steve Jobs"
```
# Dictionary-based alias expansion
resolver = AliasResolver(aliases={
"ML": "Machine Learning",
"AI": "Artificial Intelligence",
"DL": "Deep Learning",
})
resolved = resolver.resolve("ML and DL are subsets of AI")
# → "Machine Learning and Deep Learning are subsets of Artificial Intelligence"
**Key behaviours:**
- Corporate suffix normalization handles: `Inc`, `Inc.`, `Incorporated`, `Ltd`, `Limited`, `Corp`, `Corporation`, `LLC`, `GmbH`, `PLC`, and 30+ more
- `entity_type="Person"` activates last-name-first reversal, honorific removal, and suffix stripping
# Context-aware disambiguation
disambiguator = EntityDisambiguator()
result = disambiguator.disambiguate(
"Apple", context="Steve Jobs founded Apple in Cupertino"
)
# → {"entity": "Apple Inc.", "type": "Organization", "confidence": 0.96}
**Sub-normalizers:**
# Honorifics, titles, and cultural name variants
handler = NameVariantHandler()
canonical = handler.normalize("Dr. JOHN P. SMITH Jr.")
# → "John P. Smith"
```
```python
from semantica.normalize import AliasResolver, EntityDisambiguator, NameVariantHandler
## DateNormalizer
resolver = AliasResolver(aliases={
"ML": "Machine Learning",
"NLP": "Natural Language Processing",
})
resolved = resolver.resolve("ML and NLP are subfields of AI")
Parse and normalize dates from any format to ISO 8601:
disambiguator = EntityDisambiguator()
result = disambiguator.disambiguate(
"Apple",
context="Steve Jobs founded Apple in Cupertino in 1976",
)
# → {"entity": "Apple Inc.", "type": "Organization", "confidence": 0.96}
```python
from semantica.normalize import DateNormalizer
handler = NameVariantHandler()
canonical = handler.normalize("Dr. JOHN P. SMITH Jr.") # → "John P. Smith"
```
</Tab>
<Tab title="DateNormalizer">
Parses any date format and outputs ISO 8601 strings. Handles relative expressions, timezones, and date ranges:
normalizer = DateNormalizer()
```python
from semantica.normalize import DateNormalizer
dates = [
"January 1st, 2020",
"01/01/2020",
"2020-01-01T00:00:00Z",
"yesterday",
"3 weeks ago",
]
normalized = [normalizer.normalize_date(d) for d in dates]
# All → ISO 8601 strings
normalizer = DateNormalizer(
target_timezone="UTC",
output_format="%Y-%m-%d",
)
# With automatic UTC conversion
normalizer_utc = DateNormalizer(target_timezone="UTC")
utc_date = normalizer_utc.normalize_date("2024-01-01 09:00 EST")
```
dates = [
"January 1st, 2020",
"01/01/2020",
"2020-01-01T00:00:00Z",
"yesterday",
"3 weeks ago",
"Q1 2024",
]
normalized = [normalizer.normalize_date(d) for d in dates]
```
Sub-normalizers for advanced date handling:
**Sub-normalizers:**
```python
from semantica.normalize import (
TimeZoneNormalizer, RelativeDateProcessor, TemporalExpressionParser
)
```python
from semantica.normalize import TimeZoneNormalizer, RelativeDateProcessor, TemporalExpressionParser
from datetime import datetime
# Timezone conversion
tz_norm = TimeZoneNormalizer(target_tz="UTC")
utc_dt = tz_norm.normalize("2024-01-01 09:00", source_tz="America/New_York")
tz_norm = TimeZoneNormalizer(target_tz="UTC")
utc_dt = tz_norm.normalize("2024-01-01 09:00", source_tz="America/New_York")
# → datetime(2024, 1, 1, 14, 0, tzinfo=UTC)
# Relative date resolution
from datetime import datetime
processor = RelativeDateProcessor(reference_date=datetime(2025, 1, 15))
result = processor.process("3 days ago")
# → datetime(2025, 1, 12)
processor = RelativeDateProcessor(reference_date=datetime(2025, 1, 15))
result = processor.process("3 days ago") # → datetime(2025, 1, 12)
result = processor.process("next quarter") # → {"start": "2025-04-01", "end": "2025-06-30"}
# Date range parsing
parser = TemporalExpressionParser()
result = parser.parse("from January 2020 to March 2021")
# → {"start": "2020-01-01", "end": "2021-03-31", "type": "range"}
```
parser = TemporalExpressionParser()
result = parser.parse("from January 2020 to March 2021")
# → {"start": "2020-01-01", "end": "2021-03-31", "type": "range"}
## NumberNormalizer
result = parser.parse("Q2 2023")
# → {"start": "2023-04-01", "end": "2023-06-30", "type": "quarter"}
```
</Tab>
<Tab title="NumberNormalizer">
Converts number strings with units, currencies, and abbreviations to `float`:
```python
from semantica.normalize import NumberNormalizer
```python
from semantica.normalize import NumberNormalizer
normalizer = NumberNormalizer()
normalizer = NumberNormalizer()
normalizer.normalize_number("$1,234.56") # → 1234.56
normalizer.normalize_number("€42K") # → 42000.0
normalizer.normalize_number("$1.2B") # → 1200000000.0
normalizer.normalize_number("3.14e-2") # → 0.0314
normalizer.normalize_number("42%") # → 0.42
```
normalizer.normalize_number("$1,234.56") # → 1234.56
normalizer.normalize_number("€42K") # → 42000.0
normalizer.normalize_number("$1.2B") # → 1200000000.0
normalizer.normalize_number("3.14e-2") # → 0.0314
normalizer.normalize_number("42%") # → 0.42
normalizer.normalize_number("−7") # → -7.0 (minus sign, not hyphen)
```
Unit and currency conversion:
**Unit and currency conversion:**
```python
from semantica.normalize import UnitConverter, CurrencyNormalizer
```python
from semantica.normalize import UnitConverter, CurrencyNormalizer
converter = UnitConverter()
result = converter.convert(100, from_unit="km/h", to_unit="m/s")
# → 27.78
converter = UnitConverter()
result = converter.convert(100, from_unit="km/h", to_unit="m/s")
# → 27.78
# Supported categories: length, weight, volume, temperature, speed, area
categories = converter.list_categories()
categories = converter.list_categories()
# → ["length", "weight", "volume", "temperature", "speed", "area", "pressure", "energy"]
currency_norm = CurrencyNormalizer()
result = currency_norm.normalize("$42.50")
# → {"amount": 42.50, "currency": "USD", "raw": "$42.50"}
```
currency_norm = CurrencyNormalizer()
result = currency_norm.normalize("$42.50")
# → {"amount": 42.50, "currency": "USD", "raw": "$42.50"}
```
</Tab>
<Tab title="Language & Encoding">
### LanguageDetector
Identify the language of a text string. Used internally by the sentence splitter and chunker:
```python
from semantica.normalize import LanguageDetector
detector = LanguageDetector()
lang = detector.detect("Bonjour le monde")
# → {"language": "fr", "confidence": 0.98}
langs = detector.detect_top_n("This might be mixed", n=3)
# → [{"language": "en", "probability": 0.85}, ...]
results = detector.detect_batch(["Hello", "Hola", "Bonjour", "Ciao"])
supported = detector.list_supported_languages()
```
Supports 50+ languages: `en`, `de`, `fr`, `es`, `it`, `pt`, `nl`, `ru`, `zh`, `ja`, `ko`, `ar`, `hi`, `tr`, `pl`, `sv`, `da`, `no`, `fi`, and more.
### EncodingHandler
Detect and repair character encoding issues:
```python
from semantica.normalize import EncodingHandler
handler = EncodingHandler()
encoding = handler.detect_encoding(raw_bytes)
# → {"encoding": "windows-1252", "confidence": 0.73}
utf8_text = handler.to_utf8(raw_bytes)
clean = handler.remove_bom(text_with_bom)
repaired = handler.repair_encoding(garbled_text, source_encoding="cp1252")
```
**Key behaviours:**
- Encoding detection uses `chardet` internally — accuracy improves with longer input
- `to_utf8()` attempts cp1252 repair automatically when it detects mojibake patterns
- Always run `EncodingHandler` first — broken bytes cause cascading failures in every downstream normalizer
</Tab>
</Tabs>
## DataCleaner
Cleans structured record sets — useful before loading into a vector store or graph:
```python
from semantica.normalize import DataCleaner, DataValidator
cleaner = DataCleaner()
# Remove near-duplicate records
deduped = cleaner.remove_duplicates(records, similarity_threshold=0.9)
# Fill missing values
filled = cleaner.fill_missing(records, strategy="mean")
# strategy options: "mean" | "median" | "mode" | "remove"
filled = cleaner.fill_missing(
records,
strategy="mean", # "mean" | "median" | "mode" | "remove" | "constant"
constant_value=None,
)
# Validate schema
validator = DataValidator()
result = validator.validate(records, schema={"name": str, "age": int})
print(result.valid_count, result.errors)
```
## LanguageDetector
```python
from semantica.normalize import LanguageDetector
detector = LanguageDetector()
lang = detector.detect("Bonjour le monde")
# → {"language": "fr", "confidence": 0.98}
# Top N languages for mixed-language text
langs = detector.detect_top_n("This might be mixed", n=3)
# → [{"language": "en", "probability": 0.85}, ...]
# Batch detection
results = detector.detect_batch(["Hello", "Hola", "Bonjour"])
```
## EncodingHandler
```python
from semantica.normalize import EncodingHandler
handler = EncodingHandler()
encoding = handler.detect_encoding(raw_bytes)
# → {"encoding": "windows-1252", "confidence": 0.73}
utf8_text = handler.to_utf8(raw_bytes)
clean = handler.remove_bom(text_with_bom)
result = validator.validate(records, schema={"name": str, "age": int, "active": bool})
print(f"Valid: {result.valid_count}")
print(f"Invalid: {result.error_count}")
```
## Pipeline Integration
For large datasets, use the pipeline instead of per-item normalization:
```python
from semantica.pipeline import Pipeline
from semantica.pipeline import PipelineBuilder, ExecutionEngine
from semantica.ingest import FileIngestor
from semantica.normalize import TextNormalizer
from semantica.semantic_extract import NERExtractor
from semantica.llms import Groq
import os
pipeline = Pipeline()
pipeline.add_step("normalize", TextNormalizer())
result = pipeline.run(documents)
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
ingestor = FileIngestor()
normalizer = TextNormalizer(strip_html=True, normalize_unicode=True)
extractor = NERExtractor(method="llm", llm_provider=llm)
builder = PipelineBuilder()
builder.add_step("ingest", "file_ingest", handler=ingestor.ingest)
builder.add_step("normalize", "text_normalize", handler=normalizer.normalize)
builder.add_step("extract", "ner_extract", handler=extractor.extract)
builder.connect_steps("ingest", "normalize")
builder.connect_steps("normalize", "extract")
pipeline = builder.build("normalize_pipeline")
result = ExecutionEngine().execute_pipeline(pipeline, data="data/documents/")
```
## Tips and Common Pitfalls
<Warning>
**Run encoding repair before anything else.** A single cp1252 character in a UTF-8 stream silently corrupts the surrounding text. Run `EncodingHandler` or set `fix_encoding=True` on `TextNormalizer` first.
</Warning>
<Warning>
**Don't lowercase before NER.** `normalize_text(lowercase=True)` before entity extraction destroys capitalization signals that NER relies on. Apply case normalization only after extraction if needed.
</Warning>
<Tip>
**AliasResolver is order-sensitive.** If you register overlapping aliases (`"ML"` and `"ML model"`), the longer match wins. Sort aliases by length descending for predictable behaviour.
</Tip>
<Warning>
**DateNormalizer and timezone.** Without `target_timezone`, dates without timezone information are returned as naive datetime strings. In regulated pipelines (HIPAA, SOX), always set `target_timezone="UTC"` for unambiguous timestamps.
</Warning>
<Tip>
**`DataCleaner.remove_duplicates()` is not the same as `DuplicateDetector`.** `DataCleaner` operates on flat records (dicts/rows) by field-level Jaccard similarity. `DuplicateDetector` in the Deduplication module operates on graph entities with embedding-based matching. Use the latter for entity resolution.
</Tip>
<CardGroup cols={2}>
<Card title="Parse" icon="file-lines" href="parse">
Parse documents before normalization.
@@ -278,9 +457,9 @@ result = pipeline.run(documents)
Chunk normalized text for embedding.
</Card>
<Card title="Deduplication" icon="copy" href="deduplication">
Resolve duplicate entities post-normalization.
Resolve duplicate entities after normalization.
</Card>
<Card title="Pipeline" icon="gear" href="pipeline">
Include normalization as a pipeline step.
Include normalization as a named pipeline step.
</Card>
</CardGroup>
docs/reference/ontology.md
+133 -93
View File
@@ -1,51 +1,80 @@
---
title: "Ontology Module"
description: "Automated ontology generation, SHACL validation, SKOS vocabularies, alignment, diff/migration, and the visual Ontology Hub."
description: "Automated ontology generation, SHACL validation, OWL/RDF export, namespace management, and LLM-powered ontology generation."
icon: "sitemap"
---
`semantica.ontology` provides the full lifecycle for knowledge graph schemas — from auto-generation and SHACL validation to visual editing in the Ontology Hub (v0.5.0). Use it for schema design, data modeling, semantic web interoperability, and SHACL-based data quality validation.
`semantica.ontology` provides the full lifecycle for knowledge graph schemas — from auto-generation and SHACL validation to OWL/RDF export. Use it for schema design, data modeling, semantic web interoperability, and SHACL-based data quality validation.
## Exported Classes
```python
from semantica.ontology import (
OntologyGenerator, # auto-generate from KG data (6-stage pipeline)
LLMOntologyGenerator, # LLM-powered ontology generation
OntologyEngine, # unified orchestration facade
ClassInferrer, # class discovery and hierarchy building
PropertyGenerator, # property inference and XSD type mapping
SHACLGenerator, # generate SHACL shapes from ontology
OntologyValidator, # validate graphs against SHACL shapes
SHACLValidationReport, # validation report with violations list
SHACLViolation, # individual constraint violation
OWLGenerator, # OWL/RDF serialization (Turtle, XML, JSON-LD)
OntologyEvaluator, # quality evaluation: coverage, completeness
NamespaceManager, # IRI generation and namespace prefix management
OntologyAligner, # align and merge ontologies across schemas (use OntologyEngine)
AssociativeClassBuilder, # N-ary relationship intermediate class creation
NamingConventions, # PascalCase/camelCase enforcement
DomainOntologies, # pre-built domain ontologies
ingest_ontology, # load ontology from file
)
```
## What You Get
- **`OntologyManager`** — define classes, properties, relationships, and constraints
- **`OntologyGenerator`** — auto-generate ontologies from existing knowledge graph data (6-stage pipeline)
- **`SHACLGenerator`** / **`SHACLValidator`** — generate and validate SHACL shapes
- **`SKOSVocabulary`** — controlled vocabulary and taxonomy management
- **`OntologyAligner`** — align and merge ontologies across schemas
- **`OntologyDiff`** / **`OntologyMigrator`** — diff and migrate ontology versions
- **`OWLExporter`** — export to Turtle, RDF/XML, JSON-LD
- **Ontology Hub** — visual browser UI for the full ontology lifecycle (v0.5.0)
- **`LLMOntologyGenerator`** — LLM-powered ontology generation for complex domains
- **`OntologyEngine`** — unified facade that orchestrates the full ontology lifecycle
- **`SHACLGenerator`** / **`OntologyValidator`** — generate SHACL shapes and validate any graph
- **`OWLGenerator`** — serialize ontologies to Turtle, RDF/XML, JSON-LD
- **`NamespaceManager`** — IRI generation, prefix management, namespace binding
- **`OntologyEvaluator`** — coverage, completeness, and granularity quality metrics
- **`AssociativeClassBuilder`** — model N-ary relationships as intermediate OWL classes
## OntologyManager
## OntologyEngine (Unified Facade)
Define and validate a schema for your knowledge graph:
The `OntologyEngine` orchestrates the full ontology lifecycle — generation, validation, export, and versioning:
```python
from semantica.ontology import OntologyManager
from semantica.ontology import OntologyEngine
ontology = OntologyManager()
ontology.add_class("Person", properties=["name", "birth_date"])
ontology.add_class("Organization", properties=["name", "founded_date"])
ontology.add_relationship("works_for", domain="Person", range="Organization")
ontology.add_constraint("Person", "must_have_name")
engine = OntologyEngine(base_uri="https://example.org/ontology/")
# Validate a graph against the ontology
is_valid = ontology.validate_graph(kg)
# Generate ontology from KG data
ontology = engine.generate_ontology({"entities": entities, "relationships": relationships})
# Export as OWL Turtle
owl_ttl = ontology.export_owl(format="turtle")
# Validate a graph against the generated SHACL shapes
report = engine.validate(kg)
if not report.conforms:
for v in report.violations:
print(f"{v.severity}: {v.message} on {v.node}")
# Export to OWL Turtle
engine.export(ontology, "ontology.ttl", format="turtle")
```
## Auto-Generation (6-Stage Pipeline)
## OntologyGenerator (6-Stage Pipeline)
Generate an ontology automatically from your knowledge graph data:
Generate a formal ontology automatically from your knowledge graph entities and relationships:
```python
from semantica.ontology import OntologyGenerator
generator = OntologyGenerator()
ontology = generator.generate_from_graph(kg)
generator = OntologyGenerator(base_uri="https://example.org/ontology/")
ontology = generator.generate_ontology({
"entities": entities,
"relationships": relationships,
})
```
The pipeline runs through these stages in order:
@@ -62,99 +91,107 @@ The pipeline runs through these stages in order:
Generate SHACL shapes from an ontology and validate any graph against them:
```python
from semantica.ontology import SHACLGenerator, SHACLValidator
from semantica.ontology import SHACLGenerator, OntologyValidator, SHACLValidationReport, SHACLViolation
# Generate shapes
generator = SHACLGenerator()
shapes = generator.generate(ontology)
# Generate shapes from ontology
generator = SHACLGenerator()
shapes = generator.generate(ontology)
shapes_ttl = shapes.serialize(format="turtle")
# Validate a graph
validator = SHACLValidator()
report = validator.validate(kg, shapes=shapes)
# Validate a graph against the shapes
validator = OntologyValidator()
report: SHACLValidationReport = validator.validate(kg, shapes=shapes)
if not report.conforms:
violation: SHACLViolation
for violation in report.violations:
print(f"Violation: {violation.message} on {violation.node}")
print(f" Path: {violation.path}")
print(f" Severity: {violation.severity}")
print(f"{violation.severity}: {violation.message}")
print(f" Node: {violation.node}")
print(f" Path: {violation.path}")
```
## SKOS Vocabularies
## LLM-Powered Ontology Generation
Build controlled vocabularies and taxonomies using the W3C SKOS standard:
For complex or novel domains where schema patterns are hard to infer statistically:
```python
from semantica.ontology import SKOSVocabulary
from semantica.ontology import LLMOntologyGenerator
from semantica.llms import Groq
import os
vocab = SKOSVocabulary()
vocab.add_concept("Machine Learning", broader="Artificial Intelligence")
vocab.add_concept("Deep Learning", broader="Machine Learning")
vocab.add_concept("Computer Vision", broader="Deep Learning")
vocab.add_alt_label("ML", for_concept="Machine Learning")
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
skos_ttl = vocab.export(format="turtle")
```
## Ontology Alignment
Map concepts across two ontologies and merge them:
```python
from semantica.ontology import OntologyAligner
aligner = OntologyAligner()
alignment = aligner.align(source_ontology, target_ontology)
for mapping in alignment.mappings:
print(f"{mapping.source} → {mapping.target} (confidence: {mapping.confidence:.2f})")
# Merge into a unified ontology
merged = aligner.merge(source_ontology, target_ontology, alignment)
```
## Diff and Migration
Compare ontology versions and generate migration scripts for graph data:
```python
from semantica.ontology import OntologyDiff, OntologyMigrator
diff = OntologyDiff()
changes = diff.compare(ontology_v1, ontology_v2)
for change in changes:
print(f"{change.type}: {change.element} — {change.description}")
# Generate and apply a migration script
migrator = OntologyMigrator()
migration_script = migrator.generate_migration(changes)
migrator.apply(kg, migration_script)
generator = LLMOntologyGenerator(llm_provider=llm)
ontology = generator.generate(
domain_description="A biomedical ontology for clinical trial protocols",
examples=["Patient", "Trial", "Intervention", "Outcome"],
)
```
## OWL / RDF Export
```python
from semantica.ontology import OWLExporter
from semantica.ontology import OWLGenerator
exporter = OWLExporter()
exporter.export(ontology, path="ontology.ttl", format="turtle")
exporter.export(ontology, path="ontology.owl", format="xml")
exporter.export(ontology, path="ontology.json", format="json-ld")
generator = OWLGenerator()
generator.generate(ontology, path="ontology.ttl", format="turtle")
generator.generate(ontology, path="ontology.owl", format="xml")
generator.generate(ontology, path="ontology.json", format="json-ld")
```
## Namespace Management
```python
from semantica.ontology import NamespaceManager
ns = NamespaceManager(base_uri="https://example.org/")
ns.register("ex", "https://example.org/")
ns.register("schema", "https://schema.org/")
ns.register("owl", "http://www.w3.org/2002/07/owl#")
# Generate IRIs for classes and properties
class_iri = ns.generate_class_iri("Person")
property_iri = ns.generate_property_iri("worksFor")
```
## Ontology Evaluation
Measure coverage, completeness, and granularity of a generated ontology:
```python
from semantica.ontology import OntologyEvaluator
evaluator = OntologyEvaluator()
result = evaluator.evaluate(ontology, kg)
print(f"Class coverage: {result.class_coverage:.2f}")
print(f"Property coverage: {result.property_coverage:.2f}")
print(f"Completeness: {result.completeness:.2f}")
print(f"Granularity: {result.granularity:.2f}")
for gap in result.gaps:
print(f"Gap: {gap.description}")
```
## Ingest an Existing Ontology
Load and parse an ontology file for downstream use:
```python
from semantica.ontology import ingest_ontology
ontology_data = ingest_ontology("schema.ttl") # Turtle
ontology_data = ingest_ontology("schema.owl") # OWL/XML
ontology_data = ingest_ontology("schema.jsonld") # JSON-LD
```
## Ontology Hub (v0.5.0)
A visual browser UI for the full ontology lifecycle, served by `semantica.explorer`:
A visual browser UI for the full ontology lifecycle. Launch via CLI:
```bash
pip install "semantica[explorer]"
```
```python
from semantica.explorer import start_explorer
start_explorer(graph=kg, port=8080)
semantica-explorer --port 8080
# Navigate to http://localhost:8080 → Ontology Hub tab
```
@@ -162,10 +199,13 @@ Features:
- **Visual editor** — create and edit classes, properties, and relationships in the browser
- **SHACL Studio** — author and validate SHACL shapes with live feedback
- **Alignment authoring** — map concepts across ontologies with drag-and-drop
- **Health dashboard** — coverage, completeness, and constraint violation metrics
- **Version control** — snapshot, diff, and restore ontology versions
<Note>
Ontology versioning (`VersionManager`, `OntologyVersion`) has moved to `semantica.change_management`. Import from there: `from semantica.change_management import VersionManager`.
</Note>
<CardGroup cols={2}>
<Card title="Reasoning" icon="microchip" href="reasoning">
Apply inference rules over ontology axioms.
docs/reference/parse.md
+28 -2
View File
@@ -6,11 +6,37 @@ icon: "file-lines"
`semantica.parse` extracts structured text, layout, tables, and metadata from unstructured documents. `DocumentParser` handles clean machine-readable files; `DoclingParser` handles complex layouts, scanned PDFs, and multi-column documents.
## Exported Classes
```python
from semantica.parse import (
DocumentParser, # auto-detect format — delegates to format-specific parser
PDFParser, # PDF text extraction
DOCXParser, # Word .docx documents
HTMLParser, # HTML / web pages
MarkdownParser, # Markdown files
TXTParser, # plain text
JSONParser, # JSON documents
XMLParser, # XML documents
CSVParser, # CSV / TSV files
WebParser, # URL fetch + HTML parsing
EmailParser, # .eml / .msg email files
CodeParser, # source code files
# Data types
ParsedDocument, # {text, sections, tables, metadata, source_id}
DocumentMetadata, # {title, author, created_date, page_count, language, ...}
)
# Optional — requires: pip install "semantica[docling]"
from semantica.parse import DoclingParser # advanced OCR + layout analysis
```
## What You Get
- **`DocumentParser`** — standard parser for PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX
- **`DoclingParser`** — advanced parser for complex layouts, merged-cell tables, multi-column PDFs, and OCR
- **`DocumentParser`** — standard parser for PDF, DOCX, HTML, TXT, JSON, CSV, PPTX, XLSX — auto-detects format
- **`DoclingParser`** — advanced parser for complex layouts, merged-cell tables, multi-column PDFs, and OCR (optional dep)
- **`ParsedDocument`** — structured output with `text`, `sections`, `tables`, and `metadata`
- **Format-specific parsers** — `PDFParser`, `DOCXParser`, `HTMLParser`, `WebParser`, `EmailParser`, `CodeParser`, etc.
## DocumentParser
docs/reference/pipeline.md
+559 -85
View File
@@ -6,135 +6,609 @@ icon: "gear"
`semantica.pipeline` lets you chain Semantica components into reproducible, fault-tolerant workflows with parallel execution and configurable error handling. Pipelines are serializable — save them to YAML and reload in any environment.
## What You Get
## Exported Classes
- **`Pipeline`** — chain steps with parallel workers, retry policies, and failure handlers
- **`PipelineBuilder`** — fluent DSL for building pipelines with a readable chain syntax
- **`RetryPolicy`** — fixed, linear, and exponential backoff with configurable max retries
- **`FailureHandler`** — skip, stop, or retry failed documents without halting the pipeline
- **Progress tracking** — console (tqdm), WebSocket streaming, or file logging
```python
from semantica.pipeline import (
# Pipeline construction
PipelineBuilder, # DSL: add_step, connect_steps, build
Pipeline, # pipeline definition dataclass
PipelineStep, # step definition: name, step_type, handler, dependencies
StepStatus, # enum: PENDING, RUNNING, COMPLETED, FAILED, SKIPPED
PipelineSerializer, # serialize/deserialize pipeline to JSON/YAML
# Execution
ExecutionEngine, # execute_pipeline(pipeline, data) -> ExecutionResult
ExecutionResult, # {success, output, metadata, metrics, errors}
PipelineStatus, # enum: RUNNING, PAUSED, STOPPED
ProgressTracker, # get_progress(pipeline_id) -> {completed, total, pct}
# Failure handling
FailureHandler, # configure strategy: skip/retry/abort
RetryHandler, # retry with exponential backoff
FallbackHandler, # fall back to alternative step on failure
RetryPolicy, # {max_retries, backoff, jitter}
RetryStrategy, # enum: FIXED, EXPONENTIAL, LINEAR
ErrorSeverity, # enum: LOW, MEDIUM, HIGH, CRITICAL
# Parallelism
ParallelismManager, # execute_parallel(tasks, timeout) — thread or process pool
ParallelExecutionResult, # {success, result, error, task_id}
# Resource management
ResourceScheduler, # allocate_resources / release_resources
ResourceType, # enum: CPU, MEMORY, GPU, NETWORK, DISK
# Validation
PipelineValidator, # validate_pipeline(pipeline) -> ValidationResult
# Templates
PipelineTemplateManager, # get_template("full-qa") -> pre-wired Pipeline
PipelineTemplate, # template metadata dataclass
)
```
## Why Use a Pipeline?
You could wire Semantica modules together with plain Python code. Pipelines add:
<CardGroup cols={2}>
<Card title="Retry and failure handling" icon="arrow-rotate-right">
A single bad document doesn't crash a 10,000-document run.
</Card>
<Card title="Parallelism" icon="bolt">
Run extraction across multiple workers with one parameter.
</Card>
<Card title="Progress tracking" icon="chart-line">
tqdm console bar or WebSocket streaming to Explorer.
</Card>
<Card title="Reproducibility" icon="floppy-disk">
Save the exact pipeline configuration to YAML and replay on any machine.
</Card>
<Card title="Delta mode" icon="code-compare">
On re-runs, only process documents that changed since the last run.
</Card>
<Card title="Validation" icon="shield-check">
Catch misconfigured steps and dependency cycles before they fail mid-run.
</Card>
</CardGroup>
<Note>
Use plain module calls for quick scripts and notebooks. Use pipelines for anything you run repeatedly, at scale, or in production.
</Note>
<img src="/assets/img/diagrams/pipeline-flow.svg" alt="Pipeline step sequence: Ingest → Parse → Normalize → Extract → Build KG → QA → Store → Deliver" style={{ width: '100%', borderRadius: '10px', margin: '0 0 24px' }} />
## Basic Pipeline
## Quick Start
```python
from semantica.pipeline import Pipeline
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor
from semantica.kg import GraphBuilder
<Steps>
<Step title="Build a pipeline">
```python
from semantica.pipeline import PipelineBuilder
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.semantic_extract import NERExtractor
from semantica.kg import GraphBuilder
from semantica.llms import Groq
import os
pipeline = Pipeline()
pipeline.add_step("ingest", FileIngestor())
pipeline.add_step("parse", DocumentParser())
pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm))
pipeline.add_step("build_kg", GraphBuilder(merge_entities=True))
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
result = pipeline.run("data/")
kg = result.output
```
ingestor = FileIngestor()
parser = DocumentParser()
extractor = NERExtractor(method="llm", llm_provider=llm)
kg_builder = GraphBuilder(merge_entities=True)
builder = PipelineBuilder()
builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file)
builder.add_step("parse", "document_parse", handler=parser.parse)
builder.add_step("extract", "ner_extract", handler=extractor.extract)
builder.add_step("build_kg", "graph_build", handler=kg_builder.build)
builder.connect_steps("ingest", "parse")
builder.connect_steps("parse", "extract")
builder.connect_steps("extract","build_kg")
pipeline = builder.build("my_pipeline")
```
</Step>
<Step title="Validate before running">
```python
from semantica.pipeline import PipelineValidator
validator = PipelineValidator()
result = validator.validate_pipeline(pipeline)
if not result.valid:
for error in result.errors: # errors is List[str]
print(f"Error: {error}")
for warning in result.warnings:
print(f"Warning: {warning}")
```
</Step>
<Step title="Execute and inspect results">
```python
from semantica.pipeline import ExecutionEngine
engine = ExecutionEngine()
result = engine.execute_pipeline(pipeline, data="data/")
kg = result.output
print(f"Success: {result.success}")
print(f"Steps executed: {result.metrics['steps_executed']}")
print(f"Steps failed: {result.metrics['steps_failed']}")
print(f"Duration: {result.metrics['execution_time']:.1f}s")
```
</Step>
</Steps>
## Parallel Processing
Process documents concurrently across multiple workers:
Set parallelism on the builder and pass `max_workers` to `ExecutionEngine`:
```python
pipeline = Pipeline(workers=4)
from semantica.pipeline import PipelineBuilder, ExecutionEngine
pipeline.add_step("ingest", FileIngestor())
pipeline.add_step("parse", DocumentParser())
pipeline.add_step("extract", NERExtractor(), parallel=True, batch_size=10)
pipeline.add_step("build", GraphBuilder())
builder = PipelineBuilder()
builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file)
builder.add_step("parse", "document_parse", handler=parser.parse)
builder.add_step("extract", "ner_extract", handler=extractor.extract)
builder.add_step("build", "graph_build", handler=kg_builder.build)
builder.set_parallelism(4)
result = pipeline.run("data/")
pipeline = builder.build("parallel_pipeline")
engine = ExecutionEngine(max_workers=4)
result = engine.execute_pipeline(pipeline, data="data/")
```
## Retry and Error Handling
Configure retry behavior and failure strategy independently:
<Tabs>
<Tab title="Exponential backoff (recommended)">
```python
from semantica.pipeline import RetryPolicy, RetryStrategy, FailureHandler, ExecutionEngine
```python
from semantica.pipeline import Pipeline, RetryPolicy, FailureHandler
policy = RetryPolicy(
max_retries=3,
strategy=RetryStrategy.EXPONENTIAL,
initial_delay=1.0, # 1s → 2s → 4s
backoff_factor=2.0
)
retry = RetryPolicy(
max_retries=3,
backoff="exponential", # "fixed" | "linear" | "exponential"
initial_delay=1.0 # seconds before first retry
)
handler = FailureHandler()
handler.retry_policies["ner_extract"] = policy # keyed by step_type
handler = FailureHandler(
strategy="skip", # "skip" | "stop" | "retry"
log_failures=True # write failed documents to error log
)
engine = ExecutionEngine(default_max_retries=3, default_backoff_factor=2.0)
result = engine.execute_pipeline(pipeline, data="data/")
```
pipeline = Pipeline(retry_policy=retry, failure_handler=handler)
```
Best for transient API errors and rate limits — waits longer with each retry, giving upstream services time to recover.
</Tab>
<Tab title="Linear backoff">
```python
from semantica.pipeline import RetryPolicy, RetryStrategy
policy = RetryPolicy(
max_retries=3,
strategy=RetryStrategy.LINEAR,
initial_delay=2.0 # 2s → 4s → 6s
)
```
Use when the delay between retries should grow predictably — e.g., waiting for a database lock to release.
</Tab>
<Tab title="Fixed backoff">
```python
from semantica.pipeline import RetryPolicy, RetryStrategy
policy = RetryPolicy(
max_retries=5,
strategy=RetryStrategy.FIXED,
initial_delay=1.0 # 1s every attempt
)
```
Use when retrying against a service with a fixed cooldown window.
</Tab>
</Tabs>
### Failure Strategies
| Strategy | Behaviour | When to Use |
| -------- | --------- | ----------- |
| `"skip"` | Log failure, continue to next document | Production — one bad doc shouldn't stop 10k |
| `"stop"` | Raise exception immediately | Development — surface errors fast |
| `"retry"` | Retry via `RetryPolicy`, then skip | When failures are likely transient |
<Warning>
Always use `strategy="skip"` in production. A single malformed document shouldn't stop a pipeline processing thousands of documents. Inspect `result.errors` after the run to find and reprocess failures.
</Warning>
## Progress Tracking
```python
# Console progress bar (tqdm)
result = pipeline.run("data/", show_progress=True)
<Tabs>
<Tab title="Console (tqdm)">
```python
from semantica.pipeline import ExecutionEngine
# WebSocket progress — stream to Knowledge Explorer
result = pipeline.run("data/", websocket_port=8080)
engine = ExecutionEngine()
result = engine.execute_pipeline(pipeline, data="data/")
# The progress tracker outputs tqdm bars to the console during execution
```
# Inspect results
print(f"Processed: {result.processed_count}")
print(f"Failed: {result.failed_count}")
print(f"Duration: {result.duration_seconds:.1f}s")
```
Displays a live progress bar in the terminal via Semantica's built-in progress tracker. Best for scripts and CLI tools.
</Tab>
<Tab title="Live status check">
```python
from semantica.pipeline import ExecutionEngine
import threading, time
engine = ExecutionEngine()
# Run in a background thread, poll progress from main thread
def run():
engine.execute_pipeline(pipeline, data="data/")
t = threading.Thread(target=run, daemon=True)
t.start()
while t.is_alive():
progress = engine.get_progress(pipeline.name)
if progress:
print(f" {progress['completed_steps']}/{progress['total_steps']} steps — {progress['status']}")
time.sleep(2)
```
Poll `get_progress()` for live status during execution.
</Tab>
</Tabs>
## Pipeline DSL
The `PipelineBuilder` provides a fluent chain syntax that reads as a data flow:
`PipelineBuilder` uses `add_step(name, type, **config)` and `connect_steps(from, to)` to define a DAG:
```python
from semantica.pipeline import PipelineBuilder
from semantica.pipeline import PipelineBuilder, ExecutionEngine
pipeline = (
PipelineBuilder()
.ingest(FileIngestor())
.parse(DocumentParser())
.normalize()
.extract(NERExtractor(method="llm", llm_provider=llm))
.extract_relations(RelationExtractor(method="llm", llm_provider=llm))
.build_kg(merge_entities=True)
.deduplicate(strategy="semantic_v2")
.export(format="turtle", path="output.ttl")
.build()
builder = PipelineBuilder()
# Add steps — step_type is a string label, handler is the callable invoked at runtime
builder.add_step("ingest", "file_ingest", handler=ingestor.ingest_file)
builder.add_step("parse", "document_parse", handler=parser.parse)
builder.add_step("normalize", "text_normalize", handler=normalizer.normalize)
builder.add_step("extract", "ner_extract", handler=extractor.extract)
builder.add_step("rel_extract", "rel_extract", handler=rel_extractor.extract)
builder.add_step("build_kg", "graph_build", handler=kg_builder.build)
builder.add_step("deduplicate", "dedup", handler=deduplicator.deduplicate)
builder.add_step("export", "rdf_export", handler=exporter.export, format="turtle", path="output.ttl")
# Wire the data flow
builder.connect_steps("ingest", "parse")
builder.connect_steps("parse", "normalize")
builder.connect_steps("normalize", "extract")
builder.connect_steps("extract", "rel_extract")
builder.connect_steps("rel_extract", "build_kg")
builder.connect_steps("build_kg", "deduplicate")
builder.connect_steps("deduplicate", "export")
pipeline = builder.build("full_pipeline")
result = ExecutionEngine().execute_pipeline(pipeline, data="data/")
```
## Serialize and Restore Pipelines
`PipelineSerializer` converts a pipeline to JSON or dict for storage and reloads it later:
```python
from semantica.pipeline import PipelineSerializer
serializer = PipelineSerializer()
# Serialize to JSON string
json_str = serializer.serialize_pipeline(pipeline, format="json")
# Save to file
with open("pipeline_config.json", "w") as f:
f.write(json_str)
# Restore on any machine and execute
with open("pipeline_config.json") as f:
restored = serializer.deserialize_pipeline(f.read())
result = ExecutionEngine().execute_pipeline(restored, data="data/")
```
<Tip>
Serialized pipelines capture step names, types, and config — but not handler functions (callables can't be serialized). Re-register handlers on the restored steps before executing.
</Tip>
## Pre-Built Templates
`PipelineTemplateManager` wires common workflows with the correct step order — no manual wiring required:
```python
from semantica.pipeline import PipelineTemplateManager
from semantica.llms import Groq
import os
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
manager = PipelineTemplateManager()
```
<CardGroup cols={2}>
<Card title="ingest-extract-build" icon="diagram-project">
**Ingest → Parse → Extract → Build KG**
Standard knowledge base construction from documents.
```python
pipeline = manager.get_template(
"ingest-extract-build", llm_provider=llm
)
```
</Card>
<Card title="graphrag" icon="magnifying-glass">
**Ingest → Parse → Embed → Index**
Retrieval-augmented generation — builds a vector-indexed knowledge graph.
```python
pipeline = manager.get_template(
"graphrag", llm_provider=llm, vector_backend="faiss"
)
```
</Card>
<Card title="analytics" icon="chart-bar">
**Build KG → Analytics → Export Report**
Graph analysis and reporting — centrality, community detection, HTML output.
```python
pipeline = manager.get_template(
"analytics", export_format="html"
)
```
</Card>
<Card title="full-qa" icon="shield-check">
**Ingest → Normalize → Extract → Dedup → Conflicts → Build**
Production-quality KG with full data quality pipeline.
```python
pipeline = manager.get_template(
"full-qa", llm_provider=llm
)
```
</Card>
</CardGroup>
## ExecutionEngine
Fine-grained control over pipeline execution — pause, resume, cancel, and inspect live progress:
```python
from semantica.pipeline import ExecutionEngine
engine = ExecutionEngine(max_workers=4)
# pipeline.name is the pipeline ID used for all control operations
result = engine.execute_pipeline(pipeline, data="data/")
pipeline_id = pipeline.name # e.g. "my_pipeline"
# Pause after the current step finishes
engine.pause_pipeline(pipeline_id)
progress = engine.get_progress(pipeline_id)
print(f"Completed: {progress['completed_steps']}/{progress['total_steps']}")
print(f"Status: {progress['status']}")
engine.resume_pipeline(pipeline_id)
engine.stop_pipeline(pipeline_id)
```
| Method | Returns | Description |
| ------ | ------- | ----------- |
| `execute_pipeline(pipeline, data)` | `ExecutionResult` | Execute pipeline from start to finish |
| `get_pipeline_status(pipeline_id)` | `PipelineStatus` | Current state (RUNNING, PAUSED, STOPPED) |
| `get_progress(pipeline_id)` | `Dict` | `completed_steps`, `total_steps`, `progress_percentage`, `status` |
| `pause_pipeline(pipeline_id)` | `None` | Suspend after current step completes |
| `resume_pipeline(pipeline_id)` | `None` | Resume from paused state |
| `stop_pipeline(pipeline_id)` | `None` | Cancel and clean up immediately |
## PipelineValidator
Catches problems before they surface as mid-run failures:
```python
from semantica.pipeline import PipelineValidator
validator = PipelineValidator()
result = validator.validate_pipeline(pipeline)
if result.valid:
print("Pipeline is valid — safe to run")
else:
for error in result.errors: # errors is List[str]
print(f"Error: {error}")
for warning in result.warnings: # warnings is List[str]
print(f"Warning: {warning}")
```
Checks performed:
- **Dependency cycle detection** — A depends on B, B depends on A
- **Step type validation** — each step type must be registered
- **Connection integrity** — referenced step names must exist
- **Configuration completeness** — required parameters must be present
## ParallelismManager
<Tabs>
<Tab title="Thread pool (I/O-bound)">
```python
from semantica.pipeline import ParallelismManager
# use_processes=False (default) → thread pool for I/O-bound tasks
manager = ParallelismManager(max_workers=8, use_processes=False)
tasks = [{"fn": ner.extract, "args": [text]} for text in texts]
results = manager.execute_parallel(tasks, timeout=60)
# returns List[ParallelExecutionResult]
successes = [r for r in results if r.success]
failures = [r for r in results if not r.success]
```
Use thread pools for **I/O-bound** steps: web fetching, database queries, API calls.
</Tab>
<Tab title="Process pool (CPU-bound)">
```python
# use_processes=True → process pool, bypasses Python GIL
manager = ParallelismManager(max_workers=4, use_processes=True)
tasks = [{"fn": embedder.generate_embeddings, "args": [chunk]} for chunk in chunks]
results = manager.execute_parallel(tasks, timeout=120)
```
Use process pools for **CPU-bound** steps: embedding, OCR, large NER batches.
</Tab>
</Tabs>
## ResourceScheduler
Prevents memory oversubscription on large runs:
```python
from semantica.pipeline import ResourceScheduler, ExecutionEngine
scheduler = ResourceScheduler()
engine = ExecutionEngine()
resources = scheduler.allocate_resources(pipeline)
try:
result = engine.execute_pipeline(pipeline, data="data/")
finally:
scheduler.release_resources(resources)
```
## Delta Mode
Re-process only data that has changed since the last run:
```python
from semantica.pipeline import PipelineBuilder, ExecutionEngine
builder = PipelineBuilder()
# delta_mode=True tells ExecutionEngine to compute the diff between two snapshots
# and pass only changed triples to this step's handler
builder.add_step(
"ingest", "file_ingest",
handler=ingestor.ingest_file,
delta_mode=True, base_version_id="v1", target_version_id="v2"
)
builder.add_step(
"extract", "ner_extract",
handler=extractor.extract,
delta_mode=True, base_version_id="v1", target_version_id="v2"
)
builder.add_step(
"build", "graph_build",
handler=kg_builder.build,
delta_mode=False # always rebuild the merged graph
)
builder.connect_steps("ingest", "extract")
builder.connect_steps("extract", "build")
result = pipeline.run("data/")
pipeline = builder.build("delta_pipeline")
engine = ExecutionEngine()
result = engine.execute_pipeline(
pipeline,
data="data/",
version_manager=version_manager, # required for delta mode
triplet_store=triplet_store # required for delta mode
)
```
## Save and Load Pipelines
<Note>
Delta detection uses SHA-256 checksums on source content. Only sources whose checksum differs from `base_version_id` are passed to downstream steps. For pipelines that run hourly or daily against a growing corpus, delta mode eliminates redundant re-embedding and re-extraction.
</Note>
Serialize a pipeline to YAML for reproducible runs across environments:
## Schemas
```python
# Save pipeline configuration
pipeline.save("pipeline_config.yaml")
# Load and run on any machine
pipeline = Pipeline.load("pipeline_config.yaml")
result = pipeline.run("data/")
```
## Pipeline Result
<AccordionGroup>
<Accordion title="ExecutionResult schema">
```python
@dataclass
class PipelineResult:
output: Any # final step output (e.g., a KnowledgeGraph)
processed_count: int # documents successfully processed
failed_count: int # documents that failed after retries
duration_seconds: float # total wall-clock time
step_metrics: Dict # per-step timing and counts
errors: List # list of FailedDocument records
class ExecutionResult:
success: bool # True if all steps completed without failure
output: Any # output from the final pipeline step
metadata: Dict[str, Any] # {"pipeline_id": "...", "execution_time": 1.23}
metrics: Dict[str, Any] # {"steps_executed": 4, "steps_failed": 0, "execution_time": 1.23}
errors: List[str] # error messages from failed steps (empty on full success)
# Access pattern
result.success # bool
result.output # final step output
result.metadata["pipeline_id"] # pipeline name used as ID
result.metadata["execution_time"] # total wall-clock seconds
result.metrics["steps_executed"] # count of successfully completed steps
result.metrics["steps_failed"] # count of failed steps
result.errors # List[str] of error messages
```
</Accordion>
<Accordion title="PipelineStep schema">
```python
@dataclass
class PipelineStep:
name: str
step_type: str
config: Dict[str, Any]
dependencies: List[str] # names of steps this step waits for
handler: Optional[Callable]
status: StepStatus
result: Any
error: Optional[Exception]
delta_mode: bool # True = process only changed data
base_version_id: Optional[str] # snapshot ID to diff against
target_version_id: Optional[str] # snapshot ID being produced
```
</Accordion>
<Accordion title="StepStatus enum">
```python
from semantica.pipeline import StepStatus
StepStatus.PENDING # Not yet started
StepStatus.RUNNING # Currently executing
StepStatus.COMPLETED # Finished successfully
StepStatus.FAILED # Error occurred — check step.error
StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy
```
</Accordion>
</AccordionGroup>
## Tips and Common Pitfalls
<Tip>
**Use `PipelineValidator` before running in production.** It catches dependency cycles, missing step names, and misconfigured connections that would only surface as errors mid-run. Validation is instant; catching them after a 30-minute extraction job is not.
</Tip>
<Tip>
**Set `workers=` based on workload type.** Thread workers for I/O-bound steps (web fetching, DB queries), process workers for CPU-bound steps (embedding, OCR, large NER batches). Mixing pool types on the wrong step type wastes resources without speed gains.
</Tip>
<Warning>
**Use `failure_handler=FailureHandler(strategy="skip")` in production.** A single malformed document shouldn't stop a pipeline processing 10,000 documents. `skip` logs the failure and continues; inspect `result.errors` after the run to find and reprocess failed documents.
</Warning>
<Tip>
**Use templates from `PipelineTemplateManager` for common patterns.** `get_template("full-qa")` wires up normalization, deduplication, conflict detection, and graph construction in the right order — saving you from common mistakes like deduplicating before normalizing.
</Tip>
<Tip>
**Inspect `result.metrics` to find bottlenecks.** `result.metrics['steps_executed']` and `result.metrics['execution_time']` give a quick read on overall pipeline health. For per-step timing, check `step.result` on each `PipelineStep` after the run.
</Tip>
<CardGroup cols={2}>
<Card title="Ingest" icon="database" href="ingest">
First step in most pipelines.
docs/reference/provenance.md
+95 -49
View File
@@ -1,85 +1,113 @@
---
title: "Provenance Module"
description: "W3C PROV-O compliant lineage tracking, source attribution, and audit trails across all modules."
description: "W3C PROV-O compliant lineage tracking, source attribution, tamper-evident checksums, and audit trails across all modules."
icon: "link"
---
`semantica.provenance` tracks the full lineage of every fact — from raw ingestion through extraction, reasoning, and export. Compliant with W3C PROV-O, suitable for HIPAA, SOX, GDPR, and FDA 21 CFR Part 11 environments.
## Exported Classes
```python
from semantica.provenance import (
ProvenanceManager, # track entities, get lineage, export PROV-O
ProvenanceEntry, # single provenance record (entity_id, source, method, ...)
SourceReference, # rich source pointer (DOI, page, quote, URL)
ProvenanceStorage, # abstract storage backend
InMemoryStorage, # default in-memory backend
SQLiteStorage, # persistent SQLite backend for production
compute_checksum, # compute tamper-evident hash for an entry
verify_checksum, # verify integrity of a stored entry
)
# GraphBuilderWithProvenance is in semantica.kg, not semantica.provenance
from semantica.kg import GraphBuilderWithProvenance
```
## What You Get
- **`ProvenanceManager`** — track entities, relationships, and activities with source attribution
- **`ActivityTracker`** — record pipeline activities and which entities they produced or consumed
- **`ProvenanceManager`** — track entities and relationships with source attribution and lineage retrieval
- **`ProvenanceEntry`** / **`SourceReference`** — structured records with DOI, page, quote, confidence, and timestamp
- **`InMemoryStorage`** / **`SQLiteStorage`** — swappable persistence backends
- **`compute_checksum`** / **`verify_checksum`** — tamper-evident integrity verification
- **Lineage graph** — full upstream lineage from any entity back to its source document
- **W3C PROV-O export** — serialize lineage as Turtle RDF for compliance reporting
- **`GraphBuilderWithProvenance`** — drop-in replacement that auto-tracks every node and edge
- **W3C PROV-O export** — serialize lineage as Turtle RDF or JSON-LD for compliance reporting
- **`GraphBuilderWithProvenance`** (in `semantica.kg`) — drop-in replacement that auto-tracks every node and edge
## ProvenanceManager
```python
from semantica.provenance import ProvenanceManager
from semantica.provenance import ProvenanceManager, InMemoryStorage, SQLiteStorage
manager = ProvenanceManager()
# In-memory (default) — fast, not persisted across restarts
manager = ProvenanceManager(storage=InMemoryStorage())
# Track an extracted entity
# SQLite — persisted, production-ready
manager = ProvenanceManager(storage=SQLiteStorage("provenance.db"))
# Track an extracted entity (with rich source reference)
manager.track_entity(
entity_id="apple_inc",
source="annual_report_2023.pdf",
entity_type="Organization",
extraction_method="llm",
confidence=0.98
source_location="Page 12, Section 3.1",
source_quote="Apple Inc. was incorporated on January 3, 1977.",
confidence=0.98,
)
# Track an extracted relationship
manager.track_relationship(
rel_id="steve_jobs_founded_apple",
manager.track_entity(
entity_id="steve_jobs_founded_apple",
source="annual_report_2023.pdf",
extraction_method="llm",
confidence=0.92
confidence=0.92,
)
# Retrieve full lineage for any entity
lineage = manager.get_lineage("apple_inc")
print(f"Source: {lineage.source}")
print(f"Extracted: {lineage.extracted_at}")
print(f"Method: {lineage.extraction_method}")
print(f"Confidence: {lineage.confidence}")
entry = manager.get_lineage("apple_inc")
print(f"Source: {entry.source}")
print(f"Quote: {entry.source_quote}")
print(f"Confidence: {entry.confidence}")
print(f"Tracked at: {entry.tracked_at}")
```
## Activity Tracking
## SourceReference
Record pipeline activities — what was consumed and what was produced:
`SourceReference` provides a rich, citable pointer to the exact location in a source document:
```python
# Start and end an activity
activity_id = manager.start_activity(
activity_type="ner_extraction",
used=["annual_report_2023.pdf"],
generated=["apple_inc", "steve_jobs"]
from semantica.provenance import SourceReference
ref = SourceReference(
document_id="annual_report_2023.pdf",
page=12,
section="3.1",
quote="Apple Inc. was incorporated on January 3, 1977.",
url="https://investor.apple.com/sec-filings/annual-reports/",
doi="10.0000/example.doi",
)
manager.end_activity(activity_id)
# Query activities for an entity
activities = manager.get_activities(entity_id="apple_inc")
for activity in activities:
print(f"{activity.type} at {activity.started_at}")
print(f" Used: {activity.used}")
print(f" Generated: {activity.generated}")
manager.track_entity(
entity_id="apple_inc",
source_reference=ref,
confidence=0.98,
)
```
## Lineage Graph
## Tamper-Evident Checksums
Retrieve a full directed lineage graph from any entity back to its source:
Verify that provenance records have not been modified after creation:
```python
lineage_graph = manager.get_lineage_graph("apple_inc")
from semantica.provenance import compute_checksum, verify_checksum
for node in lineage_graph.nodes:
print(f"{node.id}: {node.type} — {node.timestamp}")
entry = manager.get_lineage("apple_inc")
for edge in lineage_graph.edges:
print(f"{edge.source} → {edge.target} ({edge.relation})")
# Compute and store a checksum on first write
checksum = compute_checksum(entry)
# Later: verify the entry hasn’t been altered
is_valid = verify_checksum(entry, checksum)
if not is_valid:
raise RuntimeError("Provenance record has been tampered with!")
```
## W3C PROV-O Export
@@ -99,18 +127,36 @@ manager.export_all(path="provenance.jsonld", format="json-ld")
## Integration with GraphBuilder
`GraphBuilderWithProvenance` automatically records provenance for every node and edge constructed:
`GraphBuilderWithProvenance` (from `semantica.kg`) automatically records provenance for every node and edge constructed:
```python
from semantica.kg import GraphBuilderWithProvenance
from semantica.provenance import ProvenanceManager, SQLiteStorage
builder = GraphBuilderWithProvenance(provenance=True)
result = builder.build_single_source(graph_data)
prov_manager = ProvenanceManager(storage=SQLiteStorage("provenance.db"))
builder = GraphBuilderWithProvenance(provenance_manager=prov_manager)
kg = builder.build_single_source(graph_data)
# Each node and edge has a source_id linking back to the originating document
lineage = result.provenance_manager.get_lineage("apple_inc")
print(f"Source document: {lineage.source}")
print(f"Extracted by: {lineage.extraction_method}")
# Every node and edge now has full source attribution
entry = prov_manager.get_lineage("apple_inc")
print(f"Source document: {entry.source}")
print(f"Confidence: {entry.confidence}")
```
## Enable Provenance in Extractors
```python
from semantica.semantic_extract import NERExtractor
from semantica.provenance import ProvenanceManager
prov_manager = ProvenanceManager()
ner = NERExtractor(method="llm", llm_provider=llm, provenance=True)
entities = ner.extract(text)
# Retrieve lineage for the first extracted entity
entry = prov_manager.get_lineage(entities[0]["id"])
print(f"Source: {entry.source}")
```
## Compliance Standards
docs/reference/reasoning.md
+70 -3
View File
@@ -6,6 +6,35 @@ icon: "microchip"
`semantica.reasoning` derives new knowledge from existing facts using logical rules. Every engine produces **explainable inference paths** — traceable chains of rules and facts, not black-box conclusions.
## Exported Classes
```python
from semantica.reasoning import (
# Engines
Reasoner, # IF/THEN forward-chaining facade
GraphReasoner, # inference over full KG structure
ReteEngine, # high-performance Rete pattern matching
SPARQLReasoner, # SPARQL-based RDF inference
DatalogReasoner, # recursive Horn clause fixpoint evaluation
TemporalReasoningEngine, # Allen interval algebra (13 relations)
ExplanationGenerator, # structured step-by-step explanations
# Data types
Rule, # IF/THEN rule definition
Fact, # base fact (subject, predicate, obj)
RuleType, # enum: FORWARD_CHAIN, BACKWARD_CHAIN, ...
InferenceResult, # result of infer() — contains derived_facts list
DatalogFact, # Datalog base fact (predicate, args tuple)
DatalogRule, # Datalog Horn clause ("head :- body.")
TemporalInterval, # time interval with start/end
IntervalRelation, # enum of 13 Allen relations
# Explanation types
Explanation, # conclusion + confidence + reasoning_path
ReasoningPath, # ordered list of ReasoningSteps
ReasoningStep, # single step: fact + rule_name + depth
Justification, # full justification record
)
```
## What You Get
- **`Reasoner`** — main facade for IF/THEN forward-chaining with variable substitution
@@ -16,6 +45,28 @@ icon: "microchip"
- **`TemporalReasoningEngine`** — all 13 Allen interval algebra relations for time-aware inference
- **`ExplanationGenerator`** — structured explanation paths for every derived conclusion
## Quick Start
The most common pattern: add facts + rules, run inference, explain a conclusion:
```python
from semantica.reasoning import Reasoner, Rule, Fact, RuleType, InferenceResult
reasoner = Reasoner()
reasoner.add_fact(Fact(subject="Alice", predicate="is_a", obj="Manager"))
reasoner.add_rule(Rule(
rule_type=RuleType.FORWARD_CHAIN,
conditions=[{"subject": "?x", "predicate": "is_a", "object": "Manager"}],
conclusion={"subject": "?x", "predicate": "has_authority", "object": "true"},
))
result: InferenceResult = reasoner.infer()
for fact in result.derived_facts:
print(f"{fact.subject} {fact.predicate} {fact.obj}")
print(f" via: {fact.explanation}")
```
<img src="/assets/img/diagrams/reasoning-chain.svg" alt="Forward chaining inference: known facts + IF/THEN rules produce derived facts with a full traceable explanation path" style={{ width: '100%', borderRadius: '12px', margin: '0 0 24px' }} />
## Reasoner (Main Facade)
@@ -202,22 +253,38 @@ All 13 Allen interval algebra relations are supported:
Generate structured step-by-step explanations for any derived conclusion:
```python
from semantica.reasoning import ExplanationGenerator
from semantica.reasoning import ExplanationGenerator, Explanation, ReasoningStep
generator = ExplanationGenerator(reasoner)
explanation = generator.explain(
explanation: Explanation = generator.explain(
conclusion={"subject": "John", "predicate": "has_authority", "object": "true"}
)
print(explanation.conclusion)
print(f"Conclusion: {explanation.conclusion}")
print(f"Confidence: {explanation.confidence:.2f}")
step: ReasoningStep
for step in explanation.reasoning_path.steps:
print(f" Step {step.depth}: {step.fact}")
print(f" via rule: '{step.rule_name}'")
```
## Choosing an Engine
| Engine | Best For | Termination | Complexity |
| ------ | -------- | ----------- | ---------- |
| `Reasoner` | Simple IF/THEN rules, templates | Always | Low |
| `GraphReasoner` | KG-wide structural inference | Always | Medium |
| `ReteEngine` | Large rule sets (100+ rules) | Always | Low per-match |
| `SPARQLReasoner` | RDF graphs with SPARQL endpoint | Always | Low |
| `DatalogReasoner` | Recursive rules (ancestry, reachability) | Guaranteed fixpoint | Medium |
| `TemporalReasoningEngine` | Time interval relationships | Always | Low |
<Tip>
For recursive rules (e.g. ancestor, reachability, transitivity), always use `DatalogReasoner` — it guarantees termination via semi-naive bottom-up fixpoint evaluation. `Reasoner` does not handle recursion.
</Tip>
<CardGroup cols={2}>
<Card title="Knowledge Graph" icon="diagram-project" href="kg">
The knowledge graph being reasoned over.
docs/reference/seed.md
+322 -83
View File
@@ -1,119 +1,323 @@
---
title: "Seed Module"
description: "Seed data management for initializing Knowledge Graphs from trusted, verified sources."
description: "Bootstrap Knowledge Graphs from verified, structured sources — taxonomies, reference tables, product catalogs, and domain anchors."
icon: "database"
---
`semantica.seed` provides a system for bootstrapping Knowledge Graphs with verified, structured data from trusted sources — taxonomies, reference tables, user lists, product catalogs — so you start with a reliable foundation rather than an empty graph.
`semantica.seed` gives your knowledge graph a reliable starting point. Rather than building from an empty graph and hoping extraction produces consistent reference data, you load verified, structured sources first — ISO codes, employee rosters, product catalogs, domain taxonomies — then merge freshly extracted data on top.
## Exported Classes
```python
from semantica.seed import (
SeedDataManager, # coordinator: register_source, create_foundation_graph, integrate_with_extracted
SeedDataSource, # {name, source_type, path, config} — dataclass for a registered source
SeedData, # {entities, relationships, metadata} — loaded seed data container
)
```
## What You Get
- **`SeedDataManager`** — register sources, build a foundation graph, and integrate with extracted data
- **`SeedDataSource`** — define individual data sources with format, path, and config
- **Merge strategies** — `seed_first`, `extracted_first`, `smart_merge` for combining seed and extracted data
- **Validation** — check data quality and schema compliance before loading
- **Versioning** — track and manage versions of seed data sources
<CardGroup cols={2}>
<Card title="SeedDataManager" icon="database">
Register sources, build a foundation graph, validate quality, and merge with extracted data.
</Card>
<Card title="SeedDataSource" icon="file-code">
Typed source definition supporting CSV, JSON, SQL, API, and RDF with format-specific config.
</Card>
<Card title="Foundation Graph" icon="circle-plus">
Build a foundation graph from all registered sources in one pass, ready to merge with extracted data.
</Card>
<Card title="Merge Strategies" icon="arrows-merge">
`seed_first`, `extracted_first`, and `smart_merge` with property-level conflict detection.
</Card>
<Card title="Validation" icon="shield-check">
Required field checks, ID uniqueness, type consistency, reference integrity, and encoding validation before loading.
</Card>
<Card title="Versioning" icon="clock-rotate-left">
Track seed data versions across pipeline runs and diff changes between versions.
</Card>
</CardGroup>
<Tip>
**When to use the Seed Module:**
- **Bootstrapping** — you have existing structured data (taxonomies, user lists, product catalogs) to build on
- **Reference data** — load immutable reference information (countries, ISO codes, ontology terms)
- **Testing** — load consistent, reproducible datasets for development and CI pipelines
**When to use the Seed Module:** Bootstrapping with structured reference data (taxonomies, user lists, product catalogs), loading immutable facts (ISO country codes, standard ontology terms) that extracted data should not override, ensuring test reproducibility with deterministic datasets, and anchoring entity disambiguation with canonical forms.
</Tip>
## SeedDataManager
## Quick Start
The primary interface for seed data:
<Steps>
<Step title="Register your seed sources">
```python
from semantica.seed import SeedDataManager
manager = SeedDataManager()
manager.register_source("countries", "csv", "data/countries.csv")
manager.register_source("taxonomy", "json", "data/taxonomy.json")
manager.register_source("employees", "csv", "data/employees.csv")
```
</Step>
<Step title="Build the foundation graph">
```python
foundation_kg = manager.create_foundation_graph()
print(f"Foundation nodes: {foundation_kg.node_count}")
print(f"Foundation edges: {foundation_kg.edge_count}")
```
</Step>
<Step title="Validate before loading">
```python
report = manager.validate_quality(manager.load_source("employees"))
if not report.is_valid:
for issue in report.issues:
print(f"[{issue.severity}] Row {issue.row}: {issue.message}")
else:
print(f"Validated {report.record_count} records — no issues found")
```
</Step>
<Step title="Merge with extracted data">
```python
final_kg = manager.integrate_with_extracted(
seed_graph=foundation_kg,
extracted_data=new_entities,
strategy="smart_merge",
)
print(f"Final graph: {final_kg.node_count} nodes, {final_kg.edge_count} edges")
```
</Step>
</Steps>
## SeedDataSource Types
<Tabs>
<Tab title="CSV">
```python
from semantica.seed import SeedDataSource
csv_source = SeedDataSource(
name="employees",
type="csv",
path="data/employees.csv",
config={
"delimiter": ";",
"encoding": "utf-8",
"id_column": "employee_id",
"type": "Person",
}
)
manager.register_source("employees", "csv", csv_source.path, config=csv_source.config)
```
Best for: employee rosters, product lists, reference tables.
</Tab>
<Tab title="JSON">
```python
json_source = SeedDataSource(
name="taxonomy",
type="json",
path="data/taxonomy.json",
config={"encoding": "utf-8"}
)
```
Expects an array of entity objects. Best for: taxonomies, ontology term lists, structured configs.
</Tab>
<Tab title="SQL">
```python
sql_source = SeedDataSource(
name="products",
type="sql",
path="postgresql://user:pass@localhost/db",
config={"query": "SELECT id, name, category FROM products WHERE active = true"}
)
```
Best for: live database tables — PostgreSQL, MySQL, SQLite.
</Tab>
<Tab title="API & RDF">
```python
# API source — fetch from a REST endpoint
api_source = SeedDataSource(
name="geo_codes",
type="api",
path="https://restcountries.com/v3.1/all",
config={"fields": ["name", "cca2", "region"]}
)
# RDF source — OWL ontologies or Turtle files
rdf_source = SeedDataSource(
name="domain_ontology",
type="rdf",
path="data/ontology.ttl",
config={"format": "turtle"}
)
```
API: external reference APIs (countries, currencies, geo). RDF: existing knowledge bases, OWL ontologies.
</Tab>
<Tab title="Type Reference">
| Type | Path Format | Use Case |
| ---- | ----------- | -------- |
| `csv` | File path | Employee rosters, product lists, reference tables |
| `json` | File path | Taxonomies, ontology term lists, structured configs |
| `sql` | Connection string | Live database tables — PostgreSQL, MySQL, SQLite |
| `api` | URL | External reference APIs (countries, currencies, geo) |
| `rdf` | File path | OWL ontologies, Turtle files, existing knowledge bases |
</Tab>
</Tabs>
## SeedDataManager Reference
| Method | Description |
| ------ | ----------- |
| `register_source(name, format, path)` | Add a named data source to the registry |
| `create_foundation_graph()` | Build a KG from all registered sources |
| `validate_quality(seed_data)` | Check schema compliance, required fields, and duplicates |
| `integrate_with_extracted(seed, extracted, strategy)` | Merge seed and extracted graphs |
| `export_seed_data(path, format)` | Export seed graph to RDF (`turtle`, `json-ld`), JSON, or CSV |
| `load_from_csv(path)` | Load seed records from a CSV file |
| `load_from_json(path)` | Load seed records from a JSON file |
| `list_sources()` | List all registered source names and their formats |
| `get_version(name)` | Get the current version metadata for a named source |
## Merge Strategies
<Tabs>
<Tab title="seed_first">
Seed data wins on every conflicting property. Use when seed encodes authoritative reference facts that must not be overridden.
```python
final_kg = manager.integrate_with_extracted(
seed_graph=foundation_kg,
extracted_data=new_entities,
strategy="seed_first",
)
```
Best for: ISO codes, canonical entity names, official taxonomy IDs, employee records.
</Tab>
<Tab title="extracted_first">
Extracted data overrides seed on conflicting properties. Use when new documents contain more current information than your reference data.
```python
final_kg = manager.integrate_with_extracted(
seed_graph=foundation_kg,
extracted_data=new_entities,
strategy="extracted_first",
)
```
Best for: frequently changing attributes like addresses, titles, revenue figures.
</Tab>
<Tab title="smart_merge">
Property-level merge with conflict detection — irresolvable conflicts are logged for manual review rather than silently overwritten.
```python
final_kg = manager.integrate_with_extracted(
seed_graph=foundation_kg,
extracted_data=new_entities,
strategy="smart_merge",
)
```
Best for: general-purpose pipelines where surfacing conflicts is more valuable than silently losing data.
</Tab>
</Tabs>
## Built-in Datasets
Register built-in reference datasets as named sources and load them into your foundation graph:
```python
from semantica.seed import SeedDataManager
manager = SeedDataManager()
manager.register_source("countries", "csv", "data/countries.csv")
manager.register_source("taxonomy", "json", "data/taxonomy.json")
# Build a foundation KG from all registered sources
# Register built-in reference sources by format and path
manager.register_source("countries", "csv", "data/iso_countries.csv")
manager.register_source("currencies", "json", "data/iso_currencies.json")
foundation_kg = manager.create_foundation_graph()
```
### Core Methods
| Dataset | Content |
| ------- | ------- |
| `companies` | Fortune 500 companies with type, sector, HQ |
| `countries` | ISO 3166 country codes, regions, populations |
| `currencies` | ISO 4217 codes, symbols, names |
| `person_names` | Common first/last names for synthetic data |
| Method | Description |
| ------ | ----------- |
| `register_source(name, format, location)` | Add a data source to the registry |
| `create_foundation_graph()` | Build a KG from all registered sources |
| `validate_quality(seed_data)` | Check data quality and completeness |
| `integrate_with_extracted(seed, extracted)` | Merge seed and extracted graphs |
| `export_seed_data(path, format)` | Export seed data to RDF, JSON, or CSV |
## SeedDataSource
Define a source with format-specific configuration:
```python
from semantica.seed import SeedDataSource
source = SeedDataSource(
name="taxonomy",
type="json", # "csv" | "json" | "api" | "sql"
path="taxonomy.json",
config={"encoding": "utf-8"}
)
```
## Merge Strategies
Control how seed data and extracted data are combined:
```python
final_kg = manager.integrate_seed_extracted(
seed_graph=foundation_kg,
extracted_data=new_data,
strategy="seed_first" # see options below
)
```
| Strategy | Behavior |
| -------- | -------- |
| `seed_first` | Seed data wins on conflicts — use for authoritative reference data |
| `extracted_first` | Extracted data overrides seed — use when new data is more current |
| `smart_merge` | Property-level merging with conflict detection and resolution |
## Bootstrapping a KG
Full example — load foundation data then merge with freshly ingested content:
## Full Pipeline Example
```python
from semantica.seed import SeedDataManager
from semantica.ingest import FileIngestor
from semantica.semantic_extract import NERExtractor
from semantica.parse import DocumentParser
from semantica.split import TextSplitter
from semantica.semantic_extract import NERExtractor, RelationExtractor
from semantica.kg import GraphBuilder
from semantica.llms import Groq
import os
# Build foundation from verified reference data
manager = SeedDataManager()
manager.register_source("taxonomy", "json", "taxonomy.json")
manager.register_source("employees", "csv", "employees.csv")
foundation_kg = manager.create_foundation_graph()
# Ingest and extract from new sources
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
ingestor = FileIngestor()
ner = NERExtractor(method="llm", llm_provider=llm)
sources = ingestor.ingest("news_articles/")
new_data = [ner.extract(s.content) for s in sources]
# Step 1 — Build the foundation from verified reference data
seed_manager = SeedDataManager()
seed_manager.register_source("taxonomy", "json", "data/taxonomy.json")
seed_manager.register_source("employees", "csv", "data/employees.csv")
foundation_kg = seed_manager.create_foundation_graph()
# Merge — seed data takes precedence for reference facts
final_kg = manager.integrate_seed_extracted(
# Step 2 — Ingest and extract from unstructured documents
ingestor = FileIngestor()
parser = DocumentParser()
splitter = TextSplitter(method="semantic_transformer", chunk_size=512)
ner = NERExtractor(method="llm", llm_provider=llm)
rel_ext = RelationExtractor(method="llm", llm_provider=llm)
sources = ingestor.ingest("news_articles/")
extracted_entities = []
extracted_relationships = []
for source in sources:
parsed = parser.parse(source)
chunks = splitter.split_document(parsed)
for chunk in chunks:
entities = ner.extract(chunk.text)
relationships = rel_ext.extract(chunk.text, entities=entities)
extracted_entities.extend(entities)
extracted_relationships.extend(relationships)
# Step 3 — Merge seed and extracted data
final_kg = seed_manager.integrate_with_extracted(
seed_graph=foundation_kg,
extracted_data=new_data,
strategy="seed_first"
extracted_data=extracted_entities,
strategy="seed_first",
)
print(f"Final graph: {final_kg.node_count} nodes, {final_kg.edge_count} edges")
```
## Configuration
## Versioning
Track seed data versions to detect when reference data changes between pipeline runs:
```python
manager = SeedDataManager()
manager.register_source("taxonomy", "json", "data/taxonomy.json")
version = manager.get_version("taxonomy")
print(f"Version: {version.version_id}")
print(f"Hash: {version.checksum}")
print(f"Records: {version.record_count}")
print(f"Updated: {version.last_modified}")
```
## YAML Configuration
Define sources in YAML for production deployments — no code changes needed to switch environments:
```yaml
seed:
@@ -121,22 +325,57 @@ seed:
- name: "employees"
type: "csv"
path: "./data/employees.csv"
config:
id_column: "employee_id"
type: "Person"
- name: "taxonomy"
type: "json"
path: "./data/taxonomy.json"
- name: "products"
type: "sql"
path: "${DATABASE_URL}"
config:
query: "SELECT id, name, category FROM products WHERE active = true"
merge:
strategy: "seed_first"
strategy: "smart_merge"
validation:
strict: true
required_fields: ["id", "type"]
```
Environment variable overrides:
```bash
export SEED_DATA_DIR=./data/seed
export SEED_MERGE_STRATEGY=seed_first
export SEMANTICA_SEED_DATA_DIR=./data/seed
export SEMANTICA_SEED_MERGE_STRATEGY=seed_first
```
## Tips and Common Pitfalls
<Warning>
**Load seed data before extracted data.** Seed data is your ground truth — normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values.
</Warning>
<Tip>
**Use `seed_first` merge strategy for reference data.** When seed data encodes authoritative facts (official company names, canonical taxonomy IDs, employee records), `strategy="seed_first"` ensures those values win over extracted values. Use `smart_merge` only when extracted data may be more current than the seed.
</Tip>
<Warning>
**Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast — always run it first.
</Warning>
<Tip>
**Register all sources before calling `create_foundation_graph()`.** `create_foundation_graph()` processes all registered sources in one pass. Registering a source after calling it means that source is silently excluded. Register all sources at the start of your script, then call `create_foundation_graph()` once.
</Tip>
<Tip>
**Track seed versions to detect drift.** Use `manager.get_version()` to check the checksum and record count for each registered source between pipeline runs. If a taxonomy file changes, downstream entity normalisation and deduplication thresholds may need re-tuning — don't treat seed data as static.
</Tip>
<Tip>
**Use YAML configuration for production deployments.** Hard-coding source paths in Python scripts makes environment-switching (dev → staging → prod) fragile. Declare sources in `config.yaml` under the `seed:` key and override paths with `SEMANTICA_SEED_DATA_DIR`. This way, the same code runs in every environment.
</Tip>
<CardGroup cols={2}>
<Card title="Ingest" icon="file-import" href="ingest">
Load unstructured data alongside seed data.
@@ -148,6 +387,6 @@ export SEED_MERGE_STRATEGY=seed_first
Handle duplicates during seed-extracted merge.
</Card>
<Card title="Pipeline" icon="gear" href="pipeline">
Incorporate seed loading as a pipeline step.
Incorporate seed loading as a named pipeline step.
</Card>
</CardGroup>
docs/reference/semantic_extract.md
+56 -7
View File
@@ -6,14 +6,56 @@ icon: "magnifying-glass-chart"
`semantica.semantic_extract` extracts structured information from unstructured text — the foundation of every knowledge graph in Semantica. All extractors support three modes: pattern-based (no API key), ML-based, and LLM-based.
## Exported Classes
```python
from semantica.semantic_extract import (
# Primary extractors
NamedEntityRecognizer, # full NER coordinator (confidence_threshold, merge_overlapping)
NERExtractor, # core NER implementation used by NamedEntityRecognizer
RelationExtractor, # typed relationship extraction
TripletExtractor, # (subject, predicate, object) triplet generation
EventDetector, # event detection with participants and temporal context
CoreferenceResolver, # resolve pronouns and aliases to canonical entities
# Data types
Entity, # {id, text, type, confidence, start, end}
Relation, # {subject, predicate, object, confidence}
Event, # {type, participants, temporal, location, confidence}
CoreferenceChain, # list of mentions resolving to the same entity
# Advanced
EntityClassifier, # classify entity candidates by type
CustomEntityDetector, # pattern/dictionary-based custom entity detection
TemporalEventProcessor, # extract temporal information from events
)
```
<Tip>
`NamedEntityRecognizer` is the high-level coordinator with confidence thresholding and overlap merging. `NERExtractor` is the lower-level implementation. For most use cases, start with `NERExtractor` for simplicity or `NamedEntityRecognizer` for fine-grained control.
</Tip>
## What You Get
- **`NERExtractor`** — named entity recognition: Person, Organization, Location, Date, and custom types
- **`NERExtractor`** / **`NamedEntityRecognizer`** — named entity recognition: Person, Organization, Location, Date, and custom types
- **`RelationExtractor`** — typed semantic relationships between entities (`founded_by`, `located_in`, etc.)
- **`TripletExtractor`** — direct `(subject, predicate, object)` triplet generation for RDF-ready output
- **`EventExtractor`** — event detection with participants, temporal context, and confidence scores
- **`EventDetector`** — event detection with participants, temporal context, and confidence scores
- **`CoreferenceResolver`** — resolve "Apple" and "the company" to the same entity across a document
## Quick Start
```python
from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor
from semantica.llms import Groq
import os
text = "Apple Inc. was founded by Steve Jobs in Cupertino in 1976."
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
entities = NERExtractor(method="llm", llm_provider=llm).extract(text)
relationships = RelationExtractor(method="llm", llm_provider=llm).extract(text, entities=entities)
triplets = TripletExtractor(method="llm", llm_provider=llm).extract(text)
```
<img src="/assets/img/diagrams/extraction-pipeline.svg" alt="Semantic extraction pipeline: raw text fans into NER, Relation, and Coreference extractors, then merges into a Triplet Generator" style={{ width: '100%', borderRadius: '12px', margin: '0 0 24px' }} />
## NERExtractor
@@ -97,18 +139,25 @@ triplets = trip.extract(text)
Triplets are suitable for loading directly into a triplet store or knowledge graph.
## EventExtractor
## EventDetector
Detect events with participants and temporal context:
```python
from semantica.semantic_extract import EventExtractor
from typing import List
from semantica.semantic_extract import EventDetector, Event
extractor = EventExtractor(method="llm", llm_provider=llm)
events = extractor.extract(text)
extractor = EventDetector(method="llm", llm_provider=llm)
events: List[Event] = extractor.extract(text)
for event in events:
print(f"Event type: {event.type}")
print(f"Participants: {event.participants}")
print(f"Temporal: {event.temporal}")
print(f"Confidence: {event.confidence:.2f}")
```
Output includes: event type, participants (with roles), temporal information, location, and confidence score.
Output fields per event: `type`, `participants` (with roles), `temporal`, `location`, and `confidence`.
## CoreferenceResolver
docs/reference/split.md
+388 -89
View File
@@ -1,145 +1,444 @@
---
title: "Split Module"
description: "15+ text chunking methods including recursive, semantic, entity-aware, and relation-aware splitting."
description: "15+ text chunking methods including recursive, semantic, entity-aware, relation-aware, code, and structural splitting."
icon: "scissors"
---
`semantica.split` breaks documents into chunks while preserving semantic context — critical for embedding quality in RAG systems and accurate entity extraction in NER pipelines.
`semantica.split` breaks documents into chunks that preserve semantic context. Chunking quality directly determines downstream accuracy — a poorly chunked document produces bad embeddings, missed entities, and broken relation triplets. Use the right strategy for your content type and pipeline goal.
## Why Chunking Matters
Most LLMs and embedding models have fixed context windows. Documents larger than that window must be split. But naive splitting (every 500 characters, regardless of structure) destroys semantic context:
- An entity mention like "Apple Inc." split across two chunks loses its context in both
- A relation triplet like "Steve Jobs founded Apple" split at "Steve Jobs" leaves a dangling subject
- Embedding a chunk that mixes two unrelated topics produces a centroid vector that matches neither
Semantica's chunking methods are designed to avoid these failure modes.
## Exported Classes
```python
from semantica.split import (
# Unified splitter (start here)
TextSplitter, # method=: recursive, sentence, token, semantic_transformer,
# entity_aware, relation_aware, code, structural, markdown
Splitter, # alias for TextSplitter (backward compat)
# Data type
Chunk, # {text, start_char, end_char, token_count, metadata, entities, relationships}
# Specialized chunkers
SemanticChunker, # embedding-based semantic boundary detection
StructuralChunker, # heading/section-based splits from ParsedDocument
SlidingWindowChunker, # fixed-size sliding window with overlap
TableChunker, # table-specific chunking
EntityAwareChunker, # KG: preserves named entities across chunk boundaries
RelationAwareChunker, # KG: keeps subject-predicate-object triplets intact
GraphBasedChunker, # splits based on graph community structure
OntologyAwareChunker, # splits respecting ontology concept boundaries
HierarchicalChunker, # multi-level hierarchical chunking
ProvenanceTracker, # track chunk provenance back to source document
# Convenience split functions
split_recursive, # split_recursive(text, chunk_size, chunk_overlap)
split_by_sentences, # split_by_sentences(text)
split_by_tokens, # split_by_tokens(text, chunk_size, tokenizer)
split_semantic_transformer, # split_semantic_transformer(text, threshold)
split_entity_aware, # split_entity_aware(text, entities)
split_relation_aware, # split_relation_aware(text, relationships)
)
```
## What You Get
- **`TextSplitter`** — unified interface for 9+ chunking strategies
- **Entity-aware chunking** — entity mentions never split across chunk boundaries
- **Relation-aware chunking** — subject–predicate–object triplets kept intact
- **Semantic chunking** — split at topic shift boundaries using embedding similarity
- **`Chunk`** — output object with text, token count, character offsets, and metadata
<CardGroup cols={2}>
<Card title="TextSplitter" icon="scissors">
Unified interface for 11 chunking strategies — swap methods without changing downstream code.
</Card>
<Card title="Semantic Chunking" icon="brain">
Embedding-based topic shift detection — splits only when the topic actually changes.
</Card>
<Card title="Entity-Aware Chunking" icon="user">
Entity spans never cross chunk boundaries — guaranteed by boundary adjustment.
</Card>
<Card title="Relation-Aware Chunking" icon="arrows-left-right">
Subject–predicate–object triplets kept within a single chunk for KG pipelines.
</Card>
<Card title="Code Splitting" icon="code">
AST-level boundaries (function, class, method) for source code search and analysis.
</Card>
<Card title="Chunk Object" icon="box">
Output dataclass with text, token count, character offsets, entities, and full metadata.
</Card>
</CardGroup>
## TextSplitter
## Quick Start
```python
from semantica.split import TextSplitter
<Steps>
<Step title="Choose a splitting method">
```python
from semantica.split import TextSplitter
splitter = TextSplitter(
method="semantic_transformer", # see methods table below
chunk_size=1000, # target tokens per chunk
chunk_overlap=200 # token overlap between adjacent chunks
)
splitter = TextSplitter(
method="recursive", # see Splitting Methods table
chunk_size=1000,
chunk_overlap=200,
)
```
</Step>
<Step title="Split raw text">
```python
chunks = splitter.split(text)
chunks = splitter.split(text)
for chunk in chunks:
print(f"Chunk {chunk.metadata['chunk_index']}: {chunk.text[:80]}...")
print(f" Tokens: {chunk.token_count}")
```
for chunk in chunks:
print(f"Chunk {chunk.metadata['chunk_index']} / {chunk.metadata['total_chunks']}")
print(f" Tokens: {chunk.token_count}")
print(f" Preview: {chunk.text[:80]}...")
```
</Step>
<Step title="Or split a ParsedDocument">
```python
from semantica.parse import DocumentParser
parser = DocumentParser()
parsed = parser.parse("annual_report.pdf")
splitter = TextSplitter(method="structural")
chunks = splitter.split_documents([parsed])
for chunk in chunks:
print(f"[h{chunk.metadata['heading_level']}] {chunk.metadata['section_title']}")
```
</Step>
<Step title="Batch-split a list of documents">
```python
all_chunks = splitter.split_documents(parsed_docs)
from collections import defaultdict
by_source = defaultdict(list)
for chunk in all_chunks:
by_source[chunk.metadata['source_id']].append(chunk)
```
</Step>
</Steps>
## Splitting Methods
| Method | Description | Best For |
| ------ | ----------- | -------- |
| `recursive` | Split by paragraph → sentence → word (cascading) | General purpose |
| `semantic_transformer` | Split at semantic topic boundaries via sentence transformer | RAG retrieval |
| `entity_aware` | Keep entity mentions intact across boundaries | NER pipelines |
| `relation_aware` | Keep relation triplets intact | KG construction |
| `sentence` | Split by sentence boundary | Short content |
| `token` | Split by token count (tiktoken) | LLM context windows |
| `fixed` | Fixed character count with overlap | Batch processing |
| `markdown` | Split by Markdown heading hierarchy | Documentation |
| `code` | Split by function/class/method boundaries | Code analysis |
| Method | How It Splits | Best For |
| ------ | ------------- | -------- |
| `recursive` | Paragraph → sentence → word (cascading fallback) | General-purpose default |
| `semantic_transformer` | Embeds sentences, splits at cosine similarity drops | RAG — topic coherence matters |
| `entity_aware` | Adjusts boundaries so entity spans are never cut | NER pipelines |
| `relation_aware` | Keeps subject–predicate–object triplets within one chunk | KG construction |
| `sentence` | Language-aware sentence boundary detection (NLTK/spaCy) | Short documents, Q&A |
| `token` | Exact token count via tiktoken; hard cutoff | LLM context window prep |
| `fixed` | Fixed character count with overlap; fastest, no NLP | Simple batch jobs |
| `sliding_window` | Fixed-step window — heavy overlap for dense retrieval | Bi-encoder retrieval (ColBERT, DPR) |
| `markdown` | Splits at Markdown heading levels (configurable) | Documentation, wikis, MDX |
| `structural` | Splits at `ParsedDocument.sections` boundaries | Structured PDFs and DOCX |
| `code` | AST-level splits at function / class / method boundaries | Source code search and analysis |
## Entity-Aware Chunking
## Choosing a Strategy
Entity mentions are never split across chunk boundaries, preserving context for downstream NER:
Use this decision tree before picking a method:
```python
from semantica.split import TextSplitter
from semantica.semantic_extract import NERExtractor
- **Source code?** → `code`
- **Markdown or structured doc with headings?** → `markdown` or `structural`
- **Building a KG?** → `relation_aware` (keeps triplets intact), then `entity_aware` for pure NER
- **RAG system where retrieval quality matters most?** → `semantic_transformer`
- **Dense overlap for bi-encoder retrieval (ColBERT, DPR)?** → `sliding_window`
- **Preparing prompts for a fixed-window LLM?** → `token`
- **Fast splitting with no NLP overhead?** → `recursive` or `fixed`
ner = NERExtractor()
entities = ner.extract(text)
splitter = TextSplitter(method="entity_aware")
chunks = splitter.split(text, entities=entities)
# → Each chunk contains only complete entity mentions
```
## Relation-Aware Chunking
Subject–predicate–object triplets are kept within the same chunk:
## TextSplitter Constructor
```python
from semantica.split import TextSplitter
splitter = TextSplitter(method="relation_aware")
chunks = splitter.split(text, relationships=relationships)
# → Triplets are never split across chunk boundaries
```
## Semantic Chunking
Split at topic shift boundaries detected via embedding similarity:
```python
from semantica.split import TextSplitter
from semantica.embeddings import EmbeddingGenerator
embedder = EmbeddingGenerator(model="sentence-transformers")
splitter = TextSplitter(
method="semantic_transformer",
embedder=embedder,
similarity_threshold=0.7 # split when consecutive sentence similarity drops below this
method="semantic_transformer", # chunking strategy
chunk_size=1000, # target size in tokens
chunk_overlap=200, # token overlap between adjacent chunks
tokenizer="cl100k_base", # tiktoken encoding (GPT-4 default)
min_chunk_size=50, # discard very short trailing chunks
include_metadata=True, # attach source_id, page_number, section_title
language="en", # ISO 639-1 — used by sentence boundary detector
)
chunks = splitter.split(text)
```
## Token-Based Chunking
| Parameter | Type | Default | Description |
| --------- | ---- | ------- | ----------- |
| `method` | `str` | `"recursive"` | Chunking strategy — see table above |
| `chunk_size` | `int` | `1000` | Target size in tokens (characters for `fixed`) |
| `chunk_overlap` | `int` | `200` | Token overlap between adjacent chunks |
| `tokenizer` | `str` | `"cl100k_base"` | tiktoken encoding: `"cl100k_base"` (GPT-4), `"p50k_base"` (GPT-3), `"r50k_base"` (Codex) |
| `min_chunk_size` | `int` | `0` | Discard chunks shorter than this many tokens |
| `similarity_threshold` | `float` | `0.7` | Cosine similarity cutoff for `semantic_transformer` |
| `embedder` | `EmbeddingGenerator` | `None` | Custom embedder for `semantic_transformer` |
| `include_metadata` | `bool` | `True` | Attach `source_id`, `page_number`, `section_title` to each chunk |
| `language` | `str` | `"en"` | ISO 639-1 language code for sentence boundary detection |
| `heading_levels` | `list[int]` | `[1, 2, 3]` | Heading levels to split on for `markdown` method |
| `code_units` | `list[str]` | `["function", "class"]` | AST node types to split on for `code` method |
Use tiktoken for precise token-count control when preparing LLM context windows:
## Splitting Method Details
```python
splitter = TextSplitter(
method="token",
chunk_size=512, # max tokens per chunk
chunk_overlap=50, # overlap in tokens
tokenizer="cl100k_base" # OpenAI tokenizer
)
chunks = splitter.split(text)
```
<Tabs>
<Tab title="Recursive (default)">
Tries paragraph breaks first, then sentence boundaries, then word boundaries — falling back only when the chunk exceeds `chunk_size`:
## Chunk Object
```python
splitter = TextSplitter(method="recursive", chunk_size=1000, chunk_overlap=200)
chunks = splitter.split(text)
```
**Key behaviours:**
- Preserves paragraph and sentence structure wherever possible
- Falls back gracefully — never produces chunks larger than `chunk_size`
- Overlap ensures context continuity across chunk boundaries
- Good starting point when you're unsure which method to use
</Tab>
<Tab title="Semantic">
Embeds each sentence, then splits whenever cosine similarity between consecutive sentences drops below `similarity_threshold`. Each chunk talks about one topic:
```python
from semantica.split import TextSplitter
from semantica.embeddings import EmbeddingGenerator
embedder = EmbeddingGenerator(model="sentence-transformers")
splitter = TextSplitter(
method="semantic_transformer",
embedder=embedder,
similarity_threshold=0.7, # 0.6 = more splits, 0.8 = fewer splits
chunk_size=800,
chunk_overlap=0, # not needed — chunks are already coherent
)
chunks = splitter.split(text)
```
**Key behaviours:**
- Produces variable-length chunks — some topics are short, others long
- Requires an embedder — defaults to `sentence-transformers/all-MiniLM-L6-v2` if not set
- Slower than `recursive` due to embedding computation; cache embeddings for repeated splits
- Best retrieval quality for semantic search — chunks map to single coherent topics
</Tab>
<Tab title="Entity-Aware">
Runs NER first, then adjusts chunk boundaries so no entity mention is split across two chunks:
```python
from semantica.split import TextSplitter
from semantica.semantic_extract import NERExtractor
from semantica.llms import Groq
import os
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
ner = NERExtractor(method="llm", llm_provider=llm)
entities = ner.extract(text)
splitter = TextSplitter(method="entity_aware", chunk_size=512, chunk_overlap=50)
chunks = splitter.split(text, entities=entities)
for chunk in chunks:
print(f"Chunk {chunk.metadata['chunk_index']}: {len(chunk.entities)} entities")
```
**Key behaviours:**
- Entity spans in `chunk.entities` are guaranteed to fall entirely within `chunk.text`
- Chunk sizes vary slightly from `chunk_size` — boundary adjustments are ≤ one sentence
- Works with all entity types: PERSON, ORGANIZATION, LOCATION, DATE, custom types
</Tab>
<Tab title="Relation-Aware">
Keeps subject–predicate–object triplets within the same chunk — critical for KG pipelines:
```python
from semantica.split import TextSplitter
from semantica.semantic_extract import RelationExtractor, NERExtractor
from semantica.llms import Groq
import os
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
ner = NERExtractor(method="llm", llm_provider=llm)
rel_extractor = RelationExtractor(method="llm", llm_provider=llm)
entities = ner.extract(text)
relationships = rel_extractor.extract(text, entities=entities)
splitter = TextSplitter(method="relation_aware", chunk_size=512)
chunks = splitter.split(text, relationships=relationships)
for chunk in chunks:
print(f"Chunk {chunk.metadata['chunk_index']}: {len(chunk.relationships)} triplets")
for rel in chunk.relationships:
print(f" {rel['subject']} —[{rel['predicate']}]→ {rel['object']}")
```
**Key behaviours:**
- Relation triplets in `chunk.relationships` are always fully contained within the chunk
- Implies entity-aware behaviour — both entities in a triplet are kept whole too
- Best used as the split step in a `Parse → Split → Extract → Build KG` pipeline
</Tab>
<Tab title="Code">
Parses source files with `CodeParser` and splits at AST-level boundaries:
```python
from semantica.parse import CodeParser
from semantica.split import TextSplitter
parser = CodeParser(extract_comments=True, extract_dependencies=True)
parsed = parser.parse("src/pipeline.py")
splitter = TextSplitter(
method="code",
code_units=["function", "class"], # "function" | "class" | "method" | "block"
chunk_overlap=0, # code units are self-contained
)
chunks = splitter.split_documents([parsed])
for chunk in chunks:
print(f"{chunk.metadata['unit_type']}: {chunk.metadata['unit_name']}")
print(f" Lines {chunk.start_char}–{chunk.end_char} ({chunk.token_count} tokens)")
```
**Key behaviours:**
- Requires a `ParsedDocument` from `CodeParser` — use `split_documents([parsed])`
- `chunk_overlap=0` recommended — functions and classes are logically self-contained
- If a class is too large, it is split at method boundaries automatically
- Supported languages: Python, JavaScript, TypeScript, Java, Go, Rust, C, C++, C#, Ruby, PHP, Swift
</Tab>
<Tab title="Structural & Markdown">
### Structural
Uses `ParsedDocument.sections` as natural split points — each document section becomes one chunk:
```python
from semantica.parse import DoclingParser
from semantica.split import TextSplitter
parser = DoclingParser(extract_tables=True)
parsed = parser.parse("annual_report.pdf")
splitter = TextSplitter(method="structural")
chunks = splitter.split_documents([parsed])
for chunk in chunks:
level = chunk.metadata['heading_level']
title = chunk.metadata['section_title']
print(f"{' ' * (level - 1)}[h{level}] {title} ({chunk.token_count} tokens)")
```
### Markdown
Splits at Markdown heading boundaries, configurable to specific heading levels:
```python
splitter = TextSplitter(
method="markdown",
heading_levels=[1, 2], # split at # and ## only; ### stays inline
chunk_size=800,
)
chunks = splitter.split(markdown_text)
```
</Tab>
</Tabs>
## Chunk Schema
<AccordionGroup>
<Accordion title="Chunk dataclass">
```python
@dataclass
class Chunk:
text: str # chunk text content
start_char: int # character offset in source document
end_char: int # character offset in source document
token_count: int # number of tokens
metadata: Dict # source_id, chunk_index, section_title, page_number, etc.
entities: List[Dict] # entities in chunk (entity_aware splitting only)
text: str # the chunk's text content
start_char: int # character offset of start in source document
end_char: int # character offset of end in source document
token_count: int # number of tokens (via configured tokenizer)
metadata: Dict # see metadata fields below
entities: List[Dict] # entity spans fully contained in this chunk
relationships: List[Dict] # relation triplets fully contained in this chunk
```
</Accordion>
<Accordion title="Chunk metadata fields">
| Field | Type | When Present | Description |
| ----- | ---- | ------------ | ----------- |
| `source_id` | `str` | Always | ID of the source `ParsedDocument` |
| `chunk_index` | `int` | Always | Zero-based position within the document |
| `total_chunks` | `int` | Always | Total chunks produced for this document |
| `method` | `str` | Always | Splitting method that produced this chunk |
| `section_title` | `str` | `structural`, `markdown` | Heading text of the containing section |
| `heading_level` | `int` | `structural`, `markdown` | Depth: 1 = h1, 2 = h2, … |
| `page_number` | `int` | `structural` (DoclingParser) | Source page number in PDF/DOCX |
| `unit_type` | `str` | `code` | `"function"` / `"class"` / `"method"` |
| `unit_name` | `str` | `code` | Name of the code unit, e.g. `"process_batch"` |
| `language` | `str` | `sentence`, `recursive` | ISO 639-1 code for detected text language |
| `similarity_score` | `float` | `semantic_transformer` | Cosine similarity to the adjacent chunk |
</Accordion>
</AccordionGroup>
## Tokenizer Options
| Tokenizer | Models |
| --------- | ------ |
| `cl100k_base` | GPT-4, GPT-3.5-turbo, text-embedding-ada-002 |
| `p50k_base` | GPT-3 (`text-davinci-003`), Codex |
| `r50k_base` | GPT-3 (`davinci`) |
## Pipeline Integration
```python
from semantica.pipeline import Pipeline
from semantica.ingest import FileIngestor
from semantica.parse import DocumentParser
from semantica.split import TextSplitter
from semantica.semantic_extract import NERExtractor
from semantica.llms import Groq
import os
llm = Groq(model="llama-3.3-70b-versatile", api_key=os.getenv("GROQ_API_KEY"))
pipeline = Pipeline()
pipeline.add_step("split", TextSplitter(method="semantic_transformer", chunk_size=512))
result = pipeline.run(documents)
pipeline.add_step("ingest", FileIngestor())
pipeline.add_step("parse", DocumentParser())
pipeline.add_step("split", TextSplitter(method="semantic_transformer", chunk_size=512))
pipeline.add_step("extract", NERExtractor(method="llm", llm_provider=llm))
result = pipeline.run("data/reports/")
```
## Tips and Common Pitfalls
<Warning>
**`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum — for `chunk_size=1000`, set `chunk_overlap=100` to `200`.
</Warning>
<Warning>
**Wrong tokenizer.** If you use `cl100k_base` (GPT-4) but send chunks to a model with a different vocabulary, your token counts will be wrong. Match the tokenizer to your target model.
</Warning>
<Tip>
**Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting — use `recursive` instead.
</Tip>
<Tip>
**Code units too coarse.** `code_units=["class"]` on a large codebase produces chunks too big to embed well. Use `["function", "method"]` for more granular, independently useful units.
</Tip>
<Tip>
**Set `min_chunk_size` to avoid fragment chunks.** `min_chunk_size=0` (default) can produce many tiny trailing chunks. Set to ~30–50 tokens to discard fragments that carry no retrieval value.
</Tip>
<CardGroup cols={2}>
<Card title="Parse" icon="file-lines" href="parse">
Parse documents before chunking.
Parse documents before chunking — produces sections and metadata.
</Card>
<Card title="Embeddings" icon="vector-square" href="embeddings">
Embed chunks for vector search and semantic chunking.
</Card>
<Card title="Semantic Extract" icon="magnifying-glass" href="semantic_extract">
Extract entities from individual chunks.
Extract entities and relations from individual chunks.
</Card>
<Card title="Pipeline" icon="gear" href="pipeline">
Integrate splitting as a pipeline step.
Integrate splitting as a named pipeline step.
</Card>
</CardGroup>
docs/reference/triplet_store.md
+225 -64
View File
@@ -6,33 +6,196 @@ icon: "table"
`semantica.triplet_store` provides W3C-standard RDF storage with full SPARQL query support. Use it when you need semantic web compatibility, OWL reasoning, SPARQL-based queries, or standards-compliant RDF serialization.
## Exported Classes
```python
from semantica.triplet_store import (
# Core interface
TripletStore, # unified: add_triplet, get_triplets, execute_query, bulk_load
QueryEngine, # SPARQL execution: execute_query, optimize_query, plan_query
BulkLoader, # high-volume loading with progress tracking and transaction support
# Backend stores
BlazegraphStore, # Blazegraph REST API (HTTP/HTTPS, Named Graphs, SPARQL 1.1)
JenaStore, # Apache Jena Fuseki (SPARQL 1.1, TDB2, GeoSPARQL)
RDF4JStore, # Eclipse RDF4J (SailRepository, in-memory or native)
# Convenience functions
add_triplet, # add_triplet(subject, predicate, obj)
add_triplets, # bulk: add_triplets(triplets)
get_triplets, # get_triplets(subject=None, predicate=None, obj=None)
delete_triplet, # delete_triplet(subject, predicate, obj)
execute_query, # execute_query(sparql, result_format="json")
optimize_query, # optimize_query(sparql) -> optimized SPARQL string
bulk_load, # bulk_load(file_path, format="turtle")
validate_triplets,# validate_triplets(triplets) -> ValidationResult
)
```
## What You Get
- **`TripletStore`** — unified interface for all RDF backends
- **Backends** — Blazegraph, Apache Jena (Fuseki), RDF4J
- **SPARQL** — full SELECT, CONSTRUCT, ASK, and UPDATE query support
- **Bulk loading** — efficient batch import for large triple sets
- **Import / Export** — Turtle, JSON-LD, N-Triples, RDF/XML
<CardGroup cols={2}>
<Card title="TripletStore" icon="server">
Unified interface across Blazegraph, Apache Jena (Fuseki), and RDF4J — swap backends with one parameter.
</Card>
<Card title="TripletStore (in-memory)" icon="bolt">
Zero-setup in-memory mode via `backend="memory"` for unit tests and small datasets — no server required.
</Card>
<Card title="SPARQL" icon="magnifying-glass">
Full SELECT, CONSTRUCT, ASK, and UPDATE query support with pagination for large result sets.
</Card>
<Card title="OWL Reasoning" icon="microchip">
Apache Jena supports OWL and RDFS inference natively — subclass and property chain queries automatically resolved.
</Card>
<Card title="Named Graphs" icon="diagram-project">
Isolate triples by source, dataset, or time period using named graph management.
</Card>
<Card title="Import / Export" icon="file-export">
Load and serialize to Turtle, JSON-LD, N-Triples, and RDF/XML with a single method call.
</Card>
</CardGroup>
## Basic Usage
## Quick Start
<Steps>
<Step title="Connect to a backend">
```python
from semantica.triplet_store import TripletStore
store = TripletStore(
backend="blazegraph",
endpoint="http://localhost:9999/blazegraph/sparql"
)
```
</Step>
<Step title="Add triplets">
```python
# Add a single triplet
store.add_triplet(
subject="http://example.org/apple_inc",
predicate="http://example.org/founded_by",
obj="http://example.org/steve_jobs"
)
# Bulk load a list of triplets
store.add_triplets_bulk(triplets)
```
</Step>
<Step title="Query with SPARQL">
```python
results = store.sparql("""
PREFIX ex: <http://example.org/>
SELECT ?person ?company WHERE {
?person ex:founded ?company .
?company ex:located_in ex:SiliconValley .
}
""")
for row in results:
print(row["person"], row["company"])
```
</Step>
<Step title="Export to file">
```python
store.export("output.ttl", format="turtle")
store.export("output.nt", format="nt")
store.export("output.xml", format="xml")
```
</Step>
</Steps>
## Backends
<Tabs>
<Tab title="Blazegraph">
```python
from semantica.triplet_store import TripletStore
store = TripletStore(
backend="blazegraph",
endpoint="http://localhost:9999/blazegraph/sparql",
namespace="semantica"
)
```
Best for: Wikidata-style workloads, high triple counts, SPARQL 1.1 full support.
</Tab>
<Tab title="Apache Jena">
```python
store = TripletStore(
backend="jena",
endpoint="http://localhost:3030/dataset/sparql",
update_endpoint="http://localhost:3030/dataset/update"
)
```
Best for: General RDF, standard SPARQL, production deployments needing OWL inference.
**Enable OWL reasoning:**
```python
store = TripletStore(
backend="jena",
endpoint="http://localhost:3030/dataset/sparql",
update_endpoint="http://localhost:3030/dataset/update",
reasoner="OWL", # "OWL" | "RDFS" | "OWL_MINI" | None
)
# Load an OWL ontology — subclass/property chain inferences are automatic
store.import_file("ontology.ttl", format="turtle")
store.add_triplets_bulk(data_triplets)
# Query using inferred relationships
results = store.sparql("""
SELECT ?person WHERE {
?person a ex:Employee . # inferred via subClassOf chain
}
""")
```
</Tab>
<Tab title="RDF4J">
```python
store = TripletStore(
backend="rdf4j",
server_url="http://localhost:8080/rdf4j-server",
repository_id="semantica"
)
```
Best for: Enterprise Java ecosystems, Eclipse Foundation deployments, plugin-based reasoning.
</Tab>
<Tab title="Backend Comparison">
| Backend | License | OWL Reasoning | Hosted Option | Best For |
| ------- | ------- | ------------- | ------------- | -------- |
| Blazegraph | Open source | No | Self-hosted | Wikidata-style workloads, high triple count |
| Apache Jena | Apache 2.0 | Yes (OWL/RDFS) | Self-hosted | General RDF, OWL reasoning, standard SPARQL |
| RDF4J | Eclipse 1.0 | Via plugin | Self-hosted or cloud | Enterprise Java ecosystems |
| InMemory | Built-in | No | N/A | Unit tests, small graphs, no server required |
</Tab>
</Tabs>
## Namespace Prefix Management
Register custom prefixes to keep SPARQL queries readable:
```python
from semantica.triplet_store import TripletStore
from semantica.ontology import NamespaceManager
store = TripletStore(
backend="blazegraph",
endpoint="http://localhost:9999/blazegraph/sparql"
)
ns = NamespaceManager(base_uri="http://example.org/")
ns.register("ex", "http://example.org/")
ns.register("schema", "https://schema.org/")
ns.register("owl", "http://www.w3.org/2002/07/owl#")
# Add a single triplet
store.add_triplet(
subject="http://example.org/apple_inc",
predicate="http://example.org/founded_by",
obj="http://example.org/steve_jobs"
)
store = TripletStore(backend="jena", endpoint="...")
# Bulk load a list of triplets
store.add_triplets_bulk(triplets)
# Registered prefixes are automatically prepended to every SPARQL query
results = store.sparql("""
SELECT ?company WHERE {
?person ex:works_for ?company ;
schema:name "Alice" .
}
""")
```
## SPARQL Queries
@@ -47,9 +210,6 @@ results = store.sparql("""
}
""")
for row in results:
print(row["person"], row["company"])
# CONSTRUCT — returns a graph of matched triples
graph = store.sparql_construct("""
PREFIX ex: <http://example.org/>
@@ -76,54 +236,29 @@ store.sparql_update("""
""")
```
## Backends
## SPARQL Result Pagination
For large result sets, paginate with LIMIT and OFFSET:
```python
# Blazegraph — open source, SPARQL 1.1
store = TripletStore(
backend="blazegraph",
endpoint="http://localhost:9999/blazegraph/sparql",
namespace="semantica"
)
page_size = 1000
offset = 0
# Apache Jena Fuseki — open source, widely used
store = TripletStore(
backend="jena",
endpoint="http://localhost:3030/dataset/sparql",
update_endpoint="http://localhost:3030/dataset/update"
)
# RDF4J — enterprise-grade, Eclipse Foundation
store = TripletStore(
backend="rdf4j",
server_url="http://localhost:8080/rdf4j-server",
repository_id="semantica"
)
while True:
results = store.sparql(f"""
SELECT ?s ?p ?o WHERE {{
?s ?p ?o .
}}
ORDER BY ?s
LIMIT {page_size} OFFSET {offset}
""")
if not results:
break
process_batch(results)
offset += page_size
```
## Backend Comparison
| Backend | License | Query Language | Best For |
| ------- | ------- | -------------- | -------- |
| Blazegraph | Open source | SPARQL 1.1 | Wikidata-style workloads |
| Apache Jena | Apache 2.0 | SPARQL 1.1 | General RDF, OWL reasoning |
| RDF4J | Eclipse 1.0 | SPARQL 1.1 | Enterprise, Java ecosystems |
## Import and Export
```python
# Import from file
store.import_file("ontology.ttl", format="turtle")
store.import_file("data.jsonld", format="json-ld")
store.import_file("triples.nt", format="nt")
# Export to file
store.export("output.ttl", format="turtle")
store.export("output.nt", format="nt")
store.export("output.xml", format="xml")
```
## Graph Management
## Named Graph Management
```python
# Named graphs — store triples in isolated contexts
@@ -168,6 +303,32 @@ store.import_file("output.ttl", format="turtle")
results = store.sparql("SELECT * WHERE { ?s ?p ?o } LIMIT 10")
```
## Tips and Common Pitfalls
<Tip>
**Use Apache Jena (Fuseki) for development and Blazegraph for production.** Jena runs with a single Docker command, supports OWL reasoning natively, and requires no licence. Switch to Blazegraph for high-throughput workloads by changing the `backend=` parameter — no other code changes needed.
</Tip>
<Warning>
**Paginate large SPARQL result sets.** A `SELECT * WHERE { ?s ?p ?o }` against a million-triple store can return gigabytes of data. Always include `LIMIT` and `OFFSET` in exploratory queries, and iterate with `page_size` when you need full coverage. Unbounded queries against large stores will OOM or timeout.
</Warning>
<Tip>
**Use named graphs to isolate sources.** `store.add_triplet(..., graph="http://example.org/source_A")` puts triples into a named graph. You can then query just that source, merge selectively, or clear it without touching other data — far safer than mixing all triples into the default graph.
</Tip>
<Tip>
**Register namespace prefixes before querying.** `NamespacePrefixManager` lets you write `?s ex:name ?o` instead of `?s <http://example.org/name> ?o`. Without prefixes, SPARQL queries against domain ontologies become unreadable and error-prone.
</Tip>
<Warning>
**Enable OWL reasoning only when you need it.** `reasoner="OWL"` significantly increases query planning overhead. For simple triple lookups or SPARQL SELECT queries, leave reasoning off (`reasoner=None`) and enable it only for queries that depend on class hierarchies or property chains.
</Warning>
<Tip>
**Export to Turtle before migrating backends.** If you need to move from Jena to Blazegraph (or any other store), `store.export("dump.ttl", format="turtle")` produces a portable file that any SPARQL store can import. Don't rely on backend-specific dump formats.
</Tip>
<CardGroup cols={2}>
<Card title="Export" icon="file-export" href="export">
Export knowledge graphs to RDF formats.
docs/reference/utils.md
+143 -28
View File
@@ -6,35 +6,88 @@ icon: "wrench"
`semantica.utils` provides shared infrastructure used throughout Semantica. Most users won't call it directly, but its APIs are available when you need fine-grained control over logging, validation, progress tracking, or error handling.
## Exported Classes
```python
from semantica.utils import (
# Logging
setup_logging, # configure root logger — level, format (json/text)
get_logger, # get a named logger instance
log_performance, # @decorator — logs function name, duration, exception
# Validation
validate_entity, # validate entity dict structure, raises ValidationError
validate_config, # validate config dict against schema, raises ValidationError
# Progress tracking
ProgressTracker, # class-based tracker with ETA
track_progress, # wraps any iterable with live progress bar
# Helpers
clean_text, # normalize whitespace, strip control characters
hash_data, # deterministic SHA-256 hash of any serializable object
safe_filename, # sanitize a string for use as a filename
# Exceptions
SemanticaError, # base exception for all Semantica errors
ValidationError, # raised when input fails validation
ProcessingError, # raised during extraction, graph build, or pipeline step
)
```
## What You Get
- **Logging** — structured logging with `@log_performance` decorator and quality metrics
- **Validation** — `validate_entity`, `validate_config` with a typed `ValidationError`
- **Progress tracking** — `track_progress` wraps any iterable with console, Jupyter, or file output
- **Helper functions** — `clean_text`, `hash_data`, `safe_filename`
- **Exception hierarchy** — `SemanticaError` → `ValidationError`, `ProcessingError`
<CardGroup cols={2}>
<Card title="Logging" icon="scroll">
Structured logging with `@log_performance` decorator and quality metrics via environment variables.
</Card>
<Card title="Validation" icon="shield-check">
`validate_entity` and `validate_config` with a typed `ValidationError` carrying field and value context.
</Card>
<Card title="Progress Tracking" icon="bars-progress">
`track_progress` wraps any iterable — auto-detects console vs Jupyter for the right renderer.
</Card>
<Card title="Helper Functions" icon="wrench">
`clean_text`, `hash_data`, `safe_filename`, and nested dict utilities used throughout the framework.
</Card>
<Card title="Exception Hierarchy" icon="triangle-exclamation">
`SemanticaError` → `ValidationError`, `ProcessingError` — typed exceptions for targeted recovery.
</Card>
<Card title="File Utilities" icon="file">
`read_json_file` with `ProcessingError` on failure — no boilerplate try/except around JSON I/O.
</Card>
</CardGroup>
## Logging
```python
from semantica.utils import setup_logging, get_logger, log_performance
<Steps>
<Step title="Initialize logging at application startup">
```python
from semantica.utils import setup_logging, get_logger
setup_logging(level="INFO") # "DEBUG" | "INFO" | "WARNING" | "ERROR"
logger = get_logger(__name__)
setup_logging(level="INFO") # "DEBUG" | "INFO" | "WARNING" | "ERROR"
logger = get_logger(__name__)
```
</Step>
<Step title="Instrument expensive functions with the performance decorator">
```python
from semantica.utils import log_performance, log_execution_time
@log_performance
def process_data(data):
logger.info(f"Processing {len(data)} items")
# Decorator automatically logs function name, duration, and any exception
```
@log_performance
def process_data(data):
logger.info(f"Processing {len(data)} items")
# Logs function name, duration, and any exception automatically
Configure via environment variables:
```bash
export SEMANTICA_LOG_LEVEL=DEBUG
export SEMANTICA_LOG_FORMAT=json # "json" | "text"
export SEMANTICA_PROGRESS_BAR=true
```
@log_execution_time
def expensive_step(data):
...
# Logs: "expensive_step completed in 2.34s"
```
</Step>
<Step title="Configure via environment variables">
```bash
export SEMANTICA_LOG_LEVEL=DEBUG
export SEMANTICA_LOG_FORMAT=json # "json" | "text"
export SEMANTICA_PROGRESS_BAR=true
```
</Step>
</Steps>
## Validation
@@ -72,9 +125,8 @@ for item in track_progress(items, desc="Processing documents"):
```
Supports:
- **Console** — tqdm progress bar with ETA
- **Jupyter** — notebook-compatible widget
- **Jupyter** — notebook-compatible widget (auto-detected)
- **File** — write progress to a log file
## Helper Functions
@@ -83,17 +135,46 @@ Supports:
from semantica.utils import clean_text, hash_data, safe_filename
# Normalize whitespace and strip control characters
clean = clean_text(" Hello World ") # → "Hello World"
clean = clean_text(" Hello World ") # → "Hello World"
# Deterministic SHA-256 hash of any serializable object
uid = hash_data({"key": "value"}) # → hex digest string
# Deterministic SHA-256 hash of any JSON-serializable object
uid = hash_data({"key": "value"}) # → hex digest string
# Sanitize a string for use as a filename
fname = safe_filename("My File?.txt") # → "My_File_.txt"
fname = safe_filename("My File?.txt") # → "My_File_.txt"
```
## Nested Dict Utilities
Helper functions for deep configuration access — used extensively inside `Config` and `ConfigManager`:
```python
from semantica.utils import get_nested_value, set_nested_value, merge_dicts
config = {
"processing": {"batch_size": 32, "max_workers": 4},
"llm": {"provider": "groq", "model": "llama-3.3-70b-versatile"},
}
# Dot-notation read — returns default if key path is absent
batch = get_nested_value(config, "processing.batch_size", default=16)
# → 32
# Dot-notation write
set_nested_value(config, "processing.batch_size", 64)
# Deep merge — nested keys are merged recursively
base = {"a": {"x": 1, "y": 2}, "b": 3}
overrides = {"a": {"y": 99, "z": 4}, "c": 5}
merged = merge_dicts(base, overrides, deep=True)
# → {"a": {"x": 1, "y": 99, "z": 4}, "b": 3, "c": 5}
```
## Exception Hierarchy
<AccordionGroup>
<Accordion title="Exception types and when they're raised">
```python
from semantica.utils import SemanticaError, ValidationError, ProcessingError
@@ -101,7 +182,7 @@ try:
run_pipeline(data)
except ValidationError as e:
# Input data did not pass schema validation
logger.error(f"Validation failed: {e}")
logger.error(f"Validation failed at field '{e.field}': {e.message}")
except ProcessingError as e:
# Failure during extraction or graph construction
logger.error(f"Processing failed at step {e.step}: {e}")
@@ -116,6 +197,40 @@ except SemanticaError as e:
| `ValidationError` | Input data failed schema or type validation |
| `ProcessingError` | Failure during extraction, graph build, or pipeline step |
</Accordion>
</AccordionGroup>
## File Utilities
```python
from semantica.utils import read_json_file
# Read and parse a JSON file — raises ProcessingError on failure
config = read_json_file("config.json")
```
## Tips and Common Pitfalls
<Warning>
**Call `setup_logging(level="INFO")` once at application startup.** Without it, Semantica falls back to Python's root logger, which may be silent or misconfigured. Call it before importing other Semantica modules to capture initialization messages.
</Warning>
<Tip>
**Use `@log_performance` on expensive functions.** The decorator logs function name, duration, and any raised exception automatically — no manual `time.time()` bookkeeping needed. Essential for profiling multi-step pipelines where one step is a hidden bottleneck.
</Tip>
<Tip>
**`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string — suitable as a cache key or idempotency token in pipeline steps.
</Tip>
<Tip>
**Catch `SemanticaError` as the broadest exception net.** All framework errors inherit from `SemanticaError`, so `except SemanticaError` catches validation failures, processing errors, and everything in between. Use specific subclasses for targeted recovery logic.
</Tip>
<Tip>
**`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment — the same call works in both.
</Tip>
<CardGroup cols={2}>
<Card title="Core" icon="gear" href="core">
Framework orchestration that uses Utils internally.
docs/reference/vector_store.md
+320 -61
View File
@@ -6,36 +6,118 @@ icon: "database"
`semantica.vector_store` provides a unified API for storing and searching vector embeddings across all major backends. Swap backends with a one-line change — no application code changes needed.
## What You Get
- **`VectorStore`** — unified interface across all backends
- **Backends** — FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector, in-memory
- **Hybrid search** — combine dense vector similarity with sparse keyword/metadata filtering
- **Metadata filtering** — rich filter expressions: `eq`, `ne`, `gt`, `lt`, `in`, `contains`, `$and`, `$or`
- **Namespace isolation** — multi-tenant support via isolated namespaces
- **Batch operations** — bulk add, delete, and metadata updates
## Basic Usage
## Exported Classes
```python
from semantica.vector_store import VectorStore
# In-memory (development)
store = VectorStore(backend="inmemory", dimension=768)
# FAISS (local, production)
store = VectorStore(backend="faiss", dimension=768, index_path="store.faiss")
# Add vectors
store.add_vectors(embeddings=embeddings, ids=["doc1", "doc2"], metadata=[{}, {}])
# Semantic search
results = store.search(query_vector, top_k=10)
for r in results:
print(f"{r['id']} — score: {r['score']:.3f}")
print(f" metadata: {r['metadata']}")
from semantica.vector_store import (
# Core interface
VectorStore, # unified: store_vectors, search_vectors, update_vectors, delete_vectors
VectorIndexer, # build/rebuild FAISS/ANN indices
VectorRetriever, # kNN and hybrid search
VectorManager, # store management and CRUD operations
# Backend stores
FAISSStore, # local disk / in-memory (Flat, IVF, HNSW, PQ index types)
WeaviateStore, # cloud/self-hosted, schema-aware, GraphQL queries
QdrantStore, # cloud/self-hosted, payload filtering
MilvusStore, # highly scalable, partitioning and complex queries
PineconeStore, # managed cloud vector database
PgVectorStore, # PostgreSQL with pgvector extension
# Hybrid & metadata search
HybridSearch, # fuse vector + metadata results (RRF or weighted average)
MetadataFilter, # MetadataFilter().eq("category", "science").gt("year", 2020)
SearchRanker, # configurable re-ranking after fusion
MetadataStore, # inverted index for fast metadata filtering
NamespaceManager, # multi-tenant namespace isolation
# Decision-specific helpers
DecisionEmbeddingPipeline, # end-to-end: record + embed + store + retrieve
quick_decision, # quick_decision(text, entities, outcome) — shorthand record
find_precedents, # find_precedents(scenario, k=5) — similarity search
# Convenience functions
store_vectors, # store_vectors(vectors, metadata)
search_vectors, # search_vectors(query_vector, k=10)
hybrid_search, # hybrid_search(query_vector, filter=...)
update_vectors, # update_vectors(ids, new_vectors)
delete_vectors, # delete_vectors(ids)
create_index, # create_index(index_type="hnsw", dimension=768)
)
```
## What You Get
<CardGroup cols={2}>
<Card title="VectorStore" icon="database">
Unified interface across FAISS, Pinecone, Weaviate, Qdrant, Milvus, and PgVector.
</Card>
<Card title="HybridSearch" icon="magnifying-glass">
Combine dense vector similarity with sparse keyword/BM25 filtering and configurable fusion strategies.
</Card>
<Card title="MetadataStore" icon="table">
Rich metadata indexing and schema management — query by field values without a vector.
</Card>
<Card title="NamespaceManager" icon="folder-tree">
Multi-tenant namespace isolation — structural separation, not just metadata filters.
</Card>
<Card title="Batch Operations" icon="layer-group">
Bulk add, delete, and metadata updates — automatically chunked for memory efficiency.
</Card>
<Card title="FAISS Index Types" icon="chart-scatter">
Flat, IVF, HNSW, and PQ index types with full configuration control.
</Card>
</CardGroup>
## Quick Start
<Steps>
<Step title="Create a vector store">
```python
from semantica.vector_store import VectorStore
# In-memory (development)
store = VectorStore(backend="inmemory", dimension=768)
# FAISS (local production — persists to disk)
store = VectorStore(backend="faiss", dimension=768, index_path="store.faiss")
```
</Step>
<Step title="Add vectors">
```python
# Add text documents (auto-embedded)
ids = store.add_documents(
documents=["text one", "text two"],
metadata=[{"title": "Document 1"}, {"title": "Document 2"}]
)
# Add pre-computed vectors
ids = store.store_vectors(
vectors=[embedding1, embedding2],
metadata=[{"title": "Document 1"}, {"title": "Document 2"}]
)
```
</Step>
<Step title="Search by semantic similarity">
```python
# Search by text query (auto-embeds the query)
results = store.search("machine learning", limit=10)
# Search by pre-computed vector
results = store.search_vectors(query_vector, k=10)
for r in results:
print(f"{r['id']} — score: {r['score']:.3f}")
```
</Step>
<Step title="Filter results by metadata">
```python
from semantica.vector_store import HybridSearch, MetadataFilter
mf = MetadataFilter().eq("category", "research").gt("year", 2022)
search = HybridSearch(vector_store=store)
results = search.search(query=query_vector, k=10, metadata_filter=mf)
```
</Step>
</Steps>
## Backends
<Tabs>
@@ -45,7 +127,7 @@ for r in results:
store = VectorStore(
backend="faiss",
dimension=768,
index_type="IVF", # "Flat" | "IVF" | "HNSW"
index_type="IVF", # "Flat" | "IVF" | "HNSW" | "PQ"
index_path="store.faiss"
)
```
@@ -124,65 +206,96 @@ See the [PgVector Guide](../vector_stores/pgvector) for full setup.
## Hybrid Search
Combine vector similarity with keyword/metadata filters for higher precision:
Use `HybridSearch` with a `MetadataFilter` to combine vector similarity with metadata conditions:
```python
results = store.hybrid_search(
query_vector=query_embedding,
query_text="machine learning", # keyword component
top_k=10,
alpha=0.7, # 0.0 = keyword only, 1.0 = vector only
filters={"category": "research", "year": {"$gte": 2022}}
from semantica.vector_store import HybridSearch, MetadataFilter
mf = (
MetadataFilter()
.eq("category", "research")
.gt("year", 2022)
)
search = HybridSearch(vector_store=store)
results = search.search(
query=query_vector, # np.ndarray or query string (auto-embedded)
k=10,
metadata_filter=mf
)
for r in results:
print(f"{r['id']} — score: {r['score']:.3f} metadata: {r['metadata']}")
```
## Metadata Filtering
`MetadataFilter` supports chained conditions — all conditions are ANDed:
```python
# Equality
results = store.search(query_vector, filters={"author": "John Smith"})
from semantica.vector_store import MetadataFilter
# Range
results = store.search(query_vector, filters={"date": {"$gte": "2023-01-01"}})
mf = MetadataFilter().eq("author", "John Smith") # equality
mf = MetadataFilter().ne("status", "archived") # not equal
mf = MetadataFilter().gt("year", 2022).lte("year", 2024) # range
mf = MetadataFilter().in_list("tag", ["ai", "ml"]) # set membership
mf = MetadataFilter().contains("title", "neural") # substring / list contains
# Set membership
results = store.search(query_vector, filters={"tag": {"$in": ["ai", "ml"]}})
# Compound AND
results = store.search(query_vector, filters={
"$and": [
{"category": "research"},
{"year": {"$gte": 2022}}
]
})
# Multiple conditions — all must match (AND)
mf = (
MetadataFilter()
.eq("category", "research")
.gt("year", 2022)
.contains("title", "language model")
)
```
## Namespace Isolation
Isolate vectors per tenant, project, or use case:
Use `NamespaceManager` to assign vectors to named namespaces for multi-tenant isolation:
```python
store = VectorStore(backend="faiss", dimension=768)
from semantica.vector_store import NamespaceManager, VectorStore
# Write to separate namespaces
store.add_vectors(embeddings_a, ids_a, namespace="tenant_a")
store.add_vectors(embeddings_b, ids_b, namespace="tenant_b")
store = VectorStore(backend="faiss", dimension=768)
ns_manager = NamespaceManager()
# Search is scoped to the specified namespace
results = store.search(query_vector, namespace="tenant_a")
ns_manager.create_namespace("tenant_a", description="Customer A data")
ns_manager.create_namespace("tenant_b", description="Customer B data")
# Store vectors, then assign them to a namespace
ids_a = store.store_vectors(embeddings_a, metadata=metadata_a)
for vid in ids_a:
ns_manager.add_vector_to_namespace(vid, "tenant_a")
# List all namespace names
for name in ns_manager.list_namespaces():
print(name)
ns_manager.delete_namespace("tenant_a")
```
## Batch Operations
```python
# Batch add — automatically chunked for memory efficiency
store.add_vectors_batch(embeddings_list, ids_list, batch_size=1000)
# Batch add text documents — chunked automatically by batch_size
ids = store.add_documents(
documents=large_doc_list,
metadata=large_meta_list,
batch_size=1000
)
# Batch delete
store.delete_vectors(ids=["doc1", "doc2", "doc3"])
# Batch add pre-computed vectors
ids = store.store_vectors(vectors=embeddings_list, metadata=meta_list)
# Update metadata without re-embedding
store.update_metadata("doc1", {"status": "archived", "reviewed": True})
# Delete by vector ID list
store.delete_vectors(vector_ids=["vec_0", "vec_1", "vec_2"])
# Replace vectors (re-embed then update)
store.update_vectors(
vector_ids=["vec_0"],
new_vectors=[new_embedding]
)
```
## Backend Comparison
@@ -197,6 +310,152 @@ store.update_metadata("doc1", {"status": "archived", "reviewed": True})
| PgVector | PostgreSQL | No | Limited | Postgres-native integration |
| In-memory | Process | No | No | Development, testing |
## HybridSearch
`HybridSearch` combines vector similarity with metadata filtering, and can fuse results from multiple sources:
```python
from semantica.vector_store import HybridSearch, MetadataFilter, SearchRanker
# Single-source search with metadata filter
search = HybridSearch(vector_store=store)
mf = MetadataFilter().eq("category", "research").gt("year", 2022)
results = search.search(
query=query_vector, # np.ndarray or query string
k=10,
metadata_filter=mf
)
# Multi-source fusion (RRF across multiple stores)
sources = [
{"vectors": v1, "metadata": m1, "ids": ids1},
{"vectors": v2, "metadata": m2, "ids": ids2},
]
fused = search.multi_source_search(query_vector, sources, k=10)
# Custom fusion strategy
ranker = SearchRanker(strategy="reciprocal_rank_fusion") # or "weighted_average"
fused = ranker.rank([results_list_1, results_list_2], k=60)
```
| Fusion strategy | Description |
| --------------- | ----------- |
| `reciprocal_rank_fusion` | Rank-based combination via RRF constant `k=60` — robust to score scale differences |
| `weighted_average` | Weighted average of scores — pass `weights=[0.7, 0.3]` to `rank()` |
## MetadataStore
`MetadataStore` indexes structured metadata and lets you query by field values without a vector:
```python
from semantica.vector_store import MetadataStore
meta_store = MetadataStore()
# Store and retrieve metadata
meta_store.store_metadata("doc1", {"author": "Alice", "year": 2024, "category": "research"})
meta_store.store_metadata("doc2", {"author": "Bob", "year": 2023, "category": "review"})
# Query — returns List[str] of matching vector IDs
ids = meta_store.query_metadata({"category": "research", "year": 2024})
# Get and update metadata for a specific vector
meta = meta_store.get_metadata("doc1")
meta_store.update_metadata("doc1", {"score": 0.92})
```
## NamespaceManager
Assigns vector IDs to named namespaces for multi-tenant or multi-model isolation:
```python
from semantica.vector_store import NamespaceManager
ns_manager = NamespaceManager()
ns_manager.create_namespace("tenant_a", description="Customer A data")
ns_manager.create_namespace("tenant_b", description="Customer B data")
# Assign vector IDs to a namespace after storing them
for vid in ids_a:
ns_manager.add_vector_to_namespace(vid, "tenant_a")
# Inspect namespaces
for name in ns_manager.list_namespaces(): # returns List[str]
print(name)
# Look up which namespace a vector belongs to
ns = ns_manager.get_vector_namespace("vec_0")
ns_manager.delete_namespace("tenant_a")
```
## FAISS Index Type Reference
| Index | Memory | Speed | Accuracy | When to Use |
| ----- | ------ | ----- | -------- | ----------- |
| `Flat` | High | Slow | Exact (100%) | < 100K vectors, correctness critical |
| `IVF` | Medium | Fast | ~95–98% | 100K–10M vectors, good balance |
| `HNSW` | Medium-High | Very fast | ~97–99% | Low latency, production retrieval |
| `PQ` | Low | Fast | ~90–95% | Millions of vectors, memory-constrained |
```python
# Flat — brute-force exact search
store = VectorStore(backend="faiss", dimension=768, index_type="Flat")
# IVF — inverted file index with nlist clusters
store = VectorStore(backend="faiss", dimension=768, index_type="IVF", nlist=100)
# HNSW — hierarchical navigable small world graph
store = VectorStore(backend="faiss", dimension=768, index_type="HNSW", M=16, ef_construction=200)
# PQ — product quantization for memory efficiency
store = VectorStore(backend="faiss", dimension=768, index_type="PQ", m=8)
```
## Similarity Metrics
| Metric | Constructor arg | Distance → Similarity | Best For |
| ------ | --------------- | --------------------- | -------- |
| Cosine | `metric="cosine"` | `1 - cosine_distance` | Text, embeddings |
| L2 (Euclidean) | `metric="l2"` | `1 / (1 + distance)` | Image features |
| Inner Product | `metric="ip"` | raw dot product | Recommendation systems |
```python
store = VectorStore(backend="faiss", dimension=768, metric="cosine")
```
## Tips and Common Pitfalls
<Warning>
**Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size — `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time, not at store creation.
</Warning>
<Tip>
**Use `Flat` index only for small datasets.** Flat (brute-force) search has perfect recall but O(n) query time. At 500K+ vectors, switch to `IVF` or `HNSW` — they sacrifice less than 5% recall for 100–1000x speedup.
</Tip>
<Warning>
**Don't search without normalizing first.** If you disabled `normalize=True` in `EmbeddingGenerator`, compute cosine similarity with `metric="cosine"` (which normalizes internally). Raw dot product on un-normalized vectors produces incorrect similarity rankings.
</Warning>
<Tip>
**Use `hybrid_search` for precision-sensitive workloads.** Pure vector search finds semantically similar results but may miss keyword matches important to the user. Hybrid search (vector + BM25) combines both signals — especially valuable for domain-specific terminology.
</Tip>
<Tip>
**Use `NamespaceManager` for multi-tenant applications.** Storing all tenants' vectors in the same collection and filtering by metadata at query time is slow and leaks data if a filter is accidentally omitted. Namespace isolation is both faster (smaller search space) and safer (structural isolation).
</Tip>
<Warning>
**Persist FAISS indexes to disk.** `VectorStore(backend="faiss", index_path="store.faiss")` saves the index to disk on each write. Without a path, the index is in-memory only and is lost on process exit.
</Warning>
<Tip>
**Update metadata without re-embedding.** `MetadataStore.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content.
</Tip>
<CardGroup cols={2}>
<Card title="Embeddings" icon="vector-square" href="embeddings">
Generate the vectors stored here.
docs/reference/visualization.md
+246 -122
View File
@@ -6,146 +6,270 @@ icon: "chart-bar"
`semantica.visualization` renders knowledge graphs, ontologies, embedding spaces, and temporal data as interactive HTML or static images — without launching the full Explorer server.
## Exported Classes
```python
from semantica.visualization import (
# Visualizers
KGVisualizer, # visualize_network(graph), visualize_communities(graph, communities)
OntologyVisualizer, # visualize_hierarchy(ontology), visualize_structure(ontology)
EmbeddingVisualizer, # visualize_2d_projection(embeddings, labels, method="umap")
SemanticNetworkVisualizer, # visualize_network(semantic_network)
AnalyticsVisualizer, # visualize_centrality(analytics), visualize_communities(analytics)
TemporalVisualizer, # visualize_timeline(events), visualize_evolution(snapshots)
# D3Visualizer is listed in __all__ but loaded lazily (requires d3js dependency)
# Convenience functions
visualize_kg, # visualize_kg(graph, output="interactive", method="default")
visualize_ontology, # visualize_ontology(ontology, output="interactive")
visualize_embeddings, # visualize_embeddings(embeddings, labels, method="umap")
visualize_semantic_network, # visualize_semantic_network(network)
visualize_analytics, # visualize_analytics(analytics_result)
visualize_temporal, # visualize_temporal(temporal_data)
)
```
## What You Get
- **`GraphVisualizer`** — interactive HTML (PyVis) and static image (Matplotlib) graph rendering
- **`OntologyVisualizer`** — class hierarchy and property relationship visualization
- **`EmbeddingVisualizer`** — UMAP, t-SNE, and PCA dimensionality reduction plots
- **`TemporalVisualizer`** — timeline views and animated graph evolution
- **`DistanceVisualizer`** — ego-mode neighborhood views and distance matrix heatmaps (v0.5.0)
<CardGroup cols={2}>
<Card title="KGVisualizer" icon="diagram-project">
Interactive network and community graph rendering with force, hierarchical, and circular layouts.
</Card>
<Card title="OntologyVisualizer" icon="sitemap">
Class hierarchy and property relationship visualization from any ontology.
</Card>
<Card title="EmbeddingVisualizer" icon="vector-square">
UMAP, t-SNE, and PCA dimensionality reduction plots for embedding cluster analysis.
</Card>
<Card title="TemporalVisualizer" icon="clock">
Timeline views, network evolution animation, snapshot comparison, and temporal pattern highlights.
</Card>
<Card title="AnalyticsVisualizer" icon="chart-bar">
Centrality rankings, community-colored graphs, and degree distribution histograms.
</Card>
</CardGroup>
## GraphVisualizer
## Quick Start
<Steps>
<Step title="Render a knowledge graph">
```python
from semantica.visualization import KGVisualizer
viz = KGVisualizer(layout="force", color_scheme="default")
# Interactive — opens in browser, supports hover and click
viz.visualize_network(graph, output="interactive")
```
</Step>
<Step title="Apply layout and color options">
```python
viz = KGVisualizer(layout="force", color_scheme="vibrant")
viz.visualize_network(
graph,
output="html",
file_path="graph.html",
node_color_by="type", # color nodes by entity type attribute
)
```
</Step>
<Step title="Export to static formats">
```python
# Static PNG — for reports and embedding in documents
viz.visualize_network(graph, output="png", file_path="graph.png")
# Vector SVG — for publications and scalable diagrams
viz.visualize_network(graph, output="svg", file_path="graph.svg")
```
</Step>
</Steps>
## Visualizers
<Tabs>
<Tab title="KGVisualizer">
Interactive and static knowledge graph rendering:
```python
from semantica.visualization import KGVisualizer
viz = KGVisualizer(layout="force", color_scheme="default")
# Interactive — opens in browser
viz.visualize_network(graph, output="interactive")
# Save as HTML file
viz.visualize_network(graph, output="html", file_path="graph.html")
# Static PNG
viz.visualize_network(graph, output="png", file_path="graph.png")
# Community-colored graph
viz.visualize_communities(graph, communities, file_path="communities.html")
```
**Layout options (`layout=`):**
| Layout | Description | Best For |
| ------ | ----------- | -------- |
| `force` | Physics simulation — clusters emerge naturally | General graphs |
| `hierarchical` | Top-down tree layout | Taxonomies, org charts |
| `circular` | Nodes on a circle, edges as chords | Small dense graphs |
</Tab>
<Tab title="OntologyVisualizer">
Visualize class hierarchies and property relationships:
```python
from semantica.visualization import OntologyVisualizer
viz = OntologyVisualizer()
# Full ontology graph — classes, properties, and constraints
viz.visualize(ontology, output="ontology.html")
# Class hierarchy only — cleaner for large ontologies
viz.visualize_hierarchy(ontology, output="hierarchy.html")
```
</Tab>
<Tab title="EmbeddingVisualizer">
Project high-dimensional embeddings into 2D for cluster analysis:
```python
from semantica.visualization import EmbeddingVisualizer
viz = EmbeddingVisualizer()
viz.visualize_2d_projection(
embeddings=embeddings,
labels=labels,
output="interactive",
file_path="embeddings.html",
method="umap", # "umap" | "tsne" | "pca"
)
```
| Method | Speed | Preserves | Best For |
| ------ | ----- | --------- | -------- |
| `umap` | Fast | Global + local structure | Large datasets, cluster discovery |
| `tsne` | Medium | Local structure | Tight cluster separation |
| `pca` | Very fast | Variance | Quick overview, linear structure |
</Tab>
<Tab title="TemporalVisualizer">
Visualize how a knowledge graph changes over time:
```python
from semantica.visualization import TemporalVisualizer
viz = TemporalVisualizer()
# Timeline of entity/relationship changes
viz.visualize_timeline(temporal_kg, output="interactive")
# Animated network evolution — one frame per time step
viz.visualize_network_evolution(temporal_kg, output="html", file_path="evolution.html")
# Side-by-side snapshot comparison
viz.visualize_snapshot_comparison(snap_a, snap_b, output="html", file_path="diff.html")
# Recurring temporal patterns
viz.visualize_temporal_patterns(temporal_kg, output="html", file_path="patterns.html")
```
</Tab>
<Tab title="AnalyticsVisualizer">
Visualize graph analytics results — centrality, communities, and degree distribution:
```python
from semantica.visualization import AnalyticsVisualizer
from semantica.kg import CentralityCalculator, CommunityDetector
calc = CentralityCalculator()
centrality = calc.calculate_all_centrality(kg)
detector = CommunityDetector()
communities = detector.detect_communities(kg, algorithm="louvain")
viz = AnalyticsVisualizer()
# Bar chart of top-N nodes by centrality measure
viz.visualize_centrality(centrality, metric="pagerank", top_k=20, output="centrality.html")
# Community-colored graph
viz.visualize_communities(kg, communities, output="communities.html")
# Degree distribution histogram
viz.visualize_degree_distribution(kg, output="degree_dist.html")
# Combined analytics dashboard
viz.visualize_analytics_dashboard(
kg, centrality=centrality, communities=communities,
output="analytics_dashboard.html",
)
```
</Tab>
</Tabs>
## Color Schemes
All visualizers accept a `color_scheme` parameter:
```python
from semantica.visualization import GraphVisualizer
viz = GraphVisualizer()
# Interactive HTML — opens in browser, supports hover and click
viz.visualize(graph, output="graph.html")
# Static image — for reports and export
viz.visualize(graph, output="graph.png", backend="matplotlib")
# Display inline (Jupyter or default browser)
viz.show(graph)
viz.visualize(graph, output="graph.html", color_scheme="vibrant")
```
### Layout and Styling Options
```python
viz.visualize(
graph,
output="graph.html",
layout="force_directed", # "force_directed" | "hierarchical" | "circular" | "spring"
node_color_by="type", # color nodes by entity type attribute
edge_label="relation", # show edge relationship labels
max_nodes=500 # limit rendering for large graphs
)
```
### Layout Options
| Layout | Description | Best For |
| Scheme | Description | Best For |
| ------ | ----------- | -------- |
| `force_directed` | Physics simulation — clusters emerge naturally | General graphs |
| `hierarchical` | Top-down tree layout | Taxonomies, org charts |
| `circular` | Nodes on a circle, edges as chords | Small dense graphs |
| `spring` | Spring-force layout (Fruchterman-Reingold) | Medium graphs |
| `default` | Blue-grey palette | General use |
| `vibrant` | High-contrast, saturated colours | Presentations |
| `pastel` | Soft, muted tones | Light backgrounds |
| `dark` | Dark background with bright nodes | Dark-mode dashboards |
| `light` | White background, thin edges | Publications, print |
| `colorblind` | Okabe-Ito safe palette | Accessibility |
## OntologyVisualizer
## Export Formats
Visualize class hierarchies and property relationships:
```python
from semantica.visualization import OntologyVisualizer
viz = OntologyVisualizer()
# Full ontology graph
viz.visualize(ontology, output="ontology.html")
# Class hierarchy only
viz.visualize_hierarchy(ontology, output="hierarchy.html")
```
## EmbeddingVisualizer
Project high-dimensional embeddings into 2D for cluster analysis:
```python
from semantica.visualization import EmbeddingVisualizer
viz = EmbeddingVisualizer()
viz.visualize(
embeddings=embeddings,
labels=labels,
output="embeddings.html",
method="umap" # "umap" | "tsne" | "pca"
)
```
| Method | Speed | Preserves | Best For |
| ------ | ----- | --------- | -------- |
| `umap` | Fast | Global + local structure | Large datasets, cluster discovery |
| `tsne` | Medium | Local structure | Tight cluster separation |
| `pca` | Very fast | Variance | Quick overview, linear structure |
## TemporalVisualizer
Visualize how a knowledge graph changes over time:
```python
from semantica.visualization import TemporalVisualizer
viz = TemporalVisualizer()
# Static timeline of additions and removals
viz.visualize_timeline(temporal_kg, output="timeline.html")
# Animated evolution — one frame per time step
viz.animate(temporal_kg, output="evolution.html", fps=2)
```
## DistanceVisualizer (v0.5.0)
Semantic neighborhood and distance matrix visualization from Distance Intelligence:
```python
from semantica.visualization import DistanceVisualizer
viz = DistanceVisualizer()
# Ego-mode: neighborhood of one node colored by distance band
viz.visualize_ego(
graph,
center_node="Apple Inc.",
output="ego.html",
radius=0.5 # semantic distance radius
)
# N×N distance matrix heatmap
viz.visualize_distance_matrix(
matrix=distance_matrix,
labels=node_labels,
output="distance_heatmap.html"
)
```
| Format | Interactive | Scalable | Best For |
| ------ | ----------- | -------- | -------- |
| `.html` | Yes | N/A | Web dashboards, exploratory analysis |
| `.png` | No | No | Reports, Jupyter notebooks |
| `.svg` | No | Yes | Publications, slide decks |
| `.pdf` | No | Yes | Print, compliance exports |
## Graph Explorer (Full Dashboard)
For a full browser-based UI with search, path finding, and the Ontology Hub, use `semantica.explorer`:
For a full browser-based UI with search, path finding, and the Ontology Hub, launch the Explorer via the CLI:
```python
from semantica.explorer import start_explorer
start_explorer(graph=kg, port=8080)
# Opens at http://localhost:8080
```bash
semantica explore
```
See the [Explorer reference](explorer) for the full feature set and REST API.
## Tips and Common Pitfalls
<Warning>
**Use `max_nodes=500` for large graphs.** Force-directed layouts become unreadable and very slow above ~1,000 nodes. Limit with `max_nodes=500` or filter to a subgraph (e.g., top 100 nodes by PageRank) before visualizing.
</Warning>
<Tip>
**HTML output is always the best starting point.** Interactive HTML lets you zoom, pan, hover for details, and hide node types — giving you orders of magnitude more exploratory power than a static PNG. Only export to PNG/SVG/PDF when embedding in a report.
</Tip>
<Tip>
**Use `color_scheme="colorblind"` in publications and dashboards.** The Okabe-Ito palette is readable for everyone, including the ~8% of male readers who are red-green colorblind. Reserve `vibrant` for internal presentations only.
</Tip>
<Tip>
**UMAP is faster than t-SNE at scale.** For embedding spaces with >5,000 points, UMAP completes in seconds; t-SNE may take minutes. Both produce good cluster separation — use UMAP for exploratory speed, t-SNE for final publication-quality plots.
</Tip>
<Warning>
**`TemporalVisualizer.animate()` can produce large files.** Animated HTML files include all frames and can reach dozens of MB for long time series. Use `fps=1` or reduce the number of time steps for a manageable file size.
</Warning>
<Tip>
**For interactive dashboards, prefer Explorer.** `KGVisualizer.visualize_network()` generates a self-contained HTML file. The Explorer CLI (`semantica explore`) gives a full live web app with search, filtering, path-finding, and REST API. Use Explorer for team exploration, Visualizer for standalone report embeds.
</Tip>
<CardGroup cols={2}>
<Card title="Knowledge Graph" icon="diagram-project" href="kg">
The graph being visualized.