mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: premium UI improvements — navbar links, hover effects, inline tips, accordion troubleshooting (#646)

Browse Source 
- Move Discord, GitHub, PyPI, and Follow on X links from sidebar anchors to top-right navbar
- Lock dark mode as default via appearance.strict and hide theme toggle
- Add custom.css with hover highlighting for tables, code blocks, cards, callouts, and inline code
- Move all Tips and Common Pitfalls sections inline next to their relevant content across all 25 reference docs
- Polish context.md: remove duplicates, condense callouts, upgrade Cookbooks to CardGroup
- Convert Troubleshooting and Performance Optimization sections in installation.md, cli-setup.md, explorer-setup.md, learning-more.md, and faq.md from plain headers to AccordionGroup
- Change navigation-hint Tip callouts to Info in concepts.md, faq.md, glossary.md, and modules.md
This commit is contained in:
Mohd Kaif
2026-06-17 18:59:25 +05:30
committed by GitHub
parent a326c7d3bd
commit e04dc12e6e
30 changed files with 723 additions and 593 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference/change_management.md
+16 -18
View File
@@ -97,6 +97,10 @@ icon: "clock-rotate-left"
</Step>
</Steps>
<Warning>
**Snapshot before every destructive operation.** Call `manager.create_snapshot()` before running deduplication, conflict resolution, or merge operations. `restore_snapshot()` is only possible if a snapshot exists before the change.
</Warning>
## TemporalVersionManager
Version control for knowledge graphs: snapshot, diff, and rollback.
@@ -156,6 +160,10 @@ for item in diff["entities_modified"]:
print(" %s: %s -> %s" % (field, change["from"], change["to"]))
```
<Tip>
**Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` returns a plain dict with `"summary"`, `"entities_added"`, `"entities_removed"`, and `"entities_modified"`: use the `"summary"` sub-dict to get counts and `"entities_modified"` to inspect property-level changes.
</Tip>
<Accordion title="diff() return schema">
```python
@@ -237,6 +245,10 @@ print("Properties added: ", diff["properties_added"])
The default `TemporalVersionManager()` with no arguments uses in-memory storage. Always pass `storage_path="versions.db"` or an explicit `SQLiteVersionStorage` in production: otherwise your entire version history disappears on restart.
</Warning>
<Tip>
**Use `SQLiteVersionStorage` in production.** The default in-memory storage loses all version history when the process exits. Pass `storage_path="versions.db"` to `TemporalVersionManager` or create `SQLiteVersionStorage(db_path="versions.db")` explicitly.
</Tip>
## Integrity Verification
SHA-256 checksums detect any unauthorized modification to a graph between snapshots:
@@ -311,6 +323,10 @@ print("Added: %d | Removed: %d | Modified: %d" % (
s["entities_added"], s["entities_removed"], s["entities_modified"]))
```
<Tip>
**Use `list_versions()` and `diff()` for compliance reviews.** `manager.list_versions()` returns a list of metadata dicts (with `label`, `author`, `timestamp`, `checksum`). Run `verify_checksum(snapshot)` on the dict returned by `get_version()` to confirm integrity before any export.
</Tip>
Use `verify_checksum()` before any compliance export to confirm snapshot integrity:
```python
@@ -348,24 +364,6 @@ for record in history:
</Accordion>
</AccordionGroup>
## Tips and Common Pitfalls
<Tip>
**Use `SQLiteVersionStorage` in production.** The default in-memory storage loses all version history when the process exits. Pass `storage_path="versions.db"` to `TemporalVersionManager` or create `SQLiteVersionStorage(db_path="versions.db")` explicitly.
</Tip>
<Warning>
**Snapshot before every destructive operation.** Call `manager.create_snapshot()` before running deduplication, conflict resolution, or merge operations. `restore_snapshot()` is only possible if a snapshot exists before the change.
</Warning>
<Tip>
**Use `diff()` for code review and incident investigation.** `manager.diff("v1.0", "v2.0")` returns a plain dict with `"summary"`, `"entities_added"`, `"entities_removed"`, and `"entities_modified"`: use the `"summary"` sub-dict to get counts and `"entities_modified"` to inspect property-level changes.
</Tip>
<Tip>
**Use `list_versions()` and `diff()` for compliance reviews.** `manager.list_versions()` returns a list of metadata dicts (with `label`, `author`, `timestamp`, `checksum`). Run `verify_checksum(snapshot)` on the dict returned by `get_version()` to confirm integrity before any export.
</Tip>
<CardGroup cols={2}>
<Card title="Provenance" icon="link" href="provenance">
W3C PROV-O lineage tracking.
docs/reference/conflicts.md
+28 -30
View File
@@ -139,6 +139,10 @@ Semantica's conflict detection makes disagreements explicit and actionable:
</Step>
</Steps>
<Warning>
**Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder: you lose the original source attribution.
</Warning>
## ConflictDetector
```python
@@ -160,6 +164,10 @@ conflicts = detector.detect_value_conflicts(entities, "revenue")
| `LOGICAL` | Logically inconsistent property combinations | `is_alive=True` but `death_date` set |
| `RELATIONSHIP` | Inconsistent relationship properties across sources | Edge weight 0.9 vs 0.3 from two sources |
<Warning>
**`TEMPORAL` and `LOGICAL` conflict detection is not implemented on `ConflictDetector` directly.** The `ConflictType` enum includes these types for use in custom pipelines, but the detector class only implements `detect_value_conflicts`, `detect_type_conflicts`, `detect_relationship_conflicts`, and `detect_entity_conflicts`.
</Warning>
Run targeted detection by type:
```python
@@ -199,6 +207,10 @@ for result in results:
print(" Strategy: %s Confidence: %.2f" % (result.resolution_strategy, result.confidence))
```
<Tip>
**Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with `severity == "critical"` or `severity == "high"`: high severity means the disagreement is large and the stakes of getting it wrong are high.
</Tip>
### Choosing a Resolution Strategy
<Tabs>
@@ -314,6 +326,14 @@ chain = tracker.get_traceability_chain("apple_inc")
- Credibility scores default to 0.50 for any source not explicitly set
- `SourceTracker` stores property-level provenance: so you can trace exactly which source contributed each value
<Warning>
**Always set credibility scores.** The default credibility is 0.50 for all sources. Without explicit scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`. The power of this strategy is in the differentiation.
</Warning>
<Tip>
**Combine with provenance.** The `SourceTracker` feeds directly into the [Provenance](provenance) module's audit trail. If you need to explain how a resolved value was chosen, provenance records give you the full chain.
</Tip>
## ConflictAnalyzer
```python
@@ -338,6 +358,14 @@ for t in trends:
- `analyze_conflicts()["by_source"]` includes `counts` and `top_sources`: sources appearing in many conflicts may have upstream data quality problems
- `analyze_trends()` returns a list of per-period dicts (`period`, `conflict_count`, `trend`, `trend_direction`): `trend` is `"increasing"`, `"decreasing"`, or `"stable"`
<Tip>
**Use `analyze_conflicts()["by_source"]["top_sources"]` to identify bad data feeds.** A single source appearing in many conflicts is a data quality problem upstream, not a conflict to resolve record by record. Flag it and investigate the source pipeline.
</Tip>
<Tip>
**Severity is a string label, not a score.** `ConflictDetector` assigns `"critical"`, `"high"`, or `"medium"` based on property importance and value differences. Critical fields (`id`, `name`, `type`, `revenue`) always yield `"critical"`. Domain context determines what to prioritize.
</Tip>
## InvestigationGuideGenerator
Auto-generate human-readable investigation checklists for conflicts requiring manual or expert review:
@@ -436,36 +464,6 @@ class InvestigationStep:
</Accordion>
</AccordionGroup>
## Tips and Common Pitfalls
<Warning>
**Detect before you merge, not after.** Run conflict detection on raw entity data before deduplication and graph construction. Detecting conflicts in a live graph that already contains merged entities is harder: you lose the original source attribution.
</Warning>
<Warning>
**Always set credibility scores.** The default credibility is 0.50 for all sources. Without explicit scores, `CREDIBILITY_WEIGHTED` behaves identically to `VOTING`. The power of this strategy is in the differentiation.
</Warning>
<Tip>
**Don't auto-resolve everything.** Use `MANUAL_REVIEW` for conflicts with `severity == "critical"` or `severity == "high"`: high severity means the disagreement is large and the stakes of getting it wrong are high.
</Tip>
<Warning>
**`TEMPORAL` and `LOGICAL` conflict detection is not implemented on `ConflictDetector` directly.** The `ConflictType` enum includes these types for use in custom pipelines, but the detector class only implements `detect_value_conflicts`, `detect_type_conflicts`, `detect_relationship_conflicts`, and `detect_entity_conflicts`.
</Warning>
<Tip>
**Use `analyze_conflicts()["by_source"]["top_sources"]` to identify bad data feeds.** A single source appearing in many conflicts is a data quality problem upstream, not a conflict to resolve record by record. Flag it and investigate the source pipeline.
</Tip>
<Tip>
**Severity is a string label, not a score.** `ConflictDetector` assigns `"critical"`, `"high"`, or `"medium"` based on property importance and value differences. Critical fields (`id`, `name`, `type`, `revenue`) always yield `"critical"`. Domain context determines what to prioritize.
</Tip>
<Tip>
**Combine with provenance.** The `SourceTracker` feeds directly into the [Provenance](provenance) module's audit trail. If you need to explain how a resolved value was chosen, provenance records give you the full chain.
</Tip>
<CardGroup cols={2}>
<Card title="Deduplication" icon="copy" href="deduplication">
Resolve duplicate entities before conflict detection.
docs/reference/context.md
+32 -77
View File
@@ -73,41 +73,6 @@ icon: "brain"
</CardGroup>
## Getting Started
```python
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
context = AgentContext(
vector_store=VectorStore(backend="faiss", dimension=768),
knowledge_graph=ContextGraph(advanced_analytics=True),
decision_tracking=True, # requires knowledge_graph to be set
)
# Store a fact
memory_id = context.store(
"GPT-4 outperforms GPT-3.5 on reasoning benchmarks by 40%",
metadata={"source": "openai_blog", "date": "2024-01"}
)
# Retrieve by semantic similarity
results = context.retrieve("LLM benchmark comparisons", max_results=5)
for r in results:
print("{} (score: {:.3f})".format(r["content"], r["score"]))
# Record a decision
decision_id = context.record_decision(
category="model_selection",
scenario="Choose LLM for production reasoning pipeline",
reasoning="GPT-4 benchmark advantage justifies 3x cost increase",
outcome="selected_gpt4",
confidence=0.91,
entities=["gpt-4", "gpt-3.5"],
decision_maker="pipeline_agent",
)
```
## Quick Start
<Steps>
@@ -228,9 +193,6 @@ decision_id = context.record_decision(
precedents = context.find_precedents("model selection", limit=5)
```
<Note>
`decision_tracking=True` silently no-ops unless `knowledge_graph` is also provided at construction time.
</Note>
</Tab>
<Tab title="GraphRAG Query">
Load a pre-built knowledge graph and answer complex questions with multi-hop graph traversal.
@@ -323,9 +285,13 @@ decision_id = context.record_decision(
| `advanced_analytics` | `bool` | `True` | Enables PageRank, centrality, and community analysis |
| `kg_algorithms` | `bool` | `True` | Adds path-finding and link prediction |
<Note>
`decision_tracking=True` has no effect unless `knowledge_graph` is also provided. Both must be set at construction time for decision tracking to be active.
</Note>
<Tip>
**Set `retention_days` to avoid memory bloat.** The default of `30` prunes automatically. Compliance-critical agents may need `retention_days=None` with explicit archival via `export()`.
</Tip>
<Tip>
**Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore` so the FAISS index survives process restarts.
</Tip>
### Memory Methods
@@ -344,6 +310,10 @@ decision_id = context.record_decision(
| `export(conversation_id, format)` | `str \| Dict` | Export memories as JSON or dict |
| `import_data(data, format)` | `int` | Import memories from JSON or dict |
<Tip>
**`retrieve()` uses `max_results=`, not `top_k=`.** The parameter is `max_results` (default `5`). Pass `use_graph=True` to force GraphRAG or `use_graph=False` to force vector-only retrieval regardless of whether a `knowledge_graph` is configured.
</Tip>
### Conversation Methods
```python
@@ -396,6 +366,14 @@ print("Sources used: {}".format(result["num_sources"]))
| `trace_decision_explainability(decision_id)` | `Dict` | Full explainability: causes, effects, relationship paths |
| `get_policy_engine()` | `PolicyEngine` | Access the active `PolicyEngine` instance |
<Warning>
`decision_tracking=True` requires `knowledge_graph` to also be set. Without it, `record_decision()` raises `RuntimeError`.
</Warning>
<Tip>
**Use `find_precedents()` before every significant decision.** This is how the context module prevents agents from making contradictory choices across runs. Surface precedents to the LLM as context: "we chose X for similar reasons before."
</Tip>
### Checkpoint Methods
**Ideal for auditing reasoning loops**: take a snapshot before and after a pass to see exactly what changed:
@@ -515,7 +493,7 @@ print("Reachable: {}, hops: {}".format(path["reachable"], path["hop_count"]))
```
## AgentMemory (Low-Level)
## AgentMemory
For fine-grained control over memory storage and retrieval:
@@ -636,6 +614,10 @@ print("Entities:", web["statistics"]["total_entities"])
print("Links: ", web["statistics"]["total_links"])
```
<Warning>
**`EntityLinker.link_entities()` links two entity IDs, not a list.** Call `link_entities(entity1_id, entity2_id, link_type)` to create a typed edge between two known IDs. For linking entities extracted from text, use `link(text, entities=[...])` instead.
</Warning>
`LinkedEntity` fields returned by `link()`:
| Field | Type | Description |
@@ -907,37 +889,6 @@ class EntityLink:
</Tab>
</Tabs>
## Tips and Common Pitfalls
<Warning>
**`decision_tracking=True` silently does nothing without `knowledge_graph`.** Both must be set at construction. Passing only `decision_tracking=True` without a `knowledge_graph` instance leaves the decision backend uninitialised: `record_decision()` will raise `RuntimeError`.
</Warning>
<Warning>
**Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore`: without it the FAISS index lives only in memory and is lost on shutdown.
</Warning>
<Tip>
**Use `find_precedents()` before every significant decision.** This is how the context module prevents agents from making contradictory choices across runs. Surface precedents to the LLM as context: "we chose X for similar reasons before."
</Tip>
<Tip>
**`retrieve()` uses `max_results=`, not `top_k=`.** The parameter is `max_results` (default `5`). Pass `use_graph=True` to force GraphRAG or `use_graph=False` to force vector-only retrieval regardless of whether a `knowledge_graph` is configured.
</Tip>
<Tip>
**Set `retention_days` to avoid memory bloat.** The default `AgentContext.retention_days=30` prunes automatically. Compliance-critical agents may need `retention_days=None` with explicit archival via `export()`.
</Tip>
<Tip>
**Use `checkpoint()` + `diff_checkpoints()` to audit reasoning loops.** Take a snapshot before and after a reasoning pass to see exactly which decisions and relationships were added.
</Tip>
<Warning>
**`EntityLinker.link_entities()` links two entity IDs, not a list.** Call `link_entities(entity1_id, entity2_id, link_type)` to create a typed edge between two known IDs. For linking entities extracted from text, use `link(text, entities=[...])` instead.
</Warning>
<CardGroup cols={2}>
<Card title="Vector Store" icon="database" href="vector_store">
Embedding storage backend for memory retrieval.
@@ -953,7 +904,11 @@ class EntityLink:
</Card>
</CardGroup>
### Cookbooks
- [Context Module](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb): memory and decision tracking · Intermediate
- [Advanced Context Engineering](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb): production FAISS + Neo4j setup · Advanced
<CardGroup cols={2}>
<Card title="Context Module" icon="book-open" href="https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/19_Context_Module.ipynb">
Memory and decision tracking · Intermediate
</Card>
<Card title="Advanced Context Engineering" icon="flask" href="https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/11_Advanced_Context_Engineering.ipynb">
Production FAISS + Neo4j setup · Advanced
</Card>
</CardGroup>
docs/reference/deduplication.md
+24 -26
View File
@@ -87,6 +87,10 @@ for op in operations:
))
```
<Tip>
**Normalize entity names before deduplication.** Canonical forms such as `"Apple Inc."` vs `"apple inc"` may score below threshold due to case alone. Run `EntityNormalizer` or `TextNormalizer` first for reliable matching.
</Tip>
## DuplicateDetector
Find duplicate entity pairs:
@@ -124,6 +128,14 @@ new_entities = [{"id": "4", "name": "Apple Corp.", "type": "Company"}]
candidates = detector.incremental_detect(new_entities, entities)
```
<Tip>
**Tune `similarity_threshold` before `confidence_threshold`.** The similarity threshold gates which entity pairs are even considered. The confidence threshold further filters those pairs based on multi-factor scoring. Start with `similarity_threshold=0.7` and raise it to reduce false positives.
</Tip>
<Tip>
**Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters: if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity.
</Tip>
### `detect_duplicates()` detection methods
The `method=` parameter of the `detect_duplicates()` convenience function controls how
@@ -148,6 +160,10 @@ method used internally:
| `reasons` | `List[str]` | Why they are considered duplicates |
| `metadata` | `Dict` | Additional metadata |
<Warning>
**`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score`: not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`.
</Warning>
### DuplicateGroup fields
| Field | Type | Description |
@@ -188,6 +204,10 @@ history = merger.get_merge_history()
print("Total merges performed:", len(history))
```
<Warning>
**`merge_entities()` and `EntityMerger.merge_duplicates()` return `List[MergeOperation]`, not a list of entity dicts.** Access `.merged_entity` on each operation to get the merged dict.
</Warning>
### Merge strategies
Pass as a string to `strategy=` on `merge_duplicates()` or `merge_entity_group()`:
@@ -229,6 +249,10 @@ merger.merge_strategy_manager.add_property_rule(
operations = merger.merge_duplicates(entities)
```
<Warning>
**`PropertyMergeRule` is a dataclass, not an Enum.** The merge strategy Enum is `MergeStrategy` (`KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL`). Per-property rules are added via `merger.merge_strategy_manager.add_property_rule(name, strategy)`.
</Warning>
### MergeOperation fields
| Field | Type | Description |
@@ -427,32 +451,6 @@ result = calculate_similarity(entity_a, entity_b, method="drug_name")
</Tab>
</Tabs>
## Tips and Common Pitfalls
<Warning>
**`DuplicateCandidate` fields are `entity1`, `entity2`, `similarity_score`: not `entity_a`, `entity_b`, `similarity`.** Accessing the wrong field names raises `AttributeError`.
</Warning>
<Warning>
**`merge_entities()` and `EntityMerger.merge_duplicates()` return `List[MergeOperation]`, not a list of entity dicts.** Access `.merged_entity` on each operation to get the merged dict.
</Warning>
<Warning>
**`PropertyMergeRule` is a dataclass, not an Enum.** The merge strategy Enum is `MergeStrategy` (`KEEP_FIRST`, `KEEP_LAST`, `KEEP_MOST_COMPLETE`, `KEEP_HIGHEST_CONFIDENCE`, `MERGE_ALL`). Per-property rules are added via `merger.merge_strategy_manager.add_property_rule(name, strategy)`.
</Warning>
<Tip>
**Tune `similarity_threshold` before `confidence_threshold`.** The similarity threshold gates which entity pairs are even considered. The confidence threshold further filters those pairs based on multi-factor scoring. Start with `similarity_threshold=0.7` and raise it to reduce false positives.
</Tip>
<Tip>
**Use `detect_duplicate_groups()` when you need to merge.** The `"group"` detection strategy uses union-find to form transitive clusters: if A≈B and B≈C, all three land in the same group. Plain `detect_duplicates()` returns individual pairs without transitivity.
</Tip>
<Tip>
**Normalize entity names before deduplication.** Canonical forms such as `"Apple Inc."` vs `"apple inc"` may score below threshold due to case alone. Run `EntityNormalizer` or `TextNormalizer` first for reliable matching.
</Tip>
<CardGroup cols={2}>
<Card title="Conflicts" icon="triangle-exclamation" href="conflicts">
Detect value conflicts between non-duplicate entities.
docs/reference/embeddings.md
+24 -26
View File
@@ -83,6 +83,10 @@ Semantica uses embeddings for:
<Check>
Default model is `BAAI/bge-small-en-v1.5`. Zero cost, zero GPU, works on any machine.
</Check>
<Warning>
**FastEmbed ignores the `device` parameter.** FastEmbed uses ONNX Runtime and manages its own execution providers: passing `device="cuda"` has no effect. Switch to `method="sentence_transformers"` if you need explicit GPU control.
</Warning>
</Tab>
<Tab title="Sentence-Transformers">
Broad model selection via HuggingFace. Runs locally, no API key.
@@ -103,6 +107,10 @@ Semantica uses embeddings for:
```
Popular models: `all-MiniLM-L6-v2` (fast, small), `all-mpnet-base-v2` (balanced), `BAAI/bge-large-en-v1.5` (high accuracy).
<Warning>
**Sequence length limits.** Most sentence-transformers models have a 512-token limit. Text beyond that is silently truncated. Use `TextSplitter(method="hierarchical")` + `HierarchicalPooling` for long documents.
</Warning>
</Tab>
<Tab title="BGE">
BAAI/bge models via sentence-transformers. State-of-the-art retrieval performance, runs locally.
@@ -178,6 +186,10 @@ score = generator.compare_embeddings(embeddings[0], embeddings[1], method="cosin
print(f"Similarity: {score:.3f}")
```
<Tip>
**Always use the same model for indexing and querying.** Vectors from different models are not comparable: they live in different vector spaces. Switching models requires re-embedding your entire corpus.
</Tip>
To switch provider after construction:
```python
@@ -344,6 +356,14 @@ dim = embedder.get_embedding_dimension()
- If FastEmbed or sentence-transformers is unavailable, falls back to a 128-dimensional hash-based embedding. Hash embeddings are deterministic but not semantic: do not use in production.
- Large batches are chunked internally by the underlying library to avoid OOM.
<Warning>
**Dimension mismatch.** The dimension you pass to your vector store must exactly match your embedding model's output. `BAAI/bge-small-en-v1.5` → 384, `all-MiniLM-L6-v2` → 384, `all-mpnet-base-v2` → 768, `BAAI/bge-large-en-v1.5` → 1024. Check with `embedder.get_embedding_dimension()` before creating the store.
</Warning>
<Tip>
**Fallback embeddings are not semantic.** If neither FastEmbed nor sentence-transformers loads successfully, TextEmbedder silently falls back to 128-dimensional SHA-256 hash embeddings. These are deterministic but carry no semantic meaning. Check `embedder.get_method()`: if it returns `"fallback"`, install your intended provider.
</Tip>
## Provider Stores
Use provider stores directly when you need fine-grained control over a single backend:
@@ -378,6 +398,10 @@ store = ProviderStoreFactory.create(provider="bge", model_name="BAAI/bge-large-e
`LlamaStore` exists in the module but is a placeholder: it does not connect to Ollama and always raises `ProcessingError` at embed time. Do not use it in production.
</Note>
<Warning>
**LlamaStore is not functional.** `LlamaStore` exists in the module but does not connect to Ollama. It always raises `ProcessingError` at embed time. Use `FastEmbedStore` for local ONNX-based embeddings or `BGEStore` for sentence-transformers-based local embeddings instead.
</Warning>
## Pooling Strategies
Pooling aggregates a set of embeddings into a single vector: useful when you have multiple chunk embeddings to combine:
@@ -609,32 +633,6 @@ providers = check_available_providers()
# → {"sentence_transformers": True, "fastembed": True, "openai": False}
```
## Tips and Common Pitfalls
<Warning>
**Dimension mismatch.** The dimension you pass to your vector store must exactly match your embedding model's output. `BAAI/bge-small-en-v1.5` → 384, `all-MiniLM-L6-v2` → 384, `all-mpnet-base-v2` → 768, `BAAI/bge-large-en-v1.5` → 1024. Check with `embedder.get_embedding_dimension()` before creating the store.
</Warning>
<Warning>
**LlamaStore is not functional.** `LlamaStore` exists in the module but does not connect to Ollama. It always raises `ProcessingError` at embed time. Use `FastEmbedStore` for local ONNX-based embeddings or `BGEStore` for sentence-transformers-based local embeddings instead.
</Warning>
<Warning>
**Sequence length limits.** Most sentence-transformers models have a 512-token limit. Text beyond that is silently truncated. Use `TextSplitter(method="hierarchical")` + `HierarchicalPooling` for long documents.
</Warning>
<Warning>
**FastEmbed ignores the `device` parameter.** FastEmbed uses ONNX Runtime and manages its own execution providers: passing `device="cuda"` has no effect. Switch to `method="sentence_transformers"` if you need explicit GPU control.
</Warning>
<Tip>
**Always use the same model for indexing and querying.** Vectors from different models are not comparable: they live in different vector spaces. Switching models requires re-embedding your entire corpus.
</Tip>
<Tip>
**Fallback embeddings are not semantic.** If neither FastEmbed nor sentence-transformers loads successfully, TextEmbedder silently falls back to 128-dimensional SHA-256 hash embeddings. These are deterministic but carry no semantic meaning. Check `embedder.get_method()`: if it returns `"fallback"`, install your intended provider.
</Tip>
<CardGroup cols={2}>
<Card title="Vector Store" icon="database" href="vector_store">
Store and search the generated embeddings.
docs/reference/explorer.md
+20 -22
View File
@@ -103,6 +103,10 @@ The `semantica-explorer` command accepts exactly four flags:
There are no flags for authentication, CORS, or log level in the CLI. CORS allowed origins are configured via the `EXPLORER_CORS_ORIGINS` environment variable (comma-separated, default: `http://localhost:5173,http://127.0.0.1:5173`).
</Note>
<Tip>
**CORS origins are configured via environment variable.** Set `EXPLORER_CORS_ORIGINS` to a comma-separated list of allowed origins before launching (e.g. `EXPLORER_CORS_ORIGINS="http://myapp.example.com"`).
</Tip>
```bash
# Full example
EXPLORER_CORS_ORIGINS="http://myapp.example.com" \
@@ -144,6 +148,10 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \
- **Filter by entity type**: `GET /api/graph/nodes?type=Person`
- **Semantic neighborhood**: `GET /api/graph/semantic-neighborhood?node_id=&top_k=20`
- **Distance matrix**: `POST /api/graph/distance-matrix`
<Warning>
**Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph before exporting: the force-directed layout becomes unusable on very large graphs.
</Warning>
</Tab>
<Tab title="Ontology Hub">
Ontology lifecycle management in the browser:
@@ -164,6 +172,10 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \
- **Enrich: deduplication**: `POST /api/enrich/dedup`
- **Enrich: entity extraction**: `POST /api/enrich/extract`
- **Temporal**: `GET /api/temporal/snapshot`, `GET /api/temporal/diff`, `GET /api/temporal/bounds`
<Tip>
**Use `/api/analytics/validation` to check graph quality.** The validator detects orphaned nodes, missing types, and other structural issues before you expose the graph to downstream pipelines.
</Tip>
</Tab>
<Tab title="Decisions & Provenance">
Decision tracking and provenance queries:
@@ -174,6 +186,10 @@ EXPLORER_CORS_ORIGINS="http://myapp.example.com" \
- **Compliance**: `GET /api/decisions/{id}/compliance`
- **Provenance**: `GET /api/provenance?node_id=`, `GET /api/provenance/report?node_id=`
- **Annotations**: `GET/POST /api/annotations`, `DELETE /api/annotations/{id}`
<Tip>
**Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API: pipe them into scripts to automate batch annotation, SPARQL querying, or exports.
</Tip>
</Tab>
</Tabs>
@@ -352,6 +368,10 @@ WebSocket message schema:
Event types broadcast over the WebSocket include: `connection_ack`, `pong`, and `graph_mutation` (fired when nodes or edges are added/updated/removed via import or enrichment). Send the text `"ping"` to receive a `pong` response.
<Warning>
**Session state is lost on server restart.** There is no auto-save. Call `POST /api/export` with body `{"format": "json"}` to download the current state before shutting down.
</Warning>
## Performance
| Scenario | Latency |
@@ -392,28 +412,6 @@ Semantic neighborhood requires node embeddings stored in node properties (keys `
**Session state lost after restart**
Session state is in-memory only. Use `POST /api/export` to save a JSON snapshot before shutting down.
## Tips and Common Pitfalls
<Warning>
**Filter large graphs before saving to JSON.** The CLI loads the entire JSON file into memory. For graphs > 10k nodes, filter to the relevant subgraph before exporting: the force-directed layout becomes unusable on very large graphs.
</Warning>
<Warning>
**Session state is lost on server restart.** There is no auto-save. Call `POST /api/export` with body `{"format": "json"}` to download the current state before shutting down.
</Warning>
<Tip>
**Use the REST API for automation, Explorer UI for exploration.** Explorer's REST endpoints are a stable programmatic API: pipe them into scripts to automate batch annotation, SPARQL querying, or exports.
</Tip>
<Tip>
**CORS origins are configured via environment variable.** Set `EXPLORER_CORS_ORIGINS` to a comma-separated list of allowed origins before launching (e.g. `EXPLORER_CORS_ORIGINS="http://myapp.example.com"`).
</Tip>
<Tip>
**Use `/api/analytics/validation` to check graph quality.** The validator detects orphaned nodes, missing types, and other structural issues before you expose the graph to downstream pipelines.
</Tip>
<CardGroup cols={2}>
<Card title="Context" icon="brain" href="context">
Build and save the ContextGraph that Explorer loads.
docs/reference/export.md
+28 -30
View File
@@ -116,6 +116,18 @@ export_lpg(graph, "import.cypher", method="cypher")
exporter.export_knowledge_graph(graph, "output.ttl", format="turtle")
```
<Warning>
**`export_to_rdf()` returns a string: it does not write a file.** Call `export()` or `export_knowledge_graph()` to write directly to disk.
</Warning>
<Tip>
**Use `export_to_rdf()` + string for inspection, `export()` for production.** In notebooks or debug sessions, `export_to_rdf()` is handy for quick inspection. For CI pipelines and pipelines writing files, `export()` is a single call.
</Tip>
<Tip>
**Use `turtle` for human readability, `ntriples` for streaming.** Turtle is compact and readable for debugging and sharing. N-Triples (`.nt`) is line-oriented: one triple per line: making it safe to stream, concatenate, and process with standard Unix tools.
</Tip>
**Namespace management:**
```python
@@ -166,6 +178,14 @@ export_lpg(graph, "import.cypher", method="cypher")
exporter.export(graph, "output_base")
```
<Warning>
**`ParquetExporter` and `ArrowExporter` require `pyarrow`.** Both fall back to a no-op stub class if `pyarrow` is not installed. Install with `pip install pyarrow` before using these exporters.
</Warning>
<Tip>
**Use `ParquetExporter` for downstream analytics.** Parquet preserves column types (int, float, datetime) that CSV loses and is natively supported by Spark, BigQuery, Databricks, and Snowflake. Use `compression="snappy"` for a good balance of speed and compression.
</Tip>
Requires `pyarrow`: `pip install pyarrow`. Schema is explicitly typed.
```python
@@ -215,6 +235,10 @@ export_lpg(graph, "import.cypher", method="cypher")
```
Both exporters write to a file and return `None`.
<Warning>
**`ArangoAQLExporter.export()` and `LPGExporter.export()` write to a file and return `None`.** They do not return the AQL/Cypher string. Write to a file and read it back if you need the string.
</Warning>
</Tab>
<Tab title="Visualization & OWL">
```python
@@ -285,6 +309,10 @@ export_lpg(graph, "import.cypher", method="cypher")
Available `include` columns: `source_id`, `source_type`, `target_id`, `target_type`, `hop_count`, `weighted_distance`, `semantic_similarity`, `distance_band`, `source_betweenness`, `target_betweenness`.
<Warning>
**`DistanceExporter` requires a graph at construction.** Instantiate as `DistanceExporter(graph)`, not `DistanceExporter()`. Semantic similarity columns (`semantic_similarity`) require the graph nodes to have embeddings in their properties.
</Warning>
**ReportGenerator:**
```python
@@ -349,36 +377,6 @@ The `export_csv` convenience function delegates to `CSVExporter.export()`. For p
| `"faiss"` | `faiss` | `VectorExporter` | `.faiss` | Direct FAISS index files |
| `"html"` / `"markdown"` / `"json"` / `"text"` |: | `ReportGenerator` | `.html` / `.md` / `.json` / `.txt` | Analytics reports |
## Tips and Common Pitfalls
<Warning>
**`export_to_rdf()` returns a string: it does not write a file.** Call `export()` or `export_knowledge_graph()` to write directly to disk.
</Warning>
<Warning>
**`ArangoAQLExporter.export()` and `LPGExporter.export()` write to a file and return `None`.** They do not return the AQL/Cypher string. Write to a file and read it back if you need the string.
</Warning>
<Warning>
**`DistanceExporter` requires a graph at construction.** Instantiate as `DistanceExporter(graph)`, not `DistanceExporter()`. Semantic similarity columns (`semantic_similarity`) require the graph nodes to have embeddings in their properties.
</Warning>
<Warning>
**`ParquetExporter` and `ArrowExporter` require `pyarrow`.** Both fall back to a no-op stub class if `pyarrow` is not installed. Install with `pip install pyarrow` before using these exporters.
</Warning>
<Tip>
**Use `export_to_rdf()` + string for inspection, `export()` for production.** In notebooks or debug sessions, `export_to_rdf()` is handy for quick inspection. For CI pipelines and pipelines writing files, `export()` is a single call.
</Tip>
<Tip>
**Use `turtle` for human readability, `ntriples` for streaming.** Turtle is compact and readable for debugging and sharing. N-Triples (`.nt`) is line-oriented: one triple per line: making it safe to stream, concatenate, and process with standard Unix tools.
</Tip>
<Tip>
**Use `ParquetExporter` for downstream analytics.** Parquet preserves column types (int, float, datetime) that CSV loses and is natively supported by Spark, BigQuery, Databricks, and Snowflake. Use `compression="snappy"` for a good balance of speed and compression.
</Tip>
<Tip>
**Match your export format to your consumer.** Neo4j → `cypher`; ArangoDB → `aql`; Gephi/yEd → `graphml` or `gexf`; semantic web tools → `turtle` or `json-ld`; analytics pipelines → `parquet`; zero-copy IPC → `arrow`.
</Tip>
docs/reference/graph_store.md
+32 -34
View File
@@ -108,6 +108,10 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass
store.create_node(labels=["Person"], properties={"name": "Bob"})
```
<Warning>
**Call `connect()` before any operations.** `GraphStore` does not connect automatically on construction. Either call `store.connect()` explicitly or use the context manager form `with GraphStore(...) as store:`.
</Warning>
## Quick Start
<Steps>
@@ -226,6 +230,10 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass
```
**Best for:** teams already running PostgreSQL who want graph queries without a separate service.
<Warning>
**Apache AGE requires the PostgreSQL extension installed.** `backend="age"` calls the AGE extension functions. If AGE is not installed in your PostgreSQL instance, you'll get a `ProgrammingError`. See the [Apache AGE docs](https://age.apache.org/age-manual/master/intro/setup.html) for setup.
</Warning>
</Tab>
<Tab title="Amazon Neptune">
```bash
@@ -249,6 +257,10 @@ with GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", pass
```
**Best for:** managed AWS deployments. Neptune uses the Bolt protocol for OpenCypher queries: the same query API used for Neo4j.
<Warning>
**Amazon Neptune uses `iam_auth=`, not `use_iam_auth=`.** The `AmazonNeptuneStore` and the `GraphStore` Neptune backend both use `iam_auth: bool = True` as the parameter name.
</Warning>
</Tab>
<Tab title="Backend Comparison">
@@ -311,6 +323,10 @@ if path:
print(f"Hops: {path['length']}")
```
<Tip>
**Use `create_nodes()` for bulk loading.** Individual `create_node()` calls issue one network round-trip each. `create_nodes(list)` is faster for initial graph population.
</Tip>
## QueryEngine
`QueryEngine` handles query execution and optional caching. Access it via `store.query_engine`:
@@ -354,6 +370,14 @@ engine.enable_cache()
| `enable_cache()` | `None` | Turn on caching (on by default) |
| `disable_cache()` | `None` | Turn off caching |
<Tip>
**Use `QueryEngine` caching for read-heavy workloads.** Access the engine via `store.query_engine`. Call `engine.execute(query, use_cache=True)` to cache identical queries in-process. Call `engine.clear_cache()` after writes that invalidate results.
</Tip>
<Warning>
**Use parameterized queries, never string interpolation.** `store.query("WHERE n.name = $name", parameters={"name": user_input})` prevents Cypher injection attacks. Never use `f"WHERE n.name = '{user_input}'"`.
</Warning>
## GraphAnalytics
@@ -422,6 +446,14 @@ store.create_index(label="Organization", property_name="id")
stats = store.get_stats()
```
<Warning>
**Create indexes before bulk loading.** `store.create_index(label="Person", property_name="name")` makes `MATCH` queries on `name` orders of magnitude faster. Without indexes, every query does a full scan. Create indexes first, then load data.
</Warning>
<Warning>
**`create_index` parameter is `property_name=`, not `property=`.** `store.create_index(label="Person", property_name="name")`: using `property=` will be silently ignored.
</Warning>
## Common Workflows
<Tabs>
@@ -485,40 +517,6 @@ stats = store.get_stats()
</Tab>
</Tabs>
## Tips and Common Pitfalls
<Warning>
**Call `connect()` before any operations.** `GraphStore` does not connect automatically on construction. Either call `store.connect()` explicitly or use the context manager form `with GraphStore(...) as store:`.
</Warning>
<Tip>
**Use `create_nodes()` for bulk loading.** Individual `create_node()` calls issue one network round-trip each. `create_nodes(list)` is faster for initial graph population.
</Tip>
<Warning>
**Create indexes before bulk loading.** `store.create_index(label="Person", property_name="name")` makes `MATCH` queries on `name` orders of magnitude faster. Without indexes, every query does a full scan. Create indexes first, then load data.
</Warning>
<Warning>
**Use parameterized queries, never string interpolation.** `store.query("WHERE n.name = $name", parameters={"name": user_input})` prevents Cypher injection attacks. Never use `f"WHERE n.name = '{user_input}'"`.
</Warning>
<Warning>
**`create_index` parameter is `property_name=`, not `property=`.** `store.create_index(label="Person", property_name="name")`: using `property=` will be silently ignored.
</Warning>
<Tip>
**Use `QueryEngine` caching for read-heavy workloads.** Access the engine via `store.query_engine`. Call `engine.execute(query, use_cache=True)` to cache identical queries in-process. Call `engine.clear_cache()` after writes that invalidate results.
</Tip>
<Warning>
**Apache AGE requires the PostgreSQL extension installed.** `backend="age"` calls the AGE extension functions. If AGE is not installed in your PostgreSQL instance, you'll get a `ProgrammingError`. See the [Apache AGE docs](https://age.apache.org/age-manual/master/intro/setup.html) for setup.
</Warning>
<Warning>
**Amazon Neptune uses `iam_auth=`, not `use_iam_auth=`.** The `AmazonNeptuneStore` and the `GraphStore` Neptune backend both use `iam_auth: bool = True` as the parameter name.
</Warning>
<CardGroup cols={2}>
<Card title="KG Module" icon="diagram-project" href="kg">
Build the graph before persisting it.
docs/reference/ingest.md
+24 -26
View File
@@ -56,6 +56,10 @@ for f in files:
print(f.name, f.file_type, f.size)
```
<Tip>
**`FileIngestor` is always the fastest path for local files.** It auto-detects format from extension, handles ZIP/TAR archives automatically, and reads content into `.content` bytes or the `.text` property. Use `read_content=False` when you only need file metadata.
</Tip>
For web, database, or stream sources, each ingestor exposes its own typed method:
```python
@@ -195,6 +199,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
Requires `pyarrow`: `pip install pyarrow`.
<Tip>
**Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns: critical for wide tables with hundreds of columns.
</Tip>
### XMLIngestor
XXE-safe lxml-based ingestion with optional schema validation:
@@ -220,6 +228,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
<Note>
`XMLIngestor` uses lxml with `resolve_entities=False` to prevent XML External Entity (XXE) injection attacks.
</Note>
<Warning>
**`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica: it does not block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML.
</Warning>
</Tab>
<Tab title="Web & Feed">
### WebIngestor
@@ -248,6 +260,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
Requires `beautifulsoup4`: `pip install beautifulsoup4`.
<Tip>
**Rate-limit web crawling.** `WebIngestor(delay=1.0, respect_robots=True)` is the responsible default. Without rate limiting you risk getting blocked by the target server or violating its terms of service.
</Tip>
### PublicAPIIngestor
Use this for public REST-style APIs that do not require keys or tokens:
@@ -403,6 +419,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
Requires `sqlalchemy`: `pip install sqlalchemy` plus your database driver.
<Warning>
**`DBIngestor()` takes no connection string in its constructor.** Pass the connection string to `ingest_database()`, `execute_query()`, or `export_table()` as the first positional argument: not to `DBIngestor()` itself.
</Warning>
### SnowflakeIngestor
```python
@@ -467,6 +487,10 @@ result = ingest("ontology.ttl") # -> {"ontology": OntologyData}
```
Stream processors require the appropriate client library (kafka-python, pika, boto3, pulsar-client).
<Warning>
**`StreamIngestor` methods require the target broker's client library to be installed.** `ingest_kafka` needs `kafka-python`, `ingest_rabbitmq` needs `pika`, `ingest_kinesis` needs `boto3`, and `ingest_pulsar` needs `pulsar-client`. Missing dependencies raise `ImportError` at call time, not at import time.
</Warning>
</Tab>
</Tabs>
@@ -601,32 +625,6 @@ from semantica.ingest import ingest_file
result = ingest_file("source_path", method="my_format")
```
## Tips and Common Pitfalls
<Warning>
**`DBIngestor()` takes no connection string in its constructor.** Pass the connection string to `ingest_database()`, `execute_query()`, or `export_table()` as the first positional argument: not to `DBIngestor()` itself.
</Warning>
<Tip>
**`FileIngestor` is always the fastest path for local files.** It auto-detects format from extension, handles ZIP/TAR archives automatically, and reads content into `.content` bytes or the `.text` property. Use `read_content=False` when you only need file metadata.
</Tip>
<Tip>
**Use `ParquetIngestor` instead of `FileIngestor` for structured analytical data.** Parquet ingestion preserves column types (int, float, datetime) that CSV reading loses. Use `columns=["id", "text"]` to avoid loading unused columns: critical for wide tables with hundreds of columns.
</Tip>
<Warning>
**`XMLIngestor` is XXE-safe by default.** Do not use standard `xml.etree.ElementTree` to pre-parse XML before passing to Semantica: it does not block XXE attacks. `XMLIngestor` uses lxml with `resolve_entities=False` to safely parse untrusted XML.
</Warning>
<Tip>
**Rate-limit web crawling.** `WebIngestor(delay=1.0, respect_robots=True)` is the responsible default. Without rate limiting you risk getting blocked by the target server or violating its terms of service.
</Tip>
<Warning>
**`StreamIngestor` methods require the target broker's client library to be installed.** `ingest_kafka` needs `kafka-python`, `ingest_rabbitmq` needs `pika`, `ingest_kinesis` needs `boto3`, and `ingest_pulsar` needs `pulsar-client`. Missing dependencies raise `ImportError` at call time, not at import time.
</Warning>
<CardGroup cols={2}>
<Card title="Parse" icon="file-lines" href="parse">
Parse raw sources into structured text and tables.
docs/reference/mcp_server.md
+20 -23
View File
@@ -10,7 +10,6 @@ icon: "plug"
- No Python code required after launch: configure once, use from any MCP-aware client
- Compatible with Claude Desktop, Windsurf, Cline, Continue, VS Code, Roo Code, Cursor
## Server Interface
```json
@@ -35,6 +34,10 @@ python -m semantica.mcp_server
`semantica.mcp_server` is a **stdio server process**, not a Python library. It exposes no importable classes: all interaction happens through MCP tool calls from a connected AI client.
</Tip>
<Warning>
**The server communicates over stdio: don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. All logging is written to `stderr` only. Configure log verbosity with the `SEMANTICA_LOG_LEVEL` environment variable.
</Warning>
## What You Get
<CardGroup cols={2}>
@@ -134,6 +137,10 @@ The MCP server is included in the base install: no extras required.
</CodeGroup>
<Warning>
**Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently: the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works.
</Warning>
</Step>
<Step title="Test locally before configuring your client">
```bash
@@ -156,6 +163,14 @@ The MCP server is included in the base install: no extras required.
| `SEMANTICA_KG_PATH` | *(none: in-memory graph)* | Path to a persisted graph file to load on startup |
| `SEMANTICA_LOG_LEVEL` | `WARNING` | Log verbosity: `DEBUG`, `INFO`, `WARNING` |
<Warning>
**The graph starts empty unless you set `SEMANTICA_KG_PATH`.** The MCP server creates a fresh in-memory `ContextGraph` on first use. Set `SEMANTICA_KG_PATH` to a previously saved graph file to restore state across server restarts. Without it, all data is lost when the process exits.
</Warning>
<Tip>
**Enable debug logging for troubleshooting.** Set `SEMANTICA_LOG_LEVEL=DEBUG` in your MCP client's `env` block, or run `python -m semantica.mcp_server` directly and inspect stderr output.
</Tip>
## Tools
The MCP server exposes 12 tools that any connected AI assistant can call:
@@ -294,6 +309,10 @@ Find past decisions similar to a given scenario using hybrid similarity search.
`max_results` defaults to `5`, maximum `50`.
<Tip>
**Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path: it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs.
</Tip>
</Accordion>
<Accordion title="get_causal_chain" icon="diagram-project">
@@ -440,28 +459,6 @@ The MCP server exposes three readable resources:
| `semantica://decisions/list` | All recorded decisions (up to 50) |
| `semantica://schema/info` | Server version and available tools |
## Tips and Common Pitfalls
<Warning>
**The graph starts empty unless you set `SEMANTICA_KG_PATH`.** The MCP server creates a fresh in-memory `ContextGraph` on first use. Set `SEMANTICA_KG_PATH` to a previously saved graph file to restore state across server restarts. Without it, all data is lost when the process exits.
</Warning>
<Tip>
**Use `find_precedents` before high-stakes decisions.** The tool performs hybrid similarity search across all recorded decisions. Call it at the start of any significant decision path: it surfaces past reasoning that may be directly applicable, reducing redundant work and improving consistency across agent runs.
</Tip>
<Warning>
**Configure your MCP client's `command` field exactly.** The `command` field must point to the exact executable path (use `which semantica-mcp` on macOS/Linux to find it). A wrong path fails silently: the server just doesn't appear in the tools list. Test with the raw `echo | semantica-mcp` command first to confirm the binary works.
</Warning>
<Warning>
**The server communicates over stdio: don't add logging to stdout.** Any `print()` or logger output directed to stdout will corrupt the JSON-RPC message stream. All logging is written to `stderr` only. Configure log verbosity with the `SEMANTICA_LOG_LEVEL` environment variable.
</Warning>
<Tip>
**Enable debug logging for troubleshooting.** Set `SEMANTICA_LOG_LEVEL=DEBUG` in your MCP client's `env` block, or run `python -m semantica.mcp_server` directly and inspect stderr output.
</Tip>
<CardGroup cols={2}>
<Card title="Context" icon="brain" href="context">
The ContextGraph that the MCP server operates on.
docs/reference/normalize.md
+38 -40
View File
@@ -22,7 +22,7 @@ Unstructured data is inconsistent by nature. Without normalization, the same rea
- `"Apple Inc."`, `"Apple Computer Inc."`, `"APPLE INC."`: multiple nodes, one company
- `"Jan 1st, 2020"`, `"01/01/2020"`, `"2020-01-01"`: three formats, one date
- `"$1.2B"`, `"1,200,000,000"`, `"1.2 billion USD"`: three strings, one number
- `"Hello World"` vs `"Hello\u00a0World"`: a non-breaking space that breaks string matching
- `"Hello World"` vs `"Hello World"`: a non-breaking space that breaks string matching
Normalization collapses these variants before any extractor, deduplicator, or graph builder sees the data.
@@ -51,7 +51,7 @@ from semantica.normalize import (
# Text: normalize unicode, collapse whitespace, replace smart quotes
normalizer = TextNormalizer()
clean = normalizer.normalize_text(" Hello,\u00a0 World\u2026 ")
clean = normalizer.normalize_text(" Hello,  World… ")
# → "Hello, World..."
# Date
@@ -90,6 +90,10 @@ utf8_text = handler.convert_to_utf8(raw_bytes)
# convert_to_utf8 returns a str
utf8_text = handler.convert_to_utf8(raw_bytes)
```
<Warning>
**Run encoding repair before anything else.** A single cp1252 character in a UTF-8 stream silently corrupts the surrounding text. Call `handler.convert_to_utf8(raw_bytes)` first, before any other normalizer sees the data.
</Warning>
</Step>
<Step title="TextNormalizer: unicode, whitespace, special chars">
```python
@@ -103,6 +107,10 @@ utf8_text = handler.convert_to_utf8(raw_bytes)
case="preserve",
)
```
<Warning>
**Don't lowercase before NER.** `normalize_text(text, case="lower")` before entity extraction destroys capitalization signals that NER relies on. Apply case normalization only after extraction if needed.
</Warning>
</Step>
<Step title="EntityNormalizer: canonicalize entity names">
```python
@@ -118,6 +126,10 @@ utf8_text = handler.convert_to_utf8(raw_bytes)
)
# → "Apple Inc." (if the alias_map contains it, else title-cased input)
```
<Warning>
**`EntityNormalizer` has no built-in corporate suffix expansion.** There is no automatic mapping of `"Apple Computer Inc."` → `"Apple Inc."`. To canonicalize corporate names, provide an explicit `alias_map` with lowercase keys: `EntityNormalizer(alias_map={"apple computer inc.": "Apple Inc."})`.
</Warning>
</Step>
<Step title="DateNormalizer and NumberNormalizer: parse structured values">
```python
@@ -210,13 +222,13 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
# Batch normalization
results = normalizer.process_batch(
[" hello ", "WORLD", "caf\u00e9"],
[" hello ", "WORLD", "café"],
unicode_form="NFKC",
case="lower",
)
# normalize() accepts str or List[Dict] (parsed docs from DocumentParser)
docs = [{"content": "Hello\u00a0world"}, {"content": "test text"}]
docs = [{"content": "Hello world"}, {"content": "test text"}]
normalized_docs = normalizer.normalize(docs)
```
@@ -244,14 +256,14 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
)
unicode_norm = UnicodeNormalizer()
text = unicode_norm.normalize_unicode("caf\u00e9", form="NFC")
text = unicode_norm.normalize_unicode("café", form="NFC")
ws_norm = WhitespaceNormalizer()
text = ws_norm.normalize_whitespace("Hello\t\t World\n\n")
# → "Hello World\n\n"
processor = SpecialCharacterProcessor()
text = processor.normalize_punctuation("\u2018Hello\u2019")
text = processor.normalize_punctuation("‘Hello’")
# → "'Hello'"
```
</Tab>
@@ -313,6 +325,10 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
canonical = handler.normalize_name_format("Dr. JOHN P. SMITH Jr.")
# → "John P. Smith Jr." (removes leading title)
```
<Tip>
**`AliasResolver` uses lowercase key lookup.** Register aliases with lowercase keys even if the canonical form is title-cased. The resolver converts the input to lowercase before lookup.
</Tip>
</Tab>
<Tab title="DateNormalizer">
`DateNormalizer` takes `config=None, **kwargs`. The `format` and `timezone`
@@ -427,6 +443,10 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
`detect()` requires at least 10 characters for reliable detection. On shorter text it returns the `default_language` (default: `"en"`).
</Note>
<Warning>
**`LanguageDetector.detect()` returns a `str`, not a dict.** Use `detect_with_confidence()` for `(language_code, confidence)` tuple, or `detect_multiple()` for `List[(code, confidence)]`.
</Warning>
### EncodingHandler
Detect and repair character encoding issues. Requires `chardet`: `pip install chardet`.
@@ -457,6 +477,10 @@ utf8_text = handle_encoding(raw_bytes, operation="convert")
then falls back through `latin-1`, `cp1252`, `iso-8859-1`
- Always run `EncodingHandler` first: broken bytes cause cascading failures
in every downstream normalizer
<Warning>
**`EncodingHandler.detect()` returns a `(str, float)` tuple, not a dict.** Unpack with `encoding, confidence = handler.detect(data)`.
</Warning>
</Tab>
</Tabs>
@@ -511,6 +535,14 @@ print(f"Warnings: {len(result.warnings)}")
| `validate_data(dataset, schema)` | `ValidationResult` | Validate records against a schema dict |
| `handle_missing_values(dataset, strategy)` | `List[Dict]` | Remove, fill, or impute missing values |
<Tip>
**`DataCleaner.remove_duplicates()` does not exist as a standalone method.** Use `detect_duplicates()` to get `DuplicateGroup` objects, or call `clean_data(records, remove_duplicates=True)` to remove them in-place.
</Tip>
<Tip>
**`DataCleaner` operates on flat records, not graph entities.** For entity-level semantic deduplication, use `DuplicateDetector` from the Deduplication module instead.
</Tip>
## Pipeline Integration
```python
@@ -552,40 +584,6 @@ normalized = normalize_text("Apple Inc.", method="expand_suffixes")
# → "Apple Incorporated"
```
## Tips and Common Pitfalls
<Warning>
**Run encoding repair before anything else.** A single cp1252 character in a UTF-8 stream silently corrupts the surrounding text. Call `handler.convert_to_utf8(raw_bytes)` first, before any other normalizer sees the data.
</Warning>
<Warning>
**Don't lowercase before NER.** `normalize_text(text, case="lower")` before entity extraction destroys capitalization signals that NER relies on. Apply case normalization only after extraction if needed.
</Warning>
<Warning>
**`EntityNormalizer` has no built-in corporate suffix expansion.** There is no automatic mapping of `"Apple Computer Inc."` → `"Apple Inc."`. To canonicalize corporate names, provide an explicit `alias_map` with lowercase keys: `EntityNormalizer(alias_map={"apple computer inc.": "Apple Inc."})`.
</Warning>
<Tip>
**`AliasResolver` uses lowercase key lookup.** Register aliases with lowercase keys even if the canonical form is title-cased. The resolver converts the input to lowercase before lookup.
</Tip>
<Warning>
**`LanguageDetector.detect()` returns a `str`, not a dict.** Use `detect_with_confidence()` for `(language_code, confidence)` tuple, or `detect_multiple()` for `List[(code, confidence)]`.
</Warning>
<Warning>
**`EncodingHandler.detect()` returns a `(str, float)` tuple, not a dict.** Unpack with `encoding, confidence = handler.detect(data)`.
</Warning>
<Tip>
**`DataCleaner.remove_duplicates()` does not exist as a standalone method.** Use `detect_duplicates()` to get `DuplicateGroup` objects, or call `clean_data(records, remove_duplicates=True)` to remove them in-place.
</Tip>
<Tip>
**`DataCleaner` operates on flat records, not graph entities.** For entity-level semantic deduplication, use `DuplicateDetector` from the Deduplication module instead.
</Tip>
<CardGroup cols={2}>
<Card title="Parse" icon="file-lines" href="parse">
Parse documents before normalization.
docs/reference/ontology.md
-1
View File
@@ -164,7 +164,6 @@ ontology = generator.generate_ontology_from_text(
text="A biomedical ontology for clinical trial protocols involving patients, trials, interventions, and outcomes."
)
```
```
## OWL / RDF Export
docs/reference/pipeline.md
+20 -22
View File
@@ -97,6 +97,10 @@ You could wire Semantica modules together with plain Python code. Pipelines add:
for warning in result.warnings:
print(f"Warning: {warning}")
```
<Tip>
**Use `PipelineValidator` before running in production.** It catches dependency cycles, missing step names, and misconfigured connections that would only surface as errors mid-run. Validation is instant; catching them after a 30-minute extraction job is not.
</Tip>
</Step>
<Step title="Execute and inspect results">
```python
@@ -111,6 +115,10 @@ You could wire Semantica modules together with plain Python code. Pipelines add:
print(f"Steps failed: {result.metrics['steps_failed']}")
print(f"Duration: {result.metrics['execution_time']:.1f}s")
```
<Tip>
**Inspect `result.metrics` to find bottlenecks.** `result.metrics['steps_executed']` and `result.metrics['execution_time']` give a quick read on overall pipeline health. For per-step timing, check `step.result` on each `PipelineStep` after the run.
</Tip>
</Step>
</Steps>
@@ -133,6 +141,10 @@ engine = ExecutionEngine(max_workers=4)
result = engine.execute_pipeline(pipeline, data="data/")
```
<Tip>
**Set `workers=` based on workload type.** Thread workers for I/O-bound steps (web fetching, DB queries), process workers for CPU-bound steps (embedding, OCR, large NER batches). Mixing pool types on the wrong step type wastes resources without speed gains.
</Tip>
## Retry and Error Handling
<Tabs>
@@ -196,6 +208,10 @@ result = engine.execute_pipeline(pipeline, data="data/")
In production, configure a `RetryPolicy` with limited retries so a single failing step does not stop the whole run. After execution, inspect `result.errors` to find and reprocess failed documents.
</Warning>
<Warning>
**Configure retry policies to contain failures in production.** Use `handler.set_retry_policy("step_type", RetryPolicy(max_retries=3))` so transient errors are retried without stopping the pipeline. After the run, inspect `result.errors` to find and reprocess any documents that exhausted retries.
</Warning>
## Progress Tracking
<Tabs>
@@ -349,6 +365,10 @@ The `create_pipeline_from_template(name)` method returns a configured `PipelineB
</Card>
</CardGroup>
<Tip>
**Use templates from `PipelineTemplateManager` for common patterns.** `create_pipeline_from_template("kg_construction")` wires normalization, deduplication, conflict detection, and graph construction in the correct order: saving you from common mistakes like deduplicating before normalizing.
</Tip>
## ExecutionEngine
Fine-grained control over pipeline execution: pause, resume, cancel, and inspect live progress:
@@ -563,28 +583,6 @@ StepStatus.SKIPPED # Skipped due to FailureHandler "skip" strategy
</Accordion>
</AccordionGroup>
## Tips and Common Pitfalls
<Tip>
**Use `PipelineValidator` before running in production.** It catches dependency cycles, missing step names, and misconfigured connections that would only surface as errors mid-run. Validation is instant; catching them after a 30-minute extraction job is not.
</Tip>
<Tip>
**Set `workers=` based on workload type.** Thread workers for I/O-bound steps (web fetching, DB queries), process workers for CPU-bound steps (embedding, OCR, large NER batches). Mixing pool types on the wrong step type wastes resources without speed gains.
</Tip>
<Warning>
**Configure retry policies to contain failures in production.** Use `handler.set_retry_policy("step_type", RetryPolicy(max_retries=3))` so transient errors are retried without stopping the pipeline. After the run, inspect `result.errors` to find and reprocess any documents that exhausted retries.
</Warning>
<Tip>
**Use templates from `PipelineTemplateManager` for common patterns.** `create_pipeline_from_template("kg_construction")` wires normalization, deduplication, conflict detection, and graph construction in the correct order: saving you from common mistakes like deduplicating before normalizing.
</Tip>
<Tip>
**Inspect `result.metrics` to find bottlenecks.** `result.metrics['steps_executed']` and `result.metrics['execution_time']` give a quick read on overall pipeline health. For per-step timing, check `step.result` on each `PipelineStep` after the run.
</Tip>
<CardGroup cols={2}>
<Card title="Ingest" icon="database" href="ingest">
First step in most pipelines.
docs/reference/provenance.md
+16
View File
@@ -98,6 +98,10 @@ ProvenanceManager(
If both `storage` and `storage_path` are omitted, an `InMemoryStorage` is used.
<Warning>
**`InMemoryStorage` does not persist across restarts.** Pass `storage_path="provenance.db"` or an explicit `SQLiteStorage` instance in any environment where the audit trail must survive process exits.
</Warning>
### Tracking Methods
```python
@@ -196,6 +200,10 @@ if prov:
print(prov["source_document"])
```
<Note>
`get_lineage()` returns an aggregated **dict**, not a `ProvenanceEntry`. Use `trace_lineage()` to get the raw `ProvenanceEntry` objects when you need field-level access such as `entry.checksum`.
</Note>
### Utility Methods
```python
@@ -338,6 +346,10 @@ if not is_valid:
The checksum covers `entity_id`, `entity_type`, `activity_id`, `source_document`, `timestamp`, and `confidence`.
<Tip>
**Run `verify_checksum(entry)` before any compliance export.** Pass the `ProvenanceEntry` object returned by `trace_lineage()` directly. If the stored checksum no longer matches, raise an error before the export proceeds.
</Tip>
## Bridge Axiom Translation Chains
`BridgeAxiom` and `TranslationChain` are available in `semantica.provenance.bridge_axiom` for tracking multi-layer domain translations with full coefficient attribution:
@@ -420,6 +432,10 @@ lineage = manager.get_lineage(entities[0].id)
print(lineage["source_documents"])
```
<Note>
Setting `provenance=True` on `NERExtractor` embeds metadata on the extracted entity objects — it does not automatically call `ProvenanceManager.track_entity()`. You must call `track_entity()` yourself after extraction.
</Note>
## Common Workflows
<Tabs>
docs/reference/seed.md
+17 -19
View File
@@ -61,6 +61,10 @@ icon: "database"
manager.register_source("taxonomy", "json", "data/taxonomy.json")
manager.register_source("employees", "csv", "data/employees.csv")
```
<Tip>
**Register all sources before calling `create_foundation_graph()`.** `create_foundation_graph()` processes all registered sources in one pass. Registering a source after calling it means that source is silently excluded. Register all sources at the start of your script, then call `create_foundation_graph()` once.
</Tip>
</Step>
<Step title="Build the foundation graph">
```python
@@ -82,11 +86,15 @@ icon: "database"
else:
print(f"Validated {report['metrics']['entity_count']} entities: no issues found")
```
<Warning>
**Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast: always run it first.
</Warning>
</Step>
<Step title="Merge with extracted data">
```python
from semantica.semantic_extract import NERExtractor
extractor = NERExtractor(method="ml")
new_entities = extractor.extract("Apple Inc. partners with Microsoft Corp.")
@@ -97,6 +105,10 @@ icon: "database"
merge_strategy="merge"
)
```
<Warning>
**Load seed data before extracted data.** Seed data is your ground truth: normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values.
</Warning>
</Step>
</Steps>
@@ -230,6 +242,10 @@ Different strategies for resolving conflicts during `integrate_with_extracted()`
</Tab>
</Tabs>
<Tip>
**Use `seed_first` merge strategy for reference data.** When seed data encodes authoritative facts (official company names, canonical taxonomy IDs, employee records), `merge_strategy="seed_first"` ensures those values win over extracted values. Use `merge` only when extracted data may be more current than the seed.
</Tip>
## Full Pipeline Example
```python
@@ -315,24 +331,6 @@ export SEMANTICA_SEED_DATA_DIR=./data/seed
export SEMANTICA_SEED_MERGE_STRATEGY=seed_first
```
## Tips and Common Pitfalls
<Warning>
**Load seed data before extracted data.** Seed data is your ground truth: normalised, curated, and already de-duplicated. Load it first with `create_foundation_graph()`, then merge extracted entities on top. Merging in the wrong order lets noisy extracted data overwrite trusted reference values.
</Warning>
<Tip>
**Use `seed_first` merge strategy for reference data.** When seed data encodes authoritative facts (official company names, canonical taxonomy IDs, employee records), `merge_strategy="seed_first"` ensures those values win over extracted values. Use `merge` only when extracted data may be more current than the seed.
</Tip>
<Warning>
**Validate before loading.** `manager.validate_quality(seed_data)` catches missing required fields, type inconsistencies, and duplicate IDs before they corrupt your graph. Running validation after loading means you'll need to roll back. Validation is fast: always run it first.
</Warning>
<Tip>
**Register all sources before calling `create_foundation_graph()`.** `create_foundation_graph()` processes all registered sources in one pass. Registering a source after calling it means that source is silently excluded. Register all sources at the start of your script, then call `create_foundation_graph()` once.
</Tip>
<Tip>
**Use YAML configuration for production deployments.** Hard-coding source paths in Python scripts makes environment-switching (dev → staging → prod) fragile. Declare sources in `config.yaml` under the `seed:` key and override paths with `SEMANTICA_SEED_DATA_DIR`. This way, the same code runs in every environment.
</Tip>
docs/reference/split.md
+12 -14
View File
@@ -179,6 +179,10 @@ splitter = TextSplitter(
| `relation_method` | `str` | `"ml"` | Relation extraction method for `relation_aware`: `"ml"` \| `"llm"` \| `"huggingface"` |
| `tokenizer` | `str` | `"gpt-4"` | tiktoken model name for `token` method: unrecognised names fall back to `cl100k_base` |
<Warning>
**`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum: for `chunk_size=1000`, set `chunk_overlap=100` to `200`.
</Warning>
## Splitting Method Details
<Tabs>
@@ -217,6 +221,10 @@ splitter = TextSplitter(
- Produces variable-length chunks: some topics are short, others long
- Falls back to sentence splitting if `sentence-transformers` is not installed
- Slower than `recursive` due to embedding computation; cache embeddings for repeated splits
<Tip>
**Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting: use `recursive` instead.
</Tip>
</Tab>
<Tab title="Entity-Aware">
Runs NER internally, then adjusts chunk boundaries so no entity mention is split across two chunks:
@@ -346,6 +354,10 @@ The `token` method accepts a `tokenizer=` kwarg that is passed to `tiktoken.enco
If `tiktoken` is not installed, the `token` method falls back to splitting by whitespace-separated words.
<Warning>
**Wrong tokenizer.** The `token` method passes the `tokenizer=` value to `tiktoken.encoding_for_model()`. If the model name is not recognised by tiktoken it silently falls back to `cl100k_base`. Pass a valid tiktoken model name (e.g. `"gpt-4"`, `"gpt-3.5-turbo"`) to get deterministic behaviour.
</Warning>
## Pipeline Integration
`TextSplitter` can be used standalone or composed manually with other Semantica modules. The example below shows a sequential pattern: parse a file, split the text, then extract entities from each chunk:
@@ -373,20 +385,6 @@ for chunk in chunks:
For the full pipeline orchestration API, see the [Pipeline reference](pipeline).
## Tips and Common Pitfalls
<Warning>
**`chunk_overlap` too small.** Without overlap, a fact that spans a chunk boundary is invisible in both chunks. A 10–20% overlap relative to `chunk_size` is a safe minimum: for `chunk_size=1000`, set `chunk_overlap=100` to `200`.
</Warning>
<Warning>
**Wrong tokenizer.** The `token` method passes the `tokenizer=` value to `tiktoken.encoding_for_model()`. If the model name is not recognised by tiktoken it silently falls back to `cl100k_base`. Pass a valid tiktoken model name (e.g. `"gpt-4"`, `"gpt-3.5-turbo"`) to get deterministic behaviour.
</Warning>
<Tip>
**Semantic splitting needs enough sentences.** `semantic_transformer` needs several sentences to detect topic shifts. On documents shorter than ~300 words it behaves like `sentence` splitting: use `recursive` instead.
</Tip>
<CardGroup cols={2}>
<Card title="Parse" icon="file-lines" href="parse">
Parse documents before chunking: produces sections and metadata.
docs/reference/triplet_store.md
+23 -27
View File
@@ -163,7 +163,9 @@ for row in result.bindings:
**Best for:** local development with rdflib, SPARQL read queries against a Fuseki endpoint.
**Note on inference:** `JenaStore` accepts `enable_inference=True` in config but OWL reasoning is a placeholder and does not produce inferred triples in the current implementation.
<Warning>
**`backend="jena"` OWL inference is a placeholder.** `enable_inference=True` is accepted but the inference call returns 0 inferred triples. For production OWL reasoning, use Jena Fuseki directly with its built-in reasoner configuration.
</Warning>
</Tab>
<Tab title="RDF4J">
```bash
@@ -191,6 +193,10 @@ for row in result.bindings:
</Tab>
</Tabs>
<Tip>
**Use Apache Jena for development, Blazegraph for production.** Jena initializes with rdflib in-memory: no server required for local testing. Switch to Blazegraph for high-throughput persistent workloads by changing `backend=`.
</Tip>
## Triplet Object
All store operations use the `Triplet` dataclass from `semantica.semantic_extract.types`:
@@ -215,6 +221,10 @@ t = Triplet(
| `confidence` | `float` | `1.0` | Confidence score (0–1) |
| `metadata` | `dict` | `{}` | Arbitrary metadata |
<Warning>
**`add_triplet()` takes a `Triplet` object, not keyword arguments.** Use `Triplet(subject=..., predicate=..., object=...)` from `semantica.semantic_extract.types` and pass the object: not `subject=`, `predicate=`, `obj=` to `add_triplet`.
</Warning>
## TripletStore Methods
| Method | Returns | Description |
@@ -277,6 +287,10 @@ store.execute_query("""
| `execution_time` | `float` | Seconds elapsed |
| `metadata` | `dict` | Query, graph scope, cache hit flag |
<Warning>
**`execute_query()` returns `QueryResult`, not a list.** Iterate `result.bindings`, not `result` directly. Each binding is a dict mapping variable name → `{"value": ..., "type": ...}`.
</Warning>
## SPARQL Result Pagination
For large result sets, paginate with LIMIT and OFFSET:
@@ -299,6 +313,10 @@ while True:
offset += page_size
```
<Warning>
**Paginate large SPARQL result sets.** A `SELECT * WHERE { ?s ?p ?o }` against a large store returns all triples. Always include `LIMIT` and `OFFSET` in exploratory queries. `QueryEngine` adds `LIMIT 1000` automatically unless you specify one.
</Warning>
## Named Graph Scoping
Blazegraph and RDF4J support named graphs. Scope `execute_query()` to a named graph with the `graph=` parameter:
@@ -333,6 +351,10 @@ result = store.execute_query("""
Named graph support is only available for Blazegraph and RDF4J backends. The `graph=` parameter is silently ignored for the Jena backend.
</Note>
<Tip>
**Use named graphs to isolate sources.** Pass `graph="http://example.org/source_A"` to `execute_query()` to scope a query to a specific named graph. Blazegraph and RDF4J support named graphs; Jena (rdflib backend) does not.
</Tip>
## Bulk Loading
`add_triplets()` batches writes via the internal `BulkLoader`. Access `store.bulk_loader` to configure it:
@@ -468,32 +490,6 @@ for row in result.bindings:
print(row)
```
## Tips and Common Pitfalls
<Tip>
**Use Apache Jena for development, Blazegraph for production.** Jena initializes with rdflib in-memory: no server required for local testing. Switch to Blazegraph for high-throughput persistent workloads by changing `backend=`.
</Tip>
<Warning>
**`execute_query()` returns `QueryResult`, not a list.** Iterate `result.bindings`, not `result` directly. Each binding is a dict mapping variable name → `{"value": ..., "type": ...}`.
</Warning>
<Warning>
**`add_triplet()` takes a `Triplet` object, not keyword arguments.** Use `Triplet(subject=..., predicate=..., object=...)` from `semantica.semantic_extract.types` and pass the object: not `subject=`, `predicate=`, `obj=` to `add_triplet`.
</Warning>
<Warning>
**Paginate large SPARQL result sets.** A `SELECT * WHERE { ?s ?p ?o }` against a large store returns all triples. Always include `LIMIT` and `OFFSET` in exploratory queries. `QueryEngine` adds `LIMIT 1000` automatically unless you specify one.
</Warning>
<Tip>
**Use named graphs to isolate sources.** Pass `graph="http://example.org/source_A"` to `execute_query()` to scope a query to a specific named graph. Blazegraph and RDF4J support named graphs; Jena (rdflib backend) does not.
</Tip>
<Warning>
**`backend="jena"` OWL inference is a placeholder.** `enable_inference=True` is accepted but the inference call returns 0 inferred triples. For production OWL reasoning, use Jena Fuseki directly with its built-in reasoner configuration.
</Warning>
<CardGroup cols={2}>
<Card title="Export" icon="file-export" href="export">
Export knowledge graphs to RDF formats.
docs/reference/utils.md
+21 -22
View File
@@ -67,6 +67,10 @@ Most users won't call utils directly: it's the **shared foundation** for all mod
setup_logging(level="INFO") # "DEBUG" | "INFO" | "WARNING" | "ERROR"
logger = get_logger(__name__)
```
<Warning>
**Call `setup_logging(level="INFO")` once at application startup.** Without it, Semantica falls back to Python's root logger, which may be silent or misconfigured. Call it before importing other Semantica modules to capture initialization messages.
</Warning>
</Step>
<Step title="Instrument expensive functions with the performance decorator">
```python
@@ -77,6 +81,10 @@ Most users won't call utils directly: it's the **shared foundation** for all mod
...
# Logs: "expensive_step completed in 2.34s"
```
<Tip>
**`@log_execution_time` is the performance decorator.** Apply it to any function to automatically log its name, execution time, and success/failure. `log_performance` is a lower-level function for logging metrics you've already collected: it is not a decorator.
</Tip>
</Step>
<Step title="Configure via environment variables">
```bash
@@ -119,10 +127,15 @@ for item in track_progress(items, desc="Processing documents"):
```
Supports:
- **Console**: tqdm progress bar with ETA
- **Jupyter**: notebook-compatible widget (auto-detected)
- **File**: write progress to a log file
<Tip>
**`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment: the same call works in both.
</Tip>
## Helper Functions
```python
@@ -138,6 +151,10 @@ uid = hash_data({"key": "value"}) # -> hex digest string
fname = safe_filename("My File?.txt") # -> "My_File.txt"
```
<Tip>
**`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string: suitable as a cache key or idempotency token in pipeline steps.
</Tip>
## Nested Dict Utilities
Helper functions for deep configuration access: used extensively inside `Config` and `ConfigManager`:
@@ -196,6 +213,10 @@ except SemanticaError as e:
</Accordion>
</AccordionGroup>
<Tip>
**Catch `SemanticaError` as the broadest exception net.** All framework errors inherit from `SemanticaError`, so `except SemanticaError` catches validation failures, processing errors, and everything in between. Use specific subclasses for targeted recovery logic.
</Tip>
## File Utilities
```python
@@ -205,28 +226,6 @@ from semantica.utils import read_json_file
config = read_json_file("config.json")
```
## Tips and Common Pitfalls
<Warning>
**Call `setup_logging(level="INFO")` once at application startup.** Without it, Semantica falls back to Python's root logger, which may be silent or misconfigured. Call it before importing other Semantica modules to capture initialization messages.
</Warning>
<Tip>
**`@log_execution_time` is the performance decorator.** Apply it to any function to automatically log its name, execution time, and success/failure. `log_performance` is a lower-level function for logging metrics you've already collected: it is not a decorator.
</Tip>
<Tip>
**`hash_data()` is deterministic across runs.** Given the same input dict (any JSON-serializable object), `hash_data()` always returns the same SHA-256 hex string: suitable as a cache key or idempotency token in pipeline steps.
</Tip>
<Tip>
**Catch `SemanticaError` as the broadest exception net.** All framework errors inherit from `SemanticaError`, so `except SemanticaError` catches validation failures, processing errors, and everything in between. Use specific subclasses for targeted recovery logic.
</Tip>
<Tip>
**`track_progress` auto-detects Jupyter.** In a terminal it renders a tqdm progress bar; in a Jupyter notebook it renders an interactive widget. You don't need to check the environment: the same call works in both.
</Tip>
<CardGroup cols={2}>
<Card title="Core" icon="gear" href="core">
Framework orchestration that uses Utils internally.
docs/reference/vector_store.md
+28 -30
View File
@@ -91,6 +91,14 @@ for r in results:
print(f"{r['id']}: score: {r['score']:.3f}")
```
<Warning>
**Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size: `BAAI/bge-small-en-v1.5` = 384, `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time.
</Warning>
<Tip>
**Use `add_documents()` for text, `store_vectors()` for pre-computed embeddings.** `add_documents()` auto-embeds in parallel batches. If your embeddings are already computed (e.g. from a fine-tuned model), use `store_vectors()` directly to skip re-embedding.
</Tip>
## Quick Start
<Steps>
@@ -307,6 +315,10 @@ sources = [
fused = search.multi_source_search(query_vector, sources, k=10)
```
<Tip>
**Use `HybridSearch(vector_store=store)` to avoid passing raw vectors on every call.** When `vector_store` is set, `search()` pulls vectors and metadata from the store automatically: you only need to pass the query and filter.
</Tip>
## Metadata Filtering
`MetadataFilter` supports chained conditions: all conditions are ANDed:
@@ -394,6 +406,10 @@ ns = ns_manager.get_vector_namespace("vec_0")
ns_manager.delete_namespace("tenant_a")
```
<Tip>
**Use `NamespaceManager` for multi-tenant applications.** Storing all tenants' vectors in the same collection and filtering by metadata at query time is slow and risks data leakage if a filter is accidentally omitted. Namespace isolation is both faster (smaller search space) and safer (structural isolation).
</Tip>
## Batch Operations
```python
@@ -436,6 +452,10 @@ store2.load("./vector_store_backup")
Cloud backends (Pinecone, Weaviate, Qdrant, Milvus, PgVector) manage persistence themselves. `save()`/`load()` are for the in-memory and FAISS backends only.
</Note>
<Warning>
**inmemory and faiss backends lose data on process exit without `save()`.** Call `store.save(path)` after adding vectors. Cloud backends (Pinecone, Qdrant, Weaviate, Milvus, PgVector) persist automatically.
</Warning>
## MetadataStore
`MetadataStore` indexes structured metadata and lets you query by field values without a vector:
@@ -467,6 +487,10 @@ stats = meta_store.get_stats()
# {"total_vectors": 2, "indexed_fields": 3, "field_counts": {...}}
```
<Tip>
**Update metadata without re-embedding.** `MetadataStore.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content.
</Tip>
## FAISS Index Type Reference
FAISS index type is configured by creating a `FAISSStore` directly and calling `create_index()`. Use lowercase type names:
@@ -496,6 +520,10 @@ store.create_index(index_type="pq", metric="L2", m=8)
| `hnsw` | Medium-High | Very fast | ~97–99% | Low latency, production retrieval |
| `pq` | Low | Fast | ~90–95% | Millions of vectors, memory-constrained |
<Warning>
**FAISS index type names are lowercase.** The `FAISSStore.create_index()` method expects `"flat"`, `"ivf"`, `"hnsw"`, `"pq"`: not `"Flat"`, `"IVF"`, `"HNSW"`, `"PQ"`. Uppercase values raise `ValidationError`.
</Warning>
<Note>
When using `VectorStore(backend="faiss")`, the underlying `FAISSStore` is initialised with a flat index by default. To use ivf/hnsw/pq, construct `FAISSStore` directly and call `create_index()` with the desired type.
</Note>
@@ -574,36 +602,6 @@ store.create_index(index_type="pq", metric="L2", m=8)
</Tab>
</Tabs>
## Tips and Common Pitfalls
<Warning>
**Match vector dimension to your embedding model.** The `dimension` parameter must exactly match your embedding model's output size: `BAAI/bge-small-en-v1.5` = 384, `all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768, `bge-large-en-v1.5` = 1024. A mismatch raises an error at insert time.
</Warning>
<Warning>
**FAISS index type names are lowercase.** The `FAISSStore.create_index()` method expects `"flat"`, `"ivf"`, `"hnsw"`, `"pq"`: not `"Flat"`, `"IVF"`, `"HNSW"`, `"PQ"`. Uppercase values raise `ValidationError`.
</Warning>
<Warning>
**inmemory and faiss backends lose data on process exit without `save()`.** Call `store.save(path)` after adding vectors. Cloud backends (Pinecone, Qdrant, Weaviate, Milvus, PgVector) persist automatically.
</Warning>
<Tip>
**Use `HybridSearch(vector_store=store)` to avoid passing raw vectors on every call.** When `vector_store` is set, `search()` pulls vectors and metadata from the store automatically: you only need to pass the query and filter.
</Tip>
<Tip>
**Use `add_documents()` for text, `store_vectors()` for pre-computed embeddings.** `add_documents()` auto-embeds in parallel batches. If your embeddings are already computed (e.g. from a fine-tuned model), use `store_vectors()` directly to skip re-embedding.
</Tip>
<Tip>
**Use `NamespaceManager` for multi-tenant applications.** Storing all tenants' vectors in the same collection and filtering by metadata at query time is slow and risks data leakage if a filter is accidentally omitted. Namespace isolation is both faster (smaller search space) and safer (structural isolation).
</Tip>
<Tip>
**Update metadata without re-embedding.** `MetadataStore.update_metadata(id, {...})` changes attached fields (status, tags, review date) without re-running the embedding model. Use this for state changes that don't affect semantic content.
</Tip>
<CardGroup cols={2}>
<Card title="Embeddings" icon="vector-square" href="embeddings">
Generate the vectors stored here.
docs/reference/visualization.md
+24 -26
View File
@@ -61,6 +61,10 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
</Step>
</Steps>
<Warning>
**`plotly` is required for all visualizers.** Install before use: `pip install plotly`. All visualizer methods raise `ProcessingError` if Plotly is not installed.
</Warning>
## Visualizers
<Tabs>
@@ -94,6 +98,18 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
viz.visualize_relationship_matrix(graph, output="interactive")
```
<Warning>
**Use `max_nodes` for large graphs.** Force-directed layouts become unreadable and slow above ~1,000 nodes. Filter to a subgraph before visualizing large graphs.
</Warning>
<Tip>
**HTML output is always the best starting point.** Interactive HTML lets you zoom, pan, and hover for details. Only export to PNG/SVG/PDF when embedding in a report.
</Tip>
<Tip>
**For interactive dashboards, prefer Explorer.** `KGVisualizer.visualize_network()` generates a self-contained HTML file. The Explorer CLI (`semantica-explorer`) gives a full live web app with search, filtering, path-finding, and REST API.
</Tip>
**Layout options (`layout=`):**
| Layout | Description | Best For |
@@ -148,6 +164,10 @@ Requires `plotly`: `pip install plotly`. Some exporters also need `matplotlib` o
| `umap` | Fast | Global + local structure | Large datasets, cluster discovery |
| `tsne` | Medium | Local structure | Tight cluster separation |
| `pca` | Very fast | Variance | Quick overview, linear structure |
<Tip>
**UMAP is faster than t-SNE at scale.** For embedding spaces with >5,000 points, UMAP completes in seconds; t-SNE may take minutes. Both produce good cluster separation.
</Tip>
</Tab>
<Tab title="TemporalVisualizer">
Visualize how a knowledge graph changes over time:
@@ -231,6 +251,10 @@ viz = KGVisualizer(color_scheme="vibrant")
| `light` | White background, thin edges | Publications, print |
| `colorblind` | Okabe-Ito safe palette | Accessibility |
<Tip>
**Use `color_scheme="colorblind"` in publications and dashboards.** The Okabe-Ito palette is readable for everyone, including the ~8% of readers who are red-green colorblind.
</Tip>
## Export Formats
| Format | Interactive | Scalable | Best For |
@@ -266,32 +290,6 @@ semantica-explorer --graph my_graph.json
See the [Explorer reference](explorer) for the full feature set and REST API.
## Tips and Common Pitfalls
<Warning>
**`plotly` is required for all visualizers.** Install before use: `pip install plotly`. All visualizer methods raise `ProcessingError` if Plotly is not installed.
</Warning>
<Warning>
**Use `max_nodes` for large graphs.** Force-directed layouts become unreadable and slow above ~1,000 nodes. Filter to a subgraph before visualizing large graphs.
</Warning>
<Tip>
**HTML output is always the best starting point.** Interactive HTML lets you zoom, pan, and hover for details. Only export to PNG/SVG/PDF when embedding in a report.
</Tip>
<Tip>
**Use `color_scheme="colorblind"` in publications and dashboards.** The Okabe-Ito palette is readable for everyone, including the ~8% of readers who are red-green colorblind.
</Tip>
<Tip>
**UMAP is faster than t-SNE at scale.** For embedding spaces with >5,000 points, UMAP completes in seconds; t-SNE may take minutes. Both produce good cluster separation.
</Tip>
<Tip>
**For interactive dashboards, prefer Explorer.** `KGVisualizer.visualize_network()` generates a self-contained HTML file. The Explorer CLI (`semantica-explorer`) gives a full live web app with search, filtering, path-finding, and REST API.
</Tip>
<CardGroup cols={2}>
<Card title="Knowledge Graph" icon="diagram-project" href="kg">
The graph being visualized.