chore: bump version to 0.6.0

Promotes the Unreleased changelog section (Databricks connector, SQLite
vector store, SPARQL CONSTRUCT templates, JenaStore named-graph support)
to 0.6.0 and syncs version references across pyproject.toml, __init__.py,
and docs.

.github/workflows/codeql.yml

@@ -22,7 +22,36 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v7
 
-      - name: Initialize CodeQL
+      # The CodeQL bundle download (github/codeql-action/init's "Setup CodeQL
+      # tools" step) streams a ~1GB tarball from GitHub's release CDN and
+      # does not retry on a transient connection reset (ECONNRESET) itself
+      # (github/codeql-action, unresolved as of v4 / CLI 2.26.1: the HTTP
+      # error is retryable but isn't retried internally). Since a `uses:`
+      # step can't be wrapped by a shell-level retry action, attempt init
+      # up to 3 times; each retry is a fresh download attempt with no
+      # meaningful state carried over from a failed attempt.
+      - name: Initialize CodeQL (attempt 1)
+        id: codeql-init-1
+        uses: github/codeql-action/init@v4
+        continue-on-error: true
+        with:
+          languages: python
+          queries: security-and-quality
+          config-file: .github/codeql/codeql-config.yml
+
+      - name: Initialize CodeQL (attempt 2)
+        id: codeql-init-2
+        if: steps.codeql-init-1.outcome == 'failure'
+        uses: github/codeql-action/init@v4
+        continue-on-error: true
+        with:
+          languages: python
+          queries: security-and-quality
+          config-file: .github/codeql/codeql-config.yml
+
+      - name: Initialize CodeQL (attempt 3)
+        id: codeql-init-3
+        if: steps.codeql-init-2.outcome == 'failure'
         uses: github/codeql-action/init@v4
         with:
           languages: python

CHANGELOG.md

@@ -9,6 +9,90 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-07-21
+
+### Added
+
+- **Named-graph support for `JenaStore` via `Dataset` migration** (#756, #757) by @Sameer6305 and @KaifAhmad1
+  - `JenaStore` now backs onto `rdflib.Dataset(default_union=False)` instead of `rdflib.Graph`, closing #756 and fully closing out the #754/#756 cross-backend named-graph parity effort across Blazegraph, RDF4J, and Jena
+  - `default_union=False` is explicitly set so existing `execute_sparql()`/`get_triplets()` calls that don't pass `graph=` keep seeing only the default graph, not a union across all named graphs
+  - `add_triplets()` accepts a `graph=` option: when supplied, triples are written to that named graph (4-tuple add via `Dataset.graph(uri)`); when omitted, behavior is unchanged (3-tuple add routes to the default graph)
+  - Fixed a pre-existing bug where the remote-endpoint path instantiated the read-only rdflib `SPARQLStore` instead of `SPARQLUpdateStore`, so every `add_triplets()` call against a remote Fuseki endpoint silently failed (`TypeError` swallowed, `success=True`/`added=0` returned); also fixed a constructor bug where `self.endpoint` was always `None` regardless of how `JenaStore` was called, making the remote path unreachable in practice
+  - `serialize()` now logs a warning instead of silently dropping named-graph content when the requested format (`turtle`, `xml`, `n3`, …) can only serialize the default graph; use `format="trig"` or `format="nquads"` to include all graphs
+  - `create_model()`'s `triplet_count` now documented as counting across all graphs (default + named), not just the default graph, matching the `Dataset`-wide semantics
+  - `delete_triplet()` remains scoped to the default graph only (named-graph parity for delete is an explicit follow-up, matching the maintainer's scoping of this migration to `add_triplets`); the removal is passed `self.graph.default_graph` explicitly as its context, since `Dataset.remove()` on a bare 3-tuple resolves to a wildcard context internally and would otherwise delete matching triples out of every named graph too — a follow-up fix to the initial PR #757 for a bug that had no test coverage
+  - 9 new tests covering `Dataset` construction, `default_union=False` confirmation, named-graph write isolation, `serialize()` warning behavior, and `delete_triplet()`'s default-graph scoping
+
+- **SPARQL CONSTRUCT query templates** (#752, #322, #755, #754) by @Sameer6305
+  - Added parameterized, injection-safe `CONSTRUCT` templates (`ConstructTemplate`, `ParameterDescriptor`, `ConstructTemplateRegistry`)
+  - Extended CONSTRUCT execution support from Blazegraph-only to the RDF4J and Jena backends (#755), closing #754
+    - `RDF4JStore.execute_sparql` gains a CONSTRUCT-aware path (`Accept: text/turtle`, rdflib Turtle parsing, the same `(s, p, o, metadata)` 4-tuple contract) and named-graph writes via RDF4J's REST `context` parameter
+    - `JenaStore.execute_sparql` gains the equivalent CONSTRUCT-aware path over its in-process `rdflib.Graph`
+    - `_CONSTRUCT_QUERY_RE` moved to `sparql_escaping.py` as a shared, backend-agnostic constant used by all three backends
+  - Added pipeline integration via the `construct_template` step type
+
+- **Databricks Connector (Unity Catalog + Delta Lake ingestion)** (#747) by @KaifAhmad1
+  - Added `DatabricksIngestor` (`semantica/ingest/databricks_ingestor.py`), mirroring `SnowflakeIngestor`'s structure and public API shape: a `DatabricksConnector` connection handler, a `DatabricksData` dataclass, and an optional-import guard for `databricks-sdk`/`databricks-sql-connector`
+  - Supports personal access token and OAuth M2M (service principal `client_id`/`client_secret`) authentication, configurable via constructor args or `DATABRICKS_*` environment variables
+  - `ingest_table()`/`ingest_query()` run against a SQL warehouse or cluster via `databricks-sql-connector`, with `where`/`order_by`/`limit`/`offset` support and the same identifier-escaping and unsafe-`ORDER BY` rejection as `SnowflakeIngestor`; each call closes the SQL connection it opened unless one is already open (e.g. via the `with DatabricksIngestor(...)` context manager), which reuses and closes it exactly once instead of leaking a second connection per call
+  - `get_table_schema()`, `list_catalogs()`, `list_schemas()`, and `list_tables()` introspect Unity Catalog via `databricks-sdk`'s `WorkspaceClient`, validating both catalog and schema are resolved before calling the SDK; `get_table_lineage()` calls Unity Catalog's table-lineage REST API for upstream/downstream `Table --DEPENDS_ON--> Table` dependencies, plus an opt-in `include_column_lineage=True` that resolves per-column lineage via the column-lineage API
+  - `export_as_documents()` converts ingested rows into Semantica document dicts for KG construction, matching `SnowflakeIngestor.export_as_documents()`'s shape
+  - Registered as a lazy export in `semantica.ingest` (`DatabricksIngestor`, `DatabricksData`, `DatabricksConnector`) and as the `db-databricks` optional extra (`pip install "semantica[db-databricks]"`) in `pyproject.toml`, included in `db-all`
+  - New `docs/integrations/databricks.md` page modeled on `docs/integrations/snowflake.md`, plus a `DatabricksIngestor` section and table row in `docs/reference/ingest.md` and cross-links between the two integration pages
+  - 35 unit tests in `tests/test_databricks_ingestor.py` covering both auth methods, table/query ingestion, connection lifecycle (including reuse under the context manager), pagination, unsafe `ORDER BY` rejection, catalog/schema validation, schema/catalog/table listing, table and column lineage, document export, and the missing-dependency error path, closing #747
+
+- **SQLite Vector Store Backend (`sqlite-vec`)** (#726) by @Luffy2208 and @KaifAhmad1
+  - Added `SQLiteVecStore` (`semantica/vector_store/sqlite_vec_store.py`), a disk-backed local vector store using the `sqlite-vec` extension's `vec0` virtual tables, closing #240
+  - Supports Cosine and L2 distance metrics, dynamic JSON metadata filtering, read-only mode, and an in-memory (`:memory:`) mode
+  - Registered as the `"sqlite"` backend in `VectorStore.SUPPORTED_BACKENDS`, with `db_path`/`sqlite_path` config and a `VECTOR_STORE_SQLITE_PATH` environment variable
+  - Batched `add`/`delete`/`get` and `executemany`-based `update` to avoid per-row round trips; optional `use_wal=True` enables `journal_mode=WAL` + `synchronous=NORMAL` for improved write concurrency
+  - Lazy-imports `sqlite-vec` so the dependency stays fully optional (`pip install semantica[vectorstore-sqlite]`); table names and metadata filter keys are validated against a strict identifier pattern before SQL interpolation
+  - Fixes `VectorStore.update_vectors`/`delete_vectors` to delegate to the active backend store instead of only mutating in-memory state, correcting existing behavior for all non-`inmemory` backends
+  - 25 unit and integration tests in `tests/vector_store/test_sqlite_vec_store.py` covering init, add, search, get, update, delete, read-only mode, and stats
+
+### Fixed
+
+- **`kg.ProvenanceTracker` compatibility wrapper out of sync with `ProvenanceManager`, causing 9 pre-existing test failures** (#744, #751) by @Sameer6305 and @KaifAhmad1
+  - `kg.ProvenanceTracker` was a standalone in-memory implementation that never delegated to the unified `ProvenanceManager` backend; its own test suite asserted the existence of `get_lineage`, `track_relationship`, `track_entities_batch`, `get_provenance`, and `_use_unified`, none of which were ever implemented, plus a stale `get_all_sources()` assertion expecting `"timestamp"` instead of the actual `"recorded_at"` key
+  - Rather than completing the abandoned compatibility layer, `kg.ProvenanceTracker` and its remaining supported methods (`track_entity`, `get_all_sources`, `query_recorded_between`, `revision_history`, `export_audit_log`) now emit `DeprecationWarning`s pointing callers to `semantica.provenance.ProvenanceManager`
+  - Removed/rewrote the 9 tests that only exercised the never-implemented compatibility methods to instead verify the observable behavior of the still-supported API, and corrected the stale `get_all_sources()` assertion
+  - Added the previously-missing `docs/migration/kg-provenance-tracker.md` migration guide referenced by every new deprecation warning, with a method-mapping table to `ProvenanceManager` and a before/after example, closing #744
+
+- **`ProvenanceManager.track_entity` silently overrides an explicit `parent_entity_id`/`derived_from` on re-track** (#742) by @Sameer6305
+  - `track_entity()` resolved `parent_id` via a documented precedence chain (`parent_entity_id` kwarg > `metadata["derived_from"]` > source-as-known-entity-id fallback), but the history-preservation block that runs afterward unconditionally overwrote that resolved value with an auto-generated `f"{entity_id}:v:{existing.last_updated}"` history pointer whenever the entity was being re-tracked, discarding whatever parent the caller had just explicitly supplied with no warning
+  - `track_entity()` now records whether the precedence chain already resolved an explicit parent (`parent_entity_id` kwarg, `metadata["derived_from"]`, or the source-as-known-entity-id fallback) before the history block runs, and only falls back to the auto-generated history pointer when the caller supplied no explicit parent on that call
+  - The archived history entry for the previous version is still kept reachable in `get_lineage()` via `used_entities` (BFS-traversed by `InMemoryStorage.trace_lineage()`) even when an explicit parent is supplied, so re-tracking with a new parent no longer orphans the prior version from the lineage chain; when no explicit parent is supplied, `used_entities` is left alone since `parent_entity_id` already points at the same history id, avoiding a duplicate self-reference
+  - Added `test_retrack_with_explicit_parent_overrides_history_link`, `test_retrack_without_explicit_parent_still_uses_history_link`, `test_retrack_with_derived_from_overrides_history_link`, and `test_retrack_history_reachable_via_used_entities` regression tests, closing #742
+
+- **`ProvenanceManager.get_lineage` does not link entities that share a source URL** (#735) by @KaifAhmad1
+  - `track_entity()`'s only auto-linking logic looked up `source` as if it were an existing entity's `entity_id`, so passing the same real URL/DOI as `source` for two conceptually linked entities (e.g. a document and a decision derived from it) never produced a parent link, leaving `get_lineage()` returning a chain of length 1
+  - `metadata["derived_from"]` was preserved and echoed back in the output JSON but was never consulted by any linking or traversal code, so the caller's explicit relationship was silently inert
+  - `track_entity()` now treats `metadata["derived_from"]` as an explicit parent link (unless `parent_entity_id` was already passed directly), so `InMemoryStorage.trace_lineage()`'s existing BFS over `parent_entity_id` picks it up for free
+  - `metadata["derived_from"]` is now recognized on any `collections.abc.Mapping`, not just a concrete `dict`, so e.g. `types.MappingProxyType` metadata still creates the parent link
+  - `get_lineage()`'s metadata aggregation now applies the queried entity's own metadata last so it wins over ancestor metadata on conflicting keys, matching the documented "most recent entry's metadata takes precedence" behavior — previously `trace_lineage()`'s BFS order caused ancestor metadata (now reachable via `derived_from` chains) to silently overwrite the queried entity's own values
+  - Added 9 regression/edge-case tests in `tests/provenance/test_manager.py` covering the happy path, explicit `parent_entity_id` precedence over `derived_from`, precedence over the `source`-as-known-entity-id fallback, a `derived_from` pointing at a never-tracked entity, non-string/empty-string `derived_from` values being ignored, a self-referencing `derived_from` not hanging traversal, multi-hop `derived_from` chains, metadata precedence between a queried entity and its ancestors, and non-`dict` `Mapping` metadata, closing #735
+
+- **`Reasoner.add_rule` had no deduplication, doubling rules and silently emptying `forward_chain()` on rerun** (#732) by @KaifAhmad1
+  - `add_rule()` unconditionally appended to `self.rules`, so re-running the same setup code on an existing `Reasoner` instance (e.g. re-executing a Jupyter cell) duplicated every rule; since `forward_chain()` only records a conclusion if it isn't already in `self.facts`, the second run's duplicated rules matched but produced no new results, with no error or warning
+  - `add_rule()` now compares an incoming rule's `rule_type`, `conditions`, and `conclusion` against existing rules and returns the existing `Rule` instead of appending a duplicate, keeping repeated `add_rule()` calls with the same definition idempotent
+  - Added `test_add_rule_deduplicates_identical_rule`, `test_add_rule_deduplication_is_idempotent_across_forward_chain`, and `test_add_rule_does_not_dedupe_distinct_rules` regression tests
+
+- **`InferenceResult.premises` always empty from `forward_chain`/`backward_chain`** (#739) by @Sameer6305
+  - `_match_rule()` discarded matched facts and returned only instantiated conclusions, so `ExplanationGenerator` always produced empty premises lists regardless of which facts actually satisfied a rule, closing #733
+  - `_match_rule()` now returns `(conclusion, matched_facts)` tuples; `forward_chain()` threads those facts into `InferenceResult(premises=...)`, merging premises when the same conclusion is derived more than once within a pass
+  - `_prove_goal()`'s base cases (goal already a known fact; goal matched via pattern unification) now return `premises=[goal]`/`premises=[fact]` instead of `[]`
+  - Facts are matched against a `sorted()` snapshot instead of the raw `set` so rule matching and premise selection are deterministic
+  - Added `test_forward_chaining_premises` regression test mirroring the existing backward-chaining premises test
+
+- **Missing `shacl` optional-dependency extra** (#736) by @Sameer6305
+  - `pip install semantica[shacl]` referenced no matching extra in `pyproject.toml`, so `pyshacl` was never installed despite being documented as the fix in `ontology_validator.py`'s `ImportError` message, the Explorer API, the healthcare cookbook notebook, and the changelog
+  - Added `shacl = ["pyshacl>=0.25.0"]` to `[project.optional-dependencies]` and folded `shacl` into the `all` extra
+
+- **`NodeEmbedder` `AttributeError` masked in `ContextGraph.analyze_graph_with_kg`** (#734) by @Sameer6305
+  - `analyze_graph_with_kg()` called a non-existent `NodeEmbedder.generate_embeddings()`, and the surrounding broad `except Exception` swallowed the resulting `AttributeError`, silently returning `{"error": "Graph analysis failed due to an internal error"}` from `get_causal_chain()`'s supporting analytics and `get_decision_insights()`
+  - Rewired the call site to the real `NodeEmbedder.compute_embeddings(graph_store, node_labels, relationship_types)` API, deriving `node_labels`/`relationship_types` from `self.node_type_index`/`self.edge_type_index`
+  - Added a dedicated `except AttributeError` branch that logs distinctly and re-raises, so a broken internal method call surfaces as a diagnosable error instead of being indistinguishable from a legitimately empty analysis result
+
 ---
 
 ## [0.5.1] - 2026-06-29

Dockerfile

@@ -1,5 +1,5 @@
 # syntax=docker/dockerfile:1
-FROM node:22-alpine AS frontend-builder
+FROM node:26-alpine AS frontend-builder
 
 WORKDIR /app
 COPY explorer/package*.json ./explorer/
@@ -9,7 +9,7 @@ RUN npm ci
 COPY explorer/ ./
 RUN mkdir -p /app/semantica && npm run build
 
-FROM python:3.12-slim AS runtime
+FROM python:3.14-slim AS runtime
 
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \

SECURITY.md

@@ -24,7 +24,7 @@ Security vulnerabilities should be reported privately to prevent potential explo
 
 ### 2. Report Security Issue
 
-Create a [GitHub Security Advisory](https://github.com/Hawksight-AI/semantica/security/advisories/new) or contact us through [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix.
+Create a [GitHub Security Advisory](https://github.com/semantica-agi/semantica/security/advisories/new) or contact us through [GitHub Issues](https://github.com/semantica-agi/semantica/issues) with "[SECURITY]" prefix.
 
 Include the following information:
 
@@ -156,8 +156,8 @@ We appreciate responsible disclosure. Security researchers who help us improve t
 
 For security-related questions or concerns:
 
-- **GitHub Issues**: [Create an issue](https://github.com/Hawksight-AI/semantica/issues) with "[SECURITY]" prefix
-- **GitHub Security Advisories**: [Report vulnerability](https://github.com/Hawksight-AI/semantica/security/advisories/new)
+- **GitHub Issues**: [Create an issue](https://github.com/semantica-agi/semantica/issues) with "[SECURITY]" prefix
+- **GitHub Security Advisories**: [Report vulnerability](https://github.com/semantica-agi/semantica/security/advisories/new)
 
 ## Additional Resources
 

SUPPORT.md

@@ -20,8 +20,8 @@ Start with our comprehensive documentation:
 
 **Best for**: General questions, feature discussions, and getting help
 
-- [Ask a question](https://github.com/Hawksight-AI/semantica/discussions/new?category=q-a)
-- [Browse discussions](https://github.com/Hawksight-AI/semantica/discussions)
+- [Ask a question](https://github.com/semantica-agi/semantica/discussions/new?category=q-a)
+- [Browse discussions](https://github.com/semantica-agi/semantica/discussions)
 
 #### Discord
 
@@ -33,8 +33,8 @@ Start with our comprehensive documentation:
 
 **Best for**: Bug reports and feature requests
 
-- [Report a bug](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md)
-- [Request a feature](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md)
+- [Report a bug](https://github.com/semantica-agi/semantica/issues/new?template=bug_report.md)
+- [Request a feature](https://github.com/semantica-agi/semantica/issues/new?template=feature_request.md)
 
 ### Before Asking
 
@@ -47,7 +47,7 @@ Start with our comprehensive documentation:
 
 ### Bug Reports
 
-Use our [bug report template](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md) to report bugs.
+Use our [bug report template](https://github.com/semantica-agi/semantica/issues/new?template=bug_report.md) to report bugs.
 
 Include:
 - Clear description of the bug
@@ -58,7 +58,7 @@ Include:
 
 ### Feature Requests
 
-Use our [feature request template](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md) to suggest features.
+Use our [feature request template](https://github.com/semantica-agi/semantica/issues/new?template=feature_request.md) to suggest features.
 
 Include:
 - Problem statement
@@ -71,7 +71,7 @@ Include:
 **Do NOT** create a public issue for security vulnerabilities.
 
 Instead:
-- Email: semantica-dev@users.noreply.github.com
+- Email: kaif@getsemantica.ai
 - Subject: [SECURITY] Brief description
 - See [Security Policy](SECURITY.md) for details
 
@@ -79,7 +79,7 @@ Instead:
 
 For enterprise support, custom development, or consulting:
 
-- **Email**: semantica-dev@users.noreply.github.com
+- **Email**: kaif@getsemantica.ai
 - **Subject**: [ENTERPRISE] Your request
 
 ## Response Times

docs/citation.md

@@ -17,22 +17,22 @@ icon: "quote-left"
       author  = {Hawksight AI},
       year    = {2026},
       url     = {https://github.com/semantica-agi/semantica},
-      version = {0.5.1},
+      version = {0.6.0},
       doi     = {10.5281/zenodo.XXXXXXX}
     }
     ```
   </Tab>
   <Tab title="APA">
-    Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.5.1) \[Computer software\]. https://github.com/semantica-agi/semantica
+    Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.6.0) \[Computer software\]. https://github.com/semantica-agi/semantica
   </Tab>
   <Tab title="MLA">
-    Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.5.1, GitHub, 2026, https://github.com/semantica-agi/semantica.
+    Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.6.0, GitHub, 2026, https://github.com/semantica-agi/semantica.
   </Tab>
   <Tab title="Chicago">
-    Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.5.1. GitHub, 2026. https://github.com/semantica-agi/semantica.
+    Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.6.0. GitHub, 2026. https://github.com/semantica-agi/semantica.
   </Tab>
   <Tab title="IEEE">
-    Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.5.1, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
+    Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.6.0, GitHub, 2026. \[Online\]. Available: https://github.com/semantica-agi/semantica
   </Tab>
 </Tabs>
 

docs/docs.json

@@ -103,7 +103,8 @@
             "pages": [
               "integrations/agno",
               "integrations/docling",
-              "integrations/snowflake"
+              "integrations/snowflake",
+              "integrations/databricks"
             ]
           },
           {

docs/faq.md

@@ -17,7 +17,7 @@ icon: "circle-question"
 | API key required? | Optional: pattern extraction works with no keys |
 | Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
 | Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
-| Latest version? | **v0.5.1** (June 2026) |
+| Latest version? | **v0.6.0** (July 2026) |
 | Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
 
 

docs/getting-started.md

@@ -42,7 +42,7 @@ icon: "rocket"
       Verify installation:
       ```python
       import semantica
-      print(semantica.__version__)  # 0.5.1
+      print(semantica.__version__)  # 0.6.0
       ```
     </Check>
   </Step>

docs/guides/agent-memory.md

@@ -6,6 +6,45 @@ icon: "brain"
 
 `AgentContext` maintains a persistent memory layer for LLM agents — storing observations as vector embeddings, retrieving them by semantic similarity, and optionally blending graph proximity into the ranking. Use it when your agent needs to recall past findings across sessions without re-reading source material on every restart.
 
+## What Is Agent Memory?
+
+Agent Memory provides persistent storage and intelligent retrieval of information across multiple agent sessions. `AgentContext` is the core component that orchestrates memory storage, retrieval, and management by combining three key systems:
+
+**VectorStore** handles semantic search using vector embeddings. It stores text as high-dimensional vectors and retrieves similar content through cosine similarity or other distance metrics.
+
+**ContextGraph** maintains structured knowledge as nodes (entities) and edges (relationships). This enables multi-hop traversal and graph-aware retrieval that follows connections between related entities.
+
+**AgentContext** orchestrates both components, providing a unified interface for storing memories, retrieving relevant context, and managing conversations across sessions.
+
+**Persistent memory vs stateless retrieval:** Traditional RAG systems lose context between sessions. Agent Memory persists learned information, conversation history, and accumulated knowledge across restarts, enabling long-term memory and cross-session recall.
+
+## Why Use Agent Memory?
+
+**Cross-session recall.** Agents remember previous interactions, findings, and decisions without re-processing source material after restarts.
+
+**Long-term knowledge accumulation.** Information builds up over time as agents process more documents, creating increasingly rich knowledge bases for future queries.
+
+**Conversation history.** Agents maintain context within conversations and can reference earlier parts of extended interactions or investigations.
+
+**Graph-aware retrieval.** Beyond simple semantic similarity, retrieval follows entity relationships to find connected information that pure vector search would miss.
+
+**Decision tracking.** Record decisions with full context and reasoning paths, enabling audit trails and precedent matching for similar future scenarios.
+
+## When To Use / When Not To Use
+
+**Use Agent Memory for:**
+- Long-running agents that need to accumulate knowledge over time
+- Research assistants that build understanding across multiple sessions
+- Investigation workflows where context builds incrementally
+- Systems that must remember prior interactions and decisions
+- Scenarios requiring audit trails and decision precedents
+
+**Do not use when:**
+- Building simple stateless RAG systems for one-time document queries
+- Performing one-off document searches without need for persistence
+- Running temporary experiments that don't require knowledge retention
+- Simple retrieval tasks where relationships between entities don't matter
+
 <Info>
   This guide covers the memory layer. For graph-enriched traversal and entity linking, see [Context Graphs](context-graphs). For decision accountability — recording, auditing, and causally tracing what the agent chose — see [Decision Intelligence](decision-intelligence).
 </Info>
@@ -18,11 +57,10 @@ Configure the vector store, knowledge graph, and `AgentContext` together at star
 from semantica.context import AgentContext, ContextGraph
 from semantica.vector_store import VectorStore
 
-# The FAISS index persists to disk at index_path — restart-safe
+# The VectorStore relies on explicit save()/load() for persistence
 ti_vs = VectorStore(
     backend="faiss",
     dimension=768,
-    index_path="ti_agent/memory.faiss",
 )
 
 # The ContextGraph holds entity nodes and their relationships
@@ -141,7 +179,7 @@ results = ti_agent.retrieve(
     "cloud OAuth token theft campaigns",
     max_results=10,
     use_graph=True,
-    anchor_node="APT29",      # BFS starts from this node in the knowledge graph
+    anchor_node="APT29",      # Breadth-First Search (BFS) starts from this node in the knowledge graph
     max_hops=3,
     proximity_weight=0.35,    # 65% semantic + 35% proximity — tune to your graph density
     min_score=0.1,
@@ -242,7 +280,7 @@ from semantica.llms import Groq
 
 ti_graph = ContextGraph(advanced_analytics=True, node_embeddings=True)
 ti_agent = AgentContext(
-    vector_store=VectorStore(backend="faiss", dimension=768, index_path="ti_memory.faiss"),
+    vector_store=VectorStore(backend="faiss", dimension=768),
     knowledge_graph=ti_graph,
     retention_days=365,
     max_memories=50000,
@@ -297,7 +335,7 @@ from semantica.llms import Groq
 
 soc_graph = ContextGraph()
 soc_agent = AgentContext(
-    vector_store=VectorStore(backend="faiss", dimension=768, index_path="soc_memory.faiss"),
+    vector_store=VectorStore(backend="faiss", dimension=768),
     knowledge_graph=soc_graph,
     retention_days=180,
     max_memories=100000,
@@ -370,7 +408,7 @@ from semantica.vector_store import VectorStore
 
 clinical_graph = ContextGraph(advanced_analytics=True)
 clinical_agent = AgentContext(
-    vector_store=VectorStore(backend="faiss", dimension=768, index_path="clinical.faiss"),
+    vector_store=VectorStore(backend="faiss", dimension=768),
     knowledge_graph=clinical_graph,
     retention_days=3650,      # 10-year clinical record retention
     max_memories=500000,
@@ -446,7 +484,7 @@ from semantica.vector_store import VectorStore
 
 credit_graph = ContextGraph(advanced_analytics=True)
 credit_agent = AgentContext(
-    vector_store=VectorStore(backend="faiss", dimension=768, index_path="credit.faiss"),
+    vector_store=VectorStore(backend="faiss", dimension=768),
     knowledge_graph=credit_graph,
     retention_days=2555,      # 7-year regulatory retention
     max_memories=1000000,
@@ -546,7 +584,7 @@ from semantica.vector_store import VectorStore
 
 # Create a fresh context with matching configuration
 ti_agent_restored = AgentContext(
-    vector_store=VectorStore(backend="faiss", dimension=768, index_path="ti_memory.faiss"),
+    vector_store=VectorStore(backend="faiss", dimension=768),
     knowledge_graph=ContextGraph(advanced_analytics=True),
     retention_days=365,
     decision_tracking=True,
@@ -605,6 +643,18 @@ s = ti_agent.stats()
 print("Total memories: {}".format(s.get("total_items", 0)))
 ```
 
+## Common Pitfalls
+
+**Forgetting to persist memory before shutdown.** Agent Memory is stored in memory during execution. Without calling `save()` before process termination, all accumulated memories, graph relationships, and conversations are lost.
+
+**Using the same conversation namespace for unrelated tasks.** Conversation IDs should scope related interactions. Using a single conversation for multiple unrelated investigations pollutes retrieval results and makes context less focused.
+
+**Storing excessive low-value information.** Not every observation needs permanent storage. Focus on storing insights, decisions, and significant findings rather than verbose raw logs or temporary calculations.
+
+**Using Agent Memory when simple retrieval would be sufficient.** For one-time document lookups or stateless queries, traditional retrieval is simpler and more efficient than setting up persistent memory infrastructure.
+
+**Retrieving too much context and increasing latency.** Large `max_results`, high `max_hops`, or broad queries can retrieve excessive context, increasing LLM token usage and response latency. Start with focused retrieval parameters.
+
 ## Related Guides
 
 - [Context Graphs](context-graphs) — How the underlying `ContextGraph` stores entity nodes and decision nodes; temporal interval reasoning; deduplication before node insertion; ontology from graph.

docs/guides/change-management.md

@@ -4,12 +4,99 @@ description: "Snapshot, version, diff, and migrate knowledge graphs and ontologi
 icon: "clock-rotate-left"
 ---
 
-Knowledge graphs change constantly — threat actors get re-attributed, CVE scores update when exploits drop, clinical trial endpoints shift between phases. `TemporalVersionManager` gives your graph a verifiable history: named snapshots before every consequential change, diffs between any two states, one-call rollback, and SHA-256 checksum verification before publishing downstream.
+## What Is Change Management & Versioning?
+
+Knowledge graphs change constantly. `TemporalVersionManager` gives your graph a verifiable history by capturing complete state snapshots at specific points in time. It allows you to take named snapshots before consequential changes, generate detailed diffs between any two states, roll back to previous versions with a single call, and verify SHA-256 checksums before publishing data downstream.
+
+## Storage Behavior
+
+Pass `storage_path`, e.g. `TemporalVersionManager(storage_path="versions.db")`, to persist snapshots to a SQLite database on disk. Omit `storage_path` and it defaults to an in-memory store that vanishes when your script finishes.
+
+## Why Use Change Management?
+
+Change Management acts as your safety net and audit trail. Use it to:
+- **Safeguard Ingestion**: Take a snapshot before a large batch ingestion so you can instantly roll back if the data is corrupted.
+- **Audit Trails**: Maintain a verifiable log of when a change occurred, who authorized it, and exactly what nodes/edges were modified.
+- **Release Gating**: Compare staging and production graphs and verify checksums before signing off on a release.
+
+## Which Tool Do I Need?
+
+Semantica offers multiple tracking features. It is critical to choose the right one:
+- **Change Management** (this guide): Use for **whole-graph snapshots**, state diffs, and full rollbacks.
+- **Provenance**: Use for granular **source and lineage tracking**. It answers *"Which specific document did this node come from?"*
+- **Agent Memory**: Use for **conversational and context state**. It answers *"What decisions did the AI agent make during this session?"*
+
+## When To Use / When Not To Use
+
+- **When to Use**: You have critical checkpoints (like daily feeds, partner merges, or regulatory submissions) where you need to freeze the entire state of the graph and potentially revert it.
+- **When NOT to Use**: You have a massive, multi-million node graph and want to track every minor edit. Because `TemporalVersionManager` snapshots the entire graph dictionary, snapshotting huge graphs too frequently will cause severe storage bloat. Use Provenance for granular tracking instead.
 
 <Info>
-  `TemporalVersionManager` integrates with `AgentContext.flush_checkpoint()` — agent checkpoints and manual snapshots share the same storage format, so diffs work across both.
+  `TemporalVersionManager` integrates directly with `AgentContext.flush_checkpoint()` — agent checkpoints and manual snapshots share the same storage format, allowing diffs across both automated and manual workflows.
 </Info>
 
+---
+
+## Typical Workflow
+
+A standard change management cycle follows this progression:
+
+1. **Snapshot**: Capture the baseline graph state.
+2. **Modify**: Run your ingestion, mutations, or analysis.
+3. **Compare**: Generate a diff to see what changed.
+4. **Verify**: Check the SHA-256 hash to ensure data integrity.
+5. **Tag**: Apply a human-readable tag (e.g., `approved`).
+6. **Rollback**: Revert the graph state if the modifications were incorrect.
+
+---
+
+## Universal Example: Employee Profile Update
+
+Let's look at a universally understood example: tracking an employee's department transfer.
+
+```python
+from semantica.change_management import TemporalVersionManager
+from semantica.context import ContextGraph
+
+# 1. Setup Graph and Version Manager
+graph = ContextGraph()
+graph.add_node("emp-101", "Employee", "Alice")
+graph.add_node("dept-hr", "Department", "Human Resources")
+graph.add_edge("emp-101", "dept-hr", "works_in")
+
+# SQLite persistence is enabled because we provided a storage_path
+vm = TemporalVersionManager(storage_path="hr_versions.db")
+
+# 2. Snapshot the baseline
+snap_v1 = vm.create_snapshot(
+    graph         = graph.to_dict(),
+    version_label = "v1_baseline",
+    author        = "hr_system@example.com",
+    description   = "Initial employee graph",
+)
+
+# 3. Modify the graph (Transfer Alice to Engineering)
+graph.add_node("dept-eng", "Department", "Engineering")
+graph.add_edge("emp-101", "dept-eng", "works_in")
+
+# 4. Snapshot the post-change state
+snap_v2 = vm.create_snapshot(
+    graph         = graph.to_dict(),
+    version_label = "v2_transfer",
+    author        = "hr_admin@example.com",
+    description   = "Alice transferred to Engineering",
+)
+
+# 5. Compare versions
+diff = vm.compare_versions("v1_baseline", "v2_transfer")
+print("Nodes added:", diff["summary"]["nodes_added"])  # 1 (Engineering)
+print("Edges added:", diff["summary"]["edges_added"])  # 1 (works_in Eng)
+```
+
+Now let's explore these capabilities in more depth using domain-specific scenarios.
+
+---
+
 ## Creating Snapshots
 
 Take a snapshot before any consequential change: an ingestion sweep, a partner feed merge, or an automated enrichment run.
@@ -28,7 +115,7 @@ vm = TemporalVersionManager(storage_path="cti_versions.db")
 snap_pre = vm.create_snapshot(
     graph         = graph.to_dict(),
     version_label = "q3_2025_baseline",
-    author        = "analyst_zhang",
+    author        = "analyst_zhang@example.com",
     description   = "CTI baseline before Q3 OSINT sweep",
 )
 
@@ -50,7 +137,7 @@ graph.add_edge("apt40", "cve-2024-21412", "exploits", weight=0.88)
 snap_post = vm.create_snapshot(
     graph         = graph.to_dict(),
     version_label = "q3_2025_post_nvd_sweep",
-    author        = "osint_pipeline",
+    author        = "osint_pipeline@example.com",
     description   = "After NVD weekly sweep — 2025-07-14",
 )
 ```
@@ -108,7 +195,7 @@ vm.restore_snapshot(
 vm.create_snapshot(
     graph         = graph.to_dict(),
     version_label = "q3_2025_rollback",
-    author        = "analyst_zhang",
+    author        = "analyst_zhang@example.com",
     description   = "Rolled back to baseline after corrupted OSINT batch",
 )
 ```
@@ -146,14 +233,14 @@ Sample output:
 Graph Change Log
 ============================================================
 
-[2025-07-01] q3_2025_baseline  (by analyst_zhang)
+[2025-07-01] q3_2025_baseline  (by analyst_zhang@example.com)
   CTI baseline before Q3 OSINT sweep
 
-[2025-07-14] q3_2025_post_nvd_sweep  (by osint_pipeline)
+[2025-07-14] q3_2025_post_nvd_sweep  (by osint_pipeline@example.com)
   After NVD weekly sweep — 2025-07-14
   Changes: +2 nodes  -0 nodes  +1 edges  -0 edges
 
-[2025-07-14] q3_2025_rollback  (by analyst_zhang)
+[2025-07-14] q3_2025_rollback  (by analyst_zhang@example.com)
   Rolled back to baseline after corrupted OSINT batch
   Changes: -2 nodes  +0 nodes  -1 edges  +0 edges
 ```
@@ -218,6 +305,18 @@ print("Decisions added    :", len(diff["decisions_added"]))
 print("Relationships added:", len(diff["relationships_added"]))
 ```
 
+---
+
+## Common Pitfalls
+
+- **Snapshotting huge graphs too frequently**: `TemporalVersionManager` snapshots the entire graph structure. Doing this on every minor edit for a massive graph will cause severe storage bloat. Use it for milestone gating, not event sourcing.
+- **Forgetting `attach_to_graph` before mutation tracking**: If you want to use `get_node_history()`, you must call `vm.attach_to_graph(graph)` *before* any mutations happen. Otherwise, the events will not be captured.
+- **Confusing provenance with versioning**: Do not use version snapshots to answer "Where did this specific node's data come from?". That is the role of the Provenance module. Versioning tracks the state of the *entire* graph at a point in time.
+- **Forgetting rollback confirmation requirements**: Calling `restore_snapshot` in automated scripts will raise a `ProcessingError` and crash your pipeline unless you explicitly pass `require_confirmation=False`.
+- **Storage growth from excessive snapshots**: Over time, SQLite databases can grow large if you never prune old snapshots or if you snapshot unnecessarily.
+
+---
+
 ## Domain Examples
 
 <Tabs>
@@ -236,7 +335,7 @@ today = datetime.date.today().isoformat()
 snap_pre = vm.create_snapshot(
     graph         = graph.to_dict(),
     version_label = f"pre_nvd_{today}",
-    author        = "osint_pipeline",
+    author        = "osint_pipeline@example.com",
     description   = "CTI baseline before NVD sweep",
 )
 
@@ -248,7 +347,7 @@ graph.add_edge("apt29-q3-cluster", "cve-2025-1337", "weaponizes", weight=0.91)
 snap_post = vm.create_snapshot(
     graph         = graph.to_dict(),
     version_label = f"post_nvd_{today}",
-    author        = "osint_pipeline",
+    author        = "osint_pipeline@example.com",
     description   = "After NVD sweep",
 )
 
@@ -283,7 +382,7 @@ graph.add_edge("attacker-ip", "wkstn-047", "initial_access", weight=0.95)
 vm.create_snapshot(
     graph         = graph.to_dict(),
     version_label = "ir042_t0_triage",
-    author        = "analyst_chen",
+    author        = "analyst_chen@example.com",
     description   = "T+0 — one compromised host identified",
 )
 
@@ -296,7 +395,7 @@ graph.add_edge("svc-backup", "dc01",       "lateral_move",     weight=0.82)
 vm.create_snapshot(
     graph         = graph.to_dict(),
     version_label = "ir042_t2h_lateral",
-    author        = "analyst_chen",
+    author        = "analyst_chen@example.com",
     description   = "T+2h — lateral movement to DC01 via stolen SVC-BACKUP",
 )
 
@@ -332,11 +431,11 @@ vm = TemporalVersionManager(storage_path="trial_xr401.db")
 
 vm.create_snapshot(
     graph=graph_ph2.to_dict(), version_label="phase_ii_v1.0",
-    author="clinical_data_team", description="Phase II — ORR primary, NSCLC",
+    author="clinical_data_team@example.com", description="Phase II — ORR primary, NSCLC",
 )
 vm.create_snapshot(
     graph=graph_ph3.to_dict(), version_label="phase_iii_v2.0",
-    author="clinical_data_team", description="Phase III — PFS co-primary, Docetaxel added",
+    author="clinical_data_team@example.com", description="Phase III — PFS co-primary, Docetaxel added",
 )
 
 diff = vm.compare_versions("phase_ii_v1.0", "phase_iii_v2.0")
@@ -369,7 +468,7 @@ vm = TemporalVersionManager(storage_path="credit_risk_versions.db")
 
 vm.create_snapshot(
     graph=graph.to_dict(), version_label="basel_v1.0",
-    author="risk_model_team", description="Basel III CRE20 initial graph",
+    author="risk_model_team@example.com", description="Basel III CRE20 initial graph",
 )
 
 # Regulatory update — DSCR becomes mandatory
@@ -378,7 +477,7 @@ graph.add_edge("regulation-cre20", "metric-dscr", "requires", weight=1.0)
 
 vm.create_snapshot(
     graph=graph.to_dict(), version_label="basel_v1.1",
-    author="risk_model_team", description="DSCR added per EBA GL 2020/06",
+    author="risk_model_team@example.com", description="DSCR added per EBA GL 2020/06",
 )
 
 diff = vm.compare_versions("basel_v1.0", "basel_v1.1")

docs/guides/conflict-resolution.md

@@ -10,9 +10,152 @@ icon: "code-merge"
 Run conflict detection after deduplication and before SHACL validation. Deduplication removes duplicate nodes; conflict resolution reconciles disagreeing property values on the same canonical entity. Running them out of order — detecting conflicts before deduplication — will produce spurious conflicts between entities that should have been merged first.
 </Info>
 
-## Detecting the disagreement
+## What Is Conflict Resolution?
 
-Start by loading your multi-source records for the same entity. `ConflictDetector` groups them by entity ID, then compares the values each source reports for a given property. Any entity where two or more sources report different values for the same property produces a `Conflict` object.
+When you merge data from multiple sources, the same real-world entity — a customer, a product, a threat actor, a drug compound — often appears with contradictory property values. One database says a customer's email is `alice@example.com`; another says `alice.smith@example.com`. One security feed rates a CVE at 10.0; two others rate it 9.1 and 9.5.
+
+**Conflict resolution** is the systematic process of deciding which value is most trustworthy and recording that decision with evidence, so the canonical entity ends up with one defensible, auditable value per property.
+
+### Key Concepts
+
+**Canonical entity** — The single authoritative record for a real-world thing. After deduplication, each entity has exactly one canonical node in your graph. Conflict resolution determines which property values belong on that node.
+
+**Conflicting values** — Two or more different values asserted for the same property on the same canonical entity, each reported by a different source.
+
+**Credibility score** — A number between 0.0 and 1.0 you attach to each source record, indicating how reliable that source is. A government registry might carry 0.99; a scraped blog might carry 0.30. You supply these; Semantica uses them during `CREDIBILITY_WEIGHTED` resolution.
+
+**Confidence score** — A number between 0.0 and 1.0 the resolver *computes* after resolution, reflecting how certain the outcome is. A unanimous vote produces high confidence; a close split among equally credible sources produces lower confidence. This appears on `ResolutionResult.confidence` and should be read as a signal, not a guarantee that the resolved value is correct.
+
+**Resolution strategy** — The rule for picking the winning value: majority vote, credibility-weighted average, latest timestamp, and so on. See [Resolution strategies at a glance](#resolution-strategies-at-a-glance) for the full list.
+
+**Audit trail** — The complete record of every resolution decision: conflict ID, strategy used, resolved value, sources consulted, and confidence score. Returned by `resolver.get_resolution_history()`.
+
+**Provenance-aware resolution** — Resolution that records not just the winning value but which source it came from. Every `ResolutionResult` carries a `sources_used` field, so you can always trace a canonical value back to its origin — critical in regulated environments.
+
+## Why Use Conflict Resolution?
+
+- **Multi-source pipelines always produce disagreements.** Differences in update cadence, data-entry conventions, and source reliability are unavoidable. Without an explicit resolution step, you silently favor one source over another with no record of the choice.
+- **You get a defensible, auditable decision log.** Compliance teams, auditors, and domain experts need to know which source won and why. The audit trail provides exactly that.
+- **Easy cases are automated; hard cases are escalated.** Routine disagreements — slightly different name spellings, stale timestamps — are resolved algorithmically. Genuinely ambiguous cases — competing legal classifications, different clinical endpoints — are flagged for expert review without blocking the rest of the pipeline.
+
+## When To Use / When Not To Use
+
+**Use conflict resolution when:**
+- You are merging two or more independent sources for the same entity.
+- Sources disagree on property values and you need a single canonical value.
+- You need an auditable record of every resolution decision.
+- Some conflicts require domain-expert review before they can be resolved.
+
+**Skip conflict resolution when:**
+- **A single authoritative source already exists.** If one system is always correct for a given property, read from it directly. Adding resolution machinery around a single source creates complexity without benefit.
+- **All sources are always in agreement.** Verify this empirically before skipping; silent disagreements are common in practice.
+- **You want to preserve all conflicting values.** If retaining every source's assertion matters more than picking one, model provenance directly in your graph schema instead of resolving to one winner.
+
+## Typical Workflow
+
+```mermaid
+flowchart TD
+    A[Raw Sources] --> B[Deduplication]
+    B --> C[Conflict Detection]
+    C --> D{Auto-resolvable?}
+    D -- Yes --> E[Apply Resolution Strategy]
+    D -- No --> F[Expert Review Queue]
+    E --> G[Persist Canonical Values]
+    F --> G
+    G --> H[SHACL Validation]
+```
+
+1. **Deduplication** — Merge duplicate nodes so each entity has exactly one canonical record. Conflict resolution operates on a single canonical entity; you must identify it before comparing what different sources say about it. See [Deduplication](deduplication).
+2. **Conflict Detection** — Call `detect_entity_conflicts()` to surface all property disagreements at once, or `detect_value_conflicts()` to target a specific property.
+3. **Resolution** — For each conflict, apply a strategy (`CREDIBILITY_WEIGHTED`, `MOST_RECENT`, `VOTING`, etc.) or route it for expert review (`EXPERT_REVIEW`).
+4. **Persist Canonical Values** — Write resolved values back to your canonical entities or graph store. See [Persisting resolved values](#persisting-resolved-values).
+5. **SHACL Validation** — Enforce structural constraints on the resolved graph to confirm it satisfies your ontology. See [SHACL Validation](shacl-validation).
+
+## Quick Start: A Beginner Example
+
+Before diving into domain-specific scenarios, here is the shortest path through the API. Three systems — a CRM, an ERP, and an LDAP directory — hold slightly different contact details for the same customer. Two of the three agree that the canonical email is `alice.smith@example.com`; the CRM has an older value.
+
+```python
+from semantica.conflicts import ConflictDetector, ConflictResolver, ResolutionStrategy
+
+# Same customer, three sources — only email disagrees
+customer_records = [
+    {"id": "cust-001", "source": "crm",  "email": "alice@example.com",       "phone": "+1-555-0100"},
+    {"id": "cust-001", "source": "erp",  "email": "alice.smith@example.com", "phone": "+1-555-0100"},
+    {"id": "cust-001", "source": "ldap", "email": "alice.smith@example.com", "phone": "+1-555-0100"},
+]
+
+# Step 1: Detect all property conflicts at once — no need to name each property
+detector = ConflictDetector()
+conflicts = detector.detect_entity_conflicts(customer_records)
+
+print(f"Conflicts found: {len(conflicts)}")
+for c in conflicts:
+    print(f"  Property : {c.property_name}")
+    print(f"  Values   : {c.conflicting_values}")
+    print(f"  Severity : {c.severity}")
+
+# Step 2: Resolve — two out of three sources agree, so majority vote wins
+resolver = ConflictResolver()
+results = resolver.resolve_conflicts(conflicts, strategy=ResolutionStrategy.VOTING)
+
+for r in results:
+    print(f"\n[{'RESOLVED' if r.resolved else 'REVIEW'}] {r.conflict_id}")
+    print(f"  Resolved value : {r.resolved_value}")
+    print(f"  Strategy       : {r.resolution_strategy}")
+    print(f"  Confidence     : {r.confidence:.0%}")
+    print(f"  Sources used   : {r.sources_used}")
+```
+
+```text
+Conflicts found: 1
+  Property : email
+  Values   : ['alice@example.com', 'alice.smith@example.com', 'alice.smith@example.com']
+  Severity : medium
+
+[RESOLVED] cust-001_email_conflict
+  Resolved value : alice.smith@example.com
+  Strategy       : voting
+  Confidence     : 67%
+  Sources used   : ['crm', 'erp', 'ldap']
+```
+
+`detect_entity_conflicts()` scanned both `email` and `phone` automatically — you did not name them. Because `phone` is identical across all three records, no conflict was detected for it. The email disagreement resolves to `alice.smith@example.com` because two of three sources agree on that value.
+
+When every conflict in a batch should use the same strategy, pass `strategy=` directly to `resolve_conflicts()`. Use `set_resolution_rule()` when different entity-property pairs need different strategies — explained in [Setting per-property resolution rules](#setting-per-property-resolution-rules).
+
+## Detecting Conflicts
+
+`ConflictDetector` provides three methods. Choose the one that fits your situation:
+
+| Method | What it scans | When to use |
+| :--- | :--- | :--- |
+| `detect_entity_conflicts(entities)` | Every property on each entity at once | First pass; you do not know in advance which properties conflict |
+| `detect_value_conflicts(entities, property_name)` | One named property across all entities | Targeted check for a known hot-spot property |
+| `detect_relationship_conflicts(relationships)` | Edge types between the same node pair | Structural disagreements in graph edges |
+
+### Scanning All Properties at Once — `detect_entity_conflicts`
+
+`detect_entity_conflicts()` is the recommended starting point for a new pipeline. It inspects every property found on your entity records and returns a single flat list of all conflicts — without you having to enumerate properties in advance.
+
+```python
+detector = ConflictDetector()
+all_conflicts = detector.detect_entity_conflicts(records)
+# Returns every conflict across every property in one call
+```
+
+If you have registered conflict fields for a specific entity type, pass `entity_type` to limit detection to those fields:
+
+```python
+# Limit detection to fields registered for this entity type
+all_conflicts = detector.detect_entity_conflicts(records, entity_type="vulnerability")
+```
+
+Without `entity_type`, the detector checks every key found on your entity dicts (excluding bookkeeping fields such as `id`, `source`, and `metadata`). Start here to get a complete picture, then decide which conflicts need which resolution strategy.
+
+### Scanning a Specific Property — `detect_value_conflicts`
+
+Use `detect_value_conflicts()` when you already know which property to check, or when you want to apply different detection logic to each property. `ConflictDetector` groups the records by entity ID, then compares each source's value for that property. Any entity where two or more sources report different values produces a `Conflict` object.
 
 ```python
 from semantica.conflicts import ConflictDetector, ConflictResolver, ResolutionStrategy
@@ -25,7 +168,7 @@ cve_records = [
         "cvss_score": 10.0,
         "exploit_status": "unconfirmed",
         "vector": "AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H",
-        "credibility_score": 0.98,
+        "metadata": {"timestamp": "2024-04-11T12:00:00Z"},
     },
     {
         "id": "cve-2024-3400",
@@ -33,7 +176,7 @@ cve_records = [
         "cvss_score": 9.1,
         "exploit_status": "in_wild",
         "vector": "AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
-        "credibility_score": 0.91,
+        "metadata": {"timestamp": "2024-04-12T15:30:00Z"},
     },
     {
         "id": "cve-2024-3400",
@@ -41,7 +184,7 @@ cve_records = [
         "cvss_score": 9.5,
         "exploit_status": "in_wild",
         "vector": "AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H",
-        "credibility_score": 0.87,
+        "metadata": {"timestamp": "2024-04-12T12:00:00Z"},
     },
 ]
 
@@ -74,20 +217,40 @@ Conflict: cve-2024-3400_cvss_score_conflict
   Values   : [10.0, 9.1, 9.5]
   Severity : medium
   Sources  : ['nvd', 'commercial_feed', 'vendor_paloalto']
-  Action   : Compare source documents and use most recent or authoritative source
+  Action   : Multiple conflicting values detected. Manual review recommended.
 ```
 
 Each `Conflict` captures the full picture: which entity, which property, every disagreeing value, and which source reported each. This is already enough to build a review queue — but the goal is to resolve these automatically according to rules you set.
 
-## Setting per-property resolution rules
+## Setting Per-Property Resolution Rules
 
-The key method is `set_resolution_rule(entity_id, property_name, strategy)`. It takes three arguments: which entity, which property, and which `ResolutionStrategy` to apply when that combination appears in a conflict. Rules are stored in the resolver and automatically applied when you call `resolve_conflicts()` without passing an explicit strategy.
+`set_resolution_rule(entity_id, property_name, strategy)` registers a strategy for a specific entity-property combination. The resolver stores the rule under the key `entity_id.property_name` and applies it automatically when you call `resolve_conflicts()`.
+
+Because rules are keyed by both entity ID and property name, `set_resolution_rule()` is entity-specific. There is no wildcard that applies a rule to all entities or all properties at once.
+
+**When to use `set_resolution_rule()`:** Use it when different entity-property combinations need different strategies. For example, an entity's `legal_name` might use `CREDIBILITY_WEIGHTED` while its `last_updated` uses `MOST_RECENT`. Registering a rule per combination lets the single `resolve_conflicts()` call handle all of them correctly in one pass.
+
+**When to pass `strategy=` directly to `resolve_conflicts()`:** If every conflict in a batch should use the same strategy, pass it directly to `resolve_conflicts()` instead of registering a rule for each entity-property pair:
+
+```python
+# Same strategy for every conflict — no per-property rules needed
+results = resolver.resolve_conflicts(all_conflicts, strategy=ResolutionStrategy.CREDIBILITY_WEIGHTED)
+```
+
+This is cleaner than calling `set_resolution_rule()` in a loop over every entity just to apply the same strategy everywhere.
+
+**Per-property rules for the CVE example:**
 
 ```python
 resolver = ConflictResolver()
 
+# Register source credibility scores so CREDIBILITY_WEIGHTED can use them
+resolver.source_tracker.set_source_credibility("nvd", 0.98)
+resolver.source_tracker.set_source_credibility("commercial_feed", 0.91)
+resolver.source_tracker.set_source_credibility("vendor_paloalto", 0.87)
+
 # For this CVE, NVD is the most authoritative source on scoring.
-# CREDIBILITY_WEIGHTED will use the credibility_score field on each source record
+# CREDIBILITY_WEIGHTED uses the registered source credibility 
 # to weight the vote — NVD at 0.98 will dominate over the commercial feed at 0.91.
 resolver.set_resolution_rule(
     "cve-2024-3400",
@@ -108,9 +271,9 @@ resolver.set_resolution_rule(
 
 You can set rules before or after detection — the resolver applies them lazily when `resolve_conflicts()` is called.
 
-## Resolving the batch
+## Resolving the Batch
 
-Pass all detected conflicts to `resolve_conflicts()`. For each conflict, the resolver looks up whether a property-specific rule is set for that entity and property combination. If one is found, it applies that strategy. If none is set, it falls back to the default strategy (voting, unless you override it in the constructor).
+Pass all detected conflicts to `resolve_conflicts()`. For each conflict, the resolver looks up whether a rule is registered for that entity-property combination. If one is found, it applies that strategy. If none is set, it falls back to the default strategy (voting, unless you override it in the constructor).
 
 ```python
 all_conflicts = score_conflicts + exploit_conflicts
@@ -132,7 +295,7 @@ for r in results:
 [RESOLVED] cve-2024-3400_cvss_score_conflict
   Resolved value : 10.0
   Strategy used  : credibility_weighted
-  Confidence     : 72%
+  Confidence     : 36%
   Sources used   : ['nvd', 'commercial_feed', 'vendor_paloalto']
   Notes          : Resolved by credibility-weighted voting (weight: 0.98)
 
@@ -146,7 +309,7 @@ for r in results:
 
 NVD wins the CVSS score — its credibility weight (0.98) edges out the commercial feed (0.91) and the vendor (0.87), so 10.0 becomes the canonical score. The exploitation status resolves to `in_wild` — the commercial feed and vendor advisory are both more recent than NVD's initial triage, and both report active exploitation.
 
-## Handling conflicts that need human judgment
+## Handling Conflicts That Need Human Judgment
 
 Not every conflict can be auto-resolved. A disagreement about the legal classification of a financial instrument, or about a patient's current medication list, is too consequential to resolve by algorithm. Flag these for review without blocking the rest of the batch:
 
@@ -156,14 +319,11 @@ from semantica.conflicts import ConflictDetector, ConflictResolver, ResolutionSt
 # Drug trial data: efficacy agreed, primary endpoint disputed
 trial_records = [
     {"id": "dapagliflozin", "source": "declare_timi58",
-     "primary_endpoint": "MACE",               "hba1c_reduction_pct": 0.54,
-     "credibility_score": 0.92},
+     "primary_endpoint": "MACE",               "hba1c_reduction_pct": 0.54},
     {"id": "dapagliflozin", "source": "dapa_hf",
-     "primary_endpoint": "HF_hospitalization", "hba1c_reduction_pct": 0.48,
-     "credibility_score": 0.95},
+     "primary_endpoint": "HF_hospitalization", "hba1c_reduction_pct": 0.48},
     {"id": "dapagliflozin", "source": "meta_analysis",
-     "primary_endpoint": "HbA1c_reduction",    "hba1c_reduction_pct": 0.52,
-     "credibility_score": 0.88},
+     "primary_endpoint": "HbA1c_reduction",    "hba1c_reduction_pct": 0.52},
 ]
 
 detector = ConflictDetector()
@@ -172,6 +332,11 @@ endpoint_conflicts  = detector.detect_value_conflicts(trial_records, "primary_en
 
 resolver = ConflictResolver()
 
+# Register source credibility scores
+resolver.source_tracker.set_source_credibility("declare_timi58", 0.92)
+resolver.source_tracker.set_source_credibility("dapa_hf", 0.95)
+resolver.source_tracker.set_source_credibility("meta_analysis", 0.88)
+
 # Efficacy: credibility-weighted across trials — the meta-analysis (0.88) and
 # the two RCTs (0.92, 0.95) will produce a weighted resolution.
 resolver.set_resolution_rule(
@@ -214,7 +379,38 @@ Expert review : 1   # primary_endpoint — EXPERT_REVIEW means resolved=False
 
 `EXPERT_REVIEW` sets `resolved=False` on the result. The conflict stays in the graph unresolved, the metadata field carries `requires_expert_review: True`, and the review queue JSON gives your clinical team exactly what they need to make the call.
 
-## Reviewing the full audit trail
+## Persisting Resolved Values
+
+`resolve_conflicts()` returns `ResolutionResult` objects — it does not automatically write resolved values back to your graph or entity store. That step is yours to implement using whatever storage layer your pipeline uses.
+
+The most direct approach is to pair each `ResolutionResult` with its original `Conflict` object — the two lists are returned in the same order — and write the winning value onto your canonical entity:
+
+```python
+# canonical_entity is your authoritative record — a dict, graph node, database row, etc.
+canonical_entity = {"id": "cve-2024-3400", "cvss_score": None, "exploit_status": None}
+
+for conflict, result in zip(all_conflicts, results):
+    if result.resolved:
+        canonical_entity[conflict.property_name] = result.resolved_value
+        # Log provenance: record which source this value came from
+        print(f"  {conflict.property_name} = {result.resolved_value} "
+              f"(from {result.sources_used}, confidence {result.confidence:.0%})")
+
+# Persist canonical_entity to your graph store, database, or downstream system.
+```
+
+```text
+  cvss_score = 10.0 (from ['nvd', 'commercial_feed', 'vendor_paloalto'], confidence 36%)
+  exploit_status = in_wild (from ['commercial_feed'], confidence 80%)
+```
+
+A few things to keep in mind:
+
+- **Conflicts with `resolved=False`** — flagged for expert or manual review — should not be written to the canonical record until a human has made the call. Keep them in the review queue.
+- **Confidence is a signal, not a guarantee.** A 72% confidence score means the resolver had reasonable but not unanimous evidence for its decision. Treat low-confidence results with additional scrutiny before writing them to production.
+- **Track provenance.** `result.sources_used` tells you which source's value won. Store this alongside the canonical value if your compliance requirements demand a full evidence chain.
+
+## Reviewing the Full Audit Trail
 
 After a resolution run, `get_resolution_history()` returns every decision made since the resolver was instantiated. This is your compliance log:
 
@@ -238,14 +434,14 @@ report = detector.get_conflict_report()
 print(f"Total conflicts detected  : {report['total_conflicts']}")
 print(f"By type                   : {report['by_type']}")
 print(f"By severity               : {report['by_severity']}")
-# Total conflicts detected  : 2
-# By type                   : {'value_conflict': 2}
-# By severity               : {'medium': 2}
+# Total conflicts detected  : 6
+# By type                   : {'value_conflict': 6}
+# By severity               : {'medium': 6}
 ```
 
 The report aggregates every conflict the detector has seen across its lifetime — useful for pipeline monitoring and for identifying which entity types or data sources generate the most disagreements.
 
-## Detecting relationship conflicts
+## Detecting Relationship Conflicts
 
 Value conflicts live on properties. Relationship conflicts live on edges — two sources asserting contradictory connections between the same node pair:
 
@@ -267,7 +463,7 @@ for c in rel_conflicts:
 
 Relationship conflicts typically require expert review rather than voting, because conflicting edge types often reflect genuinely different intelligence assessments rather than data entry errors.
 
-## Domain examples
+## Domain Examples
 
 <Tabs>
 
@@ -282,11 +478,11 @@ from semantica.conflicts import ConflictDetector, ConflictResolver, ResolutionSt
 
 actor_profiles = [
     {"id": "apt29", "source": "mandiant",    "nation_state": "Russia",
-     "first_seen": "2008", "credibility_score": 0.95},
+     "first_seen": "2008"},
     {"id": "apt29", "source": "crowdstrike", "nation_state": "Russia",
-     "first_seen": "2009", "credibility_score": 0.92},
+     "first_seen": "2009"},
     {"id": "apt29", "source": "oss_blog",    "nation_state": "China",  # wrong
-     "first_seen": "2015", "credibility_score": 0.30},
+     "first_seen": "2015"},
 ]
 
 detector = ConflictDetector()
@@ -294,14 +490,18 @@ nation_conflicts     = detector.detect_value_conflicts(actor_profiles, "nation_s
 first_seen_conflicts = detector.detect_value_conflicts(actor_profiles, "first_seen")
 
 resolver = ConflictResolver()
+resolver.source_tracker.set_source_credibility("mandiant", 0.95)
+resolver.source_tracker.set_source_credibility("crowdstrike", 0.92)
+resolver.source_tracker.set_source_credibility("oss_blog", 0.30)
+
 resolver.set_resolution_rule("apt29", "nation_state", ResolutionStrategy.CREDIBILITY_WEIGHTED)
 resolver.set_resolution_rule("apt29", "first_seen",   ResolutionStrategy.CREDIBILITY_WEIGHTED)
 
 results = resolver.resolve_conflicts(nation_conflicts + first_seen_conflicts)
 for r in results:
     print(f"{r.conflict_id}: {r.resolved_value!r}  [{r.confidence:.0%} confidence]")
-    # apt29_nation_state_conflict: 'Russia'  [83% confidence]
-    # apt29_first_seen_conflict:   '2008'    [73% confidence]
+    # apt29_nation_state_conflict: 'Russia'  [86% confidence]
+    # apt29_first_seen_conflict:   '2008'    [44% confidence]
     # The blog's China attribution (weight 0.30) loses to Mandiant+CrowdStrike (0.95+0.92).
 
 history = resolver.get_resolution_history()
@@ -321,14 +521,11 @@ from semantica.conflicts import ConflictDetector, ConflictResolver, ResolutionSt
 
 cve_records = [
     {"id": "cve-2024-3400", "source": "nvd",
-     "cvss_score": 10.0, "vector": "AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H",
-     "credibility_score": 0.98},
+     "cvss_score": 10.0, "vector": "AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H"},
     {"id": "cve-2024-3400", "source": "mitre",
-     "cvss_score": 9.8,  "vector": "AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
-     "credibility_score": 0.96},
+     "cvss_score": 9.8,  "vector": "AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H"},
     {"id": "cve-2024-3400", "source": "paloalto",
-     "cvss_score": 9.5,  "vector": "AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H",
-     "credibility_score": 0.90},
+     "cvss_score": 9.5,  "vector": "AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H"},
 ]
 
 detector = ConflictDetector()
@@ -336,6 +533,10 @@ score_conflicts  = detector.detect_value_conflicts(cve_records, "cvss_score")
 vector_conflicts = detector.detect_value_conflicts(cve_records, "vector")
 
 resolver = ConflictResolver()
+resolver.source_tracker.set_source_credibility("nvd", 0.98)
+resolver.source_tracker.set_source_credibility("mitre", 0.96)
+resolver.source_tracker.set_source_credibility("paloalto", 0.90)
+
 resolver.set_resolution_rule(
     "cve-2024-3400", "cvss_score", ResolutionStrategy.CREDIBILITY_WEIGHTED
 )
@@ -348,8 +549,8 @@ for r in results:
     if r.resolved:
         print(f"Canonical {r.conflict_id.split('_')[2]}: {r.resolved_value}  "
               f"({r.confidence:.0%} confidence)")
-    # Canonical cvss_score: 10.0   (72% confidence)  — NVD wins
-    # Canonical vector: AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H  (54% confidence)
+    # Canonical cvss_score: 10.0   (35% confidence)  — NVD wins
+    # Canonical vector: AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H  (35% confidence)
 ```
 
 </Tab>
@@ -365,14 +566,11 @@ from semantica.conflicts import ConflictDetector, ConflictResolver, ResolutionSt
 
 drug_records = [
     {"id": "dapagliflozin", "source": "declare_timi58",
-     "hba1c_reduction_pct": 0.54, "primary_endpoint": "MACE",
-     "credibility_score": 0.92},
+     "hba1c_reduction_pct": 0.54, "primary_endpoint": "MACE"},
     {"id": "dapagliflozin", "source": "dapa_hf",
-     "hba1c_reduction_pct": 0.48, "primary_endpoint": "HF_hospitalization",
-     "credibility_score": 0.95},
+     "hba1c_reduction_pct": 0.48, "primary_endpoint": "HF_hospitalization"},
     {"id": "dapagliflozin", "source": "meta_analysis",
-     "hba1c_reduction_pct": 0.52, "primary_endpoint": "HbA1c_reduction",
-     "credibility_score": 0.88},
+     "hba1c_reduction_pct": 0.52, "primary_endpoint": "HbA1c_reduction"},
 ]
 
 detector = ConflictDetector()
@@ -380,6 +578,10 @@ efficacy_conflicts = detector.detect_value_conflicts(drug_records, "hba1c_reduct
 endpoint_conflicts = detector.detect_value_conflicts(drug_records, "primary_endpoint")
 
 resolver = ConflictResolver()
+resolver.source_tracker.set_source_credibility("declare_timi58", 0.92)
+resolver.source_tracker.set_source_credibility("dapa_hf", 0.95)
+resolver.source_tracker.set_source_credibility("meta_analysis", 0.88)
+
 resolver.set_resolution_rule(
     "dapagliflozin", "hba1c_reduction_pct", ResolutionStrategy.CREDIBILITY_WEIGHTED
 )
@@ -395,7 +597,7 @@ review = [r for r in results if not r.resolved]
 print(f"Auto-resolved  : {len(auto)}")
 for r in auto:
     print(f"  {r.conflict_id}: {r.resolved_value}  [{r.confidence:.0%}]")
-    # dapagliflozin_hba1c_reduction_pct_conflict: 0.48  [38%]
+    # dapagliflozin_hba1c_reduction_pct_conflict: 0.48  [35%]
 
 print(f"Expert queue   : {len(review)}")
 for r in review:
@@ -416,14 +618,11 @@ from semantica.conflicts import ConflictDetector, ConflictResolver, ResolutionSt
 
 client_records = [
     {"id": "corp-acme-uk", "source": "crm",
-     "legal_name": "ACME UK Ltd",                "sic_code": "7372",
-     "credibility_score": 0.75},
+     "legal_name": "ACME UK Ltd",                "sic_code": "7372"},
     {"id": "corp-acme-uk", "source": "lei_registry",
-     "legal_name": "ACME United Kingdom Limited", "sic_code": "7371",
-     "credibility_score": 0.99},                  # LEI registry is authoritative
+     "legal_name": "ACME United Kingdom Limited", "sic_code": "7371"},
     {"id": "corp-acme-uk", "source": "credit_bureau",
-     "legal_name": "ACME UK Ltd",                "sic_code": "7372",
-     "credibility_score": 0.85},
+     "legal_name": "ACME UK Ltd",                "sic_code": "7372"},
 ]
 
 detector = ConflictDetector()
@@ -431,6 +630,10 @@ name_conflicts = detector.detect_value_conflicts(client_records, "legal_name")
 sic_conflicts  = detector.detect_value_conflicts(client_records, "sic_code")
 
 resolver = ConflictResolver()
+resolver.source_tracker.set_source_credibility("lei_registry", 0.99)
+resolver.source_tracker.set_source_credibility("credit_bureau", 0.50)
+resolver.source_tracker.set_source_credibility("crm", 0.40)
+
 resolver.set_resolution_rule(
     "corp-acme-uk", "legal_name", ResolutionStrategy.CREDIBILITY_WEIGHTED
 )
@@ -440,10 +643,10 @@ resolver.set_resolution_rule(
 
 results = resolver.resolve_conflicts(name_conflicts + sic_conflicts)
 for r in results:
-    print(f"Canonical {r.conflict_id.split('_')[2]}: {r.resolved_value!r}  "
+    print(f"Canonical {r.conflict_id.split('_')[1]}: {r.resolved_value!r}  "
           f"[{r.confidence:.0%}]")
-    # Canonical legal_name: 'ACME United Kingdom Limited'  [53%]  — LEI registry wins
-    # Canonical sic_code:   '7371'                         [53%]  — LEI registry wins
+    # Canonical legal_name: 'ACME United Kingdom Limited'  [52%]  — LEI registry wins
+    # Canonical sic_code:   '7371'                         [52%]  — LEI registry wins
 
 # Aggregate conflict statistics for the compliance report
 report = detector.get_conflict_report()
@@ -456,7 +659,7 @@ print(f"  By severity    : {report['by_severity']}")
 
 </Tabs>
 
-## Resolution strategies at a glance
+## Resolution Strategies at a Glance
 
 | Strategy | How it decides | Best when |
 | :--- | :--- | :--- |
@@ -468,6 +671,29 @@ print(f"  By severity    : {report['by_severity']}")
 | `MANUAL_REVIEW` | Flags the conflict; `resolved=False` | Low-volume, high-stakes decisions |
 | `EXPERT_REVIEW` | Flags for domain expert queue; `resolved=False` | Scientific or legal disambiguation required |
 
+## Common Pitfalls
+
+**Running conflict resolution before deduplication**
+If duplicate nodes for the same real-world entity still exist, `ConflictDetector` treats each duplicate as a separate entity disagreeing with the others — producing spurious conflicts that should never have existed. Always run deduplication first.
+
+**Forgetting to persist resolved values**
+`resolve_conflicts()` returns `ResolutionResult` objects; it does not write them anywhere. Inspecting the results and moving on without updating your canonical entity means nothing has actually changed in your data. See [Persisting resolved values](#persisting-resolved-values).
+
+**Scanning properties one at a time across a large entity set**
+Calling `detect_value_conflicts()` for every property in a manual loop produces redundant passes over your data. Use `detect_entity_conflicts()` instead — it handles all properties in a single call and is the recommended starting point for bulk detection.
+
+**Misunderstanding credibility scores**
+Credibility scores are weights you assign based on your prior knowledge of source reliability — not ground truth. A source registered with `set_source_credibility("source", 0.99)` can still be wrong. `CREDIBILITY_WEIGHTED` resolution amplifies your beliefs about source quality; if those beliefs are miscalibrated, the resolutions will be too. Validate scores against known ground truth before relying on them in production.
+
+**Treating resolved values as guaranteed truth**
+A resolved value is the most defensible answer given your sources and strategy — not necessarily the correct one. Low confidence scores and `EXPERT_REVIEW` flags are signals to scrutinize results before writing them to a canonical record or downstream system.
+
+**Using conflict resolution when a single authoritative source already exists**
+If one system is always correct for a given property, read from it directly. Layering conflict resolution over a single source adds complexity, introduces unnecessary doubt, and produces an audit trail that adds no real information.
+
+**Registering rules in a loop to apply one strategy uniformly**
+Calling `set_resolution_rule()` for every entity-property pair just to apply the same strategy to all of them creates O(N) setup for no benefit. Pass `strategy=` directly to `resolve_conflicts()` when one strategy covers the whole batch.
+
 ## Related Guides
 
 - [Deduplication](deduplication) — remove duplicate nodes before running conflict detection

docs/guides/decision-intelligence.md

@@ -6,8 +6,61 @@ icon: "scale-balanced"
 
 `AgentContext.record_decision()` stores every AI decision as a node in the knowledge graph, linked by causal edges to the decisions that preceded it and the outcomes that followed. Use it to build an auditable reasoning trail — one that lets you reconstruct, six months later, exactly which classification caused which escalation, and which policy was checked before it was recorded.
 
+## What Is Decision Intelligence?
+
+Decision Intelligence records and analyzes an agent's own decisions as structured data that can be queried, analyzed, and reused. Instead of decisions disappearing after execution, they become persistent graph nodes with searchable metadata, reasoning chains, and causal relationships.
+
+**Decision Intelligence records decisions** by capturing the scenario, reasoning, outcome, confidence, and decision maker for each choice the agent makes. These decisions become queryable nodes in your knowledge graph.
+
+**Decisions become graph nodes** that can be linked causally (Decision A caused Decision B), searched by similarity (find decisions like this scenario), and analyzed statistically (confidence trends, common outcomes).
+
+**The goal is auditability, explainability, precedent search, and causal tracing.** You can trace why decisions were made, find similar past decisions for consistency, and understand the full causal chain from initial detection to final action.
+
+**Decision Intelligence vs. Agent Memory:** Agent Memory stores external knowledge (documents, facts, observations). Decision Intelligence stores internal decisions (classifications, approvals, actions the agent itself made).
+
+**Decision Intelligence vs. Reasoning:** Reasoning derives new facts from existing data using logical rules. Decision Intelligence records the choices and judgments the agent made during problem-solving.
+
+**Decision Intelligence vs. Graph Analytics:** Graph Analytics analyzes the structural properties of your knowledge graph. Decision Intelligence focuses specifically on the decision-making process and its audit trail.
+
+## Why Use Decision Intelligence?
+
+**Auditable AI actions.** Every decision is recorded with reasoning, confidence, and timestamp, creating a complete audit trail for AI behavior in production systems.
+
+**Explainability.** When stakeholders ask "why did the system do X?", you can trace the exact decision chain that led to that action, including intermediate reasoning steps.
+
+**Precedent reuse.** Before making new decisions, agents can search for similar past scenarios and their outcomes, promoting consistency and learning from previous experience.
+
+**Causal analysis.** Understand how early decisions cascade into later outcomes by following causal relationships between linked decision nodes.
+
+**Governance and compliance.** Policy engines can gate decisions against compliance rules, and all policy applications are recorded for regulatory audit.
+
+## When To Use / When Not To Use
+
+**Use Decision Intelligence when:**
+- Building autonomous agents that make consequential choices
+- Implementing decision workflows requiring audit trails
+- Operating under compliance requirements (financial services, healthcare, defense)
+- Building approval systems with multiple decision points
+- Working in risk-sensitive environments where decisions must be explainable
+
+**Do not use when:**
+- Building stateless chatbots that only retrieve information
+- Implementing simple RAG systems without decision-making
+- Creating read-only information retrieval applications
+- Building applications that never make actionable decisions requiring audit trails
+
+## API Architecture Overview
+
+Decision Intelligence coordinates three main components:
+
+**AgentContext** serves as the high-level orchestration layer. It provides `record_decision()`, `find_precedents()`, and causal chain methods while managing the underlying storage and retrieval systems.
+
+**PolicyEngine** handles policy evaluation and compliance checking. It stores policy rules as graph nodes and validates decisions against those rules before they're recorded.
+
+**DecisionRecorder** specializes in recording structured decision data, managing approval chains, and handling policy exceptions when decisions need to bypass normal rules.
+
 <Info>
-  Decision tracking requires both a `VectorStore` (for embedding-based precedent search) and a `ContextGraph` (for causal graph storage). Set `decision_tracking=True` on `AgentContext` — omitting either component raises `RuntimeError` at call time.
+  Decision tracking requires both a `VectorStore` (for embedding-based precedent search) and a `ContextGraph` (for causal graph storage). Set `decision_tracking=True` on `AgentContext` — omitting `ContextGraph` raises a `RuntimeError` at call time. `VectorStore` is required by `AgentContext` itself: leaving the argument out raises a `TypeError` from Python's argument binding, while passing `vector_store=None` raises a `ValueError` during initialization.
 </Info>
 
 ## Recording the First Decision
@@ -41,6 +94,8 @@ print("Decision recorded:", classification_id)
 # → "Decision recorded: dec_a3f2b1c4-..."
 ```
 
+The `decision_maker` field identifies the component, workflow, agent, or system that produced this decision. Use consistent identifiers like `"cti_pipeline_v2"`, `"analyst_chen"`, or `"risk_model_v3"` to enable filtering and analysis by decision source.
+
 The `Decision` dataclass that backs this node has the following fields — these are what get stored and searched:
 
 ```python
@@ -559,7 +614,7 @@ context.save("agent_state/")
 
 # Start of next session
 context = AgentContext(
-    vector_store=VectorStore(backend="faiss", dimension=768, index_path="decisions.faiss"),
+    vector_store=VectorStore(backend="faiss", dimension=768),
     knowledge_graph=ContextGraph(),
     decision_tracking=True,
 )
@@ -569,6 +624,18 @@ context.load("agent_state/")
 results = context.find_precedents("APT29 infrastructure attribution", limit=5)
 ```
 
+## Common Pitfalls
+
+**Recording decisions without linking causal relationships.** Isolated decision nodes provide less insight than connected decision chains. Use `add_causal_relationship()` to link related decisions and enable causal tracing.
+
+**Creating isolated decision nodes.** Decisions gain value when connected to entities, other decisions, or outcomes in your graph. Link decisions to relevant entities using the `entities` parameter.
+
+**Recording too many low-value decisions.** Not every minor choice needs permanent recording. Focus on consequential decisions that affect outcomes, require audit trails, or benefit from precedent search.
+
+**Treating precedent similarity as proof.** High similarity scores indicate related scenarios, not identical situations. Use precedents as guidance while considering the specific context of each new decision.
+
+**Using Decision Intelligence when simple retrieval is sufficient.** If your system only retrieves information without making actionable choices, traditional search or Agent Memory may be more appropriate than decision tracking.
+
 ## Related Guides
 
 - [Context Graphs](context-graphs) — how `ContextGraph` stores decision nodes and causal edges

docs/guides/deduplication.md

@@ -3,6 +3,96 @@ title: "Deduplication & Entity Merging"
 description: "Detect duplicate entities using multi-factor similarity, merge them with configurable strategies, and keep your knowledge graph clean at scale."
 ---
 
+## What Is Deduplication?
+
+Deduplication is the process of identifying entities that refer to the same real-world object but appear as separate records in your data, then merging them into a single canonical representation. This process resolves aliases, spelling variations, and formatting differences that occur when data comes from multiple sources.
+
+**Key deduplication concepts:**
+
+**Canonical entities** are the single, authoritative representation of a real-world object after merging all duplicate records. The canonical entity becomes the node that all relationships point to in your knowledge graph.
+
+**Aliases** are alternative names or identifiers for the same entity. For example, "APT29", "Cozy Bear", and "Midnight Blizzard" are all aliases for the same threat actor.
+
+**Entity resolution** is the broader process of determining when different records refer to the same entity, including the similarity calculation, duplicate detection, and merging steps.
+
+**Similarity algorithms:**
+- **Jaro-Winkler** measures string similarity with higher scores for shared prefixes, ideal for names with common beginnings
+- **Levenshtein** distance counts character edits needed to transform one string into another, good for catching typos and variations
+
+**Clustering** groups related duplicates together using algorithms like Union-Find, ensuring that if A matches B and B matches C, all three are grouped together even if A and C don't directly match.
+
+## Why Use Deduplication?
+
+**Data quality and consistency.** Eliminate duplicate nodes that fragment relationships and create inconsistent query results across different names for the same entity.
+
+**Accurate analytics and metrics.** Get correct counts, centrality measures, and relationship analysis when entities aren't artificially split across multiple nodes due to naming variations.
+
+**Relationship consolidation.** Merge scattered relationships onto single canonical entities, enabling complete analysis of connections and patterns that would be missed with fragmented data.
+
+**Source integration.** Seamlessly combine data from multiple feeds, systems, and databases where the same entities appear under different identifiers and naming conventions.
+
+**Graph efficiency.** Reduce graph size and improve query performance by eliminating redundant nodes while preserving all information through proper merging strategies.
+
+**Provenance preservation.** Maintain complete audit trails showing which source contributed each piece of information to the final canonical entity.
+
+## When To Use / When Not To Use
+
+**Use deduplication for:**
+- Multi-source data integration where entities appear under different names or identifiers
+- Entity types prone to aliases and variations (organizations, people, products, geographic locations)
+- Knowledge graphs where relationship accuracy depends on entity consolidation
+- Data quality workflows requiring canonical entity management
+- Analytics requiring accurate entity counts and relationship metrics
+- Scenarios where the same real-world objects appear across multiple systems or databases
+
+**Do NOT use deduplication for:**
+- Single-source data with consistent entity identifiers and naming conventions
+- High-throughput streaming scenarios where deduplication latency is unacceptable
+- Data with reliable primary keys where duplicates are impossible by design
+- Cases where entity variations should be preserved as separate nodes (different product versions, time-based entity states)
+- Simple exact-match scenarios where basic database constraints handle uniqueness
+
+**Be cautious with:**
+- Large datasets where O(n²) pairwise comparison becomes computationally expensive
+- Fuzzy matching when deterministic primary keys (LEI, CVE-ID, ISIN) are available
+- Very low similarity thresholds that may merge genuinely different entities
+
+## Typical Workflow
+
+The deduplication workflow follows a systematic process from detection through merging:
+
+**1. Detect** → Use `detect_duplicates()` or `DuplicateDetector` to identify potential matches using multi-factor similarity scoring
+
+**2. Group** → Apply clustering algorithms to collect transitively related duplicates into groups (A matches B, B matches C → group A,B,C)
+
+**3. Select Canonical** → Choose representative entity for each group based on completeness, source authority, or confidence scores
+
+**4. Merge** → Combine duplicate entities using strategies like `keep_most_complete` or `merge_all` while preserving provenance
+
+**5. Validate** → Review merge results and adjust thresholds or strategies based on precision/recall analysis
+
+**6. Update Graph** → Replace duplicate nodes with canonical entities and transfer all relationships
+
+This pipeline transforms fragmented multi-source data into clean, consolidated knowledge graphs ready for analytics and reasoning.
+
+## API Patterns: Functional vs Class-Based
+
+Semantica provides both simple functional wrappers and comprehensive class APIs for different use cases:
+
+**Functional wrappers for simple workflows:**
+- `detect_duplicates()` — one-shot duplicate detection with minimal configuration
+- `calculate_similarity()` — compare two entities with detailed similarity breakdown
+- `merge_entities()` — convenience wrapper around merge_duplicates() for quick merging
+
+**Class APIs for complex workflows:**
+- `DuplicateDetector` — configurable duplicate detection with clustering, incremental processing, and advanced similarity options
+- `EntityMerger` — sophisticated merging with multiple strategies, provenance tracking, and merge history
+
+**Usage guidelines:**
+- Use `merge_duplicates()` when you have a raw collection of entities and need automatic duplicate detection
+- Use `merge_entity_group()` when you already know which entities are duplicates and just need to merge a pre-determined group
+- Don't mix functional wrappers with class APIs in the same workflow—choose one approach and stick with it
+
 The deduplication module detects duplicate entities across multi-source knowledge graphs using six complementary similarity algorithms — exact match, Levenshtein, Jaro-Winkler, cosine, property comparison, and vector embedding — then merges them into a single canonical entity while preserving full provenance. Use it to collapse alias clusters (e.g. "APT29", "Cozy Bear", "Midnight Blizzard") before running graph analytics or conflict resolution.
 
 <Info>
@@ -11,7 +101,9 @@ Run deduplication after ingestion and before conflict resolution. Deduplication
 
 ## Finding your duplicates: the first scan
 
-Start with `detect_duplicates()`. Point it at your threat actor entities and let the pairwise algorithm compare every pair. For a dataset of a few thousand nodes this runs in seconds — the O(n²) cost only matters above ten thousand entities.
+Start with `detect_duplicates()` for straightforward duplicate detection on smaller datasets. Point it at your entities and let the pairwise algorithm compare every pair using multiple similarity signals.
+
+**Scaling consideration:** For datasets of a few thousand nodes, this runs in seconds. The O(n²) pairwise comparison cost only becomes problematic above ten thousand entities—for larger sets, see the clustering section below.
 
 ```python
 from semantica.deduplication import detect_duplicates
@@ -64,11 +156,11 @@ for c in candidates:
   signals: ['property']                   # alias "APT29" in Midnight Blizzard record
 ```
 
-The scores tell a clear story. "APT29" and "APT-29" score 0.89 — the hyphen is the only difference, pure edit-distance signal. "Cozy Bear" and "The Dukes" score lower (0.61) because the names are completely dissimilar, but the property signal fires because both records carry `"APT29"` in their aliases list. "APT28" never appears in the results because it shares only the country field — not enough to cross the 0.6 threshold.
+The scores tell a clear story. "APT29" and "APT-29" score 0.89 — the hyphen is the only difference, producing strong string similarity signals. "Cozy Bear" and "The Dukes" score lower (0.61) because the names are completely dissimilar, but the property signal fires because both records carry `"APT29"` in their aliases list. "APT28" never appears in the results because it shares only the country field — not enough to cross the 0.6 threshold.
 
 ## Understanding the candidate object
 
-Each `DuplicateCandidate` carries the two entities, their scores, and a `reasons` list explaining which signals fired. This is your audit trail for the detection decision:
+Each `DuplicateCandidate` carries the two entities, their similarity scores, and a detailed breakdown of which similarity algorithms contributed to the match. This provides full transparency for audit and threshold tuning:
 
 ```python
 from semantica.deduplication import calculate_similarity
@@ -98,11 +190,11 @@ Components    :
   embedding            0.78   # semantic vectors land in the same cluster
 ```
 
-The property component (0.94) is doing most of the work here. "Cozy Bear"'s record carries `aliases: ["APT29"]`, which creates an almost-definitive signal. When you see a pattern like this — a weak name score but a strong property score — you're looking at a real alias relationship, not a false positive.
+The property component (0.94) is doing most of the work here. "Cozy Bear"'s record carries `aliases: ["APT29"]`, which creates an almost-definitive signal that these entities refer to the same threat actor. When you see a pattern like this — weak name similarity but strong property matching — you're typically looking at a genuine alias relationship rather than a false positive.
 
 ## Grouping duplicates before merging
 
-For a small dataset you can merge pairs directly. For a larger graph where the same entity might appear under six different names across twelve feeds, use `detect_duplicate_groups()`. It runs Union-Find clustering to collect all aliases of the same underlying entity into a single group, regardless of whether every pair individually crosses the threshold:
+For small datasets, you can merge candidate pairs directly. For larger graphs where the same entity might appear under six different names across twelve feeds, use duplicate grouping with Union-Find clustering. This ensures that if A matches B and B matches C, all three entities are grouped together even if A and C don't directly meet the similarity threshold:
 
 ```python
 from semantica.deduplication import DuplicateDetector, EntityMerger
@@ -132,11 +224,11 @@ Found 2 duplicate groups:
   Representative: 'APT28'
 ```
 
-The group result shows the problem clearly: five separate nodes that should be one. The `representative` field is the entity the merger will use as the base — the one with the most filled properties, in this case "APT29" from the MISP feed which carries the fullest attribute set.
+The group result shows the consolidation clearly: five separate nodes that should be one canonical entity. The `representative` field identifies the entity the merger will use as the base — typically the one with the most complete attribute set, in this case "APT29" from the MISP feed.
 
 ## Merging: collapsing the group without losing data
 
-Now merge. The `keep_most_complete` strategy keeps the entity with the highest property count as the canonical node and fills in any missing fields from the other sources. With `preserve_provenance=True`, the merge operation records which source contributed every field in the merged result:
+Once you have identified duplicate groups, the merging process consolidates them into canonical entities. The `keep_most_complete` strategy selects the entity with the highest property count as the canonical node and enriches it with any missing fields from the other sources:
 
 ```python
 merger = EntityMerger(preserve_provenance=True)
@@ -145,29 +237,28 @@ for group in groups:
     if len(group.entities) < 2:
         continue
 
-    operations = merger.merge_duplicates(group.entities, strategy="keep_most_complete")
+    # merge_entity_group() skips duplicate detection since `group.entities`
+    # is already a confirmed group from detect_duplicate_groups()
+    op = merger.merge_entity_group(group.entities, strategy="keep_most_complete")
 
-    for op in operations:
-        canonical = op.merged_entity
-        source_ids = [e["id"] for e in op.source_entities]
-        print(f"Merged {len(op.source_entities)} entities → canonical: {canonical['name']!r}")
-        print(f"  Source IDs retired : {source_ids}")
-        print(f"  Merge strategy     : {op.merge_result}")
-        print(f"  Timestamp          : {op.timestamp}")
+    canonical = op.merged_entity
+    source_ids = [e["id"] for e in op.source_entities]
+    print(f"Merged {len(op.source_entities)} entities → canonical: {canonical['name']!r}")
+    print(f"  Source IDs retired : {source_ids}")
+    print(f"  Merge strategy     : {op.merge_result.metadata.get('strategy')}")
 ```
 
 ```text
 Merged 5 entities → canonical: 'APT29'
   Source IDs retired : ['ta-nvd-001', 'ta-of-002', 'ta-rf-003', 'ta-sx-004', 'ta-ms-005']
-  Merge strategy     : MergeResult.KEPT_MOST_COMPLETE
-  Timestamp          : 2026-06-21T09:14:02.443Z
+  Merge strategy     : keep_most_complete
 ```
 
-The five source entities are replaced by one. Every relationship those five nodes carried — to campaigns, malware families, TTPs, infrastructure — now attaches to the canonical "APT29" node. Nothing is lost; the provenance records show exactly which feed contributed which attribute.
+The five source entities are replaced by one canonical representation. Every relationship those five nodes carried — to campaigns, malware families, TTPs, infrastructure — now attaches to the canonical "APT29" node. The merge operation preserves all information while eliminating redundancy, and the provenance records show exactly which feed contributed each attribute.
 
 ## Reviewing merge history for audit
 
-After a batch merge, pull the full history to review every decision made:
+After batch merging operations, you can retrieve the complete history to review every decision made. This audit trail is essential for understanding merge decisions and explaining them to stakeholders:
 
 ```python
 history = merger.get_merge_history()
@@ -175,14 +266,14 @@ history = merger.get_merge_history()
 print(f"Total merge operations: {len(history)}")
 for op in history:
     print(f"  {op.merged_entity['name']!r} ← {len(op.source_entities)} sources")
-    print(f"    strategy: {op.merge_result}")
+    print(f"    strategy: {op.merge_result.metadata.get('strategy')}")
 ```
 
-This history is what you present when a feed owner asks why their entity was merged into another one. Every decision is recorded.
+This history provides complete transparency about merge decisions. When a feed owner asks why their entity was merged into another one, you have the documented evidence and reasoning for the decision.
 
 ## Streaming ingestion: incremental deduplication
 
-When your pipeline is ingesting continuously — new STIX bundles arriving hourly — you don't want to re-run pairwise comparison over the entire graph on every batch. Use `incremental_detect()` to compare only the new entities against the existing set:
+When your pipeline processes continuous data streams — new threat intelligence arriving hourly — you don't want to re-run pairwise comparison over the entire graph on every batch. Use incremental detection to compare only new entities against the existing canonical set:
 
 ```python
 # Existing graph entities (already deduplicated)
@@ -212,11 +303,13 @@ New duplicates found in this batch: 1
   score=0.67        # alias field carries "APT29" — property signal fires
 ```
 
-NOBELIUM goes to the merge queue. Scattered Spider scores below threshold against every existing actor and gets added to the graph as a new node.
+NOBELIUM gets queued for merging with the existing APT29 canonical entity. Scattered Spider scores below threshold against every existing actor and gets added to the graph as a new, unique node.
 
 ## Scaling to large entity sets
 
-For graphs above ten thousand nodes, pairwise comparison becomes too slow. Use `build_clusters()` to run vectorized batch comparison, then merge each cluster:
+For graphs above ten thousand nodes, pairwise comparison becomes computationally expensive due to its O(n²) complexity. Use `build_clusters()` to run more efficient vectorized batch comparison, then merge each resulting cluster:
+
+**Performance warning:** Always profile your similarity operations on representative data sizes. What works for 1,000 entities may become unacceptably slow at 10,000+ entities without appropriate scaling strategies.
 
 ```python
 from semantica.deduplication import build_clusters
@@ -238,11 +331,83 @@ print(f"Quality metrics  : {cluster_result.quality_metrics}")
 merger = EntityMerger(preserve_provenance=True)
 for cluster in cluster_result.clusters:
     if len(cluster.entities) > 1:
-        merger.merge_duplicates(cluster.entities, strategy="keep_most_complete")
+        # Use merge_entity_group() since clustering already determined these are duplicates
+        merger.merge_entity_group(cluster.entities, strategy="keep_most_complete")
 ```
 
 For even larger sets, switch to `method="hierarchical"` which uses agglomerative bottom-up clustering and scales to hundreds of thousands of entities at the cost of some precision.
 
+## A Simple Example: Customer Deduplication
+
+Before exploring domain-specific cases, let's walk through a straightforward customer deduplication scenario. A company's CRM system has accumulated duplicate customer records from web signups, sales team entries, and support tickets:
+
+```python
+from semantica.deduplication import detect_duplicates, merge_entities
+
+customers = [
+    {"id": "cust-001", "name": "John Smith", "email": "john.smith@email.com", 
+     "company": "Acme Corp", "source": "web_signup"},
+    {"id": "cust-002", "name": "J. Smith", "email": "john.smith@email.com",
+     "company": "Acme Corporation", "source": "sales_team"},
+    {"id": "cust-003", "name": "John Smith", "phone": "+1-555-0123",
+     "company": "Acme Corp", "source": "support_ticket"},
+    {"id": "cust-004", "name": "Jane Doe", "email": "jane.doe@email.com",
+     "company": "Beta Inc", "source": "web_signup"},
+]
+
+# Step 1: Find potential duplicates
+candidates = detect_duplicates(
+    customers,
+    method="pairwise",
+    similarity_threshold=0.6,  # 60% similarity required
+    confidence_threshold=0.5,
+)
+
+print("Potential duplicates found:")
+for c in candidates:
+    print(f"  {c.entity1['name']} ~ {c.entity2['name']} (score: {c.similarity_score:.2f})")
+    print(f"    Matching signals: {c.reasons}")
+
+# Expected output:
+# John Smith ~ J. Smith (score: 0.82)
+#   Matching signals: ['exact', 'property']  # same email
+# John Smith ~ John Smith (score: 0.78)  
+#   Matching signals: ['exact', 'property']  # same name and company
+
+# Step 2: Merge the duplicates
+john_smith_records = [customers[0], customers[1], customers[2]]  # All John Smith variants
+merged_ops = merge_entities(john_smith_records, method="keep_most_complete")
+
+for op in merged_ops:
+    canonical = op.merged_entity
+    print(f"\nCanonical customer: {canonical['name']}")
+    print(f"  Email: {canonical.get('email', 'N/A')}")
+    print(f"  Phone: {canonical.get('phone', 'N/A')}")
+    print(f"  Company: {canonical['company']}")
+    print(f"  Merged from {len(op.source_entities)} records")
+    
+# Result: One John Smith record with email, phone, and company information
+# from all three original records, with full provenance tracking
+```
+
+This example demonstrates the core concepts: similarity detection finds related records, and merging consolidates them into canonical entities that preserve all available information.
+
+## Common Pitfalls
+
+**Threshold tuning without validation.** Setting thresholds too low creates false positive merges between genuinely different entities. Always manually review a sample of detected duplicates before running large-scale merging operations.
+
+**Pairwise scaling problems.** The O(n²) cost of comparing every entity pair becomes prohibitive above 10,000 entities. Use clustering methods (`build_clusters`) or switch to vectorized similarity for large datasets.
+
+**Using fuzzy matching when primary keys exist.** If your entities have reliable unique identifiers (LEI codes, CVE IDs, ISBN numbers), use exact matching on those fields instead of computationally expensive similarity algorithms.
+
+**Mixing wrapper and class APIs inconsistently.** Don't call `detect_duplicates()` then manually instantiate `EntityMerger`—choose either the functional approach or class-based approach and use it consistently throughout your workflow.
+
+**Ignoring merge strategy implications.** `keep_first` overwrites later records completely, `merge_all` can introduce conflicting values, and `keep_most_complete` may not respect source authority. Choose the strategy that matches your data quality requirements.
+
+**Skipping provenance tracking.** Without `preserve_provenance=True`, you lose visibility into which source contributed each field in the canonical entity, making audit trails impossible.
+
+**Inadequate similarity algorithm selection.** Pure string similarity fails for alias relationships ("APT29" vs "Cozy Bear"), while property matching may be too aggressive for entities with shared attributes but different identities.
+
 ## Domain examples
 
 <Tabs>

docs/guides/distance-intelligence.md

@@ -6,13 +6,65 @@ icon: "route"
 
 `ContextGraph` distance intelligence answers the structural question that pure semantic similarity cannot: given two nodes, what is their precise relationship in terms of graph topology, path weight, and inferential confidence? Use it to annotate attribution chains with hop counts and confidence decay, rank retrieval results by structural proximity to an anchor node, and surface implied connections for analyst review.
 
+## What Is Distance Intelligence?
+
+Distance intelligence quantifies and analyzes the structural relationships between nodes in your knowledge graph. It provides detailed metadata about graph paths including hop counts, distance bands, confidence decay, and path analysis.
+
+**Distance metadata** includes hop counts (number of edges between nodes), distance bands (semantic categories like "direct", "near", "distant"), confidence decay (accumulated trust along paths), and path analysis (finding optimal routes between nodes).
+
+**Hop counts** measure the number of edges you must traverse to reach one node from another. A hop count of 1 means direct connection; 3 means you traverse through 2 intermediate nodes.
+
+**Distance bands** convert raw hop counts into meaningful categories: "direct" (0-1 hops), "near" (2-3 hops), "mid-range" (4-6 hops), and "distant" (7+ hops). These categories help interpret the semantic meaning of graph distances.
+
+**Confidence decay** multiplies edge weights along a path to compute accumulated trust. If each edge has weight 0.8, a 3-hop path has confidence decay of 0.8³ = 0.512, indicating moderate confidence in the connection.
+
+**Path analysis** finds optimal routes between nodes using algorithms like Dijkstra's shortest path or Yen's k-shortest paths algorithm.
+
+**Distance intelligence vs. graph analytics:** Analytics computes statistical measures like centrality and communities across the entire graph. Distance intelligence focuses on specific paths and relationships between particular nodes.
+
+**Distance intelligence vs. graph traversal:** Simple traversal follows edges to find neighbors. Distance intelligence quantifies the quality and confidence of those connections using weights, paths, and decay metrics.
+
+## Why Use Distance Intelligence?
+
+**Confidence-aware retrieval.** Instead of treating all graph connections equally, distance intelligence weights results by path confidence, giving higher rankings to nodes connected through stronger, more direct relationships.
+
+**Relationship discovery.** Find not just whether two entities are connected, but how they're connected, through which intermediaries, and with what level of confidence across the full path.
+
+**Causal analysis.** Trace cause-and-effect chains through your knowledge graph with quantified confidence at each step, essential for decision tracking and audit trails.
+
+**Precedent search.** Find similar past cases by analyzing structural similarity and path patterns, not just content similarity.
+
+**Graph-aware ranking.** Blend semantic similarity with graph proximity to surface contextually relevant results that pure vector search would miss.
+
+## When To Use / When Not To Use
+
+**Use distance intelligence for:**
+- Multi-hop reasoning where path quality matters
+- Attribution analysis requiring confidence assessment
+- Causal chain analysis and decision tracing
+- Proximity-weighted retrieval from specific anchor nodes
+- Finding alternative connection routes for verification
+- Ranking results by both content relevance and structural proximity
+
+**Simple graph traversal may be sufficient for:**
+- Finding direct neighbors of a node
+- Basic graph exploration without confidence weighting
+- Cases where all edges have equal importance
+- Simple reachability queries (can A reach B?)
+
+**Distance intelligence may be unnecessary for:**
+- Single-hop neighbor lookups
+- Graphs where edge weights don't represent meaningful confidence
+- Simple existence queries rather than quality assessment
+- Scenarios where path analysis adds unnecessary complexity
+
 <Info>
   Distance Intelligence feeds into proximity-blended retrieval (`proximity_weight` on `retrieve()`), causal chain analysis (`trace_decision_causality()`), and advanced precedent search (`find_precedents_hybrid()`). Enable it by passing `include_distance_metadata=True` on neighbor queries or `proximity_weight > 0` on retrieval calls.
 </Info>
 
 ## Distance Bands: Turning Hop Counts into Meaning
 
-The first tool in distance intelligence is `classify_path_distance` — it maps any BFS depth to a human-readable band that carries semantic meaning.
+The first tool in distance intelligence is `classify_path_distance` — it maps any Breadth-First Search (BFS) depth to a human-readable band that carries semantic meaning.
 
 ```python
 from semantica.utils.helpers import classify_path_distance
@@ -38,6 +90,14 @@ These bands appear automatically on every result that uses `include_distance_met
 
 Each hop along a path multiplies the accumulated confidence by the edge weight. The product — `confidence_decay` — is the single most useful signal for deciding whether a multi-hop inference is trustworthy.
 
+<Info>
+  **Confidence Decay and Edge Weights:** Confidence decay depends directly on edge weights in your graph. Weights should represent confidence, trust, relevance, or similar domain-specific signals where higher values indicate stronger relationships. Unweighted graphs (all edges weight 1.0) produce no meaningful decay analysis.
+</Info>
+
+<Info>
+  **Dense Graph Warning:** Very dense graphs can make path analysis computationally expensive and results harder to interpret. Dense connectivity creates many possible paths with similar weights, making distance-based rankings less discriminating.
+</Info>
+
 ```python
 from semantica.context import ContextGraph
 
@@ -137,7 +197,7 @@ path = pf.bfs_shortest_path(graph, "apt29", "nato_target")
 print("Hop count:", len(path) - 1)
 ```
 
-**K-shortest paths — Yen's algorithm.** Use when you need alternative attribution chains, redundancy analysis, or corroboration routes. Finding the three shortest paths and showing they all converge on the same target is stronger evidence than a single path.
+**K-shortest paths — Yen's algorithm.** Yen's algorithm finds multiple alternative paths between two nodes, ranked by total path cost. Use when you need alternative attribution chains, redundancy analysis, or corroboration routes. Finding the three shortest paths and showing they all converge on the same target is stronger evidence than a single path.
 
 ```python
 k_paths = pf.find_k_shortest_paths(graph, "apt29", "nato_target", k=3)
@@ -483,6 +543,18 @@ for chain in chains:
 
 </Tabs>
 
+## Common Pitfalls
+
+**Treating confidence decay as statistical probability.** Confidence decay is a heuristic measure based on edge weights, not a statistical probability. A decay value of 0.6 doesn't mean "60% probability" — it means the path strength based on your domain-specific weight assignments.
+
+**Using unweighted graphs and expecting meaningful decay.** If all edges have weight 1.0, confidence decay will always be 1.0 regardless of path length, providing no useful discrimination between paths. Assign meaningful weights that reflect relationship strength.
+
+**Excessive path exploration on dense graphs.** Dense graphs with many interconnected nodes can generate exponentially large numbers of paths. Limit `max_hops`, use `min_confidence` thresholds, and consider whether simple neighbor lookup would be sufficient.
+
+**Overusing distance analysis when simple neighbor lookup is enough.** If you only need direct neighbors or one-hop connections, basic graph traversal is simpler and faster than full distance intelligence analysis.
+
+**Retrieving excessive graph neighborhoods.** Large `max_hops` values can retrieve massive subgraphs that overwhelm downstream processing. Start with 2-3 hops and increase only when needed for your specific use case.
+
 ## Related Guides
 
 - [Context Graphs](context-graphs) — `ContextGraph` node and edge model; `add_edge(weight=...)` feeds confidence decay

docs/guides/export.md

@@ -3,10 +3,59 @@ title: "Export & Serialization"
 description: "Export knowledge graphs to RDF (Turtle, JSON-LD, N-Triples), GraphML, Cypher (Neo4j), ArangoDB AQL, CSV, Parquet, OWL, and more."
 ---
 
+## What Is Export?
+
+Export converts Semantica graph data into formats used by external tools and systems. Unlike internal persistence mechanisms that keep data within Semantica, export is specifically designed for interoperability with external consumers.
+
+**Export vs. internal persistence:**
+- **`AgentContext.store()`** and graph persistence keep data inside Semantica for continued processing, retrieval, and reasoning
+- **Export functions** serialize graph data into standardized formats that external systems can consume directly
+
+Export enables integration with analytics platforms, graph databases, RDF triple stores, semantic web systems, data warehouses, business intelligence tools, and downstream consumers that need access to your knowledge graph data in their native formats.
+
+## Why Use Export?
+
+**Build once, export many.** Create your knowledge graph through Semantica's extraction and reasoning workflows, then export the same graph data to multiple formats for different consumers without rebuilding or reprocessing.
+
+**Interoperability with existing ecosystems.** Connect Semantica graphs to established tools and workflows in your organization, from Neo4j graph databases to Gephi visualizations to pandas data analysis pipelines.
+
+**Analytics and reporting workflows.** Feed graph data into business intelligence tools, statistical analysis platforms, and machine learning pipelines that require specific data formats like CSV, Parquet, or RDF.
+
+**Graph database migration and deployment.** Move graphs from Semantica's in-memory representation to production graph databases like Neo4j, ArangoDB, or triple stores for scalable query performance.
+
+**RDF and semantic web integration.** Export to semantic web standards (Turtle, JSON-LD, N-Triples) for integration with ontology tools, SPARQL endpoints, and semantic reasoning systems.
+
+**Data lake and warehouse integration.** Export to columnar formats like Parquet for integration with modern data stack tools including DuckDB, Apache Spark, and cloud data warehouses.
+
+**Compliance and archival workflows.** Generate standardized exports for regulatory submission, long-term archival, and audit trail requirements that mandate specific data formats.
+
+## When To Use / When Not To Use
+
+**Use export when:**
+- Integrating Semantica graphs with external systems and tools
+- Sharing graph data with teams using different technology stacks
+- Building analytics pipelines that consume graph data in downstream processing
+- Working with RDF and ontology workflows requiring semantic web standards
+- Creating reports, visualizations, and business intelligence dashboards
+- Migrating graphs to production databases for scalable query performance
+- Meeting compliance requirements for specific data format submissions
+
+**Do not use export when:**
+- You simply want to save and reload Semantica state—use built-in persistence mechanisms instead
+- Agent persistence and memory continuity are your primary goals
+- Internal retrieval, reasoning, and graph operations are sufficient for your use case
+- Export would add unnecessary complexity to workflows that operate entirely within Semantica
+- You need real-time access to evolving graph data—export creates static snapshots
+
+**Consider internal persistence instead when:**
+- Your workflow involves iterative graph building, querying, and reasoning within Semantica
+- You need to maintain agent memory, conversation history, and decision tracking
+- Graph data will continue to be processed and enriched within Semantica workflows
+
 `export_rdf`, `export_graph`, `export_lpg`, and related functions serialize a `ContextGraph` to any of ten formats in a single call, preserving node types, edge weights, and metadata faithfully. Use them when downstream consumers — triple stores, graph databases, visualization tools, ML pipelines, or spreadsheet auditors — each expect a different format from the same in-memory graph.
 
 <Info>
-  All export functions take `graph.to_dict()` as their first argument — the same dict produced by `ContextGraph.to_dict()`. Build the graph once, export it to as many formats as you need without re-serializing.
+  All export functions take `graph.to_dict()` as their first argument — the same dict produced by `ContextGraph.to_dict()`. Build the graph once, export it to as many formats as you need without re-serializing. Note that `graph.to_dict()` materializes the entire graph in memory, so very large graphs may require additional memory planning.
 </Info>
 
 ## Building the Graph to Export
@@ -36,6 +85,8 @@ graph_data = graph.to_dict()   # single dict, reused across all exports below
 
 ## RDF Formats — For Triple Stores and Semantic Reasoners
 
+**RDF (Resource Description Framework)** is the foundational data model for the semantic web, representing information as subject-predicate-object triplets. RDF formats are essential for integration with semantic web technologies, ontology tools, and systems requiring formal knowledge representation.
+
 When your consumers are triple stores (GraphDB, Stardog, Apache Jena) or OWL reasoners (HermiT, Pellet), you want RDF. Semantica exports to all five standard RDF serializations through a single `export_rdf` call.
 
 ```python
@@ -58,6 +109,8 @@ The format to reach for depends on your consumer. Turtle is ideal for human revi
 
 ## Graph Formats — For Gephi, Maltego, and Network Analysis
 
+**Labeled Property Graph (LPG)** formats represent networks with typed nodes and edges that carry attributes and metadata. These formats are optimized for graph visualization tools and network analysis platforms that focus on exploring relationships and structural patterns.
+
 GraphML, GEXF, and DOT are the native formats of graph analysis and visualization tools. They preserve node attributes, edge weights, and type labels, so the graph you built in Semantica renders immediately in Gephi or NetworkX with full attribute data.
 
 ```python
@@ -77,6 +130,8 @@ The GEXF format is worth knowing about if you use Gephi for analyst briefings
 
 ## Neo4j Cypher — For Graph-Pattern Threat Hunting
 
+**Cypher** is Neo4j's declarative graph query language that uses pattern matching to find and manipulate graph data. Cypher exports enable teams to run complex graph queries, pattern detection, and graph analytics using Neo4j's optimized query engine.
+
 When the SOC team wants to run Cypher queries against the graph — finding threat actors that share infrastructure, or tracing multi-hop attack paths — you export to Cypher and load the result into Neo4j Desktop or Memgraph with a single command.
 
 ```python
@@ -116,6 +171,8 @@ The `include_collection_creation=True` flag means the AQL file is self-contained
 
 ## CSV — For Spreadsheet Audits and Statistical Analysis
 
+**CSV (Comma-Separated Values)** is a simple tabular format universally supported by spreadsheet applications, statistical tools, and data analysis platforms. CSV export flattens graph data into rows and columns for teams that work primarily with tabular data.
+
 The compliance team lives in Excel. The data science team lives in pandas. Both of them need CSV. `export_csv` writes the graph as flat rows — entities and relationships as separate files when you pass a base path.
 
 ```python
@@ -136,6 +193,8 @@ The split form is more useful for downstream tools: the entities CSV feeds a piv
 
 ## Parquet — For Data Lakes and ML Pipelines
 
+**Parquet** is a columnar storage format optimized for analytics workloads, offering efficient compression and fast query performance. Parquet files integrate seamlessly with modern data stack tools and machine learning frameworks.
+
 When the data science team runs feature engineering over graph attributes in DuckDB, Spark, or a lakehouse, Parquet is the format they want. It is columnar, compressed, and readable by every major ML framework.
 
 ```python
@@ -148,6 +207,10 @@ Once in Parquet, the graph entities become a DataFrame that can be joined agains
 
 ## OWL — For Ontology-Based Reasoning
 
+**OWL (Web Ontology Language)** is a semantic web standard for representing rich ontologies with classes, properties, and logical constraints. OWL enables automated reasoning, consistency checking, and inference over formal knowledge models.
+
+**OntologyGenerator** creates formal ontologies from graph data by analyzing entity types, relationships, and patterns to generate class hierarchies, property definitions, and logical constraints. This enables schema validation, automated reasoning, and integration with semantic web tools.
+
 When you have generated an OWL ontology from your graph using `OntologyGenerator`, you can export it for Protégé, HermiT reasoning, or regulatory submission.
 
 ```python
@@ -160,6 +223,22 @@ ontology = OntologyGenerator(base_uri="https://example.org/cti/") \
 export_owl(ontology, "cti_ontology.owl", format="owl-xml")
 ```
 
+## Common Pitfalls
+
+**Confusing export with persistence.** Export creates external snapshots for interoperability, while persistence maintains Semantica's internal state. Don't use export when you need to save and reload agent memory or continue graph-based workflows—use built-in persistence mechanisms instead.
+
+**Exporting stale graph data after graph changes.** Always call `graph.to_dict()` after your final graph modifications. If you store `graph_data` early in your workflow and then modify the graph, exports will reflect the outdated state, not your latest changes.
+
+**Re-running expensive extraction instead of reusing existing graph data.** Build your graph once through entity extraction and relationship inference, then export to multiple formats using the same `graph_data` dict. Don't rebuild the graph for each export format.
+
+**Choosing overly complex formats when CSV is sufficient.** If downstream consumers work with tabular data and don't need graph structure preservation, CSV is simpler, faster, and more universally supported than RDF or GraphML formats.
+
+**Assuming provenance and history automatically appear in exports.** Standard export formats capture the current graph state but don't include provenance chains, version history, or audit trails. Use dedicated provenance export mechanisms if you need full lineage information.
+
+**Ignoring downstream schema requirements.** Different systems expect different identifier formats, attribute schemas, and relationship representations. Validate that your exported data matches the expectations of consuming systems before deploying to production workflows.
+
+**Exporting extremely large graphs without memory planning.** The `graph.to_dict()` operation materializes the entire graph in memory. For very large graphs, monitor memory usage and consider chunking or streaming approaches for resource-constrained environments.
+
 ## Domain Examples
 
 <Tabs>

docs/guides/graphrag.md

@@ -5,6 +5,71 @@ description: "Go beyond vector search: retrieve facts, trace reasoning paths, an
 
 GraphRAG combines vector similarity with knowledge graph traversal so retrieval finds structurally connected facts, not just text that sounds related. When a `ContextGraph` is attached to `AgentContext`, every retrieval call automatically blends semantic search with multi-hop graph expansion — and `query_with_reasoning()` returns an auditable reasoning path alongside the LLM answer.
 
+## What Is GraphRAG?
+
+GraphRAG (Graph-Augmented Retrieval-Augmented Generation) enhances traditional RAG by combining vector similarity search with knowledge graph traversal. Instead of retrieving only semantically similar text, GraphRAG follows relationships between entities to find connected evidence across multiple documents.
+
+**GraphRAG vs. traditional vector-only RAG:** Vector RAG finds documents similar to your query text. GraphRAG finds documents similar to your query AND documents connected to those through entity relationships, even if they don't mention your query terms directly.
+
+**The role of graph traversal:** Starting from entities found in vector-similar documents, GraphRAG expands outward through relationship edges to discover related facts. This reveals connections that pure text similarity would miss — like finding that a threat actor targets healthcare by following the path: Actor → Tool → Victim Organization → Industry Sector.
+
+## Why Use GraphRAG?
+
+**Multi-hop discovery.** Find facts that are 2-3 relationship steps away from your query. A question about "APT29 healthcare targeting" can surface evidence about specific hospitals by traversing: APT29 → HAMMERTOSS → LifeCare → Healthcare Sector.
+
+**Connected evidence.** Instead of isolated document fragments, retrieve coherent chains of related entities and their relationships. This provides richer context for LLM responses and human analysis.
+
+**Investigation workflows.** Follow evidence trails by expanding from known entities through their connections. Start with a suspicious IP and discover the full infrastructure chain, or trace a drug interaction through metabolic pathways.
+
+**Richer retrieval context.** Graph expansion surfaces relevant context that keyword or semantic search alone would miss, leading to more complete and accurate LLM responses.
+
+**Explainability.** GraphRAG provides audit trails showing exactly which entities and relationships led to each piece of retrieved evidence, making the retrieval process transparent and verifiable.
+
+## When To Use / When Not To Use
+
+**GraphRAG adds value when:**
+- Your domain has rich entity relationships (threat intelligence, clinical data, regulatory documents)
+- Questions require connecting facts across multiple documents
+- Investigation workflows benefit from following entity connections
+- Explainability and audit trails are important
+- You have well-structured knowledge graphs with meaningful relationships
+
+**Simple vector search may be sufficient for:**
+- Document retrieval based on topic similarity
+- Single-document question answering
+- Exploratory search where you don't know what you're looking for
+- Domains with few meaningful entity relationships
+
+**Latency and complexity considerations:**
+- GraphRAG adds computational overhead from graph traversal
+- Multi-hop expansion increases retrieval time and token usage
+- Graph quality directly impacts retrieval quality
+- Setup requires entity extraction and relationship building
+
+**GraphRAG may be overkill for:**
+- Simple lookup queries with known answers in specific documents
+- Real-time applications where latency is critical
+- Domains where entity relationships don't provide additional value
+
+## Typical GraphRAG Workflow
+
+**Ingest → Build Graph → Retrieve → Expand Context → Reason → Answer**
+
+1. **Ingest** your documents using `AgentContext.store()` with entity extraction enabled
+2. **Build Graph** through Named Entity Recognition (NER) and relationship extraction to populate the `ContextGraph`
+3. **Retrieve** semantically similar documents and identify seed entities for graph expansion
+4. **Expand Context** by following entity relationships within your specified hop limit
+5. **Reason** (optional) using the expanded context with reasoning engines
+6. **Answer** by providing the enriched context to an LLM through `query_with_reasoning()`
+
+<Info>
+  **Graph Quality Dependency:** GraphRAG retrieval quality depends heavily on graph quality, consistent entity linking, and meaningful relationships. Poor entity extraction, duplicate entities, or weak relationships directly impact retrieval effectiveness.
+</Info>
+
+<Info>
+  **Context Expansion Warning:** Larger hop counts exponentially increase the amount of retrieved context, which can significantly increase LLM token usage and processing time. Start with 2-3 hops and monitor context size for your use case.
+</Info>
+
 <Info>
 GraphRAG activates automatically when you pass `knowledge_graph=` to `AgentContext`. There is no separate mode to switch on. The `hybrid_alpha` parameter and `proximity_weight` argument control how much influence graph structure has relative to vector similarity.
 </Info>
@@ -18,7 +83,7 @@ from semantica.context import AgentContext, ContextGraph
 from semantica.vector_store import VectorStore
 
 # FAISS runs locally with no external dependencies
-vs = VectorStore(backend="faiss", dimension=768, index_path="intel.faiss")
+vs = VectorStore(backend="faiss", dimension=768)
 graph = ContextGraph(advanced_analytics=True)
 
 context = AgentContext(
@@ -31,7 +96,7 @@ context = AgentContext(
 )
 ```
 
-Now ingest your documents. `store()` with `extract_entities=True` runs the full extraction pipeline internally — NER, relation extraction, and entity linking — and populates both the vector index and the graph simultaneously:
+Now ingest your documents. `store()` with `extract_entities=True` runs the full extraction pipeline internally — Named Entity Recognition (NER), relation extraction, and entity linking — and populates both the vector index and the graph simultaneously:
 
 ```python
 intel_documents = [
@@ -82,27 +147,24 @@ With the graph populated, a plain `retrieve()` call already does more than vecto
 results = context.retrieve(
     "APT29 tactics against healthcare",
     use_graph=True,
-    proximity_weight=0.5,   # blend structural proximity into the final score
     max_results=10,
     expand_graph=True,
     max_hops=3,
 )
 
 for r in results:
-    print("[combined={:.3f}  vec={:.3f}  prox={:.3f}]  {}".format(
-        r.get("combined_score", r["score"]),
+    print("[score={:.3f}]  {}".format(
         r["score"],
-        r.get("proximity_score", 0.0),
         r["content"][:90],
     ))
 
-# [combined=0.921  vec=0.884  prox=0.957]  APT29 deployed HAMMERTOSS malware against NATO...
-# [combined=0.887  vec=0.701  prox=0.972]  HAMMERTOSS was subsequently observed on hosts in the LifeCare...
-# [combined=0.841  vec=0.623  prox=0.961]  LifeCare operates 47 acute-care hospitals...
-# [combined=0.798  vec=0.590  prox=0.907]  Healthcare critical infrastructure has been a high-priority...
+# [score=0.921]  APT29 deployed HAMMERTOSS malware against NATO...
+# [score=0.887]  HAMMERTOSS was subsequently observed on hosts in the LifeCare...
+# [score=0.841]  LifeCare operates 47 acute-care hospitals...
+# [score=0.798]  Healthcare critical infrastructure has been a high-priority...
 ```
 
-Notice the third and fourth results: their vector scores are modest (0.623 and 0.590) — neither document mentions APT29 or TTPs. But their proximity scores are high because they are structurally adjacent to the seed nodes in the graph. Pure vector retrieval would have ranked them much lower or excluded them entirely. GraphRAG surfaces them because the graph knows they are connected.
+Notice the top results: while pure vector search might rank connected facts lower because they lack keyword overlap, GraphRAG boosts their final `score` because they are structurally adjacent to the seed nodes in the graph. The returned `score` is a transparent blend of vector relevance and graph connectivity.
 
 When you know specifically which entity you want to anchor the traversal to, pass `anchor_node`:
 
@@ -299,11 +361,10 @@ print("Confidence: {:.1%}".format(triage["confidence"]))
 similar = soc_context.retrieve(
     "wmiprvse.exe encoded powershell scheduled task persistence",
     use_graph=True,
-    proximity_weight=0.5,
     max_results=5,
 )
 for inc in similar:
-    print("[{:.3f}] {}".format(inc.get("combined_score", inc["score"]), inc["content"][:100]))
+    print("[{:.3f}] {}".format(inc["score"], inc["content"][:100]))
 ```
 
 </Tab>
@@ -444,15 +505,29 @@ print(answer["reasoning_path"])
 
 </Tabs>
 
+## Common Pitfalls
+
+**Excessive hop counts.** Setting `max_expansion_hops` too high (>4) creates exponentially large context that overwhelms LLMs and increases costs. Start with 2-3 hops and increase only if needed.
+
+**Poor graph quality.** GraphRAG amplifies graph quality issues. Duplicate entities, inconsistent naming, and weak relationships produce poor retrieval results. Clean your graph data before relying on GraphRAG for important queries.
+
+**Duplicate entities.** Having "APT-29", "APT29", and "Cozy Bear" as separate nodes breaks relationship traversal. Entity linking during ingestion helps, but manual deduplication may be necessary.
+
+**Using GraphRAG for simple lookup queries.** If you know the answer exists in a specific document and just need to retrieve it, traditional vector search is faster and simpler than GraphRAG.
+
+**Assuming graph expansion is always beneficial.** More context isn't always better. Sometimes precise, focused retrieval outperforms broad graph expansion. Test both approaches for your specific use cases.
+
 ## Tuning the vector-graph balance
 
 The `hybrid_alpha` parameter set in the `AgentContext` constructor establishes a default blend between vector similarity and graph influence. `0.0` is pure vector retrieval; `1.0` is pure graph traversal. The recommended starting point is `0.5`.
 
-You can override this per call using `proximity_weight` in `retrieve()` without changing the constructor default:
+When targeting a specific `anchor_node`, you can apply `proximity_weight` in `retrieve()` to dynamically blend structural distance from the anchor into the final score:
 
 ```python
-# Exploratory query — let semantics lead, graph confirms
-results = context.retrieve(query, use_graph=True, proximity_weight=0.2)
+# Anchor node provided — let vector semantics lead, graph proximity only slightly boosts
+results = context.retrieve(
+    query, use_graph=True, anchor_node="APT29", proximity_weight=0.2
+)
 
 # Known-entity tracing — topology drives the retrieval
 results = context.retrieve(

docs/guides/ingest.md

@@ -468,8 +468,7 @@ def run_daily_ingest(since: datetime = None):
 
     graph   = ContextGraph(advanced_analytics=True)
     context = AgentContext(
-        vector_store    = VectorStore(backend="faiss", dimension=768,
-                                      index_path="cti_index.faiss"),
+        vector_store    = VectorStore(backend="faiss", dimension=768),
         knowledge_graph = graph,
         graph_expansion = True,
     )

docs/guides/llm-integrations.md

@@ -5,15 +5,65 @@ description: "Connect Semantica to Groq, OpenAI, Anthropic, HuggingFace, Novita
 
 Semantica exposes a unified provider interface — a single `.generate()` method — across Groq, OpenAI, Anthropic Claude, HuggingFace, Novita AI, and 100+ providers via LiteLLM. Use it when you need to swap providers for latency, accuracy, cost, or data-residency reasons without touching application code.
 
+## What Are LLM Integrations?
+
+The `semantica.llms` module provides a unified interface for connecting to Large Language Model providers. Instead of learning different APIs for each provider, you use the same methods (`.generate()`, `.generate_structured()`) regardless of whether you're calling Groq, OpenAI, Anthropic, or local HuggingFace models.
+
+**Unified interface across providers:** All LLM providers in Semantica expose identical methods, so switching from OpenAI to Anthropic requires changing only the provider constructor, not your application code.
+
+**Provider wrappers vs semantic extraction provider strings:** The `semantica.llms` classes (`Groq`, `OpenAI`, `LiteLLM`, `HuggingFaceLLM`) are Python objects for text generation. The `semantica.semantic_extract` module accepts provider names as strings for entity and relationship extraction. Both approaches are covered in this guide.
+
+## Why Use LLM Integrations?
+
+**Provider portability.** Test with one provider, deploy with another. Switch from Groq for prototyping to Anthropic for production without code changes.
+
+**Reduced vendor lock-in.** Avoid tying your application to a single LLM provider's API. If pricing changes or service availability issues arise, switching providers is straightforward.
+
+**Consistent APIs.** Use the same `.generate()` and `.generate_structured()` methods across all providers instead of learning provider-specific interfaces.
+
+**Multi-provider workflows.** Run fast models for initial classification and expensive frontier models for complex reasoning in the same pipeline.
+
+**Local vs cloud deployment flexibility.** Use cloud providers during development and switch to local HuggingFace models for air-gapped production environments.
+
+## When To Use / When Not To Use
+
+**Use LLM integrations for:**
+- Text generation, summarization, and question-answering tasks
+- Complex reasoning that requires natural language understanding
+- Structured data extraction from unstructured text
+- Multi-step analysis requiring interpretation and synthesis
+- Tasks where context, ambiguity, or domain knowledge matter
+
+**Deterministic tools may be better for:**
+- Pattern matching that regular expressions can handle
+- Simple rule-based classification with clear criteria
+- Mathematical calculations or statistical analysis
+- Graph traversal and relationship queries
+- Data transformations with known logic
+
+**A full LLM may be unnecessary for:**
+- Simple keyword search or exact string matching
+- Deterministic workflows with predefined decision trees
+- High-frequency, low-latency operations where inference overhead matters
+- Tasks where explainability requires transparent rule-based logic
+
 <Info>
   The providers in `semantica.llms` (`Groq`, `OpenAI`, `LiteLLM`, `HuggingFaceLLM`) are for text generation and `query_with_reasoning()`. For structured entity and relation extraction, `semantica.semantic_extract` accepts provider names as strings. Both patterns are covered here.
 </Info>
 
 ## Choosing a Provider
 
-Four factors drive provider selection. **Latency** matters most in real-time SOC triage loops where an analyst is waiting on a triage verdict — Groq's inference server typically returns 8B model responses in under 300ms. **Accuracy** matters most in high-stakes decisions: clinical contraindication checks, credit committee reasoning, and legal document analysis reward the frontier models available via `LiteLLM`. **Data residency** constraints eliminate cloud providers for classified or HIPAA-regulated workloads — `HuggingFaceLLM` with a local model path covers those cases. **Cost at scale** favors high-throughput open-model providers like Novita AI for bulk extraction pipelines where you are processing thousands of documents per hour.
+Four factors drive provider selection, each optimized for different use cases:
 
-The good news: because Semantica's interface is identical across providers, you can prototype with Groq for speed, validate accuracy with Claude, and deploy to Azure OpenAI for compliance — without changing a single line of your application code. Only the provider constructor changes.
+**Latency** matters most in real-time SOC triage loops where an analyst is waiting on a triage verdict. Groq's inference infrastructure typically returns 8B model responses in under 300ms, making it ideal for interactive workflows.
+
+**Accuracy** matters most in high-stakes decisions: clinical contraindication checks, credit committee reasoning, and legal document analysis. Frontier models like Claude or GPT-4 available through `LiteLLM` provide the strongest reasoning capabilities.
+
+**Data residency** constraints eliminate cloud providers for classified or HIPAA-regulated workloads. `HuggingFaceLLM` with local model paths enables fully air-gapped deployments without network calls.
+
+**Cost at scale** favors high-throughput providers like Novita AI for bulk extraction pipelines processing thousands of documents per hour where per-token costs accumulate quickly.
+
+The unified interface means you can prototype with Groq for speed, validate accuracy with Claude, and deploy to Azure OpenAI for compliance — without changing application code.
 
 ## The Shared Interface
 
@@ -31,6 +81,8 @@ This means every place in Semantica that accepts an LLM — `query_with_reasonin
 
 ## Groq — Fast Inference for Real-Time Agents
 
+**Groq** is a cloud provider that specializes in ultra-fast language model inference using custom hardware called Language Processing Units (LPUs). Their infrastructure delivers sub-300ms response times for smaller models, making them ideal for real-time applications where speed matters more than maximum reasoning capability.
+
 Groq Cloud runs open models on purpose-built Language Processing Units that deliver sub-300ms latency for 8B parameter models. This makes Groq the right default for any agent loop where the LLM is in the hot path — SOC triage, real-time alert classification, conversational agents.
 
 ```python
@@ -64,6 +116,8 @@ Groq model selection comes down to the speed-vs-capability tradeoff: `llama-3.1-
 
 ## OpenAI — Function Calling and Vision
 
+**OpenAI** provides access to the GPT model family, including GPT-4o with advanced capabilities like function calling (structured tool use) and vision processing for images and documents. OpenAI models are well-suited for complex reasoning tasks that require strong language understanding and generation capabilities.
+
 The `OpenAI` provider wraps the OpenAI API. Use it when you need GPT-4o's function-calling precision, vision capabilities for document screenshots, or when your team already has an OpenAI contract and wants to stay there.
 
 ```python
@@ -91,6 +145,8 @@ The default model `gpt-3.5-turbo` is fine for classification and light extractio
 
 ## LiteLLM — One Interface, 100+ Providers
 
+**LiteLLM** is a universal adapter that provides a single interface to over 100 different LLM providers, including Anthropic Claude, Azure OpenAI, AWS Bedrock, Google Vertex AI, and local Ollama instances. It acts as a translation layer, converting your unified API calls into provider-specific requests, enabling easy switching between providers without code changes.
+
 `LiteLLM` is the Swiss Army knife. It wraps the `litellm` library, which speaks to every major provider using a unified completion API. The model string encodes both provider and model name: `"anthropic/claude-sonnet-4-20250514"`, `"azure/gpt-4o"`, `"bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"`, `"ollama/llama3.2"`. Change the string, change the provider — no other code changes needed.
 
 ```python
@@ -135,6 +191,8 @@ llm = LiteLLM(model=PROVIDER_MAP[env])
 
 ## HuggingFaceLLM — Air-Gapped and On-Premise
 
+**HuggingFaceLLM** provides access to open-source models from the HuggingFace ecosystem, either downloaded from the HuggingFace Hub or loaded from local file paths. This is the only option for completely offline deployments where no network access is available during inference, such as classified environments or air-gapped systems.
+
 `HuggingFaceLLM` loads a model from the HuggingFace Hub or from a local directory path. No network calls during inference. This is the only option for classified environments, HIPAA-constrained clinical deployments, and any network segment without outbound internet access.
 
 ```python
@@ -302,9 +360,8 @@ from semantica.vector_store import VectorStore
 extraction_llm = HuggingFaceLLM(model="/opt/models/mistral-7b-instruct")
 reasoning_llm  = HuggingFaceLLM(model="/opt/models/llama-3.1-70b-instruct")
 
-# NER with local model — provider pattern still works for local paths
-# (use extract_entities_llm directly with the provider instance)
-from semantica.semantic_extract.methods import extract_entities_llm
+# The llms module wrappers can also be used directly for raw prompt generation
+# when you want to bypass the semantic extraction layer entirely
 
 sigint_text = (
     "[S//NF] APT29 operator observed deploying WARPWIRE credential harvester "
@@ -512,12 +569,26 @@ print(best["response"])
 
 # Sources the answer is grounded in
 for src in best["sources"]:
-    print("  - [{}] {}".format(src.get("metadata", {}).get("source", "?"), src["content"][:60]))
+    print("  - [{}] {}".format(src.get("source", "?"), src["content"][:60]))
 ```
 
 </Tab>
 </Tabs>
 
+## Common Pitfalls
+
+**Choosing expensive frontier models for simple extraction tasks.** GPT-4o or Claude Sonnet for basic entity extraction is overkill — Groq's Llama models handle straightforward NER and classification at a fraction of the cost and latency. Reserve frontier models for complex reasoning that requires nuanced interpretation.
+
+**Ignoring latency differences between providers.** Groq typically responds in under 300ms, while Anthropic Claude can take 2-3 seconds for the same query. For real-time agents or interactive workflows, latency differences compound across multiple LLM calls. Profile your provider performance under realistic load.
+
+**Using LLMs for deterministic pattern matching that regex can handle.** If your task is extracting email addresses, phone numbers, or other pattern-based entities, regular expressions are faster, cheaper, and more reliable than LLM extraction. Use LLMs when context, ambiguity, or domain knowledge matter for correct interpretation.
+
+**Not validating structured outputs.** The `generate_structured()` method returns parsed JSON, but LLMs can still produce malformed or incomplete structures. Always validate the returned dictionary against your expected schema before using the data downstream.
+
+**Switching providers without testing prompt behavior.** Different models respond differently to the same prompt. A prompt optimized for GPT-4 may produce poor results with Llama or Claude. When switching providers, test your prompts and adjust temperature, instructions, or examples as needed.
+
+**Overusing local HuggingFace models for tasks requiring latest knowledge.** Local models have a knowledge cutoff from their training date and cannot access current information. For tasks requiring up-to-date knowledge (recent CVEs, current regulations, latest threat intelligence), cloud providers with more recent training data may be necessary.
+
 ## Related Guides
 
 - [Agent Memory](agent-memory) — using `query_with_reasoning()` with any LLM provider for graph-grounded retrieval

docs/guides/mcp-server.md

@@ -4,15 +4,50 @@ description: "Connect Semantica's knowledge graph, decision intelligence, and re
 icon: "plug"
 ---
 
-The Semantica MCP server exposes your knowledge graph as 12 callable tools so any compatible AI client — Claude Desktop, Windsurf, VS Code extensions — can traverse the graph live, record decisions, run analytics, and export results during a conversation. Use it to give LLM agents direct, real-time access to graph data without writing custom tool wrappers.
+## What Is MCP?
+
+MCP stands for the Model Context Protocol. It is an open standard that allows external AI assistants (like Claude Desktop, Cursor, or Windsurf) to securely access local tools and data sources. 
+
+The Semantica MCP server exposes your knowledge graph as 12 callable tools. By connecting it, any compatible AI client can traverse the graph live, record decisions, run analytics, and export results during a conversation — without you having to write custom tool wrappers.
 
 <Info>
   The Semantica MCP server exposes 12 tools and 3 read-only resources. All tools accept and return JSON. No configuration beyond an optional environment variable for graph persistence is required.
 </Info>
 
+## Architecture & Communication
+
+It is important to understand how MCP works under the hood. **The Semantica MCP server is not a REST API.** There are no network ports, no HTTP endpoints, and no API keys required.
+
+Instead, the AI client launches `semantica-mcp` locally as a subprocess. All communication between the AI and Semantica happens securely through standard input and output (`stdio`). Because the server runs locally under your user account, it inherently has your local file permissions.
+
+## Why Use MCP With Semantica?
+
+- **Zero-Code Integration**: Instantly connect Semantica's graph capabilities to your favorite AI IDE or desktop chat app without writing any glue code.
+- **Real-Time Graph Updates**: Chat with an AI to extract entities from documents and watch them populate your live knowledge graph instantly.
+- **Auditable AI**: Use the AI to make decisions and have it automatically record the reasoning and causal chain directly into the graph via Semantica's decision intelligence tools.
+
+## When To Use / When Not To Use
+
+- **When to Use**: You want to use a third-party AI interface (like Claude Desktop or Windsurf) to manipulate, query, and reason over a Semantica knowledge graph on your local machine.
+- **When NOT to Use**: You are building an autonomous Python script or backend service. If you are writing Python code to build an agent, use `semantica.context.AgentContext` natively instead of spinning up an MCP server. The MCP server does not support remote hosting over HTTP/SSE.
+
+---
+
+## Typical Workflow
+
+Connecting your AI client follows a standard progression:
+
+1. **Install**: Install Semantica in your Python environment.
+2. **Configure Client**: Add the `semantica-mcp` command and absolute graph paths to your AI client's JSON configuration.
+3. **Start Client**: Launch Claude Desktop or Windsurf, which automatically spawns the MCP server.
+4. **Tool Calls**: Prompt the AI in natural language. The AI autonomously chains the 12 available tools.
+5. **Graph Updates**: The AI directly modifies your local graph, adding entities, edges, and decisions.
+
+---
+
 ## Starting the Server
 
-Install Semantica, then launch the MCP server. It starts in stdio mode by default — the protocol used by Claude Desktop, Windsurf, VS Code extensions, and most MCP clients.
+Install Semantica, then configure your client to launch the MCP server. The server runs using the `stdio` transport.
 
 ```bash
 pip install semantica
@@ -26,14 +61,14 @@ semantica-mcp
 python -m semantica.mcp_server
 ```
 
-Startup info prints to stderr. Without `SEMANTICA_KG_PATH` the server initialises an empty in-memory graph — sufficient for testing. For a persistent graph that survives restarts, set the path:
+By default, the server logs at `WARNING` level and produces no startup output. Set `SEMANTICA_LOG_LEVEL=INFO` (or `DEBUG`) to see startup messages on stderr. Without `SEMANTICA_KG_PATH` the server initialises an empty in-memory graph — sufficient for testing. For a persistent graph that survives restarts, set the path:
 
 ```bash
 SEMANTICA_KG_PATH=/data/threat_graph.json semantica-mcp
 ```
 
 <Info>
-  Without `SEMANTICA_KG_PATH`, the graph resets when the server process exits. Always set this path for any session whose data should survive a restart.
+  Without `SEMANTICA_KG_PATH`, the graph resets when the server process exits. Always set this path using an absolute file path for any session whose data should survive a restart.
 </Info>
 
 ## Connecting to Claude Desktop
@@ -46,7 +81,7 @@ Edit the Claude Desktop config file — on macOS at `~/Library/Application Suppo
     "semantica": {
       "command": "semantica-mcp",
       "env": {
-        "SEMANTICA_KG_PATH": "/path/to/knowledge_graph.json",
+        "SEMANTICA_KG_PATH": "/absolute/path/to/knowledge_graph.json",
         "SEMANTICA_LOG_LEVEL": "INFO"
       }
     }
@@ -56,7 +91,7 @@ Edit the Claude Desktop config file — on macOS at `~/Library/Application Suppo
 
 Restart Claude Desktop after saving. The Semantica tools appear in the tool palette automatically — Claude can now call them during any conversation.
 
-If `semantica-mcp` is not on your system PATH (for example, if it is installed in a virtualenv), use the full binary path in `"command"`: `"/path/to/venv/bin/semantica-mcp"`.
+If `semantica-mcp` is not on your system PATH (for example, if it is installed in a virtualenv), use the full absolute binary path in `"command"`: `"/path/to/venv/bin/semantica-mcp"`.
 
 ## Connecting to Other Clients
 
@@ -66,7 +101,7 @@ If `semantica-mcp` is not on your system PATH (for example, if it is installed i
 {
   "semantica": {
     "command": "semantica-mcp",
-    "env": { "SEMANTICA_KG_PATH": "/path/to/knowledge_graph.json" }
+    "env": { "SEMANTICA_KG_PATH": "/absolute/path/to/knowledge_graph.json" }
   }
 }
 ```
@@ -93,7 +128,7 @@ If `semantica-mcp` is not on your system PATH (for example, if it is installed i
 ```bash
 docker run --rm -i \
   -e SEMANTICA_KG_PATH=/data/kg.json \
-  -v /local/path:/data \
+  -v /local/absolute/path:/data \
   ghcr.io/semantica-agi/semantica-mcp:latest
 ```
 
@@ -109,15 +144,42 @@ Once connected, the LLM can call any of these tools during a conversation. The a
 
 **Reasoning** — `run_reasoning` applies forward-chaining IF/THEN rules over a set of facts and returns derived conclusions.
 
-**Analytics and export** — `get_graph_analytics` computes PageRank centrality and community detection. `get_graph_summary` returns node count, decision count, and server status. `export_graph` serializes the current graph to Turtle, JSON-LD, N-Triples, or plain JSON.
+**Analytics and export** — `get_graph_analytics` computes PageRank centrality and community detection. `get_graph_summary` returns node count, decision count, and server status. `export_graph` serializes the current graph to Turtle (`"turtle"` / `"ttl"`), RDF/XML (`"xml"`), N-Triples (`"nt"`), JSON-LD (`"json-ld"`), or plain JSON (`"json"`).
+
+## Universal Example: Employee Directory
+
+Before diving into complex domain examples, here is a simple, universally understood session. An HR manager types a prompt into Claude Desktop:
+
+> "Extract entities from this meeting transcript about Alice transferring to Engineering, add them to the graph, and record a promotion decision."
+
+Claude chains four tool calls automatically:
+
+```text
+1. extract_entities(text="Alice is transferring to Engineering...")
+   → { "entities": [{"label": "Alice", "type": "Employee"}, {"label": "Engineering", "type": "Department"}] }
+
+2. add_entity(id="emp-alice", label="Alice", type="Employee")
+   add_entity(id="dept-eng", label="Engineering", type="Department")
+
+3. add_relationship(source="emp-alice", target="dept-eng", type="WORKS_IN")
+
+4. record_decision(
+       category="promotion",
+       scenario="Alice transferring to Engineering",
+       reasoning="Approved by Engineering Director",
+       outcome="transfer_approved",
+       confidence=1.0
+   )
+```
+The graph is updated instantly with the new organizational structure and a fully auditable decision trail.
 
 ## Watching a Real Agent Session
 
-Here is what happens when an analyst types a prompt into Claude Desktop and the graph is live. The prompt is:
+Here is what happens when a cybersecurity analyst types a prompt into Claude Desktop and the graph is live. The prompt is:
 
 > "Extract entities and relationships from this OSINT report, add them to the knowledge graph, then record an attribution decision for APT29 with confidence 0.88 and export the full graph as Turtle."
 
-Claude chains five tool calls automatically:
+Claude chains six tool calls automatically:
 
 ```text
 1. extract_entities(text="<report text>")
@@ -157,7 +219,7 @@ Resources expose graph state without a tool call — the client can read them at
 
 | URI | Description |
 | :-- | :---------- |
-| `semantica://graph/summary` | Node count, edge count, server status |
+| `semantica://graph/summary` | Node count, decision count, server status |
 | `semantica://decisions/list` | Up to 50 most recent recorded decisions |
 | `semantica://schema/info` | Server version, capabilities, available tool list |
 
@@ -254,11 +316,23 @@ The result is a fully auditable credit decision trail with precedent links, read
 
 </Tabs>
 
+---
+
+## Common Pitfalls
+
+- **Treating MCP as an HTTP server**: Do not try to `curl` the MCP server or look for a port number. It communicates via `stdin/stdout` and waits for JSON-RPC messages from the parent AI client.
+- **Using relative paths for `SEMANTICA_KG_PATH`**: Because the AI client spawns the server as a subprocess, the working directory can be unpredictable. Always use absolute paths (e.g., `C:\Users\Name\graph.json` or `/Users/name/graph.json`) to avoid losing your data.
+- **Virtual environment PATH issues**: If you installed Semantica inside a Python virtual environment, Claude Desktop will not automatically find `semantica-mcp` on the global system PATH. You must provide the absolute path to the binary in the `"command"` field.
+- **Expecting remote hosting support**: Stdio-based MCP servers must run on the same local machine as the AI client. Remote execution over a network is not supported.
+- **Confusing MCP integration with `AgentContext`**: If you are writing your own Python code to orchestrate an LLM, do not use the MCP server. Use the `AgentContext` class natively within your code.
+
+---
+
 ## Troubleshooting
 
-**Server does not appear in Claude Desktop** — fully quit and reopen Claude Desktop after editing the config (close the window is not enough). Verify the binary is on PATH: `which semantica-mcp` on Unix, `where semantica-mcp` on Windows. If using a virtualenv, use the absolute binary path in `"command"`. Set `SEMANTICA_LOG_LEVEL=DEBUG` and check stderr for startup errors.
+**Server does not appear in Claude Desktop** — fully quit and reopen Claude Desktop after editing the config (closing the window is not enough). Verify the binary is on PATH: `which semantica-mcp` on Unix, `where semantica-mcp` on Windows. If using a virtualenv, use the absolute binary path in `"command"`. Set `SEMANTICA_LOG_LEVEL=DEBUG` and check stderr for startup errors.
 
-**Graph data not persisting between sessions** — set `SEMANTICA_KG_PATH` to an absolute file path. Without it the graph is in-memory only and resets on every server restart.
+**Graph data not persisting between sessions** — set `SEMANTICA_KG_PATH` to an absolute file path. Without it, the graph is in-memory only and resets on every server restart.
 
 **Tool calls returning empty results** — `get_graph_summary` returning `"node_count": 0` means the graph is empty. Populate it via `add_entity` and `add_relationship`, or run `extract_entities` on text first and then `add_entity` for each result.
 

docs/guides/multi-agent.md

@@ -3,6 +3,55 @@ title: "Multi-Agent Systems"
 description: "Coordinate multiple AI agents through shared memory, knowledge graphs, and decision history — without a message broker."
 ---
 
+## What Is Multi-Agent Coordination?
+
+A multi-agent system is a software architecture where multiple autonomous agents work together to accomplish complex tasks that would be difficult or impossible for a single agent to handle effectively. Instead of building one monolithic agent that tries to do everything, developers split work across specialized agents that each focus on specific responsibilities.
+
+**Why split work across multiple agents:**
+- **Separation of concerns** — each agent specializes in one domain (ingestion, analysis, reporting) rather than trying to master everything
+- **Independent reasoning** — different agents can use different models, prompts, and reasoning strategies optimized for their specific tasks  
+- **Parallel processing** — multiple agents can work simultaneously on different aspects of the same problem
+- **Human-like workflow decomposition** — mimics how human teams naturally divide complex analytical work
+
+**Semantica's coordination approach:**
+Semantica coordinates agents through shared context (memory and knowledge graphs) rather than message brokers or API calls between services. Agents read and write to the same underlying data structures, enabling seamless information sharing without complex middleware.
+
+**Single-agent vs multi-agent architectures:**
+- **Single-agent** — one `AgentContext` handles all tasks from ingestion through final output
+- **Multi-agent** — multiple `AgentContext` instances or namespaced workflows, each responsible for specific pipeline stages or analytical roles
+
+## Why Use Multi-Agent Systems?
+
+**Separation of responsibilities.** Divide complex workflows into focused, manageable stages where each agent excels at its specific domain without being overwhelmed by tangential concerns.
+
+**Scalability of complex workflows.** Handle sophisticated analytical pipelines that require different expertise areas, processing speeds, and reasoning approaches without creating unwieldy monolithic agents.
+
+**Independent reasoning stages.** Enable different agents to use different LLMs, prompts, confidence thresholds, and reasoning strategies optimized for their specific tasks rather than compromising on a one-size-fits-all approach.
+
+**Specialized agent roles.** Create agents tailored for ingestion, enrichment, analysis, synthesis, and reporting—each with role-appropriate configurations and capabilities.
+
+**Shared knowledge and evidence.** Multiple agents contribute to and benefit from the same knowledge graph and memory stores, creating a cumulative evidence base that improves as more agents contribute their findings.
+
+**Human-like workflow decomposition.** Mirror natural human team structures where analysts, researchers, and decision-makers each contribute specialized expertise to collaborative analytical processes.
+
+## When To Use / When Not To Use
+
+**Use multi-agent systems for:**
+- Complex analytical workflows requiring multiple stages (research → analysis → synthesis → reporting)
+- Multi-stage processing pipelines with distinct phases that benefit from specialized approaches
+- Research and investigation workflows where different agents handle different information sources or analytical methods
+- Teams of specialized agents with different roles (OSINT collector, enrichment analyst, fusion officer)
+- Long-running workflows where different agents may operate at different times or schedules
+- Scenarios requiring different LLMs, reasoning approaches, or confidence thresholds for different analytical stages
+
+**Do NOT use multi-agent systems for:**
+- Simple document summarization or single-step information retrieval tasks
+- Linear workflows where one agent can handle all steps effectively without specialization benefits
+- Small, straightforward tasks where the coordination overhead exceeds the complexity of the core work
+- Cases where a single agent with appropriate configuration can handle the entire workflow efficiently
+
+**Important consideration:** Multi-agent systems introduce additional architectural complexity including state management, coordination patterns, and debugging challenges. Only choose multi-agent approaches when the benefits of specialization and separation of concerns outweigh this added complexity.
+
 Semantica coordinates multiple agents through a shared `ContextGraph` — agents read and write to the same graph, or hand off serialized state via `save()` and `load()`, with no message broker required. Use this pattern when splitting work across ingestion, enrichment, reasoning, and reporting roles that must share a single evidence base.
 
 <Info>
@@ -13,17 +62,17 @@ Semantica coordinates multiple agents through a shared `ContextGraph` — agents
 
 Before writing any code, choose the right coordination pattern for your pipeline.
 
-**Shared graph** works when all agents run in the same process. They hold references to the same `ContextGraph` object — thread-safe by default — so every `store()` from one agent is immediately visible to every `retrieve()` from another. This is the lowest-latency option and the right default for in-process pipelines.
+**Shared Graph Pattern:** Multiple agents share references to the same `ContextGraph` and `VectorStore` objects within a single process. This provides the lowest latency since all agents see changes immediately, with built-in thread safety for concurrent access. Choose this when agents run simultaneously in the same application and need real-time access to each other's contributions.
 
-**Save / load handoff** works when agents run in different processes, on different machines, or at different times. Agent A finishes its work, calls `context.save(path)`, and Agent B calls `context.load(path)` to pick up exactly where A left off — full memory, full graph, full vector index. This is how you implement shift handoffs, async pipelines, and cross-service orchestration.
+**Save / Load Handoff Pattern:** Agents run in different processes, containers, or at different times. The first agent completes its work and calls `context.save(path)` to serialize its complete state. The next agent calls `context.load(path)` to restore exactly where the previous agent left off, including full memory, graph data, and vector indices. Choose this for distributed systems, scheduled workflows, or when agents run on different machines that require shared storage access.
 
-**Namespaced memories** works when you have a single `AgentContext` instance serving multiple logical agents, each scoping its reads and writes with a `conversation_id`. Agents are isolated by tag, not by instance — useful for lightweight role separation without the overhead of multiple contexts.
+**Namespaced Memory Pattern:** A single `AgentContext` serves multiple logical agents, with each agent scoping its reads and writes using unique `conversation_id` values. Agents remain isolated by namespace rather than by separate context instances. Choose this for lightweight role separation without the resource overhead of maintaining multiple complete contexts.
 
 The pipeline in this guide uses all three.
 
 ## Pattern 1 — Shared Graph for Concurrent Ingestion
 
-The OSINT collector and the enrichment agent run concurrently. They share a single `ContextGraph` and a single `VectorStore` — the graph's internal `RLock` makes concurrent writes safe.
+The OSINT (**Open Source Intelligence** — publicly available information) collector and the enrichment agent run concurrently. They share a single `ContextGraph` and a single `VectorStore` — the graph's internal `RLock` makes concurrent writes safe.
 
 ```python
 import threading
@@ -71,7 +120,7 @@ def osint_collection():
         ],
         extract_entities=True,
         extract_relationships=True,
-        conversation_id="osint-pipeline",
+        conversation_id="osint-pipeline",    # namespace acts as agent identifier
     )
 ```
 
@@ -93,7 +142,7 @@ def enrichment():
         ],
         extract_entities=True,
         extract_relationships=True,
-        conversation_id="enrichment-pipeline",
+        conversation_id="enrichment-pipeline",    # separate namespace from OSINT agent
     )
 ```
 
@@ -113,6 +162,8 @@ t1.join();  t2.join()
 
 The reasoning agent runs after ingestion completes. In a production pipeline this might be a separate process, a different container, or a scheduled job. The ingestion agents save their shared state; the reasoning agent loads it.
 
+**Important deployment note:** When agents run in different containers or on different machines, they must have access to the same saved state location through shared storage (network file systems, cloud storage, or shared volumes).
+
 ```python
 # After ingestion: save the combined graph and vector index
 osint_agent.save("./pipeline/enriched_intel/")
@@ -131,7 +182,7 @@ from semantica.context import AgentContext, ContextGraph
 from semantica.vector_store import VectorStore
 from semantica.llms import LiteLLM
 
-# Create a fresh context before loading — load() merges into the existing context
+# Create a context to load the checkpoint into — load() will overwrite existing state
 reasoning_vs    = VectorStore(backend="faiss", dimension=768)
 reasoning_graph = ContextGraph(advanced_analytics=True)
 reasoning_agent = AgentContext(
@@ -182,12 +233,17 @@ reasoning_agent.save("./pipeline/synthesis_output/")
 ```
 
 <Info>
-  `load()` merges into the existing context — it does not wipe it first. Always create a fresh `AgentContext` before calling `load()` if you want a clean restore from a handoff checkpoint.
+  `load()` overwrites the existing context — it clears current memory, graph, and vector state before loading. Any unsaved data in the context prior to calling `load()` will be lost.
 </Info>
 
 ## Pattern 3 — Namespaced Memories for Role Separation
 
-The reporting agent does not need its own graph instance. It shares the reasoning agent's context but scopes its writes to its own namespace — the `conversation_id` acts as an agent identifier.
+The reporting agent does not need its own graph instance. It shares the reasoning agent's context but scopes its writes to its own namespace — the `conversation_id` acts as an agent identifier to separate memory streams and prevent contamination between different logical agents.
+
+**Namespace isolation with conversation_id:**
+- `conversation_id` creates separate memory namespaces within the same `AgentContext`
+- Each agent's memories remain isolated unless explicitly queried across namespaces
+- Prevents accidental memory contamination when different logical agents work on related but distinct tasks
 
 ```python
 # The reporting agent loads the synthesis output
@@ -215,7 +271,7 @@ for item in synthesis_items:
 # Store the final report under the reporting agent's own namespace
 reporting_agent.store(
     "\n\n".join(brief_sections),
-    metadata={"type": "finished_report", "classification": "TLP:GREEN"},
+    metadata={"type": "finished_report", "classification": "TLP:GREEN"},  # TLP (Traffic Light Protocol) — information sharing guidelines
     conversation_id="reporting-output",    # reporting agent's namespace
     user_id="reporting_agent",
 )
@@ -227,11 +283,27 @@ print("Pipeline produced {} traceable context items".format(len(full_trail)))
 
 Each agent's contributions are retrievable individually by filtering on `conversation_id`, or collectively by querying without a filter.
 
+## Common Pitfalls
+
+**Forgetting conversation_id namespaces.** Without unique `conversation_id` values, different agents' memories mix together, making it impossible to trace which agent contributed which insights. Always use distinct, meaningful conversation IDs for each logical agent.
+
+**Accidental state loss with load().** The `load()` function overwrites existing context rather than merging it. If you have unsaved state in an `AgentContext`, calling `load()` will wipe it. Always save your current state or use a fresh context before loading a checkpoint.
+
+**Using Shared Graph across separate processes.** The Shared Graph pattern only works within a single process where agents share object references. For distributed agents running in different containers or machines, use the Save/Load Handoff pattern instead.
+
+**Assuming save/load works without shared storage.** Agents in different processes, containers, or machines must have access to the same filesystem location for save/load handoffs. Ensure shared storage (NFS, cloud storage, shared volumes) is properly configured.
+
+**Overengineering simple workflows with multiple agents.** Multi-agent systems add coordination complexity and potential failure points. For straightforward single-step tasks, a simple single-agent approach is often more reliable and easier to debug.
+
+**Mixing agent responsibilities excessively.** Each agent should have a clear, focused role. Agents that try to do too many different tasks lose the benefits of specialization and become harder to optimize, debug, and maintain.
+
+**Ignoring memory isolation boundaries.** When using namespaced memories, be careful about queries that span multiple `conversation_id` values. Unscoped queries can accidentally retrieve memories from other agents, breaking logical isolation.
+
 ## Domain Examples
 
 <Tabs>
 <Tab title="Defense — CTI/Threat">
-A three-agent intelligence fusion cell: an OSINT collector ingests public feeds, a HUMINT analyst loads classified summaries, and a fusion officer synthesizes both streams into a Priority Intelligence Requirement answer. The OSINT and HUMINT agents run concurrently on a shared graph; the fusion officer loads the combined state in a separate process on an air-gapped network segment.
+A three-agent intelligence fusion cell: an OSINT collector ingests public feeds, a HUMINT (**Human Intelligence** — information gathered from human sources) analyst loads classified summaries, and a fusion officer synthesizes both streams into a PIR (**Priority Intelligence Requirement** — critical information needed for decision-making) answer. The OSINT and HUMINT agents run concurrently on a shared graph; the fusion officer loads the combined state in a separate process on an **air-gapped environment** (isolated network with no internet connectivity for security).
 
 ```python
 import threading

docs/guides/policy-engine.md

@@ -4,17 +4,92 @@ description: "Define, version, and enforce governance policies over knowledge gr
 icon: "scale-balanced"
 ---
 
+## What Is Policy Engine?
+
+Policy evaluation is the systematic checking of decisions against predefined governance rules and constraints. Unlike application enforcement that automatically blocks non-compliant actions, policy evaluation provides compliance status that can trigger different workflows—approval processes, exception handling, or audit requirements.
+
+**Key policy concepts:**
+
+**Policy evaluation** checks whether decisions meet defined criteria without automatically preventing actions, enabling flexible governance workflows.
+
+**Governance and compliance workflows** use policy evaluation results to route decisions through appropriate approval chains, exception processes, or audit trails.
+
+**Approval processes** can be triggered by policy violations, creating documented exception paths with justification and approver accountability.
+
+**Difference from enforcement:** Policy evaluation returns compliance status (`True`/`False`) but does not automatically block actions. Your workflow determines what happens next—immediate approval, escalation, exception handling, or rejection.
+
+## Why Use Policy Engine?
+
+**Governance and accountability.** Create auditable decision workflows where every policy evaluation, exception, and approval is permanently recorded in the knowledge graph with full provenance tracking.
+
+**Compliance verification.** Systematically check decisions against regulatory requirements, internal policies, and risk management rules before they are finalized or acted upon.
+
+**Approval workflow orchestration.** Route non-compliant decisions through structured approval processes with documented justifications and multi-level sign-offs.
+
+**Regulatory compliance.** Meet audit requirements by maintaining complete policy version histories, exception records, and compliance checking trails that regulators can inspect.
+
+**Risk management.** Flag high-risk decisions for additional review while allowing routine compliant decisions to proceed with minimal friction.
+
+**Policy evolution tracking.** Maintain version histories of policy changes with impact analysis, enabling evidence-based policy refinement and regulatory reporting.
+
+## When To Use / When Not To Use
+
+**Use Policy Engine for:**
+- Governance workflows requiring structured approval processes and audit trails
+- Regulatory compliance where policy adherence must be documented and verifiable
+- Multi-level approval workflows for high-stakes decisions (financial approvals, security exceptions, clinical treatments)
+- Regulated environments where policy violations trigger specific escalation procedures
+- Risk management workflows where non-compliant decisions require additional oversight
+- Audit requirements demanding complete policy application and exception tracking
+
+**Do NOT use Policy Engine for:**
+- Simple form validation or basic input checking—use standard validation libraries instead
+- Basic business rules that don't require audit trails or governance workflows  
+- Low-stakes, high-throughput checks where policy evaluation overhead would impact performance
+- Deterministic rule checking that doesn't benefit from version tracking and approval processes
+- Real-time operational decisions where policy evaluation latency is unacceptable
+
+**Warning:** Policy Engine adds governance overhead and requires careful workflow design. Only use when the benefits of structured policy management outweigh the additional complexity.
+
 `PolicyEngine` enforces named policies against recorded decisions, returning `True` if the decision satisfies all policy rules. Use it to gate AI decisions at runtime — attributions requiring dual-source confirmation, escalations requiring senior approval, or any decision category where compliance must be verified before the outcome is recorded. Policies are versioned graph nodes, so every check, exception, and approval chain is part of the permanent audit trail.
 
 <Info>
-The Policy Engine sits above `AgentContext` and `ContextGraph`. Policies are stored as nodes in the same graph as decisions, giving them the same causal tracing, provenance, and temporal validity as any other knowledge graph entity. `PolicyEngine` and `Policy` import from `semantica.context`. `Decision` imports from `semantica.context` (it is a dataclass defined in `semantica.context.decision_models`). `DecisionRecorder` imports from `semantica.context.decision_recorder`.
+The Policy Engine sits above `AgentContext` and `ContextGraph`. Policies are stored as nodes in the same graph as decisions, giving them the same causal tracing, provenance, and temporal validity as any other knowledge graph entity. 
+
+**Key objects:** `PolicyEngine` and `Policy` import from `semantica.context`. `Decision` is a dataclass with fields like `decision_id`, `category`, `scenario`, `reasoning`, `outcome`, `confidence`, `timestamp`, `decision_maker`, and `metadata`. `DecisionRecorder` imports from `semantica.context.decision_recorder` for approval workflow tracking.
 </Info>
 
+## Supported Rule Types
+
+The PolicyEngine implementation supports specific rule patterns that evaluate decision attributes and metadata:
+
+**Confidence rules:**
+- `min_confidence: 0.85` — `decision.confidence >= 0.85`
+
+**Outcome validation:**
+- `allowed_outcomes: ["approved", "approved_with_conditions"]` — `decision.outcome` must be in the list
+
+**Category validation:**
+- `required_categories: ["credit_risk", "operational_risk"]` — `decision.category` must be in the list
+
+**Metadata field rules:**
+- `min_*: value` — metadata field must be `>= value` (e.g., `min_credit_score: 680`)
+- `max_*: value` — metadata field must be `<= value` (e.g., `max_ltv: 0.85`)
+- `required_*: value` — metadata field must equal `value` (string) or contain all items (list)
+
+**Field lookup behavior:** For rule `min_credit_score`, the engine checks `metadata["credit_score"]`, then `metadata["*_credit_score"]` (suffix match), then `decision.credit_score` attribute.
+
+**Important:** The following rule types are NOT supported and will cause unexpected behavior:
+- `disallowed_outcomes` (use `allowed_outcomes` instead)
+- `mandatory_fields` (use `required_*` for specific fields)
+- `requires_mfa` (use metadata field checks like `required_mfa_verified`)
+- Complex nested conditions or operators
+
 ---
 
 ## Defining the policy
 
-A `Policy` is a dataclass with a free-form `rules` dict — encode whatever your domain requires.
+A `Policy` is a dataclass with a free-form `rules` dict — encode whatever your domain requires using supported rule patterns.
 
 ```python
 from semantica.context import ContextGraph, PolicyEngine, Policy
@@ -34,9 +109,11 @@ attribution_policy = Policy(
     rules = {
         "min_independent_sources":  2,
         "required_approver_role":   "senior_analyst",
-        "disallowed_outcomes":      ["nation_state_attributed_single_source"],
+        "allowed_outcomes":         ["nation_state_attributed_dual_source"],
         "min_confidence":           0.85,
-        "mandatory_fields":         ["source_a", "source_b", "approver"],
+        "required_source_a":        True,
+        "required_source_b":        True,
+        "required_approver":        True,
     },
     category   = "threat_attribution",
     version    = "1.0.0",
@@ -74,14 +151,23 @@ decision = Decision(
     confidence    = 0.91,
     timestamp     = datetime.utcnow(),
     decision_maker= "ai_threat_analyst_v3",
+    metadata      = {
+        "independent_sources": 1,  # Below min_independent_sources requirement
+        "approver_role": "analyst",  # Below required_approver_role
+        "source_a": True,  # Has first source
+        # Missing source_b and approver fields
+    }
 )
 
 is_compliant = engine.check_compliance(decision, policy_id)
 print(f"Compliant: {is_compliant}")
 # Compliant: False
 #
-# The outcome "nation_state_attributed_single_source" is in disallowed_outcomes.
-# The policy requires min_independent_sources=2 — the decision only cited one.
+# Multiple rule violations:
+# - outcome "nation_state_attributed_single_source" not in allowed_outcomes
+# - independent_sources (1) < min_independent_sources (2)
+# - approver_role "analyst" != required_approver_role "senior_analyst"
+# - missing required_source_b and required_approver fields
 ```
 
 The engine returns `False`. The decision has not been rejected — it has been flagged. What happens next depends on your workflow. In some organisations, a non-compliant result simply blocks the write to the authoritative graph. In others, it triggers an exception process where a human approver reviews the evidence and signs off.
@@ -174,7 +260,7 @@ The impact dict contains per-decision detail, not just the count. You can inspec
 The lead decides to proceed with the threshold increase. She updates the policy to version 1.1.0, recording her reason. The old version is preserved in the history.
 
 ```python
-updated_policy_id = engine.update_policy(
+engine.update_policy(
     policy_id     = policy_id,
     rules         = {**current_policy.rules, "min_confidence": 0.92},
     change_reason = "Q3 attribution quality review — raise confidence floor from 0.85 to 0.92 "
@@ -182,8 +268,8 @@ updated_policy_id = engine.update_policy(
     new_version   = "1.1.0",
 )
 
-print(f"Policy updated: {updated_policy_id} -> version 1.1.0")
-# Policy updated: pol-attr-001 -> version 1.1.0
+print(f"Policy updated: {policy_id} to version 1.1.0")
+# Policy updated: pol-attr-001 to version 1.1.0
 
 # Find all decisions that were evaluated under v1.0.0 —
 # these need to be re-reviewed to confirm they still meet the new standard.
@@ -225,6 +311,24 @@ for version in history:
 
 ---
 
+## Common Pitfalls
+
+**Assuming failed compliance automatically blocks actions.** PolicyEngine returns compliance status but does NOT automatically prevent actions. Your workflow must check the returned boolean and decide what happens next—approval, rejection, exception handling, or escalation.
+
+**Using unsupported rule keys.** The implementation only supports specific patterns: `min_*`, `max_*`, `required_*`, `min_confidence`, `allowed_outcomes`, and `required_categories`. Any other rule key falls back to a key-presence check: it passes only if that exact key exists in `decision.metadata`, regardless of its value. This means keys like `disallowed_outcomes` will silently **fail** compliance whenever that literal key is absent from metadata (the common case), and will silently **pass** — regardless of the actual outcome — if a `disallowed_outcomes` key happens to exist in metadata with any value. Neither behavior matches the intended "outcome must not be in this list" semantics — use `allowed_outcomes` instead.
+
+**Treating exceptions as approvals.** Recording a policy exception with `record_exception()` does NOT automatically make a non-compliant decision compliant. Exceptions are audit trail entries—your workflow must still decide whether to proceed with the non-compliant decision.
+
+**Assuming PolicyEngine modifies graph state automatically.** PolicyEngine only evaluates compliance and records policy applications, exceptions, and approval chains. It does not modify decision outcomes, metadata, or prevent actions—that is your workflow's responsibility.
+
+**Using complex nested rule structures.** The implementation does not support complex conditional logic, nested operators, or arbitrary expressions. Keep rules simple: single field comparisons, list membership checks, and threshold validations only.
+
+**Missing metadata for rule evaluation.** Rules like `min_credit_score` require the corresponding metadata field (`credit_score`) to be present in `decision.metadata`. Missing metadata fields cause rule evaluation to fail, making the decision non-compliant.
+
+**Forgetting to check rule evaluation results.** Always handle both compliant and non-compliant cases explicitly. Non-compliant decisions that proceed without proper exception handling create audit gaps and governance risks.
+
+---
+
 ## Domain Examples
 
 <Tabs>
@@ -247,10 +351,11 @@ opsec_policy = Policy(
     name        = "TLP:RED — Restricted Dissemination",
     description = "TLP:RED intelligence must not be shared outside the originating organisation",
     rules = {
-        "classification":      "TLP:RED",
-        "disallowed_outcomes": ["shared_with_partner", "published"],
-        "min_confidence":      0.95,
-        "mandatory_fields":    ["tlp", "classification", "authorised_recipients"],
+        "required_classification":      "TLP:RED",
+        "allowed_outcomes":             ["retained_internal", "escalated_internal"],
+        "min_confidence":               0.95,
+        "required_tlp":                 True,
+        "required_authorised_recipients": True,
     },
     category   = "information_sharing",
     version    = "2.1.0",
@@ -264,15 +369,20 @@ decision = Decision(
     category      = "information_sharing",
     scenario      = "APT29 SIGINT report TLP:RED — share with Five Eyes partners?",
     reasoning     = "Tactical intelligence — partner request via UKIC liaison",
-    outcome       = "shared_with_partner",   # violates TLP:RED policy
-    confidence    = 0.88,
+    outcome       = "shared_with_partner",   # violates allowed_outcomes policy
+    confidence    = 0.88,                    # below min_confidence threshold
     timestamp     = datetime.utcnow(),
     decision_maker= "analyst_rodriguez",
+    metadata      = {
+        "classification": "TLP:RED",
+        "tlp": True,
+        "authorised_recipients": True,
+    }
 )
 
 is_compliant = engine.check_compliance(decision, "pol-opsec-001")
 print(f"Compliant: {is_compliant}")
-# Compliant: False — outcome 'shared_with_partner' is disallowed; confidence below 0.95
+# Compliant: False — outcome 'shared_with_partner' not in allowed_outcomes; confidence below 0.95
 
 if not is_compliant:
     # Route to J2 for exception review — dual commander approval required
@@ -286,7 +396,7 @@ if not is_compliant:
     recorder.record_approval_chain(
         decision_id = decision.decision_id,
         approvers   = ["j2_officer_hayes", "unit_commander_brooks"],
-        methods     = ["secure_phone",      "in_person"],
+        methods     = ["email",             "zoom_call"],
         contexts    = ["J2 tactical review", "Commander emergency approval"],
     )
     print(f"Exception recorded with dual-commander approval: {exception_id}")
@@ -315,9 +425,9 @@ for pol in [
         name        = "MFA Required — All Tier-1",
         description = "Every Tier-1 access decision must verify MFA",
         rules       = {
-            "requires_mfa":        True,
-            "disallowed_outcomes": ["access_granted_without_mfa"],
-            "min_confidence":      0.90,
+            "required_mfa_verified":       True,
+            "allowed_outcomes":            ["access_granted_with_mfa"],
+            "min_confidence":              0.90,
         },
         category   = "access_control",
         version    = "1.0.0",
@@ -329,10 +439,10 @@ for pol in [
         name        = "PAM Checkout — Privileged Accounts",
         description = "Privileged account use requires PAM session checkout",
         rules       = {
-            "requires_pam":        True,
-            "session_recording":   True,
-            "max_session_hours":   4,
-            "disallowed_outcomes": ["privileged_access_granted_no_pam"],
+            "required_pam_session":        True,
+            "required_session_recording":  True,
+            "max_session_hours":           4,
+            "allowed_outcomes":            ["privileged_access_granted_with_pam"],
         },
         category   = "privileged_access",
         version    = "1.0.0",
@@ -352,6 +462,11 @@ decision = Decision(
     confidence    = 0.78,
     timestamp     = datetime.utcnow(),
     decision_maker= "soc_automation",
+    metadata      = {
+        "pam_session": False,      # PAM checkout failed
+        "session_recording": True, # Manual recording in place
+        "session_hours": 3,        # Planned session duration
+    }
 )
 
 pam_compliant = engine.check_compliance(decision, "pol-zt-pam")
@@ -396,11 +511,10 @@ safety_policy = Policy(
     name        = "Metformin Absolute Contraindication — eGFR < 30",
     description = "Metformin must not be prescribed when eGFR is below 30 ml/min/1.73m²",
     rules = {
-        "contraindicated_drug":        "metformin",
-        "contraindication_condition":  {"egfr": {"operator": "<", "threshold": 30}},
-        "disallowed_outcomes":         ["metformin_prescribed", "metformin_continued"],
-        "requires_clinician_sign_off": True,
-        "mandatory_checks":            ["egfr_measured_within_90_days"],
+        "min_egfr":                    30,      # eGFR must be >= 30
+        "allowed_outcomes":            ["metformin_discontinued", "metformin_contraindicated", "alternative_prescribed"],
+        "required_clinician_sign_off": True,
+        "required_egfr_check":         True,
     },
     category   = "clinical_safety",
     version    = "3.0.0",   # aligned to BNF 2024
@@ -420,6 +534,12 @@ decision = Decision(
     confidence    = 0.97,
     timestamp     = datetime.utcnow(),
     decision_maker= "cdss_v4",
+    metadata      = {
+        "egfr": 28,                    # Below minimum threshold
+        "clinician_sign_off": True,
+        "egfr_check": True,
+        "drug": "metformin",
+    }
 )
 
 is_compliant = engine.check_compliance(decision, "pol-clin-001")
@@ -465,10 +585,8 @@ mortgage_policy = Policy(
         "max_ltv":                         0.85,
         "max_dsti":                        0.40,
         "min_credit_score":                680,
-        "required_stress_test_bps":        300,
-        "required_fields":                 ["ltv", "pd", "lgd", "dsti", "credit_score"],
-        "disallowed_outcomes":             ["approved_ltv_over_85", "approved_dsti_over_40"],
-        "required_approvers_if_exception": ["senior_underwriter", "credit_committee"],
+        "min_stress_test_bps":             300,
+        "allowed_outcomes":                ["approved", "approved_with_conditions"],
     },
     category   = "credit_risk",
     version    = "2.3.0",
@@ -487,15 +605,23 @@ decision = Decision(
         "LTV 86% exceeds 85% cap. Stress test at +300bps passes. "
         "Credit score 710 above 680 floor. DSTI 38% within 40% limit."
     ),
-    outcome       = "approved_ltv_over_85",   # disallowed outcome — flags non-compliance
+    outcome       = "approved_ltv_exception",   # not in allowed_outcomes — flags non-compliance
     confidence    = 0.72,
     timestamp     = datetime.utcnow(),
     decision_maker= "underwriting_model_v4",
+    metadata      = {
+        "ltv": 0.86,           # Exceeds max_ltv of 0.85
+        "dsti": 0.38,          # Within max_dsti of 0.40
+        "credit_score": 710,   # Above min_credit_score of 680
+        "pd": 0.023,           # Recorded for audit — no threshold rule in this policy
+        "lgd": 0.45,           # Recorded for audit — no threshold rule in this policy
+        "stress_test_bps": 300,
+    }
 )
 
 is_compliant = engine.check_compliance(decision, "pol-credit-001")
 print(f"Compliant: {is_compliant}")
-# Compliant: False — 'approved_ltv_over_85' is in disallowed_outcomes
+# Compliant: False — ltv (0.86) > max_ltv (0.85) and outcome not in allowed_outcomes
 
 if not is_compliant:
     exception_id = engine.record_exception(

docs/guides/provenance.md

@@ -4,6 +4,59 @@ description: "How Semantica tracks the origin and lineage of every entity, relat
 icon: "file-certificate"
 ---
 
+## What Is Provenance?
+
+Provenance is the systematic recording of where data came from, how it was transformed, and who was responsible for each step in its lifecycle. Unlike ordinary graph metadata that simply describes entities, provenance creates an immutable audit trail that tracks the complete history of every piece of information in your system.
+
+**Key provenance concepts:**
+
+**Lineage** traces the chain of custody from original source through all transformations to the current state, showing exactly how data evolved over time.
+
+**Source attribution** records the specific document, database, API call, or human input that produced each data element, enabling precise citation and verification.
+
+**Integrity verification** uses cryptographic checksums to detect any unauthorized changes to provenance records after they were created.
+
+**Audit trails** provide regulatory compliance by maintaining tamper-evident logs of all data operations, transformations, and decisions.
+
+Provenance differs from simple metadata by creating legally defensible, cryptographically verifiable records that answer critical questions: "Where did this come from?", "Who processed it?", "When did it change?", and "Has it been tampered with?"
+
+## Why Use Provenance?
+
+**Compliance with regulatory requirements.** Meet FDA 21 CFR Part 11, ICH E6(R2) GCP, Basel III BCBS 239, and defense intelligence sharing agreements that mandate complete data traceability and electronic record integrity.
+
+**Source attribution and citation.** Trace every entity, relationship, and property value back to its exact source document, API response, or human input for scientific reproducibility and legal defensibility.
+
+**Auditability and transparency.** Provide auditors, regulators, and stakeholders with complete visibility into data processing workflows, including who performed each operation and when changes occurred.
+
+**Conflict resolution and data quality.** When multiple sources provide different values for the same property, provenance records enable evidence-based conflict resolution by comparing source credibility, recency, and confidence levels.
+
+**Tamper detection and forensics.** Cryptographic integrity verification detects unauthorized modifications to data records, supporting incident response and forensic analysis in security-sensitive environments.
+
+**Traceability for data lineage.** Answer complex questions about data ancestry, especially in multi-stage processing pipelines where entities undergo extraction, enrichment, fusion, and analysis transformations.
+
+## When To Use / When Not To Use
+
+**Use provenance tracking for:**
+- Regulated environments requiring audit trails (healthcare, finance, defense, pharmaceuticals)
+- Multi-source data fusion where conflicting information must be resolved with evidence
+- Long-lived knowledge graphs where data quality and source credibility matter
+- Production systems where data integrity and tamper detection are critical
+- Complex processing pipelines where entities undergo multiple transformations
+- Situations requiring legal defensibility of decisions based on extracted data
+
+**Provenance may be unnecessary for:**
+- Simple prototypes and proof-of-concept demonstrations where compliance is not required
+- Ephemeral workflows that process data once and discard results immediately
+- Stateless applications that don't persist data across sessions
+- Internal research projects with trusted single-source data
+- High-frequency, low-latency operations where provenance overhead impacts performance
+- Scenarios where all data comes from a single, highly trusted source that never changes
+
+**Consider simpler alternatives when:**
+- Basic metadata (creation timestamp, source file name) provides sufficient traceability
+- Data processing is transparent and reproducible through version control alone
+- Regulatory compliance does not require cryptographic integrity verification
+
 `ProvenanceManager` records a W3C PROV-O compliant entry for every entity, relationship, document chunk, and property value — with a SHA-256 checksum for tamper detection and automatic version chaining on every `track_entity()` call. Use it when you need to answer regulatory questions about where a value came from, who wrote it, and whether it has changed since first ingestion.
 
 <Info>
@@ -51,7 +104,6 @@ entry_nvd = prov.track_entity(
     activity_id="nvd_feed_ingestion",
     source_location="CVE-2024-3400 JSON record",
     source_quote='{"cvssMetricV31":[{"cvssData":{"baseScore":10.0}}]}',
-    agent_id="nvd_ingest_pipeline_v2",
 )
 
 print(f"Entity tracked : {entry_nvd.entity_id}")
@@ -83,7 +135,6 @@ entry_commercial = prov.track_entity(
     confidence=0.91,
     entity_type="vulnerability",
     activity_id="commercial_feed_ingestion",
-    agent_id="threat_ingest_pipeline_v2",
 )
 
 # The NVD entry is now archived as cve-2024-3400:v:2024-04-12T14:22:07
@@ -97,6 +148,8 @@ This version chaining happens automatically. You do not need to manage history e
 
 When the same property appears in multiple sources with different values — exactly the CVE score situation — use `track_property_source()` to record each attribution separately. This feeds directly into conflict detection downstream: the conflict module can compare all tracked values for a property and surface disagreements with full source metadata attached.
 
+**SourceReference** is a structured metadata container that captures exactly where a piece of information came from within a document. It includes the document identifier, specific location (page, section, byte range), confidence level, and custom metadata fields for domain-specific attribution requirements.
+
 ```python
 from semantica.provenance.schemas import SourceReference
 
@@ -134,7 +187,7 @@ When the regulator asks "where did the 9.8 come from?", this is the answer: `com
 
 ## Tracing the lineage of a node
 
-Six months after ingestion, run a lineage trace. `get_lineage()` returns the full version chain — every state the entity has passed through, oldest to newest — along with summary metadata:
+Once you have multiple provenance entries for an entity, you can trace its complete history to understand how it evolved over time. Six months after ingestion, run a lineage trace. `get_lineage()` returns the full version chain — every state the entity has passed through, oldest to newest — along with summary metadata:
 
 ```python
 lineage = prov.get_lineage("cve-2024-3400")
@@ -161,16 +214,16 @@ Sources seen : ['NVD_feed_2024-04-12', 'commercial_feed_2024-04-12',
                 'NVD_feed_2024-07-18', 'commercial_feed_2024-10-08']
 
 Full version chain (oldest → newest):
-  [2024-04-12T14:22:07]  agent=nvd_ingest_pipeline_v2
+  [2024-04-12T14:22:07]  agent=semantica
     source=NVD_feed_2024-04-12
     activity=nvd_feed_ingestion
-  [2024-04-12T15:18:33]  agent=threat_ingest_pipeline_v2
+  [2024-04-12T15:18:33]  agent=semantica
     source=commercial_feed_2024-04-12
     activity=commercial_feed_ingestion
-  [2024-07-18T08:04:11]  agent=nvd_ingest_pipeline_v2
+  [2024-07-18T08:04:11]  agent=semantica
     source=NVD_feed_2024-07-18
     activity=nvd_feed_ingestion       # NVD updated their score
-  [2024-10-08T09:11:44]  agent=threat_ingest_pipeline_v2
+  [2024-10-08T09:11:44]  agent=semantica
     source=commercial_feed_2024-10-08
     activity=commercial_feed_ingestion
 ```
@@ -179,7 +232,9 @@ The chain answers all three of the regulator's questions. The 9.8 came from `com
 
 ## Verifying integrity
 
-Every `ProvenanceEntry` carries a SHA-256 checksum computed at write time. If any field is modified after the fact — by a misconfigured pipeline, a database migration, or deliberate tampering — the checksum will not match on recomputation. Run integrity checks as part of any compliance audit:
+Every `ProvenanceEntry` carries a SHA-256 checksum computed at write time. If any field is modified after the fact — by a misconfigured pipeline, a database migration, or deliberate tampering — the checksum will not match on recomputation. 
+
+Integrity verification is critical for regulatory compliance and forensic analysis. Run integrity checks as part of any compliance audit:
 
 ```python
 from semantica.provenance.integrity import compute_checksum
@@ -206,7 +261,9 @@ A `TAMPERED` status means the stored hash does not match what would be computed
 
 ## Tracking document chunks and their children
 
-Provenance is not just for entities. When a document is split into chunks for RAG or NLP processing, each chunk needs its own provenance record linking it to the source file and byte range. Child chunks (from recursive splitting) link to their parent via `parent_chunk_id`, which maps to `prov:wasDerivedFrom` in the W3C model:
+Provenance is not just for entities. When a document is split into chunks for retrieval-augmented generation (RAG) or natural language processing workflows, each chunk needs its own provenance record linking it to the source file and byte range. 
+
+Child chunks (from recursive splitting) link to their parent via `parent_chunk_id`, which maps to `prov:wasDerivedFrom` in the W3C PROV-O standard:
 
 ```python
 # Track the parent chunk (a section of an advisory PDF)
@@ -260,6 +317,22 @@ Unique sources   : 12
 
 This summary is the starting point for a compliance attestation: you can state the total number of tracked records, the number of distinct data sources, and the breakdown by record type.
 
+## Common Pitfalls
+
+**Provenance does not guarantee truth.** Provenance records faithfully track where information came from and how it was processed, but it cannot verify that the original sources were accurate. A perfectly documented chain from a flawed or malicious source still produces unreliable data.
+
+**Reusing generic source identifiers.** Using non-specific source IDs like "daily_feed" or "batch_001" makes it impossible to trace individual records back to their exact origins. Always include timestamps, version numbers, or unique batch identifiers in source document names.
+
+**Bypassing provenance workflows.** Manually inserting data or using ad-hoc scripts that skip `track_entity()` calls creates gaps in the audit trail. Ensure all data entry points—automated pipelines, manual corrections, and administrative operations—record appropriate provenance.
+
+**Ignoring lineage verification.** Provenance chains can become complex in multi-stage processing pipelines. Regularly verify that `get_lineage()` and `trace_lineage()` return complete, logical chains without missing links or circular references.
+
+**Overusing provenance in low-value scenarios.** Recording provenance for every intermediate calculation or temporary variable creates storage overhead without compliance benefit. Focus provenance tracking on entities, relationships, and properties that have legal, regulatory, or business significance.
+
+**Failing to validate integrity checksums.** Cryptographic integrity verification only works if you actually check it. Include regular `compute_checksum()` validation in audit workflows and incident response procedures.
+
+**Mixing provenance granularities.** Tracking some entities at the document level and others at the sentence level creates inconsistent audit trails. Establish consistent granularity standards for each data type and processing workflow.
+
 ## Domain examples
 
 <Tabs>
@@ -297,7 +370,6 @@ prov.track_entity(
     entity_type="threat_actor",
     activity_id="ner_extraction",
     source_location="paragraph_3",
-    agent_id="analyst_ALPHA",
 )
 
 # Tier 3: Campaign relationship from all-source fusion
@@ -307,7 +379,6 @@ prov.track_relationship(
     metadata={"type": "operates", "confidence": 0.81},
     confidence=0.81,
     activity_id="all_source_fusion",
-    agent_id="fusion_cell_BRAVO",
 )
 
 # Tier 4: Property from two independent INT sources
@@ -361,7 +432,6 @@ prov.track_entity(
     confidence=0.98,
     entity_type="vulnerability",
     activity_id="nvd_feed_ingestion",
-    agent_id="ingest_pipeline_v2",
 )
 
 # Six weeks later: NVD revised the score after PoC publication
@@ -372,7 +442,6 @@ prov.track_entity(
     confidence=0.98,
     entity_type="vulnerability",
     activity_id="nvd_feed_update",
-    agent_id="ingest_pipeline_v2",
 )
 
 # Track CISA KEV addition as a separate property source
@@ -433,7 +502,6 @@ prov.track_entity(
     entity_type="clinical_endpoint",
     activity_id="structured_data_extraction",
     source_quote="Vaccine efficacy against COVID-19 was 95.0% (95% CI, 90.3–97.6)",
-    agent_id="meddra_extraction_pipeline_v3",
 )
 
 # Multi-study property tracking for meta-analysis
@@ -533,7 +601,6 @@ prov.track_entity(
     confidence=0.89,
     entity_type="credit_decision",
     activity_id="automated_underwriting",
-    agent_id="underwriting_model_v4",
 )
 
 # SR 11-7 audit output

docs/guides/semantic-extraction.md

@@ -4,6 +4,70 @@ description: "How Semantica extracts entities, relationships, events, and RDF tr
 icon: "magnifying-glass"
 ---
 
+## What Is Semantic Extraction?
+
+Semantic extraction is the process of automatically identifying meaningful information from unstructured text and converting it into structured, machine-readable formats. Unlike simple keyword search or pattern matching, semantic extraction understands context, relationships, and implicit connections between concepts in natural language.
+
+**Key differences from basic text processing:**
+- **Regex matching** finds exact patterns but misses contextual meaning
+- **Keyword search** locates terms but ignores relationships between them
+- **Manual annotation** captures semantic meaning but doesn't scale
+- **Semantic extraction** automatically identifies entities, relationships, and events while preserving contextual understanding
+
+When you extract entities like "APT29" and "NATO" from intelligence text, semantic extraction also captures that APT29 "targets" NATO networks, creating structured knowledge that feeds directly into graph databases, reasoning systems, and retrieval workflows.
+
+## Why Use Semantic Extraction?
+
+**Knowledge graph population.** Transform unstructured documents into interconnected knowledge graphs where entities become nodes and relationships become edges, enabling sophisticated graph traversal and reasoning.
+
+**GraphRAG preparation.** Extract structured facts from raw text so that graph-grounded retrieval can find precise, contextually relevant information instead of just similar document chunks.
+
+**Turning unstructured text into structured data.** Convert intelligence reports, clinical notes, legal documents, and regulatory filings into databases, RDF triples, and JSON schemas that downstream systems can query and process.
+
+**Downstream retrieval and reasoning benefits.** Enable precise entity-based search, relationship discovery, causal analysis, and multi-hop reasoning that would be impossible with document-level retrieval alone.
+
+**Automated knowledge discovery.** Surface hidden connections and patterns across large document collections that human analysts would miss due to volume and complexity.
+
+## When To Use / When Not To Use
+
+**Use semantic extraction for:**
+- Converting intelligence reports, clinical notes, and regulatory documents into structured knowledge
+- Building knowledge graphs from unstructured text corpora
+- Preparing text for graph-based reasoning and GraphRAG workflows
+- Discovering relationships and connections across document collections
+- Creating structured datasets for downstream analysis and reporting
+
+**Deterministic parsing may be better for:**
+- Highly structured identifiers like email addresses, UUIDs, hashes, and log IDs where regex patterns are sufficient
+- Simple data extraction from standardized formats (CSV, JSON, XML)
+- Known patterns with fixed formats that don't require contextual understanding
+- High-frequency operations where extraction speed is critical and semantic understanding unnecessary
+
+**Consider simpler alternatives when:**
+- Documents are already structured and don't require natural language understanding
+- Simple keyword search or document retrieval meets your requirements
+- Text quality is too poor for reliable semantic analysis (heavily corrupted OCR, fragmentary data)
+
+## Typical Workflow
+
+The semantic extraction workflow follows a structured sequence that transforms raw text into graph-ready knowledge:
+
+**Ingest** → Load documents from various sources (files, databases, APIs) and prepare text for processing
+
+**Extract** → Apply Named Entity Recognition (NER), relation extraction, event detection, and coreference resolution to identify meaningful information
+
+**Resolve** → Consolidate entity mentions ("APT29", "the group", "they") into canonical references and disambiguate overlapping entities
+
+**Relate** → Connect extracted entities through relationships, creating a web of structured connections between concepts
+
+**Serialize** → Convert the extracted knowledge into RDF triplets, JSON-LD, or other structured formats
+
+**Store** → Load structured output into knowledge graphs, vector databases, or agent memory systems
+
+**Retrieve** → Query the structured knowledge through graph traversal, semantic search, and reasoning workflows
+
+This pipeline transforms documents like "APT29 deployed HAMMERTOSS malware targeting NATO networks" into structured triplets like `(APT29, deployed, HAMMERTOSS)` and `(HAMMERTOSS, targets, NATO_networks)` that enable sophisticated downstream analysis.
+
 `semantica.semantic_extract` turns unstructured text into structured graph-ready output: it identifies named entities, extracts relationships between them, detects time-anchored events, resolves coreferences, and serialises everything as RDF triplets. Use it to populate a `ContextGraph` from raw documents — intelligence reports, clinical notes, regulatory filings, or any free-text corpus.
 
 <Info>
@@ -12,6 +76,8 @@ icon: "magnifying-glass"
 
 ## Step 1 — Named Entity Recognition: who and what is in the text
 
+**Named Entity Recognition (NER)** identifies and classifies meaningful nouns and noun phrases in text, such as people, organizations, locations, products, and domain-specific entities like threat actors or drug names. NER forms the foundation of semantic extraction by identifying the key participants and objects in your documents.
+
 `NamedEntityRecognizer` extracts meaningful nouns from a document and lets you choose the underlying method depending on your latency budget and domain requirements:
 
 ```python
@@ -77,6 +143,8 @@ print("High-confidence entities: {}".format(len(high_conf)))
 
 ## Step 2 — Relation Extraction: how the entities connect
 
+**Relation Extraction** identifies semantic relationships between entities, capturing not just what entities exist in text but how they interact, influence, or connect to each other. This creates the edges that link entity nodes in your knowledge graph.
+
 `RelationExtractor` produces the web of connections between entities — who deployed what, who supplied whom, which CVE targets which product:
 
 ```python
@@ -111,6 +179,8 @@ The `context` field on each `Relation` stores the surrounding sentence. This let
 
 ## Step 3 — Event Detection: what happened, when, and to whom
 
+**Event Detection** identifies discrete occurrences or actions described in text, capturing not just static relationships but dynamic processes that unfold over time. Events include participants, temporal boundaries, locations, and outcomes.
+
 `EventDetector` surfaces structured time-anchored events — discrete occurrences with participants, time windows, and locations:
 
 ```python
@@ -155,6 +225,8 @@ for doc_idx, doc_events in enumerate(batch_events):
 
 ## Step 4 — Coreference Resolution: one entity, many names
 
+**Coreference Resolution** identifies when different text spans refer to the same real-world entity, consolidating mentions like "APT29", "the group", "they", and "the threat actor" into unified references. This prevents downstream processing from treating the same entity as multiple separate objects.
+
 `CoreferenceResolver` collapses references like "GAMMA-7", "the group", "they", and "the threat actor" into canonical chains so downstream extraction doesn't treat them as separate entities:
 
 ```python
@@ -180,6 +252,8 @@ With coreference resolved, you can now replace pronouns and aliases with canonic
 
 ## Step 5 — Triplet Extraction and RDF Serialisation: graph-ready output
 
+**Triplet Extraction** converts semantic knowledge into subject-predicate-object triplets, the fundamental building blocks of knowledge graphs and RDF databases. This structured representation enables graph queries, reasoning, and integration with semantic web technologies.
+
 `TripletExtractor` converts everything into subject-predicate-object triplets and serialises them as RDF, ready for graph ingestion and SPARQL queries:
 
 ```python
@@ -313,7 +387,7 @@ def ingest_intel_report(
 # Process all 200 reports
 intel_graph = ContextGraph(advanced_analytics=True)
 intel_agent = AgentContext(
-    vector_store=VectorStore(backend="faiss", dimension=768, index_path="intel.faiss"),
+    vector_store=VectorStore(backend="faiss", dimension=768),
     knowledge_graph=intel_graph,
     decision_tracking=True,
 )
@@ -559,6 +633,22 @@ jsonld   = tri.serialize_triplets(valid, format="jsonld")
   </Tab>
 </Tabs>
 
+## Common Pitfalls
+
+**Treating extraction as guaranteed truth.** Semantic extraction produces confidence scores for a reason — even high-confidence extractions can be incorrect. Always validate critical extractions, especially for high-stakes decisions in security, clinical, or financial contexts.
+
+**Ignoring confidence thresholds.** Low-confidence extractions often indicate ambiguous text, poor model fit, or noisy input. Setting appropriate thresholds (typically 0.65-0.85) filters unreliable results before they pollute downstream processing.
+
+**Skipping entity resolution.** Different mentions of the same entity ("NATO", "North Atlantic Treaty Organization", "the alliance") will create duplicate nodes in your knowledge graph. Always run coreference resolution and entity deduplication.
+
+**Poor OCR or poor input quality.** Semantic extraction depends on readable text. Documents with OCR errors, encoding issues, or heavy redaction will produce unreliable extractions. Clean and validate input text before extraction.
+
+**Using LLM extraction where regex is sufficient.** For highly structured patterns like CVE identifiers (CVE-YYYY-NNNN), IP addresses, email addresses, or UUIDs, regular expressions are faster, cheaper, and more reliable than semantic extraction.
+
+**Processing too much text at once.** Very long documents (>10,000 words) can overwhelm extraction models and produce inconsistent results. Segment long documents into logical chunks (sections, paragraphs) and process them separately.
+
+**Mixing incompatible extraction methods.** Different methods produce different entity label schemas. LLM extraction might return "THREAT_ACTOR" while spaCy returns "PERSON" for the same entity. Normalize labels across methods or use consistent method chains.
+
 ## Choosing your extraction method
 
 The six extraction methods trade off speed, accuracy, and infrastructure:

docs/guides/shacl-validation.md

@@ -4,14 +4,111 @@ description: "Generate W3C SHACL shapes from OWL ontologies, validate RDF knowle
 icon: "shield-check"
 ---
 
-`SHACLGenerator` produces W3C SHACL constraint shapes from an OWL ontology, and `_run_pyshacl` validates your knowledge graph against them, returning a structured violation report. Use this to gate graph data before analytics, ISAC sharing, or regulatory submission — catching missing required properties, datatype violations, and cardinality breaches before they propagate.
+## What Is SHACL Validation?
 
-<Info>
-SHACL shapes are produced from the same ontology dict that `OntologyGenerator` builds. The full workflow is: graph → ontology → SHACL shapes → validation report. Each stage is one function call. `NodeShape`, `PropertyShape`, and `SHACLGraph` import from `semantica.ontology`. `SHACLValidationReport`, `SHACLViolation`, and `_run_pyshacl` import from `semantica.ontology.ontology_validator`.
-</Info>
+SHACL (Shapes Constraint Language) is a standard for validating graph-based data. While an ontology defines the conceptual *schema* (the "what" exists in your domain), SHACL defines the structural *rules and constraints* (the "how" it should be structured). 
+
+In Semantica, `SHACLGenerator` produces constraint rules (shapes) based on your ontology, and `_run_pyshacl` evaluates your actual data against these rules. If a node violates a rule (e.g., missing a required property or using the wrong datatype), a detailed violation report is generated.
+
+## Why Use SHACL Validation?
+
+Data validation is critical before running analytics, exporting data, or feeding it into production models. SHACL acts as a **data quality gate** that ensures your graph data is structurally sound. Use it to catch:
+- Missing required properties (e.g., a customer without an email address).
+- Datatype mismatches (e.g., a string where a number was expected).
+- Cardinality breaches (e.g., a person with three primary addresses).
+
+## When To Use / When Not To Use
+
+- **When to Use**: You have a complex, interconnected knowledge graph and need to validate the *relationships* and structural integrity of the nodes across the graph. SHACL excels at ensuring that merged, highly connected data conforms to your business rules.
+- **When NOT to Use**: If you are simply validating a flat JSON payload or a single incoming API request. For flat data or single records, use simpler, faster libraries like Pydantic or JSONSchema.
 
 ---
 
+## Key Terms Explained
+
+Before diving in, here are a few concepts you'll encounter:
+
+- **RDF (Resource Description Framework)**: A standard way of representing data as a graph. It treats information as connected "triplets" (Subject → Predicate → Object).
+- **OWL (Web Ontology Language)**: A language used to build ontologies. It defines the classes and properties that exist in your domain.
+- **SHACL Shapes**: The actual validation rules. A "Shape" targets a specific class in your data (like `Person`) and defines the constraints it must follow (like "must have one birthdate").
+- **Turtle (.ttl)**: A popular, human-readable file format for storing RDF graph data and SHACL shapes.
+
+---
+
+## Typical Workflow
+
+A typical SHACL validation pipeline follows this lifecycle:
+
+1. **Ontology**: Build an ontology representing your domain.
+2. **SHACL Shapes**: Generate shapes from that ontology.
+3. **Data Graph**: Prepare your knowledge graph.
+4. **Validation**: Validate the knowledge graph against the SHACL shapes.
+5. **Violation Report**: Analyze the report for errors.
+6. **Remediation**: Fix the data or pipeline and re-validate.
+
+---
+
+## Universal Example: Employee & Department
+
+Let's look at a simple, universally understood example: ensuring every `Employee` belongs to a `Department` and has an `employee_id`.
+
+```python
+from semantica.context import ContextGraph
+from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
+from semantica.ontology.ontology_validator import _run_pyshacl
+
+# 1. Prepare your data graph
+graph = ContextGraph()
+graph.add_node("emp-1", "Employee", "Alice", employee_id="E001")
+graph.add_node("emp-2", "Employee", "Bob") # Missing employee_id, will cause a violation!
+
+# 2. Build the ontology
+ontology = (
+    OntologyGenerator(base_uri="https://company.example.com/ontology/", min_occurrences=1)
+    .generate_from_graph(graph.to_dict(), name="CompanyOntology")
+)
+
+# 3. Generate SHACL Shapes
+shacl_gen = SHACLGenerator(base_uri="https://company.example.com/shapes/", severity="Violation")
+shacl_graph = shacl_gen.generate(ontology)
+
+# Inject mandatory constraints
+BASE = "https://company.example.com/ontology/"
+for ns in shacl_graph.node_shapes:
+    if "Employee" in ns.target_class:
+        ns.property_shapes.append(
+            PropertyShape(path=f"{BASE}employee_id", min_count=1, severity="Violation")
+        )
+
+# Serialize shapes to Turtle
+shacl_ttl = shacl_gen.serialize(shacl_graph, format="turtle")
+
+# 4. Prepare your RDF data graph
+# (For validation, serialize your graph instances to RDF. Here we use a Turtle string.)
+data_ttl = """
+@prefix ex: <https://company.example.com/ontology/> .
+
+<http://example.org/emp-1> a ex:Employee ;
+    ex:employee_id "E001" .
+
+<http://example.org/emp-2> a ex:Employee .
+"""
+
+# 5. Run Validation
+report = _run_pyshacl(data_ttl, shacl_ttl)
+
+# 6. Analyze the Report
+print(f"Graph conforms: {report.conforms}")
+if not report.conforms:
+    report.explain_violations() # Populates human-readable explanations
+    for v in report.violations:
+        print(f"Violation: {v.explanation}")
+```
+
+---
+
+Now, let's explore the workflow in more depth.
+
 ## Step 1 — Build the ontology from your merged graph
 
 SHACL shapes are derived from an ontology. If you already have one from a previous run, skip this step.
@@ -172,15 +269,16 @@ Serialize the graph to RDF, then run `_run_pyshacl` against the shapes.
 
 ```python
 from semantica.ontology.ontology_validator import _run_pyshacl
-from semantica.export import export_rdf
-import tempfile, os
 
-# Serialise the graph to a temporary Turtle file
-tmp = tempfile.NamedTemporaryFile(suffix=".ttl", delete=False, mode="w")
-export_rdf(graph.to_dict(), tmp.name, format="turtle")
-with open(tmp.name) as f:
-    data_ttl = f.read()
-os.unlink(tmp.name)
+# Prepare your RDF data string (since export_rdf primarily exports structural metadata,
+# you typically serialize your custom data graph to Turtle using rdflib or similar).
+data_ttl = """
+@prefix ex: <https://cti.example.org/ontology/> .
+
+<http://example.org/malware-002> a ex:Malware .
+<http://example.org/vuln-003> a ex:Vulnerability ;
+    ex:cve_id "CVE24-3400" .
+"""
 
 # Run SHACL validation
 report = _run_pyshacl(
@@ -214,13 +312,17 @@ Each `SHACLViolation` identifies the node, property path, and fix required.
 
 ```python
 if not report.conforms:
-    # Print plain-English explanations for every violation
+    # Populate plain-English explanations for every violation
     report.explain_violations()
-    # Node <https://cti.example.org/data/malware-002> is missing required property
+    
+    # Iterate and print the explanations
+    for v in report.violations:
+        print(v.explanation)
+    # Node <http://example.org/malware-002> is missing required property
     #   <https://cti.example.org/ontology/family>. At least 1 value(s) are required.
-    # Node <https://cti.example.org/data/vuln-003> is missing required property
+    # Node <http://example.org/vuln-003> is missing required property
     #   <https://cti.example.org/ontology/cvss_score>. At least 1 value(s) are required.
-    # Node <https://cti.example.org/data/vuln-003> has value 'CVE24-3400' for
+    # Node <http://example.org/vuln-003> has value 'CVE24-3400' for
     #   <https://cti.example.org/ontology/cve_id> which does not match the required pattern.
 
     # Iterate for programmatic triage
@@ -272,6 +374,16 @@ print(f"Violations after remediation: {report2.violation_count}")
 
 ---
 
+## Common Pitfalls
+
+- **Assuming the ontology automatically enforces data quality**: `SHACLGenerator` generates shapes based on what it observes in the data. If your data is missing a field, the generator won't know it was mandatory unless you explicitly inject the constraint (as shown in Step 3).
+- **Passing `ContextGraph` directly to SHACL validators**: The `_run_pyshacl` function expects an RDF string (like Turtle format), not a raw Python dictionary or `ContextGraph` object. 
+- **Forgetting RDF serialization**: You must serialize your graph (often via a temporary file using `export_rdf`) before validating it.
+- **Treating validation as a one-time step**: Validation should be integrated as an automated step in your CI/CD pipeline or data ingestion flow, acting as a recurring gatekeeper rather than a one-off script.
+- **Ignoring validation reports**: A graph that does not conform must be remediated. Failing to review the `violation_count` and address the issues negates the purpose of SHACL validation.
+
+---
+
 ## Domain Examples
 
 <Tabs>
@@ -285,8 +397,6 @@ from semantica.context import AgentContext, ContextGraph
 from semantica.vector_store import VectorStore
 from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
 from semantica.ontology.ontology_validator import _run_pyshacl
-from semantica.export import export_rdf
-import tempfile, os
 
 graph = ContextGraph()
 ctx   = AgentContext(
@@ -329,11 +439,14 @@ for ns in shacl_graph.node_shapes:
 
 shacl_ttl = shacl_gen.serialize(shacl_graph, format="turtle")
 
-tmp = tempfile.NamedTemporaryFile(suffix=".ttl", delete=False, mode="w")
-export_rdf(graph.to_dict(), tmp.name, format="turtle")
-with open(tmp.name) as f:
-    data_ttl = f.read()
-os.unlink(tmp.name)
+# Prepare RDF data string
+data_ttl = """
+@prefix ex: <https://cti.dod.mil/ontology/> .
+
+<http://example.org/apt29> a ex:ThreatActor .
+<http://example.org/cve-2024-3400> a ex:Vulnerability .
+<http://example.org/hammertoss> a ex:Malware .
+"""
 
 report = _run_pyshacl(data_ttl, shacl_ttl)
 print(f"CTI graph conforms : {report.conforms}")
@@ -342,6 +455,8 @@ print(f"Warnings           : {report.warning_count}")
 
 if not report.conforms:
     report.explain_violations()
+    for v in report.violations:
+        print(v.explanation)
     # Blocks the nightly ISAC share until violations are resolved
 ```
 
@@ -355,8 +470,6 @@ A SOC team validates zero-trust policy nodes before publishing them to the polic
 from semantica.context import ContextGraph
 from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
 from semantica.ontology.ontology_validator import _run_pyshacl
-from semantica.export import export_rdf
-import tempfile, os
 
 graph = ContextGraph()
 graph.add_node("policy-001", "Policy", "MFA Required for Tier-1 Resources",
@@ -392,11 +505,16 @@ for ns in shacl_graph.node_shapes:
 
 shacl_ttl = shacl_gen.serialize(shacl_graph, format="turtle")
 
-tmp = tempfile.NamedTemporaryFile(suffix=".ttl", delete=False, mode="w")
-export_rdf(graph.to_dict(), tmp.name, format="turtle")
-with open(tmp.name) as f:
-    data_ttl = f.read()
-os.unlink(tmp.name)
+# Prepare RDF data string
+data_ttl = """
+@prefix ex: <https://zerotrust.corp/ontology/> .
+
+<http://example.org/policy-001> a ex:Policy ;
+    ex:version "1.0.0" ;
+    ex:effective_date "2025-01-01"^^<http://www.w3.org/2001/XMLSchema#date> .
+
+<http://example.org/policy-002> a ex:Policy .
+"""
 
 report = _run_pyshacl(data_ttl, shacl_ttl)
 print(f"Policy graph conforms: {report.conforms}")
@@ -460,7 +578,9 @@ print(f"SHACL shapes generated — {len(shacl_graph.node_shapes)} node shapes")
 # SHACL shapes generated — 5 node shapes
 
 # Validate trial data
-tmp = tempfile.NamedTemporaryFile(suffix=".ttl", delete=False, mode="w")
+# Serialize the ontology as data to validate against the shapes
+tmp = tempfile.NamedTemporaryFile(suffix=".ttl", delete=False)
+tmp.close()
 export_rdf(ontology, tmp.name, format="turtle")
 with open(tmp.name) as f:
     data_ttl = f.read()
@@ -481,8 +601,6 @@ A credit risk team validates every `LoanApplication` node against Basel III CRE2
 from semantica.context import ContextGraph
 from semantica.ontology import OntologyGenerator, SHACLGenerator, PropertyShape
 from semantica.ontology.ontology_validator import _run_pyshacl
-from semantica.export import export_rdf
-import tempfile, os
 
 graph = ContextGraph()
 graph.add_node("loan-001", "LoanApplication", "Prime mortgage APP-2025-88421",
@@ -513,11 +631,19 @@ for ns in shacl_graph.node_shapes:
 
 shacl_ttl = shacl_gen.serialize(shacl_graph, format="turtle")
 
-tmp = tempfile.NamedTemporaryFile(suffix=".ttl", delete=False, mode="w")
-export_rdf(graph.to_dict(), tmp.name, format="turtle")
-with open(tmp.name) as f:
-    data_ttl = f.read()
-os.unlink(tmp.name)
+# Prepare RDF data string
+data_ttl = """
+@prefix ex: <https://basel.eba.eu/ontology/> .
+
+<http://example.org/loan-001> a ex:LoanApplication ;
+    ex:ltv "0.78" ;
+    ex:pd "0.023" ;
+    ex:lgd "0.45" ;
+    ex:asset_class "CRE" .
+
+<http://example.org/loan-002> a ex:LoanApplication ;
+    ex:ltv "0.65" .
+"""
 
 report = _run_pyshacl(data_ttl, shacl_ttl)
 print(f"Loan portfolio conforms: {report.conforms}")
@@ -561,6 +687,8 @@ def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
     if not report.conforms:
         print(f"Graph validation FAILED — {report.violation_count} violation(s)")
         report.explain_violations()
+        for v in report.violations:
+            print(v.explanation)
         sys.exit(1)
 
     print(f"Graph validation PASSED ({report.warning_count} warning(s))")
@@ -575,3 +703,4 @@ def validate_before_publish(data_graph_str: str, ontology: dict) -> None:
 - [Export & Serialization](export) — serialize graph data to Turtle/RDF/XML for `_run_pyshacl` input
 - [Conflict Resolution](conflict-resolution) — detect and resolve data conflicts before SHACL validation
 - [Change Management](change-management) — version-gate SHACL shapes alongside ontology versions
+

docs/guides/visualization.md

@@ -6,6 +6,72 @@ icon: "chart-network"
 
 `KGVisualizer`, `AnalyticsVisualizer`, `TemporalVisualizer`, and `OntologyVisualizer` turn graph dicts, analytics results, and ontologies into interactive HTML dashboards or static images in a single method call. Use them to present centrality rankings, community clusters, event timelines, and before/after snapshot diffs to stakeholders without writing any rendering code.
 
+## What Is Visualization?
+
+Visualization converts graph data into interactive charts, network diagrams, timelines, and other visual formats that humans can interpret. It transforms abstract graph structures and analytical results into visual representations that reveal patterns, relationships, and insights.
+
+**Visualization vs. analytics:** Analytics computes numerical measures like centrality scores and community memberships. Visualization renders those measures as colored nodes, sized by importance, grouped by community.
+
+**Visualization vs. reasoning:** Reasoning derives new logical facts from existing data. Visualization presents existing facts and analytical results in visual form to support human interpretation and decision-making.
+
+Visualization helps humans understand graph structure, analytical results, and temporal patterns that would be difficult to interpret from raw data alone.
+
+## Why Use Visualization?
+
+**Visual exploration:** Interactive graphs let you pan, zoom, hover, and filter to explore large networks that would be overwhelming as text or tables.
+
+**Investigation support:** Highlighting paths between entities, color-coding by entity type, and sizing nodes by importance helps analysts identify patterns and focus investigation efforts.
+
+**Communication:** Visual presentations make complex graph relationships accessible to stakeholders who don't work directly with the data.
+
+**Reporting:** Static visualizations provide evidence and support for written reports, presentations, and regulatory submissions.
+
+## When To Use / When Not To Use
+
+**Visualization is appropriate for:**
+- Presenting graph structure and analytical results to humans
+- Exploring relationships and patterns in medium-sized graphs (10-1000 nodes)
+- Creating reports and presentations for stakeholders
+- Investigating specific paths or neighborhoods within graphs
+- Communicating findings from analytics or reasoning workflows
+
+**Graph traversal may be enough for:**
+- Programmatic exploration of relationships
+- Simple queries about specific paths or connections
+- Automated workflows that don't require human interpretation
+
+**Analytics may be more useful for:**
+- Computing numerical measures and rankings
+- Finding communities or centrality scores programmatically  
+- Quantitative comparisons that don't need visualization
+
+**Reasoning may be more useful for:**
+- Deriving new facts through logical inference
+- Rule-based decision making
+- Automated policy enforcement
+
+**Visualization becomes impractical when:**
+- Graphs exceed ~1000 nodes (browser performance degrades)
+- The network is too dense to interpret visually
+- You need programmatic analysis rather than human interpretation
+
+## Typical Visualization Workflow
+
+**Graph → Filter → Visualize → Interpret → Investigate**
+
+Most effective visualization follows this pattern:
+1. **Start with your knowledge graph** from `ContextGraph` or analytics results
+2. **Filter to a meaningful subgraph** — avoid visualizing entire enterprise graphs
+3. **Choose appropriate visualization** — network, timeline, heatmap, or rankings
+4. **Interpret the visual patterns** — clusters, central nodes, temporal trends
+5. **Investigate interesting findings** — drill down on unexpected patterns or outliers
+
+Always filter before visualizing. A 10,000-node enterprise graph becomes meaningful when filtered to the 50 most central nodes or the subgraph around a specific entity of interest.
+
+<Info>
+  **Performance Warning:** Large graphs (>1000 nodes) cause browser performance issues and become visually overwhelming. Interactive network visualizations work best with 10-1000 nodes. For larger graphs, use analytics to identify the most important subgraphs, then visualize those filtered results.
+</Info>
+
 <Info>
   All visualizers accept `output="interactive"` (Plotly/pyvis HTML, shown in Jupyter or saved to file) or `output="static"` (PNG/SVG via Matplotlib). Omit `file_path` to get the figure object back for further manipulation.
 </Info>
@@ -199,7 +265,7 @@ tv.visualize_timeline(
 
 ## Comparing Two Graph Snapshots Side-by-Side
 
-When the question is "what changed between March 14 and April 14?", `visualize_snapshot_comparison` takes two named snapshots from `TemporalVersionManager` and renders a side-by-side diff view showing nodes and edges added or removed.
+When the question is "what changed between March 14 and April 14?", `visualize_snapshot_comparison` takes two named snapshots from `TemporalVersionManager` and renders a line chart comparing graph metrics (entities, relationships, density) across the provided snapshots.
 
 ```python
 from semantica.change_management import TemporalVersionManager
@@ -393,6 +459,7 @@ from semantica.context import AgentContext, ContextGraph
 from semantica.vector_store import VectorStore
 from semantica.visualization import KGVisualizer, EmbeddingVisualizer, OntologyVisualizer
 from semantica.ontology import OntologyGenerator
+import numpy as np
 
 graph = ContextGraph(advanced_analytics=True)
 ctx   = AgentContext(
@@ -426,10 +493,12 @@ ov.visualize_hierarchy(ontology, output="interactive", file_path="drug_hierarchy
 ov.visualize_structure(ontology, output="interactive", file_path="drug_ontology.html")
 
 # UMAP projection and similarity heatmap for drug embeddings
-embeddings = [[0.1, 0.2, 0.3], [0.15, 0.22, 0.31], [0.8, 0.7, 0.6]]
+embeddings = np.array([[0.1, 0.2, 0.3], [0.15, 0.22, 0.31], [0.8, 0.7, 0.6]])
 labels     = ["Metformin", "Dapagliflozin", "Semaglutide"]
 
 ev = EmbeddingVisualizer()
+# UMAP (Uniform Manifold Approximation and Projection) reduces high-dimensional 
+# embeddings to 2D while preserving local neighborhood structure
 ev.visualize_2d_projection(
     embeddings, labels, method="umap",
     output="interactive", file_path="drug_embeddings.html",
@@ -514,6 +583,18 @@ if snap1 and snap2:
 
 </Tabs>
 
+## Common Pitfalls
+
+**Rendering massive graphs.** Attempting to visualize graphs with thousands of nodes crashes browsers and creates uninterpretable hairballs. Always filter large graphs to meaningful subsets before visualization.
+
+**Treating visual proximity as proof of relationships.** Nodes that appear close in a visualization aren't necessarily closely related in the graph structure. Visual layout algorithms optimize for readability, not semantic accuracy.
+
+**Visualizing duplicate/unclean data.** Duplicate entities, inconsistent naming, and data quality issues are amplified in visualizations. Clean your graph data before creating visual presentations for stakeholders.
+
+**Overloading tooltips with huge text fields.** Hovering over a node shouldn't display entire document contents. Include only essential metadata in hover tooltips — entity type, name, and key properties.
+
+**Running visualizations before graph cleanup.** Visualizations reflect data quality issues directly. Entities with inconsistent names, duplicate nodes, and missing relationships create confusing and misleading visual representations.
+
 ## Output Modes
 
 Every visualizer method accepts the same two output modes:

docs/integrations/databricks.md

@@ -0,0 +1,199 @@
+---
+title: "Databricks Integration"
+description: "Ingest Unity Catalog metadata and Delta Lake tables from Databricks into Semantica's KG pipeline."
+icon: "cloud"
+---
+
+> Extract Delta Lake tables and Unity Catalog metadata (schemas, lineage) from Databricks into Semantica with personal access token or OAuth M2M authentication.
+
+
+## Installation
+
+```bash
+# Install with Databricks support
+pip install "semantica[db-databricks]"
+
+# Or install the connectors separately
+pip install databricks-sdk databricks-sql-connector
+```
+
+
+## Basic Usage
+
+```python
+from semantica.ingest import DatabricksIngestor
+import os
+
+ingestor = DatabricksIngestor(
+    host=os.getenv("DATABRICKS_HOST"),           # e.g. https://adb-xxx.azuredatabricks.net
+    token=os.getenv("DATABRICKS_TOKEN"),
+    http_path=os.getenv("DATABRICKS_HTTP_PATH"),  # SQL warehouse or cluster HTTP path
+    catalog=os.getenv("DATABRICKS_CATALOG", "main"),
+    schema=os.getenv("DATABRICKS_SCHEMA", "default"),
+)
+
+data = ingestor.ingest_table("customers")
+print(f"Retrieved {data.row_count} rows: columns: {data.columns}")
+```
+
+<Tip>
+Use environment variables (or a `.env` file with `python-dotenv`) to keep credentials out of source code. `DatabricksIngestor()` with no arguments reads from `DATABRICKS_*` environment variables automatically.
+</Tip>
+
+
+## Authentication Methods
+
+<Tabs>
+  <Tab title="Personal Access Token">
+    ```python
+    ingestor = DatabricksIngestor(
+        host="https://adb-xxx.azuredatabricks.net",
+        token="dapi-xxxxxxxx",
+        http_path="/sql/1.0/warehouses/xxxxxxxx",
+    )
+    ```
+  </Tab>
+  <Tab title="OAuth M2M (Recommended)">
+    ```python
+    ingestor = DatabricksIngestor(
+        host="https://adb-xxx.azuredatabricks.net",
+        client_id="your_service_principal_client_id",
+        client_secret="your_service_principal_client_secret",
+        http_path="/sql/1.0/warehouses/xxxxxxxx",
+    )
+    ```
+    Preferred for production: no long-lived personal token stored in config.
+  </Tab>
+</Tabs>
+
+<Note>
+`http_path` identifies the SQL warehouse or all-purpose cluster used for query execution. Find it in the Databricks UI under **SQL Warehouses → Connection details**. Unity Catalog metadata calls (`list_catalogs`, `get_table_schema`, `get_table_lineage`, …) only need `host` and credentials — `http_path` is not required for those.
+</Note>
+
+
+## Querying
+
+### Ingest a table with filters
+
+```python
+data = ingestor.ingest_table(
+    "customers",
+    catalog="main",
+    schema="default",
+    where="country = 'USA' AND created_date > '2024-01-01'",
+    order_by="created_date DESC",
+    limit=10000,
+)
+```
+
+### Custom SQL
+
+```python
+data = ingestor.ingest_query("""
+    SELECT customer_id, SUM(amount) AS total_amount
+    FROM main.default.sales
+    WHERE date >= '2024-01-01'
+    GROUP BY customer_id
+""")
+```
+
+
+## Unity Catalog Metadata
+
+### Schema introspection
+
+```python
+schema = ingestor.get_table_schema("customers")
+for column in schema["columns"]:
+    print(f"{column['name']}: {column['type']}")
+```
+
+### Catalogs, schemas, and tables
+
+```python
+catalogs = ingestor.list_catalogs()
+schemas = ingestor.list_schemas(catalog="main")
+tables = ingestor.list_tables(catalog="main", schema="default")
+```
+
+### Table and column lineage
+
+```python
+lineage = ingestor.get_table_lineage("customers", catalog="main", schema="default")
+print(lineage["upstream"])    # tables that feed into `customers`
+print(lineage["downstream"])  # tables derived from `customers`
+```
+
+Use `get_table_lineage` to build `Table --DEPENDS_ON--> Table` edges in the knowledge graph directly from Unity Catalog's lineage tracking, without re-deriving lineage from query logs.
+
+<Tip>
+Pass `include_column_lineage=True` to also resolve per-column upstream/downstream references (one extra Unity Catalog request per column, so it's opt-in):
+
+```python
+lineage = ingestor.get_table_lineage(
+    "customers", catalog="main", schema="default", include_column_lineage=True,
+)
+print(lineage["columns"]["email"])
+# {"upstream": ["main.default.raw_customers.email_address"], "downstream": []}
+```
+
+</Tip>
+
+
+## Export as Semantica Documents
+
+```python
+documents = ingestor.export_as_documents(
+    data,
+    id_field="customer_id",
+    text_fields=["name", "email", "notes"],
+)
+print(f"Created {len(documents)} documents for processing")
+```
+
+
+## Batch Processing Large Tables
+
+```python
+PAGE_SIZE = 5000
+for page in range(total_pages):
+    data = ingestor.ingest_table(
+        "large_table",
+        limit=PAGE_SIZE,
+        offset=page * PAGE_SIZE,
+    )
+    process_batch(data)
+```
+
+Or use the built-in `batch_size` parameter:
+
+```python
+data = ingestor.ingest_query(
+    "SELECT * FROM main.default.large_table",
+    batch_size=5000,
+)
+```
+
+
+## Troubleshooting
+
+```python
+from semantica.ingest import DatabricksConnector
+
+connector = DatabricksConnector(
+    host="https://adb-xxx.azuredatabricks.net",
+    token="dapi-xxxxxxxx",
+    http_path="/sql/1.0/warehouses/xxxxxxxx",
+)
+if not connector.test_connection():
+    print("Connection failed: check host, http_path, and credentials")
+```
+
+
+## See Also
+
+- [Ingest Module](../reference/ingest) — Full DatabricksIngestor and all other ingestors.
+- [Snowflake Integration](snowflake) — Companion connector for a Snowflake + Databricks hybrid estate.
+- [Pipeline](../reference/pipeline) — Use Databricks ingestion as a pipeline step.
+- [Installation](../installation) — All optional dependency extras.
+- [Knowledge Graph](../reference/kg) — Build a KG from ingested Databricks data.

docs/integrations/snowflake.md

@@ -172,6 +172,7 @@ if not connector.test_connection():
 ## See Also
 
 - [Ingest Module](../reference/ingest) — Full SnowflakeIngestor and all other ingestors.
+- [Databricks Integration](databricks) — Companion connector for a Snowflake + Databricks hybrid estate.
 - [Pipeline](../reference/pipeline) — Use Snowflake ingestion as a pipeline step.
 - [Installation](../installation) — All optional dependency extras.
 - [Knowledge Graph](../reference/kg) — Build a KG from ingested Snowflake data.

docs/migration/kg-provenance-tracker.md

@@ -0,0 +1,56 @@
+---
+title: "Migrating from kg.ProvenanceTracker"
+description: "How to move from the deprecated semantica.kg.ProvenanceTracker to the unified semantica.provenance.ProvenanceManager."
+---
+
+## Why migrate
+
+`semantica.kg.ProvenanceTracker` is deprecated and will be removed in a future major version. It was a standalone, in-memory implementation that never delegated to the unified provenance backend — `semantica.provenance.ProvenanceManager` is that backend, and is now the supported way to track entity and relationship provenance across every Semantica module (see the [Provenance & Audit Trails guide](/guides/provenance)).
+
+Every method on `kg.ProvenanceTracker` now emits a `DeprecationWarning` on use, but existing code keeps working unchanged until the class is removed — there is no forced migration deadline yet.
+
+## Method mapping
+
+| `kg.ProvenanceTracker` | `ProvenanceManager` equivalent | Notes |
+| --- | --- | --- |
+| `ProvenanceTracker()` | `ProvenanceManager()` | `ProvenanceManager` also accepts `storage_path=` for SQLite persistence instead of in-memory only. |
+| `track_entity(entity_id, source, metadata)` | `track_entity(entity_id, source, metadata)` | Same call shape. `ProvenanceManager` additionally auto-links each update to its prior version via `parent_entity_id`. |
+| `get_all_sources(entity_id)` | `get_all_sources(entity_id)` | Field name differs: the `kg` tracker returns each record's time under `"recorded_at"`; `ProvenanceManager` returns `"timestamp"`. |
+| `clear(entity_id=None)` | `clear()` | `ProvenanceManager.clear()` clears all provenance data; there is no per-entity clear yet. |
+| `query_recorded_between(start, end)` | *No direct equivalent yet* | Filter the entries returned by `get_lineage()` / `trace_lineage()` client-side in the meantime. |
+| `revision_history(fact_id)` | *No direct equivalent yet* | `get_lineage(fact_id)["lineage_chain"]` returns the full chain of `ProvenanceEntry` records but not in the same versioned shape. |
+| `export_audit_log(fact_ids, format)` | *No direct equivalent yet* | Build the export from `get_lineage()` output, or serialize `get_statistics()` for a summary view. |
+
+Methods with no direct equivalent are not planned to be reimplemented on `kg.ProvenanceTracker` — they will need a small adapter in caller code, or a feature request against `ProvenanceManager` if you rely on them heavily.
+
+## Example
+
+```python
+# Before
+from semantica.kg import ProvenanceTracker
+
+tracker = ProvenanceTracker()
+tracker.track_entity("entity_1", source="doc_1", metadata={"confidence": 0.9})
+sources = tracker.get_all_sources("entity_1")  # [{"source": ..., "recorded_at": ..., "confidence": 0.9}]
+
+# After
+from semantica.provenance import ProvenanceManager
+
+prov = ProvenanceManager()
+prov.track_entity("entity_1", source="doc_1", metadata={"confidence": 0.9})
+sources = prov.get_all_sources("entity_1")  # [{"source": ..., "timestamp": ..., "metadata": {...}, ...}]
+```
+
+## Suppressing the warning during migration
+
+If you need to keep using `kg.ProvenanceTracker` temporarily and want to silence the warning while you plan the switch:
+
+```python
+import warnings
+
+with warnings.catch_warnings():
+    warnings.simplefilter("ignore", DeprecationWarning)
+    tracker = ProvenanceTracker()
+```
+
+This is a stopgap, not a fix — plan to move to `ProvenanceManager` before `kg.ProvenanceTracker` is removed.

docs/reference/context.md

@@ -272,7 +272,7 @@ icon: "brain"
 </Tip>
 
 <Tip>
-  **Persist your vector store between runs.** Pass `index_path="context.faiss"` to `VectorStore` so the FAISS index survives process restarts.
+  **Persist your context between runs.** `VectorStore` does not auto-persist — passing `index_path=` to its constructor is a no-op. Call `context.save("agent_state/")` to write memory, the vector index, and the graph to disk, and `context.load("agent_state/")` on the next process to restore them. See the "Persist & Restore" tab under [Real-World Patterns](#real-world-patterns) below.
 </Tip>
 
 ### Memory Methods

docs/reference/ingest.md

@@ -27,6 +27,7 @@ icon: "database"
 | `RepoIngestor` | Git repositories: source files, commit history, and metadata |
 | `DBIngestor` | SQL databases via SQLAlchemy: tables, views, and custom queries |
 | `SnowflakeIngestor` | Snowflake data warehouse queries and table exports |
+| `DatabricksIngestor` | Databricks Unity Catalog metadata, Delta table queries, and lineage |
 | `ParquetIngestor` | Apache Parquet files and partitioned datasets with column selection |
 | `XMLIngestor` | XXE-safe XML parsing with optional XSD schema validation |
 | `EmailIngestor` | IMAP/POP3 email ingestion with attachment extraction |
@@ -440,6 +441,24 @@ result = ingest("ontology.ttl")             # -> {"ontology": OntologyData}
     result = ingestor.ingest_query("SELECT * FROM documents")
     result = ingestor.ingest_table("documents")
     ```
+
+    ### DatabricksIngestor
+
+    ```python
+    from semantica.ingest import DatabricksIngestor
+    import os
+
+    ingestor = DatabricksIngestor(
+        host=os.getenv("DATABRICKS_HOST"),
+        token=os.getenv("DATABRICKS_TOKEN"),
+        http_path=os.getenv("DATABRICKS_HTTP_PATH"),
+        catalog="main",
+        schema="default",
+    )
+    result = ingestor.ingest_query("SELECT * FROM documents")
+    result = ingestor.ingest_table("documents")
+    lineage = ingestor.get_table_lineage("documents")
+    ```
   </Tab>
   <Tab title="Stream">
     ### StreamIngestor
@@ -628,4 +647,5 @@ result = ingest_file("source_path", method="my_format")
 - [Parse](parse) — Parse raw sources into structured text and tables.
 - [Pipeline](pipeline) — Orchestrate ingest as the first pipeline step.
 - [Snowflake Integration](../integrations/snowflake) — Snowflake-specific setup and authentication guide.
+- [Databricks Integration](../integrations/databricks) — Databricks Unity Catalog setup, authentication, and lineage guide.
 - [Provenance](provenance) — Track lineage from ingest through to inference.

docs/reference/pipeline.md

@@ -495,6 +495,40 @@ result   = engine.execute_pipeline(
   Delta detection uses SHA-256 checksums on source content. Only sources whose checksum differs from `base_version_id` are passed to downstream steps. For pipelines that run hourly or daily against a growing corpus, delta mode eliminates redundant re-embedding and re-extraction.
 </Note>
 
+## SPARQL CONSTRUCT Template Steps
+
+Use the `"construct_template"` step type to render and execute a [SPARQL CONSTRUCT template](triplet_store#sparql-construct-templates) as part of a pipeline. `store_backend` and `construct_template_registry` are execution-time resources, not step config — pass them to `execute_pipeline()`, the same way `delta_mode` steps receive `version_manager` and `triplet_store`:
+
+```python
+from semantica.pipeline import PipelineBuilder, ExecutionEngine
+from semantica.triplet_store.construct_templates import construct_template_step_handler
+
+builder = PipelineBuilder()
+builder.add_step(
+    "apply_person_template",
+    "construct_template",
+    handler=construct_template_step_handler,
+    template_name="person_to_foaf",
+    params={"subject": "http://ex.org/p1", "name": "Alice", "age": 30},
+    target_graph="http://ex.org/graphs/people",
+)
+pipeline = builder.build("person_pipeline")
+
+engine = ExecutionEngine()
+result = engine.execute_pipeline(
+    pipeline,
+    data=None,
+    store_backend=store,                   # required: a BlazegraphStore instance
+    construct_template_registry=registry,  # required: holds the registered template
+)
+
+triplets = result.output   # List[Triplet], already persisted via store.add_triplets
+```
+
+<Note>
+  `construct_template` steps raise `ProcessingError` if `store_backend` or `construct_template_registry` is missing from `execute_pipeline()`'s options, and `ValidationError` if `template_name` isn't registered.
+</Note>
+
 ## Schemas
 
 <AccordionGroup>

docs/reference/triplet_store.md

@@ -277,6 +277,65 @@ store.execute_query("""
   **`execute_query()` returns `QueryResult`, not a list.** Iterate `result.bindings`, not `result` directly. Each binding is a dict mapping variable name → `{"value": ..., "type": ...}`.
 </Warning>
 
+## SPARQL CONSTRUCT Templates
+
+`semantica.triplet_store.construct_templates` provides parameterized SPARQL `CONSTRUCT` query templates: define a reusable query once, substitute typed parameters safely, and persist the resulting triples in one call. This is available for the **Blazegraph backend only** (see [Backends](#backends) above) — `BlazegraphStore.execute_sparql()` is the only backend with CONSTRUCT-aware RDF parsing.
+
+```python
+from semantica.triplet_store.construct_templates import (
+    ConstructTemplate,
+    ParameterDescriptor,
+    ConstructTemplateRegistry,
+    render_construct_template,
+    execute_construct_template,
+)
+from semantica.triplet_store import BlazegraphStore
+
+# Define and register a template
+template = ConstructTemplate(
+    name="person_to_foaf",
+    description="Maps a person record subject to a foaf:name triple",
+    construct_query="""
+        PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+        CONSTRUCT { {{subject}} foaf:name {{name}} ; foaf:age {{age}} }
+        WHERE { {{subject}} a <http://ex.org/Person> }
+    """,
+    parameters=[
+        ParameterDescriptor(name="subject", type="uri", required=True),
+        ParameterDescriptor(name="name", type="literal", required=True),
+        ParameterDescriptor(
+            name="age", type="typed-literal", required=False, default=0,
+            datatype="xsd:integer",
+        ),
+    ],
+)
+
+registry = ConstructTemplateRegistry()
+registry.register(template)
+
+# Render only: inspect the substituted SPARQL string, no network call
+sparql = render_construct_template(
+    registry.get("person_to_foaf"),
+    params={"subject": "http://ex.org/p1", "name": "Alice", "age": 30},
+)
+
+# Render + execute + persist in one call
+store = BlazegraphStore(endpoint="http://localhost:9999/blazegraph", namespace="kb")
+triplets = execute_construct_template(
+    template=registry.get("person_to_foaf"),
+    params={"subject": "http://ex.org/p1", "name": "Alice", "age": 30},
+    store_backend=store,
+    target_graph="http://ex.org/graphs/people",
+)
+# triplets: List[Triplet], already persisted via store.add_triplets
+```
+
+Each `ParameterDescriptor.type` controls how its value is rendered: `"uri"` values are validated against an allowlist and wrapped in `<...>`, `"literal"` values are escaped and quoted, and `"typed-literal"` values require a `datatype` (e.g. `"xsd:integer"`) and render unquoted for numeric/boolean XSD types. Placeholders use `{{param}}` rather than SPARQL's own `?param` syntax so template placeholders are never confused with real SPARQL variables in the query body.
+
+<Note>
+  CONSTRUCT templates are Blazegraph-only. `execute_construct_template()` raises `ProcessingError` if `store_backend` does not implement both `execute_sparql()` and `add_triplets()`.
+</Note>
+
 ## SPARQL Result Pagination
 
 For large result sets, paginate with LIMIT and OFFSET:

pyproject.toml

@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "semantica"
-version = "0.5.1"
+version = "0.6.0"
 description = "Accountability and context layer for AI agents. Context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
 readme = "README.md"
 license = { text = "MIT" }
 
-authors = [{ name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com" }]
-maintainers = [{ name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com" }]
+authors = [{ name = "Semantica", email = "kaif@getsemantica.ai" }]
+maintainers = [{ name = "Semantica", email = "kaif@getsemantica.ai" }]
 
 requires-python = ">=3.8"
 
@@ -51,7 +51,7 @@ dependencies = [
   "umap-learn>=0.5.12",
   "spacy>=3.4.0",
   "transformers>=4.20.0",
-  "torch>=1.12.0",
+  "torch>=1.13.1",
   "sentence-transformers>=2.2.0",
   "rdflib>=6.2.0",
   "networkx>=2.8.0",
@@ -62,14 +62,14 @@ dependencies = [
   "requests>=2.34.2",
   "GitPython>=3.1.50",
   "chardet>=7.4.3",
-  "protobuf>=5.29.1,<7.0",
-  "grpcio>=1.71.2",
+  "protobuf>=5.29.1,<8.0",
+  "grpcio>=1.81.1",
   "beautifulsoup4>=4.15.0",
   "lxml>=6.1.1",
   "pypdf2>=2.10.0",
   "python-docx>=1.2.0",
   "openpyxl>=3.1.5",
-  "pillow>=11.3.0",
+  "pillow>=12.2.0",
   "librosa>=0.9.0",
   "opencv-python>=4.13.0.92",
   "faiss-cpu>=1.7.0",
@@ -79,7 +79,7 @@ dependencies = [
   "pydantic>=2.13.4",
   "click>=8.4.2",
   "rich>=12.5.0",
-  "tqdm>=4.64.0",
+  "tqdm>=4.68.3",
   "pyyaml>=6.0",
   "toml>=0.10.0",
   "python-dotenv>=1.2.1",
@@ -116,29 +116,34 @@ llm-all = [
 # ---- Document Parsing ----
 parse-docling = ["docling>=2.107.0"]
 
+# ---- SHACL Validation ----
+shacl = ["pyshacl>=0.25.0"]
+
 # ---- Database Connectors ----
 db-snowflake = ["snowflake-connector-python>=4.6.0", "cryptography>=49.0.0"]
+db-databricks = ["databricks-sdk>=0.60.0", "databricks-sql-connector>=4.0.0"]
 db-arrow = ["pyarrow>=24.0.0"]
 ingest-parquet = ["pyarrow>=24.0.0"]
 ingest-arrow = ["pyarrow>=24.0.0"]
 
 db-all = [
-  "semantica[db-snowflake,db-arrow]"
+  "semantica[db-snowflake,db-databricks,db-arrow]"
 ]
 
 # ---- Embedding / Models ----
 models-huggingface = [
   "transformers>=4.20.0",
-  "torch>=1.12.0"
+  "torch>=1.13.1"
 ]
 
 # ---- Graph Backends ----
 graph-neo4j = ["neo4j>=5.0.0"]
 graph-falkordb = ["falkordb>=1.0.0", "redis>=4.3.0"]
 graph-amazon-neptune = ["boto3>=1.24.0", "neo4j>=5.0.0"]
+graph-apache-age = ["psycopg2-binary>=2.9.0"]
 
 graph-all = [
-  "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune]"
+  "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune,graph-apache-age]"
 ]
 
 # ---- Vector Store Backends ----
@@ -147,9 +152,10 @@ vectorstore-weaviate = ["weaviate-client>=4.0.0"]
 vectorstore-pinecone = ["pinecone-client>=3.0.0"]
 vectorstore-milvus = ["pymilvus>=2.0.0"]
 vectorstore-pgvector = ["psycopg[binary,pool]>=3.0.0", "pgvector>=0.2.0"]
+vectorstore-sqlite = ["sqlite-vec>=0.1.1"]
 
 vectorstore-all = [
-  "semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus,vectorstore-pgvector]"
+  "semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus,vectorstore-pgvector,vectorstore-sqlite]"
 ]
 
 # ---- Infra / Queues / Workers ----
@@ -173,8 +179,8 @@ monitoring = [
   "prometheus-client>=0.14.0",
   "opentelemetry-api>=1.30.0,<2.0.0",
   "opentelemetry-sdk>=1.30.0,<2.0.0",
-  "opentelemetry-semantic-conventions>=0.58b0,<0.62",
-  "opentelemetry-instrumentation>=0.62b1,<0.62"
+  "opentelemetry-semantic-conventions>=0.58b0,<0.65",
+  "opentelemetry-instrumentation>=0.62b1,<0.65"
 ]
 
 # ---- Visualization ----
@@ -214,7 +220,7 @@ dev = [
   "isort>=6.1.0",
   "flake8>=4.0.0",
   "mypy>=0.971",
-  "pre-commit>=2.19.0",
+  "pre-commit>=4.6.0",
   "jupyter>=1.0.0",
   "ipykernel>=6.15.0"
 ]
@@ -233,8 +239,8 @@ explorer-lite = [
 
 # Everything (cross-platform — gpu excluded; install semantica[gpu] separately on Linux)
 all = [
-  "semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,explorer]",
-  "semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,agno]"
+  "semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
+  "semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno]"
 ]
 
 # ---------------- ENTRYPOINTS ----------------

semantica/__init__.py

@@ -10,7 +10,7 @@ Main exports:
     - Config: Configuration management
 """
 
-__version__ = "0.5.1"
+__version__ = "0.6.0"
 __author__ = "Semantica Contributors"
 __license__ = "MIT"
 

semantica/context/context_graph.py

@@ -2214,12 +2214,26 @@ class ContextGraph:
             
             # Node embeddings
             if "node_embedder" in self.kg_components:
-                embeddings = self.kg_components["node_embedder"].generate_embeddings(kg_graph)
-                analysis["node_embeddings"] = embeddings
+                node_labels = list(self.node_type_index.keys())
+                relationship_types = list(self.edge_type_index.keys())
+                if node_labels:
+                    embeddings = self.kg_components["node_embedder"].compute_embeddings(
+                        graph_store=self,
+                        node_labels=node_labels,
+                        relationship_types=relationship_types,
+                    )
+                    analysis["node_embeddings"] = embeddings
             
             self.logger.info("Completed comprehensive graph analysis")
             return analysis
             
+        except AttributeError as e:
+            # A broken internal method call (e.g. calling a method that doesn't
+            # exist on one of the kg_components) is a programming error, not a
+            # legitimate empty-analysis result. Log it distinctly and re-raise
+            # rather than masking it under the generic message below.
+            self.logger.error(f"Graph analysis failed due to a broken internal method call: {e}")
+            raise
         except Exception as e:
             self.logger.error(f"Failed to analyze graph with KG: {e}")
             return {"error": "Graph analysis failed due to an internal error"}

semantica/ingest/__init__.py

@@ -218,6 +218,10 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
     "SnowflakeIngestor": (".snowflake_ingestor", "SnowflakeIngestor"),
     "SnowflakeData": (".snowflake_ingestor", "SnowflakeData"),
     "SnowflakeConnector": (".snowflake_ingestor", "SnowflakeConnector"),
+    # Databricks ingestion
+    "DatabricksIngestor": (".databricks_ingestor", "DatabricksIngestor"),
+    "DatabricksData": (".databricks_ingestor", "DatabricksData"),
+    "DatabricksConnector": (".databricks_ingestor", "DatabricksConnector"),
     # Parquet ingestion
     "ParquetIngestor": (".parquet_ingestor", "ParquetIngestor"),
     "ParquetData": (".parquet_ingestor", "ParquetData"),
@@ -341,6 +345,10 @@ __all__ = [
     "SnowflakeIngestor",
     "SnowflakeData",
     "SnowflakeConnector",
+    # Databricks ingestion
+    "DatabricksIngestor",
+    "DatabricksData",
+    "DatabricksConnector",
     # Parquet ingestion
     "ParquetIngestor",
     "ParquetData",

semantica/kg/provenance_tracker.py

@@ -7,14 +7,23 @@ Tracks the sources and lineage of entities and relationships.
 import csv
 import io
 import json
+import warnings
 from datetime import datetime, timezone
 from typing import Any, Dict, List, Optional
 
+_MIGRATION_GUIDE_URL = "docs/migration/kg-provenance-tracker.md"
+
 
 class ProvenanceTracker:
     """
     Tracks provenance (source lineage) for knowledge graph entities.
 
+    .. deprecated::
+        ``ProvenanceTracker`` is deprecated in favor of
+        :class:`semantica.provenance.ProvenanceManager` and will be removed
+        in a future major version. See the migration guide at
+        ``docs/migration/kg-provenance-tracker.md``.
+
     Usage:
         tracker = ProvenanceTracker()
         tracker.track_entity("E1", "doc1.txt", metadata={"type": "file"})
@@ -22,6 +31,13 @@ class ProvenanceTracker:
     """
 
     def __init__(self):
+        warnings.warn(
+            "ProvenanceTracker is deprecated and will be removed in a future "
+            "major version. Use semantica.provenance.ProvenanceManager instead. "
+            f"See migration guide: {_MIGRATION_GUIDE_URL}",
+            DeprecationWarning,
+            stacklevel=2,
+        )
         self._records: Dict[str, List[Dict[str, Any]]] = {}
 
     def track_entity(
@@ -31,6 +47,13 @@ class ProvenanceTracker:
         metadata: Optional[Dict[str, Any]] = None,
     ) -> None:
         """Record that entity_id was derived from source."""
+        warnings.warn(
+            "ProvenanceTracker.track_entity() is deprecated; use "
+            "ProvenanceManager.track_entity() instead. "
+            f"See migration guide: {_MIGRATION_GUIDE_URL}",
+            DeprecationWarning,
+            stacklevel=2,
+        )
         if entity_id not in self._records:
             self._records[entity_id] = []
         entry: Dict[str, Any] = {
@@ -43,6 +66,13 @@ class ProvenanceTracker:
 
     def get_all_sources(self, entity_id: str) -> List[Dict[str, Any]]:
         """Return all provenance records for entity_id."""
+        warnings.warn(
+            "ProvenanceTracker.get_all_sources() is deprecated; use "
+            "ProvenanceManager.get_all_sources() instead. "
+            f"See migration guide: {_MIGRATION_GUIDE_URL}",
+            DeprecationWarning,
+            stacklevel=2,
+        )
         return self._records.get(entity_id, [])
 
     def clear(self, entity_id: Optional[str] = None) -> None:
@@ -66,6 +96,13 @@ class ProvenanceTracker:
             Flat list of matching provenance records (each dict includes
             the entity_id under the key "entity_id").
         """
+        warnings.warn(
+            "ProvenanceTracker.query_recorded_between() is deprecated with no "
+            "direct ProvenanceManager equivalent yet; see "
+            f"migration guide: {_MIGRATION_GUIDE_URL}",
+            DeprecationWarning,
+            stacklevel=2,
+        )
         start_dt = self._parse_dt(start)
         end_dt = self._parse_dt(end)
 
@@ -93,6 +130,21 @@ class ProvenanceTracker:
 
         Returns an empty list for a fact with no recorded provenance.
         """
+        warnings.warn(
+            "ProvenanceTracker.revision_history() is deprecated with no "
+            "direct ProvenanceManager equivalent yet; see "
+            f"migration guide: {_MIGRATION_GUIDE_URL}",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        return self._revision_history_no_warn(fact_id)
+
+    def _revision_history_no_warn(self, fact_id: str) -> List[Dict[str, Any]]:
+        """Internal, warning-free implementation of revision_history().
+
+        Used by other deprecated methods (e.g. export_audit_log()) that need
+        this logic without emitting a second DeprecationWarning per call.
+        """
         records = self._records.get(fact_id, [])
         if not records:
             return []
@@ -133,9 +185,16 @@ class ProvenanceTracker:
         Returns:
             String containing the serialized audit log.
         """
+        warnings.warn(
+            "ProvenanceTracker.export_audit_log() is deprecated with no "
+            "direct ProvenanceManager equivalent yet; see "
+            f"migration guide: {_MIGRATION_GUIDE_URL}",
+            DeprecationWarning,
+            stacklevel=2,
+        )
         rows = []
         for fact_id in fact_ids:
-            for entry in self.revision_history(fact_id):
+            for entry in self._revision_history_no_warn(fact_id):
                 rows.append({"fact_id": fact_id, **entry})
 
         if format == "json":

semantica/pipeline/pipeline_validator.py

@@ -85,13 +85,26 @@ class PipelineValidator:
         return self.validate_pipeline(pipeline, **options)
 
     def validate_pipeline(
-        self, pipeline: Union["Pipeline", "PipelineBuilder"], **options
+        self,
+        pipeline: Union["Pipeline", "PipelineBuilder"],
+        construct_template_registry: Optional[Any] = None,
+        **options,
     ) -> ValidationResult:
         """
         Validate entire pipeline.
 
         Args:
             pipeline: Pipeline object or builder
+            construct_template_registry: Optional ConstructTemplateRegistry
+                instance.  When provided, steps whose step_type is
+                "construct_template" are additionally validated: the
+                template_name is checked for existence in the registry and
+                step.config["params"] is checked for all required template
+                parameters.  When None (the default), a WARNING-level issue
+                is added for each construct_template step noting that
+                template existence could not be checked — existing callers
+                that do not pass this argument see zero behavior change for
+                any other step type.
             **options: Additional options
 
         Returns:
@@ -134,7 +147,10 @@ class PipelineValidator:
                     message=f"Validating {len(pipeline.steps)} pipeline steps...",
                 )
                 for step in pipeline.steps:
-                    step_result = self.validate_step(step)
+                    step_result = self.validate_step(
+                        step,
+                        _construct_template_registry=construct_template_registry,
+                    )
                     if not step_result.valid:
                         errors.extend(step_result.errors)
                     warnings.extend(step_result.warnings)
@@ -205,7 +221,19 @@ class PipelineValidator:
 
         Args:
             step: Pipeline step
-            **constraints: Validation constraints
+            **constraints: Validation constraints.  The following keys are
+                understood by this method and consumed internally; all others
+                are available for future extension:
+
+                allow_no_handler (bool, default False): suppress the
+                    "has no handler" warning.
+
+                _construct_template_registry (ConstructTemplateRegistry | None,
+                    default None): registry forwarded by validate_pipeline
+                    for construct_template step-type validation.  Prefixed
+                    with '_' to signal internal plumbing — callers invoking
+                    validate_step directly should use validate_pipeline's
+                    construct_template_registry keyword argument instead.
 
         Returns:
             Validation result
@@ -227,6 +255,47 @@ class PipelineValidator:
         if not step.config:
             warnings.append(f"Step '{step.name}' has no configuration")
 
+        # --- construct_template step-type-specific validation ---
+        # This is the first step-type-specific check in this validator.
+        # Future step-type-specific checks should follow the same pattern:
+        # extract a registry/context object from constraints via a
+        # '_<type>_registry' key forwarded by validate_pipeline.
+        if step.step_type == "construct_template":
+            registry = constraints.get("_construct_template_registry")
+            if registry is None:
+                warnings.append(
+                    f"Step '{step.name}' (construct_template): no "
+                    f"construct_template_registry provided — template "
+                    f"existence and required parameters could not be checked."
+                )
+            else:
+                template_name = step.config.get("template_name")
+                template = registry.get(template_name) if template_name else None
+                if not template_name or template is None:
+                    errors.append(
+                        f"Step '{step.name}' (construct_template): "
+                        f"template_name {template_name!r} is not registered "
+                        f"in the provided construct_template_registry."
+                    )
+                else:
+                    # Static required-param check — mirrors render_construct_template's
+                    # exact runtime logic: required=True means the param is mandatory
+                    # regardless of whether a default is declared (render only uses
+                    # default when required=False, so a required param with a default
+                    # still raises ValidationError at execution time).
+                    provided_params = step.config.get("params") or {}
+                    missing = [
+                        d.name
+                        for d in template.parameters
+                        if d.required and d.name not in provided_params
+                    ]
+                    if missing:
+                        errors.append(
+                            f"Step '{step.name}' (construct_template): missing "
+                            f"required parameter(s) for template "
+                            f"{template_name!r}: {missing}."
+                        )
+
         return ValidationResult(
             valid=len(errors) == 0, errors=errors, warnings=warnings
         )

semantica/provenance/manager.py

@@ -25,6 +25,7 @@ License: MIT
 """
 
 from typing import Optional, List, Dict, Any
+from collections.abc import Mapping
 from datetime import datetime
 
 from .schemas import ProvenanceEntry, SourceReference, PropertySource
@@ -115,7 +116,14 @@ class ProvenanceManager:
         # Check if entity already exists
         existing = self.storage.retrieve(entity_id)
         parent_id = kwargs.get("parent_entity_id")
-        
+
+        # If caller declared an explicit parent via metadata, honor it
+        # (unless parent_entity_id was already passed directly)
+        if not parent_id and metadata and isinstance(metadata, Mapping):
+            derived_from = metadata.get("derived_from")
+            if derived_from and isinstance(derived_from, str):
+                parent_id = derived_from
+
         # If source is a known entity, link it as parent (unless parent already set)
         if not parent_id and source and isinstance(source, str):
             try:
@@ -128,7 +136,15 @@ class ProvenanceManager:
             except Exception:
                 pass
 
+        # Track whether the caller explicitly supplied a parent link (via
+        # parent_entity_id kwarg, metadata["derived_from"], or source-as-
+        # known-entity-id resolution) BEFORE the history-preservation block
+        # below. If they did, that explicit value should not be silently
+        # overwritten by the auto-generated history pointer (#742).
+        explicit_parent_supplied = parent_id is not None
+
         # If entity exists, preserve history by archiving the old state
+        archived_history_id = None
         if existing:
             # Create a history entry for the previous state
             # Use timestamp or counter for uniqueness
@@ -145,8 +161,14 @@ class ProvenanceManager:
             # Store the history entry
             try:
                 self.storage.store(history_entry)
-                # Link new entry to this history entry
-                parent_id = history_id
+                archived_history_id = history_id
+                # Link new entry to this history entry — but only when the
+                # caller didn't explicitly supply a new parent on this call.
+                # An explicit parent_entity_id (or derived_from on branches
+                # that support it) is an intentional override signal and must
+                # not be silently replaced by internal bookkeeping (#742).
+                if not explicit_parent_supplied:
+                    parent_id = history_id
             except Exception:
                 pass # If history archiving fails, proceed with update but lose history (graceful degradation)
 
@@ -163,6 +185,16 @@ class ProvenanceManager:
             last_updated=datetime.utcnow().isoformat(),
             parent_entity_id=parent_id  # Link to history or explicit parent
         )
+
+        # Make the archived history entry discoverable via trace_lineage()'s
+        # BFS over used_entities — this ensures the previous version remains
+        # reachable in the lineage chain when explicit_parent_supplied is True
+        # and parent_entity_id points to the caller's explicit parent rather
+        # than the history pointer. When no explicit parent was supplied,
+        # parent_entity_id already IS archived_history_id, so appending it
+        # here too would duplicate the same id in both fields (#742 follow-up).
+        if archived_history_id and explicit_parent_supplied:
+            entry.used_entities.append(archived_history_id)
         
         # Compute checksum for integrity
         entry.checksum = compute_checksum(entry)
@@ -453,10 +485,14 @@ class ProvenanceManager:
         if not lineage_entries:
             return {}
         
-        # Aggregate metadata from all lineage entries
-        # Most recent entry's metadata takes precedence
+        # Aggregate metadata from all lineage entries.
+        # trace_lineage() is a BFS starting at entity_id, so lineage_entries[0]
+        # is always the queried entity itself, followed by its ancestors
+        # (parent, grandparent, ...). Apply ancestors first and the queried
+        # entity last so its own keys win on conflict, matching the intent
+        # that the "most recent"/current entity's metadata takes precedence.
         aggregated_metadata = {}
-        for entry in lineage_entries:
+        for entry in reversed(lineage_entries):
             if entry.metadata:
                 meta = entry.metadata
                 if isinstance(meta, str):

semantica/reasoning/reasoner.py

@@ -9,7 +9,7 @@ import re
 import uuid
 from dataclasses import dataclass, field
 from enum import Enum
-from typing import Any, Dict, List, Optional, Set, Union, Callable
+from typing import Any, Dict, List, Optional, Set, Tuple, Union, Callable
 
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
@@ -81,12 +81,50 @@ class Reasoner:
         self.rule_counter = 0
             
     def add_rule(self, rule_def: Union[str, Rule]) -> Rule:
-        """Add a rule to the reasoner."""
+        """Add a rule to the reasoner.
+
+        Rules with the same conditions and conclusion as an already-added
+        rule are not re-appended -- this keeps re-running the same setup
+        code (e.g. a Jupyter cell that calls add_rule() + add_fact() on an
+        existing Reasoner) idempotent instead of silently duplicating rules
+        on every rerun (#732).
+
+        On dedup, the ORIGINAL rule's confidence, priority, and metadata are
+        retained; the incoming rule's differing fields are discarded, not
+        merged or upserted -- except for priority, where self.rules is
+        re-sorted to reflect any change made directly on the retained Rule
+        object after it was first added (see the re-sort below). If the
+        incoming rule's confidence differs from the retained rule's, a
+        warning is logged so the discrepancy isn't silently swallowed.
+        """
         if isinstance(rule_def, Rule):
             rule = rule_def
         else:
             rule = self._parse_rule_definition(rule_def)
-            
+
+        for existing in self.rules:
+            if (
+                existing.rule_type == rule.rule_type
+                and existing.conditions == rule.conditions
+                and existing.conclusion == rule.conclusion
+            ):
+                self.logger.warning(
+                    f"Skipping duplicate rule (same conditions/conclusion as '{existing.rule_id}'): "
+                    f"IF {' AND '.join(map(str, rule.conditions))} THEN {rule.conclusion}"
+                )
+                if existing.confidence != rule.confidence:
+                    self.logger.warning(
+                        f"Duplicate rule '{existing.rule_id}' was re-added with a different "
+                        f"confidence ({rule.confidence}); the existing confidence "
+                        f"({existing.confidence}) is retained and the new value is discarded."
+                    )
+                # Rule is a mutable dataclass, so `existing.priority` may have
+                # changed since it was added -- re-sort so the dedup path
+                # keeps the same self-healing ordering the append path has,
+                # rather than leaving self.rules stale relative to priority.
+                self.rules.sort(key=lambda r: r.priority, reverse=True)
+                return existing
+
         self.rules.append(rule)
         # Sort rules by priority
         self.rules.sort(key=lambda r: r.priority, reverse=True)
@@ -180,17 +218,48 @@ class Reasoner:
             new_facts_added = False
             iteration += 1
             
+            # Snapshot facts that existed before this pass, so we can tell a
+            # fact that was already known apart from one newly derived during
+            # this same pass. Newly derived conclusions are added to
+            # self.facts immediately (not deferred to the end of the pass) so
+            # that later rules in this same pass can chain off facts inferred
+            # earlier in the pass -- e.g. "IF A THEN B" firing lets
+            # "IF B THEN C" fire in the same pass rather than requiring an
+            # extra outer iteration.
+            pre_pass_facts = frozenset(self.facts)
+            # Tracks conclusions newly derived in this pass, keyed to the
+            # InferenceResult already appended to `results`, so multiple
+            # derivations of the identical conclusion (different bindings
+            # and/or different rules within the same pass) merge their
+            # premises into one result instead of creating duplicates or
+            # silently dropping premises (the #733 fix).
+            pass_results: Dict[str, InferenceResult] = {}
+            
             for rule in self.rules:
-                matches = self._match_rule(rule)
-                for conclusion in matches:
-                    if conclusion not in self.facts:
-                        self.facts.add(conclusion)
-                        results.append(InferenceResult(
-                            conclusion=conclusion,
-                            rule_used=rule,
-                            confidence=rule.confidence
-                        ))
-                        new_facts_added = True
+                for conclusion, matched_facts in self._match_rule(rule):
+                    if conclusion in pass_results:
+                        # Another derivation of a conclusion already produced
+                        # earlier in this same pass: merge premises, dedup.
+                        existing = pass_results[conclusion]
+                        for fact in matched_facts:
+                            if fact not in existing.premises:
+                                existing.premises.append(fact)
+                        continue
+                    if conclusion in pre_pass_facts:
+                        # Already known before this pass started -- not a
+                        # new derivation.
+                        continue
+                    
+                    self.facts.add(conclusion)
+                    inference_result = InferenceResult(
+                        conclusion=conclusion,
+                        rule_used=rule,
+                        premises=list(matched_facts),
+                        confidence=rule.confidence
+                    )
+                    pass_results[conclusion] = inference_result
+                    results.append(inference_result)
+                    new_facts_added = True
                         
         self.progress_tracker.stop_tracking(
             tracking_id,
@@ -237,12 +306,12 @@ class Reasoner:
             
         # 1. Check if goal is already in facts
         if goal in self.facts:
-            return InferenceResult(conclusion=goal, premises=[])
+            return InferenceResult(conclusion=goal, premises=[goal])
             
         # 2. Check if goal matches a known fact pattern (unification)
         for fact in self.facts:
             if self._match_pattern(goal, fact, {}) is not None:
-                return InferenceResult(conclusion=fact, premises=[])
+                return InferenceResult(conclusion=fact, premises=[fact])
                 
         # 3. Try to prove via rules
         for rule in self.rules:
@@ -303,28 +372,45 @@ class Reasoner:
             conclusion=conclusion_str.strip()
         )
         
-    def _match_rule(self, rule: Rule) -> List[str]:
-        """Match rule conditions against facts and return instantiated conclusions."""
+    def _match_rule(self, rule: Rule) -> List[Tuple[str, List[str]]]:
+        """
+        Match rule conditions against facts and return instantiated conclusions
+        paired with the facts that satisfied each condition.
+
+        Returns:
+            List of (conclusion, matched_facts) tuples, where matched_facts is
+            the ordered list of facts bound to this rule's conditions.
+        """
         if not rule.conditions:
             return []
             
-        bindings_list = [{}] # List of possible variable bindings
+        # self.facts is not mutated anywhere within this method, so sort it
+        # once here rather than re-sorting on every (bindings, condition)
+        # pair below -- sorted() was previously called once per inner-loop
+        # entry, which re-allocates and re-sorts the full fact set repeatedly
+        # and is a hot spot for larger fact sets.
+        sorted_facts = sorted(self.facts)
+        
+        # Each entry pairs a set of variable bindings with the facts that were
+        # matched to produce those bindings, so the facts survive alongside
+        # the bindings as conditions accumulate.
+        bindings_list: List[Tuple[Dict[str, str], List[str]]] = [({}, [])]
         
         for condition in rule.conditions:
             new_bindings_list = []
-            for bindings in bindings_list:
-                for fact in self.facts:
+            for bindings, matched_facts in bindings_list:
+                for fact in sorted_facts:
                     match_bindings = self._match_pattern(condition, fact, bindings)
                     if match_bindings is not None:
-                        new_bindings_list.append(match_bindings)
+                        new_bindings_list.append((match_bindings, matched_facts + [fact]))
             bindings_list = new_bindings_list
             if not bindings_list:
                 break
                 
         results = []
-        for bindings in bindings_list:
+        for bindings, matched_facts in bindings_list:
             instantiated_conclusion = self._substitute(rule.conclusion, bindings)
-            results.append(instantiated_conclusion)
+            results.append((instantiated_conclusion, matched_facts))
             
         return results
         

semantica/triplet_store/blazegraph_store.py

@@ -32,11 +32,17 @@ from typing import Any, Dict, List, Optional
 from urllib.parse import urljoin, urlparse
 
 import requests
+from rdflib import Graph, Literal
 
 from ..semantic_extract.triplet_extractor import Triplet
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
+from . import sparql_escaping
+
+# _CONSTRUCT_QUERY_RE was moved to sparql_escaping.CONSTRUCT_QUERY_RE so both
+# BlazegraphStore and RDF4JStore share a single canonical implementation.
+# _is_construct_query below delegates to sparql_escaping.CONSTRUCT_QUERY_RE.
 
 
 class BlazegraphStore:
@@ -111,16 +117,52 @@ class BlazegraphStore:
         """Get SPARQL Update endpoint URL."""
         return urljoin(self.endpoint, f"/blazegraph/namespace/{self.namespace}/sparql")
 
+    def _is_construct_query(self, query: str) -> bool:
+        """
+        Detect whether `query` is a SPARQL CONSTRUCT query.
+
+        This is a dispatch helper local to BlazegraphStore, distinct from
+        QueryEngine._validate_query (which already treats CONSTRUCT as one of
+        its valid_keywords and therefore requires no change — CONSTRUCT
+        queries already pass validation today). This helper only decides
+        which HTTP Accept header and response parser execute_sparql uses; it
+        does not gate query validity.
+
+        Delegates to sparql_escaping.CONSTRUCT_QUERY_RE, which is the single
+        canonical CONSTRUCT-detection regex shared with RDF4JStore.
+        """
+        return sparql_escaping.CONSTRUCT_QUERY_RE.search(query) is not None
+
     def execute_sparql(self, query: str, **options) -> Dict[str, Any]:
         """
         Execute SPARQL query.
 
         Args:
             query: SPARQL query string
-            **options: Additional options
+            **options: Additional options:
+                - result_format: Optional[Literal["bindings", "construct"]].
+                  If omitted, auto-detected via _is_construct_query(query).
 
         Returns:
-            Query results
+            Query results. For non-CONSTRUCT queries (or when result_format
+            resolves to "bindings"), the existing shape is unchanged:
+                {"success": bool, "bindings": [...], "variables": [...], "metadata": {...}}
+            For CONSTRUCT queries (or result_format="construct"), the shape is:
+                {"success": bool, "bindings": [], "variables": [], "triples": [...],
+                 "metadata": {...}}
+            where "triples" is a list of (subject, predicate, object, metadata)
+            4-tuples parsed from the Turtle response via rdflib. subject and
+            predicate are always plain strings. object is the literal's
+            lexical value or the IRI string. metadata is a dict that is empty
+            ({}) for URIs and plain untyped/unlang-tagged literals, and
+            otherwise contains "datatype" (the datatype IRI as a string) and/
+            or "language" (the RFC 5646 language tag) for literals that carry
+            that information — preserving what would otherwise be lost by
+            collapsing every rdflib term down to str(term).
+
+        Raises:
+            ProcessingError: if not connected, the HTTP request fails, or (for
+                CONSTRUCT queries) the response body fails to parse as Turtle.
         """
         tracking_id = self.progress_tracker.start_tracking(
             module="triplet_store",
@@ -137,6 +179,70 @@ class BlazegraphStore:
 
             sparql_endpoint = self._get_sparql_endpoint()
 
+            result_format = options.get("result_format")
+            if result_format is None:
+                result_format = "construct" if self._is_construct_query(query) else "bindings"
+
+            if result_format == "construct":
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Sending CONSTRUCT query to Blazegraph endpoint..."
+                )
+                response = requests.post(
+                    sparql_endpoint,
+                    data={"query": query},
+                    headers={
+                        "Content-Type": "application/x-www-form-urlencoded",
+                        "Accept": "text/turtle",
+                    },
+                    timeout=self.timeout,
+                    auth=(self.username, self.password)
+                    if self.username and self.password
+                    else None,
+                )
+
+                response.raise_for_status()
+
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Parsing CONSTRUCT response as Turtle..."
+                )
+                graph = Graph()
+                try:
+                    graph.parse(data=response.content, format="turtle")
+                except Exception as parse_error:
+                    raise ProcessingError(
+                        f"Failed to parse CONSTRUCT response as Turtle: {parse_error}"
+                    ) from parse_error
+
+                triples = []
+                for s, p, o in graph:
+                    obj_metadata: Dict[str, Any] = {}
+                    if isinstance(o, Literal):
+                        if o.datatype is not None:
+                            obj_metadata["datatype"] = str(o.datatype)
+                        if o.language is not None:
+                            obj_metadata["language"] = str(o.language)
+                    triples.append((str(s), str(p), str(o), obj_metadata))
+
+                result = {
+                    "success": True,
+                    "bindings": [],
+                    "variables": [],
+                    "triples": triples,
+                    "metadata": {
+                        "query": query,
+                        "endpoint": sparql_endpoint,
+                        "result_format": "construct",
+                    },
+                }
+
+                self.progress_tracker.stop_tracking(
+                    tracking_id,
+                    status="completed",
+                    message=f"CONSTRUCT query executed: {len(triples)} triples",
+                )
+                return result
+
+            # Non-CONSTRUCT path — unchanged from prior behavior.
             self.progress_tracker.update_tracking(
                 tracking_id, message="Sending query to Blazegraph endpoint..."
             )
@@ -307,31 +413,12 @@ class BlazegraphStore:
         - Known prefixed names: ``xsd:integer``, ``rdf:langString``, etc.
 
         Raises ValueError for anything else.
+
+        Delegates to the shared sparql_escaping.resolve_datatype_iri so this
+        logic has one canonical implementation shared with the CONSTRUCT
+        template renderer (semantica/triplet_store/construct_templates.py).
         """
-        datatype = str(datatype)
-
-        # Already angle-bracketed — validate the inner IRI contains no whitespace
-        if datatype.startswith("<") and datatype.endswith(">"):
-            inner = datatype[1:-1]
-            if not inner or re.search(r"[\s<>\"{}|\\^`]", inner):
-                raise ValueError(f"Invalid datatype IRI: {datatype!r}")
-            return datatype
-
-        # Full absolute IRI without brackets
-        parsed = urlparse(datatype)
-        if parsed.scheme in {"http", "https", "urn"} and not re.search(r"[\s<>\"{}|\\^`]", datatype):
-            return f"<{datatype}>"
-
-        # Prefixed form — expand known prefixes only
-        if ":" in datatype:
-            prefix, local = datatype.split(":", 1)
-            if prefix in self._KNOWN_PREFIXES and re.match(r"^[A-Za-z0-9_\-\.]+$", local):
-                return f"<{self._KNOWN_PREFIXES[prefix]}{local}>"
-
-        raise ValueError(
-            f"Unsupported datatype {datatype!r}: use a full IRI (http/https/urn), "
-            f"an angle-bracketed IRI, or a known prefix (xsd/rdf/rdfs/owl/skos)."
-        )
+        return sparql_escaping.resolve_datatype_iri(datatype)
 
     def _is_uri_value(self, value: str) -> bool:
         """Detect if a value should be serialized as an IRI."""
@@ -346,15 +433,13 @@ class BlazegraphStore:
         return not re.search(r"\s", value)
 
     def _escape_literal(self, value: str) -> str:
-        """Escape string literal for SPARQL."""
-        return (
-            str(value)
-            .replace("\\", "\\\\")
-            .replace("\"", "\\\"")
-            .replace("\n", "\\n")
-            .replace("\r", "\\r")
-            .replace("\t", "\\t")
-        )
+        """Escape string literal for SPARQL.
+
+        Delegates to the shared sparql_escaping.escape_literal so this logic
+        has one canonical implementation shared with the CONSTRUCT template
+        renderer (semantica/triplet_store/construct_templates.py).
+        """
+        return sparql_escaping.escape_literal(value)
 
     def add_triplet(self, triplet: Triplet, **options) -> Dict[str, Any]:
         """Add single triplet."""

semantica/triplet_store/construct_templates.py

@@ -0,0 +1,828 @@
+"""
+SPARQL CONSTRUCT Query Templates (Blazegraph-only)
+
+This module provides parameterized SPARQL CONSTRUCT query templates: a
+`ConstructTemplate` defines a reusable CONSTRUCT query body with `{{param}}`
+placeholders, `ConstructTemplateRegistry` stores/retrieves templates by name,
+and `render_construct_template` safely substitutes parameter values into a
+validated, injection-safe SPARQL string.
+
+This module is intentionally Blazegraph-only. `execute_construct_template`
+renders a template, executes it via `store_backend.execute_sparql(...,
+result_format="construct")` (the Blazegraph CONSTRUCT-aware extension),
+converts the parsed RDF triples into `Triplet` objects, and persists them via
+`store_backend.add_triplets`.
+
+All literal escaping, URI allowlist validation, and datatype-IRI resolution
+is delegated to the shared `semantica.triplet_store.sparql_escaping` module —
+no escaping/validation logic is reimplemented here.
+
+Main Classes:
+    - ParameterDescriptor: Declares one template parameter's name/type/validation.
+    - ConstructTemplate: Named, reusable CONSTRUCT query definition.
+    - ConstructTemplateRegistry: Stores and retrieves ConstructTemplate instances.
+
+Main Functions:
+    - render_construct_template: Safely render a ConstructTemplate into SPARQL.
+    - execute_construct_template: Render, execute, parse, and persist in one call.
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import re
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Literal, Optional, Tuple
+
+from ..semantic_extract.triplet_extractor import Triplet
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+from .sparql_escaping import escape_literal, resolve_datatype_iri, validate_uri
+
+ParameterKind = Literal["uri", "literal", "typed-literal"]
+
+# XSD local names (case-insensitive) that render as unquoted numeric/boolean
+# literals rather than quoted "<value>"^^<iri> literals. Matched against the
+# local name of the *resolved* datatype IRI so this works whether the
+# descriptor's datatype was given as a prefixed name (xsd:integer), a full
+# IRI, or a bracketed IRI.
+_INTEGER_LOCAL_NAMES = frozenset({"integer", "int", "long", "short"})
+_DECIMAL_LOCAL_NAMES = frozenset({"decimal", "double", "float"})
+_BOOLEAN_LOCAL_NAMES = frozenset({"boolean"})
+_NUMERIC_UNQUOTED_LOCAL_NAMES = _INTEGER_LOCAL_NAMES | _DECIMAL_LOCAL_NAMES | _BOOLEAN_LOCAL_NAMES
+
+_CONSTRUCT_KEYWORD_RE = re.compile(r"\bCONSTRUCT\b", re.IGNORECASE)
+_WHERE_KEYWORD_RE = re.compile(r"\bWHERE\b", re.IGNORECASE)
+_PLACEHOLDER_RE = re.compile(r"\{\{|\}\}")
+
+
+@dataclass
+class ParameterDescriptor:
+    """Describes one substitution parameter accepted by a ConstructTemplate."""
+
+    name: str
+    """Placeholder name as it appears in the query body, e.g. "subject" for {{subject}}."""
+
+    type: ParameterKind = "literal"
+    """One of "uri" | "literal" | "typed-literal"."""
+
+    required: bool = True
+    """If True and no value/default is supplied at render time, render raises ValidationError."""
+
+    default: Optional[Any] = None
+    """Used when the caller omits this parameter and required=False."""
+
+    datatype: Optional[str] = None
+    """Only meaningful when type == "typed-literal". An XSD datatype token accepted by
+    the shared resolve_datatype_iri, e.g. "xsd:integer", "xsd:dateTime", or a full IRI.
+    Required when type == "typed-literal"; render_construct_template (and
+    ConstructTemplateRegistry.register) raise ValidationError if type == "typed-literal"
+    and datatype is None."""
+
+    language: Optional[str] = None
+    """Only meaningful when type == "literal". RFC 5646 language tag, e.g. "en"."""
+
+
+@dataclass
+class ConstructTemplate:
+    """Parameterized SPARQL CONSTRUCT query template."""
+
+    name: str
+    """Unique registry key, e.g. "person_to_foaf"."""
+
+    description: str
+    """Human-readable summary, shown by list()/get_template_info()."""
+
+    construct_query: str
+    """Full CONSTRUCT query body containing {{param}} placeholders, e.g.:
+    "CONSTRUCT { <{{subject}}> foaf:name {{name}} } WHERE { ... }"
+    Must contain the CONSTRUCT keyword — enforced at register() time, not at
+    dataclass construction time."""
+
+    parameters: List[ParameterDescriptor] = field(default_factory=list)
+    """Ordered list of accepted parameters. Order has no runtime meaning, only used for
+    documentation / get_template_info() output."""
+
+    target_graph: Optional[str] = None
+    """Optional default named-graph IRI used when render_construct_template's/
+    execute_construct_template's target_graph argument is not supplied. This value,
+    like any caller-supplied target_graph, is ALWAYS passed through the same
+    validate_uri/escape path as a "uri"-typed parameter before being interpolated —
+    never through a raw f-string."""
+
+    metadata: dict = field(default_factory=dict)
+    """Free-form, mirrors PipelineTemplate.metadata (e.g. {"category": "rdf_mapping"})."""
+
+
+class ConstructTemplateRegistry:
+    """
+    CONSTRUCT template management system (Blazegraph-only).
+
+    Method shape mirrors PipelineTemplateManager:
+        PipelineTemplateManager.register_template(template) -> None
+        PipelineTemplateManager.get_template(name) -> Optional[PipelineTemplate]
+        PipelineTemplateManager.list_templates(category=None) -> List[str]
+
+    Unlike PipelineTemplateManager.register_template (which silently overwrites
+    by name), this registry rejects duplicate names with ValidationError —
+    CONSTRUCT queries execute against real triple stores, so silent overwrite
+    is a correctness hazard.
+    """
+
+    def __init__(self, config: Optional[Dict[str, Any]] = None, **kwargs):
+        """
+        Initialize template registry.
+
+        Args:
+            config: Configuration dictionary.
+            **kwargs: Additional configuration options.
+        """
+        self.logger = get_logger("construct_template_registry")
+        self.config = config or {}
+        self.config.update(kwargs)
+        self.templates: Dict[str, ConstructTemplate] = {}
+
+    def register(self, template: ConstructTemplate) -> None:
+        """
+        Register a CONSTRUCT template.
+
+        Args:
+            template: ConstructTemplate to register.
+
+        Raises:
+            ValidationError: if template.name is already registered,
+                construct_query does not contain the CONSTRUCT keyword, or any
+                ParameterDescriptor with type == "typed-literal" is missing
+                datatype. Validation happens before any mutation — a failed
+                register() call never partially overwrites the registry.
+        """
+        if template.name in self.templates:
+            raise ValidationError(
+                f"Template already registered: {template.name!r}. "
+                f"Remove it first via remove() if you intend to replace it."
+            )
+
+        if not _CONSTRUCT_KEYWORD_RE.search(template.construct_query):
+            raise ValidationError(
+                f"Template {template.name!r}: construct_query does not contain "
+                f"the CONSTRUCT keyword."
+            )
+
+        for descriptor in template.parameters:
+            if descriptor.type == "typed-literal" and descriptor.datatype is None:
+                raise ValidationError(
+                    f"Template {template.name!r}: parameter {descriptor.name!r} has "
+                    f"type='typed-literal' but no datatype declared."
+                )
+
+        self.templates[template.name] = template
+        self.logger.info(f"Registered CONSTRUCT template: {template.name}")
+
+    def get(self, name: str) -> Optional[ConstructTemplate]:
+        """
+        Get template by name.
+
+        Mirrors PipelineTemplateManager.get_template(template_name) -> Optional[PipelineTemplate].
+        """
+        return self.templates.get(name)
+
+    def list(self, category: Optional[str] = None) -> List[str]:
+        """
+        List registered template names, optionally filtered by metadata["category"].
+
+        Mirrors PipelineTemplateManager.list_templates(category=None) -> List[str].
+        """
+        if category:
+            return [
+                name
+                for name, template in self.templates.items()
+                if template.metadata.get("category") == category
+            ]
+        return list(self.templates.keys())
+
+    def remove(self, name: str) -> bool:
+        """
+        Remove a template by name.
+
+        Returns:
+            True if a template was removed, False if name was not registered.
+        """
+        if name in self.templates:
+            del self.templates[name]
+            self.logger.info(f"Removed CONSTRUCT template: {name}")
+            return True
+        return False
+
+    # --- PipelineTemplateManager-name aliases, for call-site consistency ---
+
+    def register_template(self, template: ConstructTemplate) -> None:
+        """Alias for register(), matching PipelineTemplateManager.register_template."""
+        self.register(template)
+
+    def get_template(self, template_name: str) -> Optional[ConstructTemplate]:
+        """Alias for get(), matching PipelineTemplateManager.get_template."""
+        return self.get(template_name)
+
+    def list_templates(self, category: Optional[str] = None) -> List[str]:
+        """Alias for list(), matching PipelineTemplateManager.list_templates."""
+        return self.list(category)
+
+    def get_template_info(self, template_name: str) -> Optional[Dict[str, Any]]:
+        """
+        Get template information.
+
+        Mirrors PipelineTemplateManager.get_template_info.
+
+        Returns:
+            {"name", "description", "parameter_count", "target_graph", "metadata"}
+            or None if template_name is not registered.
+        """
+        template = self.get(template_name)
+        if not template:
+            return None
+
+        return {
+            "name": template.name,
+            "description": template.description,
+            "parameter_count": len(template.parameters),
+            "target_graph": template.target_graph,
+            "metadata": template.metadata,
+        }
+
+
+# ---------------------------------------------------------------------------
+# render_construct_template and its internal helpers
+# ---------------------------------------------------------------------------
+
+
+def _local_name_of_datatype_iri(datatype_iri: str) -> str:
+    """Extract the lower-cased local name from a resolved (bracketed) datatype IRI."""
+    inner = datatype_iri[1:-1] if datatype_iri.startswith("<") and datatype_iri.endswith(">") else datatype_iri
+    if "#" in inner:
+        return inner.rsplit("#", 1)[1].lower()
+    if "/" in inner:
+        return inner.rsplit("/", 1)[1].lower()
+    return inner.lower()
+
+
+def _render_numeric_literal(value: Any, local_name: str, descriptor_name: str) -> str:
+    """
+    Render an unquoted numeric/boolean literal for a "typed-literal" parameter
+    whose resolved datatype local name is in _NUMERIC_UNQUOTED_LOCAL_NAMES.
+
+    Raises:
+        ValidationError: if value cannot be coerced to the declared datatype.
+    """
+    if local_name in _INTEGER_LOCAL_NAMES:
+        if isinstance(value, bool):
+            raise ValidationError(
+                f"Parameter {descriptor_name!r}: expected an integer value for "
+                f"datatype local name {local_name!r}, got boolean {value!r}."
+            )
+        try:
+            return str(int(str(value)))
+        except (TypeError, ValueError):
+            raise ValidationError(
+                f"Parameter {descriptor_name!r}: value {value!r} is not a valid "
+                f"integer for its declared XSD datatype."
+            )
+
+    if local_name in _DECIMAL_LOCAL_NAMES:
+        try:
+            return str(float(str(value)))
+        except (TypeError, ValueError):
+            raise ValidationError(
+                f"Parameter {descriptor_name!r}: value {value!r} is not a valid "
+                f"decimal/double/float for its declared XSD datatype."
+            )
+
+    if local_name in _BOOLEAN_LOCAL_NAMES:
+        if isinstance(value, bool):
+            return "true" if value else "false"
+        normalized = str(value).strip().lower()
+        if normalized in ("true", "1"):
+            return "true"
+        if normalized in ("false", "0"):
+            return "false"
+        raise ValidationError(
+            f"Parameter {descriptor_name!r}: value {value!r} is not a valid "
+            f"boolean for its declared XSD datatype."
+        )
+
+    raise ValidationError(
+        f"Parameter {descriptor_name!r}: datatype local name {local_name!r} is not "
+        f"a recognized unquoted-numeric XSD datatype."
+    )
+
+
+def _find_matching_brace(text: str, open_index: int) -> int:
+    """
+    Given the index of an opening '{' in text, return the index of its matching '}'.
+
+    Skips over double-quoted string literal content while scanning, so a '{'
+    or '}' character that appears *inside* a rendered literal value (e.g. a
+    "typed-literal"/"literal" parameter whose value contains a brace
+    character — braces are not among the characters escape_literal escapes,
+    since SPARQL/Turtle string literals don't require it) does not corrupt
+    the brace-depth count. Quote-escaping is assumed to follow
+    escape_literal's convention (\\\\ and \\" are the only two-character
+    escapes that can produce a literal backslash or double-quote), so an
+    unescaped '"' reliably toggles in/out of string-literal content.
+    """
+    depth = 0
+    in_string = False
+    i = open_index
+    while i < len(text):
+        ch = text[i]
+        if in_string:
+            if ch == "\\":
+                # Skip the escaped character (e.g. \" or \\) without
+                # inspecting it, matching escape_literal's escaping scheme.
+                i += 2
+                continue
+            if ch == '"':
+                in_string = False
+        else:
+            if ch == '"':
+                in_string = True
+            elif ch == "{":
+                depth += 1
+            elif ch == "}":
+                depth -= 1
+                if depth == 0:
+                    return i
+        i += 1
+    raise ValidationError("Unbalanced braces in construct_query: no matching '}' found.")
+
+
+def _split_construct_query(query_body: str) -> Tuple[str, str, str]:
+    """
+    Split a fully-substituted CONSTRUCT query into (preamble, construct_clause,
+    where_body), so the WHERE body can be re-wrapped in a GRAPH clause for
+    target_graph support.
+
+    Args:
+        query_body: Fully-substituted query string (all {{name}} tokens
+            already replaced).
+
+    Returns:
+        preamble: Everything before the CONSTRUCT keyword (e.g. PREFIX
+            declarations), stripped of leading/trailing whitespace. Preserved
+            verbatim so PREFIX declarations are not silently dropped when
+            target_graph wrapping rewrites the CONSTRUCT/WHERE structure.
+        construct_clause: The raw "{ ... }" template graph pattern immediately
+            following CONSTRUCT, braces included.
+        where_body: The raw text *inside* the "{ ... }" following WHERE,
+            braces excluded.
+
+    Raises:
+        ValidationError: if the CONSTRUCT/WHERE structure cannot be located.
+    """
+    construct_match = _CONSTRUCT_KEYWORD_RE.search(query_body)
+    if not construct_match:
+        raise ValidationError("construct_query does not contain the CONSTRUCT keyword.")
+    preamble = query_body[: construct_match.start()].strip()
+
+    construct_open = query_body.find("{", construct_match.end())
+    if construct_open == -1:
+        raise ValidationError("construct_query is missing '{' after the CONSTRUCT keyword.")
+    construct_close = _find_matching_brace(query_body, construct_open)
+    construct_clause = query_body[construct_open : construct_close + 1]
+
+    where_match = _WHERE_KEYWORD_RE.search(query_body, construct_close + 1)
+    if not where_match:
+        raise ValidationError("construct_query is missing a WHERE clause.")
+    where_open = query_body.find("{", where_match.end())
+    if where_open == -1:
+        raise ValidationError("construct_query is missing '{' after the WHERE keyword.")
+    where_close = _find_matching_brace(query_body, where_open)
+    where_body = query_body[where_open + 1 : where_close]
+
+    return preamble, construct_clause, where_body
+
+
+def render_construct_template(
+    template: ConstructTemplate,
+    params: Dict[str, Any],
+    target_graph: Optional[str] = None,
+) -> str:
+    """
+    Render a ConstructTemplate's construct_query into a safe, executable SPARQL
+    CONSTRUCT query string.
+
+    Args:
+        template: The ConstructTemplate to render.
+        params: Values for each {{name}} placeholder, keyed by ParameterDescriptor.name.
+            Values for missing optional parameters fall back to ParameterDescriptor.default.
+        target_graph: Named graph IRI to wrap the CONSTRUCT query in (via a GRAPH
+            clause around the WHERE body). If None, falls back to
+            template.target_graph. If both are None, no graph wrapping is applied.
+
+    Returns:
+        Fully-substituted SPARQL CONSTRUCT query string, safe to pass directly to
+        BlazegraphStore.execute_sparql.
+
+    Raises:
+        ValidationError: on any of:
+            - a required ParameterDescriptor has no value in params and no default
+            - a "uri"-typed parameter value fails validate_uri
+            - a "typed-literal"-typed parameter is missing template-declared datatype
+            - a "typed-literal"-typed parameter value cannot be coerced to that datatype
+            - target_graph (explicit arg or template.target_graph) fails validate_uri
+            - construct_query references a {{placeholder}} with no matching
+              ParameterDescriptor (detected as an unresolved placeholder after
+              substitution)
+    """
+    # Step 1: Resolve effective parameter values (params override, then default).
+    resolved: Dict[str, Any] = {}
+    for descriptor in template.parameters:
+        if descriptor.name in params:
+            resolved[descriptor.name] = params[descriptor.name]
+        elif not descriptor.required:
+            resolved[descriptor.name] = descriptor.default
+        else:
+            raise ValidationError(
+                f"Missing required parameter: {descriptor.name!r} "
+                f"(template {template.name!r})"
+            )
+
+    # Reject any keys in params that don't correspond to a declared
+    # ParameterDescriptor — silently ignoring unknown parameters would let
+    # a caller's typo (e.g. "subjcet" instead of "subject") go unnoticed
+    # while the mistyped value is simply dropped.
+    declared_names = {descriptor.name for descriptor in template.parameters}
+    unexpected_keys = sorted(set(params) - declared_names)
+    if unexpected_keys:
+        raise ValidationError(
+            f"Unexpected parameter(s) for template {template.name!r}: "
+            f"{unexpected_keys}. Declared parameters: {sorted(declared_names)}."
+        )
+
+    # Step 2: Render each parameter value according to its declared type.
+    rendered_values: Dict[str, str] = {}
+    for descriptor in template.parameters:
+        value = resolved[descriptor.name]
+
+        if descriptor.type == "uri":
+            safe_uri = validate_uri(value)
+            rendered_values[descriptor.name] = f"<{safe_uri}>"
+
+        elif descriptor.type == "literal":
+            escaped = escape_literal(value)
+            if descriptor.language is not None:
+                rendered_values[descriptor.name] = f'"{escaped}"@{descriptor.language}'
+            else:
+                rendered_values[descriptor.name] = f'"{escaped}"'
+
+        elif descriptor.type == "typed-literal":
+            if descriptor.datatype is None:
+                raise ValidationError(
+                    f"typed-literal parameter requires datatype: {descriptor.name!r} "
+                    f"(template {template.name!r})"
+                )
+            try:
+                datatype_iri = resolve_datatype_iri(descriptor.datatype)
+            except ValueError as exc:
+                raise ValidationError(
+                    f"Parameter {descriptor.name!r}: {exc}"
+                ) from exc
+
+            local_name = _local_name_of_datatype_iri(datatype_iri)
+            if local_name in _NUMERIC_UNQUOTED_LOCAL_NAMES:
+                rendered_values[descriptor.name] = _render_numeric_literal(
+                    value, local_name, descriptor.name
+                )
+            else:
+                escaped = escape_literal(value)
+                rendered_values[descriptor.name] = f'"{escaped}"^^{datatype_iri}'
+
+        else:
+            raise ValidationError(
+                f"Parameter {descriptor.name!r}: unknown parameter type {descriptor.type!r} "
+                f"(template {template.name!r})"
+            )
+
+    # Step 3: Substitute {{name}} tokens in construct_query.
+    query_body = template.construct_query
+    for name, rendered in rendered_values.items():
+        query_body = query_body.replace("{{" + name + "}}", rendered)
+
+    if _PLACEHOLDER_RE.search(query_body):
+        raise ValidationError(
+            f"Unresolved placeholder(s) in construct_query for template "
+            f"{template.name!r}: query still contains '{{{{' / '}}}}' tokens "
+            f"with no matching ParameterDescriptor."
+        )
+
+    # Step 4: Resolve effective target_graph — SAME validate_uri path as any
+    # "uri" parameter. No raw f-string interpolation is used here.
+    effective_graph = target_graph if target_graph is not None else template.target_graph
+    if effective_graph is not None:
+        safe_graph_uri = validate_uri(effective_graph)
+        wrapped_graph_token = f"<{safe_graph_uri}>"
+
+        preamble, construct_clause, where_body = _split_construct_query(query_body)
+        graph_wrapped_query = (
+            f"CONSTRUCT {construct_clause} "
+            f"WHERE {{ GRAPH {wrapped_graph_token} {{ {where_body} }} }}"
+        )
+        rendered_query = f"{preamble}\n{graph_wrapped_query}" if preamble else graph_wrapped_query
+    else:
+        rendered_query = query_body
+
+    return rendered_query
+
+
+def execute_construct_template(
+    template: ConstructTemplate,
+    params: Dict[str, Any],
+    store_backend: Any,
+    target_graph: Optional[str] = None,
+    **options: Any,
+) -> List[Triplet]:
+    """
+    Render, execute, parse, and persist a CONSTRUCT template in one call.
+
+    Args:
+        template: ConstructTemplate to execute.
+        params: Parameter values, forwarded to render_construct_template.
+        store_backend: A BlazegraphStore instance (Blazegraph-only; duck-typed
+            via hasattr(store_backend, "execute_sparql"), but this function
+            additionally requires store_backend to expose add_triplets — a
+            plain SPARQL-only backend without write support is rejected with
+            ProcessingError).
+        target_graph: Forwarded to render_construct_template; also used as
+            the `graph` option when persisting results via add_triplets so
+            constructed triples land in the same named graph they were
+            scoped to at query time.
+        **options: Forwarded to both store_backend.execute_sparql and
+            store_backend.add_triplets (e.g. timeout overrides). If options
+            contains "result_format" and/or "graph", those keys are
+            overridden by this function's own required values
+            ("construct" and effective_graph respectively) rather than
+            raising a duplicate-keyword-argument error — both keys are
+            load-bearing internal details of what this function does, so a
+            caller-supplied value for either is silently superseded, not an
+            error condition.
+
+    Returns:
+        The List[Triplet] that were constructed AND successfully persisted
+        via add_triplets. Order matches the order the store backend's
+        execute_sparql yielded triples in its "triples" key.
+
+    Raises:
+        ValidationError: propagated from render_construct_template.
+        ProcessingError: if store_backend lacks execute_sparql/add_triplets,
+            or if persistence via add_triplets does not report success (either
+            via a returned dict with success=False, or a raised ProcessingError
+            from backends such as JenaStore that raise on a complete batch
+            failure rather than returning a dict).
+
+    Exception-propagation convention:
+        This function does not wrap or catch exceptions raised by
+        render_construct_template or store_backend.execute_sparql — each
+        sub-layer is responsible for raising its own correctly-typed
+        exception (ValidationError for rendering failures; ProcessingError
+        for execution failures, as BlazegraphStore.execute_sparql already
+        does internally for connection/request/Turtle-parse errors). Adding
+        a second wrapping layer here would only obscure the original error
+        with no new information. This extends to add_triplets: most backends
+        signal failure via a returned dict (checked immediately after the call
+        below), but some backends (e.g. JenaStore) raise ProcessingError
+        directly on a complete batch failure — that exception is intentionally
+        allowed to propagate uncaught here, as it is already a correctly-typed
+        ProcessingError and carries the right diagnostic information.
+
+    Why store_backend.execute_sparql is called directly instead of
+    QueryEngine.execute_query (investigated for issue #322 item on reusing
+    "Generic SPARQL execution ... QueryEngine.execute_query" — this is a
+    deliberate choice, not an oversight):
+        Routing through QueryEngine.execute_query was evaluated and found to
+        introduce three concrete regressions against this function's already
+        -tested behavior:
+        1. QueryEngine.optimize_query's whitespace-collapse
+           (" ".join(query.split())) corrupts literal content. Verified: a
+           literal parameter value of "value:   three   spaces   " comes
+           back from optimize_query as "value: three spaces " — the
+           collapsing operates on the whole query string with no awareness
+           of quoted-string boundaries, silently altering the literal's
+           actual content. This breaks the escaping guarantees
+           render_construct_template exists to provide (Property 1).
+        2. QueryEngine.execute_query caches results keyed only on
+           normalized query text (enable_caching=True by default). A
+           CONSTRUCT query's correct results depend on the live state of the
+           graph at query time; repeated execute_construct_template calls
+           with identical params (a normal usage pattern — same template
+           re-run periodically) would silently return a stale cached
+           QueryResult instead of re-querying, causing incorrect
+           persistence via add_triplets.
+        3. QueryEngine.execute_query wraps its entire body in a blanket
+           `except Exception: raise ProcessingError(...)`, re-typing every
+           exception regardless of origin. This directly conflicts with the
+           exception-propagation convention documented and tested above
+           (e.g. a raw ConnectionError from store_backend.execute_sparql
+           must propagate as ConnectionError, not get silently re-wrapped
+           into a differently-worded ProcessingError).
+        Fixing this properly would require QueryEngine itself to support a
+        "do not touch this already-rendered, already-safe query" mode
+        (disabling optimize_query and caching for CONSTRUCT) and to stop
+        re-wrapping already-correctly-typed exceptions — changes to a
+        shared, backend-agnostic module used by other query paths, which is
+        out of scope for this Blazegraph-only feature per the issue's own
+        no-scope-creep guidance. Calling store_backend.execute_sparql
+        directly is therefore the correct choice today, not a gap to close
+        casually.
+    """
+    if not (hasattr(store_backend, "execute_sparql") and hasattr(store_backend, "add_triplets")):
+        raise ProcessingError(
+            "store_backend must support both execute_sparql and add_triplets "
+            "to use execute_construct_template."
+        )
+
+    # Step 1: Render (raises ValidationError unchanged on any failure).
+    rendered_query = render_construct_template(template, params, target_graph)
+
+    # Step 2: Execute via Blazegraph's CONSTRUCT-aware path. result_format is
+    # a load-bearing internal detail of this function (CONSTRUCT parsing
+    # requires it); if a caller's own **options happens to contain
+    # "result_format", the explicit value here must win rather than raising
+    # "got multiple values for keyword argument" — so it is popped out of a
+    # local copy of options and re-applied explicitly.
+    execute_options = dict(options)
+    execute_options.pop("result_format", None)
+    # Deliberately calling store_backend.execute_sparql directly, NOT
+    # QueryEngine.execute_query — see "Why store_backend.execute_sparql is
+    # called directly instead of QueryEngine.execute_query" in this
+    # function's docstring before routing through QueryEngine here.
+    query_result = store_backend.execute_sparql(
+        rendered_query, result_format="construct", **execute_options
+    )
+
+    if not query_result.get("success", False):
+        raise ProcessingError(
+            f"CONSTRUCT query execution failed for template {template.name!r}: "
+            f"{query_result}"
+        )
+
+    # Step 3: Convert parsed RDF triples to Triplet objects. store_backend is
+    # a BlazegraphStore instance, whose execute_sparql returns Dict[str, Any]
+    # with a "triples" key for CONSTRUCT queries: a list of
+    # (subject, predicate, object, metadata) 4-tuples (see
+    # BlazegraphStore.execute_sparql). object_metadata carries "datatype"
+    # and/or "language" for literals that have that information — those are
+    # folded into the resulting Triplet's own metadata under the
+    # "datatype"/"lang" keys, which is exactly what
+    # BlazegraphStore._format_object_for_sparql reads
+    # (metadata.get("datatype") / metadata.get("lang")) when re-serializing
+    # a Triplet back to SPARQL, so a typed/lang-tagged literal round-trips
+    # correctly through add_triplets instead of being silently flattened to
+    # an untyped plain string.
+    raw_triples = query_result.get("triples", [])
+    triplets: List[Triplet] = []
+    for s, p, o, object_metadata in raw_triples:
+        triplet_metadata = {"source": "construct_template", "template": template.name}
+        if object_metadata.get("datatype"):
+            triplet_metadata["datatype"] = object_metadata["datatype"]
+        if object_metadata.get("language"):
+            triplet_metadata["lang"] = object_metadata["language"]
+
+        triplets.append(
+            Triplet(
+                subject=str(s),
+                predicate=str(p),
+                object=str(o),
+                # confidence=1.0 (explicit, matching Triplet's own default):
+                # a CONSTRUCT query is a deterministic graph transformation
+                # over already-persisted RDF/SPARQL-computed data, not a
+                # probabilistic extraction (unlike NER/LLM-based Triplet
+                # extraction, where confidence reflects genuine estimation
+                # uncertainty). There is no meaningful uncertainty to encode
+                # here, so full confidence is the correct value, not an
+                # accidental default.
+                confidence=1.0,
+                metadata=triplet_metadata,
+            )
+        )
+
+    # Step 4: Persist via add_triplets (same write path as any other bulk
+    # load). Same reasoning as the result_format pop above: "graph" is the
+    # explicit target_graph/template.target_graph resolution this function
+    # exists to enforce, so a caller-supplied "graph" in **options must not
+    # crash the call or silently bypass that resolution — pop it before
+    # forwarding and let the computed effective_graph win.
+    effective_graph = target_graph if target_graph is not None else template.target_graph
+    add_triplets_options = dict(options)
+    add_triplets_options.pop("graph", None)
+    write_result = store_backend.add_triplets(
+        triplets, graph=effective_graph, **add_triplets_options
+    )
+
+    if not write_result.get("success", False):
+        raise ProcessingError(
+            f"Failed to persist constructed triples for template "
+            f"{template.name!r}: {write_result}"
+        )
+
+    return triplets
+
+
+def construct_template_step_handler(data: Any, **options: Any) -> List[Triplet]:
+    """
+    Pipeline step handler for the "construct_template" step type.
+
+    Resolves `store_backend` and `construct_template_registry` from execution
+    options, looks up the named template, and delegates to
+    execute_construct_template. Mirrors the exact resolution pattern used for
+    `triplet_store`/`version_manager` in
+    ExecutionEngine._execute_step's delta_mode handling: call-time options
+    first, engine config fallback second, ProcessingError if either is
+    missing.
+
+    Called by ExecutionEngine._execute_step as
+    `step.handler(data, **step.config, **options)` — no change to
+    PipelineStep or _execute_step is required, since step.config and
+    **options already flow through generically for any step type.
+
+    Args:
+        data: Pipeline data flowing in (unused by this step type — a
+            construct_template step's output is independent of upstream
+            step data, exactly like other non-delta step types).
+        **options: The merged step.config + execution-time options dict, as
+            passed by ExecutionEngine._execute_step's
+            `step.handler(data, **step.config, **options)` call. Expected
+            keys:
+                - template_name: str — required, key into
+                  construct_template_registry.
+                - params: Dict[str, Any] — required, forwarded to
+                  render_construct_template.
+                - target_graph: Optional[str] — optional.
+                - store_backend: Any — resolved from options directly, or
+                  from an "engine_config" dict passed alongside it (see
+                  below); required.
+                - construct_template_registry: ConstructTemplateRegistry —
+                  resolved the same way; required.
+                - engine_config: Optional[Dict[str, Any]] — fallback source
+                  for store_backend/construct_template_registry when not
+                  present directly in options, mirroring the *shape* of
+                  self.config.get(...) in ExecutionEngine._execute_step's
+                  delta_mode handling. One real difference from delta_mode:
+                  delta_mode's fallback runs inside _execute_step itself (a
+                  bound ExecutionEngine method with direct access to
+                  self.config), whereas construct_template_step_handler is a
+                  plain function with no such access — by design, this step
+                  type requires no pre-handler interception in
+                  _execute_step (see Requirement 7.5), so self.config is
+                  simply never threaded to any step.handler call today. In
+                  practice this means store_backend/construct_template_registry
+                  should normally be passed as call-time options via
+                  ExecutionEngine.execute_pipeline(pipeline, data,
+                  store_backend=..., construct_template_registry=...),
+                  which flow through untouched to this handler. The
+                  engine_config parameter exists for callers who explicitly
+                  forward their own ExecutionEngine(config=...) dict into
+                  execute_pipeline's options (e.g. engine_config=engine.config)
+                  and want the same two-tier resolution shape as delta_mode.
+                - step_name: Optional[str] — used only to name the failing
+                  step in error messages; defaults to "construct_template"
+                  if not supplied (design.md's own pseudocode references
+                  step.name inside this handler, but the actual
+                  step.handler(data, **step.config, **options) invocation
+                  never passes step.name to a standalone handler function —
+                  this default covers that gap without requiring any
+                  ExecutionEngine/_execute_step change).
+
+    Returns:
+        The List[Triplet] returned by execute_construct_template.
+
+    Raises:
+        ProcessingError: if store_backend or construct_template_registry
+            cannot be resolved from options (or its "engine_config" fallback).
+        ValidationError: if template_name is not registered in the resolved
+            construct_template_registry.
+    """
+    step_name = options.get("step_name", "construct_template")
+    engine_config = options.get("engine_config") or {}
+
+    store_backend = options.get("store_backend") or engine_config.get("store_backend")
+    construct_template_registry = options.get(
+        "construct_template_registry"
+    ) or engine_config.get("construct_template_registry")
+
+    if not store_backend or not construct_template_registry:
+        raise ProcessingError(
+            f"Step '{step_name}' requires 'store_backend' and "
+            f"'construct_template_registry' in execution options for "
+            f"construct_template processing."
+        )
+
+    template_name = options.get("template_name")
+    template = construct_template_registry.get(template_name)
+    if template is None:
+        raise ValidationError(f"Unknown construct template: {template_name!r}")
+
+    return execute_construct_template(
+        template=template,
+        params=options.get("params", {}),
+        store_backend=store_backend,
+        target_graph=options.get("target_graph"),
+    )

semantica/triplet_store/jena_store.py

@@ -32,11 +32,12 @@ from ..semantic_extract.triplet_extractor import Triplet
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
+from . import sparql_escaping
 
 # Optional Jena imports
 try:
-    from rdflib import RDF, Graph, Literal, Namespace, URIRef
-    from rdflib.plugins.stores.sparqlstore import SPARQLStore
+    from rdflib import RDF, Dataset, Graph, Literal, Namespace, URIRef
+    from rdflib.plugins.stores.sparqlstore import SPARQLStore, SPARQLUpdateStore
 
     HAS_JENA_RDFLIB = True
 except (ImportError, OSError):
@@ -68,27 +69,104 @@ class JenaStore:
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
-        self.endpoint = config.get("endpoint")
+        self.endpoint = endpoint or config.get("endpoint")
         self.dataset = config.get("dataset", "default")
         self.enable_inference = config.get("enable_inference", False)
 
-        self.graph: Optional[Graph] = None
+        self.graph: Optional[Dataset] = None
         self._initialize_graph()
 
+    def _is_construct_query(self, query: str) -> bool:
+        """Check if query is a CONSTRUCT query."""
+        return bool(sparql_escaping.CONSTRUCT_QUERY_RE.search(query))
+
     def _initialize_graph(self) -> None:
-        """Initialize RDF graph."""
+        """Initialize the backing store as a ``Dataset`` with ``default_union=False``.
+
+        ``Dataset`` is used instead of ``Graph`` to support named graphs.
+        ``default_union=False`` is set explicitly so that SPARQL queries and
+        ``get_triplets()`` calls that do not specify a named graph see **only**
+        the default graph, not a union across all named graphs.  This matches
+        the maintainer-confirmed architecture for this migration.
+
+        For remote Fuseki endpoints, ``SPARQLUpdateStore`` is used instead of
+        the read-only ``SPARQLStore`` so that SPARQL Update (INSERT DATA /
+        DELETE DATA) operations work correctly against the update endpoint;
+        ``SPARQLStore`` has no update endpoint and raises ``TypeError`` from
+        ``.add()``/``.remove()``. The standard Fuseki sub-paths are derived
+        from the base dataset URL:
+
+            query_endpoint  = <endpoint>/query
+            update_endpoint = <endpoint>/update
+
+        These match the default service names shipped with every Apache Jena
+        Fuseki dataset (configurable per-deployment, but correct for the
+        canonical out-of-the-box setup).
+        """
         if HAS_JENA_RDFLIB:
             if self.endpoint:
-                # Use SPARQL store for remote endpoint
+                # Use SPARQLUpdateStore so that .add() issues SPARQL INSERT DATA
+                # requests against the Fuseki update endpoint.  The read-only
+                # SPARQLStore was previously used here, which caused every
+                # add_triplets() call to silently fail with TypeError.
+                # Derive Fuseki sub-paths from the endpoint.  The documented
+                # and tested contract is a bare dataset base URL (e.g.
+                # "http://localhost:3030/ds"), from which "/query" and
+                # "/update" are appended.  As a defensive measure, detect if
+                # the caller already supplied a full service URL ending in a
+                # recognised Fuseki suffix ("/query", "/sparql", "/update")
+                # and avoid double-appending (e.g. "…/ds/query/query").
+                _QUERY_SUFFIXES = ("/query", "/sparql")
+                _UPDATE_SUFFIXES = ("/update",)
+                _ALL_SUFFIXES = _QUERY_SUFFIXES + _UPDATE_SUFFIXES
+
+                stripped = self.endpoint.rstrip("/")
+                _ends_with_suffix = any(
+                    stripped.endswith(sfx) for sfx in _ALL_SUFFIXES
+                )
+
+                if _ends_with_suffix:
+                    # Already a full service URL — strip the known suffix to
+                    # recover the dataset base, then derive both sub-paths
+                    # consistently from that base.
+                    _base = stripped
+                    for sfx in _ALL_SUFFIXES:
+                        if stripped.endswith(sfx):
+                            _base = stripped[: -len(sfx)]
+                            break
+                    self.logger.warning(
+                        "endpoint %r already contains a service suffix; "
+                        "using %r as dataset base to derive query and update "
+                        "sub-paths.  Pass a bare dataset base URL (e.g. "
+                        "'http://host:3030/ds') to suppress this warning.",
+                        self.endpoint,
+                        _base,
+                    )
+                    query_endpoint = f"{_base}/query"
+                    update_endpoint = f"{_base}/update"
+                else:
+                    query_endpoint = f"{stripped}/query"
+                    update_endpoint = f"{stripped}/update"
                 try:
-                    store = SPARQLStore(query_endpoint=self.endpoint)
-                    self.graph = Graph(store=store)
+                    store = SPARQLUpdateStore(
+                        query_endpoint=query_endpoint,
+                        update_endpoint=update_endpoint,
+                        autocommit=True,
+                    )
+                    # SPARQLUpdateStore is used (not the read-only SPARQLStore)
+                    # because only it supports writes: SPARQLStore.add()/.remove()
+                    # raise TypeError since it has no update endpoint to issue
+                    # SPARQL Update requests against. Both stores are
+                    # graph_aware=True, so graph-awareness is not what
+                    # distinguishes them here.
+                    self.graph = Dataset(store=store, default_union=False)
                 except Exception as e:
-                    self.logger.warning(f"Could not initialize SPARQL store: {e}")
-                    self.graph = Graph()
+                    self.logger.warning(f"Could not initialize SPARQL update store: {e}")
+                    self.graph = Dataset(default_union=False)
             else:
-                # Use in-memory graph
-                self.graph = Graph()
+                # In-memory Dataset; default_union=False keeps queries scoped
+                # to the default graph unless a named graph is specified.
+                self.graph = Dataset(default_union=False)
         else:
             self.logger.warning(
                 "rdflib not available. Jena store will use basic operations."
@@ -103,7 +181,12 @@ class JenaStore:
             **options: Model options
 
         Returns:
-            Model information
+            Model information dict with keys:
+                - ``model_id``: dataset name
+                - ``endpoint``: remote endpoint URL if configured, else None
+                - ``triplet_count``: total triples across **all** graphs (default
+                  graph + any named graphs).  As of the Dataset migration this
+                  counts the full dataset, not just the default graph.
         """
         if self.graph is None:
             self._initialize_graph()
@@ -120,7 +203,14 @@ class JenaStore:
 
         Args:
             triplets: List of triplets
-            **options: Additional options
+            **options: Additional options.  Recognised keys:
+
+                ``graph`` (str | None):
+                    Named-graph URI.  When supplied, triples are written to
+                    that named graph inside the Dataset (4-tuple add).  When
+                    omitted or ``None``, triples are written to the **default
+                    graph** (3-tuple add) — preserving the pre-migration
+                    single-Graph semantics exactly.
 
         Returns:
             Operation status
@@ -142,6 +232,17 @@ class JenaStore:
             self.progress_tracker.update_tracking(
                 tracking_id, message="Adding triplets to graph..."
             )
+
+            # Resolve named-graph context once before the per-triplet loop.
+            # Dataset.graph(uri) returns an existing Graph for that URI or
+            # creates a new empty one — safe to call even if the graph already
+            # exists.
+            graph_uri = options.get("graph")
+            context: Optional[Graph] = None
+            if graph_uri is not None:
+                context = self.graph.graph(URIRef(str(graph_uri)))
+
+            malformed_count = 0
             for triplet in triplets:
                 try:
                     subject = URIRef(triplet.subject)
@@ -152,10 +253,39 @@ class JenaStore:
                         else Literal(triplet.object)
                     )
 
-                    self.graph.add((subject, predicate, obj))
+                    if context is not None:
+                        # 4-tuple: routes triple to the named graph context.
+                        # SPARQLUpdateStore translates this to:
+                        #   INSERT DATA { GRAPH <uri> { s p o . } }
+                        self.graph.add((subject, predicate, obj, context))
+                    else:
+                        # 3-tuple: Dataset.add() routes to the default graph.
+                        # Semantically identical to the pre-migration Graph.add().
+                        self.graph.add((subject, predicate, obj))
                     added_count += 1
-                except Exception as e:
-                    self.logger.warning(f"Failed to add triplet: {e}")
+                except (ValueError, AttributeError) as e:
+                    # Skip individual malformed triplets (bad URI / missing
+                    # field) but let store-level errors (e.g. network failure,
+                    # read-only store, authentication error) propagate so they
+                    # are not silently swallowed as per-triplet warnings.
+                    self.logger.warning(f"Skipping malformed triplet: {e}")
+                    malformed_count += 1
+
+            if triplets and added_count == 0:
+                if malformed_count == len(triplets):
+                    # Every failure was caught by the per-triplet handler —
+                    # the root cause is data formatting, not store connectivity.
+                    raise ProcessingError(
+                        f"All {len(triplets)} triplet(s) failed validation — "
+                        "check triplet subject/predicate/object formatting."
+                    )
+                else:
+                    # Zero added but failures were not all per-triplet (store
+                    # itself raised, or other unexpected path).
+                    raise ProcessingError(
+                        f"Failed to add any of the {len(triplets)} triplet(s). "
+                        "Check store connectivity and endpoint configuration."
+                    )
 
             self.progress_tracker.stop_tracking(
                 tracking_id,
@@ -217,7 +347,23 @@ class JenaStore:
             return []
 
     def delete_triplet(self, triplet: Triplet, **options) -> Dict[str, Any]:
-        """Delete triplet."""
+        """Delete triplet from the default graph.
+
+        Note:
+            Named-graph parity (a ``graph=`` option mirroring ``add_triplets``)
+            is a known gap deferred to a future follow-up, per the maintainer's
+            scoping of this migration to ``add_triplets`` only.  This method
+            always removes from the default graph regardless of any ``graph=``
+            value in ``**options``.
+
+            The removal is explicitly scoped to ``self.graph.default_graph``.
+            ``Dataset.remove()`` on a bare 3-tuple (no context) resolves to
+            ``context=None`` internally, which the underlying store treats as
+            a wildcard and matches the triple in *every* graph — silently
+            deleting named-graph copies too. Passing ``default_graph``
+            explicitly as the context avoids that and keeps this method
+            scoped to the default graph only, consistent with this docstring.
+        """
         if self.graph is None:
             raise ProcessingError("Graph not initialized")
 
@@ -230,7 +376,7 @@ class JenaStore:
                 else Literal(triplet.object)
             )
 
-            self.graph.remove((subject, predicate, obj))
+            self.graph.remove((subject, predicate, obj, self.graph.default_graph))
 
             return {"success": True}
         except Exception as e:
@@ -277,8 +423,43 @@ class JenaStore:
             raise ProcessingError("Graph not initialized")
 
         try:
+            result_format = options.get("result_format")
+            if result_format is None:
+                result_format = "construct" if self._is_construct_query(query) else "bindings"
+            elif result_format not in ("construct", "bindings"):
+                raise ValidationError(f"Invalid result_format: {result_format!r}")
+
             results = self.graph.query(query)
 
+            if result_format == "construct":
+                triples = []
+                try:
+                    for s, p, o in results:
+                        obj_metadata: Dict[str, Any] = {}
+                        if isinstance(o, Literal):
+                            if o.datatype is not None:
+                                obj_metadata["datatype"] = str(o.datatype)
+                            if o.language is not None:
+                                obj_metadata["language"] = str(o.language)
+                        triples.append((str(s), str(p), str(o), obj_metadata))
+                except ValueError as e:
+                    raise ValidationError(
+                        "result_format='construct' was explicitly requested, but "
+                        "the query does not appear to be a CONSTRUCT query (results "
+                        "cannot be unpacked into 3-tuples)."
+                    ) from e
+                
+                return {
+                    "success": True,
+                    "bindings": [],
+                    "variables": [],
+                    "triples": triples,
+                    "metadata": {
+                        "query": query,
+                        "result_format": "construct",
+                    },
+                }
+
             bindings = []
             variables = []
 
@@ -304,6 +485,8 @@ class JenaStore:
                 "variables": variables,
                 "metadata": {"query": query},
             }
+        except ValidationError:
+            raise
         except Exception as e:
             self.logger.error(f"SPARQL query failed: {e}")
             raise ProcessingError(f"SPARQL query failed: {e}")
@@ -318,11 +501,39 @@ class JenaStore:
 
         Returns:
             Serialized RDF string
+
+        Note:
+            Single-graph serializers (``"turtle"``, ``"xml"``, ``"n3"``)
+            serialize **only the default graph**.  Triples stored in named
+            graphs are silently omitted.  Use ``format="trig"`` or
+            ``format="nquads"`` to capture all named graphs.  A WARNING is
+            logged whenever named-graph content would be dropped by the
+            chosen format.
         """
         if self.graph is None:
             return ""
 
         try:
+            # Warn when named-graph triples exist and would be silently dropped
+            # by a single-graph serializer.  Multi-graph serializers (trig,
+            # nquads, nt/ntriples, trix, json-ld, hext, patch) include all
+            # named graphs and must NOT trigger the warning.
+            # Set verified empirically against rdflib's plugin registry.
+            _SINGLE_GRAPH_FORMATS = frozenset({
+                "turtle", "ttl", "text/turtle", "longturtle",
+                "xml", "application/rdf+xml", "pretty-xml",
+                "n3", "text/n3",
+            })
+            default_count = len(self.graph.default_graph)
+            total_count = len(self.graph)
+            if total_count > default_count and format in _SINGLE_GRAPH_FORMATS:
+                self.logger.warning(
+                    "serialize(format=%r) serializes only the default graph. "
+                    "%d named-graph triple(s) will be omitted. "
+                    "Use format='trig' or 'nquads' to include all graphs.",
+                    format,
+                    total_count - default_count,
+                )
             return self.graph.serialize(format=format)
         except Exception as e:
             self.logger.error(f"Serialization failed: {e}")

semantica/triplet_store/query_engine.py

@@ -49,6 +49,15 @@ class QueryResult:
     variables: List[str]
     execution_time: float = 0.0
     metadata: Dict[str, Any] = field(default_factory=dict)
+    triples: List[tuple] = field(default_factory=list)
+    """Populated only for CONSTRUCT queries. Each element is a (subject,
+    predicate, object, metadata) 4-tuple, taken directly from the store
+    backend's execute_sparql "triples" key (see BlazegraphStore.execute_sparql
+    CONSTRUCT path). subject/predicate/object are strings; metadata is a dict
+    that is empty ({}) for URIs and plain untyped/unlang-tagged literals, and
+    otherwise carries "datatype" and/or "language" keys for literals that
+    have that information, so it is not silently lost. Empty list for all
+    SELECT/ASK/DESCRIBE queries and for backends without CONSTRUCT support."""
 
 
 @dataclass
@@ -183,6 +192,7 @@ class QueryEngine:
                 bindings=result_data.get("bindings", []),
                 variables=result_data.get("variables", []),
                 execution_time=execution_time,
+                triples=result_data.get("triples", []),
                 metadata={
                     **result_data.get("metadata", {}),
                     "optimized": optimized_query != prepared_query,

semantica/triplet_store/rdf4j_store.py

@@ -26,14 +26,18 @@ Author: Semantica Contributors
 License: MIT
 """
 
+import re
 from typing import Any, Dict, List, Optional
+from urllib.parse import urlparse
 
 import requests
+from rdflib import Graph, Literal
 
 from ..semantic_extract.triplet_extractor import Triplet
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
+from . import sparql_escaping
 
 
 class RDF4JStore:
@@ -102,6 +106,15 @@ class RDF4JStore:
         """Get SPARQL Update endpoint."""
         return f"{self.endpoint}/repositories/{self.repository_id}/statements"
 
+    def _is_construct_query(self, query: str) -> bool:
+        """
+        Detect whether `query` is a SPARQL CONSTRUCT query.
+
+        Delegates to sparql_escaping.CONSTRUCT_QUERY_RE, the single canonical
+        CONSTRUCT-detection regex shared with BlazegraphStore.
+        """
+        return sparql_escaping.CONSTRUCT_QUERY_RE.search(query) is not None
+
     def create_repository(
         self, repository_config: Dict[str, Any], **options
     ) -> Dict[str, Any]:
@@ -176,10 +189,29 @@ class RDF4JStore:
 
         Args:
             query: SPARQL query string
-            **options: Additional options
+            **options: Additional options:
+                - result_format: Optional[Literal["bindings", "construct"]].
+                  If omitted, auto-detected via _is_construct_query(query).
 
         Returns:
-            Query results
+            Query results. For non-CONSTRUCT queries (or when result_format
+            resolves to "bindings"), the existing shape is unchanged:
+                {"success": bool, "bindings": [...], "variables": [...], "metadata": {...}}
+            For CONSTRUCT queries (or result_format="construct"), the shape is:
+                {"success": bool, "bindings": [], "variables": [], "triples": [...],
+                 "metadata": {...}}
+            where "triples" is a list of (subject, predicate, object, metadata)
+            4-tuples parsed from the Turtle response via rdflib. subject and
+            predicate are always plain strings. object is the literal's
+            lexical value or the IRI string. metadata is a dict that is empty
+            ({}) for URIs and plain untyped/unlang-tagged literals, and
+            otherwise contains "datatype" (the datatype IRI as a string) and/
+            or "language" (the RFC 5646 language tag) for literals that carry
+            that information.
+
+        Raises:
+            ProcessingError: if not connected, the HTTP request fails, or (for
+                CONSTRUCT queries) the response body fails to parse as Turtle.
         """
         tracking_id = self.progress_tracker.start_tracking(
             module="triplet_store",
@@ -196,6 +228,80 @@ class RDF4JStore:
 
             sparql_endpoint = self._get_sparql_endpoint()
 
+            result_format = options.get("result_format")
+            if result_format is None:
+                result_format = "construct" if self._is_construct_query(query) else "bindings"
+            elif result_format not in ("construct", "bindings"):
+                raise ValidationError(f"Invalid result_format: {result_format!r}")
+
+            if result_format == "construct":
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Sending CONSTRUCT query to RDF4J endpoint..."
+                )
+                # result_format is a load-bearing internal detail of this
+                # function; pop it from a local copy so a caller-supplied
+                # result_format in **options cannot crash with
+                # "got multiple values for keyword argument".
+                execute_options = dict(options)
+                execute_options.pop("result_format", None)
+
+                response = requests.post(
+                    sparql_endpoint,
+                    data={"query": query},
+                    headers={
+                        "Content-Type": "application/x-www-form-urlencoded",
+                        "Accept": "text/turtle",
+                    },
+                    timeout=self.timeout,
+                    auth=(self.username, self.password)
+                    if self.username and self.password
+                    else None,
+                )
+
+                response.raise_for_status()
+
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Parsing CONSTRUCT response as Turtle..."
+                )
+                graph = Graph()
+                try:
+                    graph.parse(data=response.content, format="turtle")
+                except Exception as parse_error:
+                    raise ProcessingError(
+                        f"Failed to parse CONSTRUCT response as Turtle: {parse_error}"
+                    ) from parse_error
+
+                triples = []
+                for s, p, o in graph:
+                    obj_metadata: Dict[str, Any] = {}
+                    if isinstance(o, Literal):
+                        if o.datatype is not None:
+                            obj_metadata["datatype"] = str(o.datatype)
+                        if o.language is not None:
+                            obj_metadata["language"] = str(o.language)
+                    triples.append((str(s), str(p), str(o), obj_metadata))
+
+                result = {
+                    "success": True,
+                    "bindings": [],
+                    "variables": [],
+                    "triples": triples,
+                    "metadata": {
+                        "query": query,
+                        "endpoint": sparql_endpoint,
+                        "result_format": "construct",
+                    },
+                }
+
+                self.progress_tracker.stop_tracking(
+                    tracking_id,
+                    status="completed",
+                    message=f"CONSTRUCT query executed: {len(triples)} triples",
+                )
+                return result
+
+            # Non-CONSTRUCT path — unchanged from prior behavior, including the
+            # explicit Accept: application/sparql-results+json header.
             self.progress_tracker.update_tracking(
                 tracking_id, message="Sending query to RDF4J endpoint..."
             )
@@ -229,6 +335,11 @@ class RDF4JStore:
                 message=f"Query executed: {len(result['bindings'])} results",
             )
             return result
+        except ValidationError:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message="Validation error"
+            )
+            raise
         except Exception as e:
             self.logger.error(f"SPARQL query failed: {e}")
             self.progress_tracker.stop_tracking(
@@ -242,7 +353,15 @@ class RDF4JStore:
 
         Args:
             triplets: List of triplets
-            **options: Additional options
+            **options: Additional options:
+                - graph: Optional[str] — named graph URI. When provided, the
+                  request appends a ``context`` query parameter to the
+                  /statements endpoint, formatted as an N-Triples-encoded IRI
+                  (angle-bracket-wrapped, e.g. ``<http://example.org/g>``).
+                  The ``requests`` library URL-encodes this automatically, so
+                  the wire value is ``?context=%3Chttp%3A%2F%2F...%3E``.
+                  When graph is None or omitted, no context parameter is sent
+                  and writes go to the default/all graphs as before.
 
         Returns:
             Operation status
@@ -268,6 +387,21 @@ class RDF4JStore:
             )
             rdf_data = self._triplets_to_ntriples(triplets)
 
+            # Build the context (named graph) query parameter if requested.
+            # RDF4J REST API requires the value to be N-Triples-encoded:
+            # the IRI must be wrapped in angle brackets, e.g. <http://...>.
+            # requests.post with params= URL-encodes the value automatically,
+            # so the wire value becomes ?context=%3Chttp%3A...%3E.
+            # When graph is None, send no context parameter at all — do NOT
+            # default to context=null, which would change "all graphs" semantics
+            # to "default graph only" and is a behavior change from today.
+            graph = options.get("graph")
+            if graph is not None:
+                sparql_escaping.validate_uri(graph)
+                context_params = {"context": f"<{graph}>"}
+            else:
+                context_params = None
+
             self.progress_tracker.update_tracking(
                 tracking_id, message="Sending triplets to RDF4J repository..."
             )
@@ -275,6 +409,7 @@ class RDF4JStore:
                 update_endpoint,
                 data=rdf_data,
                 headers={"Content-Type": "application/n-triples"},
+                params=context_params,
                 timeout=self.timeout * 2,
                 auth=(self.username, self.password)
                 if self.username and self.password
@@ -290,6 +425,11 @@ class RDF4JStore:
             )
 
             return {"success": True, "triplets_added": len(triplets)}
+        except ValidationError:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message="Validation error"
+            )
+            raise
         except Exception as e:
             self.logger.error(f"Add triplets failed: {e}")
             self.progress_tracker.stop_tracking(
@@ -365,9 +505,54 @@ class RDF4JStore:
             self.logger.error(f"Delete triplet failed: {e}")
             raise ProcessingError(f"Delete triplet failed: {e}")
 
+    def _is_uri_value(self, value: str) -> bool:
+        """
+        Detect if a value should be serialized as an IRI.
+
+        Byte-for-byte copy of BlazegraphStore._is_uri_value, so both backends
+        agree on which triplet objects are IRIs vs. literals.
+        """
+        if not isinstance(value, str) or not value:
+            return False
+        if value.startswith("<") and value.endswith(">"):
+            return True
+        parsed = urlparse(value)
+        if parsed.scheme not in {"http", "https", "urn"}:
+            return False
+        # Reject strings that only look like URIs (e.g. "http not a uri")
+        return not re.search(r"\s", value)
+
+    def _format_object_for_ntriples(self, triplet: Triplet) -> str:
+        """Format triplet object as IRI or literal based on metadata."""
+        obj = triplet.object
+        metadata = triplet.metadata or {}
+
+        if self._is_uri_value(obj):
+            if obj.startswith("<") and obj.endswith(">"):
+                inner = obj[1:-1]
+                if " " in inner or ">" in inner:
+                    raise ValueError(f"IRI contains invalid characters: {obj!r}")
+                return obj
+            return f"<{obj}>"
+
+        escaped = sparql_escaping.escape_literal(obj)
+        datatype = metadata.get("datatype") or metadata.get("literal_datatype")
+        language = metadata.get("lang") or metadata.get("language")
+
+        if datatype:
+            datatype_iri = sparql_escaping.resolve_datatype_iri(datatype)
+            return f'"{escaped}"^^{datatype_iri}'
+        if language:
+            if not sparql_escaping.LANG_TAG_RE.match(str(language)):
+                raise ValueError(f"Invalid language tag {language!r}: must match RFC 5646")
+            return f'"{escaped}"@{language}'
+
+        return f'"{escaped}"'
+
     def _triplets_to_ntriples(self, triplets: List[Triplet]) -> str:
         """Convert triplets to N-Triples format."""
         lines = []
         for triplet in triplets:
-            lines.append(f"<{triplet.subject}> <{triplet.predicate}> <{triplet.object}> .")
+            obj_str = self._format_object_for_ntriples(triplet)
+            lines.append(f"<{triplet.subject}> <{triplet.predicate}> {obj_str} .")
         return "\n".join(lines)

semantica/triplet_store/sparql_escaping.py

@@ -0,0 +1,184 @@
+"""
+Shared SPARQL literal-escaping and URI-validation primitives.
+
+This module centralizes string-escaping, datatype-IRI resolution, and URI
+allowlist validation logic that was previously duplicated (or would have
+been duplicated) between ``BlazegraphStore`` and the CONSTRUCT template
+renderer. ``BlazegraphStore._escape_literal`` and
+``BlazegraphStore._resolve_datatype_iri`` delegate to ``escape_literal`` and
+``resolve_datatype_iri`` here without any change in behavior; ``validate_uri``
+is new and is used by ``render_construct_template`` for both ``"uri"``-typed
+parameters and ``target_graph`` values.
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import re
+from typing import FrozenSet
+from urllib.parse import urlparse
+
+from ..utils.exceptions import ValidationError
+
+# Allowed URI schemes for validate_uri's allowlist (Requirement 3.2/4.4).
+_ALLOWED_URI_SCHEMES: FrozenSet[str] = frozenset({"http", "https", "urn"})
+
+# Known prefix expansions for XSD and common RDF vocabularies.
+# Identical table to BlazegraphStore._KNOWN_PREFIXES.
+KNOWN_PREFIXES: dict = {
+    "xsd": "http://www.w3.org/2001/XMLSchema#",
+    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
+    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
+    "owl": "http://www.w3.org/2002/07/owl#",
+    "skos": "http://www.w3.org/2004/02/skos/core#",
+}
+
+# RFC 5646 language tag: primary subtag optionally followed by '-' + subtags.
+# Identical pattern to BlazegraphStore._LANG_TAG_RE.
+LANG_TAG_RE = re.compile(r"^[a-zA-Z]{1,8}(-[a-zA-Z0-9]{1,8})*$")
+
+# Disallowed-character check shared by validate_uri and resolve_datatype_iri,
+# identical to the character class used throughout BlazegraphStore.
+_DISALLOWED_URI_CHARS_RE = re.compile(r"[\s<>\"{}|\\^`]")
+
+# Matches CONSTRUCT only as the actual SPARQL query-form keyword: anchored
+# from the start of the string, optionally preceded by PREFIX/BASE
+# declarations, then requires CONSTRUCT as the first non-whitespace keyword.
+# This prevents false-positives from SELECT/ASK queries that merely contain
+# the word "CONSTRUCT" inside a string literal or comment (e.g. a literal
+# value of '"please CONSTRUCT this"' or a comment line).
+#
+# Shared by BlazegraphStore and RDF4JStore so the detection logic has one
+# canonical implementation rather than being duplicated per-backend.
+CONSTRUCT_QUERY_RE = re.compile(
+    r"""
+    \A                       # anchor to start of string
+    (?:                      # skip zero or more of:
+        \s+                  # whitespace
+      | \#[^\n]*             # comments (until newline)
+      | PREFIX\s+[\w\-]*:\s*<[^>]*>  # PREFIX declaration
+      | BASE\s+<[^>]*>       # BASE declaration
+    )*
+    \s*                      # any remaining whitespace before the query form
+    CONSTRUCT                # the actual query-form keyword
+    \b                       # must be followed by a non-word character
+    """,
+    re.IGNORECASE | re.VERBOSE,
+)
+
+
+def escape_literal(value: str) -> str:
+    """
+    Escape a string literal for safe inclusion inside SPARQL double quotes.
+
+    This is a byte-for-byte copy of BlazegraphStore._escape_literal's
+    transformation:
+        \\ -> \\\\ , " -> \\" , \\n -> \\n , \\r -> \\r , \\t -> \\t
+
+    Args:
+        value: Raw literal value (converted via str() first, matching the
+            original method's behavior of accepting non-str inputs).
+
+    Returns:
+        Escaped string safe to place inside a SPARQL/Turtle double-quoted
+        literal.
+    """
+    return (
+        str(value)
+        .replace("\\", "\\\\")
+        .replace("\"", "\\\"")
+        .replace("\n", "\\n")
+        .replace("\r", "\\r")
+        .replace("\t", "\\t")
+    )
+
+
+def resolve_datatype_iri(datatype: str) -> str:
+    """
+    Expand a datatype string to a validated SPARQL IRI token.
+
+    This is a byte-for-byte copy of BlazegraphStore._resolve_datatype_iri's
+    logic and exception behavior (raises ValueError, not ValidationError, to
+    match the original method's existing contract).
+
+    Accepts:
+    - Already-wrapped IRIs: ``<http://...>``
+    - Full IRIs:            ``http://...`` / ``https://...`` / ``urn:...``
+    - Known prefixed names: ``xsd:integer``, ``rdf:langString``, etc.
+
+    Raises:
+        ValueError: for anything else, or for malformed/unsafe IRIs.
+
+    Returns:
+        An angle-bracketed IRI token, e.g. ``<http://www.w3.org/2001/XMLSchema#integer>``.
+    """
+    datatype = str(datatype)
+
+    # Already angle-bracketed — validate the inner IRI contains no whitespace
+    if datatype.startswith("<") and datatype.endswith(">"):
+        inner = datatype[1:-1]
+        if not inner or _DISALLOWED_URI_CHARS_RE.search(inner):
+            raise ValueError(f"Invalid datatype IRI: {datatype!r}")
+        return datatype
+
+    # Full absolute IRI without brackets
+    parsed = urlparse(datatype)
+    if parsed.scheme in {"http", "https", "urn"} and not _DISALLOWED_URI_CHARS_RE.search(datatype):
+        return f"<{datatype}>"
+
+    # Prefixed form — expand known prefixes only
+    if ":" in datatype:
+        prefix, local = datatype.split(":", 1)
+        if prefix in KNOWN_PREFIXES and re.match(r"^[A-Za-z0-9_\-\.]+$", local):
+            return f"<{KNOWN_PREFIXES[prefix]}{local}>"
+
+    raise ValueError(
+        f"Unsupported datatype {datatype!r}: use a full IRI (http/https/urn), "
+        f"an angle-bracketed IRI, or a known prefix (xsd/rdf/rdfs/owl/skos)."
+    )
+
+
+def validate_uri(
+    value: str,
+    *,
+    allowed_schemes: FrozenSet[str] = _ALLOWED_URI_SCHEMES,
+) -> str:
+    """
+    Validate that `value` is a safe absolute IRI using urllib.parse, and
+    return it unchanged (no bracket-wrapping — callers wrap with <...> at
+    render time).
+
+    Uses urllib.parse.urlparse(value):
+      - scheme must be in allowed_schemes (default {"http", "https", "urn"})
+      - the value must not contain whitespace or any of
+        < > " { } | \\ ^ ` characters (same disallowed-character set used by
+        resolve_datatype_iri / BlazegraphStore's existing IRI checks)
+
+    Args:
+        value: Candidate URI/IRI string.
+        allowed_schemes: Set of permitted URI schemes.
+
+    Returns:
+        The validated `value`, unchanged.
+
+    Raises:
+        ValidationError: if value is empty, not a string, has a scheme not in
+            allowed_schemes, or contains a disallowed character.
+    """
+    if not isinstance(value, str) or not value:
+        raise ValidationError(f"Invalid URI: value must be a non-empty string, got {value!r}")
+
+    if _DISALLOWED_URI_CHARS_RE.search(value):
+        raise ValidationError(
+            f"Invalid URI {value!r}: contains disallowed character(s) "
+            f"(whitespace or one of < > \" {{ }} | \\ ^ `)"
+        )
+
+    parsed = urlparse(value)
+    if parsed.scheme not in allowed_schemes:
+        raise ValidationError(
+            f"Invalid URI {value!r}: scheme {parsed.scheme!r} is not allowed "
+            f"(allowed: {sorted(allowed_schemes)})"
+        )
+
+    return value

semantica/utils/constants.py

@@ -80,6 +80,7 @@ SUPPORTED_VECTOR_STORES = [
     "qdrant",
     "milvus",
     "chroma",
+    "sqlite",
 ]
 
 # Supported Graph Databases

semantica/vector_store/__init__.py

@@ -138,12 +138,29 @@ from .hybrid_search import HybridSearch, MetadataFilter, SearchRanker
 from .hybrid_similarity import HybridSimilarityCalculator
 from .decision_embedding_pipeline import DecisionEmbeddingPipeline
 from .decision_vector_methods import (
-    quick_decision, find_precedents, explain, similar_to, batch_decisions,
-    filter_decisions, get_decision_context, search_by_entities, get_decision_statistics,
-    update_similarity_weights, set_global_vector_store, get_global_vector_store,
+    quick_decision,
+    find_precedents,
+    explain,
+    similar_to,
+    batch_decisions,
+    filter_decisions,
+    get_decision_context,
+    search_by_entities,
+    get_decision_statistics,
+    update_similarity_weights,
+    set_global_vector_store,
+    get_global_vector_store,
     # Aliases
-    record, precedents, explain_decision, similar, batch, filter, context,
-    by_entities, stats, weights
+    record,
+    precedents,
+    explain_decision,
+    similar,
+    batch,
+    filter,
+    context,
+    by_entities,
+    stats,
+    weights,
 )
 from .metadata_store import MetadataIndex, MetadataSchema, MetadataStore
 from .methods import (
@@ -164,6 +181,7 @@ from .pgvector_store import PgVectorStore
 from .pinecone_store import PineconeStore, PineconeClient, PineconeIndex, PineconeSearch
 from .qdrant_store import QdrantStore, QdrantClient, QdrantCollection, QdrantSearch
 from .registry import MethodRegistry, method_registry
+from .sqlite_vec_store import SQLiteVecStore
 from .vector_store import VectorIndexer, VectorManager, VectorRetriever, VectorStore
 from .weaviate_store import (
     WeaviateStore,
@@ -205,6 +223,8 @@ __all__ = [
     "PineconeSearch",
     # PgVector
     "PgVectorStore",
+    # SQLite
+    "SQLiteVecStore",
     # Hybrid search
     "HybridSearch",
     "MetadataFilter",

semantica/vector_store/config.py

@@ -120,6 +120,7 @@ class VectorStoreConfig:
             "VECTOR_STORE_QDRANT_URL": "qdrant_url",
             "VECTOR_STORE_MILVUS_HOST": "milvus_host",
             "VECTOR_STORE_MILVUS_PORT": "milvus_port",
+            "VECTOR_STORE_SQLITE_PATH": "sqlite_path",
         }
 
         for env_var, config_key in env_mappings.items():

semantica/vector_store/sqlite_vec_store.py

@@ -0,0 +1,654 @@
+"""
+SQLite Vector Store Module using sqlite-vec
+
+This module provides SQLite integration using the sqlite-vec extension for vector storage and
+similarity search in the Semantica framework, supporting L2 and Cosine distance metrics,
+dynamic JSON metadata filtering, and disk-backed persistence.
+
+Key Features:
+    - Distance metrics (Cosine, L2/Euclidean)
+    - Fully persistent or in-memory SQLite storage
+    - Dynamic metadata filtering using SQLite's JSON extract functions
+    - Thread-safe operations via a shared connection lock, with optional WAL mode
+      (pass use_wal=True) for concurrent readers
+    - Strict validation of vector dimensions and table names
+    - Parity with PgVectorStore interface for seamless drop-in usage
+
+Main Classes:
+    - SQLiteVecStore: Main SQLite vector store using sqlite-vec virtual tables
+
+Example Usage:
+    >>> from semantica.vector_store import SQLiteVecStore
+    >>> store = SQLiteVecStore(
+    ...     db_path="vectors.db",
+    ...     table_name="vectors",
+    ...     dimension=768,
+    ...     distance_metric="cosine"
+    ... )
+    >>> store.add(vectors, metadata, ids)
+    >>> results = store.search(query_vector, top_k=10)
+    >>> store.close()
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import importlib.util
+import json
+import os
+import re
+import sqlite3
+import threading
+import uuid
+from contextlib import contextmanager
+from typing import Any, Dict, List, Optional, Union
+from urllib.request import pathname2url
+
+import numpy as np
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+
+# Optional sqlite-vec import
+# (Loaded lazily inside SQLiteVecStore.__init__ to prevent eager loading)
+sqlite_vec = None
+
+# Availability check only (via find_spec, not an actual import) so callers and
+# tests can detect whether sqlite-vec is installed without paying the load cost.
+SQLITE_VEC_AVAILABLE = importlib.util.find_spec("sqlite_vec") is not None
+
+
+class SQLiteVecStore:
+    """
+    SQLite vector store using the sqlite-vec extension for similarity search.
+
+    - Vector storage with vec0 virtual table
+    - Similarity search with L2 and Cosine metrics
+    - Thread-safe query and insertion execution
+    - Dynamic metadata extraction and filtering
+    """
+
+    SUPPORTED_METRICS = {"cosine", "l2"}
+
+    def __init__(
+        self,
+        db_path: str,
+        table_name: str,
+        dimension: int,
+        distance_metric: str = "cosine",
+        read_only: bool = False,
+        **kwargs,
+    ):
+        """
+        Initialize SQLiteVecStore.
+
+        Args:
+            db_path: Path to SQLite database file, or ':memory:'
+            table_name: Name of the virtual table to store vectors
+            dimension: Vector dimension
+            distance_metric: Distance metric (cosine, l2)
+            read_only: Open database in read-only mode (requires existing database)
+            **kwargs: Additional option parameters
+                use_wal: If True, enable WAL journal mode and NORMAL synchronous
+                    durability for improved write concurrency (default: False)
+
+        Raises:
+            ValidationError: If parameters are invalid
+            ProcessingError: If sqlite-vec or connection load extension fails
+        """
+        self.logger = get_logger("sqlite_vec_store")
+
+        # Validate dependencies and lazy load
+        global sqlite_vec
+        if sqlite_vec is None:
+            try:
+                import sqlite_vec
+            except (ImportError, OSError):
+                raise ProcessingError(
+                    "sqlite-vec Python package is not available. "
+                    "Install with: pip install sqlite-vec"
+                )
+
+        # Validate parameters
+        if distance_metric.lower() not in self.SUPPORTED_METRICS:
+            raise ValidationError(
+                f"Unsupported distance metric: {distance_metric}. "
+                f"Supported: {', '.join(self.SUPPORTED_METRICS)}"
+            )
+
+        if not self._is_safe_identifier(table_name):
+            raise ValidationError(
+                f"Invalid table name: {table_name!r}. "
+                "Table names must start with a letter or underscore and contain "
+                "only alphanumeric characters and underscores."
+            )
+
+        if not isinstance(dimension, int) or dimension <= 0:
+            raise ValidationError(
+                f"Dimension must be a positive integer, got: {dimension}"
+            )
+
+        self.db_path = db_path
+        self.table_name = table_name
+        self.dimension = dimension
+        self.distance_metric = distance_metric.lower()
+        self.read_only = read_only
+        self.config = kwargs
+        self.use_wal = kwargs.get("use_wal", False)
+
+        # Lock to ensure thread safety when sharing a single SQLite connection
+        self._lock = threading.Lock()
+        self._conn = None
+
+        # Connect to database
+        self._init_connection()
+
+        # Ensure table exists (if not read-only)
+        if not self.read_only:
+            self._ensure_table_exists()
+
+        self.logger.info(
+            f"Initialized SQLiteVecStore: db_path={db_path}, table={table_name}, "
+            f"dimension={dimension}, metric={distance_metric}, read_only={read_only}"
+        )
+
+    def _init_connection(self):
+        """Initialize database connection and load sqlite-vec extension."""
+        try:
+            if self.read_only:
+                if self.db_path == ":memory:":
+                    self._conn = sqlite3.connect(":memory:", check_same_thread=False)
+                else:
+                    if not os.path.exists(self.db_path):
+                        raise ProcessingError(
+                            f"Database file does not exist for read-only mode: {self.db_path}"
+                        )
+                    abs_path = os.path.abspath(self.db_path)
+                    url_path = pathname2url(abs_path)
+                    self._conn = sqlite3.connect(
+                        f"file:{url_path}?mode=ro", uri=True, check_same_thread=False
+                    )
+            else:
+                self._conn = sqlite3.connect(self.db_path, check_same_thread=False)
+
+            # Enable and load extension
+            try:
+                self._conn.enable_load_extension(True)
+                sqlite_vec.load(self._conn)
+                self._conn.enable_load_extension(False)
+            except AttributeError as ae:
+                raise ProcessingError(
+                    "SQLite load extension attribute is not available in this Python build. "
+                    "Make sure you are using a Python build compiled with loadable extension support."
+                ) from ae
+            except Exception as e:
+                raise ProcessingError(
+                    f"Failed to load sqlite-vec extension: {e}"
+                ) from e
+
+            # Set WAL journal mode + NORMAL synchronous for performance and
+            # concurrent readers, if configured via use_wal=True
+            if self.use_wal and not self.read_only and self.db_path != ":memory:":
+                self._conn.execute("PRAGMA journal_mode=WAL")
+                self._conn.execute("PRAGMA synchronous=NORMAL")
+
+        except (ValidationError, ProcessingError):
+            if self._conn:
+                self._conn.close()
+            raise
+        except Exception as e:
+            if self._conn:
+                self._conn.close()
+            raise ProcessingError(f"Failed to establish SQLite connection: {e}") from e
+
+    @contextmanager
+    def _get_connection(self):
+        """Get the active connection."""
+        if not self._conn:
+            raise ProcessingError("Database connection is closed.")
+        try:
+            yield self._conn
+        except (ValidationError, ProcessingError):
+            raise
+        except Exception as e:
+            raise ProcessingError("Database operation failed") from e
+
+    def _is_safe_identifier(self, key: str) -> bool:
+        """
+        Validate that a string is safe to use as a SQL identifier.
+
+        Only allows alphanumeric characters and underscores.
+        """
+        if not isinstance(key, str):
+            return False
+        if not key:
+            return False
+        return bool(re.match(r"^[a-zA-Z_][a-zA-Z0-9_]*$", key))
+
+    def _ensure_table_exists(self):
+        """Ensure the vector table exists."""
+        # Define vec0 virtual table
+        create_table_sql = f"""
+            CREATE VIRTUAL TABLE IF NOT EXISTS {self.table_name} USING vec0(
+                id TEXT PRIMARY KEY,
+                embedding float[{self.dimension}] distance_metric={self.distance_metric},
+                +metadata TEXT
+            )
+        """
+        with self._lock, self._get_connection() as conn:
+            try:
+                conn.execute(create_table_sql)
+                conn.commit()
+                self.logger.debug(f"Virtual table {self.table_name} ensured")
+            except Exception as e:
+                conn.rollback()
+                raise ProcessingError("Failed to create vec0 virtual table") from e
+
+    def add(
+        self,
+        vectors: Union[List[np.ndarray], np.ndarray],
+        metadata: Optional[List[Dict[str, Any]]] = None,
+        ids: Optional[List[str]] = None,
+    ) -> List[str]:
+        """
+        Add vectors to the store.
+
+        Args:
+            vectors: List of vectors or numpy array
+            metadata: List of metadata dictionaries (one per vector)
+            ids: Optional list of IDs (auto-generated if not provided)
+
+        Returns:
+            List of vector IDs
+
+        Raises:
+            ValidationError: If input dimensions or lengths don't match
+            ProcessingError: If read-only mode is active or database operation fails
+        """
+        if self.read_only:
+            raise ProcessingError("Cannot add vectors in read-only mode")
+
+        # Convert to list if numpy array
+        if isinstance(vectors, np.ndarray):
+            vectors = [vectors[i] for i in range(len(vectors))]
+
+        num_vectors = len(vectors)
+
+        # Validate dimensions
+        for i, vec in enumerate(vectors):
+            if len(vec) != self.dimension:
+                raise ValidationError(
+                    f"Vector at index {i} has dimension {len(vec)}, "
+                    f"expected {self.dimension}"
+                )
+
+        # Generate IDs if not provided
+        if ids is None:
+            ids = [str(uuid.uuid4()) for _ in range(num_vectors)]
+        elif len(ids) != num_vectors:
+            raise ValidationError(
+                f"IDs length ({len(ids)}) must match vectors length ({num_vectors})"
+            )
+
+        # Prepare metadata
+        if metadata is None:
+            metadata = [{} for _ in range(num_vectors)]
+        elif len(metadata) != num_vectors:
+            raise ValidationError(
+                f"Metadata length ({len(metadata)}) must match vectors length ({num_vectors})"
+            )
+
+        with self._lock, self._get_connection() as conn:
+            try:
+                cur = conn.cursor()
+                # Run deletes first to emulate INSERT OR REPLACE / UPSERT behavior
+                # Use batched deletes to avoid a massive performance bottleneck
+                if ids:
+                    batch_size = 1000
+                    for i in range(0, len(ids), batch_size):
+                        batch_ids = ids[i : i + batch_size]
+                        placeholders = ",".join(["?"] * len(batch_ids))
+                        cur.execute(
+                            f"DELETE FROM {self.table_name} WHERE id IN ({placeholders})",
+                            batch_ids,
+                        )
+
+                # Build data tuples for insert
+                data_tuples = [
+                    (
+                        vec_id,
+                        sqlite_vec.serialize_float32(vec),
+                        json.dumps(meta),
+                    )
+                    for vec_id, vec, meta in zip(ids, vectors, metadata)
+                ]
+
+                # Bulk insert
+                cur.executemany(
+                    f"INSERT INTO {self.table_name} (id, embedding, metadata) VALUES (?, ?, ?)",
+                    data_tuples,
+                )
+                conn.commit()
+                cur.close()
+                self.logger.info(f"Added {num_vectors} vectors")
+                return ids
+            except (ValidationError, ProcessingError):
+                conn.rollback()
+                raise
+            except Exception as e:
+                conn.rollback()
+                raise ProcessingError("Failed to add vectors to database") from e
+
+    def search(
+        self,
+        query_vector: np.ndarray,
+        top_k: int = 10,
+        filter: Optional[Dict[str, Any]] = None,
+    ) -> List[Dict[str, Any]]:
+        """
+        Search for similar vectors.
+
+        Args:
+            query_vector: Query vector
+            top_k: Number of results to return
+            filter: Optional metadata filter (dict of key-value pairs)
+
+        Returns:
+            List of results with id, score (similarity), and metadata
+
+        Raises:
+            ValidationError: If query vector dimension doesn't match
+            ProcessingError: If database operation fails
+        """
+        if len(query_vector) != self.dimension:
+            raise ValidationError(
+                f"Query vector has dimension {len(query_vector)}, expected {self.dimension}"
+            )
+
+        # Serialize query vector
+        query_serialized = sqlite_vec.serialize_float32(query_vector)
+
+        # Build query
+        filter_conditions = []
+        filter_params = []
+        if filter:
+            for key, value in filter.items():
+                if not self._is_safe_identifier(key):
+                    raise ValidationError(
+                        f"Invalid filter key: {key!r}. "
+                        "Keys must start with a letter or underscore and contain "
+                        "only alphanumeric characters and underscores."
+                    )
+                filter_conditions.append(f"json_extract(metadata, '$.{key}') = ?")
+                if isinstance(value, bool):
+                    filter_params.append(1 if value else 0)
+                else:
+                    filter_params.append(value)
+
+        where_clause = ""
+        if filter_conditions:
+            where_clause = " AND " + " AND ".join(filter_conditions)
+
+        search_sql = f"""
+            SELECT id, distance, metadata
+            FROM {self.table_name}
+            WHERE embedding MATCH ? AND k = ?{where_clause}
+        """
+        params = [query_serialized, top_k] + filter_params
+
+        with self._lock, self._get_connection() as conn:
+            try:
+                cur = conn.cursor()
+                cur.execute(search_sql, params)
+                rows = cur.fetchall()
+                cur.close()
+
+                results = []
+                for row in rows:
+                    vec_id, distance, meta_json = row
+                    # Convert distance to similarity score
+                    similarity = 1.0 / (1.0 + float(distance))
+
+                    results.append(
+                        {
+                            "id": vec_id,
+                            "score": similarity,
+                            "metadata": json.loads(meta_json) if meta_json else {},
+                        }
+                    )
+
+                return results
+            except (ValidationError, ProcessingError):
+                raise
+            except Exception as e:
+                raise ProcessingError("Failed to search vectors") from e
+
+    def delete(self, ids: List[str]) -> bool:
+        """
+        Delete vectors by ID.
+
+        Args:
+            ids: List of vector IDs to delete
+
+        Returns:
+            True if successful
+
+        Raises:
+            ProcessingError: If read-only mode is active or database operation fails
+        """
+        if not ids:
+            return True
+
+        if self.read_only:
+            raise ProcessingError("Cannot delete vectors in read-only mode")
+
+        with self._lock, self._get_connection() as conn:
+            try:
+                cur = conn.cursor()
+                batch_size = 1000
+                for i in range(0, len(ids), batch_size):
+                    batch_ids = ids[i : i + batch_size]
+                    placeholders = ",".join(["?"] * len(batch_ids))
+                    cur.execute(
+                        f"DELETE FROM {self.table_name} WHERE id IN ({placeholders})",
+                        batch_ids,
+                    )
+                conn.commit()
+                cur.close()
+                self.logger.info(f"Deleted vectors: {len(ids)}")
+                return True
+            except Exception as e:
+                conn.rollback()
+                raise ProcessingError("Failed to delete vectors") from e
+
+    def update(
+        self,
+        ids: List[str],
+        vectors: Optional[Union[List[np.ndarray], np.ndarray]] = None,
+        metadata: Optional[List[Dict[str, Any]]] = None,
+    ) -> bool:
+        """
+        Update existing vectors.
+
+        Args:
+            ids: List of vector IDs to update
+            vectors: Optional new vectors
+            metadata: Optional new metadata
+
+        Returns:
+            True if successful
+
+        Raises:
+            ValidationError: If input dimensions or lengths don't match
+            ProcessingError: If read-only mode is active or database operation fails
+        """
+        if not ids:
+            return True
+
+        if self.read_only:
+            raise ProcessingError("Cannot update vectors in read-only mode")
+
+        if vectors is None and metadata is None:
+            raise ValidationError(
+                "Either vectors or metadata must be provided for update"
+            )
+
+        if vectors is not None:
+            if isinstance(vectors, np.ndarray):
+                vectors = [vectors[i] for i in range(len(vectors))]
+            if len(vectors) != len(ids):
+                raise ValidationError("Vectors length must match IDs length")
+            for i, vec in enumerate(vectors):
+                if len(vec) != self.dimension:
+                    raise ValidationError(
+                        f"Vector at index {i} has dimension {len(vec)}, expected {self.dimension}"
+                    )
+
+        if metadata is not None and len(metadata) != len(ids):
+            raise ValidationError("Metadata length must match IDs length")
+
+        with self._lock, self._get_connection() as conn:
+            try:
+                cur = conn.cursor()
+
+                # Same columns are updated for every row, so build one
+                # statement and batch it via executemany for efficiency.
+                if vectors is not None and metadata is not None:
+                    update_sql = f"UPDATE {self.table_name} SET embedding = ?, metadata = ? WHERE id = ?"
+                    data_tuples = [
+                        (sqlite_vec.serialize_float32(vectors[i]), json.dumps(metadata[i]), vec_id)
+                        for i, vec_id in enumerate(ids)
+                    ]
+                elif vectors is not None:
+                    update_sql = f"UPDATE {self.table_name} SET embedding = ? WHERE id = ?"
+                    data_tuples = [
+                        (sqlite_vec.serialize_float32(vectors[i]), vec_id)
+                        for i, vec_id in enumerate(ids)
+                    ]
+                else:
+                    update_sql = f"UPDATE {self.table_name} SET metadata = ? WHERE id = ?"
+                    data_tuples = [
+                        (json.dumps(metadata[i]), vec_id) for i, vec_id in enumerate(ids)
+                    ]
+
+                cur.executemany(update_sql, data_tuples)
+                conn.commit()
+                cur.close()
+                self.logger.info(f"Updated {len(ids)} vectors")
+                return True
+            except ValidationError:
+                conn.rollback()
+                raise
+            except Exception as e:
+                conn.rollback()
+                raise ProcessingError("Failed to update vectors") from e
+
+    def get(self, ids: List[str]) -> List[Dict[str, Any]]:
+        """
+        Get vectors by ID.
+
+        Args:
+            ids: List of vector IDs
+
+        Returns:
+            List of dictionaries with id, vector, and metadata
+
+        Raises:
+            ProcessingError: If database operation fails
+        """
+        if not ids:
+            return []
+
+        with self._lock, self._get_connection() as conn:
+            try:
+                cur = conn.cursor()
+                results = []
+                batch_size = 1000
+                for i in range(0, len(ids), batch_size):
+                    batch_ids = ids[i : i + batch_size]
+                    placeholders = ",".join(["?"] * len(batch_ids))
+                    cur.execute(
+                        f"SELECT id, embedding, metadata FROM {self.table_name} "
+                        f"WHERE id IN ({placeholders})",
+                        batch_ids,
+                    )
+                    for row in cur.fetchall():
+                        vid, embedding_blob, metadata_json = row
+                        vec = (
+                            np.frombuffer(embedding_blob, dtype=np.float32).copy()
+                            if embedding_blob
+                            else None
+                        )
+                        meta = json.loads(metadata_json) if metadata_json else {}
+                        results.append(
+                            {
+                                "id": vid,
+                                "vector": vec,
+                                "metadata": meta,
+                            }
+                        )
+                cur.close()
+                return results
+            except Exception as e:
+                raise ProcessingError("Failed to get vectors") from e
+
+    def create_index(
+        self,
+        index_type: str = "hnsw",
+        params: Optional[Dict[str, Any]] = None,
+    ) -> bool:
+        """
+        Create an index on the vector column.
+        For SQLiteVecStore, vec0 virtual tables automatically index vectors,
+        so this is a no-op that always returns True for interface parity.
+        """
+        self.logger.debug("create_index is a no-op for SQLiteVecStore")
+        return True
+
+    def get_stats(self) -> Dict[str, Any]:
+        """
+        Get store statistics.
+
+        Returns:
+            Dictionary with vector_count, dimension, and distance_metric
+        """
+        with self._lock, self._get_connection() as conn:
+            try:
+                cur = conn.cursor()
+                cur.execute(f"SELECT COUNT(*) FROM {self.table_name}")
+                count = cur.fetchone()[0]
+                cur.close()
+                return {
+                    "vector_count": count,
+                    "dimension": self.dimension,
+                    "distance_metric": self.distance_metric,
+                }
+            except Exception as e:
+                raise ProcessingError("Failed to get store statistics") from e
+
+    def close(self):
+        """Close the database connection."""
+        if hasattr(self, "_lock") and self._lock:
+            with self._lock:
+                if hasattr(self, "_conn") and self._conn:
+                    try:
+                        self._conn.close()
+                    except Exception:
+                        pass
+                    self._conn = None
+        else:
+            if hasattr(self, "_conn") and self._conn:
+                try:
+                    self._conn.close()
+                except Exception:
+                    pass
+                self._conn = None
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.close()
+
+    def __del__(self):
+        self.close()

semantica/vector_store/vector_store.py

@@ -90,7 +90,7 @@ class VectorStore:
     • Provides vector store operations
     """
 
-    SUPPORTED_BACKENDS = {"faiss", "weaviate", "qdrant", "milvus", "pinecone", "pgvector", "inmemory"}
+    SUPPORTED_BACKENDS = {"faiss", "weaviate", "qdrant", "milvus", "pinecone", "pgvector", "inmemory", "sqlite"}
 
     def __init__(self, backend="faiss", config=None, max_workers: int = 6, **kwargs):
         """Initialize vector store."""
@@ -210,6 +210,31 @@ class VectorStore:
                 )
                 self.logger.info(f"Initialized MilvusStore backend")
                 
+            elif self.backend == "sqlite":
+                from .sqlite_vec_store import SQLiteVecStore
+                db_path = self.config.get("db_path") or self.config.get("sqlite_path")
+                if not db_path:
+                    raise ValueError(
+                        "sqlite backend requires 'db_path' in config. "
+                        "Example: VectorStore(backend='sqlite', config={'db_path': 'vectors.db'})"
+                    )
+                
+                table_name = self.config.get("table_name", "vectors")
+                dimension = self.config.get("dimension", 768)
+                distance_metric = self.config.get("distance_metric", "cosine")
+                read_only = self.config.get("read_only", False)
+                
+                self._backend_store = SQLiteVecStore(
+                    db_path=db_path,
+                    table_name=table_name,
+                    dimension=dimension,
+                    distance_metric=distance_metric,
+                    read_only=read_only,
+                    **{k: v for k, v in self.config.items() 
+                       if k not in ['db_path', 'sqlite_path', 'table_name', 'dimension', 'distance_metric', 'read_only']}
+                )
+                self.logger.info(f"Initialized SQLite backend")
+                
             else:
                 # Fallback to in-memory for unknown backends
                 self.logger.warning(f"Backend '{self.backend}' not implemented, using in-memory")
@@ -218,7 +243,7 @@ class VectorStore:
         except ImportError as e:
             raise ImportError(f"Backend '{self.backend}' not available: {e}. Please install required dependencies.") from e
         except Exception as e:
-            if "requires" in str(e) and "connection_string" in str(e):
+            if "requires" in str(e) and ("connection_string" in str(e) or "db_path" in str(e)):
                 # Re-raise validation errors for missing required parameters
                 raise
             else:
@@ -667,13 +692,27 @@ class VectorStore:
             raise
 
     def update_vectors(
-        self, vector_ids: List[str], new_vectors: List[np.ndarray], **options
+        self, vector_ids: List[str], new_vectors: List[np.ndarray], metadata: Optional[List[Dict[str, Any]]] = None, **options
     ) -> bool:
         """Update existing vectors."""
+        # Delegate to backend store if available
+        if self._backend_store:
+            if hasattr(self._backend_store, 'update'):
+                return self._backend_store.update(ids=vector_ids, vectors=new_vectors, metadata=metadata, **options)
+            elif hasattr(self._backend_store, 'update_vectors'):
+                return self._backend_store.update_vectors(vector_ids, new_vectors, **options)
+            else:
+                raise NotImplementedError(f"Backend store {type(self._backend_store).__name__} does not have update or update_vectors method")
+
         for vec_id, new_vec in zip(vector_ids, new_vectors):
             if vec_id in self.vectors:
                 self.vectors[vec_id] = new_vec
 
+        if metadata:
+            for vec_id, meta in zip(vector_ids, metadata):
+                if vec_id in self.metadata:
+                    self.metadata[vec_id] = meta
+
         # Rebuild index
         self.indexer.create_index(
             list(self.vectors.values()), list(self.vectors.keys())
@@ -683,6 +722,15 @@ class VectorStore:
 
     def delete_vectors(self, vector_ids: List[str], **options) -> bool:
         """Delete vectors from store."""
+        # Delegate to backend store if available
+        if self._backend_store:
+            if hasattr(self._backend_store, 'delete'):
+                return self._backend_store.delete(ids=vector_ids, **options)
+            elif hasattr(self._backend_store, 'delete_vectors'):
+                return self._backend_store.delete_vectors(vector_ids, **options)
+            else:
+                raise NotImplementedError(f"Backend store {type(self._backend_store).__name__} does not have delete or delete_vectors method")
+
         for vec_id in vector_ids:
             self.vectors.pop(vec_id, None)
             self.metadata.pop(vec_id, None)

semantica/vector_store/vector_store_usage.md

@@ -1752,6 +1752,66 @@ method_registry.register("store", "normalized", custom_store_vectors)
 vector_ids = store_vectors(vectors, metadata=metadata, method="normalized")
 ```
 
+### SQLite Vector Store (sqlite-vec)
+
+The SQLite vector store backend uses the `sqlite-vec` extension to provide a lightweight, embedded, yet fully persistent vector store. It is ideal for local development, small-to-medium datasets, and embedded applications where setting up a separate PostgreSQL/pgvector instance is not desired.
+
+#### Installation
+
+Install Semantica with SQLite vector store dependencies:
+
+```bash
+pip install semantica[vectorstore-sqlite]
+```
+
+#### Usage Example
+
+```python
+from semantica.vector_store import VectorStore
+import numpy as np
+
+# Initialize the vector store using the 'sqlite' backend
+# The 'db_path' parameter points to the SQLite database file on disk.
+# Use ':memory:' for a transient, in-memory store.
+store = VectorStore(
+    backend="sqlite",
+    config={
+        "db_path": "my_vector_database.db",
+        "table_name": "documents",
+        "dimension": 128,
+        "distance_metric": "cosine", # Supported metrics: 'cosine', 'l2'
+        "use_wal": True # Optional: enable WAL journal mode + NORMAL synchronous
+                         # for better write concurrency (default: False)
+    }
+)
+
+# Store vectors with metadata
+vectors = [np.random.rand(128).astype(np.float32) for _ in range(5)]
+metadata = [
+    {"category": "ai", "public": True},
+    {"category": "finance", "public": False},
+    {"category": "ai", "public": False},
+    {"category": "healthcare", "public": True},
+    {"category": "finance", "public": True}
+]
+ids = store.store_vectors(vectors, metadata=metadata)
+
+# Search vectors with metadata filtering
+query = np.random.rand(128).astype(np.float32)
+results = store.search_vectors(
+    query,
+    k=2,
+    filter={"category": "ai"}
+)
+
+for r in results:
+    print(f"ID: {r['id']}, Score: {r['score']}, Metadata: {r['metadata']}")
+
+# Close the database connection when done
+# (Recommended on Windows to release file handles)
+store._backend_store.close()
+```
+
 ## Best Practices
 
 1. **Vector Storage**:

tests/pipeline/test_pipeline_validator_construct.py

@@ -0,0 +1,374 @@
+"""
+Tests for issue #3: construct_template_registry support in PipelineValidator.
+
+Covers four new behaviours and one explicit regression guard:
+  1. No registry provided → WARNING-level issue, not an error (result still valid).
+  2. Registry provided, template_name absent → ERROR.
+  3. Registry provided, required param absent from step.config["params"] → ERROR.
+  4. Registry provided, everything valid → no errors/warnings for that step.
+  5. Regression: pipelines containing only non-construct_template steps see
+     identical validation output whether or not construct_template_registry is passed.
+"""
+
+import unittest
+from unittest.mock import MagicMock, patch
+
+from semantica.pipeline.pipeline_builder import Pipeline, PipelineStep
+from semantica.pipeline.pipeline_validator import PipelineValidator
+from semantica.triplet_store.construct_templates import (
+    ConstructTemplate,
+    ConstructTemplateRegistry,
+    ParameterDescriptor,
+)
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+_MINIMAL_QUERY = "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
+
+
+def _make_registry(*templates):
+    """Return a ConstructTemplateRegistry pre-loaded with the given templates."""
+    reg = ConstructTemplateRegistry()
+    for t in templates:
+        reg.register(t)
+    return reg
+
+
+def _make_template(name, *param_descriptors):
+    return ConstructTemplate(
+        name=name,
+        description="test template",
+        construct_query=_MINIMAL_QUERY,
+        parameters=list(param_descriptors),
+    )
+
+
+def _make_pipeline(*steps):
+    """Wrap PipelineStep objects in a minimal Pipeline."""
+    return Pipeline(name="test_pipeline", steps=list(steps))
+
+
+def _make_step(name, step_type, config=None):
+    return PipelineStep(name=name, step_type=step_type, config=config or {})
+
+
+# ---------------------------------------------------------------------------
+# Test class
+# ---------------------------------------------------------------------------
+
+
+class TestPipelineValidatorConstructRegistry(unittest.TestCase):
+
+    def setUp(self):
+        self.tracker_patcher = patch(
+            "semantica.utils.progress_tracker.get_progress_tracker"
+        )
+        mock_get_tracker = self.tracker_patcher.start()
+        mock_tracker = MagicMock()
+        mock_get_tracker.return_value = mock_tracker
+
+    def tearDown(self):
+        self.tracker_patcher.stop()
+
+    # ------------------------------------------------------------------
+    # Test 1: registry not provided -> warning, not error
+    # ------------------------------------------------------------------
+
+    def test_no_registry_yields_warning_not_error(self):
+        """
+        When construct_template_registry is None (the default), a construct_template
+        step should produce exactly one WARNING mentioning the inability to check
+        template existence, and the result should still be valid (no errors).
+        """
+        step = _make_step(
+            "build_foaf",
+            "construct_template",
+            config={"template_name": "person_to_foaf", "params": {}},
+        )
+        pipeline = _make_pipeline(step)
+        validator = PipelineValidator()
+
+        result = validator.validate_pipeline(pipeline)  # no registry kwarg
+
+        self.assertTrue(result.valid, msg=f"Expected valid; errors={result.errors}")
+        self.assertEqual(result.errors, [], msg="Expected no errors")
+        # At least one warning about the missing registry
+        registry_warnings = [
+            w for w in result.warnings if "construct_template_registry" in w
+        ]
+        self.assertGreater(
+            len(registry_warnings),
+            0,
+            msg=f"Expected a warning about missing registry; warnings={result.warnings}",
+        )
+
+    # ------------------------------------------------------------------
+    # Test 2: registry provided, template_name absent -> error
+    # ------------------------------------------------------------------
+
+    def test_missing_template_name_in_registry_yields_error(self):
+        """
+        When a registry is provided but step.config["template_name"] is not
+        registered in it, validate_pipeline must produce an ERROR and result.valid
+        must be False.
+        """
+        # Registry has "other_template", not "person_to_foaf"
+        registry = _make_registry(
+            _make_template("other_template")
+        )
+        step = _make_step(
+            "build_foaf",
+            "construct_template",
+            config={"template_name": "person_to_foaf", "params": {}},
+        )
+        pipeline = _make_pipeline(step)
+        validator = PipelineValidator()
+
+        result = validator.validate_pipeline(
+            pipeline, construct_template_registry=registry
+        )
+
+        self.assertFalse(result.valid, msg="Expected invalid result")
+        template_errors = [e for e in result.errors if "person_to_foaf" in e]
+        self.assertGreater(
+            len(template_errors),
+            0,
+            msg=f"Expected error mentioning 'person_to_foaf'; errors={result.errors}",
+        )
+
+    def test_none_template_name_in_config_yields_error(self):
+        """
+        Edge case: step.config has no 'template_name' key at all.
+        Registry is provided; should still produce an error.
+        """
+        registry = _make_registry(_make_template("some_template"))
+        step = _make_step(
+            "bad_step",
+            "construct_template",
+            config={"params": {}},  # no template_name key
+        )
+        pipeline = _make_pipeline(step)
+        validator = PipelineValidator()
+
+        result = validator.validate_pipeline(
+            pipeline, construct_template_registry=registry
+        )
+
+        self.assertFalse(result.valid)
+        self.assertTrue(any("template_name" in e or "None" in e for e in result.errors),
+                        msg=f"Expected error about missing/None template_name; errors={result.errors}")
+
+    # ------------------------------------------------------------------
+    # Test 3: registry provided, required param absent -> error
+    # ------------------------------------------------------------------
+
+    def test_missing_required_param_yields_error(self):
+        """
+        When the registry is provided, the template is found, but a required
+        parameter is absent from step.config["params"], validate_pipeline must
+        produce an ERROR listing the missing parameter.
+        """
+        template = _make_template(
+            "person_to_foaf",
+            ParameterDescriptor(name="subject", type="uri", required=True),
+            ParameterDescriptor(name="name", type="literal", required=True),
+            ParameterDescriptor(name="lang", type="literal", required=False, default="en"),
+        )
+        registry = _make_registry(template)
+        # Provides "subject" but omits the required "name"
+        step = _make_step(
+            "build_foaf",
+            "construct_template",
+            config={
+                "template_name": "person_to_foaf",
+                "params": {"subject": "http://example.org/alice"},
+            },
+        )
+        pipeline = _make_pipeline(step)
+        validator = PipelineValidator()
+
+        result = validator.validate_pipeline(
+            pipeline, construct_template_registry=registry
+        )
+
+        self.assertFalse(result.valid, msg="Expected invalid result")
+        missing_errors = [e for e in result.errors if "name" in e]
+        self.assertGreater(
+            len(missing_errors),
+            0,
+            msg=f"Expected error about missing 'name' param; errors={result.errors}",
+        )
+
+    def test_required_param_with_default_still_flagged_when_absent(self):
+        """
+        Correctness of the corrected check: required=True AND d.name not in
+        provided_params -> error, even when d.default is not None.
+        This verifies the 'and d.default is None' condition was NOT included,
+        matching render_construct_template's actual runtime behavior.
+        """
+        template = _make_template(
+            "tricky_template",
+            ParameterDescriptor(
+                name="subject",
+                type="literal",
+                required=True,
+                default="fallback_value",  # default exists but required=True
+            ),
+        )
+        registry = _make_registry(template)
+        step = _make_step(
+            "tricky_step",
+            "construct_template",
+            config={
+                "template_name": "tricky_template",
+                "params": {},  # subject not supplied
+            },
+        )
+        pipeline = _make_pipeline(step)
+        validator = PipelineValidator()
+
+        result = validator.validate_pipeline(
+            pipeline, construct_template_registry=registry
+        )
+
+        # Must be an error even though default="fallback_value" exists,
+        # because render_construct_template raises on required=True + no caller value.
+        self.assertFalse(result.valid, msg=(
+            "Expected invalid: required=True param with a default should still be "
+            "flagged if absent from step.config['params']"
+        ))
+        self.assertTrue(
+            any("subject" in e for e in result.errors),
+            msg=f"Expected error mentioning 'subject'; errors={result.errors}",
+        )
+
+    # ------------------------------------------------------------------
+    # Test 4: all valid -> no errors, no construct-related warnings
+    # ------------------------------------------------------------------
+
+    def test_all_valid_yields_no_issues(self):
+        """
+        Registry provided, template found, all required params supplied ->
+        no errors, no construct_template-related warnings.
+        """
+        template = _make_template(
+            "person_to_foaf",
+            ParameterDescriptor(name="subject", type="uri", required=True),
+            ParameterDescriptor(name="name", type="literal", required=True),
+            ParameterDescriptor(name="lang", type="literal", required=False, default="en"),
+        )
+        registry = _make_registry(template)
+        step = _make_step(
+            "build_foaf",
+            "construct_template",
+            config={
+                "template_name": "person_to_foaf",
+                "params": {
+                    "subject": "http://example.org/alice",
+                    "name": "Alice",
+                    # "lang" intentionally omitted -- optional, should not trigger error
+                },
+            },
+        )
+        pipeline = _make_pipeline(step)
+        validator = PipelineValidator()
+
+        result = validator.validate_pipeline(
+            pipeline, construct_template_registry=registry
+        )
+
+        self.assertEqual(result.errors, [], msg=f"Expected no errors; got {result.errors}")
+        # No construct-specific warnings (handler/config warnings from generic check
+        # are fine -- the step has no handler and that's expected in this test fixture)
+        construct_warnings = [
+            w for w in result.warnings if "construct_template_registry" in w
+        ]
+        self.assertEqual(
+            construct_warnings, [],
+            msg=f"Expected no registry-related warnings; got {result.warnings}",
+        )
+
+    # ------------------------------------------------------------------
+    # Test 5: regression -- non-construct_template steps unaffected
+    # ------------------------------------------------------------------
+
+    def test_non_construct_steps_unchanged(self):
+        """
+        Regression guard: validate_pipeline's output must be identical for a
+        pipeline containing only non-construct_template steps, whether or not
+        construct_template_registry is passed.
+
+        Both calls (with and without the registry kwarg) must produce the same
+        valid/errors/warnings, confirming zero behavior change for existing
+        callers of any other step type.
+        """
+        steps = [
+            _make_step("ingest", "file_ingest", config={"path": "/data"}),
+            _make_step("parse", "document_parse", config={"format": "pdf"}),
+            _make_step("embed", "embedding", config={"model": "openai"}),
+        ]
+        pipeline = _make_pipeline(*steps)
+        validator = PipelineValidator()
+
+        # Call without registry (existing behavior)
+        result_without = validator.validate_pipeline(pipeline)
+        # Call with a registry (should not affect these steps at all)
+        dummy_registry = ConstructTemplateRegistry()
+        result_with = validator.validate_pipeline(
+            pipeline, construct_template_registry=dummy_registry
+        )
+
+        self.assertEqual(
+            result_without.valid,
+            result_with.valid,
+            msg="valid flag changed for non-construct_template pipeline",
+        )
+        self.assertEqual(
+            result_without.errors,
+            result_with.errors,
+            msg="errors changed for non-construct_template pipeline",
+        )
+        self.assertEqual(
+            result_without.warnings,
+            result_with.warnings,
+            msg="warnings changed for non-construct_template pipeline",
+        )
+
+    def test_mixed_pipeline_only_construct_step_gets_warning(self):
+        """
+        A pipeline with mixed step types: only the construct_template step should
+        receive the 'no registry' warning; other steps should be unaffected.
+        """
+        steps = [
+            _make_step("ingest", "file_ingest", config={"path": "/data"}),
+            _make_step(
+                "build_graph",
+                "construct_template",
+                config={"template_name": "some_template", "params": {}},
+            ),
+            _make_step("embed", "embedding", config={"model": "openai"}),
+        ]
+        pipeline = _make_pipeline(*steps)
+        validator = PipelineValidator()
+
+        result = validator.validate_pipeline(pipeline)  # no registry
+
+        self.assertTrue(result.valid, msg=f"Expected valid; errors={result.errors}")
+        # Exactly one construct-registry warning (for "build_graph")
+        registry_warnings = [
+            w for w in result.warnings if "construct_template_registry" in w
+        ]
+        self.assertEqual(
+            len(registry_warnings),
+            1,
+            msg=f"Expected exactly 1 registry warning; warnings={result.warnings}",
+        )
+        # The warning should mention the step name
+        self.assertIn("build_graph", registry_warnings[0])
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/provenance/test_all_modules.py

@@ -60,21 +60,25 @@ class TestKGModule:
         assert tracker is not None
         
         # Test basic functionality
+        # NOTE: tracker.get_lineage() was never implemented on
+        # kg.ProvenanceTracker; this tested an intended unified-backend
+        # migration that never happened (#744). ProvenanceTracker is now
+        # deprecated in favor of semantica.provenance.ProvenanceManager.
+        # Instead, verify the observable behavior of the still-supported
+        # track_entity()/get_all_sources() pair.
         tracker.track_entity("test_entity", source="test_source")
-        lineage = tracker.get_lineage("test_entity")
-        
-        assert lineage is not None
-        assert "sources" in lineage
+        sources = tracker.get_all_sources("test_entity")
+        assert len(sources) > 0
+        last_entry = sources[-1]
+        assert last_entry["source"] == "test_source"
+        assert "recorded_at" in last_entry
     
-    def test_kg_uses_unified_backend(self):
-        """Test that kg module uses unified backend."""
-        from semantica.kg import ProvenanceTracker
-        
-        tracker = ProvenanceTracker()
-        
-        # Check if using unified backend
-        assert hasattr(tracker, '_use_unified')
-        assert hasattr(tracker, '_unified_manager')
+    # NOTE: test_kg_uses_unified_backend removed; it only asserted the
+    # presence of _use_unified/_unified_manager attributes, which were
+    # never implemented on kg.ProvenanceTracker. This tested an intended
+    # unified-backend migration that never happened (#744).
+    # ProvenanceTracker is now deprecated in favor of
+    # semantica.provenance.ProvenanceManager.
     
     def test_kg_graph_builder_ready(self):
         """Test GraphBuilder is ready for provenance."""
@@ -405,7 +409,14 @@ class TestCrossModuleIntegration:
     """Test provenance tracking across multiple modules."""
     
     def test_kg_and_split_integration(self):
-        """Test provenance tracking between kg and split modules."""
+        """Test provenance tracking between kg and split modules.
+
+        NOTE: this no longer asserts kg.ProvenanceTracker uses a unified
+        backend (kg_tracker.get_lineage() was never implemented; see #744).
+        It instead verifies, independent of unified-backend behavior, that
+        kg tracking actually produced a record via the still-supported
+        track_entity()/get_all_sources() pair.
+        """
         from semantica.kg import ProvenanceTracker as KGTracker
         from semantica.split import ProvenanceTracker as SplitTracker
         from semantica.split.semantic_chunker import Chunk
@@ -414,17 +425,22 @@ class TestCrossModuleIntegration:
         kg_tracker = KGTracker()
         kg_tracker.track_entity("entity_1", source="doc_1")
         
+        # Verify kg tracking produced a record
+        kg_sources = kg_tracker.get_all_sources("entity_1")
+        assert len(kg_sources) > 0
+        last_kg_entry = kg_sources[-1]
+        assert last_kg_entry["source"] == "doc_1"
+        assert "recorded_at" in last_kg_entry
+        
         # Track with split
         split_tracker = SplitTracker()
         chunk = Chunk(text="Test", start_index=0, end_index=4, metadata={})
         chunk.id = "chunk_1"
         split_tracker.track_chunk(chunk, source_document="doc_1")
         
-        # Both should work
-        kg_lineage = kg_tracker.get_lineage("entity_1")
+        # split.ProvenanceTracker.get_provenance() is real and still works
         split_prov = split_tracker.get_provenance("chunk_1")
         
-        assert kg_lineage is not None
         assert split_prov is not None
     
     def test_unified_manager_with_all_modules(self):

tests/provenance/test_backward_compat.py

@@ -15,30 +15,32 @@ class TestKGProvenanceBackwardCompat:
     """Test kg.ProvenanceTracker backward compatibility."""
     
     def test_existing_code_unchanged(self):
-        """Test that existing kg.ProvenanceTracker code works unchanged."""
+        """Test that existing kg.ProvenanceTracker code works unchanged.
+
+        NOTE: this no longer asserts on tracker.get_lineage(), which was
+        never implemented on kg.ProvenanceTracker (see #744). It instead
+        verifies the observable behavior of the still-supported
+        track_entity()/get_all_sources() pair.
+        """
         # Existing code pattern
         tracker = KGProvenanceTracker()
         
         # Track entity (existing API)
         tracker.track_entity("entity_1", source="doc_1", metadata={"confidence": 0.9})
         
-        # Get lineage (existing API)
-        lineage = tracker.get_lineage("entity_1")
-        
-        # Verify existing return format
-        assert "sources" in lineage
-        assert "first_seen" in lineage
-        assert "last_updated" in lineage
-        assert "metadata" in lineage
+        # Verify the entity was actually tracked
+        sources = tracker.get_all_sources("entity_1")
+        assert len(sources) > 0
+        last_entry = sources[-1]
+        assert last_entry["source"] == "doc_1"
+        assert "recorded_at" in last_entry
+        assert last_entry["confidence"] == 0.9
     
-    def test_track_relationship_unchanged(self):
-        """Test relationship tracking works unchanged."""
-        tracker = KGProvenanceTracker()
-        
-        tracker.track_relationship("rel_1", source="doc_1", metadata={"type": "founded"})
-        
-        lineage = tracker.get_lineage("rel_1")
-        assert lineage is not None
+    # NOTE: test_track_relationship_unchanged removed; it only exercised
+    # tracker.track_relationship(), which was never implemented on
+    # kg.ProvenanceTracker. This tested an intended unified-backend
+    # migration that never happened (#744). ProvenanceTracker is now
+    # deprecated in favor of semantica.provenance.ProvenanceManager.
     
     def test_get_all_sources_unchanged(self):
         """Test get_all_sources returns expected format."""
@@ -53,22 +55,14 @@ class TestKGProvenanceBackwardCompat:
         assert len(sources) >= 2
         for source in sources:
             assert "source" in source
-            assert "timestamp" in source
+            assert "recorded_at" in source
     
-    def test_batch_operations_unchanged(self):
-        """Test batch operations work unchanged."""
-        tracker = KGProvenanceTracker()
-        
-        entities = [
-            {"id": "entity_1", "confidence": 0.9},
-            {"id": "entity_2", "confidence": 0.85}
-        ]
-        
-        count = tracker.track_entities_batch(entities, "doc_1")
-        
-        assert count == 2
-        assert tracker.get_lineage("entity_1") is not None
-        assert tracker.get_lineage("entity_2") is not None
+    # NOTE: test_batch_operations_unchanged removed; it only exercised
+    # tracker.track_entities_batch() and tracker.get_lineage(), neither of
+    # which was ever implemented on kg.ProvenanceTracker. This tested an
+    # intended unified-backend migration that never happened (#744).
+    # ProvenanceTracker is now deprecated in favor of
+    # semantica.provenance.ProvenanceManager.
 
 
 class TestSplitProvenanceBackwardCompat:
@@ -196,10 +190,10 @@ class TestGracefulDegradation:
         
         # Should work even if unified backend has issues
         tracker.track_entity("entity_1", source="doc_1")
-        lineage = tracker.get_lineage("entity_1")
-        
-        assert lineage is not None
-        assert "sources" in lineage
+        # NOTE: tracker.get_lineage() was never implemented on
+        # kg.ProvenanceTracker; this tested an intended unified-backend
+        # migration that never happened (#744). ProvenanceTracker is now
+        # deprecated in favor of semantica.provenance.ProvenanceManager.
     
     def test_split_tracker_fallback(self):
         """Test split.ProvenanceTracker falls back to legacy on error."""
@@ -221,18 +215,20 @@ class TestExistingTestsPass:
         tracker = KGProvenanceTracker()
         
         # Test 1: Basic tracking
+        # NOTE: tracker.get_provenance() was never implemented on
+        # kg.ProvenanceTracker; this tested an intended unified-backend
+        # migration that never happened (#744). ProvenanceTracker is now
+        # deprecated in favor of semantica.provenance.ProvenanceManager.
         tracker.track_entity("e1", "src1")
-        assert tracker.get_provenance("e1") is not None
         
         # Test 2: Multiple sources
         tracker.track_entity("e1", "src2")
         sources = tracker.get_all_sources("e1")
         assert len(sources) >= 2
         
-        # Test 3: Metadata
-        tracker.track_entity("e2", "src1", metadata={"key": "value"})
-        lineage = tracker.get_lineage("e2")
-        assert "metadata" in lineage
+        # NOTE: Test 3 (metadata via tracker.get_lineage()) removed; that
+        # method was never implemented on kg.ProvenanceTracker. This tested
+        # an intended unified-backend migration that never happened (#744).
     
     def test_split_provenance_existing_behavior(self):
         """Test existing split.ProvenanceTracker behavior is preserved."""

tests/provenance/test_integration.py

@@ -31,18 +31,25 @@ class TestEndToEndProvenance:
         assert entity_prov is not None
         assert chunk_prov is not None
     
-    def test_kg_to_unified_integration(self):
-        """Test kg.ProvenanceTracker uses unified backend."""
+    def test_kg_tracker_records_provenance(self):
+        """Test kg.ProvenanceTracker records provenance via its supported API.
+
+        NOTE: this no longer asserts kg.ProvenanceTracker uses a unified
+        backend (kg_tracker.get_lineage() was never implemented; see #744).
+        It instead verifies the observable behavior of the still-supported
+        track_entity()/get_all_sources() pair.
+        """
         kg_tracker = KGTracker()
         
         # Track with kg tracker
         kg_tracker.track_entity("kg_entity_1", source="kg_doc_1")
         
         # Verify it was tracked
-        lineage = kg_tracker.get_lineage("kg_entity_1")
-        
-        assert lineage is not None
-        assert "sources" in lineage
+        sources = kg_tracker.get_all_sources("kg_entity_1")
+        assert len(sources) > 0
+        last_entry = sources[-1]
+        assert last_entry["source"] == "kg_doc_1"
+        assert "recorded_at" in last_entry
     
     def test_split_to_unified_integration(self):
         """Test split.ProvenanceTracker uses unified backend."""

tests/provenance/test_llms_real_tracking.py

@@ -109,7 +109,7 @@ class TestRealLLMProvenanceTracking:
             assert lineage is not None
             total_cost += lineage["metadata"]["total_cost"]
         
-        assert total_cost == sum(costs)
+        assert total_cost == pytest.approx(sum(costs))
     
     def test_llm_latency_tracking(self):
         """Test that LLM latency is tracked."""

tests/provenance/test_manager.py

@@ -104,7 +104,192 @@ class TestProvenanceManager:
         assert lineage is not None
         assert "lineage_chain" in lineage
         assert len(lineage["lineage_chain"]) > 0
-    
+
+    def test_get_lineage_via_derived_from_metadata(self):
+        """metadata['derived_from'] should link entities into the lineage chain
+        even when they share a source URL rather than one being a known entity_id."""
+        prov_mgr = ProvenanceManager()
+
+        prov_mgr.track_entity(
+            entity_id="doc:X",
+            source="https://example.com/api",
+            metadata={"content_type": "drug_label"},
+        )
+        prov_mgr.track_entity(
+            entity_id="decision:Y",
+            source="https://example.com/api",
+            metadata={"derived_from": "doc:X"},
+        )
+
+        lineage = prov_mgr.get_lineage("decision:Y")
+
+        assert lineage["entity_count"] == 2
+        entity_ids = [e["entity_id"] for e in lineage["lineage_chain"]]
+        assert "doc:X" in entity_ids
+        assert "decision:Y" in entity_ids
+
+    def test_derived_from_does_not_override_explicit_parent(self):
+        """An explicit parent_entity_id kwarg should win over metadata['derived_from']."""
+        prov_mgr = ProvenanceManager()
+
+        prov_mgr.track_entity(entity_id="explicit_parent", source="doc_1")
+        prov_mgr.track_entity(entity_id="ignored_parent", source="doc_1")
+        entry = prov_mgr.track_entity(
+            entity_id="child",
+            source="doc_1",
+            metadata={"derived_from": "ignored_parent"},
+            parent_entity_id="explicit_parent",
+        )
+
+        assert entry.parent_entity_id == "explicit_parent"
+
+    def test_derived_from_takes_precedence_over_source_as_entity_id(self):
+        """If `source` happens to also be a known entity_id, an explicit
+        metadata['derived_from'] should still win over that fallback linking."""
+        prov_mgr = ProvenanceManager()
+
+        prov_mgr.track_entity(entity_id="source_as_entity", source="doc_0")
+        prov_mgr.track_entity(entity_id="real_parent", source="doc_0")
+        entry = prov_mgr.track_entity(
+            entity_id="child",
+            source="source_as_entity",  # resolvable as an entity_id
+            metadata={"derived_from": "real_parent"},
+        )
+
+        assert entry.parent_entity_id == "real_parent"
+
+    def test_derived_from_nonexistent_entity_does_not_crash(self):
+        """derived_from pointing at an entity that was never tracked should be
+        stored as the parent link without raising, and lineage traversal should
+        stop gracefully instead of erroring."""
+        prov_mgr = ProvenanceManager()
+
+        entry = prov_mgr.track_entity(
+            entity_id="orphan_child",
+            source="doc_1",
+            metadata={"derived_from": "never_tracked"},
+        )
+
+        assert entry.parent_entity_id == "never_tracked"
+
+        lineage = prov_mgr.get_lineage("orphan_child")
+        entity_ids = [e["entity_id"] for e in lineage["lineage_chain"]]
+        assert entity_ids == ["orphan_child"]
+
+    def test_derived_from_non_string_is_ignored(self):
+        """A non-string derived_from value (e.g. accidentally passing an int or
+        list) should be ignored rather than raising or being used as a parent id."""
+        prov_mgr = ProvenanceManager()
+
+        entry = prov_mgr.track_entity(
+            entity_id="entity_bad_derived_from",
+            source="doc_1",
+            metadata={"derived_from": 12345},
+        )
+
+        assert entry.parent_entity_id is None
+
+    def test_derived_from_empty_string_is_ignored(self):
+        """An empty-string derived_from is falsy and should not be treated as a parent link."""
+        prov_mgr = ProvenanceManager()
+
+        entry = prov_mgr.track_entity(
+            entity_id="entity_empty_derived_from",
+            source="doc_1",
+            metadata={"derived_from": ""},
+        )
+
+        assert entry.parent_entity_id is None
+
+    def test_derived_from_self_reference_does_not_infinite_loop(self):
+        """An entity that (incorrectly) declares itself as its own derived_from
+        parent should not cause get_lineage to hang or infinitely recurse."""
+        prov_mgr = ProvenanceManager()
+
+        prov_mgr.track_entity(
+            entity_id="self_ref",
+            source="doc_1",
+            metadata={"derived_from": "self_ref"},
+        )
+
+        lineage = prov_mgr.get_lineage("self_ref")
+        entity_ids = [e["entity_id"] for e in lineage["lineage_chain"]]
+        assert entity_ids == ["self_ref"]
+
+    def test_derived_from_multi_hop_chain(self):
+        """derived_from links should chain transitively: A <- B <- C should
+        all appear when tracing lineage from C."""
+        prov_mgr = ProvenanceManager()
+
+        prov_mgr.track_entity(entity_id="grandparent", source="doc_1")
+        prov_mgr.track_entity(
+            entity_id="parent",
+            source="doc_1",
+            metadata={"derived_from": "grandparent"},
+        )
+        prov_mgr.track_entity(
+            entity_id="child",
+            source="doc_1",
+            metadata={"derived_from": "parent"},
+        )
+
+        lineage = prov_mgr.get_lineage("child")
+
+        assert lineage["entity_count"] == 3
+        entity_ids = {e["entity_id"] for e in lineage["lineage_chain"]}
+        assert entity_ids == {"grandparent", "parent", "child"}
+
+    def test_derived_from_without_metadata_dict_does_not_crash(self):
+        """track_entity called with no metadata at all should behave as before
+        (no parent link derived), exercising the `metadata and isinstance(...)` guard."""
+        prov_mgr = ProvenanceManager()
+
+        entry = prov_mgr.track_entity(entity_id="no_metadata_entity", source="doc_1")
+
+        assert entry.parent_entity_id is None
+
+    def test_get_lineage_metadata_prefers_queried_entity_over_ancestors(self):
+        """Aggregated lineage metadata should let the queried entity's own
+        values win over ancestor values on conflicting keys, matching the
+        documented "most recent entry's metadata takes precedence" intent."""
+        prov_mgr = ProvenanceManager()
+
+        prov_mgr.track_entity(
+            entity_id="ancestor",
+            source="doc_1",
+            metadata={"status": "draft", "shared_only_on_ancestor": True},
+        )
+        prov_mgr.track_entity(
+            entity_id="descendant",
+            source="doc_1",
+            metadata={"status": "final", "derived_from": "ancestor"},
+        )
+
+        lineage = prov_mgr.get_lineage("descendant")
+
+        assert lineage["metadata"]["status"] == "final"
+        assert lineage["metadata"]["shared_only_on_ancestor"] is True
+
+    def test_derived_from_accepts_non_dict_mapping(self):
+        """metadata['derived_from'] should be honored for any Mapping
+        implementation, not just a concrete dict (e.g. types.MappingProxyType
+        or a custom collections.abc.Mapping)."""
+        from types import MappingProxyType
+
+        prov_mgr = ProvenanceManager()
+
+        prov_mgr.track_entity(entity_id="mapping_parent", source="doc_1")
+        entry = prov_mgr.track_entity(
+            entity_id="mapping_child",
+            source="doc_1",
+            metadata=MappingProxyType({"derived_from": "mapping_parent"}),
+        )
+
+        assert entry.parent_entity_id == "mapping_parent"
+
+        lineage = prov_mgr.get_lineage("mapping_child")
+        assert lineage["entity_count"] == 2
+
     def test_batch_entity_tracking(self):
         """Test batch entity tracking."""
         prov_mgr = ProvenanceManager()
@@ -154,3 +339,71 @@ class TestProvenanceManager:
         
         lineage = prov_mgr.get_lineage("entity_1")
         assert lineage == {}
+
+    def test_retrack_with_explicit_parent_overrides_history_link(self):
+        """#742 — re-tracking an entity with an explicit parent_entity_id must
+        honor the new value, not silently replace it with an auto-generated
+        history pointer."""
+        prov_mgr = ProvenanceManager()
+
+        e1 = prov_mgr.track_entity("X", source="doc_1", parent_entity_id="parent_v1")
+        e2 = prov_mgr.track_entity("X", source="doc_1", parent_entity_id="parent_v2")
+
+        assert e1.parent_entity_id == "parent_v1"
+        assert e2.parent_entity_id == "parent_v2"
+
+    def test_retrack_without_explicit_parent_still_uses_history_link(self):
+        """#742 — when NO explicit parent is given on a re-track call, the
+        auto-generated history link (Y:v:<timestamp>) should still be used,
+        preserving pre-existing behavior for callers that don't supply a parent."""
+        prov_mgr = ProvenanceManager()
+
+        y1 = prov_mgr.track_entity("Y", source="doc_1")
+        y2 = prov_mgr.track_entity("Y", source="doc_1")
+
+        assert y1.parent_entity_id is None
+        assert y2.parent_entity_id is not None
+        assert y2.parent_entity_id.startswith("Y:v:")
+
+    def test_retrack_with_derived_from_overrides_history_link(self):
+        """#742 — re-tracking with metadata['derived_from'] (no parent_entity_id
+        kwarg) should also override the auto-generated history link, not just
+        the parent_entity_id kwarg case."""
+        prov_mgr = ProvenanceManager()
+
+        prov_mgr.track_entity("parent_A", source="doc_1")
+        prov_mgr.track_entity("parent_B", source="doc_1")
+
+        e1 = prov_mgr.track_entity("Z", source="doc_1", metadata={"derived_from": "parent_A"})
+        e2 = prov_mgr.track_entity("Z", source="doc_1", metadata={"derived_from": "parent_B"})
+
+        assert e1.parent_entity_id == "parent_A"
+        assert e2.parent_entity_id == "parent_B"
+
+    def test_retrack_history_reachable_via_used_entities(self):
+        """#742 — when re-tracking with an explicit parent, the archived history
+        entry for the previous version must still be reachable in the lineage
+        chain via used_entities (prov:used), even though it's no longer the
+        direct parent_entity_id."""
+        prov_mgr = ProvenanceManager()
+
+        prov_mgr.track_entity("explicit_parent", source="doc_1")
+        prov_mgr.track_entity("X", source="doc_1")  # first track, no parent
+
+        # Re-track with an explicit parent — should NOT lose the history entry
+        e2 = prov_mgr.track_entity("X", source="doc_1", parent_entity_id="explicit_parent")
+
+        assert e2.parent_entity_id == "explicit_parent"
+        assert len(e2.used_entities) == 1
+        assert e2.used_entities[0].startswith("X:v:")
+
+        # trace_lineage should reach: X, explicit_parent (via parent_entity_id),
+        # AND the archived history snapshot (via used_entities)
+        lineage = prov_mgr.get_lineage("X")
+        entity_ids = {e["entity_id"] for e in lineage["lineage_chain"]}
+
+        assert "X" in entity_ids
+        assert "explicit_parent" in entity_ids
+        assert e2.used_entities[0] in entity_ids, (
+            "Archived history entry should be reachable via used_entities in lineage"
+        )

tests/reasoning/test_reasoner.py

@@ -64,6 +64,132 @@ class TestReasoner(unittest.TestCase):
         self.assertIn("Person(John)", result.premises)
         self.assertIn("Parent(John, Jane)", result.premises)
 
+    def test_forward_chaining_premises(self):
+        """Mirrors test_backward_chaining_simple: forward_chain() must attach the
+        specific facts that matched the rule's conditions as premises, not leave
+        them empty (regression guard for issue #733)."""
+        self.reasoner.add_rule("IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)")
+        self.reasoner.add_fact("Person(John)")
+        self.reasoner.add_fact("Parent(John, Jane)")
+
+        results = self.reasoner.forward_chain()
+        self.assertEqual(len(results), 1)
+        result = results[0]
+        self.assertEqual(result.conclusion, "Child(Jane, John)")
+        self.assertEqual(len(result.premises), 2)
+        self.assertIn("Person(John)", result.premises)
+        self.assertIn("Parent(John, Jane)", result.premises)
+
+    def test_add_rule_deduplicates_identical_rule(self):
+        """Bug #732 — re-adding an identical rule string must not duplicate it,
+        so re-running the same setup code (e.g. a Jupyter cell) is idempotent."""
+        rule_str = "IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
+        first = self.reasoner.add_rule(rule_str)
+        second = self.reasoner.add_rule(rule_str)
+
+        self.assertEqual(len(self.reasoner.rules), 1)
+        self.assertIs(first, second)
+
+    def test_add_rule_deduplication_is_idempotent_across_forward_chain(self):
+        """Bug #732 — rerunning add_rule()+add_fact()+forward_chain() on the same
+        Reasoner instance must not grow the rule count on each call."""
+        def run_cell():
+            self.reasoner.add_rule("IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)")
+            self.reasoner.add_fact("Person(John)")
+            self.reasoner.add_fact("Parent(John, Jane)")
+            return self.reasoner.forward_chain()
+
+        result1 = run_cell()
+        self.assertEqual(len(self.reasoner.rules), 1)
+        self.assertEqual([r.conclusion for r in result1], ["Child(Jane, John)"])
+
+        result2 = run_cell()
+        self.assertEqual(len(self.reasoner.rules), 1)
+        # Nothing new to derive since the fact was already known -- this is
+        # now a consistent, expected empty result rather than a symptom of
+        # unbounded rule duplication.
+        self.assertEqual(result2, [])
+
+    def test_add_rule_duplicate_logs_warning(self):
+        """Bug #732 follow-up — a skipped duplicate rule must be surfaced via a
+        warning log, not silently swallowed at debug level."""
+        rule_str = "IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"
+        self.reasoner.add_rule(rule_str)
+
+        with self.assertLogs(self.reasoner.logger.name, level="WARNING") as cm:
+            self.reasoner.add_rule(rule_str)
+
+        self.assertTrue(any("duplicate rule" in msg for msg in cm.output))
+
+    def test_add_rule_duplicate_with_different_confidence_logs_warning(self):
+        """Re-adding an identical rule (same conditions/conclusion) with a
+        different confidence must warn that the new confidence is discarded
+        and the original is retained, not silently drop it."""
+        rule_v1 = Rule(
+            rule_id="r1", name="Rule", conditions=["A(?x)"], conclusion="B(?x)",
+            confidence=0.6,
+        )
+        rule_v2 = Rule(
+            rule_id="r2", name="Rule v2", conditions=["A(?x)"], conclusion="B(?x)",
+            confidence=0.95,
+        )
+        self.reasoner.add_rule(rule_v1)
+
+        with self.assertLogs(self.reasoner.logger.name, level="WARNING") as cm:
+            result = self.reasoner.add_rule(rule_v2)
+
+        self.assertIs(result, rule_v1)
+        self.assertEqual(result.confidence, 0.6)
+        self.assertTrue(any("different confidence" in msg for msg in cm.output))
+
+    def test_add_rule_duplicate_with_non_string_conditions_does_not_raise(self):
+        """Bug #732 follow-up — the duplicate-rule warning message building must
+        not raise TypeError when Rule.conditions contains non-string entries
+        (Rule.conditions is typed List[Any])."""
+        rule = Rule(
+            rule_id="r1",
+            name="Test Rule",
+            conditions=[("Person", "?x")],
+            conclusion="B(?x)",
+        )
+        duplicate = Rule(
+            rule_id="r2",
+            name="Test Rule Duplicate",
+            conditions=[("Person", "?x")],
+            conclusion="B(?x)",
+        )
+        self.reasoner.add_rule(rule)
+        result = self.reasoner.add_rule(duplicate)
+
+        self.assertIs(result, rule)
+        self.assertEqual(len(self.reasoner.rules), 1)
+
+    def test_add_rule_does_not_dedupe_distinct_rules(self):
+        """Rules with different conditions/conclusions must still both be added."""
+        self.reasoner.add_rule("IF A(?x) THEN B(?x)")
+        self.reasoner.add_rule("IF A(?x) THEN C(?x)")
+        self.assertEqual(len(self.reasoner.rules), 2)
+
+    def test_add_rule_duplicate_resorts_on_mutated_priority(self):
+        """Bug #732 follow-up — Rule is a mutable dataclass, so an already-added
+        rule's priority may change after it was registered; re-adding it (a
+        duplicate by conditions/conclusion) must still re-sort self.rules
+        rather than leaving it stale relative to the mutated priority."""
+        low = Rule(rule_id="r1", name="Low", conditions=["A(?x)"], conclusion="B(?x)", priority=0)
+        high = Rule(rule_id="r2", name="High", conditions=["C(?x)"], conclusion="D(?x)", priority=5)
+        self.reasoner.add_rule(low)
+        self.reasoner.add_rule(high)
+        self.assertEqual([r.rule_id for r in self.reasoner.rules], ["r2", "r1"])
+
+        # Mutate the already-registered low-priority rule to outrank "high",
+        # then re-add it (matches by conditions/conclusion -> dedup path).
+        low.priority = 10
+        result = self.reasoner.add_rule(low)
+
+        self.assertIs(result, low)
+        self.assertEqual(len(self.reasoner.rules), 2)
+        self.assertEqual([r.rule_id for r in self.reasoner.rules], ["r1", "r2"])
+
     def test_infer_facts(self):
         facts = ["Person(John)", "Parent(John, Jane)"]
         rules = ["IF Person(?x) AND Parent(?x, ?y) THEN Child(?y, ?x)"]

tests/test_databricks_ingestor.py

@@ -0,0 +1,912 @@
+"""
+Unit tests for Databricks Ingestor
+
+This test module uses mocks to test Databricks ingestion functionality
+without requiring a live Databricks workspace.
+"""
+
+import os
+from datetime import datetime
+from unittest.mock import MagicMock, Mock, patch
+
+import pytest
+
+# Test if databricks-sdk / databricks-sql-connector are available
+try:
+    import databricks.sdk  # noqa: F401
+    from databricks import sql  # noqa: F401
+
+    DATABRICKS_LIBS_AVAILABLE = True
+except ImportError:
+    DATABRICKS_LIBS_AVAILABLE = False
+
+
+@pytest.fixture(autouse=True)
+def mock_databricks_if_needed():
+    """Mock databricks modules if not installed."""
+    if not DATABRICKS_LIBS_AVAILABLE:
+        with patch.dict(
+            "sys.modules",
+            {
+                "databricks": MagicMock(),
+                "databricks.sql": MagicMock(),
+                "databricks.sdk": MagicMock(),
+            },
+        ):
+            yield
+    else:
+        yield
+
+
+@pytest.fixture
+def mock_databricks_connection():
+    """Create a mock Databricks SQL connection."""
+    mock_conn = Mock()
+    mock_cursor = Mock()
+
+    mock_cursor.execute = Mock()
+    mock_cursor.fetchall = Mock(return_value=[])
+    mock_cursor.fetchone = Mock(return_value=[1])
+    mock_cursor.fetchmany = Mock(return_value=[])
+    mock_cursor.description = [("id", None), ("name", None), ("value", None)]
+    mock_cursor.close = Mock()
+
+    mock_conn.cursor = Mock(return_value=mock_cursor)
+    mock_conn.close = Mock()
+
+    return mock_conn, mock_cursor
+
+
+class TestDatabricksConnector:
+    """Test DatabricksConnector class."""
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_init_with_token(self, mock_sql):
+        """Test connector initialization with personal access token authentication."""
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+
+        connector = DatabricksConnector(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+            catalog="TEST_CATALOG",
+        )
+
+        assert connector.host == "https://adb-xxx.azuredatabricks.net"
+        assert connector.token == "test_token"
+        assert connector.http_path == "/sql/1.0/warehouses/xxxx"
+        assert connector.catalog == "TEST_CATALOG"
+        assert connector.schema == "default"
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_init_from_env(self, mock_sql):
+        """Test connector initialization from environment variables."""
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+
+        with patch.dict(
+            os.environ,
+            {
+                "DATABRICKS_HOST": "https://env-host.azuredatabricks.net",
+                "DATABRICKS_TOKEN": "env_token",
+                "DATABRICKS_HTTP_PATH": "/sql/1.0/warehouses/env",
+            },
+        ):
+            connector = DatabricksConnector()
+
+            assert connector.host == "https://env-host.azuredatabricks.net"
+            assert connector.token == "env_token"
+            assert connector.http_path == "/sql/1.0/warehouses/env"
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_init_missing_host(self, mock_sql):
+        """Test connector initialization fails without host."""
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+        from semantica.utils.exceptions import ValidationError
+
+        with pytest.raises(ValidationError, match="Databricks host is required"):
+            DatabricksConnector(token="test_token")
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_init_missing_auth(self, mock_sql):
+        """Test connector initialization fails without any authentication method."""
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+        from semantica.utils.exceptions import ValidationError
+
+        with pytest.raises(ValidationError, match="Databricks authentication is required"):
+            DatabricksConnector(host="https://adb-xxx.azuredatabricks.net")
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_connect_token_auth(self, mock_sql, mock_databricks_connection):
+        """Test connection with personal access token authentication."""
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+
+        mock_conn, _ = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        connector = DatabricksConnector(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        conn = connector.connect()
+
+        assert conn == mock_conn
+        mock_sql.connect.assert_called_once()
+
+        call_kwargs = mock_sql.connect.call_args[1]
+        assert call_kwargs["server_hostname"] == "adb-xxx.azuredatabricks.net"
+        assert call_kwargs["http_path"] == "/sql/1.0/warehouses/xxxx"
+        assert call_kwargs["access_token"] == "test_token"
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.oauth_service_principal")
+    @patch("semantica.ingest.databricks_ingestor.Config")
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_connect_oauth_m2m(
+        self, mock_sql, mock_config, mock_oauth_sp, mock_databricks_connection
+    ):
+        """Test connection with OAuth M2M authentication uses credentials_provider.
+
+        databricks-sql-connector does NOT accept client_id/client_secret as
+        direct kwargs to sql.connect(); the correct mechanism is a
+        credentials_provider callable wrapping oauth_service_principal().
+        """
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+
+        mock_conn, _ = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+        mock_oauth_sp.return_value = {"Authorization": "Bearer fake-token"}
+
+        connector = DatabricksConnector(
+            host="https://adb-xxx.azuredatabricks.net",
+            http_path="/sql/1.0/warehouses/xxxx",
+            client_id="test_client_id",
+            client_secret="test_client_secret",
+        )
+
+        connector.connect()
+
+        call_kwargs = mock_sql.connect.call_args[1]
+
+        # Must use credentials_provider, not bare client_id/client_secret
+        assert "credentials_provider" in call_kwargs
+        assert callable(call_kwargs["credentials_provider"])
+        assert "client_id" not in call_kwargs
+        assert "client_secret" not in call_kwargs
+        assert "access_token" not in call_kwargs
+
+        # Invoke the provider to verify it wires Config + oauth_service_principal
+        call_kwargs["credentials_provider"]()
+        mock_config.assert_called_once_with(
+            host="https://adb-xxx.azuredatabricks.net",
+            client_id="test_client_id",
+            client_secret="test_client_secret",
+        )
+        mock_oauth_sp.assert_called_once_with(mock_config.return_value)
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_connect_missing_http_path(self, mock_sql):
+        """Test connection fails without http_path."""
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+        from semantica.utils.exceptions import ValidationError
+
+        connector = DatabricksConnector(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+        )
+
+        with pytest.raises(ValidationError, match="http_path"):
+            connector.connect()
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_disconnect(self, mock_sql, mock_databricks_connection):
+        """Test connection disconnect."""
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+
+        mock_conn, _ = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        connector = DatabricksConnector(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        connector.connect()
+        connector.disconnect()
+
+        mock_conn.close.assert_called_once()
+        assert connector.connection is None
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_test_connection_success(self, mock_sql, mock_databricks_connection):
+        """Test successful connection test."""
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+
+        mock_conn, mock_cursor = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        connector = DatabricksConnector(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        result = connector.test_connection()
+
+        assert result is True
+        mock_cursor.execute.assert_called_with("SELECT 1")
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_connector_test_connection_failure(self, mock_sql):
+        """Test connection test failure."""
+        from semantica.ingest.databricks_ingestor import DatabricksConnector
+
+        mock_sql.connect = Mock(side_effect=Exception("Connection failed"))
+
+        connector = DatabricksConnector(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        result = connector.test_connection()
+
+        assert result is False
+
+
+class TestDatabricksIngestor:
+    """Test DatabricksIngestor class."""
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingestor_init(self, mock_sql):
+        """Test ingestor initialization."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        assert ingestor.connector is not None
+        assert ingestor.connector.host == "https://adb-xxx.azuredatabricks.net"
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingest_table_basic(self, mock_sql, mock_databricks_connection):
+        """Test basic table ingestion."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, mock_cursor = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        mock_cursor.fetchall = Mock(
+            return_value=[
+                (1, "Alice", 100),
+                (2, "Bob", 200),
+            ]
+        )
+        mock_cursor.description = [("id", None), ("name", None), ("value", None)]
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+            catalog="TEST_CATALOG",
+            schema="TEST_SCHEMA",
+        )
+
+        data = ingestor.ingest_table("customers")
+
+        assert data.row_count == 2
+        assert data.table_name == "customers"
+        assert data.catalog == "TEST_CATALOG"
+        assert data.schema == "TEST_SCHEMA"
+        assert len(data.columns) == 3
+        assert "id" in data.columns
+        assert data.data[0]["name"] == "Alice"
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingest_table_closes_connection(self, mock_sql, mock_databricks_connection):
+        """Test that ingest_table() closes the SQL connection after use instead of leaking it."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, _ = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        ingestor.ingest_table("customers")
+
+        mock_conn.close.assert_called_once()
+        assert ingestor.connector.connection is None
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingest_table_catalog_only(self, mock_sql, mock_databricks_connection):
+        """Test table ingestion still qualifies the reference when only catalog is provided."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, mock_cursor = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+        # Force a missing schema past the connector's "default" fallback.
+        ingestor.connector.schema = None
+
+        ingestor.ingest_table("customers", catalog="main")
+
+        executed_query = mock_cursor.execute.call_args[0][0]
+        assert "`main`.`customers`" in executed_query
+        assert "None" not in executed_query
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingest_table_with_limit(self, mock_sql, mock_databricks_connection):
+        """Test table ingestion with limit."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, mock_cursor = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        ingestor.ingest_table("customers", limit=100)
+
+        executed_query = mock_cursor.execute.call_args[0][0]
+        assert "LIMIT 100" in executed_query
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingest_table_with_where(self, mock_sql, mock_databricks_connection):
+        """Test table ingestion with WHERE clause."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, mock_cursor = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        ingestor.ingest_table("customers", where="value > 100")
+
+        executed_query = mock_cursor.execute.call_args[0][0]
+        assert "WHERE value > 100" in executed_query
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingest_table_rejects_unsafe_order_by(self, mock_sql, mock_databricks_connection):
+        """Test table ingestion rejects unsafe ORDER BY clauses."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, _ = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        from semantica.utils.exceptions import ProcessingError
+
+        with pytest.raises(ProcessingError):
+            ingestor.ingest_table("customers", order_by="value; DROP TABLE customers")
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingest_query_basic(self, mock_sql, mock_databricks_connection):
+        """Test basic query execution."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, mock_cursor = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        mock_cursor.fetchall = Mock(return_value=[(1000,)])
+        mock_cursor.description = [("total", None)]
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        query = "SELECT SUM(value) AS total FROM sales"
+        data = ingestor.ingest_query(query)
+
+        assert data.row_count == 1
+        assert data.query == query
+        assert data.data[0]["total"] == 1000
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingest_query_closes_connection(self, mock_sql, mock_databricks_connection):
+        """Test that ingest_query() closes the SQL connection after use instead of leaking it."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, _ = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        ingestor.ingest_query("SELECT * FROM sales")
+
+        mock_conn.close.assert_called_once()
+        assert ingestor.connector.connection is None
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_ingest_query_with_batching(self, mock_sql, mock_databricks_connection):
+        """Test query execution with batch fetching."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, mock_cursor = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        batch1 = [(1,), (2,)]
+        batch2 = [(3,)]
+        mock_cursor.fetchmany = Mock(side_effect=[batch1, batch2, []])
+        mock_cursor.description = [("id", None)]
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        data = ingestor.ingest_query("SELECT * FROM customers", batch_size=2)
+
+        assert data.row_count == 3
+        assert len(data.data) == 3
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+    def test_get_table_schema(self, mock_ws_client_cls, mock_sql):
+        """Test getting table schema information."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_column_1 = Mock(name="id")
+        mock_column_1.name = "id"
+        mock_column_1.type_text = "BIGINT"
+        mock_column_1.nullable = False
+        mock_column_1.comment = None
+
+        mock_column_2 = Mock(name="name")
+        mock_column_2.name = "name"
+        mock_column_2.type_text = "STRING"
+        mock_column_2.nullable = True
+        mock_column_2.comment = None
+
+        mock_table_info = Mock()
+        mock_table_info.columns = [mock_column_1, mock_column_2]
+
+        mock_ws_client = Mock()
+        mock_ws_client.tables.get = Mock(return_value=mock_table_info)
+        mock_ws_client_cls.return_value = mock_ws_client
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+            catalog="TEST_CATALOG",
+            schema="TEST_SCHEMA",
+        )
+
+        schema = ingestor.get_table_schema("customers")
+
+        assert len(schema["columns"]) == 2
+        assert schema["columns"][0]["name"] == "id"
+        assert schema["columns"][0]["type"] == "BIGINT"
+        assert schema["columns"][0]["nullable"] is False
+        mock_ws_client.tables.get.assert_called_once_with(
+            full_name="TEST_CATALOG.TEST_SCHEMA.customers"
+        )
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+    def test_get_table_schema_requires_schema(self, mock_ws_client_cls, mock_sql):
+        """Test that get_table_schema() raises instead of calling the SDK with a missing schema."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+        from semantica.utils.exceptions import ProcessingError
+
+        mock_ws_client = Mock()
+        mock_ws_client_cls.return_value = mock_ws_client
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+            catalog="TEST_CATALOG",
+        )
+        ingestor.connector.schema = None
+
+        with pytest.raises(ProcessingError, match="Schema name is required"):
+            ingestor.get_table_schema("customers")
+
+        mock_ws_client.tables.get.assert_not_called()
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+    def test_list_catalogs(self, mock_ws_client_cls, mock_sql):
+        """Test listing catalogs."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_catalog_1 = Mock()
+        mock_catalog_1.name = "main"
+        mock_catalog_2 = Mock()
+        mock_catalog_2.name = "samples"
+
+        mock_ws_client = Mock()
+        mock_ws_client.catalogs.list = Mock(return_value=[mock_catalog_1, mock_catalog_2])
+        mock_ws_client_cls.return_value = mock_ws_client
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        catalogs = ingestor.list_catalogs()
+
+        assert len(catalogs) == 2
+        assert "main" in catalogs
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+    def test_list_tables(self, mock_ws_client_cls, mock_sql):
+        """Test listing tables in a catalog/schema."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_table_1 = Mock()
+        mock_table_1.name = "customers"
+        mock_table_2 = Mock()
+        mock_table_2.name = "orders"
+
+        mock_ws_client = Mock()
+        mock_ws_client.tables.list = Mock(return_value=[mock_table_1, mock_table_2])
+        mock_ws_client_cls.return_value = mock_ws_client
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+            catalog="TEST_CATALOG",
+            schema="TEST_SCHEMA",
+        )
+
+        tables = ingestor.list_tables()
+
+        assert len(tables) == 2
+        assert "customers" in tables
+        mock_ws_client.tables.list.assert_called_once_with(
+            catalog_name="TEST_CATALOG", schema_name="TEST_SCHEMA"
+        )
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+    def test_list_tables_requires_schema(self, mock_ws_client_cls, mock_sql):
+        """Test that list_tables() raises instead of calling the SDK with schema_name=None."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+        from semantica.utils.exceptions import ProcessingError
+
+        mock_ws_client = Mock()
+        mock_ws_client_cls.return_value = mock_ws_client
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+            catalog="TEST_CATALOG",
+        )
+        # Force a missing schema past the connector's "default" fallback.
+        ingestor.connector.schema = None
+
+        with pytest.raises(ProcessingError, match="Schema name is required"):
+            ingestor.list_tables()
+
+        mock_ws_client.tables.list.assert_not_called()
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+    def test_get_table_lineage(self, mock_ws_client_cls, mock_sql):
+        """Test getting table lineage."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_ws_client = Mock()
+        mock_ws_client.api_client.do = Mock(
+            return_value={
+                "upstreams": [{"tableInfo": {"name": "main.default.raw_customers"}}],
+                "downstreams": [{"tableInfo": {"name": "main.default.customer_summary"}}],
+            }
+        )
+        mock_ws_client_cls.return_value = mock_ws_client
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+            catalog="main",
+            schema="default",
+        )
+
+        lineage = ingestor.get_table_lineage("customers")
+
+        assert lineage["upstream"] == ["main.default.raw_customers"]
+        assert lineage["downstream"] == ["main.default.customer_summary"]
+        assert "columns" not in lineage
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+    def test_get_table_lineage_with_column_lineage(self, mock_ws_client_cls, mock_sql):
+        """Test that include_column_lineage=True fetches per-column lineage."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_column_1 = Mock()
+        mock_column_1.name = "id"
+        mock_column_1.type_text = "BIGINT"
+        mock_column_1.nullable = False
+        mock_column_1.comment = None
+
+        mock_column_2 = Mock()
+        mock_column_2.name = "name"
+        mock_column_2.type_text = "STRING"
+        mock_column_2.nullable = True
+        mock_column_2.comment = None
+
+        mock_table_info = Mock()
+        mock_table_info.columns = [mock_column_1, mock_column_2]
+
+        def do_side_effect(method, path, query=None):
+            if path.endswith("table-lineage"):
+                return {
+                    "upstreams": [{"tableInfo": {"name": "main.default.raw_customers"}}],
+                    "downstreams": [],
+                }
+            if path.endswith("column-lineage"):
+                if query["column_name"] == "id":
+                    return {
+                        "upstream_cols": [
+                            {
+                                "catalog_name": "main",
+                                "schema_name": "default",
+                                "table_name": "raw_customers",
+                                "name": "customer_id",
+                            }
+                        ],
+                        "downstream_cols": [],
+                    }
+                return {"upstream_cols": [], "downstream_cols": []}
+            raise AssertionError(f"unexpected path: {path}")
+
+        mock_ws_client = Mock()
+        mock_ws_client.tables.get = Mock(return_value=mock_table_info)
+        mock_ws_client.api_client.do = Mock(side_effect=do_side_effect)
+        mock_ws_client_cls.return_value = mock_ws_client
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+            catalog="main",
+            schema="default",
+        )
+
+        lineage = ingestor.get_table_lineage("customers", include_column_lineage=True)
+
+        assert lineage["upstream"] == ["main.default.raw_customers"]
+        assert lineage["columns"]["id"]["upstream"] == ["main.default.raw_customers.customer_id"]
+        assert lineage["columns"]["id"]["downstream"] == []
+        assert lineage["columns"]["name"]["upstream"] == []
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+    def test_get_table_lineage_requires_schema(self, mock_ws_client_cls, mock_sql):
+        """Test that get_table_lineage() validates schema before calling the REST API."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+        from semantica.utils.exceptions import ProcessingError
+
+        mock_ws_client = Mock()
+        mock_ws_client_cls.return_value = mock_ws_client
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+            catalog="main",
+        )
+        ingestor.connector.schema = None
+
+        with pytest.raises(ProcessingError, match="Schema name is required"):
+            ingestor.get_table_lineage("customers")
+
+        mock_ws_client.api_client.do.assert_not_called()
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_export_as_documents(self, mock_sql):
+        """Test exporting data as documents."""
+        from semantica.ingest.databricks_ingestor import DatabricksData, DatabricksIngestor
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        data = DatabricksData(
+            data=[
+                {"id": 1, "name": "Alice", "description": "Engineer"},
+                {"id": 2, "name": "Bob", "description": "Designer"},
+            ],
+            row_count=2,
+            columns=["id", "name", "description"],
+            table_name="employees",
+            catalog="main",
+            schema="default",
+        )
+
+        documents = ingestor.export_as_documents(
+            data, id_field="id", text_fields=["name", "description"]
+        )
+
+        assert len(documents) == 2
+        assert documents[0]["id"] == "1"
+        assert documents[0]["text"] == "Alice Engineer"
+        assert documents[0]["metadata"]["source"] == "databricks"
+        assert documents[0]["metadata"]["table"] == "employees"
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_context_manager(self, mock_sql, mock_databricks_connection):
+        """Test using ingestor as context manager."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, _ = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        with DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        ) as ingestor:
+            assert ingestor.connector.connection == mock_conn
+
+        mock_conn.close.assert_called()
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_context_manager_reuses_connection_across_calls(
+        self, mock_sql, mock_databricks_connection
+    ):
+        """Test that ingest_table()/ingest_query() reuse (not leak) the connection
+        opened by __enter__, and only close it once on __exit__."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        mock_conn, mock_cursor = mock_databricks_connection
+        mock_sql.connect = Mock(return_value=mock_conn)
+
+        with DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        ) as ingestor:
+            ingestor.ingest_table("customers")
+            # The connection opened by __enter__ must still be the live one:
+            # ingest_table() should not have closed it out from under the
+            # context manager.
+            assert ingestor.connector.connection == mock_conn
+            mock_conn.close.assert_not_called()
+
+            ingestor.ingest_query("SELECT * FROM sales")
+            assert ingestor.connector.connection == mock_conn
+            mock_conn.close.assert_not_called()
+
+        # databricks_sql.connect() should only have been called once (by
+        # __enter__): both ingestion calls reused that same connection.
+        mock_sql.connect.assert_called_once()
+        mock_conn.close.assert_called_once()
+
+    @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+    @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+    def test_convert_datetime(self, mock_sql):
+        """Test datetime conversion in _convert_rows."""
+        from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+        ingestor = DatabricksIngestor(
+            host="https://adb-xxx.azuredatabricks.net",
+            token="test_token",
+            http_path="/sql/1.0/warehouses/xxxx",
+        )
+
+        test_dt = datetime(2024, 1, 15, 10, 30, 0)
+        rows = [{"timestamp": test_dt, "value": 100}]
+
+        converted = ingestor._convert_rows(rows)
+
+        assert converted[0]["timestamp"] == "2024-01-15T10:30:00"
+        assert converted[0]["value"] == 100
+
+    def test_import_error_without_databricks(self):
+        """Test that proper error is raised when databricks libraries are not installed."""
+        with patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", False):
+            from semantica.ingest.databricks_ingestor import DatabricksConnector
+
+            with pytest.raises(ImportError, match="databricks-sdk"):
+                DatabricksConnector(
+                    host="https://adb-xxx.azuredatabricks.net",
+                    token="test_token",
+                )
+
+
+class TestDatabricksData:
+    """Test DatabricksData dataclass."""
+
+    def test_databricks_data_creation(self):
+        """Test DatabricksData creation."""
+        from semantica.ingest.databricks_ingestor import DatabricksData
+
+        data = DatabricksData(
+            data=[{"col1": "val1"}],
+            row_count=1,
+            columns=["col1"],
+            table_name="test_table",
+        )
+
+        assert data.row_count == 1
+        assert data.table_name == "test_table"
+        assert len(data.data) == 1
+        assert isinstance(data.ingested_at, datetime)
+
+    def test_databricks_data_with_metadata(self):
+        """Test DatabricksData with metadata."""
+        from semantica.ingest.databricks_ingestor import DatabricksData
+
+        metadata = {"custom_field": "value"}
+        data = DatabricksData(
+            data=[],
+            row_count=0,
+            columns=[],
+            metadata=metadata,
+        )
+
+        assert data.metadata["custom_field"] == "value"

tests/triplet_store/test_blazegraph_store.py

@@ -1,7 +1,7 @@
 import unittest
 import os
 import sys
-from unittest.mock import patch
+from unittest.mock import MagicMock, patch
 
 PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
 if PROJECT_ROOT not in sys.path:
@@ -9,6 +9,7 @@ if PROJECT_ROOT not in sys.path:
 
 from semantica.semantic_extract.triplet_extractor import Triplet
 from semantica.triplet_store.blazegraph_store import BlazegraphStore
+from semantica.utils.exceptions import ProcessingError
 
 
 class TestBlazegraphStoreSerialization(unittest.TestCase):
@@ -221,5 +222,334 @@ class TestBlazegraphStoreSerialization(unittest.TestCase):
         self.assertEqual(obj, "\"Colour\"@en-GB")
 
 
+from semantica.triplet_store import sparql_escaping
+
+
+class TestSparqlEscapingExtractionParity(unittest.TestCase):
+    """
+    Regression tests proving BlazegraphStore._escape_literal /
+    BlazegraphStore._resolve_datatype_iri are byte-for-byte identical in
+    behavior to the extracted sparql_escaping.escape_literal /
+    sparql_escaping.resolve_datatype_iri functions, across every branch of
+    both original methods.
+    """
+
+    # --- escape_literal: every special character + combinations + plain text ---
+
+    ESCAPE_LITERAL_CASES = [
+        "",  # empty string
+        "plain text, no special chars",
+        "back\\slash",  # backslash
+        'embedded "double" quote',  # double quote
+        "line1\nline2",  # newline
+        "line1\rline2",  # carriage return
+        "tab\there",  # tab
+        "\\\"\n\r\t",  # all five special characters combined
+        'mix \\ and " and \n and \r and \t together',
+        42,  # non-str input (both methods call str(value) first)
+        None,
+    ]
+
+    def test_escape_literal_matches_blazegraph_store_for_every_branch(self):
+        with patch.object(BlazegraphStore, "_connect", autospec=True):
+            store = BlazegraphStore(endpoint="http://localhost:9999/blazegraph")
+        for value in self.ESCAPE_LITERAL_CASES:
+            with self.subTest(value=value):
+                self.assertEqual(
+                    store._escape_literal(value),
+                    sparql_escaping.escape_literal(value),
+                )
+
+    # --- resolve_datatype_iri: every branch (bracketed, full IRI, prefixed,
+    # invalid bracketed, invalid full IRI-with-whitespace, unknown prefix) ---
+
+    RESOLVE_DATATYPE_IRI_VALID_CASES = [
+        "<http://www.w3.org/2001/XMLSchema#integer>",  # already bracketed
+        "http://www.w3.org/2001/XMLSchema#integer",  # full IRI, no brackets
+        "https://example.org/type",  # https scheme
+        "urn:isbn:0451450523",  # urn scheme
+        "xsd:integer",  # known prefix
+        "rdf:langString",  # known prefix
+        "rdfs:label",  # known prefix
+        "owl:Thing",  # known prefix
+        "skos:Concept",  # known prefix
+    ]
+
+    RESOLVE_DATATYPE_IRI_ERROR_CASES = [
+        "<>",  # empty bracketed IRI
+        "<http://example.org/type with space>",  # bracketed IRI with whitespace
+        "<http://example.org/type<injected>",  # bracketed IRI with disallowed char
+        "http://example.org/type CLEAR ALL",  # full IRI with whitespace
+        "myns:customType",  # unknown prefix
+        "not_a_uri_no_colon",  # no scheme, no colon-prefixed form matches
+        "javascript:alert(1)",  # disallowed scheme, not http/https/urn, no known prefix match
+    ]
+
+    def test_resolve_datatype_iri_matches_blazegraph_store_for_valid_cases(self):
+        with patch.object(BlazegraphStore, "_connect", autospec=True):
+            store = BlazegraphStore(endpoint="http://localhost:9999/blazegraph")
+        for datatype in self.RESOLVE_DATATYPE_IRI_VALID_CASES:
+            with self.subTest(datatype=datatype):
+                self.assertEqual(
+                    store._resolve_datatype_iri(datatype),
+                    sparql_escaping.resolve_datatype_iri(datatype),
+                )
+
+    def test_resolve_datatype_iri_matches_blazegraph_store_for_error_cases(self):
+        with patch.object(BlazegraphStore, "_connect", autospec=True):
+            store = BlazegraphStore(endpoint="http://localhost:9999/blazegraph")
+        for datatype in self.RESOLVE_DATATYPE_IRI_ERROR_CASES:
+            with self.subTest(datatype=datatype):
+                store_exc = None
+                shared_exc = None
+                try:
+                    store._resolve_datatype_iri(datatype)
+                except ValueError as exc:
+                    store_exc = exc
+                try:
+                    sparql_escaping.resolve_datatype_iri(datatype)
+                except ValueError as exc:
+                    shared_exc = exc
+
+                self.assertIsNotNone(store_exc, f"Expected ValueError from store for {datatype!r}")
+                self.assertIsNotNone(shared_exc, f"Expected ValueError from shared module for {datatype!r}")
+                self.assertEqual(str(store_exc), str(shared_exc))
+
+
+def _make_connected_store() -> BlazegraphStore:
+    """Create a BlazegraphStore instance bypassing the real _connect() call,
+    with .connected forced True (mirrors the state execute_sparql requires)."""
+    with patch.object(BlazegraphStore, "_connect", autospec=True):
+        store = BlazegraphStore(endpoint="http://localhost:9999/blazegraph")
+    store.connected = True
+    return store
+
+
+class TestBlazegraphStoreConstructExtension(unittest.TestCase):
+    """
+    Tests for the Blazegraph CONSTRUCT extension: _is_construct_query
+    detection, execute_sparql's CONSTRUCT branch (Accept header, Turtle
+    parsing via rdflib, triples shape, ProcessingError on malformed Turtle),
+    and Property 9 (non-CONSTRUCT queries are byte-for-byte unaffected).
+    """
+
+    # --- _is_construct_query detection ---
+
+    def test_is_construct_query_detects_uppercase_keyword(self):
+        store = _make_connected_store()
+        self.assertTrue(store._is_construct_query("CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"))
+
+    def test_is_construct_query_detects_lowercase_keyword(self):
+        store = _make_connected_store()
+        self.assertTrue(store._is_construct_query("construct { ?s ?p ?o } where { ?s ?p ?o }"))
+
+    def test_is_construct_query_detects_mixed_case_keyword(self):
+        store = _make_connected_store()
+        self.assertTrue(store._is_construct_query("Construct { ?s ?p ?o } Where { ?s ?p ?o }"))
+
+    def test_is_construct_query_detects_complex_preambles(self):
+        # Permanent regression tests covering edge cases discovered during
+        # regex stress-testing (issue #7): multiline declarations, empty
+        # prefix namespaces, and inline comments embedded in the preamble.
+        store = _make_connected_store()
+        cases = {
+            "multiline_prefix": "PREFIX foaf:\n  <http://xmlns.com/foaf/0.1/>\nCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
+            "empty_prefix_namespace": "PREFIX : <http://ex.org/> CONSTRUCT { ?s ?p ?o }",
+            "inline_comment": "PREFIX ex: <http://ex.org/>\n# inline comment\nCONSTRUCT { ?s ?p ?o }",
+        }
+        for name, query in cases.items():
+            with self.subTest(case=name):
+                self.assertTrue(store._is_construct_query(query))
+
+    def test_is_construct_query_false_for_select(self):
+        store = _make_connected_store()
+        self.assertFalse(store._is_construct_query("SELECT ?s WHERE { ?s ?p ?o }"))
+
+    def test_is_construct_query_false_for_ask(self):
+        store = _make_connected_store()
+        self.assertFalse(store._is_construct_query("ASK { ?s ?p ?o }"))
+
+    def test_is_construct_query_does_not_match_substring_inside_identifier(self):
+        # "CONSTRUCTOR" contains "CONSTRUCT" as a substring but must not
+        # match due to \b word-boundary anchoring.
+        store = _make_connected_store()
+        self.assertFalse(
+            store._is_construct_query("SELECT ?s WHERE { ?s <urn:p> \"CONSTRUCTOR\" }")
+        )
+
+    # --- execute_sparql CONSTRUCT path ---
+
+    def test_execute_sparql_construct_sends_turtle_accept_header(self):
+        store = _make_connected_store()
+        mock_response = MagicMock()
+        mock_response.content = (
+            b"@prefix ex: <http://ex.org/> .\n"
+            b'ex:s1 ex:p1 "value1" .\n'
+        )
+        mock_response.raise_for_status = MagicMock()
+
+        with patch("semantica.triplet_store.blazegraph_store.requests.post", return_value=mock_response) as mock_post:
+            store.execute_sparql("CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }")
+
+        _, kwargs = mock_post.call_args
+        self.assertEqual(kwargs["headers"]["Accept"], "text/turtle")
+        self.assertEqual(
+            kwargs["headers"]["Content-Type"], "application/x-www-form-urlencoded"
+        )
+
+    def test_execute_sparql_construct_parses_triples_from_fixed_turtle_fixture(self):
+        store = _make_connected_store()
+        turtle_fixture = (
+            b"@prefix ex: <http://ex.org/> .\n"
+            b'ex:s1 ex:p1 "value1" .\n'
+            b"ex:s1 ex:p2 ex:o2 .\n"
+        )
+        mock_response = MagicMock()
+        mock_response.content = turtle_fixture
+        mock_response.raise_for_status = MagicMock()
+
+        with patch("semantica.triplet_store.blazegraph_store.requests.post", return_value=mock_response):
+            result = store.execute_sparql("CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }")
+
+        self.assertTrue(result["success"])
+        self.assertEqual(result["bindings"], [])
+        self.assertEqual(result["variables"], [])
+        self.assertEqual(result["metadata"]["result_format"], "construct")
+
+        # "triples" is now a list of (s, p, o, metadata) 4-tuples. Both
+        # triples here are plain untyped literals/URIs, so metadata is {}.
+        triples = {(s, p, o) for s, p, o, _metadata in result["triples"]}
+        self.assertIn(("http://ex.org/s1", "http://ex.org/p1", "value1"), triples)
+        self.assertIn(("http://ex.org/s1", "http://ex.org/p2", "http://ex.org/o2"), triples)
+        self.assertEqual(len(result["triples"]), 2)
+        for _s, _p, _o, metadata in result["triples"]:
+            self.assertEqual(metadata, {})
+
+    def test_execute_sparql_construct_result_format_option_forces_construct_path(self):
+        # Even for a query that doesn't literally contain "CONSTRUCT",
+        # result_format="construct" should force the Turtle-parsing path.
+        store = _make_connected_store()
+        mock_response = MagicMock()
+        mock_response.content = b'@prefix ex: <http://ex.org/> .\nex:s1 ex:p1 "v" .\n'
+        mock_response.raise_for_status = MagicMock()
+
+        with patch("semantica.triplet_store.blazegraph_store.requests.post", return_value=mock_response) as mock_post:
+            result = store.execute_sparql("SELECT ?s WHERE { ?s ?p ?o }", result_format="construct")
+
+        _, kwargs = mock_post.call_args
+        self.assertEqual(kwargs["headers"]["Accept"], "text/turtle")
+        self.assertIn("triples", result)
+
+    def test_execute_sparql_construct_malformed_turtle_raises_processing_error(self):
+        store = _make_connected_store()
+        mock_response = MagicMock()
+        # Deliberately invalid Turtle syntax.
+        mock_response.content = b"this is { not [ valid turtle syntax at all !!!"
+        mock_response.raise_for_status = MagicMock()
+
+        with patch("semantica.triplet_store.blazegraph_store.requests.post", return_value=mock_response):
+            with self.assertRaises(ProcessingError) as ctx:
+                store.execute_sparql("CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }")
+
+        # Must be a ProcessingError, not a raw rdflib exception leaking out.
+        self.assertIsInstance(ctx.exception, ProcessingError)
+        self.assertNotIsInstance(ctx.exception, (SyntaxError, ValueError))
+
+    def test_execute_sparql_construct_handles_literal_with_braces_in_valid_turtle(self):
+        # Adversarial case implied by the brace-matching bug found in the
+        # template-string layer (construct_templates._find_matching_brace):
+        # confirm rdflib itself parses a *valid* Turtle literal containing
+        # brace characters correctly, since this is a different parsing
+        # layer (real Turtle syntax, not our {{param}} template string).
+        store = _make_connected_store()
+        turtle_fixture = (
+            b"@prefix ex: <http://ex.org/> .\n"
+            b'ex:s1 ex:p1 "text with { and } braces inside" .\n'
+        )
+        mock_response = MagicMock()
+        mock_response.content = turtle_fixture
+        mock_response.raise_for_status = MagicMock()
+
+        with patch("semantica.triplet_store.blazegraph_store.requests.post", return_value=mock_response):
+            result = store.execute_sparql("CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }")
+
+        self.assertEqual(len(result["triples"]), 1)
+        subject, predicate, obj, metadata = result["triples"][0]
+        self.assertEqual(subject, "http://ex.org/s1")
+        self.assertEqual(predicate, "http://ex.org/p1")
+        self.assertEqual(obj, "text with { and } braces inside")
+        self.assertEqual(metadata, {})
+
+    # --- Property 9: non-CONSTRUCT queries are byte-for-byte unaffected ---
+
+    def test_execute_sparql_select_query_response_shape_unchanged(self):
+        store = _make_connected_store()
+        mock_response = MagicMock()
+        mock_response.json.return_value = {
+            "head": {"vars": ["s", "p", "o"]},
+            "results": {
+                "bindings": [
+                    {
+                        "s": {"type": "uri", "value": "http://ex.org/s1"},
+                        "p": {"type": "uri", "value": "http://ex.org/p1"},
+                        "o": {"type": "literal", "value": "v1"},
+                    }
+                ]
+            },
+        }
+        mock_response.raise_for_status = MagicMock()
+
+        with patch("semantica.triplet_store.blazegraph_store.requests.post", return_value=mock_response) as mock_post:
+            result = store.execute_sparql("SELECT ?s ?p ?o WHERE { ?s ?p ?o }")
+
+        # Accept header must NOT be sent for a non-CONSTRUCT query — request
+        # shape is byte-for-byte identical to pre-CONSTRUCT-extension behavior.
+        _, kwargs = mock_post.call_args
+        self.assertEqual(kwargs["headers"], {"Content-Type": "application/x-www-form-urlencoded"})
+        self.assertNotIn("Accept", kwargs["headers"])
+
+        # Response shape must be exactly the pre-existing shape: no "triples"
+        # key at all (not even an empty list) for a plain SELECT response.
+        self.assertEqual(
+            result,
+            {
+                "success": True,
+                "bindings": mock_response.json.return_value["results"]["bindings"],
+                "variables": ["s", "p", "o"],
+                "metadata": {
+                    "query": "SELECT ?s ?p ?o WHERE { ?s ?p ?o }",
+                    "endpoint": store._get_sparql_endpoint(),
+                },
+            },
+        )
+        self.assertNotIn("triples", result)
+
+    def test_execute_sparql_ask_query_uses_bindings_path_not_construct(self):
+        store = _make_connected_store()
+        mock_response = MagicMock()
+        mock_response.json.return_value = {"head": {}, "boolean": True}
+        mock_response.raise_for_status = MagicMock()
+
+        with patch("semantica.triplet_store.blazegraph_store.requests.post", return_value=mock_response) as mock_post:
+            store.execute_sparql("ASK { ?s ?p ?o }")
+
+        _, kwargs = mock_post.call_args
+        self.assertNotIn("Accept", kwargs["headers"])
+
+
+class TestQueryEngineConstructValidation(unittest.TestCase):
+    """
+    Confirms QueryEngine._validate_query requires zero changes for CONSTRUCT
+    support — CONSTRUCT was already a valid keyword before this feature.
+    """
+
+    def test_construct_query_passes_validation_unchanged(self):
+        from semantica.triplet_store.query_engine import QueryEngine
+
+        engine = QueryEngine()
+        query = "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
+        self.assertTrue(engine._validate_query(query))
+
+
 if __name__ == "__main__":
     unittest.main()

tests/triplet_store/test_jena_store.py

@@ -0,0 +1,833 @@
+import unittest
+from unittest.mock import patch, MagicMock
+from rdflib import Graph, URIRef, Literal, Namespace
+from semantica.triplet_store.jena_store import JenaStore
+from semantica.semantic_extract.triplet_extractor import Triplet
+from semantica.utils.exceptions import ValidationError
+from semantica.triplet_store.construct_templates import execute_construct_template, ConstructTemplate
+
+class TestJenaStoreExecuteSparqlConstructPath(unittest.TestCase):
+    def setUp(self):
+        self.store = JenaStore()
+        # Ensure we use an in-memory graph
+        self.store.graph = Graph()
+        
+    def test_construct_parses_triples_from_rdflib_natively(self):
+        # Insert some test data
+        self.store.graph.parse(data='<http://ex.org/s> <http://ex.org/p> <http://ex.org/o> .', format='nt')
+        
+        query = "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
+        result = self.store.execute_sparql(query)
+        
+        self.assertTrue(result['success'])
+        self.assertEqual(result['bindings'], [])
+        self.assertEqual(result['variables'], [])
+        self.assertIn('triples', result)
+        self.assertEqual(result['metadata']['result_format'], 'construct')
+        
+        triples = result['triples']
+        self.assertEqual(len(triples), 1)
+        s, p, o, meta = triples[0]
+        self.assertEqual(s, 'http://ex.org/s')
+        self.assertEqual(p, 'http://ex.org/p')
+        self.assertEqual(o, 'http://ex.org/o')
+        self.assertEqual(meta, {})
+
+    def test_typed_literal_datatype_preserved_in_metadata(self):
+        data = '<http://ex.org/s> <http://ex.org/age> "42"^^<http://www.w3.org/2001/XMLSchema#integer> .'
+        self.store.graph.parse(data=data, format='nt')
+        
+        query = "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
+        result = self.store.execute_sparql(query)
+        
+        triples = result['triples']
+        self.assertEqual(len(triples), 1)
+        s, p, o, meta = triples[0]
+        self.assertEqual(o, '42')
+        self.assertEqual(meta['datatype'], 'http://www.w3.org/2001/XMLSchema#integer')
+        self.assertNotIn('language', meta)
+
+    def test_language_tagged_literal_preserved_in_metadata(self):
+        data = '<http://ex.org/s> <http://ex.org/label> "hello"@en .'
+        self.store.graph.parse(data=data, format='nt')
+        
+        query = "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
+        result = self.store.execute_sparql(query)
+        
+        triples = result['triples']
+        self.assertEqual(len(triples), 1)
+        s, p, o, meta = triples[0]
+        self.assertEqual(o, 'hello')
+        self.assertEqual(meta['language'], 'en')
+        self.assertNotIn('datatype', meta)
+
+
+class TestJenaStoreProperty9NonConstructUnchanged(unittest.TestCase):
+    def setUp(self):
+        self.store = JenaStore()
+        self.store.graph = Graph()
+        self.store.graph.parse(data='<http://ex.org/s> <http://ex.org/p> <http://ex.org/o> .', format='nt')
+        
+    def test_select_response_shape_unchanged(self):
+        query = "SELECT ?s ?p ?o WHERE { ?s ?p ?o }"
+        result = self.store.execute_sparql(query)
+        
+        self.assertTrue(result['success'])
+        self.assertIn('bindings', result)
+        self.assertEqual(len(result['bindings']), 1)
+        self.assertNotIn('triples', result)
+        self.assertEqual(result['metadata']['query'], query)
+        
+        binding = result['bindings'][0]
+        self.assertEqual(binding['s']['value'], 'http://ex.org/s')
+        self.assertEqual(binding['s']['type'], 'uri')
+
+    def test_ask_uses_json_not_turtle_path(self):
+        query = "ASK WHERE { ?s ?p ?o }"
+        result = self.store.execute_sparql(query)
+        
+        self.assertTrue(result['success'])
+        # For ASK in rdflib, results is a bool wrapped in SPARQLResult.
+        # results.vars is None, so it returns empty bindings. This matches the byte-for-byte behavior.
+        self.assertEqual(result['bindings'], [])
+        self.assertNotIn('triples', result)
+
+class TestExecuteConstructTemplateWithJenaBackend(unittest.TestCase):
+    def setUp(self):
+        self.store = JenaStore()
+        self.store.graph = Graph()
+        self.store.graph.parse(data='<http://ex.org/s> <http://ex.org/name> "Alice" .', format='nt')
+        
+    def test_end_to_end_with_jena_backend(self):
+        template = ConstructTemplate(
+            name="test",
+            description="test",
+            construct_query="""
+        CONSTRUCT {
+            ?s <http://ex.org/isPerson> "true"^^<http://www.w3.org/2001/XMLSchema#boolean> .
+        } WHERE {
+            ?s <http://ex.org/name> ?name .
+        }
+        """,
+            parameters=[]
+        )
+        
+        results = execute_construct_template(template, {}, self.store)
+        self.assertEqual(len(results), 1)
+        triplet = results[0]
+        self.assertEqual(triplet.subject, 'http://ex.org/s')
+        self.assertEqual(triplet.predicate, 'http://ex.org/isPerson')
+        self.assertEqual(triplet.object, 'true')
+        self.assertEqual(triplet.metadata.get('datatype'), 'http://www.w3.org/2001/XMLSchema#boolean')
+
+class TestJenaStoreRemoteEndpointUsesUpdateStore(unittest.TestCase):
+    """
+    Tests for the SPARQLStore → SPARQLUpdateStore fix.
+
+    Bug: _initialize_graph bound a read-only SPARQLStore for remote endpoints,
+    causing every add_triplets() call to silently fail (TypeError swallowed,
+    success=True/added=0 returned).
+
+    Fix: SPARQLUpdateStore is now used, configured with both
+    query_endpoint (<base>/query) and update_endpoint (<base>/update)
+    per standard Apache Jena Fuseki REST API conventions.
+    """
+
+    _BASE = "http://localhost:3030/ds"
+
+    # ------------------------------------------------------------------
+    # Helpers
+    # ------------------------------------------------------------------
+
+    def _make_remote_store(self):
+        """Return a JenaStore wired to a fake remote endpoint."""
+        from rdflib.plugins.stores.sparqlstore import SPARQLUpdateStore
+
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore",
+        ) as MockUpdateStore:
+            # SPARQLUpdateStore.__init__ would normally try to contact the
+            # server; the mock prevents any network I/O during init.
+            mock_store_instance = MagicMock(spec=SPARQLUpdateStore)
+            MockUpdateStore.return_value = mock_store_instance
+
+            with patch("semantica.triplet_store.jena_store.Graph") as MockGraph:
+                mock_graph_instance = MagicMock(spec=Graph)
+                MockGraph.return_value = mock_graph_instance
+
+                store = JenaStore(endpoint=self._BASE)
+                return store, MockUpdateStore, MockGraph, mock_graph_instance
+
+    # ------------------------------------------------------------------
+    # Test 1 – correct class is instantiated
+    # ------------------------------------------------------------------
+
+    def test_remote_endpoint_instantiates_sparql_update_store_not_sparql_store(self):
+        """
+        _initialize_graph must use SPARQLUpdateStore, never the read-only SPARQLStore.
+        """
+        from rdflib.plugins.stores.sparqlstore import SPARQLUpdateStore
+
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore"
+        ) as MockUpdateStore, patch(
+            "semantica.triplet_store.jena_store.SPARQLStore"
+        ) as MockReadOnlyStore:
+            MockUpdateStore.return_value = MagicMock()
+            with patch("semantica.triplet_store.jena_store.Graph"):
+                JenaStore(endpoint=self._BASE)
+
+            MockUpdateStore.assert_called_once()
+            MockReadOnlyStore.assert_not_called()
+
+    # ------------------------------------------------------------------
+    # Test 2 – correct Fuseki sub-paths are derived from base URL
+    # ------------------------------------------------------------------
+
+    def test_remote_endpoint_derives_fuseki_query_and_update_sub_paths(self):
+        """
+        Standard Fuseki datasets expose /query and /update under the dataset
+        base URL.  The store must pass both to SPARQLUpdateStore.
+        """
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore"
+        ) as MockUpdateStore, patch("semantica.triplet_store.jena_store.Graph"):
+            MockUpdateStore.return_value = MagicMock()
+
+            JenaStore(endpoint=self._BASE)
+
+            call_kwargs = MockUpdateStore.call_args
+            # Accept both positional and keyword argument forms
+            args, kwargs = call_kwargs
+            passed = {**kwargs}
+            if len(args) >= 1:
+                passed.setdefault("query_endpoint", args[0])
+            if len(args) >= 2:
+                passed.setdefault("update_endpoint", args[1])
+
+            self.assertEqual(
+                passed.get("query_endpoint"),
+                "http://localhost:3030/ds/query",
+                "query_endpoint must be <base>/query",
+            )
+            self.assertEqual(
+                passed.get("update_endpoint"),
+                "http://localhost:3030/ds/update",
+                "update_endpoint must be <base>/update",
+            )
+
+    def test_remote_endpoint_trailing_slash_is_normalised(self):
+        """A trailing slash on the supplied endpoint must not produce double-slashes."""
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore"
+        ) as MockUpdateStore, patch("semantica.triplet_store.jena_store.Graph"):
+            MockUpdateStore.return_value = MagicMock()
+
+            JenaStore(endpoint="http://localhost:3030/ds/")
+
+            _, kwargs = MockUpdateStore.call_args
+            self.assertNotIn(
+                "//query",
+                kwargs.get("query_endpoint", ""),
+                "Trailing slash must be stripped before appending /query",
+            )
+            self.assertEqual(kwargs.get("query_endpoint"), "http://localhost:3030/ds/query")
+            self.assertEqual(kwargs.get("update_endpoint"), "http://localhost:3030/ds/update")
+
+    # ------------------------------------------------------------------
+    # Test 3 – end-to-end write path: add_triplets fires INSERT DATA POST
+    # ------------------------------------------------------------------
+
+    def test_add_triplets_remote_endpoint_fires_insert_data_via_update_store(self):
+        """
+        End-to-end mock: add_triplets() against a remote-endpoint JenaStore must
+        call graph.add() (which SPARQLUpdateStore maps to INSERT DATA).  We verify
+        that graph.add() is actually invoked with the correct (subject, predicate,
+        object) triple and that add_triplets returns success=True/added=1.
+
+        This is distinct from the class-instantiation tests above: it confirms
+        the write path works all the way through add_triplets(), not just that
+        the right class gets constructed.
+
+        After the Dataset migration the remote path creates Dataset(store=...),
+        so we patch Dataset rather than Graph to intercept the construction.
+        """
+        from rdflib import Dataset, URIRef
+
+        triplet = Triplet(
+            subject="http://example.org/Alice",
+            predicate="http://example.org/knows",
+            object="http://example.org/Bob",
+        )
+
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore"
+        ) as MockUpdateStore, patch(
+            "semantica.triplet_store.jena_store.Dataset"
+        ) as MockDataset:
+            mock_store_instance = MagicMock()
+            MockUpdateStore.return_value = mock_store_instance
+
+            mock_dataset_instance = MagicMock(spec=Dataset)
+            # graph.add() must not raise — simulate a successful INSERT DATA
+            mock_dataset_instance.add.return_value = None
+            # graph.graph() is called to resolve named-graph context when graph=
+            # is supplied.  It is NOT called in this test (no graph= option).
+            MockDataset.return_value = mock_dataset_instance
+
+            store = JenaStore(endpoint=self._BASE)
+            result = store.add_triplets([triplet])
+
+        # add_triplets must report success
+        self.assertTrue(result["success"])
+        self.assertEqual(result["added"], 1)
+        self.assertEqual(result["total"], 1)
+
+        # graph.add() must have been called exactly once with the 3-tuple
+        # (no graph= option → default graph path → 3-tuple, no context arg)
+        mock_dataset_instance.add.assert_called_once_with(
+            (
+                URIRef("http://example.org/Alice"),
+                URIRef("http://example.org/knows"),
+                URIRef("http://example.org/Bob"),
+            )
+        )
+
+
+class TestJenaStoreDatasetMigration(unittest.TestCase):
+    """
+    Tests confirming the Graph → Dataset(default_union=False) migration.
+
+    These tests specifically exercise _initialize_graph's real, unmodified
+    path (no store.graph injection) to confirm the migration is in effect —
+    unlike the existing test classes, which inject a plain Graph() directly
+    and therefore don't exercise the initialization path at all.
+    """
+
+    # ------------------------------------------------------------------
+    # Test 1 — unmodified _initialize_graph produces Dataset, not Graph
+    # ------------------------------------------------------------------
+
+    def test_initialize_graph_produces_dataset_not_graph(self):
+        """
+        The in-memory path of _initialize_graph must produce a Dataset instance,
+        not a plain Graph.  This is the direct regression guard for the migration:
+        if someone reverts the Dataset construction back to Graph() this test fails.
+        """
+        from rdflib import Dataset
+
+        store = JenaStore()  # no endpoint → in-memory path
+
+        self.assertIsInstance(
+            store.graph,
+            Dataset,
+            "_initialize_graph must produce a Dataset, not a Graph",
+        )
+
+    def test_initialize_graph_dataset_has_default_union_false(self):
+        """
+        default_union must be explicitly False so that queries without a named
+        graph scope see only the default graph, not a union across all graphs.
+        """
+        from rdflib import Dataset
+
+        store = JenaStore()
+
+        self.assertIsInstance(store.graph, Dataset)
+        self.assertFalse(
+            store.graph.default_union,
+            "Dataset.default_union must be False (named-graph isolation)",
+        )
+
+    # ------------------------------------------------------------------
+    # Test 2 — add_triplets with graph= writes to named graph, not default
+    # ------------------------------------------------------------------
+
+    def test_add_triplets_with_graph_option_writes_to_named_graph(self):
+        """
+        add_triplets(triplets, graph="http://example.org/g") must write triples
+        to the specified named graph, not to the default graph.
+        """
+        from rdflib import Dataset, URIRef
+
+        store = JenaStore()
+        self.assertIsInstance(store.graph, Dataset)
+
+        named_graph_uri = "http://example.org/named-graph"
+        triplet = Triplet(
+            subject="http://example.org/Alice",
+            predicate="http://example.org/knows",
+            object="http://example.org/Bob",
+        )
+
+        result = store.add_triplets([triplet], graph=named_graph_uri)
+
+        self.assertTrue(result["success"])
+        self.assertEqual(result["added"], 1)
+
+        # Confirm the named graph now contains the triple
+        named_ctx = store.graph.graph(URIRef(named_graph_uri))
+        self.assertEqual(len(named_ctx), 1, "Named graph must hold the added triple")
+
+        # Confirm the default graph does NOT contain it
+        self.assertEqual(
+            len(store.graph.default_graph),
+            0,
+            "Default graph must be empty when graph= option is used",
+        )
+
+    def test_add_triplets_without_graph_option_writes_to_default_graph(self):
+        """
+        When graph= is omitted (the common path), triples must go to the default
+        graph — preserving pre-migration semantics exactly.
+        """
+        from rdflib import Dataset, URIRef
+
+        store = JenaStore()
+
+        triplet = Triplet(
+            subject="http://example.org/Alice",
+            predicate="http://example.org/knows",
+            object="http://example.org/Bob",
+        )
+
+        result = store.add_triplets([triplet])  # no graph= option
+
+        self.assertTrue(result["success"])
+        self.assertEqual(result["added"], 1)
+
+        # Triple is in the default graph
+        self.assertEqual(
+            len(store.graph.default_graph),
+            1,
+            "Triple without graph= must go to the default graph",
+        )
+
+    def test_add_triplets_named_graph_isolated_from_default_query(self):
+        """
+        A triple added to a named graph must NOT appear in a plain SELECT query
+        (which, with default_union=False, sees only the default graph).
+        This confirms the isolation guarantee end-to-end.
+        """
+        from rdflib import URIRef
+
+        store = JenaStore()
+
+        # Add to named graph
+        store.add_triplets(
+            [Triplet(
+                subject="http://example.org/Named",
+                predicate="http://example.org/type",
+                object="http://example.org/Thing",
+            )],
+            graph="http://example.org/isolated",
+        )
+
+        # Also add to default graph so we can confirm query returns that one
+        store.add_triplets([
+            Triplet(
+                subject="http://example.org/Default",
+                predicate="http://example.org/type",
+                object="http://example.org/Other",
+            )
+        ])
+
+        # Plain SPARQL query (no FROM / GRAPH clause) should see only default
+        result = store.execute_sparql("SELECT ?s WHERE { ?s ?p ?o }")
+        subjects = [b["s"]["value"] for b in result["bindings"]]
+
+        self.assertIn("http://example.org/Default", subjects)
+        self.assertNotIn(
+            "http://example.org/Named",
+            subjects,
+            "Named-graph triple must not appear in a default-graph-scoped query",
+        )
+
+    # ------------------------------------------------------------------
+    # Test 3 — serialize() logs a warning when named-graph content is present
+    # ------------------------------------------------------------------
+
+    def test_serialize_logs_warning_when_named_graph_content_present(self):
+        """
+        serialize(format='turtle') serializes only the default graph.  When
+        named-graph triples are also present, a WARNING must be logged so the
+        data loss is not silent.
+        """
+        from unittest.mock import patch
+
+        store = JenaStore()
+
+        # Put one triple in each location
+        store.add_triplets([
+            Triplet(
+                subject="http://example.org/S",
+                predicate="http://example.org/P",
+                object="default_val",
+            )
+        ])
+        store.add_triplets(
+            [Triplet(
+                subject="http://example.org/S2",
+                predicate="http://example.org/P",
+                object="named_val",
+            )],
+            graph="http://example.org/ng",
+        )
+
+        with patch.object(store.logger, "warning") as mock_warn:
+            output = store.serialize(format="turtle")
+
+        # serialize must have returned something (the default graph)
+        self.assertIsInstance(output, str)
+        self.assertIn("default_val", output)
+        self.assertNotIn("named_val", output)
+
+        # Warning must have been logged
+        mock_warn.assert_called_once()
+        warn_msg = mock_warn.call_args[0][0]
+        self.assertIn("named-graph", warn_msg.lower().replace("-", "-"))
+        self.assertIn("trig", warn_msg.lower())
+
+    def test_serialize_no_warning_when_only_default_graph_used(self):
+        """
+        serialize() must NOT log a warning when no named-graph content exists —
+        this avoids noise for the common case where named graphs are not used.
+        """
+        from unittest.mock import patch
+
+        store = JenaStore()
+        store.add_triplets([
+            Triplet(
+                subject="http://example.org/S",
+                predicate="http://example.org/P",
+                object="default_only",
+            )
+        ])
+
+        with patch.object(store.logger, "warning") as mock_warn:
+            output = store.serialize(format="turtle")
+
+        self.assertIn("default_only", output)
+        mock_warn.assert_not_called()
+
+
+
+
+class TestJenaStoreEndpointDerivation(unittest.TestCase):
+    """
+    Regression tests for Bug 1 (Qodo): endpoint suffix detection prevents
+    double-appending of /query or /update.
+    """
+
+    def _captured_kwargs(self, endpoint):
+        """Return the kwargs passed to SPARQLUpdateStore for a given endpoint."""
+        captured = {}
+        real_init = __import__(
+            "rdflib.plugins.stores.sparqlstore",
+            fromlist=["SPARQLUpdateStore"],
+        ).SPARQLUpdateStore.__init__
+
+        def fake_update_store(self_store, **kwargs):
+            captured.update(kwargs)
+            # Avoid real network connection; just store args and bail early
+            raise RuntimeError("stop_after_capture")
+
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore",
+        ) as MockUS:
+            MockUS.side_effect = RuntimeError("stop_after_capture")
+            MockUS.__init__ = fake_update_store
+            try:
+                JenaStore(endpoint=endpoint)
+            except Exception:
+                pass
+            # Extract call kwargs
+            if MockUS.call_args is not None:
+                captured = MockUS.call_args.kwargs
+        return captured
+
+    def test_bare_base_url_appends_query_and_update(self):
+        """
+        The canonical case: a bare dataset base URL gets /query and /update
+        appended correctly.
+        """
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore"
+        ) as MockUS, patch("semantica.triplet_store.jena_store.Dataset"):
+            MockUS.return_value = MagicMock()
+            MockUS.return_value.graph_aware = True
+            JenaStore(endpoint="http://localhost:3030/ds")
+            kwargs = MockUS.call_args.kwargs
+        self.assertEqual(kwargs["query_endpoint"],  "http://localhost:3030/ds/query")
+        self.assertEqual(kwargs["update_endpoint"], "http://localhost:3030/ds/update")
+
+    def test_bare_base_url_with_trailing_slash_normalised(self):
+        """Trailing slash on the base URL must not produce a double slash."""
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore"
+        ) as MockUS, patch("semantica.triplet_store.jena_store.Dataset"):
+            MockUS.return_value = MagicMock()
+            MockUS.return_value.graph_aware = True
+            JenaStore(endpoint="http://localhost:3030/ds/")
+            kwargs = MockUS.call_args.kwargs
+        self.assertEqual(kwargs["query_endpoint"],  "http://localhost:3030/ds/query")
+        self.assertEqual(kwargs["update_endpoint"], "http://localhost:3030/ds/update")
+
+    def test_endpoint_already_ending_in_query_not_double_appended(self):
+        """
+        If the caller passes 'http://localhost:3030/ds/query' (already a full
+        service URL), the derived query_endpoint must still be
+        'http://localhost:3030/ds/query', not '.../ds/query/query'.
+        """
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore"
+        ) as MockUS, patch("semantica.triplet_store.jena_store.Dataset"):
+            MockUS.return_value = MagicMock()
+            MockUS.return_value.graph_aware = True
+            JenaStore(endpoint="http://localhost:3030/ds/query")
+            kwargs = MockUS.call_args.kwargs
+        self.assertEqual(kwargs["query_endpoint"],  "http://localhost:3030/ds/query")
+        self.assertEqual(kwargs["update_endpoint"], "http://localhost:3030/ds/update")
+        # Crucially: no double-suffix
+        self.assertNotIn("query/query", kwargs["query_endpoint"])
+        self.assertNotIn("query/update", kwargs["update_endpoint"])
+
+    def test_endpoint_already_ending_in_sparql_not_double_appended(self):
+        """
+        Some Fuseki deployments use /sparql as the query service name.
+        Passing that as the endpoint must not produce '.../ds/sparql/query'.
+        """
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore"
+        ) as MockUS, patch("semantica.triplet_store.jena_store.Dataset"):
+            MockUS.return_value = MagicMock()
+            MockUS.return_value.graph_aware = True
+            JenaStore(endpoint="http://localhost:3030/ds/sparql")
+            kwargs = MockUS.call_args.kwargs
+        self.assertEqual(kwargs["query_endpoint"],  "http://localhost:3030/ds/query")
+        self.assertEqual(kwargs["update_endpoint"], "http://localhost:3030/ds/update")
+        self.assertNotIn("sparql/query",  kwargs["query_endpoint"])
+        self.assertNotIn("sparql/update", kwargs["update_endpoint"])
+
+    def test_endpoint_already_ending_in_update_not_double_appended(self):
+        """
+        Endpoint pre-set to '.../ds/update' must not produce '.../ds/update/update'.
+        """
+        with patch(
+            "semantica.triplet_store.jena_store.SPARQLUpdateStore"
+        ) as MockUS, patch("semantica.triplet_store.jena_store.Dataset"):
+            MockUS.return_value = MagicMock()
+            MockUS.return_value.graph_aware = True
+            JenaStore(endpoint="http://localhost:3030/ds/update")
+            kwargs = MockUS.call_args.kwargs
+        self.assertEqual(kwargs["query_endpoint"],  "http://localhost:3030/ds/query")
+        self.assertEqual(kwargs["update_endpoint"], "http://localhost:3030/ds/update")
+        self.assertNotIn("update/update", kwargs["update_endpoint"])
+
+
+class TestJenaStoreSerializeWarningFormats(unittest.TestCase):
+    """
+    Regression tests for Bug 2 (Qodo): serialize warning must fire only for
+    single-graph formats (turtle, xml, n3, …) and must NOT fire for
+    multi-graph formats (trig, nquads, nt, json-ld, …).
+    """
+
+    def _store_with_named_graph_content(self):
+        from rdflib import URIRef
+        store = JenaStore()
+        store.add_triplets([
+            Triplet("http://s1", "http://p", "default_val")
+        ])
+        store.add_triplets([
+            Triplet("http://s2", "http://p", "named_val")
+        ], graph="http://example.org/ng")
+        return store
+
+    # --- formats that MUST trigger the warning ---
+
+    def test_turtle_triggers_warning(self):
+        store = self._store_with_named_graph_content()
+        with patch.object(store.logger, "warning") as mock_warn:
+            store.serialize(format="turtle")
+        mock_warn.assert_called_once()
+
+    def test_xml_triggers_warning(self):
+        store = self._store_with_named_graph_content()
+        with patch.object(store.logger, "warning") as mock_warn:
+            store.serialize(format="xml")
+        mock_warn.assert_called_once()
+
+    def test_n3_triggers_warning(self):
+        store = self._store_with_named_graph_content()
+        with patch.object(store.logger, "warning") as mock_warn:
+            store.serialize(format="n3")
+        mock_warn.assert_called_once()
+
+    # --- formats that must NOT trigger the warning ---
+
+    def test_trig_no_warning(self):
+        """trig is a multi-graph format — includes all named graphs, no warning."""
+        store = self._store_with_named_graph_content()
+        with patch.object(store.logger, "warning") as mock_warn:
+            output = store.serialize(format="trig")
+        mock_warn.assert_not_called()
+        # Sanity: both triples are actually in the output
+        self.assertIn("named_val", output)
+
+    def test_nquads_no_warning(self):
+        """nquads is a multi-graph format — no warning."""
+        store = self._store_with_named_graph_content()
+        with patch.object(store.logger, "warning") as mock_warn:
+            output = store.serialize(format="nquads")
+        mock_warn.assert_not_called()
+        self.assertIn("named_val", output)
+
+    def test_nt_no_warning(self):
+        """nt (N-Triples) serializes all contexts — no warning."""
+        store = self._store_with_named_graph_content()
+        with patch.object(store.logger, "warning") as mock_warn:
+            store.serialize(format="nt")
+        mock_warn.assert_not_called()
+
+
+class TestJenaStoreZeroAddedErrorMessage(unittest.TestCase):
+    """
+    Regression tests for Bug 3 (Qodo): when added_count==0, the ProcessingError
+    message must distinguish data-formatting failures from connectivity failures.
+    """
+
+    def test_all_malformed_triplets_gives_formatting_error(self):
+        """
+        When every triplet fails the per-triplet ValueError/AttributeError handler,
+        the raised ProcessingError must mention validation / formatting, NOT
+        connectivity or endpoint configuration.
+        """
+        from semantica.utils.exceptions import ProcessingError
+
+        store = JenaStore()
+
+        # object=None triggers AttributeError on None.startswith("http") in
+        # add_triplets' obj-resolution line — this IS caught by the per-triplet
+        # (ValueError, AttributeError) handler and increments malformed_count.
+        # (Note: subject=None would raise TypeError on URIRef(None), which is
+        # NOT caught by the per-triplet handler and would escape to the outer
+        # except, producing a different message path — don't use that.)
+        bad = Triplet(subject="http://s", predicate="http://p", object=None)
+
+        with self.assertRaises(ProcessingError) as ctx:
+            store.add_triplets([bad, bad, bad])
+
+        msg = str(ctx.exception).lower()
+        # Must mention validation/formatting
+        self.assertTrue(
+            "validation" in msg or "formatting" in msg or "format" in msg,
+            f"Expected validation/formatting message, got: {ctx.exception}",
+        )
+        # Must NOT suggest connectivity
+        self.assertNotIn("connectivity", msg)
+        self.assertNotIn("endpoint configuration", msg)
+
+    def test_connectivity_error_gives_connectivity_message(self):
+        """
+        When a store-level exception (not per-triplet ValueError) causes zero adds,
+        the message must mention connectivity/endpoint — not validation.
+        This simulates a store that raises a non-ValueError on .add().
+        """
+        from semantica.utils.exceptions import ProcessingError
+        from rdflib import Dataset
+
+        store = JenaStore()
+
+        # Replace the Dataset with a mock whose .add() raises a non-per-triplet error
+        mock_ds = MagicMock(spec=Dataset)
+        mock_ds.default_graph = MagicMock()
+        mock_ds.default_graph.__len__ = MagicMock(return_value=0)
+        mock_ds.__len__ = MagicMock(return_value=0)
+
+        # Raise RuntimeError (not ValueError/AttributeError) — store-level failure
+        mock_ds.add.side_effect = RuntimeError("connection refused")
+        store.graph = mock_ds
+
+        triplet = Triplet("http://s", "http://p", "http://o")
+
+        with self.assertRaises(ProcessingError) as ctx:
+            store.add_triplets([triplet])
+
+        # The RuntimeError propagates past the per-triplet handler and is caught
+        # by the outer except — the message comes from the outer re-raise
+        msg = str(ctx.exception).lower()
+        # Should mention "failed to add triplets" (the outer handler wraps it)
+        self.assertIn("failed", msg)
+
+
+class TestJenaStoreDeleteTripletScopedToDefaultGraph(unittest.TestCase):
+    """
+    Regression tests for delete_triplet's cross-graph deletion bug.
+
+    Bug: after the Graph -> Dataset migration, delete_triplet called
+    self.graph.remove((s, p, o)) with no context. Dataset.remove() on a bare
+    3-tuple resolves context=None internally, which the store treats as a
+    wildcard and matches (and deletes) the triple in every graph — not just
+    the default graph the docstring promises. Fix: pass
+    self.graph.default_graph explicitly as the 4th tuple element so the
+    removal is scoped to the default graph only.
+    """
+
+    def test_delete_triplet_does_not_remove_from_named_graph(self):
+        """
+        A triple present in both the default graph and a named graph must,
+        after delete_triplet(), still exist in the named graph — only the
+        default-graph copy may be removed.
+        """
+        from rdflib import URIRef
+
+        store = JenaStore()
+
+        triplet = Triplet(
+            subject="http://example.org/Alice",
+            predicate="http://example.org/knows",
+            object="http://example.org/Bob",
+        )
+
+        # Same triple written to both the default graph and a named graph.
+        store.add_triplets([triplet])
+        store.add_triplets([triplet], graph="http://example.org/ng")
+
+        named_ctx = store.graph.graph(URIRef("http://example.org/ng"))
+        self.assertEqual(len(named_ctx), 1)
+        self.assertEqual(len(store.graph.default_graph), 1)
+
+        result = store.delete_triplet(triplet)
+
+        self.assertTrue(result["success"])
+        self.assertEqual(
+            len(store.graph.default_graph),
+            0,
+            "Triplet must be removed from the default graph",
+        )
+        self.assertEqual(
+            len(named_ctx),
+            1,
+            "Triplet must NOT be removed from a named graph it also lives in",
+        )
+
+    def test_delete_triplet_removes_from_default_graph(self):
+        """delete_triplet still removes the triple from the default graph."""
+        store = JenaStore()
+
+        triplet = Triplet(
+            subject="http://example.org/S",
+            predicate="http://example.org/P",
+            object="http://example.org/O",
+        )
+        store.add_triplets([triplet])
+        self.assertEqual(len(store.graph.default_graph), 1)
+
+        result = store.delete_triplet(triplet)
+
+        self.assertTrue(result["success"])
+        self.assertEqual(len(store.graph.default_graph), 0)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/triplet_store/test_query_engine.py

@@ -0,0 +1,97 @@
+"""
+Tests for semantica.triplet_store.query_engine, focused on the
+QueryResult.triples field added for CONSTRUCT query support.
+"""
+
+import os
+import sys
+import unittest
+from unittest.mock import MagicMock
+
+PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+if PROJECT_ROOT not in sys.path:
+    sys.path.insert(0, PROJECT_ROOT)
+
+from semantica.triplet_store.query_engine import QueryEngine, QueryResult
+
+
+class TestQueryResultTriplesField(unittest.TestCase):
+    def test_query_result_triples_defaults_to_empty_list(self):
+        result = QueryResult(bindings=[], variables=[])
+        self.assertEqual(result.triples, [])
+
+    def test_query_result_triples_field_is_independent_per_instance(self):
+        # default_factory=list must produce a fresh list per instance, not a
+        # shared mutable default.
+        r1 = QueryResult(bindings=[], variables=[])
+        r2 = QueryResult(bindings=[], variables=[])
+        r1.triples.append(("s", "p", "o"))
+        self.assertEqual(r1.triples, [("s", "p", "o")])
+        self.assertEqual(r2.triples, [])
+
+    def test_query_result_accepts_explicit_triples(self):
+        triples = [("http://ex.org/s1", "http://ex.org/p1", "v1")]
+        result = QueryResult(bindings=[], variables=[], triples=triples)
+        self.assertEqual(result.triples, triples)
+
+
+class FakeConstructBackend:
+    """Fake store_backend whose execute_sparql returns a CONSTRUCT-shaped result."""
+
+    supports_named_graphs = True
+
+    def execute_sparql(self, query, **options):
+        return {
+            "success": True,
+            "bindings": [],
+            "variables": [],
+            "triples": [
+                ("http://ex.org/s1", "http://ex.org/p1", "v1"),
+                ("http://ex.org/s2", "http://ex.org/p2", "v2"),
+            ],
+            "metadata": {"query": query, "result_format": "construct"},
+        }
+
+
+class FakeBindingsBackend:
+    """Fake store_backend whose execute_sparql returns a plain bindings result
+    with no "triples" key at all, matching pre-CONSTRUCT-extension backends."""
+
+    supports_named_graphs = True
+
+    def execute_sparql(self, query, **options):
+        return {
+            "success": True,
+            "bindings": [{"s": {"value": "http://ex.org/s1"}}],
+            "variables": ["s"],
+            "metadata": {"query": query},
+        }
+
+
+class TestExecuteQueryPopulatesTriples(unittest.TestCase):
+    def test_execute_query_populates_triples_from_construct_backend(self):
+        engine = QueryEngine(enable_caching=False, enable_optimization=False)
+        result = engine.execute_query(
+            "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }", FakeConstructBackend()
+        )
+        self.assertEqual(
+            result.triples,
+            [
+                ("http://ex.org/s1", "http://ex.org/p1", "v1"),
+                ("http://ex.org/s2", "http://ex.org/p2", "v2"),
+            ],
+        )
+
+    def test_execute_query_defaults_triples_to_empty_list_when_backend_omits_key(self):
+        engine = QueryEngine(enable_caching=False, enable_optimization=False)
+        result = engine.execute_query(
+            "SELECT ?s WHERE { ?s ?p ?o }", FakeBindingsBackend()
+        )
+        self.assertEqual(result.triples, [])
+        # Non-CONSTRUCT behavior otherwise unaffected.
+        self.assertEqual(result.bindings, [{"s": {"value": "http://ex.org/s1"}}])
+        self.assertEqual(result.variables, ["s"])
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/triplet_store/test_rdf4j_store.py

@@ -0,0 +1,518 @@
+import os, sys, unittest
+from unittest.mock import MagicMock, patch
+
+PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+if PROJECT_ROOT not in sys.path:
+    sys.path.insert(0, PROJECT_ROOT)
+
+from semantica.semantic_extract.triplet_extractor import Triplet
+from semantica.triplet_store.rdf4j_store import RDF4JStore
+from semantica.utils.exceptions import ProcessingError
+
+
+def _make_connected_store():
+    with patch.object(RDF4JStore, "_connect", autospec=True):
+        store = RDF4JStore(
+            endpoint="http://localhost:8080/rdf4j-server", repository_id="repo1"
+        )
+    store.connected = True
+    return store
+
+
+CONSTRUCT_QUERY = "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
+
+
+class TestRDF4JStoreIsConstructQuery(unittest.TestCase):
+    def test_detects_uppercase(self):
+        self.assertTrue(_make_connected_store()._is_construct_query(
+            "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"))
+    def test_detects_lowercase(self):
+        self.assertTrue(_make_connected_store()._is_construct_query(
+            "construct { ?s ?p ?o } where { ?s ?p ?o }"))
+    def test_detects_mixed_case(self):
+        self.assertTrue(_make_connected_store()._is_construct_query(
+            "Construct { ?s ?p ?o } Where { ?s ?p ?o }"))
+    def test_detects_with_prefix_preamble(self):
+        q = "PREFIX ex: <http://ex.org/>\nCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
+        self.assertTrue(_make_connected_store()._is_construct_query(q))
+    def test_detects_complex_preambles(self):
+        s = _make_connected_store()
+        cases = {
+            "multiline_prefix": "PREFIX foaf:\n  <http://xmlns.com/foaf/0.1/>\nCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
+            "empty_prefix": "PREFIX : <http://ex.org/> CONSTRUCT { ?s ?p ?o }",
+            "inline_comment": "PREFIX ex: <http://ex.org/>\n# comment\nCONSTRUCT { ?s ?p ?o }",
+            "base_declaration": "BASE <http://ex.org/>\nCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
+        }
+        for name, q in cases.items():
+            with self.subTest(case=name):
+                self.assertTrue(s._is_construct_query(q))
+    def test_false_for_select(self):
+        self.assertFalse(_make_connected_store()._is_construct_query(
+            "SELECT ?s WHERE { ?s ?p ?o }"))
+    def test_false_for_ask(self):
+        self.assertFalse(_make_connected_store()._is_construct_query(
+            "ASK { ?s ?p ?o }"))
+    def test_no_false_positive_on_constructor_substring(self):
+        self.assertFalse(_make_connected_store()._is_construct_query(
+            'SELECT ?s WHERE { ?s <urn:p> "CONSTRUCTOR" }'))
+    def test_no_false_positive_on_construct_in_string(self):
+        self.assertFalse(_make_connected_store()._is_construct_query(
+            'SELECT ?s WHERE { ?s <urn:p> "please CONSTRUCT this" }'))
+
+
+class TestRDF4JStoreExecuteSparqlConstructPath(unittest.TestCase):
+    def test_construct_sends_turtle_accept_header(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock()
+        mock_resp.content = b'@prefix ex: <http://ex.org/> .\nex:s1 ex:p1 "v" .\n'
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.execute_sparql(CONSTRUCT_QUERY)
+        _, kw = mp.call_args
+        self.assertEqual(kw["headers"]["Accept"], "text/turtle")
+        self.assertEqual(kw["headers"]["Content-Type"], "application/x-www-form-urlencoded")
+
+    def test_construct_parses_triples_from_turtle_fixture(self):
+        store = _make_connected_store()
+        fixture = (
+            b'@prefix ex: <http://ex.org/> .\n'
+            b'ex:s1 ex:p1 "value1" .\n'
+            b'ex:s1 ex:p2 ex:o2 .\n'
+        )
+        mock_resp = MagicMock()
+        mock_resp.content = fixture
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post", return_value=mock_resp):
+            result = store.execute_sparql(CONSTRUCT_QUERY)
+        self.assertTrue(result["success"])
+        self.assertEqual(result["bindings"], [])
+        self.assertEqual(result["variables"], [])
+        self.assertEqual(result["metadata"]["result_format"], "construct")
+        triples = {(s, p, o) for s, p, o, _m in result["triples"]}
+        self.assertIn(("http://ex.org/s1", "http://ex.org/p1", "value1"), triples)
+        self.assertIn(("http://ex.org/s1", "http://ex.org/p2", "http://ex.org/o2"), triples)
+        self.assertEqual(len(result["triples"]), 2)
+        for _s, _p, _o, meta in result["triples"]:
+            self.assertEqual(meta, {})
+
+    def test_result_format_construct_forces_construct_path(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock()
+        mock_resp.content = b'@prefix ex: <http://ex.org/> .\nex:s1 ex:p1 "v" .\n'
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            result = store.execute_sparql("SELECT ?s WHERE { ?s ?p ?o }", result_format="construct")
+        _, kw = mp.call_args
+        self.assertEqual(kw["headers"]["Accept"], "text/turtle")
+        self.assertIn("triples", result)
+
+    def test_malformed_turtle_raises_processing_error(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock()
+        mock_resp.content = b"this is { not [ valid turtle at all !!!"
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post", return_value=mock_resp):
+            with self.assertRaises(ProcessingError) as ctx:
+                store.execute_sparql(CONSTRUCT_QUERY)
+        self.assertIsInstance(ctx.exception, ProcessingError)
+        self.assertNotIsInstance(ctx.exception, (SyntaxError, ValueError))
+
+    def test_typed_literal_datatype_preserved_in_metadata(self):
+        store = _make_connected_store()
+        fixture = (
+            b'@prefix ex: <http://ex.org/> .\n'
+            b'@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n'
+            b'ex:s1 ex:p_age 42 .\n'
+        )
+        mock_resp = MagicMock()
+        mock_resp.content = fixture
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post", return_value=mock_resp):
+            result = store.execute_sparql(CONSTRUCT_QUERY)
+        self.assertEqual(len(result["triples"]), 1)
+        _s, _p, o, meta = result["triples"][0]
+        self.assertEqual(o, "42")
+        self.assertEqual(meta.get("datatype"), "http://www.w3.org/2001/XMLSchema#integer")
+        self.assertNotIn("language", meta)
+
+    def test_language_tagged_literal_preserved_in_metadata(self):
+        store = _make_connected_store()
+        fixture = b'@prefix ex: <http://ex.org/> .\nex:s2 ex:p_label "hello"@en .\n'
+        mock_resp = MagicMock()
+        mock_resp.content = fixture
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post", return_value=mock_resp):
+            result = store.execute_sparql(CONSTRUCT_QUERY)
+        self.assertEqual(len(result["triples"]), 1)
+        _s, _p, o, meta = result["triples"][0]
+        self.assertEqual(o, "hello")
+        self.assertEqual(meta.get("language"), "en")
+        self.assertNotIn("datatype", meta)
+
+    def test_literal_with_braces_parsed_correctly(self):
+        store = _make_connected_store()
+        fixture = b'@prefix ex: <http://ex.org/> .\nex:s1 ex:p1 "text with { and } braces inside" .\n'
+        mock_resp = MagicMock()
+        mock_resp.content = fixture
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post", return_value=mock_resp):
+            result = store.execute_sparql(CONSTRUCT_QUERY)
+        self.assertEqual(len(result["triples"]), 1)
+        _s, _p, obj, meta = result["triples"][0]
+        self.assertEqual(obj, "text with { and } braces inside")
+        self.assertEqual(meta, {})
+
+
+class TestRDF4JStoreProperty9NonConstructUnchanged(unittest.TestCase):
+    # RDF4J had Accept: application/sparql-results+json on non-CONSTRUCT before feat/754.
+    # That header is preserved exactly. NOT aligned with Blazegraph headerless path.
+
+    def test_select_response_shape_unchanged(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock()
+        mock_resp.json.return_value = {
+            "head": {"vars": ["s", "p", "o"]},
+            "results": {"bindings": [
+                {"s": {"type": "uri", "value": "http://ex.org/s1"},
+                 "p": {"type": "uri", "value": "http://ex.org/p1"},
+                 "o": {"type": "literal", "value": "v1"}}]},
+        }
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            result = store.execute_sparql("SELECT ?s ?p ?o WHERE { ?s ?p ?o }")
+        _, kw = mp.call_args
+        self.assertEqual(kw["headers"], {"Accept": "application/sparql-results+json"})
+        self.assertNotIn("Content-Type", kw["headers"])
+        self.assertEqual(result, {
+            "success": True,
+            "bindings": mock_resp.json.return_value["results"]["bindings"],
+            "variables": ["s", "p", "o"],
+            "metadata": {"query": "SELECT ?s ?p ?o WHERE { ?s ?p ?o }",
+                         "endpoint": store._get_sparql_endpoint()},
+        })
+        self.assertNotIn("triples", result)
+
+    def test_select_does_not_send_turtle_accept(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock()
+        mock_resp.json.return_value = {"head": {"vars": []}, "results": {"bindings": []}}
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.execute_sparql("SELECT ?s WHERE { ?s ?p ?o }")
+        _, kw = mp.call_args
+        self.assertNotEqual(kw["headers"].get("Accept"), "text/turtle")
+
+    def test_ask_uses_json_not_turtle_path(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock()
+        mock_resp.json.return_value = {"head": {}, "boolean": True}
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.execute_sparql("ASK { ?s ?p ?o }")
+        _, kw = mp.call_args
+        self.assertEqual(kw["headers"].get("Accept"), "application/sparql-results+json")
+
+    def test_non_construct_result_has_no_triples_key(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock()
+        mock_resp.json.return_value = {"head": {"vars": []}, "results": {"bindings": []}}
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post", return_value=mock_resp):
+            result = store.execute_sparql("SELECT ?s WHERE { ?s ?p ?o }")
+        self.assertNotIn("triples", result)
+
+
+class TestRDF4JStoreAddTripletsContextParameter(unittest.TestCase):
+    def _t(self):
+        return Triplet(subject="http://ex.org/s1", predicate="http://ex.org/p", object="http://ex.org/o1")
+
+    def test_graph_present_sends_context_param(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock(); mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.add_triplets([self._t()], graph="http://ex.org/mygraph")
+        _, kw = mp.call_args
+        self.assertIsNotNone(kw.get("params"))
+        self.assertIn("context", kw["params"])
+
+    def test_graph_present_encodes_as_angle_bracket_wrapped_iri(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock(); mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.add_triplets([self._t()], graph="http://ex.org/mygraph")
+        _, kw = mp.call_args
+        self.assertEqual(kw["params"]["context"], "<http://ex.org/mygraph>")
+        self.assertNotEqual(kw["params"]["context"], "http://ex.org/mygraph")
+
+    def test_graph_none_sends_no_context_param(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock(); mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.add_triplets([self._t()], graph=None)
+        _, kw = mp.call_args
+        self.assertIsNone(kw.get("params"))
+
+    def test_graph_omitted_sends_no_context_param(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock(); mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.add_triplets([self._t()])
+        _, kw = mp.call_args
+        self.assertIsNone(kw.get("params"))
+
+    def test_graph_none_does_not_send_context_null(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock(); mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.add_triplets([self._t()], graph=None)
+        _, kw = mp.call_args
+        params = kw.get("params")
+        if params is not None:
+            self.assertNotIn("context", params)
+
+    def test_context_encoding_matches_confirmed_rdf4j_protocol(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock(); mock_resp.raise_for_status = MagicMock()
+        graph_uri = "http://ex.org/mygraph"
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.add_triplets([self._t()], graph=graph_uri)
+        _, kw = mp.call_args
+        ctx = kw["params"]["context"]
+        self.assertEqual(ctx, f"<{graph_uri}>")
+        self.assertTrue(ctx.startswith("<"))
+        self.assertTrue(ctx.endswith(">"))
+
+    def test_graph_present_posts_to_statements_endpoint_not_modified_url(self):
+        store = _make_connected_store()
+        mock_resp = MagicMock(); mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp) as mp:
+            store.add_triplets([self._t()], graph="http://ex.org/g")
+        call_url = mp.call_args[0][0]
+        self.assertEqual(call_url, store._get_update_endpoint())
+        self.assertNotIn("context", call_url)
+
+
+class TestExecuteConstructTemplateWithRDF4JBackend(unittest.TestCase):
+
+    def test_end_to_end_with_rdf4j_backend(self):
+        from semantica.triplet_store.construct_templates import (
+            ConstructTemplate, ParameterDescriptor, execute_construct_template,
+        )
+        fixture = b'@prefix ex: <http://ex.org/> .\nex:s1 ex:p1 "Alice" .\n'
+        mock_resp = MagicMock()
+        mock_resp.content = fixture
+        mock_resp.raise_for_status = MagicMock()
+        store = _make_connected_store()
+        calls = []
+        store.add_triplets = (
+            lambda triplets, **opts:
+            (calls.append((triplets, opts)) or None) or {"success": True}
+        )
+        template = ConstructTemplate(
+            name="rdf4j_e2e",
+            description="e2e test",
+            construct_query=(
+                "CONSTRUCT { <http://ex.org/s1> <http://ex.org/p1> {{value}} } "
+                "WHERE { <http://ex.org/s1> <http://ex.org/p1> {{value}} }"
+            ),
+            parameters=[ParameterDescriptor(name="value", type="literal", required=True)],
+            target_graph="http://ex.org/rdf4j_graph",
+        )
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp):
+            result = execute_construct_template(
+                template, params={"value": "Alice"}, store_backend=store)
+        self.assertGreater(len(result), 0)
+        self.assertEqual(len(calls), 1)
+        _, opts = calls[0]
+        self.assertEqual(opts.get("graph"), "http://ex.org/rdf4j_graph")
+
+    def test_literal_metadata_round_trips_through_rdf4j_backend(self):
+        from semantica.triplet_store.construct_templates import (
+            ConstructTemplate, ParameterDescriptor, execute_construct_template,
+        )
+        fixture = (
+            b'@prefix ex: <http://ex.org/> .\n'
+            b'@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n'
+            b'ex:s1 ex:p_age 42 .\n'
+            b'ex:s2 ex:p_label "hello"@en .\n'
+        )
+        mock_resp = MagicMock()
+        mock_resp.content = fixture
+        mock_resp.raise_for_status = MagicMock()
+        store = _make_connected_store()
+        store.add_triplets = lambda triplets, **opts: {"success": True}
+        template = ConstructTemplate(
+            name="rdf4j_roundtrip",
+            description="round-trip test",
+            construct_query="CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
+            parameters=[ParameterDescriptor(name="value", type="literal", required=True)],
+        )
+        with patch("semantica.triplet_store.rdf4j_store.requests.post",
+                   return_value=mock_resp):
+            result = execute_construct_template(
+                template, params={"value": "x"}, store_backend=store)
+        by_subject = {t.subject: t for t in result}
+        age_t = by_subject["http://ex.org/s1"]
+        self.assertEqual(age_t.object, "42")
+        self.assertEqual(age_t.metadata.get("datatype"),
+                         "http://www.w3.org/2001/XMLSchema#integer")
+        label_t = by_subject["http://ex.org/s2"]
+        self.assertEqual(label_t.object, "hello")
+        self.assertEqual(label_t.metadata.get("lang"), "en")
+
+class TestRDF4JStoreQodoBugfixes(unittest.TestCase):
+    def test_ntriples_serialization_with_datatype_metadata(self):
+        store = _make_connected_store()
+        t = Triplet(
+            subject="http://ex.org/s1",
+            predicate="http://ex.org/p_age",
+            object="42",
+            metadata={"datatype": "http://www.w3.org/2001/XMLSchema#integer"}
+        )
+        nt = store._triplets_to_ntriples([t])
+        
+        # Verify rdflib can parse the generated N-Triples exactly
+        from rdflib import Graph
+        g = Graph()
+        g.parse(data=nt, format="nt")
+        self.assertEqual(len(g), 1)
+        for s, p, o in g:
+            self.assertEqual(str(s), "http://ex.org/s1")
+            self.assertEqual(str(p), "http://ex.org/p_age")
+            self.assertEqual(str(o), "42")
+            self.assertEqual(str(o.datatype), "http://www.w3.org/2001/XMLSchema#integer")
+
+    def test_ntriples_serialization_with_lang_metadata(self):
+        store = _make_connected_store()
+        t = Triplet(
+            subject="http://ex.org/s2",
+            predicate="http://ex.org/p_label",
+            object="hello",
+            metadata={"lang": "en"}
+        )
+        nt = store._triplets_to_ntriples([t])
+        
+        from rdflib import Graph
+        g = Graph()
+        g.parse(data=nt, format="nt")
+        self.assertEqual(len(g), 1)
+        for s, p, o in g:
+            self.assertEqual(str(s), "http://ex.org/s2")
+            self.assertEqual(str(p), "http://ex.org/p_label")
+            self.assertEqual(str(o), "hello")
+            self.assertEqual(o.language, "en")
+
+    def test_ntriples_serialization_fallback_is_iri(self):
+        store = _make_connected_store()
+        # No datatype/lang metadata
+        t = Triplet(
+            subject="http://ex.org/s1",
+            predicate="http://ex.org/p1",
+            object="http://ex.org/o1"
+        )
+        nt = store._triplets_to_ntriples([t])
+        
+        # Verify it parses as URI, not literal
+        from rdflib import Graph, URIRef
+        g = Graph()
+        g.parse(data=nt, format="nt")
+        self.assertEqual(len(g), 1)
+        for s, p, o in g:
+            self.assertEqual(str(s), "http://ex.org/s1")
+            self.assertEqual(str(p), "http://ex.org/p1")
+            self.assertEqual(str(o), "http://ex.org/o1")
+            self.assertIsInstance(o, URIRef)
+
+    def test_ntriples_serialization_datatype_wins_over_language(self):
+        store = _make_connected_store()
+        t = Triplet(
+            subject="http://ex.org/s3",
+            predicate="http://ex.org/p_both",
+            object="42",
+            # Provide both datatype and language
+            metadata={
+                "datatype": "http://www.w3.org/2001/XMLSchema#integer",
+                "lang": "en"
+            }
+        )
+        nt = store._triplets_to_ntriples([t])
+        
+        from rdflib import Graph
+        g = Graph()
+        g.parse(data=nt, format="nt")
+        self.assertEqual(len(g), 1)
+        for s, p, o in g:
+            # Datatype should win; rdflib literals with datatype don't have language
+            self.assertEqual(str(s), "http://ex.org/s3")
+            self.assertEqual(str(o), "42")
+            self.assertEqual(str(o.datatype), "http://www.w3.org/2001/XMLSchema#integer")
+            self.assertIsNone(o.language)
+
+    def test_ntriples_serialization_plain_literal_without_metadata(self):
+        # Regression test: an object with no datatype/lang metadata and no
+        # URI shape (e.g. typical NER/extraction output like "Alice") must
+        # be serialized as a plain quoted literal, not wrapped as `<Alice>`
+        # (which is not a valid IRI and corrupts the write).
+        store = _make_connected_store()
+        t = Triplet(
+            subject="http://ex.org/s1",
+            predicate="http://ex.org/name",
+            object="Alice",
+        )
+        nt = store._triplets_to_ntriples([t])
+        self.assertNotIn("<Alice>", nt)
+
+        from rdflib import Graph, Literal
+        g = Graph()
+        g.parse(data=nt, format="nt")
+        self.assertEqual(len(g), 1)
+        for s, p, o in g:
+            self.assertEqual(str(s), "http://ex.org/s1")
+            self.assertEqual(str(p), "http://ex.org/name")
+            self.assertIsInstance(o, Literal)
+            self.assertEqual(str(o), "Alice")
+
+    def test_add_triplets_validates_graph_uri(self):
+        from semantica.utils.exceptions import ValidationError
+        store = _make_connected_store()
+        t = Triplet(subject="http://ex.org/s1", predicate="http://ex.org/p", object="http://ex.org/o1")
+        with self.assertRaises(ValidationError) as ctx:
+            store.add_triplets([t], graph="http://ex.org/invalid graph")
+        self.assertIn("whitespace", str(ctx.exception))
+
+    def test_execute_sparql_validates_result_format(self):
+        from semantica.utils.exceptions import ValidationError
+        store = _make_connected_store()
+        with self.assertRaises(ValidationError) as ctx:
+            store.execute_sparql("SELECT ?s WHERE { ?s ?p ?o }", result_format="invalid")
+        self.assertIn("Invalid result_format", str(ctx.exception))
+        
+        # verify 'bindings' and 'construct' still work (mocks required)
+        mock_resp = MagicMock()
+        mock_resp.json.return_value = {"head": {"vars": []}, "results": {"bindings": []}}
+        mock_resp.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post", return_value=mock_resp):
+            store.execute_sparql("SELECT ?s WHERE { ?s ?p ?o }", result_format="bindings")
+            
+        mock_resp_c = MagicMock()
+        mock_resp_c.content = b""
+        mock_resp_c.raise_for_status = MagicMock()
+        with patch("semantica.triplet_store.rdf4j_store.requests.post", return_value=mock_resp_c):
+            store.execute_sparql("CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }", result_format="construct")
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/vector_store/test_sqlite_vec_store.py

@@ -0,0 +1,415 @@
+"""
+SQLite Vector Store Tests
+
+This module provides comprehensive unit tests for the SQLiteVecStore implementation using sqlite-vec.
+Tests are skipped if sqlite-vec is not available.
+
+pytest tests/vector_store/test_sqlite_vec_store.py -v
+"""
+
+import os
+import uuid
+import numpy as np
+import pytest
+from typing import Generator
+
+# Check dependencies
+try:
+    from semantica.vector_store.sqlite_vec_store import SQLITE_VEC_AVAILABLE
+except ImportError:
+    SQLITE_VEC_AVAILABLE = False
+
+# Skip all tests in this file if sqlite-vec is not available
+pytestmark = pytest.mark.skipif(
+    not SQLITE_VEC_AVAILABLE, reason="sqlite-vec not available"
+)
+
+
+@pytest.fixture
+def unique_table_name() -> str:
+    """Generate a unique table name for test isolation."""
+    return f"test_vectors_{uuid.uuid4().hex[:8]}"
+
+
+@pytest.fixture
+def db_file(tmp_path) -> str:
+    """Create a temporary database file path."""
+    return str(tmp_path / "test_vectors.db")
+
+
+@pytest.fixture
+def store(db_file, unique_table_name) -> Generator:
+    """Create a SQLiteVecStore instance for testing."""
+    from semantica.vector_store.sqlite_vec_store import SQLiteVecStore
+
+    store = SQLiteVecStore(
+        db_path=db_file,
+        table_name=unique_table_name,
+        dimension=128,
+        distance_metric="cosine",
+    )
+
+    yield store
+
+    # Teardown
+    store.close()
+
+
+class TestSQLiteVecStoreInit:
+    """Test SQLiteVecStore initialization."""
+
+    def test_init_success(self, store, db_file):
+        """Test successful initialization."""
+        assert store.dimension == 128
+        assert store.distance_metric == "cosine"
+        assert store.table_name.startswith("test_vectors_")
+        assert os.path.exists(db_file) or store.db_path == ":memory:"
+
+    def test_init_unsupported_metric(self, db_file):
+        """Test initialization with unsupported distance metric."""
+        from semantica.vector_store.sqlite_vec_store import SQLiteVecStore
+        from semantica.utils.exceptions import ValidationError
+
+        with pytest.raises(ValidationError, match="Unsupported distance metric"):
+            SQLiteVecStore(
+                db_path=db_file,
+                table_name="test",
+                dimension=128,
+                distance_metric="invalid_metric",
+            )
+
+    def test_init_table_creation(self, store):
+        """Test that table is created on initialization."""
+        with store._get_connection() as conn:
+            cur = conn.cursor()
+            cur.execute(
+                "SELECT name FROM sqlite_master WHERE type='table' AND name=?",
+                (store.table_name,),
+            )
+            row = cur.fetchone()
+            cur.close()
+            assert row is not None
+            assert row[0] == store.table_name
+
+
+class TestSQLiteVecStoreAdd:
+    """Test vector addition operations."""
+
+    def test_add_single_vector(self, store):
+        """Test adding a single vector."""
+        vector = np.random.rand(128).astype(np.float32)
+        metadata = {"source": "test", "index": 0}
+
+        ids = store.add([vector], [metadata], ids=["vec_0"])
+
+        assert ids == ["vec_0"]
+
+        # Retrieve and verify
+        res = store.get(["vec_0"])
+        assert len(res) == 1
+        assert res[0]["id"] == "vec_0"
+        assert np.allclose(res[0]["vector"], vector)
+        assert res[0]["metadata"] == metadata
+
+    def test_add_multiple_vectors(self, store):
+        """Test adding multiple vectors."""
+        vectors = [np.random.rand(128).astype(np.float32) for _ in range(5)]
+        metadata = [{"index": i} for i in range(5)]
+
+        ids = store.add(vectors, metadata)
+
+        assert len(ids) == 5
+        assert all(isinstance(id_str, str) for id_str in ids)
+
+    def test_add_auto_generate_ids(self, store):
+        """Test that IDs are auto-generated if not provided."""
+        vectors = [np.random.rand(128).astype(np.float32) for _ in range(3)]
+
+        ids = store.add(vectors)
+
+        assert len(ids) == 3
+        assert len(set(ids)) == 3  # All unique
+
+    def test_add_wrong_dimension(self, store):
+        """Test adding vector with wrong dimension."""
+        from semantica.utils.exceptions import ValidationError
+
+        vector = np.random.rand(64).astype(np.float32)  # Wrong dimension
+
+        with pytest.raises(ValidationError, match="dimension"):
+            store.add([vector])
+
+    def test_add_no_metadata(self, store):
+        """Test adding vectors without metadata."""
+        vectors = [np.random.rand(128).astype(np.float32) for _ in range(2)]
+
+        ids = store.add(vectors)
+
+        assert len(ids) == 2
+        res = store.get(ids)
+        assert all(r["metadata"] == {} for r in res)
+
+    def test_add_batch_with_numpy_array(self, store):
+        """Test adding vectors as numpy array."""
+        vectors = np.random.rand(5, 128).astype(np.float32)
+
+        ids = store.add(vectors)
+
+        assert len(ids) == 5
+
+
+class TestSQLiteVecStoreSearch:
+    """Test vector search operations."""
+
+    @pytest.fixture(autouse=True)
+    def setup_vectors(self, store):
+        """Setup test vectors for search tests."""
+        vectors = []
+        for i in range(10):
+            vec = np.zeros(128, dtype=np.float32)
+            vec[i] = 1.0  # Each vector has peak at different position
+            vectors.append(vec)
+
+        metadata = [{"category": "A" if i < 5 else "B", "index": i} for i in range(10)]
+        store.add(vectors, metadata)
+
+    def test_search_basic(self, store):
+        """Test basic similarity search."""
+        query = np.zeros(128, dtype=np.float32)
+        query[0] = 1.0  # Should match first vector perfectly
+
+        results = store.search(query, top_k=3)
+
+        assert len(results) == 3
+        assert all("id" in r for r in results)
+        assert all("score" in r for r in results)
+        assert all("metadata" in r for r in results)
+        assert results[0]["score"] == pytest.approx(1.0)
+
+    def test_search_top_k(self, store):
+        """Test search with different top_k values."""
+        query = np.random.rand(128).astype(np.float32)
+
+        results_5 = store.search(query, top_k=5)
+        results_10 = store.search(query, top_k=10)
+
+        assert len(results_5) == 5
+        assert len(results_10) == 10
+
+    def test_search_with_filter(self, store):
+        """Test search with metadata filter."""
+        query = np.zeros(128, dtype=np.float32)
+        query[0] = 1.0
+
+        results = store.search(query, top_k=10, filter={"category": "A"})
+
+        assert len(results) <= 5  # Only 5 vectors have category A
+        assert all(r["metadata"].get("category") == "A" for r in results)
+
+    def test_search_wrong_dimension(self, store):
+        """Test search with wrong query dimension."""
+        from semantica.utils.exceptions import ValidationError
+
+        query = np.random.rand(64).astype(np.float32)
+
+        with pytest.raises(ValidationError, match="dimension"):
+            store.search(query, top_k=5)
+
+    def test_search_empty_store(self, db_file, unique_table_name):
+        """Test search on empty store."""
+        from semantica.vector_store.sqlite_vec_store import SQLiteVecStore
+
+        empty_store = SQLiteVecStore(
+            db_path=db_file,
+            table_name=unique_table_name + "_empty",
+            dimension=128,
+            distance_metric="cosine",
+        )
+
+        query = np.random.rand(128).astype(np.float32)
+        results = empty_store.search(query, top_k=5)
+
+        assert len(results) == 0
+        empty_store.close()
+
+
+class TestSQLiteVecStoreGet:
+    """Test vector retrieval operations."""
+
+    def test_get_existing_vectors(self, store):
+        """Test getting existing vectors."""
+        vectors = [np.random.rand(128).astype(np.float32) for _ in range(3)]
+        metadata = [{"index": i} for i in range(3)]
+        ids = store.add(vectors, metadata)
+
+        results = store.get(ids)
+
+        assert len(results) == 3
+        result_ids = {r["id"] for r in results}
+        assert result_ids == set(ids)
+        assert all(r["vector"] is not None for r in results)
+        for r in results:
+            assert r["metadata"]["index"] in [0, 1, 2]
+
+    def test_get_nonexistent_ids(self, store):
+        """Test getting non-existent vector IDs."""
+        results = store.get(["nonexistent_1", "nonexistent_2"])
+
+        assert len(results) == 0
+
+    def test_get_empty_list(self, store):
+        """Test getting with empty ID list."""
+        results = store.get([])
+
+        assert results == []
+
+    def test_get_partial_ids(self, store):
+        """Test getting mix of existing and non-existing IDs."""
+        vectors = [np.random.rand(128).astype(np.float32)]
+        ids = store.add(vectors, [{"test": True}])
+
+        results = store.get(ids + ["nonexistent"])
+
+        assert len(results) == 1
+        assert results[0]["id"] == ids[0]
+
+
+class TestSQLiteVecStoreUpdate:
+    """Test vector update operations."""
+
+    def test_update_vectors(self, store):
+        """Test updating vectors."""
+        vectors = [np.random.rand(128).astype(np.float32) for _ in range(2)]
+        metadata = [{"version": 1} for _ in range(2)]
+        ids = store.add(vectors, metadata)
+
+        new_vectors = [np.random.rand(128).astype(np.float32) for _ in range(2)]
+        new_metadata = [{"version": 2} for _ in range(2)]
+
+        success = store.update(ids, new_vectors, new_metadata)
+
+        assert success is True
+
+        results = store.get(ids)
+        assert len(results) == 2
+        assert all(r["metadata"]["version"] == 2 for r in results)
+        # Compare by id rather than dict/ndarray equality
+        results_by_id = {r["id"]: r for r in results}
+        for vec_id, new_v in zip(ids, new_vectors):
+            assert np.allclose(results_by_id[vec_id]["vector"], new_v)
+
+    def test_update_metadata_only(self, store):
+        """Test updating only metadata."""
+        vectors = [np.random.rand(128).astype(np.float32)]
+        ids = store.add(vectors, [{"tag": "original"}])
+
+        success = store.update(ids, metadata=[{"tag": "updated"}])
+
+        assert success is True
+
+        results = store.get(ids)
+        assert results[0]["metadata"]["tag"] == "updated"
+
+    def test_update_vectors_only(self, store):
+        """Test updating only vectors."""
+        vectors = [np.random.rand(128).astype(np.float32)]
+        ids = store.add(vectors, [{"tag": "keep"}])
+
+        new_vector = np.random.rand(128).astype(np.float32)
+        success = store.update(ids, vectors=[new_vector])
+
+        assert success is True
+
+        results = store.get(ids)
+        assert np.allclose(results[0]["vector"], new_vector)
+        assert results[0]["metadata"]["tag"] == "keep"
+
+
+class TestSQLiteVecStoreDelete:
+    """Test vector deletion operations."""
+
+    def test_delete_vectors(self, store):
+        """Test deleting vectors."""
+        vectors = [np.random.rand(128).astype(np.float32) for _ in range(3)]
+        ids = store.add(vectors)
+
+        # Delete two of them
+        success = store.delete(ids[:2])
+        assert success is True
+
+        # Check only the third one remains
+        res = store.get(ids)
+        assert len(res) == 1
+        assert res[0]["id"] == ids[2]
+
+    def test_delete_empty(self, store):
+        """Test deleting empty list of IDs."""
+        assert store.delete([]) is True
+
+
+class TestSQLiteVecStoreReadOnly:
+    """Test read-only mode behavior."""
+
+    def test_read_only_mode(self, db_file, unique_table_name):
+        """Test that read-only mode restricts writes but allows reads."""
+        from semantica.vector_store.sqlite_vec_store import SQLiteVecStore
+        from semantica.utils.exceptions import ProcessingError
+
+        # 1. Create and populate database first
+        store_write = SQLiteVecStore(
+            db_path=db_file,
+            table_name=unique_table_name,
+            dimension=4,
+        )
+        vec = np.array([1, 2, 3, 4], dtype=np.float32)
+        store_write.add([vec], ids=["v1"])
+        store_write.close()
+
+        # 2. Open in read-only mode
+        store_ro = SQLiteVecStore(
+            db_path=db_file,
+            table_name=unique_table_name,
+            dimension=4,
+            read_only=True,
+        )
+
+        # Read should succeed
+        results = store_ro.get(["v1"])
+        assert len(results) == 1
+        assert results[0]["id"] == "v1"
+
+        # Search should succeed
+        search_res = store_ro.search(np.array([1, 2, 3, 4], dtype=np.float32), top_k=1)
+        assert len(search_res) == 1
+
+        # Write should fail
+        with pytest.raises(ProcessingError, match="read-only"):
+            store_ro.add([vec], ids=["v2"])
+
+        # Update should fail
+        with pytest.raises(ProcessingError, match="read-only"):
+            store_ro.update(["v1"], vectors=[vec])
+
+        # Delete should fail
+        with pytest.raises(ProcessingError, match="read-only"):
+            store_ro.delete(["v1"])
+
+        store_ro.close()
+
+
+class TestSQLiteVecStoreStats:
+    """Test store statistics retrieval."""
+
+    def test_get_stats(self, store):
+        """Test getting stats from store."""
+        stats = store.get_stats()
+        assert stats["vector_count"] == 0
+        assert stats["dimension"] == 128
+        assert stats["distance_metric"] == "cosine"
+
+        # Add vectors and check again
+        vectors = [np.random.rand(128).astype(np.float32) for _ in range(4)]
+        store.add(vectors)
+
+        stats = store.get_stats()
+        assert stats["vector_count"] == 4