fix(deps): address review feedback on import exceptions and probes (#1513)

- Remove thinc direct constraint from nlp-spacy in pyproject.toml and update README/CHANGELOG
- Raise ProcessingError with install hint when UMAP is requested but unavailable in EmbeddingVisualizer
- Guard PCA and TSNE dimensionality reducers against None with actionable error messages
- Prevent keyword collisions in EmbeddingVisualizer dimensionality reduction (_reduce_dimensions)
- Add core-only install & test step to .github/workflows/ci.yml using base-deps.txt
- Prevent AttributeError on module load in repo_ingestor.py and xml_ingestor.py when optional dependencies are absent
- Make TOML loading in tests/test_issue_1513_slim_core.py portable across Python versions via UTF-8 text decode and loads
- Expand slim-core test suite to 17 test cases covering 2D/3D projections, core imports, and options handling

.github/workflows/ci.yml

@@ -97,24 +97,10 @@ jobs:
       - name: Build Explorer frontend
         working-directory: explorer
         run: npm run build
-      - name: Install Explorer backend test dependencies
+      - name: Install core package and base dependencies
         run: |
-          # Run the deterministic backend path before the all-extras CI
-          # environment is installed.  The Explorer extra supplies the
-          # production API dependencies without importing optional vector
-          # providers such as Pinecone during test collection.
-          #
-          # --no-deps + a separate hash-pinned install (rather than the old
-          # `pip install -e ".[explorer]" pytest==9.1.1`) so every fetched
-          # package is hash-verified (Scorecard Pinned-Dependencies); the
-          # local editable install itself has nothing to hash.
-          # .github/requirements/explorer-extra-py311.txt is
-          # `uv pip compile pyproject.toml --extra explorer --python-version 3.11 --constraint requirements-ci.txt --generate-hashes`
-          # - regenerate it the same way if pyproject.toml's base/explorer
-          # deps change. Resolved specifically for this job's python 3.11
-          # (see the Dockerfile's explorer-extra-py313.txt for why this
-          # can't be shared with python 3.13: audioread needs extra
-          # standard-aifc/standard-sunau hashes only on 3.13+).
+          # Verify that core semantica installs cleanly with only its base dependencies
+          # (no optional extras) and that core imports and lazy missing-dependency hints work.
           #
           # --no-deps only skips *runtime* dependency resolution - `-e .`
           # still does a PEP 517 build, which by default creates an isolated
@@ -125,8 +111,30 @@ jobs:
           # copies instead of fetching its own.
           pip install -r .github/requirements/pep517-build.txt --require-hashes
           pip install --no-deps --no-build-isolation -e .
-          pip install -r .github/requirements/explorer-extra-py311.txt --require-hashes
+          pip install -r .github/requirements/base-deps.txt --require-hashes
           pip install -r .github/requirements/pytest-tool.txt --require-hashes
+      - name: Verify core-only package importability and slim behavior
+        run: |
+          python -c "
+          import semantica
+          print('semantica', semantica.__version__, 'core installed and importable')
+          "
+          pytest -q tests/test_issue_1513_slim_core.py
+      - name: Install Explorer backend test dependencies
+        run: |
+          # Run the deterministic backend path before the all-extras CI
+          # environment is installed.  The Explorer extra supplies the
+          # production API dependencies without importing optional vector
+          # providers such as Pinecone during test collection.
+          #
+          # .github/requirements/explorer-extra-py311.txt is
+          # `uv pip compile pyproject.toml --extra explorer --python-version 3.11 --constraint requirements-ci.txt --generate-hashes`
+          # - regenerate it the same way if pyproject.toml's base/explorer
+          # deps change. Resolved specifically for this job's python 3.11
+          # (see the Dockerfile's explorer-extra-py313.txt for why this
+          # can't be shared with python 3.13: audioread needs extra
+          # standard-aifc/standard-sunau hashes only on 3.13+).
+          pip install -r .github/requirements/explorer-extra-py311.txt --require-hashes
       - name: Test deterministic Explorer backend path
         run: |
           pytest -q tests/explorer/test_explorer_deterministic_rendering_e2e.py

CHANGELOG.md

@@ -9,6 +9,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.7.0] - 2026-09-07
+
+### Changed
+
+- **Slim core dependencies: moved ~22 heavy packages to optional extras** (#1513)
+  - Core dependencies in `pyproject.toml` are now reduced to exactly 22 direct packages: `numpy`, `pandas`, `scipy`, `scikit-learn`, `rdflib`, `networkx`, `requests`, `chardet`, `protobuf`, `grpcio`, `pillow`, `pydantic`, `click`, `rich`, `tqdm`, `pyyaml`, `toml`, `python-dotenv`, `loguru`, `structlog`, `httpx`, and `pyarrow`.
+  - Heavy ML/NLP, visualization, document parsing, and ingestion packages moved into granular optional extras:
+    - `models-huggingface`: `torch`, `transformers`
+    - `embeddings-local`: `sentence-transformers`, `fastembed`, `onnxruntime`, `tokenizers`
+    - `nlp-spacy`: `spacy`
+    - `viz`: expanded to include `matplotlib`, `seaborn`, `plotly`, `ipywidgets`, `umap-learn`, alongside `pyvis`, `graphviz`, and `d3blocks`
+    - `media`: `librosa`, `opencv-python`
+    - `vectorstore-faiss`: `faiss-cpu` (also included in `vectorstore-all`)
+    - `documents`: `python-docx`, `openpyxl`, `lxml`, `beautifulsoup4`
+    - `ingest-git`: `GitPython`
+    - `graph-embeddings`: `gensim` (also included in `graph-all`)
+  - Full bundled behavior preserved via `pip install "semantica[all]"`, which includes all optional extras. Pinning `semantica<0.7.0` remains a permanent escape hatch for legacy workflows.
+  - Safe lazy construction across parsers and visualizers:
+    - `DOCXParser`, `ExcelParser`, `HTMLParser`, and `XMLParser` remain constructible without error on `__init__()`. They fail only upon calling `.parse()` with actionable error messages directing users to install `semantica[documents]`.
+    - `XMLParser` automatically falls back to standard library `xml.etree` (`_parse_with_etree`) when `lxml` is not installed, preserving XML parsing capabilities without extra dependencies.
+    - `EmbeddingVisualizer` and `OntologyVisualizer` safely guard `matplotlib` and optional reduction packages, advising `pip install 'semantica[viz]'`.
+    - `RepoIngestor` guards `GitPython` with a clear error pointing to `semantica[ingest-git]`.
+    - `PublicAPIIngestor` guards `lxml` and `_SAFE_XML_PARSER`.
+    - Updated user-facing installation hints across CLI doctor commands, node embeddings (`NodeEmbedder`), vector stores (`FAISSStore`), and model loaders.
+  - Recompiled CI lockfiles (`requirements-ci.txt`, `.github/requirements/explorer-extra-py311.txt`, `.github/requirements/explorer-extra-py313.txt`, and `.github/requirements/base-deps.txt`).
+
 ## [0.6.8] - 2026-09-05
 
 ### Added

CITATION.cff

@@ -7,7 +7,7 @@ authors:
 repository-code: "https://github.com/semantica-agi/semantica"
 url: "https://getsemantica.ai"
 license: MIT
-version: 0.6.8
+version: 0.7.0
 date-released: 2026-09-05
 keywords:
   - knowledge-graph

README.md

@@ -1480,6 +1480,14 @@ app = create_app(session=GraphSession(graph), agent_memory=memory)
 The Memories workspace is shown only when `agent_memory` is provided. Apply
 updates the supplied runtime object; it does not add disk persistence.
 
+## What's New in v0.7.0
+
+**Slim core dependencies: lightweight base install with granular optional extras** — `pip install semantica` now installs only 22 essential core dependencies, moving heavy packages into dedicated optional extras:
+- **Dramatically lighter and faster installation**: Core installation no longer pulls heavy machine learning or visualization packages by default.
+- **Granular extras**: Install only what your workload requires (`documents`, `embeddings-local`, `models-huggingface`, `nlp-spacy`, `viz`, `media`, `vectorstore-faiss`, `graph-embeddings`, `ingest-git`).
+- **Full backward compatibility**: `pip install "semantica[all]"` preserves the full bundled suite, while `semantica<0.7.0` remains a permanent escape hatch.
+- **Lazy parser construction & graceful fallbacks**: Document parsers can be constructed without extras and only raise actionable error hints upon calling `.parse()`; `XMLParser` automatically falls back to Python's standard library `xml.etree`.
+
 ---
 
 ## What's New in v0.6.8
@@ -1518,32 +1526,42 @@ Semantica is designed for environments where AI outputs must be explainable, aud
 ## Installation
 
 ```bash
-pip install semantica           # core
-pip install semantica[all]      # everything
+pip install semantica             # lightweight core (22 essential dependencies)
+pip install "semantica[all]"      # full bundled behavior with all extras
 ```
 
+> **Note for upgrades from <0.7.0**: In Semantica 0.7.0+, heavy machine learning, NLP, visualization, and document dependencies were moved into optional extras to make core installation significantly lighter and faster. If you want the previous bundled installation, install with `pip install "semantica[all]"` or pin `semantica<0.7.0`.
+
 ```bash
-pip install semantica[agno]                 # Agno multi-agent integration
-pip install semantica[crewai]               # CrewAI integration
-pip install semantica[langchain]            # LangChain / LangGraph integration
-pip install semantica[llm-litellm]          # OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Bedrock, Ollama, DeepSeek, and more
-pip install semantica[graph-neo4j]          # Neo4j graph store (LPG)
-pip install semantica[graph-falkordb]       # FalkorDB graph store (LPG)
-pip install semantica[graph-apache-age]     # Apache AGE graph store (LPG)
-pip install semantica[graph-amazon-neptune] # AWS Neptune graph store (LPG)
-pip install semantica[tripletstore-oxigraph] # Embedded in-memory/on-disk RDF store
+# Granular Extras
+pip install "semantica[documents]"          # Document parsing (docx, openpyxl, lxml, beautifulsoup4)
+pip install "semantica[embeddings-local]"   # Local embeddings (sentence-transformers, fastembed, onnxruntime)
+pip install "semantica[models-huggingface]" # HuggingFace models (transformers, torch)
+pip install "semantica[nlp-spacy]"          # spaCy NLP pipelines (spacy)
+pip install "semantica[viz]"                # Visualization (matplotlib, seaborn, plotly, pyvis, graphviz)
+pip install "semantica[media]"              # Audio & computer vision (librosa, opencv-python)
+pip install "semantica[graph-embeddings]"   # Knowledge graph embeddings (gensim / Node2Vec)
+pip install "semantica[ingest-git]"         # Git repository ingestor (GitPython)
+pip install "semantica[vectorstore-faiss]"  # FAISS vector store
+pip install "semantica[vectorstore-all]"    # All vector stores (Qdrant, Pinecone, Weaviate, FAISS, PgVector, SQLite)
+pip install "semantica[agno]"               # Agno multi-agent integration
+pip install "semantica[crewai]"             # CrewAI integration
+pip install "semantica[langchain]"          # LangChain / LangGraph integration
+pip install "semantica[llm-all]"            # All LLM provider clients
+pip install "semantica[graph-neo4j]"        # Neo4j graph store (LPG)
+pip install "semantica[graph-falkordb]"     # FalkorDB graph store (LPG)
+pip install "semantica[graph-apache-age]"   # Apache AGE graph store (LPG)
+pip install "semantica[graph-amazon-neptune]" # AWS Neptune graph store (LPG)
+pip install "semantica[tripletstore-oxigraph]" # Embedded in-memory/on-disk RDF store
 # RDF triple stores (Blazegraph, Apache Jena, Eclipse RDF4J) need no extra:
 # semantica.triplet_store talks SPARQL over HTTP using the core `requests` dependency
-pip install semantica[vectorstore-qdrant]   # Qdrant vector store
-pip install semantica[vectorstore-pinecone] # Pinecone vector store
-pip install semantica[db-snowflake]         # Snowflake
-pip install semantica[db-databricks]        # Databricks (SDK + SQL connector)
-pip install semantica[ingest-sap]           # SAP OData
-pip install semantica[ingest-parquet]       # Parquet / PyArrow
-pip install semantica[ingest-arrow]        # Apache Arrow, Feather, IPC
-pip install semantica[viz]                  # HTML interactive visualization
-pip install semantica[watch]                # Directory file watcher
-pip install semantica[explorer]             # Knowledge Explorer dashboard
+pip install "semantica[db-snowflake]"       # Snowflake
+pip install "semantica[db-databricks]"      # Databricks (SDK + SQL connector)
+pip install "semantica[ingest-sap]"         # SAP OData
+pip install "semantica[ingest-parquet]"     # Parquet / PyArrow
+pip install "semantica[ingest-arrow]"       # Apache Arrow, Feather, IPC
+pip install "semantica[watch]"              # Directory file watcher
+pip install "semantica[explorer]"           # Knowledge Explorer dashboard
 ```
 
 For production deployments, use Docker or Kubernetes rather than a local `pip install`. Set `SEMANTICA_API_KEY`, configure a persistent LPG graph store (Neo4j / FalkorDB / Apache AGE / AWS Neptune) and/or RDF triple store (Blazegraph / Apache Jena / Eclipse RDF4J), and point the vector store at a hosted backend (Qdrant / Pinecone). See [ARCHITECTURE.md](ARCHITECTURE.md) for the full deployment topology.

docs/learning-more.md

@@ -64,7 +64,7 @@ Whether you're running your first pipeline or deploying Semantica in production,
         [Temporal Graphs notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/10_Temporal_Knowledge_Graphs.ipynb): `valid_from`/`valid_until`, Allen interval algebra, point-in-time queries.
       </Step>
       <Step title="Ontology-driven knowledge bases">
-        [Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb): auto-generation, SHACL validation, Ontology Hub (v0.5.0).
+        [Ontology notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/introduction/14_Ontology.ipynb): auto-generation, SHACL validation, Ontology Hub.
       </Step>
       <Step title="Advanced visualization">
         [Complete Visualization Suite notebook](https://github.com/semantica-agi/semantica/blob/main/cookbook/advanced/03_Complete_Visualization_Suite.ipynb): UMAP, t-SNE, community layouts, embedding projections.
@@ -86,10 +86,10 @@ All settings can be overridden with environment variables: no code changes neede
 | OpenAI API Key | `OPENAI_API_KEY` | `None` |
 | Groq API Key | `GROQ_API_KEY` | `None` |
 | Anthropic API Key | `ANTHROPIC_API_KEY` | `None` |
-| Embedding Provider | `SEMANTICA_EMBEDDING_PROVIDER` | `"openai"` |
-| Graph Backend | `SEMANTICA_GRAPH_BACKEND` | `"networkx"` |
-| Log Level | `SEMANTICA_LOG_LEVEL` | `"INFO"` |
-| Log Format | `SEMANTICA_LOG_FORMAT` | `"text"` |
+| Graph Store Backend | `GRAPH_STORE_DEFAULT_BACKEND` | `"neo4j"` |
+| Vector Store Backend | `VECTOR_STORE_DEFAULT_BACKEND` | `"faiss"` |
+| Server Host | `SEMANTICA_HOST` | `"127.0.0.1"` |
+| Server API Key | `SEMANTICA_API_KEY` | `None` |
 
 
 ## Troubleshooting
@@ -146,10 +146,15 @@ Also reduce batch sizes and enable streaming ingestion for large corpora.
 Enable parallel execution and GPU acceleration:
 
 ```python
-from semantica.pipeline import Pipeline
+from semantica.pipeline import ParallelismManager, Task
 
-pipeline = Pipeline(workers=8, batch_size=32)
-pipeline.run(sources)
+# Run pipeline tasks concurrently across worker threads
+manager = ParallelismManager(max_workers=8)
+tasks = [
+    Task("task_1", lambda: "process part 1"),
+    Task("task_2", lambda: "process part 2"),
+]
+results = manager.execute_parallel(tasks)
 ```
 
 ```bash
@@ -160,19 +165,19 @@ pip install "semantica[gpu]"  # CUDA-backed embeddings
 
 <Accordion title="Windows [all] installation fails" icon="windows">
 
-Fixed in **v0.5.0**. Upgrade:
+Upgrade to the latest release:
 
 ```bash
 pip install --upgrade semantica
 ```
 
-Or install extras individually: `pip install "semantica[core]"`, then add `[llm-openai]`, `[gpu]`, etc. as needed.
+Or install extras individually: `pip install semantica`, then add `[llm-openai]`, `[gpu]`, etc. as needed.
 
 </Accordion>
 
 <Accordion title="cp1252 encoding crash on Windows" icon="windows">
 
-Fixed in **v0.5.0**. For earlier versions, set the encoding environment variable:
+Set the encoding environment variable:
 
 ```bash
 set PYTHONIOENCODING=utf-8
@@ -202,27 +207,44 @@ Use NetworkX for local development and prototyping. Switch to a persistent backe
 
 <Accordion title="Batch processing for large corpora" icon="layer-group">
 
-Process documents in batches rather than one at a time. Configure `chunk_size` based on available RAM: a good starting point is 1,000 documents per batch on a 16 GB machine.
+Process documents in batches rather than one at a time. Split large texts into chunks and extract entities in batches:
 
 ```python
-from semantica.pipeline import Pipeline
+from semantica.split import TextSplitter
+from semantica.semantic_extract import NERExtractor
 
-pipeline = Pipeline(workers=8, batch_size=32)
-pipeline.run(sources)
+document_text = "Acme Corp announced record revenue in Seattle. CEO Jane Doe presented results."
+splitter = TextSplitter(chunk_size=1000, chunk_overlap=100)
+chunks = splitter.split(document_text)
+
+extractor = NERExtractor()
+batch_entities = extractor.extract_entities_batch([c.text for c in chunks])
 ```
 
 </Accordion>
 
 <Accordion title="Deduplication v2: up to 7× faster" icon="bolt">
 
-If deduplication is a bottleneck, switch from v1 strategies to the v2 engine:
+If deduplication is a bottleneck, use candidate blocking to reduce O(n²) comparisons before similarity scoring:
 
 ```python
-resolver = EntityResolver()
-merged   = resolver.resolve(entities, strategy="semantic_v2")  # up to 7x faster
+from semantica.deduplication import DuplicateDetector, EntityMerger
+
+entities = [
+    {"id": "1", "name": "Acme Corp", "type": "Company"},
+    {"id": "2", "name": "Acme Corporation", "type": "Company"},
+    {"id": "3", "name": "Globex", "type": "Company"},
+]
+
+# Fast candidate blocking for large entity sets
+detector = DuplicateDetector(similarity_threshold=0.8)
+duplicates = detector.detect_duplicates(entities, candidate_strategy="blocking_v2")
+
+merger = EntityMerger()
+merged = merger.merge_duplicates(entities, strategy="keep_most_complete")
 ```
 
-The `blocking_v2`, `hybrid_v2`, and `semantic_v2` strategies reduce O(n²) comparisons via candidate blocking before similarity scoring.
+The `blocking_v2` and `hybrid_v2` candidate strategies filter candidate pairs before calculating fine-grained similarity.
 
 </Accordion>
 
@@ -233,8 +255,8 @@ The `blocking_v2`, `hybrid_v2`, and `semantic_v2` strategies reduce O(n²) compa
 
 - **API keys**: store in environment variables or a secrets manager; never commit them to version control; rotate on a schedule
 - **Sensitive data**: use local embedding models (Ollama, HuggingFace) for PII or classified content; avoid sending sensitive data to external APIs without data handling agreements
-- **Graph exports**: encrypt sensitive exports at rest; use the v0.5.0 SSRF-safe `base_url` validation when configuring custom LLM gateways
-- **XML ingestion**: always use `XMLIngestor` (v0.5.0), which uses the XXE-safe lxml backend; never parse untrusted XML with the standard library parser
+- **Graph exports**: encrypt sensitive exports at rest; use SSRF-safe `base_url` validation when configuring custom LLM gateways
+- **XML ingestion**: always use `XMLIngestor`, which uses the XXE-safe lxml backend; never parse untrusted XML with the standard library parser
 
 - [Cookbook](/cookbook): interactive Jupyter notebooks from beginner to advanced.
 - [FAQ](/faq): common questions answered.

integrations/google_adk/session_service.py

@@ -21,6 +21,7 @@ from typing import Any, List, Optional
 try:
     from google.adk.events import Event
     from google.adk.sessions import BaseSessionService, Session
+
     try:
         from google.adk.sessions import ListSessionsResponse
     except ImportError:
@@ -33,7 +34,7 @@ try:
     except ImportError:
         from google.adk.sessions.base_session_service import GetSessionConfig
     ADK_AVAILABLE = True
-except (ImportError, ModuleNotFoundError):
+except (ImportError, OSError):
     ADK_AVAILABLE = False
     BaseSessionService = object
     Session = Any
@@ -162,10 +163,10 @@ class SemanticaSessionService(BaseSessionService):
         return {}
 
     def _find_session_node(
-            self,
-            app_name: str,
-            user_id: str,
-            session_id: str,
+        self,
+        app_name: str,
+        user_id: str,
+        session_id: str,
     ) -> Optional[Any]:
         """Find a session node by its logical ADK session ID."""
         expected_node_id = self._node_id(app_name, user_id, session_id)
@@ -182,18 +183,18 @@ class SemanticaSessionService(BaseSessionService):
             metadata = node.get("metadata")
 
             if (
-                    isinstance(metadata, dict)
-                    and str(metadata.get("session_id")) == str(session_id)
-                    and str(metadata.get("app_name")) == str(app_name)
-                    and str(metadata.get("user_id")) == str(user_id)
+                isinstance(metadata, dict)
+                and str(metadata.get("session_id")) == str(session_id)
+                and str(metadata.get("app_name")) == str(app_name)
+                and str(metadata.get("user_id")) == str(user_id)
             ):
                 return node
 
         return None
 
     def _find_node_by_id(
-            self,
-            node_id: str,
+        self,
+        node_id: str,
     ) -> Optional[Any]:
         """Find a ContextGraph node by graph node ID."""
         for node in self.graph.find_nodes() or []:
@@ -212,13 +213,13 @@ class SemanticaSessionService(BaseSessionService):
         data = SemanticaSessionService._safe_dict(event)
 
         for field in (
-                "id",
-                "invocation_id",
-                "author",
-                "timestamp",
-                "partial",
-                "turn_complete",
-                "branch",
+            "id",
+            "invocation_id",
+            "author",
+            "timestamp",
+            "partial",
+            "turn_complete",
+            "branch",
         ):
             if field not in data and hasattr(event, field):
                 value = getattr(event, field)
@@ -231,10 +232,10 @@ class SemanticaSessionService(BaseSessionService):
         return data
 
     def _event_nodes(
-            self,
-            app_name: str,
-            user_id: str,
-            session_id: str,
+        self,
+        app_name: str,
+        user_id: str,
+        session_id: str,
     ) -> List[Any]:
         """Return all event nodes connected to a session."""
         session_node_id = self._node_id(app_name, user_id, session_id)
@@ -275,8 +276,8 @@ class SemanticaSessionService(BaseSessionService):
         return str(timestamp)
 
     def _event_from_node(
-            self,
-            node: Any,
+        self,
+        node: Any,
     ) -> Any:
         """
         Reconstruct an ADK Event from its stored metadata.
@@ -289,7 +290,7 @@ class SemanticaSessionService(BaseSessionService):
         if not event_id and graph_node_id:
             graph_node_id = str(graph_node_id)
             if graph_node_id.startswith("adk-event:"):
-                event_id = graph_node_id[len("adk-event:"):]
+                event_id = graph_node_id[len("adk-event:") :]
 
         if event_id:
             properties["id"] = event_id
@@ -310,11 +311,11 @@ class SemanticaSessionService(BaseSessionService):
 
     @staticmethod
     def _session_kwargs(
-            app_name: str,
-            user_id: str,
-            session_id: str,
-            state: Optional[dict],
-            events: Optional[List[Any]],
+        app_name: str,
+        user_id: str,
+        session_id: str,
+        state: Optional[dict],
+        events: Optional[List[Any]],
     ) -> dict:
         """Build kwargs for the ADK Session model."""
         return {
@@ -326,8 +327,8 @@ class SemanticaSessionService(BaseSessionService):
         }
 
     def _session_from_node(
-            self,
-            node: Any,
+        self,
+        node: Any,
     ) -> Session:
         """Reconstruct an ADK Session from a ContextGraph node."""
         properties = self._node_properties(node)
@@ -347,14 +348,14 @@ class SemanticaSessionService(BaseSessionService):
                     # splitting on ':' after the prefix always yields
                     # exactly 3 parts regardless of what characters the
                     # original app_name/user_id/session_id contained.
-                    parts = graph_node_id[len("adk-session:"):].split(":")
+                    parts = graph_node_id[len("adk-session:") :].split(":")
                     if len(parts) == 3:
                         decoded = [urllib.parse.unquote(part) for part in parts]
                         app_name = app_name or decoded[0]
                         user_id = user_id or decoded[1]
                         session_id = decoded[2]
                     else:
-                        session_id = graph_node_id[len("adk-session:"):]
+                        session_id = graph_node_id[len("adk-session:") :]
                 else:
                     session_id = graph_node_id
 
@@ -383,12 +384,12 @@ class SemanticaSessionService(BaseSessionService):
     # ------------------------------------------------------------------
 
     async def create_session(
-            self,
-            *,
-            app_name: str,
-            user_id: str,
-            state: Optional[dict[str, Any]] = None,
-            session_id: Optional[str] = None,
+        self,
+        *,
+        app_name: str,
+        user_id: str,
+        state: Optional[dict[str, Any]] = None,
+        session_id: Optional[str] = None,
     ) -> Session:
         """Create and persist an ADK session."""
         return await asyncio.to_thread(
@@ -396,11 +397,11 @@ class SemanticaSessionService(BaseSessionService):
         )
 
     def _create_session_sync(
-            self,
-            app_name: str,
-            user_id: str,
-            state: Optional[dict[str, Any]],
-            session_id: Optional[str],
+        self,
+        app_name: str,
+        user_id: str,
+        state: Optional[dict[str, Any]],
+        session_id: Optional[str],
     ) -> Session:
         with self._lock:
             session_id = session_id or str(uuid.uuid4())
@@ -430,12 +431,12 @@ class SemanticaSessionService(BaseSessionService):
             )
 
     async def get_session(
-            self,
-            *,
-            app_name: str,
-            user_id: str,
-            session_id: str,
-            config: Optional[GetSessionConfig] = None,
+        self,
+        *,
+        app_name: str,
+        user_id: str,
+        session_id: str,
+        config: Optional[GetSessionConfig] = None,
     ) -> Optional[Session]:
         """Retrieve an ADK session from ContextGraph."""
         return await asyncio.to_thread(
@@ -443,11 +444,11 @@ class SemanticaSessionService(BaseSessionService):
         )
 
     def _get_session_sync(
-            self,
-            app_name: str,
-            user_id: str,
-            session_id: str,
-            config: Optional[GetSessionConfig],
+        self,
+        app_name: str,
+        user_id: str,
+        session_id: str,
+        config: Optional[GetSessionConfig],
     ) -> Optional[Session]:
         with self._lock:
             node = self._find_session_node(app_name, user_id, session_id)
@@ -468,7 +469,7 @@ class SemanticaSessionService(BaseSessionService):
         # trims the already-built Session object.
         if config:
             if config.num_recent_events:
-                session.events = session.events[-config.num_recent_events:]
+                session.events = session.events[-config.num_recent_events :]
             if config.after_timestamp:
                 i = len(session.events) - 1
                 while i >= 0:
@@ -476,14 +477,14 @@ class SemanticaSessionService(BaseSessionService):
                         break
                     i -= 1
                 if i >= 0:
-                    session.events = session.events[i + 1:]
+                    session.events = session.events[i + 1 :]
 
         return session
 
     async def append_event(
-            self,
-            session: Session,
-            event: Event,
+        self,
+        session: Session,
+        event: Event,
     ) -> Event:
         """Persist an ADK event and associate it with a session."""
         # ADK's own base implementation is a no-op for partial/streaming
@@ -510,8 +511,13 @@ class SemanticaSessionService(BaseSessionService):
 
             # Verify cross-tenant security
             properties = self._node_properties(session_node)
-            if properties.get("app_name") != app_name or properties.get("user_id") != user_id:
-                raise ValueError("Cross-tenant session write denied: app_name or user_id mismatch.")
+            if (
+                properties.get("app_name") != app_name
+                or properties.get("user_id") != user_id
+            ):
+                raise ValueError(
+                    "Cross-tenant session write denied: app_name or user_id mismatch."
+                )
 
             # Apply ADK in-memory event and state delta semantics. This
             # runs inside asyncio.to_thread's worker thread, which has no
@@ -553,11 +559,11 @@ class SemanticaSessionService(BaseSessionService):
             )
 
     async def delete_session(
-            self,
-            *,
-            app_name: str,
-            user_id: str,
-            session_id: str,
+        self,
+        *,
+        app_name: str,
+        user_id: str,
+        session_id: str,
     ) -> None:
         """Delete a session and all of its graph-backed events."""
         await asyncio.to_thread(
@@ -565,10 +571,10 @@ class SemanticaSessionService(BaseSessionService):
         )
 
     def _delete_session_sync(
-            self,
-            app_name: str,
-            user_id: str,
-            session_id: str,
+        self,
+        app_name: str,
+        user_id: str,
+        session_id: str,
     ) -> None:
         with self._lock:
             session_node = self._find_session_node(app_name, user_id, session_id)
@@ -590,9 +596,9 @@ class SemanticaSessionService(BaseSessionService):
                     continue
 
                 if (
-                        edge.get("source") == session_node_id
-                        and edge.get("type") == "HAS_EVENT"
-                        and edge.get("target")
+                    edge.get("source") == session_node_id
+                    and edge.get("type") == "HAS_EVENT"
+                    and edge.get("target")
                 ):
                     event_node_ids.append(str(edge["target"]))
 
@@ -602,18 +608,18 @@ class SemanticaSessionService(BaseSessionService):
             self.graph.purge_node(session_node_id)
 
     async def list_sessions(
-            self,
-            *,
-            app_name: str,
-            user_id: Optional[str] = None,
+        self,
+        *,
+        app_name: str,
+        user_id: Optional[str] = None,
     ) -> ListSessionsResponse:
         """List sessions for an app, optionally scoped to one user."""
         return await asyncio.to_thread(self._list_sessions_sync, app_name, user_id)
 
     def _list_sessions_sync(
-            self,
-            app_name: str,
-            user_id: Optional[str],
+        self,
+        app_name: str,
+        user_id: Optional[str],
     ) -> ListSessionsResponse:
         with self._lock:
             sessions: List[Session] = []
@@ -643,4 +649,4 @@ class SemanticaSessionService(BaseSessionService):
 __all__ = [
     "ADK_AVAILABLE",
     "SemanticaSessionService",
-]
+]

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "semantica"
-version = "0.6.8"
+version = "0.7.0"
 description = "Graph-Native Infrastructure for Context and Accountable AI Systems: context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
 readme = "README.md"
 license = { text = "MIT" }
@@ -52,30 +52,13 @@ dependencies = [
   # last 3.9-compatible release line; 3.10+ is left unconstrained.
   "scikit-learn>=1.6.1,<1.7.0; python_version < '3.10'",
   "scikit-learn>=1.7.2; python_version >= '3.10'",
-  "umap-learn>=0.5.12",
-  # thinc (spacy's core dep) dropped Python 3.9 wheels at 8.3.10, and later
-  # spacy patch releases (3.8.8+) require thinc>=8.3.9-only-on-3.10+ ranges,
-  # which forces a source build that fails outright on 3.9 (see Install
-  # Matrix run history). Capping both keeps 3.9 on the last wheel-compatible
-  # pair; 3.10+ is left unconstrained to always get the latest spacy/thinc.
-  "spacy>=3.4.0,<3.8.8; python_version < '3.10'",
-  "spacy>=3.4.0; python_version >= '3.10'",
-  "thinc<8.3.5; python_version < '3.10'",
-  "transformers>=4.20.0",
-  "torch>=1.13.1",
-  "sentence-transformers>=2.2.0",
   "rdflib>=6.2.0",
   "networkx>=2.8.0",
-  "matplotlib>=3.9.4",
-  "seaborn>=0.13.2",
-  "plotly>=6.8.0",
-  "ipywidgets>=8.0.0",
   # requests dropped Python 3.9 support at 2.33.0 (requires_python >=3.10),
   # so an unqualified >=2.34.2 floor is unsatisfiable on 3.9. Cap 3.9 to the
   # last 3.9-compatible release; 3.10+ is left unconstrained.
   "requests>=2.32.5,<2.33.0; python_version < '3.10'",
   "requests>=2.34.2; python_version >= '3.10'",
-  "GitPython>=3.1.58",
   # chardet dropped Python 3.9 support at 6.0.0 (requires_python >=3.10), so
   # an unqualified >=7.4.3 floor is unsatisfiable on 3.9. Cap 3.9 to the last
   # 3.9-compatible release; 3.10+ is left unconstrained.
@@ -87,26 +70,11 @@ dependencies = [
   # 3.9-compatible release; 3.10+ is left unconstrained.
   "grpcio>=1.80.0,<1.81.0; python_version < '3.10'",
   "grpcio>=1.81.1; python_version >= '3.10'",
-  "beautifulsoup4>=4.15.0",
-  "lxml>=6.1.1",
-  "python-docx>=1.2.0",
-  "openpyxl>=3.1.5",
   # pillow dropped Python 3.9 support at 12.0.0 (requires_python >=3.10), so
   # an unqualified >=12.2.0 floor is unsatisfiable on 3.9. Cap 3.9 to the last
   # 3.9-compatible release; 3.10+ is left unconstrained.
   "pillow>=11.3.0,<12.0.0; python_version < '3.10'",
   "pillow>=12.2.0; python_version >= '3.10'",
-  "librosa>=0.9.0",
-  "opencv-python>=4.13.0.92",
-  "faiss-cpu>=1.7.0",
-  "fastembed>=0.2.0",
-  # onnxruntime stopped shipping cp39 wheels at 1.20.0 (its PyPI metadata
-  # still claims requires_python >=3.9, but no matching wheel exists), so an
-  # unqualified >=1.20.1 floor is unsatisfiable on 3.9. Cap 3.9 to the last
-  # release with a cp39 wheel; 3.10+ is left unconstrained.
-  "onnxruntime>=1.19.2,<1.20.0; python_version < '3.10'",
-  "onnxruntime>=1.20.1; python_version >= '3.10'",
-  "tokenizers>=0.15.0",
   "pydantic>=2.13.4",
   # click dropped Python 3.9 support at 8.2.0 (requires_python >=3.10), so an
   # unqualified >=8.4.2 floor is unsatisfiable on 3.9. Cap 3.9 to the last
@@ -120,7 +88,6 @@ dependencies = [
   "python-dotenv>=1.2.1",
   "loguru>=0.7.3",
   "structlog>=22.1.0",
-  "gensim>=4.4.0",
   "httpx<0.29.0",
   "pyarrow>=14.0.0"
 ]
@@ -152,6 +119,12 @@ llm-all = [
 ]
 
 # ---- Document Parsing ----
+documents = [
+  "python-docx>=1.2.0",
+  "openpyxl>=3.1.5",
+  "lxml>=6.1.1",
+  "beautifulsoup4>=4.15.0"
+]
 parse-docling = ["docling>=2.107.0"]
 
 # ---- SHACL Validation ----
@@ -165,6 +138,7 @@ db-salesforce = ["simple-salesforce>=1.12.0"]
 ingest-parquet = ["pyarrow>=24.0.0"]
 ingest-arrow = ["pyarrow>=24.0.0"]
 ingest-sap = ["requests>=2.28.0"]
+ingest-git = ["GitPython>=3.1.58"]
 
 db-all = [
   "semantica[db-snowflake,db-databricks,db-salesforce,db-arrow]"
@@ -175,21 +149,34 @@ models-huggingface = [
   "transformers>=4.20.0",
   "torch>=1.13.1"
 ]
+embeddings-local = [
+  "sentence-transformers>=2.2.0",
+  "fastembed>=0.2.0",
+  "onnxruntime>=1.19.2,<1.20.0; python_version < '3.10'",
+  "onnxruntime>=1.20.1; python_version >= '3.10'",
+  "tokenizers>=0.15.0"
+]
+nlp-spacy = [
+  "spacy>=3.4.0,<3.8.8; python_version < '3.10'",
+  "spacy>=3.4.0; python_version >= '3.10'"
+]
 
 # ---- Graph Backends ----
 graph-neo4j = ["neo4j>=5.0.0"]
 graph-falkordb = ["falkordb>=1.0.0", "redis>=4.3.0"]
 graph-amazon-neptune = ["boto3>=1.24.0", "neo4j>=5.0.0"]
 graph-apache-age = ["psycopg2-binary>=2.9.0"]
+graph-embeddings = ["gensim>=4.4.0"]
 
 graph-all = [
-  "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune,graph-apache-age]"
+  "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune,graph-apache-age,graph-embeddings]"
 ]
 
 # ---- Triplet Store Backends ----
 tripletstore-oxigraph = ["pyoxigraph>=0.5.0"]
 
 # ---- Vector Store Backends ----
+vectorstore-faiss = ["faiss-cpu>=1.7.0"]
 vectorstore-qdrant = ["qdrant-client>=1.0.0"]
 vectorstore-weaviate = ["weaviate-client>=4.0.0"]
 vectorstore-pinecone = ["pinecone>=3.0.0"]
@@ -198,7 +185,7 @@ vectorstore-pgvector = ["psycopg[binary,pool]>=3.0.0", "pgvector>=0.2.0"]
 vectorstore-sqlite = ["sqlite-vec>=0.1.1"]
 
 vectorstore-all = [
-  "semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus,vectorstore-pgvector,vectorstore-sqlite]"
+  "semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus,vectorstore-pgvector,vectorstore-sqlite,vectorstore-faiss]"
 ]
 
 # ---- Infra / Queues / Workers ----
@@ -230,7 +217,18 @@ monitoring = [
 viz = [
   "pyvis>=0.3.0",
   "graphviz>=0.21",
-  "d3blocks>=1.0.0"
+  "d3blocks>=1.0.0",
+  "matplotlib>=3.9.4",
+  "seaborn>=0.13.2",
+  "plotly>=6.8.0",
+  "ipywidgets>=8.0.0",
+  "umap-learn>=0.5.12"
+]
+
+# ---- Media ----
+media = [
+  "librosa>=0.9.0",
+  "opencv-python>=4.13.0.92"
 ]
 
 # ---- GPU ----
@@ -294,8 +292,7 @@ explorer-lite = [
 # (CVE-2026-45829) with no fixed release — including it here would fail the CI
 # dependency-audit/security gates.  Install it explicitly via ``semantica[crewai]``.
 all = [
-  "semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
-  "semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno,langchain,google-adk]"
+  "semantica[dev,viz,media,infra,cloud,monitoring,watch,llm-all,models-huggingface,embeddings-local,nlp-spacy,documents,ingest-git,graph-embeddings,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer,agno,langchain,google-adk]"
 ]
 
 # ---------------- ENTRYPOINTS ----------------

requirements-ci.txt

@@ -181,6 +181,7 @@ anyio==4.14.2 \
     #   jupyter-server
     #   langsmith
     #   openai
+    #   pinecone
     #   starlette
     #   watchfiles
 argon2-cffi==25.1.0 \
@@ -4409,15 +4410,15 @@ pillow==12.3.0 \
     #   python-pptx
     #   rapidocr
     #   torchvision
-pinecone==9.1.0 \
-    --hash=sha256:461632bb07919da32b943100b8a047c74be53a6aa15c8b7679bff7a0f834c939 \
-    --hash=sha256:6c3a6dfa577dc11aed3197e1b221e65522603e9e1f6bd27a1b504a0909b3559f \
-    --hash=sha256:d3871bd3f39cb430ae8470158dc9c5dcffbac5ae31d144d9a7c3b351ac51755f \
-    --hash=sha256:d53fe6f4978ab0642eb2d3a0ee3b2576ccfeebaa11e0690b18e67dac4e057047 \
-    --hash=sha256:e930ba819f5b7e20aac688d04c840a8b6fbc6d12630d71303bb2130881a9d169 \
-    --hash=sha256:fc71ec431108de2df1a1978d3a24ac16f74ba3d8f3265c3760f969386e8742b8 \
-    --hash=sha256:fe6aeaf6515e9021984755ebc162f643c79d98056059aab2e765962a7538818c \
-    --hash=sha256:ffae8fb7cbb4056b920586629f15b08107350be4802a5637d10b31e2ad841f9c
+pinecone==10.0.0 \
+    --hash=sha256:0994270c514b16c72ec94dd6c29ff2708b81d30ff8467e19de192a28a7c86b7e \
+    --hash=sha256:0e05956a3201b1fbb54a1861277df919318a4941797f2d87fd558ac5ec232151 \
+    --hash=sha256:2f4e3200ee3562d195802b363487dd7fe6039a8c13630fc25fa3e8726c7a8654 \
+    --hash=sha256:3ab0c843b4fb04fbac22f1b8455e389063208a50f6a95d7a90627939968198de \
+    --hash=sha256:6066bbe9a7ae1d667cde08d262deb6fbea6feb35deb9177dd47141b55bbd9833 \
+    --hash=sha256:94d4c64779f3213a5cc538d3bd10a873da192cb9b0039db56690e556ba00b55c \
+    --hash=sha256:995c06e905940b10bb2aefe653225340b5a3f56f3efb373fe07f4e57b5043705 \
+    --hash=sha256:d482ed27a805cbd4aca2660da212008dd6e41d87d255279f405ff13b725970e2
     # via semantica (pyproject.toml)
 platformdirs==4.11.7 \
     --hash=sha256:4f41487eeeeeb07f3a6625e61d9bc0ae6809f92d3386dbd74392fbb76108104d \

semantica/__init__.py

@@ -10,7 +10,7 @@ Main exports:
     - Config: Configuration management
 """
 
-__version__ = "0.6.8"
+__version__ = "0.7.0"
 __author__ = "Semantica Contributors"
 __license__ = "MIT"
 

semantica/cli.py

@@ -95,7 +95,26 @@ _ERROR_HINTS: Dict[type, str] = {
 }
 
 
+def _json_error_mode() -> bool:
+    """True when this invocation promised machine-readable stdout.
+
+    Covers both the global ``--json`` flag (stored on the CLI context) and a
+    subcommand's local ``--json`` flag (uniformly named ``local_json``).
+    """
+    ctx = click.get_current_context(silent=True)
+    if ctx is None:
+        return False
+    if ctx.params.get("local_json"):
+        return True
+    return isinstance(ctx.obj, CLIContext) and ctx.obj.json_output
+
+
 def _show_error_card(title: str, detail: str, hint: Optional[str] = None) -> None:
+    if _json_error_mode():
+        # --json promises machine-readable stdout with errors on stderr, so
+        # emit a structured error line there instead of a Rich panel.
+        click.echo(json.dumps({"error": detail, "type": title}), err=True)
+        return
     body = f"[bold]{title}[/bold]\n[{_DIM}]{detail}[/{_DIM}]"
     if hint:
         body += f"\n\n[{_KEY}]→[/{_KEY}] [{_DIM}]{hint}[/{_DIM}]"
@@ -105,7 +124,7 @@ def _show_error_card(title: str, detail: str, hint: Optional[str] = None) -> Non
 
 
 def _run_with_error_handling(action: Callable[[], None]) -> None:
-    """Run a CLI action with Rich error cards on failure."""
+    """Run a CLI action with error cards (or JSON-mode stderr errors) on failure."""
     try:
         action()
     except click.ClickException as exc:
@@ -861,10 +880,18 @@ def doctor(cli_ctx: CLIContext, local_json: bool, deep_embeddings: bool) -> None
         def _embedding_backend(method: str) -> str:
             if method == "sentence_transformers":
                 import sentence_transformers  # noqa: F401
-                note = f"importable ({importlib.metadata.version('sentence-transformers')})"
+                try:
+                    ver = importlib.metadata.version("sentence-transformers")
+                except Exception:
+                    ver = getattr(sentence_transformers, "__version__", "installed")
+                note = f"importable ({ver})"
             else:
                 import fastembed  # noqa: F401
-                note = f"importable ({importlib.metadata.version('fastembed')})"
+                try:
+                    ver = importlib.metadata.version("fastembed")
+                except Exception:
+                    ver = getattr(fastembed, "__version__", "installed")
+                note = f"importable ({ver})"
             if not deep:
                 return note
             try:
@@ -885,12 +912,12 @@ def doctor(cli_ctx: CLIContext, local_json: bool, deep_embeddings: bool) -> None
         checks.append(_check(
             "Embeddings (sentence-transformers)",
             lambda: _embedding_backend("sentence_transformers"),
-            hint="pip install sentence-transformers",
+            hint="pip install 'semantica[embeddings-local]'",
         ))
         checks.append(_check(
             "Embeddings (fastembed)",
             lambda: _embedding_backend("fastembed"),
-            hint="pip install fastembed",
+            hint="pip install 'semantica[embeddings-local]'",
         ))
 
         # LLM provider keys

semantica/context/erasure.py

@@ -14,13 +14,15 @@ not. It *composes* the existing public APIs; nothing in ``context_graph.py`` or
 ``agent_memory.py`` changes, and ``ContextGraph`` keeps its graph-scope
 contract.
 
-The property that matters is honest partial reporting. FAISS exposes no delete
-at all -- a flat FAISS index cannot remove individual vectors without a full
-rebuild -- so erasure is genuinely not completable on it today. Milvus and
-Weaviate now expose ``delete_vectors`` and are fully supported. The receipt
-says ``unsupported`` for FAISS rather than reporting a success it did not
-achieve -- a receipt that reads
-"graph: erased, memory: 14 erased, vectors: unsupported on faiss" is
+The property that matters is honest partial reporting. FAISS Flat indices now
+expose ``delete_vectors`` backed by native ``remove_ids``, so erasure is
+completable on them.  FAISS IVF indices explicitly reject deletion because
+their internal labels are not compacted after ``remove_ids``, which would
+desynchronize search results from the ``vector_ids`` mapping.  HNSW does not
+implement ``remove_ids`` at all.  Both IVF and HNSW report ``unsupported``.
+Milvus and Weaviate are also fully supported. The receipt says ``unsupported``
+rather than reporting a success it did not achieve -- a receipt that reads
+"graph: erased, memory: 14 erased, vectors: unsupported on faiss/hnsw" is
 actionable; a bare ``True`` is a compliance liability.
 
 Example:
@@ -379,8 +381,9 @@ class ErasureCoordinator:
 
         method_name, target = _vector_delete_capability(self.vector_store)
         if method_name is None:
-            # FAISS exposes no delete at all; it cannot remove vectors from a
-            # flat index without a full rebuild.
+            # FAISS HNSW does not implement remove_ids and FAISS IVF
+            # does not compact labels after remove_ids.  Only Flat indices
+            # currently support deletion via this code path.
             self.logger.warning(
                 "Vector backend %r exposes no delete; %d vector id(s) for %r "
                 "were not erased",

semantica/embeddings/provider_stores.py

@@ -212,7 +212,7 @@ class FastEmbedStore(ProviderStore):
             self.logger.info(f"Loaded FastEmbed model: {self.model_name}")
         except (ImportError, OSError):
             self.logger.warning(
-                "fastembed not available. Install with: pip install fastembed"
+                "fastembed not available. Install with: pip install 'semantica[embeddings-local]'"
             )
         except Exception as e:
             self.logger.warning(f"Failed to load FastEmbed model: {e}")

semantica/embeddings/text_embedder.py

@@ -35,6 +35,7 @@ try:
 
     SENTENCE_TRANSFORMERS_AVAILABLE = True
 except (ImportError, OSError):
+    SentenceTransformer = None
     SENTENCE_TRANSFORMERS_AVAILABLE = False
 
 try:
@@ -42,6 +43,7 @@ try:
 
     FASTEMBED_AVAILABLE = True
 except (ImportError, OSError):
+    TextEmbedding = None
     FASTEMBED_AVAILABLE = False
 
 
@@ -156,7 +158,7 @@ class TextEmbedder:
             else:
                 self.logger.warning(
                     "fastembed not available. "
-                    "Install with: pip install fastembed. "
+                    "Install with: pip install 'semantica[embeddings-local]'. "
                     "Using fallback embedding method."
                 )
         else:
@@ -178,7 +180,7 @@ class TextEmbedder:
             else:
                 self.logger.warning(
                     "sentence-transformers not available. "
-                    "Install with: pip install sentence-transformers. "
+                    "Install with: pip install 'semantica[embeddings-local]'. "
                     "Using fallback embedding method."
                 )
         

semantica/export/vector_exporter.py

@@ -421,7 +421,7 @@ class VectorExporter:
             import numpy as np
         except (ImportError, OSError):
             raise ImportError(
-                "FAISS not installed. Install with: pip install faiss-cpu or faiss-gpu"
+                "FAISS not installed. Install with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
             )
 
         # Extract vectors and IDs

semantica/ingest/__init__.py

@@ -133,7 +133,11 @@ import importlib
 from typing import TYPE_CHECKING, Any, Dict, Tuple
 
 if TYPE_CHECKING:
-    from .salesforce_ingestor import SalesforceConnector, SalesforceData, SalesforceIngestor
+    from .salesforce_ingestor import (
+        SalesforceConnector,
+        SalesforceData,
+        SalesforceIngestor,
+    )
 
 from .config import IngestConfig, ingest_config
 from .file_ingestor import (
@@ -248,32 +252,42 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
 _OPTIONAL_DEPENDENCY_MESSAGES = {
     ".repo_ingestor": (
         "Repository ingestion requires optional dependency 'GitPython'. "
-        "Install it before importing RepoIngestor or using ingest_repository()."
+        "Install it before importing RepoIngestor or using ingest_repository(). "
+        "Install it with: pip install 'semantica[ingest-git]'"
     ),
     ".web_ingestor": (
         "Web ingestion requires optional dependency 'beautifulsoup4'. "
-        "Install it before importing WebIngestor or using ingest_web()."
+        "Install it before importing WebIngestor or using ingest_web(). "
+        "Install it with: pip install 'semantica[documents]'"
     ),
     ".feed_ingestor": (
         "Feed ingestion requires optional dependency 'beautifulsoup4'. "
-        "Install it before importing FeedIngestor or using ingest_feed()."
+        "Install it before importing FeedIngestor or using ingest_feed(). "
+        "Install it with: pip install 'semantica[documents]'"
     ),
     ".email_ingestor": (
         "Email ingestion requires optional dependency 'beautifulsoup4'. "
-        "Install it before importing EmailIngestor or using ingest_email()."
+        "Install it before importing EmailIngestor or using ingest_email(). "
+        "Install it with: pip install 'semantica[documents]'"
+    ),
+    ".xml_ingestor": (
+        "XML ingestion requires optional dependency 'lxml'. "
+        "Install it before importing XMLIngestor or using ingest_xml(). "
+        "Install it with: pip install 'semantica[documents]'"
     ),
     ".parquet_ingestor": (
         "Parquet ingestion requires optional dependency 'pyarrow'. "
-        "Install it before importing ParquetIngestor or using ingest_parquet()."
+        "Install it before importing ParquetIngestor or using ingest_parquet(). "
+        "Install it with: pip install 'semantica[ingest-parquet]'"
     ),
     ".arrow_ingestor": (
         "Arrow ingestion requires optional dependency 'pyarrow'. "
-        "Install it before importing ArrowIngestor or using ingest_arrow()."
+        "Install it before importing ArrowIngestor or using ingest_arrow(). "
+        "Install it with: pip install 'semantica[ingest-arrow]'"
     ),
     ".salesforce_ingestor": (
         "Salesforce ingestion requires optional dependency 'simple-salesforce'. "
-        "Install it with: pip install \"semantica[db-salesforce]\" "
-        "or: pip install simple-salesforce>=1.12.0"
+        "Install it with: pip install 'semantica[db-salesforce]'"
     ),
 }
 
@@ -286,13 +300,55 @@ def __getattr__(name: str) -> Any:
     module_name, attr_name = _LAZY_EXPORTS[name]
     try:
         module = importlib.import_module(module_name, __name__)
-    except ModuleNotFoundError as exc:
+    except (ImportError, OSError) as exc:
         message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
         missing_name = getattr(exc, "name", None)
-        if message and missing_name in {"git", "bs4", "pyarrow", "simple_salesforce"}:
+        if message and (
+            missing_name is None
+            or any(
+                pkg in missing_name
+                for pkg in ("git", "bs4", "pyarrow", "simple_salesforce", "lxml")
+            )
+        ):
             raise ImportError(message) from exc
         raise
 
+    # Guard against backends whose modules imported cleanly with dependencies
+    # set to None; ensure probe imports (e.g. try: from semantica.ingest import ...)
+    # fail at import time rather than postponing failure to construction time.
+    if module_name == ".repo_ingestor" and name in {"RepoIngestor"}:
+        if getattr(module, "git", None) is None:
+            message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
+            if message:
+                raise ImportError(message)
+
+    if module_name == ".xml_ingestor" and name in {"XMLIngestor"}:
+        if getattr(module, "etree", None) is None:
+            message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
+            if message:
+                raise ImportError(message)
+
+    if module_name == ".parquet_ingestor" and name in {"ParquetIngestor"}:
+        if not getattr(module, "PARQUET_AVAILABLE", True):
+            message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
+            if message:
+                raise ImportError(message)
+
+    if module_name == ".arrow_ingestor" and name in {"ArrowIngestor"}:
+        if not getattr(module, "ARROW_AVAILABLE", True):
+            message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
+            if message:
+                raise ImportError(message)
+
+    if module_name == ".salesforce_ingestor" and name in {
+        "SalesforceIngestor",
+        "SalesforceConnector",
+    }:
+        if not getattr(module, "SALESFORCE_AVAILABLE", True):
+            message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
+            if message:
+                raise ImportError(message)
+
     value = getattr(module, attr_name)
     globals()[name] = value
     return value

semantica/ingest/methods.py

@@ -193,6 +193,7 @@ def _is_scp_like_repo_source(source: str) -> bool:
     """Return True for scp-like SSH remotes (``user@host:path``)."""
     return bool(_SCP_LIKE_REPO_URL_RE.match(source.strip()))
 
+
 if TYPE_CHECKING:
     from .api_ingestor import APIData
     from .arrow_ingestor import ArrowData
@@ -252,7 +253,12 @@ def ingest_file(
     if custom_method and custom_method != ingest_file:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -318,21 +324,18 @@ def ingest_parquet(
     if custom_method and custom_method != ingest_parquet:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
 
     try:
-        try:
-            from .parquet_ingestor import ParquetIngestor
-        except ModuleNotFoundError as exc:
-            if _is_missing_dependency(exc, "pyarrow"):
-                raise _missing_optional_dependency(
-                    "Parquet ingestion",
-                    "pyarrow",
-                ) from exc
-            raise
+        from .parquet_ingestor import ParquetIngestor
 
         config = ingest_config.get_method_config("parquet")
         config.update(kwargs)
@@ -397,21 +400,18 @@ def ingest_arrow(
     if custom_method and custom_method != ingest_arrow:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
 
     try:
-        try:
-            from .arrow_ingestor import ArrowIngestor
-        except ModuleNotFoundError as exc:
-            if _is_missing_dependency(exc, "pyarrow"):
-                raise _missing_optional_dependency(
-                    "Arrow ingestion",
-                    "pyarrow",
-                ) from exc
-            raise
+        from .arrow_ingestor import ArrowIngestor
 
         config = ingest_config.get_method_config("arrow")
         config.update(kwargs)
@@ -481,7 +481,12 @@ def ingest_xml(
     if custom_method and custom_method != ingest_xml:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -491,7 +496,10 @@ def ingest_xml(
 
         config = ingest_config.get_method_config("xml")
         config.update(kwargs)
-        ingestor = XMLIngestor(**config)
+        try:
+            ingestor = XMLIngestor(**config)
+        except ImportError as exc:
+            raise _missing_optional_dependency("XML ingestion", "lxml") from exc
 
         def _run_single(
             path: Union[str, Path],
@@ -511,6 +519,8 @@ def ingest_xml(
 
         return _run_single(source_path)
 
+    except ConfigurationError:
+        raise
     except Exception as e:
         logger.error(f"Failed to ingest XML: {e}")
         raise
@@ -545,7 +555,12 @@ def ingest_web(
     if custom_method and custom_method != ingest_web:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -635,7 +650,12 @@ def ingest_public_api(
     if custom_method and custom_method != ingest_public_api:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -722,7 +742,12 @@ def ingest_feed(
     if custom_method and custom_method != ingest_feed:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -791,7 +816,12 @@ def ingest_stream(
     if custom_method and custom_method != ingest_stream:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -868,26 +898,29 @@ def ingest_repository(
     if custom_method and custom_method != ingest_repository:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
 
     try:
-        try:
-            from .repo_ingestor import RepoIngestor
-        except ModuleNotFoundError as exc:
-            if _is_missing_dependency(exc, "git"):
-                raise _missing_optional_dependency(
-                    "Repository ingestion", "GitPython"
-                ) from exc
-            raise
+        from .repo_ingestor import RepoIngestor
 
         # Get config
         config = ingest_config.get_method_config("repo")
         config.update(kwargs)
 
-        ingestor = RepoIngestor(**config)
+        try:
+            ingestor = RepoIngestor(**config)
+        except ImportError as exc:
+            raise _missing_optional_dependency(
+                "Repository ingestion", "GitPython"
+            ) from exc
 
         if method == "clone" or (
             isinstance(source, str)
@@ -940,7 +973,12 @@ def ingest_email(
     if custom_method and custom_method != ingest_email:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -1019,7 +1057,12 @@ def ingest_ontology(
     if custom_method and custom_method != ingest_ontology:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -1085,7 +1128,12 @@ def ingest_database(
         if custom_method and custom_method != ingest_database:
             fallback = kwargs.pop("fallback_on_custom_error", False)
             result = call_custom_method(
-                logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+                logger,
+                method,
+                custom_method,
+                source,
+                fallback_on_custom_error=fallback,
+                **kwargs,
             )
             if result is not CUSTOM_METHOD_FELL_BACK:
                 return result
@@ -1233,105 +1281,122 @@ def ingest_salesforce(
     if custom_method and custom_method != ingest_salesforce:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source,
-            fallback_on_custom_error=fallback, **kwargs,
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
 
     try:
         from .salesforce_ingestor import SalesforceIngestor
-    except ModuleNotFoundError as exc:
-        if _is_missing_dependency(exc, "simple_salesforce"):
+
+        # Unpack credential dict (if given); everything else stays in kwargs.
+        creds: Dict[str, Any] = {}
+        if source is not None:
+            if not isinstance(source, dict):
+                raise ProcessingError(
+                    "ingest_salesforce() source must be a credential dict or None. "
+                    "Pass sobject_name / soql as keyword arguments."
+                )
+            creds = dict(source)
+
+        # Merge any ingest_config method config under "salesforce".
+        # get_method_config() now returns a copy, so this dict is safe to mutate.
+        # We build the final connector config in order of increasing priority:
+        #   1. base method config (lowest — global defaults set by operator)
+        #   2. per-call credential dict supplied via `source`
+        #   3. per-call connector params supplied as kwargs
+        # Credentials are extracted from kwargs and removed so they don't also
+        # flow into the ingest method call (which doesn't understand them).
+        _CONNECTOR_PARAMS = frozenset(
+            {
+                "username",
+                "password",
+                "security_token",
+                "domain",
+                "instance_url",
+                "session_id",
+                "api_version",
+            }
+        )
+        connector_kwargs = {k: v for k, v in kwargs.items() if k in _CONNECTOR_PARAMS}
+        for k in _CONNECTOR_PARAMS:
+            kwargs.pop(k, None)
+
+        # Build a fresh per-call config dict — never mutate the global store.
+        config: Dict[str, Any] = {
+            **ingest_config.get_method_config("salesforce"),  # base (already a copy)
+            **creds,  # source dict credentials
+            **connector_kwargs,  # kwarg credentials
+        }
+
+        try:
+            ingestor = SalesforceIngestor(**config)
+        except ImportError as exc:
             raise _missing_optional_dependency(
                 "Salesforce ingestion", "simple-salesforce"
             ) from exc
+
+        if method == "sobject":
+            sobject_name = kwargs.pop("sobject_name", None)
+            if not sobject_name:
+                raise ProcessingError(
+                    "ingest_salesforce() with method='sobject' requires "
+                    "sobject_name keyword argument."
+                )
+            return ingestor.ingest_sobject(sobject_name, **kwargs)
+
+        elif method == "query":
+            soql = kwargs.pop("soql", None)
+            if not soql:
+                raise ProcessingError(
+                    "ingest_salesforce() with method='query' requires "
+                    "soql keyword argument."
+                )
+            return ingestor.ingest_query(soql, **kwargs)
+
+        elif method == "list_sobjects":
+            return ingestor.list_sobjects()
+
+        elif method == "schema":
+            sobject_name = kwargs.pop("sobject_name", None)
+            if not sobject_name:
+                raise ProcessingError(
+                    "ingest_salesforce() with method='schema' requires "
+                    "sobject_name keyword argument."
+                )
+            return ingestor.get_sobject_schema(sobject_name)
+
+        elif method == "documents":
+            sobject_name = kwargs.pop("sobject_name", None)
+            if not sobject_name:
+                raise ProcessingError(
+                    "ingest_salesforce() with method='documents' requires "
+                    "sobject_name keyword argument."
+                )
+            id_field = kwargs.pop("id_field", "Id")
+            text_fields = kwargs.pop("text_fields", None)
+            data = ingestor.ingest_sobject(sobject_name, **kwargs)
+            return ingestor.export_as_documents(
+                data, id_field=id_field, text_fields=text_fields
+            )
+
+        else:
+            raise ProcessingError(
+                f"Unknown ingest_salesforce method: {method!r}. "
+                "Valid methods: 'sobject', 'query', 'list_sobjects', 'schema', "
+                "'documents'."
+            )
+
+    except ConfigurationError:
+        raise
+    except Exception as e:
+        logger.error(f"Failed to ingest salesforce: {e}")
         raise
-
-    # Unpack credential dict (if given); everything else stays in kwargs.
-    creds: Dict[str, Any] = {}
-    if source is not None:
-        if not isinstance(source, dict):
-            raise ProcessingError(
-                "ingest_salesforce() source must be a credential dict or None. "
-                "Pass sobject_name / soql as keyword arguments."
-            )
-        creds = dict(source)
-
-    # Merge any ingest_config method config under "salesforce".
-    # get_method_config() now returns a copy, so this dict is safe to mutate.
-    # We build the final connector config in order of increasing priority:
-    #   1. base method config (lowest — global defaults set by operator)
-    #   2. per-call credential dict supplied via `source`
-    #   3. per-call connector params supplied as kwargs
-    # Credentials are extracted from kwargs and removed so they don't also
-    # flow into the ingest method call (which doesn't understand them).
-    _CONNECTOR_PARAMS = frozenset({
-        "username", "password", "security_token", "domain",
-        "instance_url", "session_id", "api_version",
-    })
-    connector_kwargs = {k: v for k, v in kwargs.items() if k in _CONNECTOR_PARAMS}
-    for k in _CONNECTOR_PARAMS:
-        kwargs.pop(k, None)
-
-    # Build a fresh per-call config dict — never mutate the global store.
-    config: Dict[str, Any] = {
-        **ingest_config.get_method_config("salesforce"),  # base (already a copy)
-        **creds,                                           # source dict credentials
-        **connector_kwargs,                                # kwarg credentials
-    }
-
-    ingestor = SalesforceIngestor(**config)
-
-    if method == "sobject":
-        sobject_name = kwargs.pop("sobject_name", None)
-        if not sobject_name:
-            raise ProcessingError(
-                "ingest_salesforce() with method='sobject' requires "
-                "sobject_name keyword argument."
-            )
-        return ingestor.ingest_sobject(sobject_name, **kwargs)
-
-    elif method == "query":
-        soql = kwargs.pop("soql", None)
-        if not soql:
-            raise ProcessingError(
-                "ingest_salesforce() with method='query' requires "
-                "soql keyword argument."
-            )
-        return ingestor.ingest_query(soql, **kwargs)
-
-    elif method == "list_sobjects":
-        return ingestor.list_sobjects()
-
-    elif method == "schema":
-        sobject_name = kwargs.pop("sobject_name", None)
-        if not sobject_name:
-            raise ProcessingError(
-                "ingest_salesforce() with method='schema' requires "
-                "sobject_name keyword argument."
-            )
-        return ingestor.get_sobject_schema(sobject_name)
-
-    elif method == "documents":
-        sobject_name = kwargs.pop("sobject_name", None)
-        if not sobject_name:
-            raise ProcessingError(
-                "ingest_salesforce() with method='documents' requires "
-                "sobject_name keyword argument."
-            )
-        id_field = kwargs.pop("id_field", "Id")
-        text_fields = kwargs.pop("text_fields", None)
-        data = ingestor.ingest_sobject(sobject_name, **kwargs)
-        return ingestor.export_as_documents(data, id_field=id_field,
-                                            text_fields=text_fields)
-
-    else:
-        raise ProcessingError(
-            f"Unknown ingest_salesforce method: {method!r}. "
-            "Valid methods: 'sobject', 'query', 'list_sobjects', 'schema', "
-            "'documents'."
-        )
 
 
 def ingest_mcp(
@@ -1399,7 +1464,12 @@ def ingest_mcp(
     if custom_method and custom_method != ingest_mcp:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            source,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -1638,8 +1708,9 @@ def ingest(
     elif source_type == "mcp":
         return {"data": ingest_mcp(sources, method=method or "resources", **kwargs)}
     elif source_type == "salesforce":
-        return {"data": ingest_salesforce(sources,
-                                          method=method or "sobject", **kwargs)}
+        return {
+            "data": ingest_salesforce(sources, method=method or "sobject", **kwargs)
+        }
     else:
         raise ProcessingError(f"Unknown source type: {source_type}")
 

semantica/ingest/public_api_ingestor.py

@@ -28,12 +28,29 @@ from typing import Any, Dict, List, Optional, Tuple
 from urllib.parse import parse_qs, urlparse
 
 import requests
-from lxml import etree as lxml_etree
+
+try:
+    from lxml import etree as lxml_etree
+
+    _SAFE_XML_PARSER = lxml_etree.XMLParser(
+        resolve_entities=False,
+        no_network=True,
+        recover=False,
+        huge_tree=False,
+        load_dtd=False,
+        remove_comments=True,
+        remove_pis=True,
+    )
+    _LXML_SYNTAX_ERRORS: Tuple[type, ...] = (lxml_etree.XMLSyntaxError,)
+except (ImportError, OSError):
+    lxml_etree = None
+    _SAFE_XML_PARSER = None
+    _LXML_SYNTAX_ERRORS = ()
 
 try:
     from defusedxml import ElementTree as safe_xml_etree
     from defusedxml.common import DefusedXmlException
-except ModuleNotFoundError:  # pragma: no cover - fallback for minimal installs
+except (ImportError, OSError):  # pragma: no cover - fallback for minimal installs
     safe_xml_etree = None
 
     class DefusedXmlException(Exception):
@@ -71,14 +88,6 @@ AUTH_PARAM_NAMES = {
     "subscription-key",
 }
 
-_SAFE_XML_PARSER = lxml_etree.XMLParser(
-    resolve_entities=False,
-    no_network=True,
-    recover=False,
-    huge_tree=False,
-    load_dtd=False,
-)
-
 
 @dataclass
 class PublicAPIExample:
@@ -443,7 +452,9 @@ class PublicAPIIngestor(RESTIngestor):
             APIData: Normalized public API response and metadata
         """
         self._validate_endpoint(endpoint)
-        self._validate_no_auth_request(headers=headers, params=params, options=options, endpoint=endpoint)
+        self._validate_no_auth_request(
+            headers=headers, params=params, options=options, endpoint=endpoint
+        )
 
         tracking_id = self.progress_tracker.start_tracking(
             file=endpoint,
@@ -604,7 +615,9 @@ class PublicAPIIngestor(RESTIngestor):
         for endpoint in endpoints:
             try:
                 results.append(
-                    self.ingest_public_api(endpoint, method=method, **copy.deepcopy(options))
+                    self.ingest_public_api(
+                        endpoint, method=method, **copy.deepcopy(options)
+                    )
                 )
             except Exception as exc:
                 self.logger.warning(f"Failed to fetch public API {endpoint}: {exc}")
@@ -730,7 +743,7 @@ class PublicAPIIngestor(RESTIngestor):
             raise ProcessingError(
                 f"Failed to parse {detected_format.upper()} public API response"
             ) from exc
-        except (DefusedXmlException, lxml_etree.XMLSyntaxError) as exc:
+        except (DefusedXmlException, *_LXML_SYNTAX_ERRORS) as exc:
             raise ProcessingError("Failed to parse XML public API response") from exc
 
     def _detect_response_format(
@@ -770,15 +783,40 @@ class PublicAPIIngestor(RESTIngestor):
     def _parse_xml(self, xml_text: str) -> Dict[str, Any]:
         if safe_xml_etree is not None:
             root = safe_xml_etree.fromstring(xml_text)
-        else:
+        elif lxml_etree is not None and _SAFE_XML_PARSER is not None:
             root = lxml_etree.fromstring(
                 xml_text.encode("utf-8"),
                 parser=_SAFE_XML_PARSER,
             )
+            for elem in root.iter():
+                if (
+                    elem.tag is lxml_etree.Comment
+                    or elem.tag is lxml_etree.PI
+                    or getattr(elem.tag, "__name__", "")
+                    in ("Comment", "ProcessingInstruction", "PI")
+                ):
+                    continue
+                if callable(elem.tag) or not isinstance(elem.tag, str):
+                    raise ProcessingError("Failed to parse XML public API response")
+        else:
+            raise ProcessingError(
+                "XML parsing requires 'defusedxml' or 'lxml'. "
+                "Install it with: pip install 'semantica[documents]'"
+            )
         return self._element_to_dict(root)
 
     def _element_to_dict(self, element: Any) -> Dict[str, Any]:
-        children = [self._element_to_dict(child) for child in list(element)]
+        children = [
+            self._element_to_dict(child)
+            for child in list(element)
+            if not (
+                callable(child.tag)
+                or (
+                    lxml_etree is not None
+                    and (child.tag is lxml_etree.Comment or child.tag is lxml_etree.PI)
+                )
+            )
+        ]
         return {
             "tag": self._strip_namespace(element.tag),
             "attributes": {
@@ -789,7 +827,9 @@ class PublicAPIIngestor(RESTIngestor):
             "children": children,
         }
 
-    def _strip_namespace(self, value: str) -> str:
+    def _strip_namespace(self, value: Any) -> str:
+        if not isinstance(value, str):
+            return str(value)
         if value.startswith("{") and "}" in value:
             return value.split("}", 1)[1]
         return value

semantica/ingest/repo_ingestor.py

@@ -29,6 +29,8 @@ Author: Semantica Contributors
 License: MIT
 """
 
+from __future__ import annotations
+
 import ipaddress
 import os
 import re
@@ -44,7 +46,10 @@ from pathlib import Path
 from typing import Any, Dict, List, Optional, Set, Tuple, Union
 from urllib.parse import urlparse
 
-import git
+try:
+    import git
+except (ImportError, OSError):
+    git = None
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
@@ -57,9 +62,7 @@ ALLOWED_CLONE_OPTIONS: Set[str] = {"depth", "branch", "single_branch", "no_tags"
 ALLOWED_REPO_URL_SCHEMES = frozenset({"https", "http", "git", "ssh"})
 # SCP-like SSH remotes: user@host:path/to/repo.git (no scheme)
 _SCP_LIKE_REPO_URL_RE = re.compile(r"^[^@\s]+@[^:\s]+:.+$")
-_ENV_VAR_TOKEN_RE = re.compile(
-    r"\$(\{[A-Za-z_][A-Za-z0-9_]*\}|[A-Za-z_][A-Za-z0-9_]*)"
-)
+_ENV_VAR_TOKEN_RE = re.compile(r"\$(\{[A-Za-z_][A-Za-z0-9_]*\}|[A-Za-z_][A-Za-z0-9_]*)")
 # Short-lived DNS cache for host validation. This reduces repeated lookups but
 # does not eliminate DNS-rebinding / TOCTOU races between validate and clone —
 # network egress controls remain recommended.
@@ -525,6 +528,11 @@ class RepoIngestor:
             **kwargs: Additional configuration parameters (merged into config)
         """
         self.logger = get_logger("repo_ingestor")
+        if git is None:
+            raise ImportError(
+                "GitPython is required for repository ingestion. "
+                "Install it with: pip install 'semantica[ingest-git]'"
+            )
         self.config = config or {}
         self.config.update(kwargs)
 
@@ -590,10 +598,7 @@ class RepoIngestor:
         networks). Those addresses are not SSRF-sensitive.
         """
         return bool(
-            ip.is_private
-            or ip.is_loopback
-            or ip.is_link_local
-            or ip.is_unspecified
+            ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_unspecified
         )
 
     @staticmethod
@@ -620,9 +625,7 @@ class RepoIngestor:
         # or hanging lookup for one host cannot stall cache access for
         # concurrent lookups of other hosts.
         try:
-            addrinfos = socket.getaddrinfo(
-                host, None, type=socket.SOCK_STREAM
-            )
+            addrinfos = socket.getaddrinfo(host, None, type=socket.SOCK_STREAM)
         except socket.gaierror as exc:
             raise ValidationError(
                 f"Cannot resolve repository host {host!r}: {exc}"
@@ -795,9 +798,7 @@ class RepoIngestor:
                 f"Allowed schemes: {sorted(ALLOWED_REPO_URL_SCHEMES)}"
             )
         if not parsed.netloc or not host:
-            raise ValidationError(
-                f"Repository URL must include a host: {repo_url}"
-            )
+            raise ValidationError(f"Repository URL must include a host: {repo_url}")
 
         RepoIngestor._validate_repo_host(host)
 
@@ -812,9 +813,7 @@ class RepoIngestor:
             "include_extensions",
             "max_depth",
         }
-        candidate = {
-            k: v for k, v in options.items() if k not in non_git_options
-        }
+        candidate = {k: v for k, v in options.items() if k not in non_git_options}
         unsafe = set(candidate) - ALLOWED_CLONE_OPTIONS
         if unsafe:
             raise ValidationError(
@@ -893,9 +892,7 @@ class RepoIngestor:
             if "include_extensions" in options:
                 # Normalize extensions to include dot prefix
                 exts = options["include_extensions"]
-                normalized_exts = [
-                    e if e.startswith(".") else f".{e}" for e in exts
-                ]
+                normalized_exts = [e if e.startswith(".") else f".{e}" for e in exts]
                 file_filters["extensions"] = normalized_exts
 
             # Process code files
@@ -1062,14 +1059,14 @@ class RepoIngestor:
         return code_files
 
     def get_repository_info(
-        self, repo_url: str, repo: Optional[git.Repo] = None
+        self, repo_url: str, repo: Optional[Any] = None
     ) -> Dict[str, Any]:
         """
         Get repository metadata and information.
 
         Args:
             repo_url: Repository URL
-            repo: Git repository object (optional)
+            repo: Git repository object (git.Repo, optional)
 
         Returns:
             dict: Repository information
@@ -1111,6 +1108,7 @@ class RepoIngestor:
     def cleanup(self):
         """Cleanup temporary repository files."""
         if self.temp_dir and os.path.exists(self.temp_dir):
+
             def onexc(func, path, exc_info):
                 """
                 Error handler for shutil.rmtree.
@@ -1123,6 +1121,7 @@ class RepoIngestor:
                 Usage : shutil.rmtree(path, onerror=onexc)
                 """
                 import stat
+
                 if not os.access(path, os.W_OK):
                     # Is the error an access error ?
                     os.chmod(path, stat.S_IWUSR)

semantica/ingest/xml_ingestor.py

@@ -24,7 +24,10 @@ from datetime import datetime
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple, Union
 
-from lxml import etree
+try:
+    from lxml import etree
+except (ImportError, OSError):
+    etree = None
 
 from ..utils.constants import FILE_SIZE_LIMITS
 from ..utils.exceptions import ProcessingError, ValidationError
@@ -72,6 +75,11 @@ class XMLIngestor:
             **kwargs: Additional configuration values
         """
         self.logger = get_logger("xml_ingestor")
+        if etree is None:
+            raise ImportError(
+                "lxml is required for XMLIngestor. "
+                "Install it with: pip install 'semantica[documents]'"
+            )
         self.config = config or {}
         self.config.update(kwargs)
         self.progress_tracker = get_progress_tracker()
@@ -784,8 +792,8 @@ class XMLIngestor:
         first_error = errors[0] if errors else "No detailed validation error available."
         return f"{prefix} for {source}: {first_error}"
 
-    def _format_xml_error(self, exc: etree.XMLSyntaxError) -> str:
-        if exc.error_log:
+    def _format_xml_error(self, exc: Any) -> str:
+        if hasattr(exc, "error_log") and exc.error_log:
             return str(exc.error_log.last_error)
         return str(exc)
 

semantica/kg/node_embeddings.py

@@ -141,7 +141,7 @@ class NodeEmbedder:
 
         if method == "node2vec" and not GENSIM_AVAILABLE:
             raise ImportError(
-                "gensim is required for Node2Vec. Install with: pip install gensim"
+                "gensim is required for Node2Vec. Install with: pip install 'semantica[graph-embeddings]'"
             )
     
     def compute_embeddings(

semantica/parse/docx_parser.py

@@ -32,12 +32,20 @@ from dataclasses import dataclass, field
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Union
 
-from docx import Document
-from docx.document import Document as DocxDocument
-from docx.oxml.table import CT_Tbl
-from docx.oxml.text.paragraph import CT_P
-from docx.table import Table
-from docx.text.paragraph import Paragraph
+try:
+    from docx import Document
+    from docx.document import Document as DocxDocument
+    from docx.oxml.table import CT_Tbl
+    from docx.oxml.text.paragraph import CT_P
+    from docx.table import Table
+    from docx.text.paragraph import Paragraph
+except (ImportError, OSError):
+    Document = None
+    DocxDocument = None
+    CT_Tbl = None
+    CT_P = None
+    Table = None
+    Paragraph = None
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
@@ -84,7 +92,9 @@ class DOCXParser:
         self.config = config
         self.progress_tracker = get_progress_tracker()
 
-    def parse(self, file_path: Union[str, Path], pipeline_id: Optional[str] = None, **options) -> Dict[str, Any]:
+    def parse(
+        self, file_path: Union[str, Path], pipeline_id: Optional[str] = None, **options
+    ) -> Dict[str, Any]:
         """
         Parse DOCX document.
 
@@ -99,6 +109,12 @@ class DOCXParser:
         Returns:
             dict: Parsed document data
         """
+        if Document is None:
+            raise ProcessingError(
+                "python-docx is required to parse DOCX files. "
+                "Install it with: pip install 'semantica[documents]'"
+            )
+
         file_path = Path(file_path)
 
         # Track DOCX parsing

semantica/parse/excel_parser.py

@@ -33,7 +33,11 @@ from pathlib import Path
 from typing import Any, Dict, List, Optional, Union
 
 import pandas as pd
-from openpyxl import load_workbook
+
+try:
+    from openpyxl import load_workbook
+except (ImportError, OSError):
+    load_workbook = None
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
@@ -92,6 +96,12 @@ class ExcelParser:
         Returns:
             ExcelData or ExcelSheet: Parsed Excel data
         """
+        if load_workbook is None:
+            raise ProcessingError(
+                "openpyxl is required to parse Excel files. "
+                "Install it with: pip install 'semantica[documents]'"
+            )
+
         file_path = Path(file_path)
 
         # Track Excel parsing

semantica/parse/html_parser.py

@@ -33,7 +33,10 @@ from pathlib import Path
 from typing import Any, Dict, List, Optional, Union
 from urllib.parse import urljoin
 
-from bs4 import BeautifulSoup
+try:
+    from bs4 import BeautifulSoup
+except (ImportError, OSError):
+    BeautifulSoup = None
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
@@ -113,6 +116,12 @@ class HTMLParser:
         Returns:
             HTMLData: Parsed HTML data
         """
+        if BeautifulSoup is None:
+            raise ProcessingError(
+                "beautifulsoup4 is required to parse HTML files. "
+                "Install it with: pip install 'semantica[documents]'"
+            )
+
         # Track HTML parsing
         file_path = None
         if isinstance(html_content, Path) or (

semantica/parse/methods.py

@@ -123,7 +123,6 @@ Example Usage:
 from pathlib import Path
 from typing import Any, Callable, Dict, List, Optional, Union
 
-from ..utils.exceptions import ConfigurationError, ProcessingError
 from ..utils.logging import get_logger
 from ..utils.custom_methods import CUSTOM_METHOD_FELL_BACK, call_custom_method
 from .code_parser import CodeParser
@@ -178,10 +177,16 @@ def parse_document(
         >>> text = parse_document("document.pdf", method="default", extract_text=True)
     """
     custom_method = method_registry.get("document", method)
-    if custom_method:
+    if custom_method and custom_method != parse_document:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, file_path, file_type, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            file_path,
+            file_type,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -248,7 +253,8 @@ def parse_document_docling(
 
 # Register Docling method
 try:
-    from .docling_parser import DoclingParser
+    from . import docling_parser  # noqa: F401
+
     method_registry.register("document", "docling", parse_document_docling)
 except (ImportError, OSError):
     # Docling not available, skip registration
@@ -289,10 +295,17 @@ def parse_web_content(
         >>> html = parse_web_content("page.html", content_type="html", method="default")
     """
     custom_method = method_registry.get("web", method)
-    if custom_method:
+    if custom_method and custom_method != parse_web_content:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, content, content_type, base_url, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            content,
+            content_type,
+            base_url,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -345,10 +358,16 @@ def parse_structured_data(
         >>> csv_data = parse_structured_data("data.csv", data_format="csv", method="default")
     """
     custom_method = method_registry.get("structured", method)
-    if custom_method:
+    if custom_method and custom_method != parse_structured_data:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, data, data_format, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            data,
+            data_format,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -392,10 +411,15 @@ def parse_email(
         >>> headers = parse_email("email.eml", method="headers")
     """
     custom_method = method_registry.get("email", method)
-    if custom_method:
+    if custom_method and custom_method != parse_email:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, email_content, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            email_content,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -442,10 +466,16 @@ def parse_code(
         >>> structure = parse_code("script.py", method="ast")
     """
     custom_method = method_registry.get("code", method)
-    if custom_method:
+    if custom_method and custom_method != parse_code:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, file_path, language, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            file_path,
+            language,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -495,10 +525,16 @@ def parse_media(
         >>> video = parse_media("video.mp4", method="default")
     """
     custom_method = method_registry.get("media", method)
-    if custom_method:
+    if custom_method and custom_method != parse_media:
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, file_path, media_type, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            file_path,
+            media_type,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -541,10 +577,15 @@ def parse_pdf(
         >>> pages = parse_pdf("document.pdf", method="default", pages=[1, 2, 3])
     """
     custom_method = method_registry.get("document", method)
-    if custom_method:
+    if custom_method and custom_method not in (parse_pdf, parse_document):
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            file_path,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -585,10 +626,15 @@ def parse_docx(
         >>> docx = parse_docx("document.docx", method="default")
     """
     custom_method = method_registry.get("document", method)
-    if custom_method:
+    if custom_method and custom_method not in (parse_docx, parse_document):
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            file_path,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -628,10 +674,15 @@ def parse_json(file_path: Union[str, Path], method: str = "default", **kwargs) -
         >>> flattened = parse_json("data.json", method="default", flatten=True)
     """
     custom_method = method_registry.get("structured", method)
-    if custom_method:
+    if custom_method and custom_method not in (parse_json, parse_structured_data):
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            file_path,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -675,10 +726,16 @@ def parse_csv(
         >>> tab_separated = parse_csv("data.tsv", delimiter="\t", method="default")
     """
     custom_method = method_registry.get("structured", method)
-    if custom_method:
+    if custom_method and custom_method not in (parse_csv, parse_structured_data):
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, file_path, delimiter, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            file_path,
+            delimiter,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -714,10 +771,15 @@ def parse_xml(file_path: Union[str, Path], method: str = "default", **kwargs) ->
         >>> xml_data = parse_xml("data.xml", method="default")
     """
     custom_method = method_registry.get("structured", method)
-    if custom_method:
+    if custom_method and custom_method not in (parse_xml, parse_structured_data):
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            file_path,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result
@@ -762,10 +824,15 @@ def parse_image(
         >>> ocr_text = image.get("ocr_result", {}).get("text", "")
     """
     custom_method = method_registry.get("media", method)
-    if custom_method:
+    if custom_method and custom_method not in (parse_image, parse_media):
         fallback = kwargs.pop("fallback_on_custom_error", False)
         result = call_custom_method(
-            logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
+            logger,
+            method,
+            custom_method,
+            file_path,
+            fallback_on_custom_error=fallback,
+            **kwargs,
         )
         if result is not CUSTOM_METHOD_FELL_BACK:
             return result

semantica/parse/web_parser.py

@@ -33,7 +33,10 @@ from pathlib import Path
 from typing import Any, Dict, List, Optional, Union
 from urllib.parse import urljoin, urlparse
 
-from bs4 import BeautifulSoup
+try:
+    from bs4 import BeautifulSoup
+except (ImportError, OSError):
+    BeautifulSoup = None
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
@@ -187,6 +190,12 @@ class HTMLContentParser(HTMLParser):
         }
 
         # Load HTML for structure extraction
+        if BeautifulSoup is None:
+            raise ProcessingError(
+                "beautifulsoup4 is required for HTML structure extraction. "
+                "Install it with: pip install 'semantica[documents]'"
+            )
+
         if isinstance(html_content, Path) or (
             isinstance(html_content, str) and Path(html_content).exists()
         ):
@@ -243,6 +252,12 @@ class HTMLContentParser(HTMLParser):
         else:
             html_string = html_content
 
+        if BeautifulSoup is None:
+            raise ProcessingError(
+                "beautifulsoup4 is required for HTML cleaning. "
+                "Install it with: pip install 'semantica[documents]'"
+            )
+
         soup = BeautifulSoup(html_string, "html.parser")
 
         # Remove scripts and styles

semantica/parse/xml_parser.py

@@ -34,7 +34,10 @@ from dataclasses import dataclass, field
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Union
 
-from lxml import etree
+try:
+    from lxml import etree
+except (ImportError, OSError):
+    etree = None
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
@@ -105,7 +108,13 @@ class XMLParser:
         )
 
         try:
-            engine = options.get("engine", "lxml")
+            explicit_engine = options.get("engine") or self.config.get("engine")
+            engine = explicit_engine or ("lxml" if etree is not None else "etree")
+            if engine == "lxml" and etree is None:
+                raise ProcessingError(
+                    "lxml is required to parse XML with engine='lxml'. "
+                    "Install it with: pip install 'semantica[documents]'"
+                )
 
             # Load XML content
             if file_path_obj:
@@ -147,7 +156,7 @@ class XMLParser:
         self, xml_string: str, source: str, options: Dict[str, Any]
     ) -> XMLData:
         """Parse XML using lxml."""
-        parser = etree.XMLParser(remove_blank_text=True)
+        parser = etree.XMLParser(remove_blank_text=True, remove_comments=True)
         root = etree.fromstring(xml_string.encode("utf-8"), parser)
 
         # Extract namespaces
@@ -188,8 +197,10 @@ class XMLParser:
             metadata={"source": source, "engine": "etree"},
         )
 
-    def _element_to_xml_element(self, element) -> XMLElement:
+    def _element_to_xml_element(self, element) -> Optional[XMLElement]:
         """Convert lxml element to XMLElement."""
+        if not hasattr(element, "tag") or not isinstance(element.tag, str):
+            return None
         tag = element.tag
         if "}" in tag:
             namespace, tag = tag.split("}", 1)
@@ -206,12 +217,18 @@ class XMLParser:
 
         # Process children
         for child in element:
-            xml_elem.children.append(self._element_to_xml_element(child))
+            child_elem = self._element_to_xml_element(child)
+            if child_elem is not None:
+                xml_elem.children.append(child_elem)
 
         return xml_elem
 
-    def _etree_element_to_xml_element(self, element: ET.Element) -> XMLElement:
+    def _etree_element_to_xml_element(
+        self, element: ET.Element
+    ) -> Optional[XMLElement]:
         """Convert ElementTree element to XMLElement."""
+        if not hasattr(element, "tag") or not isinstance(element.tag, str):
+            return None
         tag = element.tag
         if "}" in tag:
             namespace, tag = tag.split("}", 1)
@@ -228,7 +245,9 @@ class XMLParser:
 
         # Process children
         for child in element:
-            xml_elem.children.append(self._etree_element_to_xml_element(child))
+            child_elem = self._etree_element_to_xml_element(child)
+            if child_elem is not None:
+                xml_elem.children.append(child_elem)
 
         return xml_elem
 
@@ -249,18 +268,30 @@ class XMLParser:
         xml_data = self.parse(file_path, **options)
 
         # Use lxml for XPath queries
+        if etree is None:
+            raise ProcessingError(
+                "lxml is required for find_elements (XPath queries). "
+                "Install it with: pip install 'semantica[documents]'"
+            )
+
         xml_string = (
             file_path
             if isinstance(file_path, str) and not Path(file_path).exists()
-            else Path(file_path).read_text()
+            else Path(file_path).read_text(encoding="utf-8")
         )
-        root = etree.fromstring(xml_string.encode("utf-8"))
+        parser = etree.XMLParser(remove_blank_text=True, remove_comments=True)
+        root = etree.fromstring(xml_string.encode("utf-8"), parser=parser)
 
         # Register namespaces for XPath
         namespaces = xml_data.namespaces
         elements = root.xpath(xpath, namespaces=namespaces)
 
-        return [self._element_to_xml_element(elem) for elem in elements]
+        results = []
+        for elem in elements:
+            xml_elem = self._element_to_xml_element(elem)
+            if xml_elem is not None:
+                results.append(xml_elem)
+        return results
 
     def extract_by_tag(
         self, file_path: Union[str, Path], tag_name: str, **options

semantica/semantic_extract/methods.py

@@ -209,6 +209,11 @@ def load_spacy_model(name: str):
     Raises whatever ``spacy.load`` raises (``OSError`` for a missing model), so
     callers keep their existing fallback behavior.
     """
+    if spacy is None:
+        raise ImportError(
+            "spaCy is not installed. Install with: pip install 'semantica[nlp-spacy]'"
+        )
+
     cached = _spacy_model_cache.get(name)
     if cached is not None and cached[0] is spacy:
         return cached[1]

semantica/utils/helpers.py

@@ -38,15 +38,15 @@ Example Usage:
     >>> from semantica.utils import clean_text, normalize_entities
     >>> cleaned = clean_text("  Hello   World  ")
     >>> entities = normalize_entities([{"id": "e1", "text": "John", "type": "PERSON"}])
-    >>> 
+    >>>
     >>> from semantica.utils import hash_data, safe_filename
     >>> data_hash = hash_data({"key": "value"})
     >>> safe_name = safe_filename("my file.txt")
-    >>> 
+    >>>
     >>> from semantica.utils import merge_dicts, get_nested_value
     >>> merged = merge_dicts({"a": 1}, {"b": 2}, deep=True)
     >>> value = get_nested_value(config, "database.host", default="localhost")
-    >>> 
+    >>>
     >>> from semantica.utils import retry_on_error
     >>> @retry_on_error(max_retries=3, delay=1.0)
     ... def fetch_data():
@@ -457,7 +457,9 @@ def chunk_list(items: List[Any], chunk_size: int) -> List[List[Any]]:
     Returns:
         List of chunks
     """
-    return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]    
+    return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]
+
+
 def flatten_dict(
     d: Dict[str, Any], parent_key: str = "", sep: str = "."
 ) -> Dict[str, Any]:
@@ -556,27 +558,27 @@ def safe_import(
 ) -> Tuple[Any, bool]:
     """
     Safely import an optional module, handling both ImportError and OSError.
-    
+
     This is useful for optional dependencies that may fail to import due to:
     - Missing package (ImportError)
     - DLL loading failures on Windows, e.g., PyTorch (OSError)
-    
+
     Args:
         module_name: Name of the module to import (e.g., "spacy", "docling.document_converter")
         package: Optional package name for relative imports
         default: Default value to return if import fails
         error_message: Optional custom error message for logging
-        
+
     Returns:
         Tuple of (module_or_default, success_flag):
         - If import succeeds: (imported_module, True)
         - If import fails: (default, False)
-        
+
     Example:
         >>> spacy, available = safe_import("spacy")
         >>> if available:
         ...     doc = spacy.load("en_core_web_sm")
-        >>> 
+        >>>
         >>> converter, available = safe_import("docling.document_converter", default=None)
         >>> if available:
         ...     converter = converter()
@@ -587,11 +589,13 @@ def safe_import(
         else:
             module = importlib.import_module(module_name)
         return module, True
-    except (ImportError, ModuleNotFoundError, OSError) as e:
+    except (ImportError, OSError) as e:
         if error_message:
             import sys
+
             if "logging" in sys.modules:
                 from .logging import get_logger
+
                 logger = get_logger("utils.helpers")
                 logger.debug(f"{error_message}: {e}")
         return default, False
@@ -807,9 +811,13 @@ def _is_record(value: Any) -> bool:
     exporters rather than a ``ValidationError`` at the boundary where the
     problem is visible.
     """
-    return isinstance(value, Mapping) or is_dataclass(value) or (
-        hasattr(value, "__dict__")
-        and not isinstance(value, (types.ModuleType, type))
+    return (
+        isinstance(value, Mapping)
+        or is_dataclass(value)
+        or (
+            hasattr(value, "__dict__")
+            and not isinstance(value, (types.ModuleType, type))
+        )
     )
 
 

semantica/vector_store/faiss_store.py

@@ -128,6 +128,11 @@ class FAISSIndex:
         self.index_type = index_type
         self.vector_ids: List[str] = []
         self.metadata: Dict[str, Dict[str, Any]] = {}
+        # Monotonic counter for default ID generation, mirroring FAISSStore._next_id.
+        # Persisted in the .meta.json sidecar so that load_index restores the
+        # correct value rather than deriving it from ntotal (which underestimates
+        # when vectors have been deleted and sparse gaps exist).
+        self.next_id: int = 0
 
     def add_vectors(self, vectors: np.ndarray, ids: Optional[List[str]] = None):
         """
@@ -199,6 +204,98 @@ class FAISSIndex:
         """Get metadata by ID."""
         return self.metadata.get(vector_id)
 
+    def delete_vectors(self, vector_ids_to_delete: List[str]) -> Dict[str, Any]:
+        """Remove vectors by their external string IDs.
+
+        Translates each requested external ID to its sequential internal FAISS
+        position, calls ``index.remove_ids`` with an ``IDSelectorBatch`` of
+        those positions, then updates ``vector_ids`` and ``metadata`` to match
+        the compacted index.  The invariant ``len(self.vector_ids) ==
+        self.index.ntotal`` is re-checked after the operation.
+
+        **Persistence:** the deletion is in-memory only.  Call
+        :meth:`FAISSStore.save_index` afterwards to write the updated state to
+        disk; without that call the deleted vectors will reappear on the next
+        process restart.
+
+        Args:
+            vector_ids_to_delete: External string IDs to remove.  Unknown IDs
+                are silently ignored.  Duplicate entries are deduplicated.
+
+        Returns:
+            ``{"delete_count": N}`` where *N* is the number of vectors
+            actually removed from the FAISS index (0 if none existed).
+
+        Raises:
+            NotImplementedError: If the underlying FAISS index type does not
+                support ``remove_ids`` (e.g. ``IndexHNSWFlat``).  No state is
+                mutated before this is raised.
+            ProcessingError: For any other unexpected FAISS error.
+        """
+        if not vector_ids_to_delete:
+            return {"delete_count": 0}
+
+        delete_set = set(vector_ids_to_delete)
+
+        # Map external string IDs to sequential internal FAISS positions.
+        positions = [
+            pos
+            for pos, vid in enumerate(self.vector_ids)
+            if vid in delete_set
+        ]
+        if not positions:
+            return {"delete_count": 0}
+
+        # IVF-family indices (IndexIVFFlat, etc.) do NOT compact their internal
+        # labels after remove_ids: the surviving vectors keep their original
+        # sequential labels.  The current architecture interprets search-result
+        # labels as offsets into vector_ids, so a non-compacting removal would
+        # silently return wrong external IDs and cause IndexError on labels
+        # beyond the compacted list length.  Raise NotImplementedError here so
+        # callers get STATUS_UNSUPPORTED rather than silent data corruption.
+        # (Flat and PQ indices DO compact labels, so they are safe.)
+        if FAISS_AVAILABLE and isinstance(self.index, faiss.IndexIVF):
+            raise NotImplementedError(
+                f"The underlying FAISS index type ({type(self.index).__name__}) "
+                "does not compact internal labels after remove_ids, which would "
+                "desynchronize search labels from the vector_ids mapping.  Use a "
+                "Flat index for deletion support, or rebuild the IVF index without "
+                "the deleted vectors."
+            )
+
+        sel = faiss.IDSelectorBatch(np.array(positions, dtype=np.int64))
+        try:
+            removed = self.index.remove_ids(sel)
+        except RuntimeError as exc:
+            if "not implemented" in str(exc).lower():
+                # HNSW and a handful of other index types do not implement
+                # remove_ids.  Raise NotImplementedError so callers (and the
+                # ErasureCoordinator) can distinguish "unsupported" from a
+                # transient failure worth retrying.
+                raise NotImplementedError(
+                    f"The underlying FAISS index type "
+                    f"({type(self.index).__name__}) does not support "
+                    "remove_ids().  Use a Flat index for deletion support, "
+                    "or rebuild the index without the deleted vectors."
+                ) from exc
+            raise ProcessingError(f"FAISS remove_ids failed: {exc}") from exc
+
+        # Keep state consistent: update the Python-side list and metadata
+        # dict to mirror the now-compacted FAISS array.  The list comprehension
+        # cannot raise, so the index and its metadata are always updated
+        # together (no partial-mutation window).
+        self.vector_ids = [vid for vid in self.vector_ids if vid not in delete_set]
+        for vid in delete_set:
+            self.metadata.pop(vid, None)
+
+        if len(self.vector_ids) != self.index.ntotal:
+            raise ProcessingError(
+                f"FAISSIndex invariant broken after delete_vectors: "
+                f"vector_ids={len(self.vector_ids)}, ntotal={self.index.ntotal}. "
+                "This indicates a bug in FAISS remove_ids or the deletion logic."
+            )
+        return {"delete_count": removed}
+
     def save(self, path: Union[str, Path]):
         """Save index to disk.
 
@@ -223,6 +320,7 @@ class FAISSIndex:
                 "metadata": self.metadata,
                 "dimension": self.dimension,
                 "index_type": self.index_type,
+                "next_id": self.next_id,
             },
             cls=_LosslessJSONEncoder,
         )
@@ -245,7 +343,9 @@ class FAISSIndex:
         was originally saved.
         """
         if not FAISS_AVAILABLE:
-            raise ProcessingError("FAISS not available")
+            raise ProcessingError(
+                "FAISS not available. Install with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
+            )
 
         path = Path(path)
         index = faiss.read_index(str(path))
@@ -262,6 +362,12 @@ class FAISSIndex:
             if persisted_index_type is not None:
                 index_type = persisted_index_type
 
+            # Restore the monotonic ID counter.  Older sidecar files written
+            # before this field was added will not have the key; fall back to
+            # ntotal, which equals the counter value for stores that have never
+            # had a deletion (no gaps in label space).
+            persisted_next_id = data.get("next_id")
+
             # Check for vector count vs sidecar ID count mismatch
             if len(vector_ids) != index.ntotal:
                 raise ProcessingError(
@@ -279,10 +385,28 @@ class FAISSIndex:
             )
             vector_ids = []
             metadata = {}
+            persisted_next_id = None
 
         obj = cls(index, dimension, index_type)
         obj.vector_ids = vector_ids
         obj.metadata = metadata
+        # Restore the monotonic counter.  Always clamp to at least the
+        # highest inferred vec_N ID, so a stale or corrupted persisted value
+        # (e.g. written before a deletion that shifted the gap) cannot cause
+        # future default IDs to collide with existing vector IDs.
+        _vec_nums = [
+            int(v[4:]) + 1
+            for v in vector_ids
+            if v.startswith("vec_") and v[4:].isdigit()
+        ]
+        _inferred = max(_vec_nums) if _vec_nums else index.ntotal
+        if persisted_next_id is not None:
+            # Trust the persisted value but never go below the inferred minimum
+            # (guards against stale/corrupted sidecars).
+            obj.next_id = max(int(persisted_next_id), _inferred)
+        else:
+            # Older sidecar files lack this field.  Use the inferred value.
+            obj.next_id = _inferred
         return obj
 
 
@@ -315,7 +439,7 @@ class FAISSSearch:
 
         results = []
         for i, (dist, idx) in enumerate(zip(distances[0], indices[0])):
-            if idx < len(self.index.vector_ids):
+            if idx < len(self.index.vector_ids) and idx >= 0:
                 vector_id = self.index.vector_ids[idx]
                 dist_val = float(dist)
 
@@ -360,7 +484,7 @@ class FAISSIndexBuilder:
         """
         if not FAISS_AVAILABLE:
             raise ProcessingError(
-                "FAISS is not available. Install it with: pip install faiss-cpu or faiss-gpu"
+                "FAISS is not available. Install it with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
             )
 
         # Create index based on type
@@ -422,11 +546,17 @@ class FAISSStore:
         self.index: Optional[FAISSIndex] = None
         self.index_builder = FAISSIndexBuilder(dimension)
         self.search_engine: Optional[FAISSSearch] = None
+        # Path remembered by load_index so delete_vectors can auto-save.
+        self._index_path: Optional[Path] = None
+        # Monotonic counter for default ID generation.  Incremented on every
+        # successful add, never decremented on deletion, so ids generated by
+        # consecutive add_vectors calls can never collide with surviving IDs.
+        self._next_id: int = 0
 
         # Check FAISS availability
         if not FAISS_AVAILABLE:
             self.logger.warning(
-                "FAISS not available. Install with: pip install faiss-cpu or faiss-gpu"
+                "FAISS not available. Install with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
             )
 
     def create_index(
@@ -498,13 +628,25 @@ class FAISSStore:
 
             vectors = vectors.astype(np.float32)
 
-            # Generate IDs if not provided
+            # Generate IDs if not provided.  Use a monotonic counter so
+            # that default IDs never collide with surviving IDs after a
+            # deletion (len(vector_ids) would decrease, potentially reusing
+            # a label that still exists in the index).
             if ids is None:
-                ids = [
-                    f"vec_{len(self.index.vector_ids) + i}" for i in range(len(vectors))
-                ]
+                _existing = set(self.index.vector_ids)
+                generated: List[str] = []
+                while len(generated) < len(vectors):
+                    cand = f"vec_{self._next_id}"
+                    self._next_id += 1
+                    if cand not in _existing:
+                        generated.append(cand)
+                        _existing.add(cand)
+                ids = generated
+                # Sync FAISSIndex.next_id so save() persists the correct value.
+                self.index.next_id = self._next_id
 
-            # Store metadata
+            # Assign metadata before the duplicate-skip filter so callers
+            # always get up-to-date metadata even for already-present ids.
             if metadata:
                 self.progress_tracker.update_tracking(
                     tracking_id, message="Storing metadata..."
@@ -613,7 +755,9 @@ class FAISSStore:
             FAISSIndex instance
         """
         if not FAISS_AVAILABLE:
-            raise ProcessingError("FAISS not available")
+            raise ProcessingError(
+                "FAISS not available. Install with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
+            )
 
         path = Path(path)
         if path.exists() and not _metadata_path(path).exists():
@@ -625,6 +769,12 @@ class FAISSStore:
 
         self.index = FAISSIndex.load(path, self.dimension, index_type)
         self.search_engine = FAISSSearch(self.index)
+        # Remember the path so delete_vectors can auto-save to the same location.
+        self._index_path = path
+        # Restore the monotonic counter from the sidecar (via FAISSIndex.next_id)
+        # rather than using ntotal.  After a deletion ntotal is smaller than the
+        # highest generated ID, so ntotal would cause ID collisions on the next add.
+        self._next_id = self.index.next_id
 
         self.logger.info(f"Loaded FAISS index from {path}")
         return self.index
@@ -735,10 +885,62 @@ class FAISSStore:
         """Return the number of vectors currently tracked in this store.
 
         Returns the length of the ``vector_ids`` list maintained by
-        ``FAISSIndex``.  FAISSStore does not implement vector deletion, so
-        this list is strictly append-only and is always consistent with the
-        underlying FAISS index (``index.ntotal``).
+        ``FAISSIndex``.  This list is always kept consistent with the
+        underlying FAISS index (``index.ntotal``), including after deletions.
         """
         if self.index is None:
             return 0
         return len(self.index.vector_ids)
+
+    def delete_vectors(self, vector_ids: List[str], **options) -> Dict[str, Any]:
+        """Delete vectors by their external string IDs.
+
+        Delegates to :meth:`FAISSIndex.delete_vectors`.  When the store was
+        loaded from disk via :meth:`load_index`, the updated index and sidecar
+        are written back to disk before this method returns, so the deletion is
+        durable across process restarts without the caller needing a separate
+        :meth:`save_index` call.  Note: only the ``.meta.json`` sidecar write
+        is atomic (temp-file + rename); the ``.faiss`` binary is written in
+        place.  A process crash between those two writes would leave the files
+        inconsistent, but the mismatch guard in :meth:`FAISSIndex.load` would
+        detect it on the next load rather than silently returning wrong data.
+
+        No-op deletions (all requested IDs unknown, or empty input) do not
+        trigger a disk write.
+
+        When the store was created in memory (no :meth:`load_index` call), the
+        deletion is in-memory only and the caller must invoke
+        :meth:`save_index` to persist it.
+
+        IVF indices do not support deletion because their internal labels do
+        not compact after ``remove_ids``, which would desynchronize search
+        labels from the ``vector_ids`` mapping.  HNSW indices also do not
+        support ``remove_ids``.  Both raise ``NotImplementedError``, which the
+        :class:`ErasureCoordinator` translates to ``STATUS_UNSUPPORTED``.
+
+        Args:
+            vector_ids: External string IDs to delete.  Unknown IDs are
+                silently ignored.  Duplicates are deduplicated.
+            **options: Accepted for API parity with other backends; unused.
+
+        Returns:
+            ``{"delete_count": N}``
+
+        Raises:
+            ProcessingError: If no index has been initialized.
+            NotImplementedError: If the underlying index type (IVF or HNSW)
+                does not support safe deletion.
+        """
+        if self.index is None:
+            raise ProcessingError(
+                "Index not initialized. Call create_index() first."
+            )
+        result = self.index.delete_vectors(vector_ids)
+        # If the store was loaded from disk (load_index recorded the path),
+        # persist the deletion so that the vectors cannot be resurrected by a
+        # process restart.  Only write when something was actually removed:
+        # a no-op deletion (all IDs unknown or empty list) must not trigger
+        # a full index rewrite.
+        if self._index_path is not None and result.get("delete_count", 0) > 0:
+            self.index.save(self._index_path)
+        return result

semantica/visualization/analytics_visualizer.py

@@ -85,7 +85,7 @@ class AnalyticsVisualizer:
         if px is None or go is None:
             raise ProcessingError(
                 "Plotly is required for analytics visualization. "
-                "Install with: pip install plotly"
+                "Install with: pip install 'semantica[viz]'"
             )
         if np is None:
             raise ProcessingError(

semantica/visualization/embedding_visualizer.py

@@ -1,9 +1,10 @@
 """
 Embedding Visualizer Module
 
-This module provides comprehensive visualization capabilities for vector embeddings in the
-Semantica framework, including 2D/3D dimensionality reduction projections, similarity heatmaps,
-clustering visualizations, multi-modal comparisons, and quality metrics analysis.
+This module provides comprehensive visualization capabilities for vector
+embeddings in the Semantica framework, including 2D/3D dimensionality
+reduction projections, similarity heatmaps, clustering visualizations,
+multi-modal comparisons, and quality metrics analysis.
 
 Key Features:
     - 2D and 3D dimensionality reduction (UMAP, t-SNE, PCA)
@@ -31,9 +32,8 @@ License: MIT
 """
 
 from pathlib import Path
-from typing import Any, Dict, List, Optional, Tuple, Union
+from typing import Any, Dict, List, Optional, Union
 
-import matplotlib.pyplot as plt
 import numpy as np
 
 try:
@@ -45,8 +45,12 @@ except (ImportError, OSError):
     go = None
     make_subplots = None
 
-from sklearn.decomposition import PCA
-from sklearn.manifold import TSNE
+try:
+    from sklearn.decomposition import PCA
+    from sklearn.manifold import TSNE
+except (ImportError, OSError):
+    PCA = None
+    TSNE = None
 
 try:
     import umap
@@ -57,7 +61,7 @@ from ..utils.exceptions import ProcessingError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
 from .utils.color_schemes import ColorPalette, ColorScheme
-from .utils.export_formats import export_matplotlib_figure, export_plotly_figure
+from .utils.export_formats import export_plotly_figure
 
 
 class EmbeddingVisualizer:
@@ -94,12 +98,17 @@ class EmbeddingVisualizer:
             self.color_scheme = ColorScheme.DEFAULT
         self.point_size = config.get("point_size", 5)
 
-    def _check_dependencies(self):
+    def _check_dependencies(self, require_sklearn: bool = False):
         """Check if dependencies are available."""
         if px is None or go is None:
             raise ProcessingError(
                 "Plotly is required for embedding visualization. "
-                "Install with: pip install plotly"
+                "Install with: pip install 'semantica[viz]'"
+            )
+        if require_sklearn and (PCA is None or TSNE is None):
+            raise ProcessingError(
+                "scikit-learn is required for dimensionality reduction. "
+                "Reinstall scikit-learn or install dependencies."
             )
 
     def visualize_2d_projection(
@@ -150,10 +159,12 @@ class EmbeddingVisualizer:
 
         try:
             self.logger.info(f"Visualizing 2D projection using {method}")
-            
+
             # Step 2: Data Analysis
             n_samples, n_features = embeddings.shape
-            self.logger.info(f"Embedding Analysis: {n_samples} samples, {n_features} dimensions")
+            self.logger.info(
+                f"Embedding Analysis: {n_samples} samples, {n_features} dimensions"
+            )
 
             if embeddings.shape[1] <= 2:
                 # Already 2D or less, use directly
@@ -165,28 +176,32 @@ class EmbeddingVisualizer:
                 self.progress_tracker.update_tracking(
                     tracking_id, message=f"Reducing dimensions using {method}..."
                 )
+                dim_options = dict(options)
+                n_comp = dim_options.pop("n_components", 2)
                 projected = self._reduce_dimensions(
-                    embeddings, method=method, n_components=2, **options
+                    embeddings, method=method, n_components=n_comp, **dim_options
                 )
 
             self.progress_tracker.update_tracking(
                 tracking_id, message="Generating visualization..."
             )
             result = self._visualize_2d_plotly(
-                projected, 
-                labels, 
-                output, 
-                file_path, 
+                projected,
+                labels,
+                output,
+                file_path,
                 color_by=color_by,
                 size_by=size_by,
                 hover_data=hover_data,
-                **options
+                **options,
             )
 
             self.progress_tracker.stop_tracking(
                 tracking_id,
                 status="completed",
-                message=f"2D projection visualization generated: {len(projected)} points",
+                message=(
+                    f"2D projection visualization generated: {len(projected)} points"
+                ),
             )
             return result
         except Exception as e:
@@ -237,8 +252,10 @@ class EmbeddingVisualizer:
                 self.progress_tracker.update_tracking(
                     tracking_id, message=f"Reducing dimensions using {method}..."
                 )
+                dim_options = dict(options)
+                n_comp = dim_options.pop("n_components", 3)
                 projected = self._reduce_dimensions(
-                    embeddings, method=method, n_components=3, **options
+                    embeddings, method=method, n_components=n_comp, **dim_options
                 )
 
             self.progress_tracker.update_tracking(
@@ -251,7 +268,9 @@ class EmbeddingVisualizer:
             self.progress_tracker.stop_tracking(
                 tracking_id,
                 status="completed",
-                message=f"3D projection visualization generated: {len(projected)} points",
+                message=(
+                    f"3D projection visualization generated: {len(projected)} points"
+                ),
             )
             return result
         except Exception as e:
@@ -342,7 +361,10 @@ class EmbeddingVisualizer:
                 self.progress_tracker.stop_tracking(
                     tracking_id,
                     status="completed",
-                    message=f"Similarity heatmap generated: {len(embeddings)}x{len(embeddings)} matrix",
+                    message=(
+                        f"Similarity heatmap generated: "
+                        f"{len(embeddings)}x{len(embeddings)} matrix"
+                    ),
                 )
                 return fig
             elif file_path:
@@ -404,8 +426,10 @@ class EmbeddingVisualizer:
                 self.progress_tracker.update_tracking(
                     tracking_id, message=f"Reducing dimensions using {method}..."
                 )
+                dim_options = dict(options)
+                n_comp = dim_options.pop("n_components", 2)
                 projected = self._reduce_dimensions(
-                    embeddings, method=method, n_components=2, **options
+                    embeddings, method=method, n_components=n_comp, **dim_options
                 )
 
             num_clusters = len(set(cluster_labels))
@@ -445,7 +469,10 @@ class EmbeddingVisualizer:
                 self.progress_tracker.stop_tracking(
                     tracking_id,
                     status="completed",
-                    message=f"Clustering visualization generated: {num_clusters} clusters, {len(embeddings)} points",
+                    message=(
+                        f"Clustering visualization generated: "
+                        f"{num_clusters} clusters, {len(embeddings)} points"
+                    ),
                 )
                 return fig
             elif file_path:
@@ -541,8 +568,13 @@ class EmbeddingVisualizer:
                 self.progress_tracker.update_tracking(
                     tracking_id, message=f"Reducing dimensions using {method}..."
                 )
+                dim_options = dict(options)
+                n_comp = dim_options.pop("n_components", 2)
                 projected = self._reduce_dimensions(
-                    combined_embeddings, method=method, n_components=2, **options
+                    combined_embeddings,
+                    method=method,
+                    n_components=n_comp,
+                    **dim_options,
                 )
 
             # Color by type
@@ -579,7 +611,10 @@ class EmbeddingVisualizer:
                 self.progress_tracker.stop_tracking(
                     tracking_id,
                     status="completed",
-                    message=f"Multi-modal comparison generated: {len(combined_embeddings)} embeddings",
+                    message=(
+                        f"Multi-modal comparison generated: "
+                        f"{len(combined_embeddings)} embeddings"
+                    ),
                 )
                 return fig
             elif file_path:
@@ -598,8 +633,6 @@ class EmbeddingVisualizer:
             )
             raise
 
-
-
     def _reduce_dimensions(
         self,
         embeddings: np.ndarray,
@@ -608,39 +641,56 @@ class EmbeddingVisualizer:
         **options,
     ) -> np.ndarray:
         """Reduce embedding dimensions using specified method."""
+        opts = dict(options)
+        opts.pop("n_components", None)
+
         if method == "pca":
-            pca = PCA(n_components=n_components, **options)
+            if PCA is None:
+                raise ProcessingError(
+                    "scikit-learn is required for dimensionality reduction. "
+                    "Reinstall scikit-learn or install dependencies."
+                )
+            pca = PCA(n_components=n_components, **opts)
             return pca.fit_transform(embeddings)
 
         elif method == "tsne":
-            perplexity = options.get("perplexity", min(30, len(embeddings) - 1))
+            if TSNE is None:
+                raise ProcessingError(
+                    "scikit-learn is required for dimensionality reduction. "
+                    "Reinstall scikit-learn or install dependencies."
+                )
+            perplexity = opts.pop("perplexity", min(30, len(embeddings) - 1))
+            random_state = opts.pop("random_state", 42)
             tsne = TSNE(
                 n_components=n_components,
                 perplexity=perplexity,
-                random_state=42,
-                **options,
+                random_state=random_state,
+                **opts,
             )
             return tsne.fit_transform(embeddings)
 
         elif method == "umap":
             if umap is not None:
-                n_neighbors = options.get("n_neighbors", min(15, len(embeddings) - 1))
+                n_neighbors = opts.pop("n_neighbors", min(15, len(embeddings) - 1))
                 reducer = umap.UMAP(
-                    n_components=n_components, n_neighbors=n_neighbors, **options
+                    n_components=n_components, n_neighbors=n_neighbors, **opts
                 )
                 return reducer.fit_transform(embeddings)
             else:
-                # Fallback to PCA if UMAP not available
-                self.logger.warning(
-                    "UMAP not available, using PCA. Install with: pip install umap-learn"
+                raise ProcessingError(
+                    "UMAP is required for UMAP dimensionality reduction. "
+                    "Install with: pip install 'semantica[viz]'"
                 )
-                pca = PCA(n_components=n_components)
-                return pca.fit_transform(embeddings)
 
         else:
+            if PCA is None:
+                raise ProcessingError(
+                    "scikit-learn is required for dimensionality reduction. "
+                    "Reinstall scikit-learn or install dependencies."
+                )
             # Fallback to PCA
             self.logger.warning(f"Method {method} not available, using PCA")
-            pca = PCA(n_components=n_components)
+            pca = PCA(n_components=n_components, **opts)
             return pca.fit_transform(embeddings)
 
     def _visualize_2d_plotly(

semantica/visualization/kg_visualizer.py

@@ -124,7 +124,7 @@ class KGVisualizer:
         if px is None or go is None:
             raise ProcessingError(
                 "Plotly is required for KG visualization. "
-                "Install with: pip install plotly"
+                "Install with: pip install 'semantica[viz]'"
             )
 
     def _convert_knowledge_graph(self, kg: Any) -> Dict[str, Any]:

semantica/visualization/ontology_visualizer.py

@@ -35,8 +35,14 @@ License: MIT
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Union
 
-import matplotlib.patches as mpatches
-import matplotlib.pyplot as plt
+try:
+    import matplotlib.patches as mpatches
+    import matplotlib.pyplot as plt
+    from matplotlib.patches import FancyBboxPatch
+except (ImportError, OSError):
+    mpatches = None
+    plt = None
+    FancyBboxPatch = None
 
 try:
     import plotly.express as px
@@ -47,8 +53,6 @@ except (ImportError, OSError):
     go = None
     make_subplots = None
 
-from matplotlib.patches import FancyBboxPatch
-
 try:
     import graphviz
 except (ImportError, OSError):
@@ -106,13 +110,13 @@ class OntologyVisualizer:
             if graphviz is None:
                 raise ProcessingError(
                     "Graphviz is required for DOT export. "
-                    "Install with: pip install graphviz"
+                    "Install with: pip install 'semantica[viz]'"
                 )
         else:
-            if px is None or go is None:
+            if go is None:
                 raise ProcessingError(
                     "Plotly is required for ontology visualization. "
-                    "Install with: pip install plotly"
+                    "Install with: pip install 'semantica[viz]'"
                 )
 
     def visualize_hierarchy(
@@ -892,7 +896,7 @@ class OntologyVisualizer:
         """Create Graphviz hierarchy visualization."""
         if graphviz is None:
             raise ProcessingError(
-                "Graphviz not available. Install with: pip install graphviz"
+                "Graphviz not available. Install with: pip install 'semantica[viz]'"
             )
 
         dot = graphviz.Digraph(comment="Ontology Hierarchy")

semantica/visualization/semantic_network_visualizer.py

@@ -74,7 +74,7 @@ class SemanticNetworkVisualizer:
         if px is None or go is None:
             raise ProcessingError(
                 "Plotly is required for semantic network visualization. "
-                "Install with: pip install plotly"
+                "Install with: pip install 'semantica[viz]'"
             )
 
     def visualize_network(

semantica/visualization/temporal_visualizer.py

@@ -83,7 +83,7 @@ class TemporalVisualizer:
         if px is None or go is None:
             raise ProcessingError(
                 "Plotly is required for temporal visualization. "
-                "Install with: pip install plotly"
+                "Install with: pip install 'semantica[viz]'"
             )
 
     def visualize_temporal_dashboard(

semantica_mcp/mcp/schemas.py

@@ -289,4 +289,122 @@ GET_ANALYTICS = {
     },
 }
 
+STORE_DOCUMENT = {
+    "type": "object",
+    "properties": {
+        "content": {
+            "type": "string",
+            "description": "Document text to chunk and store for semantic retrieval",
+        },
+        "source": {
+            "type": "string",
+            "description": "Provenance identifier, e.g. 'policy_manual_v2#page12'",
+        },
+        "authority": {
+            "type": "string",
+            "description": "Authority level of the content, e.g. 'official', 'draft', 'external'",
+        },
+        "version": {
+            "type": "string",
+            "description": "Document version tag used together with source as the upsert key (default: 'v1')",
+        },
+        "project": {
+            "type": "string",
+            "description": "Optional project namespace for later filtering",
+        },
+        "metadata": {
+            "type": "object",
+            "description": "Additional key-value properties stored on every chunk",
+        },
+        "chunk_size": {
+            "type": "integer",
+            "minimum": 100,
+            "description": "Chunk window in characters (default: 1000)",
+        },
+        "chunk_overlap": {
+            "type": "integer",
+            "minimum": 0,
+            "description": "Overlap between consecutive chunks in characters (default: 200)",
+        },
+    },
+    "required": ["content", "source", "authority"],
+}
+
+RETRIEVE_CONTEXT = {
+    "type": "object",
+    "properties": {
+        "query": {
+            "type": "string",
+            "description": "Natural language query to embed and search for",
+        },
+        "top_k": {
+            "type": "integer",
+            "minimum": 1,
+            "maximum": 10,
+            "description": "Maximum number of chunks to return (default: 5, capped at 10)",
+        },
+        "project": {
+            "type": "string",
+            "description": "Only return chunks stored under this project namespace (optional)",
+        },
+    },
+    "required": ["query"],
+}
+
+UPDATE_DOCUMENT = {
+    "type": "object",
+    "properties": {
+        "content": {
+            "type": "string",
+            "description": "New document text replacing the stored version",
+        },
+        "source": {
+            "type": "string",
+            "description": "Provenance identifier of the document to update",
+        },
+        "version": {
+            "type": "string",
+            "description": "Version tag identifying which stored version to replace (default: 'v1')",
+        },
+        "authority": {
+            "type": "string",
+            "description": "Updated authority level (defaults to the stored value)",
+        },
+        "project": {
+            "type": "string",
+            "description": "Updated project namespace (defaults to the stored value)",
+        },
+        "metadata": {
+            "type": "object",
+            "description": "Additional key-value properties merged into chunk metadata",
+        },
+        "chunk_size": {
+            "type": "integer",
+            "minimum": 100,
+            "description": "Chunk window in characters (default: 1000)",
+        },
+        "chunk_overlap": {
+            "type": "integer",
+            "minimum": 0,
+            "description": "Overlap between consecutive chunks in characters (default: 200)",
+        },
+    },
+    "required": ["content", "source"],
+}
+
+REMOVE_DOCUMENT = {
+    "type": "object",
+    "properties": {
+        "source": {
+            "type": "string",
+            "description": "Provenance identifier of the document to remove",
+        },
+        "version": {
+            "type": "string",
+            "description": "Version tag identifying which stored version to remove (default: 'v1')",
+        },
+    },
+    "required": ["source"],
+}
+
 EMPTY = {"type": "object", "properties": {}}

semantica_mcp/mcp/session.py

@@ -14,7 +14,15 @@ from typing import Any, Optional
 
 log = logging.getLogger("semantica.mcp.session")
 
+# Backends the retrieval tools can actually support end to end.  faiss
+# and pgvector have no metadata-scoped delete, so update_document and
+# remove_document cannot work on them; selecting them fails fast here
+# instead of blowing up mid-update.
+SUPPORTED_VECTOR_BACKENDS = ("inmemory", "sqlite")
+
 _graph: Optional[Any] = None
+_embedder: Optional[Any] = None
+_vector_store: Optional[Any] = None
 
 # Tracks whether the last graph initialisation successfully loaded the
 # configured SEMANTICA_KG_PATH file.  When True (or no path was configured)
@@ -59,6 +67,103 @@ def get_graph() -> Any:
     return _graph
 
 
+def get_embedder() -> Any:
+    """
+    Return the shared EmbeddingGenerator instance, creating it on first call.
+
+    Used by the semantic retrieval tools (#1235) to embed documents and
+    queries with one consistent model, so stored vectors and query
+    vectors always share the same dimensionality.
+    """
+    global _embedder
+    if _embedder is None:
+        from semantica.embeddings import EmbeddingGenerator
+
+        _embedder = EmbeddingGenerator()
+        log.info(
+            "Embedding generator initialised (method=%s)",
+            _embedder.get_text_method(),
+        )
+    return _embedder
+
+
+def get_vector_store() -> Any:
+    """
+    Return the shared VectorStore instance, creating it on first call.
+
+    Backend selection:
+
+    • ``SEMANTICA_VECTOR_BACKEND`` — ``inmemory`` (default) or ``sqlite``.
+      The ``sqlite`` backend additionally requires
+      ``SEMANTICA_VECTOR_DB_PATH``.  Other VectorStore backends (faiss,
+      pgvector) are rejected: they lack the metadata-scoped delete the
+      update/remove tools need.
+    • ``SEMANTICA_VECTOR_PATH`` — a *directory* previously written by
+      ``VectorStore.save()``.  If it exists, the store is loaded from it
+      on start.  Note this is a directory, unlike SEMANTICA_KG_PATH which
+      is a single JSON file.  The persisted dimension must match the
+      active embedder or startup fails — otherwise queries would either
+      error on shape mismatch or silently rank across incompatible
+      embedding spaces.
+    """
+    global _vector_store
+    if _vector_store is None:
+        from semantica.vector_store import VectorStore
+
+        backend = os.environ.get("SEMANTICA_VECTOR_BACKEND", "inmemory").strip().lower()
+        if backend not in SUPPORTED_VECTOR_BACKENDS:
+            raise ValueError(
+                f"SEMANTICA_VECTOR_BACKEND={backend!r} is not supported by the "
+                "MCP retrieval tools; supported backends: "
+                + ", ".join(SUPPORTED_VECTOR_BACKENDS)
+            )
+        config: dict = {}
+        if backend == "sqlite":
+            db_path = os.environ.get("SEMANTICA_VECTOR_DB_PATH", "").strip()
+            if not db_path:
+                raise ValueError(
+                    "SEMANTICA_VECTOR_BACKEND=sqlite requires "
+                    "SEMANTICA_VECTOR_DB_PATH to point at the database file"
+                )
+            config["db_path"] = db_path
+        # VectorStore defaults to dimension 768, which does not match the
+        # default embedding model (all-MiniLM-L6-v2 = 384, hash fallback
+        # = 128).  Always derive it from the embedder so store and
+        # queries stay consistent.
+        embedder = get_embedder()
+        config["dimension"] = embedder.text_embedder.get_embedding_dimension()
+
+        store = VectorStore(backend=backend, config=config)
+
+        vector_path = os.environ.get("SEMANTICA_VECTOR_PATH", "").strip()
+        if vector_path and os.path.isdir(vector_path):
+            try:
+                store.load(vector_path)
+                log.info("Vector store loaded from %s", vector_path)
+            except Exception as exc:
+                raise ValueError(
+                    f"Could not load vector store from {vector_path}: {exc}"
+                ) from exc
+            loaded_dim = getattr(store, "dimension", None)
+            if loaded_dim and loaded_dim != config["dimension"]:
+                raise ValueError(
+                    f"Persisted vector store at {vector_path} has dimension "
+                    f"{loaded_dim}, but the active embedder produces "
+                    f"{config['dimension']}. Re-embed the corpus or point "
+                    "SEMANTICA_VECTOR_PATH at a store built with the same model."
+                )
+
+        _vector_store = store
+        log.info("Vector store initialised (backend=%s)", backend)
+    return _vector_store
+
+
+def reset_vector_store() -> None:
+    """Reset the vector store singleton (mainly useful in tests)."""
+    global _vector_store
+    _vector_store = None
+
+
 def is_persistence_safe() -> bool:
     """Return True when it is safe to write mutations back to SEMANTICA_KG_PATH.
 

semantica_mcp/mcp/tools/__init__.py

@@ -9,6 +9,7 @@ from .export import EXPORT_TOOLS
 from .extraction import EXTRACTION_TOOLS
 from .graph import GRAPH_TOOLS
 from .reasoning import REASONING_TOOLS
+from .retrieval import RETRIEVAL_TOOLS
 
 # Ordered list — exposed to the MCP client via tools/list
 TOOL_DEFINITIONS = (
@@ -17,6 +18,7 @@ TOOL_DEFINITIONS = (
     + GRAPH_TOOLS
     + REASONING_TOOLS
     + EXPORT_TOOLS
+    + RETRIEVAL_TOOLS
 )
 
 __all__ = ["TOOL_DEFINITIONS"]

semantica_mcp/mcp/tools/retrieval.py

@@ -0,0 +1,533 @@
+"""
+Semantic retrieval tools — store, retrieve, update and remove documents
+in a vector store, combined with knowledge-graph context (#1235).
+
+Design notes:
+
+• Documents are chunked with a fixed sliding window (default 1000 chars,
+  200 overlap) and every chunk carries full provenance metadata:
+  chunk_id, source, authority, version, project, content hash, status
+  and character offsets.
+• (source, version) is the upsert key.  The content hash only decides
+  whether a re-store can be skipped as a no-op.
+• Updates and removals on the in-memory backend rebuild the store from
+  scratch (read everything, filter, clear, re-store) instead of calling
+  delete_vectors.  In-memory ids are derived from ``len(self.vectors)``
+  and fall back after a delete, so deleting then writing can overwrite
+  live data (#1029).  Rebuilding from an empty dict starts the counter
+  at zero — nothing to collide with.  The real fix for #1029 (ids that
+  never get reused) belongs in its own PR.
+• Retrieval results are combined with related graph nodes: for each hit
+  source we look up ContextGraph nodes tagged with the same
+  ``metadata.source`` and attach their 1-hop neighbours.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import logging
+import os
+from typing import Any, Dict, List, Tuple
+
+import numpy as np
+
+from ..schemas import (
+    REMOVE_DOCUMENT,
+    RETRIEVE_CONTEXT,
+    STORE_DOCUMENT,
+    UPDATE_DOCUMENT,
+)
+from ..session import get_embedder, get_graph, get_vector_store
+
+log = logging.getLogger("semantica.mcp.tools.retrieval")
+
+DEFAULT_CHUNK_SIZE = 1000
+DEFAULT_CHUNK_OVERLAP = 200
+MAX_TOP_K = 10
+FILTER_OVERFETCH = 3
+MAX_FILTER_MATCHES = 10_000
+MAX_CHUNKS_PER_DOC = 10_000
+
+# Metadata fields owned by the upsert logic.  Caller-supplied metadata
+# can add extra context but must not rewrite provenance: overwriting
+# source/version/hash/status would break the (source, version) upsert
+# key, the idempotent no-op check, and retrieval filters.
+PROTECTED_META_KEYS = frozenset(
+    {
+        "chunk_id",
+        "text",
+        "source",
+        "authority",
+        "version",
+        "hash",
+        "status",
+        "chunk_index",
+        "char_start",
+        "char_end",
+        "project",
+    }
+)
+
+
+def _chunk_text(text: str, chunk_size: int, chunk_overlap: int) -> List[Tuple[int, int, str]]:
+    """Split text into (char_start, char_end, chunk) windows."""
+    if chunk_overlap >= chunk_size:
+        raise ValueError("chunk_overlap must be smaller than chunk_size")
+    chunks: List[Tuple[int, int, str]] = []
+    start = 0
+    n = len(text)
+    while start < n:
+        end = min(start + chunk_size, n)
+        chunks.append((start, end, text[start:end]))
+        if end >= n:
+            break
+        start = end - chunk_overlap
+    return chunks
+
+
+def _chunk_id(source: str, version: str, index: int, text: str) -> str:
+    """Stable chunk id derived from the location key and chunk content."""
+    digest = hashlib.sha256(
+        f"{source}|{version}|{index}|{text}".encode("utf-8")
+    ).hexdigest()
+    return f"chk_{digest[:16]}"
+
+
+def _doc_hash(content: str) -> str:
+    return hashlib.sha256(content.encode("utf-8")).hexdigest()
+
+
+def _find_matching_rows(store: Any, source: str, version: str) -> List[Dict[str, Any]]:
+    """
+    Return rows (``{id, vector, metadata}``) matching (source, version).
+
+    The persistent branch pulls whole rows (vector included) into memory;
+    the limit keeps the scan bounded.  Documents beyond MAX_CHUNKS_PER_DOC
+    chunks are rejected at ingestion, so the cap cannot leave stale
+    chunks behind on update/remove.
+    """
+    if getattr(store, "backend", "") == "inmemory":
+        rows = []
+        for vid, vec in getattr(store, "vectors", {}).items():
+            meta = getattr(store, "metadata", {}).get(vid) or {}
+            if meta.get("source") == source and meta.get("version") == version:
+                rows.append({"id": vid, "vector": vec, "metadata": meta})
+        return rows
+    backend_store = getattr(store, "_backend_store", None)
+    if backend_store is not None and hasattr(backend_store, "filter_by_metadata"):
+        return backend_store.filter_by_metadata(
+            {"source": source, "version": version}, limit=MAX_FILTER_MATCHES
+        )
+    raise NotImplementedError(
+        f"Backend {type(backend_store).__name__} does not support metadata lookup; "
+        "cannot locate chunks for update/remove"
+    )
+
+
+def _find_matching_ids(store: Any, source: str, version: str) -> List[str]:
+    """Return every vector id whose metadata matches (source, version)."""
+    return [row["id"] for row in _find_matching_rows(store, source, version)]
+
+
+def _remove_ids(store: Any, remove_ids: List[str]) -> None:
+    """Remove vectors by id, avoiding the #1029 in-memory id collision."""
+    if getattr(store, "backend", "") == "inmemory":
+        # Full rebuild: read all, filter in memory, clear, re-store once.
+        # store_vectors derives ids from len(self.vectors), and the dicts
+        # are empty here, so the counter restarts at zero — no reuse of
+        # ids that are still referenced anywhere.
+        remove = set(remove_ids)
+        vectors = getattr(store, "vectors", {})
+        metadata = getattr(store, "metadata", {})
+        saved_vectors = dict(vectors)
+        saved_metadata = dict(metadata)
+        keep_vectors = []
+        keep_meta = []
+        for vid, vec in list(vectors.items()):
+            if vid in remove:
+                continue
+            keep_vectors.append(vec)
+            keep_meta.append(metadata.get(vid, {}))
+        vectors.clear()
+        metadata.clear()
+        try:
+            if keep_vectors:
+                store.store_vectors(keep_vectors, keep_meta)
+        except Exception:
+            # Restore the pre-rebuild state so a failed re-store does not
+            # silently drop every surviving document.
+            vectors.update(saved_vectors)
+            metadata.update(saved_metadata)
+            store.indexer.create_index(
+                list(vectors.values()), list(vectors.keys())
+            )
+            raise
+        return
+    # Persistent backends do not have the len-based id collision, so a
+    # direct delete is safe there.
+    store.delete_vectors(remove_ids)
+
+
+def _persist(store: Any) -> Any:
+    """
+    Persist the store when SEMANTICA_VECTOR_PATH is configured.
+
+    Returns ``None`` when no path is configured, ``True`` on success and
+    ``False`` when saving failed — surfaced in tool results so a caller
+    can tell an in-memory-only write from a durable one.
+    """
+    path = os.environ.get("SEMANTICA_VECTOR_PATH", "").strip()
+    if not path:
+        return None
+    try:
+        store.save(path)
+    except Exception as exc:
+        log.warning("Could not persist vector store to %s: %s", path, exc)
+        return False
+    return True
+
+
+def _node_source(meta: Any) -> str:
+    """
+    Extract a node's source tag from its metadata.
+
+    ContextGraph.add_node nests the caller-supplied metadata dict one
+    level down (``{'label': ..., 'metadata': {...}}``), while nodes added
+    through other paths may carry ``source`` directly.  Check both.
+    """
+    if not isinstance(meta, dict):
+        return ""
+    direct = meta.get("source")
+    if direct:
+        return str(direct)
+    nested = meta.get("metadata")
+    if isinstance(nested, dict):
+        return str(nested.get("source", "") or "")
+    return ""
+
+
+def _graph_relationships(sources: List[str], max_per_source: int = 3) -> List[Dict[str, Any]]:
+    """
+    Collect 1-hop graph neighbours for nodes tagged with the hit sources.
+
+    Node lookup matches the node's source tag against the stored document
+    sources.  Failures degrade to an empty list — graph context is a
+    bonus, never a hard dependency of retrieval.
+    """
+    if not sources:
+        return []
+    try:
+        graph = get_graph()
+        nodes = list(graph.find_nodes())
+    except Exception as exc:
+        log.debug("Graph context unavailable: %s", exc)
+        return []
+
+    relationships: List[Dict[str, Any]] = []
+    seen: set = set()
+    for source in sources:
+        anchor = None
+        for n in nodes:
+            if _node_source(n.get("metadata")) == source:
+                anchor = n
+                break
+        if anchor is None:
+            continue
+        try:
+            neighbors = graph.get_neighbors(anchor["id"], hops=1)
+        except Exception as exc:
+            log.debug("get_neighbors failed for %s: %s", anchor.get("id"), exc)
+            continue
+        added = 0
+        for nb in neighbors:
+            key = (anchor.get("id"), nb.get("id"), nb.get("relationship"))
+            if key in seen:
+                continue
+            seen.add(key)
+            relationships.append(
+                {
+                    "node": {
+                        "id": anchor.get("id"),
+                        "type": anchor.get("type"),
+                        "content": str(anchor.get("content") or "")[:200],
+                        "source": source,
+                    },
+                    "related": {
+                        "id": nb.get("id"),
+                        "type": nb.get("type"),
+                        "content": str(nb.get("content") or "")[:200],
+                    },
+                    "relationship": nb.get("relationship"),
+                }
+            )
+            added += 1
+            if added >= max_per_source:
+                break
+    return relationships
+
+
+def _upsert(args: dict, action: str) -> dict:
+    """Shared implementation for store_document and update_document."""
+    content = args.get("content", "")
+    source = str(args.get("source", "")).strip()
+    if not content or not source:
+        return {"error": "content and source are required"}
+    authority = str(args.get("authority", "")).strip()
+    if action == "store" and not authority:
+        return {"error": "authority is required"}
+    version = str(args.get("version", "")).strip() or "v1"
+    project = str(args.get("project", "")).strip() or None
+    chunk_size = int(args.get("chunk_size", DEFAULT_CHUNK_SIZE))
+    chunk_overlap = int(args.get("chunk_overlap", DEFAULT_CHUNK_OVERLAP))
+    if chunk_overlap >= chunk_size:
+        return {"error": "chunk_overlap must be smaller than chunk_size"}
+    extra = args.get("metadata") or {}
+    if not isinstance(extra, dict):
+        return {"error": "metadata must be an object"}
+    doc_hash = _doc_hash(content)
+
+    try:
+        store = get_vector_store()
+        embedder = get_embedder()
+
+        existing_ids = _find_matching_ids(store, source, version)
+        if action == "update" and not existing_ids:
+            return {"status": "not_found", "source": source, "version": version}
+        existing_first: Dict[str, Any] = {}
+        if existing_ids:
+            existing_first = store.get_metadata(existing_ids[0]) or {}
+            if action == "store" and existing_first.get("hash") == doc_hash:
+                # Same content already stored under (source, version) —
+                # skip re-embedding entirely.
+                return {
+                    "status": "unchanged",
+                    "source": source,
+                    "version": version,
+                    "chunk_ids": [
+                        (store.get_metadata(vid) or {}).get("chunk_id")
+                        for vid in existing_ids
+                    ],
+                }
+
+        chunks = _chunk_text(content, chunk_size, chunk_overlap)
+        if len(chunks) > MAX_CHUNKS_PER_DOC:
+            return {
+                "error": (
+                    f"document produces {len(chunks)} chunks, above the "
+                    f"{MAX_CHUNKS_PER_DOC}-chunk limit; split it into smaller "
+                    "documents or raise chunk_size"
+                )
+            }
+        vectors = np.asarray(
+            embedder.generate_embeddings([c_text for _, _, c_text in chunks])
+        )
+        if vectors.ndim == 1:
+            vectors = vectors.reshape(1, -1)
+        if vectors.shape[0] != len(chunks):
+            return {
+                "error": (
+                    f"embedder returned {vectors.shape[0]} vectors "
+                    f"for {len(chunks)} chunks"
+                )
+            }
+
+        final_authority = authority or existing_first.get("authority") or "unknown"
+        final_project = project or existing_first.get("project")
+
+        old_rows: List[Dict[str, Any]] = []
+        if existing_ids:
+            # Snapshot the rows being replaced so a failed write of the
+            # new chunks can put the old document back instead of leaving
+            # (source, version) silently empty.
+            old_rows = _find_matching_rows(store, source, version)
+            _remove_ids(store, existing_ids)
+
+        metas = []
+        chunk_ids = []
+        for idx, (start, end, c_text) in enumerate(chunks):
+            cid = _chunk_id(source, version, idx, c_text)
+            chunk_ids.append(cid)
+            meta: Dict[str, Any] = {
+                "chunk_id": cid,
+                "text": c_text,
+                "source": source,
+                "authority": final_authority,
+                "version": version,
+                "hash": doc_hash,
+                "status": "active",
+                "chunk_index": idx,
+                "char_start": start,
+                "char_end": end,
+            }
+            if final_project:
+                meta["project"] = final_project
+            for key in extra:
+                if key in PROTECTED_META_KEYS:
+                    log.debug(
+                        "Ignoring caller metadata key %r: provenance field is "
+                        "managed by the tool",
+                        key,
+                    )
+                else:
+                    meta[key] = extra[key]
+            metas.append(meta)
+
+        try:
+            store.store_vectors(list(vectors), metas)
+        except Exception:
+            if old_rows:
+                log.warning(
+                    "Storing new chunks failed for (%s, %s); restoring the "
+                    "previous document",
+                    source,
+                    version,
+                )
+                store.store_vectors(
+                    [row["vector"] for row in old_rows],
+                    [row["metadata"] for row in old_rows],
+                )
+            raise
+        persisted = _persist(store)
+        return {
+            "status": "stored" if action == "store" else "updated",
+            "source": source,
+            "version": version,
+            "chunk_ids": chunk_ids,
+            "chunk_count": len(chunk_ids),
+            "hash": doc_hash,
+            "persisted": persisted,
+        }
+    except Exception as exc:
+        log.exception("%s_document failed", action)
+        return {"error": str(exc)}
+
+
+def handle_store_document(args: dict) -> dict:
+    """Chunk a document, embed it, and store it for semantic retrieval."""
+    return _upsert(args, "store")
+
+
+def handle_update_document(args: dict) -> dict:
+    """Replace the stored content of a (source, version) document."""
+    return _upsert(args, "update")
+
+
+def handle_retrieve_context(args: dict) -> dict:
+    """Embed a query and return the most relevant stored chunks."""
+    query = str(args.get("query", "")).strip()
+    if not query:
+        return {"error": "query is required", "results": []}
+    try:
+        top_k = max(1, min(int(args.get("top_k", 5)), MAX_TOP_K))
+    except (TypeError, ValueError):
+        top_k = 5
+    project = str(args.get("project", "")).strip() or None
+
+    try:
+        store = get_vector_store()
+        query_vector = np.asarray(get_embedder().generate_embeddings([query]))[0]
+        # Over-fetch so a project filter can drop hits without starving
+        # the result list.
+        fetch_k = top_k * FILTER_OVERFETCH if project else top_k
+        raw = store.search_vectors(query_vector, k=fetch_k)
+
+        results = []
+        for hit in raw:
+            meta = hit.get("metadata") or {}
+            if project and meta.get("project") != project:
+                continue
+            results.append(
+                {
+                    "chunk_id": meta.get("chunk_id", hit.get("id")),
+                    "text": meta.get("text", ""),
+                    "score": hit.get("score"),
+                    "source": meta.get("source"),
+                    "authority": meta.get("authority"),
+                    "version": meta.get("version"),
+                    "project": meta.get("project"),
+                    "status": meta.get("status"),
+                    "hash": meta.get("hash"),
+                }
+            )
+            if len(results) >= top_k:
+                break
+
+        sources = list(dict.fromkeys(r["source"] for r in results if r["source"]))
+        return {
+            "query": query,
+            "results": results,
+            "count": len(results),
+            "graph_context": _graph_relationships(sources),
+        }
+    except Exception as exc:
+        log.exception("retrieve_context failed")
+        return {"error": str(exc), "results": []}
+
+
+def handle_remove_document(args: dict) -> dict:
+    """Remove every chunk stored under (source, version)."""
+    source = str(args.get("source", "")).strip()
+    if not source:
+        return {"error": "source is required"}
+    version = str(args.get("version", "")).strip() or "v1"
+    try:
+        store = get_vector_store()
+        existing_ids = _find_matching_ids(store, source, version)
+        if not existing_ids:
+            return {"status": "not_found", "source": source, "version": version}
+        _remove_ids(store, existing_ids)
+        persisted = _persist(store)
+        return {
+            "status": "removed",
+            "source": source,
+            "version": version,
+            "removed_chunks": len(existing_ids),
+            "persisted": persisted,
+        }
+    except Exception as exc:
+        log.exception("remove_document failed")
+        return {"error": str(exc)}
+
+
+RETRIEVAL_TOOLS = [
+    {
+        "name": "store_document",
+        "description": (
+            "Chunk a document, embed the chunks, and store them for semantic "
+            "retrieval. Keyed on (source, version); storing identical content "
+            "again is a no-op."
+        ),
+        "inputSchema": STORE_DOCUMENT,
+        "_handler": handle_store_document,
+    },
+    {
+        "name": "retrieve_context",
+        "description": (
+            "Embed a natural-language query and return the most relevant "
+            "stored chunks with scores and provenance, combined with related "
+            "knowledge-graph relationships."
+        ),
+        "inputSchema": RETRIEVE_CONTEXT,
+        "_handler": handle_retrieve_context,
+    },
+    {
+        "name": "update_document",
+        "description": (
+            "Replace the stored content of a document identified by "
+            "(source, version). Old chunks are removed and the new content "
+            "is re-chunked and re-embedded. Returns not_found when no "
+            "stored document matches (source, version)."
+        ),
+        "inputSchema": UPDATE_DOCUMENT,
+        "_handler": handle_update_document,
+    },
+    {
+        "name": "remove_document",
+        "description": (
+            "Remove every chunk stored under (source, version) from the "
+            "vector store."
+        ),
+        "inputSchema": REMOVE_DOCUMENT,
+        "_handler": handle_remove_document,
+    },
+]

tests/ingest/test_optional_imports.py

@@ -141,3 +141,134 @@ else:
     assert "ConfigurationError" in result.stdout
     assert "Parquet ingestion" in result.stdout
     assert "pyarrow" in result.stdout
+
+
+def test_repo_ingestor_probe_fails_without_gitpython() -> None:
+    result = _run_python_with_blocked_modules(
+        """
+try:
+    from semantica.ingest import RepoIngestor
+    has_git = True
+except ImportError:
+    has_git = False
+
+assert not has_git, "Expected RepoIngestor import to fail without GitPython"
+print("RepoIngestor probe passed")
+""",
+        ("git",),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "RepoIngestor probe passed" in result.stdout
+
+
+def test_xml_ingestor_probe_fails_without_lxml() -> None:
+    result = _run_python_with_blocked_modules(
+        """
+try:
+    from semantica.ingest import XMLIngestor
+    has_lxml = True
+except ImportError:
+    has_lxml = False
+
+assert not has_lxml, "Expected XMLIngestor import to fail without lxml"
+print("XMLIngestor probe passed")
+""",
+        ("lxml",),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "XMLIngestor probe passed" in result.stdout
+
+
+def test_xml_ingestion_reports_missing_lxml_when_used() -> None:
+    result = _run_python_with_blocked_modules(
+        """
+from semantica.ingest import ingest_xml
+
+try:
+    ingest_xml("catalog.xml")
+except Exception as exc:
+    print(type(exc).__name__, exc)
+else:
+    raise SystemExit("expected XML ingestion to fail without lxml")
+""",
+        ("lxml",),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "ConfigurationError" in result.stdout
+    assert "XML ingestion" in result.stdout
+    assert "lxml" in result.stdout
+
+
+def test_sibling_imports_succeed_without_optional_backends() -> None:
+    result = _run_python_with_blocked_modules(
+        """
+from semantica.ingest import (
+    CodeExtractor,
+    CodeFile,
+    CommitInfo,
+    GitAnalyzer,
+    XMLIngestionData,
+    SalesforceData,
+)
+print(
+    CodeExtractor.__name__,
+    CodeFile.__name__,
+    CommitInfo.__name__,
+    GitAnalyzer.__name__,
+    XMLIngestionData.__name__,
+    SalesforceData.__name__,
+)
+""",
+        ("git", "lxml", "simple_salesforce"),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert (
+        "CodeExtractor CodeFile CommitInfo GitAnalyzer XMLIngestionData SalesforceData"
+        in result.stdout
+    )
+
+
+def test_salesforce_ingestor_probe_fails_without_simple_salesforce() -> None:
+    result = _run_python_with_blocked_modules(
+        """
+try:
+    from semantica.ingest import SalesforceIngestor
+    has_salesforce = True
+except ImportError:
+    has_salesforce = False
+
+assert not has_salesforce, (
+    "Expected SalesforceIngestor import to fail without simple-salesforce"
+)
+print("SalesforceIngestor probe passed")
+""",
+        ("simple_salesforce",),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "SalesforceIngestor probe passed" in result.stdout
+
+
+def test_salesforce_ingestion_reports_missing_dep_when_used() -> None:
+    result = _run_python_with_blocked_modules(
+        """
+from semantica.ingest import ingest_salesforce
+
+try:
+    ingest_salesforce()
+except Exception as exc:
+    print(type(exc).__name__, exc)
+else:
+    raise SystemExit("expected Salesforce ingestion to fail without simple-salesforce")
+""",
+        ("simple_salesforce",),
+    )
+
+    assert result.returncode == 0, result.stderr
+    assert "ConfigurationError" in result.stdout
+    assert "Salesforce ingestion" in result.stdout
+    assert "simple-salesforce" in result.stdout

tests/semantic_extract/test_gemini_legacy_model_cache.py

@@ -0,0 +1,149 @@
+"""Regression tests for the legacy-SDK model-instance cache on ``GeminiProvider``.
+
+When the new ``google.genai`` package is unavailable, ``GeminiProvider`` falls
+back to the legacy ``google.generativeai`` package, whose ``GenerativeModel``
+binds its model name at construction time and whose API key lives in
+module-level state (``genai.configure()``).
+
+``GeminiProvider._legacy_client_for()`` therefore keeps a per-instance cache
+keyed by model name, so a repeated per-call ``model=`` override reuses one
+``GenerativeModel`` instead of rebuilding it on every request, and re-asserts
+``genai.configure(api_key=...)`` with this provider's own key before each use.
+
+PR #1488 (issue #1268) locked in *which* model a per-call override resolves to.
+These tests cover what it did not: that the resolved instance is built once and
+cached, and that the cache and credentials stay isolated per provider instance
+(issue #1269).
+"""
+
+import sys
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from semantica.semantic_extract.providers import GeminiProvider
+
+CONSTRUCTION_MODEL = "gemini-pro"
+OVERRIDE_MODEL = "gemini-1.5-flash"
+OTHER_MODEL = "gemini-1.5-pro"
+JSON_TEXT = '{"answer": 42}'
+
+
+def _make_provider(api_key="fake-key", model=CONSTRUCTION_MODEL):
+    """A GeminiProvider on the legacy path with the real SDK bootstrap skipped."""
+    with patch.object(GeminiProvider, "_init_client", return_value=None):
+        provider = GeminiProvider(api_key=api_key, model=model)
+    provider._use_new_genai = False
+    provider.client = MagicMock(name="construction client")
+    return provider
+
+
+@pytest.fixture
+def fake_legacy_genai(monkeypatch):
+    """Install a stand-in ``google.generativeai`` module.
+
+    Unlike the fake in ``test_gemini_model_override``, ``GenerativeModel`` here
+    does **not** cache internally: it returns a fresh mock every call and records
+    every model name it was asked to build, so a test can tell whether the
+    provider rebuilt a model or served it from its own cache. ``configure`` is a
+    plain mock so credential re-assertion is observable.
+    """
+    module = MagicMock()
+    module.build_calls = []
+
+    def build_model(name):
+        module.build_calls.append(name)
+        model = MagicMock(name=f"GenerativeModel({name})#{len(module.build_calls)}")
+        response = MagicMock()
+        response.text = JSON_TEXT
+        model.generate_content.return_value = response
+        return model
+
+    module.GenerativeModel.side_effect = build_model
+    monkeypatch.setitem(sys.modules, "google.generativeai", module)
+    return module
+
+
+class TestLegacyModelCacheReuse:
+    """``_legacy_client_for()`` builds each per-call model once, then caches it."""
+
+    def test_repeated_override_builds_one_generative_model(self, fake_legacy_genai):
+        provider = _make_provider()
+
+        provider.generate("hello", model=OVERRIDE_MODEL)
+        provider.generate("hello", model=OVERRIDE_MODEL)
+        provider.generate_structured("hello", model=OVERRIDE_MODEL)
+
+        assert fake_legacy_genai.build_calls == [OVERRIDE_MODEL]
+        assert list(provider._legacy_model_cache) == [OVERRIDE_MODEL]
+
+    def test_cache_hit_returns_the_same_instance(self, fake_legacy_genai):
+        provider = _make_provider()
+
+        first = provider._legacy_client_for(OVERRIDE_MODEL)
+        second = provider._legacy_client_for(OVERRIDE_MODEL)
+
+        assert first is second
+        assert first is provider._legacy_model_cache[OVERRIDE_MODEL]
+        assert fake_legacy_genai.build_calls == [OVERRIDE_MODEL]
+
+    def test_distinct_overrides_are_cached_separately(self, fake_legacy_genai):
+        provider = _make_provider()
+
+        provider.generate("hello", model=OVERRIDE_MODEL)
+        provider.generate("hello", model=OTHER_MODEL)
+        provider.generate("hello", model=OVERRIDE_MODEL)
+
+        assert fake_legacy_genai.build_calls == [OVERRIDE_MODEL, OTHER_MODEL]
+        assert set(provider._legacy_model_cache) == {OVERRIDE_MODEL, OTHER_MODEL}
+        assert (
+            provider._legacy_model_cache[OVERRIDE_MODEL]
+            is not provider._legacy_model_cache[OTHER_MODEL]
+        )
+
+    def test_default_model_is_not_cached_or_rebuilt(self, fake_legacy_genai):
+        provider = _make_provider(model=CONSTRUCTION_MODEL)
+        construction_client = provider.client
+
+        provider.generate("hello")
+        provider.generate("hello", model=CONSTRUCTION_MODEL)
+
+        assert fake_legacy_genai.build_calls == []
+        assert provider._legacy_model_cache == {}
+        assert construction_client.generate_content.call_count == 2
+
+
+class TestLegacyModelCacheIsolation:
+    """The cache and the legacy SDK's module-level key stay per-instance."""
+
+    def test_configure_reasserted_with_this_key_before_every_call(
+        self, fake_legacy_genai
+    ):
+        provider = _make_provider(api_key="key-A")
+
+        provider.generate("hello", model=OVERRIDE_MODEL)
+        provider.generate("hello", model=OVERRIDE_MODEL)  # cache hit still re-asserts
+
+        assert fake_legacy_genai.configure.call_count == 2
+        for call in fake_legacy_genai.configure.call_args_list:
+            assert call.kwargs == {"api_key": "key-A"}
+
+    def test_two_instances_keep_separate_caches_and_keys(self, fake_legacy_genai):
+        provider_a = _make_provider(api_key="key-A")
+        provider_b = _make_provider(api_key="key-B")
+
+        provider_a.generate("hello", model=OVERRIDE_MODEL)
+        provider_b.generate("hello", model=OVERRIDE_MODEL)
+
+        # Same model name, but each instance built and cached its own object.
+        assert fake_legacy_genai.build_calls == [OVERRIDE_MODEL, OVERRIDE_MODEL]
+        assert (
+            provider_a._legacy_model_cache[OVERRIDE_MODEL]
+            is not provider_b._legacy_model_cache[OVERRIDE_MODEL]
+        )
+        assert fake_legacy_genai.configure.call_args_list[-2].kwargs == {
+            "api_key": "key-A"
+        }
+        assert fake_legacy_genai.configure.call_args_list[-1].kwargs == {
+            "api_key": "key-B"
+        }

tests/semantic_extract/test_temporal_extraction.py

@@ -18,29 +18,40 @@ from datetime import datetime, timezone
 from unittest.mock import MagicMock, patch
 
 # ── Mock optional heavyweight dependencies before any semantica import ──────
-sys.modules.setdefault("spacy", MagicMock())
-sys.modules.setdefault("instructor", MagicMock())
-_openai_mock = MagicMock()
-sys.modules.setdefault("openai", _openai_mock)
-sys.modules.setdefault("groq", MagicMock())
-sys.modules.setdefault("sentence_transformers", MagicMock())
-sys.modules.setdefault("transformers", MagicMock())
-sys.modules.setdefault("torch", MagicMock())
+_MOCKED_MODULES = [
+    "spacy",
+    "instructor",
+    "openai",
+    "groq",
+    "sentence_transformers",
+    "transformers",
+]
+_original_modules = {k: sys.modules.get(k) for k in _MOCKED_MODULES}
+
+for k in _MOCKED_MODULES:
+    sys.modules.setdefault(k, MagicMock())
 
 sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
 
-from semantica.semantic_extract.methods import extract_relations_llm
-from semantica.semantic_extract.ner_extractor import Entity
-from semantica.semantic_extract.schemas import (
+from semantica.semantic_extract.methods import extract_relations_llm  # noqa: E402
+
+for _key, _original in _original_modules.items():
+    if _original is None:
+        sys.modules.pop(_key, None)
+    else:
+        sys.modules[_key] = _original
+
+from semantica.semantic_extract.ner_extractor import Entity  # noqa: E402
+from semantica.semantic_extract.schemas import (  # noqa: E402
     RelationsResponse,
     RelationsWithTemporalResponse,
 )
-from semantica.kg.temporal_normalizer import TemporalNormalizer
-from semantica.utils.exceptions import TemporalAmbiguityWarning
-
+from semantica.kg.temporal_normalizer import TemporalNormalizer  # noqa: E402
+from semantica.utils.exceptions import TemporalAmbiguityWarning  # noqa: E402
 
 # ── Helpers ─────────────────────────────────────────────────────────────────
 
+
 def _make_entities():
     return [
         Entity(text="Apple", label="ORG", start_char=0, end_char=5),
@@ -56,15 +67,17 @@ def _ref_date():
 # Part 1 – extract_relations_llm() temporal flag
 # ============================================================================
 
+
 class TestTemporalExtractionFlag(unittest.TestCase):
 
     def setUp(self):
         from semantica.semantic_extract.methods import _result_cache
+
         _result_cache.clear()
 
     @patch("semantica.semantic_extract.methods.create_provider")
     def test_extract_temporal_bounds_true_adds_four_fields(self, mock_create):
-        """With extract_temporal_bounds=True all four temporal keys appear in metadata."""
+        """With extract_temporal_bounds=True all four temporal keys appear."""
         mock_prov = MagicMock()
         mock_prov.is_available.return_value = True
         mock_prov.generate_typed.return_value = RelationsWithTemporalResponse(
@@ -132,7 +145,9 @@ class TestTemporalExtractionFlag(unittest.TestCase):
         self.assertNotIn("temporal_source_text", meta)
 
     @patch("semantica.semantic_extract.methods.create_provider")
-    def test_no_temporal_signal_returns_zero_confidence_and_null_dates(self, mock_create):
+    def test_no_temporal_signal_returns_zero_confidence_and_null_dates(
+        self, mock_create
+    ):
         """When LLM returns no temporal signal, confidence=0.0 and dates are null."""
         mock_prov = MagicMock()
         mock_prov.is_available.return_value = True
@@ -197,10 +212,12 @@ class TestTemporalExtractionFlag(unittest.TestCase):
 
     @patch("semantica.semantic_extract.methods.create_provider")
     def test_correct_schema_used_when_temporal_true(self, mock_create):
-        """generate_typed is called with RelationsWithTemporalResponse when flag=True."""
+        """generate_typed is called with RelationsWithTemporalResponse."""
         mock_prov = MagicMock()
         mock_prov.is_available.return_value = True
-        mock_prov.generate_typed.return_value = RelationsWithTemporalResponse(relations=[])
+        mock_prov.generate_typed.return_value = RelationsWithTemporalResponse(
+            relations=[]
+        )
         mock_create.return_value = mock_prov
 
         extract_relations_llm(
@@ -235,6 +252,7 @@ class TestTemporalExtractionFlag(unittest.TestCase):
 # Part 2 – TemporalNormalizer: relative dates
 # ============================================================================
 
+
 class TestTemporalNormalizerRelativeDates(unittest.TestCase):
 
     def setUp(self):
@@ -313,10 +331,13 @@ class TestTemporalNormalizerRelativeDates(unittest.TestCase):
 # Part 3 – TemporalNormalizer: partial / structured dates
 # ============================================================================
 
+
 class TestTemporalNormalizerPartialDates(unittest.TestCase):
 
     def setUp(self):
-        self.tn = TemporalNormalizer(reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc))
+        self.tn = TemporalNormalizer(
+            reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc)
+        )
 
     def test_year_only(self):
         result = self.tn.normalize("2021")
@@ -396,10 +417,13 @@ class TestTemporalNormalizerPartialDates(unittest.TestCase):
 # Part 4 – TemporalNormalizer: ambiguous formats
 # ============================================================================
 
+
 class TestTemporalNormalizerAmbiguity(unittest.TestCase):
 
     def setUp(self):
-        self.tn = TemporalNormalizer(reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc))
+        self.tn = TemporalNormalizer(
+            reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc)
+        )
 
     def test_ambiguous_slash_date_raises_warning_and_returns_none(self):
         with warnings.catch_warnings(record=True) as w:
@@ -432,14 +456,19 @@ class TestTemporalNormalizerAmbiguity(unittest.TestCase):
 # Part 5 – TemporalNormalizer: domain phrase map
 # ============================================================================
 
+
 class TestTemporalNormalizerDomainPhrases(unittest.TestCase):
 
     def setUp(self):
-        self.tn = TemporalNormalizer(reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc))
+        self.tn = TemporalNormalizer(
+            reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc)
+        )
 
     def _assert_recognized(self, phrase):
         result = self.tn.normalize_phrase(phrase)
-        self.assertIsNotNone(result, f"Expected phrase {phrase!r} to be recognized but got None")
+        self.assertIsNotNone(
+            result, f"Expected phrase {phrase!r} to be recognized but got None"
+        )
         return result
 
     # General / Policy
@@ -522,6 +551,7 @@ class TestTemporalNormalizerDomainPhrases(unittest.TestCase):
 # Part 6 – TemporalNormalizer: custom phrase map
 # ============================================================================
 
+
 class TestTemporalNormalizerCustomPhraseMap(unittest.TestCase):
 
     def setUp(self):
@@ -565,10 +595,12 @@ class TestTemporalNormalizerCustomPhraseMap(unittest.TestCase):
 # Part 7 – Full pipeline: extract → normalize → BiTemporalFact
 # ============================================================================
 
+
 class TestFullPipelineTemporalToBiTemporal(unittest.TestCase):
 
     def setUp(self):
         from semantica.semantic_extract.methods import _result_cache
+
         _result_cache.clear()
 
     @patch("semantica.semantic_extract.methods.create_provider")
@@ -620,10 +652,12 @@ class TestFullPipelineTemporalToBiTemporal(unittest.TestCase):
         self.assertEqual(vf[0].day, 1)
 
         # Feed into BiTemporalFact
-        fact = BiTemporalFact.from_relationship({
-            "valid_from": "2014-05-01T00:00:00Z",
-            "valid_until": None,
-        })
+        fact = BiTemporalFact.from_relationship(
+            {
+                "valid_from": "2014-05-01T00:00:00Z",
+                "valid_until": None,
+            }
+        )
         self.assertIsNotNone(fact.valid_from)
         self.assertEqual(fact.valid_from.year, 2014)
         self.assertEqual(fact.valid_from.month, 5)
@@ -660,7 +694,9 @@ class TestFullPipelineTemporalToBiTemporal(unittest.TestCase):
             extract_temporal_bounds=True,
         )
         meta = rels[0].metadata
-        tn = TemporalNormalizer(reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc))
+        tn = TemporalNormalizer(
+            reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc)
+        )
 
         vf = tn.normalize(meta["valid_from"])
         vu = tn.normalize(meta["valid_until"])

tests/test_cli_commands.py

@@ -2114,6 +2114,30 @@ class TestMCP:
         assert "Traceback" not in result.output
         assert "Invalid JSON" in result.output
 
+    def test_call_failure_global_json_mode_keeps_stdout_clean(self, runner):
+        """Under global --json, stdout must stay machine-readable: failures are
+        emitted as structured JSON on stderr, never as a Rich panel on stdout."""
+        result = runner.invoke(
+            cli_module.main,
+            ["--json", "mcp", "call", "some_tool", "--args", "{bad json}"],
+        )
+        assert result.exit_code != 0
+        assert result.stdout == ""
+        err = json.loads(result.stderr)
+        assert err["error"].startswith("Invalid JSON in --args")
+        assert err["type"] == "ClickException"
+
+    def test_call_failure_local_json_mode_keeps_stdout_clean(self, runner):
+        """The subcommand's own --json flag promises the same stream contract."""
+        result = runner.invoke(
+            cli_module.main,
+            ["mcp", "call", "some_tool", "--args", "{bad json}", "--json"],
+        )
+        assert result.exit_code != 0
+        assert result.stdout == ""
+        err = json.loads(result.stderr)
+        assert err["error"].startswith("Invalid JSON in --args")
+
     def test_call_import_error_is_clean(self, runner):
         with patch("builtins.__import__", side_effect=lambda n, *a, **k: (
             (_ for _ in ()).throw(ImportError(n))
@@ -2278,7 +2302,7 @@ class TestDoctorEmbeddings:
         checks = self._doctor_checks(runner)
         st = checks["Embeddings (sentence-transformers)"]
         assert st["status"] == "fail"
-        assert st["hint"] == "pip install sentence-transformers"
+        assert st["hint"] == "pip install 'semantica[embeddings-local]'"
 
     def test_deep_probe_detects_fallback_active(self, runner, monkeypatch):
         self._with_fake_st(monkeypatch)

tests/test_issue_1513_slim_core.py

@@ -0,0 +1,454 @@
+from pathlib import Path
+from unittest.mock import patch
+import pytest
+
+from semantica.parse.docx_parser import DOCXParser
+from semantica.parse.excel_parser import ExcelParser
+from semantica.parse.html_parser import HTMLParser
+from semantica.parse.xml_parser import XMLParser
+from semantica.utils.exceptions import ProcessingError
+
+
+def _load_toml(file_path: Path) -> dict:
+    """Load and parse a TOML file across Python 3.8-3.14+ without mode mismatches."""
+    content = file_path.read_text(encoding="utf-8")
+    try:
+        import tomllib  # Python 3.11+ standard library
+
+        return tomllib.loads(content)
+    except ImportError:
+        try:
+            import tomli  # Fast PEP 680 compatible parser for Python < 3.11
+
+            return tomli.loads(content)
+        except ImportError:
+            import toml  # Fallback toml parser
+
+            return toml.loads(content)
+
+
+def test_core_dependencies_count():
+    """pyproject.toml must contain exactly 22 unique core dependencies."""
+    repo_root = Path(__file__).resolve().parents[1]
+    data = _load_toml(repo_root / "pyproject.toml")
+    deps = data["project"]["dependencies"]
+    normalized_names = {
+        d.split(";")[0].split(">=")[0].split("<")[0].split("==")[0].strip()
+        for d in deps
+    }
+    expected_22 = {
+        "numpy",
+        "pandas",
+        "scipy",
+        "scikit-learn",
+        "rdflib",
+        "networkx",
+        "requests",
+        "chardet",
+        "protobuf",
+        "grpcio",
+        "pillow",
+        "pydantic",
+        "click",
+        "rich",
+        "tqdm",
+        "pyyaml",
+        "toml",
+        "python-dotenv",
+        "loguru",
+        "structlog",
+        "httpx",
+        "pyarrow",
+    }
+    assert normalized_names == expected_22
+    assert len(normalized_names) == 22
+
+
+def test_optional_extras_defined():
+    """All required optional extras must be declared in pyproject.toml."""
+    repo_root = Path(__file__).resolve().parents[1]
+    data = _load_toml(repo_root / "pyproject.toml")
+    extras = data["project"]["optional-dependencies"]
+    for extra in [
+        "documents",
+        "ingest-git",
+        "embeddings-local",
+        "nlp-spacy",
+        "viz",
+        "media",
+        "vectorstore-faiss",
+        "graph-embeddings",
+        "all",
+    ]:
+        assert extra in extras, f"Missing extra {extra}"
+    all_extra_str = str(extras["all"])
+    for expected_ref in [
+        "documents",
+        "ingest-git",
+        "embeddings-local",
+        "nlp-spacy",
+        "viz",
+        "media",
+        "graph-embeddings",
+        "vectorstore-all",
+    ]:
+        assert expected_ref in all_extra_str, f"Missing {expected_ref} in all"
+    # And vectorstore-faiss is in vectorstore-all
+    assert "vectorstore-faiss" in str(extras["vectorstore-all"])
+
+    # Verify nlp-spacy does not declare thinc directly (Qodo bot issue 1)
+    nlp_spacy_deps = str(extras.get("nlp-spacy", []))
+    assert "thinc" not in nlp_spacy_deps, "nlp-spacy should not directly declare thinc"
+    assert "spacy" in nlp_spacy_deps, "nlp-spacy must declare spacy"
+
+
+def test_core_modules_importable():
+    """Core modules must be importable without requiring optional extras."""
+    import semantica
+    import semantica.cli
+    import semantica.parse
+    import semantica.ingest
+    import semantica.embeddings
+    import semantica.export
+    import semantica.kg
+    import semantica.vector_store
+    import semantica.visualization
+    import semantica.semantic_extract
+    import semantica.pipeline
+
+    assert semantica.__version__ is not None
+
+
+def test_docx_parser_lazy_construction_and_parse_hint():
+    with patch("semantica.parse.docx_parser.Document", None):
+        parser = DOCXParser()
+        assert parser is not None
+        with pytest.raises(ProcessingError, match=r"semantica\[documents\]"):
+            parser.parse("nonexistent.docx")
+
+
+def test_excel_parser_lazy_construction_and_parse_hint():
+    with patch("semantica.parse.excel_parser.load_workbook", None):
+        parser = ExcelParser()
+        assert parser is not None
+        with pytest.raises(ProcessingError, match=r"semantica\[documents\]"):
+            parser.parse("nonexistent.xlsx")
+
+
+def test_html_parser_lazy_construction_and_parse_hint():
+    with patch("semantica.parse.html_parser.BeautifulSoup", None):
+        parser = HTMLParser()
+        assert parser is not None
+        with pytest.raises(ProcessingError, match=r"semantica\[documents\]"):
+            parser.parse("nonexistent.html")
+
+
+def test_xml_parser_etree_fallback():
+    with patch("semantica.parse.xml_parser.etree", None):
+        parser = XMLParser()
+        assert parser is not None
+        result = parser.parse("<root><item id='1'>Test</item></root>")
+        assert result is not None
+        assert result.root is not None
+        assert result.root.tag == "root"
+
+
+def test_xml_parser_lxml_explicit_requires_documents_extra():
+    with patch("semantica.parse.xml_parser.etree", None):
+        parser = XMLParser(engine="lxml")
+        assert parser is not None
+        with pytest.raises(ProcessingError, match=r"semantica\[documents\]"):
+            parser.parse("<root/>")
+
+
+def test_xml_ingestor_missing_hint():
+    with patch("semantica.ingest.xml_ingestor.etree", None):
+        from semantica.ingest.xml_ingestor import XMLIngestor
+
+        with pytest.raises(ImportError, match=r"semantica\[documents\]"):
+            XMLIngestor()
+
+
+def test_xml_ingestor_package_import_missing_hint():
+    import semantica.ingest as ingest_mod
+
+    ingest_mod.__dict__.pop("XMLIngestor", None)
+    with patch("semantica.ingest.xml_ingestor.etree", None):
+        with pytest.raises(ImportError, match=r"semantica\[documents\]"):
+            _ = ingest_mod.XMLIngestor
+
+
+def test_repo_ingestor_missing_hint():
+    with patch("semantica.ingest.repo_ingestor.git", None):
+        from semantica.ingest.repo_ingestor import RepoIngestor
+
+        with pytest.raises(ImportError, match=r"semantica\[ingest-git\]"):
+            RepoIngestor()
+
+
+def test_repo_ingestor_package_import_missing_hint():
+    import semantica.ingest as ingest_mod
+
+    ingest_mod.__dict__.pop("RepoIngestor", None)
+    with patch("semantica.ingest.repo_ingestor.git", None):
+        with pytest.raises(ImportError, match=r"semantica\[ingest-git\]"):
+            _ = ingest_mod.RepoIngestor
+
+
+def test_git_analyzer_package_import_succeeds_without_git():
+    import semantica.ingest as ingest_mod
+
+    ingest_mod.__dict__.pop("GitAnalyzer", None)
+    with patch("semantica.ingest.repo_ingestor.git", None):
+        analyzer_cls = ingest_mod.GitAnalyzer
+        assert analyzer_cls is not None
+        analyzer = analyzer_cls()
+        assert analyzer is not None
+
+
+def test_salesforce_ingestor_package_import_missing_hint():
+    import semantica.ingest as ingest_mod
+
+    ingest_mod.__dict__.pop("SalesforceIngestor", None)
+    with patch("semantica.ingest.salesforce_ingestor.SALESFORCE_AVAILABLE", False):
+        with pytest.raises(ImportError, match=r"semantica\[db-salesforce\]"):
+            _ = ingest_mod.SalesforceIngestor
+
+
+def test_parse_methods_dynamic_default_resolution():
+    from semantica.parse.methods import (
+        get_parse_method,
+        list_available_methods,
+        parse_document,
+    )
+
+    assert get_parse_method("document", "default") == parse_document
+    methods = list_available_methods()
+    assert "default" in methods.get("document", [])
+    assert "default" in methods.get("structured", [])
+
+
+def test_node_embedder_gensim_missing_hint():
+    with patch("semantica.kg.node_embeddings.GENSIM_AVAILABLE", False):
+        from semantica.kg.node_embeddings import NodeEmbedder
+
+        with pytest.raises(ImportError, match=r"semantica\[graph-embeddings\]"):
+            NodeEmbedder()
+
+
+def test_faiss_store_missing_hint():
+    with patch("semantica.vector_store.faiss_store.FAISS_AVAILABLE", False):
+        from semantica.vector_store.faiss_store import FAISSIndexBuilder, FAISSStore
+
+        builder = FAISSIndexBuilder(128)
+        with pytest.raises(ProcessingError, match=r"semantica\[vectorstore-faiss\]"):
+            builder.build_index("flat")
+        store = FAISSStore(128)
+        with pytest.raises(ProcessingError, match=r"semantica\[vectorstore-faiss\]"):
+            store.load_index("nonexistent.faiss")
+
+
+def test_visualization_missing_hint():
+    import numpy as np
+    from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
+
+    # Plotly is checked via px and go in _check_dependencies
+    with patch("semantica.visualization.embedding_visualizer.px", None):
+        visualizer = EmbeddingVisualizer()
+        with pytest.raises(
+            ProcessingError, match=r"Plotly is required.*semantica\[viz\]"
+        ):
+            visualizer.visualize_2d_projection(np.array([[0.1, 0.2], [0.3, 0.4]]))
+
+    with patch("semantica.visualization.embedding_visualizer.go", None):
+        visualizer = EmbeddingVisualizer()
+        with pytest.raises(
+            ProcessingError, match=r"Plotly is required.*semantica\[viz\]"
+        ):
+            visualizer.visualize_2d_projection(np.array([[0.1, 0.2], [0.3, 0.4]]))
+
+
+def test_visualization_umap_missing_hint():
+    import numpy as np
+    from unittest.mock import MagicMock
+    from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
+
+    # Stand in for Plotly so we reach dimensionality reduction
+    with patch("semantica.visualization.embedding_visualizer.px", MagicMock()), patch(
+        "semantica.visualization.embedding_visualizer.go", MagicMock()
+    ), patch("semantica.visualization.embedding_visualizer.umap", None):
+        visualizer = EmbeddingVisualizer()
+        # High-dimensional embeddings (>2D) trigger dimensionality reduction
+        # with method="umap"
+        embeddings = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
+        with pytest.raises(
+            ProcessingError, match=r"UMAP is required.*semantica\[viz\]"
+        ):
+            visualizer.visualize_2d_projection(embeddings, method="umap")
+
+        # Also verify 3D projection triggers the same actionable error on >3D embeddings
+        embeddings_4d = np.array(
+            [[0.1, 0.2, 0.3, 0.4], [0.5, 0.6, 0.7, 0.8], [0.9, 1.0, 1.1, 1.2]]
+        )
+        with pytest.raises(
+            ProcessingError, match=r"UMAP is required.*semantica\[viz\]"
+        ):
+            visualizer.visualize_3d_projection(embeddings_4d, method="umap")
+
+
+def test_visualization_sklearn_missing_hint():
+    import numpy as np
+    from unittest.mock import MagicMock
+    from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
+
+    # Stand in for Plotly so we reach dimensionality reduction
+    with patch("semantica.visualization.embedding_visualizer.px", MagicMock()), patch(
+        "semantica.visualization.embedding_visualizer.go", MagicMock()
+    ):
+        visualizer = EmbeddingVisualizer()
+        embeddings = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
+
+        # Test direct dependency check
+        with patch("semantica.visualization.embedding_visualizer.PCA", None):
+            with pytest.raises(ProcessingError, match=r"scikit-learn is required"):
+                visualizer._check_dependencies(require_sklearn=True)
+
+        with patch("semantica.visualization.embedding_visualizer.PCA", None):
+            with pytest.raises(ProcessingError, match=r"scikit-learn is required"):
+                visualizer.visualize_2d_projection(embeddings, method="pca")
+
+        with patch("semantica.visualization.embedding_visualizer.TSNE", None):
+            with pytest.raises(ProcessingError, match=r"scikit-learn is required"):
+                visualizer.visualize_2d_projection(embeddings, method="tsne")
+
+
+def test_visualization_options_collision_free():
+    """Options like n_components, perplexity must not cause keyword collisions."""
+    import numpy as np
+    from unittest.mock import MagicMock
+    from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
+
+    mock_pca = MagicMock()
+    mock_tsne = MagicMock()
+    mock_umap_cls = MagicMock()
+    mock_umap_module = MagicMock()
+    mock_umap_module.UMAP = mock_umap_cls
+
+    with patch("semantica.visualization.embedding_visualizer.PCA", mock_pca), patch(
+        "semantica.visualization.embedding_visualizer.TSNE", mock_tsne
+    ), patch(
+        "semantica.visualization.embedding_visualizer.umap", mock_umap_module
+    ), patch(
+        "semantica.visualization.embedding_visualizer.px", MagicMock()
+    ), patch(
+        "semantica.visualization.embedding_visualizer.go", MagicMock()
+    ):
+        visualizer = EmbeddingVisualizer()
+        embeddings = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
+
+        # PCA with n_components
+        visualizer.visualize_2d_projection(embeddings, method="pca", n_components=2)
+        # TSNE with perplexity and random_state
+        visualizer.visualize_2d_projection(
+            embeddings, method="tsne", perplexity=1, random_state=42
+        )
+        # UMAP with n_neighbors and min_dist
+        visualizer.visualize_2d_projection(
+            embeddings, method="umap", n_neighbors=2, min_dist=0.1
+        )
+        # 3D with n_components
+        visualizer.visualize_3d_projection(embeddings, method="pca", n_components=3)
+
+
+def test_spacy_load_missing_hint():
+    from semantica.semantic_extract.methods import load_spacy_model
+
+    with patch("semantica.semantic_extract.methods.spacy", None):
+        with pytest.raises(ImportError, match=r"semantica\[nlp-spacy\]"):
+            load_spacy_model("en_core_web_sm")
+
+
+def test_xml_parser_handles_comments():
+    xml_content = (
+        "<root><!-- top comment --><item id='1'>Value</item>"
+        "<!-- bottom comment --></root>"
+    )
+    # lxml engine
+    p_lxml = XMLParser(engine="lxml")
+    res_lxml = p_lxml.parse(xml_content)
+    assert res_lxml.root.tag == "root"
+    assert len(res_lxml.root.children) == 1
+    assert res_lxml.root.children[0].tag == "item"
+    assert res_lxml.root.children[0].text == "Value"
+
+    # etree engine
+    p_etree = XMLParser(engine="etree")
+    res_etree = p_etree.parse(xml_content)
+    assert res_etree.root.tag == "root"
+    assert len(res_etree.root.children) == 1
+    assert res_etree.root.children[0].tag == "item"
+    assert res_etree.root.children[0].text == "Value"
+
+
+def test_public_api_ingestor_handles_xml_comments():
+    from semantica.ingest.public_api_ingestor import PublicAPIIngestor
+
+    xml_content = "<root><!-- comment --><item id='1'>Value</item></root>"
+    ingestor = PublicAPIIngestor(rate_limit_delay=0)
+
+    # 1. Default (defusedxml if available)
+    parsed = ingestor._parse_xml(xml_content)
+    assert parsed["tag"] == "root"
+    assert len(parsed["children"]) == 1
+    assert parsed["children"][0]["tag"] == "item"
+    assert parsed["children"][0]["text"] == "Value"
+
+    # 2. lxml fallback
+    with patch("semantica.ingest.public_api_ingestor.safe_xml_etree", None):
+        parsed_lxml = ingestor._parse_xml(xml_content)
+        assert parsed_lxml["tag"] == "root"
+        assert len(parsed_lxml["children"]) == 1
+        assert parsed_lxml["children"][0]["tag"] == "item"
+        assert parsed_lxml["children"][0]["text"] == "Value"
+
+
+def test_huggingface_model_loader_catches_oserror():
+    import builtins
+    from unittest.mock import MagicMock
+    from semantica.semantic_extract.providers import HuggingFaceModelLoader
+
+    mock_torch = MagicMock()
+    mock_torch.Tensor = type("Tensor", (), {})
+    with patch.dict("sys.modules", {"torch": mock_torch}):
+        loader = HuggingFaceModelLoader()
+        # 1. Test ModuleNotFoundError / ImportError
+        with patch.dict("sys.modules", {"transformers": None}):
+            with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
+                loader.load_ner_model("bert-base-cased")
+
+            with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
+                loader.load_relation_model("bert-base-cased")
+
+            with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
+                loader.load_triplet_model("t5-base")
+
+        # 2. Test OSError (e.g. corrupt DLL / missing shared library)
+        real_import = builtins.__import__
+
+        def fake_import(name, *args, **kwargs):
+            if name == "transformers":
+                raise OSError("DLL load failed")
+            return real_import(name, *args, **kwargs)
+
+        try:
+            builtins.__import__ = fake_import
+            with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
+                loader.load_ner_model("bert-base-cased-oserror")
+
+            with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
+                loader.load_relation_model("bert-base-cased-oserror")
+
+            with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
+                loader.load_triplet_model("t5-base-oserror")
+        finally:
+            builtins.__import__ = real_import

tests/test_mcp_semantic_retrieval.py

@@ -0,0 +1,787 @@
+"""
+Tests for the MCP semantic retrieval tools (#1235).
+
+Covers the six acceptance behaviours proposed in the issue:
+
+1. store_document chunks content and stores it in a real supported
+   vector backend with provenance metadata (status / version / hash).
+2. retrieve_context returns semantically relevant chunks with scores
+   and provenance, combined with related graph relationships.
+3. update_document replaces stored content under (source, version).
+4. remove_document deletes every chunk of a document.
+5. Remove-then-store does not collide with surviving in-memory ids
+   (regression guard for the #1029 interaction).
+6. The same tool set works against the sqlite backend (real persistent
+   store, skipped when the sqlite_vec extension is missing).
+"""
+
+import os
+import sys
+import tempfile
+import unittest
+import zlib
+from unittest.mock import patch
+
+import numpy as np
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+import semantica_mcp.mcp.session as session
+import semantica.embeddings as _embeddings_pkg
+import semantica.vector_store.vector_store as _vs_module
+from semantica_mcp.mcp.session import get_vector_store, reset_vector_store
+from semantica_mcp.mcp.tools import TOOL_DEFINITIONS
+from semantica_mcp.mcp.tools.retrieval import (
+    _chunk_id,
+    _chunk_text,
+    handle_remove_document,
+    handle_retrieve_context,
+    handle_store_document,
+    handle_update_document,
+)
+
+
+class FakeTextEmbedder:
+    def __init__(self, dim: int = 64):
+        self.dim = dim
+
+    def get_embedding_dimension(self) -> int:
+        return self.dim
+
+
+class FakeEmbedder:
+    """
+    Deterministic keyword-bag embedder on a fixed dimension.
+
+    Same words land on the same dimensions, so a query sharing vocabulary
+    with a chunk scores higher than one that does not — enough signal for
+    ranking assertions without any model download.  crc32 keeps the
+    word-to-dimension mapping stable across processes (unlike builtin
+    hash(), whose per-process salt would make collisions flaky), and 64
+    dims keep the test keywords collision-free.
+    """
+
+    def __init__(self, dim: int = 64):
+        self.dim = dim
+        self.text_embedder = FakeTextEmbedder(dim)
+
+    def get_text_method(self) -> str:
+        return "fake"
+
+    def generate_embeddings(self, texts):
+        out = []
+        for t in texts:
+            v = np.zeros(self.dim, dtype=float)
+            for w in str(t).lower().split():
+                if w == "x":
+                    # "x" is the filler make_doc pads with; treating it
+                    # as a stopword keeps vectors keyword-driven instead
+                    # of filler-dominated.
+                    continue
+                v[zlib.crc32(w.encode("utf-8")) % self.dim] += 1.0
+            norm = np.linalg.norm(v)
+            if norm:
+                v /= norm
+            out.append(v)
+        return np.array(out)
+
+
+def make_doc(*keywords) -> str:
+    """
+    Build filler text with exactly one keyword per chunk.
+
+    With the default 1000 window / 200 overlap, chunk i covers
+    [800*i, 800*i+1000).  Keyword i is placed at 800*i + 300, which sits
+    inside chunk i only — clear of both neighbouring overlap zones.
+    Filler is spaced "x " tokens, which the fake embedder treats as a
+    stopword, so chunk vectors are keyword-driven.
+    """
+    filler = "x "
+    parts = []
+    pos = 0
+    for i, word in enumerate(keywords):
+        target = 800 * i + 300
+        parts.append(filler * ((target - pos) // 2))
+        parts.append(word + " ")
+        pos = target + len(word) + 1
+    parts.append(filler * 30)
+    return "".join(parts)
+
+
+def patch_embedding_generators():
+    """
+    Patch every EmbeddingGenerator construction site with FakeEmbedder.
+
+    The real EmbeddingGenerator probes FastEmbed / sentence-transformers
+    on init; where those packages are installed but the model is not
+    cached, the probe blocks on a full TCP connect timeout (~30s each).
+    VectorStore's in-memory branch builds one internally, so tests patch
+    both import sites to keep the suite fast and network-free.
+    """
+    return (
+        patch.object(_embeddings_pkg, "EmbeddingGenerator", FakeEmbedder),
+        patch.object(_vs_module, "EmbeddingGenerator", FakeEmbedder),
+    )
+
+
+def _clear_retrieval_env():
+    for var in ("SEMANTICA_VECTOR_PATH", "SEMANTICA_VECTOR_BACKEND", "SEMANTICA_VECTOR_DB_PATH"):
+        os.environ.pop(var, None)
+
+
+class InmemoryBackendTestBase(unittest.TestCase):
+    def setUp(self):
+        _clear_retrieval_env()
+        session._embedder = FakeEmbedder()
+        session._vector_store = None
+        session._graph = None
+        self._patches = patch_embedding_generators()
+        for p in self._patches:
+            p.start()
+
+    def tearDown(self):
+        for p in self._patches:
+            p.stop()
+        session._embedder = None
+        reset_vector_store()
+        session._graph = None
+        _clear_retrieval_env()
+
+
+class TestChunking(InmemoryBackendTestBase):
+    def test_fixed_window_with_overlap(self):
+        text = "a" * 2600
+        chunks = _chunk_text(text, 1000, 200)
+        self.assertEqual([c[:2] for c in chunks], [(0, 1000), (800, 1800), (1600, 2600)])
+        self.assertTrue(all(c == text[s:e] for s, e, c in chunks))
+
+    def test_short_text_single_chunk(self):
+        chunks = _chunk_text("short", 1000, 200)
+        self.assertEqual(chunks, [(0, 5, "short")])
+
+    def test_overlap_must_be_smaller_than_window(self):
+        with self.assertRaises(ValueError):
+            _chunk_text("abc", 200, 200)
+
+    def test_chunk_id_is_stable_and_position_sensitive(self):
+        a = _chunk_id("src", "v1", 0, "hello")
+        b = _chunk_id("src", "v1", 0, "hello")
+        c = _chunk_id("src", "v1", 1, "hello")
+        self.assertEqual(a, b)
+        self.assertNotEqual(a, c)
+
+
+class TestStoreDocument(InmemoryBackendTestBase):
+    def test_chunks_carry_provenance_metadata(self):
+        result = handle_store_document(
+            {
+                "content": make_doc("alpha", "beta"),
+                "source": "policy_manual#p12",
+                "authority": "official",
+                "version": "v2",
+                "project": "lending",
+            }
+        )
+        self.assertNotIn("error", result)
+        self.assertEqual(result["status"], "stored")
+        self.assertEqual(result["chunk_count"], len(result["chunk_ids"]))
+
+        store = get_vector_store()
+        first = next(
+            m
+            for m in store.metadata.values()
+            if m.get("source") == "policy_manual#p12" and m.get("chunk_index") == 0
+        )
+        self.assertEqual(first["chunk_id"], result["chunk_ids"][0])
+        self.assertEqual(first["authority"], "official")
+        self.assertEqual(first["version"], "v2")
+        self.assertEqual(first["project"], "lending")
+        self.assertEqual(first["status"], "active")
+        self.assertEqual(first["hash"], result["hash"])
+        self.assertEqual(first["char_start"], 0)
+
+    def test_identical_content_is_a_noop(self):
+        args = {"content": "same content", "source": "doc", "authority": "official"}
+        first = handle_store_document(args)
+        second = handle_store_document(args)
+        self.assertEqual(second["status"], "unchanged")
+        self.assertEqual(second["chunk_ids"], first["chunk_ids"])
+        self.assertEqual(get_vector_store().count(), first["chunk_count"])
+
+    def test_missing_authority_rejected(self):
+        result = handle_store_document({"content": "text", "source": "doc"})
+        self.assertIn("error", result)
+
+    def test_caller_metadata_cannot_override_provenance(self):
+        result = handle_store_document(
+            {
+                "content": make_doc("alpha"),
+                "source": "real_source",
+                "authority": "official",
+                "metadata": {
+                    "source": "spoofed_source",
+                    "authority": "backdated",
+                    "status": "tombstone",
+                    "hash": "deadbeef",
+                    "version": "v99",
+                    "project": "shadow_project",
+                    "dept": "risk",
+                },
+            }
+        )
+        self.assertNotIn("error", result)
+
+        store = get_vector_store()
+        meta = next(
+            m
+            for m in store.metadata.values()
+            if m.get("chunk_id") == result["chunk_ids"][0]
+        )
+        self.assertEqual(meta["source"], "real_source")
+        self.assertEqual(meta["authority"], "official")
+        self.assertEqual(meta["status"], "active")
+        self.assertEqual(meta["hash"], result["hash"])
+        self.assertEqual(meta["version"], "v1")
+        self.assertNotIn("project", meta)
+        # Non-provenance keys still land.
+        self.assertEqual(meta["dept"], "risk")
+
+        # Provenance stays intact, so the idempotent no-op still works.
+        again = handle_store_document(
+            {
+                "content": make_doc("alpha"),
+                "source": "real_source",
+                "authority": "official",
+            }
+        )
+        self.assertEqual(again["status"], "unchanged")
+
+    def test_non_dict_metadata_rejected(self):
+        result = handle_store_document(
+            {"content": "text", "source": "doc", "authority": "official", "metadata": ["bad"]}
+        )
+        self.assertIn("error", result)
+
+
+class TestRetrieveContext(InmemoryBackendTestBase):
+    def setUp(self):
+        super().setUp()
+        handle_store_document(
+            {
+                "content": make_doc("approval", "collateral", "interest"),
+                "source": "lending_policy",
+                "authority": "official",
+                "project": "lending",
+            }
+        )
+        handle_store_document(
+            {
+                "content": make_doc("payment", "refund"),
+                "source": "billing_faq",
+                "authority": "draft",
+                "project": "billing",
+            }
+        )
+
+    def test_relevant_chunks_ranked_with_provenance(self):
+        result = handle_retrieve_context({"query": "collateral", "top_k": 3})
+        self.assertNotIn("error", result)
+        self.assertGreater(result["count"], 0)
+        relevant = [r for r in result["results"] if r["score"] and r["score"] > 0]
+        self.assertTrue(relevant)
+        top = relevant[0]
+        self.assertIn("collateral", top["text"])
+        self.assertEqual(top["source"], "lending_policy")
+        self.assertEqual(top["authority"], "official")
+        self.assertEqual(top["version"], "v1")
+        self.assertEqual(top["status"], "active")
+        self.assertTrue(top["hash"])
+        self.assertIsInstance(top["score"], float)
+
+    def test_top_k_is_capped_at_ten(self):
+        # 13 chunks (one keyword per chunk) so the cap is actually hit;
+        # with fewer stored chunks the assertion would pass trivially.
+        handle_store_document(
+            {
+                "content": make_doc(*["k%02d" % i for i in range(1, 14)]),
+                "source": "capdoc",
+                "authority": "official",
+            }
+        )
+        result = handle_retrieve_context({"query": "k01", "top_k": 99})
+        self.assertEqual(result["count"], 10)
+
+    def test_project_filter_narrows_results(self):
+        result = handle_retrieve_context({"query": "collateral", "project": "billing"})
+        for r in result["results"]:
+            self.assertEqual(r["project"], "billing")
+
+    def test_graph_relationships_attached(self):
+        graph = session.get_graph()
+        graph.add_node(
+            node_id="policy_doc_lending_policy",
+            label="Lending policy doc",
+            node_type="Document",
+            metadata={"source": "lending_policy"},
+        )
+        graph.add_node(node_id="risk_team", label="Risk team", node_type="Team")
+        graph.add_edge(
+            source_id="policy_doc_lending_policy",
+            target_id="risk_team",
+            edge_type="OWNED_BY",
+        )
+        result = handle_retrieve_context({"query": "collateral"})
+        self.assertGreaterEqual(len(result["graph_context"]), 1)
+        rel = result["graph_context"][0]
+        self.assertEqual(rel["node"]["source"], "lending_policy")
+        self.assertEqual(rel["related"]["id"], "risk_team")
+        self.assertEqual(rel["relationship"], "OWNED_BY")
+
+    def test_empty_query_rejected(self):
+        result = handle_retrieve_context({"query": "  "})
+        self.assertIn("error", result)
+
+
+class TestUpdateDocument(InmemoryBackendTestBase):
+    def test_update_replaces_chunks(self):
+        handle_store_document(
+            {
+                "content": make_doc("oldterm", "legacy"),
+                "source": "handbook",
+                "authority": "official",
+            }
+        )
+        result = handle_update_document(
+            {
+                "content": make_doc("newterm"),
+                "source": "handbook",
+                "version": "v1",
+            }
+        )
+        self.assertEqual(result["status"], "updated")
+        self.assertEqual(result["chunk_count"], 1)
+
+        hits = handle_retrieve_context({"query": "newterm"})["results"]
+        hits = [h for h in hits if h["score"] and h["score"] > 0]
+        self.assertTrue(hits and "newterm" in hits[0]["text"])
+        stale = handle_retrieve_context({"query": "oldterm"})["results"]
+        stale = [h for h in stale if h["score"] and h["score"] > 0]
+        self.assertEqual(stale, [])
+        # Authority is inherited from the stored version when omitted.
+        self.assertEqual(hits[0]["authority"], "official")
+        self.assertEqual(get_vector_store().count(), 1)
+
+    def test_update_rolls_back_when_new_write_fails(self):
+        handle_store_document(
+            {
+                "content": make_doc("oldterm", "legacy"),
+                "source": "handbook",
+                "authority": "official",
+            }
+        )
+        store = get_vector_store()
+        real_store_vectors = store.store_vectors
+
+        def failing_write(vectors, metas):
+            if any("phoenix" in (m.get("text") or "") for m in metas):
+                raise RuntimeError("simulated write failure")
+            return real_store_vectors(vectors, metas)
+
+        with patch.object(store, "store_vectors", side_effect=failing_write):
+            result = handle_update_document(
+                {"content": make_doc("phoenix"), "source": "handbook"}
+            )
+        self.assertIn("error", result)
+        self.assertIn("simulated write failure", result["error"])
+
+        # The old document must survive the failed replacement, with no
+        # trace of the new content.
+        store = get_vector_store()
+        self.assertEqual(store.count(), 2)
+        old = [
+            h
+            for h in handle_retrieve_context({"query": "oldterm"})["results"]
+            if h["score"] and h["score"] > 0
+        ]
+        self.assertTrue(old and "oldterm" in old[0]["text"])
+        self.assertEqual(old[0]["source"], "handbook")
+        self.assertEqual(old[0]["authority"], "official")
+        phoenix = [
+            h
+            for h in handle_retrieve_context({"query": "phoenix"})["results"]
+            if h["score"] and h["score"] > 0
+        ]
+        self.assertEqual(phoenix, [])
+
+    def test_update_missing_document_reports_not_found(self):
+        result = handle_update_document(
+            {
+                "content": make_doc("neverseen"),
+                "source": "never_stored",
+                "version": "v1",
+            }
+        )
+        self.assertEqual(result["status"], "not_found")
+        self.assertEqual(result["source"], "never_stored")
+        self.assertEqual(result["version"], "v1")
+        self.assertEqual(get_vector_store().count(), 0)
+
+
+class TestRemoveDocument(InmemoryBackendTestBase):
+    def test_remove_deletes_every_chunk(self):
+        handle_store_document(
+            {
+                "content": make_doc("alpha", "beta", "gamma"),
+                "source": "docA",
+                "authority": "official",
+            }
+        )
+        result = handle_remove_document({"source": "docA"})
+        self.assertEqual(result["status"], "removed")
+        self.assertEqual(result["removed_chunks"], 3)
+        self.assertEqual(get_vector_store().count(), 0)
+        again = handle_remove_document({"source": "docA"})
+        self.assertEqual(again["status"], "not_found")
+
+    def test_remove_missing_document_reports_not_found(self):
+        result = handle_remove_document({"source": "never_stored"})
+        self.assertEqual(result["status"], "not_found")
+
+
+class TestInMemoryIdCollisionRegression(InmemoryBackendTestBase):
+    """
+    #1029 interaction guard.
+
+    In-memory vector ids are ``vec_{len(self.vectors) + i}``.  Deleting a
+    document that is NOT a suffix makes len() fall below surviving ids, so
+    the next plain write overwrites live data.  Our rebuild path must
+    prevent that: store a 1-chunk doc, then a 3-chunk doc, remove the
+    1-chunk one, then store another doc.  Without the rebuild the last
+    store lands on the surviving document's third chunk id and destroys
+    it.
+    """
+
+    def test_remove_then_store_keeps_surviving_chunks_intact(self):
+        handle_store_document(
+            {"content": make_doc("alpha"), "source": "docA", "authority": "official"}
+        )
+        handle_store_document(
+            {
+                "content": make_doc("bravo", "charlie", "delta"),
+                "source": "docB",
+                "authority": "official",
+            }
+        )
+        self.assertEqual(get_vector_store().count(), 4)
+
+        removed = handle_remove_document({"source": "docA"})
+        self.assertEqual(removed["status"], "removed")
+
+        stored = handle_store_document(
+            {"content": make_doc("echo"), "source": "docC", "authority": "official"}
+        )
+        self.assertEqual(stored["status"], "stored")
+
+        store = get_vector_store()
+        self.assertEqual(store.count(), 4)
+
+        delta_hits = handle_retrieve_context({"query": "delta"})["results"]
+        delta_hits = [h for h in delta_hits if h["score"] and h["score"] > 0]
+        self.assertTrue(delta_hits, "docB's third chunk was destroyed by an id collision")
+        self.assertIn("delta", delta_hits[0]["text"])
+        self.assertEqual(delta_hits[0]["source"], "docB")
+
+        for keyword, expected_source in (
+            ("bravo", "docB"),
+            ("charlie", "docB"),
+            ("echo", "docC"),
+        ):
+            hits = [
+                h
+                for h in handle_retrieve_context({"query": keyword})["results"]
+                if h["score"] and h["score"] > 0
+            ]
+            self.assertTrue(hits, f"expected a hit for {keyword}")
+            self.assertEqual(hits[0]["source"], expected_source)
+
+
+class TestBackendPolicy(InmemoryBackendTestBase):
+    def test_unsupported_backend_fails_fast(self):
+        # faiss/pgvector lack a metadata-scoped delete, so update/remove
+        # cannot work on them; selecting them must fail at startup, not
+        # mid-update.
+        for backend in ("faiss", "pgvector"):
+            with self.subTest(backend=backend):
+                os.environ["SEMANTICA_VECTOR_BACKEND"] = backend
+                with self.assertRaises(ValueError) as ctx:
+                    get_vector_store()
+                self.assertIn("not supported", str(ctx.exception))
+
+    def test_oversized_document_rejected_before_embedding(self):
+        # chunk_size=1 turns a 12k-char body into 12k chunks, crossing
+        # the ingestion cap without any expensive embedding work.
+        result = handle_store_document(
+            {
+                "content": "ab" * 6000,
+                "source": "bigdoc",
+                "authority": "official",
+                "chunk_size": 1,
+                "chunk_overlap": 0,
+            }
+        )
+        self.assertIn("error", result)
+        self.assertIn("chunks", result["error"])
+        self.assertEqual(get_vector_store().count(), 0)
+
+
+class TestToolRegistration(unittest.TestCase):
+    def test_retrieval_tools_are_registered(self):
+        retrieval = {
+            t["name"]: t
+            for t in TOOL_DEFINITIONS
+            if t["name"] in ("store_document", "retrieve_context", "update_document", "remove_document")
+        }
+        self.assertEqual(len(retrieval), 4)
+        for name, t in retrieval.items():
+            self.assertTrue(callable(t["_handler"]))
+            self.assertIn("required", t["inputSchema"])
+
+
+class TestSqliteBackend(unittest.TestCase):
+    def setUp(self):
+        try:
+            import sqlite_vec  # noqa: F401
+        except ImportError:
+            self.skipTest("sqlite_vec extension not installed")
+        self.tmpdir = tempfile.mkdtemp(prefix="semantica_sqlite_test_")
+        self.patches = patch_embedding_generators()
+        for p in self.patches:
+            p.start()
+        _clear_retrieval_env()
+        os.environ["SEMANTICA_VECTOR_BACKEND"] = "sqlite"
+        os.environ["SEMANTICA_VECTOR_DB_PATH"] = os.path.join(self.tmpdir, "vectors.db")
+        session._embedder = FakeEmbedder()
+        session._vector_store = None
+
+    def tearDown(self):
+        for p in self.patches:
+            p.stop()
+        session._embedder = None
+        reset_vector_store()
+        _clear_retrieval_env()
+        import shutil
+
+        shutil.rmtree(self.tmpdir, ignore_errors=True)
+
+    def test_sqlite_backend_roundtrip(self):
+        handle_store_document(
+            {
+                "content": make_doc("alpha", "beta"),
+                "source": "docS",
+                "authority": "official",
+            }
+        )
+        hits = handle_retrieve_context({"query": "beta"})["results"]
+        self.assertTrue(hits and "beta" in hits[0]["text"])
+        self.assertEqual(hits[0]["source"], "docS")
+
+        updated = handle_update_document(
+            {"content": make_doc("gamma"), "source": "docS"}
+        )
+        self.assertEqual(updated["status"], "updated")
+        # NB: score scales differ across backends (sqlite maps distance
+        # through 1/(1+d), so an orthogonal chunk still scores 0.5).
+        # Assert on text, the only backend-independent signal.
+        stale = [
+            h
+            for h in handle_retrieve_context({"query": "beta"})["results"]
+            if "beta" in (h.get("text") or "")
+        ]
+        self.assertEqual(stale, [])
+        self.assertTrue(handle_retrieve_context({"query": "gamma"})["results"])
+
+        removed = handle_remove_document({"source": "docS"})
+        self.assertEqual(removed["status"], "removed")
+        self.assertEqual(get_vector_store().count(), 0)
+
+    def test_sqlite_multi_document_isolation(self):
+        handle_store_document(
+            {
+                "content": make_doc("harbor", "vessel"),
+                "source": "nav_docs",
+                "authority": "official",
+            }
+        )
+        handle_store_document(
+            {
+                "content": make_doc("ledger", "invoice"),
+                "source": "fin_docs",
+                "authority": "draft",
+                "version": "v2",
+            }
+        )
+
+        nav = [
+            h
+            for h in handle_retrieve_context({"query": "vessel"})["results"]
+            if "vessel" in (h.get("text") or "")
+        ]
+        self.assertTrue(nav)
+        self.assertEqual(nav[0]["source"], "nav_docs")
+        self.assertEqual(nav[0]["authority"], "official")
+        self.assertEqual(nav[0]["status"], "active")
+        self.assertTrue(nav[0]["hash"])
+
+        # Updating one document must leave the other untouched.
+        updated = handle_update_document(
+            {"content": make_doc("anchor"), "source": "nav_docs"}
+        )
+        self.assertEqual(updated["status"], "updated")
+        fin = [
+            h
+            for h in handle_retrieve_context({"query": "invoice"})["results"]
+            if "invoice" in (h.get("text") or "")
+        ]
+        self.assertTrue(fin)
+        self.assertEqual(fin[0]["source"], "fin_docs")
+        self.assertEqual(fin[0]["authority"], "draft")
+        vessel_stale = [
+            h
+            for h in handle_retrieve_context({"query": "vessel"})["results"]
+            if "vessel" in (h.get("text") or "")
+        ]
+        self.assertEqual(vessel_stale, [])
+
+        # Removing the other document must leave the first intact.
+        removed = handle_remove_document({"source": "fin_docs", "version": "v2"})
+        self.assertEqual(removed["status"], "removed")
+        anchor = [
+            h
+            for h in handle_retrieve_context({"query": "anchor"})["results"]
+            if "anchor" in (h.get("text") or "")
+        ]
+        self.assertTrue(anchor and anchor[0]["source"] == "nav_docs")
+        ledger_stale = [
+            h
+            for h in handle_retrieve_context({"query": "ledger"})["results"]
+            if "ledger" in (h.get("text") or "")
+        ]
+        self.assertEqual(ledger_stale, [])
+
+    def test_sqlite_update_rolls_back_on_write_failure(self):
+        # Persistent path: removal is a direct delete_vectors, so the
+        # rollback has to re-store the snapshotted rows (plain lists,
+        # not arrays) when the new write fails.
+        handle_store_document(
+            {
+                "content": make_doc("oldterm", "legacy"),
+                "source": "handbook",
+                "authority": "official",
+            }
+        )
+        store = get_vector_store()
+        real_store_vectors = store.store_vectors
+
+        def failing_write(vectors, metas):
+            if any("phoenix" in (m.get("text") or "") for m in metas):
+                raise RuntimeError("simulated write failure")
+            return real_store_vectors(vectors, metas)
+
+        with patch.object(store, "store_vectors", side_effect=failing_write):
+            result = handle_update_document(
+                {"content": make_doc("phoenix"), "source": "handbook"}
+            )
+        self.assertIn("error", result)
+        self.assertEqual(get_vector_store().count(), 2)
+        old = [
+            h
+            for h in handle_retrieve_context({"query": "oldterm"})["results"]
+            if "oldterm" in (h.get("text") or "")
+        ]
+        self.assertTrue(old and old[0]["source"] == "handbook")
+
+    def test_sqlite_without_db_path_raises(self):
+        os.environ.pop("SEMANTICA_VECTOR_DB_PATH", None)
+        with self.assertRaises(ValueError):
+            get_vector_store()
+
+
+class TestPersistence(InmemoryBackendTestBase):
+    def test_store_persists_and_reloads(self):
+        tmpdir = tempfile.mkdtemp(prefix="semantica_vec_test_")
+        try:
+            os.environ["SEMANTICA_VECTOR_PATH"] = tmpdir
+            result = handle_store_document(
+                {"content": make_doc("persist"), "source": "docP", "authority": "official"}
+            )
+            self.assertTrue(result["persisted"])
+            self.assertTrue(os.path.isfile(os.path.join(tmpdir, "store_data.json")))
+
+            # Fresh session state: the store must reload from disk.
+            reset_vector_store()
+            hits = handle_retrieve_context({"query": "persist"})["results"]
+            self.assertTrue(hits and "persist" in hits[0]["text"])
+            self.assertEqual(hits[0]["source"], "docP")
+        finally:
+            import shutil
+
+            shutil.rmtree(tmpdir, ignore_errors=True)
+
+    def test_persist_failure_is_reported_not_silent(self):
+        tmpdir = tempfile.mkdtemp(prefix="semantica_vec_test_")
+        try:
+            os.environ["SEMANTICA_VECTOR_PATH"] = tmpdir
+            store = get_vector_store()
+            with patch.object(store, "save", side_effect=RuntimeError("disk full")):
+                result = handle_store_document(
+                    {"content": make_doc("volatile"), "source": "docV", "authority": "official"}
+                )
+            # The write itself succeeded; only the durable copy failed.
+            self.assertEqual(result["status"], "stored")
+            self.assertFalse(result["persisted"])
+        finally:
+            import shutil
+
+            shutil.rmtree(tmpdir, ignore_errors=True)
+
+    def test_reload_dimension_mismatch_rejected(self):
+        tmpdir = tempfile.mkdtemp(prefix="semantica_vec_test_")
+        try:
+            os.environ["SEMANTICA_VECTOR_PATH"] = tmpdir
+            handle_store_document(
+                {"content": make_doc("persist"), "source": "docP", "authority": "official"}
+            )
+            # A different embedder dimension must not silently rank
+            # vectors from an incompatible embedding space.
+            session._embedder = FakeEmbedder(32)
+            reset_vector_store()
+            with self.assertRaises(ValueError) as ctx:
+                get_vector_store()
+            self.assertIn("dimension", str(ctx.exception))
+        finally:
+            import shutil
+
+            shutil.rmtree(tmpdir, ignore_errors=True)
+
+    def test_corrupt_store_fails_on_startup(self):
+        """A broken persisted store must raise immediately, not silently
+        fall back to an empty store that would overwrite the user's data
+        on the first persist."""
+        tmpdir = tempfile.mkdtemp(prefix="semantica_vec_test_")
+        try:
+            # Drop a file that looks like a store directory but won't load.
+            with open(os.path.join(tmpdir, "store_data.json"), "w") as f:
+                f.write("{not valid json")
+            os.environ["SEMANTICA_VECTOR_PATH"] = tmpdir
+            reset_vector_store()
+            with self.assertRaises(ValueError) as ctx:
+                get_vector_store()
+            self.assertIn("Could not load", str(ctx.exception))
+        finally:
+            import shutil
+
+            shutil.rmtree(tmpdir, ignore_errors=True)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/vector_store/test_faiss_delete_vectors.py

@@ -0,0 +1,662 @@
+"""Tests for FAISSIndex.delete_vectors and FAISSStore.delete_vectors (#1374)."""
+
+import numpy as np
+import pytest
+
+from semantica.context.erasure import (
+    STATUS_ERASED,
+    STATUS_UNSUPPORTED,
+    ErasureCoordinator,
+)
+from semantica.utils.exceptions import ProcessingError
+from semantica.vector_store import VectorStore
+from semantica.vector_store.faiss_store import FAISSIndex, FAISSStore
+
+
+# ---------------------------------------------------------------------------
+# Fixtures and helpers
+# ---------------------------------------------------------------------------
+
+
+def _flat_index(dim: int = 3) -> "faiss.IndexFlatL2":  # noqa: F821
+    faiss = pytest.importorskip("faiss")
+    return faiss.IndexFlatL2(dim)
+
+
+def _populated_store(
+    dim: int = 3,
+    ids=("a", "b", "c", "d", "e"),
+    meta=None,
+):
+    """Return an FAISSStore with *ids* already inserted (random unit vectors)."""
+    faiss = pytest.importorskip("faiss")
+    store = FAISSStore(dimension=dim)
+    n = len(ids)
+    rng = np.random.default_rng(seed=42)
+    vectors = rng.random((n, dim)).astype(np.float32)
+    metadata = meta or [{} for _ in ids]
+    store.add_vectors(vectors, ids=list(ids), metadata=metadata)
+    return store
+
+
+def _populated_index(dim: int = 3, ids=("a", "b", "c", "d", "e")):
+    """Return a bare FAISSIndex with *ids* inserted (random unit vectors)."""
+    faiss = pytest.importorskip("faiss")
+    idx = FAISSIndex(faiss.IndexFlatL2(dim), dimension=dim)
+    n = len(ids)
+    rng = np.random.default_rng(seed=42)
+    vectors = rng.random((n, dim)).astype(np.float32)
+    idx.add_vectors(vectors, ids=list(ids))
+    return idx
+
+
+# ---------------------------------------------------------------------------
+# FAISSIndex-level unit tests
+# ---------------------------------------------------------------------------
+
+
+class TestFAISSIndexDeleteVectors:
+    def test_delete_single_existing_id(self):
+        idx = _populated_index()
+        result = idx.delete_vectors(["b"])
+        assert result == {"delete_count": 1}
+        assert "b" not in idx.vector_ids
+        assert idx.index.ntotal == len(idx.vector_ids) == 4
+
+    def test_delete_multiple_existing_ids(self):
+        idx = _populated_index()
+        result = idx.delete_vectors(["b", "d"])
+        assert result == {"delete_count": 2}
+        assert "b" not in idx.vector_ids
+        assert "d" not in idx.vector_ids
+        assert sorted(idx.vector_ids) == ["a", "c", "e"]
+        assert idx.index.ntotal == 3
+
+    def test_delete_nonexistent_id_is_noop(self):
+        idx = _populated_index()
+        result = idx.delete_vectors(["z"])
+        assert result == {"delete_count": 0}
+        assert len(idx.vector_ids) == 5
+        assert idx.index.ntotal == 5
+
+    def test_delete_empty_list_is_noop(self):
+        idx = _populated_index()
+        result = idx.delete_vectors([])
+        assert result == {"delete_count": 0}
+        assert len(idx.vector_ids) == 5
+
+    def test_delete_duplicate_ids_in_request_only_removes_once(self):
+        idx = _populated_index()
+        result = idx.delete_vectors(["b", "b", "b"])
+        assert result == {"delete_count": 1}
+        assert "b" not in idx.vector_ids
+        assert len(idx.vector_ids) == 4
+
+    def test_delete_count_reflects_actual_removal(self):
+        idx = _populated_index()
+        # "z" doesn't exist; only "a" and "c" do
+        result = idx.delete_vectors(["a", "c", "z"])
+        assert result == {"delete_count": 2}
+
+    def test_metadata_removed_for_deleted_id(self):
+        faiss = pytest.importorskip("faiss")
+        idx = FAISSIndex(faiss.IndexFlatL2(3), dimension=3)
+        vectors = np.eye(3, dtype=np.float32)[:2]
+        idx.add_vectors(vectors, ids=["x", "y"])
+        idx.metadata = {"x": {"val": 1}, "y": {"val": 2}}
+        idx.delete_vectors(["x"])
+        assert "x" not in idx.metadata
+        assert "y" in idx.metadata
+
+    def test_vector_ids_list_stays_parallel_to_faiss_ntotal(self):
+        idx = _populated_index(ids=["a", "b", "c"])
+        idx.delete_vectors(["b"])
+        assert len(idx.vector_ids) == idx.index.ntotal == 2
+
+    def test_search_does_not_return_deleted_id(self):
+        """After deletion, similarity search must not return the deleted ID."""
+        faiss = pytest.importorskip("faiss")
+        idx = FAISSIndex(faiss.IndexFlatL2(3), dimension=3)
+        vectors = np.array(
+            [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]], dtype=np.float32
+        )
+        idx.add_vectors(vectors, ids=["a", "b", "c"])
+        idx.delete_vectors(["b"])
+
+        query = np.array([[0.0, 1.0, 0.0]], dtype=np.float32)
+        distances, indices = idx.search(query, k=3)
+        # Filter both negative sentinels (-1) and out-of-range indices.
+        returned_ids = [
+            idx.vector_ids[i]
+            for i in indices[0]
+            if 0 <= i < len(idx.vector_ids)
+        ]
+        assert "b" not in returned_ids
+
+    def test_get_vector_returns_none_after_deletion(self):
+        faiss = pytest.importorskip("faiss")
+        idx = FAISSIndex(faiss.IndexFlatL2(3), dimension=3)
+        vectors = np.eye(3, dtype=np.float32)
+        idx.add_vectors(vectors, ids=["a", "b", "c"])
+        idx.delete_vectors(["b"])
+        assert idx.get_vector("b") is None
+
+    def test_get_metadata_returns_none_after_deletion(self):
+        faiss = pytest.importorskip("faiss")
+        idx = FAISSIndex(faiss.IndexFlatL2(3), dimension=3)
+        idx.add_vectors(np.eye(3, dtype=np.float32)[:2], ids=["a", "b"])
+        idx.metadata = {"a": {"k": 1}, "b": {"k": 2}}
+        idx.delete_vectors(["b"])
+        assert idx.get_metadata("b") is None
+
+    def test_add_vectors_after_deletion_works(self):
+        """Inserting new vectors after deletion maintains correct position mapping."""
+        idx = _populated_index(ids=["a", "b", "c"])
+        idx.delete_vectors(["b"])
+        new_vecs = np.array([[0.5, 0.5, 0.0]], dtype=np.float32)
+        idx.add_vectors(new_vecs, ids=["new"])
+        assert "new" in idx.vector_ids
+        assert len(idx.vector_ids) == idx.index.ntotal == 3
+
+    def test_save_load_after_deletion_preserves_state(self, tmp_path):
+        """Deletion persists correctly through save/load round-trip."""
+        _ = pytest.importorskip("faiss")
+        idx = _populated_index(ids=["a", "b", "c"])
+        idx.delete_vectors(["b"])
+
+        path = tmp_path / "idx.faiss"
+        idx.save(path)
+        loaded = FAISSIndex.load(path, dimension=3)
+
+        assert "b" not in loaded.vector_ids
+        assert sorted(loaded.vector_ids) == ["a", "c"]
+        assert loaded.index.ntotal == 2
+
+    def test_hnsw_delete_raises_not_implemented(self):
+        """HNSW does not support remove_ids; must raise NotImplementedError."""
+        faiss = pytest.importorskip("faiss")
+        hnsw = FAISSIndex(faiss.IndexHNSWFlat(4, 16), dimension=4)
+        vecs = np.random.rand(5, 4).astype(np.float32)
+        hnsw.add_vectors(vecs, ids=["a", "b", "c", "d", "e"])
+        with pytest.raises(NotImplementedError):
+            hnsw.delete_vectors(["a"])
+        # Python-side state must be untouched
+        assert len(hnsw.vector_ids) == 5
+
+    def test_ivf_delete_raises_not_implemented(self):
+        """IVF does not compact labels after remove_ids; raise NotImplementedError.
+
+        IVF surviving labels stay sparse (0,2,4 not 0,1,2), so the list-compact
+        approach used by Flat would desynchronize search labels from vector_ids.
+        """
+        faiss = pytest.importorskip("faiss")
+        dim = 4
+        train = np.random.rand(80, dim).astype(np.float32)
+        q = faiss.IndexFlatL2(dim)
+        ivf = faiss.IndexIVFFlat(q, dim, 2)
+        ivf.train(train)
+        idx = FAISSIndex(ivf, dimension=dim)
+        vecs = np.random.rand(5, dim).astype(np.float32)
+        idx.add_vectors(vecs, ids=["a", "b", "c", "d", "e"])
+        with pytest.raises(NotImplementedError):
+            idx.delete_vectors(["b"])
+        # Python-side state must be completely untouched
+        assert idx.vector_ids == ["a", "b", "c", "d", "e"]
+
+
+# ---------------------------------------------------------------------------
+# FAISSStore-level unit tests
+# ---------------------------------------------------------------------------
+
+
+class TestFAISSStoreDeleteVectors:
+    def test_delete_uninitialized_index_raises_processing_error(self):
+        store = FAISSStore(dimension=3)
+        with pytest.raises(ProcessingError, match="Index not initialized"):
+            store.delete_vectors(["a"])
+
+    def test_delete_existing_id_returns_dict(self):
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b", "c"])
+        result = store.delete_vectors(["b"])
+        assert result == {"delete_count": 1}
+
+    def test_delete_reduces_count(self):
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b", "c"])
+        assert store.count() == 3
+        store.delete_vectors(["b"])
+        assert store.count() == 2
+
+    def test_delete_multiple_ids(self):
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b", "c", "d"])
+        result = store.delete_vectors(["a", "c"])
+        assert result == {"delete_count": 2}
+        assert store.count() == 2
+
+    def test_delete_empty_input_is_noop(self):
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b"])
+        result = store.delete_vectors([])
+        assert result == {"delete_count": 0}
+        assert store.count() == 2
+
+    def test_delete_nonexistent_id_is_zero(self):
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b"])
+        result = store.delete_vectors(["z"])
+        assert result == {"delete_count": 0}
+        assert store.count() == 2
+
+    def test_duplicate_ids_in_request(self):
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b"])
+        result = store.delete_vectors(["a", "a"])
+        assert result == {"delete_count": 1}
+        assert store.count() == 1
+
+    def test_metadata_cleaned_up(self):
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(
+            ids=["a", "b"],
+            meta=[{"owner": "alice"}, {"owner": "bob"}],
+        )
+        store.delete_vectors(["a"])
+        assert store.get_metadata("a") is None
+        assert store.get_metadata("b") == {"owner": "bob"}
+
+    def test_get_vector_returns_none_after_deletion(self):
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b"])
+        store.delete_vectors(["a"])
+        assert store.get_vector("a") is None
+
+    def test_search_excludes_deleted_vector(self):
+        """search_similar must not return a deleted vector's ID."""
+        faiss = pytest.importorskip("faiss")
+        store = FAISSStore(dimension=3)
+        vectors = np.array(
+            [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]], dtype=np.float32
+        )
+        store.add_vectors(vectors, ids=["a", "b", "c"])
+        store.delete_vectors(["b"])
+        query = np.array([0.0, 1.0, 0.0], dtype=np.float32)
+        results = store.search_similar(query, k=3)
+        returned_ids = [r["id"] for r in results]
+        assert "b" not in returned_ids
+
+    def test_add_vectors_after_deletion(self):
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b", "c"])
+        store.delete_vectors(["b"])
+        vecs = np.array([[0.5, 0.5, 0.0]], dtype=np.float32)
+        store.add_vectors(vecs, ids=["new"])
+        assert store.count() == 3
+        assert store.get_vector("new") is not None
+
+    def test_save_load_after_deletion(self, tmp_path):
+        """Deleted vectors do not reappear after save/load."""
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b", "c"])
+        store.delete_vectors(["b"])
+
+        path = tmp_path / "store.faiss"
+        store.save_index(path)
+
+        fresh = FAISSStore(dimension=3)
+        fresh.load_index(path)
+
+        assert fresh.count() == 2
+        assert "b" not in fresh.index.vector_ids
+        assert fresh.get_vector("b") is None
+
+    def test_options_kwarg_is_accepted_and_ignored(self):
+        """delete_vectors(**options) must not crash even with extra kwargs."""
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a"])
+        result = store.delete_vectors(["a"], unused_option=True)
+        assert result["delete_count"] == 1
+
+    def test_hnsw_raises_not_implemented(self):
+        """FAISSStore.delete_vectors on HNSW must propagate NotImplementedError."""
+        faiss = pytest.importorskip("faiss")
+        store = FAISSStore(dimension=4)
+        store.create_index(index_type="hnsw", metric="L2")
+        vecs = np.random.rand(5, 4).astype(np.float32)
+        store.add_vectors(vecs, ids=["a", "b", "c", "d", "e"])
+        with pytest.raises(NotImplementedError):
+            store.delete_vectors(["a"])
+        # Count must be unchanged
+        assert store.count() == 5
+
+    def test_ivf_raises_not_implemented(self):
+        """FAISSStore.delete_vectors on IVF must raise NotImplementedError.
+
+        IVF remove_ids preserves original labels rather than compacting them,
+        which would desynchronize search labels from vector_ids.
+        """
+        faiss = pytest.importorskip("faiss")
+        store = FAISSStore(dimension=4)
+        # nlist=2 so we only need >= 2*39 = 78 training points
+        store.create_index(index_type="ivf", metric="L2", nlist=2)
+        train = np.random.rand(80, 4).astype(np.float32)
+        store.index.index.train(train)
+        store.add_vectors(train[:5], ids=["a", "b", "c", "d", "e"])
+        with pytest.raises(NotImplementedError):
+            store.delete_vectors(["a"])
+        # State must be completely unchanged
+        assert store.count() == 5
+
+    def test_delete_with_loaded_index_auto_saves(self, tmp_path):
+        """Deletion on a store loaded from disk auto-saves without explicit save_index."""
+        _ = pytest.importorskip("faiss")
+        # Create, populate, save
+        store = _populated_store(ids=["a", "b", "c"])
+        path = tmp_path / "store.faiss"
+        store.save_index(path)
+
+        # Load into a fresh store and delete
+        loaded = FAISSStore(dimension=3)
+        loaded.load_index(path)
+        loaded.delete_vectors(["b"])
+
+        # Reload without any additional save call — deletion must have persisted
+        reloaded = FAISSStore(dimension=3)
+        reloaded.load_index(path)
+        assert reloaded.count() == 2
+        assert "b" not in reloaded.index.vector_ids
+
+    def test_default_id_no_collision_after_deletion(self):
+        """Default vec_N IDs must not reuse a surviving ID after deletion."""
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["vec_0", "vec_1", "vec_2"])
+        # Delete the middle one; len(vector_ids) drops to 2
+        store.delete_vectors(["vec_1"])
+        assert store.count() == 2
+
+        # Add a new vector — without the monotonic counter, the default ID
+        # would be vec_2 which already exists and would be silently skipped.
+        new_vecs = np.random.rand(1, 3).astype(np.float32)
+        returned_ids = store.add_vectors(new_vecs)
+        # The returned ID must not be an existing one
+        assert returned_ids[0] not in {"vec_0", "vec_2"}, (
+            f"Default ID {returned_ids[0]} collides with a surviving ID"
+        )
+        # And the vector must actually have been inserted
+        assert store.count() == 3
+
+    def test_default_id_skip_past_explicit_id(self):
+        """Blocker: default IDs must skip over explicit IDs already in the store.
+
+        If a user inserts an explicit ``"vec_N"`` and then adds two vectors
+        without IDs, the generator must skip ``"vec_N"`` rather than
+        producing it and losing the second vector silently.
+        """
+        _ = pytest.importorskip("faiss")
+        store = FAISSStore(dimension=3)
+
+        # Explicit vec_1 first
+        store.add_vectors(np.ones((1, 3), dtype=np.float32), ids=["vec_1"])
+
+        # 2 default vectors — one would collide with vec_1 if not skipped
+        store.add_vectors(np.ones((2, 3), dtype=np.float32))
+
+        # 1 more default vector — must get a fresh ID, not re-generate a used one
+        original_meta = {vid: {"original": vid} for vid in store.index.vector_ids}
+        for vid, m in original_meta.items():
+            store.index.metadata[vid] = m
+        count_before = store.count()
+
+        ret = store.add_vectors(
+            np.ones((1, 3), dtype=np.float32), metadata=[{"new": True}]
+        )
+        new_id = ret[0]
+
+        assert store.count() == count_before + 1, (
+            f"Vector was silently skipped; count stayed {store.count()}"
+        )
+        assert new_id not in original_meta, (
+            f"Generated ID {new_id!r} collides with an already-existing ID"
+        )
+        # Surviving IDs' metadata must not be overwritten
+        for vid, m in original_meta.items():
+            assert store.index.metadata.get(vid) == m, (
+                f"Metadata for surviving {vid!r} was overwritten"
+            )
+
+    def test_stale_persisted_next_id_is_clamped_to_inferred_minimum(self, tmp_path):
+        """Regression: a stale ``next_id`` in the sidecar must be clamped to
+        at least ``max(vec_N)+1`` so that auto-save after deletion cannot
+        propagate the stale value and cause future ID collisions.
+        """
+        import json as _json
+        _ = pytest.importorskip("faiss")
+        rng = np.random.default_rng(seed=3)
+        store = FAISSStore(dimension=3)
+        store.add_vectors(rng.random((5, 3)).astype(np.float32))
+        # IDs are vec_0..vec_4, next_id=5
+        path = tmp_path / "s.faiss"
+        store.save_index(path)
+
+        # Corrupt the sidecar: set next_id to a stale low value
+        meta = _json.loads((tmp_path / "s.faiss.meta.json").read_text())
+        meta["next_id"] = 2  # stale — vec_2, vec_3, vec_4 still exist
+        (tmp_path / "s.faiss.meta.json").write_text(_json.dumps(meta))
+
+        # Load and immediately delete one vector (auto-save fires)
+        s2 = FAISSStore(dimension=3)
+        s2.load_index(path)
+        assert s2._next_id == 5, f"Stale next_id should be clamped to 5, got {s2._next_id}"
+        s2.delete_vectors(["vec_3"])  # triggers auto-save
+
+        # The sidecar must not carry the stale value forward
+        persisted = _json.loads((tmp_path / "s.faiss.meta.json").read_text())
+        assert persisted["next_id"] >= 5, (
+            f"Auto-save propagated stale next_id={persisted['next_id']} (expected >= 5)"
+        )
+
+    def test_search_does_not_return_phantom_id_when_k_exceeds_ntotal(self):
+        """Regression: when k > ntotal, FAISS returns -1 sentinel values.
+        ``-1 < len(vector_ids)`` is always True in Python, so without an
+        explicit non-negative guard ``-1`` maps to ``vector_ids[-1]``,
+        making the last vector appear as a spurious extra result.
+        """
+        faiss = pytest.importorskip("faiss")
+        store = FAISSStore(dimension=3)
+        vecs = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]], dtype=np.float32)
+        store.add_vectors(vecs, ids=["only_a", "only_b"])
+
+        # Ask for 10 neighbors but only 2 exist
+        results = store.search_similar(
+            np.array([0.0, 1.0, 0.0], dtype=np.float32), k=10
+        )
+        returned_ids = [r["id"] for r in results]
+        assert len(results) == 2, (
+            f"Expected exactly 2 results, got {len(results)}: {returned_ids}"
+        )
+        assert returned_ids.count("only_b") == 1, (
+            f"only_b appears {returned_ids.count('only_b')} time(s) — "
+            "sentinel -1 is mapping to vector_ids[-1]"
+        )
+
+    def test_next_id_persisted_across_delete_save_reload(self, tmp_path):
+        """Regression test for critical bug: delete → auto-save → reload → add.
+
+        Without persisting ``next_id`` in the sidecar, ``load_index`` would
+        set ``_next_id = ntotal`` (4 after one deletion from 5 vectors), which
+        would generate ``"vec_4"`` as the next default ID.  That ID is still
+        present in the surviving vector list, so the insertion would be
+        silently skipped, the count would not increase, and the old vector's
+        metadata would be overwritten by the new metadata.
+
+        This test pins the full lifecycle so any regression is caught
+        immediately.
+        """
+        _ = pytest.importorskip("faiss")
+        rng = np.random.default_rng(seed=7)
+        dim = 4
+
+        # Step 1: create vec_0 .. vec_4, record their embeddings
+        store1 = FAISSStore(dimension=dim)
+        vecs = rng.random((5, dim)).astype(np.float32)
+        store1.add_vectors(vecs)
+        for vid in store1.index.vector_ids:
+            store1.index.metadata[vid] = {"original": vid}
+        path = tmp_path / "idx.faiss"
+        store1.save_index(path)
+
+        # Step 2: reload → delete vec_2 (auto-saves) → reload again
+        store2 = FAISSStore(dimension=dim)
+        store2.load_index(path)
+        store2.delete_vectors(["vec_2"])   # ntotal drops to 4; auto-save triggered
+
+        store3 = FAISSStore(dimension=dim)
+        store3.load_index(path)
+
+        # Step 3: add a new vector without an explicit ID
+        new_vec = rng.random((1, dim)).astype(np.float32)
+        count_before = store3.count()
+        returned_ids = store3.add_vectors(new_vec, metadata=[{"new": True}])
+
+        # The generated ID must not collide with any surviving ID
+        surviving = set(store3.index.vector_ids[:count_before])
+        new_id = returned_ids[0]
+        assert new_id not in surviving, (
+            f"Generated ID {new_id!r} collides with surviving ID "
+            f"(surviving={sorted(surviving)})"
+        )
+
+        # The new vector must actually have been inserted
+        assert store3.count() == count_before + 1, (
+            f"Count did not increase: was {count_before}, still {store3.count()}"
+        )
+
+        # The new vector must be retrievable
+        assert store3.get_vector(new_id) is not None, (
+            f"New vector with ID {new_id!r} is not retrievable"
+        )
+
+        # The surviving vec_4's embedding must be unchanged
+        original_vec4 = vecs[4]
+        loaded_vec4 = store3.get_vector("vec_4")
+        assert loaded_vec4 is not None
+        np.testing.assert_allclose(loaded_vec4, original_vec4, atol=1e-5,
+            err_msg="vec_4 embedding was corrupted by the new add")
+
+        # The surviving vec_4's metadata must be unchanged
+        assert store3.index.metadata.get("vec_4") == {"original": "vec_4"}, (
+            f"vec_4 metadata was overwritten: {store3.index.metadata.get('vec_4')}"
+        )
+
+        # The new vector's metadata must be the new value
+        assert store3.index.metadata.get(new_id) == {"new": True}
+
+    def test_no_op_delete_does_not_rewrite_disk(self, tmp_path):
+        """A deletion of only nonexistent IDs must not call FAISSIndex.save().
+
+        Uses a spy on ``FAISSIndex.save`` rather than filesystem mtime so the
+        assertion is deterministic regardless of filesystem timestamp resolution.
+        """
+        from unittest.mock import patch
+        _ = pytest.importorskip("faiss")
+        store = _populated_store(ids=["a", "b", "c"])
+        path = tmp_path / "idx.faiss"
+        store.save_index(path)
+
+        loaded = FAISSStore(dimension=3)
+        loaded.load_index(path)
+
+        with patch.object(loaded.index, "save", wraps=loaded.index.save) as mock_save:
+            loaded.delete_vectors(["z"])  # nonexistent → delete_count 0
+            loaded.delete_vectors([])     # empty list  → delete_count 0
+            assert mock_save.call_count == 0, (
+                f"save() called {mock_save.call_count} time(s) for a no-op deletion"
+            )
+
+            # A real deletion must still trigger save()
+            loaded.delete_vectors(["b"])
+            assert mock_save.call_count == 1, (
+                f"save() was not called after a real deletion (calls={mock_save.call_count})"
+            )
+
+
+# ---------------------------------------------------------------------------
+# Facade delegation test
+# ---------------------------------------------------------------------------
+
+
+class TestFAISSFacadeDelegation:
+    def test_vector_store_facade_delegates_to_faiss_store(self):
+        """VectorStore(backend='faiss').delete_vectors() must call FAISSStore."""
+        _ = pytest.importorskip("faiss")
+        vs = VectorStore(backend="faiss", config={"dimension": 3})
+        vecs = np.eye(3, dtype=np.float32)
+        vs.store_vectors(list(vecs), metadata=[{}, {}, {}])
+        # Count before
+        assert vs._backend_store.count() == 3
+
+        result = vs.delete_vectors(["vec_0"])
+
+        assert result == {"delete_count": 1}
+        assert vs._backend_store.count() == 2
+
+
+# ---------------------------------------------------------------------------
+# ErasureCoordinator integration tests
+# ---------------------------------------------------------------------------
+
+
+class TestFAISSErasureCoordinator:
+    def _faiss_vector_store(self, dim: int = 3) -> VectorStore:
+        _ = pytest.importorskip("faiss")
+        vs = VectorStore(backend="faiss", config={"dimension": dim})
+        vecs = np.eye(dim, dtype=np.float32)
+        vs.store_vectors(list(vecs), metadata=[{}, {}, {}])
+        return vs
+
+    def test_erasure_reports_status_erased(self):
+        vs = self._faiss_vector_store()
+        # store_vectors assigns ids "vec_0", "vec_1", "vec_2"
+        vector_ids = vs._backend_store.index.vector_ids
+        coord = ErasureCoordinator(vector_store=vs)
+        receipt = coord.erase_entity(vector_ids[0], vector_ids=[vector_ids[0]])
+        assert receipt.stores["vectors"]["status"] == STATUS_ERASED
+
+    def test_erasure_backend_name_is_faiss(self):
+        vs = self._faiss_vector_store()
+        coord = ErasureCoordinator(vector_store=vs)
+        receipt = coord.erase_entity("vec_0", vector_ids=["vec_0"])
+        assert receipt.stores["vectors"]["backend"] == "faiss"
+
+    def test_erasure_receipt_is_complete_after_deletion(self):
+        vs = self._faiss_vector_store()
+        coord = ErasureCoordinator(vector_store=vs)
+        receipt = coord.erase_entity("vec_0", vector_ids=["vec_0"])
+        assert receipt.complete
+
+    def test_erasure_hnsw_reports_unsupported(self):
+        """HNSW deletion raises NotImplementedError; coordinator must report unsupported."""
+        faiss = pytest.importorskip("faiss")
+        vs = VectorStore(backend="faiss", config={"dimension": 4})
+        vs._backend_store.create_index(index_type="hnsw", metric="L2")
+        vecs = np.random.rand(5, 4).astype(np.float32)
+        vs._backend_store.add_vectors(vecs, ids=["a", "b", "c", "d", "e"])
+        coord = ErasureCoordinator(vector_store=vs)
+        receipt = coord.erase_entity("a", vector_ids=["a"])
+        assert receipt.stores["vectors"]["status"] == STATUS_UNSUPPORTED
+        assert not receipt.complete
+
+    def test_erasure_ivf_reports_unsupported(self):
+        """IVF deletion raises NotImplementedError; coordinator must report unsupported."""
+        faiss = pytest.importorskip("faiss")
+        dim = 4
+        vs = VectorStore(backend="faiss", config={"dimension": dim})
+        vs._backend_store.create_index(index_type="ivf", metric="L2", nlist=2)
+        train = np.random.rand(80, dim).astype(np.float32)
+        vs._backend_store.index.index.train(train)
+        vs._backend_store.add_vectors(train[:5], ids=["a", "b", "c", "d", "e"])
+        coord = ErasureCoordinator(vector_store=vs)
+        receipt = coord.erase_entity("a", vector_ids=["a"])
+        assert receipt.stores["vectors"]["status"] == STATUS_UNSUPPORTED
+        assert not receipt.complete

tests/visualization/test_optional_dependencies.py

@@ -38,14 +38,13 @@ class TestOptionalDependencies(unittest.TestCase):
         with import_without(
             "semantica.visualization.embedding_visualizer", "umap"
         ) as module:
-            with plotly_doubles(module), patch.object(module, "PCA") as mock_pca_class:
-                mock_pca_class.return_value.fit_transform.return_value = np.zeros((4, 2))
-
+            with plotly_doubles(module):
                 viz = module.EmbeddingVisualizer()
                 embeddings = np.array([[0, 1, 2], [1, 0, 3], [0, 0, 0], [1, 1, 1]])
-                viz.visualize_2d_projection(embeddings, method="umap")
-
-            mock_pca_class.assert_called()
+                with self.assertRaises(module.ProcessingError) as cm:
+                    viz.visualize_2d_projection(embeddings, method="umap")
+                self.assertIn("UMAP is required", str(cm.exception))
+                self.assertIn("semantica[viz]", str(cm.exception))
 
     def test_ontology_visualizer_without_graphviz(self):
         """Test OntologyVisualizer behavior when graphviz is missing."""