Compare commits

..
Author SHA1 Message Date
BingEdward c449209168 fix(cli): dispatch mcp call through semantica_mcp.mcp.server (#1368)
Route `semantica mcp call` and `semantica mcp list-tools` through the
canonical `semantica_mcp.mcp.server` packaged module instead of the
nonexistent `MCPSession` import.

Previously, `semantica mcp call` imported `MCPSession` from
`semantica_mcp.mcp.session`, which never defined it, causing every
call to fail with "MCP module not available". Meanwhile,
`semantica mcp list-tools` inspected `semantica_mcp.mcp.tools.__all__`,
which evaluated to `["TOOL_DEFINITIONS"]` and printed a single tool
named "TOOL_DEFINITIONS".

Key changes:
- Implement `semantica_mcp.mcp.server.call_tool(name, arguments)` as the
  shared in-process entry point used by both the CLI and the JSON-RPC
  `tools/call` handler, ensuring both surfaces expose identical tools.
- Add `UnknownToolError` to distinguish missing tools (JSON-RPC -32601)
  from handler-level `KeyError` exceptions (JSON-RPC -32603).
- Update `semantica mcp list-tools` to source tool names directly from
  `TOOL_DEFINITIONS` in `semantica_mcp.mcp.tools`.
- Validate `--args` payloads and reject non-object JSON inputs (arrays,
  primitives, null) with a clean `ClickException`.
- Update `tests/test_mcp_stdio_roundtrip.py` to spawn `semantica_mcp.mcp`
  instead of the pre-relocation `mcp` module, restoring clean stdio
  framing tests.

Fixes #1355
2026-09-08 02:40:53 +05:00
Sameer KadamandSameer Kadam 7be1582786 fix: correct vector store installation extras (#1529)
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
2026-09-08 01:42:20 +05:00
47 changed files with 9038 additions and 2732 deletions
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+18 -26
View File
@@ -97,10 +97,24 @@ jobs:
- name: Build Explorer frontend
working-directory: explorer
run: npm run build
- name: Install core package and base dependencies
- name: Install Explorer backend test dependencies
run: |
# Verify that core semantica installs cleanly with only its base dependencies
# (no optional extras) and that core imports and lazy missing-dependency hints work.
# Run the deterministic backend path before the all-extras CI
# environment is installed. The Explorer extra supplies the
# production API dependencies without importing optional vector
# providers such as Pinecone during test collection.
#
# --no-deps + a separate hash-pinned install (rather than the old
# `pip install -e ".[explorer]" pytest==9.1.1`) so every fetched
# package is hash-verified (Scorecard Pinned-Dependencies); the
# local editable install itself has nothing to hash.
# .github/requirements/explorer-extra-py311.txt is
# `uv pip compile pyproject.toml --extra explorer --python-version 3.11 --constraint requirements-ci.txt --generate-hashes`
# - regenerate it the same way if pyproject.toml's base/explorer
# deps change. Resolved specifically for this job's python 3.11
# (see the Dockerfile's explorer-extra-py313.txt for why this
# can't be shared with python 3.13: audioread needs extra
# standard-aifc/standard-sunau hashes only on 3.13+).
#
# --no-deps only skips *runtime* dependency resolution - `-e .`
# still does a PEP 517 build, which by default creates an isolated
@@ -111,30 +125,8 @@ jobs:
# copies instead of fetching its own.
pip install -r .github/requirements/pep517-build.txt --require-hashes
pip install --no-deps --no-build-isolation -e .
pip install -r .github/requirements/base-deps.txt --require-hashes
pip install -r .github/requirements/pytest-tool.txt --require-hashes
- name: Verify core-only package importability and slim behavior
run: |
python -c "
import semantica
print('semantica', semantica.__version__, 'core installed and importable')
"
pytest -q tests/test_issue_1513_slim_core.py
- name: Install Explorer backend test dependencies
run: |
# Run the deterministic backend path before the all-extras CI
# environment is installed. The Explorer extra supplies the
# production API dependencies without importing optional vector
# providers such as Pinecone during test collection.
#
# .github/requirements/explorer-extra-py311.txt is
# `uv pip compile pyproject.toml --extra explorer --python-version 3.11 --constraint requirements-ci.txt --generate-hashes`
# - regenerate it the same way if pyproject.toml's base/explorer
# deps change. Resolved specifically for this job's python 3.11
# (see the Dockerfile's explorer-extra-py313.txt for why this
# can't be shared with python 3.13: audioread needs extra
# standard-aifc/standard-sunau hashes only on 3.13+).
pip install -r .github/requirements/explorer-extra-py311.txt --require-hashes
pip install -r .github/requirements/pytest-tool.txt --require-hashes
- name: Test deterministic Explorer backend path
run: |
pytest -q tests/explorer/test_explorer_deterministic_rendering_e2e.py
BIN
View File
Binary file not shown.
-26
View File
@@ -9,32 +9,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.7.0] - 2026-09-07
### Changed
- **Slim core dependencies: moved ~22 heavy packages to optional extras** (#1513)
- Core dependencies in `pyproject.toml` are now reduced to exactly 22 direct packages: `numpy`, `pandas`, `scipy`, `scikit-learn`, `rdflib`, `networkx`, `requests`, `chardet`, `protobuf`, `grpcio`, `pillow`, `pydantic`, `click`, `rich`, `tqdm`, `pyyaml`, `toml`, `python-dotenv`, `loguru`, `structlog`, `httpx`, and `pyarrow`.
- Heavy ML/NLP, visualization, document parsing, and ingestion packages moved into granular optional extras:
- `models-huggingface`: `torch`, `transformers`
- `embeddings-local`: `sentence-transformers`, `fastembed`, `onnxruntime`, `tokenizers`
- `nlp-spacy`: `spacy`
- `viz`: expanded to include `matplotlib`, `seaborn`, `plotly`, `ipywidgets`, `umap-learn`, alongside `pyvis`, `graphviz`, and `d3blocks`
- `media`: `librosa`, `opencv-python`
- `vectorstore-faiss`: `faiss-cpu` (also included in `vectorstore-all`)
- `documents`: `python-docx`, `openpyxl`, `lxml`, `beautifulsoup4`
- `ingest-git`: `GitPython`
- `graph-embeddings`: `gensim` (also included in `graph-all`)
- Full bundled behavior preserved via `pip install "semantica[all]"`, which includes all optional extras. Pinning `semantica<0.7.0` remains a permanent escape hatch for legacy workflows.
- Safe lazy construction across parsers and visualizers:
- `DOCXParser`, `ExcelParser`, `HTMLParser`, and `XMLParser` remain constructible without error on `__init__()`. They fail only upon calling `.parse()` with actionable error messages directing users to install `semantica[documents]`.
- `XMLParser` automatically falls back to standard library `xml.etree` (`_parse_with_etree`) when `lxml` is not installed, preserving XML parsing capabilities without extra dependencies.
- `EmbeddingVisualizer` and `OntologyVisualizer` safely guard `matplotlib` and optional reduction packages, advising `pip install 'semantica[viz]'`.
- `RepoIngestor` guards `GitPython` with a clear error pointing to `semantica[ingest-git]`.
- `PublicAPIIngestor` guards `lxml` and `_SAFE_XML_PARSER`.
- Updated user-facing installation hints across CLI doctor commands, node embeddings (`NodeEmbedder`), vector stores (`FAISSStore`), and model loaders.
- Recompiled CI lockfiles (`requirements-ci.txt`, `.github/requirements/explorer-extra-py311.txt`, `.github/requirements/explorer-extra-py313.txt`, and `.github/requirements/base-deps.txt`).
## [0.6.8] - 2026-09-05
### Added
+1 -1
View File
@@ -7,7 +7,7 @@ authors:
repository-code: "https://github.com/semantica-agi/semantica"
url: "https://getsemantica.ai"
license: MIT
version: 0.7.0
version: 0.6.8
date-released: 2026-09-05
keywords:
- knowledge-graph
+21 -39
View File
@@ -1480,14 +1480,6 @@ app = create_app(session=GraphSession(graph), agent_memory=memory)
The Memories workspace is shown only when `agent_memory` is provided. Apply
updates the supplied runtime object; it does not add disk persistence.
## What's New in v0.7.0
**Slim core dependencies: lightweight base install with granular optional extras**`pip install semantica` now installs only 22 essential core dependencies, moving heavy packages into dedicated optional extras:
- **Dramatically lighter and faster installation**: Core installation no longer pulls heavy machine learning or visualization packages by default.
- **Granular extras**: Install only what your workload requires (`documents`, `embeddings-local`, `models-huggingface`, `nlp-spacy`, `viz`, `media`, `vectorstore-faiss`, `graph-embeddings`, `ingest-git`).
- **Full backward compatibility**: `pip install "semantica[all]"` preserves the full bundled suite, while `semantica<0.7.0` remains a permanent escape hatch.
- **Lazy parser construction & graceful fallbacks**: Document parsers can be constructed without extras and only raise actionable error hints upon calling `.parse()`; `XMLParser` automatically falls back to Python's standard library `xml.etree`.
---
## What's New in v0.6.8
@@ -1526,42 +1518,32 @@ Semantica is designed for environments where AI outputs must be explainable, aud
## Installation
```bash
pip install semantica # lightweight core (22 essential dependencies)
pip install "semantica[all]" # full bundled behavior with all extras
pip install semantica # core
pip install semantica[all] # everything
```
> **Note for upgrades from <0.7.0**: In Semantica 0.7.0+, heavy machine learning, NLP, visualization, and document dependencies were moved into optional extras to make core installation significantly lighter and faster. If you want the previous bundled installation, install with `pip install "semantica[all]"` or pin `semantica<0.7.0`.
```bash
# Granular Extras
pip install "semantica[documents]" # Document parsing (docx, openpyxl, lxml, beautifulsoup4)
pip install "semantica[embeddings-local]" # Local embeddings (sentence-transformers, fastembed, onnxruntime)
pip install "semantica[models-huggingface]" # HuggingFace models (transformers, torch)
pip install "semantica[nlp-spacy]" # spaCy NLP pipelines (spacy)
pip install "semantica[viz]" # Visualization (matplotlib, seaborn, plotly, pyvis, graphviz)
pip install "semantica[media]" # Audio & computer vision (librosa, opencv-python)
pip install "semantica[graph-embeddings]" # Knowledge graph embeddings (gensim / Node2Vec)
pip install "semantica[ingest-git]" # Git repository ingestor (GitPython)
pip install "semantica[vectorstore-faiss]" # FAISS vector store
pip install "semantica[vectorstore-all]" # All vector stores (Qdrant, Pinecone, Weaviate, FAISS, PgVector, SQLite)
pip install "semantica[agno]" # Agno multi-agent integration
pip install "semantica[crewai]" # CrewAI integration
pip install "semantica[langchain]" # LangChain / LangGraph integration
pip install "semantica[llm-all]" # All LLM provider clients
pip install "semantica[graph-neo4j]" # Neo4j graph store (LPG)
pip install "semantica[graph-falkordb]" # FalkorDB graph store (LPG)
pip install "semantica[graph-apache-age]" # Apache AGE graph store (LPG)
pip install "semantica[graph-amazon-neptune]" # AWS Neptune graph store (LPG)
pip install "semantica[tripletstore-oxigraph]" # Embedded in-memory/on-disk RDF store
pip install semantica[agno] # Agno multi-agent integration
pip install semantica[crewai] # CrewAI integration
pip install semantica[langchain] # LangChain / LangGraph integration
pip install semantica[llm-litellm] # OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Bedrock, Ollama, DeepSeek, and more
pip install semantica[graph-neo4j] # Neo4j graph store (LPG)
pip install semantica[graph-falkordb] # FalkorDB graph store (LPG)
pip install semantica[graph-apache-age] # Apache AGE graph store (LPG)
pip install semantica[graph-amazon-neptune] # AWS Neptune graph store (LPG)
pip install semantica[tripletstore-oxigraph] # Embedded in-memory/on-disk RDF store
# RDF triple stores (Blazegraph, Apache Jena, Eclipse RDF4J) need no extra:
# semantica.triplet_store talks SPARQL over HTTP using the core `requests` dependency
pip install "semantica[db-snowflake]" # Snowflake
pip install "semantica[db-databricks]" # Databricks (SDK + SQL connector)
pip install "semantica[ingest-sap]" # SAP OData
pip install "semantica[ingest-parquet]" # Parquet / PyArrow
pip install "semantica[ingest-arrow]" # Apache Arrow, Feather, IPC
pip install "semantica[watch]" # Directory file watcher
pip install "semantica[explorer]" # Knowledge Explorer dashboard
pip install semantica[vectorstore-qdrant] # Qdrant vector store
pip install semantica[vectorstore-pinecone] # Pinecone vector store
pip install semantica[db-snowflake] # Snowflake
pip install semantica[db-databricks] # Databricks (SDK + SQL connector)
pip install semantica[ingest-sap] # SAP OData
pip install semantica[ingest-parquet] # Parquet / PyArrow
pip install semantica[ingest-arrow] # Apache Arrow, Feather, IPC
pip install semantica[viz] # HTML interactive visualization
pip install semantica[watch] # Directory file watcher
pip install semantica[explorer] # Knowledge Explorer dashboard
```
For production deployments, use Docker or Kubernetes rather than a local `pip install`. Set `SEMANTICA_API_KEY`, configure a persistent LPG graph store (Neo4j / FalkorDB / Apache AGE / AWS Neptune) and/or RDF triple store (Blazegraph / Apache Jena / Eclipse RDF4J), and point the vector store at a hosted backend (Qdrant / Pinecone). See [ARCHITECTURE.md](ARCHITECTURE.md) for the full deployment topology.
+4 -4
View File
@@ -160,7 +160,7 @@ No installation or API key required. FAISS requires `pip install faiss-cpu`.
<Tab title="Pinecone">
```bash
pip install "semantica[pinecone]"
pip install "semantica[vectorstore-pinecone]"
```
```python
@@ -178,7 +178,7 @@ store = VectorStore(
<Tab title="Weaviate">
```bash
pip install "semantica[weaviate]"
pip install "semantica[vectorstore-weaviate]"
```
```python
@@ -194,7 +194,7 @@ store = VectorStore(
<Tab title="Qdrant">
```bash
pip install "semantica[qdrant]"
pip install "semantica[vectorstore-qdrant]"
```
```python
@@ -210,7 +210,7 @@ store = VectorStore(
<Tab title="PgVector">
```bash
pip install "semantica[pgvector]"
pip install "semantica[vectorstore-pgvector]"
```
```python
+83 -89
View File
@@ -21,7 +21,6 @@ from typing import Any, List, Optional
try:
from google.adk.events import Event
from google.adk.sessions import BaseSessionService, Session
try:
from google.adk.sessions import ListSessionsResponse
except ImportError:
@@ -34,7 +33,7 @@ try:
except ImportError:
from google.adk.sessions.base_session_service import GetSessionConfig
ADK_AVAILABLE = True
except (ImportError, OSError):
except (ImportError, ModuleNotFoundError):
ADK_AVAILABLE = False
BaseSessionService = object
Session = Any
@@ -163,10 +162,10 @@ class SemanticaSessionService(BaseSessionService):
return {}
def _find_session_node(
self,
app_name: str,
user_id: str,
session_id: str,
self,
app_name: str,
user_id: str,
session_id: str,
) -> Optional[Any]:
"""Find a session node by its logical ADK session ID."""
expected_node_id = self._node_id(app_name, user_id, session_id)
@@ -183,18 +182,18 @@ class SemanticaSessionService(BaseSessionService):
metadata = node.get("metadata")
if (
isinstance(metadata, dict)
and str(metadata.get("session_id")) == str(session_id)
and str(metadata.get("app_name")) == str(app_name)
and str(metadata.get("user_id")) == str(user_id)
isinstance(metadata, dict)
and str(metadata.get("session_id")) == str(session_id)
and str(metadata.get("app_name")) == str(app_name)
and str(metadata.get("user_id")) == str(user_id)
):
return node
return None
def _find_node_by_id(
self,
node_id: str,
self,
node_id: str,
) -> Optional[Any]:
"""Find a ContextGraph node by graph node ID."""
for node in self.graph.find_nodes() or []:
@@ -213,13 +212,13 @@ class SemanticaSessionService(BaseSessionService):
data = SemanticaSessionService._safe_dict(event)
for field in (
"id",
"invocation_id",
"author",
"timestamp",
"partial",
"turn_complete",
"branch",
"id",
"invocation_id",
"author",
"timestamp",
"partial",
"turn_complete",
"branch",
):
if field not in data and hasattr(event, field):
value = getattr(event, field)
@@ -232,10 +231,10 @@ class SemanticaSessionService(BaseSessionService):
return data
def _event_nodes(
self,
app_name: str,
user_id: str,
session_id: str,
self,
app_name: str,
user_id: str,
session_id: str,
) -> List[Any]:
"""Return all event nodes connected to a session."""
session_node_id = self._node_id(app_name, user_id, session_id)
@@ -276,8 +275,8 @@ class SemanticaSessionService(BaseSessionService):
return str(timestamp)
def _event_from_node(
self,
node: Any,
self,
node: Any,
) -> Any:
"""
Reconstruct an ADK Event from its stored metadata.
@@ -290,7 +289,7 @@ class SemanticaSessionService(BaseSessionService):
if not event_id and graph_node_id:
graph_node_id = str(graph_node_id)
if graph_node_id.startswith("adk-event:"):
event_id = graph_node_id[len("adk-event:") :]
event_id = graph_node_id[len("adk-event:"):]
if event_id:
properties["id"] = event_id
@@ -311,11 +310,11 @@ class SemanticaSessionService(BaseSessionService):
@staticmethod
def _session_kwargs(
app_name: str,
user_id: str,
session_id: str,
state: Optional[dict],
events: Optional[List[Any]],
app_name: str,
user_id: str,
session_id: str,
state: Optional[dict],
events: Optional[List[Any]],
) -> dict:
"""Build kwargs for the ADK Session model."""
return {
@@ -327,8 +326,8 @@ class SemanticaSessionService(BaseSessionService):
}
def _session_from_node(
self,
node: Any,
self,
node: Any,
) -> Session:
"""Reconstruct an ADK Session from a ContextGraph node."""
properties = self._node_properties(node)
@@ -348,14 +347,14 @@ class SemanticaSessionService(BaseSessionService):
# splitting on ':' after the prefix always yields
# exactly 3 parts regardless of what characters the
# original app_name/user_id/session_id contained.
parts = graph_node_id[len("adk-session:") :].split(":")
parts = graph_node_id[len("adk-session:"):].split(":")
if len(parts) == 3:
decoded = [urllib.parse.unquote(part) for part in parts]
app_name = app_name or decoded[0]
user_id = user_id or decoded[1]
session_id = decoded[2]
else:
session_id = graph_node_id[len("adk-session:") :]
session_id = graph_node_id[len("adk-session:"):]
else:
session_id = graph_node_id
@@ -384,12 +383,12 @@ class SemanticaSessionService(BaseSessionService):
# ------------------------------------------------------------------
async def create_session(
self,
*,
app_name: str,
user_id: str,
state: Optional[dict[str, Any]] = None,
session_id: Optional[str] = None,
self,
*,
app_name: str,
user_id: str,
state: Optional[dict[str, Any]] = None,
session_id: Optional[str] = None,
) -> Session:
"""Create and persist an ADK session."""
return await asyncio.to_thread(
@@ -397,11 +396,11 @@ class SemanticaSessionService(BaseSessionService):
)
def _create_session_sync(
self,
app_name: str,
user_id: str,
state: Optional[dict[str, Any]],
session_id: Optional[str],
self,
app_name: str,
user_id: str,
state: Optional[dict[str, Any]],
session_id: Optional[str],
) -> Session:
with self._lock:
session_id = session_id or str(uuid.uuid4())
@@ -431,12 +430,12 @@ class SemanticaSessionService(BaseSessionService):
)
async def get_session(
self,
*,
app_name: str,
user_id: str,
session_id: str,
config: Optional[GetSessionConfig] = None,
self,
*,
app_name: str,
user_id: str,
session_id: str,
config: Optional[GetSessionConfig] = None,
) -> Optional[Session]:
"""Retrieve an ADK session from ContextGraph."""
return await asyncio.to_thread(
@@ -444,11 +443,11 @@ class SemanticaSessionService(BaseSessionService):
)
def _get_session_sync(
self,
app_name: str,
user_id: str,
session_id: str,
config: Optional[GetSessionConfig],
self,
app_name: str,
user_id: str,
session_id: str,
config: Optional[GetSessionConfig],
) -> Optional[Session]:
with self._lock:
node = self._find_session_node(app_name, user_id, session_id)
@@ -469,7 +468,7 @@ class SemanticaSessionService(BaseSessionService):
# trims the already-built Session object.
if config:
if config.num_recent_events:
session.events = session.events[-config.num_recent_events :]
session.events = session.events[-config.num_recent_events:]
if config.after_timestamp:
i = len(session.events) - 1
while i >= 0:
@@ -477,14 +476,14 @@ class SemanticaSessionService(BaseSessionService):
break
i -= 1
if i >= 0:
session.events = session.events[i + 1 :]
session.events = session.events[i + 1:]
return session
async def append_event(
self,
session: Session,
event: Event,
self,
session: Session,
event: Event,
) -> Event:
"""Persist an ADK event and associate it with a session."""
# ADK's own base implementation is a no-op for partial/streaming
@@ -511,13 +510,8 @@ class SemanticaSessionService(BaseSessionService):
# Verify cross-tenant security
properties = self._node_properties(session_node)
if (
properties.get("app_name") != app_name
or properties.get("user_id") != user_id
):
raise ValueError(
"Cross-tenant session write denied: app_name or user_id mismatch."
)
if properties.get("app_name") != app_name or properties.get("user_id") != user_id:
raise ValueError("Cross-tenant session write denied: app_name or user_id mismatch.")
# Apply ADK in-memory event and state delta semantics. This
# runs inside asyncio.to_thread's worker thread, which has no
@@ -559,11 +553,11 @@ class SemanticaSessionService(BaseSessionService):
)
async def delete_session(
self,
*,
app_name: str,
user_id: str,
session_id: str,
self,
*,
app_name: str,
user_id: str,
session_id: str,
) -> None:
"""Delete a session and all of its graph-backed events."""
await asyncio.to_thread(
@@ -571,10 +565,10 @@ class SemanticaSessionService(BaseSessionService):
)
def _delete_session_sync(
self,
app_name: str,
user_id: str,
session_id: str,
self,
app_name: str,
user_id: str,
session_id: str,
) -> None:
with self._lock:
session_node = self._find_session_node(app_name, user_id, session_id)
@@ -596,9 +590,9 @@ class SemanticaSessionService(BaseSessionService):
continue
if (
edge.get("source") == session_node_id
and edge.get("type") == "HAS_EVENT"
and edge.get("target")
edge.get("source") == session_node_id
and edge.get("type") == "HAS_EVENT"
and edge.get("target")
):
event_node_ids.append(str(edge["target"]))
@@ -608,18 +602,18 @@ class SemanticaSessionService(BaseSessionService):
self.graph.purge_node(session_node_id)
async def list_sessions(
self,
*,
app_name: str,
user_id: Optional[str] = None,
self,
*,
app_name: str,
user_id: Optional[str] = None,
) -> ListSessionsResponse:
"""List sessions for an app, optionally scoped to one user."""
return await asyncio.to_thread(self._list_sessions_sync, app_name, user_id)
def _list_sessions_sync(
self,
app_name: str,
user_id: Optional[str],
self,
app_name: str,
user_id: Optional[str],
) -> ListSessionsResponse:
with self._lock:
sessions: List[Session] = []
@@ -649,4 +643,4 @@ class SemanticaSessionService(BaseSessionService):
__all__ = [
"ADK_AVAILABLE",
"SemanticaSessionService",
]
]
+39 -36
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "semantica"
version = "0.7.0"
version = "0.6.8"
description = "Graph-Native Infrastructure for Context and Accountable AI Systems: context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
readme = "README.md"
license = { text = "MIT" }
@@ -52,13 +52,30 @@ dependencies = [
# last 3.9-compatible release line; 3.10+ is left unconstrained.
"scikit-learn>=1.6.1,<1.7.0; python_version < '3.10'",
"scikit-learn>=1.7.2; python_version >= '3.10'",
"umap-learn>=0.5.12",
# thinc (spacy's core dep) dropped Python 3.9 wheels at 8.3.10, and later
# spacy patch releases (3.8.8+) require thinc>=8.3.9-only-on-3.10+ ranges,
# which forces a source build that fails outright on 3.9 (see Install
# Matrix run history). Capping both keeps 3.9 on the last wheel-compatible
# pair; 3.10+ is left unconstrained to always get the latest spacy/thinc.
"spacy>=3.4.0,<3.8.8; python_version < '3.10'",
"spacy>=3.4.0; python_version >= '3.10'",
"thinc<8.3.5; python_version < '3.10'",
"transformers>=4.20.0",
"torch>=1.13.1",
"sentence-transformers>=2.2.0",
"rdflib>=6.2.0",
"networkx>=2.8.0",
"matplotlib>=3.9.4",
"seaborn>=0.13.2",
"plotly>=6.8.0",
"ipywidgets>=8.0.0",
# requests dropped Python 3.9 support at 2.33.0 (requires_python >=3.10),
# so an unqualified >=2.34.2 floor is unsatisfiable on 3.9. Cap 3.9 to the
# last 3.9-compatible release; 3.10+ is left unconstrained.
"requests>=2.32.5,<2.33.0; python_version < '3.10'",
"requests>=2.34.2; python_version >= '3.10'",
"GitPython>=3.1.58",
# chardet dropped Python 3.9 support at 6.0.0 (requires_python >=3.10), so
# an unqualified >=7.4.3 floor is unsatisfiable on 3.9. Cap 3.9 to the last
# 3.9-compatible release; 3.10+ is left unconstrained.
@@ -70,11 +87,26 @@ dependencies = [
# 3.9-compatible release; 3.10+ is left unconstrained.
"grpcio>=1.80.0,<1.81.0; python_version < '3.10'",
"grpcio>=1.81.1; python_version >= '3.10'",
"beautifulsoup4>=4.15.0",
"lxml>=6.1.1",
"python-docx>=1.2.0",
"openpyxl>=3.1.5",
# pillow dropped Python 3.9 support at 12.0.0 (requires_python >=3.10), so
# an unqualified >=12.2.0 floor is unsatisfiable on 3.9. Cap 3.9 to the last
# 3.9-compatible release; 3.10+ is left unconstrained.
"pillow>=11.3.0,<12.0.0; python_version < '3.10'",
"pillow>=12.2.0; python_version >= '3.10'",
"librosa>=0.9.0",
"opencv-python>=4.13.0.92",
"faiss-cpu>=1.7.0",
"fastembed>=0.2.0",
# onnxruntime stopped shipping cp39 wheels at 1.20.0 (its PyPI metadata
# still claims requires_python >=3.9, but no matching wheel exists), so an
# unqualified >=1.20.1 floor is unsatisfiable on 3.9. Cap 3.9 to the last
# release with a cp39 wheel; 3.10+ is left unconstrained.
"onnxruntime>=1.19.2,<1.20.0; python_version < '3.10'",
"onnxruntime>=1.20.1; python_version >= '3.10'",
"tokenizers>=0.15.0",
"pydantic>=2.13.4",
# click dropped Python 3.9 support at 8.2.0 (requires_python >=3.10), so an
# unqualified >=8.4.2 floor is unsatisfiable on 3.9. Cap 3.9 to the last
@@ -88,6 +120,7 @@ dependencies = [
"python-dotenv>=1.2.1",
"loguru>=0.7.3",
"structlog>=22.1.0",
"gensim>=4.4.0",
"httpx<0.29.0",
"pyarrow>=14.0.0"
]
@@ -119,12 +152,6 @@ llm-all = [
]
# ---- Document Parsing ----
documents = [
"python-docx>=1.2.0",
"openpyxl>=3.1.5",
"lxml>=6.1.1",
"beautifulsoup4>=4.15.0"
]
parse-docling = ["docling>=2.107.0"]
# ---- SHACL Validation ----
@@ -138,7 +165,6 @@ db-salesforce = ["simple-salesforce>=1.12.0"]
ingest-parquet = ["pyarrow>=24.0.0"]
ingest-arrow = ["pyarrow>=24.0.0"]
ingest-sap = ["requests>=2.28.0"]
ingest-git = ["GitPython>=3.1.58"]
db-all = [
"semantica[db-snowflake,db-databricks,db-salesforce,db-arrow]"
@@ -149,34 +175,21 @@ models-huggingface = [
"transformers>=4.20.0",
"torch>=1.13.1"
]
embeddings-local = [
"sentence-transformers>=2.2.0",
"fastembed>=0.2.0",
"onnxruntime>=1.19.2,<1.20.0; python_version < '3.10'",
"onnxruntime>=1.20.1; python_version >= '3.10'",
"tokenizers>=0.15.0"
]
nlp-spacy = [
"spacy>=3.4.0,<3.8.8; python_version < '3.10'",
"spacy>=3.4.0; python_version >= '3.10'"
]
# ---- Graph Backends ----
graph-neo4j = ["neo4j>=5.0.0"]
graph-falkordb = ["falkordb>=1.0.0", "redis>=4.3.0"]
graph-amazon-neptune = ["boto3>=1.24.0", "neo4j>=5.0.0"]
graph-apache-age = ["psycopg2-binary>=2.9.0"]
graph-embeddings = ["gensim>=4.4.0"]
graph-all = [
"semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune,graph-apache-age,graph-embeddings]"
"semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune,graph-apache-age]"
]
# ---- Triplet Store Backends ----
tripletstore-oxigraph = ["pyoxigraph>=0.5.0"]
# ---- Vector Store Backends ----
vectorstore-faiss = ["faiss-cpu>=1.7.0"]
vectorstore-qdrant = ["qdrant-client>=1.0.0"]
vectorstore-weaviate = ["weaviate-client>=4.0.0"]
vectorstore-pinecone = ["pinecone>=3.0.0"]
@@ -185,7 +198,7 @@ vectorstore-pgvector = ["psycopg[binary,pool]>=3.0.0", "pgvector>=0.2.0"]
vectorstore-sqlite = ["sqlite-vec>=0.1.1"]
vectorstore-all = [
"semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus,vectorstore-pgvector,vectorstore-sqlite,vectorstore-faiss]"
"semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus,vectorstore-pgvector,vectorstore-sqlite]"
]
# ---- Infra / Queues / Workers ----
@@ -217,18 +230,7 @@ monitoring = [
viz = [
"pyvis>=0.3.0",
"graphviz>=0.21",
"d3blocks>=1.0.0",
"matplotlib>=3.9.4",
"seaborn>=0.13.2",
"plotly>=6.8.0",
"ipywidgets>=8.0.0",
"umap-learn>=0.5.12"
]
# ---- Media ----
media = [
"librosa>=0.9.0",
"opencv-python>=4.13.0.92"
"d3blocks>=1.0.0"
]
# ---- GPU ----
@@ -292,7 +294,8 @@ explorer-lite = [
# (CVE-2026-45829) with no fixed release — including it here would fail the CI
# dependency-audit/security gates. Install it explicitly via ``semantica[crewai]``.
all = [
"semantica[dev,viz,media,infra,cloud,monitoring,watch,llm-all,models-huggingface,embeddings-local,nlp-spacy,documents,ingest-git,graph-embeddings,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer,agno,langchain,google-adk]"
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno,langchain,google-adk]"
]
# ---------------- ENTRYPOINTS ----------------
-1
View File
@@ -181,7 +181,6 @@ anyio==4.14.2 \
# jupyter-server
# langsmith
# openai
# pinecone
# starlette
# watchfiles
argon2-cffi==25.1.0 \
+1 -1
View File
@@ -10,7 +10,7 @@ Main exports:
- Config: Configuration management
"""
__version__ = "0.7.0"
__version__ = "0.6.8"
__author__ = "Semantica Contributors"
__license__ = "MIT"
+16 -26
View File
@@ -880,18 +880,10 @@ def doctor(cli_ctx: CLIContext, local_json: bool, deep_embeddings: bool) -> None
def _embedding_backend(method: str) -> str:
if method == "sentence_transformers":
import sentence_transformers # noqa: F401
try:
ver = importlib.metadata.version("sentence-transformers")
except Exception:
ver = getattr(sentence_transformers, "__version__", "installed")
note = f"importable ({ver})"
note = f"importable ({importlib.metadata.version('sentence-transformers')})"
else:
import fastembed # noqa: F401
try:
ver = importlib.metadata.version("fastembed")
except Exception:
ver = getattr(fastembed, "__version__", "installed")
note = f"importable ({ver})"
note = f"importable ({importlib.metadata.version('fastembed')})"
if not deep:
return note
try:
@@ -912,12 +904,12 @@ def doctor(cli_ctx: CLIContext, local_json: bool, deep_embeddings: bool) -> None
checks.append(_check(
"Embeddings (sentence-transformers)",
lambda: _embedding_backend("sentence_transformers"),
hint="pip install 'semantica[embeddings-local]'",
hint="pip install sentence-transformers",
))
checks.append(_check(
"Embeddings (fastembed)",
lambda: _embedding_backend("fastembed"),
hint="pip install 'semantica[embeddings-local]'",
hint="pip install fastembed",
))
# LLM provider keys
@@ -4772,15 +4764,10 @@ def mcp_list_tools(cli_ctx: CLIContext, local_json: bool) -> None:
cli_ctx = _require_ctx(cli_ctx)
def _action() -> None:
try:
from semantica_mcp.mcp.tools import __all__ as tools
except ImportError:
tools = [
"extract_entities", "extract_relations", "build_graph",
"query_graph", "get_graph_analytics", "run_reasoning",
"record_decision", "get_decisions", "export_graph",
"validate_shacl", "get_provenance", "embed_and_search",
]
# Same catalog the server exposes via tools/list, so `list-tools`
# and `mcp start` can't drift (issue #1355).
from semantica_mcp.mcp.tools import TOOL_DEFINITIONS
tools = [t["name"] for t in TOOL_DEFINITIONS]
if _is_json(cli_ctx, local_json):
_jecho({"tools": list(tools)})
else:
@@ -4813,12 +4800,15 @@ def mcp_call(cli_ctx: CLIContext, tool_name: str, args: str, local_json: bool) -
tool_args = json.loads(args)
except json.JSONDecodeError as exc:
raise click.ClickException(f"Invalid JSON in --args: {exc}") from exc
if not isinstance(tool_args, dict):
raise click.ClickException("--args must be a JSON object")
# Dispatch through the same server `mcp start` spawns; its session
# module never defined MCPSession (issue #1355).
from semantica_mcp.mcp.server import UnknownToolError, call_tool
try:
from semantica_mcp.mcp.session import MCPSession
session = MCPSession(config=cli_ctx.config.to_dict())
result = session.call_tool(tool_name, **tool_args)
except ImportError as exc:
raise click.ClickException(f"MCP module not available: {exc}") from exc
result = call_tool(tool_name, tool_args)
except UnknownToolError as exc:
raise click.ClickException(str(exc)) from exc
if _is_json(cli_ctx, local_json):
_jecho(result if isinstance(result, (dict, list)) else {"result": str(result)})
else:
+1 -1
View File
@@ -212,7 +212,7 @@ class FastEmbedStore(ProviderStore):
self.logger.info(f"Loaded FastEmbed model: {self.model_name}")
except (ImportError, OSError):
self.logger.warning(
"fastembed not available. Install with: pip install 'semantica[embeddings-local]'"
"fastembed not available. Install with: pip install fastembed"
)
except Exception as e:
self.logger.warning(f"Failed to load FastEmbed model: {e}")
+2 -4
View File
@@ -35,7 +35,6 @@ try:
SENTENCE_TRANSFORMERS_AVAILABLE = True
except (ImportError, OSError):
SentenceTransformer = None
SENTENCE_TRANSFORMERS_AVAILABLE = False
try:
@@ -43,7 +42,6 @@ try:
FASTEMBED_AVAILABLE = True
except (ImportError, OSError):
TextEmbedding = None
FASTEMBED_AVAILABLE = False
@@ -158,7 +156,7 @@ class TextEmbedder:
else:
self.logger.warning(
"fastembed not available. "
"Install with: pip install 'semantica[embeddings-local]'. "
"Install with: pip install fastembed. "
"Using fallback embedding method."
)
else:
@@ -180,7 +178,7 @@ class TextEmbedder:
else:
self.logger.warning(
"sentence-transformers not available. "
"Install with: pip install 'semantica[embeddings-local]'. "
"Install with: pip install sentence-transformers. "
"Using fallback embedding method."
)
+1 -1
View File
@@ -421,7 +421,7 @@ class VectorExporter:
import numpy as np
except (ImportError, OSError):
raise ImportError(
"FAISS not installed. Install with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
"FAISS not installed. Install with: pip install faiss-cpu or faiss-gpu"
)
# Extract vectors and IDs
+11 -67
View File
@@ -133,11 +133,7 @@ import importlib
from typing import TYPE_CHECKING, Any, Dict, Tuple
if TYPE_CHECKING:
from .salesforce_ingestor import (
SalesforceConnector,
SalesforceData,
SalesforceIngestor,
)
from .salesforce_ingestor import SalesforceConnector, SalesforceData, SalesforceIngestor
from .config import IngestConfig, ingest_config
from .file_ingestor import (
@@ -252,42 +248,32 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
_OPTIONAL_DEPENDENCY_MESSAGES = {
".repo_ingestor": (
"Repository ingestion requires optional dependency 'GitPython'. "
"Install it before importing RepoIngestor or using ingest_repository(). "
"Install it with: pip install 'semantica[ingest-git]'"
"Install it before importing RepoIngestor or using ingest_repository()."
),
".web_ingestor": (
"Web ingestion requires optional dependency 'beautifulsoup4'. "
"Install it before importing WebIngestor or using ingest_web(). "
"Install it with: pip install 'semantica[documents]'"
"Install it before importing WebIngestor or using ingest_web()."
),
".feed_ingestor": (
"Feed ingestion requires optional dependency 'beautifulsoup4'. "
"Install it before importing FeedIngestor or using ingest_feed(). "
"Install it with: pip install 'semantica[documents]'"
"Install it before importing FeedIngestor or using ingest_feed()."
),
".email_ingestor": (
"Email ingestion requires optional dependency 'beautifulsoup4'. "
"Install it before importing EmailIngestor or using ingest_email(). "
"Install it with: pip install 'semantica[documents]'"
),
".xml_ingestor": (
"XML ingestion requires optional dependency 'lxml'. "
"Install it before importing XMLIngestor or using ingest_xml(). "
"Install it with: pip install 'semantica[documents]'"
"Install it before importing EmailIngestor or using ingest_email()."
),
".parquet_ingestor": (
"Parquet ingestion requires optional dependency 'pyarrow'. "
"Install it before importing ParquetIngestor or using ingest_parquet(). "
"Install it with: pip install 'semantica[ingest-parquet]'"
"Install it before importing ParquetIngestor or using ingest_parquet()."
),
".arrow_ingestor": (
"Arrow ingestion requires optional dependency 'pyarrow'. "
"Install it before importing ArrowIngestor or using ingest_arrow(). "
"Install it with: pip install 'semantica[ingest-arrow]'"
"Install it before importing ArrowIngestor or using ingest_arrow()."
),
".salesforce_ingestor": (
"Salesforce ingestion requires optional dependency 'simple-salesforce'. "
"Install it with: pip install 'semantica[db-salesforce]'"
"Install it with: pip install \"semantica[db-salesforce]\" "
"or: pip install simple-salesforce>=1.12.0"
),
}
@@ -300,55 +286,13 @@ def __getattr__(name: str) -> Any:
module_name, attr_name = _LAZY_EXPORTS[name]
try:
module = importlib.import_module(module_name, __name__)
except (ImportError, OSError) as exc:
except ModuleNotFoundError as exc:
message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
missing_name = getattr(exc, "name", None)
if message and (
missing_name is None
or any(
pkg in missing_name
for pkg in ("git", "bs4", "pyarrow", "simple_salesforce", "lxml")
)
):
if message and missing_name in {"git", "bs4", "pyarrow", "simple_salesforce"}:
raise ImportError(message) from exc
raise
# Guard against backends whose modules imported cleanly with dependencies
# set to None; ensure probe imports (e.g. try: from semantica.ingest import ...)
# fail at import time rather than postponing failure to construction time.
if module_name == ".repo_ingestor" and name in {"RepoIngestor"}:
if getattr(module, "git", None) is None:
message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
if message:
raise ImportError(message)
if module_name == ".xml_ingestor" and name in {"XMLIngestor"}:
if getattr(module, "etree", None) is None:
message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
if message:
raise ImportError(message)
if module_name == ".parquet_ingestor" and name in {"ParquetIngestor"}:
if not getattr(module, "PARQUET_AVAILABLE", True):
message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
if message:
raise ImportError(message)
if module_name == ".arrow_ingestor" and name in {"ArrowIngestor"}:
if not getattr(module, "ARROW_AVAILABLE", True):
message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
if message:
raise ImportError(message)
if module_name == ".salesforce_ingestor" and name in {
"SalesforceIngestor",
"SalesforceConnector",
}:
if not getattr(module, "SALESFORCE_AVAILABLE", True):
message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
if message:
raise ImportError(message)
value = getattr(module, attr_name)
globals()[name] = value
return value
+129 -200
View File
@@ -193,7 +193,6 @@ def _is_scp_like_repo_source(source: str) -> bool:
"""Return True for scp-like SSH remotes (``user@host:path``)."""
return bool(_SCP_LIKE_REPO_URL_RE.match(source.strip()))
if TYPE_CHECKING:
from .api_ingestor import APIData
from .arrow_ingestor import ArrowData
@@ -253,12 +252,7 @@ def ingest_file(
if custom_method and custom_method != ingest_file:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -324,18 +318,21 @@ def ingest_parquet(
if custom_method and custom_method != ingest_parquet:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .parquet_ingestor import ParquetIngestor
try:
from .parquet_ingestor import ParquetIngestor
except ModuleNotFoundError as exc:
if _is_missing_dependency(exc, "pyarrow"):
raise _missing_optional_dependency(
"Parquet ingestion",
"pyarrow",
) from exc
raise
config = ingest_config.get_method_config("parquet")
config.update(kwargs)
@@ -400,18 +397,21 @@ def ingest_arrow(
if custom_method and custom_method != ingest_arrow:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .arrow_ingestor import ArrowIngestor
try:
from .arrow_ingestor import ArrowIngestor
except ModuleNotFoundError as exc:
if _is_missing_dependency(exc, "pyarrow"):
raise _missing_optional_dependency(
"Arrow ingestion",
"pyarrow",
) from exc
raise
config = ingest_config.get_method_config("arrow")
config.update(kwargs)
@@ -481,12 +481,7 @@ def ingest_xml(
if custom_method and custom_method != ingest_xml:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -496,10 +491,7 @@ def ingest_xml(
config = ingest_config.get_method_config("xml")
config.update(kwargs)
try:
ingestor = XMLIngestor(**config)
except ImportError as exc:
raise _missing_optional_dependency("XML ingestion", "lxml") from exc
ingestor = XMLIngestor(**config)
def _run_single(
path: Union[str, Path],
@@ -519,8 +511,6 @@ def ingest_xml(
return _run_single(source_path)
except ConfigurationError:
raise
except Exception as e:
logger.error(f"Failed to ingest XML: {e}")
raise
@@ -555,12 +545,7 @@ def ingest_web(
if custom_method and custom_method != ingest_web:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -650,12 +635,7 @@ def ingest_public_api(
if custom_method and custom_method != ingest_public_api:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -742,12 +722,7 @@ def ingest_feed(
if custom_method and custom_method != ingest_feed:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -816,12 +791,7 @@ def ingest_stream(
if custom_method and custom_method != ingest_stream:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -898,29 +868,26 @@ def ingest_repository(
if custom_method and custom_method != ingest_repository:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .repo_ingestor import RepoIngestor
try:
from .repo_ingestor import RepoIngestor
except ModuleNotFoundError as exc:
if _is_missing_dependency(exc, "git"):
raise _missing_optional_dependency(
"Repository ingestion", "GitPython"
) from exc
raise
# Get config
config = ingest_config.get_method_config("repo")
config.update(kwargs)
try:
ingestor = RepoIngestor(**config)
except ImportError as exc:
raise _missing_optional_dependency(
"Repository ingestion", "GitPython"
) from exc
ingestor = RepoIngestor(**config)
if method == "clone" or (
isinstance(source, str)
@@ -973,12 +940,7 @@ def ingest_email(
if custom_method and custom_method != ingest_email:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -1057,12 +1019,7 @@ def ingest_ontology(
if custom_method and custom_method != ingest_ontology:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -1128,12 +1085,7 @@ def ingest_database(
if custom_method and custom_method != ingest_database:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -1281,122 +1233,105 @@ def ingest_salesforce(
if custom_method and custom_method != ingest_salesforce:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source,
fallback_on_custom_error=fallback, **kwargs,
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
try:
from .salesforce_ingestor import SalesforceIngestor
# Unpack credential dict (if given); everything else stays in kwargs.
creds: Dict[str, Any] = {}
if source is not None:
if not isinstance(source, dict):
raise ProcessingError(
"ingest_salesforce() source must be a credential dict or None. "
"Pass sobject_name / soql as keyword arguments."
)
creds = dict(source)
# Merge any ingest_config method config under "salesforce".
# get_method_config() now returns a copy, so this dict is safe to mutate.
# We build the final connector config in order of increasing priority:
# 1. base method config (lowest — global defaults set by operator)
# 2. per-call credential dict supplied via `source`
# 3. per-call connector params supplied as kwargs
# Credentials are extracted from kwargs and removed so they don't also
# flow into the ingest method call (which doesn't understand them).
_CONNECTOR_PARAMS = frozenset(
{
"username",
"password",
"security_token",
"domain",
"instance_url",
"session_id",
"api_version",
}
)
connector_kwargs = {k: v for k, v in kwargs.items() if k in _CONNECTOR_PARAMS}
for k in _CONNECTOR_PARAMS:
kwargs.pop(k, None)
# Build a fresh per-call config dict — never mutate the global store.
config: Dict[str, Any] = {
**ingest_config.get_method_config("salesforce"), # base (already a copy)
**creds, # source dict credentials
**connector_kwargs, # kwarg credentials
}
try:
ingestor = SalesforceIngestor(**config)
except ImportError as exc:
except ModuleNotFoundError as exc:
if _is_missing_dependency(exc, "simple_salesforce"):
raise _missing_optional_dependency(
"Salesforce ingestion", "simple-salesforce"
) from exc
raise
if method == "sobject":
sobject_name = kwargs.pop("sobject_name", None)
if not sobject_name:
raise ProcessingError(
"ingest_salesforce() with method='sobject' requires "
"sobject_name keyword argument."
)
return ingestor.ingest_sobject(sobject_name, **kwargs)
elif method == "query":
soql = kwargs.pop("soql", None)
if not soql:
raise ProcessingError(
"ingest_salesforce() with method='query' requires "
"soql keyword argument."
)
return ingestor.ingest_query(soql, **kwargs)
elif method == "list_sobjects":
return ingestor.list_sobjects()
elif method == "schema":
sobject_name = kwargs.pop("sobject_name", None)
if not sobject_name:
raise ProcessingError(
"ingest_salesforce() with method='schema' requires "
"sobject_name keyword argument."
)
return ingestor.get_sobject_schema(sobject_name)
elif method == "documents":
sobject_name = kwargs.pop("sobject_name", None)
if not sobject_name:
raise ProcessingError(
"ingest_salesforce() with method='documents' requires "
"sobject_name keyword argument."
)
id_field = kwargs.pop("id_field", "Id")
text_fields = kwargs.pop("text_fields", None)
data = ingestor.ingest_sobject(sobject_name, **kwargs)
return ingestor.export_as_documents(
data, id_field=id_field, text_fields=text_fields
)
else:
# Unpack credential dict (if given); everything else stays in kwargs.
creds: Dict[str, Any] = {}
if source is not None:
if not isinstance(source, dict):
raise ProcessingError(
f"Unknown ingest_salesforce method: {method!r}. "
"Valid methods: 'sobject', 'query', 'list_sobjects', 'schema', "
"'documents'."
"ingest_salesforce() source must be a credential dict or None. "
"Pass sobject_name / soql as keyword arguments."
)
creds = dict(source)
except ConfigurationError:
raise
except Exception as e:
logger.error(f"Failed to ingest salesforce: {e}")
raise
# Merge any ingest_config method config under "salesforce".
# get_method_config() now returns a copy, so this dict is safe to mutate.
# We build the final connector config in order of increasing priority:
# 1. base method config (lowest — global defaults set by operator)
# 2. per-call credential dict supplied via `source`
# 3. per-call connector params supplied as kwargs
# Credentials are extracted from kwargs and removed so they don't also
# flow into the ingest method call (which doesn't understand them).
_CONNECTOR_PARAMS = frozenset({
"username", "password", "security_token", "domain",
"instance_url", "session_id", "api_version",
})
connector_kwargs = {k: v for k, v in kwargs.items() if k in _CONNECTOR_PARAMS}
for k in _CONNECTOR_PARAMS:
kwargs.pop(k, None)
# Build a fresh per-call config dict — never mutate the global store.
config: Dict[str, Any] = {
**ingest_config.get_method_config("salesforce"), # base (already a copy)
**creds, # source dict credentials
**connector_kwargs, # kwarg credentials
}
ingestor = SalesforceIngestor(**config)
if method == "sobject":
sobject_name = kwargs.pop("sobject_name", None)
if not sobject_name:
raise ProcessingError(
"ingest_salesforce() with method='sobject' requires "
"sobject_name keyword argument."
)
return ingestor.ingest_sobject(sobject_name, **kwargs)
elif method == "query":
soql = kwargs.pop("soql", None)
if not soql:
raise ProcessingError(
"ingest_salesforce() with method='query' requires "
"soql keyword argument."
)
return ingestor.ingest_query(soql, **kwargs)
elif method == "list_sobjects":
return ingestor.list_sobjects()
elif method == "schema":
sobject_name = kwargs.pop("sobject_name", None)
if not sobject_name:
raise ProcessingError(
"ingest_salesforce() with method='schema' requires "
"sobject_name keyword argument."
)
return ingestor.get_sobject_schema(sobject_name)
elif method == "documents":
sobject_name = kwargs.pop("sobject_name", None)
if not sobject_name:
raise ProcessingError(
"ingest_salesforce() with method='documents' requires "
"sobject_name keyword argument."
)
id_field = kwargs.pop("id_field", "Id")
text_fields = kwargs.pop("text_fields", None)
data = ingestor.ingest_sobject(sobject_name, **kwargs)
return ingestor.export_as_documents(data, id_field=id_field,
text_fields=text_fields)
else:
raise ProcessingError(
f"Unknown ingest_salesforce method: {method!r}. "
"Valid methods: 'sobject', 'query', 'list_sobjects', 'schema', "
"'documents'."
)
def ingest_mcp(
@@ -1464,12 +1399,7 @@ def ingest_mcp(
if custom_method and custom_method != ingest_mcp:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
source,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, source, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -1708,9 +1638,8 @@ def ingest(
elif source_type == "mcp":
return {"data": ingest_mcp(sources, method=method or "resources", **kwargs)}
elif source_type == "salesforce":
return {
"data": ingest_salesforce(sources, method=method or "sobject", **kwargs)
}
return {"data": ingest_salesforce(sources,
method=method or "sobject", **kwargs)}
else:
raise ProcessingError(f"Unknown source type: {source_type}")
+16 -56
View File
@@ -28,29 +28,12 @@ from typing import Any, Dict, List, Optional, Tuple
from urllib.parse import parse_qs, urlparse
import requests
try:
from lxml import etree as lxml_etree
_SAFE_XML_PARSER = lxml_etree.XMLParser(
resolve_entities=False,
no_network=True,
recover=False,
huge_tree=False,
load_dtd=False,
remove_comments=True,
remove_pis=True,
)
_LXML_SYNTAX_ERRORS: Tuple[type, ...] = (lxml_etree.XMLSyntaxError,)
except (ImportError, OSError):
lxml_etree = None
_SAFE_XML_PARSER = None
_LXML_SYNTAX_ERRORS = ()
from lxml import etree as lxml_etree
try:
from defusedxml import ElementTree as safe_xml_etree
from defusedxml.common import DefusedXmlException
except (ImportError, OSError): # pragma: no cover - fallback for minimal installs
except ModuleNotFoundError: # pragma: no cover - fallback for minimal installs
safe_xml_etree = None
class DefusedXmlException(Exception):
@@ -88,6 +71,14 @@ AUTH_PARAM_NAMES = {
"subscription-key",
}
_SAFE_XML_PARSER = lxml_etree.XMLParser(
resolve_entities=False,
no_network=True,
recover=False,
huge_tree=False,
load_dtd=False,
)
@dataclass
class PublicAPIExample:
@@ -452,9 +443,7 @@ class PublicAPIIngestor(RESTIngestor):
APIData: Normalized public API response and metadata
"""
self._validate_endpoint(endpoint)
self._validate_no_auth_request(
headers=headers, params=params, options=options, endpoint=endpoint
)
self._validate_no_auth_request(headers=headers, params=params, options=options, endpoint=endpoint)
tracking_id = self.progress_tracker.start_tracking(
file=endpoint,
@@ -615,9 +604,7 @@ class PublicAPIIngestor(RESTIngestor):
for endpoint in endpoints:
try:
results.append(
self.ingest_public_api(
endpoint, method=method, **copy.deepcopy(options)
)
self.ingest_public_api(endpoint, method=method, **copy.deepcopy(options))
)
except Exception as exc:
self.logger.warning(f"Failed to fetch public API {endpoint}: {exc}")
@@ -743,7 +730,7 @@ class PublicAPIIngestor(RESTIngestor):
raise ProcessingError(
f"Failed to parse {detected_format.upper()} public API response"
) from exc
except (DefusedXmlException, *_LXML_SYNTAX_ERRORS) as exc:
except (DefusedXmlException, lxml_etree.XMLSyntaxError) as exc:
raise ProcessingError("Failed to parse XML public API response") from exc
def _detect_response_format(
@@ -783,40 +770,15 @@ class PublicAPIIngestor(RESTIngestor):
def _parse_xml(self, xml_text: str) -> Dict[str, Any]:
if safe_xml_etree is not None:
root = safe_xml_etree.fromstring(xml_text)
elif lxml_etree is not None and _SAFE_XML_PARSER is not None:
else:
root = lxml_etree.fromstring(
xml_text.encode("utf-8"),
parser=_SAFE_XML_PARSER,
)
for elem in root.iter():
if (
elem.tag is lxml_etree.Comment
or elem.tag is lxml_etree.PI
or getattr(elem.tag, "__name__", "")
in ("Comment", "ProcessingInstruction", "PI")
):
continue
if callable(elem.tag) or not isinstance(elem.tag, str):
raise ProcessingError("Failed to parse XML public API response")
else:
raise ProcessingError(
"XML parsing requires 'defusedxml' or 'lxml'. "
"Install it with: pip install 'semantica[documents]'"
)
return self._element_to_dict(root)
def _element_to_dict(self, element: Any) -> Dict[str, Any]:
children = [
self._element_to_dict(child)
for child in list(element)
if not (
callable(child.tag)
or (
lxml_etree is not None
and (child.tag is lxml_etree.Comment or child.tag is lxml_etree.PI)
)
)
]
children = [self._element_to_dict(child) for child in list(element)]
return {
"tag": self._strip_namespace(element.tag),
"attributes": {
@@ -827,9 +789,7 @@ class PublicAPIIngestor(RESTIngestor):
"children": children,
}
def _strip_namespace(self, value: Any) -> str:
if not isinstance(value, str):
return str(value)
def _strip_namespace(self, value: str) -> str:
if value.startswith("{") and "}" in value:
return value.split("}", 1)[1]
return value
+22 -21
View File
@@ -29,8 +29,6 @@ Author: Semantica Contributors
License: MIT
"""
from __future__ import annotations
import ipaddress
import os
import re
@@ -46,10 +44,7 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Set, Tuple, Union
from urllib.parse import urlparse
try:
import git
except (ImportError, OSError):
git = None
import git
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
@@ -62,7 +57,9 @@ ALLOWED_CLONE_OPTIONS: Set[str] = {"depth", "branch", "single_branch", "no_tags"
ALLOWED_REPO_URL_SCHEMES = frozenset({"https", "http", "git", "ssh"})
# SCP-like SSH remotes: user@host:path/to/repo.git (no scheme)
_SCP_LIKE_REPO_URL_RE = re.compile(r"^[^@\s]+@[^:\s]+:.+$")
_ENV_VAR_TOKEN_RE = re.compile(r"\$(\{[A-Za-z_][A-Za-z0-9_]*\}|[A-Za-z_][A-Za-z0-9_]*)")
_ENV_VAR_TOKEN_RE = re.compile(
r"\$(\{[A-Za-z_][A-Za-z0-9_]*\}|[A-Za-z_][A-Za-z0-9_]*)"
)
# Short-lived DNS cache for host validation. This reduces repeated lookups but
# does not eliminate DNS-rebinding / TOCTOU races between validate and clone —
# network egress controls remain recommended.
@@ -528,11 +525,6 @@ class RepoIngestor:
**kwargs: Additional configuration parameters (merged into config)
"""
self.logger = get_logger("repo_ingestor")
if git is None:
raise ImportError(
"GitPython is required for repository ingestion. "
"Install it with: pip install 'semantica[ingest-git]'"
)
self.config = config or {}
self.config.update(kwargs)
@@ -598,7 +590,10 @@ class RepoIngestor:
networks). Those addresses are not SSRF-sensitive.
"""
return bool(
ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_unspecified
ip.is_private
or ip.is_loopback
or ip.is_link_local
or ip.is_unspecified
)
@staticmethod
@@ -625,7 +620,9 @@ class RepoIngestor:
# or hanging lookup for one host cannot stall cache access for
# concurrent lookups of other hosts.
try:
addrinfos = socket.getaddrinfo(host, None, type=socket.SOCK_STREAM)
addrinfos = socket.getaddrinfo(
host, None, type=socket.SOCK_STREAM
)
except socket.gaierror as exc:
raise ValidationError(
f"Cannot resolve repository host {host!r}: {exc}"
@@ -798,7 +795,9 @@ class RepoIngestor:
f"Allowed schemes: {sorted(ALLOWED_REPO_URL_SCHEMES)}"
)
if not parsed.netloc or not host:
raise ValidationError(f"Repository URL must include a host: {repo_url}")
raise ValidationError(
f"Repository URL must include a host: {repo_url}"
)
RepoIngestor._validate_repo_host(host)
@@ -813,7 +812,9 @@ class RepoIngestor:
"include_extensions",
"max_depth",
}
candidate = {k: v for k, v in options.items() if k not in non_git_options}
candidate = {
k: v for k, v in options.items() if k not in non_git_options
}
unsafe = set(candidate) - ALLOWED_CLONE_OPTIONS
if unsafe:
raise ValidationError(
@@ -892,7 +893,9 @@ class RepoIngestor:
if "include_extensions" in options:
# Normalize extensions to include dot prefix
exts = options["include_extensions"]
normalized_exts = [e if e.startswith(".") else f".{e}" for e in exts]
normalized_exts = [
e if e.startswith(".") else f".{e}" for e in exts
]
file_filters["extensions"] = normalized_exts
# Process code files
@@ -1059,14 +1062,14 @@ class RepoIngestor:
return code_files
def get_repository_info(
self, repo_url: str, repo: Optional[Any] = None
self, repo_url: str, repo: Optional[git.Repo] = None
) -> Dict[str, Any]:
"""
Get repository metadata and information.
Args:
repo_url: Repository URL
repo: Git repository object (git.Repo, optional)
repo: Git repository object (optional)
Returns:
dict: Repository information
@@ -1108,7 +1111,6 @@ class RepoIngestor:
def cleanup(self):
"""Cleanup temporary repository files."""
if self.temp_dir and os.path.exists(self.temp_dir):
def onexc(func, path, exc_info):
"""
Error handler for shutil.rmtree.
@@ -1121,7 +1123,6 @@ class RepoIngestor:
Usage : shutil.rmtree(path, onerror=onexc)
"""
import stat
if not os.access(path, os.W_OK):
# Is the error an access error ?
os.chmod(path, stat.S_IWUSR)
+3 -11
View File
@@ -24,10 +24,7 @@ from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple, Union
try:
from lxml import etree
except (ImportError, OSError):
etree = None
from lxml import etree
from ..utils.constants import FILE_SIZE_LIMITS
from ..utils.exceptions import ProcessingError, ValidationError
@@ -75,11 +72,6 @@ class XMLIngestor:
**kwargs: Additional configuration values
"""
self.logger = get_logger("xml_ingestor")
if etree is None:
raise ImportError(
"lxml is required for XMLIngestor. "
"Install it with: pip install 'semantica[documents]'"
)
self.config = config or {}
self.config.update(kwargs)
self.progress_tracker = get_progress_tracker()
@@ -792,8 +784,8 @@ class XMLIngestor:
first_error = errors[0] if errors else "No detailed validation error available."
return f"{prefix} for {source}: {first_error}"
def _format_xml_error(self, exc: Any) -> str:
if hasattr(exc, "error_log") and exc.error_log:
def _format_xml_error(self, exc: etree.XMLSyntaxError) -> str:
if exc.error_log:
return str(exc.error_log.last_error)
return str(exc)
+1 -1
View File
@@ -141,7 +141,7 @@ class NodeEmbedder:
if method == "node2vec" and not GENSIM_AVAILABLE:
raise ImportError(
"gensim is required for Node2Vec. Install with: pip install 'semantica[graph-embeddings]'"
"gensim is required for Node2Vec. Install with: pip install gensim"
)
def compute_embeddings(
+7 -23
View File
@@ -32,20 +32,12 @@ from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
try:
from docx import Document
from docx.document import Document as DocxDocument
from docx.oxml.table import CT_Tbl
from docx.oxml.text.paragraph import CT_P
from docx.table import Table
from docx.text.paragraph import Paragraph
except (ImportError, OSError):
Document = None
DocxDocument = None
CT_Tbl = None
CT_P = None
Table = None
Paragraph = None
from docx import Document
from docx.document import Document as DocxDocument
from docx.oxml.table import CT_Tbl
from docx.oxml.text.paragraph import CT_P
from docx.table import Table
from docx.text.paragraph import Paragraph
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
@@ -92,9 +84,7 @@ class DOCXParser:
self.config = config
self.progress_tracker = get_progress_tracker()
def parse(
self, file_path: Union[str, Path], pipeline_id: Optional[str] = None, **options
) -> Dict[str, Any]:
def parse(self, file_path: Union[str, Path], pipeline_id: Optional[str] = None, **options) -> Dict[str, Any]:
"""
Parse DOCX document.
@@ -109,12 +99,6 @@ class DOCXParser:
Returns:
dict: Parsed document data
"""
if Document is None:
raise ProcessingError(
"python-docx is required to parse DOCX files. "
"Install it with: pip install 'semantica[documents]'"
)
file_path = Path(file_path)
# Track DOCX parsing
+1 -11
View File
@@ -33,11 +33,7 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Union
import pandas as pd
try:
from openpyxl import load_workbook
except (ImportError, OSError):
load_workbook = None
from openpyxl import load_workbook
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
@@ -96,12 +92,6 @@ class ExcelParser:
Returns:
ExcelData or ExcelSheet: Parsed Excel data
"""
if load_workbook is None:
raise ProcessingError(
"openpyxl is required to parse Excel files. "
"Install it with: pip install 'semantica[documents]'"
)
file_path = Path(file_path)
# Track Excel parsing
+1 -10
View File
@@ -33,10 +33,7 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from urllib.parse import urljoin
try:
from bs4 import BeautifulSoup
except (ImportError, OSError):
BeautifulSoup = None
from bs4 import BeautifulSoup
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
@@ -116,12 +113,6 @@ class HTMLParser:
Returns:
HTMLData: Parsed HTML data
"""
if BeautifulSoup is None:
raise ProcessingError(
"beautifulsoup4 is required to parse HTML files. "
"Install it with: pip install 'semantica[documents]'"
)
# Track HTML parsing
file_path = None
if isinstance(html_content, Path) or (
+26 -93
View File
@@ -123,6 +123,7 @@ Example Usage:
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional, Union
from ..utils.exceptions import ConfigurationError, ProcessingError
from ..utils.logging import get_logger
from ..utils.custom_methods import CUSTOM_METHOD_FELL_BACK, call_custom_method
from .code_parser import CodeParser
@@ -177,16 +178,10 @@ def parse_document(
>>> text = parse_document("document.pdf", method="default", extract_text=True)
"""
custom_method = method_registry.get("document", method)
if custom_method and custom_method != parse_document:
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
file_path,
file_type,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, file_path, file_type, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -253,8 +248,7 @@ def parse_document_docling(
# Register Docling method
try:
from . import docling_parser # noqa: F401
from .docling_parser import DoclingParser
method_registry.register("document", "docling", parse_document_docling)
except (ImportError, OSError):
# Docling not available, skip registration
@@ -295,17 +289,10 @@ def parse_web_content(
>>> html = parse_web_content("page.html", content_type="html", method="default")
"""
custom_method = method_registry.get("web", method)
if custom_method and custom_method != parse_web_content:
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
content,
content_type,
base_url,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, content, content_type, base_url, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -358,16 +345,10 @@ def parse_structured_data(
>>> csv_data = parse_structured_data("data.csv", data_format="csv", method="default")
"""
custom_method = method_registry.get("structured", method)
if custom_method and custom_method != parse_structured_data:
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
data,
data_format,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, data, data_format, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -411,15 +392,10 @@ def parse_email(
>>> headers = parse_email("email.eml", method="headers")
"""
custom_method = method_registry.get("email", method)
if custom_method and custom_method != parse_email:
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
email_content,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, email_content, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -466,16 +442,10 @@ def parse_code(
>>> structure = parse_code("script.py", method="ast")
"""
custom_method = method_registry.get("code", method)
if custom_method and custom_method != parse_code:
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
file_path,
language,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, file_path, language, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -525,16 +495,10 @@ def parse_media(
>>> video = parse_media("video.mp4", method="default")
"""
custom_method = method_registry.get("media", method)
if custom_method and custom_method != parse_media:
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
file_path,
media_type,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, file_path, media_type, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -577,15 +541,10 @@ def parse_pdf(
>>> pages = parse_pdf("document.pdf", method="default", pages=[1, 2, 3])
"""
custom_method = method_registry.get("document", method)
if custom_method and custom_method not in (parse_pdf, parse_document):
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
file_path,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -626,15 +585,10 @@ def parse_docx(
>>> docx = parse_docx("document.docx", method="default")
"""
custom_method = method_registry.get("document", method)
if custom_method and custom_method not in (parse_docx, parse_document):
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
file_path,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -674,15 +628,10 @@ def parse_json(file_path: Union[str, Path], method: str = "default", **kwargs) -
>>> flattened = parse_json("data.json", method="default", flatten=True)
"""
custom_method = method_registry.get("structured", method)
if custom_method and custom_method not in (parse_json, parse_structured_data):
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
file_path,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -726,16 +675,10 @@ def parse_csv(
>>> tab_separated = parse_csv("data.tsv", delimiter="\t", method="default")
"""
custom_method = method_registry.get("structured", method)
if custom_method and custom_method not in (parse_csv, parse_structured_data):
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
file_path,
delimiter,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, file_path, delimiter, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -771,15 +714,10 @@ def parse_xml(file_path: Union[str, Path], method: str = "default", **kwargs) ->
>>> xml_data = parse_xml("data.xml", method="default")
"""
custom_method = method_registry.get("structured", method)
if custom_method and custom_method not in (parse_xml, parse_structured_data):
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
file_path,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
@@ -824,15 +762,10 @@ def parse_image(
>>> ocr_text = image.get("ocr_result", {}).get("text", "")
"""
custom_method = method_registry.get("media", method)
if custom_method and custom_method not in (parse_image, parse_media):
if custom_method:
fallback = kwargs.pop("fallback_on_custom_error", False)
result = call_custom_method(
logger,
method,
custom_method,
file_path,
fallback_on_custom_error=fallback,
**kwargs,
logger, method, custom_method, file_path, fallback_on_custom_error=fallback, **kwargs
)
if result is not CUSTOM_METHOD_FELL_BACK:
return result
+1 -16
View File
@@ -33,10 +33,7 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from urllib.parse import urljoin, urlparse
try:
from bs4 import BeautifulSoup
except (ImportError, OSError):
BeautifulSoup = None
from bs4 import BeautifulSoup
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
@@ -190,12 +187,6 @@ class HTMLContentParser(HTMLParser):
}
# Load HTML for structure extraction
if BeautifulSoup is None:
raise ProcessingError(
"beautifulsoup4 is required for HTML structure extraction. "
"Install it with: pip install 'semantica[documents]'"
)
if isinstance(html_content, Path) or (
isinstance(html_content, str) and Path(html_content).exists()
):
@@ -252,12 +243,6 @@ class HTMLContentParser(HTMLParser):
else:
html_string = html_content
if BeautifulSoup is None:
raise ProcessingError(
"beautifulsoup4 is required for HTML cleaning. "
"Install it with: pip install 'semantica[documents]'"
)
soup = BeautifulSoup(html_string, "html.parser")
# Remove scripts and styles
+10 -41
View File
@@ -34,10 +34,7 @@ from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
try:
from lxml import etree
except (ImportError, OSError):
etree = None
from lxml import etree
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
@@ -108,13 +105,7 @@ class XMLParser:
)
try:
explicit_engine = options.get("engine") or self.config.get("engine")
engine = explicit_engine or ("lxml" if etree is not None else "etree")
if engine == "lxml" and etree is None:
raise ProcessingError(
"lxml is required to parse XML with engine='lxml'. "
"Install it with: pip install 'semantica[documents]'"
)
engine = options.get("engine", "lxml")
# Load XML content
if file_path_obj:
@@ -156,7 +147,7 @@ class XMLParser:
self, xml_string: str, source: str, options: Dict[str, Any]
) -> XMLData:
"""Parse XML using lxml."""
parser = etree.XMLParser(remove_blank_text=True, remove_comments=True)
parser = etree.XMLParser(remove_blank_text=True)
root = etree.fromstring(xml_string.encode("utf-8"), parser)
# Extract namespaces
@@ -197,10 +188,8 @@ class XMLParser:
metadata={"source": source, "engine": "etree"},
)
def _element_to_xml_element(self, element) -> Optional[XMLElement]:
def _element_to_xml_element(self, element) -> XMLElement:
"""Convert lxml element to XMLElement."""
if not hasattr(element, "tag") or not isinstance(element.tag, str):
return None
tag = element.tag
if "}" in tag:
namespace, tag = tag.split("}", 1)
@@ -217,18 +206,12 @@ class XMLParser:
# Process children
for child in element:
child_elem = self._element_to_xml_element(child)
if child_elem is not None:
xml_elem.children.append(child_elem)
xml_elem.children.append(self._element_to_xml_element(child))
return xml_elem
def _etree_element_to_xml_element(
self, element: ET.Element
) -> Optional[XMLElement]:
def _etree_element_to_xml_element(self, element: ET.Element) -> XMLElement:
"""Convert ElementTree element to XMLElement."""
if not hasattr(element, "tag") or not isinstance(element.tag, str):
return None
tag = element.tag
if "}" in tag:
namespace, tag = tag.split("}", 1)
@@ -245,9 +228,7 @@ class XMLParser:
# Process children
for child in element:
child_elem = self._etree_element_to_xml_element(child)
if child_elem is not None:
xml_elem.children.append(child_elem)
xml_elem.children.append(self._etree_element_to_xml_element(child))
return xml_elem
@@ -268,30 +249,18 @@ class XMLParser:
xml_data = self.parse(file_path, **options)
# Use lxml for XPath queries
if etree is None:
raise ProcessingError(
"lxml is required for find_elements (XPath queries). "
"Install it with: pip install 'semantica[documents]'"
)
xml_string = (
file_path
if isinstance(file_path, str) and not Path(file_path).exists()
else Path(file_path).read_text(encoding="utf-8")
else Path(file_path).read_text()
)
parser = etree.XMLParser(remove_blank_text=True, remove_comments=True)
root = etree.fromstring(xml_string.encode("utf-8"), parser=parser)
root = etree.fromstring(xml_string.encode("utf-8"))
# Register namespaces for XPath
namespaces = xml_data.namespaces
elements = root.xpath(xpath, namespaces=namespaces)
results = []
for elem in elements:
xml_elem = self._element_to_xml_element(elem)
if xml_elem is not None:
results.append(xml_elem)
return results
return [self._element_to_xml_element(elem) for elem in elements]
def extract_by_tag(
self, file_path: Union[str, Path], tag_name: str, **options
-5
View File
@@ -209,11 +209,6 @@ def load_spacy_model(name: str):
Raises whatever ``spacy.load`` raises (``OSError`` for a missing model), so
callers keep their existing fallback behavior.
"""
if spacy is None:
raise ImportError(
"spaCy is not installed. Install with: pip install 'semantica[nlp-spacy]'"
)
cached = _spacy_model_cache.get(name)
if cached is not None and cached[0] is spacy:
return cached[1]
File diff suppressed because it is too large Load Diff
+13 -21
View File
@@ -38,15 +38,15 @@ Example Usage:
>>> from semantica.utils import clean_text, normalize_entities
>>> cleaned = clean_text(" Hello World ")
>>> entities = normalize_entities([{"id": "e1", "text": "John", "type": "PERSON"}])
>>>
>>>
>>> from semantica.utils import hash_data, safe_filename
>>> data_hash = hash_data({"key": "value"})
>>> safe_name = safe_filename("my file.txt")
>>>
>>>
>>> from semantica.utils import merge_dicts, get_nested_value
>>> merged = merge_dicts({"a": 1}, {"b": 2}, deep=True)
>>> value = get_nested_value(config, "database.host", default="localhost")
>>>
>>>
>>> from semantica.utils import retry_on_error
>>> @retry_on_error(max_retries=3, delay=1.0)
... def fetch_data():
@@ -457,9 +457,7 @@ def chunk_list(items: List[Any], chunk_size: int) -> List[List[Any]]:
Returns:
List of chunks
"""
return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]
return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]
def flatten_dict(
d: Dict[str, Any], parent_key: str = "", sep: str = "."
) -> Dict[str, Any]:
@@ -558,27 +556,27 @@ def safe_import(
) -> Tuple[Any, bool]:
"""
Safely import an optional module, handling both ImportError and OSError.
This is useful for optional dependencies that may fail to import due to:
- Missing package (ImportError)
- DLL loading failures on Windows, e.g., PyTorch (OSError)
Args:
module_name: Name of the module to import (e.g., "spacy", "docling.document_converter")
package: Optional package name for relative imports
default: Default value to return if import fails
error_message: Optional custom error message for logging
Returns:
Tuple of (module_or_default, success_flag):
- If import succeeds: (imported_module, True)
- If import fails: (default, False)
Example:
>>> spacy, available = safe_import("spacy")
>>> if available:
... doc = spacy.load("en_core_web_sm")
>>>
>>>
>>> converter, available = safe_import("docling.document_converter", default=None)
>>> if available:
... converter = converter()
@@ -589,13 +587,11 @@ def safe_import(
else:
module = importlib.import_module(module_name)
return module, True
except (ImportError, OSError) as e:
except (ImportError, ModuleNotFoundError, OSError) as e:
if error_message:
import sys
if "logging" in sys.modules:
from .logging import get_logger
logger = get_logger("utils.helpers")
logger.debug(f"{error_message}: {e}")
return default, False
@@ -811,13 +807,9 @@ def _is_record(value: Any) -> bool:
exporters rather than a ``ValidationError`` at the boundary where the
problem is visible.
"""
return (
isinstance(value, Mapping)
or is_dataclass(value)
or (
hasattr(value, "__dict__")
and not isinstance(value, (types.ModuleType, type))
)
return isinstance(value, Mapping) or is_dataclass(value) or (
hasattr(value, "__dict__")
and not isinstance(value, (types.ModuleType, type))
)
+4 -8
View File
@@ -343,9 +343,7 @@ class FAISSIndex:
was originally saved.
"""
if not FAISS_AVAILABLE:
raise ProcessingError(
"FAISS not available. Install with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
)
raise ProcessingError("FAISS not available")
path = Path(path)
index = faiss.read_index(str(path))
@@ -484,7 +482,7 @@ class FAISSIndexBuilder:
"""
if not FAISS_AVAILABLE:
raise ProcessingError(
"FAISS is not available. Install it with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
"FAISS is not available. Install it with: pip install faiss-cpu or faiss-gpu"
)
# Create index based on type
@@ -556,7 +554,7 @@ class FAISSStore:
# Check FAISS availability
if not FAISS_AVAILABLE:
self.logger.warning(
"FAISS not available. Install with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
"FAISS not available. Install with: pip install faiss-cpu or faiss-gpu"
)
def create_index(
@@ -755,9 +753,7 @@ class FAISSStore:
FAISSIndex instance
"""
if not FAISS_AVAILABLE:
raise ProcessingError(
"FAISS not available. Install with: pip install 'semantica[vectorstore-faiss]' (or 'semantica[gpu]' for CUDA)"
)
raise ProcessingError("FAISS not available")
path = Path(path)
if path.exists() and not _metadata_path(path).exists():
@@ -85,7 +85,7 @@ class AnalyticsVisualizer:
if px is None or go is None:
raise ProcessingError(
"Plotly is required for analytics visualization. "
"Install with: pip install 'semantica[viz]'"
"Install with: pip install plotly"
)
if np is None:
raise ProcessingError(
+40 -90
View File
@@ -1,10 +1,9 @@
"""
Embedding Visualizer Module
This module provides comprehensive visualization capabilities for vector
embeddings in the Semantica framework, including 2D/3D dimensionality
reduction projections, similarity heatmaps, clustering visualizations,
multi-modal comparisons, and quality metrics analysis.
This module provides comprehensive visualization capabilities for vector embeddings in the
Semantica framework, including 2D/3D dimensionality reduction projections, similarity heatmaps,
clustering visualizations, multi-modal comparisons, and quality metrics analysis.
Key Features:
- 2D and 3D dimensionality reduction (UMAP, t-SNE, PCA)
@@ -32,8 +31,9 @@ License: MIT
"""
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from typing import Any, Dict, List, Optional, Tuple, Union
import matplotlib.pyplot as plt
import numpy as np
try:
@@ -45,12 +45,8 @@ except (ImportError, OSError):
go = None
make_subplots = None
try:
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
except (ImportError, OSError):
PCA = None
TSNE = None
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
try:
import umap
@@ -61,7 +57,7 @@ from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from .utils.color_schemes import ColorPalette, ColorScheme
from .utils.export_formats import export_plotly_figure
from .utils.export_formats import export_matplotlib_figure, export_plotly_figure
class EmbeddingVisualizer:
@@ -98,17 +94,12 @@ class EmbeddingVisualizer:
self.color_scheme = ColorScheme.DEFAULT
self.point_size = config.get("point_size", 5)
def _check_dependencies(self, require_sklearn: bool = False):
def _check_dependencies(self):
"""Check if dependencies are available."""
if px is None or go is None:
raise ProcessingError(
"Plotly is required for embedding visualization. "
"Install with: pip install 'semantica[viz]'"
)
if require_sklearn and (PCA is None or TSNE is None):
raise ProcessingError(
"scikit-learn is required for dimensionality reduction. "
"Reinstall scikit-learn or install dependencies."
"Install with: pip install plotly"
)
def visualize_2d_projection(
@@ -159,12 +150,10 @@ class EmbeddingVisualizer:
try:
self.logger.info(f"Visualizing 2D projection using {method}")
# Step 2: Data Analysis
n_samples, n_features = embeddings.shape
self.logger.info(
f"Embedding Analysis: {n_samples} samples, {n_features} dimensions"
)
self.logger.info(f"Embedding Analysis: {n_samples} samples, {n_features} dimensions")
if embeddings.shape[1] <= 2:
# Already 2D or less, use directly
@@ -176,32 +165,28 @@ class EmbeddingVisualizer:
self.progress_tracker.update_tracking(
tracking_id, message=f"Reducing dimensions using {method}..."
)
dim_options = dict(options)
n_comp = dim_options.pop("n_components", 2)
projected = self._reduce_dimensions(
embeddings, method=method, n_components=n_comp, **dim_options
embeddings, method=method, n_components=2, **options
)
self.progress_tracker.update_tracking(
tracking_id, message="Generating visualization..."
)
result = self._visualize_2d_plotly(
projected,
labels,
output,
file_path,
projected,
labels,
output,
file_path,
color_by=color_by,
size_by=size_by,
hover_data=hover_data,
**options,
**options
)
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=(
f"2D projection visualization generated: {len(projected)} points"
),
message=f"2D projection visualization generated: {len(projected)} points",
)
return result
except Exception as e:
@@ -252,10 +237,8 @@ class EmbeddingVisualizer:
self.progress_tracker.update_tracking(
tracking_id, message=f"Reducing dimensions using {method}..."
)
dim_options = dict(options)
n_comp = dim_options.pop("n_components", 3)
projected = self._reduce_dimensions(
embeddings, method=method, n_components=n_comp, **dim_options
embeddings, method=method, n_components=3, **options
)
self.progress_tracker.update_tracking(
@@ -268,9 +251,7 @@ class EmbeddingVisualizer:
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=(
f"3D projection visualization generated: {len(projected)} points"
),
message=f"3D projection visualization generated: {len(projected)} points",
)
return result
except Exception as e:
@@ -361,10 +342,7 @@ class EmbeddingVisualizer:
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=(
f"Similarity heatmap generated: "
f"{len(embeddings)}x{len(embeddings)} matrix"
),
message=f"Similarity heatmap generated: {len(embeddings)}x{len(embeddings)} matrix",
)
return fig
elif file_path:
@@ -426,10 +404,8 @@ class EmbeddingVisualizer:
self.progress_tracker.update_tracking(
tracking_id, message=f"Reducing dimensions using {method}..."
)
dim_options = dict(options)
n_comp = dim_options.pop("n_components", 2)
projected = self._reduce_dimensions(
embeddings, method=method, n_components=n_comp, **dim_options
embeddings, method=method, n_components=2, **options
)
num_clusters = len(set(cluster_labels))
@@ -469,10 +445,7 @@ class EmbeddingVisualizer:
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=(
f"Clustering visualization generated: "
f"{num_clusters} clusters, {len(embeddings)} points"
),
message=f"Clustering visualization generated: {num_clusters} clusters, {len(embeddings)} points",
)
return fig
elif file_path:
@@ -568,13 +541,8 @@ class EmbeddingVisualizer:
self.progress_tracker.update_tracking(
tracking_id, message=f"Reducing dimensions using {method}..."
)
dim_options = dict(options)
n_comp = dim_options.pop("n_components", 2)
projected = self._reduce_dimensions(
combined_embeddings,
method=method,
n_components=n_comp,
**dim_options,
combined_embeddings, method=method, n_components=2, **options
)
# Color by type
@@ -611,10 +579,7 @@ class EmbeddingVisualizer:
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=(
f"Multi-modal comparison generated: "
f"{len(combined_embeddings)} embeddings"
),
message=f"Multi-modal comparison generated: {len(combined_embeddings)} embeddings",
)
return fig
elif file_path:
@@ -633,6 +598,8 @@ class EmbeddingVisualizer:
)
raise
def _reduce_dimensions(
self,
embeddings: np.ndarray,
@@ -641,56 +608,39 @@ class EmbeddingVisualizer:
**options,
) -> np.ndarray:
"""Reduce embedding dimensions using specified method."""
opts = dict(options)
opts.pop("n_components", None)
if method == "pca":
if PCA is None:
raise ProcessingError(
"scikit-learn is required for dimensionality reduction. "
"Reinstall scikit-learn or install dependencies."
)
pca = PCA(n_components=n_components, **opts)
pca = PCA(n_components=n_components, **options)
return pca.fit_transform(embeddings)
elif method == "tsne":
if TSNE is None:
raise ProcessingError(
"scikit-learn is required for dimensionality reduction. "
"Reinstall scikit-learn or install dependencies."
)
perplexity = opts.pop("perplexity", min(30, len(embeddings) - 1))
random_state = opts.pop("random_state", 42)
perplexity = options.get("perplexity", min(30, len(embeddings) - 1))
tsne = TSNE(
n_components=n_components,
perplexity=perplexity,
random_state=random_state,
**opts,
random_state=42,
**options,
)
return tsne.fit_transform(embeddings)
elif method == "umap":
if umap is not None:
n_neighbors = opts.pop("n_neighbors", min(15, len(embeddings) - 1))
n_neighbors = options.get("n_neighbors", min(15, len(embeddings) - 1))
reducer = umap.UMAP(
n_components=n_components, n_neighbors=n_neighbors, **opts
n_components=n_components, n_neighbors=n_neighbors, **options
)
return reducer.fit_transform(embeddings)
else:
raise ProcessingError(
"UMAP is required for UMAP dimensionality reduction. "
"Install with: pip install 'semantica[viz]'"
# Fallback to PCA if UMAP not available
self.logger.warning(
"UMAP not available, using PCA. Install with: pip install umap-learn"
)
pca = PCA(n_components=n_components)
return pca.fit_transform(embeddings)
else:
if PCA is None:
raise ProcessingError(
"scikit-learn is required for dimensionality reduction. "
"Reinstall scikit-learn or install dependencies."
)
# Fallback to PCA
self.logger.warning(f"Method {method} not available, using PCA")
pca = PCA(n_components=n_components, **opts)
pca = PCA(n_components=n_components)
return pca.fit_transform(embeddings)
def _visualize_2d_plotly(
+1 -1
View File
@@ -124,7 +124,7 @@ class KGVisualizer:
if px is None or go is None:
raise ProcessingError(
"Plotly is required for KG visualization. "
"Install with: pip install 'semantica[viz]'"
"Install with: pip install plotly"
)
def _convert_knowledge_graph(self, kg: Any) -> Dict[str, Any]:
+8 -12
View File
@@ -35,14 +35,8 @@ License: MIT
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
try:
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
from matplotlib.patches import FancyBboxPatch
except (ImportError, OSError):
mpatches = None
plt = None
FancyBboxPatch = None
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
try:
import plotly.express as px
@@ -53,6 +47,8 @@ except (ImportError, OSError):
go = None
make_subplots = None
from matplotlib.patches import FancyBboxPatch
try:
import graphviz
except (ImportError, OSError):
@@ -110,13 +106,13 @@ class OntologyVisualizer:
if graphviz is None:
raise ProcessingError(
"Graphviz is required for DOT export. "
"Install with: pip install 'semantica[viz]'"
"Install with: pip install graphviz"
)
else:
if go is None:
if px is None or go is None:
raise ProcessingError(
"Plotly is required for ontology visualization. "
"Install with: pip install 'semantica[viz]'"
"Install with: pip install plotly"
)
def visualize_hierarchy(
@@ -896,7 +892,7 @@ class OntologyVisualizer:
"""Create Graphviz hierarchy visualization."""
if graphviz is None:
raise ProcessingError(
"Graphviz not available. Install with: pip install 'semantica[viz]'"
"Graphviz not available. Install with: pip install graphviz"
)
dot = graphviz.Digraph(comment="Ontology Hierarchy")
@@ -74,7 +74,7 @@ class SemanticNetworkVisualizer:
if px is None or go is None:
raise ProcessingError(
"Plotly is required for semantic network visualization. "
"Install with: pip install 'semantica[viz]'"
"Install with: pip install plotly"
)
def visualize_network(
@@ -83,7 +83,7 @@ class TemporalVisualizer:
if px is None or go is None:
raise ProcessingError(
"Plotly is required for temporal visualization. "
"Install with: pip install 'semantica[viz]'"
"Install with: pip install plotly"
)
def visualize_temporal_dashboard(
+24 -5
View File
@@ -51,6 +51,27 @@ _INTERNAL_ERROR = -32603
_TOOL_INDEX: dict[str, dict] = {t["name"]: t for t in TOOL_DEFINITIONS}
class UnknownToolError(Exception):
"""Raised by :func:`call_tool` when the tool name is not in the catalog.
A dedicated type (rather than ``KeyError``) so callers can distinguish
a bad tool name from a ``KeyError`` raised inside a handler indexing a
required argument (e.g. ``args["category"]``).
"""
def call_tool(name: str, arguments: dict) -> dict:
"""Invoke a tool in-process by name and return its raw result dict.
Shared by the JSON-RPC ``tools/call`` handler and ``semantica mcp call``
(issue #1355), so both expose exactly the same tool set.
"""
tool = _TOOL_INDEX.get(name)
if tool is None:
raise UnknownToolError(f"Unknown tool: {name}")
return tool["_handler"](arguments)
# ---------------------------------------------------------------------------
# Request handlers
# ---------------------------------------------------------------------------
@@ -85,12 +106,10 @@ def _handle_tools_call(req_id: Any, params: dict) -> dict:
name = params.get("name", "")
args = params.get("arguments", {}) or {}
tool = _TOOL_INDEX.get(name)
if tool is None:
return _err(req_id, _METHOD_NOT_FOUND, f"Unknown tool: {name}")
try:
result = tool["_handler"](args)
result = call_tool(name, args)
except UnknownToolError as exc:
return _err(req_id, _METHOD_NOT_FOUND, str(exc))
except Exception as exc:
log.exception("Tool %s raised an exception", name)
# The exception's class name (e.g. "ValidationError", "TimeoutError")
-131
View File
@@ -141,134 +141,3 @@ else:
assert "ConfigurationError" in result.stdout
assert "Parquet ingestion" in result.stdout
assert "pyarrow" in result.stdout
def test_repo_ingestor_probe_fails_without_gitpython() -> None:
result = _run_python_with_blocked_modules(
"""
try:
from semantica.ingest import RepoIngestor
has_git = True
except ImportError:
has_git = False
assert not has_git, "Expected RepoIngestor import to fail without GitPython"
print("RepoIngestor probe passed")
""",
("git",),
)
assert result.returncode == 0, result.stderr
assert "RepoIngestor probe passed" in result.stdout
def test_xml_ingestor_probe_fails_without_lxml() -> None:
result = _run_python_with_blocked_modules(
"""
try:
from semantica.ingest import XMLIngestor
has_lxml = True
except ImportError:
has_lxml = False
assert not has_lxml, "Expected XMLIngestor import to fail without lxml"
print("XMLIngestor probe passed")
""",
("lxml",),
)
assert result.returncode == 0, result.stderr
assert "XMLIngestor probe passed" in result.stdout
def test_xml_ingestion_reports_missing_lxml_when_used() -> None:
result = _run_python_with_blocked_modules(
"""
from semantica.ingest import ingest_xml
try:
ingest_xml("catalog.xml")
except Exception as exc:
print(type(exc).__name__, exc)
else:
raise SystemExit("expected XML ingestion to fail without lxml")
""",
("lxml",),
)
assert result.returncode == 0, result.stderr
assert "ConfigurationError" in result.stdout
assert "XML ingestion" in result.stdout
assert "lxml" in result.stdout
def test_sibling_imports_succeed_without_optional_backends() -> None:
result = _run_python_with_blocked_modules(
"""
from semantica.ingest import (
CodeExtractor,
CodeFile,
CommitInfo,
GitAnalyzer,
XMLIngestionData,
SalesforceData,
)
print(
CodeExtractor.__name__,
CodeFile.__name__,
CommitInfo.__name__,
GitAnalyzer.__name__,
XMLIngestionData.__name__,
SalesforceData.__name__,
)
""",
("git", "lxml", "simple_salesforce"),
)
assert result.returncode == 0, result.stderr
assert (
"CodeExtractor CodeFile CommitInfo GitAnalyzer XMLIngestionData SalesforceData"
in result.stdout
)
def test_salesforce_ingestor_probe_fails_without_simple_salesforce() -> None:
result = _run_python_with_blocked_modules(
"""
try:
from semantica.ingest import SalesforceIngestor
has_salesforce = True
except ImportError:
has_salesforce = False
assert not has_salesforce, (
"Expected SalesforceIngestor import to fail without simple-salesforce"
)
print("SalesforceIngestor probe passed")
""",
("simple_salesforce",),
)
assert result.returncode == 0, result.stderr
assert "SalesforceIngestor probe passed" in result.stdout
def test_salesforce_ingestion_reports_missing_dep_when_used() -> None:
result = _run_python_with_blocked_modules(
"""
from semantica.ingest import ingest_salesforce
try:
ingest_salesforce()
except Exception as exc:
print(type(exc).__name__, exc)
else:
raise SystemExit("expected Salesforce ingestion to fail without simple-salesforce")
""",
("simple_salesforce",),
)
assert result.returncode == 0, result.stderr
assert "ConfigurationError" in result.stdout
assert "Salesforce ingestion" in result.stdout
assert "simple-salesforce" in result.stdout
@@ -18,40 +18,29 @@ from datetime import datetime, timezone
from unittest.mock import MagicMock, patch
# ── Mock optional heavyweight dependencies before any semantica import ──────
_MOCKED_MODULES = [
"spacy",
"instructor",
"openai",
"groq",
"sentence_transformers",
"transformers",
]
_original_modules = {k: sys.modules.get(k) for k in _MOCKED_MODULES}
for k in _MOCKED_MODULES:
sys.modules.setdefault(k, MagicMock())
sys.modules.setdefault("spacy", MagicMock())
sys.modules.setdefault("instructor", MagicMock())
_openai_mock = MagicMock()
sys.modules.setdefault("openai", _openai_mock)
sys.modules.setdefault("groq", MagicMock())
sys.modules.setdefault("sentence_transformers", MagicMock())
sys.modules.setdefault("transformers", MagicMock())
sys.modules.setdefault("torch", MagicMock())
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
from semantica.semantic_extract.methods import extract_relations_llm # noqa: E402
for _key, _original in _original_modules.items():
if _original is None:
sys.modules.pop(_key, None)
else:
sys.modules[_key] = _original
from semantica.semantic_extract.ner_extractor import Entity # noqa: E402
from semantica.semantic_extract.schemas import ( # noqa: E402
from semantica.semantic_extract.methods import extract_relations_llm
from semantica.semantic_extract.ner_extractor import Entity
from semantica.semantic_extract.schemas import (
RelationsResponse,
RelationsWithTemporalResponse,
)
from semantica.kg.temporal_normalizer import TemporalNormalizer # noqa: E402
from semantica.utils.exceptions import TemporalAmbiguityWarning # noqa: E402
from semantica.kg.temporal_normalizer import TemporalNormalizer
from semantica.utils.exceptions import TemporalAmbiguityWarning
# ── Helpers ─────────────────────────────────────────────────────────────────
def _make_entities():
return [
Entity(text="Apple", label="ORG", start_char=0, end_char=5),
@@ -67,17 +56,15 @@ def _ref_date():
# Part 1 extract_relations_llm() temporal flag
# ============================================================================
class TestTemporalExtractionFlag(unittest.TestCase):
def setUp(self):
from semantica.semantic_extract.methods import _result_cache
_result_cache.clear()
@patch("semantica.semantic_extract.methods.create_provider")
def test_extract_temporal_bounds_true_adds_four_fields(self, mock_create):
"""With extract_temporal_bounds=True all four temporal keys appear."""
"""With extract_temporal_bounds=True all four temporal keys appear in metadata."""
mock_prov = MagicMock()
mock_prov.is_available.return_value = True
mock_prov.generate_typed.return_value = RelationsWithTemporalResponse(
@@ -145,9 +132,7 @@ class TestTemporalExtractionFlag(unittest.TestCase):
self.assertNotIn("temporal_source_text", meta)
@patch("semantica.semantic_extract.methods.create_provider")
def test_no_temporal_signal_returns_zero_confidence_and_null_dates(
self, mock_create
):
def test_no_temporal_signal_returns_zero_confidence_and_null_dates(self, mock_create):
"""When LLM returns no temporal signal, confidence=0.0 and dates are null."""
mock_prov = MagicMock()
mock_prov.is_available.return_value = True
@@ -212,12 +197,10 @@ class TestTemporalExtractionFlag(unittest.TestCase):
@patch("semantica.semantic_extract.methods.create_provider")
def test_correct_schema_used_when_temporal_true(self, mock_create):
"""generate_typed is called with RelationsWithTemporalResponse."""
"""generate_typed is called with RelationsWithTemporalResponse when flag=True."""
mock_prov = MagicMock()
mock_prov.is_available.return_value = True
mock_prov.generate_typed.return_value = RelationsWithTemporalResponse(
relations=[]
)
mock_prov.generate_typed.return_value = RelationsWithTemporalResponse(relations=[])
mock_create.return_value = mock_prov
extract_relations_llm(
@@ -252,7 +235,6 @@ class TestTemporalExtractionFlag(unittest.TestCase):
# Part 2 TemporalNormalizer: relative dates
# ============================================================================
class TestTemporalNormalizerRelativeDates(unittest.TestCase):
def setUp(self):
@@ -331,13 +313,10 @@ class TestTemporalNormalizerRelativeDates(unittest.TestCase):
# Part 3 TemporalNormalizer: partial / structured dates
# ============================================================================
class TestTemporalNormalizerPartialDates(unittest.TestCase):
def setUp(self):
self.tn = TemporalNormalizer(
reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc)
)
self.tn = TemporalNormalizer(reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc))
def test_year_only(self):
result = self.tn.normalize("2021")
@@ -417,13 +396,10 @@ class TestTemporalNormalizerPartialDates(unittest.TestCase):
# Part 4 TemporalNormalizer: ambiguous formats
# ============================================================================
class TestTemporalNormalizerAmbiguity(unittest.TestCase):
def setUp(self):
self.tn = TemporalNormalizer(
reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc)
)
self.tn = TemporalNormalizer(reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc))
def test_ambiguous_slash_date_raises_warning_and_returns_none(self):
with warnings.catch_warnings(record=True) as w:
@@ -456,19 +432,14 @@ class TestTemporalNormalizerAmbiguity(unittest.TestCase):
# Part 5 TemporalNormalizer: domain phrase map
# ============================================================================
class TestTemporalNormalizerDomainPhrases(unittest.TestCase):
def setUp(self):
self.tn = TemporalNormalizer(
reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc)
)
self.tn = TemporalNormalizer(reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc))
def _assert_recognized(self, phrase):
result = self.tn.normalize_phrase(phrase)
self.assertIsNotNone(
result, f"Expected phrase {phrase!r} to be recognized but got None"
)
self.assertIsNotNone(result, f"Expected phrase {phrase!r} to be recognized but got None")
return result
# General / Policy
@@ -551,7 +522,6 @@ class TestTemporalNormalizerDomainPhrases(unittest.TestCase):
# Part 6 TemporalNormalizer: custom phrase map
# ============================================================================
class TestTemporalNormalizerCustomPhraseMap(unittest.TestCase):
def setUp(self):
@@ -595,12 +565,10 @@ class TestTemporalNormalizerCustomPhraseMap(unittest.TestCase):
# Part 7 Full pipeline: extract → normalize → BiTemporalFact
# ============================================================================
class TestFullPipelineTemporalToBiTemporal(unittest.TestCase):
def setUp(self):
from semantica.semantic_extract.methods import _result_cache
_result_cache.clear()
@patch("semantica.semantic_extract.methods.create_provider")
@@ -652,12 +620,10 @@ class TestFullPipelineTemporalToBiTemporal(unittest.TestCase):
self.assertEqual(vf[0].day, 1)
# Feed into BiTemporalFact
fact = BiTemporalFact.from_relationship(
{
"valid_from": "2014-05-01T00:00:00Z",
"valid_until": None,
}
)
fact = BiTemporalFact.from_relationship({
"valid_from": "2014-05-01T00:00:00Z",
"valid_until": None,
})
self.assertIsNotNone(fact.valid_from)
self.assertEqual(fact.valid_from.year, 2014)
self.assertEqual(fact.valid_from.month, 5)
@@ -694,9 +660,7 @@ class TestFullPipelineTemporalToBiTemporal(unittest.TestCase):
extract_temporal_bounds=True,
)
meta = rels[0].metadata
tn = TemporalNormalizer(
reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc)
)
tn = TemporalNormalizer(reference_date=datetime(2025, 3, 25, tzinfo=timezone.utc))
vf = tn.normalize(meta["valid_from"])
vu = tn.normalize(meta["valid_until"])
+40 -11
View File
@@ -2090,12 +2090,16 @@ class TestMCP:
# Table renders correctly — at minimum the column header is present
assert "Tool" in result.output or "tool" in result.output.lower()
def test_list_tools_with_mock_shows_known_tools(self, runner, monkeypatch):
fake_tools = _fake_module(__all__=["extract_entities", "query_graph"])
monkeypatch.setitem(__import__("sys").modules, "semantica_mcp.mcp.tools", fake_tools)
def test_list_tools_reads_server_catalog(self, runner, monkeypatch):
"""list-tools must read TOOL_DEFINITIONS (what the server serves via
tools/list), not the module's ``__all__`` (issue #1355)."""
import semantica_mcp.mcp.tools as tools_mod
fake = [{"name": "fake_tool_from_catalog", "description": "", "inputSchema": {},
"_handler": lambda a: {}}]
monkeypatch.setattr(tools_mod, "TOOL_DEFINITIONS", fake)
result = runner.invoke(cli_module.main, ["mcp", "list-tools"])
_ok(result)
assert "extract_entities" in result.output
assert "fake_tool_from_catalog" in result.output
def test_list_tools_json(self, runner):
result = runner.invoke(cli_module.main, ["mcp", "list-tools", "--json"])
@@ -2138,14 +2142,39 @@ class TestMCP:
err = json.loads(result.stderr)
assert err["error"].startswith("Invalid JSON in --args")
def test_call_import_error_is_clean(self, runner):
with patch("builtins.__import__", side_effect=lambda n, *a, **k: (
(_ for _ in ()).throw(ImportError(n))
if n.startswith("mcp") else __import__(n, *a, **k)
)):
result = runner.invoke(cli_module.main, ["mcp", "call", "extract_entities"])
def test_call_dispatches_through_packaged_server(self, runner):
"""Regression for issue #1355: ``mcp call`` dispatches in-process through
``semantica_mcp.mcp.server`` (the server ``mcp start`` spawns) instead
of importing the nonexistent ``MCPSession``."""
result = runner.invoke(
cli_module.main, ["--json", "mcp", "call", "extract_entities"]
)
_ok(result)
# Empty args short-circuit before heavy imports; reaching the
# handler's own validation proves the dispatch path works.
assert "text is required" in result.output
def test_call_unknown_tool_fails_cleanly(self, runner):
result = runner.invoke(cli_module.main, ["mcp", "call", "no_such_tool"])
assert result.exit_code != 0
assert "Traceback" not in result.output
assert "Unknown tool" in result.output
def test_call_non_object_args_rejected(self, runner):
result = runner.invoke(
cli_module.main, ["mcp", "call", "extract_entities", "--args", "[1, 2]"]
)
assert result.exit_code != 0
assert "Traceback" not in result.output
assert "--args must be a JSON object" in result.output
def test_list_tools_json_matches_server_catalog(self, runner):
"""The CLI catalog and the MCP server catalog must be the same list."""
from semantica_mcp.mcp.tools import TOOL_DEFINITIONS
result = runner.invoke(cli_module.main, ["mcp", "list-tools", "--json"])
_ok(result)
data = _json_output(result)
assert data["tools"] == [t["name"] for t in TOOL_DEFINITIONS]
# ─── services group (backward-compat wrapper) ─────────────────────────────────
@@ -2302,7 +2331,7 @@ class TestDoctorEmbeddings:
checks = self._doctor_checks(runner)
st = checks["Embeddings (sentence-transformers)"]
assert st["status"] == "fail"
assert st["hint"] == "pip install 'semantica[embeddings-local]'"
assert st["hint"] == "pip install sentence-transformers"
def test_deep_probe_detects_fallback_active(self, runner, monkeypatch):
self._with_fake_st(monkeypatch)
-454
View File
@@ -1,454 +0,0 @@
from pathlib import Path
from unittest.mock import patch
import pytest
from semantica.parse.docx_parser import DOCXParser
from semantica.parse.excel_parser import ExcelParser
from semantica.parse.html_parser import HTMLParser
from semantica.parse.xml_parser import XMLParser
from semantica.utils.exceptions import ProcessingError
def _load_toml(file_path: Path) -> dict:
"""Load and parse a TOML file across Python 3.8-3.14+ without mode mismatches."""
content = file_path.read_text(encoding="utf-8")
try:
import tomllib # Python 3.11+ standard library
return tomllib.loads(content)
except ImportError:
try:
import tomli # Fast PEP 680 compatible parser for Python < 3.11
return tomli.loads(content)
except ImportError:
import toml # Fallback toml parser
return toml.loads(content)
def test_core_dependencies_count():
"""pyproject.toml must contain exactly 22 unique core dependencies."""
repo_root = Path(__file__).resolve().parents[1]
data = _load_toml(repo_root / "pyproject.toml")
deps = data["project"]["dependencies"]
normalized_names = {
d.split(";")[0].split(">=")[0].split("<")[0].split("==")[0].strip()
for d in deps
}
expected_22 = {
"numpy",
"pandas",
"scipy",
"scikit-learn",
"rdflib",
"networkx",
"requests",
"chardet",
"protobuf",
"grpcio",
"pillow",
"pydantic",
"click",
"rich",
"tqdm",
"pyyaml",
"toml",
"python-dotenv",
"loguru",
"structlog",
"httpx",
"pyarrow",
}
assert normalized_names == expected_22
assert len(normalized_names) == 22
def test_optional_extras_defined():
"""All required optional extras must be declared in pyproject.toml."""
repo_root = Path(__file__).resolve().parents[1]
data = _load_toml(repo_root / "pyproject.toml")
extras = data["project"]["optional-dependencies"]
for extra in [
"documents",
"ingest-git",
"embeddings-local",
"nlp-spacy",
"viz",
"media",
"vectorstore-faiss",
"graph-embeddings",
"all",
]:
assert extra in extras, f"Missing extra {extra}"
all_extra_str = str(extras["all"])
for expected_ref in [
"documents",
"ingest-git",
"embeddings-local",
"nlp-spacy",
"viz",
"media",
"graph-embeddings",
"vectorstore-all",
]:
assert expected_ref in all_extra_str, f"Missing {expected_ref} in all"
# And vectorstore-faiss is in vectorstore-all
assert "vectorstore-faiss" in str(extras["vectorstore-all"])
# Verify nlp-spacy does not declare thinc directly (Qodo bot issue 1)
nlp_spacy_deps = str(extras.get("nlp-spacy", []))
assert "thinc" not in nlp_spacy_deps, "nlp-spacy should not directly declare thinc"
assert "spacy" in nlp_spacy_deps, "nlp-spacy must declare spacy"
def test_core_modules_importable():
"""Core modules must be importable without requiring optional extras."""
import semantica
import semantica.cli
import semantica.parse
import semantica.ingest
import semantica.embeddings
import semantica.export
import semantica.kg
import semantica.vector_store
import semantica.visualization
import semantica.semantic_extract
import semantica.pipeline
assert semantica.__version__ is not None
def test_docx_parser_lazy_construction_and_parse_hint():
with patch("semantica.parse.docx_parser.Document", None):
parser = DOCXParser()
assert parser is not None
with pytest.raises(ProcessingError, match=r"semantica\[documents\]"):
parser.parse("nonexistent.docx")
def test_excel_parser_lazy_construction_and_parse_hint():
with patch("semantica.parse.excel_parser.load_workbook", None):
parser = ExcelParser()
assert parser is not None
with pytest.raises(ProcessingError, match=r"semantica\[documents\]"):
parser.parse("nonexistent.xlsx")
def test_html_parser_lazy_construction_and_parse_hint():
with patch("semantica.parse.html_parser.BeautifulSoup", None):
parser = HTMLParser()
assert parser is not None
with pytest.raises(ProcessingError, match=r"semantica\[documents\]"):
parser.parse("nonexistent.html")
def test_xml_parser_etree_fallback():
with patch("semantica.parse.xml_parser.etree", None):
parser = XMLParser()
assert parser is not None
result = parser.parse("<root><item id='1'>Test</item></root>")
assert result is not None
assert result.root is not None
assert result.root.tag == "root"
def test_xml_parser_lxml_explicit_requires_documents_extra():
with patch("semantica.parse.xml_parser.etree", None):
parser = XMLParser(engine="lxml")
assert parser is not None
with pytest.raises(ProcessingError, match=r"semantica\[documents\]"):
parser.parse("<root/>")
def test_xml_ingestor_missing_hint():
with patch("semantica.ingest.xml_ingestor.etree", None):
from semantica.ingest.xml_ingestor import XMLIngestor
with pytest.raises(ImportError, match=r"semantica\[documents\]"):
XMLIngestor()
def test_xml_ingestor_package_import_missing_hint():
import semantica.ingest as ingest_mod
ingest_mod.__dict__.pop("XMLIngestor", None)
with patch("semantica.ingest.xml_ingestor.etree", None):
with pytest.raises(ImportError, match=r"semantica\[documents\]"):
_ = ingest_mod.XMLIngestor
def test_repo_ingestor_missing_hint():
with patch("semantica.ingest.repo_ingestor.git", None):
from semantica.ingest.repo_ingestor import RepoIngestor
with pytest.raises(ImportError, match=r"semantica\[ingest-git\]"):
RepoIngestor()
def test_repo_ingestor_package_import_missing_hint():
import semantica.ingest as ingest_mod
ingest_mod.__dict__.pop("RepoIngestor", None)
with patch("semantica.ingest.repo_ingestor.git", None):
with pytest.raises(ImportError, match=r"semantica\[ingest-git\]"):
_ = ingest_mod.RepoIngestor
def test_git_analyzer_package_import_succeeds_without_git():
import semantica.ingest as ingest_mod
ingest_mod.__dict__.pop("GitAnalyzer", None)
with patch("semantica.ingest.repo_ingestor.git", None):
analyzer_cls = ingest_mod.GitAnalyzer
assert analyzer_cls is not None
analyzer = analyzer_cls()
assert analyzer is not None
def test_salesforce_ingestor_package_import_missing_hint():
import semantica.ingest as ingest_mod
ingest_mod.__dict__.pop("SalesforceIngestor", None)
with patch("semantica.ingest.salesforce_ingestor.SALESFORCE_AVAILABLE", False):
with pytest.raises(ImportError, match=r"semantica\[db-salesforce\]"):
_ = ingest_mod.SalesforceIngestor
def test_parse_methods_dynamic_default_resolution():
from semantica.parse.methods import (
get_parse_method,
list_available_methods,
parse_document,
)
assert get_parse_method("document", "default") == parse_document
methods = list_available_methods()
assert "default" in methods.get("document", [])
assert "default" in methods.get("structured", [])
def test_node_embedder_gensim_missing_hint():
with patch("semantica.kg.node_embeddings.GENSIM_AVAILABLE", False):
from semantica.kg.node_embeddings import NodeEmbedder
with pytest.raises(ImportError, match=r"semantica\[graph-embeddings\]"):
NodeEmbedder()
def test_faiss_store_missing_hint():
with patch("semantica.vector_store.faiss_store.FAISS_AVAILABLE", False):
from semantica.vector_store.faiss_store import FAISSIndexBuilder, FAISSStore
builder = FAISSIndexBuilder(128)
with pytest.raises(ProcessingError, match=r"semantica\[vectorstore-faiss\]"):
builder.build_index("flat")
store = FAISSStore(128)
with pytest.raises(ProcessingError, match=r"semantica\[vectorstore-faiss\]"):
store.load_index("nonexistent.faiss")
def test_visualization_missing_hint():
import numpy as np
from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
# Plotly is checked via px and go in _check_dependencies
with patch("semantica.visualization.embedding_visualizer.px", None):
visualizer = EmbeddingVisualizer()
with pytest.raises(
ProcessingError, match=r"Plotly is required.*semantica\[viz\]"
):
visualizer.visualize_2d_projection(np.array([[0.1, 0.2], [0.3, 0.4]]))
with patch("semantica.visualization.embedding_visualizer.go", None):
visualizer = EmbeddingVisualizer()
with pytest.raises(
ProcessingError, match=r"Plotly is required.*semantica\[viz\]"
):
visualizer.visualize_2d_projection(np.array([[0.1, 0.2], [0.3, 0.4]]))
def test_visualization_umap_missing_hint():
import numpy as np
from unittest.mock import MagicMock
from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
# Stand in for Plotly so we reach dimensionality reduction
with patch("semantica.visualization.embedding_visualizer.px", MagicMock()), patch(
"semantica.visualization.embedding_visualizer.go", MagicMock()
), patch("semantica.visualization.embedding_visualizer.umap", None):
visualizer = EmbeddingVisualizer()
# High-dimensional embeddings (>2D) trigger dimensionality reduction
# with method="umap"
embeddings = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
with pytest.raises(
ProcessingError, match=r"UMAP is required.*semantica\[viz\]"
):
visualizer.visualize_2d_projection(embeddings, method="umap")
# Also verify 3D projection triggers the same actionable error on >3D embeddings
embeddings_4d = np.array(
[[0.1, 0.2, 0.3, 0.4], [0.5, 0.6, 0.7, 0.8], [0.9, 1.0, 1.1, 1.2]]
)
with pytest.raises(
ProcessingError, match=r"UMAP is required.*semantica\[viz\]"
):
visualizer.visualize_3d_projection(embeddings_4d, method="umap")
def test_visualization_sklearn_missing_hint():
import numpy as np
from unittest.mock import MagicMock
from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
# Stand in for Plotly so we reach dimensionality reduction
with patch("semantica.visualization.embedding_visualizer.px", MagicMock()), patch(
"semantica.visualization.embedding_visualizer.go", MagicMock()
):
visualizer = EmbeddingVisualizer()
embeddings = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
# Test direct dependency check
with patch("semantica.visualization.embedding_visualizer.PCA", None):
with pytest.raises(ProcessingError, match=r"scikit-learn is required"):
visualizer._check_dependencies(require_sklearn=True)
with patch("semantica.visualization.embedding_visualizer.PCA", None):
with pytest.raises(ProcessingError, match=r"scikit-learn is required"):
visualizer.visualize_2d_projection(embeddings, method="pca")
with patch("semantica.visualization.embedding_visualizer.TSNE", None):
with pytest.raises(ProcessingError, match=r"scikit-learn is required"):
visualizer.visualize_2d_projection(embeddings, method="tsne")
def test_visualization_options_collision_free():
"""Options like n_components, perplexity must not cause keyword collisions."""
import numpy as np
from unittest.mock import MagicMock
from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
mock_pca = MagicMock()
mock_tsne = MagicMock()
mock_umap_cls = MagicMock()
mock_umap_module = MagicMock()
mock_umap_module.UMAP = mock_umap_cls
with patch("semantica.visualization.embedding_visualizer.PCA", mock_pca), patch(
"semantica.visualization.embedding_visualizer.TSNE", mock_tsne
), patch(
"semantica.visualization.embedding_visualizer.umap", mock_umap_module
), patch(
"semantica.visualization.embedding_visualizer.px", MagicMock()
), patch(
"semantica.visualization.embedding_visualizer.go", MagicMock()
):
visualizer = EmbeddingVisualizer()
embeddings = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
# PCA with n_components
visualizer.visualize_2d_projection(embeddings, method="pca", n_components=2)
# TSNE with perplexity and random_state
visualizer.visualize_2d_projection(
embeddings, method="tsne", perplexity=1, random_state=42
)
# UMAP with n_neighbors and min_dist
visualizer.visualize_2d_projection(
embeddings, method="umap", n_neighbors=2, min_dist=0.1
)
# 3D with n_components
visualizer.visualize_3d_projection(embeddings, method="pca", n_components=3)
def test_spacy_load_missing_hint():
from semantica.semantic_extract.methods import load_spacy_model
with patch("semantica.semantic_extract.methods.spacy", None):
with pytest.raises(ImportError, match=r"semantica\[nlp-spacy\]"):
load_spacy_model("en_core_web_sm")
def test_xml_parser_handles_comments():
xml_content = (
"<root><!-- top comment --><item id='1'>Value</item>"
"<!-- bottom comment --></root>"
)
# lxml engine
p_lxml = XMLParser(engine="lxml")
res_lxml = p_lxml.parse(xml_content)
assert res_lxml.root.tag == "root"
assert len(res_lxml.root.children) == 1
assert res_lxml.root.children[0].tag == "item"
assert res_lxml.root.children[0].text == "Value"
# etree engine
p_etree = XMLParser(engine="etree")
res_etree = p_etree.parse(xml_content)
assert res_etree.root.tag == "root"
assert len(res_etree.root.children) == 1
assert res_etree.root.children[0].tag == "item"
assert res_etree.root.children[0].text == "Value"
def test_public_api_ingestor_handles_xml_comments():
from semantica.ingest.public_api_ingestor import PublicAPIIngestor
xml_content = "<root><!-- comment --><item id='1'>Value</item></root>"
ingestor = PublicAPIIngestor(rate_limit_delay=0)
# 1. Default (defusedxml if available)
parsed = ingestor._parse_xml(xml_content)
assert parsed["tag"] == "root"
assert len(parsed["children"]) == 1
assert parsed["children"][0]["tag"] == "item"
assert parsed["children"][0]["text"] == "Value"
# 2. lxml fallback
with patch("semantica.ingest.public_api_ingestor.safe_xml_etree", None):
parsed_lxml = ingestor._parse_xml(xml_content)
assert parsed_lxml["tag"] == "root"
assert len(parsed_lxml["children"]) == 1
assert parsed_lxml["children"][0]["tag"] == "item"
assert parsed_lxml["children"][0]["text"] == "Value"
def test_huggingface_model_loader_catches_oserror():
import builtins
from unittest.mock import MagicMock
from semantica.semantic_extract.providers import HuggingFaceModelLoader
mock_torch = MagicMock()
mock_torch.Tensor = type("Tensor", (), {})
with patch.dict("sys.modules", {"torch": mock_torch}):
loader = HuggingFaceModelLoader()
# 1. Test ModuleNotFoundError / ImportError
with patch.dict("sys.modules", {"transformers": None}):
with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
loader.load_ner_model("bert-base-cased")
with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
loader.load_relation_model("bert-base-cased")
with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
loader.load_triplet_model("t5-base")
# 2. Test OSError (e.g. corrupt DLL / missing shared library)
real_import = builtins.__import__
def fake_import(name, *args, **kwargs):
if name == "transformers":
raise OSError("DLL load failed")
return real_import(name, *args, **kwargs)
try:
builtins.__import__ = fake_import
with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
loader.load_ner_model("bert-base-cased-oserror")
with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
loader.load_relation_model("bert-base-cased-oserror")
with pytest.raises(ImportError, match=r"semantica\[models-huggingface\]"):
loader.load_triplet_model("t5-base-oserror")
finally:
builtins.__import__ = real_import
+64
View File
@@ -0,0 +1,64 @@
"""Tests for the shared in-process tool entry point (issue #1355).
``semantica_mcp.mcp.server.call_tool`` is the dispatch used by both the
JSON-RPC ``tools/call`` handler and the ``semantica mcp call`` CLI command,
so the two surfaces cannot expose different tool sets.
"""
import os
import sys
import unittest
from unittest.mock import patch
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
from semantica_mcp.mcp import server
from semantica_mcp.mcp.server import UnknownToolError, _handle_tools_call, call_tool
class TestCallTool(unittest.TestCase):
def test_known_tool_dispatches_to_handler(self):
# Empty args hit extract_entities' own validation before any heavy
# imports, which is enough to prove dispatch reached the handler.
result = call_tool("extract_entities", {})
self.assertEqual(result["error"], "text is required")
def test_unknown_tool_raises_unknown_tool_error(self):
with self.assertRaises(UnknownToolError):
call_tool("no_such_tool", {})
def test_unknown_tool_error_is_not_a_key_error(self):
"""A handler's own KeyError (missing required arg) must remain
distinguishable from an unknown tool name."""
self.assertFalse(issubclass(UnknownToolError, KeyError))
class TestToolsCallDispatch(unittest.TestCase):
@staticmethod
def _tools_call(name, arguments):
return _handle_tools_call(1, {"name": name, "arguments": arguments})
def test_unknown_tool_returns_method_not_found(self):
response = self._tools_call("no_such_tool", {})
self.assertEqual(response["error"]["code"], -32601)
self.assertEqual(response["error"]["message"], "Unknown tool: no_such_tool")
def test_handler_key_error_is_internal_error_not_unknown_tool(self):
def _boom(args):
raise KeyError("category")
fake = {"name": "boom", "description": "", "inputSchema": {}, "_handler": _boom}
with patch.dict(server._TOOL_INDEX, {"boom": fake}):
response = self._tools_call("boom", {})
self.assertEqual(response["error"]["code"], -32603)
def test_known_tool_returns_result_content(self):
response = self._tools_call("extract_entities", {})
self.assertIn("content", response["result"])
self.assertTrue(response["result"]["isError"])
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -91,7 +91,7 @@ _INIT_REQUEST = _jsonrpc("initialize", 1, {
# ---------------------------------------------------------------------------
class TestMCPStdioFramingContract(unittest.TestCase):
"""Run 'python -m mcp' exactly as an MCP client would, over a real pipe.
"""Run 'python -m semantica_mcp.mcp' exactly as an MCP client would, over a real pipe.
Each test sends a complete JSON-RPC session through stdin and asserts that
every byte on stdout is valid JSON catching the exact failure mode from
@@ -102,7 +102,7 @@ class TestMCPStdioFramingContract(unittest.TestCase):
def _run(self, *requests: bytes) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, "-m", "mcp"],
[sys.executable, "-m", "semantica_mcp.mcp"],
input=b"".join(requests),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
@@ -38,13 +38,14 @@ class TestOptionalDependencies(unittest.TestCase):
with import_without(
"semantica.visualization.embedding_visualizer", "umap"
) as module:
with plotly_doubles(module):
with plotly_doubles(module), patch.object(module, "PCA") as mock_pca_class:
mock_pca_class.return_value.fit_transform.return_value = np.zeros((4, 2))
viz = module.EmbeddingVisualizer()
embeddings = np.array([[0, 1, 2], [1, 0, 3], [0, 0, 0], [1, 1, 1]])
with self.assertRaises(module.ProcessingError) as cm:
viz.visualize_2d_projection(embeddings, method="umap")
self.assertIn("UMAP is required", str(cm.exception))
self.assertIn("semantica[viz]", str(cm.exception))
viz.visualize_2d_projection(embeddings, method="umap")
mock_pca_class.assert_called()
def test_ontology_visualizer_without_graphviz(self):
"""Test OntologyVisualizer behavior when graphviz is missing."""