Route `semantica mcp call` and `semantica mcp list-tools` through the
canonical `semantica_mcp.mcp.server` packaged module instead of the
nonexistent `MCPSession` import.
Previously, `semantica mcp call` imported `MCPSession` from
`semantica_mcp.mcp.session`, which never defined it, causing every
call to fail with "MCP module not available". Meanwhile,
`semantica mcp list-tools` inspected `semantica_mcp.mcp.tools.__all__`,
which evaluated to `["TOOL_DEFINITIONS"]` and printed a single tool
named "TOOL_DEFINITIONS".
Key changes:
- Implement `semantica_mcp.mcp.server.call_tool(name, arguments)` as the
shared in-process entry point used by both the CLI and the JSON-RPC
`tools/call` handler, ensuring both surfaces expose identical tools.
- Add `UnknownToolError` to distinguish missing tools (JSON-RPC -32601)
from handler-level `KeyError` exceptions (JSON-RPC -32603).
- Update `semantica mcp list-tools` to source tool names directly from
`TOOL_DEFINITIONS` in `semantica_mcp.mcp.tools`.
- Validate `--args` payloads and reject non-object JSON inputs (arrays,
primitives, null) with a clean `ClickException`.
- Update `tests/test_mcp_stdio_roundtrip.py` to spawn `semantica_mcp.mcp`
instead of the pre-relocation `mcp` module, restoring clean stdio
framing tests.
Fixes#1355
- Reduce direct core dependencies in pyproject.toml from 44 to 22
- Move heavy and specialized packages into modular optional extras:
* models-huggingface: torch, transformers
* embeddings-local: sentence-transformers, fastembed, onnxruntime, tokenizers
* nlp-spacy: spacy, thinc
* viz: matplotlib, seaborn, plotly, ipywidgets, umap-learn (expanded)
* media: librosa, opencv-python
* vectorstore-faiss: faiss-cpu
* documents: python-docx, openpyxl, lxml, beautifulsoup4
* ingest-git: GitPython
* graph-embeddings: gensim
- Update semantica[all] and semantica[vectorstore-all] to encompass all extras
- Ensure lazy parser construction (DOCXParser, ExcelParser, HTMLParser, XMLParser) without error on __init__(), failing only inside .parse() with clear hints
- Add stdlib xml.etree fallback in XMLParser when lxml is missing
- Guard unguarded matplotlib imports in EmbeddingVisualizer and OntologyVisualizer
- Standardize user-facing error messages to point to pip install 'semantica[extra]'
- Bump version to 0.7.0 in pyproject.toml, semantica/__init__.py, and CITATION.cff
- Add migration notes to README.md and CHANGELOG.md
- Recompile CI and Docker requirements lockfiles
- Add dedicated test suite tests/test_issue_1513_slim_core.py
The global --json flag documents "machine-readable JSON to stdout;
errors to stderr", but commands failing inside _run_with_error_handling()
rendered a Rich error panel through the module-level Console, which
writes to stdout. Any pipeline or automation parsing stdout as JSON
broke on the first failure:
$ semantica --json mcp call extract_entities --args '[1,2' 1>/tmp/out 2>/tmp/err
exit=1 stdout_bytes=634 stderr_bytes=0
Branch in _show_error_card(), the single renderer that callers of
_run_with_error_handling() route through. When JSON mode is active
(global --json on the CLI context or a subcommand's local --json flag,
uniformly named local_json), emit a structured {"error": ..., "type": ...}
JSON line to stderr and leave stdout untouched. Exit codes and interactive
Rich panels in non-JSON mode remain unchanged.
Add regression tests verifying stdout remains empty and stderr produces
parseable JSON under both global and local --json flag scopes.
Split out of #1368 as an independent fix.
* fix(cli): dispatch reason run and store connect to real APIs
semantica reason run called Reasoner.run(), which does not exist --
the facade's API is infer_facts(facts, rules). The command now reads
nodes/relationships from the configured graph store as fact strings
(the same conventions Reasoner.add_fact applies to KG-style dicts),
loads --rules as a YAML list/mapping or plain-text lines, and reports
the inferred facts.
semantica store connect called get_graph_store_method(backend), which
is the (task, method_name) method registry, not a backend factory, so
it raised a TypeError before any connection attempt and failed
identically with or without valid credentials. It now builds the store
via GraphStore(backend=...) and probes connect(), so real
connectivity/auth errors surface.
Fixes#1354
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): use backend relationship keys, all labels, and honest engine dispatch in reason run
Review follow-ups on the reason run fact conversion:
- Relationships from every graph store backend carry start_node_id /
end_node_id; _graph_store_facts() read start_id / end_id, so each
relationship became TYPE(None, None) and relationship rules never
matched. Read the real keys and cover it with a rule that matches the
relationship fact.
- Emit a fact for every node label, not just the first, so rules on
secondary labels can match.
- reason run only executes the Reasoner facade's forward-chaining
inference; --engine values with different input paradigms (datalog,
sparql, abductive, deductive, graph) now fail with a clear message
(pointing SPARQL/Datalog at reason query) instead of reporting an
engine that never ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): harden reason run rule loading
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Sameers-MacBook-Air.local>
- Stop recommending `semantica kg build` as a workaround: it does not
persist to a configured graph store either (tracked in #1352), so
pointing users at it just traded one silent no-op for another.
- Make `--output` a real escape hatch instead of a silent bypass: it
now writes the ingested result to a .json/.jsonl/.csv file via the
existing `_write_result_output` helper, rather than being forwarded
as an ignored kwarg to the ingestor.
- Stop forwarding `--store`/`GRAPH_STORE_DEFAULT_BACKEND` into the
ingest call — they were already silently discarded downstream, so
they're now only used to decide whether to raise the "no CLI command
persists yet" error.
- Teach `_json_default` to expand dataclasses (e.g. `FileObject`) and
decode `bytes`, so `--output` produces real content instead of a
Python repr string.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds Google ADK (Agent Development Kit) support to Semantica.
`semantica_kg_tools()` and `semantica_decision_tools()` expose entity/relation extraction, graph updates, and decision recording as ADK `FunctionTool`s. `SemanticaSessionService` implements ADK session storage on top of a Semantica `ContextGraph`, so session state, events, and knowledge graph data can live in the same graph instead of keeping sessions in memory.
There were also a number of dependency and CI fixes needed to get the integration working reliably. `google-adk` is pinned to a range that avoids the CI `websockets` conflict, the deprecated `pinecone-client` dependency was replaced with `pinecone`, Windows-only dependencies now have the appropriate platform markers, and `requirements-ci.txt` was regenerated to match. A `pip-audit` pass also required updates to `google-adk` and `starlette` for known CVEs.
Some unrelated `pyproject.toml` changes had slipped in during rebases, so the previous version, dependency bounds, `ingest-sap`/LangChain entries, and package-data settings were restored.
A few bugs in the initial ADK implementation were fixed during review:
* `extract_relations()` was calling `RelationExtractor.extract_entities()`, which doesn't exist on that extractor. The failure was being caught and returned in the tool's `error` field, leaving callers with an empty relation list. It now calls the correct extraction path.
* The repo's top-level `mcp/` package shadowed the third-party `mcp` package imported by `google.adk`, causing `google.adk` imports to fail from a normal repo checkout. The local package was moved to `semantica_mcp/mcp/`.
The MCP move needed a follow-up as well. `semantica/cli.py` and four existing tests were still importing from `mcp.*`, and the modules under `semantica_mcp/mcp/` still used the old absolute imports internally. `semantica_mcp` was also missing from the setuptools package include list and had no `__init__.py`, so it wouldn't have been included in an installed package. Those imports and packaging settings are fixed now.
The session service and ADK tools also had a few other problems:
* `list_sessions()` returned a plain list instead of ADK's `ListSessionsResponse`. The original import for that type doesn't work against the installed `google-adk` package, so it was silently falling back to a stub. `user_id` was also incorrectly required instead of being optional.
* Session node IDs were built by joining `app_name`, `user_id`, and `session_id` with unescaped colons, which allowed different identities to produce the same graph node ID. Each component is now encoded before joining.
* `kg_tools.py` and `decision_tools.py` each had their own lock registry and default graph instance. Sharing a graph between the two modules therefore didn't share the lock, and using both factories without an explicit graph produced two different defaults. The shared state now lives in one module used by both.
* `add_to_graph` had a `TypeError` compatibility fallback that couldn't succeed with the current `RelationExtractor` API and could hide the original extraction error. That fallback was removed.
* `append_event` persisted partial streaming events even though ADK's base session service skips them.
* `get_session()` ignored its `config` argument, so `num_recent_events` and `after_timestamp` had no effect.
* The async session-service methods performed synchronous graph scans while holding a `threading.RLock` on the event loop thread. That work now runs in worker threads with `asyncio.to_thread()` so a slow or contended graph operation doesn't block the loop.
---
Co-authored-by: Zohaib Hassnain [109234410+ZohaibHassan16@users.noreply.github.com](mailto:109234410+ZohaibHassan16@users.noreply.github.com)
* fix(cli): write embed generate output in the format embed index reads
* Address review: structured results get their own --output writer
deduplicate --output and ontology align --output were routed through
_write_embeddings_output, a helper for numeric matrices: it rejects the
dict/list shapes these commands produce and the .csv extension deduplicate
documents. New _write_result_output serializes structured results — JSON,
JSON-lines for lists, CSV for rows — and both commands use it. embed
generate keeps the embeddings writer, whose strictness is what #994 fixed.
On the pyarrow gap: the parquet writer already fails with an actionable
message (install pyarrow or use .json). Silently writing JSON bytes to a
.parquet path would recreate #994's magic-bytes failure, so the error stays
an error and the default suggestion stays .json.
* fix(cli): improve structured output serialization
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Consolidates the remaining #994 fixes into this PR so it can fully close
the issue, per maintainer request.
From #1005 (yzxcj797):
- EmbeddingGeneratorWithProvenance.__getattr__ self-recursion guard:
accessing self._generator via attribute syntax re-entered __getattr__
forever when _generator was absent (failed __init__, pickle/copy probes
like __deepcopy__). Private-name lookups now raise AttributeError.
- 4 regression tests in TestMethodDispatchRecursion: default dispatch no
longer self-recurses for generation/text, a user-registered custom
method still takes precedence, and a bare provenance wrapper raises
AttributeError instead of RecursionError.
(The methods.py identity guards from #1005 are already present here.)
From #1006 (yzxcj797):
- doctor gains two embedding backend checks, "Embeddings
(sentence-transformers)" and "Embeddings (fastembed)". Default is a
cheap import+version check (uninstalled backend now reports fail with a
pip hint instead of invisible). --deep-embeddings (or
SEMANTICA_DOCTOR_DEEP_EMBEDDINGS=1) instantiates via TextEmbedder and
embeds a probe, catching backends that import cleanly but cannot load
(the #994 failure mode) via the hash-fallback-active signal.
_DeepEmbeddingFailure marks post-import runtime/model-load failures so
they get a remediation hint instead of a misleading pip-install hint.
- 7 tests in TestDoctorEmbeddings and TestDoctorEmbeddingHintsAndEnv.
Validation:
- tests/test_cli_commands.py: 237 passed (7 new)
- tests/test_embedding_providers.py: 9 passed (4 new)
- AST parse + import of all four modules OK
## Summary
Overhaul the CLI and all library modules to produce polished, modern
terminal output comparable to tools like uv, gh, and cargo. Rich was
already a declared dependency but barely used — this commit wires it
throughout every layer.
## Changes by layer
### semantica/cli.py — visual overhaul
- Add imports: `box`, `Panel`, `Rule`, `Syntax`, `Text` from Rich
- Add 7 style constants (`_BRAND`, `_KEY`, `_VAL`, `_DIM`, `_SUCCESS`,
`_WARN_STY`, `_TABLE_BOX`) for a consistent colour palette
- `_ok()` now prefixes output with a green ✓ checkmark
- New `_info()` helper (neutral · bullet, respects --quiet)
- New `_warn()` helper (yellow ⚠ prefix, never suppressed)
- New `_pprint()` helper: renders dicts/lists as syntax-highlighted JSON
(Rich Syntax, monokai theme) instead of raw Python repr; strings
pass through unchanged; respects --quiet
- `info` command: banner replaced with a rounded Rich Panel showing
version + tagline; component table uses SIMPLE_HEAD box
- All 7 table sites updated: `box=SIMPLE_HEAD`, `show_edge=False`,
consistent `_KEY`/`_VAL` column styles (KG Stats, Reasoning Engines,
Recent Decisions, Configured Backends, Backup Info, MCP Tools)
- `_run_build()`: `console.status(spinner="dots")` wraps the blocking
build call; skipped under --quiet / --json
- `parse`, `extract`, `embed generate`, `reason run`, `reason explain`,
`deduplicate`: each wraps its long-running operation in a status
spinner, guarded by --quiet / --json
- All 30+ `console.print(result)` calls replaced with `_pprint()`
- All raw `[yellow]Warning:[/yellow]` and "not running" patterns
replaced with the new `_warn()` / `_WARN_STY` style
### semantica/explorer/__init__.py
- Error messages use `Console(stderr=True)` with `[bold red]Error:[/bold red]`
- Graph loading wrapped in `console.status()` spinner
- Startup info replaced with a cyan-bordered Rich Panel showing URL,
API docs, and health endpoint
### Library internals — replace print() with structured logger calls
All modules below had active `print()` calls that bypassed the logging
framework, corrupted spinners, and polluted stdout in piped/programmatic
use. All replaced with appropriate `self.logger.*` calls:
- `semantica/kg/graph_builder.py` — 23 calls: entity resolution
progress, graph structure steps, GraphStore persistence timing, and
the two `='*60` completion banners → `self.logger.info/debug()`
- `semantica/semantic_extract/methods.py` — 4 verbose-mode debug
prints → `logger.debug()`
- `semantica/semantic_extract/relation_extractor.py` — progress +
error prints → `self.logger.debug/warning()` with `exc_info`
- `semantica/semantic_extract/triplet_extractor.py` — same pattern
- `semantica/semantic_extract/semantic_network_extractor.py` — batch
error prints → `self.logger.warning/error()`
- `semantica/semantic_extract/coreference_resolver.py` — error print
→ `self.logger.error()`
- `semantica/semantic_extract/providers.py` — debug print →
`self.logger.debug()`
### Tooling
- `benchmarks/benchmarks_runner.py`: Rule banner, ✓/✗/⚠ status lines,
Rule separators around regression alert
- `benchmarks/infrastructure/compare.py`: removed manual ANSI escape
codes; comparison output is now a Rich Table with SIMPLE_HEAD;
summary uses coloured Rule + styled SUCCESS/FAILURE messages
- `cookbook/advanced/snowflake_ingestion_examples.py`: `_section()`
helper using Rule; tabular data rendered as Rich Table; result lines
use ✓/✗/⚠ prefixes; logger.error already present, retained
- `docs_check.py`: `pass`/`FAIL` lines use `[bold green]` /
`[bold red]`; summary uses styled output
## Tests
- `tests/test_cli_commands.py`: fix 3 pre-existing mock mismatches
- `test_kg_stats_json_with_mock`: mock now uses `compute_metrics()`
(the method the code actually calls) instead of `get_statistics()`
- `test_dry_run_not_needed_extract_is_read_only` and
`test_stdin_input`: mock now provides `NERExtractor`,
`RelationExtractor`, `TripletExtractor`, `EventDetector`
(the classes the code imports) instead of `SemanticAnalyzer`
Result: 230/230 tests pass (was 227/230)
- `tests/verify_rich_cli.py`: new verification script; exercises all
14 command groups (92 --help checks, table rendering, dry-run
formatting, --json mode, _pprint helper); 111 pass, 0 fail
P1 — runtime-breaking API mismatches:
- decision record/list/query/trace/similar/impact/check: all six decision
commands now call decision_methods / decision_query using a GraphStore
from _get_graph_store(cli_ctx) instead of passing config= kwargs that
don't exist on the underlying API signatures.
- embed index: load vectors from the Parquet/JSON file into List[np.ndarray]
before calling create_index(), which expects vectors not a file path string.
P2 — stub implementations replaced with real logic:
- backup sync: now collects local data sources via _collect_backup_sources
and performs an incremental copy (skips files whose dst mtime >= src mtime).
- backup restore: detects .enc / tar.gz / .tar / directory, decrypts SEM1
format when --enc, extracts tar archives with leading prefix stripped, or
copies directory trees back to cwd.
P3 — correctness bugs:
- backup create: archive now includes actual config/ontology/store data files
via _collect_backup_sources; manifest records the file list.
- extract: --output now works for all formats (table/rdf/yaml), not only JSON.
- backup create: empty keyfile now raises a clear error instead of silently
producing an unencrypted archive.
- normalize: use Path.is_file() instead of Path.exists() to avoid accidentally
reading a directory that matches the input text.
- visualize: without --output, emit to stdout; do not silently write kg.html.
Minor:
- _setup_cli_logging: replace opaque _ = (quiet, json_output, exc) tuple
with del to suppress unused-variable lint.
- reason list: try to source engines from the reasoning module registry;
fall back to the hardcoded list.
- deduplicate --action report: use method="pairwise" to produce individual
pair objects with similarity scores, distinct from --action detect.
- tests: remove mixed import (from semantica.cli import main) — all 192
runner.invoke calls now use cli_module.main as CodeQL flagged.
- tests: add two focused embed-index regression tests that verify vectors
are loaded from the file before create_index is called.
Fix the export runtime mismatch where get_export_method expected the existing (task, name) registry contract but the CLI passed only the format argument.
Expands semantica/cli.py from a 2-command stub into a complete terminal
interface covering every capability described in issue #568, and ships
253 tests covering all new commands, flags, and error paths.
Co-Authored-By: KaifAhmad1 <kaifahmad087@gmail.com>
Mohd Kaif