Both skills instruct the agent to clear the result cache via
`from semantica.semantic_extract.cache import _result_cache`, but the module
exports the `ExtractionCache` singleton as `extraction_cache`. The private
name never existed, so the first step of both skills raises ImportError.
`extraction_cache.clear()` is the equivalent call.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ontology, policy, provenance, change and query skills documented classes
and methods that do not exist in the package. Every import in them fails, so
following the skill produces ImportError or AttributeError immediately:
| Documented | Reality in 0.7.0 |
| --- | --- |
| `semantica.policy.PolicyEngine` | no `semantica.policy` module; `PolicyEngine` is in `semantica.context` |
| `semantica.query.QueryEngine` | no `semantica.query` module; `QueryEngine` is in `semantica.triplet_store` |
| `semantica.ontology.OntologyManager` | no such class; use `OntologyEngine` / `OntologyValidator` |
| `semantica.provenance.ProvenanceTracer` | no such class; use `ProvenanceManager` |
| `semantica.provenance.change_tracker.ChangeTracker` | no such module; ontology versioning lives in `semantica.change_management` |
The method names were wrong too, so a path-only fix was not possible:
`.check()`, `.list_rules()`, `.trace_node()`, `.get_audit_log()`,
`.compute_diff()`, `.get_node_history()`, `.query_sparql()`, `.query_cypher()`
and `.search()` do not exist on any of the real classes.
Each skill is rewritten against signatures verified by introspection on an
installed 0.7.0. Two notes on scope:
- policy now leads with `ContextGraph.check_decision_rules()` /
`enforce_decision_policy()`, which need no graph store, and keeps
`context.PolicyEngine` as the managed-policy path.
- change previously conflated graph-state-over-time with ontology versioning.
These are separate mechanisms in 0.7.0, so the skill now documents
`ContextGraph.state_at()` and `change_management.VersionManager` separately.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(explorer): label the graph view control "Focus"
The control read "Focused" before it was ever activated, which describes a
state the selection had already reached rather than the action available.
The view mode value, tooltip, enablement and active styling are unchanged.
The legend e2e drove this button by its accessible name, so the selector
moves with the label; it now also asserts the visible text, the
selection-dependent enablement and the active state.
Closes#1551
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(explorer): name the control in the grouped-selection hint, and trim the test
"Activate Focused mode" instructed the reader to press a control that no
longer carries that name. The surrounding strings describe the mode itself,
which is still called focused, so they stay.
Drop the label and active-state assertions from the colour-legend test: the
getByRole locator already fails when the accessible name is wrong, and the
rest belonged to the control's contract rather than to the legend's.
Co-Authored-By: Claude Code <noreply@anthropic.com>
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
* fix(explorer): resolve ontology ownership on the backend
Which registered ontology owns an entity was decided twice: the backend
applies nested-namespace boundaries, while the Ontology Editor did a bare
prefix match. The two had already drifted, so a deep link to an entity in an
unregistered nested namespace selected the parent ontology whose /graph
response excludes that entity, and the selection silently failed.
/api/ontology/entity now returns owning_ontology, resolved with the same rule
the graph endpoint filters by, and the editor prefers it. The frontend
namespace guess stays as the fallback for a missing verdict, documented as
non-authoritative.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(explorer): keep the backend's no-owner verdict authoritative
Review follow-up: loadOntologyEntityOwner collapsed the backend's
explicit owning_ontology: null into undefined, re-activating the
namespace prefix guess for exactly the unregistered-nested-namespace
case this PR exists to fix. The owner verdict is now three-state
(owner / authoritative none / unavailable) and resolveEditorOntology
in the model suppresses inference on an authoritative none; only an
unavailable verdict may fall back. Model tests pin all three states.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(explorer): trust the backend to send owning_ontology
The Explorer bundle ships in the same wheel as the route that emits this
field, so the legacy-response branch could never run. Dropping it lets
the type say what the wire actually carries, leaving undefined to mean
only what it should: the request failed.
Note why the endpoint derives its ontology-URI set inline rather than
calling _known_ontology_uris, so the next reader does not consolidate a
graph scan back in.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(explorer): stop the registry default overriding a no-owner verdict
resolveEditorOntology returned string | undefined, so the caller wrote
`resolved || entries[0]?.uri` and an authoritative "nothing owns this
entity" fell straight through to an arbitrary registry entry. When that
entry happened to be the parent, the deep link opened the parent whose
graph excludes the entity — the bug the verdict exists to prevent. Only
the ordering of the registry in the earlier test hid it.
It now returns a union: unowned and unresolved both mean "no ontology to
open" but the editor treats them oppositely, so collapsing them with ||
is a type error rather than a silent regression. An unowned entity is
reported on the canvas instead of quietly opening the wrong ontology.
Also:
- /entity resolves ownership through _known_ontology_uris, the same
helper /graph uses, instead of deriving it from a get_nodes scan capped
at 999,999. Past that cap the set was silently truncated and the two
endpoints could disagree about who owns a node.
- _resolve_owning_ontology does one pass over the candidates rather than
one pass per candidate, each rescanning the whole set: 516us -> 9us at
50 ontologies, 125ms -> 138us at 800, same answers throughout. A test
pins it against _node_belongs_to_ontology so the hand-rolled version
cannot drift from the membership rule it has to mirror.
- An explicit scheme_uri is honoured even when the registry does not list
it, on both sides. Discarding it and guessing by namespace answered a
question nobody asked; an unregistered owner now surfaces as an
explicit error from /graph instead.
- A missing owning_ontology field reads as "no verdict", not as the
authoritative "nothing owns this". That claim now suppresses selection
outright, so it must not be inferred from an absent field.
Co-Authored-By: Claude Code <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
The deep-link protocol introduced in #1278 lived as bare "ontologyTab" /
"ontologyEntity" literals in five places across three files, each with its own
URLSearchParams plumbing and try/catch. Nothing tied the pieces together: in
particular the rule that a selection written under one ontology must be cleared
when the active ontology changes — otherwise a reload resolves the stale entity
and jumps back to the old ontology — was a comment at one call site with no
mechanism behind it.
ontologyUrlState.ts now owns the parameter names as private constants and
exposes the protocol as intent-named operations, with the write/clear pairing
documented where both halves live. Parsing and serialization are pure functions
over a search string, so they are covered by tests without a DOM; the window
and history.replaceState interaction stays in thin shells.
Absent parameters still read as undefined while blank ones read as empty
strings, which preserves the differing presence checks the workspace shell and
the tab selector each relied on.
One behavior change, inherited from all five original call sites: writing the
query string dropped any URL fragment, because replaceState with a bare "?..."
replaces the whole tail. updateSearch now carries window.location.hash across,
which fixes it for every writer at once — this module is the only place that
knows how the URL is written, so it is the only place the fix belongs.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
TimelinePanel fell back to a hardcoded 2030-01-01 whenever
/api/temporal/bounds reported max: null, then placed the playhead at the
midpoint of that fabricated window. session.get_temporal_bounds leaves max
open for any graph whose nodes carry valid_from instants and no
valid_until, so this was the normal response shape rather than bad data:
the header advertised a range no data supported and the workspace's first
/api/temporal/snapshot request asked about a time years ahead of the
present.
The fallback is now a `now` captured once per mount, and defaultTime is
that same `now` clamped into the range, so the initial snapshot describes
the current state. Three settings tuned for the fictional ~60-year window
follow from it: zoomMin drops from a year to a day, the timeAxis/format
pinning to 5-year ticks is removed so vis-timeline picks a granularity for
the real span, and the fixed 6-month play step becomes span/60 with a
one-day floor, which keeps a play-through at roughly 60 frames whether the
graph covers months or a decade.
The bound and step arithmetic moves into temporalScrubberBounds.ts so it
can be asserted directly, alongside the existing temporalLifecycle
predicate tests.
Closes#1536
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix(vector-store): prevent in-memory vector id reuse
* fix(vector-store): synchronize in-memory mutations
* fix(vector-store): clarify no-silent-overwrite guarantee for in-memory ID generation
The while-loop in store_vectors() already prevents any generated candidate
from landing on a live key: it breaks only when the candidate is absent
from both pre_existing (the live store at lock-entry) and within_batch
(IDs chosen earlier in the same call).
This commit:
- Renames 'existing' to 'pre_existing' and introduces 'within_batch' to
make the two-level de-dupe explicit and self-documenting.
- Tightens the break condition to 'not in pre_existing and not in
within_batch' so within-batch duplicates are also guarded.
- Adds test_auto_generated_id_never_silently_overwrites_live_vector:
stores 5, deletes 3, stores 2 more and asserts none of the new IDs
land on a surviving vec_N.
- Adds test_collision_detection_no_silent_overwrite_even_with_corrupted_counter:
rewinds _next_id to 0 with live vectors present and proves the loop
finds a free slot without overwriting either existing vector or its
metadata — the no-silent-overwrite invariant holds even under counter
corruption.
The skip-over-occupied-key behaviour for caller-inserted vec_N IDs is
intentional and preserved (test_counter_skips_explicit_vec_n_ids), matching
FAISSStore's identical pattern.
Closes reviewer finding: 'Vector collisions do not fail writes'.
* fix(vector-store): protect concurrent in-memory access
---------
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
#1540 added GHSA-4j2p-28q2-5m79 to security-scan.yml's IGNORED_VULN_IDS to
suppress the open, unpatched accelerate<=1.14.0 path traversal advisory
(Semantica doesn't call load_checkpoint_in_model/load_checkpoint_and_dispatch).
That commit's own CI run still failed: pip-audit's OSV-backed report picked
CVE-2026-69112 as the vuln's canonical `id` and demoted the GHSA id to an
alias, but the shell/JS matching only ever compared against `.id`.
List both identifiers and match against `.id` plus `.aliases` (which
pip-audit includes by default for JSON output) in the audit gate, the
"Vulnerability details" printer, and the PR-comment script, so an ignored
advisory is excluded regardless of which alias the report surfaces as
canonical. Verified against the actual failing report from run 34296586683 -
the corrected filter now yields 0 actionable vulnerabilities.
Also add osv-scanner.toml (per GHSA-4j2p-28q2-5m79's Scorecard code-scanning
remediation) so the weekly Scorecard "Vulnerabilities" check stops flagging
the same accepted, unpatched advisory.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Add GHSA-4j2p-28q2-5m79 to IGNORED_VULN_IDS in security-scan.yml.
accelerate<=1.14.0 has an open path traversal advisory (GHSA-4j2p-28q2-5m79) in load_checkpoint_in_model. 1.14.0 is currently the latest available release on PyPI, so no upstream patch exists yet. Semantica does not expose or call sharded checkpoint loading, making this non-actionable. Re-evaluate once an updated accelerate release is published.
The status panel and doctor both defaulted to reporting a 'memory' graph
store, but GraphStore._initialize_store_backend only branches on neo4j,
falkordb, neptune and age - GraphStore(backend='memory') raises
ValidationError. doctor short-circuited on that name and returned a
passing check for a store no command could construct.
Both sites now report the backend _get_graph_store actually resolves to,
and doctor probes it instead of skipping. _get_graph_store's default is
unchanged, so no existing setup behaves differently.
Refs #1481
Add a deterministic, ontology-based schema validator as a sibling to the
confidence-based ExtractionValidator. Both validators implement the same
validate_entities() and validate_relations() interfaces and return the
shared ValidationResult structure, enabling orthogonal composition across
extraction confidence and ontological conformance.
Key Additions & Behaviors:
- ExtractionSchema: Read-only view over domain ontologies (allowed concepts
and predicates with optional domain/range constraints). Supports loading
from dictionary representations (generate_ontology() or OntologyData from
OntologyIngestor) via ExtractionSchema.from_ontology(), and from OWL/Turtle
files/strings via ExtractionSchema.from_owl().
- SchemaValidator: Verifies entity labels against schema concepts and
validates relation predicates against defined domain/range constraints.
Provides filter_by_schema() and filter_relations_by_schema() to extract
conforming subsets without mutating source data.
- Domain/Range Wildcarding: Treats owl:Thing (and unqualified Thing) as
unconstrained wildcards so fallback types do not reject valid endpoints.
- Constructor Parity: Folds relation domain/range endpoint types into the
concepts set across both from_ontology() and from_owl() to ensure logical
consistency.
- Ingest Pipeline Interop: Automatically unwraps OntologyData-like objects
via duck-typing on .data.
- Resilient Endpoint Resolution: Guards malformed relations missing subject
or object endpoints without raising AttributeError, and prefers rdfs:label
over URI suffixes while supporting both owl:Class and rdfs:Class.
Part of #1510.
Repair broken GraphML and Parquet export handling in the MCP
`export_graph` tool.
Previously, both formats failed in the MCP handler: GraphML referenced a
non-existent `GraphMLExporter` class and passed unformatted graph objects,
while Parquet passed raw `ContextGraph` instances to `export()`, causing
runtime type errors and producing empty responses.
Key changes:
- Route GraphML export through `GraphExporter(format="graphml")` via
`export_knowledge_graph()` with automatic `TemporaryDirectory` cleanup.
- Harden GraphML XML generation in `GraphExporter`: declare missing
`label` and `confidence` keys with `for="all"` scope and use
`xml.sax.saxutils` (`quoteattr`, `escape`) to properly escape XML
attributes and node text.
- Route Parquet export through `ParquetExporter.export_knowledge_graph()`,
returning generated `.parquet` files base64-encoded over the JSON MCP
transport.
- Gracefully handle missing `pyarrow` dependencies with structured error
responses.
- Preserve existing JSON, CSV, and RDF export behavior.
- Add comprehensive regression tests in
`tests/test_mcp_package_export_graphml_parquet.py`.
Move the static `GET /api/decisions/causal-distance` endpoint above the
dynamic `GET /api/decisions/{decision_id}` route in the Explorer router.
Starlette evaluates routes sequentially in declaration order. Because
`/{decision_id}` was previously defined first, requests targeting
`/api/decisions/causal-distance` were captured by the parameter route
with `decision_id="causal-distance"`, causing the endpoint to query the
graph for a non-existent decision and return 404.
Changes:
- Reorder `causal_distance` above `get_decision` in
`semantica/explorer/routes/decisions.py` and add a comment guard
against future route-ordering regressions.
- Add regression test suite in
`tests/explorer/test_decisions_causal_distance_route.py` verifying
proper distance reporting, multi-hop traversal, and that unmatched
decision IDs still return 404.
Closes#1531
* fix(parse): stop infinite recursion in default method dispatch
parse_document and its five sibling dispatchers (parse_web_content,
parse_structured_data, parse_email, parse_code, parse_media) were
registered in the method registry under their own task's "default"
name. Every dispatcher begins with method_registry.get(<task>, method),
so calling e.g. parse_document(file, method="default") found itself in
the registry and re-entered infinitely until RecursionError --
`semantica parse <any file>` crashed before any parsing ran.
Drop the six self-registrations. "default" remains the built-in code
path; users can still register their own "default" (or any other name)
to override it, and the existing "docling" registration is unaffected.
Verified: `semantica parse demo.pdf` now parses successfully (was
RecursionError). No repo code or tests consume
get_parse_method("document", "default"), so removing the entries
changes no behavior besides fixing the crash.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(deps): make resolution satisfiable on Python 3.9 and add parse-pdf extra
Dependency fixes so `uv lock` / `pip install semantica[all]` resolves
across the supported matrix (3.9-3.12):
- requires-python >=3.8 was unsatisfiable (numpy>=2.0.2 needs >=3.9) and
3.9.0/3.9.1 can never resolve (every cryptography release excludes
them) -> bump to >=3.9.2 and drop the 3.8 classifier.
- Split recently-raised floors that dropped 3.9 into marker pairs
(3.9-capped / 3.10-unconstrained), following the pattern already used
for scikit-learn/requests/etc.: pyarrow extras (>=24 needs 3.10),
pre-commit 4.6, snowflake-connector 4.6, fastapi 0.129 + starlette 0.53
(older fastapi caps starlette<0.53).
- Gate docling, litellm (its only 3.9 release pins
python-dotenv==1.0.1, conflicting with the >=1.2.1 core floor), and
crewai (no un-yanked 3.9 release) to >=3.10.
Also add a parse-pdf extra: the default PDFParser requires pdfplumber,
but no extra installed it, so `semantica parse file.pdf` failed on a
default install. Follows the parse-docling convention.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(vector-store): make the qdrant backend usable through VectorStore
The qdrant backend could not be used at all through the VectorStore
facade in 0.6.8 - every path raised:
1. store_vectors() only dispatched to backend add/add_vectors methods;
QdrantStore exposes insert_vectors, so vs.store() raised
NotImplementedError. Add an insert_vectors branch (uuid-generated
ids, metadata -> payloads).
2. QdrantStore required an explicit create_collection() before any read
or write, unlike FAISSStore's automatic index creation. Lazily attach
the configured collection (config key "collection", default
"semantica_default") on first insert/search, reusing an existing one.
3. search_points() called client.search(), removed from qdrant-client in
favor of query_points() - use it when available, fall back otherwise.
4. store() silently dropped plain-string documents (it only extracted
doc.metadata), so payloads lost the source text; keep them under
payload "document".
Verified end-to-end against a Qdrant 5 server (docker): embed ->
store -> semantic search returns correctly ranked results whose payloads
carry the original documents.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(vector-store): address Qodo review findings on the qdrant write path
Four findings from the Qodo review of #1508:
1. store_vectors() returned QdrantStore.insert_vectors()' upsert status
dict although the facade promises the stored vector IDs (decision
storage indexes the result at position 0). Return the generated or
caller-supplied IDs after a successful insert instead.
2. insert_vectors() pairs points with zip(vectors, ids), so a shorter
non-empty id list silently dropped the unpaired vectors while the
completion message still reported the full batch as inserted. Reject
the mismatch with ValidationError before any write.
3. _ensure_default_collection() looked up the legacy "collection" config
key, so the documented collection_name=... option was ignored and
lazy init always fell back to semantica_default. Prefer
collection_name, keep "collection" as an alias.
4. The new parse-pdf extra was missing from both aggregate "all"
bundles, so semantica[all] still shipped without pdfplumber and the
default PDFParser raised ProcessingError on first use.
Also removes the now-stale strict xfail for qdrant's write dispatch in
test_backend_facade_contract.py — that marker exists precisely to fail
once the wiring lands.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* test(vector-store): migrate qdrant mocks to the query_points API
The qdrant search path now prefers client.query_points() (qdrant-client
removed client.search), but these tests still mocked the legacy call.
A MagicMock exposes query_points too, so the code took the modern path,
read .points off an unconfigured mock, and all three tests failed on
the branch. Return the hits in response.points as the real client does.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(ci): regenerate requirements-ci.txt for the parse-pdf extra
The CI lockfile check re-resolves pyproject.toml --extra all and diffs
the pinned versions against requirements-ci.txt. The parse-pdf extra
added pdfplumber (+pdfminer-six) to the 'all' bundle without refreshing
the lockfile, so the diff failed and the build job exited 1.
Regenerated with the command from the file header. Only the two new
pins and their 'via' comments changed - every other version is
identical. Verified locally with the CI's own check command (diff of
pkg==ver lines exits clean).
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(tests): update stale default-method-registration assertion
test_parse_methods_dynamic_default_resolution asserted that "default"
resolves through the registry to parse_document, which was true only
because of the self-registration this PR removes (it's what caused the
infinite recursion in the first place). Update the assertion to match
the intended post-fix state: "default" is not registered at all, and
falls through to the built-in dispatch path unconditionally.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: yanyu <yanyu@polixir.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
test_xml_parser_handles_comments and test_public_api_ingestor_handles_xml_comments
called XMLParser(engine="lxml") / forced the lxml fallback path unconditionally,
but the core-only CI step installs from base-deps.txt, which no longer bundles
lxml or defusedxml under this PR's slim dependency layout. Skip the lxml-only
assertions when lxml/defusedxml aren't installed, while still exercising the
always-available etree path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Route `semantica mcp call` and `semantica mcp list-tools` through the
canonical `semantica_mcp.mcp.server` packaged module instead of the
nonexistent `MCPSession` import.
Previously, `semantica mcp call` imported `MCPSession` from
`semantica_mcp.mcp.session`, which never defined it, causing every
call to fail with "MCP module not available". Meanwhile,
`semantica mcp list-tools` inspected `semantica_mcp.mcp.tools.__all__`,
which evaluated to `["TOOL_DEFINITIONS"]` and printed a single tool
named "TOOL_DEFINITIONS".
Key changes:
- Implement `semantica_mcp.mcp.server.call_tool(name, arguments)` as the
shared in-process entry point used by both the CLI and the JSON-RPC
`tools/call` handler, ensuring both surfaces expose identical tools.
- Add `UnknownToolError` to distinguish missing tools (JSON-RPC -32601)
from handler-level `KeyError` exceptions (JSON-RPC -32603).
- Update `semantica mcp list-tools` to source tool names directly from
`TOOL_DEFINITIONS` in `semantica_mcp.mcp.tools`.
- Validate `--args` payloads and reject non-object JSON inputs (arrays,
primitives, null) with a clean `ClickException`.
- Update `tests/test_mcp_stdio_roundtrip.py` to spawn `semantica_mcp.mcp`
instead of the pre-relocation `mcp` module, restoring clean stdio
framing tests.
Fixes#1355
- Remove thinc direct constraint from nlp-spacy in pyproject.toml and update README/CHANGELOG
- Raise ProcessingError with install hint when UMAP is requested but unavailable in EmbeddingVisualizer
- Guard PCA and TSNE dimensionality reducers against None with actionable error messages
- Prevent keyword collisions in EmbeddingVisualizer dimensionality reduction (_reduce_dimensions)
- Add core-only install & test step to .github/workflows/ci.yml using base-deps.txt
- Prevent AttributeError on module load in repo_ingestor.py and xml_ingestor.py when optional dependencies are absent
- Make TOML loading in tests/test_issue_1513_slim_core.py portable across Python versions via UTF-8 text decode and loads
- Expand slim-core test suite to 17 test cases covering 2D/3D projections, core imports, and options handling
- Reduce direct core dependencies in pyproject.toml from 44 to 22
- Move heavy and specialized packages into modular optional extras:
* models-huggingface: torch, transformers
* embeddings-local: sentence-transformers, fastembed, onnxruntime, tokenizers
* nlp-spacy: spacy, thinc
* viz: matplotlib, seaborn, plotly, ipywidgets, umap-learn (expanded)
* media: librosa, opencv-python
* vectorstore-faiss: faiss-cpu
* documents: python-docx, openpyxl, lxml, beautifulsoup4
* ingest-git: GitPython
* graph-embeddings: gensim
- Update semantica[all] and semantica[vectorstore-all] to encompass all extras
- Ensure lazy parser construction (DOCXParser, ExcelParser, HTMLParser, XMLParser) without error on __init__(), failing only inside .parse() with clear hints
- Add stdlib xml.etree fallback in XMLParser when lxml is missing
- Guard unguarded matplotlib imports in EmbeddingVisualizer and OntologyVisualizer
- Standardize user-facing error messages to point to pip install 'semantica[extra]'
- Bump version to 0.7.0 in pyproject.toml, semantica/__init__.py, and CITATION.cff
- Add migration notes to README.md and CHANGELOG.md
- Recompile CI and Docker requirements lockfiles
- Add dedicated test suite tests/test_issue_1513_slim_core.py
Adds native vector deletion support for FAISS Flat indices via remove_ids
and IDSelectorBatch (Closes#1374):
- Adds FAISSIndex.delete_vectors() and FAISSStore.delete_vectors() to
translate external string IDs to sequential internal positions, remove
vectors via remove_ids, and keep vector_ids and metadata parallel with
the compacted index.
- Auto-saves index and metadata sidecar after deletion when the store was
loaded from disk via load_index(), skipping redundant disk writes for
no-op deletions.
- Rejects IVF and HNSW index deletions with NotImplementedError: IVF does
not compact internal labels upon remove_ids (causing search/offset
desynchronization), and HNSW lacks remove_ids entirely. Both cleanly
map to STATUS_UNSUPPORTED in ErasureCoordinator.
Robust ID Management & Invariant Hardening:
- Generates default IDs with a monotonic candidate-check loop against
existing vector_ids, eliminating collisions and metadata overwrites
when explicit vec_N IDs coexist.
- Persists next_id in the .meta.json sidecar and clamps on load() to
max(persisted, max(vec_N) + 1), preventing stale sidecars from
re-introducing collisions across restarts or legacy migrations.
- Guards against FAISS -1 sentinels (0 <= idx < len(vector_ids)) in
search results when k > ntotal, preventing negative index wrapping
to vector_ids[-1].
- Replaces bare post-delete assert with an explicit ProcessingError check
that survives python -O.
Adds 43 comprehensive tests in test_faiss_delete_vectors.py covering
deletion, search exclusion, metadata cleanup, persistence roundtrips,
IVF/HNSW rejection, ErasureCoordinator receipts, and mock-spied
deterministic no-op saves.
Implements four semantic retrieval tools on top of VectorStore wired
into the MCP server tool registry (Closes#1235):
- store_document: Chunks content with a configurable sliding window
(1000/200 default) and attaches full provenance metadata to every
chunk (chunk_id, source, authority, version, hash, status, offsets,
and project). Re-storing identical content under the same
(source, version) is a no-op keyed on content hash to skip redundant
re-embedding.
- retrieve_context: Embeds natural-language queries, ranks scored
chunks with provenance (top_k capped at 10; project filter support),
and attaches 1-hop graph relationships for hit sources from ContextGraph.
- update_document: Replaces stored content for (source, version). Returns
not_found if the document does not exist, and snapshots existing rows
for atomic rollback if writing replacement chunks fails.
- remove_document: Deletes all chunks matching (source, version) and
returns not_found if the document does not exist.
Architecture & Session Management:
- Adds lazy session singletons get_embedder() and get_vector_store()
in semantica_mcp.mcp.session.
- Stores derive dimension directly from the active embedder to keep
indexing and query spaces aligned.
- Restricts backends to inmemory (default) and sqlite (requires
SEMANTICA_VECTOR_DB_PATH); backends lacking metadata-scoped delete
fail fast on startup.
- In-memory updates and removals safely rebuild the store to prevent
vector ID collisions (#1029).
- Fails fast on startup if SEMANTICA_VECTOR_PATH is corrupt or dimension
mismatched, preventing empty stores from overwriting existing corpora.
Hardening & Provenance Protections:
- System-owned provenance: user metadata is additive and cannot overwrite
reserved identity or provenance fields.
- Document size capped at 10,000 chunks to prevent stale chunk retention
against persistent metadata limits.
- Mutation responses return an explicit persisted flag.
Includes a comprehensive test suite in tests/test_mcp_semantic_retrieval.py
using a deterministic FakeEmbedder for network-free, hermetic execution.
The global --json flag documents "machine-readable JSON to stdout;
errors to stderr", but commands failing inside _run_with_error_handling()
rendered a Rich error panel through the module-level Console, which
writes to stdout. Any pipeline or automation parsing stdout as JSON
broke on the first failure:
$ semantica --json mcp call extract_entities --args '[1,2' 1>/tmp/out 2>/tmp/err
exit=1 stdout_bytes=634 stderr_bytes=0
Branch in _show_error_card(), the single renderer that callers of
_run_with_error_handling() route through. When JSON mode is active
(global --json on the CLI context or a subcommand's local --json flag,
uniformly named local_json), emit a structured {"error": ..., "type": ...}
JSON line to stderr and leave stdout untouched. Exit codes and interactive
Rich panels in non-JSON mode remain unchanged.
Add regression tests verifying stdout remains empty and stderr produces
parseable JSON under both global and local --json flag scopes.
Split out of #1368 as an independent fix.
#1290 bumped the runtime stage to python:3.14-slim. gensim is a base
(non-extras-gated) dependency and publishes no cp314 wheel on PyPI, so
pip falls back to building it from source, which needs a C compiler
this slim image doesn't carry:
error: [Errno 2] No such file or directory: 'gcc'
This has broken the Docker build (and Container Security Scan) on
every push to main since #1290 merged. Confirmed via
`pip index versions gensim` / the PyPI files listing that gensim 4.4.0
ships cp313 wheels but no cp314 ones; the earlier lockfile fix in
#1290 only validated that `uv pip compile` could *resolve* dependencies
for 3.14, which reads sdist metadata and doesn't need a compiler --
it doesn't validate that `pip install` can actually *build* them,
which is what fails here.
Reverts to the exact base image digest that was building successfully
before #1290 and regenerates explorer-extra-py313.txt against today's
PyPI state. Once gensim (and anything else pulled in transitively)
ships cp314 wheels, the 3.14 bump can be retried.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* docs(mcp): update stale MCP tool count from 12 to 15
TOOLS in semantica/mcp_server/__init__.py ships 15 tools, but the README
table stopped at get_graph_summary and docs/guides/mcp-server.md still
said 12 in its opening line and setup steps while saying 15 elsewhere on
the same page. Add the three missing rows (query_graph, update_node,
delete_node) to the README table and align the guide's counts. Also fix
the same stale count in the openclaw integration docstring.
Fixes#1487
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs(openclaw): fix stale MCP tool count 12 → 15 and add missing tool rows
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
* docker(deps): bump python from 3.13-slim to 3.14-slim
Bumps python from 3.13-slim to 3.14-slim.
---
updated-dependencies:
- dependency-name: python
dependency-version: 3.14-slim
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(docker): regenerate explorer-extra lockfile for Python 3.14
The base image bump to python:3.14-slim left explorer-extra-py313.txt
in place, a lockfile uv resolved specifically for Python 3.13 and
installed with --require-hashes. Re-resolving the explorer extra for
3.14 adds cloudpickle, which has no hash entry in the 3.13 file, so
the runtime stage's pip install would fail outright.
Regenerates the lockfile as explorer-extra-py314.txt and updates every
reference to it (Dockerfile, .dockerignore, container-scan.yml's path
trigger, ci.yml's comment, and the requirements README).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat(milvus): add delete_vectors to MilvusStore
Adds delete_vectors(ids) so the ErasureCoordinator can erase embeddings
on a Milvus backend. Ids are escaped with _format_milvus_value before
building the delete expression, and the backend delete count is returned
so a delete that removed nothing is distinguishable from a failure.
* test(milvus): cover delete_vectors incl. erasure integration
Unit tests assert the single-id equality expression, multi-id in
expression, id escaping, empty-id noop, missing-collection and backend
error paths. Integration tests bind MilvusStore as a VectorStore backend
and assert ErasureCoordinator reports the vector leg erased.
* test(milvus): cover vector store delete facade
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
Replace the static biomedical category key in Explorer with a live node color legend derived from the active display graph, sharing the canvas's baseColor -> color -> theme fallback resolver.
- Preserve original semantic colors on focused display clones via `semanticBaseColor` so interaction styling (selection, path, neighbor highlights) does not bleed into the legend.
- Suppress the semantic legend while distance visualization modes are active.
- Refactor plugin fallback legend panels to use the shared legend builder with composite (group, color) keys.
- Consolidate duplicate `"test:graph-workspace"` scripts in `explorer/package.json`, recovering 26 previously shadowed tests across markdown and explorer capability suites.
- Add `tests/graphColorLegend.test.ts` to workspace test runs and wire `test:graph-legend-e2e` into `.github/workflows/ci.yml`.
- Closes#1479
* fix(kg): promote synthetic relation endpoints into the graph
Relation extraction synthesizes an UNKNOWN Entity for a relationship endpoint
that is absent from the NER entity list (metadata synthetic=True), but
GraphBuilder kept only the endpoint string, so the synthetic object never
reached the entity collection. GraphValidator then reported DANGLING_EDGE,
breaking the native NER -> relation -> graph -> validation pipeline.
The relation branch now collects synthetic endpoint entities and promotes them
into the entity collection (deduplicated by id, keeping confidence=0.8 and the
synthetic metadata). A new unknown_relation_endpoint option defaults to
include; set it to "reject" to drop relationships whose endpoints are
synthetic instead.
Closes#1463
* fix(kg): reconcile synthetic endpoints after resolution
Address Qodo review feedback on #1464:
- Read unknown_relation_endpoint through the nested "config" mapping the
orchestrator passes (GraphBuilder(config=...)), so the policy actually takes
effect end to end.
- Prefer a real entity over an earlier-promoted synthetic one with the same id,
deduplicating at build tail (unhashable ids are skipped defensively).
- Tag LLM triple endpoints that match no extracted entity and promote them too,
so triplet relationships no longer leave dangling edges.
- Log when the reject policy drops a relationship.
Closes#1463
* fix(kg): key dedup on both id and entity_id; tag HF triple endpoints
Follow-up hardening on #1464's synthetic-endpoint reconcile:
- Dedup promoted endpoints against both `id` and `entity_id` (the canonical
keys the builder itself recognizes), in both the relation branch and the
build-tail reconciliation. The build-tail pass now also skips unhashable
ids defensively.
- Tag HuggingFace REBEL endpoint texts that match no extracted entity, same
as the LLM typed path, so HF triplets no longer leave dangling edges.
- Tests: use a real `Triplet`, cover the `entity_id` dedup case, exercise
`merge_entities=True` explicitly (the prior test mislabeled the default),
and lock in the dict-relationship out-of-scope contract.
* fix(kg): guard metadata=None and gate endpoint promotion scan
_real_ids and the reconcile filter both did .get("metadata", {}).get(...),
which returns None (not {}) when the key is present with a None value and
raised on entities carrying metadata=None.
The synthetic-endpoint promotion block in _process_item rebuilt existing_ids
by scanning all_entities for every relationship, making build() quadratic on
dense relation inputs. Gate it on if synthetic_endpoints so ordinary
relationships pay nothing (6000+6000 no-synthetic builds in 0.018s vs 9.04s).
* config(kg): home unknown_relation_endpoint in build config and document it
The option now has a default in the per-module build method config
(kg_methods.build.unknown_relation_endpoint) and kg_usage.md documents both
values. __init__ folds a config= dict into the option mapping once, which also
fixes entity_resolution/conflict_detection being silently dropped on the
orchestrator path; every option reads the same way now.
* fix(extract): thread entities into HF REBEL triplet path
extract_triplets_huggingface read entities from kwargs.get("entities") which
is always empty on this path, making the synthetic-endpoint tag a no-op
comparison. Declare entities as a parameter so it matches the real NER list
TripletExtractor already forwards.
* fix(kg): finalize synthetic relation endpoint handling
- Add concrete Python API (kg_config.set_method_config) and YAML config
file example to kg_usage.md; the previous text referenced only an opaque
config-file key path with no runnable code.
- Add three targeted tests to test_dangling_synthetic_endpoint.py:
* test_real_entity_with_entity_id_only_wins_over_prior_synthetic: covers
the _real_ids dedup pass when the real entity carries its id under
entity_id only (no 'id' field).
* test_llm_relation_synthetic_endpoints_promoted: exercises the full
_parse_relation_result → Relation → GraphBuilder promotion path end-to-end.
* test_reject_policy_emits_warning_when_all_rels_dropped: verifies that
the 'all relationships were dropped' warning fires correctly when the
reject policy drops every relationship from a dict-style source.
- Add import pytest (required for caplog fixture used in the new test).
- Fix missing EOF newline in the test file.
* fix(kg): make reject-policy edge drops observable
The reject policy logged at INFO and returned no machine-readable count,
making it effectively silent at WARNING-level logging thresholds.
Issue #1463 requires that rejected relationships are 'reported explicitly'.
Changes:
- Elevate the per-rejection log from INFO to WARNING so it is visible
without debug logging enabled.
- Add self._rejected_relationships counter (reset per build() call,
incremented per dropped edge).
- Expose the count in graph['metadata']['rejected_relationships'] so
callers can check the outcome programmatically without log parsing.
- Document both observability mechanisms in kg_usage.md.
- Add four focused regression tests:
* rejected count matches number of dropped edges
* include policy always yields count=0
* WARNING log is emitted once per dropped edge (not just INFO)
* counter resets between successive build() calls on the same builder
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac-11.lan>
In `trace_decision_causality()`, potential heuristic causes were collected in a list inside a loop over the decision's entities. When two decisions shared multiple entities, the earlier decision was appended once per shared entity, resulting in duplicate "influences" chains and redundant recursive subtree traversals.
Deduplicate `potential_causes` by decision ID using an insertion-ordered dict so each cause is reported and traversed exactly once, preserving deterministic iteration order across runs.
- Add regression test `test_heuristic_cause_reported_once_per_shared_entity_pair`
- Fixes#1358
* feat(explorer): complete ARIA tab pattern for markdown viewer (#1117)
The Preview/Source controls exposed role="tablist"/role="tab"/aria-selected
but never connected to the content they switch, so assistive technology could
not tell which panel the tabs controlled, and the tablist was two separate Tab
stops with no arrow-key navigation.
Wire the full pattern:
- ids from useId(), matching the GraphWorkspace search combobox, so two viewers
mounted at once cannot collide on hardcoded ids
- aria-controls on both tabs, role="tabpanel" + aria-labelledby on the panel
- roving tabindex, so the tablist is a single Tab stop
- Arrow Left/Right with wrapping, plus Home/End, all preventDefault'd
Two decisions worth recording.
Activation is manual (arrows move focus, Enter/Space selects) rather than the
automatic activation the APG suggests by default. The APG permits manual
activation where switching panels is expensive, and here it is: activating
Preview re-runs the whole markdown parse, measured at 385ms for a 1000-row GFM
table and 1433ms at 2000 rows while profiling #1118. Automatic activation would
freeze the main thread on every arrow keypress.
The panel wraps all three render branches, including the empty state, and both
tabs point aria-controls at that one id. Only the active view is ever rendered,
so per-tab panel ids would leave the inactive tab referencing an element absent
from the DOM -- and scoping the panel to the two content branches would dangle
the reference for empty nodes.
No visible focus style was added: index.css already applies a global
:focus-visible ring and tabBtnStyle does not suppress it.
Five tests cover the wiring, the empty-state branch, roving tabindex and id
uniqueness; each was confirmed to fail against the previous component. Keyboard
behaviour is not reachable from the SSR-based suite, so it was verified
separately in headless Chromium: Tab enters the tablist on the selected tab,
ArrowRight moves focus without activating, Enter and Space activate, ArrowRight
wraps, and Home/End work. All 71 tests pass and lint is unchanged.
* feat(explorer): complete markdown viewer tab accessibility
Complete the Preview/Source ARIA tab pattern with collision-safe IDs, tabpanel wiring, roving tabindex, and manual keyboard activation.
Use ref-based focus tracking to prevent keyboard navigation from triggering Markdown re-parsing, and synchronize the roving tabindex after React renders.
Add regression coverage for keyboard navigation, wrapping, ARIA labelling, and focus-state invariants.
* fix(explorer): move focusedModeRef resource-reset out of render phase
The render-phase ref write (focusedModeRef.current = defaultMode inside
the activeResourceKey !== resourceKey guard) correctly reset the ref but
triggered a react-hooks/refs lint error: refs must not be written during
render.
Replace with a useLayoutEffect([resourceKey, defaultMode]) that fires
synchronously before paint after every resource/defaultMode change. This
achieves the same invariant — focusedModeRef is reset before the no-deps
tabIndex-correction effect reads it — while satisfying the linter.
Lint: npx eslint MarkdownContentViewer.tsx -> 0 errors
Tests: npm run test:graph-workspace -> 107/107 pass
---------
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
