* feat(milvus): add delete_vectors to MilvusStore
Adds delete_vectors(ids) so the ErasureCoordinator can erase embeddings
on a Milvus backend. Ids are escaped with _format_milvus_value before
building the delete expression, and the backend delete count is returned
so a delete that removed nothing is distinguishable from a failure.
* test(milvus): cover delete_vectors incl. erasure integration
Unit tests assert the single-id equality expression, multi-id in
expression, id escaping, empty-id noop, missing-collection and backend
error paths. Integration tests bind MilvusStore as a VectorStore backend
and assert ErasureCoordinator reports the vector leg erased.
* test(milvus): cover vector store delete facade
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
Replace the static biomedical category key in Explorer with a live node color legend derived from the active display graph, sharing the canvas's baseColor -> color -> theme fallback resolver.
- Preserve original semantic colors on focused display clones via `semanticBaseColor` so interaction styling (selection, path, neighbor highlights) does not bleed into the legend.
- Suppress the semantic legend while distance visualization modes are active.
- Refactor plugin fallback legend panels to use the shared legend builder with composite (group, color) keys.
- Consolidate duplicate `"test:graph-workspace"` scripts in `explorer/package.json`, recovering 26 previously shadowed tests across markdown and explorer capability suites.
- Add `tests/graphColorLegend.test.ts` to workspace test runs and wire `test:graph-legend-e2e` into `.github/workflows/ci.yml`.
- Closes#1479
* fix(kg): promote synthetic relation endpoints into the graph
Relation extraction synthesizes an UNKNOWN Entity for a relationship endpoint
that is absent from the NER entity list (metadata synthetic=True), but
GraphBuilder kept only the endpoint string, so the synthetic object never
reached the entity collection. GraphValidator then reported DANGLING_EDGE,
breaking the native NER -> relation -> graph -> validation pipeline.
The relation branch now collects synthetic endpoint entities and promotes them
into the entity collection (deduplicated by id, keeping confidence=0.8 and the
synthetic metadata). A new unknown_relation_endpoint option defaults to
include; set it to "reject" to drop relationships whose endpoints are
synthetic instead.
Closes#1463
* fix(kg): reconcile synthetic endpoints after resolution
Address Qodo review feedback on #1464:
- Read unknown_relation_endpoint through the nested "config" mapping the
orchestrator passes (GraphBuilder(config=...)), so the policy actually takes
effect end to end.
- Prefer a real entity over an earlier-promoted synthetic one with the same id,
deduplicating at build tail (unhashable ids are skipped defensively).
- Tag LLM triple endpoints that match no extracted entity and promote them too,
so triplet relationships no longer leave dangling edges.
- Log when the reject policy drops a relationship.
Closes#1463
* fix(kg): key dedup on both id and entity_id; tag HF triple endpoints
Follow-up hardening on #1464's synthetic-endpoint reconcile:
- Dedup promoted endpoints against both `id` and `entity_id` (the canonical
keys the builder itself recognizes), in both the relation branch and the
build-tail reconciliation. The build-tail pass now also skips unhashable
ids defensively.
- Tag HuggingFace REBEL endpoint texts that match no extracted entity, same
as the LLM typed path, so HF triplets no longer leave dangling edges.
- Tests: use a real `Triplet`, cover the `entity_id` dedup case, exercise
`merge_entities=True` explicitly (the prior test mislabeled the default),
and lock in the dict-relationship out-of-scope contract.
* fix(kg): guard metadata=None and gate endpoint promotion scan
_real_ids and the reconcile filter both did .get("metadata", {}).get(...),
which returns None (not {}) when the key is present with a None value and
raised on entities carrying metadata=None.
The synthetic-endpoint promotion block in _process_item rebuilt existing_ids
by scanning all_entities for every relationship, making build() quadratic on
dense relation inputs. Gate it on if synthetic_endpoints so ordinary
relationships pay nothing (6000+6000 no-synthetic builds in 0.018s vs 9.04s).
* config(kg): home unknown_relation_endpoint in build config and document it
The option now has a default in the per-module build method config
(kg_methods.build.unknown_relation_endpoint) and kg_usage.md documents both
values. __init__ folds a config= dict into the option mapping once, which also
fixes entity_resolution/conflict_detection being silently dropped on the
orchestrator path; every option reads the same way now.
* fix(extract): thread entities into HF REBEL triplet path
extract_triplets_huggingface read entities from kwargs.get("entities") which
is always empty on this path, making the synthetic-endpoint tag a no-op
comparison. Declare entities as a parameter so it matches the real NER list
TripletExtractor already forwards.
* fix(kg): finalize synthetic relation endpoint handling
- Add concrete Python API (kg_config.set_method_config) and YAML config
file example to kg_usage.md; the previous text referenced only an opaque
config-file key path with no runnable code.
- Add three targeted tests to test_dangling_synthetic_endpoint.py:
* test_real_entity_with_entity_id_only_wins_over_prior_synthetic: covers
the _real_ids dedup pass when the real entity carries its id under
entity_id only (no 'id' field).
* test_llm_relation_synthetic_endpoints_promoted: exercises the full
_parse_relation_result → Relation → GraphBuilder promotion path end-to-end.
* test_reject_policy_emits_warning_when_all_rels_dropped: verifies that
the 'all relationships were dropped' warning fires correctly when the
reject policy drops every relationship from a dict-style source.
- Add import pytest (required for caplog fixture used in the new test).
- Fix missing EOF newline in the test file.
* fix(kg): make reject-policy edge drops observable
The reject policy logged at INFO and returned no machine-readable count,
making it effectively silent at WARNING-level logging thresholds.
Issue #1463 requires that rejected relationships are 'reported explicitly'.
Changes:
- Elevate the per-rejection log from INFO to WARNING so it is visible
without debug logging enabled.
- Add self._rejected_relationships counter (reset per build() call,
incremented per dropped edge).
- Expose the count in graph['metadata']['rejected_relationships'] so
callers can check the outcome programmatically without log parsing.
- Document both observability mechanisms in kg_usage.md.
- Add four focused regression tests:
* rejected count matches number of dropped edges
* include policy always yields count=0
* WARNING log is emitted once per dropped edge (not just INFO)
* counter resets between successive build() calls on the same builder
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac-11.lan>
In `trace_decision_causality()`, potential heuristic causes were collected in a list inside a loop over the decision's entities. When two decisions shared multiple entities, the earlier decision was appended once per shared entity, resulting in duplicate "influences" chains and redundant recursive subtree traversals.
Deduplicate `potential_causes` by decision ID using an insertion-ordered dict so each cause is reported and traversed exactly once, preserving deterministic iteration order across runs.
- Add regression test `test_heuristic_cause_reported_once_per_shared_entity_pair`
- Fixes#1358
* feat(explorer): complete ARIA tab pattern for markdown viewer (#1117)
The Preview/Source controls exposed role="tablist"/role="tab"/aria-selected
but never connected to the content they switch, so assistive technology could
not tell which panel the tabs controlled, and the tablist was two separate Tab
stops with no arrow-key navigation.
Wire the full pattern:
- ids from useId(), matching the GraphWorkspace search combobox, so two viewers
mounted at once cannot collide on hardcoded ids
- aria-controls on both tabs, role="tabpanel" + aria-labelledby on the panel
- roving tabindex, so the tablist is a single Tab stop
- Arrow Left/Right with wrapping, plus Home/End, all preventDefault'd
Two decisions worth recording.
Activation is manual (arrows move focus, Enter/Space selects) rather than the
automatic activation the APG suggests by default. The APG permits manual
activation where switching panels is expensive, and here it is: activating
Preview re-runs the whole markdown parse, measured at 385ms for a 1000-row GFM
table and 1433ms at 2000 rows while profiling #1118. Automatic activation would
freeze the main thread on every arrow keypress.
The panel wraps all three render branches, including the empty state, and both
tabs point aria-controls at that one id. Only the active view is ever rendered,
so per-tab panel ids would leave the inactive tab referencing an element absent
from the DOM -- and scoping the panel to the two content branches would dangle
the reference for empty nodes.
No visible focus style was added: index.css already applies a global
:focus-visible ring and tabBtnStyle does not suppress it.
Five tests cover the wiring, the empty-state branch, roving tabindex and id
uniqueness; each was confirmed to fail against the previous component. Keyboard
behaviour is not reachable from the SSR-based suite, so it was verified
separately in headless Chromium: Tab enters the tablist on the selected tab,
ArrowRight moves focus without activating, Enter and Space activate, ArrowRight
wraps, and Home/End work. All 71 tests pass and lint is unchanged.
* feat(explorer): complete markdown viewer tab accessibility
Complete the Preview/Source ARIA tab pattern with collision-safe IDs, tabpanel wiring, roving tabindex, and manual keyboard activation.
Use ref-based focus tracking to prevent keyboard navigation from triggering Markdown re-parsing, and synchronize the roving tabindex after React renders.
Add regression coverage for keyboard navigation, wrapping, ARIA labelling, and focus-state invariants.
* fix(explorer): move focusedModeRef resource-reset out of render phase
The render-phase ref write (focusedModeRef.current = defaultMode inside
the activeResourceKey !== resourceKey guard) correctly reset the ref but
triggered a react-hooks/refs lint error: refs must not be written during
render.
Replace with a useLayoutEffect([resourceKey, defaultMode]) that fires
synchronously before paint after every resource/defaultMode change. This
achieves the same invariant — focusedModeRef is reset before the no-deps
tabIndex-correction effect reads it — while satisfying the linter.
Lint: npx eslint MarkdownContentViewer.tsx -> 0 errors
Tests: npm run test:graph-workspace -> 107/107 pass
---------
Co-authored-by: Pravit Ampapathini <pravit.amp@gmail.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
`GET /api/ontology/graph` used to fetch every node and edge for each relevant type with `limit=2**63-1`, normalize all of it, and only then check the result against `_MAX_ANALYSIS_NODES`. So the cap limited the response size, but not the amount of work or memory needed to build it.
Using the existing `paginate_nodes`/`paginate_edges` APIs wouldn't help much either. Each call still loads and normalizes the full matching set before slicing out a page, so paging through the results would repeat that cost for every page.
This adds `GraphSession.iter_nodes` and `iter_edges`, which yield matches one at a time instead of materializing everything up front. The graph endpoint now applies the ownership filter while scanning, so nodes from other ontologies are dropped before they're normalized or kept in memory. It also stops as soon as the requested ontology's own nodes or selected edges exceed the cap.
The ownership check intentionally happens before the cap check. That way, a graph containing a large number of nodes from unrelated ontologies can't make the requested ontology appear too large to open.
The old `candidates_by_id` map is gone as well. We now only retain nodes actually owned by the requested ontology instead of building payloads for every schema-typed node in the graph.
I also split the endpoint into `_known_ontology_uris`, `_collect_core_nodes`, and `_select_structure_edges` so the main flow is a little easier to follow.
External edge endpoints were previously hydrated one at a time with an `await` for each node. With a few thousand selected edges, that meant a few thousand sequential thread dispatches. They're now hydrated concurrently with `asyncio.gather` over `to_thread` calls, using the default thread pool. The final node and edge lists are sorted before building the response, so completion order doesn't affect the output.
This isn't fully lazy all the way down. `iter_edges` still gets the full result for an edge type from `ContextGraph.find_edges` before it starts yielding, and `iter_nodes` still snapshots and sorts all node IDs for a type first. A very large type such as `rdf:type` can therefore still do work proportional to its graph-wide size before the cap gets a chance to stop the scan.
What this change avoids is the more expensive part: normalizing every schema node in the graph and keeping the full candidate payload map in memory. Making `ContextGraph.find_edges` lazy would address the remaining issue, but that's a lower-level API used elsewhere and needs its own locking design, so that's better handled separately.
The old final cap check is removed too. The streaming collectors now raise as soon as their running count exceeds `_MAX_ANALYSIS_NODES`, so by the time collection finishes, `core_node_ids` and `selected_edges` are already guaranteed to be within the limit. There's a comment at that boundary documenting the invariant instead of keeping a redundant check around.
* fix(cli): dispatch reason run and store connect to real APIs
semantica reason run called Reasoner.run(), which does not exist --
the facade's API is infer_facts(facts, rules). The command now reads
nodes/relationships from the configured graph store as fact strings
(the same conventions Reasoner.add_fact applies to KG-style dicts),
loads --rules as a YAML list/mapping or plain-text lines, and reports
the inferred facts.
semantica store connect called get_graph_store_method(backend), which
is the (task, method_name) method registry, not a backend factory, so
it raised a TypeError before any connection attempt and failed
identically with or without valid credentials. It now builds the store
via GraphStore(backend=...) and probes connect(), so real
connectivity/auth errors surface.
Fixes#1354
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): use backend relationship keys, all labels, and honest engine dispatch in reason run
Review follow-ups on the reason run fact conversion:
- Relationships from every graph store backend carry start_node_id /
end_node_id; _graph_store_facts() read start_id / end_id, so each
relationship became TYPE(None, None) and relationship rules never
matched. Read the real keys and cover it with a rule that matches the
relationship fact.
- Emit a fact for every node label, not just the first, so rules on
secondary labels can match.
- reason run only executes the Reasoner facade's forward-chaining
inference; --engine values with different input paradigms (datalog,
sparql, abductive, deductive, graph) now fail with a clear message
(pointing SPARQL/Datalog at reason query) instead of reporting an
engine that never ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): harden reason run rule loading
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Sameers-MacBook-Air.local>
The v0.6.8 release run failed at the PyPI publish step:
Checking dist/semantica-0.6.8-py3-none-any.whl.sigstore.json: ERROR
InvalidDistribution: Unknown distribution format
pypa/gh-action-pypi-publish uploads everything under packages-dir
(default dist/) with no include/exclude filter, so once the Sigstore
signing step (added in #1329) started writing dist/*.sigstore.json
alongside the wheel/sdist, publish was broken for every release from
that point on - it just never ran, since v0.6.7 was tagged two days
before #1329 merged. Confirmed nothing was uploaded to PyPI before
failing (dist/*.whl checked and passed first; the sigstore.json file
failed validation before any upload began).
Fix: run pypi-publish immediately after the package build, before the
Sigstore/attest-build-provenance steps write anything else into dist/.
The GitHub Release upload (which needs the .sigstore.json files) still
runs after signing, unaffected by the reorder.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Qodo review on #1476 caught that CITATION.cff still declared v0.6.7 /
2026-08-28, which would have made GitHub's generated citation disagree
with the package metadata and CHANGELOG this same PR bumps to v0.6.8.
Not caught by the existing release-prep process since CITATION.cff was
only added in #1266, after that process was last documented.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bump version, cut CHANGELOG's Unreleased section into 0.6.8, backfill
changelog entries for the 96 PRs merged since v0.6.7 that were missing
from it, and refresh version-dependent references in README/docs.
This release exists primarily to ship the release-signing hardening
that landed in #1266/#1329 (SLSA build-provenance attestation +
Sigstore signing, with .sigstore.json bundles attached to the GitHub
Release) — v0.6.7 was tagged two days before that fix merged, so every
release OpenSSF Scorecard's Signed-Releases check has seen so far
predates it. Cutting v0.6.8 is what actually exercises the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Stop recommending `semantica kg build` as a workaround: it does not
persist to a configured graph store either (tracked in #1352), so
pointing users at it just traded one silent no-op for another.
- Make `--output` a real escape hatch instead of a silent bypass: it
now writes the ingested result to a .json/.jsonl/.csv file via the
existing `_write_result_output` helper, rather than being forwarded
as an ignored kwarg to the ingestor.
- Stop forwarding `--store`/`GRAPH_STORE_DEFAULT_BACKEND` into the
ingest call — they were already silently discarded downstream, so
they're now only used to decide whether to raise the "no CLI command
persists yet" error.
- Teach `_json_default` to expand dataclasses (e.g. `FileObject`) and
decode `bytes`, so `--output` produces real content instead of a
Python repr string.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(context): deep-copy store results in ErasureReceipt.to_dict()
to_dict() used dict(result), a shallow copy, so a nested dict in a
per-store result (e.g. a vector backend's backend_result) was shared by
reference between the serialized payload and the live receipt. Mutating
the payload for a user-facing response silently corrupted the audit
record. Copy each store result with copy.deepcopy() instead and document
the isolation. Regression test covers nested dict mutation isolation.
* fix(context): deep-copy erasure receipt store results
---------
Co-authored-by: BinarySpecter <185640875+BinarySpecter@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Sameers-MacBook-Air.local>
* docs(setup): tighten prose across quickstart, installation, and CLI setup guides
* fix(cli-setup): correct semantica-server default binding address
The server.py main() binds to 127.0.0.1 by default and documents
SEMANTICA_HOST as the override to expose beyond localhost.
The previous documentation (from main and carried through this PR)
incorrectly stated 0.0.0.0:8000, which would lead users to believe
the server is network-accessible by default.
Fix all three occurrences in cli-setup.md:
- Installed Commands table
- When to Use Each Command prose bullet
- REST server usage example code comment
---------
Co-authored-by: AutoHarness Bot <bot@autoharness.local>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* docs(resources): tighten prose across FAQ, learning paths, citation, and license
* docs(faq): cut two leftover chained/dramatic-pause colons
"unstructured data: documents, APIs, databases: into structured..."
chained two colons in one sentence, and "...reached a conclusion:
not just what it said" is the exact "X: not Y" pattern #1426 flags
for removal. Neither was touched by this PR's original pass.
* docs_check: raise Mintlify export timeout from 300s to 600s
The "Validate Documentation" CI check has been timing out at exactly
300s on this branch, on main, and on an unrelated branch in the same
window (3 consecutive retries here, all with the same "timed out
after 300 s" error and no other diagnostic output). Local runs finish
well under the limit, so this isn't a content problem — the export
step just has no headroom left as the docs site has grown to 27
modules. Doubling the timeout gives it room without masking real
export failures, which still fail immediately with their own error.
---------
Co-authored-by: AutoHarness Bot <bot@autoharness.local>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
"Claude Desktop, ..., Cline: 15 MCP tools exposed" chained a second
colon onto the "Integrations:" label colon, the exact pattern #1426
asks this cleanup pass to remove. Split into two sentences.
Adds Google ADK (Agent Development Kit) support to Semantica.
`semantica_kg_tools()` and `semantica_decision_tools()` expose entity/relation extraction, graph updates, and decision recording as ADK `FunctionTool`s. `SemanticaSessionService` implements ADK session storage on top of a Semantica `ContextGraph`, so session state, events, and knowledge graph data can live in the same graph instead of keeping sessions in memory.
There were also a number of dependency and CI fixes needed to get the integration working reliably. `google-adk` is pinned to a range that avoids the CI `websockets` conflict, the deprecated `pinecone-client` dependency was replaced with `pinecone`, Windows-only dependencies now have the appropriate platform markers, and `requirements-ci.txt` was regenerated to match. A `pip-audit` pass also required updates to `google-adk` and `starlette` for known CVEs.
Some unrelated `pyproject.toml` changes had slipped in during rebases, so the previous version, dependency bounds, `ingest-sap`/LangChain entries, and package-data settings were restored.
A few bugs in the initial ADK implementation were fixed during review:
* `extract_relations()` was calling `RelationExtractor.extract_entities()`, which doesn't exist on that extractor. The failure was being caught and returned in the tool's `error` field, leaving callers with an empty relation list. It now calls the correct extraction path.
* The repo's top-level `mcp/` package shadowed the third-party `mcp` package imported by `google.adk`, causing `google.adk` imports to fail from a normal repo checkout. The local package was moved to `semantica_mcp/mcp/`.
The MCP move needed a follow-up as well. `semantica/cli.py` and four existing tests were still importing from `mcp.*`, and the modules under `semantica_mcp/mcp/` still used the old absolute imports internally. `semantica_mcp` was also missing from the setuptools package include list and had no `__init__.py`, so it wouldn't have been included in an installed package. Those imports and packaging settings are fixed now.
The session service and ADK tools also had a few other problems:
* `list_sessions()` returned a plain list instead of ADK's `ListSessionsResponse`. The original import for that type doesn't work against the installed `google-adk` package, so it was silently falling back to a stub. `user_id` was also incorrectly required instead of being optional.
* Session node IDs were built by joining `app_name`, `user_id`, and `session_id` with unescaped colons, which allowed different identities to produce the same graph node ID. Each component is now encoded before joining.
* `kg_tools.py` and `decision_tools.py` each had their own lock registry and default graph instance. Sharing a graph between the two modules therefore didn't share the lock, and using both factories without an explicit graph produced two different defaults. The shared state now lives in one module used by both.
* `add_to_graph` had a `TypeError` compatibility fallback that couldn't succeed with the current `RelationExtractor` API and could hide the original extraction error. That fallback was removed.
* `append_event` persisted partial streaming events even though ADK's base session service skips them.
* `get_session()` ignored its `config` argument, so `num_recent_events` and `after_timestamp` had no effect.
* The async session-service methods performed synchronous graph scans while holding a `threading.RLock` on the event loop thread. That work now runs in worker threads with `asyncio.to_thread()` so a slow or contended graph operation doesn't block the loop.
---
Co-authored-by: Zohaib Hassnain [109234410+ZohaibHassan16@users.noreply.github.com](mailto:109234410+ZohaibHassan16@users.noreply.github.com)
Google Gemini, HuggingFace, DeepSeek, and Novita AI rows were left as
a stray colon with no notes. Fill them in to match the pattern of the
other rows, sourced from semantica/semantic_extract/providers.py.
* docs(ontology): document quality gate threshold semantics
The Ontology Quality Gate section (#1397) showed a thresholds={...}
example but never explained what min_coverage, max_errors,
max_warnings, or fail_on_warnings actually mean, or that
fail_on_warnings is a separate parameter rather than a thresholds
key. Add a concise defaults/semantics table, verified against
OntologyQualityGate.DEFAULT_THRESHOLDS and __init__ in quality_gate.py.
* docs(ontology): explain thresholds as prose instead of a table
A four-row table was heavier than this needed; each threshold's
meaning reads faster as two connected sentences.
