* fix(explorer): resolve ontology ownership on the backend
Which registered ontology owns an entity was decided twice: the backend
applies nested-namespace boundaries, while the Ontology Editor did a bare
prefix match. The two had already drifted, so a deep link to an entity in an
unregistered nested namespace selected the parent ontology whose /graph
response excludes that entity, and the selection silently failed.
/api/ontology/entity now returns owning_ontology, resolved with the same rule
the graph endpoint filters by, and the editor prefers it. The frontend
namespace guess stays as the fallback for a missing verdict, documented as
non-authoritative.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(explorer): keep the backend's no-owner verdict authoritative
Review follow-up: loadOntologyEntityOwner collapsed the backend's
explicit owning_ontology: null into undefined, re-activating the
namespace prefix guess for exactly the unregistered-nested-namespace
case this PR exists to fix. The owner verdict is now three-state
(owner / authoritative none / unavailable) and resolveEditorOntology
in the model suppresses inference on an authoritative none; only an
unavailable verdict may fall back. Model tests pin all three states.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(explorer): trust the backend to send owning_ontology
The Explorer bundle ships in the same wheel as the route that emits this
field, so the legacy-response branch could never run. Dropping it lets
the type say what the wire actually carries, leaving undefined to mean
only what it should: the request failed.
Note why the endpoint derives its ontology-URI set inline rather than
calling _known_ontology_uris, so the next reader does not consolidate a
graph scan back in.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(explorer): stop the registry default overriding a no-owner verdict
resolveEditorOntology returned string | undefined, so the caller wrote
`resolved || entries[0]?.uri` and an authoritative "nothing owns this
entity" fell straight through to an arbitrary registry entry. When that
entry happened to be the parent, the deep link opened the parent whose
graph excludes the entity — the bug the verdict exists to prevent. Only
the ordering of the registry in the earlier test hid it.
It now returns a union: unowned and unresolved both mean "no ontology to
open" but the editor treats them oppositely, so collapsing them with ||
is a type error rather than a silent regression. An unowned entity is
reported on the canvas instead of quietly opening the wrong ontology.
Also:
- /entity resolves ownership through _known_ontology_uris, the same
helper /graph uses, instead of deriving it from a get_nodes scan capped
at 999,999. Past that cap the set was silently truncated and the two
endpoints could disagree about who owns a node.
- _resolve_owning_ontology does one pass over the candidates rather than
one pass per candidate, each rescanning the whole set: 516us -> 9us at
50 ontologies, 125ms -> 138us at 800, same answers throughout. A test
pins it against _node_belongs_to_ontology so the hand-rolled version
cannot drift from the membership rule it has to mirror.
- An explicit scheme_uri is honoured even when the registry does not list
it, on both sides. Discarding it and guessing by namespace answered a
question nobody asked; an unregistered owner now surfaces as an
explicit error from /graph instead.
- A missing owning_ontology field reads as "no verdict", not as the
authoritative "nothing owns this". That claim now suppresses selection
outright, so it must not be inferred from an absent field.
Co-Authored-By: Claude Code <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
* fix(vector-store): prevent in-memory vector id reuse
* fix(vector-store): synchronize in-memory mutations
* fix(vector-store): clarify no-silent-overwrite guarantee for in-memory ID generation
The while-loop in store_vectors() already prevents any generated candidate
from landing on a live key: it breaks only when the candidate is absent
from both pre_existing (the live store at lock-entry) and within_batch
(IDs chosen earlier in the same call).
This commit:
- Renames 'existing' to 'pre_existing' and introduces 'within_batch' to
make the two-level de-dupe explicit and self-documenting.
- Tightens the break condition to 'not in pre_existing and not in
within_batch' so within-batch duplicates are also guarded.
- Adds test_auto_generated_id_never_silently_overwrites_live_vector:
stores 5, deletes 3, stores 2 more and asserts none of the new IDs
land on a surviving vec_N.
- Adds test_collision_detection_no_silent_overwrite_even_with_corrupted_counter:
rewinds _next_id to 0 with live vectors present and proves the loop
finds a free slot without overwriting either existing vector or its
metadata — the no-silent-overwrite invariant holds even under counter
corruption.
The skip-over-occupied-key behaviour for caller-inserted vec_N IDs is
intentional and preserved (test_counter_skips_explicit_vec_n_ids), matching
FAISSStore's identical pattern.
Closes reviewer finding: 'Vector collisions do not fail writes'.
* fix(vector-store): protect concurrent in-memory access
---------
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Add a deterministic, ontology-based schema validator as a sibling to the
confidence-based ExtractionValidator. Both validators implement the same
validate_entities() and validate_relations() interfaces and return the
shared ValidationResult structure, enabling orthogonal composition across
extraction confidence and ontological conformance.
Key Additions & Behaviors:
- ExtractionSchema: Read-only view over domain ontologies (allowed concepts
and predicates with optional domain/range constraints). Supports loading
from dictionary representations (generate_ontology() or OntologyData from
OntologyIngestor) via ExtractionSchema.from_ontology(), and from OWL/Turtle
files/strings via ExtractionSchema.from_owl().
- SchemaValidator: Verifies entity labels against schema concepts and
validates relation predicates against defined domain/range constraints.
Provides filter_by_schema() and filter_relations_by_schema() to extract
conforming subsets without mutating source data.
- Domain/Range Wildcarding: Treats owl:Thing (and unqualified Thing) as
unconstrained wildcards so fallback types do not reject valid endpoints.
- Constructor Parity: Folds relation domain/range endpoint types into the
concepts set across both from_ontology() and from_owl() to ensure logical
consistency.
- Ingest Pipeline Interop: Automatically unwraps OntologyData-like objects
via duck-typing on .data.
- Resilient Endpoint Resolution: Guards malformed relations missing subject
or object endpoints without raising AttributeError, and prefers rdfs:label
over URI suffixes while supporting both owl:Class and rdfs:Class.
Part of #1510.
Repair broken GraphML and Parquet export handling in the MCP
`export_graph` tool.
Previously, both formats failed in the MCP handler: GraphML referenced a
non-existent `GraphMLExporter` class and passed unformatted graph objects,
while Parquet passed raw `ContextGraph` instances to `export()`, causing
runtime type errors and producing empty responses.
Key changes:
- Route GraphML export through `GraphExporter(format="graphml")` via
`export_knowledge_graph()` with automatic `TemporaryDirectory` cleanup.
- Harden GraphML XML generation in `GraphExporter`: declare missing
`label` and `confidence` keys with `for="all"` scope and use
`xml.sax.saxutils` (`quoteattr`, `escape`) to properly escape XML
attributes and node text.
- Route Parquet export through `ParquetExporter.export_knowledge_graph()`,
returning generated `.parquet` files base64-encoded over the JSON MCP
transport.
- Gracefully handle missing `pyarrow` dependencies with structured error
responses.
- Preserve existing JSON, CSV, and RDF export behavior.
- Add comprehensive regression tests in
`tests/test_mcp_package_export_graphml_parquet.py`.
Move the static `GET /api/decisions/causal-distance` endpoint above the
dynamic `GET /api/decisions/{decision_id}` route in the Explorer router.
Starlette evaluates routes sequentially in declaration order. Because
`/{decision_id}` was previously defined first, requests targeting
`/api/decisions/causal-distance` were captured by the parameter route
with `decision_id="causal-distance"`, causing the endpoint to query the
graph for a non-existent decision and return 404.
Changes:
- Reorder `causal_distance` above `get_decision` in
`semantica/explorer/routes/decisions.py` and add a comment guard
against future route-ordering regressions.
- Add regression test suite in
`tests/explorer/test_decisions_causal_distance_route.py` verifying
proper distance reporting, multi-hop traversal, and that unmatched
decision IDs still return 404.
Closes#1531
* fix(parse): stop infinite recursion in default method dispatch
parse_document and its five sibling dispatchers (parse_web_content,
parse_structured_data, parse_email, parse_code, parse_media) were
registered in the method registry under their own task's "default"
name. Every dispatcher begins with method_registry.get(<task>, method),
so calling e.g. parse_document(file, method="default") found itself in
the registry and re-entered infinitely until RecursionError --
`semantica parse <any file>` crashed before any parsing ran.
Drop the six self-registrations. "default" remains the built-in code
path; users can still register their own "default" (or any other name)
to override it, and the existing "docling" registration is unaffected.
Verified: `semantica parse demo.pdf` now parses successfully (was
RecursionError). No repo code or tests consume
get_parse_method("document", "default"), so removing the entries
changes no behavior besides fixing the crash.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(deps): make resolution satisfiable on Python 3.9 and add parse-pdf extra
Dependency fixes so `uv lock` / `pip install semantica[all]` resolves
across the supported matrix (3.9-3.12):
- requires-python >=3.8 was unsatisfiable (numpy>=2.0.2 needs >=3.9) and
3.9.0/3.9.1 can never resolve (every cryptography release excludes
them) -> bump to >=3.9.2 and drop the 3.8 classifier.
- Split recently-raised floors that dropped 3.9 into marker pairs
(3.9-capped / 3.10-unconstrained), following the pattern already used
for scikit-learn/requests/etc.: pyarrow extras (>=24 needs 3.10),
pre-commit 4.6, snowflake-connector 4.6, fastapi 0.129 + starlette 0.53
(older fastapi caps starlette<0.53).
- Gate docling, litellm (its only 3.9 release pins
python-dotenv==1.0.1, conflicting with the >=1.2.1 core floor), and
crewai (no un-yanked 3.9 release) to >=3.10.
Also add a parse-pdf extra: the default PDFParser requires pdfplumber,
but no extra installed it, so `semantica parse file.pdf` failed on a
default install. Follows the parse-docling convention.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(vector-store): make the qdrant backend usable through VectorStore
The qdrant backend could not be used at all through the VectorStore
facade in 0.6.8 - every path raised:
1. store_vectors() only dispatched to backend add/add_vectors methods;
QdrantStore exposes insert_vectors, so vs.store() raised
NotImplementedError. Add an insert_vectors branch (uuid-generated
ids, metadata -> payloads).
2. QdrantStore required an explicit create_collection() before any read
or write, unlike FAISSStore's automatic index creation. Lazily attach
the configured collection (config key "collection", default
"semantica_default") on first insert/search, reusing an existing one.
3. search_points() called client.search(), removed from qdrant-client in
favor of query_points() - use it when available, fall back otherwise.
4. store() silently dropped plain-string documents (it only extracted
doc.metadata), so payloads lost the source text; keep them under
payload "document".
Verified end-to-end against a Qdrant 5 server (docker): embed ->
store -> semantic search returns correctly ranked results whose payloads
carry the original documents.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(vector-store): address Qodo review findings on the qdrant write path
Four findings from the Qodo review of #1508:
1. store_vectors() returned QdrantStore.insert_vectors()' upsert status
dict although the facade promises the stored vector IDs (decision
storage indexes the result at position 0). Return the generated or
caller-supplied IDs after a successful insert instead.
2. insert_vectors() pairs points with zip(vectors, ids), so a shorter
non-empty id list silently dropped the unpaired vectors while the
completion message still reported the full batch as inserted. Reject
the mismatch with ValidationError before any write.
3. _ensure_default_collection() looked up the legacy "collection" config
key, so the documented collection_name=... option was ignored and
lazy init always fell back to semantica_default. Prefer
collection_name, keep "collection" as an alias.
4. The new parse-pdf extra was missing from both aggregate "all"
bundles, so semantica[all] still shipped without pdfplumber and the
default PDFParser raised ProcessingError on first use.
Also removes the now-stale strict xfail for qdrant's write dispatch in
test_backend_facade_contract.py — that marker exists precisely to fail
once the wiring lands.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* test(vector-store): migrate qdrant mocks to the query_points API
The qdrant search path now prefers client.query_points() (qdrant-client
removed client.search), but these tests still mocked the legacy call.
A MagicMock exposes query_points too, so the code took the modern path,
read .points off an unconfigured mock, and all three tests failed on
the branch. Return the hits in response.points as the real client does.
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(ci): regenerate requirements-ci.txt for the parse-pdf extra
The CI lockfile check re-resolves pyproject.toml --extra all and diffs
the pinned versions against requirements-ci.txt. The parse-pdf extra
added pdfplumber (+pdfminer-six) to the 'all' bundle without refreshing
the lockfile, so the diff failed and the build job exited 1.
Regenerated with the command from the file header. Only the two new
pins and their 'via' comments changed - every other version is
identical. Verified locally with the CI's own check command (diff of
pkg==ver lines exits clean).
Co-Authored-By: Claude Code <noreply@anthropic.com>
* fix(tests): update stale default-method-registration assertion
test_parse_methods_dynamic_default_resolution asserted that "default"
resolves through the registry to parse_document, which was true only
because of the self-registration this PR removes (it's what caused the
infinite recursion in the first place). Update the assertion to match
the intended post-fix state: "default" is not registered at all, and
falls through to the built-in dispatch path unconditionally.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: yanyu <yanyu@polixir.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
test_xml_parser_handles_comments and test_public_api_ingestor_handles_xml_comments
called XMLParser(engine="lxml") / forced the lxml fallback path unconditionally,
but the core-only CI step installs from base-deps.txt, which no longer bundles
lxml or defusedxml under this PR's slim dependency layout. Skip the lxml-only
assertions when lxml/defusedxml aren't installed, while still exercising the
always-available etree path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Route `semantica mcp call` and `semantica mcp list-tools` through the
canonical `semantica_mcp.mcp.server` packaged module instead of the
nonexistent `MCPSession` import.
Previously, `semantica mcp call` imported `MCPSession` from
`semantica_mcp.mcp.session`, which never defined it, causing every
call to fail with "MCP module not available". Meanwhile,
`semantica mcp list-tools` inspected `semantica_mcp.mcp.tools.__all__`,
which evaluated to `["TOOL_DEFINITIONS"]` and printed a single tool
named "TOOL_DEFINITIONS".
Key changes:
- Implement `semantica_mcp.mcp.server.call_tool(name, arguments)` as the
shared in-process entry point used by both the CLI and the JSON-RPC
`tools/call` handler, ensuring both surfaces expose identical tools.
- Add `UnknownToolError` to distinguish missing tools (JSON-RPC -32601)
from handler-level `KeyError` exceptions (JSON-RPC -32603).
- Update `semantica mcp list-tools` to source tool names directly from
`TOOL_DEFINITIONS` in `semantica_mcp.mcp.tools`.
- Validate `--args` payloads and reject non-object JSON inputs (arrays,
primitives, null) with a clean `ClickException`.
- Update `tests/test_mcp_stdio_roundtrip.py` to spawn `semantica_mcp.mcp`
instead of the pre-relocation `mcp` module, restoring clean stdio
framing tests.
Fixes#1355
- Remove thinc direct constraint from nlp-spacy in pyproject.toml and update README/CHANGELOG
- Raise ProcessingError with install hint when UMAP is requested but unavailable in EmbeddingVisualizer
- Guard PCA and TSNE dimensionality reducers against None with actionable error messages
- Prevent keyword collisions in EmbeddingVisualizer dimensionality reduction (_reduce_dimensions)
- Add core-only install & test step to .github/workflows/ci.yml using base-deps.txt
- Prevent AttributeError on module load in repo_ingestor.py and xml_ingestor.py when optional dependencies are absent
- Make TOML loading in tests/test_issue_1513_slim_core.py portable across Python versions via UTF-8 text decode and loads
- Expand slim-core test suite to 17 test cases covering 2D/3D projections, core imports, and options handling
- Reduce direct core dependencies in pyproject.toml from 44 to 22
- Move heavy and specialized packages into modular optional extras:
* models-huggingface: torch, transformers
* embeddings-local: sentence-transformers, fastembed, onnxruntime, tokenizers
* nlp-spacy: spacy, thinc
* viz: matplotlib, seaborn, plotly, ipywidgets, umap-learn (expanded)
* media: librosa, opencv-python
* vectorstore-faiss: faiss-cpu
* documents: python-docx, openpyxl, lxml, beautifulsoup4
* ingest-git: GitPython
* graph-embeddings: gensim
- Update semantica[all] and semantica[vectorstore-all] to encompass all extras
- Ensure lazy parser construction (DOCXParser, ExcelParser, HTMLParser, XMLParser) without error on __init__(), failing only inside .parse() with clear hints
- Add stdlib xml.etree fallback in XMLParser when lxml is missing
- Guard unguarded matplotlib imports in EmbeddingVisualizer and OntologyVisualizer
- Standardize user-facing error messages to point to pip install 'semantica[extra]'
- Bump version to 0.7.0 in pyproject.toml, semantica/__init__.py, and CITATION.cff
- Add migration notes to README.md and CHANGELOG.md
- Recompile CI and Docker requirements lockfiles
- Add dedicated test suite tests/test_issue_1513_slim_core.py
Adds native vector deletion support for FAISS Flat indices via remove_ids
and IDSelectorBatch (Closes#1374):
- Adds FAISSIndex.delete_vectors() and FAISSStore.delete_vectors() to
translate external string IDs to sequential internal positions, remove
vectors via remove_ids, and keep vector_ids and metadata parallel with
the compacted index.
- Auto-saves index and metadata sidecar after deletion when the store was
loaded from disk via load_index(), skipping redundant disk writes for
no-op deletions.
- Rejects IVF and HNSW index deletions with NotImplementedError: IVF does
not compact internal labels upon remove_ids (causing search/offset
desynchronization), and HNSW lacks remove_ids entirely. Both cleanly
map to STATUS_UNSUPPORTED in ErasureCoordinator.
Robust ID Management & Invariant Hardening:
- Generates default IDs with a monotonic candidate-check loop against
existing vector_ids, eliminating collisions and metadata overwrites
when explicit vec_N IDs coexist.
- Persists next_id in the .meta.json sidecar and clamps on load() to
max(persisted, max(vec_N) + 1), preventing stale sidecars from
re-introducing collisions across restarts or legacy migrations.
- Guards against FAISS -1 sentinels (0 <= idx < len(vector_ids)) in
search results when k > ntotal, preventing negative index wrapping
to vector_ids[-1].
- Replaces bare post-delete assert with an explicit ProcessingError check
that survives python -O.
Adds 43 comprehensive tests in test_faiss_delete_vectors.py covering
deletion, search exclusion, metadata cleanup, persistence roundtrips,
IVF/HNSW rejection, ErasureCoordinator receipts, and mock-spied
deterministic no-op saves.
Implements four semantic retrieval tools on top of VectorStore wired
into the MCP server tool registry (Closes#1235):
- store_document: Chunks content with a configurable sliding window
(1000/200 default) and attaches full provenance metadata to every
chunk (chunk_id, source, authority, version, hash, status, offsets,
and project). Re-storing identical content under the same
(source, version) is a no-op keyed on content hash to skip redundant
re-embedding.
- retrieve_context: Embeds natural-language queries, ranks scored
chunks with provenance (top_k capped at 10; project filter support),
and attaches 1-hop graph relationships for hit sources from ContextGraph.
- update_document: Replaces stored content for (source, version). Returns
not_found if the document does not exist, and snapshots existing rows
for atomic rollback if writing replacement chunks fails.
- remove_document: Deletes all chunks matching (source, version) and
returns not_found if the document does not exist.
Architecture & Session Management:
- Adds lazy session singletons get_embedder() and get_vector_store()
in semantica_mcp.mcp.session.
- Stores derive dimension directly from the active embedder to keep
indexing and query spaces aligned.
- Restricts backends to inmemory (default) and sqlite (requires
SEMANTICA_VECTOR_DB_PATH); backends lacking metadata-scoped delete
fail fast on startup.
- In-memory updates and removals safely rebuild the store to prevent
vector ID collisions (#1029).
- Fails fast on startup if SEMANTICA_VECTOR_PATH is corrupt or dimension
mismatched, preventing empty stores from overwriting existing corpora.
Hardening & Provenance Protections:
- System-owned provenance: user metadata is additive and cannot overwrite
reserved identity or provenance fields.
- Document size capped at 10,000 chunks to prevent stale chunk retention
against persistent metadata limits.
- Mutation responses return an explicit persisted flag.
Includes a comprehensive test suite in tests/test_mcp_semantic_retrieval.py
using a deterministic FakeEmbedder for network-free, hermetic execution.
The global --json flag documents "machine-readable JSON to stdout;
errors to stderr", but commands failing inside _run_with_error_handling()
rendered a Rich error panel through the module-level Console, which
writes to stdout. Any pipeline or automation parsing stdout as JSON
broke on the first failure:
$ semantica --json mcp call extract_entities --args '[1,2' 1>/tmp/out 2>/tmp/err
exit=1 stdout_bytes=634 stderr_bytes=0
Branch in _show_error_card(), the single renderer that callers of
_run_with_error_handling() route through. When JSON mode is active
(global --json on the CLI context or a subcommand's local --json flag,
uniformly named local_json), emit a structured {"error": ..., "type": ...}
JSON line to stderr and leave stdout untouched. Exit codes and interactive
Rich panels in non-JSON mode remain unchanged.
Add regression tests verifying stdout remains empty and stderr produces
parseable JSON under both global and local --json flag scopes.
Split out of #1368 as an independent fix.
* feat(milvus): add delete_vectors to MilvusStore
Adds delete_vectors(ids) so the ErasureCoordinator can erase embeddings
on a Milvus backend. Ids are escaped with _format_milvus_value before
building the delete expression, and the backend delete count is returned
so a delete that removed nothing is distinguishable from a failure.
* test(milvus): cover delete_vectors incl. erasure integration
Unit tests assert the single-id equality expression, multi-id in
expression, id escaping, empty-id noop, missing-collection and backend
error paths. Integration tests bind MilvusStore as a VectorStore backend
and assert ErasureCoordinator reports the vector leg erased.
* test(milvus): cover vector store delete facade
---------
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac.lan>
* fix(kg): promote synthetic relation endpoints into the graph
Relation extraction synthesizes an UNKNOWN Entity for a relationship endpoint
that is absent from the NER entity list (metadata synthetic=True), but
GraphBuilder kept only the endpoint string, so the synthetic object never
reached the entity collection. GraphValidator then reported DANGLING_EDGE,
breaking the native NER -> relation -> graph -> validation pipeline.
The relation branch now collects synthetic endpoint entities and promotes them
into the entity collection (deduplicated by id, keeping confidence=0.8 and the
synthetic metadata). A new unknown_relation_endpoint option defaults to
include; set it to "reject" to drop relationships whose endpoints are
synthetic instead.
Closes#1463
* fix(kg): reconcile synthetic endpoints after resolution
Address Qodo review feedback on #1464:
- Read unknown_relation_endpoint through the nested "config" mapping the
orchestrator passes (GraphBuilder(config=...)), so the policy actually takes
effect end to end.
- Prefer a real entity over an earlier-promoted synthetic one with the same id,
deduplicating at build tail (unhashable ids are skipped defensively).
- Tag LLM triple endpoints that match no extracted entity and promote them too,
so triplet relationships no longer leave dangling edges.
- Log when the reject policy drops a relationship.
Closes#1463
* fix(kg): key dedup on both id and entity_id; tag HF triple endpoints
Follow-up hardening on #1464's synthetic-endpoint reconcile:
- Dedup promoted endpoints against both `id` and `entity_id` (the canonical
keys the builder itself recognizes), in both the relation branch and the
build-tail reconciliation. The build-tail pass now also skips unhashable
ids defensively.
- Tag HuggingFace REBEL endpoint texts that match no extracted entity, same
as the LLM typed path, so HF triplets no longer leave dangling edges.
- Tests: use a real `Triplet`, cover the `entity_id` dedup case, exercise
`merge_entities=True` explicitly (the prior test mislabeled the default),
and lock in the dict-relationship out-of-scope contract.
* fix(kg): guard metadata=None and gate endpoint promotion scan
_real_ids and the reconcile filter both did .get("metadata", {}).get(...),
which returns None (not {}) when the key is present with a None value and
raised on entities carrying metadata=None.
The synthetic-endpoint promotion block in _process_item rebuilt existing_ids
by scanning all_entities for every relationship, making build() quadratic on
dense relation inputs. Gate it on if synthetic_endpoints so ordinary
relationships pay nothing (6000+6000 no-synthetic builds in 0.018s vs 9.04s).
* config(kg): home unknown_relation_endpoint in build config and document it
The option now has a default in the per-module build method config
(kg_methods.build.unknown_relation_endpoint) and kg_usage.md documents both
values. __init__ folds a config= dict into the option mapping once, which also
fixes entity_resolution/conflict_detection being silently dropped on the
orchestrator path; every option reads the same way now.
* fix(extract): thread entities into HF REBEL triplet path
extract_triplets_huggingface read entities from kwargs.get("entities") which
is always empty on this path, making the synthetic-endpoint tag a no-op
comparison. Declare entities as a parameter so it matches the real NER list
TripletExtractor already forwards.
* fix(kg): finalize synthetic relation endpoint handling
- Add concrete Python API (kg_config.set_method_config) and YAML config
file example to kg_usage.md; the previous text referenced only an opaque
config-file key path with no runnable code.
- Add three targeted tests to test_dangling_synthetic_endpoint.py:
* test_real_entity_with_entity_id_only_wins_over_prior_synthetic: covers
the _real_ids dedup pass when the real entity carries its id under
entity_id only (no 'id' field).
* test_llm_relation_synthetic_endpoints_promoted: exercises the full
_parse_relation_result → Relation → GraphBuilder promotion path end-to-end.
* test_reject_policy_emits_warning_when_all_rels_dropped: verifies that
the 'all relationships were dropped' warning fires correctly when the
reject policy drops every relationship from a dict-style source.
- Add import pytest (required for caplog fixture used in the new test).
- Fix missing EOF newline in the test file.
* fix(kg): make reject-policy edge drops observable
The reject policy logged at INFO and returned no machine-readable count,
making it effectively silent at WARNING-level logging thresholds.
Issue #1463 requires that rejected relationships are 'reported explicitly'.
Changes:
- Elevate the per-rejection log from INFO to WARNING so it is visible
without debug logging enabled.
- Add self._rejected_relationships counter (reset per build() call,
incremented per dropped edge).
- Expose the count in graph['metadata']['rejected_relationships'] so
callers can check the outcome programmatically without log parsing.
- Document both observability mechanisms in kg_usage.md.
- Add four focused regression tests:
* rejected count matches number of dropped edges
* include policy always yields count=0
* WARNING log is emitted once per dropped edge (not just INFO)
* counter resets between successive build() calls on the same builder
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Mac-11.lan>
In `trace_decision_causality()`, potential heuristic causes were collected in a list inside a loop over the decision's entities. When two decisions shared multiple entities, the earlier decision was appended once per shared entity, resulting in duplicate "influences" chains and redundant recursive subtree traversals.
Deduplicate `potential_causes` by decision ID using an insertion-ordered dict so each cause is reported and traversed exactly once, preserving deterministic iteration order across runs.
- Add regression test `test_heuristic_cause_reported_once_per_shared_entity_pair`
- Fixes#1358
`GET /api/ontology/graph` used to fetch every node and edge for each relevant type with `limit=2**63-1`, normalize all of it, and only then check the result against `_MAX_ANALYSIS_NODES`. So the cap limited the response size, but not the amount of work or memory needed to build it.
Using the existing `paginate_nodes`/`paginate_edges` APIs wouldn't help much either. Each call still loads and normalizes the full matching set before slicing out a page, so paging through the results would repeat that cost for every page.
This adds `GraphSession.iter_nodes` and `iter_edges`, which yield matches one at a time instead of materializing everything up front. The graph endpoint now applies the ownership filter while scanning, so nodes from other ontologies are dropped before they're normalized or kept in memory. It also stops as soon as the requested ontology's own nodes or selected edges exceed the cap.
The ownership check intentionally happens before the cap check. That way, a graph containing a large number of nodes from unrelated ontologies can't make the requested ontology appear too large to open.
The old `candidates_by_id` map is gone as well. We now only retain nodes actually owned by the requested ontology instead of building payloads for every schema-typed node in the graph.
I also split the endpoint into `_known_ontology_uris`, `_collect_core_nodes`, and `_select_structure_edges` so the main flow is a little easier to follow.
External edge endpoints were previously hydrated one at a time with an `await` for each node. With a few thousand selected edges, that meant a few thousand sequential thread dispatches. They're now hydrated concurrently with `asyncio.gather` over `to_thread` calls, using the default thread pool. The final node and edge lists are sorted before building the response, so completion order doesn't affect the output.
This isn't fully lazy all the way down. `iter_edges` still gets the full result for an edge type from `ContextGraph.find_edges` before it starts yielding, and `iter_nodes` still snapshots and sorts all node IDs for a type first. A very large type such as `rdf:type` can therefore still do work proportional to its graph-wide size before the cap gets a chance to stop the scan.
What this change avoids is the more expensive part: normalizing every schema node in the graph and keeping the full candidate payload map in memory. Making `ContextGraph.find_edges` lazy would address the remaining issue, but that's a lower-level API used elsewhere and needs its own locking design, so that's better handled separately.
The old final cap check is removed too. The streaming collectors now raise as soon as their running count exceeds `_MAX_ANALYSIS_NODES`, so by the time collection finishes, `core_node_ids` and `selected_edges` are already guaranteed to be within the limit. There's a comment at that boundary documenting the invariant instead of keeping a redundant check around.
* fix(cli): dispatch reason run and store connect to real APIs
semantica reason run called Reasoner.run(), which does not exist --
the facade's API is infer_facts(facts, rules). The command now reads
nodes/relationships from the configured graph store as fact strings
(the same conventions Reasoner.add_fact applies to KG-style dicts),
loads --rules as a YAML list/mapping or plain-text lines, and reports
the inferred facts.
semantica store connect called get_graph_store_method(backend), which
is the (task, method_name) method registry, not a backend factory, so
it raised a TypeError before any connection attempt and failed
identically with or without valid credentials. It now builds the store
via GraphStore(backend=...) and probes connect(), so real
connectivity/auth errors surface.
Fixes#1354
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): use backend relationship keys, all labels, and honest engine dispatch in reason run
Review follow-ups on the reason run fact conversion:
- Relationships from every graph store backend carry start_node_id /
end_node_id; _graph_store_facts() read start_id / end_id, so each
relationship became TYPE(None, None) and relationship rules never
matched. Read the real keys and cover it with a rule that matches the
relationship fact.
- Emit a fact for every node label, not just the first, so rules on
secondary labels can match.
- reason run only executes the Reasoner facade's forward-chaining
inference; --engine values with different input paradigms (datalog,
sparql, abductive, deductive, graph) now fail with a clear message
(pointing SPARQL/Datalog at reason query) instead of reporting an
engine that never ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): harden reason run rule loading
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Sameers-MacBook-Air.local>
- Stop recommending `semantica kg build` as a workaround: it does not
persist to a configured graph store either (tracked in #1352), so
pointing users at it just traded one silent no-op for another.
- Make `--output` a real escape hatch instead of a silent bypass: it
now writes the ingested result to a .json/.jsonl/.csv file via the
existing `_write_result_output` helper, rather than being forwarded
as an ignored kwarg to the ingestor.
- Stop forwarding `--store`/`GRAPH_STORE_DEFAULT_BACKEND` into the
ingest call — they were already silently discarded downstream, so
they're now only used to decide whether to raise the "no CLI command
persists yet" error.
- Teach `_json_default` to expand dataclasses (e.g. `FileObject`) and
decode `bytes`, so `--output` produces real content instead of a
Python repr string.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(context): deep-copy store results in ErasureReceipt.to_dict()
to_dict() used dict(result), a shallow copy, so a nested dict in a
per-store result (e.g. a vector backend's backend_result) was shared by
reference between the serialized payload and the live receipt. Mutating
the payload for a user-facing response silently corrupted the audit
record. Copy each store result with copy.deepcopy() instead and document
the isolation. Regression test covers nested dict mutation isolation.
* fix(context): deep-copy erasure receipt store results
---------
Co-authored-by: BinarySpecter <185640875+BinarySpecter@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: Sameer Kadam <sameerkadam@Sameers-MacBook-Air.local>
Adds Google ADK (Agent Development Kit) support to Semantica.
`semantica_kg_tools()` and `semantica_decision_tools()` expose entity/relation extraction, graph updates, and decision recording as ADK `FunctionTool`s. `SemanticaSessionService` implements ADK session storage on top of a Semantica `ContextGraph`, so session state, events, and knowledge graph data can live in the same graph instead of keeping sessions in memory.
There were also a number of dependency and CI fixes needed to get the integration working reliably. `google-adk` is pinned to a range that avoids the CI `websockets` conflict, the deprecated `pinecone-client` dependency was replaced with `pinecone`, Windows-only dependencies now have the appropriate platform markers, and `requirements-ci.txt` was regenerated to match. A `pip-audit` pass also required updates to `google-adk` and `starlette` for known CVEs.
Some unrelated `pyproject.toml` changes had slipped in during rebases, so the previous version, dependency bounds, `ingest-sap`/LangChain entries, and package-data settings were restored.
A few bugs in the initial ADK implementation were fixed during review:
* `extract_relations()` was calling `RelationExtractor.extract_entities()`, which doesn't exist on that extractor. The failure was being caught and returned in the tool's `error` field, leaving callers with an empty relation list. It now calls the correct extraction path.
* The repo's top-level `mcp/` package shadowed the third-party `mcp` package imported by `google.adk`, causing `google.adk` imports to fail from a normal repo checkout. The local package was moved to `semantica_mcp/mcp/`.
The MCP move needed a follow-up as well. `semantica/cli.py` and four existing tests were still importing from `mcp.*`, and the modules under `semantica_mcp/mcp/` still used the old absolute imports internally. `semantica_mcp` was also missing from the setuptools package include list and had no `__init__.py`, so it wouldn't have been included in an installed package. Those imports and packaging settings are fixed now.
The session service and ADK tools also had a few other problems:
* `list_sessions()` returned a plain list instead of ADK's `ListSessionsResponse`. The original import for that type doesn't work against the installed `google-adk` package, so it was silently falling back to a stub. `user_id` was also incorrectly required instead of being optional.
* Session node IDs were built by joining `app_name`, `user_id`, and `session_id` with unescaped colons, which allowed different identities to produce the same graph node ID. Each component is now encoded before joining.
* `kg_tools.py` and `decision_tools.py` each had their own lock registry and default graph instance. Sharing a graph between the two modules therefore didn't share the lock, and using both factories without an explicit graph produced two different defaults. The shared state now lives in one module used by both.
* `add_to_graph` had a `TypeError` compatibility fallback that couldn't succeed with the current `RelationExtractor` API and could hide the original extraction error. That fallback was removed.
* `append_event` persisted partial streaming events even though ADK's base session service skips them.
* `get_session()` ignored its `config` argument, so `num_recent_events` and `after_timestamp` had no effect.
* The async session-service methods performed synchronous graph scans while holding a `threading.RLock` on the event loop thread. That work now runs in worker threads with `asyncio.to_thread()` so a slow or contended graph operation doesn't block the loop.
---
Co-authored-by: Zohaib Hassnain [109234410+ZohaibHassan16@users.noreply.github.com](mailto:109234410+ZohaibHassan16@users.noreply.github.com)
The Ontology Hub editor selected a registered ontology but left the canvas empty, and opening an ontology deep link landed on the Welcome workspace instead of the editor. Two independent causes: the application shell ignored `ontologyTab`/`ontologyEntity` URL state at startup, and the editor loaded registry metadata but never fetched the selected ontology's schema nodes and structural edges. The backend now exposes a bounded schema subgraph for one ontology at `GET /api/ontology/graph?uri=...`, and the editor maps that response into React Flow nodes and edges with loading, error, selection, and layout handling.
Five things came out of review on the new endpoint and the editor that consumes it.
The edge selection originally included an edge whenever either its source or its target was a core node. That let a property owned by a completely unrelated ontology leak into the requested one just because its `rdfs:domain` or `rdfs:range` happened to point at one of the requested ontology's classes. Edges are now selected only when their source is a core node, so the requested ontology can still reference outward to external vocabulary, but nothing from an unrelated ontology gets pulled in the other direction.
The backend accepts both compact and full-IRI forms for node types (`owl:Class` and `http://www.w3.org/2002/07/owl#Class` are equivalent), but the frontend classifier only recognized the compact strings, so a full-IRI class or ontology node fell through to `"external"`, wrong panel, wrongly read-only. Classification moved into `ontologyEditorModel.ts` as `classifyNodeType`, which compacts known full IRIs before matching.
An ontology imported through the fallback RDF parser, one with no `owl:Ontology` or `skos:ConceptScheme` declaration, minted a synthetic registry URI but never created a matching graph node or set `scheme_uri` on the classes and properties it imported. `_node_belongs_to_ontology` had nothing to associate those nodes with, so `core_node_ids` ended up empty and the endpoint 404'd for a registered ontology that genuinely had data. The fallback parser now records that ownership and emits a matching `owl:Ontology` node whenever it has to synthesize a URI.
Nested namespaces that were never registered as their own ontology got silently absorbed into whichever parent prefix matched, in both directions: a fragment-delimited nested name (`<stem>/child#Term`) and a path-delimited one (`<stem>/child/Term`). The first fix only handled the fragment form; prefix ownership now only extends to names minted directly in the ontology's own namespace (`<stem>#Term` or `<stem>/Term`), and any further delimiter of either kind marks a nested vocabulary that isn't absorbed until it's registered or carries an explicit owner. Once registered, the nested namespace owns its own nodes as before.
Selecting a node in the editor writes `ontologyEntity=<id>` into the URL. Switching ontologies via the dropdown cleared the in-memory selection but left that parameter pointing at the old ontology, so a reload after switching could resolve the stale ID and jump back. The dropdown now clears the parameter on change.
Regression tests cover each fix directly: inward-edge exclusion, the full-IRI classification matrix, an end-to-end fallback-import test that forces the parser path and opens the resulting ontology, and a nested-namespace ownership matrix covering both delimiter forms in both the registered and unregistered case.
* fix mutable default argument in graph_analyzer.py
* fix mutable default argument in kg_chunkers.py
* fix mutable default argument in methods.py
* Address review: move default-init code out of docstrings, default levels in split_hierarchical
Three findings from the Qodo review:
- analyze_temporal_evolution: the 'if metrics is None' block had landed
inside the docstring, so it never executed and metrics_tracked came back
None. Moved below the docstring where it runs.
- HierarchicalChunker.__init__: the same misplacement turned the docstring
into a dead string constant and broke help()/introspection. Moved the
default-init below it.
- split_hierarchical: the signature now defaults levels to None, but the
body still ran 'in levels' membership tests — calling it without levels
raised TypeError. Defaults to the documented hierarchy, matching the
class-level default.
* test: add mutable-default regression tests for the three fixed sites
- tests/split/test_chunkers.py: TestMutableDefaultRegression (6 tests)
- split_hierarchical() default levels and chunk_sizes stay independent across calls
- HierarchicalChunker() default levels stay independent across instances
- tests/kg/test_kg.py: TestAnalyzeTemporalEvolutionMutableDefault (5 tests)
- analyze_temporal_evolution() default metrics value is canonical
- mutations to a returned metrics_tracked list do not affect the next call
- explicit metrics override is forwarded and reflected in the return value
- mutating an explicitly passed list does not corrupt a subsequent default call
All 96 tests in the two affected test files pass.
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix(erasure): ensure vector_store=False disables internal vector cascade in AgentMemory
* fix(erasure): ensure skip_vector=True does not orphan local vector ID tracking
MCP's stdio transport uses stdout for JSON-RPC framing, so anything else written there corrupts every response after it. The original #1134 bug was progress-tracker output landing on stdout during tool calls that construct a `ContextGraph`, which is exactly what happens on any request that triggers reasoning or extraction. This PR closes out the remaining pieces of that fix: loading now goes through `load_from_file()` instead of the older `load()` path on the root graph, and mutations, `record_decision`, `add_entity`, `add_relationship`, now persist back to `SEMANTICA_KG_PATH` when it's configured, in both MCP server implementations (the root `mcp/` package and the packaged `semantica.mcp_server`), not just one.
Four things came out of review on top of that.
The stdio regression test originally exercised `get_graph_summary`, which doesn't touch the progress tracker at all, so it couldn't have caught the original bug. Swapped it for `run_reasoning`: `Reasoner.infer_with_results()` calls `progress_tracker.start_tracking()` directly, the exact call site that corrupted stdout before, so this is the minimal path that actually proves the fix. The test now spawns a real `python -m mcp` subprocess, sends it a `tools/call` for `run_reasoning`, and asserts every single line on stdout parses as JSON.
Loading a corrupt or unreadable `SEMANTICA_KG_PATH` used to fail silently and fall through to an empty graph, which meant the next mutation would happily save that empty graph over the original file. Both implementations now track whether the initial load actually succeeded. If it didn't, every mutation handler refuses to save and returns an error instead, so a broken file on disk stays broken rather than getting silently replaced with nothing. An empty file is treated differently: that's a fresh destination, not a corrupt one, and starts a normal empty graph without tripping the guard.
`save_to_file` used to `open(path, 'w')` and `json.dump` directly into the destination, so a crash or disk-full error mid-write could leave a truncated file as the only copy of the graph. It now writes to a temp file in the same directory, flushes, fsyncs, and only then `os.replace`s the destination, so the destination is always either the old contents or the new contents, never a partial write. The temp file gets cleaned up if anything fails before the replace.
And since a mutation is applied to the in-memory graph before the save happens, a save failure used to leave the in-memory graph ahead of what's on disk, an entity or decision the client thinks succeeded but that never made it to the file. `record_decision`, `add_entity`, and `add_relationship` all roll back the in-memory mutation now if `save_to_file` raises, so the client-visible state and the persisted state never diverge: either both hold the change or neither does.
104 tests passing across the MCP, persistence, and progress-tracking suites.
