* fix(explorer): coerce decision timestamp to str to prevent 422 on /api/decisions
ContextGraph stores decision timestamps as POSIX floats (e.g. 1786513069.69),
but DecisionResponse.timestamp is typed Optional[str]. Pydantic strict
validation rejects the float and the whole /api/decisions endpoint returns
HTTP 422 "Invalid input", which breaks the Decisions workspace in the
Knowledge Explorer entirely (no decision can be listed).
Coerce the value to str (preserving None) in _node_to_decision so the
response validates. Verified: /api/decisions now returns 200 and the 3
sample decisions render in the Decisions workspace.
* test(explorer): cover decision timestamp coercion in _node_to_decision
Regression tests for the 422 fix in _node_to_decision. Covers the cases
that produced HTTP 422 (float / int timestamps from ContextGraph) and
the ones that must keep working (None, already-string, missing key).
Verified the suite catches the regression: with the fix reverted, the
float / int / nan / inf cases fail with the same ValidationError that
caused the 422; with the fix applied all 6 pass.
* fix(explorer): preserve decision timestamp normalization
The route-level str() cast introduced in the initial fix bypasses
DecisionResponse._normalize_timestamp, the field validator on main that
converts POSIX float epochs to ISO-8601 strings via
datetime.fromtimestamp(value, tz=UTC).isoformat().
With the cast in place the API emits raw numeric strings such as
'1786513069.69' instead of '2026-08-12T05:37:49+00:00', breaking
datetime.fromisoformat() for every caller and failing
TestRecordedDecisions::test_list_decisions_serializes_float_timestamp.
It also silently accepts nan/inf/out-of-range epochs that the validator
is designed to reject.
Restore _node_to_decision() to pass the raw stored value through
unchanged so DecisionResponse._normalize_timestamp remains the single
normalization boundary for all three affected endpoints:
GET /api/decisions
GET /api/decisions/{id}
GET /api/decisions/{id}/precedents
Rewrite test_decision_route_timestamp.py so every assertion uses
datetime.fromisoformat() to verify ISO-8601 output and explicitly
asserts ValidationError for nan, inf, -inf and out-of-range epochs.
Add three TestClient integration tests covering the full production
path: record_decision() -> float stored in graph -> HTTP GET -> JSON.
---------
Co-authored-by: administrator <administrator@administratordeMac-mini.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
* fix(explorer): repair /api/enrich/extract and the /api/decisions routes
Two Explorer API endpoints fail on every install.
/api/enrich/extract imported extract_entities and extract_relations from
semantic_extract.methods, where neither name is defined — that module ships
only the per-strategy variants (extract_entities_ml, extract_relations_regex,
...), and nothing re-exports a plain facade. The resulting ImportError was
caught and reported as "semantic_extract module not available. Ensure spacy
and transformers are installed.", so a wiring bug looked like a missing
dependency. The route now calls NamedEntityRecognizer and RelationExtractor
directly, the classes the README documents, and feeds the extracted entities
into relation extraction rather than re-deriving them. The 503 branch stays
for a genuinely absent module.
Every /api/decisions* route returned 500 once the graph held a decision:
record_decision() stores timestamp as datetime.now().timestamp(), a float,
while DecisionResponse types the field as str, so pydantic rejected the value
the library itself wrote. A before-mode field validator on DecisionResponse
normalizes float, int and datetime inputs to ISO-8601, covering every route
that builds the model instead of only the list endpoint.
The existing tests missed both: test_extract accepted 503 as a pass, and the
decision fixtures are hand-built nodes carrying no timestamp at all. Both are
tightened, and a TestRecordedDecisions class exercises the routes against
decisions created through record_decision().
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* perf(semantic_extract): cache spaCy models instead of loading one per call
extract_entities_ml(), extract_relations_similarity() and
extract_relations_dependency() called spacy.load() on every invocation, so the
model was re-read from disk and re-initialized per call. On a short sentence
that is ~120 ms of loading around ~2 ms of work, and successive calls never got
cheaper. The path is reachable from the CLI, the MCP extract_entities tool, the
pipeline ner_extract step and POST /api/enrich/extract, and process_batch()
multiplies it by the number of documents.
The module already had a cached loader for one code path — get_nlp_model() and
its _nlp_cache global — but the extraction functions bypassed it.
Adds load_spacy_model(), a process-level cache keyed by model name behind a
lock so concurrent callers do not each start a load, and routes the five call
sites through it. Errors are left uncached and propagate unchanged, so the
existing OSError fallbacks to pattern extraction still fire. get_nlp_model()
keeps its own entry: it loads with disable=["parser", "ner", "lemmatizer"] for
similarity work, so its model is not interchangeable with the NER one.
Cache entries record the spacy module object they came from. Several tests
patch methods.spacy with a mock and assert on load calls; without that guard a
name-keyed cache would hand a previous test's mock to a later one.
Measured on the same sentence, Python 3.12.13 / spacy 3.8.15 / en_core_web_sm:
extract_entities_ml() median 132 ms before, 2.1 ms after, identical entities.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(explorer): harden extraction and timestamp handling
* fix(explorer): catch OverflowError/OSError in decision timestamp validator
DecisionResponse._normalize_timestamp only guarded against NaN/inf via
math.isfinite(), but datetime.fromtimestamp() raises OverflowError or
OSError for finite epoch values outside the platform's representable
range (e.g. milliseconds stored where seconds were expected). Those
exceptions escaped the pydantic validator unhandled, reintroducing an
unhandled 500 on /api/decisions* for exactly the bug class this PR
closes. Also exclude bool from the numeric branch, since bool is an
int subclass and was being silently coerced to epoch 0/1.
* docs: add changelog entry for PR #886 (explorer extract/decisions fixes)
Documents the extraction 503, decisions timestamp 500, and folded-in
spaCy caching fixes, plus the review-round hardening from Sameer6305
and the timestamp overflow/bool fix from this follow-up commit.
---------
Co-authored-by: joseedson18jc <joseedson18jc@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
CORSMiddleware doesn't cover WebSocket handshakes at all (Starlette's CORS
support only wraps HTTP), so under SEMANTICA_ALLOW_ANONYMOUS=true --
the mode docker-compose.dev.yml ships -- is_valid_api_key's anonymous
bypass accepted a /ws/graph-updates connection from any origin. Loopback
binding isn't a boundary against a browser: any page the operator has
open can still reach ws://localhost:8000/ws/graph-updates directly, and
ConnectionManager.broadcast sends every graph_mutation to every
connected socket with no per-connection scoping. Combined with
/api/import accepting multipart/form-data (a CORS-safelisted content
type that skips preflight), a hostile page could write to the graph
over REST and read the result back over the unauthenticated WebSocket
-- demonstrated end-to-end in the report with a real client.
Not affected: any deployment with SEMANTICA_API_KEY configured -- the
handshake already rejects without a valid key in that mode. This is an
anonymous-mode-only, development-configuration exposure.
Fix: check the handshake's Origin header against
app.state.explorer_settings['allowed_origins'], the same list
CORSMiddleware already enforces for HTTP, before the key check. A
missing Origin (native/CLI clients, which never set the header --
only browsers do) is still allowed through, since the browser is the
only threat this closes.
4 new tests in test_explorer_auth.py: hostile Origin rejected under
anonymous mode; hostile Origin rejected even with a correct key
(Origin is checked before the key, so a leaked key alone can't
hijack the socket); an allowlisted Origin still connects under
anonymous mode; a missing Origin still connects under anonymous mode
(native clients keep working). Full explorer suite: 226 passed.
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Qodo's re-review confirmed the multi-IP fallback fix but kept the proxy
finding open: logging-and-falling-back when a proxy applies still let
the DNS-pinning protection be silently skipped under proxy
configuration, rather than enforcing a clear policy either way.
Implemented Qodo's preferred option: proxies are now disabled outright
for this SSRF-sensitive fetcher via session.trust_env = False, so
HTTP_PROXY/HTTPS_PROXY/NO_PROXY env vars are never consulted in the
first place (a configured proxy would perform its own DNS resolution
of the target host outside this process's control, reopening the
DNS check-then-use race pinning exists to close). The adapter also
keeps a fail-closed backstop: if a proxy is somehow still configured
despite trust_env=False (e.g. set explicitly by future code), it now
raises a clear 502 instead of silently connecting through the proxy
unpinned.
_validate_fetch_url's destination classification (blocking private/
internal targets) is unaffected either way — it runs before any of
this and doesn't depend on proxy configuration.
4 new tests: trust_env is disabled on every pinned session; an
HTTP_PROXY env var pointed at an address that would fail if contacted
is confirmed genuinely unused (real local-server fetch still succeeds
directly); and the fail-closed backstop actually raises when a proxy
is forced onto the session. Full explorer + triplet_store suite: 572
passed.
Four findings from PR #916's automated review, all addressed:
- CodeQL (HIGH): the test HTTPS server's SSLContext allowed TLSv1/TLSv1.1
by not setting a minimum version. Added
ssl_ctx.minimum_version = ssl.TLSVersion.TLSv1_2.
- github-code-quality: unused `cryptography` local in
_make_self_signed_cert — importorskip's return value was never used.
- Qodo (reliability): _validate_fetch_url() only returned the first
validated IP, and _make_pinned_session() pinned to just that one
address, so a fetch would fail outright if the first-returned A/AAAA
record happened to be unreachable even though a later one would work.
_validate_fetch_url() now returns every validated IP (deduplicated,
in resolution order); _make_pinned_session() takes the full list and
falls back through each one via a custom Connection._new_conn
override, matching the fallback behavior a normal DNS-resolving
connection would already get for free. Verified with a real test:
pin to an unreachable loopback address followed by a real one, confirm
the fetch still succeeds by falling back; and a real test confirming
it still raises (rather than silently re-resolving the hostname) when
every pinned address is unreachable.
- Qodo (security): when an HTTP(S) proxy applies, the adapter falls back
to the unpinned path rather than pinning. This is a real, but
architecturally unavoidable, limitation from the client side: for a
forward proxy, the *proxy* performs its own DNS resolution of the
target host on the application's behalf, a resolution this process
has no visibility into or control over — there's no client-side pin
that closes that race. _validate_fetch_url's destination
classification still fully applies either way; only the secondary
DNS-pinning hardening doesn't extend through a proxy. Added an info
log when this fallback path is taken so it's observable rather than
silent, and expanded the code comment to make the reasoning explicit
for the next reader/reviewer rather than looking like an oversight.
Tests: 3 new tests in test_ontology_dns_pinning.py (multi-IP fallback
success, all-unreachable failure, deduplicated multi-record resolution).
Full explorer + triplet_store suite: 569 passed.
Two follow-up hardening items flagged as secondary/deferred during
GHSA-8c7v-62gr-hj6g and GHSA-8vgg-8mr4-r236's fixes:
1. DNS check-then-use (TOCTOU) window in the ontology URL fetcher.
_validate_fetch_url() resolved and validated a hostname once, but
_fetch_url_sync() then let requests resolve the same hostname again
independently at connect time — a low-TTL or rebinding DNS answer
could differ between the two lookups, reopening the SSRF window the
validation exists to close.
_validate_fetch_url() now returns the validated IP, and a new
_make_pinned_session() builds a per-hop requests.Session whose
connection pool is pinned directly to that IP (bypassing DNS
resolution for the connection entirely), while explicitly restoring
the real hostname as the outgoing HTTP Host header and, for HTTPS,
the TLS SNI server_hostname/assert_hostname — so the connection
reaches the validated IP but still presents (and is verified
against) the real hostname's identity, keeping virtual hosting and
certificate validation correct.
Note: an earlier version of this fix set `_dns_host` post-construction
assuming it was decoupled from `host`, matching some other urllib3
releases; in the installed version (2.7.0), `host` is a property
that reads/writes `_dns_host` directly, so that approach silently
changed the Host header too. Verified with a real (non-mocked) local
HTTP server, a real local HTTPS server with a self-signed cert
(proving SNI/cert-hostname verification checks the real hostname,
not the pinned IP), and a negative control confirming a hostname/cert
mismatch is still correctly rejected — not silently bypassed.
2. Pre-wrapped object IRIs skipped full validation in
_format_object_for_sparql/_format_object_for_ntriples (Blazegraph,
RDF4J). A triplet object already wrapped in `<...>` only had its
inner content checked for a literal space or `>`, not run through
sparql_escaping.validate_uri() like the unwrapped-object branch —
flagged by automated review during GHSA-8vgg-8mr4-r236's fix. Both
branches now validate identically.
Tests: tests/explorer/test_ontology_dns_pinning.py (6 tests, including
2 real local-server end-to-end checks and 2 real-TLS checks with a
generated self-signed cert, gracefully skipped if `cryptography` isn't
installed); updated tests/explorer/test_ontology_ssrf.py for the new
per-hop session construction; 4 new tests in
tests/triplet_store/test_sparql_injection.py for the object-IRI fix.
Full explorer + triplet_store suite: 566 passed.
Two follow-up fixes to the initial ReDoS patch (CodeQL py/polynomial-redos
#1897), raised during code review:
--- Fix 1: _PREFIX_DECL regression — inline prologues and CRLF (#review-1) ---
The first ReDoS fix replaced the ambiguous trailing \s* with [ \t]*(?:\n|$),
but that introduced a behavioral regression:
* Inline prologues — PREFIX ex: <...> SELECT ... on a single line were no
longer stripped because the mandatory (?:\n|$) anchor never matched when
non-whitespace content followed the IRI on the same line.
* CRLF line endings — PREFIX ex: <...>\r\n failed because \r is not in
[ \t]* and the anchor expected a bare \n.
Root cause: the end-of-line anchor was unnecessary; the only thing needed
to eliminate backtracking ambiguity is ensuring the IRI body character class
and the trailing whitespace quantifier are disjoint.
Fix: change the IRI body from <[^>]*> to <[^>\r\n]*>, which:
- excludes CR and LF from the IRI match (semantically correct — SPARQL
IRIs cannot span line boundaries)
- makes [^>\r\n]* and the trailing [ \t]* have zero character overlap,
eliminating all backtracking ambiguity without any end-of-line anchor
No anchor is used, so both inline prologues and CRLF/LF endings work
naturally. ReDoS payloads (base< + !< x 10,000) still complete in <1 ms.
--- Fix 2: oversized-query length guard obscured error (#review-2) ---
The initial patch placed the _SPARQL_MAX_QUERY_LEN guard inside
_is_read_only_query(), which caused execute_sparql() to return the same
generic 'Only SELECT' error for both genuinely disallowed query types and
oversized inputs. Clients could not distinguish the two rejection reasons.
Fix: move the length check out of _is_read_only_query() and into
execute_sparql() as an explicit early gate, alongside the other resource
limits (_SPARQL_MAX_ROWS, _SPARQL_MAX_GRAPH_NODES). Oversized queries now
return a specific message naming the limit, the received length, and the
remediation step. _is_read_only_query() is documented to be length-agnostic.
_SPARQL_MAX_QUERY_LEN is relocated to the resource-limits block with the
other constants.
--- Tests added ---
tests/test_security_regression.py:
- test_inline_prefix_before_select_allowed (Fix 1 regression)
- test_crlf_line_endings_with_prefix (Fix 1 regression)
- test_crlf_multiple_prefixes_then_select (Fix 1 regression)
- test_inline_prefix_before_insert_still_blocked (Fix 1 security check)
- test_long_valid_query_not_rejected_by_is_read_only (Fix 2 separation)
tests/explorer/test_sparql_route.py:
- test_oversized_query_returns_distinct_length_error (Fix 2 error message)
- test_oversized_query_never_touches_the_graph (Fix 2 short-circuit)
- test_query_exactly_at_length_limit_is_accepted (Fix 2 boundary)
All 82 tests pass.
Two issues in the last round of commits:
1. explorer/auth.py added a new, opt-in APIKeyAuthMiddleware
(EXPLORER_API_KEY) and wired it into create_app(), but in doing so
removed the Depends(require_auth) dependency from every router and
deleted the /ws/graph-updates handshake check entirely. The new
middleware also fails OPEN (allows all requests) when its key is
unset, the opposite of require_auth's fail-closed design. Since
GHSA-j4mq-hprp-987v (the unauthenticated-Explorer-API advisory) is
already merged into main via require_auth, this would have reverted
a merged Critical fix the moment this branch merges. Removed
explorer/auth.py, restored the per-router dependencies and the
WebSocket auth check. Kept auth.py's one genuine improvement (adding
X-API-Key to the CORS allow_headers list) by folding it into the
existing CORS middleware config.
2. sparql.py's new _is_read_only_query() hardening (comment/PREFIX
stripping + forbidden-keyword scan) used `#[^\n]*` to strip SPARQL
comments, but a bare '#' also appears inside standard RDF namespace
IRIs (e.g. ".../1999/02/22-rdf-syntax-ns#") — the regex struck
everything after that '#' as a "comment", corrupting the query and
rejecting any legitimate SELECT using rdf:/rdfs:-style PREFIX
declarations. Confirmed by the fact the new hardening's own inlined
test copy failed against two of its own cases. Fixed by only
treating '#' as a comment-start at line-start or after whitespace,
which distinguishes ".../ns#" (preceded by a word character) from an
actual comment (preceded by whitespace/newline in every realistic
case, including the attacker's own comment-hiding PoC). Also fixed
the companion PREFIX/BASE regex, which required a prefix-name token
between the keyword and the IRI even for bare `BASE <...>`
declarations (which have none).
tests/test_security_regression.py's SPARQL section now imports the real
_is_read_only_query instead of maintaining a parallel inlined copy that
had silently drifted from — and shared the same bug as — the real
implementation; removed its TestAPIKeyAuth class (tested the now-deleted
auth.py) since equivalent, more thorough coverage already exists in
tests/explorer/test_explorer_auth.py. Updated tests/explorer/test_sparql_route.py's
multi-statement-injection test to reflect that the keyword scan now
catches "SELECT ... ; DROP ALL" itself rather than relying on rdflib's
parser, and added a new test confirming the parser still catches
multi-statement syntax that doesn't contain any forbidden keyword.
Full explorer/vector_store/security-regression/age_store suite: 543
passed (the only failures are 6 pre-existing, unrelated Pinecone-client
mocking issues).
* security: require API-key auth on all Explorer API routes (GHSA-j4mq-hprp-987v)
Every Explorer route (bulk import/export, delete, LLM-backed ontology
generation, SPARQL, etc.) was mounted with no authentication, and both
server entrypoints bind 0.0.0.0 by default. Anyone reaching the port got
full read/write/delete on the graph.
- Add require_auth dependency (explorer/dependencies.py): checks
X-API-Key against SEMANTICA_API_KEY, fails closed with 503 if
unconfigured (not silently anonymous), 401 on wrong/missing key.
SEMANTICA_ALLOW_ANONYMOUS=true opts out explicitly for local dev.
- Wire dependencies=[Depends(require_auth)] into all 11 API routers in
both explorer/app.py and server.py. /health, /api/info, static assets,
and the SPA catch-all stay public.
- /ws/graph-updates handshake now checks the same key via header or
?api_key= query param (browsers can't set custom WS headers) before
accepting the connection.
- Default bind changed from 0.0.0.0 to 127.0.0.1 in server.py's main()
and cli.py's `server start`; the CLI warns if a non-loopback host is
passed explicitly without a key configured.
- Startup logging reports the resolved auth mode in both app factories.
- Document/generate SEMANTICA_API_KEY in the deploy recipes that expose
a public endpoint by default: docker-compose, Railway, Fly, Render.
Added tests/explorer/test_explorer_auth.py covering fail-closed default,
wrong/missing/correct key, anonymous opt-in, public-route exemptions, and
the WS handshake. Added tests/explorer/conftest.py defaulting the
pre-existing ~200 explorer tests to SEMANTICA_ALLOW_ANONYMOUS=true so
they keep exercising route logic without needing a key.
* fix CORS
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
- vector_store.save(): use v.tolist() instead of list(v) so numpy float32
vectors round-trip through JSON instead of raising TypeError.
- ontology._fetch_url_sync(): resolve relative Location headers via urljoin
before re-validating (previously any relative redirect was rejected
outright), and close every response instead of leaking the connection
across redirect hops.
- sparql.execute_sparql(): move _build_rdflib_graph inside the handler's
error handling so the graph-size cap returns a clean SparqlResponse
error instead of an unhandled 500.
- add regression tests for all three.
- SQLiteStorage now migrates an existing (pre-#825) provenance.db in place
via ALTER TABLE ADD COLUMN for any columns introduced since, instead of
only ever running CREATE TABLE IF NOT EXISTS. Without this, opening an
older database with the new code would break on the first insert/select
since the row width and _row_to_entry's fixed indices grew past the old
schema. Added test_migrates_pre_existing_old_schema_database.
- verify_chain() now also checks that sequence_id is exactly the
predecessor's plus one (no gap, no duplicate), in addition to the existing
previous_checksum comparison. Hardens against the narrow case where
compute_checksum()'s deliberate exclusion of entity_id could let two
distinct rows coincidentally share a checksum, which alone would let a
checksum-only comparison miss a gap. Added
test_verify_chain_detects_tampered_sequence_gap.
- Explorer provenance route: edge ids now include direction
(f"{src}-{eid}-{direction}") to match the seen_edges dedupe key, which
already included it. The same (src, target) pair can legitimately appear
in both the upstream and downstream chains (cycles/overlap), and without
this the two edges collided on the same id. Added
test_add_chain_edges_ids_distinguish_direction.
- Removed an unused `Any` import in parse_provenance.py.
- Removed unused `validate_skos_hierarchy` import from
test_ontology_subissue3.py (flake8 F401); the test uses a `wraps=` spy
on the real add_nodes_and_edges instead of calling the helper directly.
- refresh_ontology tests now percent-encode the ontology URI with
urllib.parse.quote before interpolating it into the {ontology_uri:path}
request path, matching the already-encoded unknown-uri refresh test in
the same file instead of embedding a raw http://... URI with slashes.
- Reworded the cyclic-SKOS refresh test's comment and section header:
GraphSession.add_nodes_and_edges() documents pre-write validation and
lock-based mutual exclusion, not transactional rollback, so "atomic"
was replaced with "single combined add_nodes_and_edges() call" to avoid
implying rollback guarantees that don't exist.
Verified: tests/explorer/test_ontology_subissue3.py (34 passed) and
tests/explorer/ (204 passed), no regressions.
- validate_skos_hierarchy() re-walked every existing hierarchy edge in
the graph on each write, so one pre-existing cycle anywhere would
block all unrelated future SKOS writes. It now only traverses
concepts touched by the edges actually being written, while still
checking those against existing edges for cross-boundary cycles.
- In /api/ontology/load, `except HTTPException: raise` sat after a
broader `except Exception` clause that already matched HTTPException,
so a 422 raised after a successful OntologyIngestor parse was
silently swallowed and retried via the fallback RDF parser instead of
reaching the caller. Reordered the except clauses.
Co-authored-by: mikemikimike <13286568797@163.com>
- Extend ProvenanceNode in semantica/explorer/schemas.py with audit evidence fields: source_document, source_location, source_quote, confidence, and checksum.
- Update _transform_audit_lineage in semantica/explorer/routes/provenance.py to populate these evidence fields for each lineage node from ProvenanceEntry records, while keeping default None values for orphan nodes.
- Include source_document, confidence, and checksum in markdown report rendering (_render_markdown) so exported markdown reports surface attribution and integrity evidence.
- Add unit test test_provenance_audit_evidence_fields_preserved in test_provenance_manager_wiring.py verifying that evidence fields are present across /api/provenance JSON responses and exported JSON/markdown reports.
- Update verify_checksum and compute_checksum in semantica/provenance/integrity.py to support both ProvenanceEntry objects and serialized dictionary entries.
- Add integrity_verified flag computed via verify_checksum to the dictionary returned by ProvenanceManager.get_lineage().
- Update _build_provenance in semantica/explorer/routes/provenance.py to verify every returned lineage entry before labeling the result as source=audit. If verification fails due to missing checksums or corrupted records, log a warning and fall back cleanly to graph traversal.
- Add unit test test_provenance_manager_wiring_checksum_failure_falls_back in test_provenance_manager_wiring.py verifying that tampered lineage entries trigger fallback to source=graph_traversal.
- Fix _transform_audit_lineage to classify all non-downstream ancestor derivation edges as 'upstream' instead of 'lateral', correcting multi-hop lineage direction in JSON and markdown reports.
- Add GraphSession.set_provenance_storage_path() to explicitly reject conflicting preconfigured storage paths or path mutations after provenance_manager initialization.
- Update create_app() to call active_session.set_provenance_storage_path(prov_path), preventing silent retention of conflicting paths or un-redirectable cached managers.
- Remove unused logging import in app.py.
- Add comprehensive unit tests in test_provenance_manager_wiring.py for upstream edge classification, markdown report grouping, conflicting path rejection, and manager initialization lockouts.
- Explorer's /api/provenance now queries the audit-grade ProvenanceManager
(SQLite-backed, checksummed) first, falling back to the naive 2-hop
graph traversal when no audit records exist for a node.
- Fixed a process-global mutable-state risk in the initial approach:
provenance storage path is threaded per-session via GraphSession,
not via ProvenanceManager's global set_default_storage_path classmethod.
- Added source: 'audit' | 'graph_traversal' to the response so callers
can distinguish which path served the data.
- Documented a known limitation: ProvenanceManager currently only
traces upstream/ancestor lineage, not descendants — the naive
fallback remains the only source for downstream relationships until
ProvenanceManager gains a reverse lookup (tracked separately).
- Warns (rather than silently no-ops) if a provided session's
provenance_manager was already constructed before create_app()
applied a provenance_storage_path.
- Never lets a provenance-manager failure crash the route; degrades
to the naive path with a logged warning instead.
Tests: 5 new tests in test_provenance_manager_wiring.py covering the
audit path, empty-record fallback, storage-failure degradation, app
startup wiring, and cross-session storage isolation. Full
tests/explorer/ + tests/provenance/ suite passing, order-invariant.
Extracts the row-cap-and-truncate loop (duplicated between the
CONSTRUCT/DESCRIBE and SELECT branches) into a shared _cap_rows()
helper, and adds a test for the previously-uncovered CONSTRUCT/DESCRIBE
truncation path. Addresses review nits on PR #805.
- Revert create_ontology silently falling back to a near-empty ontology on
generation failure; restores the HTTPException(500) behavior from #770/#787
that this PR had accidentally undone (and re-enables TestOntologyCreateFailures)
- Fold sh:Warning/sh:Info severity pySHACL results into the /shacl/validate
response's violations array instead of silently dropping them, so a
non-conforming report is never returned with an empty violations list
- Share a single nodes/edges fetch between _generated_shacl_for_uri and
_data_graph_turtle_for_uri via new _fetch_analysis_graph(), so /health
no longer re-queries and re-truncation-checks the same ontology twice
sparql.py handles direct SPARQL query execution against the live graph with no test coverage anywhere in the repo. Adds coverage for the read-only allowlist (the actual security boundary here), row/timeout limits, error handling, and RDF projection fidelity.
207 alone is indistinguishable from 200 to callers that only check
response.ok, so /api/analytics now raises 500 when every requested
metric fails and reserves 207 for genuine partial failure. Adds
regression tests for the temporal, analytics, and ontology-create
failure paths introduced in this PR, and logs the fix in the
changelog's Unreleased section.
- gcp/cloudrun-service.yaml: add comment + README sed one-liner so PROJECT_ID
is substituted before gcloud run services replace (was a literal placeholder
that caused image-pull failure on the declarative deploy path)
- azure/main.parameters.json: replace wildcard allowedOrigins "*" with a
REPLACE_ME placeholder; add README note to set the real URL after first deploy
- kubernetes/networkpolicy.yaml + helm networkpolicy template: add from: selector
(ingress-nginx namespace + same-namespace pods) so ingress is no longer
allow-all; restrict egress to FalkorDB port 6379 and DNS port 53 instead of
the allow-all egress: - {} wildcard
- helm/values.yaml: expose networkPolicy.ingressNamespace and falkordbPort values
- kubernetes/deployment.yaml: add secretRef for knowledge-explorer-secrets so
FALKORDB_PASSWORD is actually injected into the container
- app.py: add _mutation_bridge_installed guard to prevent closure stacking when
the same GraphSession is passed to create_app() more than once; remove
duplicate app.state.allowed_origins assignment (single source of truth is
app.state.explorer_settings); add comment on falkordb_host/port dead config
- tests: update allowed_origins assertions to use explorer_settings dict
- .checkov.yaml: remove global CKV_K8S_21/28/30 suppressions; rely on per-file
inline checkov:skip comments in cloudrun-service.yaml so future real K8s
manifests are not silently exempted
- GCP: remove --allow-unauthenticated, restrict ingress to
internal-and-cloud-load-balancing, replace wildcard ALLOWED_ORIGINS=*
with a substitution variable (_ALLOWED_ORIGINS) so operators supply a
real URL at deploy time; same fix in cloudrun-service.yaml
- Fly.io: replace hardcoded FALKORDB_HOST=localhost with the correct
.internal private-network hostname pattern; update README accordingly
- docker-compose.dev.yml: add missing top-level networks: block so the
frontend service can join the semantica network without --file layering
- K8s/Helm: add readOnlyRootFilesystem: true + runAsUser: 1000 to
container securityContext; mount an emptyDir /tmp so uvicorn can write
temp files
- app.py: fix _read_explorer_settings() or-chain, use in os.environ
checks so an explicit ALLOWED_ORIGINS="" produces an empty allow-list
instead of silently falling through to localhost defaults; remove dead
app.state.falkordb_host/port attributes
- docs: update four locations that still documented {"status":"healthy"}
to reflect the new {"status":"ok"} health response
- tests: update test assertion to read falkordb settings from
app.state.explorer_settings instead of removed top-level attributes
* fix(explorer): resolve blank dashboard UI and ship frontend bundle in wheel
Fixes#631 — the Explorer server started successfully but the browser showed
a blank page because semantica/static/ was gitignored and never present after
a fresh install or clone.
Changes:
- ci.yml / release.yml: add Node 20 setup + npm ci && npm run build before
python -m build so every wheel contains a CI-built frontend bundle
- pyproject.toml: add package-data patterns (static/*, static/assets/*) so
setuptools includes the bundle in the wheel; add MANIFEST.in for sdist coverage
- app.py: replace silent empty-HTML fallback with a 200 page that clearly
explains the missing bundle and links to /docs; fix CORS allow_credentials
to default false, gated behind EXPLORER_CORS_CREDENTIALS env var to prevent
credentialed cross-origin requests on unauthenticated endpoints
- __init__.py: warn at startup when --host is non-loopback (unauthenticated
network exposure)
- explorer/README.md: full rewrite covering pip-install mode (primary path,
no Node required) and dev-server mode (contributors), CLI flags, env vars,
workspace table, troubleshooting for the blank-page symptom
- README.md: update Knowledge Explorer section with correct command and link
to the new setup guide
* fix(explorer): set build.target esnext to fix esbuild CI failure
esbuild >=0.28 (forced via npm overrides) conflicts with Vite 6 defaults on
Linux CI — it tries to lower destructuring syntax for the implicit browser
target list but errors out. Explicit target: 'esnext' tells esbuild to emit
native syntax unchanged, bypassing the transpilation error entirely. Safe for
a developer tool that runs in modern browsers.
* test(explorer): verify packaged frontend bundle
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
Backend (semantica/explorer/routes/ontology.py):
- suggest-alignments: add TF-IDF character-ngram embeddings via sklearn
(SimilarityCalculator-compatible cosine scoring) so embedding_similarity
is populated in results; combined score = 0.4*label + 0.6*embedding when
available, falling back to label-only when sklearn is absent
- suggest-alignments: add token-overlap prefilter before SequenceMatcher so
zero-Jaccard pairs are skipped without computing full similarity; add
_MAX_ENTITIES_PER_SIDE=500 per-ontology cap on top of the existing
_MAX_ANALYSIS_NODES global cap
- suggest-alignments: remove dead try/except OntologyEngine.create_alignment
block that always failed silently (no TripletStore configured); replace
with a comment explaining the intentional ephemeral-only storage model
- health: replace O(alignments x entities) any() scans for alignment coverage
with O(1) set membership checks via assessed_ids
- shacl/validate: run rdflib.Graph().parse(format='turtle') syntax check on
the submitted Turtle before returning; invalid syntax now raises 422 instead
of returning a misleading unavailable/success response
Frontend:
- AlignmentsTab: add pairwise alignment matrix section that groups recorded
alignments by (source_ontology, target_ontology) pair; each cell shows
color-coded relation badges per RELATION_COLORS; clicking a badge populates
the create/edit form for quick editing; matrix is shown when at least two
ontologies are loaded
- ShaclStudio: add selectedShapeId state and fullShacl ref; each shape row in
the library is now a clickable button that extracts its Turtle block from
the full SHACL and pre-populates the Monaco editor; a "View all" toggle
restores the full SHACL; selected shape ID is shown in the editor header
- GraphWorkspace: fix viewMode race in external focus effect — call
setSelectedNodeId directly instead of going through focusNode(), which
captured a stale viewMode in its closure; remove focusNode from the
dependency array since it is no longer called
Tests (14 passing, was 11):
- Add test_suggest_alignments_returns_embedding_similarity: asserts
embedding_similarity is non-null when sklearn is available
- Add test_shacl_validate_rejects_invalid_turtle_syntax: asserts 422 on
syntactically invalid Turtle
- Add test_health_alignment_coverage_uses_set_lookup: asserts alignment
dimension score is non-zero after recording an alignment, verifying the
O(1) set lookup path works correctly end-to-end
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Backend:
- Replace false conforms=True SHACL stub with status=unavailable always;
live validation cannot be wired until OntologyEngine.validate_graph is
connected to a data graph — a stub that returns conforms=True misleads
users editing shapes
- Cap node/edge fetches in health, suggest-alignments, and SHACL generation
at _MAX_ANALYSIS_NODES (5 000) with a logger.warning when the graph
exceeds the limit; unbounded limit=999_999 fetches cause OOM on large graphs
- Set SHACL health dimension score to 0.0 (was 70.0) when status=unavailable;
exclude unavailable dimensions from the total_score average so they neither
inflate nor deflate the result
- Allow alignments to reference external/unloaded URIs (e.g. schema.org)
without raising 404; label falls back to URI fragment or caller-supplied
source_label/target_label fields added to OntologyAlignmentRequest
- Fix _alignment_id to use uuid.NAMESPACE_OID instead of NAMESPACE_URL;
the composite key is not a URL
- Fix _summarize_shapes to normalise \r\n before splitting on .\n so shape
parsing works correctly on Windows line endings
Frontend:
- Wrap handleSave/handleSuggest/handleRemove/handleAcceptSuggestion in
useCallback in AlignmentsTab for consistency with sibling components
- Add ephemeral-storage banner in AlignmentsTab warning that alignments are
session-memory-only and not persisted across restarts
- Fix exportReport in HealthTab to append/remove anchor from document before
clicking and defer URL.revokeObjectURL to avoid Blob URL leak in some browsers
- Derive health dimension grid column count from health.dimensions.length
instead of the hardcoded repeat(5, ...) that breaks if the backend adds
or removes a dimension
- Add minimal Monarch tokenizer for the Monaco turtle language registration
in ShaclStudio so prefix declarations, IRIs, SHACL properties, comments,
and string literals are syntax-highlighted; previously the editor rendered
as plain text despite theme rules being defined
Tests (11 passing, was 5):
- Rename test_shacl_validate_has_stable_contract to
test_shacl_validate_returns_unavailable and assert status == unavailable
- Add test_shacl_validate_rejects_empty_turtle (expects 422)
- Add test_health_returns_404_for_unknown_ontology
- Add test_health_shacl_dimension_is_zero_when_unavailable with total_score check
- Add test_delete_unknown_alignment_returns_404
- Add test_alignment_upsert_is_idempotent (verifies ID stability and created_at
preservation across updates)
- Add test_alignment_accepts_external_uri (verifies no 404 for schema.org URIs)
- Relax test_alignment_suggestions_are_ranked label assertions to substring
checks so the test survives similarity algorithm changes
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
- Align _coerce_embedding_vector inner dict-probe key list with
_extract_node_embeddings outer key list (add 'embeddings', reorder to
generic-first) so nested embedding dicts resolve consistently.
- Add TODO comment on _extract_node_embeddings to cache per-session
graph revision and avoid O(N) re-scan on every semantic request.
- Add deprecation docstrings to legacy path-segment routes
(/node/{id}/path and /node/{id}/semantic-neighborhood) documenting
the known slash-in-ID limitation and pointing to the query-param
alternatives.
- Extract _FakeSimilarity to module level so it is shared without
duplication across test classes.
- Rewrite test_legacy_semantic_neighborhood_still_works_for_simple_ids
as a fully isolated TestClient session instead of mutating the
shared module-scoped 'client' fixture, preventing cross-test
state pollution.
- Extract _make_slash_node_session helper to reduce boilerplate in the
slash-safe route tests.
Co-Authored-By: ZohaibHassan16 <109234410+ZohaibHassan16@users.noreply.github.com>
Co-Authored-By: KaifAhmad1 <98801504+KaifAhmad1@users.noreply.github.com>
- Extend PathResponse with hop_count (len(path)-1) and distance_band
("direct"|"near"|"mid-range"|"distant") as first-class API fields
- Add classify_path_distance() to semantica/utils/helpers.py as the
single source of truth for hop-count thresholds; both the route and
the visualizer import from it, eliminating duplicate threshold logic
- Populate hop_count and distance_band in find_path route via
classify_path_distance(); remove local _classify_distance() copy
- Add highlight_path: list[str] param to KGVisualizer.visualize_network;
path edges rendered as a distance-aware orange trace (opacity and
stroke width scale from direct→distant: 1.0/4px to 0.35/1.5px)
- Fix bidirectional edge lookup: only forward pairs (A→B) along the
path are added to path_edge_set; reverse back-edges in directed
graphs are no longer incorrectly highlighted
- Add logger.warning when highlight_path contains node IDs absent from
the layout position map, surfacing silent no-op mismatches
- Extend frontend PathResponse type in GraphInspectorPanel.tsx and
GraphWorkspaceShell.tsx with hop_count: number and distance_band
literal union to match the updated API contract
- Add 10 new tests: 2 API-level and 8 unit tests covering all four
band boundaries (0, 1, 2, 3, 4, 6, 7, 20 hops); 104 explorer tests
pass, 0 failures introduced
- Update CHANGELOG.md
- PathFinder.bfs_shortest_path() and dijkstra_shortest_path() gain a
directed: bool = True parameter. When False, a temporary undirected
view (graph.to_undirected()) is used for traversal only; the original
directed edges are preserved and returned in the response.
- _make_undirected_view() helper added to PathFinder; falls back safely
for non-NetworkX graph types.
- GET /api/graph/node/{id}/path exposes ?directed=false query param.
- PathResponse gains a directed: bool field echoing the mode used.
- Route now returns 404 on empty path (previously returned 200 with
path: []).
- 21 new tests: 12 unit (TestBidirectionalPathFinding) + 9 API-level
(TestBidirectionalPathRoute). All 120 tests pass.
- CHANGELOG updated under [Unreleased].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- routes/graph.py: raise HTTPException(404) for missing nodes; wrap path_finder call in try/except → HTTPException(404)
- routes/decisions.py: raise HTTPException(404) on chain, compliance, precedents sub-routes
- routes/annotations.py: raise HTTPException(404) on create (missing node) and delete (missing annotation)
- routes/enrich.py: raise HTTPException(404/503/422) for missing nodes, unavailable services, and extraction errors
- routes/temporal.py: fix TemporalPatternDetector method name detect_patterns → detect_temporal_patterns
- explorer/app.py: root / now serves built index.html when available, falls back to minimal HTML shell; add HTMLResponse import
- tests/explorer/test_explorer_api.py: test_extract accepts 503 (spacy/transformers not installed is a valid service state)
All 45 explorer API integration tests pass.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
