semantica/mcp_server/__init__.py was fixed to stop hardcoding 0.4.0,
but the separate top-level mcp/ package (run via `python -m
mcp.server`, documented in mcp/__init__.py as a supported way to
configure Claude Desktop/Windsurf/etc. from a source checkout) still
hardcoded 0.4.0 in three places: mcp/__init__.py, mcp/server.py, and
mcp/resources/registry.py.
Reuses semantica.__version__ directly, matching the pattern just
adopted in semantica/mcp_server/__init__.py, so both implementations
stay in sync with the package version going forward.
- pinecone_store: call self.index.describe_index_stats() instead of the
nonexistent self.describe_index_stats(), and use a unit query vector
instead of an all-zero vector so filter_by_metadata() works on
cosine-metric indexes (the library's own default)
- pgvector_store: apply the existing lowercase true/false bool handling
to the list-filter branch too, and use the jsonb ?| operator so
list-valued metadata fields match on intersection instead of being
compared as a single JSON-text blob
- sqlite_vec_store: use json_each() with a json_type guard so list-valued
metadata fields match on intersection, mirroring the in-memory
backend's set-intersection semantics
- faiss_store: filter_by_metadata(limit=0) now returns [] instead of one
result
- milvus_store: reject NaN/Infinity filter values up front with a clear
ValidationError instead of building an invalid expression that gets
silently swallowed
- update the #848 FAISS NotImplementedError test to reflect that FAISS
now implements real filter_by_metadata() (this PR's whole point)
- add regression tests for each fix; sqlite tests run against the real
sqlite-vec extension
Qodo's re-review confirmed the multi-IP fallback fix but kept the proxy
finding open: logging-and-falling-back when a proxy applies still let
the DNS-pinning protection be silently skipped under proxy
configuration, rather than enforcing a clear policy either way.
Implemented Qodo's preferred option: proxies are now disabled outright
for this SSRF-sensitive fetcher via session.trust_env = False, so
HTTP_PROXY/HTTPS_PROXY/NO_PROXY env vars are never consulted in the
first place (a configured proxy would perform its own DNS resolution
of the target host outside this process's control, reopening the
DNS check-then-use race pinning exists to close). The adapter also
keeps a fail-closed backstop: if a proxy is somehow still configured
despite trust_env=False (e.g. set explicitly by future code), it now
raises a clear 502 instead of silently connecting through the proxy
unpinned.
_validate_fetch_url's destination classification (blocking private/
internal targets) is unaffected either way — it runs before any of
this and doesn't depend on proxy configuration.
4 new tests: trust_env is disabled on every pinned session; an
HTTP_PROXY env var pointed at an address that would fail if contacted
is confirmed genuinely unused (real local-server fetch still succeeds
directly); and the fail-closed backstop actually raises when a proxy
is forced onto the session. Full explorer + triplet_store suite: 572
passed.
Four findings from PR #916's automated review, all addressed:
- CodeQL (HIGH): the test HTTPS server's SSLContext allowed TLSv1/TLSv1.1
by not setting a minimum version. Added
ssl_ctx.minimum_version = ssl.TLSVersion.TLSv1_2.
- github-code-quality: unused `cryptography` local in
_make_self_signed_cert — importorskip's return value was never used.
- Qodo (reliability): _validate_fetch_url() only returned the first
validated IP, and _make_pinned_session() pinned to just that one
address, so a fetch would fail outright if the first-returned A/AAAA
record happened to be unreachable even though a later one would work.
_validate_fetch_url() now returns every validated IP (deduplicated,
in resolution order); _make_pinned_session() takes the full list and
falls back through each one via a custom Connection._new_conn
override, matching the fallback behavior a normal DNS-resolving
connection would already get for free. Verified with a real test:
pin to an unreachable loopback address followed by a real one, confirm
the fetch still succeeds by falling back; and a real test confirming
it still raises (rather than silently re-resolving the hostname) when
every pinned address is unreachable.
- Qodo (security): when an HTTP(S) proxy applies, the adapter falls back
to the unpinned path rather than pinning. This is a real, but
architecturally unavoidable, limitation from the client side: for a
forward proxy, the *proxy* performs its own DNS resolution of the
target host on the application's behalf, a resolution this process
has no visibility into or control over — there's no client-side pin
that closes that race. _validate_fetch_url's destination
classification still fully applies either way; only the secondary
DNS-pinning hardening doesn't extend through a proxy. Added an info
log when this fallback path is taken so it's observable rather than
silent, and expanded the code comment to make the reasoning explicit
for the next reader/reviewer rather than looking like an oversight.
Tests: 3 new tests in test_ontology_dns_pinning.py (multi-IP fallback
success, all-unreachable failure, deduplicated multi-record resolution).
Full explorer + triplet_store suite: 569 passed.
Two follow-up hardening items flagged as secondary/deferred during
GHSA-8c7v-62gr-hj6g and GHSA-8vgg-8mr4-r236's fixes:
1. DNS check-then-use (TOCTOU) window in the ontology URL fetcher.
_validate_fetch_url() resolved and validated a hostname once, but
_fetch_url_sync() then let requests resolve the same hostname again
independently at connect time — a low-TTL or rebinding DNS answer
could differ between the two lookups, reopening the SSRF window the
validation exists to close.
_validate_fetch_url() now returns the validated IP, and a new
_make_pinned_session() builds a per-hop requests.Session whose
connection pool is pinned directly to that IP (bypassing DNS
resolution for the connection entirely), while explicitly restoring
the real hostname as the outgoing HTTP Host header and, for HTTPS,
the TLS SNI server_hostname/assert_hostname — so the connection
reaches the validated IP but still presents (and is verified
against) the real hostname's identity, keeping virtual hosting and
certificate validation correct.
Note: an earlier version of this fix set `_dns_host` post-construction
assuming it was decoupled from `host`, matching some other urllib3
releases; in the installed version (2.7.0), `host` is a property
that reads/writes `_dns_host` directly, so that approach silently
changed the Host header too. Verified with a real (non-mocked) local
HTTP server, a real local HTTPS server with a self-signed cert
(proving SNI/cert-hostname verification checks the real hostname,
not the pinned IP), and a negative control confirming a hostname/cert
mismatch is still correctly rejected — not silently bypassed.
2. Pre-wrapped object IRIs skipped full validation in
_format_object_for_sparql/_format_object_for_ntriples (Blazegraph,
RDF4J). A triplet object already wrapped in `<...>` only had its
inner content checked for a literal space or `>`, not run through
sparql_escaping.validate_uri() like the unwrapped-object branch —
flagged by automated review during GHSA-8vgg-8mr4-r236's fix. Both
branches now validate identically.
Tests: tests/explorer/test_ontology_dns_pinning.py (6 tests, including
2 real local-server end-to-end checks and 2 real-TLS checks with a
generated self-signed cert, gracefully skipped if `cryptography` isn't
installed); updated tests/explorer/test_ontology_ssrf.py for the new
per-hop session construction; 4 new tests in
tests/triplet_store/test_sparql_injection.py for the object-IRI fix.
Full explorer + triplet_store suite: 566 passed.
Two issues in the last round of commits:
1. explorer/auth.py added a new, opt-in APIKeyAuthMiddleware
(EXPLORER_API_KEY) and wired it into create_app(), but in doing so
removed the Depends(require_auth) dependency from every router and
deleted the /ws/graph-updates handshake check entirely. The new
middleware also fails OPEN (allows all requests) when its key is
unset, the opposite of require_auth's fail-closed design. Since
GHSA-j4mq-hprp-987v (the unauthenticated-Explorer-API advisory) is
already merged into main via require_auth, this would have reverted
a merged Critical fix the moment this branch merges. Removed
explorer/auth.py, restored the per-router dependencies and the
WebSocket auth check. Kept auth.py's one genuine improvement (adding
X-API-Key to the CORS allow_headers list) by folding it into the
existing CORS middleware config.
2. sparql.py's new _is_read_only_query() hardening (comment/PREFIX
stripping + forbidden-keyword scan) used `#[^\n]*` to strip SPARQL
comments, but a bare '#' also appears inside standard RDF namespace
IRIs (e.g. ".../1999/02/22-rdf-syntax-ns#") — the regex struck
everything after that '#' as a "comment", corrupting the query and
rejecting any legitimate SELECT using rdf:/rdfs:-style PREFIX
declarations. Confirmed by the fact the new hardening's own inlined
test copy failed against two of its own cases. Fixed by only
treating '#' as a comment-start at line-start or after whitespace,
which distinguishes ".../ns#" (preceded by a word character) from an
actual comment (preceded by whitespace/newline in every realistic
case, including the attacker's own comment-hiding PoC). Also fixed
the companion PREFIX/BASE regex, which required a prefix-name token
between the keyword and the IRI even for bare `BASE <...>`
declarations (which have none).
tests/test_security_regression.py's SPARQL section now imports the real
_is_read_only_query instead of maintaining a parallel inlined copy that
had silently drifted from — and shared the same bug as — the real
implementation; removed its TestAPIKeyAuth class (tested the now-deleted
auth.py) since equivalent, more thorough coverage already exists in
tests/explorer/test_explorer_auth.py. Updated tests/explorer/test_sparql_route.py's
multi-statement-injection test to reflect that the keyword scan now
catches "SELECT ... ; DROP ALL" itself rather than relying on rdflib's
parser, and added a new test confirming the parser still catches
multi-statement syntax that doesn't contain any forbidden keyword.
Full explorer/vector_store/security-regression/age_store suite: 543
passed (the only failures are 6 pre-existing, unrelated Pinecone-client
mocking issues).
The previous rework of the redirect loop closed the response on each
redirect hop but dropped the try/finally around the success path, so the
terminal response (the one actually read and returned) was left
unclosed, leaking the connection back to the pool unclosed under load.
Triplet.subject and Triplet.predicate (and, in some builders, .object)
were interpolated directly into SPARQL update/query strings in the
Blazegraph and RDF4J stores, and into a SELECT filter in the Jena store.
A subject containing '>' closes the '<...>' IRI token early, so the rest
of the value is parsed as more SPARQL. Entity names are document text in
the normal ingest pipeline, so anyone whose content gets processed could
append operations like CLEAR ALL, running with the application's store
credentials.
Applied the existing sparql_escaping.validate_uri (already used by
anzo_store.py, the one backend that was already hardened) at every
subject/predicate/object interpolation site:
- blazegraph_store.py: _build_insert_data, _triplets_to_rdf (unreachable
dead code today but same fix applied for consistency/future-proofing),
bulk_load's graph option, get_triplets's filter, delete_triplet.
- rdf4j_store.py: _triplets_to_ntriples, get_triplets's filter,
delete_triplet. (add_triplets's graph option was already validated.)
- jena_store.py: get_triplets's filter — the only vulnerable site;
add_triplets/delete_triplet already use rdflib's native Python API
(Graph.add/.remove with URIRef) rather than building query strings, so
they were never exploitable this way.
Added tests/triplet_store/test_sparql_injection.py (12 tests) reproducing
the advisory's own injection payload against all three backends' write
and read paths, asserting the malicious query is never built or sent.
Full triplet_store suite (330 tests) passes with no regressions.
Note: while adding read-path test coverage, found that jena_store.py's
get_triplets() WHERE-clause filter syntax is malformed SPARQL (missing a
FILTER()/separator before the equality conditions) — a pre-existing
correctness bug unrelated to this fix, worth a separate follow-up.
- vector_store.save(): use v.tolist() instead of list(v) so numpy float32
vectors round-trip through JSON instead of raising TypeError.
- ontology._fetch_url_sync(): resolve relative Location headers via urljoin
before re-validating (previously any relative redirect was rejected
outright), and close every response instead of leaking the connection
across redirect hops.
- sparql.execute_sparql(): move _build_rdflib_graph inside the handler's
error handling so the graph-size cap returns a clean SparqlResponse
error instead of an unhandled 500.
- add regression tests for all three.
The 1.0 / (1.0 + max(0.0, 1.0 - score)) normalization added in the last
commit clamped every raw score >= 1.0 to an identical 1.0, collapsing
result ranking for dot-product-metric indexes (unbounded), which cosine
(bounded to [-1, 1]) never exercised. Replaced with x/(1+|x|) rescaled
to (0, 1), which is strictly monotonic for any real score.
Also adds regression tests for scores >= 1 and a CHANGELOG entry.
self.indexer is only set for backend="inmemory", so save()/load() still
raised AttributeError for persistent backends (faiss, qdrant, etc.) even
after this PR's getattr() guards on self.vectors/self.metadata, since the
unguarded `self.indexer` access happened first. Guard it the same way and
delegate to the backend store's native save_index/load_index (currently
only FAISSStore implements these) so persistent-backend saves actually
persist instead of silently no-oping.
_get_candidate_embeddings()'s expand-and-retry loop widens the search
pool (up to limit*10) when post-filtering leaves too few candidates.
If the backend keeps returning a full page and filtered matches never
reach `limit`, the loop exited via the while condition instead of the
break branch, so the pre-loop empty embeddings/metadata/scores lists
were returned instead of the matches actually found in the final
iteration. This silently returned [] for filtered queries against
large persistent-backend stores even when matches existed - exactly
the scenario this PR adds support for.
Falls back to the last collected batch instead of discarding it.
Also documents this PR and #839 in the changelog.
Adds an Unreleased/Fixed entry for #841 (closes#840) — QdrantStore
search results were keyed "payload" instead of "metadata", breaking
HybridSearch.filter_by_metadata() for Qdrant results.
Adds an Unreleased/Added entry for #838 (closes#834), including the
follow-up fix that preserves ImportError for a missing pyoxigraph
install instead of masking it as a generic ProcessingError.
Upstream moved the v4 tag to 5595ccaf912efad79be6eef63a5619ff05969be3
(v4.37.6), which the repo's own verify-action-pins.sh now (correctly)
flags as a mismatch against the previously-pinned commit. Pre-existing
drift unrelated to #830/#836, but it was failing this PR's required
"verify" check, so fixing it here.
- Wire the Explorer frontend's node --test suites (test:graph-store,
test:graph-workspace, and the new test:plugin-registry regression
test) into CI. Previously only `npm run build` ran, so none of the
frontend tests -- including this fix's own regression coverage --
executed anywhere except a contributor's local machine.
- Broaden the diagnostics dedup's structureLayer comparison to also
cover disabledReason/curveCount/bridgeCurveCount/backboneCurveCount,
not just cacheKey/lastDrawAt/enabled, so a disabledReason-only
transition doesn't leave the dev diagnostics panel stale.
- Legacy top_k kwarg was read but not removed from options, so it got
forwarded via **options into VectorStore.search_vectors(), colliding
with backends that call search(..., top_k=k, **options) (e.g. sqlite,
pgvector) and raising "got multiple values for keyword argument
'top_k'". Now popped instead of just read.
- VectorStore.search_vectors()'s dispatch only recognized backend
methods named search/search_similar, so HybridSearch's delegation
still hit NotImplementedError for qdrant/milvus/pinecone, which name
their method search_vectors() with a differently-named count
parameter (limit vs k). Added a third dispatch branch that binds the
count positionally so it works regardless of the backend's parameter
name.
- Backend-delegated results defaulted a missing "distance" to the raw
score, silently reusing the local path's cosine-similarity convention
(distance = 1 - score) even for backends using unrelated metrics
(L2, inner product). A missing distance is now left as None instead
of a fabricated, metric-inconsistent value.
HybridSearch.search() directly accessed self.vector_store.vectors, a dict
that VectorStore only creates for backend="inmemory". Every other backend
(faiss, weaviate, qdrant, milvus, pinecone, pgvector, sqlite) raised
AttributeError. It now delegates to VectorStore.search_vectors() for
non-inmemory backends, applying metadata_filter as a post-filter and
normalizing results to a consistent {id, score, distance, metadata} shape.
Also fixes two related bugs surfaced while testing the backend-delegated
path end to end:
- vector_ids could remain None when explicit vectors/metadata were passed
without vector_ids, crashing downstream indexing.
- query_vector passed as a plain list crashed backend stores (e.g.
FAISSStore.search_similar) that call .ndim on it; now normalized to a
numpy array up front.
And in vector_store.py: VectorStore.store_vectors() silently dropped
metadata for FAISS (and any add_vectors-only backend) because it called
add_vectors(vectors, **options) without forwarding metadata, even though
FAISSStore.add_vectors() accepts it. This blocked HybridSearch's metadata
filtering from working at all against FAISS.
Fixes#833
- SQLiteStorage now migrates an existing (pre-#825) provenance.db in place
via ALTER TABLE ADD COLUMN for any columns introduced since, instead of
only ever running CREATE TABLE IF NOT EXISTS. Without this, opening an
older database with the new code would break on the first insert/select
since the row width and _row_to_entry's fixed indices grew past the old
schema. Added test_migrates_pre_existing_old_schema_database.
- verify_chain() now also checks that sequence_id is exactly the
predecessor's plus one (no gap, no duplicate), in addition to the existing
previous_checksum comparison. Hardens against the narrow case where
compute_checksum()'s deliberate exclusion of entity_id could let two
distinct rows coincidentally share a checksum, which alone would let a
checksum-only comparison miss a gap. Added
test_verify_chain_detects_tampered_sequence_gap.
- Explorer provenance route: edge ids now include direction
(f"{src}-{eid}-{direction}") to match the seen_edges dedupe key, which
already included it. The same (src, target) pair can legitimately appear
in both the upstream and downstream chains (cycles/overlap), and without
this the two edges collided on the same id. Added
test_add_chain_edges_ids_distinguish_direction.
- Removed an unused `Any` import in parse_provenance.py.
Part A - high-stakes trust blockers:
- Invalidation tombstones via ProvenanceManager.invalidate() (archive-then-append,
never mutates or deletes) instead of hard delete
- Hash-chained integrity: sequence_id/previous_checksum chain every entry to its
predecessor; new verify_chain() detects wholesale row deletion that a lone
per-row checksum cannot
- Typed Agent (AgentRecord: agent_type/is_automated) and Activity (ActivityRecord:
start/end timing), wired through all 18 *_provenance.py wrappers
- Split parent_entity_id into previous_version_id (correction) vs derived_from_id
(cross-source derivation), additive alongside the legacy combined field
- Downstream/descendant lineage traversal (get_descendants/trace_descendants,
reverse BFS) closing the dead direction="downstream" code path in the
Explorer's provenance route
- Qualified Association+hadRole and Invalidation in export_prov()
- New CLI: provenance invalidate|verify-chain|descendants
Part B - general PROV-O spec completeness:
- Qualified Generation/Usage/Derivation in export_prov()
- wasAssociatedWith, actedOnBehalfOf, wasInformedBy relations
- Bitemporal fields (valid_from/valid_until/revision_type/supersedes) plus
revision_history()/query_recorded_between(), closing the deprecated
kg.ProvenanceTracker's "no direct equivalent yet" migration gaps
- prov:Bundle/hadMember membership via bundle_id
- Configurable base_uri (--base-uri CLI flag), shared by RDFExporter's
NamespaceManager and OWLExporter's default ontology_uri so KG/OWL/PROV
exports co-resolve under one namespace instead of three hardcoded ones
Bugs fixed along the way:
- agent_id was a dead field: no track_* method read it from kwargs
- track_entities_batch silently absorbed typed kwargs into the metadata blob
- compute_checksum() had to exclude entity_id itself: hashing it made
track_entity's versioning-archive relabel permanently orphan any entry
already chained from the pre-relabel checksum, a false-positive "broken
chain" for a legitimate rename
- InMemoryStorage.get_chain_head() ignored the committed head whenever the
current transaction had staged entries, corrupting the next chain link
- several new ProvenanceEntry fields were wired into the dataclass and
export_prov() but not into SQLiteStorage's DDL/INSERT/row-mapping;
InMemoryStorage masked the gap. Added a permanent round-trip regression
test to catch this class of bug for future field additions
Flagged, not fixed (separate pre-existing issues, out of scope for #825):
- pipeline/pipeline_provenance.py imports a nonexistent module and wraps a
Pipeline dataclass with no run() method
- most *_provenance.py wrappers' backing classes are themselves missing or
incomplete (context_manager, deduplicator, normalizer, etc.)
- kg_provenance.py passes entity_type inside metadata={} instead of as a
top-level track_entity() kwarg across most of its call sites
- validate_skos_hierarchy() re-walked every existing hierarchy edge in
the graph on each write, so one pre-existing cycle anywhere would
block all unrelated future SKOS writes. It now only traverses
concepts touched by the edges actually being written, while still
checking those against existing edges for cross-boundary cycles.
- In /api/ontology/load, `except HTTPException: raise` sat after a
broader `except Exception` clause that already matched HTTPException,
so a 422 raised after a successful OntologyIngestor parse was
silently swallowed and retried via the fallback RDF parser instead of
reaching the caller. Reordered the except clauses.
Co-authored-by: mikemikimike <13286568797@163.com>
Signature introspection and the resulting call were sharing one
try/except, so a genuine bug inside a backend's get_causal_chain
(raising an unrelated TypeError) was misread as a signature mismatch,
causing an identical retry call before the real error surfaced.
Split introspection from the call so a successfully-introspected call
happens exactly once; the trial-and-error cascade now only runs when
inspect.signature itself fails. Also adds the CHANGELOG entry for
#781/#817, which was missing.
Two review findings on #807/#812:
- retrieve() and trace_lineage() were routed through transaction()'s
BEGIN IMMEDIATE, so plain reads took SQLite's writer lock and
serialized behind every other read/write, defeating the WAL
concurrency this PR was meant to add. They now use a dedicated
_read_connection() (configured, no explicit BEGIN).
- track_entity()/track_chunk() swallowed all internal storage
exceptions unconditionally, so a single item's failure inside
track_entities_batch()/track_chunks_batch()'s shared transaction
never reached the batch loop's per-item except, inflating
tracked_count for entries that were never persisted. Both now
re-raise when called with a shared _conn (batch context) while
still degrading gracefully on standalone calls.
Added regression tests for both, corrected the CHANGELOG entry and
docs that described the prior (overly broad) behavior.
- README: get_table_lineage() takes table_name first, then catalog/schema
keyword args — the example had them in the wrong order, which would have
queried lineage for the wrong fully-qualified table when copy-pasted.
- modules.md: the ingest example used DatabricksIngestor without importing
it, causing a NameError if copy-pasted as-is.
- guides/ingest.md: corrected the claim that Databricks/Snowflake ingestors
return "the same shape as DBIngestor" — DBIngestor.execute_query() returns
a raw List[Dict] with no wrapper, unlike DatabricksData/SnowflakeData.
Makes enterprise lakehouse/warehouse ingestion (Databricks Unity Catalog +
Delta Lake, Snowflake) a first-class, prominently documented capability
across the README and guides, and adds matching runnable examples to
docs/guides/ingest.md. Also fixes several pre-existing inaccuracies caught
while auditing the ingest module docs against the actual source:
WebIngestor has no ingest_urls() (only singular ingest_url()), XMLIngestor's
XSD option is schema_path (not validate_xsd) and belongs on ingest() not the
constructor, and the "Available ingestors" list was missing DatabricksIngestor
while listing several classes not actually exported from semantica.ingest.
Extracts the row-cap-and-truncate loop (duplicated between the
CONSTRUCT/DESCRIBE and SELECT branches) into a shared _cap_rows()
helper, and adds a test for the previously-uncovered CONSTRUCT/DESCRIBE
truncation path. Addresses review nits on PR #805.
- Revert create_ontology silently falling back to a near-empty ontology on
generation failure; restores the HTTPException(500) behavior from #770/#787
that this PR had accidentally undone (and re-enables TestOntologyCreateFailures)
- Fold sh:Warning/sh:Info severity pySHACL results into the /shacl/validate
response's violations array instead of silently dropping them, so a
non-conforming report is never returned with an empty violations list
- Share a single nodes/edges fetch between _generated_shacl_for_uri and
_data_graph_turtle_for_uri via new _fetch_analysis_graph(), so /health
no longer re-queries and re-truncation-checks the same ontology twice
- track_entity() no longer aliases a caller-supplied used_entities list
(it stored the reference directly and later mutated it via .append())
- Remove dead fallback branches in orchestrator.py/manager.py that
duplicated what Config.get()'s dotted-path resolution already does
- Add local --dry-run to `provenance audit` for parity with
`provenance export`
- `provenance check --strict` now warns instead of printing a success
checkmark before raising on a failed check
Add an Unreleased/Added entry for #786 covering the new export/import
Markdown format, idempotency and rollback guarantees, and the
Explorer/ContextGraph scoping decision from #765.
The set-state-in-effect refactor inlined each initial-fetch effect as a
standalone `fetchInitial`, duplicating the logic of the existing
reload/fetchOverview/fetchRegistry/loadVersions callbacks instead of
reusing them (required, since eslint-plugin-react-hooks v7 flags calling
an outside setState-touching function directly from an effect body, even
through an async gap - verified via a local lint probe). The duplicates
dropped the setError/flashMsg calls the originals had, so a failed
initial page load in AlignmentsTab, KGOverviewTab, OntologyManager, and
VersionsTab now failed silently instead of showing an error - a
regression of the exact bug #767/#790 fixed for these same files.
Also fixes LineageDiagram only clearing nodes/edges when the new
activeId was falsy, leaving the previous lineage view's stale diagram
on screen while switching directly between two ids.
The single-line checkov:skip=CKV_K8S_21 comment added in ed44260 was 286
characters, exceeding the repo's yamllint line-length rule (max 120,
.pre-commit-config.yaml). Split into three short comment lines: the skip
directive itself, then the rationale, in service.yaml, deployment.yaml,
and configmap.yaml.
Checkov's helm framework renders the chart without a namespace override,
so metadata.namespace (set to .Release.Namespace, bound only at install
time) always resolves to "default" and trips CKV_K8S_21 on service.yaml,
deployment.yaml, and configmap.yaml even though the chart is
namespace-agnostic by design.
Suppressed via per-file checkov:skip comments, following the same
convention already used for the Cloud Run false positives in
deploy/gcp/cloudrun-service.yaml.
Previously retryCount never reset on success (removed in adb4613 to
avoid resetting mid-Suspense-fallback), so unrelated transient errors
across a session could permanently exhaust the 3-retry budget even
though each prior retry had actually recovered. Now the counter resets
via a short settle timer after a retry stays error-free, avoiding both
the premature-reset and never-reset failure modes.
KGOverviewTab dropped the nodes-fetch 207 warning whenever stats also
returned 207; HealthTab and AlignmentsTab still had the exact
silent-swallow pattern this PR set out to fix elsewhere in the same
folder. Also documents all of #790's fixes in the changelog.
207 alone is indistinguishable from 200 to callers that only check
response.ok, so /api/analytics now raises 500 when every requested
metric fails and reserves 207 for genuine partial failure. Adds
regression tests for the temporal, analytics, and ontology-create
failure paths introduced in this PR, and logs the fix in the
changelog's Unreleased section.
Promotes the Unreleased changelog section (Databricks connector, SQLite
vector store, SPARQL CONSTRUCT templates, JenaStore named-graph support)
to 0.6.0 and syncs version references across pyproject.toml, __init__.py,
and docs.
- TemporalGraphQuery.query_time_range() and RDFExporter.export() both
expect {entities/relationships} (or {relationships} with source_id/
target_id keys), not ContextGraph.to_dict()'s {nodes, edges} shape.
Map the output before passing it in, and add an actual temporally-
bounded edge to the Temporal Intelligence example so the query has
something to find.
- add_causal_relationship() only accepts relationship_type values of
CAUSED, INFLUENCED, or PRECEDENT_FOR; replace the invented "triggers"/
"enables" values used in Decision Intelligence and the audit-trail
recipe, which would otherwise raise ValueError immediately.
PipelineBuilder.add_step() returns the created PipelineStep, not the
builder, so chaining .add_step().add_step() raised AttributeError.
Only connect_steps() and set_parallelism() return the builder and can
be chained.
Consolidates the platform reference into a single, premium README with
collapsible module/recipe sections so the docs and the deep-dive reference
no longer live in two places. Every code example was checked against the
actual semantica/ source and corrected where the API had drifted:
resolve_conflicts, register_source, ValidationResult.valid, clean_data,
execute_pipeline, ParquetExporter/LPGExporter/ReportGenerator calls,
graph.to_dict(), TemporalGraphQuery/TemporalNormalizer usage, the
Reasoner/ExplanationGenerator API, and the REST endpoint paths. Also
removed duplicated titles, snippets, and repeated example scenarios that
had crept in during the merge.
The CodeQL Analyze Python job failed on the #757 merge commit with
ECONNRESET while streaming the CodeQL bundle download in
codeql-action/init's "Setup CodeQL tools" step. This is unrelated to
the merged code — it's a known, currently-unaddressed gap in
codeql-action: the download error is retryable but the action doesn't
retry it internally (confirmed via codeql-action's issue tracker and
changelog).
Since a `uses:` step can't be wrapped by a shell-level retry action,
Initialize CodeQL now runs up to 3 times, cascading to the next
attempt only if the previous one failed, so the common case (success
on attempt 1) costs nothing extra.
Dataset.remove() on a bare 3-tuple resolves context=None internally,
which the underlying store treats as a wildcard and deletes the
matching triple from every graph, not just the default graph the
docstring promises. Pass self.graph.default_graph explicitly as the
context so delete_triplet stays scoped to the default graph, matching
the isolation guarantee default_union=False is meant to provide.
Also corrects a misleading comment: SPARQLStore is graph_aware=True
too, so graph-awareness isn't what requires SPARQLUpdateStore here —
it's SPARQLStore being read-only (.add()/.remove() raise TypeError).
Adds regression tests and a CHANGELOG entry for PR #757.
_format_object_for_ntriples decided IRI vs. literal purely from the
presence of datatype/lang metadata, defaulting anything without it to
<obj>. Any plain literal object (e.g. typical NER/extraction output
like "Alice", or an untyped Turtle literal round-tripped through the
new CONSTRUCT path) was wrapped as an invalid IRI instead of a quoted
literal, diverging from BlazegraphStore's _is_uri_value-first check.
Port _is_uri_value from BlazegraphStore so RDF4JStore checks whether
the object is actually URI-shaped before falling back to literal
handling, with a plain-quoted-literal fallback instead of <obj>.
Replace remaining Hawksight-AI GitHub org links and the old
semantica-dev noreply email with the current semantica-agi org
and kaif@getsemantica.ai contact, so security/support contacts
match pyproject.toml.
Documents the 9 pre-existing test failures caused by never-implemented
kg.ProvenanceTracker compatibility methods, the deprecation fix, and
the follow-up migration guide addition in this PR.
Every deprecation warning added in this PR (and the class docstring)
points to docs/migration/kg-provenance-tracker.md, but the file was
never added, so the reference was dead. Adds the guide with a
method-mapping table to semantica.provenance.ProvenanceManager.
Add graph-apache-age extra (psycopg2-binary) which was previously
undeclared despite age_store.py depending on it, wire it into
graph-all, and document install commands for FalkorDB/AGE/Neptune
alongside Neo4j. Note that RDF triple stores need no extra since they
talk SPARQL over HTTP via the core `requests` dependency. Also align
README's "Triplet Stores" table label to "Triple Stores (RDF)" to
match the standard term used elsewhere in the docs, while keeping the
TripletStore interface name in backticks.
Semantica already ships both an RDF triplet-store stack (Blazegraph,
Apache Jena, Eclipse RDF4J via a unified TripletStore/SPARQL interface)
and an LPG graph-store stack (Neo4j, FalkorDB, Apache AGE, AWS Neptune
via Cypher), but the README only surfaced the LPG side. Add a hero
highlight line, a "What Semantica gives you" bullet, and split the
Features-at-a-Glance table row so both formats and all backends are
named explicitly.
- get_table_lineage() gains include_column_lineage=True, resolving
per-column upstream/downstream references via Unity Catalog's
column-lineage API (one request per column, opt-in)
- DatabricksConnector.connect() now reuses an already-open connection
instead of opening a second one; ingest_table()/ingest_query() only
close the connection they opened themselves, so using the ingestor
as a context manager no longer leaks the connection opened by
__enter__
- get_table_schema()/get_table_lineage()/list_tables() now validate
both catalog and schema are resolved before calling Unity Catalog,
matching list_tables()'s existing catalog check
- 8 new regression tests (35 total)
Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.
Closes#747
Review follow-up: only append archived_history_id to used_entities when
explicit_parent_supplied is True. Previously it was appended unconditionally,
so the no-explicit-parent re-track path ended up with the same history id in
both parent_entity_id and used_entities, duplicating the reference in
get_lineage() output.
- get_lineage() aggregated metadata by iterating trace_lineage()'s BFS
order and calling dict.update() on each entry, so ancestor metadata
(now reachable via derived_from chains) could overwrite the queried
entity's own metadata on conflicting keys. Reverse the iteration so
the queried entity (always lineage_entries[0]) is applied last and
wins, matching the documented "most recent entry's metadata takes
precedence" intent.
- track_entity()'s derived_from guard only accepted a concrete dict,
silently ignoring other collections.abc.Mapping implementations
(e.g. types.MappingProxyType). Switch the isinstance check to
Mapping so any mapping-like metadata is honored.
Addresses Qodo review findings on PR #741.
track_entity() only auto-linked a parent by looking up `source` as an
existing entity_id, so two entities sharing a real source URL (e.g. a
document and a decision derived from it) never got connected, and
metadata["derived_from"] was stored but never consulted by any linking
or traversal code.
track_entity() now treats metadata["derived_from"] as an explicit
parent link (unless parent_entity_id was already passed directly), so
the existing BFS in trace_lineage() picks it up for free.
Closes#735
Rule is a mutable dataclass, so an already-registered rule's priority
could change after being added; the dedup early-return skipped the
priority re-sort, so re-adding a rule after mutating its priority
left self.rules stale relative to that change. The duplicate branch
now re-sorts before returning, matching the append path.
- add_rule()'s duplicate-skip path now logs at warning level instead
of debug, so a skipped duplicate is visible by default rather than
silent in typical logging configs
- The duplicate-rule log message now stringifies conditions via
map(str, ...) before joining, since Rule.conditions is List[Any]
and non-string entries would otherwise raise TypeError
add_rule() unconditionally appended to self.rules, so re-running the
same setup code on an existing Reasoner instance (e.g. re-executing a
Jupyter cell) duplicated every rule; forward_chain() would then match
the duplicated rules but silently return no new results since the
conclusions were already in self.facts, with no error or warning.
add_rule() now compares an incoming rule's rule_type, conditions, and
conclusion against existing rules and returns the existing Rule
instead of appending a duplicate, keeping repeated add_rule() calls
with the same definition idempotent.
'By default, initializing ... with storage_path=...' read as if passing
storage_path were the default, contradicting the very next sentence
about the no-argument in-memory default. Rephrased so the in-memory
default isn't undercut by the first sentence.
Step 5's illustrative explain_violations() output still referenced the
old cti.example.org/data/... node URIs after Step 4's data_ttl was
rewritten to use example.org/... URIs. Also removes now-unused
export_rdf/tempfile/os imports left over from replacing dynamic RDF
export with inline Turtle strings in five of the code examples.
Consolidate the hero around a single category claim (Context and
Accountability Layer for AI agents), drop the named-competitor
comparison table (LangChain/LlamaIndex/Mem0/Zep/Palantir Foundry),
remove GitHub alert-box tips/notes, cut redundant module/changelog
sections, strip vanity feature counts, and trim em dashes for a
cleaner, more premium read.
- Add SQLITE_VEC_AVAILABLE flag via importlib.util.find_spec so the test
suite's skipif actually reflects whether sqlite-vec is installed; it was
previously undefined, causing all sqlite vector store tests to be
silently skipped regardless of installation state.
- Actually apply PRAGMA synchronous=NORMAL alongside journal_mode=WAL when
use_wal=True, matching the documented behavior; document use_wal as an
opt-in kwarg in the docstring and usage guide.
- Correct _is_safe_identifier error messages (regex never allowed hyphens).
- Batch get() and update() with IN(...)/executemany instead of per-id
round trips, consistent with add()/delete().
- Fix flaky test_update_vectors assertion that relied on list.index()
over dicts containing numpy arrays.
- Reorder sqlite_vec_store import alphabetically in vector_store/__init__.py.
Co-Authored-By: Luffy2208 <209925020+Luffy2208@users.noreply.github.com>
Fix stale/incorrect weight and confidence figures in the conflict
resolution guide that don't match actual resolver output, and update
a leftover credibility_score field reference in Common Pitfalls.
The merging example told readers to use merge_entity_group() for
already-confirmed duplicate groups, but the code right below it still
called merge_duplicates() on group.entities, which re-runs duplicate
detection redundantly. Update the call to match the stated guidance.
- Correct the unsupported-rule-key pitfall: absent keys fail compliance,
present keys (any value) pass — the previous wording had this backwards.
- Remove required_ltv/pd/lgd/dsti/credit_score: True from the mortgage
example. required_* checks equality against the given value, so True
against a real numeric field silently marks compliant decisions as
non-compliant (verified: a fully passing decision still returned False).
The min_/max_ rules already enforce presence of ltv/dsti/credit_score.
FileObject.content is raw bytes; AgentContext.store() only accepts str/list and raises ValueError on bytes, so the previous fix commit broke both domain examples.
KaifAhmad1