* feat(triplet_store): add Altair Anzo triplet store backend
Adds AnzoStore as a fourth peer to BlazegraphStore/RDF4JStore/JenaStore,
speaking plain SPARQL 1.1 over HTTP (no new dependency needed). The one
structural difference from the existing backends is that Anzo addresses
data by a dataset/graphmart URI rather than a short namespace/repository
name, so the endpoint path percent-encodes it. Reuses the shared
sparql_escaping.py helpers and wires "anzo" into TripletStore's backend
dispatch and config env vars.
Closes#813
* fix(triplet_store): correct AnzoStore SPARQL syntax and validate IRIs
Addresses review findings from Qodo and Codex on PR #814:
- get_triplets(): constraints are now expressed via FILTER(...) instead of
bare equality expressions appended inside the WHERE group graph pattern
(e.g. "?s ?p ?o ?s = <...>"), which is not valid SPARQL and was rejected
by standards-compliant endpoints.
- bulk_load(): named-graph inserts now nest the GRAPH block inside the
INSERT DATA braces (INSERT DATA { GRAPH <g> { ... } }) per the SPARQL 1.1
Update grammar, instead of "INSERT DATA GRAPH <g> { ... }".
- bulk_load()/_build_insert_data()/delete_triplet()/get_triplets() now
validate subject/predicate/graph URIs via sparql_escaping.validate_uri
before interpolating them into SPARQL Update/Query strings, closing an
injection path where a value containing ">" or "}" could break out of
the intended <...> token.
- Corrected the store_type docstring/usage example: Anzo's linked-data-set
store type is "lds", not "dataset".
Extended tests/triplet_store/test_anzo_store.py with coverage for the
corrected query shapes and the new validation/injection-rejection paths
(38 tests total, up from 32). Full tests/triplet_store/ suite: 299/299
passing.
* test(triplet_store): expand AnzoStore regression coverage
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* refactor(provenance): centralize duplicated store-and-swallow logic into _save_entry() (closes#784)
- Added ProvenanceManager._save_entry(entry) as the single shared
checksum-compute + storage.store() + graceful-failure-swallow
pipeline, previously duplicated identically across track_entity,
track_relationship, track_chunk, and track_property_source.
- track_entities_batch/track_chunks_batch already delegate to
track_entity/track_chunk in a loop, so they inherit the fix for
free — left untouched, confirmed no direct duplication there.
- Byte-for-byte preserves today's swallow-and-continue behavior and
comment text; this is an architecture-only refactor. The silent-
failure behavior itself is unchanged and out of scope here — a fix
to it now only needs to happen in one place instead of four.
- Added 4 new regression tests (previously 0 of the 4 single-item
methods had failure-path coverage) proving storage.store() raising
is still caught and each method still returns its ProvenanceEntry.
Tests: tests/provenance/ 228 passed (+4 new), tests/explorer/test_provenance_manager_wiring.py 8 passed. 236/236, 0 failed.
* fix(provenance): drop out-of-transaction store attempt in track_entity fallback
The _save_entry refactor changed track_entity's pre-build exception
fallback (entry is None branch) to call _save_entry(), which makes a
real self.storage.store(entry) call. The original code only computed
a checksum here and never attempted storage again, since this branch
fires when something already failed before the entry was built inside
the atomic transaction. Storing outside that transaction bypasses the
BEGIN IMMEDIATE serialization #807 added, risking the same race it
fixed. Restored checksum-only behavior and added a regression test
asserting storage.store is not called on this path.
Also removed an untested hasattr(_store_with_conn) defensive branch
added during the refactor that wasn't in the original code, and added
a changelog entry.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Two review findings on #807/#812:
- retrieve() and trace_lineage() were routed through transaction()'s
BEGIN IMMEDIATE, so plain reads took SQLite's writer lock and
serialized behind every other read/write, defeating the WAL
concurrency this PR was meant to add. They now use a dedicated
_read_connection() (configured, no explicit BEGIN).
- track_entity()/track_chunk() swallowed all internal storage
exceptions unconditionally, so a single item's failure inside
track_entities_batch()/track_chunks_batch()'s shared transaction
never reached the batch loop's per-item except, inflating
tracked_count for entries that were never persisted. Both now
re-raise when called with a shared _conn (batch context) while
still degrading gracefully on standalone calls.
Added regression tests for both, corrected the CHANGELOG entry and
docs that described the prior (overly broad) behavior.
- Extend ProvenanceNode in semantica/explorer/schemas.py with audit evidence fields: source_document, source_location, source_quote, confidence, and checksum.
- Update _transform_audit_lineage in semantica/explorer/routes/provenance.py to populate these evidence fields for each lineage node from ProvenanceEntry records, while keeping default None values for orphan nodes.
- Include source_document, confidence, and checksum in markdown report rendering (_render_markdown) so exported markdown reports surface attribution and integrity evidence.
- Add unit test test_provenance_audit_evidence_fields_preserved in test_provenance_manager_wiring.py verifying that evidence fields are present across /api/provenance JSON responses and exported JSON/markdown reports.
- Update verify_checksum and compute_checksum in semantica/provenance/integrity.py to support both ProvenanceEntry objects and serialized dictionary entries.
- Add integrity_verified flag computed via verify_checksum to the dictionary returned by ProvenanceManager.get_lineage().
- Update _build_provenance in semantica/explorer/routes/provenance.py to verify every returned lineage entry before labeling the result as source=audit. If verification fails due to missing checksums or corrupted records, log a warning and fall back cleanly to graph traversal.
- Add unit test test_provenance_manager_wiring_checksum_failure_falls_back in test_provenance_manager_wiring.py verifying that tampered lineage entries trigger fallback to source=graph_traversal.
- Fix _transform_audit_lineage to classify all non-downstream ancestor derivation edges as 'upstream' instead of 'lateral', correcting multi-hop lineage direction in JSON and markdown reports.
- Add GraphSession.set_provenance_storage_path() to explicitly reject conflicting preconfigured storage paths or path mutations after provenance_manager initialization.
- Update create_app() to call active_session.set_provenance_storage_path(prov_path), preventing silent retention of conflicting paths or un-redirectable cached managers.
- Remove unused logging import in app.py.
- Add comprehensive unit tests in test_provenance_manager_wiring.py for upstream edge classification, markdown report grouping, conflicting path rejection, and manager initialization lockouts.
- Explorer's /api/provenance now queries the audit-grade ProvenanceManager
(SQLite-backed, checksummed) first, falling back to the naive 2-hop
graph traversal when no audit records exist for a node.
- Fixed a process-global mutable-state risk in the initial approach:
provenance storage path is threaded per-session via GraphSession,
not via ProvenanceManager's global set_default_storage_path classmethod.
- Added source: 'audit' | 'graph_traversal' to the response so callers
can distinguish which path served the data.
- Documented a known limitation: ProvenanceManager currently only
traces upstream/ancestor lineage, not descendants — the naive
fallback remains the only source for downstream relationships until
ProvenanceManager gains a reverse lookup (tracked separately).
- Warns (rather than silently no-ops) if a provided session's
provenance_manager was already constructed before create_app()
applied a provenance_storage_path.
- Never lets a provenance-manager failure crash the route; degrades
to the naive path with a logged warning instead.
Tests: 5 new tests in test_provenance_manager_wiring.py covering the
audit path, empty-record fallback, storage-failure degradation, app
startup wiring, and cross-session storage isolation. Full
tests/explorer/ + tests/provenance/ suite passing, order-invariant.
Extracts the row-cap-and-truncate loop (duplicated between the
CONSTRUCT/DESCRIBE and SELECT branches) into a shared _cap_rows()
helper, and adds a test for the previously-uncovered CONSTRUCT/DESCRIBE
truncation path. Addresses review nits on PR #805.
- Revert create_ontology silently falling back to a near-empty ontology on
generation failure; restores the HTTPException(500) behavior from #770/#787
that this PR had accidentally undone (and re-enables TestOntologyCreateFailures)
- Fold sh:Warning/sh:Info severity pySHACL results into the /shacl/validate
response's violations array instead of silently dropping them, so a
non-conforming report is never returned with an empty violations list
- Share a single nodes/edges fetch between _generated_shacl_for_uri and
_data_graph_turtle_for_uri via new _fetch_analysis_graph(), so /health
no longer re-queries and re-truncation-checks the same ontology twice
sparql.py handles direct SPARQL query execution against the live graph with no test coverage anywhere in the repo. Adds coverage for the read-only allowlist (the actual security boundary here), row/timeout limits, error handling, and RDF projection fidelity.
- Add entries alias in get_lineage() return dictionary so CLI and programmatic callers can access lineage entries via either key
- Update lineage() wrapper method to fallback to lineage_chain when entries is missing
- Add assertions in test_cli_lineage confirming lineage and entries lists are non-empty
- Add _default_storage_path, set_default_storage_path(), and test-isolation context manager default_storage_path() in ProvenanceManager
- Accept config kwarg in ProvenanceManager.__init__ to fix CLI initialization bug
- Implement audit_log(), lineage(), export_prov(), and check() on ProvenanceManager matching cli.py expectations
- Wire provenance.storage_path in Semantica.__init__ before pipeline stages execute
- Add comprehensive unit tests in tests/provenance/test_manager.py for CLI methods and test isolation
207 alone is indistinguishable from 200 to callers that only check
response.ok, so /api/analytics now raises 500 when every requested
metric fails and reserves 207 for genuine partial failure. Adds
regression tests for the temporal, analytics, and ontology-create
failure paths introduced in this PR, and logs the fix in the
changelog's Unreleased section.
Dataset.remove() on a bare 3-tuple resolves context=None internally,
which the underlying store treats as a wildcard and deletes the
matching triple from every graph, not just the default graph the
docstring promises. Pass self.graph.default_graph explicitly as the
context so delete_triplet stays scoped to the default graph, matching
the isolation guarantee default_union=False is meant to provide.
Also corrects a misleading comment: SPARQLStore is graph_aware=True
too, so graph-awareness isn't what requires SPARQLUpdateStore here —
it's SPARQLStore being read-only (.add()/.remove() raise TypeError).
Adds regression tests and a CHANGELOG entry for PR #757.
1. Endpoint derivation regression: Detect if self.endpoint already contains
a Fuseki service suffix (/query, /update, /sparql) to prevent double-appending
(e.g., /ds/query/query). If it does, derive the base and construct both
paths properly.
2. Misleading serialize warning: Limit the named-graph data loss warning
to single-graph serializer formats (turtle, xml, n3, etc.). Multi-graph
formats (trig, nquads, nt) will correctly serialize all graphs without warning.
3. Zero-added error misdiagnosis: Track malformed triples accurately in
add_triplets(). If every triplet fails the local validation (ValueError/
AttributeError), raise a formatting-oriented ProcessingError instead of
assuming a store connectivity issue.
Includes comprehensive regression tests for all three cases.
Migrates JenaStore from rdflib.Graph to rdflib.Dataset with default_union=False
explicitly set, per maintainer-confirmed architecture for issue #756.
Changes:
- _initialize_graph: construct self.graph as Dataset(default_union=False) for
the in-memory path, and Dataset(store=SPARQLUpdateStore(...), default_union=False)
for the remote path. SPARQLUpdateStore.graph_aware=True satisfies Dataset's
hard requirement. Both paths verified against rdflib source.
- add_triplets: accept and honor graph= option. When supplied, Dataset.graph(uri)
creates/retrieves the named-graph context and the triple is written via a
4-tuple (which SPARQLUpdateStore maps to INSERT DATA { GRAPH <uri> { ... } }).
When graph= is omitted, the 3-tuple path routes to Dataset's default graph,
preserving pre-migration semantics exactly.
- serialize: add WARNING log when named-graph content would be silently dropped
by a single-graph serializer (turtle/xml/n3). Log includes triple count and
recommends trig/nquads formats. No warning when only the default graph is used.
- create_model: document that triplet_count now counts triples across all graphs
(default + named) as a consequence of this migration. Semantics shift made
visible, not silent.
- delete_triplet: document that graph= parity is a known gap, deferred to a
future follow-up per maintainer's stated scope (add_triplets only).
Decisions applied:
1. triplet_count semantics shift: documented in create_model docstring
2. delete_triplet graph= parity: explicitly out of scope, noted in docstring
3. Existing store.graph=Graph() tests: left unchanged; new tests added
to cover the real _initialize_graph path
Tests added (TestJenaStoreDatasetMigration):
- test_initialize_graph_produces_dataset_not_graph
- test_initialize_graph_dataset_has_default_union_false
- test_add_triplets_with_graph_option_writes_to_named_graph
- test_add_triplets_without_graph_option_writes_to_default_graph
- test_add_triplets_named_graph_isolated_from_default_query
- test_serialize_logs_warning_when_named_graph_content_present
- test_serialize_no_warning_when_only_default_graph_used
Also updated test_add_triplets_remote_endpoint_fires_insert_data_via_update_store
to patch Dataset instead of Graph (the remote path now creates Dataset(store=...)).
Full suite: 269 passed, 0 failed (tests/triplet_store/ + tests/pipeline/)
The remote-endpoint path in _initialize_graph was instantiating the read-only
rdflib SPARQLStore, causing every add_triplets() call against a remote Fuseki
endpoint to silently fail: SPARQLStore.add() raises TypeError which was swallowed
by the broad except Exception per-triplet handler and returned as success=True/added=0.
Changes:
- Import SPARQLUpdateStore alongside SPARQLStore
- _initialize_graph: use SPARQLUpdateStore(query_endpoint=<base>/query,
update_endpoint=<base>/update) per standard Fuseki REST API conventions
- Fix constructor: self.endpoint=config.get('endpoint') always returned None
because the named positional 'endpoint' param captures the kwarg before **config;
now uses endpoint or config.get('endpoint')
- Narrow per-triplet except to (ValueError, AttributeError); add ProcessingError
when entire batch fails to prevent misleading success=True/added=0 return
Tests added (TestJenaStoreRemoteEndpointUsesUpdateStore): 4 new test cases
Full suite: 262 passed (tests/triplet_store/ + tests/pipeline/)
_format_object_for_ntriples decided IRI vs. literal purely from the
presence of datatype/lang metadata, defaulting anything without it to
<obj>. Any plain literal object (e.g. typical NER/extraction output
like "Alice", or an untyped Turtle literal round-tripped through the
new CONSTRUCT path) was wrapped as an invalid IRI instead of a quoted
literal, diverging from BlazegraphStore's _is_uri_value-first check.
Port _is_uri_value from BlazegraphStore so RDF4JStore checks whether
the object is actually URI-shaped before falling back to literal
handling, with a plain-quoted-literal fallback instead of <obj>.
Extends CONSTRUCT support to JenaStore, which uses rdflib.Graph natively rather
than an HTTP protocol - CONSTRUCT results come as native 3-tuples with no
Accept-header/parsing dance needed, unlike Blazegraph/RDF4J.
- CONSTRUCT-aware execute_sparql: reuses shared sparql_escaping.CONSTRUCT_QUERY_RE,
extracts datatype/language from rdflib Literal objects into the same 4-tuple
metadata contract used by Blazegraph/RDF4J
- Non-CONSTRUCT path (SELECT/ASK) confirmed byte-for-byte unchanged (Property 9)
- execute_construct_template confirmed backend-agnostic against JenaStore, zero
changes needed
- Named-graph support explicitly out of scope - JenaStore wraps a single
rdflib.Graph with no named-graph concept; add_triplets continues to silently
ignore graph= exactly as before. Tracked separately as a follow-up issue
requiring a Graph -> ConjunctiveGraph/Dataset migration.
Extends the Blazegraph-only CONSTRUCT support from #322 (commit 4f2c6c82's
approved pattern) to RDF4JStore:
- CONSTRUCT-aware execute_sparql: Accept: text/turtle, rdflib Turtle parsing,
4-tuple (s, p, o, metadata) contract with datatype/language preservation
- Named-graph writes via RDF4J's REST context parameter, N-Triples-encoded
(angle-bracket-wrapped IRI), confirmed against RDF4J's Protocol.java source
- graph=None preserves existing behavior exactly (no context param sent,
not context=null - verified as a distinct, deliberate choice)
- _CONSTRUCT_QUERY_RE moved to sparql_escaping.py as a shared, backend-agnostic
constant; Blazegraph now delegates to it, zero behavioral change confirmed
- execute_construct_template (construct_templates.py) required zero changes -
confirmed backend-agnostic via end-to-end integration tests against RDF4JStore
29 new tests, full suite 245/245 passing. Jena support remains out of scope
for this PR - tracked separately in #754's remaining scope.
Implements #322: ConstructTemplate/ParameterDescriptor/ConstructTemplateRegistry
with injection-safe {{param}} rendering, Blazegraph CONSTRUCT-aware execute_sparql
extension, execute_construct_template (render->execute->parse->persist), and a
construct_template pipeline step. RDF4J/Jena support deferred to a follow-up issue.
Closes#322
- get_table_lineage() gains include_column_lineage=True, resolving
per-column upstream/downstream references via Unity Catalog's
column-lineage API (one request per column, opt-in)
- DatabricksConnector.connect() now reuses an already-open connection
instead of opening a second one; ingest_table()/ingest_query() only
close the connection they opened themselves, so using the ingestor
as a context manager no longer leaks the connection opened by
__enter__
- get_table_schema()/get_table_lineage()/list_tables() now validate
both catalog and schema are resolved before calling Unity Catalog,
matching list_tables()'s existing catalog check
- 8 new regression tests (35 total)
Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.
Closes#747
Sameer6305