- Replace direct .vectors and .metadata access with VectorStore.search_vectors().
- Add a fallback in HybridSimilarityCalculator (via ind_similar_decisions) to use the search score when backend vector databases do not natively return the raw vector array.
- Fix get_decision_statistics to gracefully fall back when .metadata is not fully supported by the underlying DB.
- Add regression tests utilizing the real FAISS and inmemory backends directly without mocking.
QdrantStore.search_vectors() returned results keyed by "payload" while
HybridSearch and PineconeStore both expect/return "metadata". This silently
dropped metadata from Qdrant results and caused HybridSearch.filter_by_metadata
to reject every candidate when a filter was applied (empty result sets).
Fixes#840
* fix(vector_store): stop dropping metadata for add_vectors-only backends
VectorStore.store_vectors() previously discarded the metadata argument
whenever the backend only exposed add_vectors() (e.g. FAISSStore), even
though add_vectors() supports it. Now metadata is forwarded, and is only
passed when the backend's add_vectors() signature actually accepts it
(checked via inspect.signature), avoiding a TypeError for stricter
backend signatures.
Fixes#832
* fix(vector_store): guard signature introspection in store_vectors
inspect.signature() can raise ValueError/TypeError for some callables
(e.g. certain C-implemented or dynamically built methods). Wrap the
add_vectors() signature probe in try/except, consistent with the same
pattern already used in ProvenanceManager.trace_lineage(), defaulting
to attempting to pass metadata when introspection fails.
* docs(changelog): document VectorStore metadata-drop fix (#832, #835)
* test(vector_store): add regression coverage for metadata forwarding
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
- SQLiteStorage now migrates an existing (pre-#825) provenance.db in place
via ALTER TABLE ADD COLUMN for any columns introduced since, instead of
only ever running CREATE TABLE IF NOT EXISTS. Without this, opening an
older database with the new code would break on the first insert/select
since the row width and _row_to_entry's fixed indices grew past the old
schema. Added test_migrates_pre_existing_old_schema_database.
- verify_chain() now also checks that sequence_id is exactly the
predecessor's plus one (no gap, no duplicate), in addition to the existing
previous_checksum comparison. Hardens against the narrow case where
compute_checksum()'s deliberate exclusion of entity_id could let two
distinct rows coincidentally share a checksum, which alone would let a
checksum-only comparison miss a gap. Added
test_verify_chain_detects_tampered_sequence_gap.
- Explorer provenance route: edge ids now include direction
(f"{src}-{eid}-{direction}") to match the seen_edges dedupe key, which
already included it. The same (src, target) pair can legitimately appear
in both the upstream and downstream chains (cycles/overlap), and without
this the two edges collided on the same id. Added
test_add_chain_edges_ids_distinguish_direction.
- Removed an unused `Any` import in parse_provenance.py.
Part A - high-stakes trust blockers:
- Invalidation tombstones via ProvenanceManager.invalidate() (archive-then-append,
never mutates or deletes) instead of hard delete
- Hash-chained integrity: sequence_id/previous_checksum chain every entry to its
predecessor; new verify_chain() detects wholesale row deletion that a lone
per-row checksum cannot
- Typed Agent (AgentRecord: agent_type/is_automated) and Activity (ActivityRecord:
start/end timing), wired through all 18 *_provenance.py wrappers
- Split parent_entity_id into previous_version_id (correction) vs derived_from_id
(cross-source derivation), additive alongside the legacy combined field
- Downstream/descendant lineage traversal (get_descendants/trace_descendants,
reverse BFS) closing the dead direction="downstream" code path in the
Explorer's provenance route
- Qualified Association+hadRole and Invalidation in export_prov()
- New CLI: provenance invalidate|verify-chain|descendants
Part B - general PROV-O spec completeness:
- Qualified Generation/Usage/Derivation in export_prov()
- wasAssociatedWith, actedOnBehalfOf, wasInformedBy relations
- Bitemporal fields (valid_from/valid_until/revision_type/supersedes) plus
revision_history()/query_recorded_between(), closing the deprecated
kg.ProvenanceTracker's "no direct equivalent yet" migration gaps
- prov:Bundle/hadMember membership via bundle_id
- Configurable base_uri (--base-uri CLI flag), shared by RDFExporter's
NamespaceManager and OWLExporter's default ontology_uri so KG/OWL/PROV
exports co-resolve under one namespace instead of three hardcoded ones
Bugs fixed along the way:
- agent_id was a dead field: no track_* method read it from kwargs
- track_entities_batch silently absorbed typed kwargs into the metadata blob
- compute_checksum() had to exclude entity_id itself: hashing it made
track_entity's versioning-archive relabel permanently orphan any entry
already chained from the pre-relabel checksum, a false-positive "broken
chain" for a legitimate rename
- InMemoryStorage.get_chain_head() ignored the committed head whenever the
current transaction had staged entries, corrupting the next chain link
- several new ProvenanceEntry fields were wired into the dataclass and
export_prov() but not into SQLiteStorage's DDL/INSERT/row-mapping;
InMemoryStorage masked the gap. Added a permanent round-trip regression
test to catch this class of bug for future field additions
Flagged, not fixed (separate pre-existing issues, out of scope for #825):
- pipeline/pipeline_provenance.py imports a nonexistent module and wraps a
Pipeline dataclass with no run() method
- most *_provenance.py wrappers' backing classes are themselves missing or
incomplete (context_manager, deduplicator, normalizer, etc.)
- kg_provenance.py passes entity_type inside metadata={} instead of as a
top-level track_entity() kwarg across most of its call sites
* fix(agno): surface unevaluable policy rules (#778)
* fix(agno): fixed qodo reviews
check_policy previously let unevaluable policy rules silently return
compliant=True with no signal (issue #778): a rule referencing a field
missing from decision_data, or a rule string not matching the expected
<field> <op> <value> format, both fell through _eval_rule's `return
True` and were treated as passed.
Both now raise ValueError, which routes through check_policy's existing
exception handler and surfaces as a `warnings` entry instead. compliant/
violations semantics are unchanged for every case that previously worked
correctly; an unevaluable rule is not counted as a violation since it's
genuinely unknown whether it would have passed.
Follow-up fixes from code review:
- policy_rules decoded via json.loads without checking it was a list;
a JSON-encoded bare string decoded to a Python str, so iterating it
evaluated one "rule" per character, amplifying a single input-shape
mistake into a wall of per-character warnings. A decoded string is
now treated as a single rule; any other non-list shape or non-string
list element produces exactly one warning instead.
- _eval_rule used `data.get(field) is None` to detect a missing field,
which can't distinguish an absent key from a key present with JSON
null - both produced the same "undefined field" warning. Field
presence is now checked with `field not in data` first, and a
present-but-null value gets its own distinct message.
Added regression tests for all of the above in
tests/integrations/agno/test_decision_kit.py (38 tests in the file,
128 passing across tests/integrations/agno/).
* fix(agno): reject non-object decision_data in check_policy
check_policy only validated that decision_data was well-formed JSON,
not that it decoded to an object. When it decoded to a list, `field
not in data` in _eval_rule silently became list-membership testing
of values instead of a dict key check - e.g. "confidence" not in
["confidence", 0.95] evaluates to False - so a matching rule fell
through to data["confidence"], raising a raw internal TypeError
("list indices must be integers or slices, not str") instead of any
meaningful diagnostic. Numbers, strings, and bools produced similarly
opaque TypeErrors deep inside _eval_rule.
check_policy now checks isinstance(data, dict) right after decoding
and rejects any other shape with a single clear violations entry,
the same way it already rejects malformed JSON.
Added 5 regression tests in tests/integrations/agno/test_decision_kit.py
covering list/number/string/bool/null decision_data shapes (43 tests
in the file, 133 passing across tests/integrations/agno/).
Addresses Copilot PR review comment on the #778 fix branch.
* docs(changelog): reference PR #822 in the check_policy changelog entry
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
- Removed unused `validate_skos_hierarchy` import from
test_ontology_subissue3.py (flake8 F401); the test uses a `wraps=` spy
on the real add_nodes_and_edges instead of calling the helper directly.
- refresh_ontology tests now percent-encode the ontology URI with
urllib.parse.quote before interpolating it into the {ontology_uri:path}
request path, matching the already-encoded unknown-uri refresh test in
the same file instead of embedding a raw http://... URI with slashes.
- Reworded the cyclic-SKOS refresh test's comment and section header:
GraphSession.add_nodes_and_edges() documents pre-write validation and
lock-based mutual exclusion, not transactional rollback, so "atomic"
was replaced with "single combined add_nodes_and_edges() call" to avoid
implying rollback guarantees that don't exist.
Verified: tests/explorer/test_ontology_subissue3.py (34 passed) and
tests/explorer/ (204 passed), no regressions.
- validate_skos_hierarchy() re-walked every existing hierarchy edge in
the graph on each write, so one pre-existing cycle anywhere would
block all unrelated future SKOS writes. It now only traverses
concepts touched by the edges actually being written, while still
checking those against existing edges for cross-boundary cycles.
- In /api/ontology/load, `except HTTPException: raise` sat after a
broader `except Exception` clause that already matched HTTPException,
so a 422 raised after a successful OntologyIngestor parse was
silently swallowed and retried via the fallback RDF parser instead of
reaching the caller. Reordered the except clauses.
Co-authored-by: mikemikimike <13286568797@163.com>
- split.ProvenanceTracker: Check _unified_manager.track_chunk() return value and fall back to legacy storage when None is returned (storage failure)
- split.ProvenanceTracker: Initialize legacy stores (_provenance_store and _chunk_registry) unconditionally in __init__ so legacy fallback storage works safely even when initialized with use_unified=True
- split.ProvenanceTracker: Update get_provenance() to check legacy storage when no record is found in unified storage, ensuring fallback-tracked chunks remain retrievable
- SourceTracker: Change track_sources_batch() condition from if entry is not None: to if ok: to respect the boolean return contract (-> bool) of track_entity_source(), track_property_source(), and track_relationship_source()
- Tests: Add regression test test_unified_tracking_returns_none_triggers_fallback and update test_track_sources_batch_failure_not_counted to test boolean failure return_value=False
Signature introspection and the resulting call were sharing one
try/except, so a genuine bug inside a backend's get_causal_chain
(raising an unrelated TypeError) was misread as a signature mismatch,
causing an identical retry call before the real error surfaced.
Split introspection from the call so a successfully-introspected call
happens exactly once; the trial-and-error cascade now only runs when
inspect.signature itself fails. Also adds the CHANGELOG entry for
#781/#817, which was missing.
- Add safe input validation and bounds clamping on max_depth (1..100) to prevent DoS/memory exhaustion
- Use inspect.signature for accurate keyword dispatch with precise TypeError fallback
- Prevent masking of genuine internal TypeError exceptions inside graph backends
- Add security and input hardening regression tests
- Support legacy (depth kwarg) and positional-only get_causal_chain backend signatures in fallback path
- Add regression tests for signature compatibility
- Return explicit error dictionary when graph lacks get_causal_chain instead of silent empty list
- Forward direction and max_depth in fallback graph.get_causal_chain call
- Add regression tests for error signaling and parameter forwarding
* feat(triplet_store): add Altair Anzo triplet store backend
Adds AnzoStore as a fourth peer to BlazegraphStore/RDF4JStore/JenaStore,
speaking plain SPARQL 1.1 over HTTP (no new dependency needed). The one
structural difference from the existing backends is that Anzo addresses
data by a dataset/graphmart URI rather than a short namespace/repository
name, so the endpoint path percent-encodes it. Reuses the shared
sparql_escaping.py helpers and wires "anzo" into TripletStore's backend
dispatch and config env vars.
Closes#813
* fix(triplet_store): correct AnzoStore SPARQL syntax and validate IRIs
Addresses review findings from Qodo and Codex on PR #814:
- get_triplets(): constraints are now expressed via FILTER(...) instead of
bare equality expressions appended inside the WHERE group graph pattern
(e.g. "?s ?p ?o ?s = <...>"), which is not valid SPARQL and was rejected
by standards-compliant endpoints.
- bulk_load(): named-graph inserts now nest the GRAPH block inside the
INSERT DATA braces (INSERT DATA { GRAPH <g> { ... } }) per the SPARQL 1.1
Update grammar, instead of "INSERT DATA GRAPH <g> { ... }".
- bulk_load()/_build_insert_data()/delete_triplet()/get_triplets() now
validate subject/predicate/graph URIs via sparql_escaping.validate_uri
before interpolating them into SPARQL Update/Query strings, closing an
injection path where a value containing ">" or "}" could break out of
the intended <...> token.
- Corrected the store_type docstring/usage example: Anzo's linked-data-set
store type is "lds", not "dataset".
Extended tests/triplet_store/test_anzo_store.py with coverage for the
corrected query shapes and the new validation/injection-rejection paths
(38 tests total, up from 32). Full tests/triplet_store/ suite: 299/299
passing.
* test(triplet_store): expand AnzoStore regression coverage
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
* refactor(provenance): centralize duplicated store-and-swallow logic into _save_entry() (closes#784)
- Added ProvenanceManager._save_entry(entry) as the single shared
checksum-compute + storage.store() + graceful-failure-swallow
pipeline, previously duplicated identically across track_entity,
track_relationship, track_chunk, and track_property_source.
- track_entities_batch/track_chunks_batch already delegate to
track_entity/track_chunk in a loop, so they inherit the fix for
free — left untouched, confirmed no direct duplication there.
- Byte-for-byte preserves today's swallow-and-continue behavior and
comment text; this is an architecture-only refactor. The silent-
failure behavior itself is unchanged and out of scope here — a fix
to it now only needs to happen in one place instead of four.
- Added 4 new regression tests (previously 0 of the 4 single-item
methods had failure-path coverage) proving storage.store() raising
is still caught and each method still returns its ProvenanceEntry.
Tests: tests/provenance/ 228 passed (+4 new), tests/explorer/test_provenance_manager_wiring.py 8 passed. 236/236, 0 failed.
* fix(provenance): drop out-of-transaction store attempt in track_entity fallback
The _save_entry refactor changed track_entity's pre-build exception
fallback (entry is None branch) to call _save_entry(), which makes a
real self.storage.store(entry) call. The original code only computed
a checksum here and never attempted storage again, since this branch
fires when something already failed before the entry was built inside
the atomic transaction. Storing outside that transaction bypasses the
BEGIN IMMEDIATE serialization #807 added, risking the same race it
fixed. Restored checksum-only behavior and added a regression test
asserting storage.store is not called on this path.
Also removed an untested hasattr(_store_with_conn) defensive branch
added during the refactor that wasn't in the original code, and added
a changelog entry.
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Two review findings on #807/#812:
- retrieve() and trace_lineage() were routed through transaction()'s
BEGIN IMMEDIATE, so plain reads took SQLite's writer lock and
serialized behind every other read/write, defeating the WAL
concurrency this PR was meant to add. They now use a dedicated
_read_connection() (configured, no explicit BEGIN).
- track_entity()/track_chunk() swallowed all internal storage
exceptions unconditionally, so a single item's failure inside
track_entities_batch()/track_chunks_batch()'s shared transaction
never reached the batch loop's per-item except, inflating
tracked_count for entries that were never persisted. Both now
re-raise when called with a shared _conn (batch context) while
still degrading gracefully on standalone calls.
Added regression tests for both, corrected the CHANGELOG entry and
docs that described the prior (overly broad) behavior.
- Extend ProvenanceNode in semantica/explorer/schemas.py with audit evidence fields: source_document, source_location, source_quote, confidence, and checksum.
- Update _transform_audit_lineage in semantica/explorer/routes/provenance.py to populate these evidence fields for each lineage node from ProvenanceEntry records, while keeping default None values for orphan nodes.
- Include source_document, confidence, and checksum in markdown report rendering (_render_markdown) so exported markdown reports surface attribution and integrity evidence.
- Add unit test test_provenance_audit_evidence_fields_preserved in test_provenance_manager_wiring.py verifying that evidence fields are present across /api/provenance JSON responses and exported JSON/markdown reports.
- Update verify_checksum and compute_checksum in semantica/provenance/integrity.py to support both ProvenanceEntry objects and serialized dictionary entries.
- Add integrity_verified flag computed via verify_checksum to the dictionary returned by ProvenanceManager.get_lineage().
- Update _build_provenance in semantica/explorer/routes/provenance.py to verify every returned lineage entry before labeling the result as source=audit. If verification fails due to missing checksums or corrupted records, log a warning and fall back cleanly to graph traversal.
- Add unit test test_provenance_manager_wiring_checksum_failure_falls_back in test_provenance_manager_wiring.py verifying that tampered lineage entries trigger fallback to source=graph_traversal.
- Fix _transform_audit_lineage to classify all non-downstream ancestor derivation edges as 'upstream' instead of 'lateral', correcting multi-hop lineage direction in JSON and markdown reports.
- Add GraphSession.set_provenance_storage_path() to explicitly reject conflicting preconfigured storage paths or path mutations after provenance_manager initialization.
- Update create_app() to call active_session.set_provenance_storage_path(prov_path), preventing silent retention of conflicting paths or un-redirectable cached managers.
- Remove unused logging import in app.py.
- Add comprehensive unit tests in test_provenance_manager_wiring.py for upstream edge classification, markdown report grouping, conflicting path rejection, and manager initialization lockouts.
- Explorer's /api/provenance now queries the audit-grade ProvenanceManager
(SQLite-backed, checksummed) first, falling back to the naive 2-hop
graph traversal when no audit records exist for a node.
- Fixed a process-global mutable-state risk in the initial approach:
provenance storage path is threaded per-session via GraphSession,
not via ProvenanceManager's global set_default_storage_path classmethod.
- Added source: 'audit' | 'graph_traversal' to the response so callers
can distinguish which path served the data.
- Documented a known limitation: ProvenanceManager currently only
traces upstream/ancestor lineage, not descendants — the naive
fallback remains the only source for downstream relationships until
ProvenanceManager gains a reverse lookup (tracked separately).
- Warns (rather than silently no-ops) if a provided session's
provenance_manager was already constructed before create_app()
applied a provenance_storage_path.
- Never lets a provenance-manager failure crash the route; degrades
to the naive path with a logged warning instead.
Tests: 5 new tests in test_provenance_manager_wiring.py covering the
audit path, empty-record fallback, storage-failure degradation, app
startup wiring, and cross-session storage isolation. Full
tests/explorer/ + tests/provenance/ suite passing, order-invariant.
Sameer Kadam