fix(provenance): stop reads from taking the writer lock, fix batch count inflation

Two review findings on #807/#812:

- retrieve() and trace_lineage() were routed through transaction()'s
  BEGIN IMMEDIATE, so plain reads took SQLite's writer lock and
  serialized behind every other read/write, defeating the WAL
  concurrency this PR was meant to add. They now use a dedicated
  _read_connection() (configured, no explicit BEGIN).

- track_entity()/track_chunk() swallowed all internal storage
  exceptions unconditionally, so a single item's failure inside
  track_entities_batch()/track_chunks_batch()'s shared transaction
  never reached the batch loop's per-item except, inflating
  tracked_count for entries that were never persisted. Both now
  re-raise when called with a shared _conn (batch context) while
  still degrading gracefully on standalone calls.

Added regression tests for both, corrected the CHANGELOG entry and
docs that described the prior (overly broad) behavior.
This commit is contained in:
KaifAhmad1
2026-07-29 12:54:43 +05:30
parent dabeb0e833
commit f99241ca88
6 changed files with 123 additions and 8 deletions
+1 -1
View File
@@ -84,7 +84,7 @@ prov = ProvenanceManager(storage=SQLiteStorage("audit.db"))
For any regulated deployment — security operations, clinical data, financial risk — use `storage_path`. A SQLite file can be backed up, versioned, and queried with standard tools without requiring a server.
<Note>
`SQLiteStorage` automatically configures Write-Ahead Logging (`WAL`), `busy_timeout=5000`, and `synchronous=NORMAL`, and executes operations in atomic immediate transactions (`BEGIN IMMEDIATE`). Furthermore, `ProvenanceManager` automatically supports custom storage backends overriding only `trace_lineage(self, entity_id)` without requiring `max_depth` in their signature.
`SQLiteStorage` automatically configures Write-Ahead Logging (`WAL`), `busy_timeout=5000`, and `synchronous=NORMAL`, and executes read-modify-write operations (like `track_entity()`) in atomic immediate transactions (`BEGIN IMMEDIATE`); plain reads (`retrieve()`, `trace_lineage()`) use a separate connection without an explicit write lock so they don't serialize behind writers. Furthermore, `ProvenanceManager` automatically supports custom storage backends overriding only `trace_lineage(self, entity_id)` without requiring `max_depth` in their signature.
</Note>
## Recording provenance when ingesting data
+1 -1
View File
@@ -324,7 +324,7 @@ manager = ProvenanceManager(storage_path="provenance.db")
`SQLiteStorage` creates the database and indexes automatically on first use.
- **Atomicity & Concurrency**: Configures Write-Ahead Logging (`PRAGMA journal_mode=WAL`), `PRAGMA busy_timeout=5000`, and `PRAGMA synchronous=NORMAL`. Each public method call opens a single connection and executes inside an immediate write transaction (`BEGIN IMMEDIATE`), ensuring read-modify-write sequences are serialized across concurrent connections without leaving open file handles across calls.
- **Atomicity & Concurrency**: Configures Write-Ahead Logging (`PRAGMA journal_mode=WAL`), `PRAGMA busy_timeout=5000`, and `PRAGMA synchronous=NORMAL`. Read-modify-write methods (`track_entity()`, `store()`) open a single connection and execute inside an immediate write transaction (`BEGIN IMMEDIATE`), ensuring these sequences are serialized across concurrent connections without leaving open file handles across calls. Plain reads (`retrieve()`, `trace_lineage()`) use a separate connection with no explicit write lock, so concurrent reads don't serialize behind writers or each other.
- **Backward Compatibility**: Custom storage subclasses overriding `trace_lineage(self, entity_id)` remain backward compatible; `ProvenanceManager` inspects the override signature and automatically calls it with one argument if `max_depth` is unsupported.
## Tamper-Evident Checksums