mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
fix(provenance): stop reads from taking the writer lock, fix batch count inflation
Two review findings on #807/#812: - retrieve() and trace_lineage() were routed through transaction()'s BEGIN IMMEDIATE, so plain reads took SQLite's writer lock and serialized behind every other read/write, defeating the WAL concurrency this PR was meant to add. They now use a dedicated _read_connection() (configured, no explicit BEGIN). - track_entity()/track_chunk() swallowed all internal storage exceptions unconditionally, so a single item's failure inside track_entities_batch()/track_chunks_batch()'s shared transaction never reached the batch loop's per-item except, inflating tracked_count for entries that were never persisted. Both now re-raise when called with a shared _conn (batch context) while still degrading gracefully on standalone calls. Added regression tests for both, corrected the CHANGELOG entry and docs that described the prior (overly broad) behavior.
This commit is contained in:
@@ -84,7 +84,7 @@ prov = ProvenanceManager(storage=SQLiteStorage("audit.db"))
|
||||
For any regulated deployment — security operations, clinical data, financial risk — use `storage_path`. A SQLite file can be backed up, versioned, and queried with standard tools without requiring a server.
|
||||
|
||||
<Note>
|
||||
`SQLiteStorage` automatically configures Write-Ahead Logging (`WAL`), `busy_timeout=5000`, and `synchronous=NORMAL`, and executes operations in atomic immediate transactions (`BEGIN IMMEDIATE`). Furthermore, `ProvenanceManager` automatically supports custom storage backends overriding only `trace_lineage(self, entity_id)` without requiring `max_depth` in their signature.
|
||||
`SQLiteStorage` automatically configures Write-Ahead Logging (`WAL`), `busy_timeout=5000`, and `synchronous=NORMAL`, and executes read-modify-write operations (like `track_entity()`) in atomic immediate transactions (`BEGIN IMMEDIATE`); plain reads (`retrieve()`, `trace_lineage()`) use a separate connection without an explicit write lock so they don't serialize behind writers. Furthermore, `ProvenanceManager` automatically supports custom storage backends overriding only `trace_lineage(self, entity_id)` without requiring `max_depth` in their signature.
|
||||
</Note>
|
||||
|
||||
## Recording provenance when ingesting data
|
||||
|
||||
@@ -324,7 +324,7 @@ manager = ProvenanceManager(storage_path="provenance.db")
|
||||
|
||||
`SQLiteStorage` creates the database and indexes automatically on first use.
|
||||
|
||||
- **Atomicity & Concurrency**: Configures Write-Ahead Logging (`PRAGMA journal_mode=WAL`), `PRAGMA busy_timeout=5000`, and `PRAGMA synchronous=NORMAL`. Each public method call opens a single connection and executes inside an immediate write transaction (`BEGIN IMMEDIATE`), ensuring read-modify-write sequences are serialized across concurrent connections without leaving open file handles across calls.
|
||||
- **Atomicity & Concurrency**: Configures Write-Ahead Logging (`PRAGMA journal_mode=WAL`), `PRAGMA busy_timeout=5000`, and `PRAGMA synchronous=NORMAL`. Read-modify-write methods (`track_entity()`, `store()`) open a single connection and execute inside an immediate write transaction (`BEGIN IMMEDIATE`), ensuring these sequences are serialized across concurrent connections without leaving open file handles across calls. Plain reads (`retrieve()`, `trace_lineage()`) use a separate connection with no explicit write lock, so concurrent reads don't serialize behind writers or each other.
|
||||
- **Backward Compatibility**: Custom storage subclasses overriding `trace_lineage(self, entity_id)` remain backward compatible; `ProvenanceManager` inspects the override signature and automatically calls it with one argument if `max_depth` is unsupported.
|
||||
|
||||
## Tamper-Evident Checksums
|
||||
|
||||
Reference in New Issue
Block a user