mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
docs: update provenance documentation for SQLiteStorage WAL and batch tracking (#807)
This commit is contained in:
@@ -83,6 +83,10 @@ prov = ProvenanceManager(storage=SQLiteStorage("audit.db"))
|
||||
|
||||
For any regulated deployment — security operations, clinical data, financial risk — use `storage_path`. A SQLite file can be backed up, versioned, and queried with standard tools without requiring a server.
|
||||
|
||||
<Note>
|
||||
`SQLiteStorage` automatically configures Write-Ahead Logging (`WAL`), `busy_timeout=5000`, and `synchronous=NORMAL`, and executes operations in atomic immediate transactions (`BEGIN IMMEDIATE`). Furthermore, `ProvenanceManager` automatically supports custom storage backends overriding only `trace_lineage(self, entity_id)` without requiring `max_depth` in their signature.
|
||||
</Note>
|
||||
|
||||
## Recording provenance when ingesting data
|
||||
|
||||
The moment data enters your graph is the moment provenance must be recorded. `track_entity()` captures the source document, the timestamp, the operator or pipeline that ran the extraction, a verbatim quote from the source, and a confidence score. It returns a `ProvenanceEntry` with a SHA-256 checksum computed automatically.
|
||||
|
||||
@@ -155,13 +155,15 @@ prop_entry = manager.track_property_source(
|
||||
|
||||
### Batch Tracking
|
||||
|
||||
Batch tracking methods process items in blocks (default `batch_size=1000`) inside a shared transaction per block. Only entities or chunks that successfully commit to storage are added to the returned count, preventing rolled-back entries from inflating success counts.
|
||||
|
||||
```python
|
||||
entities = [
|
||||
{"id": "entity_1", "confidence": 0.9},
|
||||
{"id": "entity_2", "confidence": 0.85},
|
||||
]
|
||||
count = manager.track_entities_batch(entities, source="doc_1")
|
||||
# Returns the number of entities successfully tracked
|
||||
# Returns the number of entities successfully tracked and committed
|
||||
|
||||
chunks = [
|
||||
{"id": "chunk_0", "start_index": 0, "end_index": 500},
|
||||
@@ -322,6 +324,9 @@ manager = ProvenanceManager(storage_path="provenance.db")
|
||||
|
||||
`SQLiteStorage` creates the database and indexes automatically on first use.
|
||||
|
||||
- **Atomicity & Concurrency**: Configures Write-Ahead Logging (`PRAGMA journal_mode=WAL`), `PRAGMA busy_timeout=5000`, and `PRAGMA synchronous=NORMAL`. Each public method call opens a single connection and executes inside an immediate write transaction (`BEGIN IMMEDIATE`), ensuring read-modify-write sequences are serialized across concurrent connections without leaving open file handles across calls.
|
||||
- **Backward Compatibility**: Custom storage subclasses overriding `trace_lineage(self, entity_id)` remain backward compatible; `ProvenanceManager` inspects the override signature and automatically calls it with one argument if `max_depth` is unsupported.
|
||||
|
||||
## Tamper-Evident Checksums
|
||||
|
||||
`compute_checksum` and `verify_checksum` are auto-used by `track_entity` and all other tracking methods. You can also call them directly:
|
||||
|
||||
Reference in New Issue
Block a user