mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-08-29 04:26:38 +00:00
Merge remote-tracking branch 'origin/master' into dshw/pr-deepseek-harness-deepseek-harness-2698
This commit is contained in:
_Kerman
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session/session-persistence-sqlite/README.md
|
||||
README.md: ec42419a132a26c1f23ab99ab3da1db97a5483b0
|
||||
README.zh.md: 67c6bcfbec92f8f5b91150fb108c65a287906db9
|
||||
README.md: ba005d79771bcbc2c0c1632da77d694aa3a18c07
|
||||
README.zh.md: 96a396d237a8abf263c50c46c8c7b23054d6e7aa
|
||||
|
||||
@@ -2,40 +2,39 @@
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
A SQLite durable session-persistence backend — a second `SessionPersistence` provider ([session persistence](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md)) satisfying the same contract as `dsh-session-persistence-jsonl` (append-only, contiguous-seq, lazy materialization, interrupted-turn close on load), expressed over `node:sqlite` rows instead of file bytes.
|
||||
An opt-in SQLite `SessionPersistence` provider. It stores eligible `assistant/chunk` runs in packed physical rows, selectively Zstandard-compresses large payloads, and delta-encodes provenance sequences while restoring the exact logical `SessionEvent[]`. No shipped composition selects it; deployments mount this package explicitly and provide its database path.
|
||||
|
||||
`locate(meta)` returns `undefined`: all sessions share one database, so there is no honest independent per-session transcript path.
|
||||
`locate(meta)` returns `undefined` because every session shares one database. The provider exposes no per-session raw artifact.
|
||||
|
||||
## Storage model
|
||||
|
||||
Each `SessionEvent` maps 1:1 onto a row in an `events` table `(session_id, seq, type, time, data, source_event_seqs, surface_op)` — `data` is the event payload as JSON text, so the row shape is the event verbatim (including `assistant/chunk`, keeping `seq` contiguous). The two `TEXT` columns `source_event_seqs` and `surface_op` are nullable; they store the event's optional surface-metadata fields (see [session surface](../../../.agents/notes/implemented/architecture/2026-06-18-session-surface.md)). Out-of-log metadata (`SessionHeader`), a per-materialization incarnation id, and a monotonic per-log revision live in a `sessions` row; `createdAt` is a non-negative safe integer stored in a strict `INTEGER` column. A singleton state row carries the immutable store id. A `sessions` row is written only by the first `append` — its existence is the lazy-materialization signal (`list` reports exactly the sessions that have a row).
|
||||
Schema 17 keeps ordinary ROWID tables and the composite `events(session_id, seq)` primary-key index. Scalar rows store one logical event. Packed rows use `text-chunks`, `reasoning-chunks`, or `tool-call-chunks` as the physical `type`; `seq` and `time` identify the first represented event, and `data` holds the shared packed-chunk payload. Packed rows set `ignorable=0` as a physical discriminator and leave `source_event_seqs` and `surface_op` as `NULL`; scalar rows use `ignorable=1` only for logical ignorable events and `NULL` otherwise. A future ignorable logical event may therefore reuse a storage-tag name without being decoded as a packed row. These tags are storage records, not `SessionEventMap` members.
|
||||
|
||||
The repository's Node range supports unflagged `node:sqlite`. The database enables foreign keys and uses the configured journal mode (`wal` by default; use a rollback mode where WAL shared-memory files are unsuitable). `PRAGMA application_id` identifies the canonical persistence database, and `PRAGMA user_version` stores its layout version. A fresh database must have no application identity or user-defined schema objects; initialization creates every table and stamps both pragmas in one transaction. Non-pristine unversioned databases, foreign application identities, and every non-current version reject before journal-mode mutation because this unreleased format has no migrations.
|
||||
Schema 17 owns its codec locally rather than importing another persistence format's mutable implementation. Only exact, consecutive same-block text, reasoning, or tool-call delta forms pack. Unknown fields, surface metadata, sequence gaps, incompatible block/call identity, and unsafe timestamps remain scalar. A packed row represents at most 1,024 events and at most 1 MiB of uncompressed UTF-8 `data`; longer runs are partitioned without changing logical events. Reads reconstruct every original sequence number, timestamp, token boundary, argument fragment, and payload before returning data to the persistence coordinator.
|
||||
|
||||
On filesystems with POSIX modes, the backend requests mode `0700` for missing directories and exclusively creates a missing database with mode `0600` before SQLite opens it; the process umask may further restrict both. New WAL, shared-memory, and persistent rollback-journal sidecars receive the database's resulting owner-only mode. Existing directories, database files, and sidecars keep their modes; filesystem setup errors other than an existing database fail initialization. These defaults prevent incidental exposure through a permissive process umask, but do not protect database confidentiality or integrity when another principal can replace the database entry in its parent directory.
|
||||
Serialized `data` smaller than 4 KiB stays as SQLite `TEXT`. At or above that threshold, the writer uses Zstandard level 3 and stores a `BLOB` only when the frame is smaller than the original text; the reader decompresses it before UTF-8 validation and JSON parsing. `source_event_seqs` remains the complete ordered provenance array. Its first sequence is an unsigned varint and each subsequent sequence is a signed delta encoded with ZigZag varints, stored as a `BLOB`; no source is omitted or converted to a range.
|
||||
|
||||
## Contract semantics over rows
|
||||
Each append holds `BEGIN IMMEDIATE`, validates the bounded physical tail, packs only the new durable batch, inserts those records, and increments the session revision once. Normal appends never delete or replace an earlier event row. The default 200 ms write-behind window therefore compresses high-frequency streams while the physical write volume stays proportional to newly durable batches rather than repeatedly rewriting a growing packed value. A storage-level logical-tail check rejects a stale writer before mutation.
|
||||
|
||||
- **Append = a transaction.** `append` runs `BEGIN`/`COMMIT` around the batch: it materializes the `sessions` row (if still lazy) and INSERTs every event, asserting the contiguous-seq contract first (the first event's `seq` must equal the stored next-seq). A mid-batch failure (a UNIQUE violation on a duplicated seq) rolls back entirely, so the stored log and the in-memory cursor stay consistent. (`load()` already balanced the stored log, so `append` never has to repair a crash tail.)
|
||||
- **Lazy materialization.** `create()` records intent in memory only — no row is written until the first `append`. A created-but-never-appended session has no `sessions` row, so it is absent from `list()` (which reports exactly the sessions that have a row).
|
||||
- **Interrupted-turn close on load.** `load()` implements the shared [crash-recovery contract](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md): preserve the valid interrupted turn, append its synthetic closing events in one transaction, and remove only a torn tail row. Committed parse errors or sequence gaps make the session unloadable. Because recovery mutates stored rows, the next append starts from a balanced log and accurate cursor.
|
||||
- **Non-mutating inspection.** `inspect()` returns an immutable balanced logical view and may synthesize recovery closers in memory, without deleting a torn tail row, appending recovery rows, or changing the lightweight revision.
|
||||
- **Lightweight revisions.** `listSnapshots(signal?)` combines the immutable store and database-file identity, a per-materialization incarnation id, and a per-session counter incremented in each mutating transaction. A full-prefix read captures that revision and its event rows in one read transaction, while `readStoredRevision()` queries only the session row to validate retained preparations. This keeps unchanged observations stable without parsing event rows and distinguishes independent stores and recreated same-id logs. It checks cancellation before and after shared readiness and the synchronous metadata query; the query itself is non-preemptible.
|
||||
Full reads scan physical rows in first-logical-sequence order. A reverse pass finds the last valid `turn/end` without retaining decoded copies of every physical row; the forward pass decodes and validates one physical row at a time into the returned logical event array. `readFrom(id, fromSeq)` examines packed predecessors only within the maximum row span and anchors the suffix at the earliest one that may contain `fromSeq`; this includes an event range that starts inside a packed row, detects overlapping physical corruption, and does not parse unrelated earlier scalar rows. A malformed packed row is all-or-nothing: committed corruption rejects, while a torn final row is deleted from its physical base during mutating recovery. Repair re-reads the tail under the write lock and rejects a stale marker before deleting anything. Packed `data` that exceeds the schema byte limit rejects before JSON parsing.
|
||||
|
||||
## Schema compatibility
|
||||
|
||||
A pristine database initializes directly at schema 17. Older schemas, foreign application identities, non-pristine unversioned databases, and incompatible schema objects reject; this pre-release provider supplies no migration. Every statement and fixed pragma lives in a packaged `.sql` resource; values use SQLite parameters and runtime code never assembles query text.
|
||||
|
||||
## Configuration (schemastery)
|
||||
|
||||
```ts
|
||||
interface Config {
|
||||
path: string // SQLite database file path, or ':memory:' for an in-process DB
|
||||
journalMode?: 'wal' | 'delete' | 'truncate' | 'persist' // journal_mode pragma; default 'wal'
|
||||
preparedSessionCacheSize?: number // positive integer; default 5
|
||||
writeBatchMaxDelayMs?: number // positive integer; default 200; maximum 2_147_483_647
|
||||
path: string
|
||||
journalMode?: 'wal' | 'delete' | 'truncate' | 'persist'
|
||||
busyTimeoutMs?: number
|
||||
preparedSessionCacheSize?: number
|
||||
writeBatchMaxDelayMs?: number
|
||||
}
|
||||
```
|
||||
|
||||
## Write path
|
||||
|
||||
Like the JSONL backend, the plugin copies each frozen `session/event` into one controller per live session. The first pending event starts the configured fixed batching window, and later events join without resetting it. Expiry starts one transaction; events admitted during that write form a separately bounded follow-up batch. `session/flush` cancels the wait and drains current and pending batches. The controller persists a fork's seed once, keeps a write cursor so resume never re-appends stored events, and seeds live sessions on apply because HMR does not replay `session/created`. Dispose drains every retained controller before closing the database. Every event remains a separate SQLite row; batching only groups more INSERTs into one transaction and revision increment.
|
||||
`journalMode` defaults to `wal`, `busyTimeoutMs` defaults to `5,000`, `preparedSessionCacheSize` defaults to `5`, and `writeBatchMaxDelayMs` defaults to `200`. The timeout bounds each synchronous SQLite lock wait. Because SQLite may return `SQLITE_BUSY` immediately while changing journal mode, cold open yields between attempts and starts no further attempt after an open-relative retry cutoff. An in-progress synchronous SQLite call may finish after that cutoff. The provider disables trusted schemas and memory-mapped I/O on every connection, then reads both settings back. The selected journal mode is also read back and must match; in-memory databases explicitly accept SQLite's `memory` result. After selecting the journal, the provider pins `synchronous=FULL` and verifies it so SQLite build defaults cannot weaken committed-append durability. On POSIX, the database parent and file must be owned by the current user, the parent must not be group/world-writable, and the file must have no group/world permissions. Symbolic links and non-regular files reject. Windows also rejects symbolic links and non-regular files, but deployments remain responsible for restricting the directory and file ACLs to the harness user. Path and ownership failures reject plugin initialization. Node SQLite loads lazily on the first persistence operation; the import suppresses only Node 22's exact SQLite `ExperimentalWarning`. Store-identity and schema failures reject that operation before data is exposed or mutated.
|
||||
|
||||
## Model Experience
|
||||
|
||||
@@ -43,20 +42,22 @@ Like the JSONL backend, the plugin copies each frozen `session/event` into one c
|
||||
|
||||
#### What the model sees
|
||||
|
||||
SQLite storage contributes no live prompt or schema. Loading restores the same surface history as JSONL and preserves prior headers for reconstruction; the new loop composes its current envelope. Recovery balances an assistant request without a durable call with `TOOL_NOT_STARTED`; a durable call without a result becomes `TOOL_OUTCOME_UNKNOWN`, which tells the model to retry only read-only or idempotent work and to verify possible side effects or ask the user. Row metadata and raw chunks are not messages.
|
||||
Nothing specific to SQLite. Resume restores the same logical events and derived messages as JSONL; physical packed tags never reach prompts, tools, replay, or live `session/event` delivery.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Zero live-request tokens. Resume restores retained history and pays the current envelope, plus the quoted repair result for each interrupted call.
|
||||
Zero live-request tokens. Resume pays only for the retained logical history and current request envelope.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
SQLite storage does not mutate live request prefixes. A resumed loop can reuse provider cache only when its reconstructed history, current envelope, and model route match; crash-repair results append.
|
||||
Physical packing does not mutate request prefixes. Provider cache reuse depends on the reconstructed history, current envelope, and model route exactly as with other persistence backends.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **`DatabaseSync` is synchronous** — every append transaction blocks the event loop for its duration; acceptable for local stores, a throughput ceiling for busy multi-session servers.
|
||||
- **Write contention has no wait or retry policy** — the backend sets no busy timeout and retries no locked-database error, so another connection holding a write transaction makes the operation reject immediately.
|
||||
- **Only a pristine new database or the current owned `SCHEMA_VERSION` opens** — unversioned schema objects, foreign application identities, and every other schema version are rejected rather than migrated (unreleased software; no persisted user data to preserve).
|
||||
- **Nothing deletes stored sessions** — rows accumulate until removed externally (the seam has no deletion API; `ON DELETE CASCADE` is wired for such out-of-band cleanup).
|
||||
- **TODO:** this backend talks to `node:sqlite` directly. If a cordis database service (`cordis/db` / a `@cordisjs` SQL driver plugin) is adopted, route through that instead of holding a raw `DatabaseSync` here — the contract surface (`SessionPersistence`) would not change, only the storage driver.
|
||||
- **Interim SQLite-specific design** — This efficiency-focused implementation is informed by [morlay/session-persistence-rdb](https://github.com/morlay/session-persistence-rdb). A unified relational-database design with multiple backends and configurable schemas is deferred; neither schema stability nor migration support is guaranteed during pre-release development.
|
||||
- **Packing follows durable batch boundaries** — compatible runs split by the write-behind window or an explicit flush remain separate physical records; this avoids rewriting prior rows at the cost of a timing-dependent packing ratio.
|
||||
- **Synchronous compression** — Node's SQLite and Zstandard calls block the JavaScript thread; the 4 KiB threshold limits per-frame work for small records.
|
||||
- **`DatabaseSync` blocks the event loop** — physical row reduction does not make SQLite operations asynchronous.
|
||||
- **Busy waits block the event loop** — SQLite waits inside synchronous `DatabaseSync` calls; only a busy journal-mode transition yields between attempts, and the open-relative cutoff prevents another attempt rather than interrupting an active call.
|
||||
- **External SQL readers must understand physical tags** — supported consumers read through this provider rather than treating every `events.type` as a logical event type.
|
||||
- **No deletion or background historical compaction** — normal appends are insert-only.
|
||||
|
||||
@@ -2,61 +2,62 @@
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
SQLite 持久会话存储后端：第二个 `SessionPersistence` 提供方（见[会话持久化](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md)），满足与 `dsh-session-persistence-jsonl` 相同的约定（仅追加、连续 seq、延迟实体化、在 load 时关闭中断轮次），但用 `node:sqlite` 行而非文件字节表达。
|
||||
一个可选启用的 SQLite `SessionPersistence` 提供方。它将符合条件的 `assistant/chunk` 连续段存入打包后的物理行，对大型 payload 选择性应用 Zstandard 压缩，并对来源序列进行 delta 编码，同时恢复完全一致的逻辑 `SessionEvent[]`。随产品交付的组合均不选择它；部署方需显式挂载本包并提供数据库路径。
|
||||
|
||||
`locate(meta)` 返回 `undefined`：所有会话共享一个数据库，因此不存在真实、独立的逐会话 transcript（文本记录）路径。
|
||||
`locate(meta)` 返回 `undefined`，因为所有会话共享同一个数据库。该提供方不暴露逐会话原始产物。
|
||||
|
||||
## 存储模型
|
||||
|
||||
每个 `SessionEvent` 1:1 映射到 `events` 表中的一行 `(session_id, seq, type, time, data, source_event_seqs, surface_op)`；`data` 是作为 JSON 文本的事件 payload，因此行结构就是原始事件本身（包括 `assistant/chunk`，保持 `seq` 连续）。两个 `TEXT` 列 `source_event_seqs` 和 `surface_op` 可为空，存储事件可选接口元数据字段（见[会话接口](../../../.agents/notes/implemented/architecture/2026-06-18-session-surface.md)）。日志外元数据（`SessionHeader`）、每实体化 incarnation id 和每日志单调修订位于 `sessions` 行；`createdAt` 是存储在 strict `INTEGER` 列中的非负安全整数。单例状态行携带不可变存储 id。`sessions` 行只由第一次 `append` 写入，其存在性是延迟实体化信号（`list` 精确报告有行的会话）。
|
||||
Schema 17 保留普通 ROWID 表以及复合主键索引 `events(session_id, seq)`。标量行存储一个逻辑事件。打包行把 `text-chunks`、`reasoning-chunks` 或 `tool-call-chunks` 用作物理 `type`；`seq` 与 `time` 标识所表示的第一个事件，`data` 保存共享的分片打包 payload。打包行把 `ignorable=0` 用作物理判别值，并让 `source_event_seqs` 与 `surface_op` 保持 `NULL`；标量行仅在逻辑事件可忽略时使用 `ignorable=1`，否则使用 `NULL`。因此，未来的可忽略逻辑事件即使复用了某个存储标签名称，也不会被解码为打包行。这些标签属于存储记录，而不是 `SessionEventMap` 成员。
|
||||
|
||||
仓库支持的 Node 范围可不加 flag 使用 `node:sqlite`。数据库启用外键，并使用已配置 journal mode（默认 `wal`；WAL 共享内存文件不适用时使用 rollback mode）。`PRAGMA application_id` 标识规范持久化数据库，`PRAGMA user_version` 存储布局版本。新数据库必须没有 application identity 或用户定义 schema 对象；初始化在一个事务中创建全部表并盖上两个 pragma。非 pristine 无版本数据库、外部 application identity 和所有非当前版本在 journal-mode 变更前均会被拒绝，因为该未发布格式无迁移。
|
||||
Schema 17 在本包内拥有 codec，不导入其他持久化格式中可变的实现。只有字段完全匹配、连续且属于同一分片块的文本、推理或工具调用 delta 才会打包。未知字段、surface 元数据、序列缺口、不兼容的块／调用身份以及不安全时间戳仍以标量行存储。一个打包行最多表示 1,024 个事件，未压缩 UTF-8 `data` 最多 1 MiB；更长的连续段会在不改变逻辑事件的前提下分割。读取会在向持久化协调器返回数据前，重建每个原始序列号、时间戳、token 边界、参数片段和 payload。
|
||||
|
||||
在具有 POSIX mode 的文件系统上，后端为缺失目录请求 mode `0700`，并在 SQLite 打开前以 mode `0600` 排他创建缺失数据库；进程 umask 可进一步限制两者。新 WAL、共享内存和持久 rollback-journal sidecar 获得数据库最终的仅所有者 mode。现有目录、数据库文件和 sidecar 保留原 mode；除已存在数据库外的文件系统设置错误会使初始化失败。这些默认值防止宽松进程 umask 造成的意外暴露，但当其他 principal 能替换父目录中的数据库条目时，不保护数据库机密性或完整性。
|
||||
序列化后的 `data` 小于 4 KiB 时保持为 SQLite `TEXT`。达到或超过该阈值时，写入方会使用 Zstandard level 3，并且只在 frame 小于原文本的情况下存储 `BLOB`；读取方会先解压，再执行 UTF-8 校验和 JSON 解析。`source_event_seqs` 仍是完整且有序的来源数组。第一个序列使用无符号 varint，后续序列使用 ZigZag varint 编码的有符号差值，并存为 `BLOB`；不会省略任何来源，也不会把数组转换成范围。
|
||||
|
||||
## 行上的约定语义
|
||||
每次追加持有 `BEGIN IMMEDIATE`，验证有界物理尾部，只打包新的持久批次，插入这些记录，并把会话 revision 递增一次。普通追加绝不删除或替换既有事件行。默认 200 毫秒写后缓冲窗口因此仍能压缩高频流，而物理写入量与新增持久批次成正比，不会反复改写不断增长的打包值。存储层逻辑尾部检查会在陈旧写入方执行变更前拒绝该写入。
|
||||
|
||||
- **Append = 事务。**`append` 围绕批次运行 `BEGIN`/`COMMIT`：它实体化 `sessions` 行（如果仍未实体化），并 INSERT 每个事件，首先断言连续 seq 约定（第一个事件 `seq` 必须等于已存储 next-seq）。批次中失败（重复 seq 上的 UNIQUE 违规）会完全回滚，使已存储日志和内存游标保持一致。（`load()` 已平衡已存储日志，因此 `append` 不必修复崩溃尾部。）
|
||||
- **延迟实体化。**`create()` 只在内存记录意图，第一次 `append` 前不写行。已创建但从未 append 的会话没有 `sessions` 行，因此不在 `list()` 中（它精确报告有行的会话）。
|
||||
- **在 load 时关闭中断轮次。**`load()` 实现共享[崩溃恢复约定](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md)：保留有效中断轮次，在一个事务中追加合成关闭事件，并只移除撕裂尾部行。已提交解析错误或序列缺口使会话无法加载。恢复会变更已存储行，因此下一次 append 从平衡日志和准确游标开始。
|
||||
- **非修改式检查。**`inspect()` 返回不可变、平衡的逻辑视图，并可在内存中合成恢复 closer，但不会删除撕裂尾部行、追加恢复行或更改轻量修订。
|
||||
- **轻量修订。**`listSnapshots(signal?)` 组合不可变存储与数据库文件身份、每实体化 incarnation id，以及在每个变更事务中递增的每会话计数器。完整前缀读取在同一个读事务中捕获该 revision 及其事件行，`readStoredRevision()` 则只查询 session 行来校验保留的 preparation。它在不解析事件行的情况下保持未变观察稳定，并区分独立存储和重建的同 id 日志。它在共享就绪和同步元数据查询前后检查取消；查询本身不可抢占。
|
||||
完整读取按首个逻辑序列号的顺序扫描物理行。反向扫描会定位最后一个有效 `turn/end`，但不会保留每个物理行的解码副本；正向扫描则逐行解码并校验，写入最终返回的逻辑事件数组。`readFrom(id, fromSeq)` 只检查最大行跨度内的打包前驱，并把后缀锚定在可能包含 `fromSeq` 的最早前驱；这样既可包含从打包行内部开始的事件范围，也能检测相互重叠的物理损坏，而不会解析无关的更早标量行。畸形打包行按全有或全无处理：已提交区域中的损坏会拒绝读取，最终撕裂行则在可变恢复期间从其物理起点删除。修复会在持有写锁时重新读取尾部，并在删除任何数据前拒绝陈旧 marker。打包 `data` 超出 schema 字节上限时，会在解析 JSON 前拒绝。
|
||||
|
||||
## Schema 兼容性
|
||||
|
||||
全新数据库直接初始化为 schema 17。旧 schema、外部 application identity、非空未版本化数据库以及不兼容 schema 对象都会被拒绝；这个预发布提供方不提供迁移。每条语句和固定 pragma 都位于随包发布的 `.sql` 资源中；值使用 SQLite 参数，运行时代码不会拼装查询文本。
|
||||
|
||||
## 配置（schemastery）
|
||||
|
||||
```ts
|
||||
interface Config {
|
||||
path: string // SQLite database file path, or ':memory:' for an in-process DB
|
||||
journalMode?: 'wal' | 'delete' | 'truncate' | 'persist' // journal_mode pragma; default 'wal'
|
||||
preparedSessionCacheSize?: number // positive integer; default 5
|
||||
writeBatchMaxDelayMs?: number // positive integer; default 200; maximum 2_147_483_647
|
||||
path: string
|
||||
journalMode?: 'wal' | 'delete' | 'truncate' | 'persist'
|
||||
busyTimeoutMs?: number
|
||||
preparedSessionCacheSize?: number
|
||||
writeBatchMaxDelayMs?: number
|
||||
}
|
||||
```
|
||||
|
||||
## 写入路径
|
||||
|
||||
与 JSONL 后端一样，插件将每个冻结的 `session/event` 复制到对应活动会话的 controller 中，每个活动会话各有一个 controller。第一个待处理事件会开启配置的固定批处理窗口，后续事件会加入但不会重置截止时间。窗口到期后会启动一个事务；该次写入期间接纳的事件会形成另一个独立有界的后续批次。`session/flush` 会取消等待并排空当前与待处理批次。Controller 会持久化一次 fork 种子，并保留写入游标，使恢复操作绝不重新 append 已存储事件；它还会在 apply 时为活动会话设置初始状态，因为 HMR（热模块替换）不回放 `session/created`。dispose（资源释放）会在关闭数据库前排空每个保留的 controller。每个事件仍各占一行 SQLite 记录；批处理只把更多 INSERT 归入同一个事务和同一次修订版本递增。
|
||||
`journalMode` 默认为 `wal`，`busyTimeoutMs` 默认为 `5,000`，`preparedSessionCacheSize` 默认为 `5`，`writeBatchMaxDelayMs` 默认为 `200`。该超时限制每次同步 SQLite 锁等待的时长。SQLite 在切换 journal mode 时可能立即返回 `SQLITE_BUSY`，因此冷打开会在尝试之间让出执行，并在从打开时开始计算的重试截止点后不再发起新尝试。正在执行的同步 SQLite 调用可能在该截止点之后才完成。提供方会在每个连接上禁用可信 schema 与内存映射 I/O，然后读回这两项设置。提供方还会读回所选 journal mode 并要求它匹配；内存数据库显式接受 SQLite 返回的 `memory`。选择 journal 后，提供方会把 `synchronous` 固定为 `FULL` 并验证该设置，避免 SQLite 构建默认值削弱已提交追加的持久性。在 POSIX 上，数据库父目录和文件必须归当前用户所有，父目录不得允许组或其他用户写入，文件不得授予组或其他用户任何权限。符号链接和非普通文件会被拒绝。Windows 同样拒绝符号链接与非普通文件，但部署方仍负责把目录和文件 ACL 限制给 harness 用户。路径与所有权错误会拒绝插件初始化。Node SQLite 在第一次持久化操作时才加载；导入时只抑制 Node 22 精确的 SQLite `ExperimentalWarning`。存储身份与 schema 错误会在暴露或变更数据前拒绝该操作。
|
||||
|
||||
## 模型体验
|
||||
|
||||
### 恢复的对话历史
|
||||
|
||||
#### 模型看到的内容
|
||||
#### 模型看到什么
|
||||
|
||||
SQLite 存储不会向当前请求提供提示词或 schema。加载会恢复与 JSONL 相同的呈现历史，并保留之前的 header 用于重建；新 loop 组合当前 envelope。恢复会用 `TOOL_NOT_STARTED` 平衡没有已持久化调用的 assistant 请求；已有持久化调用但无结果时则变为 `TOOL_OUTCOME_UNKNOWN`，它要求模型只重试只读或幂等工作，并验证可能的副作用或询问用户。行元数据和原始分片不会成为消息。
|
||||
没有 SQLite 特有内容。恢复得到与 JSONL 相同的逻辑事件和派生消息；物理打包标签绝不会进入 prompt、工具、回放或实时 `session/event` 投递。
|
||||
|
||||
#### Token 影响
|
||||
|
||||
SQLite 存储不会增加当前请求的 token 用量。恢复会还原已保留的历史，并产生当前 envelope 以及每个中断调用所附、以引用形式呈现的修复结果文本所产生的 token 开销。
|
||||
实时请求增加零 token。恢复只为保留的逻辑历史和当前请求 envelope 付出 token。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
SQLite 存储不修改当前请求前缀。只有重建历史、当前 envelope 和模型路由匹配时，恢复 loop 才能重用提供方缓存；崩溃修复结果会追加到末尾。
|
||||
物理打包不会改变请求前缀。与其他持久化后端相同，提供方 cache 复用取决于重建历史、当前 envelope 和模型路由。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
## 已知限制与延期工作
|
||||
|
||||
- **`DatabaseSync` 是同步的**：每个 append 事务在整个期间阻塞事件循环；对本地存储可接受，对繁忙多会话服务器是吞吐上限。
|
||||
- **写入争用无等待或重试策略**：后端不设置 busy timeout，也不重试 locked-database 错误，因此其他连接持有写事务时操作立即拒绝。
|
||||
- **只有 pristine 新数据库或当前自有 `SCHEMA_VERSION` 才能打开**：无版本 schema 对象、外部 application identity 和所有其他 schema 版本被拒绝，而不是迁移（未发布软件，无持久用户数据需要保留）。
|
||||
- **不删除已存储会话**：行会累积，直到外部移除（seam 无删除接口；`ON DELETE CASCADE` 已为这种带外清理配置）。
|
||||
- **TODO：** 该后端直接调用 `node:sqlite`。如果采用 Cordis 数据库服务（`cordis/db` / `@cordisjs` SQL driver 插件），应改为通过该服务路由，而不在此直接持有 `DatabaseSync`；约定接口（`SessionPersistence`）不会变，只更换存储驱动。
|
||||
- **过渡性的 SQLite 专用设计**——这一以效率为重点的实现参考了 [morlay/session-persistence-rdb](https://github.com/morlay/session-persistence-rdb)。支持多种后端与可配置 schema 的统一关系数据库设计尚待后续完善；预发布开发阶段不保证 schema 稳定性或迁移支持。
|
||||
- **打包服从持久批次边界**——被写后缓冲窗口或显式 flush 分开的兼容连续段会保留为不同物理记录；这以打包率受时序影响为代价，避免改写既有行。
|
||||
- **同步压缩**——Node 的 SQLite 与 Zstandard 调用都会阻塞 JavaScript 线程；4 KiB 阈值限制了小型记录的逐 frame 工作。
|
||||
- **`DatabaseSync` 会阻塞事件循环**——减少物理行不会使 SQLite 操作变为异步。
|
||||
- **繁忙等待会阻塞事件循环**——SQLite 会在同步 `DatabaseSync` 调用内等待；只有繁忙的 journal-mode 切换会在两次尝试之间让出执行，而且从打开时计算的截止点只阻止新尝试，不会中断正在执行的调用。
|
||||
- **外部 SQL 读取方必须理解物理标签**——受支持的消费方通过本提供方读取，而不是把每个 `events.type` 都当作逻辑事件类型。
|
||||
- **没有删除或后台历史压缩**——普通追加只做插入。
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-persistence-sqlite",
|
||||
"description": "SQLite durable session persistence backend for the DeepSeek Harness",
|
||||
"description": "SQLite durable session persistence with physical chunk-row packing",
|
||||
"version": "0.1.0-rc.7",
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
@@ -28,11 +28,13 @@
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"resources/sql/**/*.sql",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:^",
|
||||
"@deepseek-ai/cordis": "workspace:^"
|
||||
@@ -41,9 +43,13 @@
|
||||
"@deepseek-ai/schemastery": "workspace:^"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/cordis-plugin-include": "workspace:^",
|
||||
"@deepseek-ai/cordis-plugin-loader": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:^",
|
||||
"@deepseek-ai/cordis": "workspace:^"
|
||||
"@deepseek-ai/cordis": "workspace:^",
|
||||
"typescript": "^6.0.3"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
BEGIN IMMEDIATE;
|
||||
@@ -0,0 +1 @@
|
||||
BEGIN;
|
||||
@@ -0,0 +1 @@
|
||||
COMMIT;
|
||||
@@ -0,0 +1,2 @@
|
||||
DELETE FROM events
|
||||
WHERE session_id = ? AND seq >= ?;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA foreign_keys = ON;
|
||||
@@ -0,0 +1,3 @@
|
||||
INSERT INTO events
|
||||
(session_id, seq, type, time, data, source_event_seqs, surface_op, ignorable)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?);
|
||||
@@ -0,0 +1,2 @@
|
||||
INSERT INTO persistence_state (singleton, store_id)
|
||||
VALUES (1, ?);
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA journal_mode = DELETE;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA journal_mode = PERSIST;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA journal_mode = TRUNCATE;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA journal_mode = WAL;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA mmap_size = 0;
|
||||
@@ -0,0 +1 @@
|
||||
ROLLBACK;
|
||||
@@ -0,0 +1,30 @@
|
||||
CREATE TABLE persistence_state (
|
||||
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
|
||||
store_id TEXT NOT NULL
|
||||
) STRICT;
|
||||
|
||||
CREATE TABLE sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
version INTEGER NOT NULL,
|
||||
created_at INTEGER NOT NULL,
|
||||
cwd TEXT,
|
||||
parent_session TEXT,
|
||||
seed_length INTEGER,
|
||||
origin TEXT,
|
||||
delegation_depth INTEGER,
|
||||
agent_preset TEXT,
|
||||
incarnation TEXT NOT NULL,
|
||||
revision INTEGER NOT NULL
|
||||
) STRICT;
|
||||
|
||||
CREATE TABLE events (
|
||||
session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
|
||||
seq INTEGER NOT NULL,
|
||||
type TEXT NOT NULL,
|
||||
time INTEGER NOT NULL,
|
||||
data ANY NOT NULL,
|
||||
source_event_seqs ANY,
|
||||
surface_op TEXT,
|
||||
ignorable INTEGER CHECK (ignorable IS NULL OR ignorable IN (0, 1)),
|
||||
PRIMARY KEY (session_id, seq)
|
||||
) STRICT;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA application_id;
|
||||
@@ -0,0 +1,4 @@
|
||||
SELECT seq, type, time, data, source_event_seqs, surface_op, ignorable
|
||||
FROM events
|
||||
WHERE session_id = ? AND seq >= ?
|
||||
ORDER BY seq;
|
||||
@@ -0,0 +1,4 @@
|
||||
SELECT seq, type, time, data, source_event_seqs, surface_op, ignorable
|
||||
FROM events
|
||||
WHERE session_id = ?
|
||||
ORDER BY seq;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA mmap_size;
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
SELECT seq, type, time, data, source_event_seqs, surface_op, ignorable
|
||||
FROM events
|
||||
WHERE session_id = ? AND seq >= ? AND seq < ?
|
||||
AND type IN ('text-chunks', 'reasoning-chunks', 'tool-call-chunks')
|
||||
AND ignorable = 0
|
||||
ORDER BY seq;
|
||||
@@ -0,0 +1,4 @@
|
||||
SELECT type, name, tbl_name, sql
|
||||
FROM sqlite_schema
|
||||
WHERE name NOT GLOB 'sqlite_*'
|
||||
ORDER BY type, name;
|
||||
@@ -0,0 +1,4 @@
|
||||
SELECT id, version, created_at, cwd, parent_session, seed_length, origin,
|
||||
delegation_depth, agent_preset, incarnation, revision
|
||||
FROM sessions
|
||||
WHERE id = ?;
|
||||
@@ -0,0 +1,3 @@
|
||||
SELECT id, version, created_at, cwd, parent_session, seed_length, origin,
|
||||
delegation_depth, agent_preset, incarnation, revision
|
||||
FROM sessions;
|
||||
@@ -0,0 +1,3 @@
|
||||
SELECT store_id
|
||||
FROM persistence_state
|
||||
WHERE singleton = 1;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA synchronous;
|
||||
@@ -0,0 +1,5 @@
|
||||
SELECT seq, type, time, data, source_event_seqs, surface_op, ignorable
|
||||
FROM events
|
||||
WHERE session_id = ?
|
||||
ORDER BY seq DESC
|
||||
LIMIT ?;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA trusted_schema;
|
||||
@@ -0,0 +1,3 @@
|
||||
SELECT COUNT(*) AS count
|
||||
FROM sqlite_schema
|
||||
WHERE name NOT GLOB 'sqlite_*';
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA user_version;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA application_id = 1146308688;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA user_version = 17;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA synchronous = FULL;
|
||||
@@ -0,0 +1 @@
|
||||
PRAGMA trusted_schema = OFF;
|
||||
@@ -0,0 +1,3 @@
|
||||
UPDATE sessions
|
||||
SET revision = revision + 1
|
||||
WHERE id = ?;
|
||||
@@ -0,0 +1,13 @@
|
||||
INSERT INTO sessions
|
||||
(id, version, created_at, cwd, parent_session, seed_length, origin,
|
||||
delegation_depth, agent_preset, incarnation, revision)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 0)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
version = excluded.version,
|
||||
created_at = excluded.created_at,
|
||||
cwd = excluded.cwd,
|
||||
parent_session = excluded.parent_session,
|
||||
seed_length = excluded.seed_length,
|
||||
origin = excluded.origin,
|
||||
delegation_depth = excluded.delegation_depth,
|
||||
agent_preset = excluded.agent_preset;
|
||||
@@ -0,0 +1,343 @@
|
||||
/**
|
||||
* Schema-17 physical chunk-row codec. This package owns the durable tags,
|
||||
* validation, and row-size limits independently from other persistence formats.
|
||||
* @module @deepseek-ai/dsh-session-persistence-sqlite/codec
|
||||
*/
|
||||
|
||||
import type { StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
|
||||
/* jscpd:ignore-start -- schema 17 deliberately owns a frozen physical codec;
|
||||
* importing or sharing the JSONL codec would let that format mutate this database interpreter. */
|
||||
type DeltaKind = 'text-delta' | 'reasoning-delta' | 'tool-call-delta'
|
||||
type DeltaEvent = SessionEvent<'assistant/chunk'>
|
||||
|
||||
interface RunDataBase {
|
||||
readonly turn: number
|
||||
readonly step: number
|
||||
readonly index: number
|
||||
readonly dt: number[]
|
||||
}
|
||||
|
||||
interface TextRunData extends RunDataBase {
|
||||
readonly texts: string[]
|
||||
}
|
||||
|
||||
interface ToolCallRunData extends RunDataBase {
|
||||
readonly id: Extract<StreamChunk, { type: 'tool-call-delta' }>['id']
|
||||
readonly name?: string
|
||||
readonly args: string[]
|
||||
}
|
||||
|
||||
/** One schema-17 packed physical record. */
|
||||
export type ChunkRow =
|
||||
| { readonly type: 'text-chunks'; readonly seq0: number; readonly time0: number; readonly data: TextRunData }
|
||||
| { readonly type: 'reasoning-chunks'; readonly seq0: number; readonly time0: number; readonly data: TextRunData }
|
||||
| { readonly type: 'tool-call-chunks'; readonly seq0: number; readonly time0: number; readonly data: ToolCallRunData }
|
||||
|
||||
/** One scalar event or schema-17 packed physical record. */
|
||||
export type StorageRecord = SessionEvent | ChunkRow
|
||||
|
||||
/** Minimum eligible members in a packed physical record. */
|
||||
export const MIN_PACKED_ROW_MEMBERS = 3
|
||||
/** Maximum logical members represented by one packed physical record. */
|
||||
export const MAX_PACKED_ROW_MEMBERS = 1_024
|
||||
/** Maximum UTF-8 bytes in one packed physical record's data column. */
|
||||
export const MAX_PACKED_DATA_BYTES = 1_048_576
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === 'object' && value !== null
|
||||
}
|
||||
|
||||
function hasExactKeys(value: object, keys: readonly string[]): boolean {
|
||||
return Object.keys(value).length === keys.length && keys.every(key => Object.hasOwn(value, key))
|
||||
}
|
||||
|
||||
function classify(event: SessionEvent): DeltaKind | undefined {
|
||||
if (event.type !== 'assistant/chunk') return undefined
|
||||
if (!hasExactKeys(event, ['type', 'seq', 'time', 'data'])) return undefined
|
||||
if (!Number.isSafeInteger(event.seq) || event.seq < 0 || !Number.isSafeInteger(event.time)) return undefined
|
||||
const data: unknown = event.data
|
||||
if (!isRecord(data) || !hasExactKeys(data, ['turn', 'step', 'chunk'])) return undefined
|
||||
if (typeof data.turn !== 'number' || typeof data.step !== 'number') return undefined
|
||||
const chunk = data.chunk
|
||||
if (!isRecord(chunk) || typeof chunk.index !== 'number') return undefined
|
||||
switch (chunk.type) {
|
||||
case 'text-delta':
|
||||
case 'reasoning-delta':
|
||||
return hasExactKeys(chunk, ['type', 'index', 'text']) && typeof chunk.text === 'string'
|
||||
? chunk.type
|
||||
: undefined
|
||||
case 'tool-call-delta': {
|
||||
const validKeys = hasExactKeys(chunk, ['type', 'index', 'id', 'argumentsDelta'])
|
||||
|| (hasExactKeys(chunk, ['type', 'index', 'id', 'name', 'argumentsDelta'])
|
||||
&& typeof chunk.name === 'string')
|
||||
return validKeys && typeof chunk.id === 'string' && typeof chunk.argumentsDelta === 'string'
|
||||
? chunk.type
|
||||
: undefined
|
||||
}
|
||||
default:
|
||||
return undefined
|
||||
}
|
||||
}
|
||||
|
||||
function toolCallOf(event: DeltaEvent): { readonly id: string; readonly name?: string } {
|
||||
return event.data.chunk as { readonly id: string; readonly name?: string }
|
||||
}
|
||||
|
||||
function indexOf(event: DeltaEvent): number {
|
||||
return (event.data.chunk as { readonly index: number }).index
|
||||
}
|
||||
|
||||
function continues(previous: DeltaEvent, next: DeltaEvent, kind: DeltaKind): boolean {
|
||||
if (next.seq !== previous.seq + 1 || !Number.isSafeInteger(next.time - previous.time)) return false
|
||||
if (next.data.turn !== previous.data.turn || next.data.step !== previous.data.step) return false
|
||||
if (indexOf(next) !== indexOf(previous)) return false
|
||||
if (kind !== 'tool-call-delta') return true
|
||||
const left = toolCallOf(previous)
|
||||
const right = toolCallOf(next)
|
||||
return left.id === right.id
|
||||
&& Object.hasOwn(left, 'name') === Object.hasOwn(right, 'name')
|
||||
&& left.name === right.name
|
||||
}
|
||||
|
||||
function buildRow(kind: DeltaKind, run: readonly DeltaEvent[]): ChunkRow {
|
||||
const first = run[0] as DeltaEvent
|
||||
const base = {
|
||||
turn: first.data.turn,
|
||||
step: first.data.step,
|
||||
index: indexOf(first),
|
||||
dt: run.slice(1).map((event, index) => event.time - (run[index] as DeltaEvent).time),
|
||||
}
|
||||
const envelope = { seq0: first.seq, time0: first.time }
|
||||
if (kind === 'tool-call-delta') {
|
||||
const call = toolCallOf(first)
|
||||
return {
|
||||
type: 'tool-call-chunks',
|
||||
...envelope,
|
||||
data: {
|
||||
...base,
|
||||
id: call.id as Extract<StreamChunk, { type: 'tool-call-delta' }>['id'],
|
||||
...Object.hasOwn(call, 'name') ? { name: call.name as string } : {},
|
||||
args: run.map(event => (event.data.chunk as { readonly argumentsDelta: string }).argumentsDelta),
|
||||
},
|
||||
}
|
||||
}
|
||||
const data = {
|
||||
...base,
|
||||
texts: run.map(event => (event.data.chunk as { readonly text: string }).text),
|
||||
}
|
||||
return kind === 'text-delta'
|
||||
? { type: 'text-chunks', ...envelope, data }
|
||||
: { type: 'reasoning-chunks', ...envelope, data }
|
||||
}
|
||||
|
||||
function packedDataBytes(row: ChunkRow): number {
|
||||
return Buffer.byteLength(JSON.stringify(row.data))
|
||||
}
|
||||
|
||||
function emitBoundedRun(out: StorageRecord[], kind: DeltaKind, completeRun: readonly DeltaEvent[]): void {
|
||||
let offset = 0
|
||||
while (completeRun.length - offset >= MIN_PACKED_ROW_MEMBERS) {
|
||||
let low = MIN_PACKED_ROW_MEMBERS
|
||||
let high = Math.min(completeRun.length - offset, MAX_PACKED_ROW_MEMBERS)
|
||||
const largest = buildRow(kind, completeRun.slice(offset, offset + high))
|
||||
if (packedDataBytes(largest) <= MAX_PACKED_DATA_BYTES) {
|
||||
out.push(largest)
|
||||
offset += high
|
||||
continue
|
||||
}
|
||||
high -= 1
|
||||
let accepted = 0
|
||||
let acceptedRow: ChunkRow | undefined
|
||||
while (low <= high) {
|
||||
const middle = Math.floor((low + high) / 2)
|
||||
const candidate = buildRow(kind, completeRun.slice(offset, offset + middle))
|
||||
if (packedDataBytes(candidate) <= MAX_PACKED_DATA_BYTES) {
|
||||
accepted = middle
|
||||
acceptedRow = candidate
|
||||
low = middle + 1
|
||||
} else {
|
||||
high = middle - 1
|
||||
}
|
||||
}
|
||||
if (accepted === 0) {
|
||||
out.push(completeRun[offset] as DeltaEvent)
|
||||
offset += 1
|
||||
continue
|
||||
}
|
||||
/* v8 ignore next -- accepted is set only with its same-branch candidate. */
|
||||
out.push(acceptedRow ?? malformed(kind, 'bounded encoder lost its accepted row'))
|
||||
offset += accepted
|
||||
}
|
||||
out.push(...completeRun.slice(offset))
|
||||
}
|
||||
|
||||
/**
|
||||
* Pack eligible logical chunk runs into bounded schema-17 records.
|
||||
* @param events - logical events in sequence order.
|
||||
* @returns scalar and packed physical records in equivalent order.
|
||||
*/
|
||||
export function packChunkRuns(events: readonly SessionEvent[]): StorageRecord[] {
|
||||
const out: StorageRecord[] = []
|
||||
let kind: DeltaKind | undefined
|
||||
let run: DeltaEvent[] = []
|
||||
const flush = (): void => {
|
||||
if (kind === undefined) out.push(...run)
|
||||
else emitBoundedRun(out, kind, run)
|
||||
kind = undefined
|
||||
run = []
|
||||
}
|
||||
for (const event of events) {
|
||||
const nextKind = classify(event)
|
||||
if (nextKind === undefined) {
|
||||
flush()
|
||||
out.push(event)
|
||||
continue
|
||||
}
|
||||
const delta = event as DeltaEvent
|
||||
const previous = run.at(-1)
|
||||
if (nextKind === kind && previous !== undefined && continues(previous, delta, nextKind)) {
|
||||
run.push(delta)
|
||||
continue
|
||||
}
|
||||
flush()
|
||||
kind = nextKind
|
||||
run = [delta]
|
||||
}
|
||||
flush()
|
||||
return out
|
||||
}
|
||||
|
||||
function malformed(tag: string, reason: string): never {
|
||||
throw new Error(`malformed ${tag} storage row: ${reason}`)
|
||||
}
|
||||
|
||||
function validateRunData(
|
||||
tag: string,
|
||||
data: Record<string, unknown>,
|
||||
payloadKey: 'texts' | 'args',
|
||||
serializedBytes?: number,
|
||||
): string[] {
|
||||
if (typeof data.turn !== 'number' || typeof data.step !== 'number' || typeof data.index !== 'number') {
|
||||
malformed(tag, 'turn/step/index must be numbers')
|
||||
}
|
||||
const payload = data[payloadKey]
|
||||
if (!Array.isArray(payload)
|
||||
|| payload.length < MIN_PACKED_ROW_MEMBERS
|
||||
|| payload.length > MAX_PACKED_ROW_MEMBERS
|
||||
|| payload.some(member => typeof member !== 'string')) {
|
||||
malformed(tag, `${payloadKey} must contain ${MIN_PACKED_ROW_MEMBERS}..${MAX_PACKED_ROW_MEMBERS} strings`)
|
||||
}
|
||||
const gaps = data.dt
|
||||
if (!Array.isArray(gaps) || gaps.some(gap => !Number.isSafeInteger(gap))) {
|
||||
malformed(tag, 'dt must be an array of safe integers')
|
||||
}
|
||||
if (gaps.length !== payload.length - 1) malformed(tag, 'dt length must match the member count')
|
||||
if ((serializedBytes ?? Buffer.byteLength(JSON.stringify(data))) > MAX_PACKED_DATA_BYTES) {
|
||||
malformed(tag, `data exceeds ${MAX_PACKED_DATA_BYTES} UTF-8 bytes`)
|
||||
}
|
||||
return payload as string[]
|
||||
}
|
||||
|
||||
function validateRow(
|
||||
value: Record<string, unknown>,
|
||||
tag: ChunkRow['type'],
|
||||
serializedBytes?: number,
|
||||
): ChunkRow {
|
||||
if (!hasExactKeys(value, ['type', 'seq0', 'time0', 'data'])) malformed(tag, 'invalid envelope fields')
|
||||
if (!Number.isSafeInteger(value.seq0) || (value.seq0 as number) < 0) malformed(tag, 'seq0 must be non-negative')
|
||||
if (!Number.isSafeInteger(value.time0)) malformed(tag, 'time0 must be a safe integer')
|
||||
const data = value.data
|
||||
if (!isRecord(data)) malformed(tag, 'data must be an object')
|
||||
let payload: string[]
|
||||
if (tag === 'tool-call-chunks') {
|
||||
const withName = hasExactKeys(data, ['turn', 'step', 'index', 'id', 'name', 'dt', 'args'])
|
||||
if (!withName && !hasExactKeys(data, ['turn', 'step', 'index', 'id', 'dt', 'args'])) {
|
||||
malformed(tag, 'invalid tool-call data fields')
|
||||
}
|
||||
if (typeof data.id !== 'string' || (withName && typeof data.name !== 'string')) {
|
||||
malformed(tag, 'id and optional name must be strings')
|
||||
}
|
||||
payload = validateRunData(tag, data, 'args', serializedBytes)
|
||||
} else {
|
||||
if (!hasExactKeys(data, ['turn', 'step', 'index', 'dt', 'texts'])) malformed(tag, 'invalid text data fields')
|
||||
payload = validateRunData(tag, data, 'texts', serializedBytes)
|
||||
}
|
||||
if (!Number.isSafeInteger((value.seq0 as number) + payload.length - 1)) malformed(tag, 'member seqs exceed safe integers')
|
||||
let time = value.time0 as number
|
||||
for (const gap of data.dt as number[]) {
|
||||
time += gap
|
||||
if (!Number.isSafeInteger(time)) malformed(tag, 'member times exceed safe integers')
|
||||
}
|
||||
return value as unknown as ChunkRow
|
||||
}
|
||||
|
||||
function expandRow(row: ChunkRow): SessionEvent[] {
|
||||
const members = row.type === 'tool-call-chunks' ? row.data.args : row.data.texts
|
||||
const events: SessionEvent[] = []
|
||||
let time = row.time0
|
||||
for (let index = 0; index < members.length; index += 1) {
|
||||
if (index > 0) time += row.data.dt[index - 1] as number
|
||||
let chunk: StreamChunk
|
||||
switch (row.type) {
|
||||
case 'text-chunks':
|
||||
chunk = { type: 'text-delta', index: row.data.index, text: members[index] as string }
|
||||
break
|
||||
case 'reasoning-chunks':
|
||||
chunk = { type: 'reasoning-delta', index: row.data.index, text: members[index] as string }
|
||||
break
|
||||
case 'tool-call-chunks':
|
||||
chunk = {
|
||||
type: 'tool-call-delta',
|
||||
index: row.data.index,
|
||||
id: row.data.id,
|
||||
...Object.hasOwn(row.data, 'name') ? { name: row.data.name as string } : {},
|
||||
argumentsDelta: members[index] as string,
|
||||
}
|
||||
break
|
||||
}
|
||||
events.push({
|
||||
type: 'assistant/chunk',
|
||||
seq: row.seq0 + index,
|
||||
time,
|
||||
data: { turn: row.data.turn, step: row.data.step, chunk },
|
||||
})
|
||||
}
|
||||
return events
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode one scalar or packed schema-17 record.
|
||||
* @param value - parsed physical-record value.
|
||||
* @returns the represented logical events.
|
||||
*/
|
||||
export function decodeStorageRecord(value: unknown): SessionEvent[] {
|
||||
if (!isRecord(value)) return [value as SessionEvent]
|
||||
const tag = value.type
|
||||
if (tag !== 'text-chunks' && tag !== 'reasoning-chunks' && tag !== 'tool-call-chunks') {
|
||||
return [value as SessionEvent]
|
||||
}
|
||||
return expandRow(validateRow(value, tag))
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode one packed row from its exact uncompressed data value. The byte bound
|
||||
* rejects oversized input before JSON parsing and avoids serializing it again.
|
||||
* @param tag - validated packed physical type.
|
||||
* @param seq0 - first represented logical sequence number.
|
||||
* @param time0 - first represented logical timestamp.
|
||||
* @param serializedData - decoded SQLite data-column text.
|
||||
* @returns the represented logical events.
|
||||
*/
|
||||
export function decodeSerializedChunkRow(
|
||||
tag: ChunkRow['type'],
|
||||
seq0: number,
|
||||
time0: number,
|
||||
serializedData: string,
|
||||
): SessionEvent[] {
|
||||
const bytes = Buffer.byteLength(serializedData)
|
||||
if (bytes > MAX_PACKED_DATA_BYTES) malformed(tag, `data exceeds ${MAX_PACKED_DATA_BYTES} UTF-8 bytes`)
|
||||
return expandRow(validateRow({ type: tag, seq0, time0, data: JSON.parse(serializedData) as unknown }, tag, bytes))
|
||||
}
|
||||
/* jscpd:ignore-end */
|
||||
@@ -0,0 +1,276 @@
|
||||
/**
|
||||
* Fixed physical-record compression for SQLite. Schema-owned functions
|
||||
* encode logical events and decode tagged rows before persistence consumers
|
||||
* observe them.
|
||||
* @module @deepseek-ai/dsh-session-persistence-sqlite/compression
|
||||
*/
|
||||
|
||||
import { TextDecoder } from 'node:util'
|
||||
import { constants, zstdCompressSync, zstdDecompressSync } from 'node:zlib'
|
||||
import type { SessionEvent, SurfaceEventType } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
decodeSerializedChunkRow,
|
||||
type ChunkRow,
|
||||
MAX_PACKED_DATA_BYTES,
|
||||
type StorageRecord,
|
||||
} from './codec.ts'
|
||||
import type { EventRow } from './schema.ts'
|
||||
|
||||
/** One physical row ready for SQLite parameter binding. */
|
||||
export interface BoundRecord {
|
||||
readonly seq: number
|
||||
readonly type: string
|
||||
readonly time: number
|
||||
readonly data: string | Uint8Array
|
||||
readonly sourceEventSeqs: Uint8Array | null
|
||||
readonly surfaceOp: string | null
|
||||
readonly ignorable: number | null
|
||||
}
|
||||
|
||||
/** Small values stay as SQLite text to avoid per-frame CPU and byte overhead. */
|
||||
export const ZSTD_DATA_THRESHOLD_BYTES = 4_096
|
||||
|
||||
const MAX_SAFE_INTEGER = BigInt(Number.MAX_SAFE_INTEGER)
|
||||
const MAX_ZIGZAG_INTEGER = MAX_SAFE_INTEGER * 2n
|
||||
const UTF8_DECODER = new TextDecoder('utf-8', { fatal: true })
|
||||
const ZSTD_COMPRESSION_LEVEL = 3
|
||||
const PACKED_ROW_SENTINEL = 0
|
||||
|
||||
const CHUNK_TAGS = ['text-chunks', 'reasoning-chunks', 'tool-call-chunks'] as const
|
||||
type ChunkTag = typeof CHUNK_TAGS[number]
|
||||
|
||||
function isChunkTag(value: string): value is ChunkTag {
|
||||
return (CHUNK_TAGS as readonly string[]).includes(value)
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode one physical SQLite row into its complete logical event span.
|
||||
* @param row - detached SQLite event row.
|
||||
* @returns every logical event represented by the row.
|
||||
*/
|
||||
export function decodeRow(row: EventRow): SessionEvent[] {
|
||||
if (row.ignorable !== PACKED_ROW_SENTINEL) return [decodeScalarRow(row)]
|
||||
if (!isChunkTag(row.type)) {
|
||||
throw new Error(`malformed ${row.type} storage row: packed discriminator requires a chunk tag`)
|
||||
}
|
||||
if (row.source_event_seqs !== null || row.surface_op !== null) {
|
||||
throw new Error(`malformed ${row.type} storage row: packed surface fields must be null`)
|
||||
}
|
||||
return decodeSerializedChunkRow(
|
||||
row.type,
|
||||
row.seq,
|
||||
row.time,
|
||||
decodeData(row.data, MAX_PACKED_DATA_BYTES),
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a storage record to SQLite column values.
|
||||
* @param record - scalar event or packed chunk record.
|
||||
* @returns column values for one physical insert.
|
||||
*/
|
||||
export function bindRecord(record: StorageRecord): BoundRecord {
|
||||
if (isChunkRow(record)) {
|
||||
return {
|
||||
seq: record.seq0,
|
||||
type: record.type,
|
||||
time: record.time0,
|
||||
data: encodeData(JSON.stringify(record.data)),
|
||||
sourceEventSeqs: null,
|
||||
surfaceOp: null,
|
||||
ignorable: PACKED_ROW_SENTINEL,
|
||||
}
|
||||
}
|
||||
const event = record
|
||||
const surface = event as SessionEvent<SurfaceEventType>
|
||||
return {
|
||||
seq: event.seq,
|
||||
type: event.type,
|
||||
time: event.time,
|
||||
data: encodeData(JSON.stringify(event.data)),
|
||||
sourceEventSeqs: surface.sourceEventSeqs === undefined
|
||||
? null
|
||||
: encodeSourceEventSeqs(surface.sourceEventSeqs),
|
||||
surfaceOp: surface.surfaceOp === undefined ? null : JSON.stringify(surface.surfaceOp),
|
||||
ignorable: event.ignorable === true ? 1 : null,
|
||||
}
|
||||
}
|
||||
|
||||
function encodeData(serialized: string): string | Uint8Array {
|
||||
const bytes = Buffer.from(serialized)
|
||||
if (bytes.length < ZSTD_DATA_THRESHOLD_BYTES) return serialized
|
||||
const compressed = zstdCompressSync(bytes, {
|
||||
params: { [constants.ZSTD_c_compressionLevel]: ZSTD_COMPRESSION_LEVEL },
|
||||
})
|
||||
return compressed.length < bytes.length ? compressed : serialized
|
||||
}
|
||||
|
||||
function decodeData(value: string | Uint8Array, maxOutputLength?: number): string {
|
||||
if (typeof value === 'string') return value
|
||||
const decoded = maxOutputLength === undefined
|
||||
? zstdDecompressSync(value)
|
||||
: zstdDecompressSync(value, { maxOutputLength })
|
||||
return UTF8_DECODER.decode(decoded)
|
||||
}
|
||||
|
||||
function encodeSourceEventSeqs(values: readonly number[]): Uint8Array {
|
||||
const bytes: number[] = []
|
||||
let previous = 0n
|
||||
for (let index = 0; index < values.length; index += 1) {
|
||||
const sourceSeq = values[index] as number
|
||||
if (!Number.isSafeInteger(sourceSeq) || sourceSeq < 0) {
|
||||
throw new TypeError('sourceEventSeqs must contain non-negative safe integers')
|
||||
}
|
||||
const value = BigInt(sourceSeq)
|
||||
const encoded = index === 0
|
||||
? value
|
||||
: value >= previous
|
||||
? (value - previous) * 2n
|
||||
: ((previous - value) * 2n) - 1n
|
||||
appendVarint(bytes, encoded)
|
||||
previous = value
|
||||
}
|
||||
return Buffer.from(bytes)
|
||||
}
|
||||
|
||||
function appendVarint(bytes: number[], value: bigint): void {
|
||||
let remaining = value
|
||||
while (remaining >= 0x80n) {
|
||||
bytes.push(Number(remaining & 0x7fn) | 0x80)
|
||||
remaining >>= 7n
|
||||
}
|
||||
bytes.push(Number(remaining))
|
||||
}
|
||||
|
||||
function decodeSourceEventSeqs(bytes: Uint8Array): number[] {
|
||||
const values: number[] = []
|
||||
let previous = 0n
|
||||
let offset = 0
|
||||
while (offset < bytes.length) {
|
||||
const first = values.length === 0
|
||||
const decoded = readVarint(bytes, offset, first ? MAX_SAFE_INTEGER : MAX_ZIGZAG_INTEGER)
|
||||
offset = decoded.offset
|
||||
const delta = first
|
||||
? decoded.value
|
||||
: (decoded.value & 1n) === 0n
|
||||
? decoded.value / 2n
|
||||
: -((decoded.value + 1n) / 2n)
|
||||
const value = first ? delta : previous + delta
|
||||
if (value < 0n || value > MAX_SAFE_INTEGER) {
|
||||
throw new Error('malformed source_event_seqs storage value: decoded seq is out of range')
|
||||
}
|
||||
values.push(Number(value))
|
||||
previous = value
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
function readVarint(
|
||||
bytes: Uint8Array,
|
||||
offset: number,
|
||||
limit: bigint,
|
||||
): { readonly value: bigint; readonly offset: number } {
|
||||
let value = 0n
|
||||
let shift = 0n
|
||||
while (offset < bytes.length) {
|
||||
const byte = bytes[offset] as number
|
||||
offset += 1
|
||||
value |= BigInt(byte & 0x7f) << shift
|
||||
if ((byte & 0x80) === 0) {
|
||||
if (shift > 0n && (byte & 0x7f) === 0) {
|
||||
throw new Error('malformed source_event_seqs storage value: non-canonical varint')
|
||||
}
|
||||
if (value > limit) {
|
||||
throw new Error('malformed source_event_seqs storage value: varint is out of range')
|
||||
}
|
||||
return { value, offset }
|
||||
}
|
||||
shift += 7n
|
||||
if (shift > 56n) {
|
||||
throw new Error('malformed source_event_seqs storage value: varint is out of range')
|
||||
}
|
||||
}
|
||||
throw new Error('malformed source_event_seqs storage value: truncated varint')
|
||||
}
|
||||
|
||||
function isChunkRow(record: StorageRecord): record is ChunkRow {
|
||||
return isChunkTag(record.type) && 'seq0' in record && !('seq' in record)
|
||||
}
|
||||
|
||||
function decodeScalarRow(row: EventRow): SessionEvent {
|
||||
const surfaceFields = {
|
||||
...row.source_event_seqs === null
|
||||
? {}
|
||||
: { sourceEventSeqs: decodeSourceEventSeqs(row.source_event_seqs) },
|
||||
...row.surface_op === null
|
||||
? {}
|
||||
: { surfaceOp: JSON.parse(row.surface_op) as SessionEvent<SurfaceEventType>['surfaceOp'] },
|
||||
}
|
||||
return {
|
||||
type: row.type as SessionEvent['type'],
|
||||
seq: row.seq,
|
||||
time: row.time,
|
||||
data: JSON.parse(decodeData(row.data)) as SessionEvent['data'],
|
||||
...surfaceFields,
|
||||
...row.ignorable === 1 ? { ignorable: true as const } : {},
|
||||
} as SessionEvent
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate and flatten physical rows into their logical prefix. A malformed
|
||||
* row or logical gap is committed corruption when a later valid turn end
|
||||
* exists; otherwise it starts a removable physical tail.
|
||||
* @param rows - physical rows ordered by their first logical sequence.
|
||||
* @param base - logical sequence expected from the first selected row.
|
||||
* @returns the contiguous logical prefix and optional physical deletion base.
|
||||
*/
|
||||
export function scanRows(
|
||||
rows: readonly EventRow[],
|
||||
base = 0,
|
||||
): { preserved: SessionEvent[]; tornFrom?: number } {
|
||||
let lastTurnEndRow = -1
|
||||
for (let index = rows.length - 1; index >= 0; index -= 1) {
|
||||
try {
|
||||
if (decodeRow(rows[index] as EventRow).some(event => event.type === 'turn/end')) {
|
||||
lastTurnEndRow = index
|
||||
break
|
||||
}
|
||||
} catch {
|
||||
// A malformed row cannot prove that an earlier physical prefix committed.
|
||||
}
|
||||
}
|
||||
|
||||
const preserved: SessionEvent[] = []
|
||||
let expected = base
|
||||
for (let rowIndex = 0; rowIndex < rows.length; rowIndex += 1) {
|
||||
const physical = rows[rowIndex] as EventRow
|
||||
let logicalEvents: SessionEvent[] | undefined
|
||||
try {
|
||||
logicalEvents = decodeRow(physical)
|
||||
} catch {
|
||||
// The committed-prefix rule below owns whether this invalid row is fatal or repairable.
|
||||
}
|
||||
if (logicalEvents === undefined) {
|
||||
if (rowIndex <= lastTurnEndRow) {
|
||||
throw new Error(`corrupt session log: invalid committed physical row at seq ${physical.seq}`)
|
||||
}
|
||||
return { preserved, tornFrom: physical.seq }
|
||||
}
|
||||
let contiguous = true
|
||||
for (const event of logicalEvents) {
|
||||
if (event.seq !== expected) {
|
||||
contiguous = false
|
||||
break
|
||||
}
|
||||
expected += 1
|
||||
}
|
||||
if (!contiguous) {
|
||||
if (rowIndex <= lastTurnEndRow) {
|
||||
throw new Error(`corrupt session log: invalid committed physical row at seq ${physical.seq}`)
|
||||
}
|
||||
return { preserved, tornFrom: physical.seq }
|
||||
}
|
||||
preserved.push(...logicalEvents)
|
||||
}
|
||||
return { preserved }
|
||||
}
|
||||
@@ -1,99 +1,45 @@
|
||||
/**
|
||||
* SQLite durable session-persistence backend. It maps each session header and
|
||||
* event to rows, and delegates write-path orchestration to
|
||||
* {@link PersistenceCoordinator}. It has no independent per-session artifact,
|
||||
* so its locator returns `undefined`.
|
||||
* Opt-in SQLite persistence provider. Logical sessions remain unchanged;
|
||||
* the physical backend packs eligible chunk runs into schema-17 rows.
|
||||
* @module @deepseek-ai/dsh-session-persistence-sqlite
|
||||
*/
|
||||
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import { Context, Service } from '@deepseek-ai/cordis'
|
||||
import z from '@deepseek-ai/schemastery'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { statSync } from 'node:fs'
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import { mkdir, open } from 'node:fs/promises'
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import type {
|
||||
SessionEvent,
|
||||
SessionHeader,
|
||||
SessionId,
|
||||
SessionPreparation,
|
||||
} from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
DEFAULT_PREPARED_SESSION_CACHE_SIZE, DEFAULT_WRITE_BATCH_MAX_DELAY_MS, MAX_WRITE_BATCH_DELAY_MS,
|
||||
decodeStoredSessionHeader, SessionPersistence, SessionPersistenceRevision,
|
||||
SessionPersistenceRevisionConflictError, PersistenceCoordinator,
|
||||
type PersistenceBackend, type SessionLocation, type SessionPersistenceSnapshot,
|
||||
type SessionInspection, type SessionPersistenceRevision as PersistenceRevision,
|
||||
type StoredEventRead, type StoredSessionSource,
|
||||
DEFAULT_PREPARED_SESSION_CACHE_SIZE,
|
||||
DEFAULT_WRITE_BATCH_MAX_DELAY_MS,
|
||||
MAX_WRITE_BATCH_DELAY_MS,
|
||||
PersistenceCoordinator,
|
||||
SessionPersistence,
|
||||
type SessionInspection,
|
||||
type SessionLocation,
|
||||
type SessionPersistenceSnapshot,
|
||||
} from '@deepseek-ai/dsh-session-persistence'
|
||||
import { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SurfaceEventType, SessionHeader, SessionPreparation } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
type JournalMode, openDatabase, rowToStoredMeta, scanRows, type EventRow, type SessionRow,
|
||||
} from './schema.ts'
|
||||
import type { JournalMode } from './schema.ts'
|
||||
import { SqliteStore } from './store.ts'
|
||||
|
||||
export { SCHEMA_VERSION } from './schema.ts'
|
||||
|
||||
/**
|
||||
* Serialize an event's optional envelope fields for SQL binding. The surface
|
||||
* fields are nullable TEXT columns — null when the event has no surface
|
||||
* metadata (non-surface events, events written before surface support); the
|
||||
* ignorable marker is a nullable INTEGER column — `1` iff the envelope carries
|
||||
* `ignorable: true`.
|
||||
*/
|
||||
function envelopeBindings(event: SessionEvent): [string | null, string | null, number | null] {
|
||||
const se = event as SessionEvent<SurfaceEventType>
|
||||
return [
|
||||
se.sourceEventSeqs ? JSON.stringify(se.sourceEventSeqs) : null,
|
||||
se.surfaceOp !== undefined ? JSON.stringify(se.surfaceOp) : null,
|
||||
event.ignorable === true ? 1 : null,
|
||||
]
|
||||
}
|
||||
|
||||
/** Build the source-qualified revision shared by full and lightweight reads. */
|
||||
function sqliteRevision(storeIdentity: string, row: SessionRow): PersistenceRevision {
|
||||
return SessionPersistenceRevision(
|
||||
`${storeIdentity}:incarnation:${row.incarnation}:revision:${row.revision}`,
|
||||
)
|
||||
}
|
||||
|
||||
interface SqliteStoredPrefix {
|
||||
readonly meta: unknown
|
||||
readonly events: unknown[]
|
||||
readonly revision: PersistenceRevision
|
||||
readonly tornMarker?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Exclusively create a missing database file with owner-only permissions.
|
||||
* Existing files retain their modes, and errors other than `EEXIST` propagate.
|
||||
* `DatabaseSync` reopens by path, so this does not protect confidentiality or
|
||||
* integrity when another principal can replace the database entry in its parent
|
||||
* directory.
|
||||
*/
|
||||
async function createDatabaseFile(path: string): Promise<void> {
|
||||
try {
|
||||
const handle = await open(path, 'wx', 0o600)
|
||||
await handle.close()
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error
|
||||
}
|
||||
}
|
||||
/** Default wait for another SQLite connection's write reservation. */
|
||||
export const DEFAULT_BUSY_TIMEOUT_MS = 5_000
|
||||
/** Largest busy timeout accepted by SQLite's signed millisecond interface. */
|
||||
export const MAX_BUSY_TIMEOUT_MS = 2_147_483_647
|
||||
|
||||
/** Plugin configuration. */
|
||||
export interface Config {
|
||||
/**
|
||||
* Filesystem path to the SQLite database file. The special value `:memory:`
|
||||
* opens an in-process database (tests). On filesystems with POSIX modes,
|
||||
* missing directories and databases are created owner-only; existing path
|
||||
* modes are preserved. Filesystem setup errors other than an existing database
|
||||
* fail initialization. The backend does not protect confidentiality or
|
||||
* integrity when another principal can replace the database entry in its
|
||||
* parent directory.
|
||||
*/
|
||||
/** SQLite database path, or `:memory:` for an in-process database. */
|
||||
path: string
|
||||
/**
|
||||
* SQLite `journal_mode` pragma. `wal` (the default) is the recorded
|
||||
* durability model; pick a rollback-journal mode (`delete`/`truncate`/
|
||||
* `persist`) on filesystems where WAL's shared-memory files do not work
|
||||
* (network mounts). See {@link JournalMode}.
|
||||
*/
|
||||
/** Durable SQLite journal mode; defaults to `wal`. */
|
||||
journalMode?: JournalMode
|
||||
/** Maximum wait for another SQLite connection's lock; defaults to 5,000 ms. */
|
||||
busyTimeoutMs?: number
|
||||
/** Maximum cold Session preparations retained for history-to-resume reuse. */
|
||||
preparedSessionCacheSize?: number
|
||||
/** Fixed live-event coalescing window; not a backend completion deadline. */
|
||||
@@ -101,84 +47,49 @@ export interface Config {
|
||||
}
|
||||
|
||||
/**
|
||||
* The SQLite persistence backend. Load as a plugin; it registers as
|
||||
* `ctx.sessionPersistence` and (via the coordinator) installs the write-path
|
||||
* listeners. Its torn-tail marker is the seq to delete from.
|
||||
* SQLite `SessionPersistence` provider with a schema-owned physical codec.
|
||||
*/
|
||||
export class SqliteSessionPersistence extends SessionPersistence implements PersistenceBackend<number> {
|
||||
export class SqliteSessionPersistence extends SessionPersistence {
|
||||
override readonly supportsRawArtifacts = false
|
||||
override readonly name = 'session-persistence-sqlite'
|
||||
|
||||
static inject = ['sessions']
|
||||
|
||||
static Config: z<Config> = z.object({
|
||||
path: z.string().required(),
|
||||
journalMode: z.union(['wal', 'delete', 'truncate', 'persist'] as const).default('wal'),
|
||||
busyTimeoutMs: z.number().step(1).min(0).max(MAX_BUSY_TIMEOUT_MS).default(DEFAULT_BUSY_TIMEOUT_MS),
|
||||
preparedSessionCacheSize: z.number().step(1).min(1).default(DEFAULT_PREPARED_SESSION_CACHE_SIZE),
|
||||
writeBatchMaxDelayMs: z.number().step(1).min(1).max(MAX_WRITE_BATCH_DELAY_MS)
|
||||
.default(DEFAULT_WRITE_BATCH_MAX_DELAY_MS),
|
||||
})
|
||||
|
||||
/**
|
||||
* Backend label for the coordinator's dispose diagnostics. Intentionally
|
||||
* shadows cordis `Service.name` (set to `'sessionPersistence'` by the base);
|
||||
* see the JSONL backend for why this does not affect service resolution.
|
||||
*/
|
||||
override readonly name = 'session-persistence-sqlite'
|
||||
|
||||
private db!: DatabaseSync
|
||||
private storeIdentity!: string
|
||||
private ready: Promise<void>
|
||||
private coordinator: PersistenceCoordinator<number>
|
||||
private readonly store: SqliteStore
|
||||
private readonly coordinator: PersistenceCoordinator<number>
|
||||
|
||||
constructor(ctx: Context, public config: Config) {
|
||||
super(ctx)
|
||||
// Programmatic wrappers may construct the backend without Schemastery normalization.
|
||||
const preparedSessionCacheSize = config.preparedSessionCacheSize
|
||||
?? DEFAULT_PREPARED_SESSION_CACHE_SIZE
|
||||
const writeBatchMaxDelayMs = config.writeBatchMaxDelayMs
|
||||
?? DEFAULT_WRITE_BATCH_MAX_DELAY_MS
|
||||
// Open asynchronously so directory creation does not block plugin apply;
|
||||
// every storage hook awaits the same readiness promise.
|
||||
this.ready = this.openDb(config.path, (config as Required<Config>).journalMode)
|
||||
this.coordinator = new PersistenceCoordinator<number>(this.ctx, this, {
|
||||
this.store = new SqliteStore({
|
||||
path: config.path,
|
||||
journalMode: config.journalMode ?? 'wal',
|
||||
busyTimeoutMs: config.busyTimeoutMs ?? DEFAULT_BUSY_TIMEOUT_MS,
|
||||
})
|
||||
this.coordinator = new PersistenceCoordinator(this.ctx, this.store, {
|
||||
preparedSessionCacheSize,
|
||||
writeBatchMaxDelayMs,
|
||||
})
|
||||
}
|
||||
|
||||
private async openDb(path: string, journalMode: JournalMode): Promise<void> {
|
||||
const actual = path === ':memory:' ? path : resolve(path)
|
||||
if (actual !== ':memory:') {
|
||||
await mkdir(dirname(actual), { recursive: true, mode: 0o700 })
|
||||
await createDatabaseFile(actual)
|
||||
}
|
||||
this.db = openDatabase(actual, journalMode)
|
||||
try {
|
||||
const row = this.db.prepare(
|
||||
'SELECT store_id FROM persistence_state WHERE singleton = 1',
|
||||
).get() as { store_id: string } | undefined
|
||||
/* v8 ignore next -- openDatabase inserts the singleton before returning. */
|
||||
if (row === undefined) {
|
||||
throw new Error(`session database at "${actual}" has no store identity`)
|
||||
}
|
||||
if (row.store_id.length === 0) {
|
||||
throw new Error(`session database at "${actual}" has no valid store identity`)
|
||||
}
|
||||
if (actual !== ':memory:') {
|
||||
const identity = statSync(actual, { bigint: true })
|
||||
this.storeIdentity = `file:${identity.dev}:${identity.ino}:${identity.birthtimeNs}:store:${row.store_id}`
|
||||
} else {
|
||||
this.storeIdentity = `memory:store:${row.store_id}`
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
this.db.close()
|
||||
throw error
|
||||
}
|
||||
/** Reject self-contained path and ownership failures without loading Node SQLite. */
|
||||
protected async [Service.init](): Promise<void> {
|
||||
await this.store.validatePath()
|
||||
}
|
||||
|
||||
// --- SessionPersistence service API (delegated to the coordinator) ---
|
||||
|
||||
/** SQLite has one database, not an independent local artifact per session. */
|
||||
/** SQLite has one database, not an independent per-session artifact. */
|
||||
locate(_meta: SessionHeader): SessionLocation | undefined {
|
||||
return undefined
|
||||
}
|
||||
@@ -203,342 +114,20 @@ export class SqliteSessionPersistence extends SessionPersistence implements Pers
|
||||
return this.coordinator.inspect(id, signal)
|
||||
}
|
||||
|
||||
readFrom(id: SessionId, fromSeq: number, signal?: AbortSignal): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
readFrom(
|
||||
id: SessionId,
|
||||
fromSeq: number,
|
||||
signal?: AbortSignal,
|
||||
): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return this.coordinator.readFrom(id, fromSeq, signal)
|
||||
}
|
||||
|
||||
// One method serves both public `list` and the backend hook; delegating it to
|
||||
// the coordinator would call this hook recursively.
|
||||
|
||||
// --- PersistenceBackend hooks (the SQLite storage primitives) ---
|
||||
|
||||
/** Open repeatable reads over one SQLite row revision. */
|
||||
async openStored(id: SessionId, signal?: AbortSignal): Promise<StoredSessionSource<number> | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
const row = this.rowFor(id)
|
||||
if (row === undefined) return undefined
|
||||
const revision = sqliteRevision(this.storeIdentity, row)
|
||||
return {
|
||||
meta: rowToStoredMeta(row),
|
||||
revision,
|
||||
readEvents: (options = {}): StoredEventRead<number> => {
|
||||
return this.createStoredEventRead(
|
||||
async () => {
|
||||
const fromSeq = options.fromSeq ?? 0
|
||||
const stored = fromSeq === 0
|
||||
? await this.readPrefix(id, signal)
|
||||
: await this.readSuffix(id, fromSeq, signal)
|
||||
if (stored === undefined || stored.revision !== revision) {
|
||||
throw new SessionPersistenceRevisionConflictError(
|
||||
`session "${id}" changed while reading revision ${revision}`,
|
||||
)
|
||||
}
|
||||
return stored
|
||||
},
|
||||
() => true,
|
||||
signal,
|
||||
)
|
||||
},
|
||||
}
|
||||
list(signal?: AbortSignal): Promise<SessionHeader[]> {
|
||||
return this.store.list(signal)
|
||||
}
|
||||
|
||||
/** Read one row's revision without loading its events. */
|
||||
async readStoredRevision(id: SessionId, signal?: AbortSignal): Promise<PersistenceRevision | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
const row = this.rowFor(id)
|
||||
return row === undefined ? undefined : sqliteRevision(this.storeIdentity, row)
|
||||
}
|
||||
|
||||
/**
|
||||
* Seek-capable suffix read: SQL selects `seq >= fromSeq` directly, so the
|
||||
* read scales with the suffix, not the log. Torn rows past the preserved
|
||||
* region are dropped, never repaired (non-mutating read).
|
||||
*/
|
||||
private async readSuffix(id: SessionId, fromSeq: number, signal?: AbortSignal): Promise<SqliteStoredPrefix | undefined> {
|
||||
return this.readStoredEvents(id, fromSeq, false, signal)
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a session's row and ordered events at one SQLite snapshot. The
|
||||
* torn-tail marker is the seq from which a never-committed tail must be deleted
|
||||
* (`scanRows` already returns it as `number | undefined`).
|
||||
*/
|
||||
private async readPrefix(id: SessionId, signal?: AbortSignal): Promise<SqliteStoredPrefix | undefined> {
|
||||
return this.readStoredEvents(id, 0, true, signal)
|
||||
}
|
||||
|
||||
private async readStoredEvents(
|
||||
id: SessionId,
|
||||
fromSeq: number,
|
||||
includeTornMarker: boolean,
|
||||
signal?: AbortSignal,
|
||||
): Promise<SqliteStoredPrefix | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
this.db.exec('BEGIN')
|
||||
let snapshot: { row: SessionRow; eventRows: EventRow[] } | undefined
|
||||
try {
|
||||
const row = this.rowFor(id)
|
||||
if (row !== undefined) {
|
||||
const statement = fromSeq === 0
|
||||
? this.db.prepare('SELECT seq, type, time, data, source_event_seqs, surface_op, ignorable FROM events WHERE session_id = ? ORDER BY seq')
|
||||
: this.db.prepare('SELECT seq, type, time, data, source_event_seqs, surface_op, ignorable FROM events WHERE session_id = ? AND seq >= ? ORDER BY seq')
|
||||
const eventRows = (fromSeq === 0
|
||||
? statement.all(id)
|
||||
: statement.all(id, fromSeq)) as unknown as EventRow[]
|
||||
snapshot = { row, eventRows }
|
||||
}
|
||||
this.db.exec('COMMIT')
|
||||
} catch (error: unknown) {
|
||||
this.db.exec('ROLLBACK')
|
||||
throw error
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
if (snapshot === undefined) return undefined
|
||||
const { row, eventRows } = snapshot
|
||||
const { preserved, tornFrom } = scanRows(eventRows, fromSeq)
|
||||
return {
|
||||
meta: rowToStoredMeta(row),
|
||||
events: preserved,
|
||||
revision: sqliteRevision(this.storeIdentity, row),
|
||||
...includeTornMarker && tornFrom !== undefined ? { tornMarker: tornFrom } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Durably append a batch in ONE transaction: materialize the sessions row (if
|
||||
* lazy) and INSERT every event, or roll back entirely. The transaction is the
|
||||
* atomicity + durability boundary, so a mid-batch failure (a UNIQUE violation
|
||||
* on a duplicated seq) leaves the stored log untouched.
|
||||
*/
|
||||
async appendBatch(meta: SessionHeader, events: readonly SessionEvent[], isMaterialized: boolean): Promise<void> {
|
||||
await this.ready
|
||||
const insertEvent = this.db.prepare(
|
||||
'INSERT INTO events (session_id, seq, type, time, data, source_event_seqs, surface_op, ignorable) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
|
||||
)
|
||||
this.db.exec('BEGIN')
|
||||
try {
|
||||
if (!isMaterialized) this.writeRow(meta)
|
||||
for (const event of events) {
|
||||
const [surfaceSeqs, surfaceOp, ignorable] = envelopeBindings(event)
|
||||
insertEvent.run(meta.id, event.seq, event.type, event.time, JSON.stringify(event.data), surfaceSeqs, surfaceOp, ignorable)
|
||||
}
|
||||
this.db.prepare('UPDATE sessions SET revision = revision + 1 WHERE id = ?').run(meta.id)
|
||||
this.db.exec('COMMIT')
|
||||
} catch (error) {
|
||||
this.db.exec('ROLLBACK')
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Make a crash repair durable in ONE transaction: DELETE the torn tail (from
|
||||
* `tornMarker`) and INSERT the synthetic `closers`. After COMMIT the stored rows
|
||||
* == the balanced log.
|
||||
*/
|
||||
async commitRepair(meta: SessionHeader, tornMarker: number | undefined, closers: readonly SessionEvent[]): Promise<void> {
|
||||
await this.ready
|
||||
this.db.exec('BEGIN')
|
||||
try {
|
||||
if (tornMarker !== undefined) {
|
||||
this.db.prepare('DELETE FROM events WHERE session_id = ? AND seq >= ?').run(meta.id, tornMarker)
|
||||
}
|
||||
if (closers.length > 0) {
|
||||
const insertEvent = this.db.prepare(
|
||||
'INSERT INTO events (session_id, seq, type, time, data, source_event_seqs, surface_op, ignorable) VALUES (?, ?, ?, ?, ?, ?, ?, ?)',
|
||||
)
|
||||
for (const event of closers) {
|
||||
const [surfaceSeqs, surfaceOp, ignorable] = envelopeBindings(event)
|
||||
insertEvent.run(meta.id, event.seq, event.type, event.time, JSON.stringify(event.data), surfaceSeqs, surfaceOp, ignorable)
|
||||
}
|
||||
}
|
||||
if (tornMarker !== undefined || closers.length > 0) {
|
||||
this.db.prepare('UPDATE sessions SET revision = revision + 1 WHERE id = ?').run(meta.id)
|
||||
}
|
||||
this.db.exec('COMMIT')
|
||||
} catch (error) {
|
||||
// The DELETE+INSERT cannot collide (a row at a closer's seq is preserved or
|
||||
// deleted as torn first); this rolls back a DB-level failure (disk full,
|
||||
// etc.), unreachable in test.
|
||||
/* v8 ignore start */
|
||||
this.db.exec('ROLLBACK')
|
||||
throw error
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
|
||||
/** Stage a streamed replacement, then compare and swap it in one transaction. */
|
||||
async replaceStored(
|
||||
expectedRevision: PersistenceRevision,
|
||||
meta: SessionHeader,
|
||||
events: AsyncIterable<SessionEvent>,
|
||||
): Promise<void> {
|
||||
await this.ready
|
||||
const observed = this.rowFor(meta.id)
|
||||
if (observed === undefined
|
||||
|| sqliteRevision(this.storeIdentity, observed) !== expectedRevision) {
|
||||
throw new SessionPersistenceRevisionConflictError(
|
||||
`session "${meta.id}" changed before replacement of revision ${expectedRevision}`,
|
||||
)
|
||||
}
|
||||
if (meta.cwd !== (observed.cwd ?? undefined)) {
|
||||
throw new Error(`replacement for session "${meta.id}" changes its stored identity`)
|
||||
}
|
||||
const staging = `format_upgrade_${randomUUID().replaceAll('-', '')}`
|
||||
this.db.exec(`
|
||||
CREATE TEMP TABLE ${staging} (
|
||||
seq INTEGER PRIMARY KEY,
|
||||
type TEXT NOT NULL,
|
||||
time INTEGER NOT NULL,
|
||||
data TEXT NOT NULL,
|
||||
source_event_seqs TEXT,
|
||||
surface_op TEXT,
|
||||
ignorable INTEGER
|
||||
) STRICT
|
||||
`)
|
||||
try {
|
||||
const stageEvent = this.db.prepare(
|
||||
`INSERT INTO ${staging} (seq, type, time, data, source_event_seqs, surface_op, ignorable) VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
for await (const event of events) {
|
||||
const [surfaceSeqs, surfaceOp, ignorable] = envelopeBindings(event)
|
||||
stageEvent.run(event.seq, event.type, event.time, JSON.stringify(event.data), surfaceSeqs, surfaceOp, ignorable)
|
||||
}
|
||||
|
||||
this.db.exec('BEGIN IMMEDIATE')
|
||||
let began = true
|
||||
try {
|
||||
const row = this.rowFor(meta.id)
|
||||
if (row === undefined
|
||||
|| sqliteRevision(this.storeIdentity, row) !== expectedRevision) {
|
||||
this.db.exec('ROLLBACK')
|
||||
began = false
|
||||
throw new SessionPersistenceRevisionConflictError(
|
||||
`session "${meta.id}" changed before replacement of revision ${expectedRevision}`,
|
||||
)
|
||||
}
|
||||
if (meta.cwd !== (row.cwd ?? undefined)) {
|
||||
throw new Error(`replacement for session "${meta.id}" changes its stored identity`)
|
||||
}
|
||||
|
||||
this.db.prepare('DELETE FROM events WHERE session_id = ?').run(meta.id)
|
||||
this.db.prepare(`
|
||||
INSERT INTO events
|
||||
(session_id, seq, type, time, data, source_event_seqs, surface_op, ignorable)
|
||||
SELECT ?, seq, type, time, data, source_event_seqs, surface_op, ignorable
|
||||
FROM ${staging}
|
||||
ORDER BY seq
|
||||
`).run(meta.id)
|
||||
this.db.prepare(`
|
||||
UPDATE sessions SET
|
||||
version = ?,
|
||||
created_at = ?,
|
||||
cwd = ?,
|
||||
parent_session = ?,
|
||||
seed_length = ?,
|
||||
origin = ?,
|
||||
delegation_depth = ?,
|
||||
agent_preset = ?,
|
||||
revision = revision + 1
|
||||
WHERE id = ?
|
||||
`).run(
|
||||
meta.version,
|
||||
meta.createdAt,
|
||||
meta.cwd ?? null,
|
||||
meta.parentSession ?? null,
|
||||
meta.seedLength ?? null,
|
||||
meta.origin ?? null,
|
||||
meta.delegationDepth ?? null,
|
||||
meta.agentPreset ?? null,
|
||||
meta.id,
|
||||
)
|
||||
this.db.exec('COMMIT')
|
||||
began = false
|
||||
} catch (error: unknown) {
|
||||
if (began) this.db.exec('ROLLBACK')
|
||||
throw error
|
||||
}
|
||||
} finally {
|
||||
this.db.exec(`DROP TABLE IF EXISTS ${staging}`)
|
||||
}
|
||||
}
|
||||
|
||||
/** List all materialized sessions' metadata (every row is a materialized session). */
|
||||
async list(signal?: AbortSignal): Promise<SessionHeader[]> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
const rows = this.db
|
||||
.prepare('SELECT * FROM sessions')
|
||||
.all() as unknown as SessionRow[]
|
||||
signal?.throwIfAborted()
|
||||
return rows.map(row => decodeStoredSessionHeader(rowToStoredMeta(row), SessionId(row.id)))
|
||||
}
|
||||
|
||||
/** List metadata with a source-qualified monotonic revision per session. */
|
||||
async listSnapshots(signal?: AbortSignal): Promise<SessionPersistenceSnapshot[]> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
const rows = this.db.prepare('SELECT * FROM sessions').all() as unknown as SessionRow[]
|
||||
signal?.throwIfAborted()
|
||||
return rows.map(row => ({
|
||||
header: decodeStoredSessionHeader(rowToStoredMeta(row), SessionId(row.id)),
|
||||
revision: SessionPersistenceRevision(
|
||||
`${this.storeIdentity}:incarnation:${row.incarnation}:revision:${row.revision}`,
|
||||
),
|
||||
}))
|
||||
}
|
||||
|
||||
/** Close the database handle (awaited by the coordinator's dispose, post-drain). */
|
||||
async close(): Promise<void> {
|
||||
await this.ready
|
||||
this.db.close()
|
||||
}
|
||||
|
||||
// --- row helpers ---
|
||||
|
||||
/** Fetch a session's row, or undefined if absent. */
|
||||
private rowFor(id: SessionId): SessionRow | undefined {
|
||||
return this.db.prepare('SELECT * FROM sessions WHERE id = ?').get(id) as unknown as SessionRow | undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert-or-replace a session's metadata row. The only caller is the first
|
||||
* materializing `appendBatch`, so writing the row IS the materialization (its
|
||||
* existence is the signal `list` reads).
|
||||
*/
|
||||
private writeRow(meta: SessionHeader): void {
|
||||
this.db.prepare(`
|
||||
INSERT INTO sessions
|
||||
(id, version, created_at, cwd, parent_session, seed_length, origin, delegation_depth, agent_preset, incarnation, revision)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 0)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
version = excluded.version,
|
||||
created_at = excluded.created_at,
|
||||
cwd = excluded.cwd,
|
||||
parent_session = excluded.parent_session,
|
||||
seed_length = excluded.seed_length,
|
||||
origin = excluded.origin,
|
||||
delegation_depth = excluded.delegation_depth,
|
||||
agent_preset = excluded.agent_preset
|
||||
`).run(
|
||||
meta.id,
|
||||
meta.version,
|
||||
meta.createdAt,
|
||||
meta.cwd ?? null,
|
||||
meta.parentSession ?? null,
|
||||
meta.seedLength ?? null,
|
||||
meta.origin ?? null,
|
||||
meta.delegationDepth ?? null,
|
||||
meta.agentPreset ?? null,
|
||||
randomUUID(),
|
||||
)
|
||||
listSnapshots(signal?: AbortSignal): Promise<SessionPersistenceSnapshot[]> {
|
||||
return this.store.listSnapshots(signal)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -15,8 +15,8 @@ export const name = 'session-persistence-sqlite-invariant'
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: persistence correctness requires backend round-trip and crash-tail tests;
|
||||
* this package exposes no continuously observable in-process relation.
|
||||
* No runtime invariant: physical packing is observable only by database
|
||||
* round-trip and row-count checks, not a continuous in-process relation.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
|
||||
@@ -1,87 +1,86 @@
|
||||
/**
|
||||
* Schema + load-time helpers for the SQLite session-persistence backend: the
|
||||
* DDL (a store-identity row, `sessions` metadata, and a 1:1 `events` row per
|
||||
* `SessionEvent`), the database open/configure step, and the last-`turn/end`
|
||||
* cut that gives the SQLite backend the SAME crash-tail-on-load semantics as
|
||||
* the JSONL backend.
|
||||
*
|
||||
* @module dsh-session-persistence-sqlite/schema
|
||||
* SQLite schema ownership and durable-row validation.
|
||||
* @module @deepseek-ai/dsh-session-persistence-sqlite/schema
|
||||
*/
|
||||
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import type { SessionEvent, SessionHeader, SurfaceOp } from '@deepseek-ai/dsh-session'
|
||||
import { isAbsolute } from 'node:path'
|
||||
import { performance } from 'node:perf_hooks'
|
||||
import type { DatabaseSync } from 'node:sqlite'
|
||||
import { setTimeout as delay } from 'node:timers/promises'
|
||||
import {
|
||||
SessionId,
|
||||
type SessionHeader,
|
||||
} from '@deepseek-ai/dsh-session'
|
||||
import { sql } from './sql.ts'
|
||||
|
||||
/**
|
||||
* The on-disk schema version. Bumped only on a breaking change to the table
|
||||
* layout; orthogonal to a session's own `version` (which versions the EVENT
|
||||
* vocabulary, stored per session in the `sessions` row).
|
||||
*/
|
||||
export const SCHEMA_VERSION = 15
|
||||
|
||||
/** SQLite application id protecting unrelated databases from persistence writes. */
|
||||
/** Current physical-record schema with packed and compressed event rows. */
|
||||
export const SCHEMA_VERSION = 17
|
||||
/** Application id reserved for DeepSeek Harness SQLite session databases. */
|
||||
export const SESSION_PERSISTENCE_SQLITE_APPLICATION_ID = 0x44534850
|
||||
|
||||
/**
|
||||
* A row of the `sessions` table — the out-of-log metadata ({@link SessionHeader}).
|
||||
* The row's EXISTENCE is the materialization signal: it is written only by the
|
||||
* first `append` (lazy materialization), so a created-but-never-appended
|
||||
* session has no row and is absent from `list`, mirroring the JSONL
|
||||
* backend's "no file until first append".
|
||||
*/
|
||||
/** A materialized session's metadata and monotonic revision. */
|
||||
export interface SessionRow {
|
||||
id: string
|
||||
version: number
|
||||
created_at: number
|
||||
cwd: string | null
|
||||
parent_session: string | null
|
||||
seed_length: number | null
|
||||
origin: 'subagent' | null
|
||||
/** Stable identity assigned when this log is materialized. */
|
||||
incarnation: string
|
||||
/** Monotonic log-change token incremented in each mutating transaction. */
|
||||
revision: number
|
||||
delegation_depth: number | null
|
||||
agent_preset: string | null
|
||||
readonly id: string
|
||||
readonly version: number
|
||||
readonly created_at: number
|
||||
readonly cwd: string | null
|
||||
readonly parent_session: string | null
|
||||
readonly seed_length: number | null
|
||||
readonly origin: 'subagent' | null
|
||||
readonly incarnation: string
|
||||
readonly revision: number
|
||||
readonly delegation_depth: number | null
|
||||
readonly agent_preset: string | null
|
||||
}
|
||||
|
||||
/** An `events` table row: one `SessionEvent` mapped 1:1 (`data` is JSON text). */
|
||||
/** One physical event row; packed rows may represent multiple logical events. */
|
||||
export interface EventRow {
|
||||
seq: number
|
||||
type: string
|
||||
time: number
|
||||
data: string
|
||||
/** JSON-encoded `number[]` — the event's sourceEventSeqs, or null. */
|
||||
source_event_seqs: string | null
|
||||
/** JSON-encoded `SurfaceOp` — how the event entered the surface, or null. */
|
||||
surface_op: string | null
|
||||
/** `1` iff the event carries the envelope's `ignorable: true` marker, else null. */
|
||||
ignorable: number | null
|
||||
readonly seq: number
|
||||
readonly type: string
|
||||
readonly time: number
|
||||
readonly data: string | Uint8Array
|
||||
readonly source_event_seqs: Uint8Array | null
|
||||
readonly surface_op: string | null
|
||||
readonly ignorable: number | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Journal modes the backend will run under. `wal` is the default and the
|
||||
* durability model the persistence ADR records; the rollback-journal modes
|
||||
* (`delete`/`truncate`/`persist`) exist for filesystems where WAL's
|
||||
* shared-memory files do not work (network mounts). `memory`/`off` are
|
||||
* excluded: dropping journal durability silently contradicts what this
|
||||
* backend promises.
|
||||
*/
|
||||
/** Durable journal modes accepted by the backend. */
|
||||
export type JournalMode = 'wal' | 'delete' | 'truncate' | 'persist'
|
||||
|
||||
interface SchemaObjectRow {
|
||||
readonly type: string
|
||||
readonly name: string
|
||||
readonly tbl_name: string
|
||||
readonly sql: string
|
||||
}
|
||||
|
||||
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/iu
|
||||
const JOURNAL_BUSY_RETRY_INTERVAL_MS = 10
|
||||
type DatabaseSyncConstructor = typeof import('node:sqlite')['DatabaseSync']
|
||||
|
||||
/**
|
||||
* Open the database and apply its schema and pragmas. An empty database with a
|
||||
* zero `user_version` is initialized at {@link SCHEMA_VERSION}; a nonempty
|
||||
* unversioned database and every other non-current version reject rather than
|
||||
* being migrated in place.
|
||||
* @param path - the SQLite database file to open (created when absent).
|
||||
* Open and validate a SQLite session database.
|
||||
* @param Database - lazily imported Node SQLite constructor.
|
||||
* @param path - SQLite path, including `:memory:`.
|
||||
* @param journalMode - validated journal pragma.
|
||||
* @returns the open handle with pragmas applied and all three tables ensured.
|
||||
* @param busyTimeoutMs - validated maximum wait for a competing SQLite lock.
|
||||
* @returns the configured database handle.
|
||||
* @throws when connection settings, schema ownership, or SQLite setup cannot be validated.
|
||||
*/
|
||||
export function openDatabase(path: string, journalMode: JournalMode): DatabaseSync {
|
||||
const db = new DatabaseSync(path)
|
||||
export async function openDatabase(
|
||||
Database: DatabaseSyncConstructor,
|
||||
path: string,
|
||||
journalMode: JournalMode,
|
||||
busyTimeoutMs: number,
|
||||
): Promise<DatabaseSync> {
|
||||
const deadline = performance.now() + busyTimeoutMs
|
||||
const db = new Database(path, { timeout: busyTimeoutMs })
|
||||
try {
|
||||
configureDatabase(db, path, journalMode)
|
||||
configureConnectionSecurity(db, path)
|
||||
configureDatabase(Database, db, path)
|
||||
await selectJournalMode(db, path, journalMode, deadline)
|
||||
configureDurability(db, path)
|
||||
return db
|
||||
} catch (error: unknown) {
|
||||
db.close()
|
||||
@@ -89,192 +88,337 @@ export function openDatabase(path: string, journalMode: JournalMode): DatabaseSy
|
||||
}
|
||||
}
|
||||
|
||||
function configureDatabase(db: DatabaseSync, path: string, journalMode: JournalMode): void {
|
||||
db.exec('PRAGMA foreign_keys = ON')
|
||||
function configureConnectionSecurity(db: DatabaseSync, path: string): void {
|
||||
db.exec(sql('trusted-schema-off'))
|
||||
const trustedSchema = integerField(db.prepare(sql('select-trusted-schema')).get(), 'trusted_schema')
|
||||
/* v8 ignore next 3 -- supported SQLite versions return the fixed setting. */
|
||||
if (trustedSchema !== 0) {
|
||||
throw new Error(`session database at "${path}" retained trusted_schema=${trustedSchema}, expected 0`)
|
||||
}
|
||||
db.exec(sql('mmap-off'))
|
||||
if (path === ':memory:') return
|
||||
const mmapSize = integerField(db.prepare(sql('select-mmap-size')).get(), 'mmap_size')
|
||||
/* v8 ignore next 3 -- supported file-backed SQLite connections return the fixed setting. */
|
||||
if (mmapSize !== 0) {
|
||||
throw new Error(`session database at "${path}" retained mmap_size=${mmapSize}, expected 0`)
|
||||
}
|
||||
}
|
||||
|
||||
function configureDatabase(
|
||||
Database: DatabaseSyncConstructor,
|
||||
db: DatabaseSync,
|
||||
path: string,
|
||||
): void {
|
||||
db.exec(sql('foreign-keys-on'))
|
||||
let began = false
|
||||
try {
|
||||
db.exec('BEGIN IMMEDIATE')
|
||||
db.exec(sql('begin-immediate'))
|
||||
began = true
|
||||
// Validate while holding the write lock so no other connection can change
|
||||
// schema ownership between inspection and initialization.
|
||||
const { user_version: onDisk } = db.prepare('PRAGMA user_version').get() as { user_version: number }
|
||||
const { application_id: applicationId } = db.prepare('PRAGMA application_id').get() as { application_id: number }
|
||||
const { count: userObjectCount } = db.prepare(
|
||||
"SELECT COUNT(*) AS count FROM sqlite_schema WHERE name NOT GLOB 'sqlite_*'",
|
||||
).get() as { count: number }
|
||||
const onDisk = integerField(db.prepare(sql('select-user-version')).get(), 'user_version')
|
||||
const applicationId = integerField(db.prepare(sql('select-application-id')).get(), 'application_id')
|
||||
const userObjectCount = integerField(db.prepare(sql('select-user-object-count')).get(), 'count')
|
||||
if (onDisk === 0 && (applicationId !== 0 || userObjectCount > 0)) {
|
||||
throw new Error(`session database at "${path}" has an unversioned schema or application identity`)
|
||||
}
|
||||
if (onDisk !== 0 && onDisk !== SCHEMA_VERSION) {
|
||||
throw new Error(`session database at "${path}" has schema version ${onDisk}, incompatible with this build (${SCHEMA_VERSION})`)
|
||||
throw new Error(
|
||||
`session database at "${path}" has schema version ${onDisk}, incompatible with this build (${SCHEMA_VERSION})`,
|
||||
)
|
||||
}
|
||||
if (onDisk === SCHEMA_VERSION && applicationId !== SESSION_PERSISTENCE_SQLITE_APPLICATION_ID) {
|
||||
if (onDisk !== 0 && applicationId !== SESSION_PERSISTENCE_SQLITE_APPLICATION_ID) {
|
||||
throw new Error(
|
||||
`session database at "${path}" has application id ${applicationId}, expected ${SESSION_PERSISTENCE_SQLITE_APPLICATION_ID}`,
|
||||
)
|
||||
}
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS persistence_state (
|
||||
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
|
||||
store_id TEXT NOT NULL
|
||||
) STRICT;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
version INTEGER NOT NULL,
|
||||
created_at INTEGER NOT NULL,
|
||||
cwd TEXT,
|
||||
parent_session TEXT,
|
||||
seed_length INTEGER,
|
||||
origin TEXT,
|
||||
delegation_depth INTEGER,
|
||||
agent_preset TEXT,
|
||||
incarnation TEXT NOT NULL,
|
||||
revision INTEGER NOT NULL
|
||||
) STRICT;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS events (
|
||||
session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
|
||||
seq INTEGER NOT NULL,
|
||||
type TEXT NOT NULL,
|
||||
time INTEGER NOT NULL,
|
||||
data TEXT NOT NULL,
|
||||
source_event_seqs TEXT,
|
||||
surface_op TEXT,
|
||||
ignorable INTEGER,
|
||||
PRIMARY KEY (session_id, seq)
|
||||
) STRICT
|
||||
`)
|
||||
db.prepare(
|
||||
'INSERT OR IGNORE INTO persistence_state (singleton, store_id) VALUES (1, ?)',
|
||||
).run(randomUUID())
|
||||
if (onDisk === 0) {
|
||||
db.exec(`PRAGMA application_id = ${SESSION_PERSISTENCE_SQLITE_APPLICATION_ID}`)
|
||||
db.exec(`PRAGMA user_version = ${SCHEMA_VERSION}`)
|
||||
}
|
||||
db.exec('COMMIT')
|
||||
if (onDisk === 0) initializeDatabase(db)
|
||||
validateRequiredSchema(Database, db, path)
|
||||
db.exec(sql('commit'))
|
||||
began = false
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore next -- a BEGIN failure leaves no transaction to roll back. */
|
||||
/* v8 ignore else -- a failed begin leaves no transaction to roll back. */
|
||||
if (began) {
|
||||
/* v8 ignore next 5 -- preserve the original schema failure if SQLite also refuses rollback. */
|
||||
/* v8 ignore next 5 -- retain the original ownership failure if rollback fails too. */
|
||||
try {
|
||||
db.exec('ROLLBACK')
|
||||
db.exec(sql('rollback'))
|
||||
} catch {
|
||||
// The original SQLite failure remains the actionable cause.
|
||||
// The original database-ownership failure remains actionable.
|
||||
}
|
||||
}
|
||||
throw error
|
||||
}
|
||||
// The validated union is safe to interpolate into a non-bindable PRAGMA.
|
||||
// Apply it only after ownership validation and initialization commit.
|
||||
db.exec(`PRAGMA journal_mode = ${journalMode.toUpperCase()}`)
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconstruct the {@link SessionHeader} from a `sessions` row.
|
||||
* @param row - the `sessions` table row.
|
||||
* @returns the header, `NULL` columns mapped to omitted optional fields.
|
||||
*/
|
||||
export function rowToMeta(row: SessionRow): SessionHeader {
|
||||
if (!Number.isSafeInteger(row.created_at) || row.created_at < 0) {
|
||||
throw new Error('stored session createdAt must be a non-negative safe integer')
|
||||
}
|
||||
return rowToStoredMeta(row) as SessionHeader
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconstruct parsed logical header JSON without applying the current Session
|
||||
* format type. The versioned decoder owns structural migration and validation.
|
||||
* @param row - stored session row.
|
||||
* @returns logical header fields represented by the physical schema.
|
||||
*/
|
||||
export function rowToStoredMeta(row: SessionRow): unknown {
|
||||
return {
|
||||
version: row.version,
|
||||
id: row.id,
|
||||
createdAt: row.created_at,
|
||||
...row.cwd !== null ? { cwd: row.cwd } : {},
|
||||
...row.parent_session !== null ? { parentSession: row.parent_session } : {},
|
||||
...row.seed_length !== null ? { seedLength: row.seed_length } : {},
|
||||
...row.origin !== null ? { origin: row.origin } : {},
|
||||
...row.delegation_depth !== null ? { delegationDepth: row.delegation_depth } : {},
|
||||
...row.agent_preset !== null ? { agentPreset: row.agent_preset } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconstruct a {@link SessionEvent} from an `events` row (parses `data`).
|
||||
* @param row - the `events` table row; `data` and the surface columns hold JSON text.
|
||||
* @returns the reconstructed event; throws when a JSON column fails to parse
|
||||
* ({@link scanRows} treats that as a hole, not corruption, in the tail).
|
||||
*/
|
||||
export function rowToEvent(row: EventRow): SessionEvent {
|
||||
// Surface-metadata fields are conditional on the event type in the type
|
||||
// system; spread them so each variant gets only the fields it declares.
|
||||
const surfaceFields = {
|
||||
...row.source_event_seqs !== null ? { sourceEventSeqs: JSON.parse(row.source_event_seqs) as number[] } : {},
|
||||
...row.surface_op !== null ? { surfaceOp: JSON.parse(row.surface_op) as SurfaceOp } : {},
|
||||
}
|
||||
const ignorableField = row.ignorable === 1 ? { ignorable: true as const } : {}
|
||||
return {
|
||||
type: row.type as SessionEvent['type'],
|
||||
seq: row.seq,
|
||||
time: row.time,
|
||||
data: JSON.parse(row.data) as SessionEvent['data'],
|
||||
...surfaceFields,
|
||||
...ignorableField,
|
||||
} as SessionEvent
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the preserved prefix of ordered event rows. Fully written rows in an
|
||||
* interrupted final turn remain in the prefix. The first unparsable row or seq
|
||||
* gap after the last `turn/end` marks a tolerated torn tail; the same hole in
|
||||
* the committed region rejects.
|
||||
*
|
||||
* @param rows - one session's event rows, ordered by seq ascending.
|
||||
* @param base - the seq the first row is expected to carry; `0` for a whole
|
||||
* log, or the requested `fromSeq` for a seekable suffix read.
|
||||
* @returns the preserved event prefix, plus `tornFrom` — the seq the physical
|
||||
* delete starts at — when a torn tail exists.
|
||||
*/
|
||||
export function scanRows(rows: readonly EventRow[], base = 0): { preserved: SessionEvent[]; tornFrom?: number } {
|
||||
// Pass 1: parse each row's data; a row whose data is not valid JSON is a hole.
|
||||
// (The seq/type COLUMNS are always present even when `data` is corrupt.)
|
||||
interface Parsed { ok: boolean; event?: SessionEvent }
|
||||
const parsed: Parsed[] = rows.map((row) => {
|
||||
async function selectJournalMode(
|
||||
db: DatabaseSync,
|
||||
path: string,
|
||||
journalMode: JournalMode,
|
||||
deadline: number,
|
||||
): Promise<void> {
|
||||
let result: unknown
|
||||
while (true) {
|
||||
try {
|
||||
return { ok: true, event: rowToEvent(row) }
|
||||
} catch {
|
||||
return { ok: false }
|
||||
result = db.prepare(sql(journalResource(journalMode))).get()
|
||||
break
|
||||
} catch (error: unknown) {
|
||||
const remainingMs = Math.max(0, Math.ceil(deadline - performance.now()))
|
||||
if (!isSqliteBusy(error) || remainingMs === 0) throw error
|
||||
await delay(Math.min(JOURNAL_BUSY_RETRY_INTERVAL_MS, remainingMs))
|
||||
if (performance.now() >= deadline) throw error
|
||||
}
|
||||
}
|
||||
const selected = stringField(result, 'journal_mode').toLowerCase()
|
||||
const expected = path === ':memory:' ? 'memory' : journalMode
|
||||
/* v8 ignore next 3 -- SQLite returns the selected mode from these fixed, valid pragmas. */
|
||||
if (selected !== expected) {
|
||||
throw new Error(`session database at "${path}" selected journal mode ${selected}, expected ${expected}`)
|
||||
}
|
||||
}
|
||||
|
||||
function configureDurability(db: DatabaseSync, path: string): void {
|
||||
db.exec(sql('synchronous-full'))
|
||||
const synchronous = integerField(db.prepare(sql('select-synchronous')).get(), 'synchronous')
|
||||
/* v8 ignore next 3 -- supported SQLite versions return the fixed setting. */
|
||||
if (synchronous !== 2) {
|
||||
throw new Error(`session database at "${path}" retained synchronous=${synchronous}, expected FULL (2)`)
|
||||
}
|
||||
}
|
||||
|
||||
function isSqliteBusy(error: unknown): boolean {
|
||||
return typeof error === 'object'
|
||||
&& error !== null
|
||||
&& Reflect.get(error, 'errcode') === 5
|
||||
}
|
||||
|
||||
function journalResource(mode: JournalMode):
|
||||
| 'journal-mode-wal'
|
||||
| 'journal-mode-delete'
|
||||
| 'journal-mode-truncate'
|
||||
| 'journal-mode-persist' {
|
||||
switch (mode) {
|
||||
case 'wal': return 'journal-mode-wal'
|
||||
case 'delete': return 'journal-mode-delete'
|
||||
case 'truncate': return 'journal-mode-truncate'
|
||||
case 'persist': return 'journal-mode-persist'
|
||||
}
|
||||
}
|
||||
|
||||
function initializeDatabase(db: DatabaseSync): void {
|
||||
db.exec(sql('schema'))
|
||||
db.prepare(sql('insert-persistence-state')).run(randomUUID())
|
||||
db.exec(sql('set-application-id'))
|
||||
db.exec(sql('set-user-version-17'))
|
||||
}
|
||||
|
||||
let canonicalSchema: readonly SchemaObjectRow[] | undefined
|
||||
|
||||
function expectedSchema(Database: DatabaseSyncConstructor): readonly SchemaObjectRow[] {
|
||||
if (canonicalSchema !== undefined) return canonicalSchema
|
||||
const reference = new Database(':memory:')
|
||||
try {
|
||||
reference.exec(sql('foreign-keys-on'))
|
||||
reference.exec(sql('schema'))
|
||||
canonicalSchema = schemaObjects(reference)
|
||||
return canonicalSchema
|
||||
} finally {
|
||||
reference.close()
|
||||
}
|
||||
}
|
||||
|
||||
function schemaObjects(db: DatabaseSync): SchemaObjectRow[] {
|
||||
return db.prepare(sql('select-schema-objects')).all().map((value) => {
|
||||
const row = record(value, 'schema object')
|
||||
return {
|
||||
type: stringField(row, 'type'),
|
||||
name: stringField(row, 'name'),
|
||||
tbl_name: stringField(row, 'tbl_name'),
|
||||
sql: normalizeSql(stringField(row, 'sql')),
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// The last index that is a valid `turn/end` — holes through a closed turn
|
||||
// are always committed corruption.
|
||||
let lastTurnEnd = -1
|
||||
for (let i = parsed.length - 1; i >= 0; i--) {
|
||||
if (parsed[i]?.ok && rows[i]?.type === 'turn/end') { lastTurnEnd = i; break }
|
||||
function normalizeSql(value: string): string {
|
||||
return value.replaceAll(/\s+/gu, ' ').trim()
|
||||
}
|
||||
|
||||
function validateRequiredSchema(
|
||||
Database: DatabaseSyncConstructor,
|
||||
db: DatabaseSync,
|
||||
path: string,
|
||||
): void {
|
||||
if (JSON.stringify(schemaObjects(db)) !== JSON.stringify(expectedSchema(Database))) {
|
||||
throw new Error(`session database at "${path}" does not contain the required schema objects`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Recheck schema ownership inside the caller's mutation transaction.
|
||||
* @param Database - constructor used to validate the canonical schema.
|
||||
* @param db - open owned database with an active immediate transaction.
|
||||
* @param path - database location used in ownership diagnostics.
|
||||
* @throws when another writer changed the application identity, schema, or version.
|
||||
*/
|
||||
export function validateSchemaForMutation(
|
||||
Database: DatabaseSyncConstructor,
|
||||
db: DatabaseSync,
|
||||
path: string,
|
||||
): void {
|
||||
const version = integerField(db.prepare(sql('select-user-version')).get(), 'user_version')
|
||||
const applicationId = integerField(db.prepare(sql('select-application-id')).get(), 'application_id')
|
||||
if (applicationId !== SESSION_PERSISTENCE_SQLITE_APPLICATION_ID) {
|
||||
throw new Error(
|
||||
`session database application id changed before mutation (expected ${SESSION_PERSISTENCE_SQLITE_APPLICATION_ID}, got ${applicationId})`,
|
||||
)
|
||||
}
|
||||
validateRequiredSchema(Database, db, path)
|
||||
if (version !== SCHEMA_VERSION) {
|
||||
throw new Error(`session database schema changed before mutation (expected ${SCHEMA_VERSION}, got ${version})`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode and validate one durable session row.
|
||||
* @param value - value returned by SQLite.
|
||||
* @returns a validated session row.
|
||||
*/
|
||||
export function decodeSessionRow(value: unknown): SessionRow {
|
||||
const row = record(value, 'stored session metadata')
|
||||
const id = nonemptyStringField(row, 'id')
|
||||
const version = safeIntegerField(row, 'version')
|
||||
const cwd = nullableStringField(row, 'cwd')
|
||||
if (cwd !== null && !isAbsolute(cwd)) throw new Error('stored session cwd must be absolute')
|
||||
const parent = nullableStringField(row, 'parent_session')
|
||||
const origin = nullableStringField(row, 'origin')
|
||||
if (origin !== null && origin !== 'subagent') throw new Error('stored session origin must be subagent or null')
|
||||
const incarnation = nonemptyStringField(row, 'incarnation')
|
||||
if (!UUID.test(incarnation)) throw new Error('stored session incarnation must be a UUID')
|
||||
return {
|
||||
id,
|
||||
version,
|
||||
created_at: nonnegativeSafeIntegerField(row, 'created_at'),
|
||||
cwd,
|
||||
parent_session: parent,
|
||||
seed_length: nullableNonnegativeSafeIntegerField(row, 'seed_length'),
|
||||
origin,
|
||||
delegation_depth: nullableNonnegativeSafeIntegerField(row, 'delegation_depth'),
|
||||
agent_preset: nullableStringField(row, 'agent_preset'),
|
||||
incarnation,
|
||||
revision: nonnegativeSafeIntegerField(row, 'revision'),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode and validate one durable event row before JSON interpretation.
|
||||
* @param value - value returned by SQLite.
|
||||
* @returns a validated physical event row.
|
||||
*/
|
||||
export function decodeEventRow(value: unknown): EventRow {
|
||||
const row = record(value, 'stored event')
|
||||
const ignorable = nullableSafeIntegerField(row, 'ignorable')
|
||||
if (ignorable !== null && ignorable !== 0 && ignorable !== 1) {
|
||||
throw new Error('stored event ignorable must be 0, 1, or null')
|
||||
}
|
||||
return {
|
||||
seq: nonnegativeSafeIntegerField(row, 'seq'),
|
||||
type: nonemptyStringField(row, 'type'),
|
||||
time: safeIntegerField(row, 'time'),
|
||||
data: stringOrBlobField(row, 'data'),
|
||||
source_event_seqs: nullableBlobField(row, 'source_event_seqs'),
|
||||
surface_op: nullableStringField(row, 'surface_op'),
|
||||
ignorable,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate the singleton identity read from durable storage.
|
||||
* @param value - value returned by SQLite.
|
||||
* @returns the UUID store identity.
|
||||
*/
|
||||
export function decodeStoreIdentity(value: unknown): string {
|
||||
const identity = nonemptyStringField(value, 'store_id')
|
||||
if (!UUID.test(identity)) throw new Error('stored store_id must be a UUID')
|
||||
return identity
|
||||
}
|
||||
|
||||
// Preserve the contiguous prefix, including a complete interrupted turn;
|
||||
// holes through the last committed boundary throw, while later holes stop.
|
||||
const preserved: SessionEvent[] = []
|
||||
for (let i = 0; i < rows.length; i++) {
|
||||
const p = parsed[i]
|
||||
if (!p?.ok || p.event === undefined) {
|
||||
if (i <= lastTurnEnd) throw new Error(`corrupt session log: unparsable committed event at seq ${rows[i]?.seq}`)
|
||||
break // torn tail fragment after the last turn/end — stop, tolerate
|
||||
}
|
||||
if (p.event.seq !== base + i) {
|
||||
if (i <= lastTurnEnd) throw new Error(`corrupt session log: seq gap in committed region (expected ${base + i}, got ${p.event.seq})`)
|
||||
break // gap after the last turn/end — torn tail, stop
|
||||
}
|
||||
preserved.push(p.event)
|
||||
/**
|
||||
* Reconstruct an immutable session header from a validated metadata row.
|
||||
* @param row - validated stored metadata row.
|
||||
* @returns the session header.
|
||||
*/
|
||||
export function rowToMeta(row: SessionRow): SessionHeader {
|
||||
return {
|
||||
version: row.version,
|
||||
id: SessionId(row.id),
|
||||
createdAt: row.created_at,
|
||||
...row.cwd === null ? {} : { cwd: row.cwd },
|
||||
...row.parent_session === null ? {} : { parentSession: SessionId(row.parent_session) },
|
||||
...row.seed_length === null ? {} : { seedLength: row.seed_length },
|
||||
...row.origin === null ? {} : { origin: row.origin },
|
||||
...row.delegation_depth === null ? {} : { delegationDepth: row.delegation_depth },
|
||||
...row.agent_preset === null ? {} : { agentPreset: row.agent_preset },
|
||||
}
|
||||
}
|
||||
|
||||
function record(value: unknown, label: string): Record<string, unknown> {
|
||||
if (typeof value !== 'object' || value === null) throw new Error(`${label} must be an object`)
|
||||
return value as Record<string, unknown>
|
||||
}
|
||||
|
||||
function stringField(value: unknown, key: string): string {
|
||||
const field = record(value, 'SQLite row')[key]
|
||||
if (typeof field !== 'string') throw new Error(`stored ${key} must be a string`)
|
||||
return field
|
||||
}
|
||||
|
||||
function nonemptyStringField(value: unknown, key: string): string {
|
||||
const field = stringField(value, key)
|
||||
if (field.length === 0) throw new Error(`stored ${key} must not be empty`)
|
||||
return field
|
||||
}
|
||||
|
||||
function nullableStringField(value: unknown, key: string): string | null {
|
||||
const field = record(value, 'SQLite row')[key]
|
||||
if (field === null) return null
|
||||
if (typeof field !== 'string') throw new Error(`stored ${key} must be a string or null`)
|
||||
return field
|
||||
}
|
||||
|
||||
function stringOrBlobField(value: unknown, key: string): string | Uint8Array {
|
||||
const field = record(value, 'SQLite row')[key]
|
||||
if (typeof field === 'string' || field instanceof Uint8Array) return field
|
||||
throw new Error(`stored ${key} must be a string or blob`)
|
||||
}
|
||||
|
||||
function nullableBlobField(value: unknown, key: string): Uint8Array | null {
|
||||
const field = record(value, 'SQLite row')[key]
|
||||
if (field === null || field instanceof Uint8Array) return field
|
||||
throw new Error(`stored ${key} must be a blob or null`)
|
||||
}
|
||||
|
||||
function integerField(value: unknown, key: string): number {
|
||||
const field = record(value, 'SQLite row')[key]
|
||||
if (!Number.isSafeInteger(field)) throw new Error(`stored ${key} must be a safe integer`)
|
||||
return field as number
|
||||
}
|
||||
|
||||
function safeIntegerField(value: unknown, key: string): number {
|
||||
return integerField(value, key)
|
||||
}
|
||||
|
||||
function nonnegativeSafeIntegerField(value: unknown, key: string): number {
|
||||
const field = integerField(value, key)
|
||||
if (field < 0) throw new Error(`stored ${key} must be non-negative`)
|
||||
return field
|
||||
}
|
||||
|
||||
function nullableSafeIntegerField(value: unknown, key: string): number | null {
|
||||
const field = record(value, 'SQLite row')[key]
|
||||
if (field === null) return null
|
||||
if (!Number.isSafeInteger(field)) throw new Error(`stored ${key} must be a safe integer or null`)
|
||||
return field as number
|
||||
}
|
||||
|
||||
// Any rows past the preserved prefix are a never-committed torn tail; their
|
||||
// first seq is the deletion point for load's physical repair.
|
||||
return preserved.length < rows.length ? { preserved, tornFrom: base + preserved.length } : { preserved }
|
||||
function nullableNonnegativeSafeIntegerField(value: unknown, key: string): number | null {
|
||||
const field = nullableSafeIntegerField(value, key)
|
||||
if (field !== null && field < 0) throw new Error(`stored ${key} must be non-negative or null`)
|
||||
return field
|
||||
}
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
/**
|
||||
* Closed, package-owned SQL resource loading for SQLite.
|
||||
* @module @deepseek-ai/dsh-session-persistence-sqlite/sql
|
||||
*/
|
||||
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const SQL_RESOURCES = [
|
||||
'begin',
|
||||
'begin-immediate',
|
||||
'commit',
|
||||
'delete-events-from',
|
||||
'foreign-keys-on',
|
||||
'insert-event',
|
||||
'insert-persistence-state',
|
||||
'journal-mode-delete',
|
||||
'journal-mode-persist',
|
||||
'journal-mode-truncate',
|
||||
'journal-mode-wal',
|
||||
'mmap-off',
|
||||
'rollback',
|
||||
'schema',
|
||||
'select-application-id',
|
||||
'select-events',
|
||||
'select-events-from',
|
||||
'select-mmap-size',
|
||||
'select-packed-predecessors',
|
||||
'select-schema-objects',
|
||||
'select-session',
|
||||
'select-sessions',
|
||||
'select-store-id',
|
||||
'select-synchronous',
|
||||
'select-tail-events',
|
||||
'select-trusted-schema',
|
||||
'select-user-object-count',
|
||||
'select-user-version',
|
||||
'set-application-id',
|
||||
'set-user-version-17',
|
||||
'synchronous-full',
|
||||
'trusted-schema-off',
|
||||
'update-session-revision',
|
||||
'upsert-session',
|
||||
] as const
|
||||
|
||||
/** A resource basename selected exclusively by package code. */
|
||||
export type SqlResourceName = typeof SQL_RESOURCES[number]
|
||||
|
||||
const cache = new Map<SqlResourceName, string>()
|
||||
|
||||
/**
|
||||
* Load an immutable SQL statement by closed resource name.
|
||||
* @param name - package-owned resource basename.
|
||||
* @returns the resource text.
|
||||
*/
|
||||
export function sql(name: SqlResourceName): string {
|
||||
const cached = cache.get(name)
|
||||
if (cached !== undefined) return cached
|
||||
const statement = readFileSync(
|
||||
fileURLToPath(new URL(`../resources/sql/${name}.sql`, import.meta.url)),
|
||||
'utf8',
|
||||
)
|
||||
cache.set(name, statement)
|
||||
return statement
|
||||
}
|
||||
@@ -0,0 +1,624 @@
|
||||
/**
|
||||
* SQLite storage primitives: transactional append-batch packing, physical
|
||||
* reads, schema validation, revisions, repair, and lifecycle closure.
|
||||
* @module @deepseek-ai/dsh-session-persistence-sqlite/store
|
||||
*/
|
||||
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { statSync } from 'node:fs'
|
||||
import { lstat, mkdir, open } from 'node:fs/promises'
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import type { DatabaseSync, StatementSync } from 'node:sqlite'
|
||||
import {
|
||||
SessionId,
|
||||
type SessionEvent,
|
||||
type SessionHeader,
|
||||
} from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
decodeStoredSessionHeader,
|
||||
SessionPersistenceRevision,
|
||||
SessionPersistenceRevisionConflictError,
|
||||
type PersistenceBackend,
|
||||
type SessionPersistenceRevision as PersistenceRevision,
|
||||
type SessionPersistenceSnapshot,
|
||||
type StoredEventRead,
|
||||
type StoredEventReadCompletion,
|
||||
type StoredEventReadOptions,
|
||||
type StoredSessionSource,
|
||||
} from '@deepseek-ai/dsh-session-persistence'
|
||||
import {
|
||||
MAX_PACKED_ROW_MEMBERS,
|
||||
packChunkRuns,
|
||||
} from './codec.ts'
|
||||
import {
|
||||
bindRecord,
|
||||
decodeRow,
|
||||
scanRows,
|
||||
type BoundRecord,
|
||||
} from './compression.ts'
|
||||
import {
|
||||
type EventRow,
|
||||
type JournalMode,
|
||||
decodeEventRow,
|
||||
decodeSessionRow,
|
||||
decodeStoreIdentity,
|
||||
openDatabase,
|
||||
validateSchemaForMutation,
|
||||
rowToMeta,
|
||||
type SessionRow,
|
||||
} from './schema.ts'
|
||||
import { sql } from './sql.ts'
|
||||
|
||||
/** Storage options resolved by the service provider. */
|
||||
export interface SqliteStoreOptions {
|
||||
readonly path: string
|
||||
readonly journalMode: JournalMode
|
||||
readonly busyTimeoutMs: number
|
||||
}
|
||||
|
||||
/** A stored session's header, valid event prefix, and revision at one snapshot. */
|
||||
interface SqliteStoredPrefix {
|
||||
readonly meta: SessionHeader
|
||||
readonly events: SessionEvent[]
|
||||
readonly revision: PersistenceRevision
|
||||
readonly tornMarker?: number
|
||||
}
|
||||
|
||||
/** A stored session's suffix (events at or past a seq) and its snapshot revision. */
|
||||
interface SqliteStoredSuffix {
|
||||
readonly meta: SessionHeader
|
||||
readonly events: SessionEvent[]
|
||||
readonly revision: PersistenceRevision
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the standard lazy event stream and EOF metadata around one backend
|
||||
* read, mirroring the service's protected helper for standalone stores.
|
||||
* @param load - revision-checked batch loader owned by the backend.
|
||||
* @param include - whether one loaded event belongs in this physical read.
|
||||
* @param signal - optional cancellation checked between yielded events.
|
||||
* @returns an independently consumable event read.
|
||||
*/
|
||||
function createStoredEventRead<TornMarker>(
|
||||
load: () => Promise<{ readonly events: readonly unknown[]; readonly tornMarker?: TornMarker }>,
|
||||
include: (event: unknown) => boolean,
|
||||
signal?: AbortSignal,
|
||||
): StoredEventRead<TornMarker> {
|
||||
const completed = Promise.withResolvers<StoredEventReadCompletion<TornMarker>>()
|
||||
const events = (async function* (): AsyncIterable<unknown> {
|
||||
try {
|
||||
const batch = await load()
|
||||
for (const event of batch.events) {
|
||||
signal?.throwIfAborted()
|
||||
if (include(event)) yield event
|
||||
}
|
||||
completed.resolve(batch.tornMarker === undefined ? {} : { tornMarker: batch.tornMarker })
|
||||
} catch (error: unknown) {
|
||||
completed.reject(error)
|
||||
throw error
|
||||
}
|
||||
})()
|
||||
return { events, completed: completed.promise }
|
||||
}
|
||||
|
||||
/** SQLite implementation of the coordinator's physical backend hooks. */
|
||||
export class SqliteStore implements PersistenceBackend<number> {
|
||||
readonly name = 'session-persistence-sqlite'
|
||||
private db!: DatabaseSync
|
||||
private databaseConstructor!: typeof import('node:sqlite')['DatabaseSync']
|
||||
private storeIdentity!: string
|
||||
private databasePath!: string
|
||||
private opened = false
|
||||
private pathReady: Promise<void> | undefined
|
||||
private ready: Promise<void> | undefined
|
||||
|
||||
constructor(private readonly options: SqliteStoreOptions) {}
|
||||
|
||||
/**
|
||||
* Validate filesystem ownership without importing or opening Node SQLite.
|
||||
* @returns settlement of the store's one path-validation operation.
|
||||
*/
|
||||
validatePath(): Promise<void> {
|
||||
this.pathReady ??= this.preparePath(this.options.path)
|
||||
return this.pathReady
|
||||
}
|
||||
|
||||
/**
|
||||
* Lazily open and validate the database on first persistence use.
|
||||
* @returns settlement of the store's one database-open operation.
|
||||
*/
|
||||
open(): Promise<void> {
|
||||
this.ready ??= this.openDb()
|
||||
return this.ready
|
||||
}
|
||||
|
||||
private async preparePath(path: string): Promise<void> {
|
||||
const actual = path === ':memory:' ? path : resolve(path)
|
||||
if (actual !== ':memory:') {
|
||||
await mkdir(dirname(actual), { recursive: true, mode: 0o700 })
|
||||
await validateParentDirectory(dirname(actual))
|
||||
await validateDatabaseFileIfPresent(actual)
|
||||
}
|
||||
this.databasePath = actual
|
||||
}
|
||||
|
||||
private async openDb(): Promise<void> {
|
||||
await this.validatePath()
|
||||
if (this.databasePath !== ':memory:') {
|
||||
await createDatabaseFile(this.databasePath)
|
||||
await validateDatabaseFile(this.databasePath)
|
||||
}
|
||||
const { DatabaseSync } = await loadNodeSqlite()
|
||||
this.databaseConstructor = DatabaseSync
|
||||
this.db = await openDatabase(
|
||||
DatabaseSync,
|
||||
this.databasePath,
|
||||
this.options.journalMode,
|
||||
this.options.busyTimeoutMs,
|
||||
)
|
||||
try {
|
||||
const row = this.db.prepare(sql('select-store-id')).get()
|
||||
if (row === undefined) {
|
||||
throw new Error(`session database at "${this.databasePath}" has no valid store identity`)
|
||||
}
|
||||
let storeId: string
|
||||
try {
|
||||
storeId = decodeStoreIdentity(row)
|
||||
} catch (error: unknown) {
|
||||
throw new Error(`session database at "${this.databasePath}" has no valid store identity`, { cause: error })
|
||||
}
|
||||
if (this.databasePath === ':memory:') {
|
||||
this.storeIdentity = `memory:store:${storeId}`
|
||||
} else {
|
||||
const identity = statSync(this.databasePath, { bigint: true })
|
||||
this.storeIdentity = `file:${identity.dev}:${identity.ino}:${identity.birthtimeNs}:store:${storeId}`
|
||||
}
|
||||
this.opened = true
|
||||
} catch (error: unknown) {
|
||||
this.db.close()
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
async loadStored(id: SessionId, signal?: AbortSignal): Promise<SqliteStoredPrefix | undefined> {
|
||||
await this.observe(signal)
|
||||
const snapshot = this.readTransaction(() => {
|
||||
const row = this.rowFor(id)
|
||||
if (row === undefined) return undefined
|
||||
const eventRows = this.db.prepare(sql('select-events')).all(id).map(decodeEventRow)
|
||||
return { row, eventRows }
|
||||
})
|
||||
signal?.throwIfAborted()
|
||||
if (snapshot === undefined) return undefined
|
||||
const scanned = scanRows(snapshot.eventRows)
|
||||
return {
|
||||
meta: rowToMeta(snapshot.row),
|
||||
events: scanned.preserved,
|
||||
revision: sqliteRevision(this.storeIdentity, snapshot.row),
|
||||
...scanned.tornFrom === undefined ? {} : { tornMarker: scanned.tornFrom },
|
||||
}
|
||||
}
|
||||
|
||||
async readStoredRevision(id: SessionId, signal?: AbortSignal): Promise<PersistenceRevision | undefined> {
|
||||
await this.observe(signal)
|
||||
const row = this.rowFor(id)
|
||||
signal?.throwIfAborted()
|
||||
return row === undefined ? undefined : sqliteRevision(this.storeIdentity, row)
|
||||
}
|
||||
|
||||
async loadStoredFrom(id: SessionId, fromSeq: number, signal?: AbortSignal): Promise<SqliteStoredSuffix | undefined> {
|
||||
await this.observe(signal)
|
||||
const snapshot = this.readTransaction(() => {
|
||||
const row = this.rowFor(id)
|
||||
if (row === undefined) return undefined
|
||||
return { row, ...this.physicalSpanFrom(id, fromSeq) }
|
||||
})
|
||||
signal?.throwIfAborted()
|
||||
if (snapshot === undefined) return undefined
|
||||
const { preserved } = scanRows(snapshot.eventRows, snapshot.base)
|
||||
return {
|
||||
meta: rowToMeta(snapshot.row),
|
||||
events: preserved.filter(event => event.seq >= fromSeq),
|
||||
revision: sqliteRevision(this.storeIdentity, snapshot.row),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Open repeatable reads over one row revision. Each event reader reproduces
|
||||
* this revision or rejects when a concurrent writer changed the row.
|
||||
* @param id - persisted session id to resolve.
|
||||
* @param signal - optional cancellation for backend read work.
|
||||
* @returns the source, or `undefined` when the session has no stored row.
|
||||
*/
|
||||
async openStored(id: SessionId, signal?: AbortSignal): Promise<StoredSessionSource<number> | undefined> {
|
||||
await this.observe(signal)
|
||||
const row = this.rowFor(id)
|
||||
signal?.throwIfAborted()
|
||||
if (row === undefined) return undefined
|
||||
const revision = sqliteRevision(this.storeIdentity, row)
|
||||
return {
|
||||
meta: rowToMeta(row),
|
||||
revision,
|
||||
readEvents: (options: StoredEventReadOptions = {}): StoredEventRead<number> => {
|
||||
const fromSeq = options.fromSeq ?? 0
|
||||
return createStoredEventRead(
|
||||
async () => {
|
||||
const stored = fromSeq === 0
|
||||
? await this.loadStored(id, signal)
|
||||
: await this.loadStoredFrom(id, fromSeq, signal)
|
||||
if (stored === undefined || stored.revision !== revision) {
|
||||
throw new SessionPersistenceRevisionConflictError(
|
||||
`session "${id}" changed while reading revision ${revision}`,
|
||||
)
|
||||
}
|
||||
return stored
|
||||
},
|
||||
() => true,
|
||||
signal,
|
||||
)
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
async appendBatch(
|
||||
meta: SessionHeader,
|
||||
events: readonly SessionEvent[],
|
||||
isMaterialized: boolean,
|
||||
): Promise<void> {
|
||||
await this.open()
|
||||
if (events.length === 0) return
|
||||
this.db.exec(sql('begin-immediate'))
|
||||
try {
|
||||
validateSchemaForMutation(this.databaseConstructor, this.db, this.databasePath)
|
||||
const tailRows = this.tailRows(meta.id)
|
||||
const currentLast = this.logicalLastEvent(meta.id, tailRows)
|
||||
const expected = currentLast === undefined ? 0 : currentLast.seq + 1
|
||||
const first = events[0] as SessionEvent
|
||||
if (first.seq !== expected) {
|
||||
throw new Error(`session ${meta.id} append starts at seq ${first.seq}, stored next seq is ${expected}`)
|
||||
}
|
||||
if (!isMaterialized) this.writeRow(meta)
|
||||
|
||||
const insert = this.insertStatement()
|
||||
for (const record of packChunkRuns(events)) this.insertRecord(insert, meta.id, bindRecord(record))
|
||||
this.incrementRevision(meta.id)
|
||||
this.db.exec(sql('commit'))
|
||||
} catch (error: unknown) {
|
||||
this.rollback(error, 'append')
|
||||
}
|
||||
}
|
||||
|
||||
async commitRepair(
|
||||
meta: SessionHeader,
|
||||
tornMarker: number | undefined,
|
||||
closers: readonly SessionEvent[],
|
||||
): Promise<void> {
|
||||
await this.open()
|
||||
if (tornMarker === undefined && closers.length === 0) return
|
||||
this.db.exec(sql('begin-immediate'))
|
||||
try {
|
||||
validateSchemaForMutation(this.databaseConstructor, this.db, this.databasePath)
|
||||
const row = this.rowFor(meta.id)
|
||||
if (row === undefined) throw new Error(`session ${meta.id} metadata row is missing`)
|
||||
const currentRows = this.db.prepare(sql('select-events')).all(meta.id).map(decodeEventRow)
|
||||
const current = scanRows(currentRows)
|
||||
if (tornMarker !== undefined) {
|
||||
if (current.tornFrom !== tornMarker) {
|
||||
throw new Error(`session ${meta.id} repair is stale: physical tail no longer starts at seq ${tornMarker}`)
|
||||
}
|
||||
this.db.prepare(sql('delete-events-from'))
|
||||
.run(meta.id, tornMarker)
|
||||
} else if (current.tornFrom !== undefined) {
|
||||
throw new Error(`session ${meta.id} repair omitted current torn tail at seq ${current.tornFrom}`)
|
||||
}
|
||||
if (closers.length > 0) {
|
||||
const expected = current.preserved.at(-1)?.seq === undefined
|
||||
? 0
|
||||
: (current.preserved.at(-1) as SessionEvent).seq + 1
|
||||
if (closers[0]?.seq !== expected) {
|
||||
throw new Error(`session ${meta.id} repair is stale: closer starts at seq ${closers[0]?.seq}, stored next seq is ${expected}`)
|
||||
}
|
||||
const insert = this.insertStatement()
|
||||
for (const closer of closers) this.insertRecord(insert, meta.id, bindRecord(closer))
|
||||
}
|
||||
this.incrementRevision(meta.id)
|
||||
this.db.exec(sql('commit'))
|
||||
} catch (error: unknown) {
|
||||
this.rollback(error, 'repair')
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Atomically replace one exact stored revision with a complete current log.
|
||||
* The streamed events are staged in memory, then the swap commits in one
|
||||
* transaction that rechecks the revision and storage identity.
|
||||
* @param expectedRevision - exact source revision decoded by the caller.
|
||||
* @param meta - complete current-format header.
|
||||
* @param events - complete current-format event stream.
|
||||
*/
|
||||
async replaceStored(
|
||||
expectedRevision: PersistenceRevision,
|
||||
meta: SessionHeader,
|
||||
events: AsyncIterable<SessionEvent>,
|
||||
): Promise<void> {
|
||||
await this.open()
|
||||
const observed = this.rowFor(meta.id)
|
||||
if (observed === undefined
|
||||
|| sqliteRevision(this.storeIdentity, observed) !== expectedRevision) {
|
||||
throw new SessionPersistenceRevisionConflictError(
|
||||
`session "${meta.id}" changed before replacement of revision ${expectedRevision}`,
|
||||
)
|
||||
}
|
||||
if (meta.cwd !== (observed.cwd ?? undefined)) {
|
||||
throw new Error(`replacement for session "${meta.id}" changes its stored identity`)
|
||||
}
|
||||
// Stage the complete replacement before the swap transaction so a failed
|
||||
// or cancelled stream leaves the stored log untouched.
|
||||
const staged: SessionEvent[] = []
|
||||
for await (const event of events) staged.push(event)
|
||||
const records = packChunkRuns(staged)
|
||||
this.db.exec(sql('begin-immediate'))
|
||||
try {
|
||||
validateSchemaForMutation(this.databaseConstructor, this.db, this.databasePath)
|
||||
const row = this.rowFor(meta.id)
|
||||
if (row === undefined
|
||||
|| sqliteRevision(this.storeIdentity, row) !== expectedRevision) {
|
||||
throw new SessionPersistenceRevisionConflictError(
|
||||
`session "${meta.id}" changed before replacement of revision ${expectedRevision}`,
|
||||
)
|
||||
}
|
||||
if (meta.cwd !== (row.cwd ?? undefined)) {
|
||||
throw new Error(`replacement for session "${meta.id}" changes its stored identity`)
|
||||
}
|
||||
this.db.prepare(sql('delete-events-from')).run(meta.id, 0)
|
||||
const insert = this.insertStatement()
|
||||
for (const record of records) this.insertRecord(insert, meta.id, bindRecord(record))
|
||||
this.db.prepare(sql('upsert-session')).run(
|
||||
meta.id,
|
||||
meta.version,
|
||||
meta.createdAt,
|
||||
meta.cwd ?? null,
|
||||
meta.parentSession ?? null,
|
||||
meta.seedLength ?? null,
|
||||
meta.origin ?? null,
|
||||
meta.delegationDepth ?? null,
|
||||
meta.agentPreset ?? null,
|
||||
randomUUID(),
|
||||
)
|
||||
this.incrementRevision(meta.id)
|
||||
this.db.exec(sql('commit'))
|
||||
} catch (error: unknown) {
|
||||
this.rollback(error, 'replacement')
|
||||
}
|
||||
}
|
||||
|
||||
async list(signal?: AbortSignal): Promise<SessionHeader[]> {
|
||||
await this.observe(signal)
|
||||
const rows = this.sessionRows()
|
||||
signal?.throwIfAborted()
|
||||
return rows.map(row => decodeStoredSessionHeader(rowToMeta(row), SessionId(row.id)))
|
||||
}
|
||||
|
||||
/**
|
||||
* Return every materialized header with its source-qualified revision.
|
||||
* @param signal - optional cancellation before or after the metadata query.
|
||||
* @returns stored headers and revisions without loading event rows.
|
||||
*/
|
||||
async listSnapshots(signal?: AbortSignal): Promise<SessionPersistenceSnapshot[]> {
|
||||
await this.observe(signal)
|
||||
const rows = this.sessionRows()
|
||||
signal?.throwIfAborted()
|
||||
return rows.map(row => ({
|
||||
header: decodeStoredSessionHeader(rowToMeta(row), SessionId(row.id)),
|
||||
revision: sqliteRevision(this.storeIdentity, row),
|
||||
}))
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (this.ready === undefined) {
|
||||
if (this.pathReady !== undefined) await Promise.allSettled([this.pathReady])
|
||||
return
|
||||
}
|
||||
await Promise.allSettled([this.ready])
|
||||
if (!this.opened) return
|
||||
this.opened = false
|
||||
this.db.close()
|
||||
}
|
||||
|
||||
private rowFor(id: SessionId): SessionRow | undefined {
|
||||
const value = this.db.prepare(sql('select-session')).get(id)
|
||||
return value === undefined ? undefined : decodeSessionRow(value)
|
||||
}
|
||||
|
||||
private async observe(signal: AbortSignal | undefined): Promise<void> {
|
||||
signal?.throwIfAborted()
|
||||
await this.open()
|
||||
signal?.throwIfAborted()
|
||||
}
|
||||
|
||||
private readTransaction<T>(read: () => T): T {
|
||||
this.db.exec(sql('begin'))
|
||||
try {
|
||||
const value = read()
|
||||
this.db.exec(sql('commit'))
|
||||
return value
|
||||
} catch (error: unknown) {
|
||||
this.rollback(error, 'read')
|
||||
}
|
||||
}
|
||||
|
||||
private sessionRows(): SessionRow[] {
|
||||
return this.db.prepare(sql('select-sessions')).all().map(decodeSessionRow)
|
||||
}
|
||||
|
||||
private rollback(error: unknown, operation: string): never {
|
||||
try {
|
||||
this.db.exec(sql('rollback'))
|
||||
} catch (rollbackError: unknown) {
|
||||
/* v8 ignore next -- requires SQLite to fail both an operation and its immediate rollback. */
|
||||
throw new AggregateError([error, rollbackError], `${this.name} ${operation} failed and rollback also failed`)
|
||||
}
|
||||
throw error
|
||||
}
|
||||
|
||||
private incrementRevision(id: SessionId): void {
|
||||
const updated = this.db.prepare(sql('update-session-revision'))
|
||||
.run(id)
|
||||
/* v8 ignore next -- materialized writes follow coordinator create(); other writes upsert in this transaction. */
|
||||
if (Number(updated.changes) !== 1) throw new Error(`session ${id} metadata row is missing`)
|
||||
}
|
||||
|
||||
private tailRows(id: SessionId): EventRow[] {
|
||||
const tail = this.db.prepare(sql('select-tail-events')).all(id, 2).map(decodeEventRow).reverse()
|
||||
if (tail.length === 0) return []
|
||||
return this.physicalSpanFrom(id, (tail[0] as EventRow).seq).eventRows
|
||||
}
|
||||
|
||||
/** Select the bounded physical span that may represent `fromSeq`. */
|
||||
private physicalSpanFrom(
|
||||
id: SessionId,
|
||||
fromSeq: number,
|
||||
): { readonly base: number; readonly eventRows: EventRow[] } {
|
||||
const packedFloor = Math.max(0, fromSeq - MAX_PACKED_ROW_MEMBERS + 1)
|
||||
const packedPredecessors = this.db.prepare(sql('select-packed-predecessors'))
|
||||
.all(id, packedFloor, fromSeq)
|
||||
.map(decodeEventRow)
|
||||
let base = fromSeq
|
||||
for (const predecessor of packedPredecessors) {
|
||||
try {
|
||||
const last = decodeRow(predecessor).at(-1)
|
||||
if (last !== undefined && last.seq >= fromSeq) base = Math.min(base, predecessor.seq)
|
||||
} catch {
|
||||
// A malformed bounded predecessor may cover fromSeq; include it so the scanner fails closed.
|
||||
base = Math.min(base, predecessor.seq)
|
||||
}
|
||||
}
|
||||
const eventRows = this.db.prepare(sql('select-events-from')).all(id, base).map(decodeEventRow)
|
||||
return { base, eventRows }
|
||||
}
|
||||
|
||||
private logicalLastEvent(id: SessionId, tailRows: readonly EventRow[]): SessionEvent | undefined {
|
||||
if (tailRows.length === 0) return undefined
|
||||
const { preserved, tornFrom } = scanRows(tailRows, (tailRows[0] as EventRow).seq)
|
||||
if (tornFrom !== undefined) throw new Error(`session ${id} has an invalid physical tail at seq ${tornFrom}`)
|
||||
return preserved.at(-1)
|
||||
}
|
||||
|
||||
private insertStatement(): StatementSync {
|
||||
return this.db.prepare(sql('insert-event'))
|
||||
}
|
||||
|
||||
private insertRecord(insert: StatementSync, id: SessionId, record: BoundRecord): void {
|
||||
insert.run(
|
||||
id,
|
||||
record.seq,
|
||||
record.type,
|
||||
record.time,
|
||||
record.data,
|
||||
record.sourceEventSeqs,
|
||||
record.surfaceOp,
|
||||
record.ignorable,
|
||||
)
|
||||
}
|
||||
|
||||
private writeRow(meta: SessionHeader): void {
|
||||
this.db.prepare(sql('upsert-session')).run(
|
||||
meta.id,
|
||||
meta.version,
|
||||
meta.createdAt,
|
||||
meta.cwd ?? null,
|
||||
meta.parentSession ?? null,
|
||||
meta.seedLength ?? null,
|
||||
meta.origin ?? null,
|
||||
meta.delegationDepth ?? null,
|
||||
meta.agentPreset ?? null,
|
||||
randomUUID(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
function sqliteRevision(storeIdentity: string, row: SessionRow): PersistenceRevision {
|
||||
return SessionPersistenceRevision(
|
||||
`${storeIdentity}:incarnation:${row.incarnation}:revision:${row.revision}`,
|
||||
)
|
||||
}
|
||||
|
||||
async function createDatabaseFile(path: string): Promise<void> {
|
||||
try {
|
||||
const handle = await open(path, 'wx', 0o600)
|
||||
await handle.close()
|
||||
} catch (error: unknown) {
|
||||
if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error
|
||||
}
|
||||
}
|
||||
|
||||
async function validateParentDirectory(path: string): Promise<void> {
|
||||
const parent = await lstat(path)
|
||||
if (parent.isSymbolicLink() || !parent.isDirectory()) {
|
||||
throw new Error(`session database parent "${path}" must be a real directory`)
|
||||
}
|
||||
const uid = process.getuid?.()
|
||||
/* v8 ignore start -- Windows exposes neither process.getuid nor meaningful
|
||||
* uid/mode bits; POSIX tests cover owner and mode rejection. */
|
||||
if (uid !== undefined && (parent.uid !== uid || (parent.mode & 0o022) !== 0)) {
|
||||
throw new Error(`session database parent "${path}" must be owned by the current user and not group/world-writable`)
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
|
||||
async function validateDatabaseFile(path: string): Promise<void> {
|
||||
const file = await lstat(path)
|
||||
if (file.isSymbolicLink() || !file.isFile()) {
|
||||
throw new Error(`session database "${path}" must be a regular file, not a symbolic link`)
|
||||
}
|
||||
const uid = process.getuid?.()
|
||||
/* v8 ignore start -- Windows exposes neither process.getuid nor meaningful
|
||||
* uid/mode bits; POSIX tests cover owner and mode rejection. */
|
||||
if (uid !== undefined && (file.uid !== uid || (file.mode & 0o077) !== 0)) {
|
||||
throw new Error(`session database "${path}" must be owned by the current user and accessible only by that user`)
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
|
||||
async function validateDatabaseFileIfPresent(path: string): Promise<void> {
|
||||
try {
|
||||
await validateDatabaseFile(path)
|
||||
} catch (error: unknown) {
|
||||
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error
|
||||
}
|
||||
}
|
||||
|
||||
let nodeSqlite: Promise<typeof import('node:sqlite')> | undefined
|
||||
|
||||
/** Load Node SQLite once so concurrent stores share one warning-filter lifetime. */
|
||||
function loadNodeSqlite(): Promise<typeof import('node:sqlite')> {
|
||||
nodeSqlite ??= importNodeSqlite()
|
||||
return nodeSqlite
|
||||
}
|
||||
|
||||
/** Import Node 22's SQLite dependency without its process-wide experimental warning. */
|
||||
async function importNodeSqlite(): Promise<typeof import('node:sqlite')> {
|
||||
const emitWarning = Reflect.get(process, 'emitWarning')
|
||||
/* v8 ignore start -- Node 22 alone emits this warning; primary coverage runs on Node 24. */
|
||||
const filteredEmitWarning = (warning: string | Error, ...args: unknown[]): void => {
|
||||
const message = warning instanceof Error ? warning.message : warning
|
||||
const first = args[0]
|
||||
const type = warning instanceof Error
|
||||
? warning.name
|
||||
: typeof first === 'string'
|
||||
? first
|
||||
: typeof first === 'object' && first !== null && 'type' in first
|
||||
? first.type
|
||||
: undefined
|
||||
if (message === 'SQLite is an experimental feature and might change at any time'
|
||||
&& type === 'ExperimentalWarning') return
|
||||
Reflect.apply(emitWarning, process, [warning, ...args])
|
||||
}
|
||||
Reflect.set(process, 'emitWarning', filteredEmitWarning)
|
||||
try {
|
||||
return await import('node:sqlite')
|
||||
} finally {
|
||||
Reflect.set(process, 'emitWarning', emitWarning)
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
import { execFile } from 'node:child_process'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { promisify } from 'node:util'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
const repoRoot = fileURLToPath(new URL('../../../../', import.meta.url))
|
||||
const builtBundle = fileURLToPath(new URL('../lib/index.js', import.meta.url))
|
||||
const execFileAsync = promisify(execFile)
|
||||
|
||||
const probe = String.raw`
|
||||
import { resolve } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
const load = path => import(pathToFileURL(resolve(path)).href);
|
||||
const [{ Context }, { default: SessionStore }, { default: Sqlite }] = await Promise.all([
|
||||
load('vendor/cordis/lib/index.js'),
|
||||
load('packages/core/session/lib/index.js'),
|
||||
load('packages/session/session-persistence-sqlite/lib/index.js'),
|
||||
]);
|
||||
const ctx = new Context();
|
||||
await ctx.plugin(SessionStore);
|
||||
await ctx.plugin(Sqlite, { path: ':memory:' });
|
||||
console.log(JSON.stringify(await ctx.sessionPersistence.list()));
|
||||
await ctx.fiber.dispose();
|
||||
`
|
||||
|
||||
describe.skipIf(!existsSync(builtBundle))('SQLite built package', () => {
|
||||
it('loads packaged SQL resources from the published entry', async () => {
|
||||
const { stdout, stderr } = await execFileAsync(process.execPath, ['--input-type=module', '-e', probe], {
|
||||
cwd: repoRoot,
|
||||
timeout: 15_000,
|
||||
})
|
||||
expect(stderr).toBe('')
|
||||
expect(JSON.parse(stdout) as unknown).toEqual([])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,25 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
|
||||
vi.mock('node:zlib', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('node:zlib')>()
|
||||
return {
|
||||
...actual,
|
||||
zstdCompressSync: (input: ArrayBufferView) => Buffer.alloc(input.byteLength + 1),
|
||||
}
|
||||
})
|
||||
|
||||
import { bindRecord, ZSTD_DATA_THRESHOLD_BYTES } from '../src/compression.ts'
|
||||
|
||||
describe('SQLite compression fallback', () => {
|
||||
it('keeps large data as text when its Zstandard frame is not smaller', () => {
|
||||
const event = {
|
||||
type: 'assistant/message',
|
||||
seq: 0,
|
||||
time: 1,
|
||||
data: { text: 'x'.repeat(ZSTD_DATA_THRESHOLD_BYTES) },
|
||||
} as unknown as SessionEvent
|
||||
|
||||
expect(typeof bindRecord(event).data).toBe('string')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,357 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { zstdCompressSync } from 'node:zlib'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import { CallId, type StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import {
|
||||
decodeStorageRecord,
|
||||
MAX_PACKED_DATA_BYTES,
|
||||
MAX_PACKED_ROW_MEMBERS,
|
||||
packChunkRuns,
|
||||
type StorageRecord,
|
||||
} from '../src/codec.ts'
|
||||
import {
|
||||
bindRecord,
|
||||
decodeRow,
|
||||
scanRows,
|
||||
ZSTD_DATA_THRESHOLD_BYTES,
|
||||
} from '../src/compression.ts'
|
||||
import type { EventRow } from '../src/schema.ts'
|
||||
|
||||
function chunk(seq: number, text = `token-${seq}`): SessionEvent {
|
||||
return {
|
||||
type: 'assistant/chunk',
|
||||
seq,
|
||||
time: 1_000 + seq,
|
||||
data: {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
chunk: { type: 'text-delta', index: 0, text },
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function event(seq: number, time: number, value: StreamChunk, turn = 1, step = 1): SessionEvent {
|
||||
return { type: 'assistant/chunk', seq, time, data: { turn, step, chunk: value } }
|
||||
}
|
||||
|
||||
function row(record: StorageRecord): EventRow {
|
||||
const bound = bindRecord(record)
|
||||
return {
|
||||
seq: bound.seq,
|
||||
type: bound.type,
|
||||
time: bound.time,
|
||||
data: bound.data,
|
||||
source_event_seqs: bound.sourceEventSeqs,
|
||||
surface_op: bound.surfaceOp,
|
||||
ignorable: bound.ignorable,
|
||||
}
|
||||
}
|
||||
|
||||
describe('SQLite compression', () => {
|
||||
it('stores a 100-member run in one row and restores every logical event', () => {
|
||||
const events = Array.from({ length: 100 }, (_, index) => chunk(index))
|
||||
const records = packChunkRuns(events)
|
||||
expect(records).toHaveLength(1)
|
||||
expect(records[0]?.type).toBe('text-chunks')
|
||||
expect(scanRows(records.map(row)).preserved).toEqual(events)
|
||||
})
|
||||
|
||||
it('partitions long and large runs within schema-owned row limits', () => {
|
||||
const long = Array.from({ length: MAX_PACKED_ROW_MEMBERS + 3 }, (_, index) => chunk(index))
|
||||
const longRecords = packChunkRuns(long)
|
||||
expect(longRecords).toHaveLength(2)
|
||||
expect(scanRows(longRecords.map(row)).preserved).toEqual(long)
|
||||
|
||||
const large = Array.from({ length: 4 }, (_, index) => chunk(index, 'x'.repeat(300_000)))
|
||||
const largeRecords = packChunkRuns(large)
|
||||
expect(largeRecords).toHaveLength(2)
|
||||
for (const record of largeRecords) {
|
||||
if (record.type.endsWith('-chunks')) {
|
||||
expect(Buffer.byteLength(JSON.stringify(record.data))).toBeLessThanOrEqual(MAX_PACKED_DATA_BYTES)
|
||||
}
|
||||
}
|
||||
expect(scanRows(largeRecords.map(row)).preserved).toEqual(large)
|
||||
|
||||
const individuallyLarge = Array.from({ length: 3 }, (_, index) => chunk(index, 'x'.repeat(400_000)))
|
||||
expect(packChunkRuns(individuallyLarge)).toEqual(individuallyLarge)
|
||||
|
||||
const byteBound = Array.from({ length: 10 }, (_, index) => chunk(index, 'x'.repeat(150_000)))
|
||||
const byteBoundRecords = packChunkRuns(byteBound)
|
||||
expect(byteBoundRecords.length).toBeGreaterThan(1)
|
||||
expect(scanRows(byteBoundRecords.map(row)).preserved).toEqual(byteBound)
|
||||
})
|
||||
|
||||
it('packs every owned kind and preserves optional tool-call names', () => {
|
||||
const events = [
|
||||
...[0, 1, 2].map(seq => event(seq, seq, { type: 'reasoning-delta', index: 1, text: `${seq}` })),
|
||||
...[3, 4, 5].map(seq => event(seq, seq, {
|
||||
type: 'tool-call-delta', index: 2, id: CallId('named'), name: 'write', argumentsDelta: `${seq}`,
|
||||
})),
|
||||
...[6, 7, 8].map(seq => event(seq, seq, {
|
||||
type: 'tool-call-delta', index: 3, id: CallId('unnamed'), argumentsDelta: `${seq}`,
|
||||
})),
|
||||
]
|
||||
const records = packChunkRuns(events)
|
||||
expect(records.map(record => record.type)).toEqual([
|
||||
'reasoning-chunks', 'tool-call-chunks', 'tool-call-chunks',
|
||||
])
|
||||
expect(records.flatMap(decodeStorageRecord)).toEqual(events)
|
||||
})
|
||||
|
||||
it('keeps every off-format delta scalar and splits incompatible runs', () => {
|
||||
const malformed = (seq: number, data: unknown): SessionEvent => ({
|
||||
type: 'assistant/chunk', seq, time: 10 + seq, data,
|
||||
} as SessionEvent)
|
||||
const values: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
{ ...chunk(1), extra: true } as unknown as SessionEvent,
|
||||
{ ...chunk(-1), seq: -1 },
|
||||
{ ...chunk(3), time: 1.5 },
|
||||
malformed(4, 'data'),
|
||||
malformed(5, { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'x' }, extra: 1 }),
|
||||
malformed(6, { turn: '1', step: 1, chunk: { type: 'text-delta', index: 0, text: 'x' } }),
|
||||
malformed(7, { turn: 1, step: 1, chunk: 'chunk' }),
|
||||
malformed(8, { turn: 1, step: 1, chunk: { type: 'text-delta', index: '0', text: 'x' } }),
|
||||
malformed(9, { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 1 } }),
|
||||
malformed(10, { turn: 1, step: 1, chunk: { type: 'tool-call-delta', index: 0, id: 1, argumentsDelta: 'x' } }),
|
||||
malformed(11, { turn: 1, step: 1, chunk: { type: 'tool-call-delta', index: 0, id: 'id', name: 1, argumentsDelta: 'x' } }),
|
||||
malformed(12, { turn: 1, step: 1, chunk: { type: 'usage', index: 0, totalTokens: 1 } }),
|
||||
]
|
||||
expect(packChunkRuns(values)).toEqual(values)
|
||||
|
||||
const gap = [chunk(0), chunk(1), chunk(3)]
|
||||
const step = [chunk(0), chunk(1), event(2, 2, { type: 'text-delta', index: 0, text: 'x' }, 1, 2)]
|
||||
const block = [chunk(0), chunk(1), event(2, 2, { type: 'text-delta', index: 1, text: 'x' })]
|
||||
const unsafeTime = [
|
||||
event(0, Number.MIN_SAFE_INTEGER, { type: 'text-delta', index: 0, text: 'a' }),
|
||||
event(1, Number.MAX_SAFE_INTEGER, { type: 'text-delta', index: 0, text: 'b' }),
|
||||
event(2, Number.MAX_SAFE_INTEGER, { type: 'text-delta', index: 0, text: 'c' }),
|
||||
]
|
||||
const toolName = [0, 1, 2].map(seq => event(seq, seq, {
|
||||
type: 'tool-call-delta', index: 0, id: CallId('id'),
|
||||
...seq === 2 ? {} : { name: 'write' }, argumentsDelta: 'x',
|
||||
}))
|
||||
for (const events of [gap, step, block, unsafeTime, toolName]) {
|
||||
expect(packChunkRuns(events)).toEqual(events)
|
||||
}
|
||||
})
|
||||
|
||||
it.each([
|
||||
['extra envelope field', { type: 'text-chunks', seq0: 0, time0: 1, data: {}, extra: true }],
|
||||
['negative sequence', { type: 'text-chunks', seq0: -1, time0: 1, data: {} }],
|
||||
['fractional time', { type: 'text-chunks', seq0: 0, time0: 1.5, data: {} }],
|
||||
['primitive data', { type: 'text-chunks', seq0: 0, time0: 1, data: 'bad' }],
|
||||
['text fields', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: [], args: [] } }],
|
||||
['non-numeric placement', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: '1', step: 1, index: 0, dt: [0, 0], texts: ['a', 'b', 'c'] } }],
|
||||
['non-array members', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: [0, 0], texts: 'abc' } }],
|
||||
['too few members', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: [0], texts: ['a', 'b'] } }],
|
||||
['too many members', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: Array(1_024).fill(0), texts: Array(1_025).fill('a') } }],
|
||||
['non-string member', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: [0, 0], texts: ['a', 1, 'c'] } }],
|
||||
['invalid gaps', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: [0, 0.5], texts: ['a', 'b', 'c'] } }],
|
||||
['non-array gaps', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: '00', texts: ['a', 'b', 'c'] } }],
|
||||
['gap arity', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: [0], texts: ['a', 'b', 'c'] } }],
|
||||
['oversized data', { type: 'text-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: [0, 0], texts: ['x'.repeat(400_000), 'x'.repeat(400_000), 'x'.repeat(400_000)] } }],
|
||||
['sequence overflow', { type: 'text-chunks', seq0: Number.MAX_SAFE_INTEGER, time0: 1, data: { turn: 1, step: 1, index: 0, dt: [0, 0], texts: ['a', 'b', 'c'] } }],
|
||||
['time overflow', { type: 'text-chunks', seq0: 0, time0: Number.MAX_SAFE_INTEGER, data: { turn: 1, step: 1, index: 0, dt: [1, 0], texts: ['a', 'b', 'c'] } }],
|
||||
['tool fields', { type: 'tool-call-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, dt: [0, 0], args: ['a', 'b', 'c'] } }],
|
||||
['tool id', { type: 'tool-call-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, id: 1, dt: [0, 0], args: ['a', 'b', 'c'] } }],
|
||||
['tool name', { type: 'tool-call-chunks', seq0: 0, time0: 1, data: { turn: 1, step: 1, index: 0, id: 'id', name: 1, dt: [0, 0], args: ['a', 'b', 'c'] } }],
|
||||
])('rejects malformed packed data: %s', (_label, record) => {
|
||||
expect(() => decodeStorageRecord(record)).toThrow(/malformed .* storage row/)
|
||||
})
|
||||
|
||||
it('decodes the schema-17 row vocabulary without another package codec', () => {
|
||||
const fixture: EventRow = {
|
||||
seq: 7,
|
||||
type: 'text-chunks',
|
||||
time: 90,
|
||||
data: JSON.stringify({ turn: 2, step: 3, index: 1, dt: [2, -1], texts: ['a', 'b', 'c'] }),
|
||||
source_event_seqs: null,
|
||||
surface_op: null,
|
||||
ignorable: 0,
|
||||
}
|
||||
expect(decodeRow(fixture)).toEqual([
|
||||
{ ...chunk(7, 'a'), time: 90, data: { turn: 2, step: 3, chunk: { type: 'text-delta', index: 1, text: 'a' } } },
|
||||
{ ...chunk(8, 'b'), time: 92, data: { turn: 2, step: 3, chunk: { type: 'text-delta', index: 1, text: 'b' } } },
|
||||
{ ...chunk(9, 'c'), time: 91, data: { turn: 2, step: 3, chunk: { type: 'text-delta', index: 1, text: 'c' } } },
|
||||
])
|
||||
expect(decodeStorageRecord('scalar')).toEqual(['scalar'])
|
||||
expect(decodeStorageRecord(chunk(0))).toEqual([chunk(0)])
|
||||
})
|
||||
|
||||
it('rejects surface columns on packed rows', () => {
|
||||
const packed = row(packChunkRuns([chunk(0), chunk(1), chunk(2)])[0]!)
|
||||
const invalid: EventRow[] = [
|
||||
{ ...packed, source_event_seqs: Buffer.alloc(0) },
|
||||
{ ...packed, surface_op: '"append"' },
|
||||
]
|
||||
for (const candidate of invalid) {
|
||||
expect(() => decodeRow(candidate)).toThrow(/surface fields must be null/)
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects the packed discriminator on a scalar event type', () => {
|
||||
const scalar = row({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } })
|
||||
expect(() => decodeRow({ ...scalar, ignorable: 0 }))
|
||||
.toThrow(/packed discriminator requires a chunk tag/)
|
||||
})
|
||||
|
||||
it.each(['text-chunks', 'reasoning-chunks', 'tool-call-chunks'])(
|
||||
'preserves an ignorable logical event named %s as a scalar row',
|
||||
(type) => {
|
||||
const logical = {
|
||||
type,
|
||||
seq: 0,
|
||||
time: 1,
|
||||
data: { future: true },
|
||||
ignorable: true,
|
||||
} as unknown as SessionEvent
|
||||
const physical = row(logical)
|
||||
expect(physical.ignorable).toBe(1)
|
||||
expect(decodeRow(physical)).toEqual([logical])
|
||||
},
|
||||
)
|
||||
|
||||
it('compresses large data and delta-encodes complete provenance arrays', () => {
|
||||
const sources = Array.from({ length: 2_000 }, (_, index) => index + 10)
|
||||
const event = {
|
||||
type: 'assistant/message',
|
||||
seq: sources.at(-1)! + 1,
|
||||
time: 1,
|
||||
data: { text: 'x'.repeat(ZSTD_DATA_THRESHOLD_BYTES * 2) },
|
||||
sourceEventSeqs: sources,
|
||||
surfaceOp: 'append',
|
||||
} as unknown as SessionEvent
|
||||
const bound = bindRecord(event)
|
||||
expect(bound.data).toBeInstanceOf(Uint8Array)
|
||||
expect(bound.sourceEventSeqs).toBeInstanceOf(Uint8Array)
|
||||
expect(bound.sourceEventSeqs?.byteLength).toBeLessThan(Buffer.byteLength(JSON.stringify(sources)))
|
||||
expect(decodeRow(row(event))).toEqual([event])
|
||||
|
||||
const small = bindRecord({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } })
|
||||
expect(typeof small.data).toBe('string')
|
||||
})
|
||||
|
||||
it('round-trips empty, descending, and maximum-safe provenance deltas', () => {
|
||||
for (const sources of [
|
||||
[],
|
||||
[Number.MAX_SAFE_INTEGER - 1, 0, Number.MAX_SAFE_INTEGER - 2],
|
||||
]) {
|
||||
const event = {
|
||||
type: 'assistant/message',
|
||||
seq: Number.MAX_SAFE_INTEGER,
|
||||
time: 1,
|
||||
data: {},
|
||||
sourceEventSeqs: sources,
|
||||
surfaceOp: 'append',
|
||||
} as unknown as SessionEvent
|
||||
expect(decodeRow(row(event))).toEqual([event])
|
||||
}
|
||||
})
|
||||
|
||||
it.each([-1, 0.5])('rejects invalid provenance sequence %s before encoding', (sourceSeq) => {
|
||||
const event = {
|
||||
type: 'assistant/message',
|
||||
seq: 1,
|
||||
time: 1,
|
||||
data: {},
|
||||
sourceEventSeqs: [sourceSeq],
|
||||
surfaceOp: 'append',
|
||||
} as unknown as SessionEvent
|
||||
expect(() => bindRecord(event)).toThrow(/non-negative safe integers/)
|
||||
})
|
||||
|
||||
it('rejects malformed compressed and delta-encoded values', () => {
|
||||
const scalar = row({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } })
|
||||
expect(() => decodeRow({ ...scalar, data: Buffer.from('not zstd') })).toThrow()
|
||||
expect(() => decodeRow({ ...scalar, source_event_seqs: Buffer.from([0x80]) }))
|
||||
.toThrow(/truncated varint/)
|
||||
expect(() => decodeRow({ ...scalar, source_event_seqs: Buffer.from([0x80, 0x00]) }))
|
||||
.toThrow(/non-canonical varint/)
|
||||
expect(() => decodeRow({ ...scalar, source_event_seqs: Buffer.from([0x00, 0x01]) }))
|
||||
.toThrow(/decoded seq is out of range/)
|
||||
expect(() => decodeRow({ ...scalar, source_event_seqs: Buffer.from([
|
||||
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x0f, 0x02,
|
||||
]) })).toThrow(/decoded seq is out of range/)
|
||||
expect(() => decodeRow({ ...scalar, source_event_seqs: Buffer.from([
|
||||
0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x10,
|
||||
]) })).toThrow(/varint is out of range/)
|
||||
expect(() => decodeRow({ ...scalar, source_event_seqs: Buffer.alloc(9, 0x80) }))
|
||||
.toThrow(/varint is out of range/)
|
||||
})
|
||||
|
||||
it('rejects an oversized packed data column before JSON decoding', () => {
|
||||
const oversized: EventRow = {
|
||||
seq: 0,
|
||||
type: 'text-chunks',
|
||||
time: 1,
|
||||
data: ' '.repeat(MAX_PACKED_DATA_BYTES + 1),
|
||||
source_event_seqs: null,
|
||||
surface_op: null,
|
||||
ignorable: 0,
|
||||
}
|
||||
expect(() => decodeRow(oversized)).toThrow(/data exceeds/)
|
||||
})
|
||||
|
||||
it('bounds packed data while decompressing', () => {
|
||||
const serialized = JSON.stringify({
|
||||
turn: 1,
|
||||
step: 1,
|
||||
index: 0,
|
||||
dt: [0, 0],
|
||||
texts: ['x'.repeat(MAX_PACKED_DATA_BYTES), 'b', 'c'],
|
||||
})
|
||||
const oversized: EventRow = {
|
||||
seq: 0,
|
||||
type: 'text-chunks',
|
||||
time: 1,
|
||||
data: zstdCompressSync(serialized),
|
||||
source_event_seqs: null,
|
||||
surface_op: null,
|
||||
ignorable: 0,
|
||||
}
|
||||
expect(() => decodeRow(oversized)).toThrow(/Buffer larger than/)
|
||||
})
|
||||
|
||||
it('distinguishes removable and committed physical corruption', () => {
|
||||
const start = row({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } })
|
||||
const skipped = row({ type: 'step/start', seq: 2, time: 2, data: { turn: 1, step: 1 } })
|
||||
expect(scanRows([start, skipped])).toEqual({ preserved: [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
], tornFrom: 2 })
|
||||
|
||||
const end = row({
|
||||
type: 'turn/end',
|
||||
seq: 3,
|
||||
time: 3,
|
||||
data: { turn: 1, reason: { kind: 'completed' } },
|
||||
})
|
||||
expect(() => scanRows([start, skipped, end])).toThrow(/invalid committed physical row at seq 2/)
|
||||
|
||||
const malformed = {
|
||||
...row(packChunkRuns([chunk(0), chunk(1), chunk(2)])[0]!),
|
||||
data: '{not json',
|
||||
}
|
||||
const committedEnd = row({
|
||||
type: 'turn/end',
|
||||
seq: 1,
|
||||
time: 4,
|
||||
data: { turn: 1, reason: { kind: 'completed' } },
|
||||
})
|
||||
expect(() => scanRows([malformed, committedEnd]))
|
||||
.toThrow(/invalid committed physical row at seq 0/)
|
||||
})
|
||||
|
||||
it('treats a malformed packed tail as one removable physical row', () => {
|
||||
const malformed: EventRow = {
|
||||
seq: 0,
|
||||
type: 'text-chunks',
|
||||
time: 1,
|
||||
data: JSON.stringify({ turn: 1, step: 1, index: 0, dt: [], texts: ['a', 'b'] }),
|
||||
source_event_seqs: null,
|
||||
surface_op: null,
|
||||
ignorable: 0,
|
||||
}
|
||||
expect(scanRows([malformed])).toEqual({ preserved: [], tornFrom: 0 })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,273 @@
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import fc from 'fast-check'
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import { mkdtemp, rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import { CallId, type StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { type SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionPersistence } from '@deepseek-ai/dsh-session-persistence'
|
||||
import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
import SessionPersistenceSqlite from '@deepseek-ai/dsh-session-persistence-sqlite'
|
||||
import { meta } from '../../session-persistence/tests/contract.ts'
|
||||
import { testSql } from './test-sql.ts'
|
||||
|
||||
type BackendName = 'jsonl-zstd' | 'sqlite'
|
||||
|
||||
interface MountedBackend {
|
||||
readonly persistence: SessionPersistence
|
||||
dispose(): Promise<void>
|
||||
}
|
||||
|
||||
const directories: string[] = []
|
||||
afterEach(async () => {
|
||||
for (const directory of directories.splice(0)) {
|
||||
await rm(directory, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
async function freshDirectory(prefix: string): Promise<string> {
|
||||
const directory = await mkdtemp(join(tmpdir(), prefix))
|
||||
directories.push(directory)
|
||||
return directory
|
||||
}
|
||||
|
||||
async function mount(name: BackendName, root: string): Promise<MountedBackend> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
switch (name) {
|
||||
case 'jsonl-zstd': {
|
||||
const fiber = await ctx.plugin(SessionPersistenceJsonl, { root: join(root, 'jsonl') })
|
||||
return { persistence: ctx.sessionPersistence, dispose: async () => { await fiber.dispose() } }
|
||||
}
|
||||
case 'sqlite': {
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: join(root, 'sessions.db') })
|
||||
return { persistence: ctx.sessionPersistence, dispose: async () => { await fiber.dispose() } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function closedChunkLog(
|
||||
entries: readonly { readonly chunk: StreamChunk; readonly time: number; readonly ignorable?: true }[],
|
||||
): SessionEvent[] {
|
||||
const chunks = entries.map(({ chunk, time, ignorable }, index): SessionEvent => ({
|
||||
type: 'assistant/chunk',
|
||||
seq: index + 2,
|
||||
time,
|
||||
data: { turn: 1, step: 1, chunk },
|
||||
...ignorable === true ? { ignorable } : {},
|
||||
}))
|
||||
return [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
{ type: 'step/start', seq: 1, time: 2, data: { turn: 1, step: 1 } },
|
||||
...chunks,
|
||||
{ type: 'step/end', seq: chunks.length + 2, time: 3, data: { turn: 1, step: 1 } },
|
||||
{
|
||||
type: 'turn/end',
|
||||
seq: chunks.length + 3,
|
||||
time: 4,
|
||||
data: { turn: 1, reason: { kind: 'completed' } },
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
function packingMatrixLog(): SessionEvent[] {
|
||||
const entries: { chunk: StreamChunk; time: number; ignorable?: true }[] = [
|
||||
...Array.from({ length: 5 }, (_, index) => ({
|
||||
chunk: { type: 'text-delta' as const, index: 0, text: `text-${index}` },
|
||||
time: 1_000 + index,
|
||||
})),
|
||||
...Array.from({ length: 4 }, (_, index) => ({
|
||||
chunk: { type: 'reasoning-delta' as const, index: 1, text: `reason-${index}` },
|
||||
time: 990 - index,
|
||||
})),
|
||||
...Array.from({ length: 4 }, (_, index) => ({
|
||||
chunk: {
|
||||
type: 'tool-call-delta' as const,
|
||||
index: 2,
|
||||
id: CallId('named-call'),
|
||||
name: 'write',
|
||||
argumentsDelta: `{${index}`,
|
||||
},
|
||||
time: 2_000 + index,
|
||||
})),
|
||||
...Array.from({ length: 3 }, (_, index) => ({
|
||||
chunk: {
|
||||
type: 'tool-call-delta' as const,
|
||||
index: 3,
|
||||
id: CallId('unnamed-call'),
|
||||
argumentsDelta: `${index}}`,
|
||||
},
|
||||
time: 3_000 + index,
|
||||
})),
|
||||
{ chunk: { type: 'block-start', index: 4, blockType: 'text' }, time: 4_000 },
|
||||
{ chunk: { type: 'text-delta', index: 4, text: 'short-a' }, time: 4_001 },
|
||||
{ chunk: { type: 'text-delta', index: 4, text: 'short-b' }, time: 4_002 },
|
||||
{ chunk: { type: 'text-delta', index: 5, text: 'scalar-envelope' }, time: 4_003, ignorable: true },
|
||||
{ chunk: { type: 'finish', reason: { kind: 'stop' } }, time: 4_004 },
|
||||
]
|
||||
return closedChunkLog(entries)
|
||||
}
|
||||
|
||||
function storageTagCollisionLog(): SessionEvent[] {
|
||||
return [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
...['text-chunks', 'reasoning-chunks', 'tool-call-chunks'].map((type, index) => ({
|
||||
type,
|
||||
seq: index + 1,
|
||||
time: index + 2,
|
||||
data: { future: true },
|
||||
ignorable: true as const,
|
||||
}) as unknown as SessionEvent),
|
||||
{ type: 'turn/end', seq: 4, time: 5, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
]
|
||||
}
|
||||
|
||||
function batches(events: readonly SessionEvent[], sizes: readonly number[]): SessionEvent[][] {
|
||||
const result: SessionEvent[][] = []
|
||||
let offset = 0
|
||||
let index = 0
|
||||
while (offset < events.length) {
|
||||
const size = sizes[index % sizes.length] as number
|
||||
result.push(events.slice(offset, offset + size))
|
||||
offset += size
|
||||
index += 1
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
async function verifyBackend(
|
||||
name: BackendName,
|
||||
root: string,
|
||||
events: readonly SessionEvent[],
|
||||
sizes: readonly number[],
|
||||
): Promise<void> {
|
||||
const header = { ...meta('differential', '/work'), delegationDepth: 0 }
|
||||
let mounted = await mount(name, root)
|
||||
try {
|
||||
await mounted.persistence.create(header)
|
||||
for (const batch of batches(events, sizes)) {
|
||||
await mounted.persistence.append(header.id, batch)
|
||||
}
|
||||
expect(await mounted.persistence.inspect(header.id), name).toEqual({ meta: header, events })
|
||||
expect(await mounted.persistence.list(), name).toEqual([header])
|
||||
const revision = (await mounted.persistence.listSnapshots())[0]?.revision
|
||||
for (let fromSeq = 0; fromSeq <= events.length + 1; fromSeq += 1) {
|
||||
expect((await mounted.persistence.readFrom(header.id, fromSeq)).events, `${name} seq ${fromSeq}`)
|
||||
.toEqual(events.slice(fromSeq))
|
||||
}
|
||||
expect((await mounted.persistence.listSnapshots())[0]?.revision, name).toBe(revision)
|
||||
} finally {
|
||||
await mounted.dispose()
|
||||
}
|
||||
|
||||
mounted = await mount(name, root)
|
||||
try {
|
||||
expect(await mounted.persistence.inspect(header.id), `${name} reopen`).toEqual({ meta: header, events })
|
||||
} finally {
|
||||
await mounted.dispose()
|
||||
}
|
||||
}
|
||||
|
||||
const streamChunkArbitrary: fc.Arbitrary<StreamChunk> = fc.oneof(
|
||||
fc.record({ type: fc.constant<'text-delta'>('text-delta'), index: fc.nat(2), text: fc.string() }),
|
||||
fc.record({ type: fc.constant<'reasoning-delta'>('reasoning-delta'), index: fc.nat(2), text: fc.string() }),
|
||||
fc.record({
|
||||
type: fc.constant<'tool-call-delta'>('tool-call-delta'),
|
||||
index: fc.nat(2),
|
||||
id: fc.constantFrom(CallId('call-1'), CallId('call-2')),
|
||||
argumentsDelta: fc.string(),
|
||||
}),
|
||||
fc.record({
|
||||
type: fc.constant<'tool-call-delta'>('tool-call-delta'),
|
||||
index: fc.nat(2),
|
||||
id: fc.constantFrom(CallId('call-1'), CallId('call-2')),
|
||||
name: fc.constantFrom('read', 'write'),
|
||||
argumentsDelta: fc.string(),
|
||||
}),
|
||||
fc.record({
|
||||
type: fc.constant<'block-start'>('block-start'),
|
||||
index: fc.nat(2),
|
||||
blockType: fc.constant<'text'>('text'),
|
||||
}),
|
||||
fc.record({ type: fc.constant<'finish'>('finish'), reason: fc.constant({ kind: 'stop' as const }) }),
|
||||
)
|
||||
|
||||
const randomWorkload = fc.record({
|
||||
entries: fc.array(fc.record({
|
||||
chunk: streamChunkArbitrary,
|
||||
time: fc.oneof(
|
||||
{ weight: 4, arbitrary: fc.integer({ min: 0, max: 10_000 }) },
|
||||
{ weight: 1, arbitrary: fc.integer({ min: Number.MIN_SAFE_INTEGER, max: Number.MAX_SAFE_INTEGER }) },
|
||||
),
|
||||
ignorable: fc.option(fc.constant<true>(true), { nil: undefined }),
|
||||
}), { maxLength: 30 }),
|
||||
batchSizes: fc.array(fc.integer({ min: 1, max: 8 }), { minLength: 1, maxLength: 8 }),
|
||||
}).map(({ entries, batchSizes }) => ({
|
||||
events: JSON.parse(JSON.stringify(closedChunkLog(entries.map(({ chunk, time, ignorable }) => ({
|
||||
chunk,
|
||||
time,
|
||||
...ignorable === true ? { ignorable } : {},
|
||||
}))))) as SessionEvent[],
|
||||
batchSizes,
|
||||
}))
|
||||
|
||||
describe('SQLite cross-backend differential behavior', () => {
|
||||
it('preserves ignorable logical events whose names match physical storage tags', async () => {
|
||||
const events = storageTagCollisionLog()
|
||||
const directory = await freshDirectory('dsh-sqlite-storage-tag-collision-')
|
||||
const root = join(directory, 'sqlite')
|
||||
await verifyBackend('sqlite', root, events, [2, 1])
|
||||
const db = new DatabaseSync(join(root, 'sessions.db'), { readOnly: true })
|
||||
try {
|
||||
expect(db.prepare(testSql('count-physical-types')).all()).toEqual([])
|
||||
expect(db.prepare(testSql('count-ignorable-events')).get()).toEqual({ count: 3 })
|
||||
} finally {
|
||||
db.close()
|
||||
}
|
||||
})
|
||||
|
||||
it('matches JSONL/Zstandard for every packed kind, scalar fallback, suffix, partition, and reopen', async () => {
|
||||
const events = packingMatrixLog()
|
||||
for (const [partitionIndex, sizes] of [[events.length], [1], [2, 1, 5, 3]].entries()) {
|
||||
const directory = await freshDirectory(`dsh-sqlite-matrix-${partitionIndex}-`)
|
||||
for (const name of ['jsonl-zstd', 'sqlite'] as const) {
|
||||
const root = join(directory, name)
|
||||
await verifyBackend(name, root, events, sizes)
|
||||
if (name === 'sqlite') {
|
||||
const db = new DatabaseSync(join(root, 'sessions.db'), { readOnly: true })
|
||||
try {
|
||||
expect(db.prepare(testSql('count-physical-types')).all()).toEqual([
|
||||
[
|
||||
{ type: 'reasoning-chunks', count: 1 },
|
||||
{ type: 'text-chunks', count: 1 },
|
||||
{ type: 'tool-call-chunks', count: 2 },
|
||||
],
|
||||
[],
|
||||
[
|
||||
{ type: 'reasoning-chunks', count: 1 },
|
||||
{ type: 'text-chunks', count: 1 },
|
||||
{ type: 'tool-call-chunks', count: 1 },
|
||||
],
|
||||
][partitionIndex])
|
||||
expect(db.prepare(testSql('count-ignorable-events')).get())
|
||||
.toEqual({ count: 1 })
|
||||
} finally {
|
||||
db.close()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}, 30_000)
|
||||
|
||||
it('matches JSONL/Zstandard across randomized logical logs and append partitions', async () => {
|
||||
await fc.assert(fc.asyncProperty(randomWorkload, async ({ events, batchSizes }) => {
|
||||
const directory = await freshDirectory('dsh-sqlite-property-')
|
||||
for (const name of ['jsonl-zstd', 'sqlite'] as const) {
|
||||
await verifyBackend(name, join(directory, name), events, batchSizes)
|
||||
}
|
||||
}), { numRuns: 100, seed: 0x5A17E })
|
||||
}, 60_000)
|
||||
|
||||
})
|
||||
+1
@@ -0,0 +1 @@
|
||||
ALTER TABLE events ADD COLUMN unexpected TEXT;
|
||||
@@ -0,0 +1,2 @@
|
||||
SELECT COUNT(*) AS count
|
||||
FROM events;
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
SELECT COUNT(*) AS count
|
||||
FROM events
|
||||
WHERE ignorable = 1;
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
SELECT COUNT(*) AS count
|
||||
FROM events
|
||||
WHERE type = 'text-chunks' AND ignorable = 0;
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
SELECT type, COUNT(*) AS count
|
||||
FROM events
|
||||
WHERE type IN ('text-chunks', 'reasoning-chunks', 'tool-call-chunks')
|
||||
AND ignorable = 0
|
||||
GROUP BY type
|
||||
ORDER BY type;
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
CREATE TABLE persistence_state (singleton ANY, store_id ANY);
|
||||
CREATE TABLE sessions (
|
||||
id ANY, version ANY, created_at ANY, cwd ANY, parent_session ANY,
|
||||
seed_length ANY, origin ANY, delegation_depth ANY, agent_preset ANY,
|
||||
incarnation ANY, revision ANY
|
||||
);
|
||||
CREATE TABLE events (
|
||||
session_id ANY, seq ANY, type ANY, time ANY, data ANY,
|
||||
source_event_seqs ANY, surface_op ANY, ignorable ANY
|
||||
);
|
||||
INSERT INTO persistence_state (singleton, store_id)
|
||||
VALUES (1, '00000000-0000-4000-8000-000000000000');
|
||||
PRAGMA application_id = 1146308688;
|
||||
PRAGMA user_version = 17;
|
||||
+1
@@ -0,0 +1 @@
|
||||
CREATE TABLE unrelated (value TEXT);
|
||||
+1
@@ -0,0 +1 @@
|
||||
DELETE FROM persistence_state;
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
DELETE FROM events
|
||||
WHERE session_id = ?;
|
||||
@@ -0,0 +1,3 @@
|
||||
UPDATE persistence_state
|
||||
SET store_id = ''
|
||||
WHERE singleton = 1;
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
INSERT INTO events (session_id, seq, type, time, data, ignorable)
|
||||
VALUES (?, ?, ?, ?, ?, ?);
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
SELECT COUNT(*) AS rows,
|
||||
COALESCE(MAX(length(CAST(data AS BLOB))), 0) AS largest
|
||||
FROM events;
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
PRAGMA foreign_keys = OFF;
|
||||
ALTER TABLE events RENAME TO strict_events;
|
||||
CREATE TABLE events (
|
||||
session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
|
||||
seq INTEGER NOT NULL,
|
||||
type TEXT NOT NULL,
|
||||
time INTEGER NOT NULL,
|
||||
data TEXT NOT NULL,
|
||||
source_event_seqs TEXT,
|
||||
surface_op TEXT,
|
||||
ignorable INTEGER,
|
||||
PRIMARY KEY (session_id, seq)
|
||||
);
|
||||
DROP TABLE strict_events;
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
SELECT seq, rowid
|
||||
FROM events
|
||||
ORDER BY seq;
|
||||
@@ -0,0 +1,4 @@
|
||||
SELECT rowid, seq, type, time, data, source_event_seqs, surface_op, ignorable
|
||||
FROM events
|
||||
WHERE session_id = ?
|
||||
ORDER BY seq;
|
||||
@@ -0,0 +1,5 @@
|
||||
SELECT seq, type, data
|
||||
FROM events
|
||||
WHERE session_id = ?
|
||||
ORDER BY seq DESC
|
||||
LIMIT 1;
|
||||
+1
@@ -0,0 +1 @@
|
||||
PRAGMA user_version;
|
||||
+1
@@ -0,0 +1 @@
|
||||
PRAGMA application_id = 12345;
|
||||
+1
@@ -0,0 +1 @@
|
||||
PRAGMA user_version = 15;
|
||||
+1
@@ -0,0 +1 @@
|
||||
PRAGMA user_version = 16;
|
||||
+1
@@ -0,0 +1 @@
|
||||
PRAGMA user_version = 17;
|
||||
+3
@@ -0,0 +1,3 @@
|
||||
UPDATE sessions
|
||||
SET origin = 'external', delegation_depth = -1, seed_length = -1
|
||||
WHERE id = ?;
|
||||
@@ -0,0 +1,102 @@
|
||||
import { readdir, readFile } from 'node:fs/promises'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import ts from 'typescript'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
const PACKAGE_ROOT = fileURLToPath(new URL('../', import.meta.url))
|
||||
const SQL_LITERAL = /^\s*(?:ALTER|ATTACH|BEGIN|COMMIT|CREATE|DELETE|DETACH|DROP|INSERT|PRAGMA|REINDEX|RELEASE|ROLLBACK|SAVEPOINT|SELECT|UPDATE|VACUUM|WITH)\s/iu // eslint-disable-line @stylistic/max-len
|
||||
|
||||
async function filesUnder(path: string): Promise<string[]> {
|
||||
const entries = await readdir(path, { withFileTypes: true })
|
||||
return (await Promise.all(entries.map(async entry => entry.isDirectory()
|
||||
? filesUnder(`${path}/${entry.name}`)
|
||||
: [`${path}/${entry.name}`]))).flat()
|
||||
}
|
||||
|
||||
function sqlLiteralText(node: ts.Node): string | undefined {
|
||||
if (ts.isStringLiteral(node) || ts.isNoSubstitutionTemplateLiteral(node)) return node.text
|
||||
if (node.kind === ts.SyntaxKind.TemplateHead) {
|
||||
return (node as ts.Node & { readonly text: string }).text
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
function isOwnedSqlSource(node: ts.Expression | undefined, source: ts.SourceFile): boolean {
|
||||
if (node === undefined) return false
|
||||
if (ts.isCallExpression(node)
|
||||
&& ts.isIdentifier(node.expression)
|
||||
&& (node.expression.text === 'sql' || node.expression.text === 'testSql')) return true
|
||||
if (!ts.isIdentifier(node) || node.text !== 'source') return false
|
||||
const call = node.parent
|
||||
if (!ts.isCallExpression(call)
|
||||
|| call.arguments.length !== 1
|
||||
|| call.arguments[0] !== node
|
||||
|| !ts.isPropertyAccessExpression(call.expression)
|
||||
|| call.expression.expression.kind !== ts.SyntaxKind.SuperKeyword
|
||||
|| call.expression.name.text !== 'prepare') return false
|
||||
let method: ts.Node | undefined = node.parent
|
||||
while (method !== undefined && !ts.isMethodDeclaration(method)) method = method.parent
|
||||
if (method === undefined
|
||||
|| method.name.getText(source) !== 'prepare'
|
||||
|| method.parameters.length !== 1
|
||||
|| method.parameters[0]?.name.getText(source) !== 'source') return false
|
||||
let classNode: ts.Node | undefined = method.parent
|
||||
while (classNode !== undefined && !ts.isClassExpression(classNode)) classNode = classNode.parent
|
||||
if (classNode === undefined || classNode.name?.text !== 'JournalFailureDatabase') return false
|
||||
const guard = method.body?.statements[0]
|
||||
if (guard === undefined
|
||||
|| !ts.isIfStatement(guard)
|
||||
|| !ts.isBinaryExpression(guard.expression)
|
||||
|| guard.expression.operatorToken.kind !== ts.SyntaxKind.ExclamationEqualsEqualsToken
|
||||
|| guard.expression.left.getText(source) !== 'source'
|
||||
|| guard.expression.right.getText(source) !== "sql('journal-mode-wal')") return false
|
||||
return ts.isReturnStatement(guard.thenStatement)
|
||||
&& guard.thenStatement.expression === call
|
||||
}
|
||||
|
||||
describe('SQLite SQL resource boundary', () => {
|
||||
it('keeps statements and query assembly out of TypeScript files', async () => {
|
||||
const files = (await Promise.all([
|
||||
filesUnder(`${PACKAGE_ROOT}/src`),
|
||||
filesUnder(`${PACKAGE_ROOT}/tests`),
|
||||
])).flat().filter(path => path.endsWith('.ts'))
|
||||
const violations: string[] = []
|
||||
for (const path of files) {
|
||||
const source = ts.createSourceFile(path, await readFile(path, 'utf8'), ts.ScriptTarget.Latest, true)
|
||||
const usesNodeSqlite = source.statements.some(statement => ts.isImportDeclaration(statement)
|
||||
&& ts.isStringLiteral(statement.moduleSpecifier)
|
||||
&& statement.moduleSpecifier.text === 'node:sqlite')
|
||||
const visit = (node: ts.Node): void => {
|
||||
const literal = sqlLiteralText(node)
|
||||
if (literal !== undefined && SQL_LITERAL.test(literal)) {
|
||||
violations.push(`${path}:${source.getLineAndCharacterOfPosition(node.getStart()).line + 1}: SQL literal`)
|
||||
}
|
||||
// Awaited prepare() is SessionPersistence; DatabaseSync.prepare() is synchronous.
|
||||
if (usesNodeSqlite
|
||||
&& ts.isCallExpression(node)
|
||||
&& ts.isPropertyAccessExpression(node.expression)
|
||||
&& (node.expression.name.text === 'exec'
|
||||
|| (node.expression.name.text === 'prepare' && !ts.isAwaitExpression(node.parent)))) {
|
||||
const argument = node.arguments[0]
|
||||
if (!isOwnedSqlSource(argument, source)) {
|
||||
violations.push(`${path}:${source.getLineAndCharacterOfPosition(node.getStart()).line + 1}: unowned query source`)
|
||||
}
|
||||
}
|
||||
ts.forEachChild(node, visit)
|
||||
}
|
||||
visit(source)
|
||||
}
|
||||
expect(violations).toEqual([])
|
||||
})
|
||||
|
||||
it('keeps resource text static instead of interpolated', async () => {
|
||||
const files = (await Promise.all([
|
||||
filesUnder(`${PACKAGE_ROOT}/resources/sql`),
|
||||
filesUnder(`${PACKAGE_ROOT}/tests/resources/sql`),
|
||||
])).flat()
|
||||
for (const path of files) {
|
||||
expect(path.endsWith('.sql')).toBe(true)
|
||||
expect(await readFile(path, 'utf8')).not.toContain('${')
|
||||
}
|
||||
})
|
||||
})
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,32 @@
|
||||
/** Test-only loader for fixed SQLite fixtures. */
|
||||
|
||||
import { readFileSync } from 'node:fs'
|
||||
|
||||
export type TestSqlName =
|
||||
| 'add-unexpected-column'
|
||||
| 'count-events'
|
||||
| 'count-ignorable-events'
|
||||
| 'count-packed-events'
|
||||
| 'count-physical-types'
|
||||
| 'create-loose-schema'
|
||||
| 'create-unrelated-table'
|
||||
| 'delete-persistence-state'
|
||||
| 'delete-session-events'
|
||||
| 'empty-store-id'
|
||||
| 'insert-corrupt-event'
|
||||
| 'measure-write-traffic'
|
||||
| 'replace-events-with-nonstrict-table'
|
||||
| 'select-last-event'
|
||||
| 'select-event-rowids'
|
||||
| 'select-event-rows'
|
||||
| 'select-user-version'
|
||||
| 'set-application-id-12345'
|
||||
| 'set-user-version-15'
|
||||
| 'set-user-version-16'
|
||||
| 'set-user-version-17'
|
||||
| 'update-invalid-session-metadata'
|
||||
|
||||
/** Load one fixed test SQL resource. */
|
||||
export function testSql(name: TestSqlName): string {
|
||||
return readFileSync(new URL(`./resources/sql/${name}.sql`, import.meta.url), 'utf8')
|
||||
}
|
||||
@@ -20,6 +20,9 @@
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../session-persistence"
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user