deepseek-harness/packages/session/session-persistence

@deepseek-ai/dsh-session-persistence

English | 中文

Session persistence is a capability seam. The abstract SessionPersistence service (ctx.sessionPersistence) is its Service Definition. It requires a persistence backend to store, reload, and list sessions durably without defining the storage implementation. The seam follows the dsh-shell roles (capability seams): this package owns the Service Definition, a sibling package owns the Service Provider, and Consumers inject the service.

The persisted unit IS the existing SessionEvent (event-sourced model — the log is the single source of truth), so there is no parallel "persisted message" type. Metadata that is NOT replayable conversation state (format version, cwd, lineage, seed boundary, origin, delegation depth) travels separately as SessionHeader, owned by dsh-session and re-exported here.

Service API (ctx.sessionPersistence)

Method	Contract
locate(meta): SessionLocation | undefined	Resolve an absolute per-session artifact target without I/O or materialization. Backends without an independent local artifact return undefined.
supportsRawArtifacts: boolean	State explicitly whether this backend exposes one verbatim artifact per session. Consumers check this capability before calling readRaw; false is not session absence.
readRaw(id, signal?): Promise<SessionRawArtifact | undefined>	Read a supported backend's own artifact text verbatim, decoded from its physical encoding but never reconstructed from events. undefined means only that the requested artifact is absent; an unsupported backend rejects.
create(meta): Promise<void>	Register a new session's metadata. MAY defer the physical write until the first append (lazy materialization).
ensureMaterialized(session): Promise<void>	Explicitly make an exact live session durable even with zero events, without inventing an event. Lifecycle frontends use this only when the empty session itself is a resumable resource; ordinary creation remains lazy.
append(id, events): Promise<void>	Durably persist a batch. Append-only; first event seq == stored next-seq after any repair; rejects non-JSON-serializable data naming the offending type.
prepare(id, signal?): Promise<SessionPreparation>	Reserve the exact unpublished Session used by resume. A coordinator reuses an earlier inspection when available, commits pending recovery, and releases an unpublished reservation back to its bounded cache on disposal.
load(id): Promise<{ meta; events }>	Return an immutable balanced logical log after decoding a supported format path and committing any format replacement plus cold recovery. A live load first flushes its snapshot and rejects while its turn is open; a cold load preserves an interrupted final turn and durably closes it with synthetic tool/result/step/end?/turn/end {interrupted} events. Only a torn tail fragment is dropped; committed corruption and malformed records reject as SessionPersistenceCorruptionError, while an unsupported format version or an event type unknown to this build (without the envelope's ignorable marker) refuses as SessionFormatUnsupportedError, naming the refusal direction and the raw log path when the backend keeps one artifact per session.
inspect(id, signal?): Promise<{ meta; events }>	Return an upgraded, validated, deeply frozen logical view without committing recovery or publishing a Session. A cold view receives in-memory synthetic recovery closers while its physical torn tail remains untouched; an already-live view is its current immutable snapshot and may contain an open turn. Coordinator-backed implementations retain the exact cold unpublished Session in a bounded LRU for later prepare, but discard and reload it when the stored revision changes. Same-id inspections share an in-flight read.
readFrom(id, fromSeq, signal?): Promise<{ meta; events }>	Return valid stored events with seq >= fromSeq without preparation caching, truncation, closers, or coordinator state. A fromSeq at or past the stored end returns an empty event list; a negative or non-safe-integer fromSeq rejects. Current-format reads request a suffix from the backend; a format migration requires the complete source and applies fromSeq only after migration. Sequential media may still scan framing before filtering, while seek-capable media can avoid reading earlier rows. Intended for checkpoint consumers that apply only events after a stored sequence number.
list(signal?): Promise<SessionHeader[]>	Lightweight listing from metadata, no full-log parse. The optional signal cancels backend listing work. A zero-event session is absent until a consumer explicitly materializes it.
listSnapshots(signal?): Promise<SessionPersistenceSnapshot[]>	Lightweight metadata plus an opaque branded per-log revision, without loading event logs. A revision stays equal while that log and its backing store are unchanged, changes after append or mutating load repair, and cannot collide solely because two stores use the same local counter. The optional signal requests cancellation of backend discovery work; first-party backends settle any started listing work before rejecting so an awaited call is quiescent.

Invariants every backend must honor

• Append-only; a crashed turn is closed, not truncated. Flushed events are never rewritten. A crash can leave an unclosed final turn whose events are real and possibly large; load preserves them and durably appends synthetic closers (a risk-classified error tool/result per unanswered assistant call, then step/end?+turn/end {interrupted}) to balance the log and keep the rehydrated history a valid provider transcript. Only a never-fully-written torn tail fragment is discarded.
• Contiguous seq. load rejects a seq gap/parse error in the MIDDLE of the log; append's first seq must equal the stored next-seq.
• JSON-serializable data. append materializes each direct/replay batch through the shared one-pass lossless-JSON boundary. Live Session events are already deep-frozen, but the write coordinator still copies each event into a persistence-owned buffer.
• Durability. append returns only once the batch is durable.

The write coordinator

PersistenceCoordinator owns per-id state and serialization, one bounded write controller per live session, lazy materialization, crash-tail repair, session adoption, and quiescent disposal. A first-party backend composes one, implements the small PersistenceBackend storage hook interface, and delegates its stateful methods. JSONL and SQLite therefore share lifecycle correctness while retaining different storage primitives; see the coordinator Agent Note, flush-controller simplification, and bounded batching decision.

Each session/event copies its event into the session controller. The first pending event starts a fixed batching window; later events join without resetting its deadline. The configured writeBatchMaxDelayMs bounds this intentional wait, not event-loop, initialization, serialized-operation, or backend latency. Events admitted during a write form a new bounded batch. session/flush cancels the wait and is a shared quiescence barrier that drains events admitted while it runs. A background failure is logged once, retains the ordered batch, and pauses automatic retry; a new event starts a fresh window, while explicit flush or backend teardown retries immediately and surfaces a repeated failure.

Crash repair is cold-only. For a live id, load(id) snapshots the authoritative in-memory log, waits for that snapshot to become durable, and returns it only when balanced; an open live turn rejects instead of receiving synthetic interruption closers. For a cold id, inspection reads, validates, freezes, and constructs one unpublished Session; repeated inspection reuses that object graph only while its source revision remains current. prepare(id) performs the same check before repair, reserves the exact Session, commits any pending torn-tail/interrupted-turn repair, and returns it for publication. HMR adoption opens the same revision-bound source, applies the coordinator's cwd check, and never closes the active turn.

Format decoding and upgrades

Every logical read opens a repeatable StoredSessionSource containing an untrusted header, an exact revision, and a readEvents() factory. The static decoder chooses a complete adjacent-version path, creates one migration instance per version, calls header() once, calls event() once per input record, and calls optional finish() after EOF. It then validates the final header and events as the current format. inspect() and readFrom() do not write. Cold continuation and live adoption replace a converted source through the backend's revision compare-and-swap, then reopen it; a concurrent change discards the decoded result and restarts from the new source. The session-log versioning Agent Note owns the rationale and refusal rules.

A future vN→vN+1 change adds src/format-migrations/vN-to-vN+1.ts, exports its class from the static SESSION_FORMAT_MIGRATIONS array, and increments SESSION_FORMAT_VERSION. Static from/to identify adjacent versions; instance fields retain header and cross-event state. header() validates and converts the old header, event() returns exactly one lossless-JSON event with the input event's seq, and optional finish() validates state that can be settled only at EOF. Header-only reads do not call finish(). A migration that changes facts consumed by a projection also increments that projection's stateVersion; persistence does not invalidate every projection cache entry. Backends and the coordinator remain version-independent.

The v0 decoder also recognizes the bounded pre-versioning variants recorded by the pre-identity message and pre-react-loop session decisions, and normalizes the historical compact/start, compact/summary, compact/end, and compact/prune names to their canonical compaction/* names. These compatibility transforms are not format migrations.

When a live session emits session/disposed, the coordinator waits for its controller, serializes a final drain, then releases state owned by that exact Session object. Failed retirement leaves the controller in the live-session map, so backend teardown can retry it. Backend teardown stops event admission first, flushes every remaining controller, awaits per-id operations, and only then closes the storage handle.

The side-effect-free locate, lightweight listSnapshots, and per-id readStoredRevision queries remain backend-owned because they describe storage topology and revision identity rather than write orchestration. listSnapshots(signal?) passes the caller's exact signal into backend discovery so observers can cancel that work without detaching it.

The PersistenceBackend<TornMarker> hooks (the only contract between the coordinator and storage):

Hook	Role
name	Backend label for the dispose-failure AggregateError.
openStored(id, signal?)	Open an untrusted header plus repeatable event readers bound to one exact source revision. Each readEvents({ fromSeq? }) reproduces that revision and exposes backend-owned torn-tail metadata only after EOF, or rejects with SessionPersistenceRevisionConflictError when the source changed.
readStoredRevision(id, signal?)	Read the current source-qualified revision for one id without loading its event log. It uses the same revision representation as openStored and returns undefined when the id is absent.
appendBatch(meta, events, isMaterialized)	Durably append a contiguous batch, lazily materializing ATOMICALLY when not yet materialized.
materializeHeader?(meta)	Durably create a header-only artifact for ensureMaterialized; required by providers that support durable empty sessions.
commitRepair(meta, tornMarker, closers)	Make a crash repair durable: truncate the torn tail (iff tornMarker !== undefined — a marker may be falsy, e.g. seq/offset 0) and append closers. NOT required to be atomic. Used by load (truncate + closers) and live-adoption (truncate only).
replaceStored(expectedRevision, meta, events)	Atomically replace one exact revision with a complete current-format header and event stream. Revision and stored identity checks occur at the commit boundary — immediately before the atomic rename on JSONL, inside the replacing transaction on SQLite; the checks add no cross-process writer exclusion. A mismatch rejects with SessionPersistenceRevisionConflictError.
list(signal?)	List all stored metadata, observing optional cancellation.
close?()	Optional lifecycle teardown (e.g. close a db handle), awaited after the dispose drain.

The coordinator asserts the stored id and compares stored/live cwd before repair or live adoption. Its inspect() path takes ownership of fresh backend values, validates and freezes them once, and retains at most the configured number of unpublished Sessions without calling commitRepair. A retained source is reused or repaired only when its revision still equals readStoredRevision; otherwise the coordinator reloads it. This freshness check does not add cross-process writer exclusion. Revision retries converge when the durable log remains unchanged for one read/check round trip; continuous external writers can delay load, inspect, or prepare. The tornMarker is fully OPAQUE: the coordinator only tests !== undefined and round-trips it to commitRepair, never inspecting its value (the JSONL backend uses the byte offset to truncate to, the SQLite backend the seq to delete from). A third-party backend MAY implement the abstract service directly without the coordinator, but it must provide the same non-mutating inspection and trustworthy lightweight snapshot revisions. See the write-coordinator Agent Note.

Metadata and location types

Re-exported from dsh-session: SessionHeader (immutable session metadata: version, id, createdAt, cwd?, parentSession?, seedLength?, origin?, delegationDepth?). SessionLocation is { readonly kind: string; readonly path: string }; its path is an absolute backend target, not proof that the artifact exists or contains an unflushed turn.

Model Experience

Resumed conversation history

What the model sees

This seam adds no prompt or schema. Resume restores stored surface events as message history; stored request headers reconstruct earlier calls, while the new loop composes the current system prompt, tools, and session prefix for its next request. Crash repair marks an assistant request without a durable call as TOOL_NOT_STARTED; a durable call without a result becomes TOOL_OUTCOME_UNKNOWN, whose text lets the model retry read-only or idempotent work but directs it to verify side effects or ask the user instead of retrying blindly.

Token effect

Zero tokens during ordinary persistence. Resume restores retained history cost and pays the current request envelope normally; each repaired call adds the quoted retained error text.

KV Cache effect

Persistence does not mutate live request prefixes. A resumed loop can reuse provider cache only when its reconstructed history, current envelope, and model route match; crash-repair results append without rewriting earlier history.

Known Limitations and Deferred Work

• No deletion or retention API — pruning stored sessions is out-of-band backend maintenance.
• list() is unpaginated and unfiltered — it returns every stored session's header; fine for local stores, unindexed at scale.
• Repair-time synthetic closers are the only crash story — a backend must synthesize tool/result/step/end/turn/end closers on load; there is no partial-turn resume that continues an interrupted turn instead of closing it.