Session format v2 embeds each attempt's compact stream in
assistant/message and assistant/attempt, but Host and client consumers
still expanded it into per-member TimedStreamChunk arrays and did
per-member work; expandAssistantStream materializes the full array before
find/toReversed/break can answer. Session Stats (the projection phase of
every Session open), the token meter's usage and provider-assembly folds,
the subagent output fold, and the Session Controller image lookup still
paid O(members) allocation and time per settlement.
The Chat and Trajectory definitions were already settled from
message.content on master; the remaining per-member folds stay.
dsh-llm now exports record-level readers (first token, visible content,
visible text, last raw chunk of a type, raw chunks of a type, joined
text, run-aware assembly, per-run first-token/first-visible times) that
scan the compact records once with early exit. Session Stats reads
assistantStreamFirstTokenTime, the token meter reads
lastAssistantStreamChunk(stream, 'usage') and assembles through
assembleAssistantStream, the subagent output fold appends
joinAssistantStreamText, and the Session Controller scans
assistantStreamChunks(stream, 'block-end').
expandAssistantStream is deliberately not memoized: retaining expansions
costs roughly ten times the compact stream for the Session's lifetime.
It remains the validating path at durable boundaries.
Synthetic 200-turn v0 migration benchmark, median of five: first-open
projection 28.0 ms -> 5.4 ms, first-open total 76.9 -> 50.0 ms, peak RSS
137.2 -> 94.9 MB; reopen projection 17.8 -> 5.6 ms; all phase budgets and
the 128 MB heap constraint keep passing.
The migrating first open measured 1,963 ms on the CI runner under the
128 MB heap limit; 4,000 ms keeps a 2x margin while the repeated-snapshot
implementation still fails by heap exhaustion and would need ~10 s.
The migrating first open of the synthetic log costs about 1 s on the
reference machine under the 128 MB heap limit and about twice that on the
CI runner, so 3,000 ms keeps headroom while staying far below the ~5 s
(~10 s on CI) of the repeated-snapshot implementation.
Add a benchmark lane (`vitest.bench.config.ts`, `pnpm run test:bench`,
gate mode `ci-bench`) and a required `node 24 / benchmarks` CI job that
runs it alone. Benchmarks synthesize their input in-process from fixed
parameters and fail on documented budgets:
- `open-generation.bench.ts`: a 200-turn released-v0 log with 500 text
and 125 reasoning deltas per reply (127,400 events, ~2.8 MB) encoded
through the frozen v0 codec; the migrating first `open()` must finish
within 2,000 ms in a child process capped at 128 MB of old space, and a
fresh process must open the published current generation within 500 ms.
- `conversation-fold.bench.client.ts`: 200 replies whose compact streams
hold 2,000 text + 500 reasoning deltas each, folded through every Chat
Definition by the real assembler; the fold must finish within 150 ms
and stay within 3x the fold of the same window with 100 deltas per
reply.
On this commit both gates fail: the migration exhausts the 128 MB heap
(4.8 s and 696 MB peak RSS without the cap; the pre-stack decode of the
same bytes took 34 ms and 168 MB) and the fold scales 11x with the delta
count. The stacked fixes bring both paths to O(records).
Write handles now hold a durable lease (session.lock.json beside the
log): a random owner token, diagnostic pid, and an expiry. Acquisition
wins by exclusive create; a second process's create or write open rejects
while the record is unrenewed for less than leaseTtlMs (default 5 min),
and takes over after that — a crashed holder is waited out, never
reclaimed by pid. The holder renews every leaseRenewIntervalMs (default
4 min); a renewal that finds a foreign, vanished, or expired record — or
fails outright — marks the lease lost permanently, so every later
append/flush rejects with SessionOwnershipLostError while reads continue.
Close releases the record; read handles never touch it. Takeover of an
expired record is eventually exclusive: a replaced holder stops within
one renewal interval.
Refs #3245
`SESSION_FORMAT_VERSION` is the only value that names the Session format
generation. Two other version fields had moved to 2 alongside it:
- The `dsh_session_log` request extension's outer schema `version` returns
to 1 and its Session header projection keeps `seedLength`, derived from
the logical inherited cut, so the external wire is unchanged in this PR;
`sessionFormatVersion` alone identifies the embedded generation.
- The `sessionStats` projection unit's `stateVersion` returns to 1: the
projection cache binds every checkpoint to the format generation, so a
generation change discards old rows without a unit version bump.
Test doubles and the standalone Client fixture that build a current
header now spell `SESSION_FORMAT_VERSION` instead of the literal.
The README file table listed four of the six source modules and omitted
the two largest, so the package read as an oversized identity conversion.
The summary now states that most of the source is the frozen released
v0/v1 event vocabulary, why the edge refuses a malformed historical log
before the installed current restorer runs, and that later edges reuse
those shapes; the table gains `payload-validation.ts` and
`relationships.ts` rows.
`session[.vN].jsonl` was assembled independently by the JSONL provider
(twice), the session-log export archive, and the recorded-session fixture
helpers. `dsh-session-format` now owns `sessionFormatLogFilename` and
`parseSessionFormatLogFilename`; the three consumers append only their
compression suffix, and the migration note names the owner.
`snapshotSessionFormatJson` re-implemented the lossless JSON walk that
`dsh-util-values` already publishes as `snapshotJsonValue` + `deepFreeze`,
and the frozen v0 relationship validator carried a third structural JSON
comparison next to `deepEqualJson`. Both now delegate; the format package
reports one `is not lossless JSON` diagnostic per labelled subject instead
of ten member-specific ones, and enumerable accessors whose values survive
a JSON round trip are accepted like `JSON.stringify` accepts them.
The JSONL backend rethrows storage errno failures and abort signals from the
source read of a pending migration unwrapped; a test now drives both through
the stat seam so the classification stays covered. The navigation-panes Web
e2e derives the exported archive entry from `SESSION_FORMAT_VERSION`, which
`session-log-export` uses for its `session.v<N>.jsonl` name. The module graph
records the `session-log-export` → `session` peer edge.
`session-format-v1-to-v2` takes the `0.1.2-rc.1` root version that master
requires, and `defineReleasedPayloadDisposition` drops the `@param` tag for a
parameter it no longer has.
`session-log-export` names its archive entry after `SESSION_FORMAT_VERSION`,
a runtime value of the shared `dsh-session` instance, so `dsh-session` becomes
a peer dependency and the export is classified as peer-required in the package
dependency policy. The three Session format packages take the `0.1.2-rc.1`
root version that master now requires.
The repository-only benchmark, its unit-suite spec, the root script, and the
README section are removed; the accepted acceptance run's figures stay in the
embedded-assistant-streams note as recorded facts. The note also records why
the migration edge reuses the `dsh-llm` stream helpers instead of frozen
copies and why its target validation re-checks message/stream agreement
itself. A validation test that recomputed `RELEASED_V2_EVENT_TYPES` from the
same expression as its definition is dropped.
`ensureJsonlGenerationCurrent` returned a live Zstandard body iterator that
every production caller disposed unread before decoding the same bytes again;
the header read now decodes only the first frame and owns no decoder past its
return. The JSONL backend reuses the generation module's stable-revision read
instead of keeping a second copy, `encodeCurrent` encodes the validated
current artifact it is documented to receive instead of re-running the
migration chain on it, and the test-only `inspectVersion` catalog method and
the `migrateSessionSnapshotFixture` duplicate of
`prepareSessionSnapshotFixtureForComparison` are removed.
Yichen Jiang