10 KiB
description, kind
| description | kind |
|---|---|
| Keyless LLM replay plugin for snapshot tests, for test authors booting the real agent against recorded model transcripts. | package-reference |
@deepseek-ai/dsh-llm-replay
English | 中文
Summary
dsh-llm-replay makes snapshot tests run without an API key: it installs a replay LLM adapter that serves model streams reconstructed from a recorded session JSONL fixture, so a test boots the real agent against a fixed transcript. The fixture is a projection of the persisted session log — assistant/chunk events group into per-call chunk sequences, and an explicitly marked local compaction call replays as one canonical stream. A replay.override.json sidecar covers what a log cannot reconstruct: a throw before any chunk, a cancel/hang, or an injected retry. Live sessions bind to recorded scripts by first-call order, so parent-and-subagent scenarios each get their own script. It is the model source behind the ACP and headless snapshot suites and the Web browser e2e lane.
Table of Contents
- Use this package
- Understand the implementation
- Further Exploration
- Model Experience
- Known Limitations and Deferred Work
- Dev Note
Use this package
This package gives a keyless test a real agent with a fixed model transcript: mount it in place of a real LLM adapter, point it at a recorded fixture, and run the scenario exactly as if the model had produced the recorded output.
Mounting it
With providers configured, the plugin registers a replay-only adapter whose catalog is available to scenarios that exercise model discovery; without providers, it installs the catch-all llm/stream waterfall used by tests that do not need discovery:
- id: llm-replay
name: '@deepseek-ai/dsh-llm-replay'
config:
providers:
- id: deepseek-official
name: DeepSeek
retryPolicy:
mode: normal
backoff:
initialDelayMs: 1
maxDelayMs: 1
jitterRatio: 0
models:
- id: deepseek-v4-flash
contextWindow: 128000
- id: deepseek-v4-pro
# file/overrideFile/childFiles default to $DSH_SNAPSHOT_FILE /
# $DSH_SNAPSHOT_OVERRIDE / $DSH_SNAPSHOT_CHILD_FILES, set by the snapshot
# harness per scenario.
| Field | Default | Meaning |
|---|---|---|
file |
$DSH_SNAPSHOT_FILE |
Path to the primary (parent) session.jsonl fixture; required (config or env) |
overrideFile |
$DSH_SNAPSHOT_OVERRIDE |
Optional ReplayOverrideDoc sidecar for the primary session |
childFiles |
$DSH_SNAPSHOT_CHILD_FILES |
Recorded subagent child-session logs for a nested scenario |
providers |
— | Optional replay-only provider and model catalog; a model may declare contextWindow, text/image modalities, and positive imageRequestTokens when image-capable; invalid values fail at load and routes never perform provider I/O |
paceMs |
— (burst) | Optional per-chunk delay in ms for genuinely incremental delivery |
The generated configuration catalog is the exhaustive source for every accepted field and its JSDoc.
How the fixture works
The fixture is a projection of a persisted session log (<scenario>/session.jsonl) produced by running the real agent once — this plugin does not record. It keeps the header and every event payload but omits body seq/time envelopes (seq0/time0 for packed rows); replay restores contiguous synthetic envelopes while parsing, and one file cannot mix projected and complete body rows. Runtime persistence continues to write complete logs. Replay derives each model call's chunk sequence from assistant/chunk events, so a recorded fixture replays the same logical stream the live model produced. A fixture may carry its request/header content tokenized to {{system}}/{{tools}}; replay is indifferent because derivation reads only the chunk and summary events plus the line-0 session header.
Nested agents
A scenario where a parent agent delegates to in-process subagents records one log per session: the parent (session.jsonl) plus one per child (session.1.jsonl, …). Live session ids are freshly random each run, so replay binds each live session to a recorded script by first-call order: the first live session to make a model call claims the first script, the next new session the next, and so on, with each session advancing its own cursor. More distinct live sessions than recorded scripts fails loud.
Failure modes and overrides
When replay serves deepseek-official with ctx.deepseekLlmApiExtensions, it prepares and accepts those fields after selecting a valid script entry and before yielding the first chunk. This mirrors the live adapter's post-2xx commit point, so durable acceptance watermarks and SDK event notifications behave the same in recording and replay. Replay supplies a synthetic { messages: [] } base body: it proves acceptance side effects, not prepared field bytes.
Two failure modes are not reconstructable from assistant/chunk alone — a pure throw before any chunk (for example an HTTP 401, where the log holds only a turn/end {error}) and a cancel/hang. A scenario that needs those supplies an optional sidecar (<scenario>/replay.override.json) that either replaces the derived script with a bare ReplayEntry[] or augments it with { patches: [{ at, entry }] }, which keeps every derived call and swaps the named 0-based call indexes; at equal to the derived length appends the retry attempt after an injected transient throw. A throw entry accepts DeepSeek request extensions when it has prefix chunks; a zero-chunk throw defaults to pre-2xx non-acceptance and may set accepted: true for a post-2xx failure. A hang entry may name readyFile, which replay writes before waiting for cancellation so an external driver can cancel deterministically.
What can go wrong
- The fixture is not fully consumed —
assertConsumed()at teardown turns a scenario that silently drove fewer model calls than recorded into a crisp diagnostic; call it when installing replay directly in a test. - An unrecorded session makes a call — replay fails loud and tells you to re-record the scenario.
- A scripted placeholder matches nothing —
{{fromRequest:<regex>}}resolution validates the pattern and the request corpus and fails loud on no match, an invalid pattern, or an unterminated placeholder.
Understand the implementation
Implementation internals — click to expand
This section explains the design of the replay plugin; the observable behavior is fully covered in Use this package.
Design
Replay is built on one idea: the projected session log is the fixture. deriveReplayScript parses the JSONL header (for id/createdAt ordering facts) and splits assistant/chunk events at every finish chunk, keyed by (turn, step), so each recorded stream() call becomes one chunks entry; an assistant group without a finish chunk is the fingerprint of a thrown stream() and must be expressed through an override sidecar. A compaction/summary carrying llmStreamCall: true and a complete rawOutput replays as one canonical successful stream at that event's position. Scripted strings may embed {{fromRequest:<regex>}}; at stream time each placeholder resolves against the live request's string leaves, taking the pattern's last match and its first capture group (or the whole match) in place.
Source map
| File | Role |
|---|---|
src/index.ts |
Types, fixture derivation, override validation, placeholder resolution, session binding, installLlmReplay, and the plugin export |
src/invariant.ts |
Invariant companion (no runtime invariant; the stream grammar is checked by the LLM companion and derivation tests) |
Binding and stream flow
installLlmReplay loads the ordered scripts, then installs either a routed replay adapter (when providers is non-empty) or a catch-all llm/stream waterfall listener. Each live stream() call is keyed by its calling session id: a new session claims the next unclaimed script (parent first, because it streams before it can delegate), and calls without a sessionId share one anonymous session bound to the primary script. The returned ReplayHandle carries a disposer for HMR safety and assertConsumed(), which throws unless every recorded script bound to a live session and every bound cursor drained.
Further Exploration
Read these pages when the package-level contract is not enough. They move from the replay adapter to the harness that records fixtures and the loop that consumes streams.
- session-snapshot — the snapshot support that records fixtures and drives replay, record, and refresh modes.
- LLM package — the provider stream contract and adapter registry replay implements.
- Testing policy — the keyless snapshot tier and when it is required.
- Test-support group map — sibling harnesses and support packages.
Model Experience
None, as this keyless test adapter sends no request to a provider model; it only replays recorded assistant chunks into the test loop.
KV Cache effect
None; this package neither assembles nor sends a provider request.
Known Limitations and Deferred Work
These limits define when replay cannot stand in for a live model. They are current package constraints, not a task backlog.
- First-call-order script binding assumes sequential delegation — a cut that runs sibling subagents concurrently would bind live sessions to recorded scripts non-deterministically; a stronger keying is deferred until such a scenario exists.
- Only ordinary loop chunks and marked local compaction outputs are derivable — a pure pre-chunk throw, a cancel/hang, or an unmarked external summarizer call needs the
replay.override.jsonsidecar; replacement and patch forms affect only the primary session, and child scripts still derive from their logs.
Dev Note
Working context for maintainers — click to expand
None.