mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-08-29 04:26:38 +00:00
fix(review): harden web replay verification
Validate replay sidecars and cross-copy failure facts, make browser console tripwires and macOS temp paths deterministic, and wait for asynchronous TUI resume details. Keep the owning docs, translations, and generated catalog aligned.
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: 1d96028e8e9255518b4e5127f0aeeaa4ee68b411
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: e07fce4b62c05b1b4774e6d1758321e3b7bd315c
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: c4e34b3f44162c7021cb25681eea7e49ac78f672
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: 466e1c0fc16aac21b87b68cfedaec4fb22a417e2
|
||||
|
||||
@@ -10,7 +10,7 @@ The web GUI ships as a real assembled chain — chromium page → client plugin
|
||||
|
||||
## Decision
|
||||
|
||||
`pnpm run test:web` carries a keyless, deterministic browser e2e lane under `apps/web/tests/`: recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay` against the real in-process web composition, asserting a normalized conversation aria golden plus in-process world state. No new package; the product deltas are additive `dsh-llm-replay` surfaces (`paceMs`, `ReplayHandle`, and the `{ patches }` override form: indexed augmentation over the derived script so a sidecar expresses "call N throws / hangs, everything else replays as recorded" without copying recorded chunks), one `dsh-llm` fix the retry scenario exposed (a carried `failure` snapshot is honored on any Error — the `instanceof` gate dropped provider codes across dual package copies, source-plane replay over a lib-plane boot), and the `llm-retry` row the web composition was missing.
|
||||
`pnpm run test:web` carries a keyless, deterministic browser e2e lane under `apps/web/tests/`: recorded session-log fixtures replay through `@deepseek-ai/dsh-llm-replay` against the real in-process web composition, with normalized aria goldens for user-visible states and in-process assertions for durable world state. The supporting product contracts are `dsh-llm-replay` pacing, consumption checks, and validated indexed override patches; cross-package `dsh-llm` failures retain validated provider facts through own data properties; and the shipped web composition mounts `llm-retry` for transient model failures.
|
||||
|
||||
### Scaffold: `apps/web/tests/scaffold.ts`
|
||||
|
||||
@@ -32,25 +32,17 @@ Every scenario fails on any pageerror and on the client's connection-loss/gap-re
|
||||
|
||||
### Expected outputs
|
||||
|
||||
At least one committed golden per scenario, and one per DISTINCT end-state for the interactive scenarios (cancel/error/retry, waiting/answered, mid-steer/settled, panel-open, post-reload): a normalized `ariaSnapshot()` of the scenario's owning region — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride root-context session events inline (which tool call produced which durable result, whether `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping.
|
||||
Scenarios with a stable owning region commit a normalized `ariaSnapshot()` for each distinct user-visible state; cross-region workspace-management states instead use semantic DOM assertions plus authoritative host-state checks. UUID, cwd, workspace basename, and duration volatility collapse to stable tokens; captures poll until consecutive normalized reads agree. Role and text anchors remain semantic guards around the reviewable goldens and own cross-region states directly. World-state assertions use root-context session events rather than a second committed log golden because the ACP, headless, and TUI suites already pin the persisted-log surface through the same loop and persistence. `refresh` is the sole golden writer; a missing replay golden fails with the regeneration command.
|
||||
|
||||
The typecheck plane split is structural: the three files that boot the host spine (`scaffold`, `replay-round-trip.e2e`, and `seeded-history.e2e`) are excluded from the client-registered `apps/web` project. Those files and their shared `support.ts` are included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges.
|
||||
The typecheck plane split is structural: the host scaffold, its support module, and every web spec that boots or inspects the host composition are excluded from the client-registered `apps/web` project and included file-by-file in `tsconfig.host.json`. One program cannot hold both sides of the Cordis `Context` merges.
|
||||
|
||||
### Modes and fixtures
|
||||
|
||||
`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` tokenization; a follow-up keyless refresh regenerates the aria goldens. Every prompting scenario's fixture was recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point with no run-local `rpcId`). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred.
|
||||
`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless). Prompting specs separate drive steps shared by all modes from replay/refresh assertions; record mode drives the live composer, harvests the in-memory session header and events, scrubs request headers, and tokenizes run-local session, cwd, and RPC identities. A follow-up keyless refresh regenerates aria goldens. Each prompt is checked against its fixture's recorded `user/message`, and each scenario directory has a closed inventory whose JSONL files are scrub fixed points. Web fixtures scrub headers everywhere and pin no header class; see Deferred.
|
||||
|
||||
### Scenarios
|
||||
### Coverage contract
|
||||
|
||||
1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (the bash call's durable result is exactly `WEB_E2E_OK\n`, completed `turn/end`, >10 chunk events).
|
||||
2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it (group row → session row, collapsed by default), opening renders tool cards and text purely from the log through the implicit cold-resume attach inside `session.history` — zero model calls in replay, so no binding constraints; record mode drives the same turn live (real `read` tool against seeded workspace files) to produce the seed.
|
||||
3. **`live-interactions`** — one tool-free recorded turn serves three replay-only scenarios through override sidecars whose CONTENT is authored in the spec and minted as a per-run file in a spec-owned temp dir (the derived success entry for the retry append is re-derived from the fixture via `deriveReplayScript`, never copied into a committed sidecar). Cancel: a `{ patches }` `hang` with a `readyFile` marker — the marker's existence proves the stream is parked mid-turn before the test clicks Stop, making mid-stream cancellation deterministic by construction (`turn/end` reason `aborted`, composer re-enabled). AUTH error: a pre-chunk `throw` outside llm-retry's retryable set (`turn/end` reason `error`, zero `llm/retry` events, composer recovers). SERVER retry: `throw` at call 0 + the fixture's own success appended at 1, proving llm-retry end-to-end in the browser via the durable `llm/retry` record (`request/header` logs only on change, so attempt count is invisible there). Each scenario pins its terminal surface as a golden: `cancel.expected.md` (frozen `partial`, 已停止 marker), `error-auth.expected.md` (the prompt bubble alone — the committed artifact of the web-error-surface gap, the diff that flips when error rendering lands), `retry.expected.md` (indistinguishable from a clean completion — retries are deliberately invisible in the transcript).
|
||||
4. **`question-composer`** — the shipped composition's resident `ask_user_question` takeover: a recorded turn blocks mid-step on the real userInteraction seam, the composer (`[data-question-key]`) renders in the browser, the test answers through it (the ONE sanctioned place a drive step reacts to model content: the turn cannot complete without the answer, in record and replay alike), and the tool result carries the chosen label. Goldens: the composer's stable waiting state (`ui.expected.md`) and the answered transcript (`answered.expected.md` — the question resolved into its tool round trip plus the final reply, takeover gone).
|
||||
5. **`steering`** — mid-turn steer while the question composer blocks the step (the deterministic mid-turn window; no timing dependence). The composer locks while running, so the steer POSTs `session.prompt` `mode:'steer'` from the page over the same same-origin `/api` wire the client uses (`TODO(web-steer-composer)`: drive a composer gesture once one exists); everything downstream is product — gateway → `Agent.steer` → step-boundary drain → durable `steering/message` → SSE → badged interjection bubble. Record-mode fixture honesty: the recording is rejected unless the live model's final reply obeys an instruction only the steering message carries. Goldens pin the timing semantics visually: `mid-steer.expected.md` captures the accepted-but-invisible state (the loop drains steering only at the step boundary, so no interjection bubble exists while the question still blocks — if the client ever renders pending steers eagerly, this golden flips first) and `settled.expected.md` the badged bubble plus obeying reply.
|
||||
6. **`navigation-panes`** — one rich two-turn seed (turn 1: bash + two parallel reads in one assistant message; turn 2: a markdown-heavy reply) rendered cold through the seeded-history pattern (zero model calls), serving four surfaces: sidebar search (client-side title filter — asserted only after the durable title lands with the attach baseline, because a cold `SessionSummary` carries no title and search matches the `displayTitle` the user sees; negative query empties the tree, positive narrows, clear restores), the Trajectory tab (turn sections + the step group's tool mix plus the view-area aria golden), the Waterfall tab (span stats + one lane per span — the P-I fold counts a turn-0 prologue span because only assistant/steering nodes carry a turn number, pinned as-is), and the details column (the bash toolview row routes click to openDetails; open/closed is asserted on the frame's `data-details-collapsed` attribute because close collapses the grid column to width 0 without unmounting the subtree). Goldens: `trajectory.expected.md` and `waterfall.expected.md` (each tab's view area) plus `details-open.expected.md` (the open panel: tool-name header, Input args, Output result).
|
||||
7. **`lifecycle-chrome`** — one tiny recorded text turn drives three whole-page concerns. Workspace flow over the real wire: the empty-state hero's first send materializes a real Workspace + Session (the jsdom `workspace-flow.snapshot.ts` suite pins this state machine over the fixture client; this scenario pins it through HTTP RPC + SSE + the gateway), proven durably by the session header's cwd being the create-by-name target `<workspaceRoot>/workspace`, plus the hero waiting-state aria golden. Reload recovery: collapse the sidebar (persisted `dsh.layout.panels`), `page.reload`, and the surface comes back whole from persistence alone — layout collapsed, selection restored (`dsh.sessions.current`), the recorded turn re-rendered from `session.history` with zero model calls (the drained replay cursor makes any stray request fail loud at close), and `reloaded.expected.md` pins the rebuilt conversation region — rendering the same settled transcript from persistence alone IS the recovery claim. Dark mode: the scenario drives the ThemeService's DOM contract seam directly — the `body[data-ds-dark-theme]` attribute — and pins the shipped cascade (alias token flips, a painted surface repaints, removal restores the light sample exactly), independent of the settings surface whose real user gesture `settings-chrome` owns; per the scope ruling there is no theme/layout golden (aria is color-blind).
|
||||
8. **`settings-chrome`** — the settings surface (#644), zero model calls on a blank frame. The modal shell: sidebar-foot trigger (`aria-haspopup`/`aria-expanded`) opens `role=dialog` 设置, General active by default with the skeleton rows plus the functional Language and Appearance rows (dialog aria golden), section switch moves `aria-current` to the deliberately empty Models, closes via Escape and the header close button. The Appearance row is the REAL theme gesture (retiring the lifecycle scenario's `TODO(web-theme-gesture)`): clicking 深色 runs the whole chain — `aria-pressed`, persisted `dsh.theme`, `body[data-ds-dark-theme]`, alias-token flip — and survives reload; `system` follows the emulated OS scheme both ways (`page.emulateMedia`), and the spec restores the light default for inter-spec hygiene. The Language row switches the settings-scoped copy to English (`dsh.locale` persisted, dialog re-registers as Settings/General/Appearance), survives reload, and restores zh — only the settings namespaces are localized today, so the scenario asserts exactly that surface. Intentional reloads tear the SSE stream, so the spec drains exactly the reconnect warnings its own reloads caused; the tripwire still fails on any unexpected connection loss.
|
||||
9. **`workspace-management`** — the workspace browser operations (#643), zero model calls (workspace.create/rename are host RPCs; the one session row comes from re-seeding seeded-history's committed seed, so no new fixture is recorded). Create-by-name twice through the region-header + dialog (`workspace.create` mkdirs and prepends to the durable registry — asserted host-side via `ctx.workspace.list()`). Rename end to end: the hover-revealed row-actions menu (the button is `display:none` until its row hovers) → Rename dialog → the duplicate-name pre-check raises the inline `role=alert` and disables the primary button before any wire call → a fresh name goes through the `workspace.rename` RPC, updates the row, persists on the host, and survives reload. The flat "In one list" view: the Group by menu flips the section label to Sessions, drops group headers (seeded session becomes a top-level row), persists in `dsh.workspace.view` across reload, and the spec restores grouped mode. The session hover card renders after the dwell (display-only, no aria role — text anchors) and closes when the pointer leaves. Deliberately NOT driven: the visual-only menu rows this iteration ships inert (session Rename/Fork/Delete, workspace Delete) and drag reorder — see Deferred.
|
||||
The lane covers three behavior families. Live-turn scenarios pin ordinary tool execution, cancellation, non-retryable failure, transient retry, resident questions, and mid-turn steering; synchronization uses durable events, `whenIdle()`, or an explicit replay marker rather than delays. Cold-history scenarios seed through the real persistence API and cover history rendering, sidebar search, trajectory and waterfall views, and tool details without model calls. Browser-lifecycle scenarios cover first-send workspace materialization, reload recovery, layout persistence, theme and locale preferences, and workspace create/rename/view operations. Each family asserts the browser surface and the authoritative host state; a stray model call or under-consumed fixture fails teardown.
|
||||
|
||||
### CI stance
|
||||
|
||||
@@ -70,7 +62,7 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
|
||||
|
||||
**Placeholder `DEEPSEEK_API_KEY` + replay interception instead of disabling the adapter row.** Rejected despite zero composition change and two in-tree precedents: it satisfies `llm-deepseek`'s fail-loud key check with a lie and leaves a dead adapter mounted-but-intercepted; the disabled row (the ACP overlay's move) is honest keylessness and fails loud at the earliest resolvable point.
|
||||
|
||||
**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios a factory generalizes from one consumer while the genuinely shared logic is already exported from gated packages. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free.
|
||||
**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and the scenario-specific interactions have not produced a stable browser-free contract beyond the helpers already exported from gated packages and the local scaffold. Reconsider when a second web-shaped consumer or demonstrably repeated lifecycle code establishes that contract.
|
||||
|
||||
**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on root-context events keep the world-verification duty.
|
||||
|
||||
@@ -80,11 +72,11 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
|
||||
|
||||
**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement.
|
||||
|
||||
**A client `data-dsh-busy` settled signal.** Deferred: the multi-condition settled polls proved sufficient at two scenarios and the host-side `whenIdle` barrier does the heavy lifting. Re-entry trigger: the first settled-poll flake, or a scenario needing a state the DOM does not expose.
|
||||
**A client `data-dsh-busy` settled signal.** Deferred: the host-side `whenIdle` barrier plus stable DOM polls cover the current scenarios. Reconsider after the first settled-poll flake or when a required state is not observable in the DOM.
|
||||
|
||||
## Testing
|
||||
|
||||
The lane itself: `pnpm run test:web` runs every scenario keylessly alongside the existing smoke pair; `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` re-records a scenario's fixture against the live model; `DSH_SNAPSHOT=refresh` rewrites the aria goldens keylessly. `paceMs` validation, pacing floor, abort-during-pace, both `assertConsumed` failure shapes, and the `{ patches }` acceptance/rejection paths (index swap keeps siblings, `at == length` appends, out-of-range/non-integer loud) are pinned in `packages/support/llm-replay/tests/llm-replay.spec.ts`.
|
||||
`pnpm run test:web` runs the lane keylessly. `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` records a prompting scenario against the live model, and `DSH_SNAPSHOT=refresh` rewrites aria goldens keylessly. `dsh-llm-replay` unit coverage pins pacing, cancellation, consumption diagnostics, sidecar validation, indexed replacement, and the single append position.
|
||||
|
||||
## Deferred
|
||||
|
||||
@@ -93,7 +85,7 @@ The lane itself: `pnpm run test:web` runs every scenario keylessly alongside the
|
||||
- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses.
|
||||
- **Web error surface**: the client consumes no `agent/error` frames and a pre-chunk failure freezes no partial, so a non-retryable provider failure renders no error copy — the user sees the send simply stop. The AUTH scenario pins the current contract (no crash, composer recovers, turn logged `error`) and `FIXME(web-error-surface)` marks where visible error text gets asserted once the UI grows an error rendering.
|
||||
- **Composer steering gesture**: the input locks while running (stop-or-wait), so the steering scenario steers over the wire from the page; `TODO(web-steer-composer)` upgrades the drive step to a real composer gesture when the product grows one.
|
||||
- **Drag session reorder**: `workspace.insertSessionBefore` (manual ordering, #643) has no browser scenario yet — it needs two sessions materialized in ONE workspace (a two-script recorded fixture) plus synthesized HTML5 drag events; add it when that surface changes or regresses. The inert menu rows (session Rename/Fork/Delete, workspace Delete) get scenarios when they gain behavior.
|
||||
- **Drag session reorder**: `workspace.insertSessionBefore` has no browser scenario; it needs two sessions materialized in one workspace plus synthesized HTML5 drag events. Add it when that surface changes or regresses. The inert session Rename/Fork/Delete and workspace Delete menu rows get scenarios when they gain behavior.
|
||||
|
||||
## Consequences
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
## 决策
|
||||
|
||||
`pnpm run test:web` 携带 `apps/web/tests/` 下的无密钥、确定性浏览器 e2e 车道:录制的会话日志 fixture 经 `@deepseek-ai/dsh-llm-replay` 对真实进程内 web 组合回放,断言规范化后的会话区 aria 预期输出加进程内世界状态。不新增包(package);产品侧增量为 `dsh-llm-replay` 的增量接口(`paceMs`、`ReplayHandle`,以及 `{ patches }` 覆写形式:对派生脚本按索引增补,使一份 sidecar 无需复制已录分片即可表达「第 N 次调用抛错/挂起,其余照录回放」),一处由重试场景暴露的 `dsh-llm` 修复(携带的 `failure` 快照对任何 Error 都生效——此前的 `instanceof` 判定会在两份包副本并存时丢弃提供方错误码,即源码平面回放叠在 lib 平面 boot 之上的情形),以及 web 组合此前缺失的 `llm-retry` 行。
|
||||
`pnpm run test:web` 携带 `apps/web/tests/` 下的无密钥、确定性浏览器 e2e 车道:录制的会话日志 fixture 经 `@deepseek-ai/dsh-llm-replay` 对真实进程内 web 组合回放;用户可见状态使用规范化的 aria 预期输出,持久世界状态则使用进程内断言。配套的产品契约包括 `dsh-llm-replay` 的节奏控制、消费检查与已校验的索引式覆写 patch;跨包的 `dsh-llm` 失败通过自有数据属性保留经校验的提供方信息;已交付的 web 组合挂载 `llm-retry`,以处理瞬态模型失败。
|
||||
|
||||
### Scaffold:`apps/web/tests/scaffold.ts`
|
||||
|
||||
@@ -32,25 +32,17 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
### 预期输出
|
||||
|
||||
每场景至少一份提交的预期输出,交互类场景则每个不同终态各一份(取消/错误/重试、等待/已作答、steer 中途/安定、面板打开、重新加载后):该场景所属区域的规范化 `ariaSnapshot()`——uuid/cwd/工作区目录名/时长归一为稳定 token,在安定里程碑处轮询至两次相等再采集——外加几条 role/文本锚断言,让保语义的组件重写在预期输出可评审地变动时仍保持绿色锚点。aria 树是 client 规则「断言用户所见,绝不断言类名」的机械化。世界状态断言内联在根上下文的会话事件上(哪次工具调用产生了哪项已持久化的工具结果、`turn/end` 是否完成)而不是第二份提交的日志预期输出:持久化日志表面已由 ACP/headless/TUI 套件经同一循环和持久化钉住,在此重复钉住会违背分层纪律、翻倍刷新成本。`refresh` 是预期输出的唯一写入者——回放模式下预期输出缺失会连同修复命令一起报错,而不是静默自举。
|
||||
具有稳定所属区域的场景会为每个不同的用户可见状态提交一份规范化的 `ariaSnapshot()`;跨区域的工作区管理状态则使用语义 DOM 断言和权威的 host 状态检查。UUID、cwd、工作区目录名与时长等易变内容会归一为稳定 token;采集过程持续轮询,直到连续两次规范化读取结果相同。Role 与文本锚点继续充当可评审预期输出周围的语义防线,并直接覆盖跨区域状态。世界状态断言使用根上下文的会话事件,而不是第二份提交的日志预期输出,因为 ACP、headless 与 TUI 套件已经通过同一循环和持久化钉住持久化日志表面。`refresh` 是预期输出的唯一写入者;回放模式下缺少预期输出时,测试会连同重新生成命令一起失败。
|
||||
|
||||
类型检查平面切分是结构性的:启动 host 主干的三个文件(`scaffold`、`replay-round-trip.e2e` 和 `seeded-history.e2e`)被排除出注册在 client 侧的 `apps/web` 工程。这三个文件及其共享的 `support.ts` 逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。
|
||||
类型检查平面切分是结构性的:host scaffold、其支持模块,以及每个启动或检查 host 组合的 web spec 都会从注册在 client 侧的 `apps/web` 工程中排除,并逐文件纳入 `tsconfig.host.json`。一个程序不能同时持有 Cordis `Context` 合并的两侧。
|
||||
|
||||
### 模式与 fixture
|
||||
|
||||
`DSH_SNAPSHOT` 以内联 spec 分支选择 replay(默认,无密钥)、record(带密钥)或 refresh(无密钥)——TUI 的形态,不是套件工厂:两个场景撑不起 acp-snapshot 工厂机制,且真正共享的部分已被导出(`scrubRequestHeaders`、`parseSessionLog`、`installLlmReplay`)。每个 spec 切分为驱动步骤(输入、发送、`whenTurnSettled`——所有模式都执行,绝不等待模型内容选择器,因此 record 不会因真实模型答法不同而挂起)与断言步骤(仅 replay/refresh)。Record = 经真实输入框实时驱动 + 采收内存中的 `session.header`/`session.events`(TUI 的 `rawSessionLog` 形态——无需文件解压)+ `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` token 化;随后一次无密钥 refresh 重新生成各份 aria 预期输出。每个发起提示的场景,其 fixture 都经此流程对本组装录制。一条漂移防线把每个 spec 的驱动提示词与 fixture 录制的 `user/message` 绑定。fixture 清单防线保持每个场景目录封闭(精确文件集合,每个 JSONL 都是脱敏不动点,不含当次运行的 `rpcId`)。Web fixture 全部脱敏请求头且不钉任何头类别,沿用 TUI 先例而非[钉住请求头](2026-07-06-pin-request-header-content-in-one-scenario.md)的严格读法——见「暂缓」。
|
||||
`DSH_SNAPSHOT` 选择 replay(默认,无密钥)、record(带密钥)或 refresh(无密钥)。发起提示的 spec 将所有模式共用的驱动步骤与仅供 replay/refresh 使用的断言分开;record 模式驱动真实输入框,采收内存中的会话 header 与事件,脱敏请求头,并 token 化当次运行的会话、cwd 与 RPC 标识。随后一次无密钥 refresh 重新生成 aria 预期输出。每条提示词都会与 fixture 中录制的 `user/message` 核对;每个场景目录都采用封闭清单,其中每个 JSONL 都是脱敏不动点。Web fixture 全部脱敏请求头且不钉任何 header 类别;见「暂缓」。
|
||||
|
||||
### 场景
|
||||
### 覆盖契约
|
||||
|
||||
1. **`replay-round-trip`**——新会话,经真实输入框发送提示词,回放流式输出推理(reasoning)+ 一次在临时工作区真实执行的 `bash` 工具调用 + 最终文本(15ms 节奏)。断言安定后的 markdown、aria 预期输出与内联世界状态(这次 bash 调用的已持久化工具结果严格等于 `WEB_E2E_OK\n`、完成的 `turn/end`、>10 个分片事件)。
|
||||
2. **`seeded-history`**——冷播种一份已录会话;侧栏列出它(分组行 → 会话行,默认折叠),打开后纯凭日志经 `session.history` 内的隐式冷恢复挂载渲染工具卡片与文本——replay 下零模型调用,因此没有任何绑定约束;record 模式实时驱动同一轮(真实 `read` 工具读取播种的工作区文件)来产出种子。
|
||||
3. **`live-interactions`**——一段不含工具调用的已录轮次经覆写 sidecar 承载三个仅回放的场景:sidecar 的内容本身写在 spec 里,每次运行时在 spec 自有的临时目录中生成文件(重试追加所用的派生成功条目经 `deriveReplayScript` 从 fixture 重新派生,绝不复制进已提交的 sidecar)。取消:一个带 `readyFile` 标记的 `{ patches }` `hang`——标记文件的存在证明流在测试点击 Stop 之前已停驻在轮次中途,使流中取消按构造即确定(`turn/end` 原因为 `aborted`,输入框重新启用)。AUTH 错误:一次落在 llm-retry 可重试集合之外的分片前 `throw`(`turn/end` 原因为 `error`,零条 `llm/retry` 事件,输入框恢复可用)。SERVER 重试:第 0 次调用 `throw` + 在第 1 次调用处追加 fixture 自身的成功条目,凭持久的 `llm/retry` 记录在浏览器中端到端证明 llm-retry(`request/header` 仅在变化时记录,因此尝试次数在那里不可见)。每个场景都把各自的终态表面钉为一份预期输出:`cancel.expected.md`(冻结的 `partial`、「已停止」标记)、`error-auth.expected.md`(仅有提示词气泡——web-error-surface 缺口的已提交产物,错误渲染落地时翻转的那份 diff)、`retry.expected.md`(与一次干净完成无从区分——重试在文本记录中刻意不可见)。
|
||||
4. **`question-composer`**——已交付组合中常驻的 `ask_user_question` 接管:一段已录轮次在真实的 userInteraction seam 上阻塞于步骤中途,提问输入框(`[data-question-key]`)在浏览器中渲染,测试经它作答(这是驱动步骤对模型内容作出反应的唯一获准之处:没有这个回答,轮次无法完成,record 与 replay 皆然),工具结果携带所选的 label。预期输出:提问输入框稳定的等待态(`ui.expected.md`)与已作答的文本记录(`answered.expected.md`——提问已落定为其工具往返加最终回复,接管消失)。
|
||||
5. **`steering`**——在提问输入框阻塞该步骤时做轮次中途 steering(中途引导),此即确定性的轮次中途窗口,不依赖任何时序。输入框在运行期间锁定,因此这一 steer 由页面经客户端所用的同一条同源 `/api` wire POST `session.prompt` `mode:'steer'`(`TODO(web-steer-composer)`:待有输入框手势后改为驱动它);下游的一切都是产品路径——gateway → `Agent.steer` → 步骤边界排空 → 持久的 `steering/message` → SSE → 带徽标的插话气泡。record 模式的 fixture 诚实性:除非真实模型的最终回复遵循了一条只有 steering 消息才携带的指令,否则该次录制被拒绝。预期输出以可视方式钉住这一时序语义:`mid-steer.expected.md` 捕捉「已接受但不可见」的状态(循环仅在步骤边界才排空 steering,因此提问仍在阻塞时不存在插话气泡——若 client 日后提前渲染待处理的 steer,这份预期输出会最先翻转),`settled.expected.md` 则捕捉带徽标的气泡加遵循指令的回复。
|
||||
6. **`navigation-panes`**——一份内容丰富的双轮次种子(轮次 1:同一条 assistant 消息内的 bash + 两次并行 read;轮次 2:一段 markdown 密集的回复)经 seeded-history 模式冷渲染(零模型调用),承载四个表面:侧栏搜索(客户端标题过滤;仅在持久的标题随 attach 基线一同到达后才断言,因为冷的 `SessionSummary` 不携带标题,而搜索匹配的是用户所见的 `displayTitle`;反例查询清空整棵树,正例查询收窄,清除后复原)、Trajectory 标签页(轮次分节 + 步骤组的工具构成,外加视图区 aria 预期输出)、Waterfall 标签页(span 统计 + 每个 span 一条泳道;只有 assistant/steering 节点携带轮次编号,因此 P-I 折叠会将一个轮次 0 的序幕 span 计入,按原样钉住)与详情列(bash 工具视图行把点击路由到 openDetails;打开/关闭状态断言在 frame 的 `data-details-collapsed` 属性上,因为关闭把网格列收缩到宽度 0 而不卸载子树)。预期输出:`trajectory.expected.md` 与 `waterfall.expected.md`(各自标签页的视图区),外加 `details-open.expected.md`(打开的面板:工具名标题、Input 参数、Output 结果)。
|
||||
7. **`lifecycle-chrome`**——一段极小的已录纯文本轮次驱动三个整页关注点。真实 wire 上的 Workspace 动线:空态 hero 的首次发送物化出真实的 Workspace + Session(jsdom 的 `workspace-flow.snapshot.ts` 套件基于 fixture 客户端钉住这一状态机;本场景则经 HTTP RPC + SSE + gateway 钉住它),其持久证据是会话头部的 cwd 恰为按名创建的目标 `<workspaceRoot>/workspace`,外加 hero 等待态的 aria 预期输出。重新加载恢复:折叠侧栏(持久化于 `dsh.layout.panels`),`page.reload`,整个表面纯凭持久化完整归来——布局保持折叠,选中项恢复(`dsh.sessions.current`),已录轮次从 `session.history` 重新渲染且零模型调用(已耗尽的回放游标使任何离群请求都在 close 时大声失败),且 `reloaded.expected.md` 钉住重建后的会话区——纯凭持久化渲染出同一份安定的文本记录,这本身就是恢复主张。暗色模式:本场景直接驱动 ThemeService 的 DOM 契约 seam(即 `body[data-ds-dark-theme]` 属性),并钉住已交付的级联(alias token 翻转,某个实际绘制的表面重绘,移除该属性则精确还原亮色采样值),且独立于设置表面——该表面的真实用户手势归 `settings-chrome` 管;按范围裁定,主题/布局不设预期输出(aria 感知不到颜色)。
|
||||
8. **`settings-chrome`**——设置表面(#644),空白 frame 上零模型调用。模态框外壳:侧栏底部的触发按钮(`aria-haspopup`/`aria-expanded`)打开 `role=dialog` 的「设置」,默认激活「通用设置」,其中既有骨架行,也有具备实际功能的「语言」与「外观」两行(对话框 aria 预期输出);分节切换把 `aria-current` 移到刻意留空的「模型」分节;经 Escape 与头部的「关闭」按钮均可关闭。「外观」行是真正的主题手势(lifecycle 场景的 `TODO(web-theme-gesture)` 就此撤除):点击「深色」跑通整条链路(`aria-pressed`、持久化的 `dsh.theme`、`body[data-ds-dark-theme]`、alias token 翻转)并在重新加载后存续;`system` 双向跟随所模拟的操作系统配色方案(`page.emulateMedia`),该 spec 还会恢复「浅色」默认值以保证 spec 之间互不污染。「语言」行把设置范围内的文案切换为 English(`dsh.locale` 持久化,对话框重新注册为 Settings/General/Appearance),在重新加载后存续,最后恢复为「中文」——目前本地化只覆盖设置命名空间,因此该场景断言的恰是这一表面。有意的重新加载会撕断 SSE 流,因此该 spec 恰好只排空自身重新加载引发的重连警告;任何意外的连接丢失仍会触发绊线失败。
|
||||
9. **`workspace-management`**——工作区浏览器操作(#643),零模型调用(workspace.create/rename 是 host 侧 RPC;唯一的会话行来自重新播种 seeded-history 已提交的种子,因此没有录制任何新 fixture)。经区域头部的「+」对话框按名创建两次(`workspace.create` 会 mkdir 并把新项前插到持久注册表——host 侧经 `ctx.workspace.list()` 断言)。端到端的重命名:悬停显露的行操作菜单(按钮在所在行悬停之前是 `display:none`)→ Rename 对话框 → 重名预检在发出任何 wire 调用之前就亮出内联 `role=alert` 并禁用主按钮 → 换一个全新名称则走 `workspace.rename` RPC,更新该行、在 host 上持久化并在重新加载后存续。扁平的「In one list」视图:Group by 菜单把分节标签翻转为 Sessions,去掉分组头(播种的会话成为顶层行),在 `dsh.workspace.view` 中持久化并跨重新加载存续,该 spec 最后恢复分组模式。会话悬停卡片在驻留延时后渲染(纯展示,无 aria role——用文本锚定),指针移开即关闭。刻意不驱动:本次迭代以无行为形态交付的纯视觉菜单行(会话的 Rename/Fork/Delete、工作区的 Delete)与拖拽重排——见「暂缓」。
|
||||
该车道覆盖三类行为。实时轮次场景钉住普通工具执行、取消、不可重试失败、瞬态重试、常驻提问与轮次中途 steering;同步依赖持久事件、`whenIdle()` 或显式回放标记,而不使用延时。冷历史场景通过真实持久化 API 播种,在不调用模型的情况下覆盖历史渲染、侧栏搜索、Trajectory 与 Waterfall 视图及工具详情。浏览器生命周期场景覆盖首次发送时物化工作区、重新加载恢复、布局持久化、主题与语言偏好,以及工作区的创建、重命名和视图操作。每类场景都断言浏览器表面和权威的 host 状态;离群的模型调用或未耗尽的 fixture 会使拆卸失败。
|
||||
|
||||
### CI 立场
|
||||
|
||||
@@ -70,7 +62,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
**用占位 `DEEPSEEK_API_KEY` + 回放拦截替代禁用适配器行。** 尽管零组合改动且树内有两处先例仍被否决:它用谎言满足 `llm-deepseek` 的快速失败密钥检查,还留下一个挂载却被拦截的死适配器;禁用行(ACP overlay 的同款做法)是诚实的无密钥,并在最早可解析点快速失败。
|
||||
|
||||
**`packages/support/web-snapshot` 包 + `defineWebSnapshotSuite` 工厂。** 已否决:驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100%,且两个场景就上工厂是从单一消费方过度泛化,真正共享的逻辑已从受门禁的包中导出。重启条件:出现第二个 web 形态消费方,或 ≥6 个场景的内联分支被证实各自漂移;届时包边界将画在无浏览器一侧。
|
||||
**`packages/support/web-snapshot` 包 + `defineWebSnapshotSuite` 工厂。** 已否决:驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100%,且除受门禁的包已导出的辅助工具与本地 scaffold 外,这些场景专用交互尚未形成稳定的无浏览器契约。出现第二个 web 形态消费方,或被证实重复的生命周期代码确立该契约后,再重新考虑。
|
||||
|
||||
**第二份提交的规范化会话日志预期输出。** 已否决:日志表面已由 ACP/headless/TUI 套件经同一循环与持久化钉住;在此只会翻倍刷新成本并重复测试下层。内联在根上下文事件上的世界状态断言保住了验证世界的义务。
|
||||
|
||||
@@ -80,11 +72,11 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
**以真实模型浏览器测试充当无密钥车道。** 已否决:按构造即不确定;被调研的前车之鉴(open-webui)长出无界超时后被删除。带密钥的 W5 冒烟仍是真实模型侧的补充。
|
||||
|
||||
**客户端 `data-dsh-busy` 安定信号。** 暂缓:两个场景下多条件安定轮询已经够用,host 侧 `whenIdle` 屏障承担了重活。重启条件:第一次安定轮询抖动,或某场景需要等待 DOM 不暴露的状态。
|
||||
**客户端 `data-dsh-busy` 安定信号。** 暂缓:host 侧 `whenIdle` 屏障配合稳定 DOM 轮询,足以覆盖当前场景。第一次安定轮询抖动,或必要状态在 DOM 中不可观察时,再重新考虑。
|
||||
|
||||
## Testing
|
||||
|
||||
车道自身:`pnpm run test:web` 与既有冒烟对一起无密钥运行所有场景;`DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` 对真实模型重录某场景的 fixture;`DSH_SNAPSHOT=refresh` 无密钥重写各份 aria 预期输出。`paceMs` 校验、节奏下限、节奏中中止、`assertConsumed` 的两种失败形态,以及 `{ patches }` 的接受/拒绝路径(按索引换入保留邻项、`at == length` 追加、越界/非整数大声失败)钉在 `packages/support/llm-replay/tests/llm-replay.spec.ts`。
|
||||
`pnpm run test:web` 无密钥运行该车道。`DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` 对真实模型录制一个发起提示的场景,`DSH_SNAPSHOT=refresh` 则无密钥重写 aria 预期输出。`dsh-llm-replay` 单元覆盖率钉住节奏控制、取消、消费诊断、sidecar 校验、按索引替换与唯一的追加位置。
|
||||
|
||||
## 暂缓
|
||||
|
||||
@@ -93,7 +85,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
- **恢复后追问场景**:真实 wire 上的历史/实时缝合路径;当该代码变更或回归时作为独立场景补充。
|
||||
- **Web 错误表面**:客户端不消费任何 `agent/error` 帧,分片前的失败也没有可冻结的部分输出,因此不可重试的提供方失败不渲染任何错误文案——用户看到的只是发送就此停住。AUTH 场景钉住当前契约(不崩溃、输入框恢复可用、轮次记录为 `error`),`FIXME(web-error-surface)` 标记了待 UI 长出错误渲染后断言可见错误文本的位置。
|
||||
- **输入框 steering 手势**:输入在运行期间锁定(只能停止或等待),因此 steering 场景从页面走 wire 做 steer;`TODO(web-steer-composer)` 待产品长出真实的输入框手势后,把驱动步骤升级为该手势。
|
||||
- **拖拽会话重排**:`workspace.insertSessionBefore`(手动排序,#643)尚无浏览器场景——它需要在同一个工作区里物化两个会话(一份双脚本的已录 fixture)外加合成的 HTML5 拖拽事件;当该表面变更或回归时再补充。无行为的菜单行(会话的 Rename/Fork/Delete、工作区的 Delete)待长出行为后获得各自的场景。
|
||||
- **拖拽会话重排**:`workspace.insertSessionBefore` 尚无浏览器场景;它需要在同一个工作区里物化两个会话,并合成 HTML5 拖拽事件。当该表面变更或回归时再补充。无行为的会话 Rename/Fork/Delete 和工作区 Delete 菜单行待获得行为后再补充场景。
|
||||
|
||||
## 后果
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ import { chromium } from 'playwright'
|
||||
import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts,
|
||||
acknowledgeReloadConnectionLoss, assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts,
|
||||
launchWebScaffold, recordFixture, watchConsole, webSnapshotMode, type WebScaffold,
|
||||
} from './scaffold.ts'
|
||||
import { saveFailureShot } from './support.ts'
|
||||
@@ -103,8 +103,10 @@ describe('web e2e: lifecycle & chrome (workspace flow / reload / dark mode)', ()
|
||||
// (persisted under dsh.layout.panels) before reloading.
|
||||
await page.getByRole('button', { name: 'Collapse sidebar' }).click()
|
||||
await expect.poll(() => page.getByRole('button', { name: 'Open sidebar' }).count(), { timeout: 10_000 }).toBe(1)
|
||||
const warningStart = tripwire.warnings.length
|
||||
await page.reload({ waitUntil: 'load' })
|
||||
await page.waitForSelector('[class*="frame"]', { timeout: 30_000 })
|
||||
acknowledgeReloadConnectionLoss(tripwire, warningStart)
|
||||
// Layout persisted: the sidebar comes back collapsed.
|
||||
await expect.poll(() => page.getByRole('button', { name: 'Open sidebar' }).count(), { timeout: 10_000 }).toBe(1)
|
||||
// Selection persisted (dsh.sessions.current) and history replayed: the
|
||||
@@ -155,6 +157,7 @@ describe('web e2e: lifecycle & chrome (workspace flow / reload / dark mode)', ()
|
||||
}, 60_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('keeps the fixture inventory closed', async () => {
|
||||
expect(tripwire.warnings).toEqual([])
|
||||
await assertFixtureInventory(SNAPSHOT_DIR, ['session.jsonl', 'hero.expected.md', 'reloaded.expected.md'])
|
||||
})
|
||||
})
|
||||
|
||||
@@ -142,6 +142,7 @@ describe('web e2e: live-turn interactions (cancel / error / retry)', () => {
|
||||
const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold!.workspaceCwd)
|
||||
await compareOrRefreshGolden(CANCEL_EXPECTED, snapshot, MODE)
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
expect(tripwire.warnings).toEqual([])
|
||||
}, 120_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('surfaces a non-retryable AUTH failure without retrying', async () => {
|
||||
@@ -167,6 +168,7 @@ describe('web e2e: live-turn interactions (cancel / error / retry)', () => {
|
||||
const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold!.workspaceCwd)
|
||||
await compareOrRefreshGolden(ERROR_EXPECTED, snapshot, MODE)
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
expect(tripwire.warnings).toEqual([])
|
||||
}, 120_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('recovers a transient SERVER failure through llm-retry and completes', async () => {
|
||||
@@ -194,6 +196,7 @@ describe('web e2e: live-turn interactions (cancel / error / retry)', () => {
|
||||
const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold!.workspaceCwd)
|
||||
await compareOrRefreshGolden(RETRY_EXPECTED, snapshot, MODE)
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
expect(tripwire.warnings).toEqual([])
|
||||
}, 120_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('keeps the fixture inventory closed', async () => {
|
||||
|
||||
@@ -71,8 +71,8 @@ describe('web e2e: resident question composer round trip', () => {
|
||||
await expect.poll(() => composer.getByText('Which color do you prefer?').count(), { timeout: 10_000 }).toBeGreaterThan(0)
|
||||
|
||||
if (MODE !== 'record') {
|
||||
// Golden of the composer's waiting state (the transcript region golden
|
||||
// is #612's job; this pins the question surface).
|
||||
// This golden owns the stable question surface; the answered-state
|
||||
// golden below owns the resulting transcript.
|
||||
const snapshot = await captureStableAria(page, '[data-question-key]', scaffold.workspaceCwd)
|
||||
await compareOrRefreshGolden(UI_EXPECTED, snapshot, MODE)
|
||||
}
|
||||
@@ -98,6 +98,7 @@ describe('web e2e: resident question composer round trip', () => {
|
||||
const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd)
|
||||
await compareOrRefreshGolden(ANSWERED_EXPECTED, snapshot, MODE)
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
expect(tripwire.warnings).toEqual([])
|
||||
}, 200_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('keeps the fixture inventory closed', async () => {
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
// (the plugin-row path discards the ReplayHandle; the direct install keeps
|
||||
// assertConsumed for the teardown fixture-consumption check).
|
||||
import { existsSync, readFileSync } from 'node:fs'
|
||||
import { mkdtemp, readFile, readdir, rm, utimes, writeFile } from 'node:fs/promises'
|
||||
import { mkdtemp, readFile, readdir, realpath, rm, utimes, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join, resolve } from 'node:path'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
@@ -141,7 +141,7 @@ export async function launchWebScaffold(options: LaunchOptions = {}): Promise<We
|
||||
throw new Error('web e2e record mode needs DEEPSEEK_API_KEY (env or repo-root .env)')
|
||||
}
|
||||
}
|
||||
const workspaceCwd = await mkdtemp(join(tmpdir(), 'dsh-web-e2e-ws-'))
|
||||
const workspaceCwd = await realpath(await mkdtemp(join(tmpdir(), 'dsh-web-e2e-ws-')))
|
||||
let persistenceRoot: string
|
||||
try {
|
||||
persistenceRoot = await mkdtemp(join(tmpdir(), 'dsh-web-e2e-sessions-'))
|
||||
@@ -453,3 +453,17 @@ export function watchConsole(page: Page): { warnings: string[]; pageErrors: stri
|
||||
page.on('pageerror', (error) => { pageErrors.push(String(error)) })
|
||||
return { warnings, pageErrors }
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove only connection-loss warnings emitted after an intentional reload.
|
||||
* Earlier warnings and all gap-repair/discontinuity warnings remain fatal.
|
||||
* @param tripwire - the live console-warning collector.
|
||||
* @param warningStart - warning count captured immediately before reloading.
|
||||
*/
|
||||
export function acknowledgeReloadConnectionLoss(
|
||||
tripwire: ReturnType<typeof watchConsole>,
|
||||
warningStart: number,
|
||||
): void {
|
||||
const reloadWarnings = tripwire.warnings.splice(warningStart)
|
||||
tripwire.warnings.push(...reloadWarnings.filter(text => !/connection lost/i.test(text)))
|
||||
}
|
||||
|
||||
@@ -12,7 +12,7 @@ import { chromium } from 'playwright'
|
||||
import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'
|
||||
import { join } from 'node:path'
|
||||
import {
|
||||
assertFixtureInventory, captureStableAria, compareOrRefreshGolden,
|
||||
acknowledgeReloadConnectionLoss, assertFixtureInventory, captureStableAria, compareOrRefreshGolden,
|
||||
launchWebScaffold, watchConsole, webSnapshotMode, type WebScaffold,
|
||||
} from './scaffold.ts'
|
||||
import { saveFailureShot } from './support.ts'
|
||||
@@ -36,17 +36,6 @@ describe('web e2e: settings modal, appearance gesture, language switch', () => {
|
||||
await page.waitForSelector('[class*="frame"]', { timeout: 30_000 })
|
||||
}, 120_000)
|
||||
|
||||
/**
|
||||
* An INTENTIONAL reload tears the SSE stream mid-flight, so the dying
|
||||
* page's reconnect note is expected — drain exactly those entries so the
|
||||
* tripwire still fails the spec on any UNEXPECTED connection loss.
|
||||
*/
|
||||
const drainReloadWarnings = (): void => {
|
||||
const kept = tripwire.warnings.filter(text => !/connection lost/i.test(text))
|
||||
tripwire.warnings.length = 0
|
||||
tripwire.warnings.push(...kept)
|
||||
}
|
||||
|
||||
afterAll(async () => {
|
||||
await browser?.close()
|
||||
await scaffold?.close()
|
||||
@@ -114,9 +103,10 @@ describe('web e2e: settings modal, appearance gesture, language switch', () => {
|
||||
await page.keyboard.press('Escape')
|
||||
|
||||
// Reload: the preference survives boot (restore + presenter initial apply).
|
||||
const warningStart = tripwire.warnings.length
|
||||
await page.reload({ waitUntil: 'load' })
|
||||
await page.waitForSelector('[class*="frame"]', { timeout: 30_000 })
|
||||
drainReloadWarnings()
|
||||
acknowledgeReloadConnectionLoss(tripwire, warningStart)
|
||||
await page.emulateMedia({ colorScheme: 'light' })
|
||||
const reloaded = await readState()
|
||||
expect(reloaded.attr).toBe(true)
|
||||
@@ -158,9 +148,10 @@ describe('web e2e: settings modal, appearance gesture, language switch', () => {
|
||||
expect(await page.evaluate(() => localStorage.getItem('dsh.locale'))).toBe('en')
|
||||
// Reload keeps English; then restore zh so shared page state (and the
|
||||
// other specs' 设置-anchored selectors + goldens) see the default again.
|
||||
const warningStart = tripwire.warnings.length
|
||||
await page.reload({ waitUntil: 'load' })
|
||||
await page.waitForSelector('[class*="frame"]', { timeout: 30_000 })
|
||||
drainReloadWarnings()
|
||||
acknowledgeReloadConnectionLoss(tripwire, warningStart)
|
||||
const enTrigger = page.getByRole('button', { name: 'Settings' })
|
||||
await enTrigger.waitFor({ timeout: 10_000 })
|
||||
await enTrigger.click()
|
||||
|
||||
@@ -162,6 +162,7 @@ describe('web e2e: mid-turn steering lands durably and visibly', () => {
|
||||
const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd)
|
||||
await compareOrRefreshGolden(SETTLED_EXPECTED, snapshot, MODE)
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
expect(tripwire.warnings).toEqual([])
|
||||
}, 200_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('keeps the fixture inventory closed', async () => {
|
||||
|
||||
@@ -12,7 +12,7 @@ import type { Browser, Page } from 'playwright'
|
||||
import { chromium } from 'playwright'
|
||||
import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'
|
||||
import {
|
||||
assertFixtureInventory, launchWebScaffold, seedSession, watchConsole,
|
||||
acknowledgeReloadConnectionLoss, assertFixtureInventory, launchWebScaffold, seedSession, watchConsole,
|
||||
webSnapshotMode, type WebScaffold,
|
||||
} from './scaffold.ts'
|
||||
import { saveFailureShot } from './support.ts'
|
||||
@@ -45,17 +45,6 @@ describe('web e2e: workspace management (create / rename / flat view / hover car
|
||||
await page.waitForSelector('[class*="frame"]', { timeout: 30_000 })
|
||||
}, 120_000)
|
||||
|
||||
/**
|
||||
* An INTENTIONAL reload tears the SSE stream mid-flight, so the dying
|
||||
* page's reconnect note is expected — drain exactly those entries so the
|
||||
* tripwire still fails the spec on any UNEXPECTED connection loss.
|
||||
*/
|
||||
const drainReloadWarnings = (): void => {
|
||||
const kept = tripwire.warnings.filter(text => !/connection lost/i.test(text))
|
||||
tripwire.warnings.length = 0
|
||||
tripwire.warnings.push(...kept)
|
||||
}
|
||||
|
||||
afterAll(async () => {
|
||||
await browser?.close()
|
||||
await scaffold?.close()
|
||||
@@ -108,9 +97,10 @@ describe('web e2e: workspace management (create / rename / flat view / hover car
|
||||
expect(await page.getByText('alpha-ws', { exact: true }).count()).toBe(0)
|
||||
// Host durability, then reload: the projection is rebuilt from the wire.
|
||||
expect(scaffold.ctx.workspace.list().map(workspace => workspace.title)).toContain('gamma-ws')
|
||||
const warningStart = tripwire.warnings.length
|
||||
await page.reload({ waitUntil: 'load' })
|
||||
await page.waitForSelector('[class*="frame"]', { timeout: 30_000 })
|
||||
drainReloadWarnings()
|
||||
acknowledgeReloadConnectionLoss(tripwire, warningStart)
|
||||
await expect.poll(() => page.getByText('gamma-ws', { exact: true }).count(), { timeout: 15_000 }).toBeGreaterThanOrEqual(1)
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
}, 90_000)
|
||||
@@ -129,9 +119,10 @@ describe('web e2e: workspace management (create / rename / flat view / hover car
|
||||
await expect.poll(() => page.locator('[role="treeitem"]').count(), { timeout: 10_000 }).toBeGreaterThanOrEqual(1)
|
||||
expect(await page.evaluate(() => localStorage.getItem('dsh.workspace.view'))).toContain('flat')
|
||||
// Persisted across reload; then restore grouped for inter-spec hygiene.
|
||||
const warningStart = tripwire.warnings.length
|
||||
await page.reload({ waitUntil: 'load' })
|
||||
await page.waitForSelector('[class*="frame"]', { timeout: 30_000 })
|
||||
drainReloadWarnings()
|
||||
acknowledgeReloadConnectionLoss(tripwire, warningStart)
|
||||
await expect.poll(() => page.getByText('Ungrouped', { exact: true }).count(), { timeout: 15_000 }).toBe(0)
|
||||
await page.getByRole('button', { name: 'Group by' }).click()
|
||||
await page.getByRole('menuitem', { name: 'WorkSpace' }).click()
|
||||
|
||||
@@ -688,7 +688,7 @@ export interface ReplayModelConfig {
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/support/llm-replay/src/index.ts:497`](../packages/support/llm-replay/src/index.ts)
|
||||
Source: [`packages/support/llm-replay/src/index.ts:590`](../packages/support/llm-replay/src/index.ts)
|
||||
|
||||
## `@deepseek-ai/dsh-llm-retry`
|
||||
|
||||
|
||||
@@ -47,13 +47,10 @@ export function markLlmAdapterFailure(
|
||||
const error = value instanceof Error
|
||||
? value as Error & { code?: string }
|
||||
: new HarnessError(String(value), 'UNKNOWN', { cause: value })
|
||||
// The own `failure` data property is the serializable boundary contract:
|
||||
// validated field-by-field and cross-checked against the error's own code,
|
||||
// then honored on ANY Error — an instanceof gate here would drop the facts
|
||||
// exactly when class identity is lost (a second copy of this package in
|
||||
// the process, e.g. a source-plane test harness over a lib-plane boot).
|
||||
// Cross-package copies preserve own data but not class identity. Trust the
|
||||
// carried facts only when both own properties agree after validation.
|
||||
const carried = ownFailureSnapshot(error)
|
||||
const failure = carried !== undefined && carried.code === foreignErrorCode(error) ? carried : Object.freeze({
|
||||
const failure = carried !== undefined && carried.code === ownErrorCode(error) ? carried : Object.freeze({
|
||||
message: errorMessage(error),
|
||||
code: harnessErrorCode(error),
|
||||
})
|
||||
@@ -61,13 +58,12 @@ export function markLlmAdapterFailure(
|
||||
return error
|
||||
}
|
||||
|
||||
/** Read a foreign error's `code` for the cross-check without letting an SDK accessor replace the primary failure. */
|
||||
function foreignErrorCode(error: Error & { code?: string }): unknown {
|
||||
/** Read a foreign error's own data-backed `code` without invoking accessors. */
|
||||
function ownErrorCode(error: Error): unknown {
|
||||
try {
|
||||
return error.code
|
||||
} catch (_sdkCodeGetter) {
|
||||
// An unreadable code cannot confirm the carried facts describe this
|
||||
// error; the caller falls back to the normalized snapshot.
|
||||
const descriptor = Object.getOwnPropertyDescriptor(error, 'code')
|
||||
return descriptor !== undefined && 'value' in descriptor ? descriptor.value : undefined
|
||||
} catch (_sdkPropertyTrap) {
|
||||
return undefined
|
||||
}
|
||||
}
|
||||
|
||||
@@ -291,6 +291,34 @@ describe('LlmService', () => {
|
||||
expect(facts).not.toBe(carried)
|
||||
})
|
||||
|
||||
it('keeps validated failure facts across package copies with matching own codes', async () => {
|
||||
const original = Object.assign(new Error('provider busy'), {
|
||||
code: 'RATE_LIMIT',
|
||||
failure: {
|
||||
message: 'provider busy',
|
||||
code: 'RATE_LIMIT',
|
||||
status: 429,
|
||||
providerRetryAfterMs: 1_500,
|
||||
requestId: 'req-cross-copy',
|
||||
},
|
||||
})
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ThrowingAdapter(original))
|
||||
|
||||
const stream = ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })
|
||||
await expect((async () => {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
})()).rejects.toBe(original)
|
||||
expect(llmFailureOf(stream, original)).toEqual({
|
||||
message: 'provider busy',
|
||||
code: 'RATE_LIMIT',
|
||||
status: 429,
|
||||
providerRetryAfterMs: 1_500,
|
||||
requestId: 'req-cross-copy',
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps an unknown SDK Error exact without trusting its private code or accessors', async () => {
|
||||
const original = Object.assign(new Error('socket closed'), { code: 'ECONNRESET' })
|
||||
Object.defineProperty(original, 'failure', {
|
||||
@@ -324,10 +352,7 @@ describe('LlmService', () => {
|
||||
expect(llmFailureOf(stream, original)).toEqual({ message: 'LLM adapter failed', code: 'UNKNOWN' })
|
||||
})
|
||||
|
||||
it('keeps an SDK Error exact when a valid failure payload rides a hostile code accessor', async () => {
|
||||
// The carried-facts cross-check reads error.code; a throwing accessor
|
||||
// there must fall back to the normalized snapshot instead of replacing
|
||||
// the original adapter error with the accessor exception.
|
||||
it('keeps an SDK Error exact without trusting accessor-backed carried facts', async () => {
|
||||
const original = Object.assign(new Error('busy'), {
|
||||
failure: { message: 'busy', code: 'SERVER', status: 503 },
|
||||
})
|
||||
@@ -345,6 +370,46 @@ describe('LlmService', () => {
|
||||
expect(llmFailureOf(stream, original)).toEqual({ message: 'busy', code: 'UNKNOWN' })
|
||||
})
|
||||
|
||||
it('does not trust carried facts matched only by an inherited code', async () => {
|
||||
class InheritedCodeError extends Error {
|
||||
get code(): string { return 'SERVER' }
|
||||
}
|
||||
const original = Object.assign(new InheritedCodeError('busy'), {
|
||||
failure: { message: 'busy', code: 'SERVER', status: 503 },
|
||||
})
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ThrowingAdapter(original))
|
||||
const stream = ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })
|
||||
|
||||
await expect((async () => {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
})()).rejects.toBe(original)
|
||||
expect(llmFailureOf(stream, original)).toEqual({ message: 'busy', code: 'UNKNOWN' })
|
||||
})
|
||||
|
||||
it('keeps an SDK Error exact when code descriptor inspection is trapped', async () => {
|
||||
const target = Object.assign(new Error('busy'), {
|
||||
code: 'SERVER',
|
||||
failure: { message: 'busy', code: 'SERVER', status: 503 },
|
||||
})
|
||||
const original = new Proxy(target, {
|
||||
getOwnPropertyDescriptor(value, property) {
|
||||
if (property === 'code') throw new Error('SDK code descriptor trap')
|
||||
return Reflect.getOwnPropertyDescriptor(value, property)
|
||||
},
|
||||
})
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ThrowingAdapter(original))
|
||||
const stream = ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })
|
||||
|
||||
await expect((async () => {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
})()).rejects.toBe(original)
|
||||
expect(llmFailureOf(stream, original)).toEqual({ message: 'busy', code: 'UNKNOWN' })
|
||||
})
|
||||
|
||||
it('falls back safely when SDK objects trap failure inspection or expose malformed facts', async () => {
|
||||
const propertyTrap = new Proxy(new HarnessError('descriptor trapped', 'SERVER'), {
|
||||
getOwnPropertyDescriptor(target, property) {
|
||||
|
||||
@@ -71,12 +71,13 @@ export interface Scenario {
|
||||
recorded: boolean
|
||||
/**
|
||||
* Whether replay is driven by a hand-written `replay.override.json` sidecar
|
||||
* (a `ReplayEntry[]` that REPLACES the script derived from `session.jsonl`)
|
||||
* — the throw/hang cases chunks cannot express. The fixture guard requires
|
||||
* the sidecar exactly when this is set: the harness forwards the file purely
|
||||
* on existence, so an unregistered stray sidecar would silently replace the
|
||||
* derived script — the guard fails loud on either mismatch. Defaults to
|
||||
* false (replay derives from the fixture's `assistant/chunk` events).
|
||||
* (a `ReplayOverrideDoc` that replaces or patches the script derived from
|
||||
* `session.jsonl`) — the throw/hang cases chunks cannot express. The fixture
|
||||
* guard requires the sidecar exactly when this is set: the harness forwards
|
||||
* the file purely on existence, so an unregistered stray sidecar would
|
||||
* silently alter the derived script. The guard fails loud on either
|
||||
* mismatch. Defaults to false (replay derives from the fixture's
|
||||
* `assistant/chunk` events).
|
||||
*/
|
||||
overridden?: boolean
|
||||
/**
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: f184e271ff9e68760db43cfe79d4f39be81ef00f
|
||||
README.zh.md: a47bc81ab747fcdc130d535e116979e45304b319
|
||||
README.md: ce0758641f3d49a54b29415ed449e43043840f9a
|
||||
README.zh.md: 47a2b9aa211b44c4e476a1adf5a9a72d927cd0ed
|
||||
|
||||
@@ -10,7 +10,7 @@ Its consumers are the ACP, headless `stream-json`, and TUI snapshot suites plus
|
||||
|
||||
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
|
||||
|
||||
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`) that either REPLACES the derived script (a bare `ReplayEntry[]`) or AUGMENTS it (`{ patches: [{ at, entry }] }`: keep every JSONL-derived call, swap only the named 0-based call indexes; `at` equal to the derived length appends — the slot for the retry attempt that follows an injected transient throw). A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
|
||||
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`) that either replaces the derived script (a bare `ReplayEntry[]`) or augments it (`{ patches: [{ at, entry }] }`: keep every JSONL-derived call and swap the named 0-based call indexes; `at` equal to the derived length appends the retry attempt after an injected transient throw). Patch indexes must be unique. The override document, each patch and entry, and every chunk discriminant are validated when the file loads. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
|
||||
|
||||
## Nested agents: per-session keying
|
||||
|
||||
@@ -23,7 +23,7 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
| Key | Type | Default | Notes |
|
||||
|---|---|---|---|
|
||||
| `file` | string | `$DSH_SNAPSHOT_FILE` | Path to the primary (parent) `session.jsonl` fixture. Required (config or env). |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional path to a `ReplayEntry[]` sidecar that replaces the PRIMARY session's derived script. |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional `ReplayOverrideDoc` sidecar for the primary session: a bare `ReplayEntry[]` replaces its derived script, while `{ patches }` augments it by call index. |
|
||||
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | Recorded subagent child-session logs for a nested scenario; empty for a single-session scenario. |
|
||||
| `providers` | `ReplayProviderConfig[]` | — | Optional replay-only provider and model catalog. Each model may publish `contextWindow`; configured routes dispatch through the replay adapter and never perform provider I/O. |
|
||||
| `paceMs` | number | — (burst) | Optional per-chunk delay in ms so downstream transports (e.g. the web SSE mux observed by a real browser) see genuinely incremental delivery. A realism knob only — tests must not depend on it for correctness. Non-negative integer; abort during a pace wait cancels the stream promptly. |
|
||||
@@ -48,9 +48,9 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
|
||||
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns a `ReplayHandle` (`dispose()` for HMR safety plus `assertConsumed()`, the teardown check that every recorded script bound to a live session and every bound cursor drained — turning a scenario that silently drove fewer model calls than recorded into a crisp diagnostic). Use this in tests to drive replay without the Loader or env vars.
|
||||
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
|
||||
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the PRIMARY session only (sidecar override if present, else derived from the JSONL; fail-loud if the fixture is missing).
|
||||
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the primary session only (validated sidecar replacement/patches if present, else derived from the JSONL; fail-loud if the fixture is missing).
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
|
||||
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
|
||||
- Types `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
|
||||
|
||||
## Plugin export shape
|
||||
|
||||
@@ -67,4 +67,4 @@ None; this package neither assembles nor sends a provider request.
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **First-call-order script binding assumes sequential delegation** — a cut that runs sibling subagents concurrently (or a compaction summarize call landing mid-run) would bind live sessions to recorded scripts non-deterministically; a stronger keying is deferred until such a scenario exists (`XXX(concurrent-subagents)`).
|
||||
- **Only chunk-producing calls are derivable** — a pure pre-chunk throw or a cancel/hang scenario needs the `replay.override.json` sidecar; the override replaces the PRIMARY session's script only.
|
||||
- **Only chunk-producing calls are derivable** — a pure pre-chunk throw or a cancel/hang scenario needs the `replay.override.json` sidecar. Replacement and patch forms affect only the primary session; child scripts still derive from their logs.
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
|
||||
Fixture 就是持久化会话日志(`<scenario>/session.jsonl`)。其 `assistant/chunk` 事件携带每个 `StreamChunk`,因此按 `(turn, step)` 对其分组可重建每次 `stream()` 调用的分片序列(每个 loop 步骤一次模型调用)。因此,录制操作是「运行一次真实 agent 并收集 `.jsonl`」,由快照 harness 完成;该插件不执行录制。Fixture 的 `request/header` 内容可能被 token 化为 `{{system}}`/`{{tools}}`(harness 在一个场景中固定该内容,并擦除其余场景);回放对此并不关心,因为派生只读取 `assistant/chunk` 事件和第 0 行会话 header。
|
||||
|
||||
有两种失败 mode 无法仅从 `assistant/chunk` 重建:在任何分片前纯抛出(例如 HTTP 401,日志只包含 `turn/end {error}` 而没有分片),以及 cancel/hang(是时序,而非分片内容)。需要这些的场景提供可选 sidecar(`<scenario>/replay.override.json`),它要么替换派生脚本(裸 `ReplayEntry[]`),要么增补派生脚本(`{ patches: [{ at, entry }] }`:保留全部由 JSONL 派生的调用,仅在点名的调用索引处换入,索引从 0 计;`at` 等于派生长度时为追加,正是注入的瞬态抛出之后那次重试尝试所占的槽位)。`hang` 条目可以指定 `readyFile`;在其前缀分片到达 loop 后、等待取消前,回放会写入该空标记,使外部驱动器可以在不观察展示更新的情况下确定性取消。
|
||||
有两种失败 mode 无法仅从 `assistant/chunk` 重建:在任何分片前纯抛出(例如 HTTP 401,日志只包含 `turn/end {error}` 而没有分片),以及 cancel/hang(是时序,而非分片内容)。需要这些的场景提供可选 sidecar(`<scenario>/replay.override.json`),它要么替换派生脚本(裸 `ReplayEntry[]`),要么增补派生脚本(`{ patches: [{ at, entry }] }`:保留全部由 JSONL 派生的调用,仅在点名的调用索引处换入,索引从 0 计;`at` 等于派生长度时为追加,正是注入的瞬态抛出之后那次重试尝试所占的槽位)。Patch 索引必须互不重复。覆写文档、每个 patch 与每个条目,以及每个分片的判别字段都会在文件加载时接受校验。`hang` 条目可以指定 `readyFile`;在其前缀分片到达 loop 后、等待取消前,回放会写入该空标记,使外部驱动器可以在不观察展示更新的情况下确定性取消。
|
||||
|
||||
## 嵌套 agent:每会话键控
|
||||
|
||||
@@ -23,7 +23,7 @@ Fixture 就是持久化会话日志(`<scenario>/session.jsonl`)。其 `assis
|
||||
| 键 | 类型 | 默认值 | 说明 |
|
||||
|---|---|---|---|
|
||||
| `file` | string | `$DSH_SNAPSHOT_FILE` | 主(父)`session.jsonl` fixture 的路径。必需(配置或 env)。 |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | 替换主会话派生脚本的 `ReplayEntry[]` sidecar 可选路径。 |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | 主会话的可选 `ReplayOverrideDoc` sidecar:裸 `ReplayEntry[]` 替换其派生脚本,`{ patches }` 则按调用索引增补该脚本。 |
|
||||
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | 嵌套场景中已记录的 subagent 子会话日志;单会话场景为空。 |
|
||||
| `providers` | `ReplayProviderConfig[]` | 无 | 可选的仅回放提供方和模型目录。每个模型可以发布 `contextWindow`;已配置路由通过回放适配器分派,绝不执行提供方 I/O。 |
|
||||
| `paceMs` | number | 无(突发) | 可选的每分片毫秒延迟,使下游传输(例如真实浏览器观察的 web SSE mux)看到真正的增量传递。它只是仿真开关,测试不得依赖它保证正确性。值必须是非负整数;pace 等待期间中止会迅速取消流。 |
|
||||
@@ -48,9 +48,9 @@ Fixture 就是持久化会话日志(`<scenario>/session.jsonl`)。其 `assis
|
||||
|
||||
- `installLlmReplay(ctx, config)`:安装已配置回放适配器或 catch-all `llm/stream` 监听器;返回 `ReplayHandle`(包含用于 HMR 安全的 `dispose()`,以及 `assertConsumed()` 拆卸检查;后者确保每个已记录脚本都绑定到实时会话,且每个已绑定游标都已耗尽,从而将场景静默驱动的模型调用少于记录数转换为明确诊断)。在测试中使用它,可以不通过 Loader 或 env var 驱动回放。
|
||||
- `loadSessionScripts(config)`:解析场景的有序 `SessionScript[]` (主级 + 子级),准备按首次调用顺序绑定到实时会话。
|
||||
- `loadReplayScript(config)`:只解析主会话的 `ReplayEntry[]` (如果存在则使用 sidecar override,否则从 JSONL 派生;fixture 缺失时快速失败)。
|
||||
- `loadReplayScript(config)`:只解析主会话的 `ReplayEntry[]` (如果存在则使用经校验的 sidecar 替换或 patch,否则从 JSONL 派生;fixture 缺失时快速失败)。
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)`:将已记录会话日志转换为脚本并读取其 header `id`/`createdAt` 的纯辅助工具。派生分组必须以 `finish` 分片结束;没有该分片的分组是已抛出 `stream()` 的指纹,必须改用 override sidecar 表达。
|
||||
- 类型 `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`。
|
||||
- 类型 `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`。
|
||||
|
||||
## 插件导出形态
|
||||
|
||||
@@ -67,4 +67,4 @@ Fixture 就是持久化会话日志(`<scenario>/session.jsonl`)。其 `assis
|
||||
## 已知限制与待完成工作
|
||||
|
||||
- **首次调用顺序脚本绑定假设串行委托**:并发运行同级 subagent 的 cut(或运行中落地的压缩摘要调用)会非确定性地将实时会话绑定到已记录脚本;在这种场景出现前暂不实现更强的键控(`XXX(concurrent-subagents)`)。
|
||||
- **只有生产分片的调用可派生**:纯分片前抛出或 cancel/hang 场景需要 `replay.override.json` sidecar;override 只替换主会话的脚本。
|
||||
- **只有生产分片的调用可派生**:纯分片前抛出或 cancel/hang 场景需要 `replay.override.json` sidecar。替换和 patch 两种形式都只影响主会话;子会话脚本仍从各自日志派生。
|
||||
|
||||
@@ -59,7 +59,7 @@ export interface ReplayConfig {
|
||||
*/
|
||||
file: string
|
||||
/**
|
||||
* Optional sidecar for the PRIMARY session: a bare `ReplayEntry[]` REPLACES
|
||||
* Optional sidecar for the PRIMARY session: a bare `ReplayEntry[]` replaces
|
||||
* the derived script; `{ patches }` keeps it and swaps the named call
|
||||
* indexes ({@link ReplayOverrideDoc}). Used by single-session scenarios not
|
||||
* expressible as `assistant/chunk` (throw-before-chunk, cancel/hang,
|
||||
@@ -214,13 +214,105 @@ export interface ReplayOverridePatch {
|
||||
}
|
||||
|
||||
/**
|
||||
* Override sidecar document: either the legacy whole-script replacement (a
|
||||
* Override sidecar document: either a whole-script replacement (a
|
||||
* bare `ReplayEntry[]`) or the augmentation form `{ patches }`, which keeps
|
||||
* the JSONL-derived script and swaps only the named call indexes — the shape
|
||||
* for "turn N errors, everything else replays as recorded".
|
||||
*/
|
||||
export type ReplayOverrideDoc = ReplayEntry[] | { patches: ReplayOverridePatch[] }
|
||||
|
||||
const REPLAY_CHUNK_TYPES = new Set<StreamChunk['type']>([
|
||||
'block-start',
|
||||
'text-delta',
|
||||
'reasoning-delta',
|
||||
'tool-call-delta',
|
||||
'block-end',
|
||||
'usage',
|
||||
'finish',
|
||||
])
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === 'object' && value !== null && !Array.isArray(value)
|
||||
}
|
||||
|
||||
function hasExactKeys(value: Record<string, unknown>, keys: readonly string[]): boolean {
|
||||
return Object.keys(value).length === keys.length && keys.every(key => Object.hasOwn(value, key))
|
||||
}
|
||||
|
||||
function invalidOverride(file: string, location: string, detail: string): never {
|
||||
throw new Error(`llm-replay: invalid override ${file}: ${location} ${detail}`)
|
||||
}
|
||||
|
||||
function readChunks(value: unknown, file: string, location: string): StreamChunk[] {
|
||||
if (!Array.isArray(value)) invalidOverride(file, location, 'chunks must be an array')
|
||||
for (const [index, chunk] of value.entries()) {
|
||||
if (!isRecord(chunk)
|
||||
|| typeof chunk['type'] !== 'string'
|
||||
|| !REPLAY_CHUNK_TYPES.has(chunk['type'] as StreamChunk['type'])) {
|
||||
invalidOverride(file, `${location}.chunks[${index}]`, 'must have a known StreamChunk type')
|
||||
}
|
||||
}
|
||||
return value as StreamChunk[]
|
||||
}
|
||||
|
||||
function readReplayEntry(value: unknown, file: string, location: string): ReplayEntry {
|
||||
if (!isRecord(value)) invalidOverride(file, location, 'must be an object')
|
||||
switch (value['kind']) {
|
||||
case 'chunks': {
|
||||
if (!hasExactKeys(value, ['kind', 'chunks'])) invalidOverride(file, location, 'has invalid chunks-entry fields')
|
||||
return { kind: 'chunks', chunks: readChunks(value['chunks'], file, location) }
|
||||
}
|
||||
case 'throw': {
|
||||
if (!hasExactKeys(value, ['kind', 'chunks', 'message', 'code'])) {
|
||||
invalidOverride(file, location, 'has invalid throw-entry fields')
|
||||
}
|
||||
if (typeof value['message'] !== 'string' || value['message'].length === 0) {
|
||||
invalidOverride(file, location, 'message must be a non-empty string')
|
||||
}
|
||||
if (typeof value['code'] !== 'string' || value['code'].length === 0) {
|
||||
invalidOverride(file, location, 'code must be a non-empty string')
|
||||
}
|
||||
return {
|
||||
kind: 'throw',
|
||||
chunks: readChunks(value['chunks'], file, location),
|
||||
message: value['message'],
|
||||
code: value['code'],
|
||||
}
|
||||
}
|
||||
case 'hang': {
|
||||
const readyFile = value['readyFile']
|
||||
const keys = readyFile === undefined ? ['kind'] : ['kind', 'readyFile']
|
||||
if (!hasExactKeys(value, keys)) invalidOverride(file, location, 'has invalid hang-entry fields')
|
||||
if (readyFile !== undefined && (typeof readyFile !== 'string' || readyFile.length === 0)) {
|
||||
invalidOverride(file, location, 'readyFile must be a non-empty string')
|
||||
}
|
||||
return { kind: 'hang', ...(readyFile === undefined ? {} : { readyFile }) }
|
||||
}
|
||||
default:
|
||||
return invalidOverride(file, location, `has unknown kind ${JSON.stringify(value['kind'])}`)
|
||||
}
|
||||
}
|
||||
|
||||
function readOverrideDoc(value: unknown, file: string): ReplayOverrideDoc {
|
||||
if (Array.isArray(value)) return value.map((entry, index) => readReplayEntry(entry, file, `entry ${index}`))
|
||||
if (!isRecord(value) || !hasExactKeys(value, ['patches']) || !Array.isArray(value['patches'])) {
|
||||
return invalidOverride(file, 'document', 'must be a ReplayEntry[] or { patches: [...] }')
|
||||
}
|
||||
return {
|
||||
patches: value['patches'].map((value, index): ReplayOverridePatch => {
|
||||
const location = `patch ${index}`
|
||||
if (!isRecord(value) || !hasExactKeys(value, ['at', 'entry'])) {
|
||||
return invalidOverride(file, location, 'must contain exactly at and entry')
|
||||
}
|
||||
const at = value['at']
|
||||
if (typeof at !== 'number' || !Number.isSafeInteger(at) || at < 0) {
|
||||
return invalidOverride(file, location, 'at must be a non-negative safe integer')
|
||||
}
|
||||
return { at, entry: readReplayEntry(value['entry'], file, `${location}.entry`) }
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Load the PRIMARY session's replay script: the sidecar override when present
|
||||
* (whole-script replacement or `{ patches }` augmentation over the derived
|
||||
@@ -231,20 +323,22 @@ export type ReplayOverrideDoc = ReplayEntry[] | { patches: ReplayOverridePatch[]
|
||||
*/
|
||||
export function loadReplayScript(config: ReplayConfig): ReplayEntry[] {
|
||||
if (config.overrideFile !== undefined && existsSync(config.overrideFile)) {
|
||||
const parsed: unknown = JSON.parse(readFileSync(config.overrideFile, 'utf8'))
|
||||
if (Array.isArray(parsed)) return parsed as ReplayEntry[]
|
||||
const doc = parsed as { patches?: unknown }
|
||||
if (typeof parsed !== 'object' || parsed === null || !Array.isArray(doc.patches)) {
|
||||
throw new Error(`llm-replay: override must be a ReplayEntry[] or { patches: [...] }: ${config.overrideFile}`)
|
||||
}
|
||||
const doc = readOverrideDoc(JSON.parse(readFileSync(config.overrideFile, 'utf8')) as unknown, config.overrideFile)
|
||||
if (Array.isArray(doc)) return doc
|
||||
const script = deriveScriptFromFile(config.file)
|
||||
for (const patch of doc.patches as ReplayOverridePatch[]) {
|
||||
if (!Number.isInteger(patch.at) || patch.at < 0 || patch.at > script.length) {
|
||||
const derivedLength = script.length
|
||||
const seenIndexes = new Set<number>()
|
||||
for (const patch of doc.patches) {
|
||||
if (patch.at > derivedLength) {
|
||||
throw new Error(
|
||||
`llm-replay: override patch index ${String(patch.at)} out of range `
|
||||
+ `(derived script has ${script.length} call(s); == length appends): ${config.overrideFile}`,
|
||||
+ `(derived script has ${derivedLength} call(s); == length appends): ${config.overrideFile}`,
|
||||
)
|
||||
}
|
||||
if (seenIndexes.has(patch.at)) {
|
||||
throw new Error(`llm-replay: duplicate override patch index ${patch.at}: ${config.overrideFile}`)
|
||||
}
|
||||
seenIndexes.add(patch.at)
|
||||
script[patch.at] = patch.entry
|
||||
}
|
||||
return script
|
||||
@@ -397,9 +491,8 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined,
|
||||
})
|
||||
/* v8 ignore next -- unreachable: the hang promise only ever rejects (on abort), never resolves; control never reaches here */
|
||||
return
|
||||
/* v8 ignore next -- sidecar entries are validated before they reach the closed local union. */
|
||||
default:
|
||||
// Closed local union: an unknown kind means malformed (hand-edited or
|
||||
// drifted) sidecar data — fail loud with a runtime diagnostic.
|
||||
return assertNever(entry, 'llm-replay replay entry')
|
||||
}
|
||||
}
|
||||
|
||||
@@ -203,11 +203,11 @@ describe('loadReplayScript', () => {
|
||||
expect(() => loadReplayScript({ file: join(dir, 'absent.jsonl') })).toThrow(/fixture not found/)
|
||||
})
|
||||
|
||||
it('throws when the override is not a JSON array', () => {
|
||||
it('rejects an override document that is neither supported form', () => {
|
||||
writeFileSync(file, sessionJsonl([]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
writeFileSync(overrideFile, '{"not":"array"}', 'utf8')
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/ReplayEntry\[\] or \{ patches/)
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/document must be a ReplayEntry\[\] or \{ patches/)
|
||||
})
|
||||
|
||||
it('patches form: swaps the named call index and keeps derived siblings', () => {
|
||||
@@ -249,11 +249,46 @@ describe('loadReplayScript', () => {
|
||||
it('patches form: an out-of-range index fails loud with the derived length', () => {
|
||||
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
for (const at of [2, -1, 1.5]) {
|
||||
writeFileSync(overrideFile, JSON.stringify({ patches: [{ at, entry: { kind: 'hang' } }] }), 'utf8')
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/patch index .* out of range/)
|
||||
writeFileSync(overrideFile, JSON.stringify({ patches: [{ at: 2, entry: { kind: 'hang' } }] }), 'utf8')
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/patch index 2 out of range.*1 call/s)
|
||||
})
|
||||
|
||||
it('validates patch and entry shapes at the file boundary', () => {
|
||||
writeFileSync(file, sessionJsonl([]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
const invalid: Array<{ doc: unknown; message: RegExp }> = [
|
||||
{ doc: null, message: /document must be/ },
|
||||
{ doc: { patches: [null] }, message: /patch 0 must contain exactly at and entry/ },
|
||||
{ doc: { patches: [{ at: -1, entry: { kind: 'hang' } }] }, message: /at must be a non-negative safe integer/ },
|
||||
{ doc: { patches: [{ at: 1.5, entry: { kind: 'hang' } }] }, message: /at must be a non-negative safe integer/ },
|
||||
{ doc: [42], message: /entry 0 must be an object/ },
|
||||
{ doc: [{ kind: 'chunks', chunks: 'nope' }], message: /chunks must be an array/ },
|
||||
{ doc: [{ kind: 'chunks', chunks: [], extra: true }], message: /invalid chunks-entry fields/ },
|
||||
{ doc: [{ kind: 'chunks', chunks: [{ type: 'bogus' }] }], message: /known StreamChunk type/ },
|
||||
{ doc: [{ kind: 'throw', chunks: [], message: 'nope', code: 'AUTH', extra: true }], message: /invalid throw-entry fields/ },
|
||||
{ doc: [{ kind: 'throw', chunks: [], message: '', code: 'AUTH' }], message: /message must be a non-empty string/ },
|
||||
{ doc: [{ kind: 'throw', chunks: [], message: 'nope', code: '' }], message: /code must be a non-empty string/ },
|
||||
{ doc: [{ kind: 'hang', extra: true }], message: /invalid hang-entry fields/ },
|
||||
{ doc: [{ kind: 'hang', readyFile: 1 }], message: /readyFile must be a non-empty string/ },
|
||||
{ doc: [{ kind: 'bogus' }], message: /unknown kind/ },
|
||||
]
|
||||
for (const { doc, message } of invalid) {
|
||||
writeFileSync(overrideFile, JSON.stringify(doc), 'utf8')
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(message)
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects duplicate patch indexes instead of silently taking the last one', () => {
|
||||
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
writeFileSync(overrideFile, JSON.stringify({
|
||||
patches: [
|
||||
{ at: 0, entry: { kind: 'hang' } },
|
||||
{ at: 0, entry: { kind: 'throw', chunks: [], message: 'busy', code: 'SERVER' } },
|
||||
],
|
||||
}), 'utf8')
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/duplicate override patch index 0/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('installLlmReplay (through the real LlmService)', () => {
|
||||
@@ -409,16 +444,14 @@ describe('installLlmReplay (through the real LlmService)', () => {
|
||||
.toEqual([{ type: 'finish', reason: { kind: 'stop' } }])
|
||||
})
|
||||
|
||||
it('throws on a malformed sidecar entry kind (the assertNever guard)', async () => {
|
||||
it('rejects a malformed sidecar entry kind before installing replay', async () => {
|
||||
writeFileSync(file, sessionJsonl([]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
// A kind the union does not know — hand-edited/drifted sidecar data.
|
||||
writeFileSync(overrideFile, JSON.stringify([{ kind: 'bogus' }]), 'utf8')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file, overrideFile })
|
||||
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })))
|
||||
.rejects.toThrow(/llm-replay replay entry/)
|
||||
expect(() => installLlmReplay(ctx, { file, overrideFile })).toThrow(/unknown kind/)
|
||||
})
|
||||
|
||||
it('rejects a hang entry when the signal fires DURING the wait (abort listener path)', async () => {
|
||||
|
||||
@@ -998,8 +998,9 @@ describe('resume command and /resume', () => {
|
||||
await tick(); await tick()
|
||||
result.terminal.send('Fallback target')
|
||||
result.terminal.send('\r')
|
||||
await tick()
|
||||
expect(result.terminal.output).toContain('This host cannot hand off in place. Exit and run:')
|
||||
await vi.waitFor(() => {
|
||||
expect(result.terminal.output).toContain('This host cannot hand off in place. Exit and run:')
|
||||
})
|
||||
expect(result.terminal.output).toContain('dsh --resume fallback-session')
|
||||
expect(result.terminal.stopped).toBe(0)
|
||||
await dispose(result)
|
||||
@@ -1019,8 +1020,9 @@ describe('resume command and /resume', () => {
|
||||
await tick(); await tick()
|
||||
result.terminal.send('No fallback target')
|
||||
result.terminal.send('\r')
|
||||
await tick()
|
||||
expect(result.terminal.output).toContain('Session is resumable, but this host cannot hand it off in place')
|
||||
await vi.waitFor(() => {
|
||||
expect(result.terminal.output).toContain('Session is resumable, but this host cannot hand it off in place')
|
||||
})
|
||||
await dispose(result)
|
||||
})
|
||||
|
||||
|
||||
Reference in New Issue
Block a user