diff --git a/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.i18n.yaml b/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.i18n.yaml index 22cf5283e3..bfd2d60086 100644 --- a/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.i18n.yaml +++ b/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.md -2026-09-04-session-open-performance-gate.md: 2820c9d7d0e5b7d9382c7f8d6540154440175f26 -2026-09-04-session-open-performance-gate.zh.md: 965b9035074504870bcb2f1ca8166962c264d75a +2026-09-04-session-open-performance-gate.md: 2937a2aec1dbddb31fde82d2617d69852a611d90 +2026-09-04-session-open-performance-gate.zh.md: b6608ca79d07c3e9fb00d62801038ecefbdf944c diff --git a/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.md b/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.md index 2820c9d7d0..2937a2aec1 100644 --- a/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.md +++ b/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.md @@ -12,13 +12,13 @@ Measuring only `SessionPersistence.open()` does not stably describe the result f ## Decision -Linux pull requests run a required `node 24 / benchmarks` job that executes `pnpm run check:ci:bench` → `pnpm run test:bench`. The private `@deepseek-ai/dsh-benchmarks` workspace owns benchmark-only dependencies. The command first builds workspace libraries and dedicated workers under `benchmarks/.dsh-build/`, then invokes `vitest.bench.config.ts`. The [standard hosted runner decision](2026-09-06-standard-hosted-benchmark-runner.md) owns runner selection and the outer job timeout. The job runs the benchmark lane alone; Vitest runs one file at a time and only prepares input, starts measurement children, aggregates results, and enforces budgets. Every timed CPU path executes compiled JavaScript under plain Node with `NODE_OPTIONS` removed and no TypeScript loader; bare workspace imports therefore resolve from `benchmarks/node_modules` through package exports to built `lib/` entries. +Linux pull requests run a required `node 24 / benchmarks` job that executes `pnpm run check:ci:bench` → `pnpm run test:bench`. The private `@deepseek-ai/dsh-benchmarks` workspace owns benchmark-only dependencies. The command first builds workspace libraries and dedicated workers under `benchmarks/.dsh-build/`, then invokes `vitest.bench.config.ts`. The [standard hosted runner decision](2026-09-06-standard-hosted-benchmark-runner.md) owns runner selection and the outer job timeout. The job runs the benchmark lane alone; Vitest runs one file at a time and only prepares input, starts measurement children, aggregates results, and enforces budgets. Every timed Node CPU path executes compiled JavaScript under plain Node with `NODE_OPTIONS` removed and no TypeScript loader; bare workspace imports therefore resolve from `benchmarks/node_modules` through package exports to built `lib/` entries. Required performance gates live under top-level `benchmarks/`, grouped by measured user path rather than package ownership. Host files use `*.bench.ts`, Client-face files use `*.bench.client.ts`, and scenario-specific workers and fixtures stay beside their benchmark without a benchmark suffix. Package-local `.perf.ts` files remain non-gating diagnostics; `scripts/` owns orchestration rather than benchmark cases. The Session benchmarks synthesize a released-v0 input from fixed parameters: 200 turns with 500 text deltas and 125 reasoning deltas per turn, for 127,400 logical events. The input uses Zstandard with fixed logical-row grouping and frame partitioning, so every run processes the same events, bytes, and frame distribution. The fixture constructs the immutable released-v0 physical rows directly instead of depending on a current-runtime historical encoder; compression and every measured read or migration entry point still use production code. Setup writes the input into a private temporary directory for each sample before timing starts; benchmarks never use recorded Sessions. -Every Session endpoint runs at two user-lifecycle points. `first-open` starts with only the released V0 generation and therefore includes migration and successor publication. Setup produces `post-upgrade-reopen` once through that same production migration outside measurement, then copies both the unchanged V0 predecessor and published V2 successor into each sample root. Reopen samples use a fresh process, so they measure an upgraded user's later disk open without migration or process-local caches. +Every Session endpoint runs at two user-lifecycle points. `first-open` starts with only the released V0 generation and includes migration; read-only consumers do not publish a successor, while writable Agent resume does. Setup produces `post-upgrade-reopen` once through that same production migration outside measurement, then copies both the unchanged V0 predecessor and published V2 successor into each sample root. Reopen samples use a fresh process, so they measure an upgraded user's later disk open without migration or process-local caches. Each access-kind and endpoint sample runs in a fresh compiled Node child process. Module imports, Host service initialization, and fixture preparation finish before measurement; the measured process performs no extra parse warm-up. Normal-heap mode runs five independent samples, reports every sample plus minimum, median, and maximum, and enforces access-specific fixed budgets against the median. Another child runs the same path under a fixed 128 MB old-space limit and checks only that it completes; extra GC caused by the constrained heap does not enter the normal timing baseline. @@ -26,7 +26,7 @@ The lane contains three independent Session-opening benchmarks and retains the C | Benchmark | Measured path | Timing metrics | |---|---|---| -| Phase profile | Executes the real persistence open, handle read, Session restore, and projection for both first open and post-upgrade reopen | `openMs`, `readMs`, `sessionRestoreMs`, and `projectionMs` each have a fixed budget; encoding, writes, verification, and publication awaited by migration all belong to first-open `openMs` | +| Phase profile | Executes the real persistence open, handle read, Session restore, and projection for both first open and post-upgrade reopen | `openMs`, `readMs`, `sessionRestoreMs`, and `projectionMs` each have a fixed budget; read-only migration belongs to first-open `openMs`; successor encoding, verification, and publication belong to writable Agent resume | | First history | Reads each access kind through the Host Session history controller until it produces the first paginated snapshot | Separate first-open and reopen end-to-end budgets; each includes source stat, reading, restoration, projection, pagination, and snapshot construction, while first open additionally includes migration; both exclude Gateway network transport, Client fold, and browser paint | | Agent resume | Calls `ctx.agents.resume()` for each access kind until Agent creation, setup, publication, and loop startup finish | Separate first-open and reopen end-to-end budgets; neither path runs after first-history or reuses that benchmark's cache | | Client fold | Folds small and large v2 history windows through the real `ConversationNodeAssembler` and every Chat Definition | The large window's absolute time and scaling relative to the small window each have a fixed budget | @@ -102,4 +102,4 @@ The calibrated source budgets are: Every pull request pays for one required Linux job; its Session portion runs several short-lived child processes in exchange for cold caches, isolated V8 heaps, explicit GC state, and attributable failures. The repository-level benchmark tree accepts deliberate cross-package test dependencies without changing product package manifests. The fixed Zstandard workload covers both event volume and frame topology; first-open measurements protect the one-time upgrade experience, reopen measurements prevent regressions in later opens, phase budgets locate cost, first-history budgets protect user-visible waiting, Agent-resume budgets and post-GC deltas protect complete cold activation and resident memory, and the 128 MB mode protects the transient allocation ceiling. -The gate does not measure network transfer, browser rendering, or recorded Sessions, and it is not a continuous performance-trend system. A Node or runner change requires resampling the same workload and reviewing the budgets; a business-implementation change must not relax a budget without new positive and negative control data. +The Session and Node-fold scenarios do not measure network transfer, browser rendering, or recorded Sessions, and they are not a continuous performance-trend system. [Frontend performance budgets](2026-09-06-frontend-performance-budgets.md) own browser workflow measurements. A Node or runner change requires resampling the same workload and reviewing the budgets; a business-implementation change must not relax a budget without new positive and negative control data. diff --git a/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.zh.md b/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.zh.md index 965b903507..b6608ca79d 100644 --- a/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.zh.md +++ b/.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.zh.md @@ -12,13 +12,13 @@ Session format v2 的推出改变了两条成本随模型输出增长的路径 ## 决定 -Linux pull request 运行必需的 `node 24 / benchmarks` job,执行 `pnpm run check:ci:bench` → `pnpm run test:bench`。私有 `@deepseek-ai/dsh-benchmarks` workspace 拥有 benchmark 专属依赖。该命令先构建 workspace library 和 `benchmarks/.dsh-build/` 下的专用 worker,再调用 `vitest.bench.config.ts`。[标准托管运行器决策](2026-09-06-standard-hosted-benchmark-runner.zh.md)拥有运行器选择及外层 job 超时。该 job 单独运行 benchmark lane;Vitest 逐文件运行,只负责准备输入、启动测量子进程、汇总结果和执行预算断言。每条被计时的 CPU 路径都以纯 Node 执行编译后的 JavaScript,并移除 `NODE_OPTIONS` 且不加载 TypeScript runtime;workspace 裸导入因此从 `benchmarks/node_modules` 通过 package exports 解析到构建后的 `lib/` 入口。 +Linux pull request 运行必需的 `node 24 / benchmarks` job,执行 `pnpm run check:ci:bench` → `pnpm run test:bench`。私有 `@deepseek-ai/dsh-benchmarks` workspace 拥有 benchmark 专属依赖。该命令先构建 workspace library 和 `benchmarks/.dsh-build/` 下的专用 worker,再调用 `vitest.bench.config.ts`。[标准托管运行器决策](2026-09-06-standard-hosted-benchmark-runner.zh.md)拥有运行器选择及外层 job 超时。该 job 单独运行 benchmark lane;Vitest 逐文件运行,只负责准备输入、启动测量子进程、汇总结果和执行预算断言。每条被计时的 Node CPU 路径都以纯 Node 执行编译后的 JavaScript,并移除 `NODE_OPTIONS` 且不加载 TypeScript runtime;workspace 裸导入因此从 `benchmarks/node_modules` 通过 package exports 解析到构建后的 `lib/` 入口。 必需性能 gate 位于顶层 `benchmarks/`,按被测用户路径而非 package 归属组织。Host 文件使用 `*.bench.ts`,Client 面文件使用 `*.bench.client.ts`,场景专属 worker 与 fixture 留在对应 benchmark 旁且不带 benchmark 后缀。包内 `.perf.ts` 文件仍是非门禁诊断;`scripts/` 负责编排而不承载 benchmark case。 Session benchmark 使用固定参数合成 released-v0 输入:200 轮,每轮 500 个 text delta 与 125 个 reasoning delta,共 127,400 个逻辑事件。输入使用 Zstandard,并固定 logical rows 的分组与 frame 拆分,使每次运行处理相同的事件、字节与 frame 分布。fixture 直接构造不可变的 released-v0 physical rows,不依赖当前 runtime 的历史 encoder;压缩以及所有被测读取和 migration 入口仍使用生产代码。输入在计时前写入每个样本独占的临时目录;benchmark 不使用录制的 Session。 -每个 Session endpoint 都针对用户生命周期中的两个时点运行。`first-open` 最初只有 released V0 generation,因此包含 migration 与后继 generation 发布。测试准备阶段在计时外通过同一套生产 migration 生成一次 `post-upgrade-reopen`,再把未改动的 V0 前代和已发布的 V2 后继一起复制到每个样本目录。Reopen 样本使用全新进程,因此测量用户升级完成后的磁盘再次打开,不包含 migration 或进程内 cache。 +每个 Session endpoint 都针对用户生命周期中的两个时点运行。`first-open` 最初只有 released V0 generation,包含 migration;只读消费者不发布后继文件,可写 Agent resume 才会发布。测试准备阶段在计时外通过同一套生产 migration 生成一次 `post-upgrade-reopen`,再把未改动的 V0 前代和已发布的 V2 后继一起复制到每个样本目录。Reopen 样本使用全新进程,因此测量用户升级完成后的磁盘再次打开,不包含 migration 或进程内 cache。 每个 access kind 与 endpoint 的样本都在全新、已编译的 Node 子进程中运行。模块加载、Host 服务初始化和 fixture 准备在测量开始前完成;测量进程不执行额外的预热解析。正常堆模式运行五个独立样本,报告全部样本及最小值、中位数和最大值,并以中位数执行各访问状态独立的固定预算。另一个子进程使用固定 128 MB old-space 上限运行同一路径,只判断能否完成;低堆限制引起的额外 GC 不进入正常时间基线。 @@ -26,7 +26,7 @@ Session benchmark 使用固定参数合成 released-v0 输入:200 轮,每轮 | Benchmark | 被测路径 | 时间指标 | |---|---|---| -| 阶段剖面 | 分别为 first open 与 post-upgrade reopen 执行真实 persistence open、handle read、Session restore 与 projection | `openMs`、`readMs`、`sessionRestoreMs`、`projectionMs` 各自使用固定预算;migration 所等待的编码、写入、verify 与 publish 全部归入 first-open `openMs` | +| 阶段剖面 | 分别为 first open 与 post-upgrade reopen 执行真实 persistence open、handle read、Session restore 与 projection | `openMs`、`readMs`、`sessionRestoreMs`、`projectionMs` 各自使用固定预算;只读 migration 归入 first-open `openMs`;后继编码、verify 与 publish 属于可写 Agent resume | | 首屏历史 | 两种 access kind 分别经 Host Session history controller 读取到首个分页 snapshot | First open 与 reopen 各有一个端到端预算;均包含 source stat、读取、Session restore、projection、分页与 snapshot 构造,first open 还包含 migration;两者都不包含 Gateway 网络传输、Client fold 或浏览器 paint | | Agent resume | 对两种 access kind 分别调用 `ctx.agents.resume()`,直到 Agent 创建、setup、发布与 loop 启动完成 | First open 与 reopen 各有一个端到端预算;两条路径都不与首屏历史串行,也不依赖它留下的 cache | | Client fold | 大小两个 v2 history window 经真实 `ConversationNodeAssembler` 与全部 Chat Definition fold | 大窗口的绝对时间与相对小窗口的缩放比各自使用固定预算 | @@ -50,7 +50,7 @@ Session benchmark 使用固定参数合成 released-v0 输入:200 轮,每轮 | First open | 栈前参考版本 | 249.0 ms | 253.8 ms | 100.7 ms | 26.1 MB | 完成 | | First open | 重复 snapshot 退化实现 | 4,197.5 ms | 4,284.8 ms | 4,197.9 ms | 4.4 MB | 堆耗尽 | | Post-upgrade reopen | 栈前参考版本 | 251.1 ms | 253.8 ms | 100.7 ms | 26.1 MB | 完成 | -| Post-upgrade reopen | 重复 snapshot 退化实现 | 49.2 ms | 50.4 ms | 43.8 ms | 完成 | +| Post-upgrade reopen | 重复 snapshot 退化实现 | 49.2 ms | 50.4 ms | 43.8 ms | 4.5 MB | 完成 | 栈前实现以 V0 作为当前格式,因此 first open 不改变磁盘表示;它的原生 V0 首屏历史与 Agent resume 测量同时适用于两个生命周期行。 @@ -102,4 +102,4 @@ Session benchmark 使用固定参数合成 released-v0 输入:200 轮,每轮 每个 pull request 多付出一个必需 Linux job;该 job 的 Session 部分运行多个短生命周期子进程,以换取冷 cache、独立 V8 heap、明确 GC 状态和可归因的失败。仓库级 benchmark 目录接受有意的跨包测试依赖,而不修改产品 package manifest。固定 Zstandard workload 同时覆盖事件规模与 frame 拓扑;first-open 测量保护一次性升级体验,reopen 测量防止后续打开退化,四阶段预算定位成本归属,首屏预算保护用户可见等待,Agent resume 预算与 GC 后增量保护完整冷恢复及常驻内存,128 MB 模式保护瞬时分配上限。 -该 gate 不测量网络传输、浏览器渲染或真实录制 Session,也不是持续性能趋势系统。Node 或 runner 变化需要用同一 workload 重新采样并评审预算;修改业务实现时不得顺带放宽预算而不提供新的正反例数据。 +Session 与 Node-fold 场景不测量网络传输、浏览器渲染或真实录制 Session,也不是持续性能趋势系统。[前端性能预算](2026-09-06-frontend-performance-budgets.zh.md)拥有浏览器工作流测量。Node 或 runner 变化需要用同一 workload 重新采样并评审预算;修改业务实现时不得顺带放宽预算而不提供新的正反例数据。 diff --git a/.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.i18n.yaml b/.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.i18n.yaml new file mode 100644 index 0000000000..06aa949170 --- /dev/null +++ b/.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.md +2026-09-06-frontend-performance-budgets.md: 0ebe97db9532c4922d2e0e8f2bd41613b9e80b6e +2026-09-06-frontend-performance-budgets.zh.md: f58afd5aae763b145887204a63dd5b90d16566b4 diff --git a/.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.md b/.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.md new file mode 100644 index 0000000000..0ebe97db95 --- /dev/null +++ b/.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.md @@ -0,0 +1,55 @@ +# Agent Note: Frontend large-session performance budgets + +Status: implemented + +English | [中文](2026-09-06-frontend-performance-budgets.zh.md) + +## Problem + +A fast Node conversation fold does not prove that a browser paints a long conversation or remains responsive while a response streams. Active reconnect also reconstructs a different representation from settled history: a compact prefix becomes public per-chunk Client entries. The [Session performance policy](2026-09-04-session-open-performance-gate.md) supplies an isolated CI job but does not measure these user paths. + +## Decision + +The existing serial benchmark inventory includes two frontend owners: [active reconnect](../../../../benchmarks/active-stream-reconnect/README.md) and a [browser workflow](../../../../benchmarks/long-session-browser/README.md). The browser workflow combines cold open, older-page navigation, first Trajectory activation, return to Chat, and a paced response with trusted keyboard input into one sequential scenario. These are endpoints of one workflow, not independent cold scenarios. The settled conversation-fold benchmark remains unchanged. + +`build:bench` keeps the Node-only library and worker build. `test:bench` additionally builds the Web shell before running all cases; the required benchmark CI job provisions Chromium. Browser cases reuse the shipped-composition Web scaffold with private temporary roots and an atomically assigned loopback port. Only the nondeterministic model is replaced by synthetic replay. The scaffold Host runs under the existing Vitest source resolver; measured Client rendering runs built bundles in fresh Chromium processes. Browser wall times therefore include this test Host, transport, Playwright actionability, and rendering, and are not claims about a published Host process. + +The browser input contains 240 closed turns, 40 tool results, and 20 code fences, plus mixed-language prose and reasoning. Nine older-page actions exhaust this input from its observed 25-turn initial window; the readiness probe follows mounted turn growth rather than duplicating the pagination algorithm. Each sample uses a fresh scaffold and browser. Setup, seeding, browser launch, initial shell load, and sidebar expansion are excluded from open timing. Open ends at transcript availability and an editable composer; page and navigation timings end at their target DOM state. Two animation frames include a rendering opportunity, not hardware presentation or a guarantee that every offscreen node painted. + +The continuation sends 120 text deltas at 8 ms replay pacing. It records click-to-first-visible-reply, trusted draft typing while the completion marker is absent, complete reply wall time through settled persistence, and Chromium main-thread task duration. The complete wall budget adds the fixed 992 ms scripted pacing to a scaled overhead allowance; input and completion have their own enforced budgets. Post-GC browser heap and DOM counts remain diagnostics because one endpoint does not prove a leak. + +Reconnect uses three fresh compiled plain-Node children. Each creates a 100,000-delta reasoning prefix with distinct timestamps and two compact records before timing `ClientAssistantStream.replace()`. GC precedes the baseline and follows replacement while the result remains reachable; replacement time excludes both collections. The report consumes the result after collection and checks that the next dense live frame remains accepted. This measures reconstruction, not transport, rendering, or an entire reconnect workflow. + +## Calibration + +Three-sample medians on the arm64 reference machine, Node 24.19 and Chromium 149.0.7827.55, at product revision `925e012340`, establish the baseline below. An isolated repeat follows a complete workflow smoke. Each browser sample reports raw endpoint values and every page; the paging verdict uses the median of the sample maxima. Reconnect reports all child measurements. Source reference constants round above observed values; the shared 2× time scale and 1.25× variance allowance produce CI limits. Memory uses only variance allowance. The existing scale comes from Node CI calibration, not a measured x64 browser comparison; browser-specific runner calibration remains an explicit gap. + +| Endpoint | Measured median | Reference allowance | CI limit | +|---|---:|---:|---:| +| Browser open | 166.77 ms | 200 ms | 500 ms | +| Slowest older page | 245.52 ms | 260 ms | 650 ms | +| First Trajectory | 133.45 ms | 160 ms | 400 ms | +| First reply | 1033.07 ms | 1100 ms | 2750 ms | +| Stream main-thread task | 1668.46 ms | 1800 ms | 4500 ms | +| Draft typing | 228.08 ms | 500 ms | 1250 ms | +| Complete response | 1699.26 ms | 1000 ms overhead + 992 ms pacing | 3492 ms | +| Reconnect replacement | 13.83 ms | 16 ms | 40 ms | +| Reconnect retained heap | 23.03 MiB | 24 MiB | 30 MiB | + +Draft typing spans 138.80–420.28 ms across the three isolated samples; its reference covers that observed spread instead of treating the median as a per-keystroke bound. No budget is an environment override. Temporary zero allowances exercise every rejection path; these negative controls prove enforcement, not an optimization or a historical regression. + +## Alternatives considered + +**Use the Node fold as paint evidence.** Rejected because it never performs DOM mutation, layout, or browser scheduling. The focused reconnect case likewise makes no GUI speed claim. + +**Promote the entire manual browser diagnostic into CI.** Rejected because its 1,000-session sidebar and 100-turn soak cover a much broader workload. The bounded required case reuses its shipped scaffold and measurement approach without importing a test module or changing the manual inventory. + +**Coalesce active reconnect chunks.** Rejected as a benchmark shortcut: Client entries expose per-member ordering and timestamps to conversation definitions. The benchmark retains that production behavior; reducing retained entries requires a separate semantic design, not copied product algorithms or a synthetic approximation. + +**Measure stream CPU alone.** Rejected because transport stalls and final-settlement delays can leave main-thread CPU low. The independent input, first-reply, and complete-wall budgets cover those waits. + +## Consequences + +The benchmark layer changes no product implementation or user-visible behavior. It adds approximately fifteen seconds of local browser/reconnect execution plus Web build and browser provisioning to the existing isolated CI lane. A fresh browser discards previous caches, but each workflow deliberately retains its own loaded history and previously activated Trajectory during continuation. + +The baseline is independently mergeable and protects current performance; optimization layers tighten budgets only with repeated measurements and focused semantic tests. It does not cover sidebar cardinality, an hours-long soak, GPU presentation, real model latency, a published Host launch, or reconnect rendering. The manual Web diagnostic and existing functional browser tests retain those separate responsibilities. The existing Session performance note remains active because it owns Node calibration and persistence rationale; this note extends rather than supersedes it. diff --git a/.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.zh.md b/.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.zh.md new file mode 100644 index 0000000000..f58afd5aae --- /dev/null +++ b/.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.zh.md @@ -0,0 +1,55 @@ +# Agent Note: 前端长 Session 性能预算 + +Status: implemented + +[English](2026-09-06-frontend-performance-budgets.md) | 中文 + +## 问题 + +Node 对话折叠很快,并不能证明浏览器能绘制长对话或在流式回复期间保持响应。活跃重连还会重建与已结算历史不同的表示:紧凑前缀变成公开的逐 chunk Client 条目。[Session 性能策略](2026-09-04-session-open-performance-gate.zh.md)提供隔离的 CI job,但不测量这些用户路径。 + +## 决策 + +现有串行基准清单包含两个前端所有者:[活跃重连](../../../../benchmarks/active-stream-reconnect/README.zh.md)和[浏览器工作流](../../../../benchmarks/long-session-browser/README.zh.md)。浏览器工作流在一个顺序场景中组合冷打开、更早分页导航、首次激活 Trajectory、返回 Chat,以及伴随真实键盘输入的有节奏回复。这些是同一工作流的测量终点,而不是相互独立的冷场景。已结算对话折叠基准保持不变。 + +`build:bench` 保留仅 Node 的 library 与 worker 构建。`test:bench` 额外构建 Web shell 后再运行所有用例;必需的基准 CI job 安装 Chromium。浏览器用例复用产品组合的 Web scaffold,使用私有临时目录和原子分配的回环端口。只有不确定的模型被合成重放替代。scaffold Host 通过现有 Vitest 源码解析器运行;被测 Client 渲染在全新 Chromium 进程中执行构建后的 bundle。因此浏览器壁钟时间包含测试 Host、传输、Playwright 可交互性等待及渲染,不代表发布版 Host 进程。 + +浏览器输入包含 240 个已关闭轮次、40 个工具结果和 20 个代码块,以及混合语言正文和推理。从观察到的初始 25 轮窗口开始,九次更早分页操作读完该输入;就绪探针跟踪已挂载轮次增长,不复制分页算法。每个样本使用全新 scaffold 和浏览器。环境准备、数据播种、浏览器启动、初始 shell 加载及侧栏展开不计入打开时间。打开测量在对话可用且输入框可编辑时结束;分页与导航测量在目标 DOM 状态出现时结束。两次动画帧包含一次渲染机会,不代表硬件显示或保证每个屏幕外节点都已绘制。 + +续接以 8 ms 重放间隔发送 120 个文本 delta。它记录点击到首段可见回复的时间、完成标记尚未出现时的真实草稿键入、直到持久化结算的完整回复壁钟时间,以及 Chromium 主线程任务时间。完整壁钟预算在缩放后的额外开销额度上加固定的 992 ms 脚本节奏;输入和完成均有独立执行的预算。强制 GC 后的浏览器 heap 和 DOM 数量仍仅供诊断,因为单个终点不能证明泄漏。 + +重连使用三个全新编译后的纯 Node 子进程。各进程在计时 `ClientAssistantStream.replace()` 前创建包含不同时间戳、两条紧凑记录和 100,000 个 delta 的推理前缀。在基线前执行 GC,并在结果仍可达时于替换后再次 GC;替换时间不含两次回收。报告在回收后消费结果,并检查下一个稠密序号的实时 frame 仍被接受。这测量重建,不测量传输、渲染或完整重连工作流。 + +## 校准 + +在 arm64 参考机器、Node 24.19、Chromium 149.0.7827.55 和产品版本 `925e012340` 上,三个样本的中位数建立下表基线。完整工作流 smoke 后执行一次隔离重复测量。每个浏览器样本报告原始终点数据和每一页;分页判定使用各样本最大值的中位数。重连报告全部子进程测量。源码参考常量向上取整覆盖观察值;共享的 2× 时间倍率和 1.25× 方差余量产生 CI 限制。内存仅使用方差余量。现有倍率来自 Node CI 校准,并非实测 x64 浏览器对比;浏览器专用 runner 校准仍是明确缺口。 + +| 终点 | 实测中位数 | 参考额度 | CI 限制 | +|---|---:|---:|---:| +| 浏览器打开 | 166.77 ms | 200 ms | 500 ms | +| 最慢更早分页 | 245.52 ms | 260 ms | 650 ms | +| 首次 Trajectory | 133.45 ms | 160 ms | 400 ms | +| 首段回复 | 1033.07 ms | 1100 ms | 2750 ms | +| 流式主线程任务 | 1668.46 ms | 1800 ms | 4500 ms | +| 草稿键入 | 228.08 ms | 500 ms | 1250 ms | +| 完整回复 | 1699.26 ms | 1000 ms 额外开销 + 992 ms 节奏 | 3492 ms | +| 重连替换 | 13.83 ms | 16 ms | 40 ms | +| 重连保留 heap | 23.03 MiB | 24 MiB | 30 MiB | + +三个隔离样本中的草稿键入时间为 138.80–420.28 ms;参考额度覆盖观察到的波动,而不把中位数作为单次按键上限。预算不能通过环境变量覆盖。临时零额度覆盖每条拒绝路径;这些负向对照证明预算执行,而非优化或历史回归。 + +## 考虑过的替代方案 + +**用 Node 折叠作为绘制证据。** 拒绝,因为它不执行 DOM 修改、布局或浏览器调度。聚焦重连用例同样不声称 GUI 提速。 + +**把整个手动浏览器诊断提升到 CI。** 拒绝,因为其 1,000 Session 侧栏和 100 轮 soak 覆盖更广的工作负载。受限的必需用例复用其产品 scaffold 和测量方式,不导入测试模块,也不改变手动清单。 + +**合并活跃重连 chunk。** 不能作为基准捷径:Client 条目向对话定义公开每个成员的顺序和时间戳。基准保留该生产行为;减少保留条目需要独立的语义设计,而非复制产品算法或使用合成近似。 + +**只测量流式 CPU。** 拒绝,因为传输停顿和最终结算延迟可能不增加主线程 CPU。独立的输入、首段回复和完整壁钟预算覆盖这些等待。 + +## 影响 + +基准层不改变产品实现或用户可见行为。它在现有隔离 CI lane 中增加约十五秒的本地浏览器与重连执行,以及 Web 构建和浏览器安装成本。全新浏览器丢弃此前的缓存,但每个工作流刻意在续接期间保留自身已加载历史和曾激活的 Trajectory。 + +基线可独立合并并保护现有性能;优化层只有在重复测量与聚焦语义测试支持下才收紧预算。它不覆盖侧栏数量级、数小时 soak、GPU 显示、真实模型延迟、发布版 Host 启动或重连渲染。手动 Web 诊断和现有功能浏览器测试继续各负其责。现有 Session 性能记录保持活跃,因为它拥有 Node 校准和持久化理由;本记录扩展而不替代它。 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c81db5a69f..03afdec2ca 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -206,6 +206,9 @@ jobs: - name: Install (immutable) run: pnpm install --frozen-lockfile + - name: Install benchmark browser + run: pnpm --filter @deepseek-ai/dsh-benchmarks exec playwright install --with-deps chromium + - name: Run performance benchmarks env: DSH_GATE_VERBOSE: '1' diff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.md index 933d4d25bf..2f40d70afb 100644 --- a/benchmarks/AGENTS.md +++ b/benchmarks/AGENTS.md @@ -5,6 +5,7 @@ This tree owns required, repository-level performance gates whose measured user - Organize benchmarks by measured user path, one directory per path. Do not mirror the package tree. - Host cases use `*.bench.ts`; Client-face cases use `*.bench.client.ts`. Worker, fixture, and support modules do not carry a benchmark suffix. - The private `@deepseek-ai/dsh-benchmarks` workspace owns benchmark-only dependencies. `test:bench` builds workspace libraries and `benchmarks/.dsh-build/` workers before Vitest orchestration. Timed CPU work runs in those workers under plain Node, without a TypeScript loader; runtime package imports must resolve to built `lib/` entries. +- Browser workflow cases drive built Client bundles through the shared shipped-composition Web scaffold. Report its source-resolved test Host separately from published-Host evidence; two animation frames prove a rendering opportunity, not hardware presentation. Use fresh browsers and private scaffold worlds per sample. - Synthesize fixed inputs from reviewed constants. Never use recorded Sessions, user material, ambient repositories, or network services. - Run process-level wall-clock and retained-memory samples in fresh children with private `mkdtemp` roots. Pure synchronous folds create a fresh object graph per sample and must not mutate process-global state. Bound every child, await exit, and remove owned roots after failure as well as success. - Record reference-machine expectations separately from the shared CI time scale and variance headroom. Do not apply the time scale to memory or dimensionless ratios. diff --git a/benchmarks/active-stream-reconnect/README.i18n.yaml b/benchmarks/active-stream-reconnect/README.i18n.yaml new file mode 100644 index 0000000000..b4a785f27e --- /dev/null +++ b/benchmarks/active-stream-reconnect/README.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write benchmarks/active-stream-reconnect/README.md +README.md: 2f10512f144b923df2d89ff2766c4acf1059a652 +README.zh.md: b0c97ea7f06281a71f4e633b9f60f5f5b2ebf4fa diff --git a/benchmarks/active-stream-reconnect/README.md b/benchmarks/active-stream-reconnect/README.md new file mode 100644 index 0000000000..2f10512f14 --- /dev/null +++ b/benchmarks/active-stream-reconnect/README.md @@ -0,0 +1,7 @@ +# Active Assistant reconnect benchmark + +English | [中文](README.zh.md) + +[reconnect.bench.client.ts](reconnect.bench.client.ts) measures the production Client fold when a reconnect carries an unfinished 100,000-delta reasoning prefix. A compiled private adapter reaches `ClientAssistantStream.replace()` without adding product exports. Three fresh plain-Node workers synthesize the compact baseline before timing; replacement time and retained heap after forced GC have separate median budgets. The next dense live frame must still be accepted. + +Build with `pnpm run build:bench`, then select `benchmarks/active-stream-reconnect` in `vitest.bench.config.ts`. This focused Node workload neither builds nor measures browser rendering. [Frontend performance budgets](../../.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.md) records calibration and exclusions. diff --git a/benchmarks/active-stream-reconnect/README.zh.md b/benchmarks/active-stream-reconnect/README.zh.md new file mode 100644 index 0000000000..b0c97ea7f0 --- /dev/null +++ b/benchmarks/active-stream-reconnect/README.zh.md @@ -0,0 +1,7 @@ +# 活跃 Assistant 重连基准 + +[English](README.md) | 中文 + +[reconnect.bench.client.ts](reconnect.bench.client.ts) 测量重连携带未完成的 100,000 个 reasoning delta 前缀时,生产 Client 的折叠成本。编译后的私有适配器调用 `ClientAssistantStream.replace()`,不增加产品导出。三个全新纯 Node worker 在计时前合成紧凑 baseline;替换时间与强制 GC 后的保留 heap 分别执行中位数预算检查。下一个稠密序号的实时 frame 仍须被接受。 + +通过 `pnpm run build:bench` 构建,再在 `vitest.bench.config.ts` 中选择 `benchmarks/active-stream-reconnect`。该聚焦 Node workload 既不构建也不测量浏览器渲染。[前端性能预算](../../.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.zh.md)记录校准与排除项。 diff --git a/benchmarks/active-stream-reconnect/reconnect.bench.client.ts b/benchmarks/active-stream-reconnect/reconnect.bench.client.ts new file mode 100644 index 0000000000..374173495b --- /dev/null +++ b/benchmarks/active-stream-reconnect/reconnect.bench.client.ts @@ -0,0 +1,34 @@ +/** Required baseline budgets for reconnecting during a large active Assistant stream. */ +import { join } from 'node:path' +import { expect, it } from 'vitest' +import { runBuiltBenchmarkWorker } from '../support/built-worker.ts' +import { ciTimeBudget, PERFORMANCE_BUDGET_HEADROOM } from '../support/calibration.ts' +import type { ReconnectReport } from './reconnect.worker.client.ts' + +const REFERENCE_REPLACE_MS = 16 +const REFERENCE_RETAINED_MB = 24 +const SAMPLES = 3 + +it('reconstructs a 100000-delta live prefix within baseline time and retained-memory budgets', async () => { + const samples: ReconnectReport[] = [] + for (let sample = 0; sample < SAMPLES; sample++) { + const run = await runBuiltBenchmarkWorker({ + worker: join(import.meta.dirname, '../.dsh-build/active-stream-reconnect/reconnect.worker.js'), + exposeGc: true, timeoutMs: 30000, + }) + expect(run.timedOut, run.stderr).toBe(false) + expect(run.signal, run.stderr).toBeNull() + expect(run.exitCode, run.stderr).toBe(0) + if (run.report === undefined) throw new Error('reconnect worker omitted report') + expect(run.report.nextFrame).toBe('transient') + expect(run.report.entries).toBeGreaterThan(0) + samples.push(run.report) + } + const replaceMs = samples.map(sample => sample.replaceMs).toSorted((a, b) => a - b)[1]! + const retainedMb = samples.map(sample => sample.retainedMb).toSorted((a, b) => a - b)[1]! + const budgetMs = ciTimeBudget(REFERENCE_REPLACE_MS) + const budgetMb = REFERENCE_RETAINED_MB * PERFORMANCE_BUDGET_HEADROOM + console.log(JSON.stringify({ benchmark: 'active-stream-reconnect', samples, median: { replaceMs, retainedMb }, referenceMs: REFERENCE_REPLACE_MS, referenceMb: REFERENCE_RETAINED_MB, budgetMs, budgetMb })) + expect.soft(replaceMs).toBeLessThanOrEqual(budgetMs) + expect.soft(retainedMb).toBeLessThanOrEqual(budgetMb) +}) diff --git a/benchmarks/active-stream-reconnect/reconnect.worker.client.ts b/benchmarks/active-stream-reconnect/reconnect.worker.client.ts new file mode 100644 index 0000000000..a1a315a263 --- /dev/null +++ b/benchmarks/active-stream-reconnect/reconnect.worker.client.ts @@ -0,0 +1,49 @@ +/** Compiled production Client fold for a reconnect during a long Assistant attempt. */ +import { performance } from 'node:perf_hooks' +import { AssistantStreamAccumulator } from '@deepseek-ai/dsh-llm/assistant-stream' +import { LlmAttemptId } from '@deepseek-ai/dsh-llm/brand' +import type { SessionAssistantStreamBaseline } from '@deepseek-ai/dsh-api-session-controller/types' +// The Client implementation has no plain-Node export; only this adapter is bundled. +import { ClientAssistantStream } from '../../packages/api/session-controller/src/client/sessions/assistant-stream.ts' +import { assertBuiltBenchmarkRuntime } from '../support/built-worker.ts' + +/** Measurements of replace() only; fixture construction and forced GC are excluded. */ +export interface ReconnectReport { + readonly deltas: number + readonly records: number + readonly entries: number + readonly replaceMs: number + readonly retainedMb: number + readonly nextFrame: string | undefined +} + +assertBuiltBenchmarkRuntime(import.meta.url, { + '@deepseek-ai/dsh-llm/assistant-stream': import.meta.resolve('@deepseek-ai/dsh-llm/assistant-stream'), +}) +const deltas = 100000 +const accumulator = new AssistantStreamAccumulator() +accumulator.push({ time: 1700000000000, chunk: { type: 'block-start', index: 0, blockType: 'reasoning' } }) +for (let index = 0; index < deltas; index++) { + accumulator.push({ time: 1700000000001 + index, chunk: { type: 'reasoning-delta', index: 0, text: 'token ' } }) +} +const attemptId = LlmAttemptId('synthetic-reconnect') +const nextIndex = deltas + 1 +const baseline: SessionAssistantStreamBaseline = { + revision: nextIndex + 1, + activeAttempt: { + attemptId, startedAfterSeq: -1, turn: 1, step: 1, nextIndex, + stream: JSON.parse(JSON.stringify(accumulator.snapshot())) as NonNullable['stream'], + }, +} +if (globalThis.gc === undefined) throw new Error('reconnect benchmark requires --expose-gc') +globalThis.gc() +const before = process.memoryUsage().heapUsed +const client = new ClientAssistantStream() +const start = performance.now() +const visible = client.replace([], baseline) +const replaceMs = performance.now() - start +globalThis.gc() +const retainedMb = (process.memoryUsage().heapUsed - before) / 1048576 +const next = client.acceptFrame({ type: 'chunk', attemptId, revision: nextIndex + 2, index: nextIndex, time: 1700000000001 + deltas, chunk: { type: 'reasoning-delta', index: 0, text: 'suffix' } }) +const report: ReconnectReport = { deltas, records: baseline.activeAttempt!.stream.length, entries: visible.length, replaceMs, retainedMb, nextFrame: next?.type } +process.stdout.write(JSON.stringify(report) + '\n') diff --git a/benchmarks/long-session-browser/README.i18n.yaml b/benchmarks/long-session-browser/README.i18n.yaml new file mode 100644 index 0000000000..e95961a823 --- /dev/null +++ b/benchmarks/long-session-browser/README.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write benchmarks/long-session-browser/README.md +README.md: 53385706703736a76568b0b55141d40731068ed2 +README.zh.md: 5e709ec6bbd53a420fcf8ad4f4f414b4552549b6 diff --git a/benchmarks/long-session-browser/README.md b/benchmarks/long-session-browser/README.md new file mode 100644 index 0000000000..5338570670 --- /dev/null +++ b/benchmarks/long-session-browser/README.md @@ -0,0 +1,17 @@ +# Long-session browser benchmark + +English | [中文](README.zh.md) + +This reference describes the required Chromium workflow in [long-session.bench.ts](long-session.bench.ts). It opens a synthetic 240-turn Session, loads every older page, visits Trajectory, returns to Chat, and submits a paced reply while typing another draft. The shipped Web scaffold owns the isolated home, persistence, replay adapter, and loopback listener; Chromium loads the built Web artifacts, not a replacement development server. + +## Run + +`pnpm run test:bench` builds libraries, workers, and Web artifacts before running the serial benchmark inventory. With artifacts already built, select this directory through `pnpm exec vitest run --config vitest.bench.config.ts benchmarks/long-session-browser`. Install Chromium through the benchmark workspace before the first run. + +## Measurements + +Three fresh browser processes and scaffold worlds produce raw samples and median verdicts. Open and paging end after the expected transcript state and two animation frames; this includes a rendering opportunity, not a hardware presentation timestamp. Paging reports every page and gates the median of each sample’s slowest page. Stream reports first visible reply, trusted draft typing, complete reply wall time, and Chromium main-thread task duration. Heap after forced GC and DOM counts are diagnostics, not leak budgets. + +The fixture contains mixed-language prompts, prose, reasoning, 20 code fences, and 40 synthetic tool results. No model, tool, external network, recorded Session, or private Harness home supplies its content. Streaming uses 120 text deltas at 8 ms replay pacing through the real composer, agent loop, transport, and persistence. + +The [decision record](../../.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.md) owns calibration, exclusions, and alternatives. The larger [manual diagnostic](../../apps/web/tests/complex-history.perf.ts) remains separate. diff --git a/benchmarks/long-session-browser/README.zh.md b/benchmarks/long-session-browser/README.zh.md new file mode 100644 index 0000000000..5e709ec6bb --- /dev/null +++ b/benchmarks/long-session-browser/README.zh.md @@ -0,0 +1,17 @@ +# 长 Session 浏览器基准 + +[English](README.md) | 中文 + +本文说明 [long-session.bench.ts](long-session.bench.ts) 中必需的 Chromium 工作流。它打开一个合成的 240 轮 Session,加载所有更早的分页,访问 Trajectory,返回 Chat,并在流式回复期间输入下一条草稿。随产品维护的 Web scaffold 拥有隔离的主目录、持久化、重放适配器和回环监听器;Chromium 加载构建后的 Web 产物,而非替代开发服务器。 + +## 运行 + +`pnpm run test:bench` 先构建 library、worker 和 Web 产物,再串行运行基准清单。产物已构建时,通过 `pnpm exec vitest run --config vitest.bench.config.ts benchmarks/long-session-browser` 选择此目录。首次运行前,通过 benchmark workspace 安装 Chromium。 + +## 测量 + +三个全新浏览器进程与 scaffold 环境产生原始样本及中位数判定。打开和分页在预期对话状态出现且经过两次动画帧后结束;这包含一次渲染机会,而非硬件显示时间戳。分页报告每一页,并对各样本最慢分页时间的中位数执行预算检查。流式报告首段可见回复、真实草稿键入、完整回复壁钟时间和 Chromium 主线程任务时间。强制 GC 后的 heap 与 DOM 数量仅供诊断,不作为泄漏预算。 + +fixture(测试前置数据)包含混合语言提示、正文、推理、20 个代码块和 40 个合成工具结果。其内容不来自模型、工具、外部网络、录制 Session 或私有 Harness 主目录。流式回复以 8 ms 重放间隔发送 120 个文本 delta,经过真实输入框、agent loop(智能体循环)、传输与持久化。 + +[决策记录](../../.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.zh.md)拥有校准、排除项与替代方案。更大规模的[手动诊断](../../apps/web/tests/complex-history.perf.ts)保持独立。 diff --git a/benchmarks/long-session-browser/long-session.bench.ts b/benchmarks/long-session-browser/long-session.bench.ts new file mode 100644 index 0000000000..7a4120778b --- /dev/null +++ b/benchmarks/long-session-browser/long-session.bench.ts @@ -0,0 +1,139 @@ +/** Required browser budgets for opening, paging and continuing synthetic long history. */ +import { mkdtemp, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { performance } from 'node:perf_hooks' +import { chromium, type Page, type CDPSession } from 'playwright' +import { expect, it } from 'vitest' +import { launchWebScaffold, seedSession, watchConsole, webSnapshotMode } from '../../apps/web/tests/scaffold.ts' +import { newEnglishPage } from '../../apps/web/tests/support.ts' +import { ciTimeBudget } from '../support/calibration.ts' +import { HISTORY_TURNS, SESSION_ID, FIRST, DONE, DELTAS, PACE_MS, syntheticHistory, syntheticReply } from './synthetic-history.ts' + +const SAMPLES = 3 +const TAIL = '[data-chat-flow-key^="9:turn-tail"]' +const REFERENCE = { open: 200, page: 260, trajectory: 160, first: 1100, streamTask: 1800, input: 500, streamWall: 1000 } +const REPLAY_DURATION_MS = (DELTAS + 4) * PACE_MS + +async function painted(page: Page): Promise { + // Two rAF callbacks include a rendering opportunity, not a GPU presentation timestamp. + await page.evaluate(() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(() => resolve())))) +} + +async function measure(page: Page, action: () => Promise): Promise { + const start = performance.now() + await action() + await painted(page) + return performance.now() - start +} + +async function taskMs(cdp: CDPSession): Promise { + const result = await cdp.send('Performance.getMetrics') + const metric = result.metrics.find(metric => metric.name === 'TaskDuration') + if (metric === undefined) throw new Error('Chromium TaskDuration missing') + return metric.value * 1000 +} + +function median(values: number[]): number { + return values.toSorted((a, b) => a - b)[Math.floor(values.length / 2)]! +} + +it('opens, pages, navigates and streams into a 240-turn browser history', async () => { + if (webSnapshotMode() !== 'replay') throw new Error('browser benchmarks require keyless replay mode') + const samples: { open: number; page: number; trajectory: number; first: number; streamTask: number; streamWall: number; input: number; heapMb: number; nodes: number }[] = [] + for (let sample = 0; sample < SAMPLES; sample++) { + const failures: unknown[] = [] + const root = await mkdtemp(join(tmpdir(), 'dsh-browser-benchmark-')) + try { + const replayOverride = join(root, 'reply.json') + await writeFile(replayOverride, JSON.stringify([{ kind: 'chunks', chunks: syntheticReply() }])) + const scaffold = await launchWebScaffold({ replayFixture: join(root, 'override-only.jsonl'), replayOverride, paceMs: PACE_MS, replayContextWindow: 10000000 }) + try { + await seedSession(scaffold, syntheticHistory(), SESSION_ID) + const browser = await chromium.launch({ headless: true }) + try { + const page = await newEnglishPage(browser) + const consoleWatch = watchConsole(page) + page.setDefaultTimeout(30000) + await page.goto(scaffold.authenticatedUrl, { waitUntil: 'load' }) + expect(new URL(page.url()).origin).toBe(scaffold.baseUrl) + console.log(JSON.stringify({ benchmark: 'long-session-browser/server', url: scaffold.baseUrl, browser: browser.version(), sample })) + await page.waitForSelector('[class*="frame"]') + await page.getByRole('treeitem').first().click() + const result = page.getByRole('treeitem').nth(1) + await result.waitFor() + const open = await measure(page, async () => { + await result.click() + await page.locator(TAIL).last().waitFor() + await page.locator('[data-composer-input][contenteditable="true"]').last().waitFor() + }) + const pages: number[] = [] + const initialTurns = await page.locator(TAIL).count() + expect(initialTurns).toBeGreaterThan(0) + expect(initialTurns).toBeLessThan(HISTORY_TURNS) + let count = initialTurns + while (count < HISTORY_TURNS) { + pages.push(await measure(page, async () => { + await page.getByRole('button', { name: 'Load earlier', exact: true }).click() + await page.waitForFunction(({ selector, previous }) => document.querySelectorAll(selector).length > previous, { selector: TAIL, previous: count }) + })) + count = await page.locator(TAIL).count() + } + const trajectory = await measure(page, async () => { + await page.getByRole('tab', { name: 'Trajectory', exact: true }).click() + await page.getByRole('searchbox', { name: 'Search trajectory', exact: true }).waitFor() + await page.getByRole('row').last().waitFor() + }) + await page.getByRole('tab', { name: 'Chat', exact: true }).click() + await page.waitForFunction(selector => document.querySelectorAll(selector).length === 240, TAIL) + const composer = page.locator('[data-composer-input][contenteditable="true"]').last() + await composer.fill('Continue the synthetic review and summarize the validation. '.repeat(30)) + const cdp = await page.context().newCDPSession(page) + await cdp.send('Performance.enable') + const beforeTask = await taskMs(cdp) + const settled = scaffold.whenTurnSettled(60000).then( + () => ({ ok: true as const }), + (error: unknown) => ({ ok: false as const, error }), + ) + const started = performance.now() + await page.getByRole('button', { name: 'Send message', exact: true }).click() + await page.getByText(FIRST, { exact: false }).last().waitFor() + await painted(page) + const first = performance.now() - started + expect(await page.getByText(DONE, { exact: false }).count()).toBe(0) + // Trusted keyboard input while the response is live, rather than a synthetic heartbeat. + const input = await measure(page, async () => { + await composer.click() + await page.keyboard.type('next synthetic question') + await expect.poll(() => composer.textContent()).toBe('next synthetic question') + }) + await page.getByText(DONE, { exact: false }).last().waitFor() + const settlement = await settled + if (!settlement.ok) throw settlement.error + await painted(page) + const streamWall = performance.now() - started + const streamTask = await taskMs(cdp) - beforeTask + await cdp.send('HeapProfiler.collectGarbage') + const metrics = (await cdp.send('Performance.getMetrics')).metrics + const heap = metrics.find(metric => metric.name === 'JSHeapUsedSize') + if (heap === undefined) throw new Error('Chromium heap metric missing') + samples.push({ open, page: Math.max(...pages), trajectory, first, streamTask, streamWall, input, heapMb: heap.value / 1048576, nodes: await page.locator('*').count() }) + console.log(JSON.stringify({ benchmark: 'long-session-browser/sample', sample, initialTurns, pages, ...samples.at(-1) })) + expect(consoleWatch.pageErrors).toEqual([]) + expect(consoleWatch.warnings).toEqual([]) + } catch (error) { failures.push(error) } finally { + await browser.close().catch((error: unknown) => failures.push(error)) + } + } catch (error) { failures.push(error) } finally { + await scaffold.close().catch((error: unknown) => failures.push(error)) + } + } catch (error) { failures.push(error) } finally { + await rm(root, { recursive: true, force: true }).catch((error: unknown) => failures.push(error)) + } + if (failures.length > 0) throw new AggregateError(failures, 'browser benchmark failed') + } + const aggregate = Object.fromEntries(Object.keys(REFERENCE).map(key => [key, median(samples.map(sample => sample[key as keyof typeof REFERENCE]))])) + const budgets = Object.fromEntries(Object.entries(REFERENCE).map(([key, value]) => [key, ciTimeBudget(value) + (key === 'streamWall' ? REPLAY_DURATION_MS : 0)])) + console.log(JSON.stringify({ benchmark: 'long-session-browser/median', turns: HISTORY_TURNS, deltas: DELTAS, paceMs: PACE_MS, samples, aggregate, referenceMs: REFERENCE, budgets })) + for (const [key, value] of Object.entries(aggregate)) expect.soft(value, key).toBeLessThanOrEqual(budgets[key]!) +}) diff --git a/benchmarks/long-session-browser/synthetic-history.ts b/benchmarks/long-session-browser/synthetic-history.ts new file mode 100644 index 0000000000..1d3b22aedb --- /dev/null +++ b/benchmarks/long-session-browser/synthetic-history.ts @@ -0,0 +1,69 @@ +/** Synthetic current-generation history and paced reply for browser measurements. */ +import { createAssistantMessage, createUserMessage, createToolResultMessage, ToolCallId } from '@deepseek-ai/dsh-llm' +import type { StreamChunk } from '@deepseek-ai/dsh-llm' +import { Session, SessionId, SESSION_FORMAT_VERSION } from '@deepseek-ai/dsh-session' +import type {} from '@deepseek-ai/dsh-session-title' + +/** Closed turns in the browser history workload. */ +export const HISTORY_TURNS = 240 +/** Identity private to each isolated scaffold. */ +export const SESSION_ID = 'benchmark-browser-history' +const TITLE = 'SYNTHETIC_BROWSER_HISTORY' +/** First streamed text marker. */ +export const FIRST = 'SYNTHETIC_REPLY_FIRST' +/** Last streamed text marker. */ +export const DONE = 'SYNTHETIC_REPLY_DONE' +/** Paced text chunks per continuation. */ +export const DELTAS = 120 +/** Replay delay per stream chunk, in milliseconds. */ +export const PACE_MS = 8 + +/** Create mixed prose, code, reasoning and tool history without reading user data. + * @returns Current Session JSONL accepted by the shared Web seeder. + */ +export function syntheticHistory(): string { + const session = Session.create(SessionId(SESSION_ID)) + for (let turn = 1; turn <= HISTORY_TURNS; turn++) { + session.append('turn/start', { turn }) + const user = session.append('user/message', createUserMessage({ + content: [{ type: 'text', text: 'Review synthetic change ' + String(turn) + ': 检查增量渲染。 '.repeat(30) }], + source: { kind: 'user' }, + }), { surfaceOp: 'append' }) + if (turn === 1) session.append('session/title', { title: TITLE, messageSeqs: [user.seq], source: { kind: 'fallback' } }) + session.append('step/start', { turn, step: 1 }) + const callId = ToolCallId('synthetic-tool-' + String(turn)) + const tool = turn % 6 === 0 + const code = turn % 12 === 0 + ? '\n\n```ts\n' + Array.from({ length: 60 }, (_, i) => 'const value' + String(i) + ' = ' + String(i)).join('\n') + '\n```' + : '' + session.append('assistant/message', { + turn, step: 1, stream: [], + message: createAssistantMessage({ + source: { provider: 'deepseek-official', model: 'deepseek-v4-flash' }, + content: [ + { type: 'reasoning', text: 'Compare the synthetic module and test. '.repeat(40) }, + { type: 'text', text: 'Synthetic answer ' + String(turn) + '. ' + 'Preserve ordering and validate the output. '.repeat(30) + code }, + ...tool ? [{ type: 'tool-call' as const, id: callId, name: 'synthetic_tool', arguments: '{"path":"src/example.ts"}' }] : [], + ], + }), + usage: { inputTokens: 4000, outputTokens: 800 }, + }, { surfaceOp: 'append' }) + if (tool) { + const call = session.append('tool/call', { turn, step: 1, callId, name: 'synthetic_tool', arguments: '{"path":"src/example.ts"}' }) + session.append('tool/result', { turn, step: 1, message: createToolResultMessage({ + callId, isError: false, content: [{ type: 'text', text: 'Synthetic tool output line.\n'.repeat(160) }], + }) }, { surfaceOp: 'append', sourceEventSeqs: [call.seq] }) + } + session.append('step/end', { turn, step: 1 }) + session.append('turn/end', { turn, reason: { kind: 'completed' } }) + } + return [JSON.stringify({ type: 'session', version: SESSION_FORMAT_VERSION, id: '{{sessionId}}', createdAt: 1700000000000, cwd: '{{cwd}}', isSeeded: false, delegationDepth: 0 }), ...session.snapshotEvents().map(event => JSON.stringify(event)), ''].join('\n') +} + +/** Create one paced response; replay owns delays outside the browser. + * @returns Stream chunks ending in a visible completion marker. + */ +export function syntheticReply(): StreamChunk[] { + const deltas = Array.from({ length: DELTAS }, (_, i) => i === 0 ? FIRST + ' ' : i === DELTAS - 1 ? DONE : 'Synthetic response ' + String(i) + '. ') + return [{ type: 'block-start', index: 0, blockType: 'text' }, ...deltas.map(text => ({ type: 'text-delta' as const, index: 0, text })), { type: 'block-end', index: 0, block: { type: 'text', text: deltas.join('') } }, { type: 'usage', usage: { inputTokens: 4000, outputTokens: 800 } }, { type: 'finish', reason: { kind: 'stop' } }] +} diff --git a/benchmarks/package.json b/benchmarks/package.json index 2cf0eec369..cc8c60cf71 100644 --- a/benchmarks/package.json +++ b/benchmarks/package.json @@ -5,6 +5,8 @@ "private": true, "type": "module", "devDependencies": { + "playwright": "^1.49.0", + "@deepseek-ai/dsh-llm-replay": "workspace:^", "@deepseek-ai/cordis": "workspace:^", "@deepseek-ai/dsh-agent": "workspace:^", "@deepseek-ai/dsh-agent-loop": "workspace:^", diff --git a/benchmarks/tsdown.config.ts b/benchmarks/tsdown.config.ts index 3c8d6a9ab5..2b7a7d2bb9 100644 --- a/benchmarks/tsdown.config.ts +++ b/benchmarks/tsdown.config.ts @@ -14,6 +14,13 @@ const shared = { /** Compile measured benchmark workers while keeping workspace packages on their built `lib` entries. */ export default defineConfig([ + { + ...shared, + entry: { 'reconnect.worker': 'active-stream-reconnect/reconnect.worker.client.ts' }, + outDir: '.dsh-build/active-stream-reconnect', + clean: true, + tsconfig: 'tsconfig.client.json', + }, { ...shared, entry: { diff --git a/package.json b/package.json index 801b00c765..296b0d20ca 100644 --- a/package.json +++ b/package.json @@ -37,7 +37,7 @@ "test:coverage": "vitest run --coverage", "test:coverage:partitioned": "tsx scripts/run-coverage-partitions.ts", "test:e2e": "vitest run --config vitest.e2e.config.ts", - "test:bench": "npm run build:bench && npm run test:bench:built", + "test:bench": "npm run build:bench && npm run build:web && npm run test:bench:built", "test:bench:built": "vitest run --config vitest.bench.config.ts", "test:expected": "vitest run --config vitest.expected.config.ts", "test:expected:refresh": "DSH_SNAPSHOT=refresh vitest run --config vitest.expected.config.ts", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index cf86833ddb..dadcc170fe 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -581,6 +581,9 @@ importers: '@deepseek-ai/dsh-llm': specifier: workspace:^ version: link:../packages/llm/llm + '@deepseek-ai/dsh-llm-replay': + specifier: workspace:^ + version: link:../packages/test-support/llm-replay '@deepseek-ai/dsh-sdk-client': specifier: workspace:^ version: link:../packages/sdk/client @@ -620,6 +623,9 @@ importers: '@deepseek-ai/dsh-typert-protocol': specifier: workspace:^ version: link:../packages/typert/protocol + playwright: + specifier: ^1.49.0 + version: 1.61.1 native/landlock-run: devDependencies: diff --git a/vitest.bench.config.ts b/vitest.bench.config.ts index 81a4652718..5e355d85d6 100644 --- a/vitest.bench.config.ts +++ b/vitest.bench.config.ts @@ -3,8 +3,9 @@ import { defineConfig } from 'vitest/config' import { standardDecoratorPlugin, vitestExecArgv } from './vitest.shared.ts' /** - * CI performance gate. Vitest orchestrates compiled plain-Node workers under - * `.dsh-build/benchmarks/`; timed product work never runs through its source transform. + * CI performance gate. Node CPU cases use compiled plain-Node workers under + * `benchmarks/.dsh-build/`; browser cases drive built Client artifacts through + * the shared shipped-composition Web scaffold. * Files run one at a time so a measurement never shares the CPU with another * benchmark. */