From 7a1cf325b8d6d758f97743cbac865328db1b0f99 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Sun, 6 Sep 2026 14:14:30 +0800 Subject: [PATCH 1/2] docs(skills): add evidence-driven performance optimization workflow --- ...vidence-driven-performance-skill.i18n.yaml | 6 ++ ...09-06-evidence-driven-performance-skill.md | 29 ++++++ ...06-evidence-driven-performance-skill.zh.md | 29 ++++++ .agents/skills/dsh-speed-up-perf/SKILL.md | 92 +++++++++++++++++++ .../references/pr-evidence.md | 59 ++++++++++++ 5 files changed, 215 insertions(+) create mode 100644 .agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.i18n.yaml create mode 100644 .agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md create mode 100644 .agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.zh.md create mode 100644 .agents/skills/dsh-speed-up-perf/SKILL.md create mode 100644 .agents/skills/dsh-speed-up-perf/references/pr-evidence.md diff --git a/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.i18n.yaml b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.i18n.yaml new file mode 100644 index 0000000000..ba3a54c1c1 --- /dev/null +++ b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md +2026-09-06-evidence-driven-performance-skill.md: 66f04f242afc2e70a6af7f8b6558e763b2e33617 +2026-09-06-evidence-driven-performance-skill.zh.md: b82aa37cb017054e2f35a97af50b71763f0cd86a diff --git a/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md new file mode 100644 index 0000000000..66f04f242a --- /dev/null +++ b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md @@ -0,0 +1,29 @@ +# Agent Note: Evidence-driven performance optimization workflow + +Status: implemented + +English | [中文](2026-09-06-evidence-driven-performance-skill.zh.md) + +## Problem + +Performance work can improve an isolated phase while moving cost into another phase, retaining more data, or skipping required behavior. Historical PR descriptions also retain abandoned implementations and estimates, so copying their apparent solution can restore a rejected design instead of addressing a current bottleneck. + +## Decision + +The [dsh-speed-up-perf skill](../../../skills/dsh-speed-up-perf/SKILL.md) guides broad surveys toward bounded, measured user paths. It combines focused attribution with independently timed backend and browser endpoints, synthetic workload distributions, comparable cold/warm and retained-memory conditions, and negative controls for tightened budgets. Its [evidence reference](../../../skills/dsh-speed-up-perf/references/pr-evidence.md) distinguishes merged implementations, superseded proposals, author-reported measurements, and estimates. + +The workflow requires behavior evidence independently of timing: model-visible logs, durable generation and publication rules, stream ordering, cancellation, and disposal remain obligations. Authorized private corpus inspection yields only aggregate workload inspiration; committed inputs and published artifacts contain synthetic material. Optimization PRs carry their tighter budgets, while a preceding benchmark layer can protect the measured baseline and remain independently mergeable. + +The [Session-opening performance-gate decision](../testing/2026-09-04-session-open-performance-gate.md) retains ownership of lane mechanics and calibration. The [simplification skill](../../../skills/dsh-find-simplifications/SKILL.md) retains ownership of deletion-oriented surveys. Neither is superseded: this workflow adds performance-specific candidate selection, measurement comparability, and stopping criteria rather than replacing their decisions. + +## Alternatives considered + +**Optimize suspicious code before measuring.** Rejected because local complexity does not identify dominant user cost and cannot establish improvement or regression protection. + +**Treat historical speedups as reusable prescriptions.** Rejected because representation, ownership, and lifecycle requirements change. Historical evidence generates hypotheses; current production paths and fresh measurements decide whether a change applies. + +**Use only microbenchmarks or only end-to-end timing.** Rejected because isolated phases can omit moved work, while aggregate timing alone cannot locate its cause. Both are required at the scope appropriate to the selected problem. + +## Consequences + +The skill adds no runtime behavior, benchmark implementation, or new CI policy. Its validation is document/link consistency and skill metadata; each future optimization supplies executable measurements and functional evidence at its owner. The finite scenario/fix scope prevents a broad performance request from becoming an unrelated architectural rewrite. diff --git a/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.zh.md b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.zh.md new file mode 100644 index 0000000000..b82aa37cb0 --- /dev/null +++ b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.zh.md @@ -0,0 +1,29 @@ +# Agent Note: 以证据驱动的性能优化工作流 + +Status: implemented + +[English](2026-09-06-evidence-driven-performance-skill.md) | 中文 + +## Problem + +性能工作可能改善某个独立阶段,却把成本转移到另一阶段、保留更多数据,或跳过必要行为。历史 PR(Pull Request)描述也可能保留已放弃的实现和估计值,因此照搬其表面方案可能恢复已否决的设计,而不是解决当前瓶颈。 + +## Decision + +[dsh-speed-up-perf skill](../../../skills/dsh-speed-up-perf/SKILL.md)(技能)引导广泛调查收敛到范围明确、可测量的用户路径。它结合聚焦的成本归因与独立计时的后端和浏览器端点、合成负载分布、可比较的冷态/热态与保留内存条件,以及收紧预算的负向对照。其[证据参考](../../../skills/dsh-speed-up-perf/references/pr-evidence.md)区分已合并实现、已被替代的提案、作者报告的测量值和估计值。 + +该工作流要求独立于计时的行为证据:模型可见日志、持久化代际和发布规则、流顺序、取消及 dispose(资源释放)仍是必须满足的要求。获授权的私有语料检查仅提供聚合负载启发;提交的输入和发布的产物包含合成材料。优化 PR 携带收紧后的预算,而前置基准测试层可以保护已测基线并保持独立可合并。 + +[会话打开性能门禁决策](../testing/2026-09-04-session-open-performance-gate.zh.md)继续负责测试通道机制与校准。[简化 skill](../../../skills/dsh-find-simplifications/SKILL.md)继续负责以删除为目标的调查。两者均未被替代:本工作流增加面向性能的候选选择、测量可比性和停止条件,而不替换它们的决策。 + +## Alternatives considered + +**先优化可疑代码,再测量。** 否决,因为局部复杂度不能确定主要用户成本,也无法证明改善或防止回归。 + +**把历史加速方案当作可复用处方。** 否决,因为表示方式、所有权和生命周期要求会变化。历史证据用于产生假设;当前生产路径与新的测量决定改动是否适用。 + +**只使用微基准测试,或只使用端到端计时。** 否决,因为独立阶段可能遗漏被转移的工作,而总计时无法定位原因。两者都需要在与所选问题相符的范围内使用。 + +## Consequences + +该 skill 不增加运行时行为、基准测试实现或新的 CI 策略。其验证涵盖文档/链接一致性和 skill 元数据;后续每项优化在其所属位置提供可执行测量与功能证据。有限的场景/修复范围防止广泛性能请求演变成无关的架构重写。 diff --git a/.agents/skills/dsh-speed-up-perf/SKILL.md b/.agents/skills/dsh-speed-up-perf/SKILL.md new file mode 100644 index 0000000000..68eed10147 --- /dev/null +++ b/.agents/skills/dsh-speed-up-perf/SKILL.md @@ -0,0 +1,92 @@ +--- +name: dsh-speed-up-perf +description: 'Use when investigating or optimizing DeepSeek Harness performance, designing realistic synthetic benchmarks or CI performance gates, profiling long Sessions or Web responsiveness, or turning performance PR evidence into measured behavior-preserving fixes.' +--- + +# Speed Up DeepSeek Harness + +Turn a broad “make it faster” request into reproducible user-path measurements and small, evidence-backed fixes. This is guidance, not a quota or a script: survey broadly, follow measured cost, and reject attractive changes that do not improve the workload users actually run. + +## Establish scope and current authority + +Read [AGENTS.md](../../../AGENTS.md), [architecture](../../../docs/architecture.md), [testing policy](../../../docs/testing.md), [defensive patterns](../../../docs/defensive-patterns.md), and the affected packages’ instructions and Agent Notes. Use [CI test reliability](../dsh-ci-test-reliability/SKILL.md) for processes, clocks, browser tests, and asynchronous cleanup. + +Agree on the user-visible endpoint, workload range, resource constraints, acceptable minor behavior differences, and stopping rule. Keep backend and browser end-to-end measurements separate: a fast history iterator or Client fold does not prove fast transport, paint, scrolling, or input response. Exclude model/network latency when measuring local overhead, and state that exclusion rather than calling the result complete product latency. + +Inspect the exact current base, not just the running checkout. Study final merged diffs, owning source, tests, and resolved review threads; a PR body can describe an abandoned implementation. Separate merged, closed-unmerged, superseded, estimated, and newly measured evidence. The [PR evidence reference](references/pr-evidence.md) supplies historical leads, not authority to reintroduce their implementations. + +## Survey user paths, then rank candidates + +Delegate independent domains when breadth helps; require measurements and production call sites, not guesses. Useful domains include: + +- Cold profile startup, first historical read, current-generation reopen, and writable resume. +- Many-turn and tool-heavy history, large individual messages/results, child Session listing, and repeated navigation among Sessions. +- Initial history transport and fold, first usable browser paint, older-page loading, scrolling, tool expansion, and inactive-view activation. +- Live streaming and reconnect, including a long active attempt, interleaved tool work, settlement, cancellation, and teardown. + +Vary independent cost drivers: bytes, durable events, compact records, raw deltas, turns, tools, children, and visible DOM nodes are different quantities. Do not call a large count of tiny identical messages “realistic” without checking which user operation it stresses. Include typical and tail workloads, but avoid a combinatorial matrix with no decision value. + +Rank candidates by observed user latency, CPU/allocations, retained memory, occurrence, and confidence. For each, name the production consumer, the repeated work, the expected complexity, the smallest falsifiable intervention, and the behavior that must remain stable. A suspicious loop, unused cache, or large file alone is not evidence of a bottleneck. + +## Build realistic synthetic benchmarks first + +Follow [benchmarks/AGENTS.md](../../../benchmarks/AGENTS.md) and the [performance-gate decision](../../notes/implemented/testing/2026-09-04-session-open-performance-gate.md). Extend the existing required lane rather than creating competing calibration or reporting infrastructure. Package-local diagnostics remain beside their owner; cross-package required cases live under the measured user path in `benchmarks/`. + +If the user authorizes local corpus inspection, extract only aggregate workload characteristics. Never copy prompts, outputs, paths, identities, IDs, credentials, recordings, or recognizable snippets into fixtures, logs, screenshots, PRs, or artifacts. Generate fixed inputs from reviewed constants; no benchmark depends on the user’s home, ambient repository, network service, or private data. + +Before implementation, record a measurement card: + +| Field | Required decision | +|---|---| +| User operation | Exact action and externally observable completion condition | +| Workload | Fixed dimensions, distributions, construction seed/constants, and why they exercise ordinary and tail use | +| Entry path | Production calls/composition and built artifacts; mocked external boundaries | +| Clock | Included setup, cold/warm state, timing start/end, and excluded costs | +| Memory | Reachable endpoint objects, baseline, GC policy, retained versus transient limits | +| Verdict | Raw samples, chosen aggregate, calibrated absolute/ratio/memory limits, and negative control | +| Behavior | Owning functional tests/snapshots and permitted minor differences | + +Measure built JavaScript under plain Node for CPU workers; source-loader overhead and module resolution are not the shipped path. Browser cases use built product assets and the supported `dsh` profile through the existing test harness. Do not add a production export solely for measurement or copy the algorithm into a “benchmark implementation.” + +Use fresh children and private temporary roots for cold/process-memory samples. Warm samples explicitly retain the intended cache; never let fixture setup secretly warm a cold scenario. Keep the same input, validations, completion condition, and reachable output on both sides. A parse-and-discard baseline is not comparable with validated retained history. + +Report all samples and the aggregate that decides the result. Use the existing shared time calibration and reviewed variance headroom; do not scale bytes, counts, or dimensionless ratios by CPU speed. Budgets are source constants, not environment overrides. Serialize measured work against other owned CPU-heavy jobs; measure reference and candidate under comparable conditions. Do not widen a budget or select a lucky run to hide a regression. + +Measure end-to-end latency independently from component phases. Track retained memory with intended objects still reachable, and transient pressure separately through constrained-heap completion or an appropriate peak measurement. Faster execution with unbounded retention is not an automatic win. + +For browser responsiveness, use real browser input and observe the resulting UI update. Include the final stall in frame/input measurements, distinguish scheduled timers from actual input, and bound synthetic producers so catch-up bursts do not invent a different workload. State whether first paint, scrolling, paging, live updates, and activated-but-hidden views are covered. Node folds, fake DOMs, and custom heartbeat events alone cannot establish browser responsiveness. + +## Prove the regression, then remove work + +Run the unoptimized workload before changing production code. Save the command, revision, runtime/platform, fixture dimensions, raw measurements, and verdict. Reduce a failing scenario until it still exercises the real bottleneck, then rank falsifiable hypotheses before patching. Use profiles, allocation samples, work counts, or phase timings to distinguish them. + +Common patterns worth testing, not automatic prescriptions: + +- Keep compact representations compact through downstream readers; avoid per-delta objects when the consumer needs settled content or one aggregate. +- Remove duplicate parsing, copying, freezing, and validation only after identifying the actual ownership and trust transition. Typed same-process borrowing is not permission to weaken durable or wire parsing. +- Stream artifact transformations and bound intermediate state rather than retaining every generation. Include publication, verification, and writable-readiness obligations where the user operation requires them. +- Separate read-only preparation from write/publication work without moving awaited work past a correctness-required endpoint. +- Defer inactive-view and collapsed-detail work; measure first activation and retained state too. Deferral is not deletion, and viewport highlighting is not full virtualization. +- Stabilize identities and narrow subscriptions so one changed node does not invalidate an entire history; preserve update ordering and immediate-event behavior. +- Prefer a suitable data structure to repeated shifting, scanning, or rebuilding. Measure the whole consumer path, not just the isolated container operation. +- Use revision-keyed reuse or singleflight only with explicit invalidation, bounded retention, independent waiter cancellation, and disposal ownership. Avoid caching expanded representations merely to make repeated benchmarks look fast. + +Change one causal factor at a time. Re-run both the focused scenario and its end-to-end parent. Require a negative control: the tightened assertion fails on the original implementation or a controlled reintroduction of the targeted cost. A threshold so generous that the regression passes is not protection; a budget below a verified noise floor is not reliable either. + +## Preserve behavior and resource ownership + +Performance measurements complement functional evidence; they do not replace it. Run or add the narrow owning tests for output, ordering, paging, stream indexes, errors, cancellation, concurrency, and disposal as applicable. Preserve model-visible/logged equivalence, released-generation immutability, atomic publication, required validation, and writable readiness. Do not silently truncate history, skip tool results, disable invariants, or change lifecycle semantics to reach a number. + +State any deliberate minor visible difference and verify it through the owning keyless snapshot. For a product-visible GUI change, include the required browser evidence/GIF. Keep functional expectations independent of benchmark internals; benchmark assertions need enough evidence to reach the real endpoint, not a second semantic test suite. + +Reject an optimization when gains disappear end-to-end, a typical workload regresses materially, complexity outweighs a small gain, or cancellation/retention/durability cannot be explained and tested. Record the rejected hypothesis briefly instead of expanding scope to justify it. + +## Deliver a bounded, reviewable result + +Use [Agent Note rules](../../notes/README.md) for durable rationale, alternatives, calibration, exclusions, and remaining risks. Check relevant notes for supersession without turning performance work into a corpus-wide prose cleanup. Keep the reusable procedure here and scenario-specific truth with its benchmark or package owner. + +When the task requests stacked PRs, choose layers before editing and use official GitHub stacks and separate worktrees. Keep each layer mergeable: benchmark infrastructure can protect the measured baseline; the optimization layer carries its fix, functional coverage, and tighter budget. Independent bottlenecks may use separate stacks. Fix a finding in its owning layer before propagating upward. + +Apply [pre-push checks](../dsh-pre-push-checks/SKILL.md), report only executed evidence, and inspect CI rather than assuming local timing proves runner stability. After marking ready, evaluate review findings against code and executable evidence; reply with the reason or fix and resolve addressed threads. Do not dismiss a report merely because it came from a bot. + +Summarize each result as: workload → before/after absolute values and ratio → endpoint and memory semantics → behavior evidence → negative control → exact checks → exclusions. Separate author-reported historical numbers, fresh local measurements, and CI evidence. Stop at the agreed scenario/fix scope; retain a short ranked follow-up list instead of chasing unrelated opportunities. diff --git a/.agents/skills/dsh-speed-up-perf/references/pr-evidence.md b/.agents/skills/dsh-speed-up-perf/references/pr-evidence.md new file mode 100644 index 0000000000..6713439b25 --- /dev/null +++ b/.agents/skills/dsh-speed-up-perf/references/pr-evidence.md @@ -0,0 +1,59 @@ +# Performance PR evidence + +## Scope + +This reference records historical evidence for choosing performance experiments, not current runtime authority. PR measurements are author-reported and were not rerun for this survey. Source inspection used revision `bdbf976558f54d6bfa775b875702dcebc355a554`; PR bodies, selected diffs, and review discussions were read through GitHub. No private Session contents were inspected or copied. Consult the current [benchmark instructions](../../../../benchmarks/AGENTS.md), [testing policy](../../../../docs/testing.md), source, and owning Agent Notes before implementation. + +## Session performance series + +| Primary source | Scenario and reported result | Reusable finding | +|---|---|---| +| [#3535](https://github.com/deepseek-harness/deepseek-harness/pull/3535), merged | Synthetic first open, first Host history, cold Agent resume, and Client fold. The [final design comment](https://github.com/deepseek-harness/deepseek-harness/pull/3535#issuecomment-5552779119) reports negative controls: first-open 4,394 ms against 550 ms; first-history 4,452 against 550; Agent resume 4,333 against 450; all three first-open 128 MB checks exhausted heap; Client fold 123.9 ms / 10.84× against 40 ms / 3.125×. | Establish executable positive and negative controls before optimization. The original PR body describes an earlier package-local design; the merged diff and final comment describe centralized compiled workers. #3587 was folded back into this PR. | +| [#3536](https://github.com/deepseek-harness/deepseek-harness/pull/3536), closed unmerged | Repeated artifact snapshots/freezes/validation occupied approximately 70% of profiled CPU. A 127,400-event synthetic open fell from 4,734–4,921 ms to 707–823 ms with identity reuse. | This implementation was superseded by streaming migration. Do not revive its snapshot registry without proving that repeated whole-artifact work still exists. | +| [#3585](https://github.com/deepseek-harness/deepseek-harness/pull/3585), merged | On one 116,228,655-byte historical input, physical decode fell from 7.527 s / 7,219 MB peak RSS to 1.467 s / 908 MB. Whole-artifact migration exhausted heap before availability; streaming migration plus serial publication completed in 6.241 s, with 2.107 GB peak RSS and 477 MB retained heap. Settled Client fold of 500,000 deltas took 3.2 ms. | Keep packed runs compact through synchronous stateful stages, retain only the final event array, and let one outer scheduler own yielding. Derive settled UI from final message content instead of replaying every delta. Attribution estimates in the PR overlap and cannot be added. | +| [#3586](https://github.com/deepseek-harness/deepseek-harness/pull/3586), merged | Same current-v2 file, three-sample medians: opening snapshot 2,011.4→1,027.9 ms; Session restore 598.5→16.0 ms; retained heap 1,025.3→478.7 MB. Historical opening snapshot measured 3,109.7 ms; writable Agent resume 4,888.6 ms, including 2,023.8 ms publication. | Separate read-only preparation from write publication. Transfer explicit immutable ownership instead of cloning/freezing at every consumer. Share preparation by source revision, with caller-local cancellation. | +| [#3537](https://github.com/deepseek-harness/deepseek-harness/pull/3537), merged | Five-sample medians on the synthetic 200-turn Session: first-open projection 28.0→5.4 ms, total 76.9→50.0 ms, peak RSS 137.2→94.9 MB; reopen total 37.7→24.4 ms. | Compact-record readers answer stats, usage, text, and image questions without allocating expanded streams. Expanded-stream caching can retain approximately ten times the compact representation for the Session lifetime. The final PR excludes Chat/Trajectory changes already supplied by #3585. | + +The source inspection revision predates #3537. Its results therefore come from its merged PR, not from running the inspected checkout. + +## Benchmark infrastructure at the inspected revision + +The [benchmark workspace instructions](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/AGENTS.md) require fixed synthetic inputs, production entry points, private temporary roots, bounded children, and cleanup after failures. Cross-package cases are grouped by user path; package-local diagnostics remain separate. + +- The [required PR job](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/.github/workflows/ci.yml#L159-L203) selects Node 24 and runs the benchmark command alone, with a 30-minute deadline. [Package scripts](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/package.json#L40-L65) build libraries and workers before Vitest orchestration; measured work uses plain Node and built package exports, not TSX. +- The [Session matrix](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/session-open/session-open.bench.ts#L273-L355) contains 12 cases: two disk states × three endpoints × normal-heap or constrained-heap execution. Disk states are historical-v0 first open and fresh-process current-v2 reopen. Endpoints are four-phase preparation, first Host history snapshot, and cold Agent resume. Normal timings use five-process medians; separate 128 MB old-space children check completion. +- The [worker timing boundaries](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/session-open/session-open.worker.ts#L160-L287) exclude imports, Host initialization, and fixture setup. First history ends at the first iterator snapshot, before live promotion, network transfer, Client fold, or paint. Forced-GC retained memory keeps the measured endpoint reachable. CPU, peak RSS, external memory, and array buffers are reported; only Agent retained heap has a separate resident-memory budget. +- [Calibration](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/support/calibration.ts) multiplies reference times by 2 for CI and 1.25 for variance. Memory and dimensionless ratios receive no machine-time multiplier. CI uses reviewed constants, not a historical checkout on every run. [Client fold](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/conversation-fold/conversation-fold.bench.client.ts) uses 200 replies, equal event/record counts with different embedded delta counts, minimum-of-three samples, and absolute plus scaling limits. +- The [manual Web performance inventory](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/vitest.web.perf.config.ts) and [browser stress inventory](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/vitest.web-stress.config.ts) are explicitly outside default CI. Their existence does not imply required browser latency coverage. + +The benchmark Agent Note at this revision still says historical read-only phase opens include publication. The actual worker calls read access; #3586 separates preparation from publication. Use the worker and persistence source for that timing distinction. + +## Additional merged performance work + +| Primary source | Evidence | Pattern or limitation | +|---|---|---| +| [#2587](https://github.com/deepseek-harness/deepseek-harness/pull/2587) | Historical 416,756-event workload: 696 packed records; client history 4,682→276 ms; sampled additional V8 peak 612.5→199.4 MB. | Preserve compact representation through validation, indexing, and folding. These historical cardinalities are not the later v2 format. The [successor diagnostic](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/packages/client/ui-conversation/tests/history-transport.perf.client.ts#L410-L480) compares embedded streams and equal final-state digests. | +| [#3331](https://github.com/deepseek-harness/deepseek-harness/pull/3331) | 10,000 collapsed Tool rows with 1 KiB JSON: 22.5→7.5 ms, retained heap 12.2→1.6 MiB. Inactive Trajectory with 10,000 individual flushes: 4,082→15.5 ms. | Defer unused JSON parsing and target materialization; share historical Inbox arrays. Context folding and first target activation still cost work. | +| [#3391](https://github.com/deepseek-harness/deepseek-harness/pull/3391) | Per-node subscriptions, stable identities, CSS replacing layout reads, bounded Trajectory inputs, and batched publication. | Its 10,000-node timing table is explicitly estimated, not measured browser evidence. | +| [#3383](https://github.com/deepseek-harness/deepseek-harness/pull/3383) | Viewport-triggered syntax highlighting. | One-shot deferral is not virtualization: visited blocks retain token DOM. | +| [#3292](https://github.com/deepseek-harness/deepseek-harness/pull/3292) | Two-million-item FIFO drain: 9.656 ms median, excluding enqueue. | Deque avoids array-shift copying; it does not provide admission control or backpressure. | +| [#1161](https://github.com/deepseek-harness/deepseek-harness/pull/1161) | Keyless 100,000-reasoning-chunk browser stress, paced at 128 chunks per 16 ms, with heartbeat and scheduled-event latency budgets. | Scheduled synthetic events are not real pointer/keyboard interactions; producer catch-up and heartbeat endpoints need scrutiny. | + +The [complex-history diagnostic](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/apps/web/tests/complex-history.perf.ts) supplies broader synthetic building blocks: 1,000 sidebar Sessions, 500 turns/tools, a 24-turn initial window, continued turns, history/search/warm switches, and a 100-turn soak. It records browser task/layout/style time, DOM/listener counts, and forced-GC heap. Its timings are observational rather than required performance verdicts. + +## Review pitfalls worth retaining + +- Compare equivalent work and ownership. The [#2587 baseline review](https://github.com/deepseek-harness/deepseek-harness/pull/2587#discussion_r3803082730) requires full-response validation and retained parsed arrays on both sides, not parse-and-discard on one side. +- Preserve exact cancellation and revision semantics. [#3586 cancellation](https://github.com/deepseek-harness/deepseek-harness/pull/3586#discussion_r3940578092) lets started publication settle, then rejects the canceled write open and releases its lease; [source drift](https://github.com/deepseek-harness/deepseek-harness/pull/3586#discussion_r3940569241) invalidates preparation rather than silently replacing returned history. +- Record validation tradeoffs explicitly. [#3585](https://github.com/deepseek-harness/deepseek-harness/pull/3585#discussion_r3940324353) deferred full historical scalar payload-member validation for performance; #3586 distinguishes ordinary current-file restore from strict publication-worker replay. [#3537 readers](https://github.com/deepseek-harness/deepseek-harness/pull/3537#discussion_r3942974015) trust typed records rather than validating arbitrary durable data. +- Measure the producer as well as the renderer. [#1161 review](https://github.com/deepseek-harness/deepseek-harness/pull/1161#discussion_r3699970161) identifies catch-up bursts; [heartbeat review](https://github.com/deepseek-harness/deepseek-harness/pull/1161#discussion_r3699970162) identifies final-stall undercount. These are measurement caveats, not newly reproduced failures. +- An isolated job does not guarantee an isolated physical host. [#3535 failover review](https://github.com/deepseek-harness/deepseek-harness/pull/3535#discussion_r3927945561) warns that multiple runners on one standby VM can contaminate wall-clock budgets. + +## Bounded scenario hypotheses + +These candidates require profiling; they are not claims of defects. + +1. Add tool-heavy and many-turn distributions beside the text/reasoning Session workload. Keep read-only opening, current reopen, and writable resume distinct, with component attribution and independent end-to-end clocks. +2. Measure many-child-Session list/observe-to-resume work before changing caches. #3586 identifies bulk subagent body reads as remaining work; avoid optimizing an already removed artifact-snapshot design. +3. Measure browser first usable paint, older-page loading, scrolling, expanding one large result, and first Trajectory activation separately from Host snapshot latency. Reuse complex-history fixtures while adding explicit, calibrated verdicts and real interaction probes. +4. Isolate reconnect during a long active Assistant attempt. The [inspected reconnect implementation](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/packages/api/session-controller/src/client/sessions/assistant-stream.ts#L52-L94) expands a compact baseline into transient entries, unlike settled history. Preserve ordering, attempt identity, and next-index semantics in any experiment. From 6cc750b0067c38e8fdc0920a1f78a1858264558a Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Sun, 6 Sep 2026 14:50:53 +0800 Subject: [PATCH 2/2] docs(skills): route performance evidence through decision note --- ...vidence-driven-performance-skill.i18n.yaml | 4 +- ...09-06-evidence-driven-performance-skill.md | 21 ++++++- ...06-evidence-driven-performance-skill.zh.md | 29 +++++++-- .agents/skills/dsh-speed-up-perf/SKILL.md | 4 +- .../references/pr-evidence.md | 59 ------------------- 5 files changed, 48 insertions(+), 69 deletions(-) delete mode 100644 .agents/skills/dsh-speed-up-perf/references/pr-evidence.md diff --git a/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.i18n.yaml b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.i18n.yaml index ba3a54c1c1..53de712240 100644 --- a/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.i18n.yaml +++ b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md -2026-09-06-evidence-driven-performance-skill.md: 66f04f242afc2e70a6af7f8b6558e763b2e33617 -2026-09-06-evidence-driven-performance-skill.zh.md: b82aa37cb017054e2f35a97af50b71763f0cd86a +2026-09-06-evidence-driven-performance-skill.md: 5b15cce1adbd7ff47e5668f7332cba8d1b59e5fe +2026-09-06-evidence-driven-performance-skill.zh.md: c1fbbd76740badd87ed0a95218e2c4082f2c0e8b diff --git a/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md index 66f04f242a..5b15cce1ad 100644 --- a/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md +++ b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md @@ -10,12 +10,31 @@ Performance work can improve an isolated phase while moving cost into another ph ## Decision -The [dsh-speed-up-perf skill](../../../skills/dsh-speed-up-perf/SKILL.md) guides broad surveys toward bounded, measured user paths. It combines focused attribution with independently timed backend and browser endpoints, synthetic workload distributions, comparable cold/warm and retained-memory conditions, and negative controls for tightened budgets. Its [evidence reference](../../../skills/dsh-speed-up-perf/references/pr-evidence.md) distinguishes merged implementations, superseded proposals, author-reported measurements, and estimates. +The [dsh-speed-up-perf skill](../../../skills/dsh-speed-up-perf/SKILL.md) guides broad surveys toward bounded, measured user paths. It combines focused attribution with independently timed backend and browser endpoints, synthetic workload distributions, comparable cold/warm and retained-memory conditions, and negative controls for tightened budgets. The historical evidence below distinguishes merged implementations, superseded proposals, author-reported measurements, and estimates. The workflow requires behavior evidence independently of timing: model-visible logs, durable generation and publication rules, stream ordering, cancellation, and disposal remain obligations. Authorized private corpus inspection yields only aggregate workload inspiration; committed inputs and published artifacts contain synthetic material. Optimization PRs carry their tighter budgets, while a preceding benchmark layer can protect the measured baseline and remain independently mergeable. The [Session-opening performance-gate decision](../testing/2026-09-04-session-open-performance-gate.md) retains ownership of lane mechanics and calibration. The [simplification skill](../../../skills/dsh-find-simplifications/SKILL.md) retains ownership of deletion-oriented surveys. Neither is superseded: this workflow adds performance-specific candidate selection, measurement comparability, and stopping criteria rather than replacing their decisions. +## Historical evidence + +These are author-reported historical measurements, not benchmarks rerun for this workflow. Final merged diffs and owning source take precedence over original PR descriptions. The rejected intermediate proposal is retained only to explain why identity registries are not a general prescription. + +| Evidence | Measured path and result | Reusable lesson | +|---|---|---| +| [#3535](https://github.com/deepseek-harness/deepseek-harness/pull/3535), merged | The [final benchmark design](https://github.com/deepseek-harness/deepseek-harness/pull/3535#issuecomment-5552779119) reports a 4,394 ms first-open negative control against 550 ms, first-history 4,452 against 550, resume 4,333 against 450, and 128 MB heap failures. Client fold: 123.9 ms / 10.84× against 40 ms / 3.125×. | Built-JS user-path gates and positive/negative controls matter more than an earlier PR-body design. | +| [#3536](https://github.com/deepseek-harness/deepseek-harness/pull/3536), closed unmerged | Repeated snapshot/freeze work occupied about 70% of profiled CPU; synthetic open improved from 4,734–4,921 to 707–823 ms. | Streaming migration superseded this identity-registry proposal. Do not revive it without current ownership evidence. | +| [#3585](https://github.com/deepseek-harness/deepseek-harness/pull/3585), merged | Historical physical decode: 7.527 s / 7,219 MB peak RSS to 1.467 s / 908 MB; streaming migration with serial publication: 6.241 s, 2.107 GB peak, 477 MB retained. Settled 500,000-delta Client fold: 3.2 ms. | Keep representations compact across consumers; bound intermediate state. Attribution estimates overlap and cannot be added. | +| [#3586](https://github.com/deepseek-harness/deepseek-harness/pull/3586), merged | Current-v2 opening snapshot: 2,011.4→1,027.9 ms; restore: 598.5→16 ms; retained heap: 1,025.3→478.7 MB. | Separate read-only preparation from awaited write publication; share immutable ownership with revision-keyed preparation and caller-local cancellation. | +| [#3537](https://github.com/deepseek-harness/deepseek-harness/pull/3537), merged | Synthetic 200-turn projection: 28→5.4 ms; total: 76.9→50 ms; peak RSS: 137.2→94.9 MB. | Read stats, usage, text and image references per compact record. Expanded-stream caching retains unnecessary representation cost. Chat/Trajectory belong to the preceding migration change. | +| [#2587](https://github.com/deepseek-harness/deepseek-harness/pull/2587), merged | Historical 416,756 events represented by 696 records: client history 4,682→276 ms; sampled additional V8 peak 612.5→199.4 MB. | Preserve compactness through validation and folding; [baseline review](https://github.com/deepseek-harness/deepseek-harness/pull/2587#discussion_r3803082730) requires equal validation and retained output, not parse-and-discard. | +| [#3331](https://github.com/deepseek-harness/deepseek-harness/pull/3331), merged | 10,000 collapsed tool rows: 22.5→7.5 ms, retained 12.2→1.6 MiB; inactive Trajectory flushes: 4,082→15.5 ms. | Defer unused parsing and materialization; first activation and retained Context still cost work. | +| [#3391](https://github.com/deepseek-harness/deepseek-harness/pull/3391) and [#3383](https://github.com/deepseek-harness/deepseek-harness/pull/3383), merged | Narrow subscriptions, stable identities, batched publication, and viewport-triggered highlighting. The 10,000-node timing table is estimated, not browser measurement. | Deferral is not virtualization: visited token DOM remains retained. | +| [#3292](https://github.com/deepseek-harness/deepseek-harness/pull/3292), merged | Two-million-item FIFO drain: 9.656 ms median, excluding enqueue. | A deque removes shift copying, not queue admission or backpressure obligations. | +| [#1161](https://github.com/deepseek-harness/deepseek-harness/pull/1161), merged | Keyless 100,000-chunk browser stress at 128 chunks per 16 ms. | [Producer catch-up](https://github.com/deepseek-harness/deepseek-harness/pull/1161#discussion_r3699970161) and [final heartbeat stalls](https://github.com/deepseek-harness/deepseek-harness/pull/1161#discussion_r3699970162) can distort measurements; scheduled events are not trusted keyboard/pointer input. | + +The [cancellation review](https://github.com/deepseek-harness/deepseek-harness/pull/3586#discussion_r3940578092), [source-revision review](https://github.com/deepseek-harness/deepseek-harness/pull/3586#discussion_r3940569241), and [typed-reader review](https://github.com/deepseek-harness/deepseek-harness/pull/3537#discussion_r3942974015) illustrate why removing repeated work does not authorize deleting validation or publication obligations. A [standby-runner review](https://github.com/deepseek-harness/deepseek-harness/pull/3535#discussion_r3927945561) distinguishes a dedicated job from an isolated physical host. + ## Alternatives considered **Optimize suspicious code before measuring.** Rejected because local complexity does not identify dominant user cost and cannot establish improvement or regression protection. diff --git a/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.zh.md b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.zh.md index b82aa37cb0..c1fbbd7674 100644 --- a/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.zh.md +++ b/.agents/notes/implemented/process/2026-09-06-evidence-driven-performance-skill.zh.md @@ -4,19 +4,38 @@ Status: implemented [English](2026-09-06-evidence-driven-performance-skill.md) | 中文 -## Problem +## 问题 性能工作可能改善某个独立阶段,却把成本转移到另一阶段、保留更多数据,或跳过必要行为。历史 PR(Pull Request)描述也可能保留已放弃的实现和估计值,因此照搬其表面方案可能恢复已否决的设计,而不是解决当前瓶颈。 -## Decision +## 决定 -[dsh-speed-up-perf skill](../../../skills/dsh-speed-up-perf/SKILL.md)(技能)引导广泛调查收敛到范围明确、可测量的用户路径。它结合聚焦的成本归因与独立计时的后端和浏览器端点、合成负载分布、可比较的冷态/热态与保留内存条件,以及收紧预算的负向对照。其[证据参考](../../../skills/dsh-speed-up-perf/references/pr-evidence.md)区分已合并实现、已被替代的提案、作者报告的测量值和估计值。 +[dsh-speed-up-perf skill](../../../skills/dsh-speed-up-perf/SKILL.md)(技能)引导广泛调查收敛到范围明确、可测量的用户路径。它结合聚焦的成本归因与独立计时的后端和浏览器端点、合成负载分布、可比较的冷态/热态与保留内存条件,以及收紧预算的负向对照。下方历史证据区分已合并实现、已被替代的提案、作者报告的测量值和估计值。 该工作流要求独立于计时的行为证据:模型可见日志、持久化代际和发布规则、流顺序、取消及 dispose(资源释放)仍是必须满足的要求。获授权的私有语料检查仅提供聚合负载启发;提交的输入和发布的产物包含合成材料。优化 PR 携带收紧后的预算,而前置基准测试层可以保护已测基线并保持独立可合并。 [会话打开性能门禁决策](../testing/2026-09-04-session-open-performance-gate.zh.md)继续负责测试通道机制与校准。[简化 skill](../../../skills/dsh-find-simplifications/SKILL.md)继续负责以删除为目标的调查。两者均未被替代:本工作流增加面向性能的候选选择、测量可比性和停止条件,而不替换它们的决策。 -## Alternatives considered +## 历史证据 + +这些是作者报告的历史测量,并非为本工作流重新运行的基准测试。最终合并差异与所属源码优先于最初 PR 描述。保留已否决的中间提案,仅用于解释为何身份注册表不是通用处方。 + +| 证据 | 测量路径与结果 | 可复用经验 | +|---|---|---| +| [#3535](https://github.com/deepseek-harness/deepseek-harness/pull/3535),已合并 | [最终基准设计](https://github.com/deepseek-harness/deepseek-harness/pull/3535#issuecomment-5552779119)报告首次打开负向对照 4,394 ms,预算 550 ms;首屏历史 4,452,预算 550;恢复 4,333,预算 450;128 MB 堆检查失败。Client fold:123.9 ms / 10.84×,预算 40 ms / 3.125×。 | built-JS 用户路径门禁与正/负向对照比早期 PR 正文设计更重要。 | +| [#3536](https://github.com/deepseek-harness/deepseek-harness/pull/3536),关闭未合并 | 重复 snapshot/freeze 工作占采样 CPU 的约 70%;合成打开从 4,734–4,921 改善为 707–823 ms。 | 流式迁移替代了该身份注册表提案。没有当前所有权证据时,不恢复它。 | +| [#3585](https://github.com/deepseek-harness/deepseek-harness/pull/3585),已合并 | 历史物理解码:7.527 s / 7,219 MB 峰值 RSS 降至 1.467 s / 908 MB;流式迁移加串行发布:6.241 s,2.107 GB 峰值,477 MB 保留。已结算的 500,000-delta Client fold:3.2 ms。 | 跨消费者保持紧凑表示;限制中间状态。归因估计重叠,不能相加。 | +| [#3586](https://github.com/deepseek-harness/deepseek-harness/pull/3586),已合并 | 当前 v2 打开快照:2,011.4→1,027.9 ms;恢复:598.5→16 ms;保留堆:1,025.3→478.7 MB。 | 分离只读准备与必须等待的写发布;通过按修订号共享准备和调用方局部取消共享不可变所有权。 | +| [#3537](https://github.com/deepseek-harness/deepseek-harness/pull/3537),已合并 | 合成 200 轮投影:28→5.4 ms;总计:76.9→50 ms;峰值 RSS:137.2→94.9 MB。 | 按紧凑记录读取统计、usage、文本和图像引用。展开流缓存保留不必要的表示成本。Chat/Trajectory 属于前置迁移改动。 | +| [#2587](https://github.com/deepseek-harness/deepseek-harness/pull/2587),已合并 | 历史 416,756 事件由 696 记录表示:Client 历史 4,682→276 ms;采样额外 V8 峰值 612.5→199.4 MB。 | 验证和折叠过程保持紧凑;[基线审查](https://github.com/deepseek-harness/deepseek-harness/pull/2587#discussion_r3803082730)要求相同验证与保留输出,而不是解析后丢弃。 | +| [#3331](https://github.com/deepseek-harness/deepseek-harness/pull/3331),已合并 | 10,000 个折叠工具行:22.5→7.5 ms,保留 12.2→1.6 MiB;非活动 Trajectory 刷新:4,082→15.5 ms。 | 延迟未使用的解析和实体化;首次激活与保留 Context 仍有成本。 | +| [#3391](https://github.com/deepseek-harness/deepseek-harness/pull/3391) 和 [#3383](https://github.com/deepseek-harness/deepseek-harness/pull/3383),已合并 | 缩小订阅范围、稳定身份、批量发布和视口触发高亮。10,000 节点计时表是估计,不是浏览器测量。 | 延迟不等于虚拟化:访问过的 token DOM 仍被保留。 | +| [#3292](https://github.com/deepseek-harness/deepseek-harness/pull/3292),已合并 | 两百万条 FIFO 排空:中位数 9.656 ms,不含入队。 | deque 删除 shift 复制,不删除队列准入或背压义务。 | +| [#1161](https://github.com/deepseek-harness/deepseek-harness/pull/1161),已合并 | 无密钥的 100,000-chunk 浏览器压力测试,每 16 ms 推送 128 个 chunk。 | [生产者追赶](https://github.com/deepseek-harness/deepseek-harness/pull/1161#discussion_r3699970161)和[最后一次心跳停顿](https://github.com/deepseek-harness/deepseek-harness/pull/1161#discussion_r3699970162)可能扭曲测量;定时派发事件不是真实键盘/指针输入。 | + +[取消审查](https://github.com/deepseek-harness/deepseek-harness/pull/3586#discussion_r3940578092)、[源修订审查](https://github.com/deepseek-harness/deepseek-harness/pull/3586#discussion_r3940569241)和[类型化读取器审查](https://github.com/deepseek-harness/deepseek-harness/pull/3537#discussion_r3942974015)说明删除重复工作不等于允许删除验证或发布义务。[备用 runner 审查](https://github.com/deepseek-harness/deepseek-harness/pull/3535#discussion_r3927945561)区分独立 job 与隔离的物理主机。 + +## 考虑过的替代方案 **先优化可疑代码,再测量。** 否决,因为局部复杂度不能确定主要用户成本,也无法证明改善或防止回归。 @@ -24,6 +43,6 @@ Status: implemented **只使用微基准测试,或只使用端到端计时。** 否决,因为独立阶段可能遗漏被转移的工作,而总计时无法定位原因。两者都需要在与所选问题相符的范围内使用。 -## Consequences +## 后果 该 skill 不增加运行时行为、基准测试实现或新的 CI 策略。其验证涵盖文档/链接一致性和 skill 元数据;后续每项优化在其所属位置提供可执行测量与功能证据。有限的场景/修复范围防止广泛性能请求演变成无关的架构重写。 diff --git a/.agents/skills/dsh-speed-up-perf/SKILL.md b/.agents/skills/dsh-speed-up-perf/SKILL.md index 68eed10147..fbd5d262e1 100644 --- a/.agents/skills/dsh-speed-up-perf/SKILL.md +++ b/.agents/skills/dsh-speed-up-perf/SKILL.md @@ -13,7 +13,7 @@ Read [AGENTS.md](../../../AGENTS.md), [architecture](../../../docs/architecture. Agree on the user-visible endpoint, workload range, resource constraints, acceptable minor behavior differences, and stopping rule. Keep backend and browser end-to-end measurements separate: a fast history iterator or Client fold does not prove fast transport, paint, scrolling, or input response. Exclude model/network latency when measuring local overhead, and state that exclusion rather than calling the result complete product latency. -Inspect the exact current base, not just the running checkout. Study final merged diffs, owning source, tests, and resolved review threads; a PR body can describe an abandoned implementation. Separate merged, closed-unmerged, superseded, estimated, and newly measured evidence. The [PR evidence reference](references/pr-evidence.md) supplies historical leads, not authority to reintroduce their implementations. +Inspect the exact current base, not just the running checkout. Study final merged diffs, owning source, tests, and resolved review threads; a PR body can describe an abandoned implementation. Separate merged, closed-unmerged, superseded, estimated, and newly measured evidence. The [performance workflow decision and evidence](../../notes/implemented/process/2026-09-06-evidence-driven-performance-skill.md) supply historical leads, not authority to reintroduce their implementations. ## Survey user paths, then rank candidates @@ -50,7 +50,7 @@ Measure built JavaScript under plain Node for CPU workers; source-loader overhea Use fresh children and private temporary roots for cold/process-memory samples. Warm samples explicitly retain the intended cache; never let fixture setup secretly warm a cold scenario. Keep the same input, validations, completion condition, and reachable output on both sides. A parse-and-discard baseline is not comparable with validated retained history. -Report all samples and the aggregate that decides the result. Use the existing shared time calibration and reviewed variance headroom; do not scale bytes, counts, or dimensionless ratios by CPU speed. Budgets are source constants, not environment overrides. Serialize measured work against other owned CPU-heavy jobs; measure reference and candidate under comparable conditions. Do not widen a budget or select a lucky run to hide a regression. +Report all samples and the aggregate that decides the result. For the Node lane, use the existing shared time calibration and reviewed variance headroom; do not scale bytes, counts, or dimensionless ratios by CPU speed. Keep manual browser diagnostics threshold-free. A required browser performance case needs an explicit lane decision and repeated measurements on its actual CI browser/runner before adopting timing budgets; the Node machine multiplier alone is not browser calibration. Budgets are source constants, not environment overrides. Serialize measured work against other owned CPU-heavy jobs; measure reference and candidate under comparable conditions. Do not widen a budget or select a lucky run to hide a regression. Measure end-to-end latency independently from component phases. Track retained memory with intended objects still reachable, and transient pressure separately through constrained-heap completion or an appropriate peak measurement. Faster execution with unbounded retention is not an automatic win. diff --git a/.agents/skills/dsh-speed-up-perf/references/pr-evidence.md b/.agents/skills/dsh-speed-up-perf/references/pr-evidence.md deleted file mode 100644 index 6713439b25..0000000000 --- a/.agents/skills/dsh-speed-up-perf/references/pr-evidence.md +++ /dev/null @@ -1,59 +0,0 @@ -# Performance PR evidence - -## Scope - -This reference records historical evidence for choosing performance experiments, not current runtime authority. PR measurements are author-reported and were not rerun for this survey. Source inspection used revision `bdbf976558f54d6bfa775b875702dcebc355a554`; PR bodies, selected diffs, and review discussions were read through GitHub. No private Session contents were inspected or copied. Consult the current [benchmark instructions](../../../../benchmarks/AGENTS.md), [testing policy](../../../../docs/testing.md), source, and owning Agent Notes before implementation. - -## Session performance series - -| Primary source | Scenario and reported result | Reusable finding | -|---|---|---| -| [#3535](https://github.com/deepseek-harness/deepseek-harness/pull/3535), merged | Synthetic first open, first Host history, cold Agent resume, and Client fold. The [final design comment](https://github.com/deepseek-harness/deepseek-harness/pull/3535#issuecomment-5552779119) reports negative controls: first-open 4,394 ms against 550 ms; first-history 4,452 against 550; Agent resume 4,333 against 450; all three first-open 128 MB checks exhausted heap; Client fold 123.9 ms / 10.84× against 40 ms / 3.125×. | Establish executable positive and negative controls before optimization. The original PR body describes an earlier package-local design; the merged diff and final comment describe centralized compiled workers. #3587 was folded back into this PR. | -| [#3536](https://github.com/deepseek-harness/deepseek-harness/pull/3536), closed unmerged | Repeated artifact snapshots/freezes/validation occupied approximately 70% of profiled CPU. A 127,400-event synthetic open fell from 4,734–4,921 ms to 707–823 ms with identity reuse. | This implementation was superseded by streaming migration. Do not revive its snapshot registry without proving that repeated whole-artifact work still exists. | -| [#3585](https://github.com/deepseek-harness/deepseek-harness/pull/3585), merged | On one 116,228,655-byte historical input, physical decode fell from 7.527 s / 7,219 MB peak RSS to 1.467 s / 908 MB. Whole-artifact migration exhausted heap before availability; streaming migration plus serial publication completed in 6.241 s, with 2.107 GB peak RSS and 477 MB retained heap. Settled Client fold of 500,000 deltas took 3.2 ms. | Keep packed runs compact through synchronous stateful stages, retain only the final event array, and let one outer scheduler own yielding. Derive settled UI from final message content instead of replaying every delta. Attribution estimates in the PR overlap and cannot be added. | -| [#3586](https://github.com/deepseek-harness/deepseek-harness/pull/3586), merged | Same current-v2 file, three-sample medians: opening snapshot 2,011.4→1,027.9 ms; Session restore 598.5→16.0 ms; retained heap 1,025.3→478.7 MB. Historical opening snapshot measured 3,109.7 ms; writable Agent resume 4,888.6 ms, including 2,023.8 ms publication. | Separate read-only preparation from write publication. Transfer explicit immutable ownership instead of cloning/freezing at every consumer. Share preparation by source revision, with caller-local cancellation. | -| [#3537](https://github.com/deepseek-harness/deepseek-harness/pull/3537), merged | Five-sample medians on the synthetic 200-turn Session: first-open projection 28.0→5.4 ms, total 76.9→50.0 ms, peak RSS 137.2→94.9 MB; reopen total 37.7→24.4 ms. | Compact-record readers answer stats, usage, text, and image questions without allocating expanded streams. Expanded-stream caching can retain approximately ten times the compact representation for the Session lifetime. The final PR excludes Chat/Trajectory changes already supplied by #3585. | - -The source inspection revision predates #3537. Its results therefore come from its merged PR, not from running the inspected checkout. - -## Benchmark infrastructure at the inspected revision - -The [benchmark workspace instructions](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/AGENTS.md) require fixed synthetic inputs, production entry points, private temporary roots, bounded children, and cleanup after failures. Cross-package cases are grouped by user path; package-local diagnostics remain separate. - -- The [required PR job](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/.github/workflows/ci.yml#L159-L203) selects Node 24 and runs the benchmark command alone, with a 30-minute deadline. [Package scripts](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/package.json#L40-L65) build libraries and workers before Vitest orchestration; measured work uses plain Node and built package exports, not TSX. -- The [Session matrix](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/session-open/session-open.bench.ts#L273-L355) contains 12 cases: two disk states × three endpoints × normal-heap or constrained-heap execution. Disk states are historical-v0 first open and fresh-process current-v2 reopen. Endpoints are four-phase preparation, first Host history snapshot, and cold Agent resume. Normal timings use five-process medians; separate 128 MB old-space children check completion. -- The [worker timing boundaries](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/session-open/session-open.worker.ts#L160-L287) exclude imports, Host initialization, and fixture setup. First history ends at the first iterator snapshot, before live promotion, network transfer, Client fold, or paint. Forced-GC retained memory keeps the measured endpoint reachable. CPU, peak RSS, external memory, and array buffers are reported; only Agent retained heap has a separate resident-memory budget. -- [Calibration](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/support/calibration.ts) multiplies reference times by 2 for CI and 1.25 for variance. Memory and dimensionless ratios receive no machine-time multiplier. CI uses reviewed constants, not a historical checkout on every run. [Client fold](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/benchmarks/conversation-fold/conversation-fold.bench.client.ts) uses 200 replies, equal event/record counts with different embedded delta counts, minimum-of-three samples, and absolute plus scaling limits. -- The [manual Web performance inventory](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/vitest.web.perf.config.ts) and [browser stress inventory](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/vitest.web-stress.config.ts) are explicitly outside default CI. Their existence does not imply required browser latency coverage. - -The benchmark Agent Note at this revision still says historical read-only phase opens include publication. The actual worker calls read access; #3586 separates preparation from publication. Use the worker and persistence source for that timing distinction. - -## Additional merged performance work - -| Primary source | Evidence | Pattern or limitation | -|---|---|---| -| [#2587](https://github.com/deepseek-harness/deepseek-harness/pull/2587) | Historical 416,756-event workload: 696 packed records; client history 4,682→276 ms; sampled additional V8 peak 612.5→199.4 MB. | Preserve compact representation through validation, indexing, and folding. These historical cardinalities are not the later v2 format. The [successor diagnostic](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/packages/client/ui-conversation/tests/history-transport.perf.client.ts#L410-L480) compares embedded streams and equal final-state digests. | -| [#3331](https://github.com/deepseek-harness/deepseek-harness/pull/3331) | 10,000 collapsed Tool rows with 1 KiB JSON: 22.5→7.5 ms, retained heap 12.2→1.6 MiB. Inactive Trajectory with 10,000 individual flushes: 4,082→15.5 ms. | Defer unused JSON parsing and target materialization; share historical Inbox arrays. Context folding and first target activation still cost work. | -| [#3391](https://github.com/deepseek-harness/deepseek-harness/pull/3391) | Per-node subscriptions, stable identities, CSS replacing layout reads, bounded Trajectory inputs, and batched publication. | Its 10,000-node timing table is explicitly estimated, not measured browser evidence. | -| [#3383](https://github.com/deepseek-harness/deepseek-harness/pull/3383) | Viewport-triggered syntax highlighting. | One-shot deferral is not virtualization: visited blocks retain token DOM. | -| [#3292](https://github.com/deepseek-harness/deepseek-harness/pull/3292) | Two-million-item FIFO drain: 9.656 ms median, excluding enqueue. | Deque avoids array-shift copying; it does not provide admission control or backpressure. | -| [#1161](https://github.com/deepseek-harness/deepseek-harness/pull/1161) | Keyless 100,000-reasoning-chunk browser stress, paced at 128 chunks per 16 ms, with heartbeat and scheduled-event latency budgets. | Scheduled synthetic events are not real pointer/keyboard interactions; producer catch-up and heartbeat endpoints need scrutiny. | - -The [complex-history diagnostic](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/apps/web/tests/complex-history.perf.ts) supplies broader synthetic building blocks: 1,000 sidebar Sessions, 500 turns/tools, a 24-turn initial window, continued turns, history/search/warm switches, and a 100-turn soak. It records browser task/layout/style time, DOM/listener counts, and forced-GC heap. Its timings are observational rather than required performance verdicts. - -## Review pitfalls worth retaining - -- Compare equivalent work and ownership. The [#2587 baseline review](https://github.com/deepseek-harness/deepseek-harness/pull/2587#discussion_r3803082730) requires full-response validation and retained parsed arrays on both sides, not parse-and-discard on one side. -- Preserve exact cancellation and revision semantics. [#3586 cancellation](https://github.com/deepseek-harness/deepseek-harness/pull/3586#discussion_r3940578092) lets started publication settle, then rejects the canceled write open and releases its lease; [source drift](https://github.com/deepseek-harness/deepseek-harness/pull/3586#discussion_r3940569241) invalidates preparation rather than silently replacing returned history. -- Record validation tradeoffs explicitly. [#3585](https://github.com/deepseek-harness/deepseek-harness/pull/3585#discussion_r3940324353) deferred full historical scalar payload-member validation for performance; #3586 distinguishes ordinary current-file restore from strict publication-worker replay. [#3537 readers](https://github.com/deepseek-harness/deepseek-harness/pull/3537#discussion_r3942974015) trust typed records rather than validating arbitrary durable data. -- Measure the producer as well as the renderer. [#1161 review](https://github.com/deepseek-harness/deepseek-harness/pull/1161#discussion_r3699970161) identifies catch-up bursts; [heartbeat review](https://github.com/deepseek-harness/deepseek-harness/pull/1161#discussion_r3699970162) identifies final-stall undercount. These are measurement caveats, not newly reproduced failures. -- An isolated job does not guarantee an isolated physical host. [#3535 failover review](https://github.com/deepseek-harness/deepseek-harness/pull/3535#discussion_r3927945561) warns that multiple runners on one standby VM can contaminate wall-clock budgets. - -## Bounded scenario hypotheses - -These candidates require profiling; they are not claims of defects. - -1. Add tool-heavy and many-turn distributions beside the text/reasoning Session workload. Keep read-only opening, current reopen, and writable resume distinct, with component attribution and independent end-to-end clocks. -2. Measure many-child-Session list/observe-to-resume work before changing caches. #3586 identifies bulk subagent body reads as remaining work; avoid optimizing an already removed artifact-snapshot design. -3. Measure browser first usable paint, older-page loading, scrolling, expanding one large result, and first Trajectory activation separately from Host snapshot latency. Reuse complex-history fixtures while adding explicit, calibrated verdicts and real interaction probes. -4. Isolate reconnect during a long active Assistant attempt. The [inspected reconnect implementation](https://github.com/deepseek-harness/deepseek-harness/blob/bdbf976558f54d6bfa775b875702dcebc355a554/packages/api/session-controller/src/client/sessions/assistant-stream.ts#L52-L94) expands a compact baseline into transient entries, unlike settled history. Preserve ordering, attempt identity, and next-index semantics in any experiment.