mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-08-29 04:26:38 +00:00
fix(fixtures): repair the cancel-fixture abort error placement; exempt pwsh-local coverage on pwsh-less hosts
The hand-edit in 348ab41151 put the seq-17 abort error inside the tool-result message and dropped a closing brace, breaking every JSONL consumer; tool-calls.ts appends 'error' as a data-level sibling of 'message' (the seq-19 shape), so the fixture now matches the emitter. Coverage: pwsh-local's executor suites self-skip without a real pwsh, which left per-file 100% unreachable on pwsh-less contributor hosts (mirror of the existing windowsCoverageExclusions contract). A PATH-only probe exempts only pwsh-local/src/index.ts; CI runners ship pwsh and still enforce the full bar. docs/testing.md (+zh, pairing re-recorded) names the prerequisite; the testing.md budget rises 1120->1150 because the coverage-gate contract genuinely grew.
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/testing.md
|
||||
testing.md: d460d9ba76eab84b79e8c66454325d03d7c5908e
|
||||
testing.zh.md: 821e49b6593b924e754d5ed0d588dd1b5b566051
|
||||
testing.md: 5e766bb684d9d76a1c9e854a2aef95796a31ec12
|
||||
testing.zh.md: 21b3f98e9ddbbb6f77621cff9ce206ae17125bc3
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning
|
||||
## Tiers
|
||||
|
||||
- **Unit** (`pnpm run test`): vitest over package and example specs under their `tests/**` directories plus repository script specs under `scripts/**/*.spec.ts`; tests stay with the code area they exercise. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, concurrency races, and permanent contract regressions (see `packages/core/agent-loop/tests/contract-regressions.spec.ts`).
|
||||
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
|
||||
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped. Per-file 100% on `packages/bash/pwsh-local/src` needs a real `pwsh`: without one its executor suites self-skip and `vitest.config.ts` exempts the file so pwsh-less hosts stay green, while CI runners ship pwsh and enforce the full bar.
|
||||
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
|
||||
- **Snapshot** (`pnpm run test:snapshot`): keyless expected outputs cover external behavior — transport contracts and presentation, while persisted logs pin assembled backend behavior. ACP boots the real automation-server example, replays a recorded session, and diffs normalized JSON-RPC plus the re-persisted log ([ACP snapshot Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)); headless pins `stream-json` through its real one-shot process. TUI journeys replay primary/child JSONL through the real loop and tools, then project ANSI into semantic terminal-state outputs; package snapshots retain transient states and a real PTY covers the process boundary ([TUI snapshot Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript changes and `pnpm run test:snapshot:refresh` when replay input remains valid; review every JSONL and expected-output diff. One ACP scenario (`text-turn`) pins full system-prompt/tool-schema content; other fixtures tokenize it so an edit churns one line ([pinned-header Agent Note](../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **Web browser snapshot** (`pnpm run test:web`; required Linux PR gate): Chromium compares replayed browser output with `apps/web/tests/snapshots/`. CI forces read-only `DSH_SNAPSHOT=replay`, never writing expected outputs; record/refresh stay local and every diff is reviewed ([web e2e lane](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md), [CI gate decision](../.agents/notes/implemented/testing/2026-07-30-web-browser-snapshot-ci-gate.md)). `test:web` [builds first](../.agents/notes/implemented/bug-fix/2026-07-28-themed-scrollbars-and-reserved-gutter.md) for plugin CSS.
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@
|
||||
## 层级
|
||||
|
||||
- **单元测试**(`pnpm run test`):vitest 运行包(package)和示例各自的 `tests/**` 目录下的测试,以及匹配 `scripts/**/*.spec.ts` 的仓库脚本测试;测试文件与其所覆盖的代码区域放在一起。每个注册表都有一个 HMR(热模块替换)安全测试(dispose(资源释放)贡献的 fiber,断言清理完成)。优先覆盖边界情况、错误路径、事件顺序、并发竞态,以及永久性契约回归(见 `packages/core/agent-loop/tests/contract-regressions.spec.ts`)。
|
||||
- **覆盖率门禁**(`pnpm run test:coverage`):门禁级运行,对 `packages/*/*/src` 按文件 100% 覆盖。未覆盖的行往往是门禁正确标记出的死代码(应删除),而非需要补写的测试。行覆盖率是必要条件,但永远不是充分条件:它证明行被执行过,不证明功能按交付预期工作。
|
||||
- **覆盖率门禁**(`pnpm run test:coverage`):门禁级运行,对 `packages/*/*/src` 按文件 100% 覆盖。未覆盖的行往往是门禁正确标记出的死代码(应删除),而非需要补写的测试。行覆盖率是必要条件,但永远不是充分条件:它证明行被执行过,不证明功能按交付预期工作。`packages/bash/pwsh-local/src` 的按文件 100% 覆盖需要真实的 `pwsh`:缺少它时其 executor 套件会自动跳过,`vitest.config.ts` 会豁免该文件以使无 pwsh 的主机保持绿色,而 CI runner 自带 pwsh,仍按完整标准执行门禁。
|
||||
- **真实 API e2e**(`pnpm run test:e2e`):带密钥测试调用真实提供方 API,包括 DeepSeek 模型以及各提供方特有的冒烟测试;这些测试各自由自己的密钥控制(`EXA_API_KEY`、`PERPLEXITY_API_KEY` 等),缺少密钥时套件会自动跳过,使 keyless CI 保持绿色([真实 API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md))。
|
||||
- **快照**(`pnpm run test:snapshot`):无密钥预期输出覆盖对外行为(传输契约与呈现),持久化日志则固定组装后的后端行为。ACP 启动真实的自动化服务器示例、回放录制会话,并对归一化 JSON-RPC 与重新持久化的日志执行 diff([ACP 快照 Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md));headless 通过真实单次运行进程固定 `stream-json`。TUI 旅程通过真实循环与工具回放主会话与子会话 JSONL,再将 ANSI 投影为语义化终端状态输出;包级快照保留瞬态状态,真实 PTY 覆盖进程边界([TUI 快照 Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md))。当模型 transcript(文本记录)发生变化时使用 `pnpm run test:snapshot:record`,回放输入仍然有效时使用 `pnpm run test:snapshot:refresh`;请审查每一处 JSONL 与预期输出差异。一个 ACP 场景(`text-turn`)固定完整的系统提示词与工具 schema 内容;其他 fixture(测试前置数据)将其 token 化,因此修改只会扰动一行([pinned-header Agent Note](../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md))。
|
||||
- **Web 浏览器快照**(`pnpm run test:web`;必需的 Linux PR(Pull Request)门禁):Chromium 将回放后的浏览器输出与 `apps/web/tests/snapshots/` 比较。CI 强制只读的 `DSH_SNAPSHOT=replay`,绝不写入预期输出;record/refresh 留在本地,每处 diff 都须评审([web e2e 车道](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md)、[CI 门禁决策](../.agents/notes/implemented/testing/2026-07-30-web-browser-snapshot-ci-gate.md))。`test:web` 会[先构建](../.agents/notes/implemented/bug-fix/2026-07-28-themed-scrollbars-and-reserved-gutter.md)以交付插件 CSS。
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
{"type":"assistant/chunk","seq":14,"time":1785487611319,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}}
|
||||
{"type":"assistant/message","seq":15,"time":1785487611319,"data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"call_wait","name":"bash","arguments":"{\"command\":\"node -e \\\"require('node:fs').writeFileSync('started.txt', 'started'); setInterval(() => {}, 1000)\\\"\",\"description\":\"Wait until cancellation\"}"},{"type":"tool-call","id":"call_skipped","name":"bash","arguments":"{\"command\":\"printf skipped > skipped.txt\",\"description\":\"Write skipped marker\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash"},"id":"bd630f41-b45d-4183-a785-1ff6e7049b62"},"usage":{"inputTokens":10,"outputTokens":10}},"sourceEventSeqs":[7,8,9,10,11,12,13,14],"surfaceOp":"append"}
|
||||
{"type":"tool/call","seq":16,"time":1785487611319,"data":{"turn":1,"step":1,"callId":"call_wait","name":"bash","arguments":"{\"command\":\"node -e \\\"require('node:fs').writeFileSync('started.txt', 'started'); setInterval(() => {}, 1000)\\\"\",\"description\":\"Wait until cancellation\"}"}}
|
||||
{"type":"tool/result","seq":17,"time":1785487611378,"data":{"turn":1,"step":1,"message":{"source":{"kind":"tool","callId":"call_wait"},"content":[{"type":"tool-result","toolCallId":"call_wait","content":[{"type":"text","text":"Error: tool call aborted"}],"isError":true}],"role":"user","id":"252903b2-b4e1-4a33-81d8-d5befefcb27e","error":{"name":"AbortError","code":"ABORTED"}},"sourceEventSeqs":[16],"surfaceOp":"append"}
|
||||
{"type":"tool/result","seq":17,"time":1785487611378,"data":{"turn":1,"step":1,"message":{"source":{"kind":"tool","callId":"call_wait"},"content":[{"type":"tool-result","toolCallId":"call_wait","content":[{"type":"text","text":"Error: tool call aborted"}],"isError":true}],"role":"user","id":"252903b2-b4e1-4a33-81d8-d5befefcb27e"},"error":{"name":"AbortError","code":"ABORTED"}},"sourceEventSeqs":[16],"surfaceOp":"append"}
|
||||
{"type":"tool/call","seq":18,"time":1785487611378,"data":{"turn":1,"step":1,"callId":"call_skipped","name":"bash","arguments":"{\"command\":\"printf skipped > skipped.txt\",\"description\":\"Write skipped marker\"}"}}
|
||||
{"type":"tool/result","seq":19,"time":1785487611378,"data":{"turn":1,"step":1,"message":{"source":{"kind":"tool","callId":"call_skipped"},"content":[{"type":"tool-result","toolCallId":"call_skipped","content":[{"type":"text","text":"Error: tool call aborted before dispatch"}],"isError":true}],"role":"user","id":"282c5c8c-7296-4545-8014-e6393b351436"},"error":{"name":"AbortError","code":"ABORTED_BEFORE_DISPATCH"}},"sourceEventSeqs":[18],"surfaceOp":"append"}
|
||||
{"type":"step/end","seq":20,"time":1785487611378,"data":{"turn":1,"step":1}}
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"docs/architecture.md": 2160,
|
||||
"docs/cordis-primer.md": 600,
|
||||
"docs/defensive-patterns.md": 550,
|
||||
"docs/testing.md": 1120,
|
||||
"docs/testing.md": 1150,
|
||||
"examples/AGENTS.md": 310,
|
||||
"packages/AGENTS.md": 675,
|
||||
"packages/README.md": 920
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import tsconfigPaths from 'vite-tsconfig-paths'
|
||||
import { defineConfig } from 'vitest/config'
|
||||
import { vitestExecArgv } from './vitest.shared.ts'
|
||||
@@ -39,6 +40,18 @@ const windowsCoverageExclusions = process.platform === 'win32'
|
||||
]
|
||||
: []
|
||||
|
||||
// Mirrors windowsCoverageExclusions: pwsh-local's run/start/lifecycle suites
|
||||
// self-skip without a real pwsh (executor.spec.ts hasPwsh), leaving this file
|
||||
// far below per-file 100% on pwsh-less hosts; the exemption keeps those hosts
|
||||
// green while CI runners ship pwsh and still enforce the full bar. The probe
|
||||
// is deliberately PATH-only (narrower than the suites' resolvePwshPath): a
|
||||
// win32 host where only install-location pwsh or 5.1 resolves forfeits the
|
||||
// exemption while the suites still run, so the gate can only get stricter,
|
||||
// never falsely green.
|
||||
const pwshCoverageExclusions = spawnSync('pwsh', ['-NoLogo', '-NoProfile', '-NonInteractive', '-Command', '$true'], { encoding: 'utf8' }).status === 0
|
||||
? []
|
||||
: ['packages/bash/pwsh-local/src/index.ts']
|
||||
|
||||
const testIncludes = [
|
||||
'packages/*/*/tests/**/*.spec.{ts,tsx}',
|
||||
'apps/*/tests/**/*.spec.ts',
|
||||
@@ -203,6 +216,7 @@ export default defineConfig({
|
||||
'packages/ui/tui/src/index.ts',
|
||||
...windowsUnsupportedPackages.map(path => `${path}/src/**/*.ts`),
|
||||
...windowsCoverageExclusions,
|
||||
...pwshCoverageExclusions,
|
||||
],
|
||||
// 100% or it doesn't merge (docs/testing.md: excessive tests are welcome).
|
||||
// Per-file so a well-covered big file can't subsidize a bare one.
|
||||
|
||||
Reference in New Issue
Block a user