test(snapshot): centralize ACP session corpus

This commit is contained in:
Tianyi Cui
2026-08-24 21:36:31 +08:00
parent 30762b63c9
commit 5c67cf898c
376 changed files with 27 additions and 12 deletions
+1 -1
View File
@@ -123,7 +123,7 @@ Real-API tests and demos read `DEEPSEEK_API_KEY`, optional `DEEPSEEK_BASE_URL`,
- **Tests describe behavior, not correctness.** Change obsolete behavior with its tests; explain why in the PR.
- **Non-trivial changes MUST include an Agent Note in the same PR;** only mechanical/local edits are exempt ([scope](.agents/notes/README.md#when-to-write-one)). Archived notes are frozen: never edit or treat them as current authority ([archive policy](.agents/notes/README.md#archiving-and-deletion)).
- **Client UI copy is locale-owned.** Route product text through typed dictionaries and `t` or localized primitive props; `verify-client-ui-i18n` rejects hardcoded copy ([decision](.agents/notes/implemented/architecture/2026-08-23-locale-owned-client-ui-copy.md)).
- **Testing policy** — [docs/testing.md](docs/testing.md). Every non-trivial model- or product-user-visible change updates a keyless runnable-example snapshot; package, e2e-only, and mock-only tests do not substitute. Fixtures replay on macOS/Linux; fix fixtures, not normalizers.
- **Testing policy** — [docs/testing.md](docs/testing.md). Every non-trivial model- or product-user-visible change updates a keyless recorded-session snapshot; [snapshot ownership](snapshots/AGENTS.md) reserves the top-level tree for session-driven cases and keeps other expected output owner-local. Fixtures replay on macOS/Linux; fix fixtures, not normalizers.
- **Design each tool's UI presentation up front.** Host presenters stay pure; Web cards derive from raw events and persisted result metadata ([cookbook](docs/cookbook/adding-a-tool.md)).
- **Plan unit, e2e, and snapshot coverage** for capability seams, lifecycle paths, and transcript output; include missing snapshot-harness support in the same change.
- **Both SDKs project the loop.** Agent-loop, session-lifecycle, and `SessionEventMap` changes update the TypeScript and Python SDK expected outputs in the same PR; `pnpm run test` covers neither ([surfaces](docs/testing.md#when-a-snapshot-test-is-required)).
+1 -1
View File
@@ -13,7 +13,7 @@ import {
} from './scaffold.ts'
import { newEnglishPage, saveFailureShot } from './support.ts'
const FIXTURE = fileURLToPath(new URL('../../../examples/acp-agent/tests/snapshots/cancel-tool-calls/session.jsonl', import.meta.url))
const FIXTURE = fileURLToPath(new URL('../../../snapshots/acp/cancel-tool-calls/session.jsonl', import.meta.url))
const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/bash-abort-row', import.meta.url))
const UI_EXPECTED = join(SNAPSHOT_DIR, 'ui.expected.md')
const MODE = webSnapshotMode()
+1 -1
View File
@@ -12,7 +12,7 @@ import {
} from './scaffold.ts'
import { newEnglishPage, saveFailureShot } from './support.ts'
const FIXTURE = fileURLToPath(new URL('../../../examples/acp-agent/tests/snapshots/skill-load/session.jsonl', import.meta.url))
const FIXTURE = fileURLToPath(new URL('../../../snapshots/acp/skill-load/session.jsonl', import.meta.url))
const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/skill-tool-row', import.meta.url))
const UI_EXPECTED = fileURLToPath(new URL('./snapshots/skill-tool-row/ui.expected.md', import.meta.url))
const MODE = webSnapshotMode()
+2 -2
View File
@@ -22,8 +22,8 @@ const MODE = webSnapshotMode()
const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/workflow-run', import.meta.url))
const UI_LIVE_EXPECTED = join(SNAPSHOT_DIR, 'ui-live.expected.md')
const UI_EXPECTED = join(SNAPSHOT_DIR, 'ui.expected.md')
const PARENT_FIXTURE = join(REPO_ROOT, 'examples/acp-agent/tests/snapshots/workflow-run/session.jsonl')
const CHILD_FIXTURE = join(REPO_ROOT, 'examples/acp-agent/tests/snapshots/workflow-run/session.1.jsonl')
const PARENT_FIXTURE = join(REPO_ROOT, 'snapshots/acp/workflow-run/session.jsonl')
const CHILD_FIXTURE = join(REPO_ROOT, 'snapshots/acp/workflow-run/session.1.jsonl')
const CHILD_PROMPT = 'Reply with exactly the word WF_CHILD_OK and nothing else.'
describe.skipIf(MODE === 'record')('web e2e: durable workflow run in Chat', () => {
+2 -2
View File
@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/testing.md
testing.md: 80f973045a2a2a090da1ec56d5e9da99bb7ec50d
testing.zh.md: 897f0434989f4e461e9affc2475dc1ad32278872
testing.md: 9b0ed0a103b1ca9f149d289cabcc2b0ba9bc909f
testing.zh.md: 18e72deed19b7ecd981123e933f23cbbf7ba2c87
+1 -1
View File
@@ -46,4 +46,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword
## When a snapshot test is required
Every non-trivial model-, protocol-, or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP automation scenarios use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-session-snapshot`](../packages/test-support/session-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the internal canonical-event JSONL snapshots and replay fixtures. The `pwsh-tool-turn` ACP scenario boots real `pwsh` and skips where it is absent. Completed interactive-terminal journeys use JSONL-driven scenarios under `apps/cli/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/`. The two SDKs project the agent loop, session lifecycle, and `SessionEventMap` independently, so changing any of those updates both: `examples/python-sdk-agent/tests/snapshots/` owns the TypeScript client; `scripts/snapshots/python-sdk-single-exe/` owns the Python client, which only the required `python-runtime` CI job runs. New capability seams, lifecycle variants, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation.
Every non-trivial model-, protocol-, or human-visible change adds or updates a keyless scenario in the same PR through its owning recorded-session suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP automation scenarios live under `snapshots/acp/`, driven by the [`dsh-session-snapshot`](../packages/test-support/session-snapshot/README.md) ACP adapter; `examples/headless-agent` owns the internal canonical-event JSONL snapshots and replay fixtures. The `pwsh-tool-turn` ACP scenario boots real `pwsh` and skips where it is absent. Completed interactive-terminal journeys use JSONL-driven scenarios under `apps/cli/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/`. The two SDKs project the agent loop, session lifecycle, and `SessionEventMap` independently, so changing any of those updates both: `examples/python-sdk-agent/tests/snapshots/` owns the TypeScript client; `scripts/snapshots/python-sdk-single-exe/` owns the Python client, which only the required `python-runtime` CI job runs. New capability seams, lifecycle variants, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation.
+1 -1
View File
@@ -46,4 +46,4 @@ e2e 断言应重新运行命令或从外部重新读取文件;对 agent 自身
## 何时需要快照测试
每项非平凡的模型可见、协议可见或人类可见变更,都必须在同一 PR 中,通过可运行示例所属的快照套件添加或更新无密钥场景。包测试、e2e 断言、mock 与仅测试组合、PR 理由都不能取代组装后的 transcript;必要时应扩展 harness。ACP 自动化场景使用 `examples/<name>/tests/snapshots/`即基于 [`dsh-session-snapshot`](../packages/test-support/session-snapshot/README.zh.md) 套件工厂的场景表(`examples/acp-agent` 为主套件)`examples/headless-agent` 拥有内部规范事件 JSONL 快照与回放 fixture。`pwsh-tool-turn` ACP 场景启动真实 `pwsh`,在无 `pwsh` 的主机上跳过。已完成的交互式终端旅程使用 `apps/cli/tests/snapshots/` 下由 JSONL 驱动的场景;瞬态呈现使用包内语义矩阵,输入、Loader 选择或终端清理发生变化时还要添加 PTY 用例。浏览器渲染的 Web GUI 旅程使用上述 Web 应用快照套件。两个 SDK 各自独立地投影 agent loop、会话生命周期与 `SessionEventMap`,因此改动其中任何一项都要同时更新两者:`examples/python-sdk-agent/tests/snapshots/` 拥有 TypeScript 客户端;`scripts/snapshots/python-sdk-single-exe/` 拥有 Python 客户端,且只有必需的 `python-runtime` CI 作业会运行它。新的能力 seam、生命周期变体或 transcript 呈现接口在计划阶段就要列出每个覆盖层级,并在实现前验证 harness 能够表达它们。
每项非平凡的模型可见、协议可见或人类可见变更,都必须在同一 PR 中,通过所属录制会话套件添加或更新无密钥场景。包测试、e2e 断言、mock 与仅测试组合、PR 理由都不能取代组装后的 transcript;必要时应扩展 harness。ACP 自动化场景位于 `snapshots/acp/` [`dsh-session-snapshot`](../packages/test-support/session-snapshot/README.zh.md) ACP 适配器驱动`examples/headless-agent` 拥有内部规范事件 JSONL 快照与回放 fixture。`pwsh-tool-turn` ACP 场景启动真实 `pwsh`,在无 `pwsh` 的主机上跳过。已完成的交互式终端旅程使用 `apps/cli/tests/snapshots/` 下由 JSONL 驱动的场景;瞬态呈现使用包内语义矩阵,输入、Loader 选择或终端清理发生变化时还要添加 PTY 用例。浏览器渲染的 Web GUI 旅程使用上述 Web 应用快照套件。两个 SDK 各自独立地投影 agent loop、会话生命周期与 `SessionEventMap`,因此改动其中任何一项都要同时更新两者:`examples/python-sdk-agent/tests/snapshots/` 拥有 TypeScript 客户端;`scripts/snapshots/python-sdk-single-exe/` 拥有 Python 客户端,且只有必需的 `python-runtime` CI 作业会运行它。新的能力 seam、生命周期变体或 transcript 呈现接口在计划阶段就要列出每个覆盖层级,并在实现前验证 harness 能够表达它们。
+1 -1
View File
@@ -88,7 +88,7 @@ const PRODUCT_SUBAGENT_RESULT_DIAGNOSTIC_CONFIG = fileURLToPath(
new URL('../subagent-result-diagnostic.cordis.yml', import.meta.url),
)
const FS_DIFF_BOUND_CONFIG = fileURLToPath(new URL('./fs-diff-bound.cordis.yml', import.meta.url))
const SNAPSHOTS_DIR = join(dirname(fileURLToPath(import.meta.url)), 'snapshots')
const SNAPSHOTS_DIR = fileURLToPath(new URL('../../../snapshots/acp/', import.meta.url))
const PACKED_CHUNKS_SOURCE = 'hook-cc-pretool-deny'
async function prepareEditingCordisSkillWorkspace(cwd: string): Promise<void> {
+2
View File
@@ -23,9 +23,11 @@ const PATTERNS = [
'packages/*/*.md',
'packages/*/*/*.md',
'examples/**/system-prompt.expected.md',
'snapshots/**/system-prompt.expected.md',
'packages/**/system-prompt.expected.md',
'AGENTS.md',
'packages/AGENTS.md',
'snapshots/AGENTS.md',
]
/** A located hard-wrap: a prose paragraph spanning more than one source line. */
+13
View File
@@ -0,0 +1,13 @@
# AGENTS.md — Recorded-session snapshots
This tree contains only tests whose committed session JSONL is replay input and expected persisted output. Keep non-session ARIA, geometry, generator, CLI, and unit expected output with its owning app, script, or package as a golden.
Every process under test starts through the `dsh` CLI with a shipped profile and optional scenario patches. Test clients may drive a public protocol or browser interface; do not add another application entrypoint, hidden CLI mode, or executable scenario driver.
Each scenario owns or explicitly references one primary `session.jsonl` plus contiguous child files. The owner alone records or refreshes it. Shared references are read-only, acyclic, and used only when another interface intentionally renders the same recorded behavior.
Committed sessions are normalization fixed points. Replace volatile identities with typed relationship-preserving tokens, replace request system prompts and tool schemas with tokens, and keep exactly one readable sidecar owner per header class. Never redact arbitrary user or tool text merely because it resembles an identifier.
Workspace seeds stay scenario-local. A scenario that mutates the workspace commits and compares its expected final workspace; model prose and tool-result text do not prove the external effect.
`pnpm run test:snapshot` replays without writes. Recording and refresh use the explicit snapshot scripts, and every resulting JSONL, prompt, schema, protocol, UI, and workspace diff is reviewed before commit.
@@ -23,6 +23,8 @@ Use the web_search tool to discover current information on the web. The required
Use goal tools for one long-running completion objective in the current session. create_goal may infer goal intent from a direct human request in any language; do not create a goal for routine single-turn work. Call get_goal before update_goal and copy its exact goal_id and revision. After session resume or fork, an active goal is disarmed: when a human asks to continue or resume in any wording or language, use update_goal action resume to rearm it. Mark complete only when the objective is actually achieved. Mark blocked only after the same blocking condition persists for at least 3 consecutive goal rounds, and report that concrete condition in blocked_reason; difficulty, uncertainty, or useful remaining work is not blocked.
Use the workflow tool ONLY when the user explicitly asks for a workflow or for large multi-agent orchestration: you write a JavaScript script (the tool description documents the exact format) that fans work out across many subagents with phases and structured results. For one or two delegations, prefer plain subagent calls.
# Dynamic Cordis Plugins
Dynamic Cordis plugins temporarily extend the current DSH process. A Plugin uses apply(ctx) to consume Services, listen to Events, provide Services, register model Tools, or register browser UI in Slots.
@@ -129,8 +131,6 @@ return {
- After a technical failure, use cordis_inspect_self to read the exact Package source and its message/stack. Define a corrected Package under the same Plugin and retry autonomously.
- Use the cordis-plugin-development Skill for other failure causes, repair procedures, and complete extension patterns.
Use the workflow tool ONLY when the user explicitly asks for a workflow or for large multi-agent orchestration: you write a JavaScript script (the tool description documents the exact format) that fans work out across many subagents with phases and structured results. For one or two delegations, prefer plain subagent calls.
Use the ralph tool ONLY when the direct human explicitly asks for a Ralph loop or fresh-agent iterative execution. Each Ralph round starts a fresh child with no conversation seed and uses the shared workspace as durable memory. Completion and blockers are worker reports, not independent evaluation. Use same-session goal tools for ordinary long-running objectives, and plain subagents or workflows for bounded delegation and fan-out.
Use subagent in the background by default. Start independent delegations together in one assistant message and continue useful work while they run. Set `run_in_background: false` only when your next action depends on that subagent's result. When a background run settles, the runtime sends you a notice containing its outcome and any final assistant message.

Some files were not shown because too many files have changed in this diff Show More