mengzhx/deepseek-harness
1
0
Fork 0
mirror of https://github.com/deepseek-ai/deepseek-harness.git synced 2026-08-29 04:26:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(python): pin the minimal composition's model-visible surface

Browse Source 
The Python lane never compared what the minimal composition shows the
model: the mock model only asserted system-role messages, and the
advanced snapshot tokenizes the assembled system prompt and tool
schemas.

The sdk-minimal scenario now records model-visible.json — every model
request's advertised tool schemas verbatim and its message list, with
system and user text kept and assistant/tool payloads reduced to call
identity so the expected output replays on macOS and Linux. It excludes
the dynamic runtime-context snapshot, which the same composition emits
on macOS and not on Linux (#2488).

AGENTS.md and the testing policy name both SDKs as independent
projections of the agent loop, session lifecycle, and SessionEventMap.
This commit is contained in:
Yichen Jiang
2026-08-17 16:49:14 +08:00
parent 3083461524
commit 8692a1b76b
16 changed files with 652 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing.md
+1 -1
View File
@@ -46,4 +46,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword
## When a snapshot test is required
Every non-trivial model-, protocol-, or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP automation scenarios use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/test-support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the internal canonical-event JSONL snapshots and replay fixtures. The `pwsh-tool-turn` ACP scenario boots real `pwsh` and skips where it is absent. Completed interactive-terminal journeys use JSONL-driven scenarios under `apps/cli/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/`. New capability seams, lifecycle variants, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation.
Every non-trivial model-, protocol-, or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP automation scenarios use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/test-support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the internal canonical-event JSONL snapshots and replay fixtures. The `pwsh-tool-turn` ACP scenario boots real `pwsh` and skips where it is absent. Completed interactive-terminal journeys use JSONL-driven scenarios under `apps/cli/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/`. The two SDKs project the agent loop, session lifecycle, and `SessionEventMap` independently, so changing any of those updates both: `examples/jsonrpc-agent/tests/snapshots/` owns the TypeScript client; `scripts/snapshots/python-sdk-single-exe/` owns the Python client, which only the required `python-runtime` CI job runs. New capability seams, lifecycle variants, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation.