mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-08-29 04:26:38 +00:00
test(python): gate installed runtime wheels across release targets (#2953)
* test(python): exercise installed wheels as black boxes Add an installed-wheel mode that refuses source/editable imports, repository working directories, mismatched SDK/runtime versions, unpinned runtime dependencies, and executables outside the installed runtime distribution. The mode resolves the wheel-owned executable itself, so callers cannot accidentally prove an explicit checkout artifact. Add a real-API scenario that drives two tool-using turns through the public synchronous SDK, verifies the file bytes outside the agent, checks completed turn/tool events and persistence, and projects provider failures without retaining credential-bearing error text. The existing deterministic scenario set remains the keyless behavior oracle. Refs #2952. * ci(python): require installed-wheel checks on every release target Move the complete deterministic runtime scenarios behind construction and clean installation of the SDK and matching runtime wheels. Each native leg runs outside the checkout with source-resolution environment variables removed; Linux manylinux smokes assert the same installed provenance. Expand the required pull-request call from Linux x64 to Linux x64, Linux arm64, and macOS arm64. Trusted heads receive only DEEPSEEK_API_KEY_EXTERNAL for a fail-loud live two-turn smoke on each carrier, while fork and Dependabot heads retain the full keyless path without exposing secrets. Pin the reusable secret declaration, matrix call, aggregate dependency, untrusted-head condition, and live/keyless commands in the workflow contract test. Refs #2952. * docs(testing): make installed wheels the Python CI authority Record the clean-wheel provenance boundary, complete keyless scenario set, trusted real-API contract, secret handling, and three-target required topology in a new implemented testing decision. Update the SEA distribution and portable-CI authorities plus the Python contributor reference to describe the same current state. Archive the fully superseded Linux-x64-only decision after consolidating its rationale and alternatives into the new owner. Preserve its bilingual triplet as a sealed historical snapshot and redirect every active current-state reference. Refs #2952.
This commit is contained in:
@@ -27,14 +27,16 @@ uv run --project python/sdk pytest
|
||||
|
||||
`python/sdk/tests/test_bundled_runtime.py` 会运行可用的内置载体;某个载体的产物尚未构建时,会跳过该载体。仓库级测试政策见 [测试](../docs/testing.zh.md)。
|
||||
|
||||
该套件面向的是伪造的运行时对端。`scripts/smoke-python-runtime.py` 面向真实的打包运行时;必需的 `python-runtime` CI 任务会用新构建的可执行文件运行全部场景:
|
||||
该套件面向的是伪造的运行时对端。`scripts/smoke-python-runtime.py` 面向打包运行时。必需的 `python-runtime` CI 任务会构建每个已发布原生目标,把匹配的 SDK wheel 包与运行时 wheel 包安装进新的 Python 3.10 虚拟环境,在 checkout 外清除 `PYTHONPATH` 与 `DSH_RUNTIME_MODE` 后运行,证明两个模块及可执行文件都来自这些 distribution,然后运行全部 keyless 场景。聚焦的本地源码 SDK 运行可以选择一个已构建可执行文件与场景:
|
||||
|
||||
```sh
|
||||
uv run --project python/sdk python scripts/smoke-python-runtime.py \
|
||||
--scenario sdk-minimal --exe dist-exe/dsh-jsonrpc-agent-pkg-macos-arm64
|
||||
```
|
||||
|
||||
其中两个场景会比对 `scripts/snapshots/python-sdk-single-exe/` 下已提交的期望输出。`minimal/model-visible.json` 固定了签入的极简组合所组装的系统提示词、对外公布的工具 schema 以及模型可见消息,因此插件一旦贡献出计划外的系统分段或 user 消息,该任务即失败;它会丢弃动态运行时上下文快照——同一组合在 macOS 上会发出它,在 Linux 上不会([#2488](https://github.com/deepseek-harness/deepseek-harness/issues/2488))。`advanced/` 固定 SDK 结果与持久化的会话日志。重新运行对应场景时加上 `--update-snapshots`,并在提交前审阅该差异。
|
||||
其中三个场景会比对 `scripts/snapshots/python-sdk-single-exe/` 下已提交的期望输出。`minimal/model-visible.json` 固定了签入的极简组合所组装的系统提示词、对外公布的工具 schema 以及模型可见消息,因此插件一旦贡献出计划外的系统分段或 user 消息,该任务即失败;它会丢弃动态运行时上下文快照——同一组合在 macOS 上会发出它,在 Linux 上不会([#2488](https://github.com/deepseek-harness/deepseek-harness/issues/2488))。`advanced/` 固定一个复杂进程的 SDK 结果及父/子会话日志。`restart/` 针对同一持久化根目录启动两个完整 SDK 运行时进程,并固定其彼此隔离的模型历史、高层结果与独立持久日志。重新运行对应场景时加上 `--update-snapshots`,并在提交前审阅该差异。
|
||||
|
||||
可信拉取请求还会在每个原生目标上运行 `--scenario sdk-live --installed-wheel`。该场景面向 `https://api.deepseek.com` 执行两个使用工具的轮次,从外部验证已创建文件,并在仓库密钥缺失时失败而不是自行 skip。Fork 与 Dependabot 拉取请求会运行完整的 keyless 安装后 wheel 路径,但不会获得密钥。
|
||||
|
||||
交互式冒烟测试需要环境变量或仓库根目录 `.env` 中存在 `DEEPSEEK_API_KEY`:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user