mengzhx/deepseek-harness
1
0
Fork 0
mirror of https://github.com/deepseek-ai/deepseek-harness.git synced 2026-08-29 04:26:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(python): gate installed runtime wheels across release targets (#2953)

Browse Source 
* test(python): exercise installed wheels as black boxes

Add an installed-wheel mode that refuses source/editable imports, repository working directories, mismatched SDK/runtime versions, unpinned runtime dependencies, and executables outside the installed runtime distribution. The mode resolves the wheel-owned executable itself, so callers cannot accidentally prove an explicit checkout artifact.

Add a real-API scenario that drives two tool-using turns through the public synchronous SDK, verifies the file bytes outside the agent, checks completed turn/tool events and persistence, and projects provider failures without retaining credential-bearing error text. The existing deterministic scenario set remains the keyless behavior oracle.

Refs #2952.

* ci(python): require installed-wheel checks on every release target

Move the complete deterministic runtime scenarios behind construction and clean installation of the SDK and matching runtime wheels. Each native leg runs outside the checkout with source-resolution environment variables removed; Linux manylinux smokes assert the same installed provenance.

Expand the required pull-request call from Linux x64 to Linux x64, Linux arm64, and macOS arm64. Trusted heads receive only DEEPSEEK_API_KEY_EXTERNAL for a fail-loud live two-turn smoke on each carrier, while fork and Dependabot heads retain the full keyless path without exposing secrets.

Pin the reusable secret declaration, matrix call, aggregate dependency, untrusted-head condition, and live/keyless commands in the workflow contract test.

Refs #2952.

* docs(testing): make installed wheels the Python CI authority

Record the clean-wheel provenance boundary, complete keyless scenario set, trusted real-API contract, secret handling, and three-target required topology in a new implemented testing decision. Update the SEA distribution and portable-CI authorities plus the Python contributor reference to describe the same current state.

Archive the fully superseded Linux-x64-only decision after consolidating its rationale and alternatives into the new owner. Preserve its bilingual triplet as a sealed historical snapshot and redirect every active current-state reference.

Refs #2952.
This commit is contained in:
Tianyi Cui
2026-08-23 16:53:55 +08:00
committed by GitHub
parent 9f9f160854
commit 3c1c6a89b1
24 changed files with 727 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/development.md
+4 -2
View File
@@ -27,14 +27,16 @@ uv run --project python/sdk pytest
`python/sdk/tests/test_bundled_runtime.py` exercises available bundled carriers and skips a carrier when its artifact has not been built. For repository-wide test policy, see [Testing](../docs/testing.md).
That suite drives fake runtime peers. `scripts/smoke-python-runtime.py` drives the real packaged runtime instead, and the required `python-runtime` CI job runs every scenario against a freshly built executable:
That suite drives fake runtime peers. `scripts/smoke-python-runtime.py` drives the packaged runtime instead. The required `python-runtime` CI job builds every published native target, installs the matching SDK and runtime wheels into a new Python 3.10 virtual environment, runs outside the checkout with `PYTHONPATH` and `DSH_RUNTIME_MODE` unset, proves that both modules and the executable came from those distributions, and then runs every keyless scenario. A focused local source-SDK run can select one built executable and scenario:
```sh
uv run --project python/sdk python scripts/smoke-python-runtime.py \
--scenario sdk-minimal --exe dist-exe/dsh-jsonrpc-agent-pkg-macos-arm64
```
Two scenarios compare committed expected output under `scripts/snapshots/python-sdk-single-exe/`. `minimal/model-visible.json` pins the checked-in minimal composition's assembled system prompts, advertised tool schemas, and model-visible messages, so a plugin that contributes an unintended system section or user message fails the job; it drops the dynamic runtime-context snapshot, which the same composition emits on macOS and not on Linux ([#2488](https://github.com/deepseek-harness/deepseek-harness/issues/2488)). `advanced/` pins the SDK result and the persisted session logs. Rerun the owning scenario with `--update-snapshots` and review that diff before committing it.
Three scenarios compare committed expected output under `scripts/snapshots/python-sdk-single-exe/`. `minimal/model-visible.json` pins the checked-in minimal composition's assembled system prompts, advertised tool schemas, and model-visible messages, so a plugin that contributes an unintended system section or user message fails the job; it drops the dynamic runtime-context snapshot, which the same composition emits on macOS and not on Linux ([#2488](https://github.com/deepseek-harness/deepseek-harness/issues/2488)). `advanced/` pins one complex process's SDK result and parent/child session logs. `restart/` launches two complete SDK runtime processes against one persistence root and snapshots their isolated model histories, high-level results, and separate durable logs. Rerun the owning scenario with `--update-snapshots` and review that diff before committing it.
Trusted pull requests also run `--scenario sdk-live --installed-wheel` on every native target. That scenario performs two tool-using turns against `https://api.deepseek.com`, verifies the created file externally, and fails when the repository secret is absent instead of self-skipping. Fork and Dependabot pull requests run the complete keyless installed-wheel path but receive no key.
An interactive smoke test needs `DEEPSEEK_API_KEY` in the environment or repository-root `.env`: