test(python): gate installed runtime wheels across release targets (#2953)

* test(python): exercise installed wheels as black boxes

Add an installed-wheel mode that refuses source/editable imports, repository working directories, mismatched SDK/runtime versions, unpinned runtime dependencies, and executables outside the installed runtime distribution. The mode resolves the wheel-owned executable itself, so callers cannot accidentally prove an explicit checkout artifact.

Add a real-API scenario that drives two tool-using turns through the public synchronous SDK, verifies the file bytes outside the agent, checks completed turn/tool events and persistence, and projects provider failures without retaining credential-bearing error text. The existing deterministic scenario set remains the keyless behavior oracle.

Refs #2952.

* ci(python): require installed-wheel checks on every release target

Move the complete deterministic runtime scenarios behind construction and clean installation of the SDK and matching runtime wheels. Each native leg runs outside the checkout with source-resolution environment variables removed; Linux manylinux smokes assert the same installed provenance.

Expand the required pull-request call from Linux x64 to Linux x64, Linux arm64, and macOS arm64. Trusted heads receive only DEEPSEEK_API_KEY_EXTERNAL for a fail-loud live two-turn smoke on each carrier, while fork and Dependabot heads retain the full keyless path without exposing secrets.

Pin the reusable secret declaration, matrix call, aggregate dependency, untrusted-head condition, and live/keyless commands in the workflow contract test.

Refs #2952.

* docs(testing): make installed wheels the Python CI authority

Record the clean-wheel provenance boundary, complete keyless scenario set, trusted real-API contract, secret handling, and three-target required topology in a new implemented testing decision. Update the SEA distribution and portable-CI authorities plus the Python contributor reference to describe the same current state.

Archive the fully superseded Linux-x64-only decision after consolidating its rationale and alternatives into the new owner. Preserve its bilingual triplet as a sealed historical snapshot and redirect every active current-state reference.

Refs #2952.
This commit is contained in:
Tianyi Cui
2026-08-23 16:53:55 +08:00
committed by GitHub
parent 9f9f160854
commit 3c1c6a89b1
24 changed files with 727 additions and 47 deletions
+56 -14
View File
@@ -21,10 +21,14 @@ on:
required: false
default: false
ci:
description: Run as the required Linux x64 Python runtime pull-request check.
description: Run as the required all-target Python runtime pull-request check.
type: boolean
required: false
default: false
secrets:
DEEPSEEK_API_KEY_EXTERNAL:
description: Real DeepSeek API key for trusted installed-wheel pull-request tests.
required: false
workflow_dispatch:
inputs:
targets:
@@ -243,13 +247,6 @@ jobs:
echo "exe=$exe" >> "$GITHUB_OUTPUT"
echo "wheel=$wheel" >> "$GITHUB_OUTPUT"
- name: Full-turn SDK, executable snapshot, and direct-binary smoke
run: >-
uv run --python 3.10 --group test --project python/sdk
python scripts/smoke-python-runtime.py
--scenario all
--exe "${{ steps.runtime.outputs.exe }}"
- name: Build release-shaped runtime wheel
run: >-
python scripts/build-python-release.py
@@ -273,10 +270,53 @@ jobs:
"$RUNNER_TEMP/dsh-sdk-smoke/bin/python" -m pip install \
"dist-python/$SDK_WHEEL" \
"dist-python/$RUNTIME_WHEEL"
"$RUNNER_TEMP/dsh-sdk-smoke/bin/python" scripts/smoke-python-runtime.py \
--scenario sdk-default
"$RUNNER_TEMP/dsh-sdk-smoke/bin/python" scripts/smoke-python-runtime.py \
--scenario sdk-mcp
- name: Run installed-wheel keyless black-box tests
run: |
set -euo pipefail
blackbox_root="$RUNNER_TEMP/dsh-sdk-blackbox"
mkdir -p "$blackbox_root"
cd "$blackbox_root"
env -u PYTHONPATH -u DSH_RUNTIME_MODE \
"$RUNNER_TEMP/dsh-sdk-smoke/bin/python" \
"$GITHUB_WORKSPACE/scripts/smoke-python-runtime.py" \
--scenario all \
--installed-wheel
- name: Preflight installed-wheel real API test
if: >-
inputs.ci
&& (github.event_name != 'pull_request'
|| !(github.event.pull_request.head.repo.fork
|| github.event.pull_request.user.login == 'dependabot[bot]'))
env:
DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY_EXTERNAL }}
run: |
set -euo pipefail
if [ -z "${DEEPSEEK_API_KEY:-}" ]; then
echo "::error::DEEPSEEK_API_KEY_EXTERNAL is empty; the installed-wheel real API test cannot self-skip."
exit 1
fi
- name: Run installed-wheel real API black-box test
if: >-
inputs.ci
&& (github.event_name != 'pull_request'
|| !(github.event.pull_request.head.repo.fork
|| github.event.pull_request.user.login == 'dependabot[bot]'))
env:
DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY_EXTERNAL }}
DEEPSEEK_BASE_URL: https://api.deepseek.com
run: |
set -euo pipefail
blackbox_root="$RUNNER_TEMP/dsh-sdk-blackbox-live"
mkdir -p "$blackbox_root"
cd "$blackbox_root"
env -u PYTHONPATH -u DSH_RUNTIME_MODE \
"$RUNNER_TEMP/dsh-sdk-smoke/bin/python" \
"$GITHUB_WORKSPACE/scripts/smoke-python-runtime.py" \
--scenario sdk-live \
--installed-wheel
- name: Check Linux GLIBC requirements
if: runner.os == 'Linux'
@@ -314,8 +354,10 @@ jobs:
docker run --rm -e RUNTIME_WHEEL -e SDK_WHEEL -e DSH_TELEMETRY_DISABLED -v "$PWD:/work" -w /work "$image" bash -euxo pipefail -c '
/opt/python/cp310-cp310/bin/python -m venv /tmp/dsh-sdk
/tmp/dsh-sdk/bin/python -m pip install "/work/dist-python/$SDK_WHEEL" "/work/dist-python/$RUNTIME_WHEEL"
/tmp/dsh-sdk/bin/python /work/scripts/smoke-python-runtime.py --scenario sdk-default
/tmp/dsh-sdk/bin/python /work/scripts/smoke-python-runtime.py --scenario sdk-mcp
mkdir -p /tmp/dsh-sdk-manylinux-smoke
cd /tmp/dsh-sdk-manylinux-smoke
env -u PYTHONPATH -u DSH_RUNTIME_MODE /tmp/dsh-sdk/bin/python /work/scripts/smoke-python-runtime.py --scenario sdk-default --installed-wheel
env -u PYTHONPATH -u DSH_RUNTIME_MODE /tmp/dsh-sdk/bin/python /work/scripts/smoke-python-runtime.py --scenario sdk-mcp --installed-wheel
'
- uses: actions/upload-artifact@v7
+7 -6
View File
@@ -294,17 +294,18 @@ jobs:
- name: Run complete keyless Python suite
run: uv run --python 3.10 --group test --project python/sdk pytest
# One native target makes the complete release-shaped Python path required
# without duplicating platform-independent behavior across the release matrix.
# The reusable builder owns the executable, snapshot, wheel, clean-install,
# GLIBC, and manylinux checks; release validation retains all native targets.
# The reusable builder owns each published executable, wheel, clean-install,
# keyless black-box, and trusted real-API path. All native release targets are
# required because a platform wheel cannot be validated by another carrier.
python-runtime:
if: github.event_name == 'pull_request'
name: python runtime / release-shaped Linux x64
name: python runtime / release-shaped matrix
uses: ./.github/workflows/build-exe-for-python-sdk.yml
with:
targets: node24-linux-x64
targets: node24-linux-x64,node24-linux-arm64,node24-macos-arm64
ci: true
secrets:
DEEPSEEK_API_KEY_EXTERNAL: ${{ secrets.DEEPSEEK_API_KEY_EXTERNAL }}
# The pull-request Windows signals cover complementary hosts. The two fast
# win32 toolchain surfaces (workspace build, production site) execute with