diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml index b3206b074e..b98eafb852 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md -2026-07-26-ci-failover-runbook.md: 659a3aa2a506e07e2788422a85a21b36cb79cfa9 -2026-07-26-ci-failover-runbook.zh.md: fc81e30918a4580f64e4b272e84c5b8926f0a53d +2026-07-26-ci-failover-runbook.md: f5711fd7e9c32f7ca555e06bb59b67e2129745d6 +2026-07-26-ci-failover-runbook.zh.md: 05803ff3ad8cee57c2b78a25f2940157e243cb64 diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md index 659a3aa2a5..f5711fd7e9 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md @@ -6,11 +6,11 @@ English | [中文](2026-07-26-ci-failover-runbook.zh.md) ## Problem -The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) run on the hosted enterprise 32-core pools; the required verdict job that aggregates them (`all checks passed`) runs on standard `ubuntu-latest`; the [native Windows jobs](2026-08-08-native-windows-pull-request-ci.md) run on the hosted `dsh-windows-2025-16core` larger runner. When the enterprise pools degrade — jobs queue indefinitely or the enterprise labels vanish — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. **Scope: two independent switches, one per platform.** `DSH_CI_FAILOVER_LINUX` recovers an enterprise Linux-pool outage (the three required Linux workers plus the `all checks passed` verdict); `DSH_CI_FAILOVER_WINDOWS` recovers a hosted Windows-pool outage (the native Windows jobs and eligible Windows x64 Python runtime CI builds; see [the Python runtime note](../../proposed/process/2026-09-06-python-runtime-windows-selfhosted.md)). A Linux-pool outage need not retarget Windows jobs and vice versa. The verdict's other required dependencies (`node-compat`, `python-sdk`, `windows`) stay on standard hosted runners by design (the portable boundary); in a broader GitHub-hosted capacity failure that also takes out the standard pools, those dependencies still block `all checks passed`. An outage therefore needs a switch any responder with repository write access can throw without merging anything. +The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) run on the hosted enterprise 32-core pools; the required verdict job that aggregates them (`all checks passed`) runs on standard `ubuntu-latest`; the [native Windows jobs](2026-08-08-native-windows-pull-request-ci.md) run on the hosted `dsh-windows-2025-16core` larger runner. When the enterprise pools degrade — jobs queue indefinitely or the enterprise labels vanish — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. **Scope: two independent switches, one per platform.** `DSH_CI_FAILOVER_LINUX` recovers an enterprise Linux-pool outage (the three required Linux workers plus the `all checks passed` verdict); `DSH_CI_FAILOVER_WINDOWS` recovers a hosted Windows-pool outage (the native Windows jobs). A Linux-pool outage need not retarget Windows jobs and vice versa. The verdict's other required dependencies (`node-compat`, `python-sdk`, `windows`) stay on standard hosted runners by design (the portable boundary); in a broader GitHub-hosted capacity failure that also takes out the standard pools, those dependencies still block `all checks passed`. An outage therefore needs a switch any responder with repository write access can throw without merging anything. ## Decision -Each of the three required Linux worker jobs, the native Windows jobs, and the `all checks passed` verdict job — which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through a repository variable, and the switch is split by platform so an outage on one platform does not retarget the other. The three Linux workers and the `all checks passed` verdict (whose `needs` are the required Linux workers and which runs on the `vm-backup` pool) resolve through `DSH_CI_FAILOVER_LINUX`; the native Windows jobs and eligible Windows x64 Python runtime CI builds resolve through `DSH_CI_FAILOVER_WINDOWS`. Unset, they default to their hosted pools; selecting `selfhosted` is an explicit operator choice. Set to `selfhosted` by any repository writer, the corresponding jobs retarget onto the in-house self-hosted pool: under `DSH_CI_FAILOVER_LINUX`, the Linux jobs and verdict move onto the `vm-backup` pool, snapshot concurrency drops to the shared-VM bound, and the hosted-path pnpm cache restores are skipped; under `DSH_CI_FAILOVER_WINDOWS`, the native Windows jobs and eligible Windows x64 Python runtime CI builds move onto the `dsh-win-ci` pool. Each switch is writer-manageable repository state, not a merge, so it works while every check is red. The in-house pools' readiness is continuously re-proven by the `serial / linux (self-hosted standby)` and `serial / windows (self-hosted standby)` lanes, which run the complete unsharded aggregates on every master push. +Each of the three required Linux worker jobs, the native Windows jobs, and the `all checks passed` verdict job — which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through a repository variable, and the switch is split by platform so an outage on one platform does not retarget the other. The three Linux workers and the `all checks passed` verdict (whose `needs` are the required Linux workers and which runs on the `vm-backup` pool) resolve through `DSH_CI_FAILOVER_LINUX`; the native Windows jobs resolve through `DSH_CI_FAILOVER_WINDOWS`. Unset, they default to their hosted pools; selecting `selfhosted` is an explicit operator choice. Set to `selfhosted` by any repository writer, the corresponding jobs retarget onto the in-house self-hosted pool: under `DSH_CI_FAILOVER_LINUX`, the Linux jobs and verdict move onto the `vm-backup` pool, snapshot concurrency drops to the shared-VM bound, and the hosted-path pnpm cache restores are skipped; under `DSH_CI_FAILOVER_WINDOWS`, the native Windows jobs move onto the `dsh-win-ci` pool. Each switch is writer-manageable repository state, not a merge, so it works while every check is red. The in-house pools' readiness is continuously re-proven by the `serial / linux (self-hosted standby)` and `serial / windows (self-hosted standby)` lanes, which run the complete unsharded aggregates on every master push. `ci-master.yml` exempts exactly one event from `cancel-in-progress` (`${{ github.event_name != 'push' }}`), so one master push does not cancel the drill still running from the previous one. Each drill runs its complete unsharded aggregate with one gate worker, which takes longer than the interval between master merges; under unconditional cancellation a drill is superseded before reaching a verdict and the lane yields no readiness evidence for a responder to check. @@ -28,7 +28,7 @@ The decision belongs at workflow level because cancellation applies to the whole #### Windows pool -`dsh-win-ci`: 32 always-on runner instances (scheduled tasks `GH-Runner-01`…`GH-Runner-32`) on the in-house Windows CI server (one 96-core / 580 GB machine). Labels: `[self-hosted, dsh-win-ci, windows]`. The image must preinstall Node 24, pnpm, Git (with Git Bash on `PATH`, i.e. `C:\Program Files\Git\bin` — the `bash` tool spawns `bash` by name), PowerShell 7, and enable Developer Mode for symlink support. Python runtime CI additionally requires preinstalled Python on `PATH` with `venv` and `ensurepip`; [the Python runtime note](../../proposed/process/2026-09-06-python-runtime-windows-selfhosted.md) owns its private setup and validation. The general-purpose Windows workspaces and pnpm store must both live on a ReFS volume (`F:`): those installs pass `--package-import-method=clone` on ReFS, which needs that volume layout and the `@reflink/reflink` native module that the system corepack pnpm carries (see [the Windows ReFS store note](../../archived/process/2026-08-30-windows-refs-store-block-clone-install.md)); a rebuilt runner without this layout fails the Windows build gates with TS6231. Check the latest `serial / windows (self-hosted standby)` run before switching: a green standby verifies the pool can execute `check:ci:windows-complete` end-to-end. +`dsh-win-ci`: 32 always-on runner instances (scheduled tasks `GH-Runner-01`…`GH-Runner-32`) on the in-house Windows CI server (one 96-core / 580 GB machine). Labels: `[self-hosted, dsh-win-ci, windows]`. The image must preinstall Node 24, pnpm, Git (with Git Bash on `PATH`, i.e. `C:\Program Files\Git\bin` — the `bash` tool spawns `bash` by name), PowerShell 7, and enable Developer Mode for symlink support. The general-purpose Windows workspaces and pnpm store must both live on a ReFS volume (`F:`): those installs pass `--package-import-method=clone` on ReFS, which needs that volume layout and the `@reflink/reflink` native module that the system corepack pnpm carries (see [the Windows ReFS store note](../../archived/process/2026-08-30-windows-refs-store-block-clone-install.md)); a rebuilt runner without this layout fails the Windows build gates with TS6231. Check the latest `serial / windows (self-hosted standby)` run before switching: a green standby verifies the pool can execute `check:ci:windows-complete` end-to-end. ### Switch (any repository writer, ~1 minute, no merge) @@ -36,7 +36,7 @@ The two switches are independent: flip only the one whose platform is degraded. 1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER_LINUX` (Linux pool outage) or `DSH_CI_FAILOVER_WINDOWS` (Windows pool outage), value `selfhosted`. 2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued. -3. That is the entire switch. Under Linux failover the workflow also drops `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 for the shared VM and skips the hosted-path pnpm cache restores because the VM's persistent store serves warm installs. Coverage uses the same four single-worker instrumented partitions and two exempt workers on both Linux pools. The Windows switch retargets the native Windows jobs and eligible Windows x64 Python runtime CI builds; the latter use job-private tooling and caches and skip hosted cache restore/save steps as described in [the Python runtime note](../../proposed/process/2026-09-06-python-runtime-windows-selfhosted.md). +3. That is the entire switch. Under Linux failover the workflow also drops `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 for the shared VM and skips the hosted-path pnpm cache restores because the VM's persistent store serves warm installs. Coverage uses the same four single-worker instrumented partitions and two exempt workers on both Linux pools. The Windows switch has no concurrency or cache branches; it only retargets the native Windows jobs' pool. #**Dependabot exception.** Both switches' selectors deliberately exclude `dependabot[bot]`: under failover, Dependabot PRs stay queued for the hosted pool rather than executing dependency-supplied code on the persistent VMs. A Dependabot PR that remains queued during an outage is expected behavior, not a failed switch; it completes when the hosted pool recovers. @@ -49,7 +49,7 @@ Capacity includes the master standby, main-CI jobs, and three release-rehearsal ### Switch back -Delete the `DSH_CI_FAILOVER_LINUX` or `DSH_CI_FAILOVER_WINDOWS` variable (or set it to anything other than `selfhosted`). New runs resolve back to their hosted pools, including eligible Windows x64 Python runtime CI builds when the Windows switch is cleared. Remove any extra instances that were registered during the incident. +Delete the `DSH_CI_FAILOVER_LINUX` or `DSH_CI_FAILOVER_WINDOWS` variable (or set it to anything other than `selfhosted`). New runs resolve back to their hosted pools. Remove any extra instances that were registered during the incident. ### Trust boundary diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md index fc81e30918..05803ff3ad 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md @@ -6,11 +6,11 @@ Status: implemented ## 问题 -[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 工作作业(`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`)运行在托管的企业级 32 核池上;聚合它们的必需判定作业(`all checks passed`)运行在标准 `ubuntu-latest` 上;[原生 Windows 作业](2026-08-08-native-windows-pull-request-ci.zh.md)运行在托管的 `dsh-windows-2025-16core` 大型运行器上。当企业池发生故障——作业无限排队或企业标签消失——所有开启的拉取请求都无法合并,而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。**适用范围:两个独立开关,每个平台一个。**`DSH_CI_FAILOVER_LINUX` 恢复企业级 Linux 池故障(三个必需的 Linux 工作作业加 `all checks passed` 判定作业);`DSH_CI_FAILOVER_WINDOWS` 恢复托管 Windows 池故障(原生 Windows 作业及符合条件的 Windows x64 Python runtime CI 构建;见 [Python runtime 说明](../../proposed/process/2026-09-06-python-runtime-windows-selfhosted.zh.md))。Linux 池故障无需重定向 Windows 作业,反之亦然。判定作业的其余必需依赖(`node-compat`、`python-sdk`、`windows`)按设计留在标准托管运行器上(可移植边界);若更大范围的 GitHub 托管容量故障连标准池一并击倒,这些依赖仍会阻塞 `all checks passed`。因此故障需要一个任何具备仓库写权限的响应者都能在不合并任何代码的情况下触发的开关。 +[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 工作作业(`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`)运行在托管的企业级 32 核池上;聚合它们的必需判定作业(`all checks passed`)运行在标准 `ubuntu-latest` 上;[原生 Windows 作业](2026-08-08-native-windows-pull-request-ci.zh.md)运行在托管的 `dsh-windows-2025-16core` 大型运行器上。当企业池发生故障——作业无限排队或企业标签消失——所有开启的拉取请求都无法合并,而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。**适用范围:两个独立开关,每个平台一个。**`DSH_CI_FAILOVER_LINUX` 恢复企业级 Linux 池故障(三个必需的 Linux 工作作业加 `all checks passed` 判定作业);`DSH_CI_FAILOVER_WINDOWS` 恢复托管 Windows 池故障(原生 Windows 作业)。Linux 池故障无需重定向 Windows 作业,反之亦然。判定作业的其余必需依赖(`node-compat`、`python-sdk`、`windows`)按设计留在标准托管运行器上(可移植边界);若更大范围的 GitHub 托管容量故障连标准池一并击倒,这些依赖仍会阻塞 `all checks passed`。因此故障需要一个任何具备仓库写权限的响应者都能在不合并任何代码的情况下触发的开关。 ## 决策 -三个必需的 Linux 工作作业、原生 Windows 作业,以及 `all checks passed` 判定作业(若不随切换,即使全部工作作业通过,它仍会滞留在故障池的队列中)——各自通过仓库变量解析运行器池,且开关按平台拆分,使一个平台的故障不会重定向另一个平台。三个 Linux 工作作业与 `all checks passed` 判定作业(其 `needs` 是必需的 Linux 工作作业,且运行在 `vm-backup` 池上)通过 `DSH_CI_FAILOVER_LINUX` 解析;原生 Windows 作业及符合条件的 Windows x64 Python runtime CI 构建通过 `DSH_CI_FAILOVER_WINDOWS` 解析。未设置变量时默认使用各自的托管池;选择 `selfhosted` 是运维人员的明确操作;由任何具备写权限的协作者设为 `selfhosted` 时,对应作业切换到公司自有的自托管池:`DSH_CI_FAILOVER_LINUX` 下,Linux 作业与判定作业切到 `vm-backup` 池,快照并发降到共享虚拟机上限,并跳过托管路径的 pnpm 缓存恢复;`DSH_CI_FAILOVER_WINDOWS` 下,原生 Windows 作业及符合条件的 Windows x64 Python runtime CI 构建切到 `dsh-win-ci` 池。每个开关都是写者可管理的仓库状态而非一次合并,因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 与 `serial / windows (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。 +三个必需的 Linux 工作作业、原生 Windows 作业,以及 `all checks passed` 判定作业(若不随切换,即使全部工作作业通过,它仍会滞留在故障池的队列中)——各自通过仓库变量解析运行器池,且开关按平台拆分,使一个平台的故障不会重定向另一个平台。三个 Linux 工作作业与 `all checks passed` 判定作业(其 `needs` 是必需的 Linux 工作作业,且运行在 `vm-backup` 池上)通过 `DSH_CI_FAILOVER_LINUX` 解析;原生 Windows 作业通过 `DSH_CI_FAILOVER_WINDOWS` 解析。未设置变量时默认使用各自的托管池;选择 `selfhosted` 是运维人员的明确操作;由任何具备写权限的协作者设为 `selfhosted` 时,对应作业切换到公司自有的自托管池:`DSH_CI_FAILOVER_LINUX` 下,Linux 作业与判定作业切到 `vm-backup` 池,快照并发降到共享虚拟机上限,并跳过托管路径的 pnpm 缓存恢复;`DSH_CI_FAILOVER_WINDOWS` 下,原生 Windows 作业切到 `dsh-win-ci` 池。每个开关都是写者可管理的仓库状态而非一次合并,因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 与 `serial / windows (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。 `ci-master.yml` 只豁免一个事件不做取消(`${{ github.event_name != 'push' }}`),因此一次 master 推送不会取消上一次推送留下的、仍在运行的演练。每次演练以单门禁工作进程执行完整的未分片聚合流程,耗时长于 master 合并的间隔;在无条件取消下,演练会在得出结论前被后续运行取代,该通道无法产出供响应者查看的就绪证据。 @@ -28,7 +28,7 @@ Status: implemented #### Windows 池 -`dsh-win-ci`:公司内部 Windows CI 服务器(一台 96 核 / 580 GB 机器)上 32 个常驻运行器实例(计划任务 `GH-Runner-01`…`GH-Runner-32`)。标签:`[self-hosted, dsh-win-ci, windows]`。镜像必须预装 Node 24、pnpm、Git(Git Bash 在 `PATH` 上,即 `C:\Program Files\Git\bin`——`bash` 工具按名称 spawn `bash`)、PowerShell 7,并为符号链接支持启用开发人员模式。Python runtime CI 还要求预装可通过 `PATH` 调用且提供 `venv` 与 `ensurepip` 的 Python;[Python runtime 说明](../../proposed/process/2026-09-06-python-runtime-windows-selfhosted.zh.md) 负责其私有准备流程与验证。通用 Windows 通道的工作区与 pnpm store 必须都位于 ReFS 卷(`F:`)上:这些安装步骤在 ReFS 上传递 `--package-import-method=clone`,这需要该卷布局以及系统 corepack pnpm 携带的 `@reflink/reflink` 原生模块(见 [Windows ReFS store note](../../archived/process/2026-08-30-windows-refs-store-block-clone-install.md));没有此布局的重建运行器会在 Windows 构建门禁阶段以 TS6231 失败。切换前先看 `serial / windows (self-hosted standby)` 最近一次运行:绿色热备验证该池能端到端执行 `check:ci:windows-complete`。 +`dsh-win-ci`:公司内部 Windows CI 服务器(一台 96 核 / 580 GB 机器)上 32 个常驻运行器实例(计划任务 `GH-Runner-01`…`GH-Runner-32`)。标签:`[self-hosted, dsh-win-ci, windows]`。镜像必须预装 Node 24、pnpm、Git(Git Bash 在 `PATH` 上,即 `C:\Program Files\Git\bin`——`bash` 工具按名称 spawn `bash`)、PowerShell 7,并为符号链接支持启用开发人员模式。通用 Windows 通道的工作区与 pnpm store 必须都位于 ReFS 卷(`F:`)上:这些安装步骤在 ReFS 上传递 `--package-import-method=clone`,这需要该卷布局以及系统 corepack pnpm 携带的 `@reflink/reflink` 原生模块(见 [Windows ReFS store note](../../archived/process/2026-08-30-windows-refs-store-block-clone-install.md));没有此布局的重建运行器会在 Windows 构建门禁阶段以 TS6231 失败。切换前先看 `serial / windows (self-hosted standby)` 最近一次运行:绿色热备验证该池能端到端执行 `check:ci:windows-complete`。 ### 切换步骤(任何具备写权限的协作者,约 1 分钟,无需合并) @@ -36,7 +36,7 @@ Status: implemented 1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**:名称 `DSH_CI_FAILOVER_LINUX`(Linux 池故障)或 `DSH_CI_FAILOVER_WINDOWS`(Windows 池故障),值 `selfhosted`。 2. 重新触发必需作业,使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向,也无法原地 re-run,因此对于本手册所述的无限排队故障,应取消卡住的运行并 re-run all jobs,或推送一个新提交;“Re-run failed jobs”只有在作业真正失败(而非仍在排队)时才有用。 -3. 切换到此完成。Linux 故障切换状态下,工作流还会把 `DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12,以限制共享虚拟机上的争抢,并跳过托管路径的 pnpm 缓存恢复,因为虚拟机的持久 store 会直接提供热安装。覆盖率在两个 Linux 池上都使用 4 个单 worker 插桩分区与 2 个豁免 worker。Windows 开关重定向原生 Windows 作业及符合条件的 Windows x64 Python runtime CI 构建;后者使用作业私有的工具与缓存,并跳过托管缓存恢复/保存步骤,详见 [Python runtime 说明](../../proposed/process/2026-09-06-python-runtime-windows-selfhosted.zh.md)。 +3. 切换到此完成。Linux 故障切换状态下,工作流还会把 `DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12,以限制共享虚拟机上的争抢,并跳过托管路径的 pnpm 缓存恢复,因为虚拟机的持久 store 会直接提供热安装。覆盖率在两个 Linux 池上都使用 4 个单 worker 插桩分区与 2 个豁免 worker。Windows 开关没有并发或缓存分支;它只重定向原生 Windows 作业的运行器池。 #**Dependabot 例外。**两个开关的选择器都刻意排除了 `dependabot[bot]`:故障切换期间,Dependabot 拉取请求继续在托管池排队,而不是把依赖项提供的代码放到持久化虚拟机上执行。故障期间 Dependabot PR 持续排队是预期行为而非切换失败;托管池恢复后它会自行完成。 @@ -49,7 +49,7 @@ Linux 开关启用期间,容量需覆盖 master 热备、主 CI 作业,以 ### 切回 -删除 `DSH_CI_FAILOVER_LINUX` 或 `DSH_CI_FAILOVER_WINDOWS` 变量(或改为 `selfhosted` 以外的任何值),新的运行即解析回各自的托管池,清除 Windows 开关时也包括符合条件的 Windows x64 Python runtime CI 构建。若故障期间追加注册过实例,将其移除。 +删除 `DSH_CI_FAILOVER_LINUX` 或 `DSH_CI_FAILOVER_WINDOWS` 变量(或改为 `selfhosted` 以外的任何值),新的运行即解析回各自的托管池。若故障期间追加注册过实例,将其移除。 ### 信任边界 diff --git a/.agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.i18n.yaml b/.agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.i18n.yaml new file mode 100644 index 0000000000..c7bff1c862 --- /dev/null +++ b/.agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.md +2026-09-06-python-runtime-windows-hosted.md: ca2f02e8bac8a90be2b10bd6d7ae0b68215152ae +2026-09-06-python-runtime-windows-hosted.zh.md: e1d2ca1a65de19a6604f0848de23fe5cc100e87f diff --git a/.agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.md b/.agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.md new file mode 100644 index 0000000000..ca2f02e8ba --- /dev/null +++ b/.agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.md @@ -0,0 +1,25 @@ +# Agent Note: Windows Python runtime CI stays on GitHub-hosted Windows + +Status: implemented + +English | [中文](2026-09-06-python-runtime-windows-hosted.zh.md) + +## Problem + +The Windows x64 target in [build-exe-for-python-sdk.yml](../../../../.github/workflows/build-exe-for-python-sdk.yml) started resolving through `DSH_CI_FAILOVER_WINDOWS=selfhosted` for trusted pull-request CI when #3629 added the failover selector and the job-private Windows toolchain. The shared `dsh-win-ci` pool did not make the lane more reliable. On 2026-09-06 the installed-wheel smoke passed at 09:12 on `dsh-win-ci-16` for [an earlier commit of the same pull request](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34023970384), then failed at 10:06 on `dsh-win-ci-21` for [another pull request](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34026500701) and at 10:46 on `dsh-win-ci-04` for [the same pull request](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34028339888/job/101473395734), where `smoke_sdk_profile_plugin`'s packaged `dsh plugin add` child exited without output while the Linux and macOS cells of that run passed. The migration proposal ([#3629](https://github.com/deepseek-harness/deepseek-harness/pull/3629)) remained `proposed` because its throughput and shared-load acceptance criteria were never measured. + +## Decision + +The Windows x64 target always uses its hosted `matrix.runner` — `windows-2025` for pull-request CI — with the standard setup-python toolchain, the pnpm cache restore, and the pkg cache. The failover selector, the job-private Python setup step, the self-hosted dependency install and post-step cleanup, the private setup script, and the routing spec from #3629 are removed. `DSH_CI_FAILOVER_WINDOWS=selfhosted` again retargets only the native Windows jobs in [ci.yml](../../../../.github/workflows/ci.yml); the [failover runbook](2026-07-26-ci-failover-runbook.md) and [python/development.md](../../../../python/development.md) describe hosted-only runtime builds. The migration's UTF-8 mode exports existed because the persistent host used a GBK default code page; hosted images provide the locale the lane previously ran under. + +## Alternatives considered + +**Keep the failover routing.** Rejected: the shared pool reproduced the same silent installed-wheel child death twice in one day while the migrated inventory's throughput acceptance stayed open, and routing a correctness lane through failover state couples it to an unrelated pool-outage switch. + +**Fix the shared pool instead.** Left to pool operators: the observed failures are subprocesses dying without output, not a missing image prerequisite, and the same image serves the native Windows failover jobs. + +**Retain the job-private toolchain on hosted images.** Rejected: the private uv/Python download exists to avoid mutating a persistent shared host; disposable hosted images already provide the registered Python 3.10 toolchain the pre-migration lane used. + +## Consequences + +Every qualifying pull request again pays GitHub-hosted Windows capacity for the runtime build, and the job-private setup and cleanup machinery — including the bounded filesystem retries — is gone with the lane. In exchange each build runs on a disposable host with the proven toolchain and hosted caches, and the Windows failover switch covers only the native Windows jobs as documented before the migration. A future self-hosted attempt must re-validate throughput and failure reproducibility on the actual pool before any routing change. diff --git a/.agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.zh.md b/.agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.zh.md new file mode 100644 index 0000000000..e1d2ca1a65 --- /dev/null +++ b/.agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.zh.md @@ -0,0 +1,25 @@ +# Agent Note: Windows Python runtime CI 保留在 GitHub 托管 Windows 上 + +Status: implemented + +[English](2026-09-06-python-runtime-windows-hosted.md) | 中文 + +## 问题 + +当 #3629 加入故障切换选择器与作业私有的 Windows 工具链后,[build-exe-for-python-sdk.yml](../../../../.github/workflows/build-exe-for-python-sdk.yml) 中的 Windows x64 目标开始对受信任的 PR CI 通过 `DSH_CI_FAILOVER_WINDOWS=selfhosted` 解析运行器。共享的 `dsh-win-ci` 池并未让该通道更可靠。2026-09-06,安装后 wheel 冒烟测试在 09:12 于 `dsh-win-ci-16` 上为[同一拉取请求的较早提交](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34023970384)通过,随后 10:06 在 `dsh-win-ci-21` 上为[另一个拉取请求](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34026500701)失败,10:46 在 `dsh-win-ci-04` 上为[同一拉取请求](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34028339888/job/101473395734)失败——`smoke_sdk_profile_plugin` 打包的 `dsh plugin add` 子进程无输出即退出,而该次运行的 Linux 与 macOS 单元均通过。迁移提案([#3629](https://github.com/deepseek-harness/deepseek-harness/pull/3629))保持 `proposed`,因为其吞吐量与共享负载验收标准从未实测。 + +## 决策 + +Windows x64 目标始终使用托管的 `matrix.runner`——PR CI 为 `windows-2025`——配以标准 setup-python 工具链、pnpm 缓存恢复与 pkg 缓存。来自 #3629 的故障切换选择器、作业私有 Python 准备步骤、自托管依赖安装与后置清理、私有准备脚本及路由测试均被移除。`DSH_CI_FAILOVER_WINDOWS=selfhosted` 再次只重定向 [ci.yml](../../../../.github/workflows/ci.yml) 中的原生 Windows 作业;[故障切换手册](2026-07-26-ci-failover-runbook.zh.md)与 [python/development.zh.md](../../../../python/development.zh.md) 描述仅托管的 runtime 构建。迁移中的 UTF-8 模式导出之所以存在,是因为持久主机使用 GBK 默认代码页;托管镜像提供该通道此前运行的区域设置。 + +## 已考虑的替代方案 + +**保留故障切换路由。** 不采用:共享池同一天两次复现相同的安装后 wheel 子进程无声死亡,而迁移清单的吞吐量验收仍然悬置;并且把正确性通道路由进故障切换状态,会使其耦合到无关的池故障开关。 + +**改为修复共享池。** 交由池运维者处理:观测到的失败是无输出即退出的子进程,而非镜像前置条件缺失;同一镜像还服务原生 Windows 故障切换作业。 + +**在托管镜像上保留作业私有工具链。** 不采用:私有 uv/Python 下载的存在理由是不修改持久共享主机;一次性托管镜像已提供迁移前通道使用的已注册 Python 3.10 工具链。 + +## 后果 + +每个符合条件的拉取请求再次为 runtime 构建支付 GitHub 托管 Windows 容量,作业私有准备与清理机制(包括有界文件系统重试)随通道一同移除。交换来的是每次构建运行在带标准工具链与托管缓存的一次性主机上,且 Windows 故障切换开关只覆盖迁移前文档所述的原生 Windows 作业。未来的自托管尝试必须在任何路由变更前,对实际池重新验证吞吐量与失败可复现性。 diff --git a/.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.i18n.yaml b/.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.i18n.yaml deleted file mode 100644 index df6194d4a1..0000000000 --- a/.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.i18n.yaml +++ /dev/null @@ -1,6 +0,0 @@ -# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each -# side as of the last confirmed-consistent state. Both languages carry equal authority; -# after editing either side, bring the other along and re-record with: -# pnpm run verify-translation-pairing --write .agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.md -2026-09-06-python-runtime-windows-selfhosted.md: 36ff1ab3fc94c9731f342ef9ff0541a1166e8108 -2026-09-06-python-runtime-windows-selfhosted.zh.md: 868d7fc800947305297db49be28c09836837a29c diff --git a/.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.md b/.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.md deleted file mode 100644 index 36ff1ab3fc..0000000000 --- a/.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.md +++ /dev/null @@ -1,42 +0,0 @@ -# Agent Note: Job-private Windows Python runtime CI - -Status: proposed - -English | [中文](2026-09-06-python-runtime-windows-selfhosted.zh.md) - -## Problem - -The native Python runtime matrix consumes hosted Windows capacity, but moving its build unchanged onto shared persistent runners would modify machine installation state and reuse user-level caches. The [CI failover runbook](../../implemented/process/2026-07-26-ci-failover-runbook.md) remains the owner of the existing general-purpose lanes and pool prerequisites; the [native Windows CI note](../../implemented/process/2026-08-08-native-windows-pull-request-ci.md) owns the independent Wine/native topology. This proposal covers only Python runtime builds. - -The [read-only prerequisite probe](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34012679056) found native Windows x64, Python 3.14.7 with venv/ensurepip, and enabled Developer Mode, but no Python toolcache. Linux lacked Docker, which both manylinux steps require. These observations permit a Windows-only experiment, not a claim that the runtime build passes. - -## Proposal - -Route only the Windows x64 target in [the runtime workflow](../../../../.github/workflows/build-exe-for-python-sdk.yml) to the persistent pool when `inputs.ci && !inputs.release`, the repository is the canonical repository, and the event is a same-repository non-fork, non-Dependabot PR. `DSH_CI_FAILOVER_WINDOWS=selfhosted` enables this routing; an unset or different value keeps the lane hosted. Release/manual builds, other events, Linux/macOS targets, planning, and the SDK-wheel job remain hosted. Throughput comparison and concurrent-job/cancellation acceptance remain pending. - -The [native setup probe](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34013261224/job/101432611073) downloads Python 3.10.20, verifies command resolution and a seeded smoke venv, asserts unchanged registered Python installations and Developer Mode, and proves job-root deletion. Windows recursive removal needs bounded retries after an observed non-empty-directory failure. The workflow additionally clears the exported compile-cache path and resets temporary-directory variables before action post-steps; focused tests pin those assignments, which are not part of the cited probe commit. The focused routing tests pass, and an inverted failover condition produces three expected failures before restoration. The first full native run builds the executable and wheel but fails when Python reads UTF-8 Session JSONL with the host GBK default. The setup exports Python UTF-8 mode and UTF-8 standard streams; a local forced-ASCII-locale subprocess reproduces the default-decoding failure and verifies the setting. The [corrected native Windows job](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34014942421/job/101437029350) completed successfully in 523 seconds, including executable and release-shaped wheel builds, installed-wheel keyless/live-API tests, upload, private-root cleanup, and action post-steps. - -The [private setup script](../../../../scripts/setup-python-runtime-windows.ps1) bootstraps uv 0.11.23 inside a temporary venv using the preinstalled interpreter, then downloads managed Python 3.10 into a unique job directory with `--no-bin --no-registry`. It creates a seeded tooling venv without further Python downloads. These flags exist in the [pinned uv source](https://github.com/astral-sh/uv/blob/3cdf50e0924f1ace7a92ddbac98b12a958b87688/crates/uv-cli/src/lib.rs#L6672-L6713); the [implementation](https://github.com/astral-sh/uv/blob/3cdf50e0924f1ace7a92ddbac98b12a958b87688/crates/uv/src/commands/python/install.rs#L667-L723) suppresses executable links and registry registration. CI checks Developer Mode rather than enabling it. - -The job owns its pnpm store, pkg/npm/node-gyp/Python/Node caches and temporary test directories. Dependency imports use copy rather than links into a shared store; hosted cache restore/save steps are skipped. An always-run cleanup removes only the recorded job root. Checkout does not persist credentials. These are resource-isolation measures, not protection against malicious code running under the same Windows account. - -## Alternatives considered - -**An independent Python failover switch.** Rejected for this shared host: reusing `DSH_CI_FAILOVER_WINDOWS` lets responders recover the platform with one switch and adds no variable. The trade-off is coupled placement: enabling native Windows failover also adds eligible Python runtime builds and their cold private-tool/cache setup load to the same host; clearing it returns both workloads to hosted pools. - -**Cold setup-python with a private toolcache.** Rejected: the concrete Python 3.10.11 [Windows release installer](https://github.com/actions/python-versions/blob/98e79473eb342d6f43487a289ca633620404742e/installers/win-setup-template.ps1#L21-L70) removes matching machine/current-user installation records and installs for all users. A private directory does not isolate that registry state. - -**Administrator-preprovisioned Python 3.10.** Viable with enforced cache-hit-only use and private dependency environments, but the measured pool does not supply it. Portable uv avoids requiring a host installation change. - -**Migrate Linux simultaneously.** Deferred until administrator-approved Docker provisioning and manylinux validation; skipping either manylinux step would weaken the wheel compatibility check. - -## Acceptance criteria - -- Selector tests prove hosted routing for release/manual, foreign/fork/Dependabot events, non-Windows targets, and an unset or unknown switch value. -- A trusted native Windows run builds the executable and release-shaped wheel, passes installed-wheel keyless and required live-API tests, and uploads the wheel without global Python or registry writes. -- Concurrent jobs use distinct cache/tool roots; success, failure, and cancellation exercise cleanup without deleting another job’s paths. -- Compare elapsed time and shared-pool load against hosted Windows before claiming cost or throughput improvement. Until then this note remains proposed. - -## Risks - -Private stores and copy imports trade warm-cache speed and disk space for bounded mutation. Portable Python can select a different 3.10 patch from setup-python. Downloads remain external dependencies; hard runner termination can prevent cleanup. Shared-account trust and pool availability remain operational limits, and the hosted fallback does not prove self-hosted readiness. diff --git a/.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.zh.md b/.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.zh.md deleted file mode 100644 index 868d7fc800..0000000000 --- a/.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.zh.md +++ /dev/null @@ -1,42 +0,0 @@ -# Agent Note: 作业私有的 Windows Python runtime CI - -Status: proposed - -[English](2026-09-06-python-runtime-windows-selfhosted.md) | 中文 - -## 问题 - -原生 Python runtime 矩阵消耗托管 Windows 容量,但将构建原样迁移到共享常驻运行器会修改机器安装状态并复用用户级缓存。[CI 故障切换手册](../../implemented/process/2026-07-26-ci-failover-runbook.zh.md) 继续负责现有通用通道与运行器池前置条件;[原生 Windows CI 说明](../../implemented/process/2026-08-08-native-windows-pull-request-ci.zh.md) 负责独立的 Wine/原生拓扑。本提案仅覆盖 Python runtime 构建。 - -[只读前置条件探测](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34012679056) 发现 Windows 为原生 x64,Python 3.14.7 提供 venv/ensurepip,开发人员模式已启用,但没有 Python 工具缓存。Linux 缺少两个 manylinux 步骤都依赖的 Docker。这些观测允许开展仅针对 Windows 的实验,并不证明 runtime 构建能够通过。 - -## 提案 - -仅当 `inputs.ci && !inputs.release`、仓库为规范仓库,且事件为同仓库非 fork、非 Dependabot 的 PR(Pull Request)时,将 [runtime 工作流](../../../../.github/workflows/build-exe-for-python-sdk.yml) 的 Windows x64 目标路由到常驻运行器池。`DSH_CI_FAILOVER_WINDOWS=selfhosted` 启用此路由;未设置或其他值使通道留在托管运行器。发布/手动构建、其他事件、Linux/macOS 目标、规划作业与 SDK wheel 包作业继续使用托管运行器。吞吐量对比及并发作业/取消验收仍待完成。 - -[原生准备探测](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34013261224/job/101432611073) 下载 Python 3.10.20,验证命令解析与包含 pip 的冒烟 venv,断言已注册的 Python 安装与开发人员模式不变,并证明作业根目录已删除。观测到目录非空的删除失败后,Windows 递归删除使用有限重试。工作流另外在 action 后置步骤前清除导出的编译缓存路径并重置临时目录变量;定向测试固定这些赋值,它们不属于引用的探测提交。定向路由测试通过,反转故障切换条件会产生三个预期失败,随后恢复条件。首次完整原生运行成功构建可执行文件与 wheel 包,但 Python 用主机默认 GBK 编码读取 UTF-8 Session JSONL 时失败。准备脚本导出 Python UTF-8 模式与 UTF-8 标准流;本地强制 ASCII locale 的子进程复现默认解码失败并验证设置。[修复后的原生 Windows 作业](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34014942421/job/101437029350) 在 523 秒内成功完成,包括可执行文件与发布形态 wheel 包构建、安装后 wheel 包的无密钥/真实 API 测试、上传、私有根目录清理及 action 后置步骤。 - -[私有准备脚本](../../../../scripts/setup-python-runtime-windows.ps1) 使用预装解释器,在临时 venv 内引导安装 uv 0.11.23,再通过 `--no-bin --no-registry` 将托管 Python 3.10 下载到唯一的作业目录。它创建包含初始工具包的工具 venv,禁止进一步下载 Python。[固定版本的 uv 源码](https://github.com/astral-sh/uv/blob/3cdf50e0924f1ace7a92ddbac98b12a958b87688/crates/uv-cli/src/lib.rs#L6672-L6713) 提供这些参数;[实现](https://github.com/astral-sh/uv/blob/3cdf50e0924f1ace7a92ddbac98b12a958b87688/crates/uv/src/commands/python/install.rs#L667-L723) 禁止创建可执行文件链接与注册表登记。CI 检查开发人员模式,不负责启用它。 - -作业独占其 pnpm 存储、pkg/npm/node-gyp/Python/Node 缓存以及临时测试目录。依赖导入使用复制,而不是指向共享存储的链接;跳过托管缓存恢复/保存步骤。始终执行的清理步骤仅删除记录的作业根目录。检出不保留凭据。这些措施隔离资源,不能防御同一 Windows 账户下运行的恶意代码。 - -## 已考虑的替代方案 - -**独立的 Python 故障切换开关。** 对这台共享主机不采用:复用 `DSH_CI_FAILOVER_WINDOWS` 让响应者用一个开关恢复整个平台,不新增变量。代价是部署位置相互绑定:启用原生 Windows 故障切换也会把符合条件的 Python runtime 构建及其私有工具/缓存冷启动负载加到同一主机上;清除开关则让两类工作负载都回到托管池。 - -**使用私有工具缓存冷启动 setup-python。** 不采用:具体的 Python 3.10.11 [Windows 发布安装器](https://github.com/actions/python-versions/blob/98e79473eb342d6f43487a289ca633620404742e/installers/win-setup-template.ps1#L21-L70) 会删除匹配的机器/当前用户安装记录,并为所有用户安装。私有目录无法隔离这些注册表状态。 - -**由管理员预装 Python 3.10。** 强制仅使用缓存命中路径并采用私有依赖环境时可行,但观测到的运行器池并未提供它。便携 uv 避免要求修改主机安装。 - -**同时迁移 Linux。** 推迟到管理员批准 Docker 部署并完成 manylinux 验证之后;跳过任一 manylinux 步骤都会削弱 wheel 包兼容性检查。 - -## 验收标准 - -- 选择器测试证明发布/手动、外部仓库/fork/Dependabot 事件、非 Windows 目标及未设置或未知的开关值均使用托管路由。 -- 一次可信的原生 Windows 运行构建可执行文件与发布形态 wheel 包,通过安装后 wheel 包的无密钥测试及必需的真实 API 测试,并上传 wheel 包,期间不写全局 Python 或注册表。 -- 并发作业使用不同的缓存/工具根目录;成功、失败与取消路径均执行清理且不删除其他作业的路径。 -- 在宣称成本或吞吐量改善之前,对比托管 Windows 的耗时与共享池负载。此前本说明保持 proposed 状态。 - -## 风险 - -私有存储与复制导入以热缓存速度和磁盘空间换取受限的修改范围。便携 Python 可能选择与 setup-python 不同的 3.10 补丁版本。下载仍依赖外部服务;运行器被强制终止可能阻止清理。共享账户信任与运行器池可用性仍是运维限制,托管回退也不能证明自托管运行器已就绪。 diff --git a/.github/workflows/build-exe-for-python-sdk.yml b/.github/workflows/build-exe-for-python-sdk.yml index b26a7c8ca6..c00241b0e0 100644 --- a/.github/workflows/build-exe-for-python-sdk.yml +++ b/.github/workflows/build-exe-for-python-sdk.yml @@ -150,19 +150,7 @@ jobs: build: needs: [plan, sdk-wheel] name: ${{ matrix.target }} - # Release and manual builds retain disposable hosted images. Only trusted CI - # may use the persistent Windows host; Linux requires an unavailable Docker daemon. - runs-on: >- - ${{ inputs.ci && !inputs.release - && github.repository == 'deepseek-harness/deepseek-harness' - && github.event_name == 'pull_request' - && github.event.pull_request.head.repo.full_name == github.repository - && !github.event.pull_request.head.repo.fork - && github.event.pull_request.user.login != 'dependabot[bot]' - && matrix.target == 'node24-win-x64' - && vars.DSH_CI_FAILOVER_WINDOWS == 'selfhosted' - && fromJSON('["self-hosted", "dsh-win-ci", "windows", "x64"]') - || matrix.runner }} + runs-on: ${{ matrix.runner }} timeout-minutes: 45 strategy: fail-fast: false @@ -170,21 +158,13 @@ jobs: include: ${{ fromJSON(needs.plan.outputs.matrix) }} steps: - uses: actions/checkout@v6 - with: - persist-credentials: false - - - name: Prepare private Windows Python toolchain - id: private-windows - if: runner.os == 'Windows' && runner.environment == 'self-hosted' - shell: pwsh - run: ./scripts/setup-python-runtime-windows.ps1 - uses: pnpm/action-setup@v4 with: dest: ${{ runner.temp }}/setup-pnpm-js-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }} - name: Enable Windows Developer Mode (symlink support) - if: runner.os == 'Windows' && runner.environment != 'self-hosted' + if: runner.os == 'Windows' shell: pwsh run: >- reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock" @@ -195,22 +175,18 @@ jobs: - uses: actions/setup-node@v6 with: node-version: 24 - cache: ${{ runner.environment != 'self-hosted' && 'pnpm' || '' }} - package-manager-cache: false + cache: pnpm - uses: actions/setup-python@v6.3.0 - if: runner.environment != 'self-hosted' with: python-version: '3.10' - name: Install Python build tooling - if: runner.environment != 'self-hosted' run: python -m pip install uv==0.11.23 # Cache pkg's target Node binary; lockfile changes roll the # exact key while the restore prefix can seed its replacement. - uses: actions/cache@v4 - if: runner.environment != 'self-hosted' with: path: ~/.pkg-cache key: pkg-fetch-${{ matrix.target }}-${{ hashFiles('pnpm-lock.yaml') }} @@ -218,16 +194,8 @@ jobs: pkg-fetch-${{ matrix.target }}- - name: Install (immutable) - if: runner.environment != 'self-hosted' run: pnpm install --frozen-lockfile - - name: Install private Windows dependencies (immutable) - if: runner.os == 'Windows' && runner.environment == 'self-hosted' - shell: pwsh - run: | - pnpm install --frozen-lockfile --package-import-method=copy - if ($LASTEXITCODE -ne 0) { throw 'Private Windows dependency installation failed.' } - - name: Rebuild Linux node-pty against manylinux 2.28 if: runner.os == 'Linux' env: @@ -512,22 +480,3 @@ jobs: path: dist-python/${{ steps.runtime-posix.outputs.wheel || steps.runtime-windows.outputs.wheel }} if-no-files-found: error retention-days: 7 - - # Node action posts consume temp/compile-cache paths. pnpm post skips - # pruning without run_install; no Python/pkg/npm subprocess runs after cleanup. - - name: Remove private Windows toolchain and test directories - if: always() && steps.private-windows.outputs.root != '' - shell: pwsh - env: - PRIVATE_ROOT: ${{ steps.private-windows.outputs.root }} - run: | - Set-Location $env:GITHUB_WORKSPACE - $env:TMP = $env:RUNNER_TEMP - $env:TEMP = $env:RUNNER_TEMP - Remove-Item Env:NODE_COMPILE_CACHE -ErrorAction SilentlyContinue - "NODE_COMPILE_CACHE=" >> $env:GITHUB_ENV - "TMP=$env:RUNNER_TEMP" >> $env:GITHUB_ENV - "TEMP=$env:RUNNER_TEMP" >> $env:GITHUB_ENV - node -e "const fs = require('node:fs'); const root = process.env.PRIVATE_ROOT; if (fs.lstatSync(root, { throwIfNoEntry: false })?.isSymbolicLink()) fs.unlinkSync(root); else fs.rmSync(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 })" - if ($LASTEXITCODE -ne 0) { throw 'Private Windows job directory removal failed.' } - if (Test-Path -LiteralPath $env:PRIVATE_ROOT) { throw 'Private Windows job directory survived cleanup.' } diff --git a/python/development.i18n.yaml b/python/development.i18n.yaml index 245eb796aa..f1ada26c49 100644 --- a/python/development.i18n.yaml +++ b/python/development.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write python/development.md -development.md: 56770042d381566e29bee2451b76dcb6ab0852fd -development.zh.md: 010dd06de6245f9dbcf8cee74ee2846d4027d3ce +development.md: aa0144d7eaa66711d0f08316d4445da060918ca8 +development.zh.md: a35f6fc8de1bdbd282fd8999a1440fde0c400b34 diff --git a/python/development.md b/python/development.md index 56770042d3..aa0144d7ea 100644 --- a/python/development.md +++ b/python/development.md @@ -15,8 +15,6 @@ pnpm exec tsx scripts/build-exe-for-python-sdk.ts Use `--skip-build` when the required `lib/` artifacts already exist, or `--targets=node24-linux-x64,node24-linux-arm64,node24-macos-arm64,node24-macos-x64,node24-win-x64` to select platforms. Build each target on its native architecture. Products land in `dist-exe/` and the script syncs the selected carriers into `python/sdk-runtime/`. Windows emits `.exe` and `-rg.exe`; macOS also syncs the matching spawn helper required by `node-pty`. -CI-only Windows x64 builds can use the self-hosted pool when `DSH_CI_FAILOVER_WINDOWS=selfhosted`: only same-repository non-fork, non-Dependabot pull requests qualify. The job downloads Python 3.10 into a private temporary directory without registering it in Windows, isolates build caches and test environments, and removes that directory after success or failure. Release and manual builds, Linux and macOS targets, and the SDK-wheel helper retain hosted runners. See the [runner isolation proposal](../.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.md) for image prerequisites and validation limits. - ## Validate the SDK Keep the virtual environment outside `python/`, install the test group, and run the Python suite: diff --git a/python/development.zh.md b/python/development.zh.md index 010dd06de6..a35f6fc8de 100644 --- a/python/development.zh.md +++ b/python/development.zh.md @@ -15,8 +15,6 @@ pnpm exec tsx scripts/build-exe-for-python-sdk.ts 所需 `lib/` 产物已存在时使用 `--skip-build`;如需选择平台,请使用 `--targets=node24-linux-x64,node24-linux-arm64,node24-macos-arm64,node24-macos-x64,node24-win-x64`。每个目标都应在其原生架构上构建。产物写入 `dist-exe/`,脚本会将所选载体同步到 `python/sdk-runtime/`。Windows 会生成 `.exe` 与 `-rg.exe`;macOS 构建还会同步 `node-pty` 所需的配套 spawn 辅助程序。 -仅用于 CI 的 Windows x64 构建可在 `DSH_CI_FAILOVER_WINDOWS=selfhosted` 时使用自托管池:只有同仓库、非 fork、非 Dependabot 的拉取请求符合条件。作业将 Python 3.10 下载到私有临时目录而不在 Windows 中注册它,隔离构建缓存与测试环境,并在成功或失败后删除该目录。发布与手动构建、Linux 与 macOS 目标,以及 SDK wheel 辅助作业仍使用托管运行器。镜像前提与验证限制见[运行器隔离提案](../.agents/notes/proposed/process/2026-09-06-python-runtime-windows-selfhosted.zh.md)。 - ## 验证 SDK 请将虚拟环境放在 `python/` 之外,安装测试组,然后运行 Python 测试套件: diff --git a/scripts/python-runtime-selfhosted.spec.ts b/scripts/python-runtime-selfhosted.spec.ts deleted file mode 100644 index 29a8302480..0000000000 --- a/scripts/python-runtime-selfhosted.spec.ts +++ /dev/null @@ -1,167 +0,0 @@ -import { spawnSync } from 'node:child_process' -import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs' -import { tmpdir } from 'node:os' -import { resolve } from 'node:path' -import { runInNewContext } from 'node:vm' -import * as yaml from 'js-yaml' -import { describe, expect, it } from 'vitest' - -const root = resolve(import.meta.dirname, '..') -const workflow = yaml.load(readFileSync(resolve(root, '.github/workflows/build-exe-for-python-sdk.yml'), 'utf8')) as { - jobs: Record }> }> -} -const build = workflow.jobs.build! -const selector = build['runs-on'].slice(3, -2).trim() -const windows = ['self-hosted', 'dsh-win-ci', 'windows', 'x64'] - -function context() { - return { - inputs: { ci: true, release: false }, - github: { - repository: 'deepseek-harness/deepseek-harness', - event_name: 'pull_request', - ref: 'refs/pull/42/merge', - event: { pull_request: { - head: { repo: { full_name: 'deepseek-harness/deepseek-harness', fork: false } }, - user: { login: 'contributor' }, - } }, - }, - matrix: { target: 'node24-win-x64', runner: 'windows-2025' }, - vars: { DSH_CI_FAILOVER_WINDOWS: 'selfhosted' }, - fromJSON: JSON.parse, - } -} - -function route(value: ReturnType, expression = selector): unknown { - // These canonical-case fixtures share JS/Actions comparison results; Actions also ignores string case. - // This evaluates the selected syntax, not GitHub's complete expression language. - return runInNewContext(expression, value, { timeout: 1000 }) -} - -describe('Python runtime self-hosted routing', () => { - it('routes same-repository member PRs to native x64 Windows', () => { - expect(route(context())).toEqual(windows) - }) - - it.each([ - ['release caller', (value: ReturnType) => { value.inputs.release = true }], - ['non-CI caller', (value: ReturnType) => { value.inputs.ci = false }], - ['manual dispatch', (value: ReturnType) => { value.github.event_name = 'workflow_dispatch' }], - ['pull_request_target', (value: ReturnType) => { value.github.event_name = 'pull_request_target' }], - ['unknown event', (value: ReturnType) => { value.github.event_name = '' }], - ['fork', (value: ReturnType) => { value.github.event.pull_request.head.repo.fork = true }], - ['different repository head', (value: ReturnType) => { value.github.event.pull_request.head.repo.full_name = 'someone/fork' }], - ['different caller repository', (value: ReturnType) => { value.github.repository = 'someone/fork' }], - ['Dependabot author', (value: ReturnType) => { value.github.event.pull_request.user.login = 'dependabot[bot]' }], - ['disabled failover', (value: ReturnType) => { value.vars.DSH_CI_FAILOVER_WINDOWS = '' }], - ['unknown failover value', (value: ReturnType) => { value.vars.DSH_CI_FAILOVER_WINDOWS = 'hosted' }], - ['master push', (value: ReturnType) => { value.github.event_name = 'push'; value.github.ref = 'refs/heads/master' }], - ['branch push', (value: ReturnType) => { value.github.event_name = 'push'; value.github.ref = 'refs/heads/topic' }], - ['tag push', (value: ReturnType) => { value.github.event_name = 'push'; value.github.ref = 'refs/tags/python-v1' }], - ] as const)('keeps %s on the hosted fallback', (_name, change) => { - const value = context() - change(value) - expect(route(value)).toBe('windows-2025') - }) - - it.each([ - ['node24-linux-x64', 'ubuntu-latest'], - ['node24-linux-arm64', 'ubuntu-24.04-arm'], - ['node24-macos-arm64', 'macos-latest'], - ['node24-macos-x64', 'macos-15-intel'], - ])('keeps %s hosted even with failover enabled', (target, runner) => { - const value = context() - value.matrix = { target, runner } - expect(route(value)).toBe(runner) - }) - - it('keeps setup helper jobs on hosted images', () => { - expect(workflow.jobs.plan!['runs-on']).toBe('ubuntu-latest') - expect(workflow.jobs['sdk-wheel']!['runs-on']).toBe('ubuntu-latest') - }) - - it('isolates setup before pnpm and excludes shared installers and cache archives', () => { - const privateSetup = build.steps.findIndex(step => step.id === 'private-windows') - expect(privateSetup).toBeGreaterThan(0) - expect(privateSetup).toBeLessThan(build.steps.findIndex(step => step.uses?.startsWith('pnpm/action-setup@'))) - for (const step of build.steps.filter(step => step.uses?.startsWith('actions/setup-python@') || step.uses?.startsWith('actions/cache@') || step.name === 'Install Python build tooling')) { - expect(step.if).toBe("runner.environment != 'self-hosted'") - } - expect(build.steps.find(step => step.name?.startsWith('Enable Windows'))?.if).toBe("runner.os == 'Windows' && runner.environment != 'self-hosted'") - expect(build.steps.find(step => step.uses?.startsWith('actions/setup-node@'))?.with?.cache).toContain("runner.environment != 'self-hosted'") - expect(build.steps.at(-1)).toMatchObject({ if: "always() && steps.private-windows.outputs.root != ''", shell: 'pwsh' }) - expect(build.steps.find(step => step.uses?.startsWith('actions/setup-node@'))?.with?.['package-manager-cache']).toBe(false) - expect(build.steps.find(step => step.name === 'Install (immutable)')?.if).toBe("runner.environment != 'self-hosted'") - expect(build.steps.find(step => step.name === 'Install private Windows dependencies (immutable)')).toMatchObject({ - if: "runner.os == 'Windows' && runner.environment == 'self-hosted'", - shell: 'pwsh', - }) - expect(build.steps.find(step => step.name === 'Install private Windows dependencies (immutable)')?.run).toContain('pnpm install --frozen-lockfile --package-import-method=copy') - const cleanup = build.steps.at(-1)!.run! - expect(cleanup).toContain('"NODE_COMPILE_CACHE=" >> $env:GITHUB_ENV') - expect(cleanup).toContain('"TMP=$env:RUNNER_TEMP" >> $env:GITHUB_ENV') - expect(cleanup).toContain('"TEMP=$env:RUNNER_TEMP" >> $env:GITHUB_ENV') - expect(cleanup).toContain('maxRetries: 10, retryDelay: 100') - }) - - it.each(['root', 'nested', 'absent'] as const)('cleans a %s job directory without deleting another target', (location) => { - const temp = mkdtempSync(resolve(tmpdir(), 'python-runtime-cleanup-')) - try { - const target = resolve(temp, 'other-job') - const owned = resolve(temp, 'owned') - mkdirSync(target) - writeFileSync(resolve(target, 'sentinel'), 'preserve') - if (location === 'nested') mkdirSync(owned) - if (location !== 'absent') symlinkSync(target, location === 'root' ? owned : resolve(owned, 'link'), 'junction') - const command = /node -e "([^"\n]+)"/.exec(build.steps.at(-1)!.run!)?.[1] - expect(command).toBeDefined() - const result = spawnSync(process.execPath, ['-e', command!], { - env: { ...process.env, PRIVATE_ROOT: owned, NODE_COMPILE_CACHE: '' }, - encoding: 'utf8', - timeout: 10000, - }) - expect(result.error).toBeUndefined() - expect(result.signal).toBeNull() - expect(result.status, result.stderr).toBe(0) - expect(existsSync(owned)).toBe(false) - expect(readFileSync(resolve(target, 'sentinel'), 'utf8')).toBe('preserve') - } finally { - rmSync(temp, { recursive: true, force: true }) - } - }) - - it('reads UTF-8 Session JSONL independently of the host locale', () => { - const setup = readFileSync(resolve(root, 'scripts/setup-python-runtime-windows.ps1'), 'utf8') - const utf8 = /PYTHONUTF8 = '([^']+)'/.exec(setup)?.[1] - expect(utf8).toBe('1') - const result = spawnSync(process.platform === 'win32' ? 'python' : 'python3', ['-c', [ - 'import pathlib, tempfile, sys', - 'assert sys.flags.utf8_mode == 1', - 'with tempfile.TemporaryDirectory(prefix="python-runtime-encoding-") as root:', - ' log = pathlib.Path(root) / "session.jsonl"', - ' text = chr(0x2014) + chr(0x4e2d)', - ' log.write_bytes(text.encode("utf-8"))', - ' assert log.read_text() == text', - ].join('\n')], { - env: { ...process.env, LC_ALL: 'C', LANG: 'C', PYTHONCOERCECLOCALE: '0', PYTHONUTF8: utf8 }, - encoding: 'utf8', - timeout: 10000, - }) - expect(result.error).toBeUndefined() - expect(result.signal).toBeNull() - expect(result.status, result.stderr).toBe(0) - }) - - it('pins portable Python without registry or shared cache writes', () => { - const setup = readFileSync(resolve(root, 'scripts/setup-python-runtime-windows.ps1'), 'utf8') - expect(setup).toContain('--no-bin --no-registry 3.10') - expect(setup).toContain('--managed-python --no-python-downloads --seed') - expect(setup).toContain('UV_PYTHON_INSTALL_REGISTRY') - expect(setup).toContain('PNPM_CONFIG_STORE_DIR') - expect(setup).toContain('PKG_CACHE_PATH') - expect(setup.indexOf('$bootstrapScripts >> $env:GITHUB_PATH')).toBeLessThan(setup.indexOf('$toolingScripts >> $env:GITHUB_PATH')) - expect(setup).toContain('AllowDevelopmentWithoutDevLicense -ErrorAction SilentlyContinue') - expect(setup).toContain('$null -eq $devMode -or') - expect(setup).not.toMatch(/reg add|Set-ItemProperty|InstallAllUsers/) - }) -}) diff --git a/scripts/setup-python-runtime-windows.ps1 b/scripts/setup-python-runtime-windows.ps1 deleted file mode 100644 index f8f053f272..0000000000 --- a/scripts/setup-python-runtime-windows.ps1 +++ /dev/null @@ -1,57 +0,0 @@ -# Prepare a job-private Python 3.10 toolchain without Windows installer or registry writes. -$ErrorActionPreference = 'Stop' -$root = Join-Path $env:RUNNER_TEMP ("python-runtime-" + [guid]::NewGuid().ToString('N')) -New-Item -ItemType Directory -Path $root | Out-Null -"root=$root" >> $env:GITHUB_OUTPUT - -$privateEnvironment = @{ - # Session JSONL and SDK pipes use UTF-8, including on Chinese Windows images. - PYTHONUTF8 = '1' - PYTHONIOENCODING = 'utf-8' - TMP = $root - TEMP = $root - UV_CACHE_DIR = (Join-Path $root 'uv-cache') - UV_PYTHON_INSTALL_DIR = (Join-Path $root 'python') - UV_PYTHON_INSTALL_BIN = '0' - UV_PYTHON_INSTALL_REGISTRY = '0' - UV_NO_CONFIG = '1' - PIP_CACHE_DIR = (Join-Path $root 'pip-cache') - npm_config_cache = (Join-Path $root 'npm-cache') - npm_config_devdir = (Join-Path $root 'node-gyp') - PNPM_CONFIG_PACKAGE_IMPORT_METHOD = 'copy' - PKG_CACHE_PATH = (Join-Path $root 'pkg-cache') - PNPM_CONFIG_STORE_DIR = (Join-Path $root 'pnpm-store') - NODE_COMPILE_CACHE = (Join-Path $root 'node-compile-cache') -} -foreach ($entry in $privateEnvironment.GetEnumerator()) { - [Environment]::SetEnvironmentVariable($entry.Key, $entry.Value, 'Process') - "$($entry.Key)=$($entry.Value)" >> $env:GITHUB_ENV -} - -if ([Runtime.InteropServices.RuntimeInformation]::OSArchitecture -ne 'X64') { - throw 'Python runtime CI requires a native x64 Windows host.' -} -$devMode = Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\AppModelUnlock' -Name AllowDevelopmentWithoutDevLicense -ErrorAction SilentlyContinue -if ($null -eq $devMode -or $devMode.AllowDevelopmentWithoutDevLicense -ne 1) { - throw 'The self-hosted Windows image must enable Developer Mode before CI.' -} - -$bootstrap = Join-Path $root 'bootstrap' -python -m venv $bootstrap -if ($LASTEXITCODE -ne 0) { throw 'The self-hosted Windows image requires Python with venv and ensurepip.' } -$bootstrapScripts = Join-Path $bootstrap 'Scripts' -& (Join-Path $bootstrapScripts 'python.exe') -m pip --isolated --disable-pip-version-check --no-cache-dir install uv==0.11.23 -if ($LASTEXITCODE -ne 0) { throw 'Job-private uv installation failed.' } -$uv = Join-Path $bootstrapScripts 'uv.exe' -& $uv python install --install-dir $env:UV_PYTHON_INSTALL_DIR --no-bin --no-registry 3.10 -if ($LASTEXITCODE -ne 0) { throw 'Job-private Python 3.10 download failed.' } -$tooling = Join-Path $root 'tooling' -& $uv venv --python 3.10 --managed-python --no-python-downloads --seed $tooling -if ($LASTEXITCODE -ne 0) { throw 'Job-private Python 3.10 environment creation failed.' } -$toolingScripts = Join-Path $tooling 'Scripts' -$python = Join-Path $toolingScripts 'python.exe' -& $python -c 'import platform, sys; assert sys.version_info[:2] == (3, 10); assert platform.machine() == "AMD64"; print(sys.version); print(sys.executable)' -if ($LASTEXITCODE -ne 0) { throw 'Job-private Python version or architecture is incorrect.' } -# Actions prepends each entry: the last appended directory wins Python lookup. -$bootstrapScripts >> $env:GITHUB_PATH -$toolingScripts >> $env:GITHUB_PATH