diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml index 8d50d7a5f9..3798f5cdcd 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md -2026-07-26-ci-failover-runbook.md: b24996a4ba4dfaa4b26f88519a61f45c81efb5b5 -2026-07-26-ci-failover-runbook.zh.md: ee7339d70e4796c367f97490ea57468687464b3f +2026-07-26-ci-failover-runbook.md: 6123592e7419efe3ab514fc0e93df267137e5afa +2026-07-26-ci-failover-runbook.zh.md: c5406211406e11205df4254f88e61bafbdf15f03 diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md index b24996a4ba..6123592e74 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md @@ -10,7 +10,7 @@ The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.ym ## Decision -Each of the three required Linux worker jobs, the independent native Windows job, and the `all checks passed` verdict job — which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through a repository variable, and the switch is split by platform so an outage on one platform does not retarget the other. The three Linux workers and the `all checks passed` verdict (whose `needs` are the required Linux workers and which runs on the `vm-backup` pool) resolve through `DSH_CI_FAILOVER_LINUX`; the native Windows job resolves through `DSH_CI_FAILOVER_WINDOWS`. Unset (normal), they run on the hosted enterprise pools. Set to `selfhosted` by any repository writer, the corresponding jobs retarget onto the in-house self-hosted pool: under `DSH_CI_FAILOVER_LINUX`, the Linux jobs and verdict move onto the `vm-backup` pool, snapshot concurrency drops to the shared-VM bound, and the hosted-path pnpm cache restores are skipped; under `DSH_CI_FAILOVER_WINDOWS`, the native Windows job moves onto the `dsh-win-ci` pool. Each switch is writer-manageable repository state, not a merge, so it works while every check is red. The in-house pools' readiness is continuously re-proven by the `serial / linux (self-hosted standby)` and `serial / windows (self-hosted standby)` lanes, which run the complete unsharded aggregates on every master push. +Each of the three required Linux worker jobs, the independent native Windows job, and the `all checks passed` verdict job — which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through a repository variable, and the switch is split by platform so an outage on one platform does not retarget the other. The three Linux workers and the `all checks passed` verdict (whose `needs` are the required Linux workers and which runs on the `vm-backup` pool) resolve through `DSH_CI_FAILOVER_LINUX`; the native Windows job resolves through `DSH_CI_FAILOVER_WINDOWS`. Unset, they default to the hosted pools; selecting `selfhosted` is an explicit operator choice. Set to `selfhosted` by any repository writer, the corresponding jobs retarget onto the in-house self-hosted pool: under `DSH_CI_FAILOVER_LINUX`, the Linux jobs and verdict move onto the `vm-backup` pool, snapshot concurrency drops to the shared-VM bound, and the hosted-path pnpm cache restores are skipped; under `DSH_CI_FAILOVER_WINDOWS`, the native Windows job moves onto the `dsh-win-ci` pool. Each switch is writer-manageable repository state, not a merge, so it works while every check is red. The in-house pools' readiness is continuously re-proven by the `serial / linux (self-hosted standby)` and `serial / windows (self-hosted standby)` lanes, which run the complete unsharded aggregates on every master push. `ci-master.yml` exempts exactly one event from `cancel-in-progress` (`${{ github.event_name != 'push' }}`), so one master push does not cancel the drill still running from the previous one. Each drill runs its complete unsharded aggregate with one gate worker, which takes longer than the interval between master merges; under unconditional cancellation a drill is superseded before reaching a verdict and the lane yields no readiness evidence for a responder to check. @@ -18,9 +18,13 @@ The exemption is narrower than "a drill always finishes", in two ways. GitHub ke The decision belongs at workflow level because cancellation applies to the whole superseded run: a job-level `concurrency` group does not exempt its job. The negated form is load-bearing rather than cosmetic: naming `pull_request` alone would also stop cancelling `workflow_dispatch`, and each runner benchmark fans out to twelve larger runners for up to fifteen minutes inside this same group on master, so a re-dispatch would queue ahead of a drill instead of replacing a stale measurement. What bounds the cost is that a master push in `ci-master.yml` carries only `wine-apt-cache` and these two drills; the pull-request jobs live in the separate `ci.yml` (which does not see `push`), and the benchmarks are `workflow_dispatch`-gated within `ci-master.yml`. `scripts/ci-workflow.spec.ts` pins that push-reachable set — classifying by exact condition, since a negated event test mentions the event it excludes — so a new push-reachable job cannot quietly start accumulating uncancelled runs. +### Release rehearsals share the Linux switch + +`DSH_CI_FAILOVER_LINUX=selfhosted` also routes the credential-free dependency-layout job and both dsh/vendor pack jobs onto `vm-backup` for eligible same-repository PRs and master pushes. Their [release rehearsal decision](2026-09-06-release-rehearsal-selfhosted.md) owns the stricter event eligibility and hosted manual dispatch. This coupling is intentional: keeping the variable set to save release minutes also keeps the eligible main-CI Linux jobs self-hosted. Clearing it returns both workloads to their hosted targets for subsequent runs; publication stays hosted regardless. + ### What the in-house pool is -`vm-backup`: one 64-core VM, six always-on systemd-managed runner instances. Its image must preinstall Playwright Chromium's Linux system packages; CI downloads the lockfile-selected browser but never runs `apt` on this persistent shared host. Check the latest `serial / linux (self-hosted standby)` run before switching: its aggregate includes browser replay, so a green standby verifies both ordinary capacity and this browser prerequisite. +`vm-backup`: one shared VM with multiple always-on systemd-managed runner instances. Registrations share its CPU, memory, and disk; their count is not a count of independent machines. Its image must preinstall Playwright Chromium's Linux system packages; CI downloads the lockfile-selected browser but never runs `apt` on this persistent shared host. Check the latest `serial / linux (self-hosted standby)` run before switching: its aggregate includes browser replay, so a green standby verifies both ordinary capacity and this browser prerequisite. #### Windows pool @@ -40,7 +44,7 @@ The two switches are independent: flip only the one whose platform is degraded. ## Capacity during failover -Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /` (the globs also catch `.runner_migrated`/`.credentials_migrated`, which GitHub writes on migrated runners and which equally trigger the already-configured refusal) — then run `config.sh` (copying `.runner`/`.credentials` verbatim makes it refuse with "already configured"), and **start the listener**: `sudo ./svc.sh install ubuntu && sudo ./svc.sh start`. Registration alone leaves the runner offline; only a started service adds capacity. About a minute per instance. +Capacity includes the master standby, main-CI jobs, and three release-rehearsal jobs for each eligible PR or master push while the Linux switch is set. The release workflows do not cancel running rehearsals when another run arrives, so overlapping refs can add sustained build, pack, and install load. Check current CPU, memory, disk, and queue pressure before extending self-hosted operation; extra registrations on this VM add scheduling slots, not machine resources. Do not infer spare capacity from the standby alone. When host resources permit extra registrations, use an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /` (the globs also catch `.runner_migrated`/`.credentials_migrated`, which GitHub writes on migrated runners and which equally trigger the already-configured refusal) — then run `config.sh` (copying `.runner`/`.credentials` verbatim makes it refuse with "already configured"), and **start the listener**: `sudo ./svc.sh install ubuntu && sudo ./svc.sh start`. Registration alone leaves the runner offline; a started service adds a scheduling slot, not CPU or memory. ### Switch back @@ -55,7 +59,7 @@ The variables are writer-manageable repository state; a pull request event itsel **Merge a workflow change to switch pools.** Rejected because the outage that motivates the switch is exactly the state in which no PR can merge: the required checks are the ones failing. A repository variable is writer-manageable state that takes effect on re-run without a merge. -**Keep the self-hosted pool always in the required path.** Rejected because it trades hosted-pool availability for the in-house VM's, moving a single point of failure rather than adding a fallback. The variables keep the hosted pools primary and the self-hosted pools proven, one-action standbys; splitting them by platform means an outage on one platform does not retarget the other. +**Keep the self-hosted pool always in the required path.** Rejected because it trades hosted-pool availability for the in-house VM's, moving a single point of failure rather than adding a fallback. The unset defaults retain hosted targets and the switches provide a reversible, operator-selected self-hosted path; splitting them by platform means an outage on one platform does not retarget the other. ## Consequences diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md index ee7339d70e..c540621140 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md @@ -10,7 +10,7 @@ Status: implemented ## 决策 -三个必需的 Linux 工作作业、独立的原生 Windows 作业,以及 `all checks passed` 判定作业(若不随切换,即使全部工作作业通过,它仍会滞留在故障池的队列中)——各自通过仓库变量解析运行器池,且开关按平台拆分,使一个平台的故障不会重定向另一个平台。三个 Linux 工作作业与 `all checks passed` 判定作业(其 `needs` 是必需的 Linux 工作作业,且运行在 `vm-backup` 池上)通过 `DSH_CI_FAILOVER_LINUX` 解析;原生 Windows 作业通过 `DSH_CI_FAILOVER_WINDOWS` 解析。变量不存在(正常)时它们运行在托管企业池上;由任何具备写权限的协作者设为 `selfhosted` 时,对应作业切换到公司自有的自托管池:`DSH_CI_FAILOVER_LINUX` 下,Linux 作业与判定作业切到 `vm-backup` 池,快照并发降到共享虚拟机上限,并跳过托管路径的 pnpm 缓存恢复;`DSH_CI_FAILOVER_WINDOWS` 下,原生 Windows 作业切到 `dsh-win-ci` 池。每个开关都是写者可管理的仓库状态而非一次合并,因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 与 `serial / windows (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。 +三个必需的 Linux 工作作业、独立的原生 Windows 作业,以及 `all checks passed` 判定作业(若不随切换,即使全部工作作业通过,它仍会滞留在故障池的队列中)——各自通过仓库变量解析运行器池,且开关按平台拆分,使一个平台的故障不会重定向另一个平台。三个 Linux 工作作业与 `all checks passed` 判定作业(其 `needs` 是必需的 Linux 工作作业,且运行在 `vm-backup` 池上)通过 `DSH_CI_FAILOVER_LINUX` 解析;原生 Windows 作业通过 `DSH_CI_FAILOVER_WINDOWS` 解析。未设置变量时默认使用托管池;选择 `selfhosted` 是运维人员的明确操作;由任何具备写权限的协作者设为 `selfhosted` 时,对应作业切换到公司自有的自托管池:`DSH_CI_FAILOVER_LINUX` 下,Linux 作业与判定作业切到 `vm-backup` 池,快照并发降到共享虚拟机上限,并跳过托管路径的 pnpm 缓存恢复;`DSH_CI_FAILOVER_WINDOWS` 下,原生 Windows 作业切到 `dsh-win-ci` 池。每个开关都是写者可管理的仓库状态而非一次合并,因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 与 `serial / windows (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。 `ci-master.yml` 只豁免一个事件不做取消(`${{ github.event_name != 'push' }}`),因此一次 master 推送不会取消上一次推送留下的、仍在运行的演练。每次演练以单门禁工作进程执行完整的未分片聚合流程,耗时长于 master 合并的间隔;在无条件取消下,演练会在得出结论前被后续运行取代,该通道无法产出供响应者查看的就绪证据。 @@ -18,9 +18,13 @@ Status: implemented 这个决定必须放在工作流级:取消作用于被取代的整个运行,作业级 `concurrency` 组并不能豁免其所属作业。采用否定式写法而非仅指名 `pull_request`,是有实质作用的:后者会连 `workflow_dispatch` 一起停止取消,而每次运行器基准测试会在 master 上的同一并发组内同时占用 12 台大规格运行器、最长 15 分钟,届时重复派发会排在演练之前,而不是替换掉已过时的测量。成本之所以可控,是因为 `ci-master.yml` 中一次 master 推送只承载 `wine-apt-cache` 和这两条演练;拉取请求作业位于独立的 `ci.yml`(不监听 `push`),而基准测试在 `ci-master.yml` 内受 `workflow_dispatch` 门控。`scripts/ci-workflow.spec.ts` 会锁定这个推送可达集合——按条件精确匹配,因为否定式事件判断会包含它所排除的事件名——使新的推送可达作业无法悄悄开始累积未取消的运行。 +### 发布演练共用 Linux 开关 + +`DSH_CI_FAILOVER_LINUX=selfhosted` 还会将符合条件的同仓库 PR 和 master 推送中的无凭据依赖布局作业与 dsh/vendor 两个打包作业路由到 `vm-backup`。[发布演练决策](2026-09-06-release-rehearsal-selfhosted.zh.md) 负责更严格的事件准入规则及保留托管的手动触发。这种耦合是有意的:持续设置变量来节省发布分钟,也会让符合条件的主 CI Linux 作业持续使用自托管。清除变量会让两类负载的后续运行返回各自的托管目标;发布操作始终保留托管。 + ### 自有池是什么 -`vm-backup`:一台 64 核虚拟机,6 个常驻 systemd 管理的运行器实例。其镜像必须预装 Playwright Chromium 的 Linux 系统软件包;CI 会下载锁文件选定的浏览器,但绝不在这台持久化共享主机上运行 `apt`。切换前先看 `serial / linux (self-hosted standby)` 最近一次运行:其聚合流程包含浏览器回放,因此绿色热备同时验证常规容量和这项浏览器先决条件。 +`vm-backup`:一台共享虚拟机,运行多个常驻 systemd 管理的运行器实例。注册实例共享 CPU、内存和磁盘;实例数量不代表独立机器数量。其镜像必须预装 Playwright Chromium 的 Linux 系统软件包;CI 会下载锁文件选定的浏览器,但绝不在这台持久化共享主机上运行 `apt`。切换前先看 `serial / linux (self-hosted standby)` 最近一次运行:其聚合流程包含浏览器回放,因此绿色热备同时验证常规容量和这项浏览器先决条件。 #### Windows 池 @@ -40,7 +44,7 @@ Status: implemented ## 切换期间的容量 -6 个常驻实例可承接正常 PR 流量(该池平时唯一的稳态负载是每次 master 推送一个串行热备作业,故障切换时几乎全池可用)。若仍出现排队,用组织级注册 token(组织 Settings → Actions → Runners → New runner)追加注册实例。复制现有 runner 目录时**必须排除身份文件**——`rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /`(通配同时排除 `.runner_migrated`/`.credentials_migrated`——GitHub 会在迁移过的运行器上写入这些文件,它们同样会触发 already-configured 拒绝)——再跑 `config.sh`(原样拷贝 `.runner`/`.credentials` 会使其以 "already configured" 拒绝),然后**启动监听器**:`sudo ./svc.sh install ubuntu && sudo ./svc.sh start`。仅注册不会上线;只有启动了服务的 runner 才会增加容量。每个约一分钟。 +Linux 开关启用期间,容量需覆盖 master 热备、主 CI 作业,以及每个符合条件的 PR 或 master 推送的三个发布演练作业。发布工作流不会因为新运行到来而取消正在执行的演练,因此不同引用的重叠运行会增加持续的构建、打包和安装负载。延长自托管运行前,检查当前 CPU、内存、磁盘和队列压力;同一虚拟机上新增注册只增加调度槽位,不增加机器资源。不能只依据热备负载推断空闲容量。主机资源允许增加注册实例时,使用组织级注册 token(组织 Settings → Actions → Runners → New runner)。复制现有 runner 目录时**必须排除身份文件**——`rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /`(通配同时排除 `.runner_migrated`/`.credentials_migrated`——GitHub 会在迁移过的运行器上写入这些文件,它们同样会触发 already-configured 拒绝)——再跑 `config.sh`(原样拷贝 `.runner`/`.credentials` 会使其以 "already configured" 拒绝),然后**启动监听器**:`sudo ./svc.sh install ubuntu && sudo ./svc.sh start`。仅注册不会上线;启动服务增加的是调度槽位,而非 CPU 或内存。 ### 切回 @@ -55,7 +59,7 @@ Status: implemented **通过合并一次工作流改动来切换池。** 否决,因为触发切换的故障状态恰恰是任何 PR 都无法合并的状态:必需检查正是失败的那些。仓库变量是写者可管理的状态,重跑即生效,无需合并。 -**让自托管池长期处于必需路径中。** 否决,因为这是拿托管池的可用性去换自有虚拟机的可用性,只是搬移了单点故障而非增加回退。这些变量让托管池保持主路径,自托管池作为一个经过验证、一步即可启用的热备;按平台拆分意味着一个平台的故障不会重定向另一个平台。 +**让自托管池长期处于必需路径中。** 否决,因为这是拿托管池的可用性去换自有虚拟机的可用性,只是搬移了单点故障而非增加回退。未设置变量时默认保留托管目标,开关提供由运维人员选择、可逆的自托管路径;按平台拆分意味着一个平台的故障不会重定向另一个平台。 ## 后果 diff --git a/.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.i18n.yaml b/.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.i18n.yaml new file mode 100644 index 0000000000..a8ce01be68 --- /dev/null +++ b/.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.md +2026-09-06-release-rehearsal-selfhosted.md: 415ae4716e9bc0ae9b165afc807f6f41e8a57e04 +2026-09-06-release-rehearsal-selfhosted.zh.md: a6fa441d01e66cea998d77a9b1be588053ac60a5 diff --git a/.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.md b/.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.md new file mode 100644 index 0000000000..415ae4716e --- /dev/null +++ b/.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.md @@ -0,0 +1,27 @@ +# Agent Note: trusted release rehearsals on persistent Linux runners + +Status: implemented + +English | [中文](2026-09-06-release-rehearsal-selfhosted.zh.md) + +## Problem + +Dependency-layout and release-pack rehearsals consume hosted Linux minutes without requiring npm or API credentials. Moving arbitrary pull-request code or credentialed publication onto a persistent shared host would weaken isolation; reusing a checkout without cleaning would also weaken the packed-payload proof. + +## Decision + +The two jobs in [release.yml](../../../../.github/workflows/release.yml) and the pack job in [release-vendor.yml](../../../../.github/workflows/release-vendor.yml) select the existing self-hosted Linux pool only with the writer-controlled `DSH_CI_FAILOVER_LINUX` repository variable set to `selfhosted`. The selector requires the canonical repository and a non-Dependabot actor, then admits only master pushes or same-repository, non-fork PRs whose author is not Dependabot. Manual dispatch always selects `ubuntu-24.04`, as do all other rejected contexts. The [failover runbook](2026-07-26-ci-failover-runbook.md) owns the platform switches and standby operation. Release rehearsals intentionally share the Linux switch with main CI: enabling or disabling it routes both workloads, not releases independently. Unset remains the hosted default; hosted-minute savings occur only while an operator selects `selfhosted`, whether for an outage or a longer-running cost choice. + +The runner labels are `[self-hosted, linux, x64, vm-backup]`. Runner registrations share one VM, not independent machine capacity. Each job uses its runner-private temporary volume for Node compile cache and node-gyp headers before pnpm setup, and a pnpm setup destination qualified by run, attempt, and job. `TMPDIR` also points to `runner.temp`, so temporary npm consumers stay outside the checkout but inside runner cleanup even when a killed process cannot execute `finally`. The persistent pnpm store stays outside checkout cleanup; only GitHub-hosted runners restore the remote store cache. Neither rehearsal workflow saves remote caches. + +Checkout explicitly cleans ignored and untracked output before immutable installation and the existing builds. Full tag history, pack concurrency, dependency checks, tarball verification, and artifact retention remain unchanged. The packed-install verifier creates a fresh consumer outside the checkout, installs tarballs with npm, removes inherited Node resolution hooks, and deletes the consumer in `finally`; a warm pnpm store cannot substitute workspace links or stale build output for a tarball payload. The [npm release decision](2026-08-10-npm-release-sequences.md) still owns release families and publication. Both manual publish workflows remain entirely hosted and gain no credentials or registry changes here. + +## Alternatives considered + +Always-hosted rehearsals avoid persistent-host risk but retain all hosted minutes. Always-self-hosted rehearsals remove the portable fallback. A scheduling job or reusable workflow adds another logical job and hides the three short setup sequences. Allowing manual dispatch on arbitrary refs gives a maintainer action broader persistent-host access than the explicit event trust rule. + +## Consequences + +Unsetting the variable or changing it away from `selfhosted` routes subsequent eligible jobs to hosted Ubuntu. This is an operator-selected fallback, not automatic runner-health detection or failover for already queued jobs. The shared VM can still contend with other trusted jobs, and repository writers remain responsible for code admitted to its persistent trust domain. No workflow provisions host packages or changes global host configuration. + +[scripts/tests/ci-release-selfhosted.spec.ts](../../../../scripts/tests/ci-release-selfhosted.spec.ts) evaluates the committed selectors with trusted events and negative controls for forks, Dependabot, other repositories, non-master pushes, dispatches, missing PR data, and disabled switches. It pins setup ordering, checkout cleanup, hosted-only remote cache access, publication isolation, and the retained commands. Real release-build and packed-install execution remains the PR CI verification owner; selector tests do not claim to reproduce those builds. diff --git a/.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.zh.md b/.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.zh.md new file mode 100644 index 0000000000..a6fa441d01 --- /dev/null +++ b/.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.zh.md @@ -0,0 +1,27 @@ +# Agent Note: 在持久化 Linux 运行器上执行受信任的发布演练 + +Status: implemented + +[English](2026-09-06-release-rehearsal-selfhosted.md) | 中文 + +## Problem + +依赖布局检查和发布打包演练消耗托管 Linux 分钟,但不需要 npm 或 API 凭据。将任意拉取请求代码或携带凭据的发布任务放到持久化共享主机会削弱隔离;复用未经清理的检出目录也会削弱打包载荷验证。 + +## Decision + +[release.yml](../../../../.github/workflows/release.yml) 的两个作业和 [release-vendor.yml](../../../../.github/workflows/release-vendor.yml) 的打包作业仅在写权限维护者控制的仓库变量 `DSH_CI_FAILOVER_LINUX` 设为 `selfhosted` 时选择现有 Linux 自托管池。选择器要求当前仓库为正式仓库且触发者不是 Dependabot,然后只接纳 master 推送,或作者不是 Dependabot 的同仓库、非 fork PR(Pull Request)。手动触发始终选择 `ubuntu-24.04`,其他不满足条件的上下文也一样。[故障切换手册](2026-07-26-ci-failover-runbook.zh.md) 负责按平台划分的开关与热备操作。发布演练有意与主 CI 共用 Linux 开关:启用或禁用会同时路由两类负载,不能独立切换发布演练。未设置时仍默认使用托管池;只有运维人员选择 `selfhosted` 期间才节省托管分钟,无论该选择用于故障恢复还是持续的成本控制。 + +运行器标签为 `[self-hosted, linux, x64, vm-backup]`。运行器注册共享一台虚拟机,不代表独立机器容量。每个作业在 pnpm 设置前将 Node 编译缓存与 node-gyp 头文件放在运行器私有临时卷上,pnpm 设置目标路径包含运行、重试次数和作业标识。`TMPDIR` 也指向 `runner.temp`,因此临时 npm 消费目录既在检出目录之外,也在运行器清理范围之内,即使进程被强杀而无法执行 `finally` 也一样。持久化 pnpm 存储位于检出清理范围之外;只有 GitHub 托管运行器恢复远端存储缓存。两个演练工作流都不保存远端缓存。 + +检出操作显式清理被忽略和未跟踪的输出,再执行锁定依赖安装与现有构建。完整标签历史、打包并发、依赖检查、压缩包验证和产物保留期均保持不变。打包安装验证器在检出目录外创建全新的消费目录,用 npm 安装压缩包,移除继承的 Node 解析钩子,并在 `finally` 中删除消费目录;预热 pnpm 存储无法用工作区链接或过期构建输出代替压缩包载荷。[npm 发布决策](2026-08-10-npm-release-sequences.zh.md) 仍负责发布族与发布操作。两个手动发布工作流全部保留在托管运行器上,本改动不增加凭据,也不改变注册表。 + +## Alternatives considered + +始终使用托管演练可以避免持久化主机风险,但会保留全部托管分钟。始终自托管则失去可移植回退。增加调度作业或可复用工作流会多出一个逻辑作业,并隐藏三个简短的设置序列。允许任意引用的手动触发,会让维护者操作获得比明确事件信任规则更广的持久化主机访问权限。 + +## Consequences + +取消变量或将其改为非 `selfhosted` 值,会将后续符合条件的作业路由到托管 Ubuntu。这是运维人员选择的回退,不会自动探测运行器健康,也不会切换已排队的作业。共享虚拟机仍可能与其他受信任作业竞争资源;仓库写权限维护者仍对进入持久化信任域的代码负责。工作流不安装主机系统包,也不修改全局主机配置。 + +[scripts/tests/ci-release-selfhosted.spec.ts](../../../../scripts/tests/ci-release-selfhosted.spec.ts) 使用受信任事件和 fork、Dependabot、其他仓库、非 master 推送、手动触发、缺失 PR 数据、禁用开关等负向对照求值已提交的选择器。测试固定设置顺序、检出清理、仅托管运行器访问远端缓存、发布隔离和保留命令。真实发布构建与打包安装执行仍由 PR CI 验证;选择器测试不声称重现这些构建。 diff --git a/.github/workflows/release-vendor.yml b/.github/workflows/release-vendor.yml index 8e1d531a03..ddf3c3f96c 100644 --- a/.github/workflows/release-vendor.yml +++ b/.github/workflows/release-vendor.yml @@ -29,21 +29,40 @@ env: jobs: pack: name: Pack npm tarballs - runs-on: ubuntu-24.04 + # Persistent runners accept only trusted, credential-free rehearsals. + runs-on: >- + ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' + && github.repository == 'deepseek-harness/deepseek-harness' + && github.actor != 'dependabot[bot]' + && ((github.event_name == 'push' && github.ref == 'refs/heads/master') + || (github.event_name == 'pull_request' + && github.event.pull_request.head.repo.full_name == github.repository + && github.event.pull_request.head.repo.fork == false + && github.event.pull_request.user.login != 'dependabot[bot]')) + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'ubuntu-24.04' }} steps: # Complete history: the release scripts read tags. - uses: actions/checkout@v6 with: fetch-depth: 0 persist-credentials: false + clean: true + + - name: Configure runner-private caches + run: | + echo "NODE_COMPILE_CACHE=${{ runner.temp }}/node-compile-cache" >> "$GITHUB_ENV" + echo "npm_config_devdir=${{ runner.temp }}/node-gyp" >> "$GITHUB_ENV" + echo "TMPDIR=${{ runner.temp }}" >> "$GITHUB_ENV" - uses: pnpm/action-setup@v4 with: - dest: ${{ runner.temp }}/setup-pnpm + dest: ${{ runner.temp }}/setup-pnpm-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }} - uses: actions/setup-node@v6 with: node-version: ${{ env.PRIMARY_NODE_VERSION }} + package-manager-cache: false - name: Configure pnpm store path id: pnpm-store @@ -54,6 +73,7 @@ jobs: echo "path=$store_path" >> "$GITHUB_OUTPUT" - uses: actions/cache/restore@v4 + if: runner.environment == 'github-hosted' with: path: ${{ steps.pnpm-store.outputs.path }} key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index ad6c64ae67..1e6662149e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -28,19 +28,38 @@ env: jobs: dependencies: name: Dependency layout - runs-on: ubuntu-24.04 + # Persistent runners accept only trusted, credential-free rehearsals. + runs-on: >- + ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' + && github.repository == 'deepseek-harness/deepseek-harness' + && github.actor != 'dependabot[bot]' + && ((github.event_name == 'push' && github.ref == 'refs/heads/master') + || (github.event_name == 'pull_request' + && github.event.pull_request.head.repo.full_name == github.repository + && github.event.pull_request.head.repo.fork == false + && github.event.pull_request.user.login != 'dependabot[bot]')) + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'ubuntu-24.04' }} steps: - uses: actions/checkout@v6 with: persist-credentials: false + clean: true + + - name: Configure runner-private caches + run: | + echo "NODE_COMPILE_CACHE=${{ runner.temp }}/node-compile-cache" >> "$GITHUB_ENV" + echo "npm_config_devdir=${{ runner.temp }}/node-gyp" >> "$GITHUB_ENV" + echo "TMPDIR=${{ runner.temp }}" >> "$GITHUB_ENV" - uses: pnpm/action-setup@v4 with: - dest: ${{ runner.temp }}/setup-pnpm + dest: ${{ runner.temp }}/setup-pnpm-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }} - uses: actions/setup-node@v6 with: node-version: ${{ env.PRIMARY_NODE_VERSION }} + package-manager-cache: false - name: Configure pnpm store path id: pnpm-store @@ -51,6 +70,7 @@ jobs: echo "path=$store_path" >> "$GITHUB_OUTPUT" - uses: actions/cache/restore@v4 + if: runner.environment == 'github-hosted' with: path: ${{ steps.pnpm-store.outputs.path }} key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} @@ -68,21 +88,40 @@ jobs: pack: name: Pack npm tarballs - runs-on: ubuntu-24.04 + # Persistent runners accept only trusted, credential-free rehearsals. + runs-on: >- + ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' + && github.repository == 'deepseek-harness/deepseek-harness' + && github.actor != 'dependabot[bot]' + && ((github.event_name == 'push' && github.ref == 'refs/heads/master') + || (github.event_name == 'pull_request' + && github.event.pull_request.head.repo.full_name == github.repository + && github.event.pull_request.head.repo.fork == false + && github.event.pull_request.user.login != 'dependabot[bot]')) + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'ubuntu-24.04' }} steps: # Complete history: the release scripts read tags. - uses: actions/checkout@v6 with: fetch-depth: 0 persist-credentials: false + clean: true + + - name: Configure runner-private caches + run: | + echo "NODE_COMPILE_CACHE=${{ runner.temp }}/node-compile-cache" >> "$GITHUB_ENV" + echo "npm_config_devdir=${{ runner.temp }}/node-gyp" >> "$GITHUB_ENV" + echo "TMPDIR=${{ runner.temp }}" >> "$GITHUB_ENV" - uses: pnpm/action-setup@v4 with: - dest: ${{ runner.temp }}/setup-pnpm + dest: ${{ runner.temp }}/setup-pnpm-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }} - uses: actions/setup-node@v6 with: node-version: ${{ env.PRIMARY_NODE_VERSION }} + package-manager-cache: false - name: Configure pnpm store path id: pnpm-store @@ -93,6 +132,7 @@ jobs: echo "path=$store_path" >> "$GITHUB_OUTPUT" - uses: actions/cache/restore@v4 + if: runner.environment == 'github-hosted' with: path: ${{ steps.pnpm-store.outputs.path }} key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} diff --git a/docs/development.i18n.yaml b/docs/development.i18n.yaml index 14b86dfa27..6c7316250a 100644 --- a/docs/development.i18n.yaml +++ b/docs/development.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/development.md -development.md: 0f9e76f30e6fc65f242b7f4ce6e1bef79e3451b9 -development.zh.md: 6e98acf9d9cda5271b98c3afbf4b915673fde4d0 +development.md: a57c99d606a73cb938f339e080ab0ba05913902a +development.zh.md: 5439fec59cb7c245d655393690bf6c848cdf20fa diff --git a/docs/development.md b/docs/development.md index 0f9e76f30e..a57c99d606 100644 --- a/docs/development.md +++ b/docs/development.md @@ -122,6 +122,8 @@ Contributors can opt into the comprehensive local gate set with `pnpm run check: The keyless [CI workflow](../.github/workflows/ci.yml) groups independent gates into broad lanes and runs a smaller compatibility signal across supported Node versions. Artifact consumers wait for one build within their lane. The separate real-API workflow runs `pnpm run test:e2e` with its configured worker bound. See [scripts/run-gates.ts](../scripts/run-gates.ts) and the workflow files for the current gate and job inventory. +The credential-free dsh dependency-layout and dsh/vendor pack rehearsals use the existing Linux self-hosted pool only when `DSH_CI_FAILOVER_LINUX=selfhosted` and the event is a trusted master push or same-repository, non-fork, non-Dependabot pull request. All other cases, including manual dispatch, use `ubuntu-24.04`; manual publication stays hosted. See the [release rehearsal runner decision](../.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.md) for persistent-store isolation and fallback limits. + ### Daily commands The root [contributor instructions](../AGENTS.md#commands) summarize common commands, while [`package.json`](../package.json) and [scripts/run-gates.ts](../scripts/run-gates.ts) own the current script and gate inventories. Select the smallest checks that cover the changed surface. Documentation changes use `pnpm run doc-sync`; package-public behavior changes also update the owning README or JSDoc, and built-artifact checks require `pnpm run build` first. diff --git a/docs/development.zh.md b/docs/development.zh.md index 6e98acf9d9..5439fec59c 100644 --- a/docs/development.zh.md +++ b/docs/development.zh.md @@ -126,6 +126,8 @@ vendor manifest 守卫检查 `vendor/*/src` 下的改动是否连同对应的 `v keyless [CI 工作流](../.github/workflows/ci.yml) 将独立门禁分组到若干宽粒度 lane,并在受支持的 Node 版本上运行一组较小的兼容性检查。产物消费方在各自 lane 内等待一次 build。单独的真实 API 工作流按其配置的 worker 上限运行 `pnpm run test:e2e`。当前门禁和 job 清单以 [scripts/run-gates.ts](../scripts/run-gates.ts) 和工作流文件为准。 +不带凭据的 dsh 依赖布局检查与 dsh/vendor 打包演练仅在 `DSH_CI_FAILOVER_LINUX=selfhosted`,且事件为受信任的 master 推送或同仓库、非 fork、非 Dependabot 拉取请求时使用现有 Linux 自托管池。其余情况(包括手动触发)均使用 `ubuntu-24.04`;手动发布仍使用托管运行器。持久化存储隔离与回退限制见[发布演练运行器决策](../.agents/notes/implemented/process/2026-09-06-release-rehearsal-selfhosted.zh.md)。 + ### 日常命令 根目录的[贡献者说明](../AGENTS.md#commands)概述常用命令,[`package.json`](../package.json) 与 [scripts/run-gates.ts](../scripts/run-gates.ts) 则负责当前脚本和门禁清单。请选择覆盖变更表面的最小检查集。文档变更使用 `pnpm run doc-sync`;包公开行为变更还需更新所属 README 或 JSDoc,而基于构建产物的检查需要先运行 `pnpm run build`。 diff --git a/scripts/tests/ci-release-selfhosted.spec.ts b/scripts/tests/ci-release-selfhosted.spec.ts new file mode 100644 index 0000000000..99ce317248 --- /dev/null +++ b/scripts/tests/ci-release-selfhosted.spec.ts @@ -0,0 +1,150 @@ +/** Release rehearsal routing and persistent-runner isolation, without executing release builds. */ +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { runInNewContext } from 'node:vm' +import { load } from 'js-yaml' +import { describe, expect, it } from 'vitest' + +const root = resolve(import.meta.dirname, '../..') +const repository = 'deepseek-harness/deepseek-harness' +const selfhosted = ['self-hosted', 'linux', 'x64', 'vm-backup'] +const hosted = 'ubuntu-24.04' + +interface Step { + name?: string + uses?: string + run?: string + if?: string + with?: Record +} +interface Workflow { + on: Record + permissions: Record + concurrency?: Record + jobs: Record +} + +function workflow(file: string): Workflow { + return load(readFileSync(resolve(root, '.github/workflows', file), 'utf8')) as Workflow +} + +// This canonical-case corpus has matching Actions/JavaScript comparison results. +// This is not an Actions interpreter: string case-folding and general coercion differ. +// Missing context properties use the Actions empty-string value. +function evaluate(expression: string, context: Record): unknown { + const source = expression.trim().replace(/^\$\{\{|\}\}$/g, '') + .replace(/\b(?:github|vars|runner)(?:\.[a-zA-Z_][a-zA-Z_0-9]*)+/g, + key => JSON.stringify(context[key] ?? '')) + return runInNewContext(source, { fromJSON: JSON.parse }, { timeout: 1000 }) as unknown +} + +const trustedPr = { + 'vars.DSH_CI_FAILOVER_LINUX': 'selfhosted', + 'github.repository': repository, + 'github.actor': 'maintainer', + 'github.event_name': 'pull_request', + 'github.ref': 'refs/pull/42/merge', + 'github.event.pull_request.head.repo.full_name': repository, + 'github.event.pull_request.head.repo.fork': false, + 'github.event.pull_request.user.login': 'contributor', +} +const trustedPush = { + 'vars.DSH_CI_FAILOVER_LINUX': 'selfhosted', + 'github.repository': repository, + 'github.actor': 'maintainer', + 'github.event_name': 'push', + 'github.ref': 'refs/heads/master', +} +const fallbackCases: Array<[string, Record]> = [ + ['unset switch', { ...trustedPr, 'vars.DSH_CI_FAILOVER_LINUX': '' }], + ['hosted switch', { ...trustedPr, 'vars.DSH_CI_FAILOVER_LINUX': 'hosted' }], + ['unknown switch', { ...trustedPr, 'vars.DSH_CI_FAILOVER_LINUX': 'true' }], + ['fork PR', { ...trustedPr, 'github.event.pull_request.head.repo.full_name': 'outsider/fork', 'github.event.pull_request.head.repo.fork': true }], + ['different head repository', { ...trustedPr, 'github.event.pull_request.head.repo.full_name': 'outsider/repo' }], + ['fork flag', { ...trustedPr, 'github.event.pull_request.head.repo.fork': true }], + ['Dependabot author rerun by maintainer', { ...trustedPr, 'github.event.pull_request.user.login': 'dependabot[bot]' }], + ['Dependabot PR actor', { ...trustedPr, 'github.actor': 'dependabot[bot]' }], + ['Dependabot push actor', { ...trustedPush, 'github.actor': 'dependabot[bot]' }], + ['non-master push', { ...trustedPush, 'github.ref': 'refs/heads/topic' }], + ['tag push', { ...trustedPush, 'github.ref': 'refs/tags/dsh-v1.0.0' }], + ['push in another repository', { ...trustedPush, 'github.repository': 'outsider/fork' }], + ['dispatch on master', { ...trustedPush, 'github.event_name': 'workflow_dispatch' }], + ['dispatch on topic', { ...trustedPush, 'github.event_name': 'workflow_dispatch', 'github.ref': 'refs/heads/topic' }], + ['dispatch on tag', { ...trustedPush, 'github.event_name': 'workflow_dispatch', 'github.ref': 'refs/tags/dsh-v1.0.0' }], + ['pull_request_target', { ...trustedPr, 'github.event_name': 'pull_request_target' }], + ['missing PR payload', { ...trustedPush, 'github.event_name': 'pull_request' }], +] + +for (const [file, jobIds] of [['release.yml', ['dependencies', 'pack']], ['release-vendor.yml', ['pack']]] as const) { + describe(file, () => { + const release = workflow(file) + it('preserves the logical jobs, rehearsal events and read-only permission', () => { + expect(Object.keys(release.jobs)).toEqual(jobIds) + expect(release.on).toEqual({ pull_request: null, push: { branches: ['master'] }, workflow_dispatch: null }) + expect(release.permissions).toEqual({ contents: 'read' }) + expect(release.concurrency).toEqual({ group: '${{ github.workflow }}-${{ github.ref }}', 'cancel-in-progress': false }) + }) + for (const jobId of jobIds) { + describe(jobId, () => { + const job = release.jobs[jobId]! + it('routes trusted PRs and master pushes onto the existing Linux pool', () => { + expect(evaluate(job['runs-on'], trustedPr)).toEqual(selfhosted) + expect(evaluate(job['runs-on'], trustedPush)).toEqual(selfhosted) + expect(evaluate(job['runs-on'], { ...trustedPush, 'vars.DSH_CI_FAILOVER_LINUX': '' })).toBe(hosted) + }) + it.each(fallbackCases)('keeps %s hosted', (_name, context) => { + expect(evaluate(job['runs-on'], context)).toBe(hosted) + }) + it('cleans stale checkout output and isolates setup before any pnpm invocation', () => { + expect(job.steps[0]).toMatchObject({ uses: 'actions/checkout@v6', with: { clean: true, 'persist-credentials': false } }) + const cacheIndex = job.steps.findIndex(step => step.run?.includes('NODE_COMPILE_CACHE=')) + const pnpmIndex = job.steps.findIndex(step => step.uses?.startsWith('pnpm/') || /\bpnpm\b/.test(step.run ?? '')) + expect(cacheIndex).toBeGreaterThan(0) + expect(cacheIndex).toBeLessThan(pnpmIndex) + expect(job.steps[cacheIndex]?.run).toContain('echo "NODE_COMPILE_CACHE=${{ runner.temp }}/node-compile-cache" >> "$GITHUB_ENV"') + expect(job.steps[cacheIndex]?.run).toContain('echo "npm_config_devdir=${{ runner.temp }}/node-gyp" >> "$GITHUB_ENV"') + expect(job.steps[cacheIndex]?.run).toContain('echo "TMPDIR=${{ runner.temp }}" >> "$GITHUB_ENV"') + expect(job.steps.find(step => step.uses === 'pnpm/action-setup@v4')?.with?.dest) + .toBe('${{ runner.temp }}/setup-pnpm-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}') + expect(job.steps.find(step => step.name === 'Install (immutable)')?.run).toBe('pnpm install --frozen-lockfile') + }) + it('uses the persistent store without remote cache reads or writes on self-hosted', () => { + expect(job.steps.find(step => step.name === 'Configure pnpm store path')?.run).toContain('store_root="$HOME/.local/share/pnpm/store"') + const caches = job.steps.filter(step => step.uses?.startsWith('actions/cache')) + expect(caches.map(step => step.uses)).toEqual(['actions/cache/restore@v4']) + for (const step of caches) { + expect(evaluate(step.if!, { 'runner.environment': 'self-hosted' })).toBe(false) + expect(evaluate(step.if!, { 'runner.environment': 'github-hosted' })).toBe(true) + } + const nodeSetup = job.steps.find(step => step.uses === 'actions/setup-node@v6') + expect(nodeSetup?.with?.cache).toBeUndefined() + expect(nodeSetup?.with?.['package-manager-cache']).toBe(false) + }) + it('retains the dependency and pack verification commands', () => { + const commands = job.steps.flatMap(step => step.run === undefined ? [] : [step.run]) + if (jobId === 'dependencies') { + expect(commands).toContain('pnpm run verify-package-dependencies') + expect(commands).toContain('pnpm run verify-npm-install-layout') + } else { + const family = file === 'release.yml' ? 'dsh' : 'vendor' + const output = family === 'dsh' ? 'dist/npm' : 'dist/npm-vendor' + expect(job.steps[0]?.with?.['fetch-depth']).toBe(0) + expect(commands).toContain('pnpm run release:verify --family ' + family) + expect(commands).toContain('pnpm run ' + (family === 'dsh' ? 'build:official' : 'build:lib:host')) + expect(commands).toContain('pnpm run release:pack --family ' + family + ' --out ' + output + ' --concurrency 8') + expect(commands).toContain('pnpm run release:verify-packed-install --family ' + family + ' --from ' + output + + (family === 'dsh' ? ' --from dist/npm-vendor --from dist/npm-landlock' : '')) + expect(job.steps.at(-1)).toMatchObject({ uses: 'actions/upload-artifact@v4', with: { path: output + '/*', 'retention-days': 7 } }) + } + expect(JSON.stringify(job)).not.toMatch(/secrets\.|release:publish|npm-publish/) + }) + }) + } + }) +} + +it.each(['release-publish.yml', 'release-vendor-publish.yml'])('keeps %s manual and entirely hosted', (file) => { + const publish = workflow(file) + expect(publish.on).toEqual({ workflow_dispatch: null }) + for (const job of Object.values(publish.jobs)) expect(job['runs-on']).toBe(hosted) +})