perf(test): widen transform corpus sharding

2026-08-29 04:26:38 +00:00 · 2026-08-22 20:10:55 +08:00
parent d27a5f2967
commit d39b6c638b
7 changed files with 16 additions and 16 deletions
@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-31-coverage-exempt-heavy-suites.md
-2026-07-31-coverage-exempt-heavy-suites.md: 4d080eea4b1e4afde998bf2397dce7a9f30e484e
-2026-07-31-coverage-exempt-heavy-suites.zh.md: 9cd759d01d37245f9b91cadbadf8b929bc797225
+2026-07-31-coverage-exempt-heavy-suites.md: d0b9e92b1f122211fa1baaa9514190fc8bd10cc7
+2026-07-31-coverage-exempt-heavy-suites.zh.md: bcb9e11e9283146a98780dd173579c3f261031e3
@@ -23,7 +23,7 @@ Linux coverage CI and native Windows CI use [in-job partitioned coverage](2026-0

 `scripts/coverage-exempt.ts` is the single roster point, holding the membership contract and the filter/exclude pairs so the two sides cannot drift.

-`transform-corpus.spec.ts` discovers the complete built-bundle set once, assigns every path to exactly one of four Node-loader children, and asserts the shard union before launch. The test-support pair and the ACL/win32-process pair retain their original order in one shard because their pinned Vitest-state and Koffi exemptions depend on preceding module state.
+`transform-corpus.spec.ts` discovers the complete built-bundle set once, assigns every path to exactly one of eight Node-loader children, and asserts the shard union before launch. The test-support pair and the ACL/win32-process pair retain their original order in one shard because their pinned Vitest-state and Koffi exemptions depend on preceding module state.

 ### The roster, reconciled entry by entry

@@ -54,7 +54,7 @@ Coverage-result invariance therefore does not rest on humans maintaining the ros
 - **CLI `--exclude` to drop the exempt suites from the instrumented gate.** Proven ineffective: vitest 4's `cliExclude` does not participate in per-project include resolution, so under a multi-project config the exempt suites stayed selected; the env + config route replaced it.
 - **Lowering worker counts or raising gate concurrency.** Measured ineffective during the incident: the lane's wall clock was pinned by the longest tail files (aggregate/wall ≈ 4× effective parallelism), and the concurrency knobs moved nothing in either direction.
 - **Cross-runner sharding (`--shard` + blob merge).** Rejected because a matrix, artifact pipeline, and merge job would add a second workflow topology. The selected [in-job partitioning](2026-08-18-in-job-partitioned-coverage.md) uses Vitest shards only as local single-worker processes inside the existing job.
- **Keep the transform corpus in one Node process.** Rejected because its serial loader becomes the Windows heavy gate's longest tail under host contention. Four local children retain the same file set, per-file oracle, loader-sensitive affinities, and one blocking Vitest verdict.
+- **Keep the transform corpus in one Node process.** Rejected because its serial loader becomes the Windows heavy gate's longest tail under host contention. Eight local children retain the same file set, per-file oracle, loader-sensitive affinities, and one blocking Vitest verdict.
 - **Deleting or skipping the heavy suites.** Rejected: they are the sole correctness evidence for the typert generator and the scripts tooling; running them uninstrumented in parallel preserves the full signal.

 ## Verification
@@ -63,13 +63,13 @@ Measured on CI (16-core runner): the gate segment went from 424 seconds to the t

 The Web Worker corpus entry is pinned by an eight-partition aggregate that runs all 15,250 tests and reports 100% for 45,959 statements, 28,116 branches, 9,781 functions, and 40,550 lines. A focused instrumented corpus run records no package source from its child process; the paired list check proves the spec is absent from the instrumented inventory and present in the uninstrumented inventory.

-The four-child corpus run checks the same 239 native Windows bundles with 234 exact export matches, four pinned loader exemptions, one sentinel refusal, and no drift. The ARM64 VM measures 25.06 seconds for the sharded Vitest path versus 29.59 seconds for the unsharded checker; the complete x64 job remains the contended-host timing proof.
+The eight-child corpus run checks the same 239 native Windows bundles with 234 exact export matches, four pinned loader exemptions, one sentinel refusal, and no drift. The ARM64 VM measures 25.44 seconds for the sharded Vitest path versus 29.59 seconds for the unsharded checker; the complete x64 job remains the contended-host timing proof.

 ## Consequences

 - The exempt suites execute without adding instrumentation cost to the thresholded gate; partitioned wall-clock measurements belong to the [in-job partitioning decision](2026-08-18-in-job-partitioned-coverage.md).
 - Native Windows schedules the exempt suites after instrumented coverage and overlaps them with observational checks; Linux retains the parallel coverage split.
- The corpus suite uses four child Node loaders but emits one blocking test result; its affinity roster is part of the exemption oracle and must move with affected bundles.
+- The corpus suite uses eight child Node loaders but emits one blocking test result; its affinity roster is part of the exemption oracle and must move with affected bundles.
 - `DSH_GATE_CONCURRENCY` has two schedulable gates in this lane again, so the aggregate scheduler is no longer a pass-through.
 - Adding a heavy suite to the roster requires the membership audit above; a wrong entry fails the instrumented gate loudly rather than eroding coverage silently.
 - The exempt suites no longer appear in the coverage report's file list of contributors; their correctness signal lives solely in the uninstrumented gate's pass/fail.
@@ -23,7 +23,7 @@ Linux 覆盖率 CI 与原生 Windows CI 在插桩门禁内部使用 [job 内分

 `scripts/coverage-exempt.ts` 是唯一名单点，集中持有成员资格约定与 filter/exclude 配对，防止两侧漂移。

-`transform-corpus.spec.ts` 只发现一次完整的已构建 bundle 集合，把每条路径恰好分配给四个 Node loader 子进程之一，并在启动前断言分片并集。test-support 对与 ACL/win32-process 对在同一分片内保留原始顺序，因为它们固定的 Vitest 状态与 Koffi 豁免依赖前序模块状态。
+`transform-corpus.spec.ts` 只发现一次完整的已构建 bundle 集合，把每条路径恰好分配给八个 Node loader 子进程之一，并在启动前断言分片并集。test-support 对与 ACL/win32-process 对在同一分片内保留原始顺序，因为它们固定的 Vitest 状态与 Koffi 豁免依赖前序模块状态。

 ### 豁免名单与逐项对账

@@ -54,7 +54,7 @@ per-file 100% 阈值本身就是豁免名单的守卫，名单错误无法静默
 - **CLI `--exclude` 从插桩 gate 剔除豁免套件。** 实证无效：vitest 4 的 `cliExclude` 不参与 per-project include 解析，多 project 配置下豁免套件仍被选中，故改走 env + config。
 - **降低 worker 数或提高 gate 并发。** 事故期间实测无效：lane 墙钟被尾部最长文件钉死（聚合/墙钟 ≈ 4× 有效并行），并发旋钮两个方向都动不了尾巴。
 - **跨 runner 分片（`--shard` + blob 合并）。** 不予采用，因为 matrix、产物流水线和合并 job 会引入第二套工作流拓扑。所选的 [job 内分区](2026-08-18-in-job-partitioned-coverage.zh.md)只把 Vitest shard 用作既有 job 内的本地单 worker 进程。
- **让转换语料库保留在一个 Node 进程中。** 不予采用，因为串行 loader 在宿主争用下成为 Windows 重型门禁的最长尾部。四个本地子进程保留相同文件集、逐文件判定器、对 loader 敏感的亲和顺序，以及一个阻断性 Vitest 判定。
+- **让转换语料库保留在一个 Node 进程中。** 不予采用，因为串行 loader 在宿主争用下成为 Windows 重型门禁的最长尾部。八个本地子进程保留相同文件集、逐文件判定器、对 loader 敏感的亲和顺序，以及一个阻断性 Vitest 判定。
 - **直接删除或跳过重型套件。** 拒绝：它们是 typert generator 与 scripts 工具的唯一正确性证据，无插桩并排执行保住全部信号。

 ## Verification
@@ -63,13 +63,13 @@ CI 实测（16 核 runner）：拆分前 gate 段 424 秒，拆分后两 gate

 Web Worker 语料库条目由八分区聚合固定：它执行全部 15,250 个测试，并对 45,959 条语句、28,116 个分支、9,781 个函数和 40,550 行报告 100%。聚焦的插桩语料库运行不会记录其子进程中的包源码；配对名单检查证明该 spec 不在插桩清单中，但存在于无插桩清单中。

-四子进程语料库运行检查相同的 239 个原生 Windows bundle，得到 234 个精确 export 匹配、四个固定 loader 豁免、一次 sentinel 拒绝和零漂移。ARM64 虚拟机上，分片 Vitest 路径耗时 25.06 秒，未分片检查器耗时 29.59 秒；完整 x64 job 仍负责证明宿主争用下的耗时。
+八子进程语料库运行检查相同的 239 个原生 Windows bundle，得到 234 个精确 export 匹配、四个固定 loader 豁免、一次 sentinel 拒绝和零漂移。ARM64 虚拟机上，分片 Vitest 路径耗时 25.44 秒，未分片检查器耗时 29.59 秒；完整 x64 job 仍负责证明宿主争用下的耗时。

 ## Consequences

 - 豁免套件在执行时不会向阈值门禁叠加插桩开销；分区墙钟数据由 [job 内分区决策](2026-08-18-in-job-partitioned-coverage.zh.md)负责记录。
 - 原生 Windows 在插桩覆盖率后调度豁免套件，并让它们与观测性检查重叠；Linux 保留并行覆盖率拆分。
- 语料库套件使用四个 Node loader 子进程，但只产生一个阻断性测试结果；其亲和名单属于豁免判定器，受影响 bundle 移动时必须同步更新。
+- 语料库套件使用八个 Node loader 子进程，但只产生一个阻断性测试结果；其亲和名单属于豁免判定器，受影响 bundle 移动时必须同步更新。
 - `DSH_GATE_CONCURRENCY` 在本 lane 重新拥有两个可调度对象，聚合调度器不再是直通。
 - 向名单新增重型套件必须完成上述成员资格对账；错误条目会让插桩 gate 大声失败，而不是静默侵蚀覆盖率。
 - 豁免套件不再出现在覆盖率报告的贡献文件列表中；其正确性信号完全由无插桩 gate 的红绿承载。
@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-08-18-in-job-partitioned-coverage.md
-2026-08-18-in-job-partitioned-coverage.md: 761dcb147b79b47962ad68f9d16f6fb6ad4985dd
-2026-08-18-in-job-partitioned-coverage.zh.md: 8a3cd2077b5f125636f2f064620599646ae320fc
+2026-08-18-in-job-partitioned-coverage.md: cc312b53937dea8eec8207c95c7354be64375e15
+2026-08-18-in-job-partitioned-coverage.zh.md: 88892e485d89fec9d731409fa6d1f6d7f7c15d93
@@ -32,7 +32,7 @@ A normal failed test still emits a blob through `--coverage.reportOnFailure`, al

 Completed native Windows comparisons measured two partitions near 405 seconds and sixteen partitions at 112.66–122.01 seconds, but the sixteen-way schedule could put more than twenty active execution units beside build and exempt coverage on a 16-core runner. Eight partitions keep separate-process isolation while accepting a longer feedback path for a materially lower peak. Two Linux samples measured the conservative two-partition configuration at 276.68 and 282.27 seconds; that configuration was stable but halved the ordinary path's four instrumented workers. Four partitions restore that fan-out, for six total coverage execution units on the 16-core hosted runner and at most 36 across the failover VM's six runner instances. These values come from completed runs or fixed capacity bounds; an unfinished run crossing an arbitrary elapsed-time mark is not evidence for increasing concurrency.

-The native ARM64 VM runs the full transform corpus in 29.59 seconds without coverage partitions and in 25.06 seconds through the four-child Vitest path. A concurrent self-hosted x64 job stretched the former serial test to 279.13 seconds while one instrumented partition reached 442.45 seconds, which is why the Windows graph separates the partition and exempt phases instead of increasing partition count.
+The native ARM64 VM runs the full transform corpus in 29.59 seconds without coverage partitions and in 25.44 seconds through the eight-child Vitest path. A concurrent self-hosted x64 job stretched the former serial test to 279.13 seconds while one instrumented partition reached 442.45 seconds, which is why the Windows graph separates the partition and exempt phases instead of increasing partition count.

 ## Alternatives considered

@@ -32,7 +32,7 @@ Status: implemented

 已完成的原生 Windows 对比中，双分区耗时约 405 秒，16 分区耗时 112.66–122.01 秒，但 16 路调度与构建、豁免覆盖率并行时，会在 16 核运行器上形成超过 20 个活动执行单元。8 个分区继续保留独立进程隔离，同时接受更长的反馈路径，以显著降低峰值。两个 Linux 样本中，保守的双分区配置耗时 276.68 秒和 282.27 秒；该配置运行稳定，却把普通路径原有的 4 个插桩 worker 减半。4 个分区恢复这份并发，使 16 核托管 runner 上的覆盖率执行单元总数为 6，故障切换虚拟机的 6 个 runner 实例最多合计 36 个执行单元。这些数值来自完整运行或固定容量上限；运行尚未结束时跨过任意耗时刻度，不构成增加并发的证据。

-原生 ARM64 虚拟机在没有覆盖率分区时用 29.59 秒运行完整转换语料库，通过四子进程 Vitest 路径时用 25.06 秒。一个并发运行的自托管 x64 job 把此前的串行测试拉长到 279.13 秒，同时一个插桩分区达到 442.45 秒；因此 Windows 门禁图分离分区阶段与豁免阶段，而不是增加分区数量。
+原生 ARM64 虚拟机在没有覆盖率分区时用 29.59 秒运行完整转换语料库，通过八子进程 Vitest 路径时用 25.44 秒。一个并发运行的自托管 x64 job 把此前的串行测试拉长到 279.13 秒，同时一个插桩分区达到 442.45 秒；因此 Windows 门禁图分离分区阶段与豁免阶段，而不是增加分区数量。

 ## 曾考虑的替代方案

@@ -11,7 +11,7 @@
 * exemptions as stale. The gate's own note applies to itself: a gate whose
 * verdict depends on how it was launched is not a gate.
 *
- * Four Node-loader processes divide the discovered files, and the union check
+ * Eight Node-loader processes divide the discovered files, and the union check
 * proves that each bundle appears once. The two test-support bundles and the
 * ACL/win32-process pair stay in one ordered shard because their pinned loader
 * exemptions depend on the same preceding module state as the unsharded
@@ -26,7 +26,7 @@ import { expect, test } from 'vitest'

 const runner = fileURLToPath(new URL('./transform-corpus-check.ts', import.meta.url))
 const repositoryRoot = fileURLToPath(new URL('../../../../../', import.meta.url))
-const corpusShards = 4
+const corpusShards = 8
 const shardAffinity = new Set([
  'packages/test-support/acp-snapshot/lib/index.js',
  'packages/test-support/client-runtime/lib/index.js',