docs(testing): add platform-semantics and lane-budget rules to the skill

Two failure classes the repository paid for are not covered by the isolation,
synchronization, and teardown rules already in the skill.

A value the operating system owns is not guaranteed to return as written. A
test may write one back only where the assertion tolerates that write-back
failing; where the assertion depends on it, the expected value comes from a
fresh read. NTFS truncating a fractional-millisecond mtime and Windows folding
environment variable name case are the two instances seen so far.

A describe or case timeout overrides the runner's --testTimeout rather than
yielding to it, so a value below the lane budget lowers what CI granted, while
the same literal reads as a widening where the host default is smaller. The
hook budget travels with the test budget, and a case asserting a timeout keeps
its outer wait far larger than the timeout under test. Restoring a granted
budget, or sizing a bounded retry to measured contention, is named as distinct
from the masking fixes the skill rejects.

The diagnosis reference gains the platform differences under its platform
class, a classification path for self-hosted pools that expose no host metrics,
and the stopping rule for a signature no available host can reproduce.
This commit is contained in:
Yichen Jiang
2026-08-29 13:54:59 +08:00
parent 94c714d813
commit 596a13d1cb
5 changed files with 32 additions and 4 deletions
@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-08-28-ci-test-reliability-skill.md
2026-08-28-ci-test-reliability-skill.md: 5edf2aa769047cb5d4052a1338f2cc3886d9e0f9
2026-08-28-ci-test-reliability-skill.zh.md: fd1a6d584f3c4686761f8c94ba99ee0d1c9b0233
2026-08-28-ci-test-reliability-skill.md: 1c8e0389dfa6e3eb3a6e04e994e6400d19a673ac
2026-08-28-ci-test-reliability-skill.zh.md: ef36eab497bee874cd117488ddf4895c7edd0ed1
@@ -16,6 +16,8 @@ The testing policy owns test tiers, defensive patterns own runtime lifecycle rul
The skill requires agents to model concurrency beyond one Vitest process, allocate live resources atomically, separate stable fixture identities from ephemeral transport addresses, synchronize on observable state, restore global mutations exactly, and await teardown to quiescence. Regression evidence matches the owned risk: negative controls for guards, deterministic barriers for races, concurrent independent processes for host-resource isolation, and external observations instead of component self-reports.
Two rules cover the failures the repository has actually paid for. A value the operating system owns is not guaranteed to return as written, so a test may write one back only where the assertion tolerates that write-back failing; where the assertion depends on it, the expected value comes from a fresh read. And a suite timeout overrides the runner flag rather than yielding to it, so a suite bound by process creation takes the lane budget, raises the hook budget with it, and keeps an outer wait far larger than any timeout under test. Restoring a granted budget or sizing a bounded retry to measured contention is therefore not a masking fix.
The diagnosis-only workflow lives in a separate reference so ordinary authoring does not load Actions triage procedure. It compares passing and failing evidence before classifying host collisions, incomplete lifecycle, global contamination, load-sensitive synchronization, platform or entry-path failures, product races, provider transience, or runner infrastructure.
[dsh-pre-push-checks](../../../skills/dsh-pre-push-checks/SKILL.md) conditionally consults the reliability skill before selecting commands, while [dsh-code-review](../../../skills/dsh-code-review/SKILL.md) applies it when reviewing risky tests. Command selection and general PR review remain with those existing skills.
@@ -16,6 +16,8 @@ DeepSeek Harness 会在并发的 Vitest 文件、worker 进程、仓库 gate 与
该 Skill 要求 agent 建模单个 Vitest 进程之外的并发,原子分配实时资源,把稳定 fixture 标识与临时传输地址分开,按可观察状态同步,精确恢复全局变更,并等待 teardown 达到静止状态。回归证据与所持有的风险匹配:guard 使用负向控制,竞态使用确定性 barrier,宿主机资源隔离使用并发独立进程,并以外部观察代替组件自述。
另有两条规则覆盖仓库已经付出过代价的失败。操作系统拥有的值不保证按写入的样子返回,因此只有在断言容忍写回失败时,测试才可以把它写回去;断言依赖写回成功时,期望值取自重新读取。以及套件级 timeout 覆盖而不是让位于 runner 的 flag,因此受进程创建约束的套件取 lane 预算、连同 hook 预算一起抬高,并让外层等待远大于任何被测超时。据此,恢复已被授予的预算、或按实测争抢标定一个有界重试,都不属于掩盖式修复。
仅用于诊断的流程放在单独 reference 中,因此普通编写任务不会加载 Actions 分诊步骤。它会先比较成功与失败证据,再对宿主机冲突、未完成生命周期、全局状态污染、负载敏感同步、平台或入口路径失败、产品竞态、provider 瞬时故障或 runner 基础设施进行分类。
[dsh-pre-push-checks](../../../skills/dsh-pre-push-checks/SKILL.md) 在选择命令前按条件引用可靠性 Skill,[dsh-code-review](../../../skills/dsh-code-review/SKILL.md) 则在 review 高风险测试时应用它。命令选择与通用 PR review 仍由这些现有 Skill 负责。
@@ -53,6 +53,25 @@ Prefer an injected dependency or instance-local adapter. When mutation is requir
- keep an `afterEach` fallback when failure before the local `finally` is plausible;
- intercept the narrowest exact request or call that the fixture owns.
## Respect platform-owned semantics
CI runs the same suite on Windows and on POSIX hosts, and a value the operating system owns does not always come back the way a test wrote it.
- Writing a value back is safe only when the assertion tolerates the write-back failing. Restoring a file's `mtime` to prove that a fingerprint invalidates anyway holds everywhere; restoring it to prove that a record stays valid assumes a lossless round trip, which NTFS's 100-nanosecond ticks do not give a fractional millisecond. When the assertion depends on the restoration, take the expected value from a fresh read rather than from the remembered one.
- Windows matches environment variable names case-insensitively, so a fixture seeding `http_proxy` and `HTTP_PROXY` as separate keys holds one entry there.
- Windows releases file handles asynchronously, so a rename or removal that completes at once on a POSIX host needs a bounded retry sized to the observed contention.
- Windows has no POSIX permission or signal semantics. A case that depends on them takes an explicit platform skip naming the reason, rather than an assertion weakened everywhere.
Prefer an observation that holds on every platform. When a case genuinely cannot, exclude it on that platform explicitly.
## Budget timeouts against the lane
A `describe` or case timeout overrides the runner's `--testTimeout` instead of yielding to it, so a value below the lane's budget lowers what CI already granted — and the same literal reads as a widening on a host whose default is smaller. A suite bound by process creation takes the lane budget; a tighter value carries the reason it is tighter.
Raise the hook budget with the test budget. Setup and teardown pay the same contention, so lifting only the case budget moves a contended failure into `afterEach`.
Where a timeout is the subject, keep the outer wait far larger than the timeout under test. A case proving that a 20 ms deadline fires must not race the harness's own wait, or load decides which deadline reports first.
## Synchronize on state
A fixed sleep is not evidence that setup completed or cleanup settled.
@@ -75,6 +94,7 @@ Calling `abort()`, `close()`, or `kill()` without awaiting the owned completion
- For a new static or corpus guard, temporarily introduce the rejected case and observe the intended failure.
- For a race, use barriers to prove overlap; repeated execution alone is not a race test.
- For ports, sockets, shared paths, subprocesses, or other host resources, run independent test processes concurrently when cross-process isolation is part of the fix.
- Where a fixture spawns with its own deadline, assert that no signal or timeout ended the child before asserting its exit status, so a killed child reports as a timeout instead of as a status mismatch.
- Verify external state, events, files, logs, exits, or disposal instead of trusting the component's self-report.
Stress runs supplement a deterministic regression; they do not replace one.
@@ -93,6 +113,8 @@ Do not present these as root-cause fixes for deterministic local tests:
Retries remain valid for documented transient external-provider tests under the real-API policy. Keep that exception at the external boundary.
Restoring a budget is not masking. Raising a suite to the lane budget it already had, or sizing a bounded retry to the contention actually measured on the runner, names the awaited work and returns what the lane granted; neither invents headroom around an unexamined wait.
## Diagnose existing flakes
For an existing probabilistic CI failure, read [the CI flake diagnosis workflow](references/ci-flake-diagnosis.md). A diagnosis-only request remains read-only: report the cause and evidence unless the user also asks for a fix.
@@ -18,10 +18,10 @@ Classify from recorded evidence, not from the eventual fix:
- **Incomplete lifecycle:** teardown returns before children, workers, streams, servers, or callbacks reach quiescence; later output or mutations appear in another test.
- **Process-global contamination:** outcome depends on test order or leaked `process.env`, `cwd`, fake timers, globals, mocks, locale, or module state.
- **Load-sensitive synchronization:** a sleep, polling interval, or assumed event-loop turn substitutes for observable readiness or completion.
- **Platform or entry-path mismatch:** the failure consistently follows an operating system, shell, filesystem rule, source/build mode, or executable entry.
- **Platform or entry-path mismatch:** the failure consistently follows an operating system, shell, filesystem rule, source/build mode, or executable entry. Timestamp precision, environment variable name case, handle-release timing, and permission semantics all differ between Windows and POSIX hosts, so a case passing on macOS says nothing about the Windows lane.
- **Product concurrency defect:** the test controls its resources, reproduces deterministically with explicit overlap, and exposes a race in shipped behavior.
- **External-provider transience:** the failure is owned by a live API or network boundary and matches its documented retry policy.
- **Runner infrastructure:** checkout, dependency download, disk, host process, or runner service fails independently of the test command. Require direct runner evidence before assigning this class.
- **Runner infrastructure:** checkout, dependency download, disk, host process, or runner service fails independently of the test command. Require direct runner evidence before assigning this class. Where a self-hosted pool exposes no host metrics, say so and classify from what the logs do carry: one signature repeating across unrelated branches on one pool is evidence of shared-host contention even when the host cannot be inspected.
If evidence supports more than one independent fact, report each one. Do not collapse a timeout, signal, exit code, and assertion into a single inferred outcome.
@@ -37,6 +37,8 @@ Start with the owning test file or focused test name. Increase concurrency only
Match the active Vitest config, environment knobs, source/build mode, and platform. Do not lower a production timeout or add random load merely to manufacture a different failure.
Where the signature belongs to a platform the available host cannot run, the ladder stops at the last reachable rung. Record that limit rather than substituting a passing run on another platform, then use CI as the reproduction, changing one suspected owner per run so the result stays attributable.
For a suspected race, replace probabilistic timing with a barrier at the contested transition. For a suspected host collision, prove simultaneous acquisition of the same identifier or prove that atomic unique allocation removes the conflict.
## Fix at the owner