The Testing sentence's subject attached the three rebinds to 'the fix' rather than
to the fixture that performs them; reword to 'pinned by a case that rebinds' and
mirror zh ('由一个…用例钉住'), re-recording the pairing.
53 KiB
Agent Note: Settlement, framing, and lifecycle fixes in the CPython backend
Status: implemented
English | 中文
Problem
The CPython subprocess backend for Code Mode, built on the fd-3 frame protocol, resolves every program outcome as a CodeRunResult, rejects run() only for seam misuse, and disposes to quiescence so no subprocess that stays in the child's process group outlives the fiber (a descendant that escapes the group with setsid() is the documented exception — see the package README's Known Limitations). A sequence of review passes surfaced defects that broke those contracts in ways unit coverage did not catch — each hid behind a /* v8 ignore */, a captured-callable that read as a fix but was not, a memory effect invisible through the seam, a load-time bound that double-counted, a process-group escalation that a survivor could outlast, a cross-event-loop completion that silently deadlocked, a synchronous throw outside the settlement path, or a transport boundary rendered as a log boundary. Most behavioral fixes ship with a test that fails without them; nine do not, and are called out as such — the chunked frame read (a syscall-count improvement with no cross-platform-deterministic failure), the confirmed-empty finalize (its only seam-observable effect, a frozen heartbeat, freezes the instant SIGKILL is delivered, which the pre-fix finalize-on-delivery code also produced, and the discriminating probe is the signal-0 check the Alternatives reject as cross-environment-unreliable), the shared stdout/stderr budget (its only seam-observable difference is which entry boundary a mid-stream flush lands on, and that depends on the relative arrival timing of two independent OS pipes, which os.sched_yield does not make deterministic; the per-pipe memory bounds it strengthens ARE covered by the single-pipe flood tests), the flush_line join-clear-push reorder (it lowers the settlement-flush peak from three copies to two, but the 12x load gate already covers the three-copy newline path, so every gate-admitted config stays within the address space under both orders and no seam-observable difference exists — the memory effect is inside the Python child, unmeasurable through the seam like the shared-budget case), pacing binding replies (its in-tree case only asserts the framed replies still round-trip; the peak it removes lives inside the host's fd-3 writable buffer, unseen through the seam, so the 32.0 MiB → 0.0 MiB reduction is measurable only out-of-tree), dropping a late binding resolution before snapshot (its three assertions all also hold pre-fix, because sendReply already dropped after-settlement values — just later than the snapshot), the done-value TOCTOU pre-encoding (a concurrent mutation racing the encode cannot be deterministically constructed through the seam — its daemon-mutation regression only asserts the result is never a worker-exit, which is probabilistic and non-discriminating, so under the existing no-fail-before-with-a-reason precedent it is registered as no-fail-before), the stray-UTF-8 budget-flush retention (a budget flush landing exactly on a multibyte boundary is not schedulable through the seam; it is cross-referenced as v8-ignored), and the late-rejection settled guard (a rejection arriving after the run has already settled cannot be deterministically constructed from the seam).
Decision
Independent corrections, each in the package that owns the defect.
Boot-write failure no longer rejects run()
In src/index.ts the fd-3 boot-frame write is the last statement of run()'s synchronous setup. Its catch calls finish(), and finish() reads wallTimer and onAbort and — through settle() — live. Those bindings are const and were declared AFTER the boot-write, so on a synchronous write failure finish() touched them in their temporal dead zone and threw a ReferenceError. That escaped the Promise executor and REJECTED run(), violating the seam's "outcomes resolve" contract: the caller saw a thrown error instead of the worker-exit the catch constructs. The boot-write block is now emitted after wallTimer, onAbort, and live are initialized, and the /* v8 ignore */ that had hidden the branch from coverage is removed so the catch is measured.
Log capture is serialized against settlement
In py/bootstrap.py the settlement flush_out()/flush_err() on the main coroutine read and clear each stream's _pending list and mutate the shared LogBuffer ledger. Model code may start daemon threads whose print/write mutate the same state concurrently. Capturing the bound method (out_stream.flush_line) fixed only WHICH callable settlement invokes, not what it reads mid-flight: an interleaved flush could join a _pending list being mutated under it, corrupting the ledger and costing the done frame — stranding the run to the wall clock. LogBuffer now owns one re-entrant lock shared by both streams; _LogStream.write and flush_line, and LogBuffer.push, take it, so the whole read-modify-write is atomic across threads.
Fd-3 residual is copied, not viewed
Also in src/index.ts, after the newline loop over a Buffer.concat of the pending fd-3 chunks, the leftover partial line was carried forward as the subarray VIEW it was sliced to. A view keeps the entire concat backing allocation alive, so a large frame followed by a tiny trailing fragment pinned a whole frame's worth of memory while pendingBytes — set to the fragment's length — reported far less than was retained. The residual is now detached into a fresh right-sized Buffer via the exported detachResidual helper, letting the concat allocation be collected and keeping pendingBytes an honest measure.
Output-cap load bound is ceiling minus envelope, not divided by six
The load-time check that rejects a maxLogBytes/maxValueBytes larger than one fd-3 frame can carry divided the frame ceiling by six for worst-case escape expansion. But both budgets are metered in ALREADY-ESCAPED serialized bytes — the host log ledger charges the serialized cost via jsonStringCostUpTo (which walks to the cap without allocating the escaped copy), checkDoneValue measures the escaped form, and the producing-side _cap_message also caps by serialized cost — so a payload admitted under the cap occupies at most cap + envelope on the wire; escaping is inside the charge and must not be multiplied in again. The bound is now FRAME_CEILING_BYTES - FRAME_ENVELOPE_BYTES, and the unused MAX_JSON_ESCAPE_EXPANSION constant is gone. The old bound was not unsafe — it under-admitted — but it silently forbade legitimate large caps. The same load check also rejects a NON-INTEGER maxLogBytes/maxValueBytes: the child reads each budget through int(...), which floors a float, so maxLogBytes: 3.5 would truncate at 3 bytes child-side while the host meters the fraction — the two sides enforcing different public config. Rejecting the float at load keeps them in step, matching the worker backend.
Same-group survivors are reaped before the fiber goes quiescent
A model program can leave a descendant in the child's OWN process group (no setsid, so kill(-pid) reaches it) that ignores SIGTERM but releases the inherited stdout/stderr/fd-3 pipes. The leader then exits, its close fires because the pipes drained, and settlement runs while that descendant is still alive. kill() arms an unref'd SIGKILL timer after SIGTERM; the fix is that settle() no longer resolves the run's finished promise — nor drops the run from live — immediately when an escalation is in flight. Instead, when killing is set and the process group is not yet empty (process.kill(-pid, 0) does not throw ESRCH), it polls the group on a REF'd timer, bounded by graceMs + CLOSE_REAP_MARGIN_MS, and both drops the run from live and resolves finished only once the group has emptied. The ref'd poll is the load-bearing part: it keeps the host event loop alive until the SIGKILL has actually reaped the group, so even a short-lived host — a one-shot headless run, a config subprocess — cannot exit and reparent the survivor to init. Deferring the live removal is what makes a dispose() racing a just-resolved run() still await the survivor: dropping the run from live at settlement (before the reap) would let teardown snapshot an empty set and return while the descendant lived. In the normal case (the leader was the only member) the first probe returns ESRCH and settlement finalizes with zero added latency. teardown() awaits each run's finished, so disposal is genuinely quiescent, matching its JSDoc — including for a run that already resolved.
Settlement also CANCELS the SIGKILL timer the moment the group is confirmed empty (the normal path, and when the poll sees the survivor gone). Leaving it armed would expose a PID-reuse hazard: a kill(-pid) left pending for up to graceMs after the leader was reaped could hit a RECYCLED pgid once the kernel reused the leader's pid, SIGKILLing an unrelated group (killGroup swallowing ESRCH does not help — the danger is precisely the kill that SUCCEEDS against a reused group). Clearing it on the empty probe bounds the reuse window to only the genuine-survivor case, where the group cannot be empty to reuse.
The window the cleared timer cannot cover is closed by an IDENTITY check inside killGroup. Every signal it sends is a raw process.kill(-child.pid, sig), which — unlike child.kill() — has no handle guard, so it would reach a recycled pgid during the interval between the leader being reaped and close firing (measured at 3039 ms with a pipe-holding descendant). The leader's start time is therefore read once at spawn (/proc/<pid>/stat field 22) and re-read before each signal, with two rulings: a reading that is PRESENT AND DIFFERENT means the number now belongs to another process, so the signal is withheld; an ABSENT reading means the leader was already reaped, which is the ordinary case for every escalation — its /proc entry is gone while the group it led can still hold the survivor this teardown exists to reap — so the signal proceeds. Absent is also the constant reading on a platform with no /proc, where the guard is inert and the pre-existing behavior stands. Reading absent as a mismatch is not hypothetical: the first version did, which withheld the grace SIGKILL and the poll deadline's SIGKILL, and the three same-group heartbeat cases went red on the Linux coverage lane while passing on Darwin, where the reader always returns undefined.
The reap poll also handles a host event loop BLOCKED past both timers. If a synchronous computation holds the loop from before the poll was scheduled until after its deadline, both the poll timer and the grace-window SIGKILL timer are overdue when the loop resumes, and Node runs the earlier-scheduled poll first — so the grace SIGKILL may never have fired. The deadline branch therefore sends SIGKILL ITSELF (idempotent if the timer already ran) rather than cancelling the unfired escalation, then grants ONE more CLOSE_REAP_MARGIN_MS and keeps polling until the group is confirmed empty, because finalizing on mere signal delivery would declare quiescence while the group is still dying. The outer bound on the wait is therefore graceMs + 2 * CLOSE_REAP_MARGIN_MS. A final hard bound finalizes if that extra margin elapses with the group still non-empty; that branch carries a /* v8 ignore */ because it is reachable only where a SIGKILL'd survivor lingers as a zombie and is never wait()'d — a container whose PID 1 does not reap orphans — which cannot be built deterministically across CI platforms. The ignore's reason states that environment dependence rather than claiming the branch cannot run, cross-referencing the Alternatives entry that rejected the signal-0 reap assertion for the same reason.
RLIMIT clamps against the inherited soft limit, not only the hard
In py/bootstrap.py _clamped bounded a requested (soft, hard) rlimit pair by the inherited HARD limit alone. A deployment that inherited a soft limit below the requested one — say inherited (100, 200), requested (150, 160) — got back (150, 160), RAISING the effective soft from 100 to 150: for RLIMIT_AS that loosens the memory ceiling, for RLIMIT_CPU it defers SIGXCPU, both violating "strictest of configured and inherited". _clamped now clamps each side against its own inherited counterpart (RLIM_INFINITY imposing no ceiling), then pins soft under hard so setrlimit never sees an inverted pair. The settlement-time CPU recheck (die_if_cpu_exhausted) follows the same rule: it compares spent CPU against the EFFECTIVE clamped cpu_soft, not the configured cpuSeconds, so a program that traps SIGXCPU and burns past a stricter inherited soft before returning is reported as a timeout rather than a false success. The SIGXCPU diagnostic no longer names the configured cpuSeconds as the effective budget — under a stricter inherited soft that number is wrong — and instead reports that CPU time was exhausted at "at most the configured N seconds", which holds whichever limit fired.
Concurrent binding replies are paced against fd 3
sendReply ignored proto.write's false return, so a program resolving several large values in one asyncio.gather round encoded every reply in the same turn and queued all of them in fd 3's writable buffer. Binding resolution carries no seam-level byte cap to bound that, and the failure kills the HOST process rather than failing the run: measured on a 64 KiB-highWaterMark pipe, eight 4 MiB replies buffered 32.0 MiB at once. Replies now go through a queue that encodes and writes one frame at a time, awaiting drain when the pipe is full, which measured a 0.0 MiB peak for the same shape. The encode happens inside the loop so a queued reply the run no longer needs is dropped by the settled check without ever being serialized. The same settled predicate also guards the reply callback AFTER await fn(...) but BEFORE snapshotJsonValue, so a wide value that resolves after settlement is dropped before its width is walked — the host does not expand a late value for a run whose outcome is already fixed.
Pacing changes nothing the model can observe. The child matches each reply to its call by id from a pump that reads fd 3 continuously, so arrival order was never observable, and the bindings themselves still run concurrently -- only the host's peak memory and the flush timing change. That is also why serializing is not a narrowing of the seam's concurrency contract, which was the reason this was first deferred; that reasoning was wrong.
Binding replies complete on the calling loop's thread
Also in py/bootstrap.py, a binding reply Future is created on the loop that ran dispatch. When the model calls a binding from a worker THREAD via asyncio.run(tools.x(...)), that Future belongs to the thread's loop, not the main loop where _pump_replies reads the reply. asyncio.Future is not thread-safe: completing it from another thread does not wake its own loop, so the direct set_result/set_exception left the awaiting thread stranded and the run degraded to a wall-clock timeout. Each pending entry now records its Future's loop alongside the Future, and _pump_replies completes it via that loop's call_soon_threadsafe. The shared pending/next_id state is guarded by a threading.Lock held across the id claim, the fd-3 write, and the counter advance, so concurrent callers cannot interleave frames out of the id order the host requires. call_soon_threadsafe onto a loop that has already CLOSED (the worker thread finished and abandoned its call before the reply arrived) raises RuntimeError; that schedule is wrapped so the moot reply is dropped rather than letting the exception end the pump task and strand every later reply.
The blocking frame reader reads in chunks, not byte by byte
ProtocolChannel.read_frame — used for the boot and run handshake frames — read through FileIO.readline() on the unbuffered (buffering=0) fd, which issues one os.read(1) per byte. The run frame arrives AFTER RLIMIT_CPU is in force, so a legitimate multi-megabyte program burned seconds of CPU in millions of single-byte syscalls before ast.parse ran — potentially exhausting the budget on the read alone. It now reads in _READ_CHUNK_BYTES chunks into the same _pending residual buffer the async reader already uses (the wrapping os.fdopen object is gone; both readers call os.read(self._fd, ...) directly), so the read cost is trivial and read-ahead past a newline is preserved for the next frame. Both readers track a running scan offset (find(b"\n", scanned)) so a large frame accumulated across many chunks is scanned once, not re-scanned from index 0 per chunk — a chunked rescan would have replaced the byte-at-a-time cost with an O(N²) memchr cost on the same large-frame path.
Synchronous spawn failure resolves worker-exit, not reject
Also in src/index.ts, spawn is called before the settlement Promise executor exists. Node defers only a fixed set of spawn errnos (EACCES, EAGAIN, EMFILE, ENFILE, ENOENT) to an asynchronous error event, which the settlement path already turns into a worker-exit; every other errno throws SYNCHRONOUSLY from spawn. A pythonBin longer than the platform PATH_MAX passes the load-time validation (non-empty, no NUL) but makes spawn throw ENAMETOOLONG here — outside the executor — so run() REJECTED instead of resolving, violating resolve-don't-reject, and left this run's just-materialized staging directory on disk since only settle() removes it. The spawn call and the fd-3 narrowing are now wrapped: a synchronous throw removes the staging directory and resolves the same worker-exit class (python spawn error: …) the async error event produces.
Stray pipe output is aggregated by line, not by transport chunk
Also in src/index.ts, native stdout/stderr bytes (C-extension writes, os.write past the pipe buffer) were pushed to logs one entry per Node data chunk. logs entries are joined with \n downstream (Code Mode), so a single newline-free write larger than one pipe read — arriving as several data chunks — read back with model-visible newlines inserted at arbitrary transport boundaries. Capture now accumulates raw Buffer chunks (the same shape as the fd-3 reader, and for the same reasons: a string += accumulator re-copies the whole residual per chunk and scanning it from index 0 each chunk is a second quadratic — both O(N²) on a large newline-free write), splits on the raw 0x0a byte, and admits one entry per complete line. A newline never appears inside a UTF-8 multibyte sequence, so decoding each split line is safe without a streaming decoder. Three separate bounds keep the residual from exhausting host memory, each mirroring the fd-3 reader: the fragment list SEALS into finished blocks past MAX_PENDING_CHUNKS so a program pacing single-byte os.writes cannot accumulate millions of live Buffer objects (whose per-object overhead no byte count sees); the residual is flushed when the COMBINED running SERIALIZED cost of both pipes — tracked through accrueStrayCost, which decodes UTF-8 structurally across chunks so a byte that renders as U+FFFD is charged the three bytes that replacement character serializes to — would cross the budget, so a control-char or illegal-UTF-8 flood flushes at a fraction of the raw bytes rather than accumulating a full budget's worth of raw bytes first, and stdout and stderr are metered together rather than each against the full budget (which would let both retain nearly a budget's worth at once, doubling the peak); and once the ledger has truncated, buffering stops so nothing accumulates for output that can never be admitted. accrueStrayCost charging illegal bytes their U+FFFD width is the fix for a byte that never begins a valid sequence (0x80–0xC1, 0xF5–0xFF), a multibyte sequence that breaks before completing, or a structurally-complete but ILLEGAL sequence: toString('utf8') renders each of those bytes as its own U+FFFD (3 bytes), so it validates each lead's first-continuation range (WHATWG: E0→A0-BF, ED→80-9F, F0→90-BF, F4→80-8F, others 80-BF) and charges 3 per byte of any sequence outside it. Charging the raw 1 undercounted a b"\xff" flood threefold, and charging only the structural width undercounted a CESU-8 surrogate (ED A0 80) or overlong (E0 80 80) threefold just as cheaply, letting the residual grow to a full budget's worth of raw bytes before flushing and, near a large maxLogBytes, expand toward a ~1 GiB peak in the flush's concat plus toString. The per-entry charge on the admitted string is metered by SERIALIZED cost through jsonStringCostUpTo, which walks the string to the cap and stops — the previous Buffer.byteLength(JSON.stringify(text)) allocated the whole escaped form first, so a near-budget control-char-dense line under a large maxLogBytes could momentarily allocate over a gigabyte just to measure it. jsonStringCostUpTo (the string-walking function, reached by a forged log frame whose text JSON.parse produced) charges a LONE surrogate the full six escaped bytes (\uXXXX under ES2019 well-formed JSON.stringify), not the three bytes Buffer.byteLength reports for its U+FFFD rendering, so a \ud800 flood is not undercharged by half; accrueStrayCost walks raw bytes and never sees a surrogate as such — a CESU-8-encoded surrogate reaches it as three bytes its per-lead range check rejects, each charged 3 (total 9), matching what toString('utf8') renders. The residual is flushed on the pipe's end and also explicitly in the closeDeadline handler before it destroys the streams: a setsid escapee holding the pipes open forces settlement through that path without an end, so a final newline-free os.write(1, …) the leader emitted before exiting would otherwise be dropped from logs.
An incompatible output-budget/addressSpaceMb pair is rejected at load
The child (py/bootstrap.py) builds, charges, and frames a maxLogBytes log entry or a maxValueBytes completion value under RLIMIT_AS, and both ledgers trigger on CHARACTER count against a serialized-BYTE budget. An astral character is one character but four bytes of CPython str storage and four UTF-8 bytes, and the heaviest path holds THREE such copies at once: a single sys.stdout.write(line + "\n") keeps the caller's text argument (alive for the whole write call, ~4×), the line slice handed to LogBuffer.push (~4×), and the text.encode("utf-8") copy _push_locked takes to charge and ship it (~4×) — a peak of ~12× the budget. The settlement flush_line path holds only two (its "".join(...) and that encode copy — it drops the pending chunks before pushing), so the newline path is the binding worst case. When a budget approaches addressSpaceMb, a LEGITIMATE near-budget output breaches the address space during that build-and-encode and dies as worker-exit instead of truncating (log) or failing as output-limit (value). Metering every child write against the address space at runtime is the wrong fix: an exact serialized-cost check on the hot path is either a full encode (the allocation being avoided) or a per-character Python loop (which burns the CPU budget — a 10 MB legitimate write hits SIGXCPU under cpuSeconds: 1). Both trade one resource bound for another. Instead src/index.ts rejects the incompatible pair at LOAD: each budget times OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE (twelve — the three simultaneous ~4× copies of the newline path) must fit the address space LEFT after a fixed INTERPRETER_BASELINE_BYTES reservation for the interpreter's own footprint, with a >= so a budget whose worst-case peak exactly equals that room is rejected (that peak plus the reserved baseline is the whole address space, the RLIMIT_AS edge). flush_line was also made to drop the pending chunks BEFORE its push, matching the newline path's join-clear-push order, so it holds at most the join and its encode rather than three copies. The baseline is reserved SEPARATELY from the multiple because it is a fixed cost, not one that scales with the budget: folding it into the multiple would leave a budget sized right at addressSpaceMb / 12 admitted while its peak plus the interpreter still overran. Both maxLogBytes and maxValueBytes are gated symmetrically; the value path's build-and-encode is the same shape. The check runs on every platform, not just where RLIMIT_AS is enforced: the incompatibility is a property of the config values, so a Linux deployment OOMs regardless of the host that assembled the config, and a uniform load-time rejection is the fail-loud contract (Darwin skips only the runtime setrlimit). This eliminates the class at the config seam rather than patching the write path, so _LogStream keeps its original character-count buffering (a valid lower bound on serialized cost, memory-safe once the budget fits the address space). The value path enforces the same discipline in a second place: _check_done_value (the byte meter) and _encode_json_plain (the frame encoder) walk in O(DEPTH), not O(width). Each container pushes ONE cursor frame that pulls its children one at a time rather than one traversal tuple or stack entry per child — a flat [0] * 6_000_000 serializes to ~12 MB but a per-element walk allocates ~400 MB of bookkeeping (~28× the serialized size, far past the 12× the gate reserves), so a value the meter admits could OOM on the walk's own frames. With the cursor, the only width-proportional allocation is the output string the meter already bounded.
The host gate validates against the CONFIGURED addressSpaceMb, but a launch environment can inherit a STRICTER RLIMIT_AS (a ulimit -v wrapper below addressSpaceMb), which the bootstrap's _clamped correctly lowers the EFFECTIVE limit to — leaving the budgets sized for a ceiling the child never gets. So bootstrap.py re-checks both budgets against the effective clamped soft limit after applying it, mirroring the host gate's multiple and baseline, and raises at boot (caught by the setrlimit-phase handler and reported as exception, the same class as any other resource-limit-application failure) rather than letting a near-budget output OOM mid-run. The two child constants are kept in step with the host's by the shared reasoning, not a wire field.
One residual write-path copy is fixed alongside, independent of the config gate: _LogStream.write's newline branch buffered the whole unterminated tail after the last newline (text[pos:]) into _pending before the flush trigger could bound it, so an early newline followed by a huge tail ("\n" + "A" * 30 MiB) made a second full copy of the model's own string — the RLIMIT_AS death the path exists to avoid, and one the config gate does not cover because the tail can far exceed maxLogBytes. The tail is now sliced to a remaining + 4-character prefix (anything past remaining characters cannot be admitted, the char count being a lower bound on the serialized cost), which the flush trigger then rejects with the marker.
The completion value and error are pre-encoded at their validation point
In py/bootstrap.py, _done_with_value now returns the whole terminal frame as a PRE-ENCODED JSON string on the success path: the admitted value is serialized once here, at the validation point inside _run's try, as '{"type": "done", "value": ' + _encode_json_plain(value) + "}". The program can keep mutating a returned list/dict from a daemon thread or signal handler after it returns, so a second traversal held at a later point would be a TOCTOU — a concurrent mutation into a non-JSON type would let that later encode throw outside the settlement handler and downgrade a settled run host-side to worker-exit. Serializing once, inside the try that wraps this call, closes the window: if a concurrent mutation makes the encode throw, the exception handler classifies it as an exception, and once the string is produced the frame is written verbatim with no further touching of the live value.
send_done (a local function inside _run) writes the pre-encoded string through a BOUND channel.write_encoded, and encodes a dict error frame through a bound _encode_json_plain before writing it — it never calls channel.send_sync, whose body re-resolves self.write_encoded and the module-level _encode_json_plain at call time. _encode_json_plain and channel.write_encoded are bound into locals before the program runs, for the same reason flush_out/flush_err/safe_model_traceback are: the program runs as __main__, so import __main__; __main__.ProtocolChannel.send_sync = boom or __main__._encode_json_plain = boom would otherwise re-resolve the send/encode to a rebranded callable at call time and, when that replacement raises, skip the done frame and downgrade a settled verdict to a host-side worker-exit.
A budget flush retains an unfinished trailing multibyte sequence
Also in src/index.ts, flushStray(stray, retainPartialTail) withholds an unfinished multibyte tail from the decode on the BUDGET-triggered flush (the combined-cost threshold in captureStray): when the residual ends on a partial UTF-8 lead sequence (stray.utf8.expected > 0), the leading byte plus the continuations consumed so far (≤3 bytes) are detached from the frame as the new residual, and only the complete prefix is admitted and decoded. Nothing is admitted when the whole residual is a single unfinished sequence, so a legal, un-finished character is never rendered as U+FFFD in a released, un-truncated entry, and no bogus empty entry is pushed. The withheld tail is re-accrued from a FRESH stray.utf8 state — metering it against the post-flush expected > 0 state would charge the carried lead byte as an illegal continuation — so the next chunk continues the walk correctly and the pipe's cost/UTF-8 state is rebuilt over the retained tail. The end/closeDeadline paths pass false and decode the FULL residual unchanged, because there a trailing incomplete sequence is real truncated input and the U+FFFD is the honest render.
A late binding rejection returns before formatting the error
Also in src/index.ts, the binding-rejection catch branch now checks settled and returns BEFORE formatting messageOf(error). A rejection that arrives after maxWallMs, an abort, or dispose has already settled the run would otherwise have messageOf(error) run hostile toString/message getters — spending host heap and time on a run whose outcome is already fixed — before sendReply peeks at settled. Dropping the framed reply early spares that waste. The running loop's otherwise-mostly-linear reply drain also reads by a head cursor into the queue array instead of shift()ing each entry, so a large asyncio.gather of wide bindings awaiting fd 3's drain drains in linear time rather than O(n²) from repeated re-slicing.
Testing
tests/boot-write-failure.spec.tsmocksspawnso the fd-3 pipe throws on the boot write — the one path a real subprocess cannot be coerced into — and assertsrun()resolves aworker-exitrather than rejecting. A sibling case makes the mockedspawnthrow SYNCHRONOUSLY and assertsrun()still resolves aworker-exitand removes its staging directory, keyed off the exact bootstrap path the mockedspawnreceived in its argv so a sibling worker's concurrent staging cannot flake it. Both are isolated in this spec so the real-subprocess suite is untouched.tests/residual-detach.spec.tsunit-testsdetachResidual: the carried copy equals the residual, owns a backing store sized to its own length (fixture kept above Node's Buffer pool threshold), and does not share the source frame'sArrayBuffer.tests/runtime.spec.ts— the output-cap case asserts theceiling - envelopebound (268435392) and its message. A daemon-thread case drives four threads emitting unterminated writes through settlement's flush. A native-write case writes 200 KiB with no newline viaos.writeunder a raisedmaxLogBytesand asserts it reads back as EXACTLY one log entry (proving stray output is aggregated by line, not split at pipe-chunk boundaries); a companion writesb"one\ntwo\nthree"and asserts three entries (proving real newlines still delimit). A newline-free-flood case writes 2 MiB under a 4 KiBmaxLogBytesand asserts the capture ends at the truncation marker and stays under budget (proving the residual is bounded by the ledger, not buffered whole); a NUL-flood companion writes 4000 newline-free NULs under the same budget and asserts truncation (proving the residual is charged by SERIALIZED cost, ~6× raw, measured without allocating the escaped copy); an illegal-UTF-8 case paces single-byte\xffwrites under a 3072-byte budget withBuffer.concatwrapped to measure the peak merged buffer, asserting it stays under 2048 (charged at the U+FFFD width 3 the residual flushes near 1024 raw bytes; a raw-byte undercount would let it reach ~3072, so the bound discriminates); a CESU-8/overlong case paces the structurally-well-formed but illegalED A0 80one byte at a time and asserts the same peak bound (charged at the true 9 per sequence it flushes early; charging the structural width 3 triples the peak, so reverting the per-lead range check turns it red); a broken-multibyte case writes a 3-byte lead then a fresh ASCII byte in separate chunks and asserts both a capturedAand a U+FFFD (exercisingaccrueStrayCost's cross-chunk broken-sequence branch); a post-truncation case writes a 108-byte payload (under the smallest PIPE_BUF, so one atomic write) whose first line exhausts a 64-byte budget and asserts the second line is dropped (exercising the post-truncation admit no-op in onedatacallback, no v8-ignore); a short-escape case writes a line mixing a tab, quote, backslash, a\uXXXXcontrol, a multibyte character, and ASCII, asserting it round-trips verbatim (exercising every branch ofjsonStringCostUpTo); a reassembly case writes a payload spanning every valid multibyte lead class (E0-range, plain 3-byte, F0, and F4) past the pipe buffer and asserts it round-trips with no U+FFFD (exercisingaccrueStrayCost's per-lead ranges and cross-chunk reassembly); a lone-surrogate case forges an fd-3logframe flooding 1000\ud800escapes under a 4 KiB budget and asserts truncation (the count sits in the window where charging 3 bytes would admit and 6 bytes truncates, proving the surrogate is charged its full escaped width); a stray-sealing case paces 60000 single-byte newline-freeos.write(1, …)calls under a raised budget withBuffer.concatwrapped to measure copy volume, asserting the trickle coalesces to one entry and the cumulative copy stays under a measured 256 KiB threshold (the sealed shape copies ~120 KB, the re-merge shape ~538 KB, so reverting the seal to a re-merge turns the assertion red — proving the fragment list seals into blocks pastMAX_PENDING_CHUNKS). A closeDeadline-flush case has the leader write a newline-free diagnostic then spawn asetsidorphan holding the pipes open, and asserts the diagnostic survives inlogs(proving the residual is flushed before the deadline destroys the streams). The same-group reap case spawns a SIGTERM-ignoring same-group descendant that releases the pipes and bumps a heartbeat file; the test asserts the heartbeat STOPS after the grace-window SIGKILL — an assertion robust whether the killed descendant is reaped or lingers as a zombie, so it holds where PID 1 does not wait() orphans. A dispose-after-resolve case assertsdispose()of a completed run with a same-group survivor returns only after the survivor stops executing (proving the run stays inliveuntil its group is reaped), with anexpect(afterDispose).toBeGreaterThan(0)guard so the frozen-heartbeat assertion cannot pass vacuously when the file was never written. A deadline case busy-blocks the event loop past both timers and asserts the survivor's heartbeat freezes (proving the poll's deadline arm sends SIGKILL itself rather than cancelling the unfired escalation). The cross-loop case runs a binding from a worker thread's ownasyncio.runloop while the main coroutine yields withawait asyncio.sleep, asserting the reply round-trips instead of timing out; a companion case abandons a thread's call so its loop closes, then answers it before a later binding — asserting the pump survives the closed-loopcall_soon_threadsafe(host-gated ordering makes it deterministic, fail-before hangs the later binding to the wall clock). The inherited-soft-limit case runs the interpreter through aulimit -S -twrapper that sets a CPU soft limit belowcpuSecondsand asserts the appliedRLIMIT_CPUsoft is the inherited value, not the configured one (CPU rather than address space, since macOS ignoresulimit -v); a companion inherits a 1 s CPU soft, has the program trap SIGXCPU and busy-loop past it, and asserts the settlement recheck reports a timeout — proving the recheck uses the effective soft, not the configuredcpuSeconds. A control-heavy-diagnostic case raises a NUL-flood exception under a smallmaxValueBytesand asserts the serialized frame fits (proving the diagnostic is metered by serialized cost). A tail-copy case (maxLogBytes: 256,addressSpaceMb: 384) has the program build a tail in a variable and write"\n" + tailwheretailis 150 MiB — construction peaks at ~2× (~300 MiB, within the address space, so the model's own allocation succeeds and any OOM belongs to the defect path), and the pre-fix whole-tail re-buffer added a third ~150 MiB copy past 384 MiB; the sliced prefix lets the run truncate and complete (Linux-only RLIMIT_AS repro, macOS happy path — the fixture's own construction must fit the address space, a general rule for these RLIMIT_AS cases). An output-budget/address-space case asserts amaxLogBytesof 50 MB AND amaxValueBytesof 50 MB each reject at load against a 256 MiBaddressSpaceMb(past the room left after the interpreter baseline when multiplied by the worst-case 12) while the default caps against 512 MiB load, gating both budgets symmetrically; a discriminating case asserts a 48 MiBmaxLogBytesagainst a 512 MiBaddressSpaceMbrejects — 48×8 = 384 MiB fits the 448 MiB budgetable (the old 8× multiple wrongly admitted it) but 48×12 = 576 MiB does not. The ~12× peak the multiple covers is the NEWLINE path's single near-budget write — the caller's own string, the line slice, and the encode copy live at once; the settlement flush is no longer the binding case, becauseflush_linedrops the pending chunks before its push and so holds two copies rather than three. An inherited-RLIMIT_AS case runs the interpreter through aulimit -v 131072wrapper with a 32 MiBmaxLogBytesthe configured 512 MiBaddressSpaceMbadmits, and asserts the boot re-check rejects it as anexceptionwhose message names the inherited RLIMIT_AS (the 128 MiB inherited limit leaves too little after the baseline; Linux-only, macOS ignoresulimit -vand the run proceeds). A non-integer-budget case asserts a fractionalmaxLogBytes/maxValueBytesrejects at load. A combined-peak case (maxLogBytes: 32 MiB,maxValueBytes: 32 MiB,addressSpaceMb: 512— each budget admitted alone at 12×) writes ~33M newline-free astral characters (buffered, unflushed) then returns ~33M astral characters, and asserts the run settles asoutput-limit(the value is itself over its 32 MiB budget); pre-fix the unflushed log pending plus the value's build-and-encode peak added past the 512 MiB address space and OOM'd, so flushing the logs before framing the value is what lets the value check complete (Linux-only RLIMIT_AS repro; on macOS the over-budget value reports output-limit under both orders). A wide-value case (maxValueBytes: 20 MiB,addressSpaceMb: 384) returns[0] * 6_000_000— ~12 MB of JSON, under the 20 MiB budget, so it must round-trip; pre-fix the O(width) walk allocated ~400 MB of per-element traversal tuples and encoder stack entries (~28× the serialized size, past the 12× the gate reserves) and OOM'd on a value the meter admitted, while the O(depth) cursor keeps the only width-proportional allocation the output string itself (Linux-only RLIMIT_AS repro; the fixture stays within the address space so it is honest on macOS too). A wide-BINDING-ARGUMENT case (addressSpaceMb: 384) calls a binding with[0] * 6_000_000and asserts the length echoes back:_lossless_json_violationruns on model-built arguments that no child-side budget bounds first, and its per-member tuples measured 459.1 MiB against 0.0 MiB for the cursor. A backtracking case returns a 4 MiB string from a binding and asserts it round-trips: the old scalar regex retained engine state proportional to the string's width (146 MiB at 1 MiB, 557.8 MiB at 4 MiB, past the default 512 MiB), which raised MemoryError inside_pump_repliesand stranded the call to the wall clock. A control-heavy metering case returns 8M NULs under a 16 MiBmaxValueBytesand assertsoutput-limit, notexception: charging by counting instead of materializing the escaped form measured 19.1 MiB against 228.9 MiB for an identical byte count. An addressSpaceMb-baseline case asserts 64 MiB and 32 MiB reject at load with a message namingaddressSpaceMb, rather than the budget loop's negative admissible limit. A process-identity case asserts the leader's start time reads stably on Linux and reports undefined on Darwin, the guard that keeps a recycled pgid from receiving this run's SIGTERM. A paced-replies case resolves eight 4 MiB values in oneasyncio.gatherround and asserts the frames round-trip; the peak the fix removes (32.0 MiB buffered → 0.0 MiB) lives in the host's fd-3 writable buffer, invisible through the seam, so this case pins the round-trip and the no-regression match but is measured for its peak only out-of-tree. A late-drop case settles the run onmaxWallMsand resolves the pending binding afterwards, asserting atimeoutresult, an undefined value, and that the late path actually ran — the three assertions all hold pre-fix becausesendReplyalready dropped after-settlement values, just later, so the case pins the ordering, not a seam-observable behavior. A binding-all-names case (rebinds every name the failure path uses) asserts a realValueErrorsurvives send-done binding — a tested fix pinned by a case that rebinds__main__.ProtocolChannel.send_sync,__main__.ProtocolChannel.write_encoded, and__main__._encode_json_plain— the three names the shippedsend_donewould resolve late if it looked them up at call time — and pins thedoneframe against that call-time look-up skipping it; the done-value TOCTOU pre-encoding, the stray-UTF-8 budget-flush retention, and the late-rejection settled guard are counted among the nine no-fail-before fixes (reasons in the Problem section), not pinned by a fail-before test.
Alternatives considered
Leave the boot-write /* v8 ignore */ and fix only the ordering. Rejected: the ignore is what let the TDZ regression ship uncaught. Removing it makes the catch a measured branch, so per-file 100% coverage now proves the failure path is exercised.
Fix the flush race by capturing more bound methods. Rejected: this is the approach that already failed. Binding a callable fixes reference resolution, not concurrent access to the mutable state the callable reads. Only mutual exclusion over the shared ledger closes the race.
Guard the residual with a size threshold (copy only large frames). Rejected: the branch runs once per newline-bearing read, the copy is bounded by the residual's own length (always a partial line), and a threshold adds a tunable and a second code path for no measurable saving. An unconditional right-sized copy is simpler and always correct.
Assert the residual memory effect through the seam. Rejected: the retained allocation is not observable through CodeRunResult, so a black-box test could not distinguish fixed from unfixed. Extracting detachResidual makes the backing-store invariant a deterministic unit test instead.
Reap the same-group survivor with a fire-and-forget unref'd SIGKILL timer alone. Rejected: an unref'd timer does not keep the host alive, so a host that exits within the grace window (a one-shot run, a config subprocess) never fires the SIGKILL and the survivor is reparented to init — the same "no subprocess outlives the fiber" violation in a different shape, and teardown's "await each child's exit" JSDoc would be false. Awaiting the group's death on a ref'd poll keeps the host alive exactly long enough to reap, at zero cost in the common empty-group case.
Assert the reap with process.kill(pid, 0) throwing ESRCH. Rejected: a SIGKILL'd process lingers as a zombie until its parent wait()s it, and in a container whose PID 1 does not reap orphans the signal-0 probe keeps succeeding, so the assertion would false-fail cross-environment. A heartbeat file that stops advancing detects "no longer executing," which a reaped process and a zombie both satisfy.
Complete the cross-loop Future with a plain set_result and rely on the GIL. Rejected: the GIL serializes bytecode but does not make asyncio.Future cross-loop-safe — completing a Future from a thread other than its loop's does not schedule its callbacks or wake the loop. call_soon_threadsafe on the owning loop is the documented mechanism.
Leave the SIGKILL timer armed after settlement (the earlier same-group fix). Rejected: an unref'd timer left to fire up to graceMs after the leader was reaped can kill(-pid) a RECYCLED pgid, striking an unrelated group; the danger is the kill that succeeds, which killGroup's ESRCH swallow cannot prevent. Clearing the timer once the group is confirmed empty bounds the reuse window to the genuine-survivor case, where the group is not empty to reuse.
Clamp rlimits by the inherited hard limit only. Rejected: that silently RAISES an inherited soft limit stricter than the request, loosening the very containment the clamp exists to preserve. Clamping each side against its own inherited bound (then pinning soft under hard) keeps the strictest of configured and inherited on both.
Bill the host-side capMessage backstop by serialized cost, matching the child's _cap_message. Rejected: the two caps guard different things. _cap_message's output re-crosses fd 3 as a JSON string, so its escaped width is what the frame ceiling bounds — serialized billing is required there. capMessage's output goes straight into CodeRunResult.error.message and never re-crosses a frame-bounded channel, so the honest measure of what it retains is the raw byte length of the model-visible string. An honest child has already capped by serialized cost and raw length ≤ serialized cost, so a well-formed message passes unchanged; a forged control-heavy message could serialize to ~6× its raw length, but since it travels no capped channel, billing it by that inflated wire width would truncate a legitimately-sized diagnostic for no containment gain. Each side's JSDoc documents the split and points at the other.
Push stray pipe output one entry per data chunk. Rejected: logs entries are joined with \n downstream, so a transport chunk boundary would become a model-visible newline — a single native write split across pipe reads would read back with spurious line breaks. Aggregating by real newline (raw-chunk buffer + split on 0x0a) matches the child's line-granular log frames; the ledger still bounds a newline-free flood by admitting-and-truncating the residual when it would cross the budget.
Enforce the fd-3 frame ceiling per-frame (split before the counter check) to avoid a batch-edge false reject. Rejected: the ceiling check reads the byte counter BEFORE any Buffer.concat, precisely so a hostile program cannot force ~2× the 256 MiB ceiling of host memory (the counter and the join are a second copy of everything held). Splitting first to bill a single frame would Buffer.concat an over-ceiling frame before rejecting it, reintroducing that doubling — two regression tests assert the pre-concat order for exactly this reason. The batch-edge false reject the per-frame order would fix (a legitimate near-cap frame whose newline-bearing chunk also carries the next frame's leading bytes nudging the counter over the ceiling for one pipe read) is reachable only when maxLogBytes/maxValueBytes is configured within one pipe read of the 256 MiB ceiling — orders of magnitude past the 32/64 KiB defaults. The memory-safety bound against hostile input at any config takes precedence over a false reject reachable only at a pathological near-ceiling config; the counter's over-count and this trade-off are documented at the check.
Flush the two stray pipes in residual-arrival order when the combined budget crosses. Rejected: stdout and stderr are independent OS streams whose data events already interleave nondeterministically with each other and with the child's own fd-3 log frames. The seam's CodeRunResult.logs JSDoc reads "in order", which the surrounding text scopes to program-emission order WITHIN a stream — ordering ACROSS concurrent streams is inherently best-effort here, since no host-side flush order can reconstruct the true interleaving the kernel already lost, so preserving a residual's arrival order at the flush buys nothing. A fixed drain order is as valid as any. Tracking a per-residual arrival tick to drain the earlier pipe first would add a branch whose two sides fire only on the relative timing of two OS pipes, which os.sched_yield does not make deterministic, so the branch could not be covered without a flaky test — cost with no observable contract benefit.
Meter the child log ledger against the address space at runtime instead of rejecting the config at load. Rejected: an exact serialized-cost check on every child write is either a full encode — the very allocation an oversized write cannot afford, which the ledger's cheap pre-check exists to avoid — or a per-character Python loop, which burns the CPU budget (a 10 MB legitimate write hits SIGXCPU under cpuSeconds: 1). Each runtime approach trades the memory bound for another resource bound on the hot path. The address-space breach is a property of the maxLogBytes/addressSpaceMb pair, not of any particular write, so rejecting the incompatible pair once at load eliminates the whole class without any per-write cost and keeps _LogStream's original character-count buffering, which is memory-safe once the budget fits the address space.
Consequences
The seam's resolve-don't-reject contract holds on the boot-write path and the synchronous-spawn-failure path, both with measured coverage, and neither strands a staging directory. Log capture is thread-safe at the cost of one re-entrant lock acquisition per write and flush, and stray native output is delimited by its own newlines rather than by transport chunks. Fd-3 residual memory is bounded by the actual retained bytes, and both frame readers scan an accumulating frame once rather than quadratically. The output caps admit every value a frame can carry and reject a non-integer budget at load. Disposal is genuinely quiescent against a same-group survivor — bounded by graceMs + 2 * CLOSE_REAP_MARGIN_MS, zero-cost when the group is already empty, with the SIGKILL timer cleared once the group empties so a stale kill cannot strike a recycled pgid — RLIMIT enforcement keeps the strictest of configured and inherited on both soft and hard (and the SIGXCPU diagnostic no longer names a budget the host cannot guarantee), bindings called from model-created threads complete instead of timing out, and the handshake frame reader no longer burns the CPU budget on a large program. Every behavioral fix carries a test that fails without it, except the nine called out in the Problem section — the chunked frame read (a syscall-count improvement), the confirmed-empty finalize (whose only seam-observable effect freezes at signal delivery, which the pre-fix code also produced), the shared stdout/stderr budget (whose only seam-observable difference turns on nondeterministic cross-pipe arrival timing), the flush_line reorder (whose lowered peak stays within what the 12x gate already admits, so no config behaves differently), pacing binding replies (the 32.0 MiB → 0.0 MiB peak reduction lives inside the host's fd-3 writable buffer, unmeasurable through the seam), dropping a late binding resolution before snapshot (its three assertions all hold pre-fix, so it is not a fail-before case), the done-value TOCTOU pre-encoding (its concurrent-mutation race is not deterministically constructible through the seam, and the daemon-mutation regression's only assertion is probabilistic), the stray-UTF-8 budget-flush retention (a budget flush landing on a multibyte boundary is not schedulable through the seam — v8-ignored), and the late-rejection settled guard (a rejection arriving after settlement is not deterministically constructible from the seam) — so a future regression on the rest goes red.