fix(code-runtime-python): charge illegal UTF-8 by its U+FFFD width on both log paths

The host stray-capture cost function charged illegal UTF-8 bytes (0x80-0xC1,
0xF5-0xFF, and orphaned multibyte leads) the raw 1, but toString('utf8')
renders each as U+FFFD (3 serialized bytes). A b"\xff" flood was undercounted
threefold, so the residual grew to a full budget's worth of raw bytes before
flushing and, near a large maxLogBytes, expanded toward a ~1 GiB peak in the
flush's concat plus toString. Replace serializedBufferCost with accrueStrayCost,
a cross-chunk UTF-8 walker that charges each byte its decoded serialized width;
carry its sequence state on each StrayBuffer.

The child _LogStream had the same-family bug: its early-flush trigger compared
_pending_chars (character count) against remaining (a serialized-byte budget),
so a 30M-NUL newline-free flood stayed under a 50 MB char trigger yet encoded to
~180 MB at settlement, breaching RLIMIT_AS as worker-exit. Track _pending_cost
via the _JSON_BYTE_COST table and trigger on it; keep _pending_chars for the
char-based slice bounds.

Correct the note's surrogate claim (only the string-walking jsonStringCostUpTo
charges a lone surrogate six bytes; the byte walker never sees one). Shrink the
post-truncation fixture below PIPE_BUF for a deterministic single callback. List
the shared stdout/stderr budget as a third honest fail-before exception
(cross-pipe arrival timing is nondeterministic). Add illegal-UTF-8,
broken-multibyte, and child-log-flood regression tests; sync the zh pair.
This commit is contained in:
Chinesezjc
2026-08-31 14:22:37 +08:00
committed by Tianyi Cui
parent 45814baf26
commit dbff8ffba3
6 changed files with 224 additions and 44 deletions
@@ -747,6 +747,70 @@ describe('PythonCodeRuntime — programs and bindings', () => {
expect(result.logs.at(-1)).toBe(logTruncationMarker(4096))
})
it('bounds a newline-free NUL flood through sys.stdout by serialized cost, not char count', async () => {
// `_LogStream` (the child's sys.stdout wrapper) buffers newline-free writes
// and early-flushes once the pending tail can no longer fit the ledger.
// Charging that trigger by CHARACTER count undercharged a control-char flood
// by up to 6x: 30M NUL chars stay under a 50 MB char-count trigger yet
// serialize to ~180 MB, which the settlement flush then allocated at once —
// breaching a 64 MB RLIMIT_AS and surfacing as worker-exit instead of the
// truncation marker. Driving the flood through sys.stdout.write (not
// os.write, which bypasses the wrapper into host stray capture) exercises the
// in-child stream. On Linux CI the pre-fix trigger dies on RLIMIT_AS; the
// serialized-cost trigger flushes while running, so the run completes and
// ends at the marker. (RLIMIT_AS is skipped on Darwin — bootstrap.py — so the
// worker-exit repro is Linux-only; locally this asserts the happy path.)
const { runtime } = await setup({ maxLogBytes: 50_000_000, addressSpaceMb: 64, maxWallMs: 20_000 })
const result = await runtime.run({
program: ['import sys', 'sys.stdout.write("\\x00" * 30_000_000)', 'return None'].join('\n'),
bindings: [],
})
expect(result.error).toBeUndefined()
expect(result.logs.at(-1)).toBe(logTruncationMarker(50_000_000))
})
it('bounds an illegal-UTF-8 native residual by its U+FFFD-decoded cost', async () => {
// Every 0xFF byte is illegal in any UTF-8 sequence, so `toString('utf8')`
// renders each as U+FFFD (3 serialized bytes). `accrueStrayCost` must charge
// that 3, not the raw 1: otherwise the newline-free residual grows to a full
// budget's worth of RAW bytes before flushing — a ~3x undercount that near a
// large maxLogBytes retains hundreds of MiB then expands toward a ~1 GiB peak
// in flushStray's concat + toString. Paced single-byte writes (each its own
// `data` chunk, like the sealing case) expose the sub-chunk accrual: charged
// at 3 the residual crosses a 3072-byte budget after ~1024 bytes and flushes;
// charged at 1 it would need ~3072 bytes, so the peak residual triples. The
// largest merged buffer is the discriminator.
const realConcat = Buffer.concat.bind(Buffer)
let maxConcat = 0
Buffer.concat = (list: readonly Uint8Array[], total?: number): Buffer<ArrayBuffer> => {
const merged = realConcat(list, total)
if (merged.length > maxConcat) maxConcat = merged.length
return merged
}
let result: CodeRunResult
try {
const { runtime } = await setup({ maxLogBytes: 3072, maxWallMs: 30_000 })
result = await runtime.run({
program: [
'import os',
'for _ in range(6000):',
' os.write(1, b"\\xff")',
' os.sched_yield()',
'return None',
].join('\n'),
bindings: [],
})
} finally {
Buffer.concat = realConcat
}
expect(result.error).toBeUndefined()
expect(result.logs.at(-1)).toBe(logTruncationMarker(3072))
// Charged at 3, the residual flushes around 1024 raw bytes; the largest
// merged buffer stays well under 2048. A raw-byte undercount would let it
// reach ~3072 before flushing, so 2048 discriminates.
expect(maxConcat).toBeLessThan(2048)
})
it('charges a lone surrogate its full six escaped bytes, not three', async () => {
// A forged `log` frame carrying `\ud800` escapes materializes lone
// surrogates after JSON.parse. `Buffer.byteLength` of U+FFFD is 3, but
@@ -779,10 +843,14 @@ describe('PythonCodeRuntime — programs and bindings', () => {
// exhausts maxLogBytes: the first line's admit truncates and marks the
// ledger, and the second line's admit — reached in the same `data` callback
// — must be the post-truncation no-op. Proves that branch is exercised, so
// it carries no v8-ignore.
// it carries no v8-ignore. Kept to 109 bytes (< the smallest PIPE_BUF, 512 on
// macOS) so the whole payload lands in ONE atomic write and one `data`
// callback — the two newlines cannot split across callbacks and leave the
// branch un-exercised, which would be a hard-to-attribute per-file coverage
// flake. 103 payload bytes still exceed the 64-byte budget, so it truncates.
const { runtime } = await setup({ maxLogBytes: 64 })
const result = await runtime.run({
program: ['import os', 'os.write(1, b"A" * 5000 + b"\\nSECOND\\n")', 'return None'].join('\n'),
program: ['import os', 'os.write(1, b"A" * 100 + b"\\nSECOND\\n")', 'return None'].join('\n'),
bindings: [],
})
expect(result.error).toBeUndefined()
@@ -790,6 +858,29 @@ describe('PythonCodeRuntime — programs and bindings', () => {
expect(result.logs.join('\n')).not.toContain('SECOND')
})
it('charges a broken multibyte sequence its U+FFFD bytes, split across pipe chunks', async () => {
// A 3-byte lead (0xE4) whose continuation never arrives — the next byte is a
// fresh ASCII 'A' — must be costed as U+FFFD (3) for the orphaned lead, not
// folded into a phantom character. Driven byte-by-byte so the lead and the
// breaking byte land in separate `data` chunks, exercising accrueStrayCost's
// cross-chunk broken-sequence branch. The run completes and the bytes are
// captured (rendered U+FFFD by toString), proving the walk resynchronizes.
const { runtime } = await setup({ maxLogBytes: 1024 })
const result = await runtime.run({
program: [
'import os',
'os.write(1, b"\\xe4")',
'os.sched_yield()',
'os.write(1, b"A\\n")',
'return None',
].join('\n'),
bindings: [],
})
expect(result.error).toBeUndefined()
expect(result.logs.join('')).toContain('A')
expect(result.logs.join('')).toContain('')
})
it('charges the exact serialized cost of short-escape and quote/backslash characters', async () => {
// Exercises every branch of jsonStringCostUpTo's per-character cost: a tab
// and other C0 controls with short JSON forms (\t etc., 2 bytes), a quote