mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-09-11 04:00:38 +00:00
docs(code-runtime-python): align stale frame-ceiling prose with the 64 MiB parse cap; pin pythonBin resolution
The review's doc drift items: the orphan receive-ceiling JSDoc, the frame-ceiling
references in index.ts/bootstrap.py/tests, and the README's 'dropped, stranding
to the wall clock' phrasing (the run now settles as a worker-exit) are all
updated to the 64 MiB FRAME_PARSE_CAP_BYTES semantics; the README notes the
>64 MiB binding-argument residual as a worker-exit trip of the same cap. A
regression case resolves a basename pythonBin against a PATH whose first entry
is relative ('.') and asserts the absolute entry is used.
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/code-runtime/code-runtime-python/README.md
|
||||
README.md: 63fd9c5b03361a1970730cbf402daf5761ea7dc4
|
||||
README.zh.md: 8c0d119de131cba93224a81f7d3267daec7c8bd3
|
||||
README.md: 507c1d78a1e25636792a077f657326f4e884ac46
|
||||
README.zh.md: e4284e852be26b0d19dc4b8bd484b1d72bac9129
|
||||
|
||||
@@ -39,7 +39,7 @@ No direct invalidation; the named consumer owns any request-prefix changes.
|
||||
- **PID-reuse protection is inert on macOS** — `readProcessStart` reads `/proc/<pid>/stat`, which Darwin does not provide, so the identity re-check that guards `killGroup` against signalling a recycled pgid always passes there; `killGroup` signals the pgid without the identity re-check on macOS rather than paying a `ps` fork on a teardown path. The process-group teardown and the `closeDeadline` bound still contain the run.
|
||||
- **C-ext stdio buffers are not drained at settlement.** The child runs with `-u` (unbuffered), so `sys.__stdout__`/`sys.__stderr__` and `os.write` bytes are visible to the host's stray capture immediately; but a C extension's private C-stdio (`FILE*`) buffering is outside the interpreter, and its unwritten bytes are lost when the host SIGTERMs the child after the done frame. Model code should flush C-level stdio explicitly before returning if it must survive.
|
||||
|
||||
- **An fd-3 frame whose raw length exceeds 64 MiB is dropped before decoding.** The receive path caps raw frames at `FRAME_PARSE_CAP_BYTES` before `toString`/`JSON.parse` (a compact wide frame near the 256 MiB wire ceiling could decode to far more host memory than the wire admitted). `maxLogBytes`/`maxValueBytes` are load-bounded to that parser cap so an honest child's frames always fit; a model-constructed binding ARGUMENT above 64 MiB (a value with no seam-level budget) is likewise dropped, stranding that call to the wall clock — an accepted residual of the same OOM guard.
|
||||
- **An fd-3 frame whose raw length exceeds 64 MiB settles the run as a worker-exit.** The receive path caps raw frames at `FRAME_PARSE_CAP_BYTES` before `toString`/`JSON.parse` (a compact wide frame could decode to far more host memory than the wire bytes admitted). `maxLogBytes`/`maxValueBytes` are load-bounded to that parser cap so an honest child's frames always fit; a model-constructed binding ARGUMENT above 64 MiB (a value with no seam-level budget) trips the same cap — an accepted residual of the OOM guard.
|
||||
|
||||
- **A truncated log's serialized array runs to `maxLogBytes` plus the marker.** The truncation marker is envelope, not payload — it rides uncharged so it can always be emitted — and the outer-array envelope is reserved one byte in the ledger. A truncated run with admitted entries therefore serializes its `logs` array to at most `maxLogBytes + marker + 1`; the marker alone fits any admissible budget (the 64-byte floor guarantees it).
|
||||
- **A descendant that calls `setsid()` / `start_new_session=True` escapes teardown.** Termination signals the child's process group with `kill(-pid)`; a descendant that moves itself into a fresh session is no longer in that group and no signal reaches it. If it also releases the inherited stdout/stderr/fd-3 pipes, the leader's `close` still settles the run, and after the `closeDeadline` bound the fiber goes quiescent while that orphan keeps running. This is the containment boundary, not a security one — model code has bash-equivalent trust, and a bash tool can `setsid` away just the same. Reaching such an orphan would require tracking every descendant pid (as the bash-local backend's process-inspector does) and is deferred; the process-group teardown reaps everything that stays in the group.
|
||||
|
||||
@@ -39,7 +39,7 @@ host 与 CPython 子进程在子进程的 fd 3 上交换一个无版本号的 JS
|
||||
- **PID 复用防护在 macOS 上失效** —— `readProcessStart` 读取 `/proc/<pid>/stat`,Darwin 不提供它,因此防止 `killGroup` 对已回收的 pgid 发信号的同一性复检在那里恒通过;`killGroup` 在 macOS 上不经同一性复检直接对 pgid 发信号,而非在拆卸路径上付出一次 `ps` fork。进程组拆卸与 `closeDeadline` 上界仍约束该次运行。
|
||||
- **C 扩展的 stdio 缓冲在结算时不被排空。** 子进程以 `-u`(无缓冲)运行,因此 `sys.__stdout__`/`sys.__stderr__` 与 `os.write` 的字节立即可见;但 C 扩展私有的 C-stdio(`FILE*`)缓冲在解释器之外,其未写出的字节会在宿主于 done 帧后 SIGTERM 子进程时丢失。模型代码若需保留,应在返回前显式 flush C 层 stdio。
|
||||
|
||||
- **原始长度超过 64 MiB 的 fd-3 帧会在解码前被丢弃。** 接收路径在 `toString`/`JSON.parse` 之前把原始帧限制在 `FRAME_PARSE_CAP_BYTES`(接近 256 MiB 线上上限的紧凑宽帧解码后可能占用远超线上字节的宿主内存)。`maxLogBytes`/`maxValueBytes` 在加载期被限制到该解析器上限,因此诚实子进程的帧总能放得下;模型构造的超过 64 MiB 的 binding 实参(一个在 seam 层没有预算的值)同样被丢弃,把该次调用搁置到墙钟——这是同一 OOM 防护的已接受残余。
|
||||
- **原始长度超过 64 MiB 的 fd-3 帧会让本次运行以 worker-exit 结算。** 接收路径在 `toString`/`JSON.parse` 之前把原始帧限制在 `FRAME_PARSE_CAP_BYTES`(紧凑宽帧解码后可能占用远超线上字节的宿主内存)。`maxLogBytes`/`maxValueBytes` 在加载期被限制到该解析器上限,因此诚实子进程的帧总能放得下;模型构造的超过 64 MiB 的 binding 实参(一个在 seam 层没有预算的值)会触发同一上限——这是该 OOM 防护的已接受残余。
|
||||
|
||||
- **截断日志的序列化数组会到 `maxLogBytes` 加标记为止。** 截断标记是 envelope 而非 payload——它不计费地随行,因此总能发出——而外层数组外壳在账本中预留了一字节。因此带已放行条目的截断运行,其 `logs` 数组序列化后至多为 `maxLogBytes + marker + 1`;标记单独能放进任何可接受的预算(64 字节下限保证这一点)。
|
||||
- **调用 `setsid()` / `start_new_session=True` 的后代会逃出 teardown。** 终止是用 `kill(-pid)` 向子进程的进程组发信号;一个把自己移入新会话的后代已不在该进程组内,任何信号都到不了它。若它同时释放了继承而来的 stdout/stderr/fd-3 管道,leader 的 `close` 仍会结算该次运行,在 `closeDeadline` 到界之后 fiber 变为完全停稳,而那个孤儿仍在运行。这是 containment 边界,而非安全边界——模型代码具有等同 bash 的信任级别,一个 bash 工具同样能 `setsid` 逃逸。要够到这样的孤儿需要追踪每一个后代 pid(如 bash-local 后端的 process-inspector 所做),此项已推迟;进程组 teardown 会回收所有留在组内的进程。
|
||||
|
||||
@@ -2169,7 +2169,7 @@ def _cap_message(message: str, max_bytes: int) -> str:
|
||||
by raw UTF-8 length: the message crosses fd 3 inside a JSON frame where
|
||||
control characters escape up to sixfold (a NUL is one raw byte but six as
|
||||
``\\u0000``), so a raw-length cap of ``maxValueBytes`` could serialize to
|
||||
roughly six times that and breach the 256 MiB frame ceiling — the silent
|
||||
roughly six times that and breach the 64 MiB frame parse cap — the silent
|
||||
``worker-exit`` inversion the load-time cap check exists to prevent, and a
|
||||
several-hundred-MiB escape allocation besides. The seam's load bound admits
|
||||
``maxValueBytes`` up to ``parse-cap - envelope`` on the premise that both the
|
||||
|
||||
@@ -192,20 +192,21 @@ function materializePyScripts(): string {
|
||||
}
|
||||
|
||||
/**
|
||||
* The fd-3 receive ceiling for one unframed line: a pure host-memory-safety
|
||||
* The fd-3 receive cap for one unframed line: a pure host-memory-safety
|
||||
* bound, NOT an output budget. Binding `call` frames legitimately carry large
|
||||
* arguments (the seam puts no byte cap on binding traffic), so the ceiling
|
||||
* arguments (the seam puts no byte cap on binding traffic), so the cap
|
||||
* must sit far above any plausible frame while still stopping a hostile
|
||||
* newline-free flood from growing the host accumulator without bound — the
|
||||
* child's RLIMIT_AS bounds the child, not the host string. 256 MiB mirrors
|
||||
* the order of the worker backend's default outer-output cap and V8's string
|
||||
* ceiling neighborhood; completion values have their own `maxValueBytes`
|
||||
* check at the `done` handler, deliberately decoupled from this. Not a config
|
||||
* knob because it is an internal framing invariant, not a deployment choice.
|
||||
* child's RLIMIT_AS bounds the child, not the host string. 64 MiB mirrors
|
||||
* the order of the worker backend's default outer-output cap while keeping
|
||||
* decode amplification (see FRAME_PARSE_CAP_BYTES) a bounded factor of the
|
||||
* wire bytes; completion values have their own `maxValueBytes` check at the
|
||||
* `done` handler, deliberately decoupled from this. Not a config knob because
|
||||
* it is an internal framing invariant, not a deployment choice.
|
||||
*/
|
||||
/**
|
||||
* A frame's RAW length is capped before JSON.parse: the 256 MiB fd-3 wire
|
||||
* ceiling bounds the bytes, not the decoded structure, and a compact wide
|
||||
* A frame's RAW length is capped before JSON.parse: the 64 MiB fd-3 frame
|
||||
* parse cap bounds the bytes, not the decoded structure, and a compact wide
|
||||
* frame near that ceiling (e.g. a huge array of tiny elements) could decode to
|
||||
* far more host memory than the wire admitted — an OOM inside the receive
|
||||
* path. 64 MiB raw admits every legal config (the widest in-tree completion
|
||||
@@ -607,7 +608,7 @@ function accrueStrayCost(buf: Buffer, state: Utf8CostState): number {
|
||||
*/
|
||||
function capMessage(message: string, maxValueBytes: number): string {
|
||||
// Code-unit bounds BEFORE any encode, so a forged done frame carrying a
|
||||
// message anywhere below the 256 MiB fd-3 frame ceiling cannot force a
|
||||
// message anywhere below the 64 MiB fd-3 frame parse cap cannot force a
|
||||
// full-length UTF-8 copy under a 32 KiB cap. One UTF-16 code unit encodes to
|
||||
// at least one UTF-8 byte and at most three: three for a non-ASCII BMP
|
||||
// character, two apiece for the pair halves sharing an astral code point's
|
||||
@@ -1077,8 +1078,8 @@ export class PythonCodeRuntime extends CodeRuntime {
|
||||
// pair contributes two of the four bytes its code point encodes to),
|
||||
// and the JSON form adds two quotes on top of the separator byte. So
|
||||
// `text.length + 3` never exceeds the true cost, and a forged `log`
|
||||
// frame carrying a control-heavy string anywhere below the 256 MiB
|
||||
// frame ceiling truncates here instead of allocating a
|
||||
// frame carrying a control-heavy string anywhere below the 64 MiB
|
||||
// frame parse cap truncates here instead of allocating a
|
||||
// hundreds-of-megabytes escaped copy under a small maxLogBytes.
|
||||
if (text.length + 3 > logBudget) {
|
||||
logsTruncated = true
|
||||
@@ -1321,8 +1322,8 @@ export class PythonCodeRuntime extends CodeRuntime {
|
||||
// pipe read of the 64 MiB cap, orders of magnitude past the 32/64 KiB
|
||||
// defaults.
|
||||
//
|
||||
// The cap used HERE is FRAME_PARSE_CAP_BYTES, not the 256 MiB wire
|
||||
// ceiling, and ONLY when the held bytes are still a single unframed
|
||||
// The cap used HERE is FRAME_PARSE_CAP_BYTES, not the old 256 MiB
|
||||
// wire ceiling, and ONLY when the held bytes are still a single unframed
|
||||
// line (this chunk carries no newline, and earlier newline-bearing
|
||||
// chunks were joined immediately): a frame between 64 MiB and the
|
||||
// ceiling would otherwise be fully `Buffer.concat`-ed (a second copy
|
||||
@@ -1360,7 +1361,7 @@ export class PythonCodeRuntime extends CodeRuntime {
|
||||
// 53.7 GB that way, and 64 MiB copies 2.2 TB. Here each byte is copied
|
||||
// once into its block and never again, so the total stays linear, and the
|
||||
// block list is itself bounded — every block holds at least
|
||||
// `MAX_PENDING_CHUNKS - 1` bytes, so reaching the 256 MiB ceiling admits
|
||||
// `MAX_PENDING_CHUNKS - 1` bytes, so reaching the 64 MiB cap admits
|
||||
// at most a few hundred thousand of them.
|
||||
// Sealing runs ONLY on a newline-free chunk, and after the newline
|
||||
// branch below: a chunk carrying a newline must reach the join (and its
|
||||
@@ -1453,7 +1454,7 @@ export class PythonCodeRuntime extends CodeRuntime {
|
||||
// next legitimate id exactly `nextCallId`.
|
||||
//
|
||||
// Retaining a set instead let a program write an unbounded run of unique
|
||||
// forged ids, each below the 256 MiB per-frame ceiling so nothing
|
||||
// forged ids, each below the 64 MiB per-frame parse cap so nothing
|
||||
// rejected them, and grow host memory for the whole run. Accepting any
|
||||
// id above a high-water mark would have been just as wrong in the other
|
||||
// direction: one forged `{"id": 9999}` would starve every honest call
|
||||
@@ -1513,7 +1514,7 @@ export class PythonCodeRuntime extends CodeRuntime {
|
||||
// completion must cross intact rather than dying on stringify
|
||||
// recursion; bounded because it stops at the cap without
|
||||
// materializing the encoding, rejecting a forged value anywhere
|
||||
// below the 256 MiB frame ceiling before it forces host-side copies.
|
||||
// below the 64 MiB frame parse cap before it forces host-side copies.
|
||||
// The seam forbids substituting a rendered/truncated value, so an
|
||||
// oversized value fails the run as output-limit and a non-lossless
|
||||
// number as invalid-output. The value is JSON-plain by construction
|
||||
@@ -1535,8 +1536,8 @@ export class PythonCodeRuntime extends CodeRuntime {
|
||||
const fn = record && Object.hasOwn(record, message.name) ? record[message.name] : undefined
|
||||
if (typeof fn !== 'function') {
|
||||
// `call.global` and `call.name` are attacker-controlled strings
|
||||
// with no byte cap of their own — only the 256 MiB fd-3 frame
|
||||
// ceiling — so each is sliced to `maxValueBytes` CODE UNITS
|
||||
// with no byte cap of their own — only the 64 MiB fd-3 frame
|
||||
// parse cap — so each is sliced to `maxValueBytes` CODE UNITS
|
||||
// BEFORE it reaches the template. Interpolating them whole would
|
||||
// copy them into the message, `JSON.stringify` would copy the
|
||||
// escaped form, `encodeJsonPlain` the frame, and the pipe write
|
||||
|
||||
@@ -103,7 +103,7 @@ describe('PythonCodeRuntime — seam descriptors and misuse', () => {
|
||||
|
||||
it('rejects an output cap whose payload could not cross the frame ceiling', async () => {
|
||||
// The caps budget a payload that must arrive inside ONE fd-3 frame, and the
|
||||
// 256 MiB framing ceiling is fixed. A larger cap is unsatisfiable rather
|
||||
// 64 MiB frame parse cap is fixed. A larger cap is unsatisfiable rather
|
||||
// than generous: a completion the cap admits arrives as an over-ceiling
|
||||
// frame and fails the run as `worker-exit`, inverting the `output-limit`
|
||||
// the cap describes. Both budgets are metered in already-escaped serialized
|
||||
@@ -141,6 +141,27 @@ describe('PythonCodeRuntime — seam descriptors and misuse', () => {
|
||||
.rejects.toThrow(/pythonBin must be a non-empty path without NUL bytes/)
|
||||
})
|
||||
|
||||
it('skips relative PATH entries when resolving a basename pythonBin', async () => {
|
||||
// resolvePythonBin must return an absolute path: a RELATIVE PATH entry
|
||||
// ('.' here) would otherwise resolve the basename against the host CWD —
|
||||
// spawn() then tries './python3' from the test process's directory, where
|
||||
// no interpreter exists, surfacing an ENOENT worker-exit instead of a
|
||||
// normal run. The relative entry is skipped and the absolute entry used.
|
||||
const cp = await import('node:child_process')
|
||||
const nodePath = await import('node:path')
|
||||
const pythonDir = nodePath.dirname(cp.execFileSync('which', ['python3'], { encoding: 'utf8' }).trim())
|
||||
vi.stubEnv('PATH', `.:${pythonDir}`)
|
||||
try {
|
||||
const { runtime, fiber } = await setup({ pythonBin: 'python3', maxWallMs: 30_000 })
|
||||
const result = await runtime.run({ program: 'return 1', bindings: [] })
|
||||
expect(result.error).toBeUndefined()
|
||||
expect(result.value).toBe(1)
|
||||
await fiber.dispose()
|
||||
} finally {
|
||||
vi.unstubAllEnvs()
|
||||
}
|
||||
}, 45_000)
|
||||
|
||||
it('rejects a timer budget setTimeout would silently clamp to 1 ms', async () => {
|
||||
// Node stores a setTimeout delay as a signed 32-bit value and substitutes
|
||||
// 1 ms for anything larger, inverting the knob's meaning: a huge maxWallMs
|
||||
@@ -1735,7 +1756,7 @@ describe('PythonCodeRuntime — programs and bindings', () => {
|
||||
}, 15_000)
|
||||
|
||||
it('rejects an fd-3 frame whose raw length exceeds the parse cap before joining it', async () => {
|
||||
// The 256 MiB wire ceiling bounds the RAW frame bytes, not the decoded
|
||||
// The 64 MiB frame parse cap bounds the RAW frame bytes, not the decoded
|
||||
// structure; a compact wide frame near that ceiling could decode to far
|
||||
// more host memory. The unframed-buffer counter is checked against
|
||||
// FRAME_PARSE_CAP_BYTES BEFORE the Buffer.concat join, so an oversized
|
||||
@@ -2186,7 +2207,7 @@ describe('PythonCodeRuntime — programs and bindings', () => {
|
||||
|
||||
it('bounds an unknown-binding diagnostic built from a forged call frame', async () => {
|
||||
// `call.global` and `call.name` carry no byte cap of their own, only the
|
||||
// 256 MiB fd-3 frame ceiling, and the reply interpolated them raw: one copy
|
||||
// 64 MiB fd-3 frame parse cap, and the reply interpolated them raw: one copy
|
||||
// into the template result, one into the `JSON.stringify` escape, one into
|
||||
// the `encodeJsonPlain` frame, one into the pipe write. Slicing each field
|
||||
// to `maxValueBytes` code units first makes an 8 MiB forged name a
|
||||
@@ -3045,7 +3066,7 @@ describe('PythonCodeRuntime — hostile peer', () => {
|
||||
it('drops forged call frames whose ids are not the next in sequence, retaining no per-id state', async () => {
|
||||
// The host used to remember every answered id in a Set, so a program could
|
||||
// write an unbounded run of unique forged ids — each frame far below the
|
||||
// 256 MiB ceiling, so nothing rejected them — and grow host memory for the
|
||||
// 64 MiB cap, so nothing rejected them — and grow host memory for the
|
||||
// whole run. Ids are consecutive from 0, so one counter replaces the set.
|
||||
//
|
||||
// The discriminator is that the forgeries must not be answered. Each names a
|
||||
@@ -3764,7 +3785,7 @@ describe('PythonCodeRuntime — hostile peer', () => {
|
||||
// The marker branch bypasses `admit`, so retaining the frame's own text put
|
||||
// attacker-controlled bytes into `logs` with no cap at all: measured, a 1 MiB
|
||||
// forged text was retained whole under `maxLogBytes: 64`, and the only bound
|
||||
// left was the 256 MiB frame ceiling. The host emits its own marker instead,
|
||||
// left was the 64 MiB frame parse cap. The host emits its own marker instead,
|
||||
// so the retained size is fixed regardless of what the program sent.
|
||||
const forgedBytes = 1024 * 1024
|
||||
const { runtime } = await setup({ maxLogBytes: 64, maxWallMs: 20_000 })
|
||||
|
||||
Reference in New Issue
Block a user