diff --git a/.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.i18n.yaml b/.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.i18n.yaml new file mode 100644 index 0000000000..76dcf9e149 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.md +2026-07-31-code-runtime-python-settlement-fixes.md: ef7772c10cece314bc6e77da55525f2521101cec +2026-07-31-code-runtime-python-settlement-fixes.zh.md: 07110b18783988c69a5e1e1c976db2fec9e234c1 diff --git a/.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.md b/.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.md new file mode 100644 index 0000000000..ef7772c10c --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.md @@ -0,0 +1,105 @@ +# Agent Note: Three settlement and framing fixes in the CPython backend + +Status: implemented + +English | [中文](2026-07-31-code-runtime-python-settlement-fixes.zh.md) + +## Problem + +The [CPython subprocess backend](2026-07-17-code-runtime-python.md) for Code Mode +resolves every program outcome as a `CodeRunResult` and rejects `run()` only for +seam misuse. Three defects broke that contract in ways unit coverage did not +surface, because each hid behind a `/* v8 ignore */`, a captured-callable +comment that read as a fix but was not, or a memory effect invisible through the +seam. They were found by review of the backend as it stood, not by a failing +test, so each fix ships with a test that fails without it. + +## Decision + +Three independent corrections, each in the package that owns the defect. + +### Boot-write failure no longer rejects run() + +In [`src/index.ts`](../../../../packages/code-runtime/code-runtime-python/src/index.ts) +the fd-3 boot-frame write is the last statement of `run()`'s synchronous setup. +Its `catch` calls `finish()`, and `finish()` reads `wallTimer` and `onAbort` and +— through `settle()` — `live`. Those bindings are `const` and were declared +AFTER the boot-write, so on a synchronous write failure `finish()` touched them +in their temporal dead zone and threw a `ReferenceError`. That escaped the +Promise executor and REJECTED `run()`, violating the seam's "outcomes resolve" +contract: the caller saw a thrown error instead of the `worker-exit` the catch +constructs. The boot-write block is now emitted after `wallTimer`, `onAbort`, and +`live` are initialized, and the `/* v8 ignore */` that had hidden the branch from +coverage is removed so the catch is measured. + +### Log capture is serialized against settlement + +In [`py/bootstrap.py`](../../../../packages/code-runtime/code-runtime-python/py/bootstrap.py) +the settlement `flush_out()`/`flush_err()` on the main coroutine read and clear +each stream's `_pending` list and mutate the shared `LogBuffer` ledger. Model +code may start daemon threads whose `print`/`write` mutate the same state +concurrently. Capturing the bound method (`out_stream.flush_line`) fixed only +WHICH callable settlement invokes, not what it reads mid-flight: an interleaved +flush could join a `_pending` list being mutated under it, corrupting the ledger +and costing the `done` frame — stranding the run to the wall clock. `LogBuffer` +now owns one re-entrant lock shared by both streams; `_LogStream.write` and +`flush_line`, and `LogBuffer.push`, take it, so the whole read-modify-write is +atomic across threads. + +### Fd-3 residual is copied, not viewed + +Also in `src/index.ts`, after the newline loop over a `Buffer.concat` of the +pending fd-3 chunks, the leftover partial line was carried forward as the +`subarray` VIEW it was sliced to. A view keeps the entire concat backing +allocation alive, so a large frame followed by a tiny trailing fragment pinned a +whole frame's worth of memory while `pendingBytes` — set to the fragment's +length — reported far less than was retained. The residual is now detached into a +fresh right-sized `Buffer` via the exported `detachResidual` helper, letting the +concat allocation be collected and keeping `pendingBytes` an honest measure. + +## Testing + +- `tests/boot-write-failure.spec.ts` mocks `spawn` so the fd-3 pipe throws on the + boot write — the one path a real subprocess cannot be coerced into — and + asserts `run()` resolves a `worker-exit` rather than rejecting. Isolated in its + own spec so the real-subprocess suite is untouched. +- `tests/residual-detach.spec.ts` unit-tests `detachResidual`: the carried copy + equals the residual, owns a backing store sized to its own length, and does not + share the source frame's `ArrayBuffer`. +- `tests/runtime.spec.ts` adds a real-subprocess case where four daemon threads + emit unterminated writes up to the moment the body returns and settlement + flushes, repeated so the interleave lands; the run must complete cleanly. A + pure data race has no single bad input to reject, so this maximizes overlap + rather than asserting a deterministic rejection. + +## Alternatives considered + +**Leave the boot-write `/* v8 ignore */` and fix only the ordering.** Rejected: +the ignore is what let the TDZ regression ship uncaught. Removing it makes the +catch a measured branch, so per-file 100% coverage now proves the failure path +is exercised. + +**Fix the flush race by capturing more bound methods.** Rejected: this is the +approach that already failed. Binding a callable fixes reference resolution, not +concurrent access to the mutable state the callable reads. Only mutual exclusion +over the shared ledger closes the race. + +**Guard the residual with a size threshold (copy only large frames).** Rejected: +the branch runs once per newline-bearing read, the copy is bounded by the +residual's own length (always a partial line), and a threshold adds a tunable +and a second code path for no measurable saving. An unconditional right-sized +copy is simpler and always correct. + +**Assert the residual memory effect through the seam.** Rejected: the retained +allocation is not observable through `CodeRunResult`, so a black-box test could +not distinguish fixed from unfixed. Extracting `detachResidual` makes the +backing-store invariant a deterministic unit test instead. + +## Consequences + +The seam's resolve-don't-reject contract now holds on the boot-write path, and +its coverage is measured rather than ignored. Log capture is thread-safe at the +cost of one re-entrant lock acquisition per write and flush — negligible against +the os.write already on that path. Fd-3 residual memory is bounded by the actual +retained bytes, and `pendingBytes` measures what it claims. Each fix carries a +test that fails without it, so a future regression on any of the three goes red. diff --git a/.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.zh.md b/.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.zh.md new file mode 100644 index 0000000000..07110b1878 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.zh.md @@ -0,0 +1,45 @@ +# Agent Note: CPython 后端的三处结算与分帧修复 + +Status: implemented + +[English](2026-07-31-code-runtime-python-settlement-fixes.md) | 中文 + +## Problem + +用于 Code Mode 的 [CPython 子进程后端](2026-07-17-code-runtime-python.md)把每个程序结果都 resolve 成一个 `CodeRunResult`,仅在 seam 被误用时才 reject `run()`。三个缺陷以单元测试覆盖率无法暴露的方式破坏了这一契约,因为它们各自藏在一处 `/* v8 ignore */` 之后、藏在一条读起来像修复但实际并非修复的"捕获可调用对象"注释之后,或藏在一处透过 seam 不可见的内存效应之后。这些缺陷是通过审查当时的后端代码发现的,而非由某个失败的测试发现,因此每处修复都附带一个在缺少该修复时会失败的测试。 + +## Decision + +三处相互独立的修正,各自位于拥有对应缺陷的包中。 + +### Boot-write failure no longer rejects run() + +在 [`src/index.ts`](../../../../packages/code-runtime/code-runtime-python/src/index.ts) 中,fd-3 引导帧写入是 `run()` 同步初始化阶段的最后一条语句。它的 `catch` 会调用 `finish()`,而 `finish()` 读取 `wallTimer` 和 `onAbort`,并通过 `settle()` 读取 `live`。这些绑定是 `const`,且声明在引导写入之后,因此在同步写入失败时,`finish()` 会在它们处于暂时性死区(temporal dead zone)时访问它们,从而抛出一个 `ReferenceError`。该错误逃出了 Promise executor 并 reject 了 `run()`,违反了 seam 的"结果一律 resolve"契约:调用方看到的是一个被抛出的错误,而不是 catch 构造的 `worker-exit`。现在引导写入代码块被放到 `wallTimer`、`onAbort` 和 `live` 初始化之后,并且那处曾把该分支从覆盖率中隐藏的 `/* v8 ignore */` 已被移除,从而使该 catch 被纳入度量。 + +### Log capture is serialized against settlement + +在 [`py/bootstrap.py`](../../../../packages/code-runtime/code-runtime-python/py/bootstrap.py) 中,主协程上的结算 `flush_out()`/`flush_err()` 会读取并清空各个流的 `_pending` 列表,并修改共享的 `LogBuffer` 账本。模型代码可能启动一些 daemon 线程,其 `print`/`write` 会并发地修改同一状态。捕获绑定方法(`out_stream.flush_line`)只解决了结算调用哪个可调用对象的问题,而没有解决它在执行途中读取什么的问题:一次交错的 flush 可能拼接一个正在其下被修改的 `_pending` 列表,从而破坏账本并丢失 `done` 帧,使该次运行一直拖到墙钟超时。现在 `LogBuffer` 持有一把由两个流共享的可重入锁;`_LogStream.write` 和 `flush_line`,以及 `LogBuffer.push`,都会获取该锁,因此整个读-改-写过程在多线程间是原子的。 + +### Fd-3 residual is copied, not viewed + +同样在 `src/index.ts` 中,在对待处理 fd-3 分片的 `Buffer.concat` 结果按换行符做循环之后,剩余的不完整行被以它被切出的 `subarray` 视图形式向前传递。视图会使整个 concat 的底层分配保持存活,因此一个大帧后面跟着一个极小的尾部片段,会钉住整整一帧大小的内存,而 `pendingBytes`(被设为该片段的长度)报告的值远小于实际保留的内存。现在,残余数据通过导出的 `detachResidual` 辅助函数被分离到一个大小恰当的新 `Buffer` 中,从而让 concat 分配得以被回收,并使 `pendingBytes` 成为一个诚实的度量值。 + +## Testing + +- `tests/boot-write-failure.spec.ts` 对 `spawn` 做 mock,使 fd-3 管道在引导写入时抛出异常(这是真实子进程无法被迫进入的唯一路径),并断言 `run()` resolve 出一个 `worker-exit` 而非 reject。它被隔离在自己的 spec 中,因此真实子进程测试套件不受影响。 +- `tests/residual-detach.spec.ts` 对 `detachResidual` 做单元测试:向前传递的副本与残余数据相等、拥有一个大小与其自身长度一致的底层存储,并且不与源帧的 `ArrayBuffer` 共享。 +- `tests/runtime.spec.ts` 新增一个真实子进程用例:四个 daemon 线程持续发出未结束的写入,直到函数体返回、结算执行 flush 的那一刻,并反复运行以让交错真正出现;该次运行必须干净地完成。纯数据竞态没有单一的坏输入可供 reject,因此该测试最大化重叠而非断言一个确定性的 reject。 + +## Alternatives considered + +**保留引导写入处的 `/* v8 ignore */`,只修复顺序。** 已否决:正是那处 ignore 让这个 TDZ 回归得以未被发现地进入代码库。移除它使该 catch 成为被度量的分支,因此按文件计的 100% 覆盖率现在能证明该失败路径确实被执行。 + +**通过捕获更多绑定方法来修复 flush 竞态。** 已否决:这正是已经失败过的做法。绑定一个可调用对象解决的是引用解析,而不是对该可调用对象所读取的可变状态的并发访问。只有对共享账本施加互斥才能消除该竞态。 + +**用大小阈值来保护残余数据(只复制大帧)。** 已否决:该分支在每次包含换行符的读取时运行一次,复制的规模受残余数据自身长度约束(始终是一个不完整行),而阈值会引入一个可调参数和第二条代码路径,却换不来任何可度量的节省。无条件地做大小恰当的复制更简单,且始终正确。 + +**通过 seam 断言残余数据的内存效应。** 已否决:被保留的分配透过 `CodeRunResult` 不可观测,因此黑盒测试无法区分已修复与未修复。转而抽取出 `detachResidual`,把底层存储的不变量变成一个确定性的单元测试。 + +## Consequences + +现在 seam 的"只 resolve、不 reject"契约在引导写入路径上得以成立,且其覆盖率是被度量而非被忽略的。日志捕获现在是线程安全的,代价是每次写入和 flush 都要获取一次可重入锁,这相对于该路径上已有的 os.write 可以忽略不计。fd-3 残余数据的内存现在受实际保留的字节数约束,且 `pendingBytes` 度量的正是它所声称的值。每处修复都附带一个在缺少它时会失败的测试,因此这三处中任何一处未来若发生回归都会变红。 diff --git a/docs/config-catalog.md b/docs/config-catalog.md index ec077edd10..59b9ef4acf 100644 --- a/docs/config-catalog.md +++ b/docs/config-catalog.md @@ -356,6 +356,47 @@ export interface Config { Source: [`packages/client/hmr/src/index.ts:31`](../packages/client/hmr/src/index.ts) + + +## `@deepseek-ai/dsh-code-runtime-python` + +```ts config-catalog +/** Plugin config: every cap, changeable from `cordis.yml` (no hardcoded tunables). */ +export interface Config { + /** + * RLIMIT_CPU in whole seconds (a positive integer — `setrlimit` in the child + * rejects a float). The child sets the soft limit to `cpuSeconds` and the + * hard limit to `cpuSeconds + 1`: the kernel delivers SIGXCPU at the soft + * limit, which the host classifies as a `timeout`; the +1s hard limit is a + * SIGKILL backstop for a program that traps SIGXCPU. Granularity is seconds — + * a coarser counterpart to the worker backend's millisecond `computeMs`. + */ + cpuSeconds?: number + /** Wall-clock ceiling in milliseconds; backstops CPU time for programs awaiting a promise nobody resolves. */ + maxWallMs?: number + /** + * RLIMIT_AS in mebibytes; caps address space so a runaway allocation fails + * cleanly. Not applied on Darwin, where the dyld shared cache mapped into + * every process at exec exceeds any practical cap and the kernel rejects + * the call; `cpuSeconds` and `maxWallMs` still bound the run there. + */ + addressSpaceMb?: number + /** Shared byte budget for captured log text (host-side ledger). */ + maxLogBytes?: number + /** Byte cap for the completion value. */ + maxValueBytes?: number + /** SIGTERM→SIGKILL grace period on kill, matching bash-local's default. */ + graceMs?: number + /** + * Absolute path or basename of the CPython interpreter to spawn. Resolved + * through `PATH` when a basename is given. + */ + pythonBin?: string +} +``` + +Source: [`packages/code-runtime/code-runtime-python/src/index.ts:44`](../packages/code-runtime/code-runtime-python/src/index.ts) + ## `@deepseek-ai/dsh-code-runtime-worker-thread` @@ -3403,7 +3444,6 @@ Imported as libraries by other packages; a `cordis.yml` cannot load them. - `@deepseek-ai/dsh-client-ui-slots` ([`packages/client/ui-slots/src/index.ts`](../packages/client/ui-slots/src/index.ts)) - `@deepseek-ai/dsh-client-web` ([`packages/client/web/src/index.ts`](../packages/client/web/src/index.ts)) - `@deepseek-ai/dsh-cmdline` ([`packages/boot/cmdline/src/index.ts`](../packages/boot/cmdline/src/index.ts)) -- `@deepseek-ai/dsh-code-runtime-python` ([`packages/code-runtime/code-runtime-python/src/index.ts`](../packages/code-runtime/code-runtime-python/src/index.ts)) - `@deepseek-ai/dsh-deque` ([`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts)) - `@deepseek-ai/dsh-experimental-agent-team-profile` ([`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts)) - `@deepseek-ai/dsh-experimental-agent-team-web-profile` ([`packages/experimental/agent-team-web-profile/src/index.ts`](../packages/experimental/agent-team-web-profile/src/index.ts)) diff --git a/docs/module-graph.i18n.yaml b/docs/module-graph.i18n.yaml index cbec50a70c..281ec87fee 100644 --- a/docs/module-graph.i18n.yaml +++ b/docs/module-graph.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/module-graph.md -module-graph.md: 2229efcd7e76b3b224eb307ee7de9ecea0ad85d7 -module-graph.zh.md: 3edca4ea261f68593973149b21e72e4a25d7a504 +module-graph.md: e4bbd01f7000e88a4dc982f5f8c7986176968ec6 +module-graph.zh.md: f046b59e7412bb3db0ea66fccc39294e90c66f1d diff --git a/docs/module-graph.md b/docs/module-graph.md index 2229efcd7e..e4bbd01f70 100644 --- a/docs/module-graph.md +++ b/docs/module-graph.md @@ -378,7 +378,6 @@ flowchart TD pkg_sdk_app --> pkg_invariants pkg_sdk_minimal --> pkg_invariants pkg_code_runtime --> pkg_invariants - pkg_code_runtime_python --> pkg_invariants pkg_credentials --> pkg_invariants pkg_e2b --> pkg_invariants pkg_experimental_agent_team_profile --> pkg_invariants @@ -472,6 +471,9 @@ flowchart TD pkg_app_boot --> pkg_invariants pkg_app_boot --> pkg_launch_environment pkg_app_boot --> pkg_system_prompt + pkg_code_runtime_python --> pkg_invariants + pkg_code_runtime_python --> pkg_session + pkg_code_runtime_python --> pkg_timeout pkg_code_runtime_worker_thread --> pkg_code_runtime pkg_code_runtime_worker_thread --> pkg_invariants pkg_code_runtime_worker_thread --> pkg_session @@ -1372,7 +1374,6 @@ flowchart TD | [`sdk-app`](../packages/bundle/sdk-app) | `bundle` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`sdk-minimal`](../packages/bundle/sdk-minimal) | `bundle` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`code-runtime`](../packages/code-runtime/code-runtime) | `code-runtime` | [`invariants`](../packages/runtime-diagnostics/invariants) | -| [`code-runtime-python`](../packages/code-runtime/code-runtime-python) | `code-runtime` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`credentials`](../packages/credentials/credentials) | `credentials` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`e2b`](../packages/e2b/e2b) | `e2b` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | [`invariants`](../packages/runtime-diagnostics/invariants) | @@ -1415,6 +1416,7 @@ flowchart TD | [`spill`](../packages/spill/spill) | `spill` | [`brand`](../packages/util/brand), [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) | | [`attachment-local`](../packages/attachment/attachment-local) | `attachment` | [`attachment`](../packages/attachment/attachment), [`home-paths`](../packages/util/home-paths), [`invariants`](../packages/runtime-diagnostics/invariants) | | [`app-boot`](../packages/boot/app-boot) | `boot` | [`home-paths`](../packages/util/home-paths), [`invariants`](../packages/runtime-diagnostics/invariants), [`launch-environment`](../packages/util/launch-environment), [`system-prompt`](../packages/core/system-prompt) | +| [`code-runtime-python`](../packages/code-runtime/code-runtime-python) | `code-runtime` | [`invariants`](../packages/runtime-diagnostics/invariants), [`session`](../packages/core/session), [`timeout`](../packages/util/timeout) | | [`code-runtime-worker-thread`](../packages/code-runtime/code-runtime-worker-thread) | `code-runtime` | [`code-runtime`](../packages/code-runtime/code-runtime), [`invariants`](../packages/runtime-diagnostics/invariants), [`session`](../packages/core/session), [`timeout`](../packages/util/timeout) | | [`persona`](../packages/preset/persona) | `preset` | [`invariants`](../packages/runtime-diagnostics/invariants), [`system-prompt`](../packages/core/system-prompt) | | [`sandbox`](../packages/sandbox/sandbox) | `sandbox` | [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) | diff --git a/docs/module-graph.zh.md b/docs/module-graph.zh.md index 3edca4ea26..f046b59e74 100644 --- a/docs/module-graph.zh.md +++ b/docs/module-graph.zh.md @@ -380,7 +380,6 @@ flowchart TD pkg_sdk_app --> pkg_invariants pkg_sdk_minimal --> pkg_invariants pkg_code_runtime --> pkg_invariants - pkg_code_runtime_python --> pkg_invariants pkg_credentials --> pkg_invariants pkg_e2b --> pkg_invariants pkg_experimental_agent_team_profile --> pkg_invariants @@ -474,6 +473,9 @@ flowchart TD pkg_app_boot --> pkg_invariants pkg_app_boot --> pkg_launch_environment pkg_app_boot --> pkg_system_prompt + pkg_code_runtime_python --> pkg_invariants + pkg_code_runtime_python --> pkg_session + pkg_code_runtime_python --> pkg_timeout pkg_code_runtime_worker_thread --> pkg_code_runtime pkg_code_runtime_worker_thread --> pkg_invariants pkg_code_runtime_worker_thread --> pkg_session @@ -1374,7 +1376,6 @@ flowchart TD | [`sdk-app`](../packages/bundle/sdk-app) | `bundle` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`sdk-minimal`](../packages/bundle/sdk-minimal) | `bundle` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`code-runtime`](../packages/code-runtime/code-runtime) | `code-runtime` | [`invariants`](../packages/runtime-diagnostics/invariants) | -| [`code-runtime-python`](../packages/code-runtime/code-runtime-python) | `code-runtime` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`credentials`](../packages/credentials/credentials) | `credentials` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`e2b`](../packages/e2b/e2b) | `e2b` | [`invariants`](../packages/runtime-diagnostics/invariants) | | [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | [`invariants`](../packages/runtime-diagnostics/invariants) | @@ -1417,6 +1418,7 @@ flowchart TD | [`spill`](../packages/spill/spill) | `spill` | [`brand`](../packages/util/brand), [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) | | [`attachment-local`](../packages/attachment/attachment-local) | `attachment` | [`attachment`](../packages/attachment/attachment), [`home-paths`](../packages/util/home-paths), [`invariants`](../packages/runtime-diagnostics/invariants) | | [`app-boot`](../packages/boot/app-boot) | `boot` | [`home-paths`](../packages/util/home-paths), [`invariants`](../packages/runtime-diagnostics/invariants), [`launch-environment`](../packages/util/launch-environment), [`system-prompt`](../packages/core/system-prompt) | +| [`code-runtime-python`](../packages/code-runtime/code-runtime-python) | `code-runtime` | [`invariants`](../packages/runtime-diagnostics/invariants), [`session`](../packages/core/session), [`timeout`](../packages/util/timeout) | | [`code-runtime-worker-thread`](../packages/code-runtime/code-runtime-worker-thread) | `code-runtime` | [`code-runtime`](../packages/code-runtime/code-runtime), [`invariants`](../packages/runtime-diagnostics/invariants), [`session`](../packages/core/session), [`timeout`](../packages/util/timeout) | | [`persona`](../packages/preset/persona) | `preset` | [`invariants`](../packages/runtime-diagnostics/invariants), [`system-prompt`](../packages/core/system-prompt) | | [`sandbox`](../packages/sandbox/sandbox) | `sandbox` | [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) | diff --git a/packages/code-runtime/code-runtime-python/package.json b/packages/code-runtime/code-runtime-python/package.json index cd5aa2996c..29f9d21fd9 100644 --- a/packages/code-runtime/code-runtime-python/package.json +++ b/packages/code-runtime/code-runtime-python/package.json @@ -33,10 +33,17 @@ "license": "MIT", "peerDependencies": { "@deepseek-ai/dsh-invariants": "workspace:^", + "@deepseek-ai/dsh-session": "workspace:^", + "@deepseek-ai/dsh-timeout": "workspace:^", "@deepseek-ai/cordis": "workspace:^" }, + "dependencies": { + "@deepseek-ai/schemastery": "workspace:^" + }, "devDependencies": { "@deepseek-ai/dsh-invariants": "workspace:^", + "@deepseek-ai/dsh-session": "workspace:^", + "@deepseek-ai/dsh-timeout": "workspace:^", "@deepseek-ai/cordis": "workspace:^" } } diff --git a/packages/code-runtime/code-runtime-python/py/bootstrap.py b/packages/code-runtime/code-runtime-python/py/bootstrap.py new file mode 100644 index 0000000000..707a50858e --- /dev/null +++ b/packages/code-runtime/code-runtime-python/py/bootstrap.py @@ -0,0 +1,1711 @@ +"""CPython bootstrap for dsh-code-runtime-python. + +Reads a :class:`BootMessage` on fd 3, applies resource limits and log capture, +reads a :class:`RunMessage`, runs the model program as the body of an async +function (top-level ``await`` and ``return`` both work; the returned value is +the completion), and posts a terminal :class:`DoneMessage`. The program calls +host functions through the ``tools`` (or other namespace) proxy, whose attribute +and subscript access return awaitables that ride binding messages over fd 3. + +This module runs under ``python3 -I`` with an empty environment and +``sys.path`` containing only its own directory. +""" + +from __future__ import annotations + +import asyncio +import ast +import io +import json +import math +import os +import re +import resource +import signal +import sys +import threading +import traceback +from decimal import Decimal +from pathlib import Path +from typing import Any + +# ``python3 -I`` (isolated) drops the script directory from ``sys.path`` so +# the sibling ``protocol.py`` is invisible by default. Restore it explicitly +# before importing. +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from protocol import PROTOCOL_FD, log_truncation_marker # noqa: E402 + +# Read size for the async fd-3 reader. One `os.read` returns whatever the pipe +# holds, so this only bounds a single syscall's copy, not a frame: a larger frame +# simply takes more reads. 64 KiB matches the usual pipe capacity. +_READ_CHUNK_BYTES = 65536 + +# Code-unit ceiling on the exception class name interpolated into the LAST-resort +# failure diagnostic. A metaclass `__name__` property can return any length, and +# that construction runs outside the guard that would otherwise absorb a +# MemoryError, so the name is sliced before it is copied. Generous enough that no +# real class name is touched. +_MAX_FALLBACK_NAME_CHARS = 200 + + +# --------------------------------------------------------------------------- +# Log buffer — Python-side ledger for captured text. +# --------------------------------------------------------------------------- + + +class LogBuffer: + """Ordered text capture under one shared byte budget. + + Once the budget is exhausted the buffer emits exactly one in-band + truncation marker via ``sink`` and silently drops everything after. The + cap is a blast-radius bound; "how much was lost" intentionally stays + unmeasured. + """ + + def __init__(self, max_bytes: int, sink) -> None: + self._max_bytes = max_bytes + self._remaining = max_bytes + self._truncated = False + # Re-entrant so a caller may hold it across a compound read-modify-write + # (``_LogStream.write`` reads ``remaining`` several times and then calls + # ``push`` while still holding it). One lock is shared by this buffer and + # every stream that funnels into it: model code may start daemon threads + # that keep calling ``print`` after the program body returns, and the + # settlement ``flush_line`` on the main coroutine reads and mutates the + # same ``_pending``/ledger state. Without a shared lock the flush could + # interleave with a concurrent ``write`` — dropping or double-counting a + # line, or costing the ``done`` frame on a mangled ledger. Fixing which + # callable runs (binding ``out_stream.flush_line``) does not fix what it + # reads. + self._lock = threading.RLock() + # ``sink(text, truncated=False)``. The marker is emitted with + # ``truncated=True`` so the host can stop its own capture at the same + # point rather than treating the marker as ordinary program output: the + # two ledgers exhaust independently, and one entry larger than + # ``max_bytes`` sends only the marker while the host budget is still + # nearly empty. + self._sink = sink + + @property + def lock(self) -> "threading.RLock": + """The shared re-entrant lock guarding this ledger and its streams' buffers.""" + + return self._lock + + @property + def remaining(self) -> int: + """Serialized bytes still admissible; zero once truncated (streams use this to bound their own buffering, where a character count is a valid lower bound).""" + + return 0 if self._truncated else self._remaining + + def push(self, text: str) -> None: + with self._lock: + self._push_locked(text) + + def _push_locked(self, text: str) -> None: + if self._truncated: + return + # Cheap lower bound FIRST: one char is at least one UTF-8 byte and the + # JSON form adds two quotes plus the separator, so a single print() far + # above the budget truncates without ever encoding it — the full encode + # would allocate a second equally large string and could turn a + # truncatable log into an RLIMIT_AS death. + if len(text) + 3 > self._remaining: + self._truncated = True + self._sink(log_truncation_marker(self._max_bytes), truncated=True) + return + # A model print() can emit a lone surrogate; strict UTF-8 throws on it + # here. Replace it rather than escaping it the way :func:`_dump_string` + # preserves one inside a completion VALUE: log text is already a + # truncatable, substituting channel (the byte cap replaces the tail with + # a marker), and the ledger below charges the RAW UTF-8 bytes, which + # would undercharge the six-byte escape by half. Bounded: the text + # passed the length check, so this encodes at most ~4x remaining. + try: + raw = text.encode("utf-8") + except UnicodeEncodeError: + raw = text.encode("utf-8", errors="replace") + text = raw.decode("utf-8") + # Charge the SERIALIZED cost — the JSON string form's bytes plus one + # separator byte — exactly as the host ledger does. Charging the raw + # UTF-8 length instead undercharges control-heavy text, whose JSON + # escaping expands it up to sixfold (a NUL costs one raw byte but six + # as its ``\uXXXX`` escape): a NUL flood sized to fit ``maxLogBytes`` + # raw would serialize to roughly six times the shared cap, and the child + # could then die on RLIMIT_AS (reported host-side as ``worker-exit``) + # instead of emitting the truncation marker. The +1 also floors an empty + # entry above zero, so a flood of blank ``print()`` lines exhausts the + # budget instead of emitting unbounded zero-cost log frames. + cost = _json_string_cost(raw) + 1 + if cost > self._remaining: + self._truncated = True + self._sink(log_truncation_marker(self._max_bytes), truncated=True) + return + self._remaining -= cost + self._sink(text) + + +class _LogStream(io.TextIOBase): + """A newline-coalescing text stream backed by a :class:`LogBuffer`. + + Installed as ``sys.stdout`` / ``sys.stderr`` before executing the model + program. ``print(...)`` calls ``write`` once per argument, separator, and + newline, so a raw one-push-per-write stream would emit + ``["a", " ", "b", "\\n"]`` for ``print("a", "b")`` — and Code Mode renders + ``logs`` with ``join('\\n')``, turning that into spurious blank lines. This + stream instead buffers writes and pushes one LogBuffer entry per completed + LINE (the text up to each ``\\n``, newline stripped), so the rendered join + reproduces ``a b``. Any unterminated tail is flushed by :meth:`flush_line` + after the program settles. + """ + + def __init__(self, logs: LogBuffer) -> None: + super().__init__() + self._logs = logs + # list-of-chunks, joined only at a newline or flush: repeated + # ``print("x", end="")`` must not concatenate quadratically. + self._pending: list[str] = [] + self._pending_chars = 0 + + def writable(self) -> bool: # noqa: D401 -- inherited contract + return True + + def write(self, text: str) -> int: # noqa: D401 -- inherited contract + # Serialize the whole read-modify-write against the settlement flush and + # any other thread's write: model code may spawn daemon threads that keep + # printing after the program body returns, and this method reads + # ``remaining`` and mutates ``_pending``/the ledger across many steps. The + # lock is the buffer's and is re-entrant, so the ``push`` calls below + # (which re-acquire it) do not deadlock. + with self._logs.lock: + return self._write_locked(text) + + def _write_locked(self, text: str) -> int: + # Drop an empty write instead of buffering it. An empty chunk adds no + # character, so the budget check below can never fire on it: + # ``while True: sys.stdout.write("")`` would append one list slot per + # call with `_pending_chars` pinned at 0, growing unbounded long after + # the log ledger was exhausted (about 3.7 M slots per CPU second here) + # until RLIMIT_AS turned the allocation into a MemoryError — reported as + # the program's own exception rather than the intended bounded-log + # behavior. Returning here also keeps `flush_line` from pushing a + # spurious empty log entry for a program whose only writes were empty. + if not text: + return 0 + if "\n" in text: + # Scan `text` in place; the buffered chunks are joined ONLY into the + # first line. Joining the pending chunks with the whole write first + # made a second copy of that write, which an over-budget write + # cannot afford (measured under a 400 MiB addressSpaceMb: one + # buffered character followed by a 340 MiB write died on MemoryError + # inside the join, reported as the program's own exception, and the + # retained chunks made the settlement `flush_line` fail the same way + # — costing the `done` frame and turning the run into a wall-clock + # timeout instead of the promised truncation marker). + length = len(text) + pos = 0 + if self._pending: + newline = text.index("\n") + if self._pending_chars + newline + 3 > self._logs.remaining: + # The reconstructed first line cannot fit the ledger, so + # LogBuffer would reject it whole: copy only the prefix that + # fails its cheap bound and drop the chunks. The slice is + # bounded HERE, not inside the helper: `text[:newline]` on a + # 340 MiB newline-terminated write is the same full copy the + # join was (measured: MemoryError inside `sys.stdout.write` + # under a 400 MiB addressSpaceMb), and `remaining + 4` + # characters are all the helper can use. + self._push_bounded_prefix(text[: min(newline, self._logs.remaining + 4)]) + else: + self._pending.append(text[:newline]) + line = "".join(self._pending) + self._pending = [] + self._pending_chars = 0 + self._logs.push(line) + pos = newline + 1 + # Scan by offset and STOP once the ledger is exhausted: a single + # write of many newlines (``print("\n" * 1000000)``) would otherwise + # re-slice the tail once per line and keep pushing long after + # LogBuffer truncated, burning the CPU budget on discarded lines. + # `remaining` reads 0 the instant the buffer truncates, so the loop + # exits immediately; the unscanned tail is simply dropped. + while pos < length and self._logs.remaining > 0: + newline = text.find("\n", pos) + if newline < 0: + break + # Bound the SLICE the same way LogBuffer bounds the encode: a + # first line far above the ledger would be copied whole before + # push could reject it, and that copy is the allocation an + # over-budget write cannot afford. Copy only a budget-sized + # prefix, which push still rejects on its own cheap bound (the + # prefix is longer than `remaining`), so the marker is emitted + # and the oversized line is never materialized. + if newline - pos + 3 > self._logs.remaining: + self._logs.push(text[pos:pos + self._logs.remaining + 4]) + break + self._logs.push(text[pos:newline]) + pos = newline + 1 + if pos < length: + if self._logs.remaining > 0: + tail = text[pos:] + self._pending.append(tail) + self._pending_chars = len(tail) + else: + # The ledger ran out with text still unscanned, so that text + # IS being dropped and the run must say so. One push is + # enough and is bounded: `remaining` is 0, so LogBuffer's + # cheap length lower bound rejects immediately, emits the + # marker, and never encodes the tail — and a push after the + # marker is already out returns without emitting a second. + # Reaching 0 EXACTLY (65 one-character lines against the + # default 3-byte-per-entry serialized charge) leaves + # `_truncated` unset, so without this the tail vanished with + # no marker at all. Sliced to a budget-sized prefix, not the + # whole tail: the tail can be hundreds of megabytes and the + # copy would be the RLIMIT_AS death this bound exists to + # avoid, while push only needs enough characters to fail its + # own cheap length check. + self._logs.push(text[pos:pos + self._logs.remaining + 4]) + else: + self._pending.append(text) + self._pending_chars += len(text) + # A newline-free flood must hit the budget while running, not at + # settlement: once the buffered tail alone can no longer fit the + # ledger (chars lower-bound the serialized cost), push it through — LogBuffer + # truncates, emits the marker once, and swallows everything after. + if self._pending_chars > self._logs.remaining: + self._push_bounded_prefix() + return len(text) + + def _push_bounded_prefix(self, extra: str = "") -> None: + # Reached only when the buffered characters already exceed what the + # ledger admits, so LogBuffer is certain to reject on its cheap length + # bound and emit the marker. Copy a budget-sized PREFIX rather than the + # joined whole: ``sys.stdout.write("x")`` followed by one newline-free + # 340 MiB write leaves two chunks whose join is a second copy of the + # payload, and under a tight addressSpaceMb that join raises MemoryError + # from inside `write` — surfacing as the program's own exception, or, + # while the oversized chunks stayed retained, again from `flush_line` + # after the program settled, which cost the `done` frame and turned the + # run into a wall-clock timeout instead of the promised truncation + # marker. + # + # The chunks are dropped BEFORE the push so neither this call nor the + # settlement flush can repeat the allocation, and dropping the text is + # exactly what the marker reports. `remaining + 4` is the shortest + # prefix that still fails LogBuffer's ``len(text) + 3 > remaining`` + # check; the accumulation stops there, so the copy is bounded by the log + # budget however large the pending chunks are. + limit = self._logs.remaining + 4 + parts: list[str] = [] + total = 0 + for chunk in (*self._pending, extra): + parts.append(chunk[: limit - total]) + total += len(parts[-1]) + if total >= limit: + break + self._pending = [] + self._pending_chars = 0 + self._logs.push("".join(parts)) + + def flush(self) -> None: # noqa: D401 -- inherited contract + # ``TextIOBase.flush`` is a no-op, so without this override an explicit + # ``print(..., flush=True)`` or ``sys.stdout.flush()`` left the text in + # `_pending` with nothing to drain it except `flush_line` after the + # program settles. A run that then hangs or is killed never reaches that + # call: ``print("before hang", end="", flush=True)`` followed by an + # infinite loop returned `logs: []`, losing the one diagnostic the + # program deliberately committed. Forwarding makes an explicit flush emit + # the pending entry immediately, which is what the caller asked for; a + # newline-terminated write already emitted on its own. + self.flush_line() + + def flush_line(self) -> None: + """Push any buffered text not terminated by a newline (also serves explicit flushes).""" + + # Same shared, re-entrant lock as ``write``: the settlement flush on the + # main coroutine and a daemon thread's concurrent ``write`` both touch + # ``_pending`` and the ledger, so this read-and-clear must be atomic + # against them. + with self._logs.lock: + if self._pending: + self._logs.push("".join(self._pending)) + self._pending = [] + self._pending_chars = 0 + + +# --------------------------------------------------------------------------- +# Fd-3 channel — line-framed JSON. +# --------------------------------------------------------------------------- + + +class ProtocolChannel: + """Blocking readers and synchronous writers over the fd-3 protocol pipe. + + Writes are unbuffered and go straight to the fd, so ``send_sync`` is safe + from inside model code (which may run outside an asyncio task) and from + background tasks alike. The single writer is serialized by CPython's GIL + plus one os.write per frame (POSIX guarantees atomicity for writes below + ``PIPE_BUF``, and our frames are short JSON lines). + """ + + def __init__(self, fd: int) -> None: + # Unbuffered binary I/O so we never lose frames to an idle flush. + self._reader = os.fdopen(fd, "rb", buffering=0, closefd=False) + self._fd = fd + # Residual bytes read past a frame's newline. Held here, not in the + # reading coroutine: the reply pump is cancelled once `done` is posted, + # and read-ahead sitting in a local would be lost with it. + self._pending = bytearray() + # Serializes writers: os.write releases the GIL, and a frame larger + # than PIPE_BUF is neither atomic nor guaranteed fully consumed by one + # call — without the lock, model-created threads printing while a big + # completion frame drains could interleave bytes mid-frame. + self._write_lock = threading.Lock() + + def read_frame(self) -> dict[str, Any] | None: + """Read one JSON-line frame (iteratively decoded). ``None`` on EOF. + + Blocking. Used for the two frames read BEFORE the model program starts + (``boot`` and ``run``), where blocking is what the handshake wants. Reply + frames arriving during the program go through :meth:`read_frame_async`, + which must not occupy a thread. + """ + + line = self._reader.readline() + if not line: + return None + return _decode_json_plain(line.decode("utf-8")) + + async def read_frame_async(self) -> dict[str, Any] | None: + """Await one JSON-line frame without occupying a thread. ``None`` on EOF. + + ``loop.run_in_executor(None, read_frame)`` was the obvious spelling and + the wrong one: the default executor spins up its first thread the moment + the program awaits a binding, and on Linux/glibc that thread's 8 MiB + stack plus a 64 MiB per-thread malloc arena reservation are charged to + ``RLIMIT_AS`` — measured, the child's mappings went from 30.34 MiB to + 102.39 MiB across one ``await tools.*``. Since the limit is already in + force, that ~72 MiB comes straight out of the run's ``addressSpaceMb``: + under a small limit the thread cannot start at all and a legitimate + binding call hangs to ``maxWallMs``, and under a larger one an allocation + that should have fit dies as ``MemoryError``. This is the same accounting + the settlement-time CPU recheck was designed around, where a sampling + thread cost the same 72 MiB. + `loop.add_reader` watches the fd instead, so no thread exists. + + Bytes past a frame's newline belong to the next frame, so the residual + lives on the CHANNEL rather than in this coroutine: the pump is cancelled + once ``done`` is posted, and a local buffer would discard whatever it had + read ahead. + """ + + loop = asyncio.get_event_loop() + while True: + newline = self._pending.find(b"\n") + if newline >= 0: + line = bytes(self._pending[:newline]) + del self._pending[: newline + 1] + return _decode_json_plain(line.decode("utf-8")) + ready = loop.create_future() + # `add_reader` only reports readability; the read itself happens here, + # and `os.read` returns whatever is buffered without waiting for more. + loop.add_reader(self._fd, lambda: ready.done() or ready.set_result(None)) + try: + await ready + finally: + loop.remove_reader(self._fd) + chunk = os.read(self._fd, _READ_CHUNK_BYTES) + if not chunk: + # EOF. Any partial line is dropped, matching how the host drops a + # frame that never completed. + return None + self._pending.extend(chunk) + + def send_sync(self, message: dict[str, Any]) -> None: + """Post one frame synchronously. + + Encoded with the iterative :func:`_encode_json_plain` (not + ``json.dumps``, whose per-level recursion would raise + ``RecursionError`` on a deeply nested completion or call argument the + depth-unbounded ``CodeJsonValue`` contract admits). NaN/Infinity still + raise ``ValueError`` — they would serialize as non-standard tokens + that Node's ``JSON.parse`` rejects, silently dropping the frame, and a + call would then hang until the wall clock instead of failing fast. + Callers turn the ``ValueError`` into their own contract error + (dispatch raises the lossless-JSON message). + """ + + payload = (_encode_json_plain(message) + "\n").encode("utf-8") + # Full-write loop under the writer lock: one os.write may consume only + # part of a frame beyond PIPE_BUF (64 KiB logs / 32 KiB completions / + # uncapped call args exceed it), and a partial or interleaved frame is + # dropped host-side as malformed JSON — the run would then hang to the + # wall clock. + with self._write_lock: + view = memoryview(payload) + while view: + view = view[os.write(self._fd, view):] + + +# --------------------------------------------------------------------------- +# Tools proxy — turns ``await tools.name(args)`` into a fd-3 call frame. +# --------------------------------------------------------------------------- + + +class _Namespace: + """A proxy for one binding namespace: every declared name routes to the bridge. + + Names arrive from :class:`BootMessage.namespaces`. Both attribute access + (``tools.name``) and subscript access (``tools["my-tool"]`` — the SDK's + escape hatch for exotic or reserved names, which are legal function names + on the wire) return a coroutine factory that posts a ``call`` frame and + awaits the matching ``reply``. An undeclared name raises ``AttributeError`` + (attribute) or ``KeyError`` (subscript), matching the worker backend's + own-property discipline. + + ``__getattribute__`` (not ``__getattr__``) intercepts attribute access so a + declared name ALWAYS reaches the bridge — even one that collides with an + inherited attribute like ``__class__``, which ordinary lookup would resolve + on ``object`` before ``__getattr__`` ever ran. Internal state lives under + name-mangled ``_Namespace__*`` attributes; a declared binding with such a + name still wins (declared-names check runs first). + """ + + def __init__(self, global_name: str, names: list[str], dispatch) -> None: + self.__global = global_name + self.__names = set(names) + self.__dispatch = dispatch + + def __call_for(self, name: str): + dispatch = object.__getattribute__(self, "_Namespace__dispatch") + global_name = object.__getattribute__(self, "_Namespace__global") + + async def call(args: Any) -> Any: + return await dispatch(global_name, name, args) + + return call + + def __getattribute__(self, name: str): + # Declared names route to the bridge unconditionally — before Python + # can resolve an inherited attribute (``__class__``) or our own + # internals. Everything else falls through to normal lookup so the + # proxy machinery itself keeps working. + names = object.__getattribute__(self, "_Namespace__names") + if name in names: + return object.__getattribute__(self, "_Namespace__call_for")(name) + return object.__getattribute__(self, name) + + def __getattr__(self, name: str): + # Reached only when normal lookup found nothing (declared names were + # already intercepted above), so this is always an undeclared tool. + raise AttributeError( + f"tool {name!r} is not declared in namespace " + f"{object.__getattribute__(self, '_Namespace__global')!r}" + ) + + def __getitem__(self, name: str): + names = object.__getattribute__(self, "_Namespace__names") + if name not in names: + raise KeyError( + f"tool {name!r} is not declared in namespace " + f"{object.__getattribute__(self, '_Namespace__global')!r}" + ) + return object.__getattribute__(self, "_Namespace__call_for")(name) + + +class _BindingRejection(Exception): + """Internal reply-pump rejection, converted by ``dispatch`` into the + namespace's declared error class (or ``RuntimeError``) so the marker type + itself never reaches model code.""" + + +def _make_error_class(name: str, member_name_property: str) -> type: + """Mint one program-visible rejection class per the seam's + ``CodeBindingErrorClass`` contract: instances carry the failed member name + under ``member_name_property`` and render as their message.""" + + def __init__(self, member_name: str, message: str) -> None: # noqa: N807 + Exception.__init__(self, message) + setattr(self, member_name_property, member_name) + + return type(name, (Exception,), {"__init__": __init__}) + + +def _clamped(which: int, soft: int, hard: int) -> tuple[int, int]: + """Bound a requested (soft, hard) rlimit pair by the inherited hard limit. + + An unprivileged process may lower a hard limit but never raise it, so a + harness already started under a tighter ceiling (``ulimit -v`` below + ``addressSpaceBytes``, or a CPU cap below ``cpuSeconds`` + 1) would make + ``setrlimit`` raise ``ValueError`` and fail every run — despite the + inherited limit being STRONGER than the one requested. Clamping keeps the + stricter of the two, which still satisfies the containment contract. + ``RLIM_INFINITY`` compares as -1, so it is special-cased rather than + treated as the smallest bound. + """ + inherited = resource.getrlimit(which)[1] + if inherited == resource.RLIM_INFINITY: + return (soft, hard) + return (min(soft, inherited), min(hard, inherited)) + + +# --------------------------------------------------------------------------- +# Main. +# --------------------------------------------------------------------------- + + +async def _run(channel: ProtocolChannel) -> None: + # 1. Boot handshake. + boot = channel.read_frame() + if boot is None or boot.get("type") != "boot": + raise RuntimeError("bootstrap: expected boot frame on fd 3") + + # A limit that cannot be applied must fail the run as a diagnosable done + # frame, not a bare traceback + exit(1): running the program UNCAPPED would + # silently void the containment contract, and the host can only relay what + # rides the protocol. + try: + # SIGXCPU's default disposition (how the soft CPU limit stops the child) + # dumps core, and the child inherits the host's RLIMIT_CORE — a CPU + # timeout would otherwise write a large memory-bearing core file into + # the workspace. Forbid core dumps first so the timeout path leaves none. + resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) + # Soft limit at cpuSeconds fires SIGXCPU (its default disposition + # terminates the child; the host classifies that close as a timeout). + # Hard limit at +1s is a SIGKILL backstop for a program that traps + # SIGXCPU and keeps burning CPU. + cpu_soft, cpu_hard = _clamped( + resource.RLIMIT_CPU, boot["cpuSeconds"], boot["cpuSeconds"] + 1 + ) + resource.setrlimit(resource.RLIMIT_CPU, (cpu_soft, cpu_hard)) + # Darwin maps the multi-GB dyld shared cache into every process at + # exec, so any practical RLIMIT_AS cap sits below current usage and + # the kernel rejects it — the child would die here on every run. Skip + # the address-space cap there; RLIMIT_CPU and the host's wall-clock + # ceiling still bound the run. + if sys.platform != "darwin": + addr_bytes = int(boot["addressSpaceBytes"]) + resource.setrlimit( + resource.RLIMIT_AS, _clamped(resource.RLIMIT_AS, addr_bytes, addr_bytes) + ) + except BaseException as exc: # noqa: BLE001 -- report every failure to host + channel.send_sync( + { + "type": "done", + "error": { + "kind": "exception", + # Exception-only rendering: format_exc() would embed the + # absolute installed bootstrap.py path in model-visible + # durable output, leaking host paths into transcripts. + "message": "bootstrap: applying resource limits failed\n" + + "".join( + traceback.format_exception_only(type(exc), exc) + ), + }, + } + ) + return + + logs = LogBuffer( + int(boot["maxLogBytes"]), + sink=lambda text, truncated=False: channel.send_sync( + {"type": "log", "text": text, **({"truncated": True} if truncated else {})} + ), + ) + + # 2. Wire the tools proxies and the ack. + pending: dict[int, asyncio.Future[Any]] = {} + next_id = 0 + + error_classes: dict[str, type] = {} + + async def dispatch(global_name: str, name: str, args: Any) -> Any: + nonlocal next_id + error_class = error_classes.get(global_name) + + def call_failure(message: str) -> BaseException: + # The namespace's declared rejection contract (e.g. Code Mode's + # ToolCallError with .toolName) when present; RuntimeError keeps + # the pre-errorClass behavior for namespaces that declared none. + if error_class is not None: + return error_class(name, message) + return RuntimeError(message) + + # Validate the argument shape before claiming an id, so a rejected call + # leaves no gap in the sequence the host checks. json.dumps would coerce + # a non-string dict key or non-finite float rather than raise (allow_nan + # is off, but key coercion still slips through), silently corrupting what + # the tool receives. Reject up front through the call's error contract. + violation = _lossless_json_violation(args) + if violation is not None: + raise call_failure(f"binding arguments must be lossless JSON ({violation})") + # Ids are consecutive from 0 with NO gaps: the host answers a `call` only + # when its id is the exact successor of the last one, which bounds the + # state it retains to a single number. A frame that never reaches the + # host must therefore not consume an id, so the counter advances only + # once the write has succeeded. + call_id = next_id + fut: asyncio.Future[Any] = asyncio.get_event_loop().create_future() + pending[call_id] = fut + try: + channel.send_sync( + { + "type": "call", + "id": call_id, + "global": global_name, + "name": name, + "args": args, + } + ) + except (TypeError, ValueError) as exc: + pending.pop(call_id, None) + raise call_failure( + f"binding arguments must be lossless JSON: {exc}" + ) from exc + next_id += 1 + try: + return await fut + except _BindingRejection as exc: + raise call_failure(str(exc)) from None + + namespaces: dict[str, Any] = {} + for entry in boot["namespaces"]: + namespaces[entry["global"]] = _Namespace( + entry["global"], entry["names"], dispatch + ) + declared = entry.get("errorClass") + if declared: + error_class = _make_error_class( + declared["name"], declared["memberNameProperty"] + ) + error_classes[entry["global"]] = error_class + # The class is program-visible under its own name so model code + # can `except ToolCallError as e:` and read the member property. + namespaces[declared["name"]] = error_class + + channel.send_sync({"type": "boot-ack"}) + + # 3. Start a reply-pump task before the run message: replies can arrive + # interleaved with the run's own binding traffic. + reply_task = asyncio.get_event_loop().create_task(_pump_replies(channel, pending)) + + # 4. Read the run message. + run = channel.read_frame() + if run is None or run.get("type") != "run": + reply_task.cancel() + raise RuntimeError("bootstrap: expected run frame on fd 3") + + program: str = run["program"] + + # 5. Install log capture — ``print``, tracebacks, and ordinary ``sys.stdout`` + # writes funnel into the LogBuffer. The real fds stay open (host uses + # stderr for stray-byte accounting) but the Python-visible streams point + # at the buffer. + sys.stdout = _LogStream(logs) # type: ignore[assignment] + sys.stderr = _LogStream(logs) # type: ignore[assignment] + out_stream, err_stream = sys.stdout, sys.stderr + + # 6. Compile the program as the body of an async function, matching the + # seam contract (`CodeRunRequest.program` is an async-function body: top-level + # `await` and `return` both work, and the returned value is the completion). + # AST-splicing the parsed body into an `async def` keeps every statement's + # original line number, so a traceback points at the model's own source. + ns: dict[str, Any] = { + "__name__": "__main__", + "__builtins__": __builtins__, + **namespaces, + } + # Read the enforcement callable and its budget into this frame's locals + # BEFORE the program runs: model code can rebind this module's globals + # (the bootstrap IS ``__main__``), and a frame local is not a module + # attribute, so a later ``__main__._DIE_IF_CPU_EXHAUSTED = ...`` cannot + # change which callable the post-check below invokes. This defeats the + # one-line rebind, not a determined `sys._getframe` walk; the unforgeable + # bounds are the RLIMIT_CPU hard limit and the host wall clock + # (see _make_cpu_enforcer). + die_if_cpu_exhausted = _DIE_IF_CPU_EXHAUSTED + cpu_seconds = int(boot["cpuSeconds"]) + # Same capture, same reason, for the failure path and the send that follows + # it. The reporter was a module-global lookup inside the `except` block, so + # ``import __main__; __main__._SAFE_MODEL_TRACEBACK = ...`` put model code + # there with no guard around it; the flush and send were attribute lookups + # on the stream and channel CLASSES, which ``__main__._LogStream.flush_line + # = ...`` rebinds just as easily. All four run AFTER the handler, where a + # throw costs the `done` frame and the host reports a wall-clock timeout + # instead of the model's exception. Binding the callables now fixes what + # runs; what they in turn reach is closed over in _make_failure_reporter. + safe_model_traceback = _SAFE_MODEL_TRACEBACK + flush_out = out_stream.flush_line + flush_err = err_stream.flush_line + send_done = channel.send_sync + max_value_bytes = int(boot["maxValueBytes"]) + done: dict[str, Any] + try: + module = ast.parse(program) + wrapper = ast.AsyncFunctionDef( + name="__dsh_main__", + args=ast.arguments( + posonlyargs=[], args=[], vararg=None, + kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[], + ), + body=module.body or [ast.Pass()], + decorator_list=[], + returns=None, + ) + # Anchor the synthetic wrapper on the first real statement (or line 1 for + # an empty program) so fix_missing_locations does not stamp it at 0. + anchor = module.body[0] if module.body else ast.parse("pass").body[0] + ast.copy_location(wrapper, anchor) + wrapped = ast.Module(body=[wrapper], type_ignores=[]) + ast.fix_missing_locations(wrapped) + code = compile(wrapped, "", "exec") + exec(code, ns) # noqa: S102 -- defines __dsh_main__; executing model code is the point + value = await ns["__dsh_main__"]() + die_if_cpu_exhausted(cpu_seconds) + done = _done_with_value(value, max_value_bytes) + except BaseException as exc: # noqa: BLE001 -- report every failure to host + done = { + "type": "done", + "error": { + "kind": "exception", + # Cap the diagnostic BEFORE it crosses the wire: a program can + # raise with a gigabytes-long message, and formatting/sending + # it whole would allocate on both sides before the host's own + # cap runs. Byte-cap at maxValueBytes with the host's marker + # text so the truncated diagnostic reads identically wherever + # the cap was applied. The rendering is wrapped because the + # `done` send below sits outside this handler: a throw while + # formatting would skip it and strand the host on fd 3 until + # maxWallMs (see _make_failure_reporter). + "message": safe_model_traceback(exc, max_value_bytes), + }, + } + + # Flush any print output not terminated by a newline (a traceback always + # ends in one, but `print(x, end="")` or a bare write may not), so the + # final partial line is not silently dropped. + flush_out() + flush_err() + reply_task.cancel() + send_done(done) + + +async def _pump_replies( + channel: ProtocolChannel, pending: dict[int, asyncio.Future[Any]] +) -> None: + """Background task: read reply frames and settle pending futures. + + Cancelled after ``done`` is posted. Unknown ids and post-settlement replies + are ignored (mirrors the worker backend's hostile-peer stance, though here + the host is the trusted side; the guards defend against races). + """ + + while True: + frame = await channel.read_frame_async() + if frame is None: + return + if frame.get("type") != "reply": + continue + fut = pending.pop(frame.get("id"), None) + if fut is None or fut.done(): + continue + if frame.get("ok"): + fut.set_result(frame.get("value")) + else: + fut.set_exception(_BindingRejection(str(frame.get("message")))) + + +_SCALAR_RE = re.compile( + r'"(?:[^"\\]|\\.)*"|-?\d+(?:\.\d+)?(?:[eE][+-]?\d+)?|true|false|null' +) + + +def _decode_json_plain(text: str) -> Any: + """Parse one JSON document iteratively (no per-level recursion). + + ``json.loads`` recurses per nesting level and raises ``RecursionError`` + around ~10k levels, but a binding reply is depth-unbounded by the seam + contract — the host's iterative encoder happily produces documents + ``json.loads`` cannot read back. Scalars (numbers, strings with escapes) + are delegated to ``json.loads`` one token at a time, so their grammar and + semantics stay CPython's own; only the container structure is parsed here + with an explicit stack. Raises ``ValueError`` on malformed input; frames + come from the TRUSTED host, so strictness mirrors ``json.loads`` without + extra hostile-input hardening. + """ + + length = len(text) + + def skip_ws(i: int) -> int: + while i < length and text[i] in " \t\n\r": + i += 1 + return i + + def scalar(i: int): + match = _SCALAR_RE.match(text, i) + if match is None: + raise ValueError(f"invalid JSON at offset {i}") + return json.loads(match.group(0)), match.end() + + def string_key(i: int): + key, end = scalar(i) + if not isinstance(key, str): + raise ValueError(f"object key must be a string at offset {i}") + end = skip_ws(end) + if end >= length or text[end] != ":": + raise ValueError(f"expected ':' at offset {end}") + return key, skip_ws(end + 1) + + # Frames: a list, or (dict, pending key). `value`/`have_value` carry each + # completed value up to its parent frame. + stack: list[Any] = [] + value: Any = None + have_value = False + i = skip_ws(0) + while True: + if not have_value: + ch = text[i] if i < length else "" + if ch == "[": + i = skip_ws(i + 1) + if i < length and text[i] == "]": + i += 1 + value, have_value = [], True + else: + stack.append([]) + continue + elif ch == "{": + i = skip_ws(i + 1) + if i < length and text[i] == "}": + i += 1 + value, have_value = {}, True + else: + key, i = string_key(i) + stack.append(({}, key)) + continue + else: + value, i = scalar(i) + have_value = True + if not stack: + i = skip_ws(i) + if i != length: + raise ValueError(f"trailing data at offset {i}") + return value + top = stack[-1] + i = skip_ws(i) + ch = text[i] if i < length else "" + if isinstance(top, list): + top.append(value) + if ch == ",": + i = skip_ws(i + 1) + have_value = False + elif ch == "]": + i += 1 + stack.pop() + value = top + else: + raise ValueError(f"expected ',' or ']' at offset {i}") + else: + container, key = top + container[key] = value + if ch == ",": + key, i = string_key(skip_ws(i + 1)) + stack[-1] = (container, key) + have_value = False + elif ch == "}": + i += 1 + stack.pop() + value = container + else: + raise ValueError(f"expected ',' or '}}' at offset {i}") + + +def _encode_json_plain(value: Any) -> str: + """Encode JSON-plain data iteratively, byte-identical to compact ``json.dumps``. + + ``json.dumps`` recurses one Python frame per nesting level and raises + ``RecursionError`` a few thousand levels deep, but the seam's + ``CodeJsonValue`` has no depth limit — a valid deeply nested completion or + call argument below the byte budget must cross intact (the host uses the + same iterative idiom in ``protocol.ts``). Accepts what the callers already + validated or constructed: ``None``/``bool``/``int``/finite ``float``/ + ``str``, exact ``list``/``tuple``, and exact ``dict`` with ``str`` keys. + Scalar encoding delegates to ``json.dumps`` (string escaping, float repr) + so the bytes match; non-finite floats still raise ``ValueError`` exactly + like ``allow_nan=False``. + + Containers are classified by EXACT type and traversed through the unbound + built-in methods rather than the instance's own: a ``dict``/``list`` + subclass can override ``items``, ``keys``, ``__iter__``, ``__len__``, or + ``__getitem__``, and the validators only see the container it subclasses, + so an instance-method call here could emit different data than the walk + that metered and approved it. ``_check_done_value`` and + ``_lossless_json_violation`` reject subclasses outright, so this path only + ever sees exact containers; classifying on exact type keeps that agreement + checkable at one glance instead of resting on the caller. + """ + + chunks: list[str] = [] + # Each frame is either a literal string to emit or a value to expand. + stack: list[Any] = [value] + while stack: + current = stack.pop() + current_type = type(current) + if current_type is _Emit: + chunks.append(current.text) + elif current_type is list or current_type is tuple: + count = len(current) + chunks.append("[") + stack.append(_Emit("]")) + for index in range(count - 1, -1, -1): + if index < count - 1: + stack.append(_Emit(",")) + stack.append(current[index]) + elif current_type is dict: + chunks.append("{") + stack.append(_Emit("}")) + items = list(dict.items(current)) + for index in range(len(items) - 1, -1, -1): + key, item = items[index] + if index < len(items) - 1: + stack.append(_Emit(",")) + stack.append(item) + stack.append(_Emit(_dump_scalar(key) + ":")) + else: + chunks.append(_dump_scalar(current)) + return "".join(chunks) + + +def _dump_scalar(value: Any) -> str: + """One scalar as compact JSON, byte-compatible with the host's encoder. + + ``ensure_ascii=False`` keeps non-ASCII text as raw UTF-8 — the default + backslash-u escaping would make the child count ``"é"`` as 8 bytes where + the host meter (and the worker backend) count its UTF-8 JSON form as 4, + splitting the budget the two sides are supposed to share. Strings route + through :func:`_dump_string`, which restores the escaping for the one class + of character UTF-8 cannot hold. Floats route through :func:`_dump_float` + because CPython's ``repr`` and ECMAScript's Number-to-String disagree on + spelling. + + Dispatch is on EXACT type, matching the validators: a ``float`` subclass + reaching :func:`_dump_float` would have its overridden ``__repr__`` read as + the number's digits, so ``F(2.5)`` whose ``__repr__`` returns ``"1.0"`` + would serialize as ``1``. ``json.dumps`` then refuses any subclass by + ``TypeError`` instead of emitting a value nothing validated; the callers + reject subclasses first, so this is the encoder refusing to be the place a + validation gap turns into corrupted output. + """ + + if type(value) is float: + return _dump_float(value) + if type(value) is str: + return _dump_string(value) + if value is None or type(value) is bool or type(value) is int: + return json.dumps(value, ensure_ascii=False, allow_nan=False) + raise TypeError(f"unsupported type ({type(value).__name__})") + + +# A surrogate code unit, and an adjacent high-low pair. Python stores an astral +# character as ONE code point, so a surrogate reaching these patterns is either +# lone or half of a pair the program spelled out code unit by code unit. +_SURROGATE = re.compile("[\ud800-\udfff]") +_SURROGATE_PAIR = re.compile("[\ud800-\udbff][\udc00-\udfff]") + + +def _combine_surrogate_pair(match: re.Match[str]) -> str: + """Fold one spelled-out high-low pair into the astral code point it names.""" + + high, low = match.group(0) + return chr(0x10000 + ((ord(high) - 0xD800) << 10) + (ord(low) - 0xDC00)) + + +def _dump_string(text: str) -> str: + """One string as compact JSON, byte-identical to the host's ``JSON.stringify``. + + ``ensure_ascii=False`` cannot render a surrogate code unit: UTF-8 has no + encoding for one, so the frame write would raise and the run would strand + until the wall clock. JSON carries it as the ASCII escape ``\\ud800``, which + the host's ``JSON.parse`` reads back as the same UTF-16 code unit and its + ``JSON.stringify`` re-emits identically — so the shared seam + (``CodeJsonValue``, ``snapshotJsonValue``, the worker backend) keeps a + lone-surrogate string instead of failing the value. An adjacent high-low + pair is folded into its astral code point FIRST: the host holds strings as + UTF-16, where those two code units and the single character are the same + string, and the raw 4-byte form is what the host would emit — escaping the + halves separately would charge 12 bytes against a budget the host meters at + 4. Every remaining surrogate is lone and becomes six ASCII bytes, matching + the host exactly. + @param text: the string to encode. + @return: its compact JSON form, always UTF-8-encodable. + """ + + rendered = json.dumps(text, ensure_ascii=False) + if _SURROGATE.search(rendered) is None: + return rendered + return _SURROGATE.sub( + lambda match: "\\u%04x" % ord(match.group(0)), + _SURROGATE_PAIR.sub(_combine_surrogate_pair, rendered), + ) + + +# How many bytes each byte that needs escaping adds beyond its raw self, as a +# ready-made (byte, surcharge) list so :func:`_json_string_cost` walks no +# branches per pass. ``"`` and ``\\`` take a one-character prefix; the five C0 +# controls with a shorthand (``\\b\\f\\n\\r\\t``) likewise; every other C0 +# control becomes a six-character ``\\uXXXX``. +_JSON_ESCAPE_SURCHARGES = [ + (bytes((byte,)), 1 if byte in b'"\\\b\f\n\r\t' else 5) + for byte in [*range(0x20), ord('"'), ord("\\")] +] + + +def _json_string_cost(raw: bytes) -> int: + """UTF-8 byte length of one string's JSON form, WITHOUT building that form. + + Used by :class:`LogBuffer` to charge a log entry what it will actually cost + on the wire. Building ``json.dumps(text)`` to measure it would allocate a + second copy up to six times the original — the very allocation the ledger's + cheap pre-check exists to avoid, and enough to breach ``RLIMIT_AS`` on a + large control-heavy line. Counts exactly what :func:`_dump_scalar`'s + ``ensure_ascii=False`` output holds: the two quotes, each escaped byte's + surcharge from :data:`_JSON_ESCAPE_SURCHARGES`, and the raw bytes themselves + (non-ASCII stays raw, so its UTF-8 length already counts). Uses a fixed + number of C-level ``count`` passes — allocating nothing, unlike a + ``translate`` filter — because the caller admits up to ~4x the remaining + budget of bytes here and a per-byte Python loop over it would cost more than + the encode being avoided. + @param raw: the entry's UTF-8 bytes. + @return: the byte length of its JSON string form, quotes included. + """ + + extra = 0 + for byte, surcharge in _JSON_ESCAPE_SURCHARGES: + extra += raw.count(byte) * surcharge + return len(raw) + 2 + extra + + +def _dump_float(value: float) -> str: + """One finite float in ECMAScript ``Number::toString`` spelling. + + CPython's ``repr`` and the host's ``String(number)`` name the same double + differently: ``1.0`` is ``"1.0"`` here but ``"1"`` there, ``1e-07`` pads the + exponent the host writes as ``1e-7``, and ``1e+21``/``2**60`` differ again. + Since the child meters the completion value against ``maxValueBytes`` and + the host re-meters the frame it parses, any spelling difference splits the + shared budget: ``return 1.0`` under ``maxValueBytes: 1`` used to be reported + as ``output-limit`` by the child while the host would have counted the + one-byte ``1`` it actually receives. Both sides also emit these bytes (the + child through :func:`_encode_json_plain`, the host through + ``encodeJsonPlain``), so the fix has to be in the shared speller, not in the + meter. + + Implements ECMA-262 ``Number::toString`` radix 10 directly: ``repr`` + already yields the shortest round-tripping decimal digits, and ``Decimal`` + splits them into the significand ``s`` (``digits``, ``k`` of them) and + decimal exponent ``n`` the spec's cases select on. The integral values above + the JS safe range take the host's BigInt branch, whose exact digits differ + from the shortest-round-trip form (``2**60`` prints ``...846976``, not + ``...847000``). + """ + + if value != value or value in (float("inf"), float("-inf")): + # json.dumps(allow_nan=False) raises the same way; the callers reject + # non-finite floats before metering, so this is unreachable defense. + raise ValueError("Out of range float values are not JSON compliant") + if value == 0.0: + # Covers -0.0 too; callers reject it as non-lossless before this point. + return "0" + if value < 0: + return "-" + _dump_float(-value) + if value.is_integer() and value > float(2**53 - 1): + # The host's BigInt branch: exact digits, not shortest-round-trip. + return str(int(value)) + parts = Decimal(repr(value)).normalize().as_tuple() + digits = "".join(str(digit) for digit in parts.digits) + k = len(digits) + n = parts.exponent + k + if k <= n <= 21: + return digits + "0" * (n - k) + if 0 < n <= 21: + return digits[:n] + "." + digits[n:] + if -6 < n <= 0: + return "0." + "0" * -n + digits + exponent = ("+" if n - 1 >= 0 else "-") + str(abs(n - 1)) + return (digits if k == 1 else digits[0] + "." + digits[1:]) + "e" + exponent + + +def _check_done_value(value: Any, max_bytes: int): + """Meter a completion value's JSON byte size AND validate its lossless-JSON + shape in one bounded post-order walk; return ``None`` when it passes. + + Folds what was formerly a losslessness walk followed by a separate byte + meter into one pass. Running the losslessness walk first materialized one + traversal tuple per element before any size cap: ``return [0] * 2000000`` + under ``maxValueBytes: 64`` allocated millions of frames (an RLIMIT_AS + death) before the meter could reject it. Folding the byte bound into the + walk rejects over-budget BEFORE enqueuing a container's children — every + element is at least one JSON byte — so the walk stays O(cap). Same + JS-double-exact integer boundary, cycle detection (a leave marker pops each + container off ``on_path``), and type rejections as + :func:`_lossless_json_violation`, and the same byte accounting as + :func:`_encode_json_plain`. :func:`_lossless_json_violation` stays for the + binding-argument path, which carries no size cap. + + EVERY type here is matched EXACTLY, containers and scalars alike, so a + subclass is rejected as an unsupported type rather than admitted by + ``isinstance``. A subclass can override the operators and methods this walk + and the encoder call, and they need not agree: a populated ``dict`` + subclass whose ``items()`` returns ``[]`` would meter as ``{}``; a ``float`` + subclass overriding ``__repr__`` passes the non-finite and negative-zero + checks by its real value but serializes as whatever the override says, since + :func:`_dump_float` reads ``repr``; an ``int`` subclass overriding ``__gt__`` + and ``__lt__`` slips past the JS-safe-range bound while ``json.dumps`` + emits its true C-level digits, so ``2**53 + 1`` reaches the host as + ``...992``; a ``str`` subclass overriding ``__len__`` returns 0 from the + pre-encode lower bound and admits an arbitrarily large string. In each case + the value the host receives differs from the one this walk approved. The + worker backend rejects the equivalent shapes by prototype identity and + ``typeof`` (``hasPlainObjectPrototype`` in ``worker-json.ts``); a ``bool`` + is checked before ``int`` because it is an ``int`` subclass that IS + lossless JSON. + + Returns ``("invalid-output", message)`` for a non-lossless value, + ``("output-limit", message)`` once the size crosses ``max_bytes``, or + ``None`` when the value is lossless JSON within budget. + """ + + js_safe = 2**53 - 1 + + def invalid(reason: str): + return ("invalid-output", f"program completion must be lossless JSON ({reason})") + + over_budget = ("output-limit", f"completion value exceeded {max_bytes} bytes") + + total = 0 + on_path: set[int] = set() + # Each frame is (value, is_leave): a leave frame pops its container off the path. + stack: list[tuple[Any, bool]] = [(value, False)] + while stack: + current, is_leave = stack.pop() + if is_leave: + on_path.discard(id(current)) + continue + if current is None or type(current) is bool: + total += len(_dump_scalar(current).encode("utf-8")) + elif type(current) is str: + # Lower-bound BEFORE materializing the escaped form: every character + # is at least one UTF-8 byte plus the two quotes, so a huge or + # control-heavy string (whose escaped copy expands severalfold) is + # rejected without allocating that copy. + if total + len(current) + 2 > max_bytes: + return over_budget + # A lone surrogate has no UTF-8 form but a lossless JSON one — the + # ASCII ``\uXXXX`` escape :func:`_dump_string` emits — so it is + # metered, not rejected, matching the shared seam. + total += len(_dump_string(current).encode("utf-8")) + elif type(current) is int: + # The canonical boundary accepts every JS-double-exact value: an int + # outside +-2**53-1 is fine IFF the double round-trip is exact. + if current > js_safe or current < -js_safe: + try: + exact = int(float(current)) == current + except OverflowError: + exact = False + if not exact: + return invalid("integer not exactly representable as a JavaScript number") + total += len(_dump_scalar(current).encode("utf-8")) + elif type(current) is float: + if current != current or current in (float("inf"), float("-inf")): + return invalid("non-finite float") + # JSON turns -0.0 into a sign the host parses back to JS -0; the + # canonical boundary rejects it, so this side must too. + if current == 0.0 and math.copysign(1.0, current) < 0: + return invalid("negative zero") + total += len(_dump_scalar(current).encode("utf-8")) + elif type(current) is list: + if id(current) in on_path: + return invalid("circular reference") + count = len(current) + total += 2 + (count - 1 if count > 1 else 0) + # Reject over-budget BEFORE enqueuing children: every element + # serializes to at least one byte, so a wide flat forgery fails here + # without materializing millions of leave frames first. + if total + count > max_bytes: + return over_budget + on_path.add(id(current)) + stack.append((current, True)) + stack.extend((child, False) for child in current) + elif type(current) is dict: + if id(current) in on_path: + return invalid("circular reference") + # ``len`` without materializing ``current.items()``: that list + # allocates one tuple per member before the bound below could run, + # recreating the spike the bound exists to stop. + count = len(current) + total += 2 + (count - 1 if count > 1 else 0) + # Same pre-enqueue bound: each entry contributes a quoted key + # (>= 2 bytes), a colon, and a >= 1-byte value. + if total + count * 4 > max_bytes: + return over_budget + on_path.add(id(current)) + stack.append((current, True)) + for key, item in current.items(): + # Only an EXACT str key survives: bool and int coerce or raise, + # and a str SUBCLASS can override the ``__len__`` the bound + # below reads while the encoder emits its real characters. + if type(key) is not str: + return invalid(f"non-string dict key ({type(key).__name__})") + # The same string lower bound, before escaping the key. + if total + len(key) + 3 > max_bytes: + return over_budget + total += len(_dump_scalar(key).encode("utf-8")) + 1 + stack.append((item, False)) + else: + # tuple, set, or any other type: not round-trippable JSON. + return invalid(f"unsupported type ({type(current).__name__})") + if total > max_bytes: + return over_budget + return None + + +class _Emit: + """A pre-rendered fragment on :func:`_encode_json_plain`'s explicit stack.""" + + __slots__ = ("text",) + + def __init__(self, text: str) -> None: + self.text = text + + +def _lossless_json_violation(value: Any) -> str | None: + """Return why ``value`` is not lossless JSON, or ``None`` when it is. + + ``json.dumps`` succeeding is NOT proof of losslessness: it coerces a + non-string ``dict`` key to its string form (``{1: "a", "1": "b"}`` collapses + to one key, silently dropping data), emits non-standard ``NaN``/``Infinity`` + tokens without ``allow_nan=False``, and accepts integers outside JavaScript's + safe range (``9007199254740993`` becomes ``...992`` once the host parses the + frame into a JS number). Validate the shape up front so a coercive or lossy + value fails as ``invalid-output`` instead of round-tripping to something the + program did not compute. Iterative so deep nesting cannot overflow the stack, + and it tracks the container ancestry on the current path so a cyclic value is + reported at once rather than spinning until the CPU budget. Only JSON-plain + types survive: ``None``/``bool``/JS-safe ``int``/finite ``float``/``str``, + exact ``list``, and exact ``dict`` with ``str`` keys. Every type matches + EXACTLY, containers and scalars alike, for the reason + :func:`_check_done_value` documents: a subclass can override the operators + and methods a traversal calls, so an ``isinstance`` admission here would + approve one shape and let the encoder emit another. + """ + + # The canonical boundary accepts every JS-double-exact value: an int + # outside +-2**53-1 is fine IFF the double round-trip is exact (2**53 or + # 2**60 survive; 2**53+1 rounds), matching the worker backend. + js_safe = 2**53 - 1 + # Post-order walk with an explicit "leave" marker: a container's id is added + # to `on_path` when entered and removed when left, so a back-edge to an + # ancestor (a cycle) is detected without rejecting a legitimately shared + # acyclic subtree. + on_path: set[int] = set() + # Each frame is (value, is_leave): a leave frame pops its container off the path. + stack: list[tuple[Any, bool]] = [(value, False)] + while stack: + current, is_leave = stack.pop() + if is_leave: + on_path.discard(id(current)) + continue + if current is None or type(current) is bool: + continue + if type(current) is str: + # Every string is lossless JSON. A lone surrogate has no UTF-8 form, + # but JSON carries the code unit as its ASCII ``\uXXXX`` escape and + # :func:`_dump_string` emits exactly that, so the host receives the + # same code unit the program passed — the same acceptance + # ``CodeJsonValue``, ``snapshotJsonValue``, and the worker backend + # already give it. + continue + if type(current) is int: + if current > js_safe or current < -js_safe: + try: + exact = int(float(current)) == current + except OverflowError: + exact = False + if not exact: + return "integer not exactly representable as a JavaScript number" + continue + if type(current) is float: + if current != current or current in (float("inf"), float("-inf")): + return "non-finite float" + # JSON serialization turns -0.0 into 0 (or "-0.0" text that the + # host parses to JS -0), silently changing the sign bit either + # way; the repository's canonical lossless-JSON boundary and the + # worker backend both reject it, so this side must too. + if current == 0.0 and math.copysign(1.0, current) < 0: + return "negative zero" + continue + if type(current) is list or type(current) is dict: + if id(current) in on_path: + return "circular reference" + on_path.add(id(current)) + stack.append((current, True)) + if type(current) is dict: + for key in current: + # Only an EXACT str key survives: int, float, None, and + # tuple keys coerce or raise, and a str subclass can carry + # overrides the encoder does not honor. + if type(key) is not str: + return f"non-string dict key ({type(key).__name__})" + stack.extend((child, False) for child in current.values()) + else: + stack.extend((child, False) for child in current) + continue + return f"unsupported type ({type(current).__name__})" + return None + + +def _make_cpu_enforcer() -> Any: + """Build the CPU post-check over closure-held primitives. + + This bootstrap IS ``__main__``, so model code can reach every one of its + module globals: ``import __main__; __main__._X = ...`` rebinds the name the + enforcement would otherwise read at call time, which a plain module-level + function plus module-level captures made a one-line defeat. The primitives + therefore live in this factory's locals, which become closure cells of the + returned function, and :func:`_run` binds the returned function into a + local of its own frame BEFORE executing the program, so no assignment to + ``__main__`` changes which callable runs or what it calls. Capture happens + at import time, before model code runs, so the captured + ``resource.getrusage``/``signal.signal``/``os.kill`` are the real builtins. + + This raises the cost of defeating the check; it does not make it + unreachable, and nothing in-process could. A cell is writable through + ``fn.__closure__[i].cell_contents``, and ``sys._getframe`` walks to + :func:`_run`'s frame and reads its locals, so a program determined to + tamper still can — consistent with this backend's documented posture, where + the in-process interpreter is containment rather than a security boundary + (§Trust posture in the Code Mode RFC). The bounds that model code cannot + forge are outside the interpreter: the RLIMIT_CPU HARD limit at + ``cpuSeconds + 1``, whose SIGKILL is undeliverable to a handler and + unraisable by a process that cannot raise its own hard limit, and the + host's wall-clock ceiling. This check exists to convert the two cases those + miss — a program that traps SIGXCPU and settles inside the soft-to-hard + gap, and a program that spends the budget in DESCENDANTS the kernel never + charged to this process — from a reported SUCCESS into the same `timeout` + an untrapped program gets. + + @returns The one-argument enforcement callable, taking `cpuSeconds`. + """ + + getrusage = resource.getrusage + rusage_self = resource.RUSAGE_SELF + rusage_children = resource.RUSAGE_CHILDREN + set_signal = signal.signal + sig_dfl = signal.SIG_DFL + sigxcpu = signal.SIGXCPU + kill = os.kill + getpid = os.getpid + + def die_if_cpu_exhausted(cpu_seconds: int) -> None: + """Die by re-delivered SIGXCPU when the CPU budget is already spent. + + Two cases reach here as a would-be SUCCESS. A model program can trap + SIGXCPU and return during the one-second soft-to-hard gap. And + ``RLIMIT_CPU`` is PER-PROCESS, inherited fresh by every child, so a + program calling ``subprocess`` or ``os.fork`` multiplies the run's CPU + budget by the number of descendants it starts: measured with + ``cpuSeconds: 1``, two sequential busy children burned 2.0 + CPU-seconds and the parent, which had accrued almost no CPU of its own + while blocked in ``subprocess.wait``, still returned a completion. + The meter is therefore ``RUSAGE_SELF + RUSAGE_CHILDREN``, the kernel's + own aggregate, which accumulates the CPU of every REAPED descendant + (grandchildren included, verified). + + ``getrusage`` is the kernel's own meter (unforgeable from model code), + and dying by SIGXCPU with the default disposition restored gives the + host the same kernel-authoritative close signal as the untrapped soft + limit — classified as `timeout`, after which the host's process-group + SIGTERM/SIGKILL teardown reaches any surviving descendants. Runs AFTER + the model program settled, so a program can re-trap SIGXCPU between + this SIG_DFL and the kill only by running more code, which it no longer + does. A program that tampers with this callable instead (see + :func:`_make_cpu_enforcer` on why in-process state cannot be hidden) + buys at most the remaining soft-to-hard gap: one more CPU second, after + which the hard limit's SIGKILL lands with no handler possible. + + Checking at settle time rather than sampling mid-run is deliberate: + both mid-run designs perturb the run they measure. A sampling thread + cost 72 MiB of virtual address space in the child (8 MiB stack plus a + 64 MiB glibc per-thread malloc arena reservation; measured 30.23 MiB of + mappings without it against 102.37 MiB with it), and ``RLIMIT_AS`` + counts reserved space, so it silently shrank every run's + `addressSpaceMb`. A ``SIGALRM`` interval timer costs no mappings but + makes the program's own syscalls return short under PEP 475 — measured + a 64 MiB ``os.write`` returning 65536 — which corrupts fd-3 framing. + The cost of checking only at settle time is that a descendant's CPU is + detected after it is spent, not while it runs; the host's wall-clock + ceiling bounds that interval, and a program that never reaps its child + is bounded by the wall clock alone, since ``RUSAGE_CHILDREN`` counts + only reaped descendants (verified: a still-running child contributes + 0.0). + + @param cpu_seconds The `cpuSeconds` budget the soft RLIMIT_CPU used. + """ + + own = getrusage(rusage_self) + kids = getrusage(rusage_children) + spent = own.ru_utime + own.ru_stime + kids.ru_utime + kids.ru_stime + if spent >= cpu_seconds: + set_signal(sigxcpu, sig_dfl) + kill(getpid(), sigxcpu) + + return die_if_cpu_exhausted + + +_DIE_IF_CPU_EXHAUSTED = _make_cpu_enforcer() + + +_TRUNCATION_MARKER = "… [truncated]" + +# The marker's own UTF-8 size, reserved out of the cap rather than added on top +# of it. Byte-identical to the host's TRUNCATION_MARKER_BYTES; the ellipsis is +# three bytes, so this is 15, not the string's 13 characters. +_TRUNCATION_MARKER_BYTES = len(_TRUNCATION_MARKER.encode("utf-8")) + + +def _cap_message(message: str, max_bytes: int) -> str: + """Byte-cap a diagnostic, appending the same marker the host uses. + + Encoded with ``errors="replace"`` first: a model exception message can + contain an unpaired surrogate (``raise Exception("\\ud800")``), and a + strict encode would throw while BUILDING the failure frame — the run + would then strand until the wall clock instead of reporting the + exception. Then a UTF-8 slice with a trailing partial sequence dropped + by ``errors="ignore"``; the marker text matches the host-side + ``capMessage`` so a truncated diagnostic reads identically wherever the + cap was applied. + + The marker's bytes come OUT of ``max_bytes``, so the returned string as a + whole honors the cap; retaining a full cap of text and then appending the + marker would exceed the bound this function enforces, and the host meters + the same field again on arrival. A ``max_bytes`` below the marker's own + size leaves no room for message text and yields the marker alone, so the + true bound is ``max(max_bytes, 15)`` — reporting that truncation happened + is worth those 15 bytes. + """ + + raw = message.encode("utf-8", errors="replace") + if len(raw) <= max_bytes: + return raw.decode("utf-8") + budget = max(0, max_bytes - _TRUNCATION_MARKER_BYTES) + return raw[:budget].decode("utf-8", errors="ignore") + _TRUNCATION_MARKER + + +# Fixed safety/liveness bound, not a tunable: a model can raise an exception +# with an arbitrarily deep __cause__/__context__ chain, and both the rendering +# walk and format() are linear in chain length. Capping how many links get +# RENDERED keeps traceback formatting from consuming the whole wall budget. +# 100 links is far beyond any legible human traceback. +_MAX_TRACEBACK_CHAIN = 100 + +# Diagnostic used when rendering the failure itself fails. Built from a fixed +# literal plus the exception CLASS name, never from the exception's own str. +_UNRENDERABLE_DIAGNOSTIC = "" + + +def _model_traceback(exc: BaseException, max_bytes: int) -> str: + """Format a model-program failure with only the MODEL's own frames. + + Bootstrap frames carry host-absolute paths — meaningless to the model and + unstable across machines, so transcripts pinning them cannot replay. They + appear not only as a leading prefix (the bootstrap's ``exec``/``await``) + but also interleaved and trailing: an uncaught binding rejection re-raised + by ``dispatch`` puts bootstrap frames AFTER the model's, and chained + ``__cause__``/``__context__`` exceptions carry their own stacks. Filter + every non-```` frame across the whole chain rather than trimming a + prefix. A failure with no model frame anywhere (e.g. a SyntaxError raised + by ``compile``) keeps the standard exception-only rendering. + + Rendering is bounded to ``_MAX_TRACEBACK_CHAIN`` links, cut on the + ``TracebackException`` COPY, and a marker line announces the truncation. + Nothing here touches the live exception: an exception class overriding + ``__setattr__`` would run MODEL code from inside the caller's failure + handler, and a throw there costs the ``done`` frame (see + ``_safe_model_traceback``). ``TracebackException`` instances hold no such + hooks, so clearing their links runs no model code. The walk is iterative, + so a deep chain cannot overflow the recursion limit. + + ``from_exception`` still copies the WHOLE live chain, at a higher per-link + cost than building it took. That is bounded by the child's ``RLIMIT_AS``: + the model must materialize every link (exception object plus traceback) + before raising, so a chain long enough for the copy to matter is already + near the address-space cap, and a ``MemoryError`` in the copy lands in the + caller's fallback rather than stranding the run. + """ + + te = traceback.TracebackException.from_exception(exc) + # One iterative pass over the copy does both jobs: keep only frames + # on every linked exception and group member, and cut the chain at the cap. + found = False + truncated = False + pending = [(te, 1)] + while pending: + entry, depth = pending.pop() + kept = [f for f in entry.stack if f.filename == ""] + entry.stack = traceback.StackSummary.from_list(kept) + found = found or bool(kept) + # 3.11+ exception groups (a binding failure inside asyncio.TaskGroup) + # carry member stacks under `exceptions`, not the dunder links; a group + # member counts as a link so the cap bounds nesting through both edges. + members = getattr(entry, "exceptions", None) or () + if depth >= _MAX_TRACEBACK_CHAIN: + if entry.__cause__ is not None or entry.__context__ is not None or members: + truncated = True + entry.__cause__ = None + entry.__context__ = None + if members: + entry.exceptions = None + continue + for linked in (entry.__cause__, entry.__context__): + if linked is not None: + pending.append((linked, depth + 1)) + for member in members: + pending.append((member, depth + 1)) + + def emit(): + if found: + yield from te.format() + else: + yield from traceback.format_exception_only(type(exc), exc) + if truncated: + yield f"[dsh-code-runtime-python] exception chain truncated at {_MAX_TRACEBACK_CHAIN} links\n" + + return _join_bounded(emit(), max_bytes) + + +def _make_failure_reporter() -> Any: + """Build the failure-diagnostic renderer over closure-held primitives. + + The returned callable renders a model failure diagnostic that cannot itself + raise. The caller sends the ``done`` frame AFTER its ``except BaseException`` + block, so anything thrown while rendering the diagnostic skips the send + entirely: the host then blocks on fd 3 until ``maxWallMs`` and reports a + timeout instead of the exception that actually happened. Rendering runs + model code by design (``format()`` reaches ``__str__``, ``__repr__`` and + ``__notes__``) and allocates under ``RLIMIT_AS``, so it must be treated as + able to throw. + + The fallback names the exception CLASS and a fixed literal — no ``str(exc)`` + and no ``format_exception_only``, both of which reach the model's + ``__str__``. A ``__name__`` that is not exactly ``str`` (a metaclass + property can return anything, or raise) is discarded rather than + formatted, so no override runs on this path either. + + The factory exists for the same reason :func:`_make_cpu_enforcer` does: this + bootstrap IS ``__main__``, so ``import __main__; __main__._X = ...`` rebinds + any module global a call-time lookup would read. On this path a rebind is + worst — the handler's own reporter, and everything the reporter reaches, + would run model code outside any guard, and a throw there costs the ``done`` + frame. The traceback formatter, the byte cap and the fallback literal + therefore become closure cells captured at import time, before model code + runs, and :func:`_run` binds the returned callable into a local of its own + frame. A frame local is not a module attribute, so no assignment to + ``__main__`` changes which callable runs or what it calls. This defeats the + one-line rebind, not a determined ``sys._getframe`` walk; the unforgeable + bound is the host wall clock. + """ + + cap_message = _cap_message + model_traceback = _model_traceback + unrenderable = _UNRENDERABLE_DIAGNOSTIC + + def safe_model_traceback(exc: BaseException, max_bytes: int) -> str: + try: + return cap_message(model_traceback(exc, max_bytes), max_bytes) + except BaseException: # noqa: BLE001 -- a throw here would cost the done frame + pass + try: + raw_name = type(exc).__name__ + # Slice BEFORE interpolating. A metaclass `__name__` property can + # return an arbitrarily long string, and both the f-string and + # `cap_message`'s encode would copy it whole — under a tight + # RLIMIT_AS either allocation can raise MemoryError, and this is the + # LAST fallback, so a throw here costs the `done` frame outright and + # the run misreports as an exit or a timeout. The slice is a + # code-unit prefix, which bounds the bytes at 4x, and the following + # `cap_message` still applies the exact byte cap. + name = raw_name[:_MAX_FALLBACK_NAME_CHARS] if type(raw_name) is str else "" + except BaseException: # noqa: BLE001 -- a raising __name__ must not cost the done frame + name = "" + # Wrapped for the same reason: `cap_message` encodes, and its allocation + # is the only step left that can still fail. The fixed literal needs no + # budget, so it can always be delivered. + try: + return cap_message(f"{name}: {unrenderable}", max_bytes) + except BaseException: # noqa: BLE001 -- the done frame outranks the diagnostic's detail + return unrenderable + + return safe_model_traceback + + +_SAFE_MODEL_TRACEBACK = _make_failure_reporter() + + +def _join_bounded(lines, max_bytes: int) -> str: + """Join formatter output, stopping once the budget is comfortably passed. + + ``format()`` yields lines lazily; consuming it whole for an exception + carrying a huge message would materialize the full text only for + ``_cap_message`` to throw it away — enough over-shoot to exhaust + ``RLIMIT_AS``. Stop after the accumulated CHARACTER count passes the byte + budget (chars lower-bound UTF-8 bytes); the caller's ``_cap_message`` + does the exact byte-level cut. + """ + + chunks: list[str] = [] + total = 0 + for line in lines: + # A single yielded line can itself dwarf the budget (the exception + # message rides in one line): keep only the prefix it can ever need. + if len(line) > max_bytes + 1: + line = line[: max_bytes + 1] + chunks.append(line) + total += len(line) + if total > max_bytes: + break + return "".join(chunks) + + +def _done_with_value(value: Any, max_value_bytes: int) -> dict[str, Any]: + """Build the terminal done frame under the seam's lossless-JSON contract. + + A completion value returned by the program (``None`` when it returns + nothing) that is not lossless JSON fails the run as ``invalid-output``; a + serialized value beyond ``max_value_bytes`` fails as ``output-limit``. + Substituting a ``repr`` or truncated string would be a silent lie about + what the program computed, so both paths refuse instead (mirroring the + worker backend's contract). ``None`` crosses as an exact JSON ``null``. + """ + + # One bounded walk folds the losslessness check and the byte meter (mirrors + # the host's checkDoneValue): the former split ran the full losslessness + # walk first, materializing one tuple per element for a wide completion + # before the size cap could reject it — an RLIMIT_AS death on a value the + # meter would have refused. send_sync later encodes the admitted value, + # whose size the walk proved within budget. Iterative like the encoder, so a + # valid completion deeper than the recursion limit still checks. + rejection = _check_done_value(value, max_value_bytes) + if rejection is not None: + kind, message = rejection + return {"type": "done", "error": {"kind": kind, "message": message}} + return {"type": "done", "value": value} + + +def main() -> None: + channel = ProtocolChannel(PROTOCOL_FD) + asyncio.run(_run(channel)) + + +if __name__ == "__main__": + main() diff --git a/packages/code-runtime/code-runtime-python/src/index.ts b/packages/code-runtime/code-runtime-python/src/index.ts index 8a3e99f2d1..b40e466cc4 100644 --- a/packages/code-runtime/code-runtime-python/src/index.ts +++ b/packages/code-runtime/code-runtime-python/src/index.ts @@ -1,5 +1,9 @@ /** - * CPython subprocess code runtime for the DeepSeek Harness code-execution seam. + * CPython subprocess code runtime: a fresh `python3` process runs each model program under an + * asyncio event loop with top-level ``await``. Binding calls travel on fd 3 as JSON-lines, + * leaving stdout/stderr free for the program's own output. This is containment, not a security + * boundary: model code has bash-equivalent trust, contained by an empty environment, RLIMIT_CPU + * + RLIMIT_AS, wall-clock timeout, and SIGTERM→grace→SIGKILL on the process group. * * The package owns the versionless fd-3 wire protocol between the Node host and * the CPython subprocess. The protocol's host-side codec and hostile-frame @@ -8,6 +12,24 @@ * @module @deepseek-ai/dsh-code-runtime-python */ +import { spawn } from 'node:child_process' +import { StringDecoder } from 'node:string_decoder' +import { accessSync, copyFileSync, constants as fsConstants, mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { delimiter, dirname, isAbsolute, join } from 'node:path' +import { fileURLToPath } from 'node:url' +import type { Duplex } from 'node:stream' +import { Context } from 'cordis' +import z from 'schemastery' +import { CodeRuntime, DUNDER_MEMBER, PORTABLE_RESERVED_WORDS, RESERVED_BINDING_GLOBALS, RESERVED_ERROR_MEMBERS } from '@deepseek-ai/dsh-code-runtime' +import type { CodeBindingErrorClass, CodeBindingFunction, CodeJsonValue, CodeRunFailure, CodeRunRequest, CodeRunResult } from '@deepseek-ai/dsh-code-runtime' +import { snapshotJsonValue } from '@deepseek-ai/dsh-session' +import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout' +import type { BootMessage, ChildToHost, ReplyMessage } from './protocol.ts' +import { checkDoneValue, encodeJsonPlain, hasUnsafeIntegerToken, logTruncationMarker, validateChildFrame } from './protocol.ts' + +// Re-export the fd-3 wire vocabulary so the runtime and its tests share one +// import surface; the protocol layer owns the definitions. export type { BootMessage, ChildToHost, ReplyMessage } from './protocol.ts' export { checkDoneValue, @@ -17,3 +39,1115 @@ export { logTruncationMarker, validateChildFrame, } from './protocol.ts' + +/** Plugin config: every cap, changeable from `cordis.yml` (no hardcoded tunables). */ +export interface Config { + /** + * RLIMIT_CPU in whole seconds (a positive integer — `setrlimit` in the child + * rejects a float). The child sets the soft limit to `cpuSeconds` and the + * hard limit to `cpuSeconds + 1`: the kernel delivers SIGXCPU at the soft + * limit, which the host classifies as a `timeout`; the +1s hard limit is a + * SIGKILL backstop for a program that traps SIGXCPU. Granularity is seconds — + * a coarser counterpart to the worker backend's millisecond `computeMs`. + */ + cpuSeconds?: number + /** Wall-clock ceiling in milliseconds; backstops CPU time for programs awaiting a promise nobody resolves. */ + maxWallMs?: number + /** + * RLIMIT_AS in mebibytes; caps address space so a runaway allocation fails + * cleanly. Not applied on Darwin, where the dyld shared cache mapped into + * every process at exec exceeds any practical cap and the kernel rejects + * the call; `cpuSeconds` and `maxWallMs` still bound the run there. + */ + addressSpaceMb?: number + /** Shared byte budget for captured log text (host-side ledger). */ + maxLogBytes?: number + /** Byte cap for the completion value. */ + maxValueBytes?: number + /** SIGTERM→SIGKILL grace period on kill, matching bash-local's default. */ + graceMs?: number + /** + * Absolute path or basename of the CPython interpreter to spawn. Resolved + * through `PATH` when a basename is given. + */ + pythonBin?: string +} + +/** {@link Config} with all defaults filled. */ +type ResolvedConfig = Required + +/** + * The seam's language-portable identifier subset (see + * `CodeBindingNamespace.global`) — identical to Python's identifier grammar, + * so the shared contract needs no per-backend mapping here. + */ +const IDENTIFIER = /^[A-Za-z_][A-Za-z0-9_]*$/ + +/** + * The seam's cross-language reserved-word union: the portable-identifier + * contract promises a namespace list valid here is valid on every backend, so + * a JS keyword like `typeof` is refused even though it is a legal Python name. + */ +const RESERVED_NAMES = PORTABLE_RESERVED_WORDS + +/** + * The seam's shared backend-owned globals (`console` is the worker's slot; + * `__dsh_main__`/`__builtins__`/`__name__` are this bootstrap's wrapper and + * seeded module globals). Shared so a namespace list valid on one backend is + * valid on all — colliding with an owned slot would be silently overwritten + * (or overwrite builtins), so the seam rejects them up front. + */ +const RUNTIME_OWNED_GLOBALS = RESERVED_BINDING_GLOBALS + +/** + * The seam's shared error-member exclusions (`RESERVED_ERROR_MEMBERS` + + * dunder-form names) — enforced identically here and in the worker backend so + * an errorClass valid on one backend is valid on all. Several dunders are + * constrained CPython descriptors whose `setattr` raises while constructing + * the very rejection it was meant to carry; the exact set is an interpreter + * version detail, hence the dunder-wide rule at the seam. + */ +const EXCEPTION_RESERVED_MEMBERS = RESERVED_ERROR_MEMBERS + +const DUNDER = DUNDER_MEMBER + +/** + * The `py/` scripts the interpreter must be able to open: the entry script plus + * every module it imports from its own directory. Kept beside the built JS so a + * consumer package with `files: ['lib', 'py']` ships both. + */ +const PY_SCRIPTS = ['bootstrap.py', 'protocol.py'] + +/** + * Copy the `py/` scripts to a real filesystem directory and return the entry + * script's path there. + * + * The interpreter is an EXTERNAL process, so it can only open paths the OS + * resolves. Inside the single-file Python-SDK executable, `import.meta.url` + * resolves into pkg's virtual filesystem, which Node reads through its patched + * `fs` but `python3` cannot see at all — the spawn fails with ENOENT on a path + * that exists as far as the host is concerned. `bootstrap.py` additionally + * inserts its own directory on `sys.path` to import the sibling `protocol.py`, + * so both files must land in the SAME real directory. + * + * The copy is unconditional rather than gated on a bundled-runtime probe: the + * read goes through Node's `fs` either way, and one code path means the + * packaged deployment runs what the tests exercise. Placement is under + * `os.tmpdir()` with `0o700` keeps the scripts off other users' reach, but NOT + * the model's: the child runs as the same UID as the host, so a program can + * rewrite the very files it was started from. Hence one copy per RUN, discarded + * at settlement — a rewrite then damages only the run that performed it, which + * is what fresh-subprocess-per-run already promises. Sharing one copy across + * runs made an overwritten `bootstrap.py` break the next run. + * + * Deliberately SYNCHRONOUS. An `await` here would open an async boundary in + * `execute` before the run is registered in `live` and before the abort + * listener is installed, so a disposal or an abort landing in that window would + * be missed: `teardown` would see no runs and return while the continuation + * went on to spawn a subprocess, and an `addEventListener('abort')` installed + * afterwards does not replay an event that already fired. Three small + * filesystem operations per run are not worth that class of race, and `execute` + * already runs synchronously up to `spawn`. + * + * A failed copy removes the directory here, so a partial attempt never outlives + * the call that made it; a successful one is the caller's to remove, which it + * derives from the returned path. + * + * @returns the absolute path of the materialized entry script. + */ +function materializePyScripts(): string { + const dir = mkdtempSync(join(tmpdir(), 'dsh-code-runtime-python-')) + const source = fileURLToPath(new URL('../py/', import.meta.url)) + try { + for (const name of PY_SCRIPTS) copyFileSync(join(source, name), join(dir, name)) + } catch (error: unknown) { + try { + rmSync(dir, { recursive: true, force: true }) + } catch { + // Swallows only a failure to remove the partial staging directory. The + // caller reports the copy failure that got us here, which is the + // diagnosable one; nothing else can act on a temp dir we cannot unlink. + } + throw error + } + return join(dir, 'bootstrap.py') +} + +/** + * The fd-3 receive ceiling for one unframed line: a pure host-memory-safety + * bound, NOT an output budget. Binding `call` frames legitimately carry large + * arguments (the seam puts no byte cap on binding traffic), so the ceiling + * must sit far above any plausible frame while still stopping a hostile + * newline-free flood from growing the host accumulator without bound — the + * child's RLIMIT_AS bounds the child, not the host string. 256 MiB mirrors + * the order of the worker backend's default outer-output cap and V8's string + * ceiling neighborhood; completion values have their own `maxValueBytes` + * check at the `done` handler, deliberately decoupled from this. Not a config + * knob because it is an internal framing invariant, not a deployment choice. + */ +const FRAME_CEILING_BYTES = 256 * 1024 * 1024 + +/** + * Fragments the unframed fd-3 buffer may hold before they are coalesced into + * one Buffer, bounding retained per-chunk overhead that {@link + * FRAME_CEILING_BYTES} cannot see: that ceiling meters payload bytes, while + * each chunk is a distinct Buffer with its own object and backing store. A + * program writing single bytes without a newline produced one chunk per write. + * 1024 keeps the overhead a small constant factor of the payload while leaving + * normal pipe-sized reads (which arrive in far fewer, much larger chunks) + * untouched. A framing invariant, not a deployment choice. + */ +const MAX_PENDING_CHUNKS = 1024 + +/** + * Bytes a frame spends on its own JSON structure around a capped payload, used + * to bound `maxLogBytes`/`maxValueBytes` against {@link FRAME_CEILING_BYTES}. + * The widest carrier is `{"type":"log","text":"","truncated":true}` at 41 + * bytes; 64 rounds that up so adding a field to either frame does not silently + * invalidate the bound. A protocol constant, not a deployment choice. + */ +const FRAME_ENVELOPE_BYTES = 64 + +/** + * The most bytes one payload character can occupy once JSON-escaped: a control + * character renders as `\uXXXX`. Used with {@link FRAME_ENVELOPE_BYTES} to turn + * the frame ceiling into an admissible output cap. + */ +const MAX_JSON_ESCAPE_EXPANSION = 6 + +/** + * Extra time added to `graceMs` before the post-kill close-deadline force-settles + * a run whose `close` never fires (a setsid-escaped orphan holds our inherited + * stdio; see the `closeDeadline` arm in {@link PythonCodeRuntime.execute}). It + * covers the OS reaping the killed child itself after SIGKILL — not a deployment + * choice but a fixed safety margin, so it is a constant rather than a config knob. + */ +const CLOSE_REAP_MARGIN_MS = 2_000 + +/** + * Extract a human message from an unknown thrown value. + * + * `String(error)` runs the value's own conversion, and a host binding may reject + * with an object whose `Symbol.toPrimitive` or `toString` throws. One call site + * is a detached async reply callback, where that throw escapes as an unhandled + * rejection: the reply frame is never written, the program stays blocked on + * `await`, and the run degrades to a `maxWallMs` timeout (a Node host without an + * `unhandledRejection` listener exits outright). The conversion is therefore + * wrapped, with a fixed literal as the fallback — the value already proved it + * cannot be rendered, so nothing derived from it is safe to try. + * + * `Error.message` is typed `string` but is a plain writable property, so a + * rejecting binding can hand back an `Error` carrying any value there. The + * `Error` arm therefore goes through the same conversion rather than returning + * `message` verbatim: the returned string crosses the wire under + * `encodeJsonPlain`'s JSON-plain precondition, where a cyclic object grows the + * encoder stack until the host exhausts memory and any other unsupported value + * prevents the reply frame outright. + * + * The same conversion renders abort reasons, which reach an `AbortSignal` + * listener: Node reports a throw from such a listener as an uncaught exception, + * so an unwrapped conversion there can terminate the host with the run left + * unsettled. + * + * @param error The thrown value, of unknown shape. + * @returns The value's message or string form; a fixed placeholder when its own + * conversion throws. + */ +function messageOf(error: unknown): string { + try { + return String(error instanceof Error ? error.message : error) + } catch { + // Swallows only a throw from the value's own `message` getter or string + // conversion. Nothing else runs inside the try, and the placeholder is a + // literal, so this cannot throw again. + return '' + } +} + +/** + * Resolve `pythonBin` to an absolute path against the CURRENT process `PATH`, + * BEFORE the child spawns with an empty environment. A basename (the default + * `python3`) would otherwise fail: `env: {}` drops `PATH`, so Node's own lookup + * falls back to the platform default (`/usr/bin:/bin`) and misses interpreters + * that live only on the caller's `PATH` (Nix, pyenv, Homebrew, conda). An + * absolute or explicitly relative path is used verbatim. When no `PATH` entry + * holds an executable match, the original value is returned unchanged so the + * spawn produces its normal ENOENT `error` event (a settled `worker-exit`), + * not a thrown exception here. + * @param bin - the configured interpreter (absolute path or bare command). + * @returns an absolute path when resolvable, else `bin` unchanged. + */ +function resolvePythonBin(bin: string): string { + if (isAbsolute(bin) || bin.includes('/')) return bin + const path = process.env.PATH + /* v8 ignore next -- PATH is set in every environment the runtime boots in; the guard is defensive. */ + if (path === undefined) return bin + for (const dir of path.split(delimiter)) { + // An empty PATH segment (a `::`, implicitly CWD on POSIX) is skipped so a + // basename never resolves against the working directory; normal PATHs + // carry no empty segment. + /* v8 ignore next -- normal PATHs carry no empty segment. */ + if (dir === '') continue + const candidate = join(dir, bin) + try { + accessSync(candidate, fsConstants.X_OK) + return candidate + } catch { + // Not executable here; try the next PATH entry. + } + } + return bin +} + +/** The marker appended when a diagnostic message is byte-capped host-side. */ +const TRUNCATION_MARKER = '… [truncated]' + +/** + * The marker's own UTF-8 byte length, reserved out of the budget so a capped + * message stays WITHIN `maxValueBytes` rather than exceeding it by the marker. + * The ellipsis is 3 bytes, so this is 15, not the string's 13 code units. + */ +const TRUNCATION_MARKER_BYTES = Buffer.byteLength(TRUNCATION_MARKER, 'utf8') + +/** + * Cap a done-frame `error.message` to `maxValueBytes` host-side: a forged done + * frame can carry an arbitrarily long message, so truncate by byte length and + * append the shared marker on overflow. Completion VALUES are never truncated + * — the seam forbids substitution, so an oversized value fails the run as + * `output-limit` instead (see the done case in `execute`). + * + * The marker's bytes are RESERVED from the budget, not added on top: the whole + * returned string, marker included, is at most `maxValueBytes` bytes. Appending + * the marker after retaining a full budget's worth of text would overrun the + * very cap this function exists to enforce. The one exception is a configured + * cap SMALLER than the marker itself, which leaves no room for message text at + * all; the marker alone is returned there, so the bound is + * `max(maxValueBytes, 15)`. Reporting the truncation is worth those 15 bytes, + * and the default cap is 32 KiB. + * @param message - the error message from an inbound (possibly forged) done frame. + * @param maxValueBytes - the configured completion-value budget, reused here. + * @returns the message unchanged, or its byte-capped form on overflow. + */ +function capMessage(message: string, maxValueBytes: number): string { + // Code-unit bounds BEFORE any encode, so a forged done frame carrying a + // message anywhere below the 256 MiB fd-3 frame ceiling cannot force a + // full-length UTF-8 copy under a 32 KiB cap. One UTF-16 code unit encodes to + // at least one UTF-8 byte and at most three: three for a non-ASCII BMP + // character, two apiece for the pair halves sharing an astral code point's + // four bytes, and three for a LONE surrogate, which `Buffer.from` renders as + // U+FFFD. So at most maxValueBytes/3 code units cannot overflow the cap and + // need no encode at all... + if (message.length * 3 <= maxValueBytes) return message + // ...and nothing past the first maxValueBytes code units can fit inside it, + // so only that prefix is ever encoded — at most 3 * maxValueBytes bytes. + const keep = Math.min(message.length, maxValueBytes) + const whole = keep === message.length + const bytes = Buffer.from(whole ? message : message.slice(0, keep), 'utf8') + // A message that fits is measured against the WHOLE cap: it gets no marker, + // so reserving marker bytes here would truncate text that was within budget. + if (whole && bytes.length <= maxValueBytes) return message + // Past this point the message IS being truncated, so the marker WILL be + // appended and its bytes come out of the cap instead of sitting on top of it. + const budget = Math.max(0, maxValueBytes - TRUNCATION_MARKER_BYTES) + // Trim back to the last complete UTF-8 sequence: a cut through a multibyte + // character would decode as U+FFFD — corrupting the diagnostic AND + // exceeding the byte cap, since the replacement character itself encodes + // to three bytes. Continuation bytes are 0b10xxxxxx; at most three of them + // precede a lead byte. + // + // This also covers a code-unit prefix ending on a HIGH SURROGATE whose low + // half sits outside it, which `Buffer.from` encodes as U+FFFD: that orphan + // occupies the last three bytes of `bytes`, and `bytes` is at least + // `maxValueBytes + 2` long here (one byte per retained unit, three for the + // orphan), so it starts past `budget` and is always cut. Reserving the + // marker is what makes that hold; cutting at `maxValueBytes` itself did not, + // and needed an explicit surrogate check. + let end = Math.min(budget, bytes.length) + while (end > 0 && ((bytes[end] as number) & 0b1100_0000) === 0b1000_0000) end-- + return `${bytes.subarray(0, end).toString('utf8')}${TRUNCATION_MARKER}` +} + +/** + * Copy an fd-3 line residual into a fresh, right-sized Buffer so it no longer + * shares the joined-frame allocation it was sliced from. + * + * After the newline loop over a `Buffer.concat` of the pending chunks, the + * leftover partial line is a `subarray` VIEW onto that concat's backing store. + * A view keeps the ENTIRE backing allocation alive for as long as it is + * retained, so carrying the view forward as the next pending chunk would pin a + * whole large frame's worth of memory behind a tiny trailing fragment — and the + * `pendingBytes` counter, set to the fragment's own length, would no longer + * measure the memory actually held. `Buffer.from` allocates exactly + * `residual.length` bytes and copies, letting the concat allocation be + * collected; an empty residual carries nothing forward. + * @param residual - the leftover slice after the last newline (a view). + * @returns the pending-chunk list to carry forward: `[copy]`, or `[]` when empty. + */ +export function detachResidual(residual: Buffer): Buffer[] { + return residual.length > 0 ? [Buffer.from(residual)] : [] +} + +/** One namespace after seam validation: its callables plus the optional typed-rejection contract. */ +interface ValidatedNamespace { + functions: Record + errorClass?: CodeBindingErrorClass +} + +/** + * One in-flight run's host-side state, tracked for disposal so teardown can + * fail every live run as `abort` and AWAIT each child's exit. + */ +interface LiveRun { + kill(sig: NodeJS.Signals): void + settle(failure: CodeRunFailure): void + finished: Promise +} + +/** + * The shipped {@link CodeRuntime} backend registering as `codeRuntime`. Every + * cap is validated config; every long-running operation honors the request's + * `AbortSignal`; every disposer awaits child-process exit. + */ +export class PythonCodeRuntime extends CodeRuntime { + static Config: z = z.object({ + cpuSeconds: z.number().default(60), + maxWallMs: z.number().default(600_000), + addressSpaceMb: z.number().default(512), + maxLogBytes: z.number().default(65_536), + maxValueBytes: z.number().default(32_768), + graceMs: z.number().default(3_000), + pythonBin: z.string().default('python3'), + }) + + readonly language = 'python' + readonly isolation = 'process' + + private readonly config: ResolvedConfig + private readonly live = new Set() + private disposed = false + + /* jscpd:ignore-start -- parallel to code-runtime-worker: sibling backends keep symmetric constructor/teardown/run shapes. */ + constructor(ctx: Context, config: Config) { + super(ctx) + // Reject at load on Windows: the bootstrap imports the POSIX-only `resource` + // module for RLIMIT_CPU/RLIMIT_AS, spawns with a positional fd 3, and + // terminates via negative-PID process-group signals — none of which exist + // on Windows. Registering ctx.codeRuntime there would let assembly succeed + // and defer the failure to the first run. The asymmetry with the worker + // backend is intentional: that backend is cross-platform; this one is not. + if (process.platform === 'win32') { + throw new Error('dsh-code-runtime-python: this backend requires a Unix platform (POSIX rlimits, fd-3 stdio, process-group signals); it cannot run on Windows') + } + this.config = config as ResolvedConfig + for (const [key, value] of Object.entries(this.config)) { + if (typeof value === 'number' && !(Number.isFinite(value) && value > 0)) { + throw new Error(`dsh-code-runtime-python: config.${key} must be a positive number, got ${String(value)}`) + } + } + // cpuSeconds crosses to the child's setrlimit(RLIMIT_CPU) raw; a float + // raises TypeError inside every child (a late per-run failure). Reject it + // at load. Other numeric caps are consumed as numbers host-side or + // int()-truncated in the bootstrap, so they need no integer gate. + if (!Number.isInteger(this.config.cpuSeconds)) { + throw new Error(`dsh-code-runtime-python: config.cpuSeconds must be a positive integer, got ${String(this.config.cpuSeconds)}`) + } + // Finite is not the same as representable as an rlimit. `cpuSeconds` and its + // `+ 1` hard limit both cross to `setrlimit` as integers, and `1e100` clears + // `Number.isInteger` while being far past the safe range, so it cannot round + // -trip: the child sees a different number than was configured. The `+ 1` is + // what gets checked because that is the larger of the two values sent. + if (!Number.isSafeInteger(this.config.cpuSeconds + 1)) { + throw new Error(`dsh-code-runtime-python: config.cpuSeconds must be at most ${Number.MAX_SAFE_INTEGER - 1} (it and its +1 hard limit cross to setrlimit as exact integers), got ${String(this.config.cpuSeconds)}`) + } + // `addressSpaceMb` is multiplied by 1 MiB before it is framed, and a large + // finite value overflows to `Infinity` there — which `encodeJsonPlain` + // renders as `null`, so the child receives no limit at all and every run + // ends in a bootstrap exception rather than a load-time configuration error. + // Checking the DERIVED byte count is what catches it; the input itself looks + // ordinary. Safe-integer, not merely finite, since the value must survive + // the JSON round trip exactly. + if (!Number.isSafeInteger(this.config.addressSpaceMb * 1024 * 1024)) { + throw new Error(`dsh-code-runtime-python: config.addressSpaceMb must be at most ${Math.floor(Number.MAX_SAFE_INTEGER / (1024 * 1024))} (its byte count crosses the wire as an exact integer), got ${String(this.config.addressSpaceMb)}`) + } + // `pythonBin` reaches `spawn` as the executable path, where two values the + // string schema admits fail late and unhelpfully. An empty string makes + // `spawn` throw `ERR_INVALID_ARG_VALUE` synchronously, and an embedded NUL + // throws `ERR_INVALID_ARG_TYPE` — both from inside `run()`, so the method + // REJECTS instead of resolving the `worker-exit` the seam promises for a + // child that cannot start. An empty basename also makes `resolvePythonBin` + // probe every PATH directory itself for the X_OK bit. Both are + // self-contained configuration errors, so they fail at load. + if (this.config.pythonBin === '' || this.config.pythonBin.includes('\0')) { + throw new Error(`dsh-code-runtime-python: config.pythonBin must be a non-empty path without NUL bytes, got ${JSON.stringify(this.config.pythonBin)}`) + } + // `maxWallMs` and `graceMs` are armed with setTimeout, which clamps any + // delay past MAX_TIMER_DELAY_MS to 1 ms without a word — turning a + // generous ceiling into an instant timeout and a generous grace period into + // an instant SIGKILL. `graceMs` is checked against the margin the + // close-deadline adds on top, since that sum is what gets armed. + if (this.config.maxWallMs > MAX_TIMER_DELAY_MS) { + throw new Error(`dsh-code-runtime-python: config.maxWallMs must not exceed ${MAX_TIMER_DELAY_MS} (setTimeout clamps a larger delay to 1ms), got ${String(this.config.maxWallMs)}`) + } + if (this.config.graceMs + CLOSE_REAP_MARGIN_MS > MAX_TIMER_DELAY_MS) { + throw new Error(`dsh-code-runtime-python: config.graceMs must not exceed ${MAX_TIMER_DELAY_MS - CLOSE_REAP_MARGIN_MS} (its close deadline adds ${CLOSE_REAP_MARGIN_MS}ms, and setTimeout clamps a larger delay to 1ms), got ${String(this.config.graceMs)}`) + } + // The output caps are budgets for a payload that has to cross fd 3 inside + // one frame, and the framing ceiling is fixed. A cap above what a frame can + // carry is unsatisfiable: a completion or log entry that the cap admits + // arrives as an over-ceiling frame and fails the run as `worker-exit` + // instead of the `output-limit` the cap describes — a silent inversion, so + // it fails at load. The bound subtracts the frame's own envelope, since the + // ceiling covers the whole line: worst case is every payload byte escaping + // to six (`\uXXXX` per control character), so the admissible cap is + // `(ceiling - envelope) / 6`. + for (const key of ['maxLogBytes', 'maxValueBytes'] as const) { + const limit = Math.floor((FRAME_CEILING_BYTES - FRAME_ENVELOPE_BYTES) / MAX_JSON_ESCAPE_EXPANSION) + if (this.config[key] > limit) { + throw new Error(`dsh-code-runtime-python: config.${key} must not exceed ${limit} (a payload that large cannot cross the ${FRAME_CEILING_BYTES}-byte fd-3 frame ceiling, so the run would fail as worker-exit rather than output-limit), got ${String(this.config[key])}`) + } + } + ctx.effect(() => () => this.teardown(), 'python code-runtime teardown') + } + + /** + * Dispose to quiescence: fail every in-flight run as aborted and AWAIT each + * child's exit so no subprocess outlives the fiber. + */ + private async teardown(): Promise { + this.disposed = true + const runs = [...this.live] + for (const run of runs) run.settle({ kind: 'abort', message: 'runtime disposed' }) + // Awaiting `finished` is also what clears staging: that promise resolves + // inside the run's own `settle`, which removes its directory first. So there + // is deliberately no sweep here — a second pass could only ever find an + // empty set, and an unreachable cleanup path is worse than none, since it + // reads as the real guarantee while never running. + await Promise.all(runs.map(run => run.finished)) + } + + /** + * Execute one program in a fresh Python subprocess. Program outcomes resolve + * with `result.error`; the method rejects only for seam misuse. + */ + async run(request: CodeRunRequest): Promise { + if (this.disposed) throw new Error('dsh-code-runtime-python: run() after disposal') + const bindings = this.validateBindings(request) + if (request.signal?.aborted) { + return { logs: [], error: { kind: 'abort', message: messageOf(request.signal.reason) } } + } + let bootstrapPath: string + try { + // The interpreter is an external process, so the entry script has to sit + // on the real filesystem; see materializePyScripts. One copy PER RUN, + // synchronously, so no async boundary opens before `execute` registers the + // run and installs the abort listener. + bootstrapPath = materializePyScripts() + } catch (error: unknown) { + // A full or read-only temp filesystem, or a packaged asset the deployment + // failed to ship, is a SUBSTRATE failure — the same class as a child that + // cannot start. The seam permits rejection only for misuse, so this + // resolves as `worker-exit` rather than throwing out of `run()`. + return { logs: [], error: { kind: 'worker-exit', message: `failed to stage the python bootstrap: ${messageOf(error)}` } } + } + return await this.execute(request, bindings, bootstrapPath) + } + /* jscpd:ignore-end */ + + /** + * Reject (seam misuse) malformed binding namespaces: non-identifier or + * reserved globals/error classes, duplicates, and colliding or + * runtime-owned injected globals. + */ + private validateBindings(request: CodeRunRequest): Map { + const bindings = new Map() + // Every name the bootstrap injects into the program's one global namespace: + // namespace globals plus error-class names. They must be a collision-free + // set that avoids the runtime's own slots, or a later injection silently + // overwrites an earlier one (or the completion/builtins slot) and the run + // fails obscurely at execution time. + const injectedGlobals = new Set() + const claimGlobal = (name: string, role: string): void => { + if (RUNTIME_OWNED_GLOBALS.has(name)) { + throw new Error(`dsh-code-runtime-python: ${role} ${JSON.stringify(name)} collides with a runtime-owned global`) + } + if (injectedGlobals.has(name)) { + throw new Error(`dsh-code-runtime-python: ${role} ${JSON.stringify(name)} collides with another injected global`) + } + injectedGlobals.add(name) + } + for (const namespace of request.bindings) { + if (!IDENTIFIER.test(namespace.global) || RESERVED_NAMES.has(namespace.global)) { + throw new Error(`dsh-code-runtime-python: binding global ${JSON.stringify(namespace.global)} is not a usable Python identifier`) + } + if (bindings.has(namespace.global)) { + throw new Error(`dsh-code-runtime-python: duplicate binding global ${JSON.stringify(namespace.global)}`) + } + claimGlobal(namespace.global, 'binding global') + // The error class becomes a program global and its member property an + // attribute name, so both face the Python identifier rules; the member + // additionally must be assignable on a BaseException instance. + const errorClass = namespace.errorClass + if (errorClass) { + if (!IDENTIFIER.test(errorClass.name) || RESERVED_NAMES.has(errorClass.name)) { + throw new Error(`dsh-code-runtime-python: errorClass.name ${JSON.stringify(errorClass.name)} is not a usable Python identifier`) + } + // Any non-empty own attribute name is settable via setattr (the + // program reads exotic names like `tool-name` with getattr), matching + // the seam contract and the worker backend — only the seam-excluded + // and protocol-reserved members below are refused. + if (errorClass.memberNameProperty.length === 0) { + throw new Error('dsh-code-runtime-python: errorClass.memberNameProperty must be a non-empty attribute name') + } + if (EXCEPTION_RESERVED_MEMBERS.has(errorClass.memberNameProperty) || DUNDER.test(errorClass.memberNameProperty)) { + throw new Error(`dsh-code-runtime-python: errorClass.memberNameProperty ${JSON.stringify(errorClass.memberNameProperty)} is a reserved error member and cannot be assigned`) + } + claimGlobal(errorClass.name, 'errorClass.name') + } + bindings.set(namespace.global, { functions: namespace.functions, ...errorClass ? { errorClass } : {} }) + } + return bindings + } + + /** Spawn the child for one validated run and drive it to settlement. */ + private execute( + request: CodeRunRequest, + bindings: Map, + bootstrapPath: string, + ): Promise { + // This run's own staging directory, removed at settlement. + const bootstrapDir = dirname(bootstrapPath) + // Explicit pipe count of 4 puts the framed-JSON channel at fd 3 in the child. + // Resolve the interpreter against the current PATH first: the child's empty + // env would otherwise strip PATH and miss a basename python3 (see resolvePythonBin). + const child = spawn(resolvePythonBin(this.config.pythonBin), ['-I', bootstrapPath], { + env: {}, + detached: true, // Own process group — kill(-pid, sig) reaches subprocesses the model program spawns. + stdio: ['pipe', 'pipe', 'pipe', 'pipe'], + }) + + // Fd 3 is a duplex pipe carrying protocol frames. Node types extra stdio + // entries as `Stream | null`; the runtime shape with `'pipe'` is a duplex, + // so we narrow at the boundary rather than smearing casts below. Stdout + // and stderr are guaranteed non-null under `'pipe'` and typed as such. + const proto = child.stdio[3] as Duplex | null + /* v8 ignore next 3 -- `'pipe'` stdio always populates fd 3; guarding Node's `Stream | null` typing widening. */ + if (proto === null) { + throw new Error('dsh-code-runtime-python: python subprocess spawned without a fd-3 pipe') + } + + return new Promise((resolve) => { + let settled = false + const logs: string[] = [] + + // One host-side ledger covers normal frames, forged frames, and stray stdout bytes. + let logBudget = this.config.maxLogBytes + let logsTruncated = false + const admit = (text: string): void => { + /* v8 ignore next -- post-truncation admits no-op; needs child to keep streaming after ledger drops. */ + if (logsTruncated) return + // Each entry is charged its SERIALIZED cost — JSON.stringify's quotes + // and escapes plus one separator byte — because the seam bounds the + // serialized outer logs payload, and control characters expand + // several-fold under JSON escaping (a "\x00" flood would otherwise + // admit 6x its charge). The charge also puts a floor under an empty + // entry (its two quotes plus separator), so a `while True: print()` + // flood of zero-byte lines exhausts the ledger instead of growing the + // retained array without ever touching the budget. The one fixed + // truncation-marker entry is envelope, not payload, and rides + // uncharged. + // + // Cheap lower bound FIRST, before the escaped copy exists: every + // UTF-16 code unit costs at least one serialized byte (an ASCII + // character is one byte; a control character is six as `\uXXXX`; a + // non-ASCII BMP character is two or three; each half of a surrogate + // pair contributes two of the four bytes its code point encodes to), + // and the JSON form adds two quotes on top of the separator byte. So + // `text.length + 3` never exceeds the true cost, and a forged `log` + // frame carrying a control-heavy string anywhere below the 256 MiB + // frame ceiling truncates here instead of allocating a + // hundreds-of-megabytes escaped copy under a small maxLogBytes. + if (text.length + 3 > logBudget) { + logsTruncated = true + logs.push(logTruncationMarker(this.config.maxLogBytes)) + return + } + // Past the lower bound the escape expands the string at most sixfold, + // so this copy is bounded by ~6x the remaining budget. + const cost = Buffer.byteLength(JSON.stringify(text), 'utf8') + 1 + if (cost > logBudget) { + logsTruncated = true + logs.push(logTruncationMarker(this.config.maxLogBytes)) + return + } + logBudget -= cost + logs.push(text) + } + + // Stray-byte capture: anything the child writes to its stdout/stderr + // (native prints, C-extension writes) still counts against the ledger. + // One STREAMING decoder per pipe: a multibyte UTF-8 sequence can span + // two chunks (native writes, os.write past the pipe buffer), and + // decoding each chunk independently would corrupt both halves into + // replacement characters. StringDecoder holds the partial sequence + // until its continuation bytes arrive; the pipes are separate byte + // streams, so they cannot share one decoder. + const strayOut = new StringDecoder('utf8') + const strayErr = new StringDecoder('utf8') + const captureStray = (decoder: StringDecoder, chunk: Buffer): void => { + const text = decoder.write(chunk) + // Empty only when the chunk is nothing but a partial multibyte + // sequence — needs a pipe boundary INSIDE one character, which cannot + // be forced deterministically from the child side. + /* v8 ignore next */ + if (text.length > 0) admit(text) + } + child.stdout.on('data', (chunk: Buffer) => { captureStray(strayOut, chunk) }) + child.stderr.on('data', (chunk: Buffer) => { captureStray(strayErr, chunk) }) + // Flush each decoder when its pipe ends: output that STOPS mid-sequence + // (native code killed between bytes) leaves the partial character in + // the decoder, and end() renders it as U+FFFD rather than dropping the + // evidence. `end` fires before `close` settles the run, so the flush is + // admitted into `logs`. + const flushStray = (decoder: StringDecoder): void => { + const tail = decoder.end() + if (tail.length > 0) admit(tail) + } + child.stdout.on('end', () => { flushStray(strayOut) }) + child.stderr.on('end', () => { flushStray(strayErr) }) + + // Line-framed JSON reader over fd 3. The unframed buffer is bounded: a + // hostile program can loop `os.write(3, b"A"*4096)` with no newline to + // exhaust HOST memory, which the child's RLIMIT_AS does not cover. It is + // a memory-safety bound only: legitimate `call` frames may be large + // (binding traffic has no seam byte cap), so it never keys off + // maxValueBytes. + // Buffered as raw chunks with a running byte counter: appending is O(1) + // per chunk (a string `+=` accumulator would re-copy the whole prefix on + // every pipe chunk — quadratic on a large frame), joins happen only when + // a newline actually arrived, and the ceiling check reads the counter. + let pendingChunks: Buffer[] = [] + // Fragments already merged into finished blocks. Kept separate from + // `pendingChunks` so sealing never re-copies what earlier seals produced; + // the two together are the unframed buffer, and `pendingBytes` counts both. + let sealedBlocks: Buffer[] = [] + let pendingBytes = 0 + proto.on('data', (chunk: Buffer) => { + // Once settled, stop accumulating: a hostile child that keeps flooding + // fd 3 between finish() and close must not regrow the host buffer. + /* v8 ignore next -- post-settlement data needs the child to outrace close after we decided. */ + if (settled) return + pendingChunks.push(chunk) + pendingBytes += chunk.length + // Check the counter BEFORE the join, not the joined line afterwards: + // Buffer.concat allocates a second copy of everything held, so a line + // measured after the concat had already cost twice the ceiling — the + // ceiling this check exists to enforce. The counter is exact and free, + // and the retained chunks are released here so the rejected payload is + // not still held while the run settles. + // Reading the counter rather than the line length also charges the + // whole unframed buffer, which over-counts by at most the newline- + // bearing chunk's own length (one pipe read): the residual carried in + // is always a partial line, so nothing but the current line can be + // larger than that. + if (pendingBytes > FRAME_CEILING_BYTES) { + pendingChunks = [] + sealedBlocks = [] + pendingBytes = 0 + finish({ error: { kind: 'worker-exit', message: `protocol frame exceeded ${FRAME_CEILING_BYTES} bytes on fd 3` } }) + return + } + // Bound the FRAGMENT COUNT as well as the byte total, but only AFTER the + // ceiling check above: sealing first would `Buffer.concat` an already + // over-ceiling payload and allocate a second copy of it before the + // rejection ran, which is the peak-memory doubling that check exists to + // prevent. + // + // Fragment count needs its own bound because the ceiling meters payload + // bytes only, while each retained chunk is a separate Buffer with object + // and backing-store overhead no byte count sees: 5000 single-byte + // newline-free writes produced 5000 chunks holding 5031 bytes, so a + // program pacing such writes could accumulate millions of objects inside + // the wall budget and exhaust the host heap far below the ceiling. + // + // Sealing appends to a list of finished blocks instead of re-merging + // everything held. Concatenating the whole buffer at each threshold + // re-copied the entire accumulated prefix every time, so the cumulative + // copy volume was quadratic, not the amortized O(1) an earlier revision + // of this comment claimed: 10 MiB trickled a byte at a time copies + // 53.7 GB that way, and 64 MiB copies 2.2 TB. Here each byte is copied + // once into its block and never again, so the total stays linear, and the + // block list is itself bounded — every block holds at least + // `MAX_PENDING_CHUNKS - 1` bytes, so reaching the 256 MiB ceiling admits + // at most a few hundred thousand of them. + if (pendingChunks.length >= MAX_PENDING_CHUNKS) { + sealedBlocks.push(Buffer.concat(pendingChunks)) + pendingChunks = [] + } + if (chunk.includes(0x0a)) { + let buffered = Buffer.concat(sealedBlocks.length > 0 ? [...sealedBlocks, ...pendingChunks] : pendingChunks) + sealedBlocks = [] + let newline: number + while ((newline = buffered.indexOf(0x0a)) >= 0) { + const line = buffered.subarray(0, newline) + buffered = buffered.subarray(newline + 1) + /* v8 ignore next -- an empty line comes only from a forged `\n\n` write. */ + if (line.length === 0) continue + const text = line.toString('utf8') + // JSON.parse would silently ROUND an integer token outside the + // safe range before validation could see it, so a forged frame + // could smuggle a corrupted value into a dispatch or completion. + // An honest child never emits one (its validator rejects unsafe + // ints), so such a frame is hostile traffic: drop it like any + // other junk frame. + if (hasUnsafeIntegerToken(text)) continue + let parsed: unknown + try { + parsed = JSON.parse(text) as unknown + } catch { + continue // Junk frames drop silently (hostile-peer stance). + } + const message = validateChildFrame(parsed) + if (message) handleFrame(message) + } + // Carry the residual forward as a fresh, right-sized copy, NOT the + // `subarray` view: a view keeps the whole joined-frame allocation from + // the `Buffer.concat` above alive, so a large frame followed by a tiny + // trailing fragment would pin megabytes while `pendingBytes` reported + // only the fragment's length. See {@link detachResidual}. + pendingChunks = detachResidual(buffered) + pendingBytes = buffered.length + } + }) + + // Duplicate-call suppression against the honest child's id SEQUENCE, not + // a set of every id seen. `dispatch` sends consecutive ids from 0 with no + // gaps — it advances its counter only after the write succeeds, so a call + // rejected before reaching the wire consumes nothing — which makes the + // next legitimate id exactly `nextCallId`. + // + // Retaining a set instead let a program write an unbounded run of unique + // forged ids, each below the 256 MiB per-frame ceiling so nothing + // rejected them, and grow host memory for the whole run. Accepting any + // id above a high-water mark would have been just as wrong in the other + // direction: one forged `{"id": 9999}` would starve every honest call + // after it. The exact successor is the only test that both bounds the + // retained state to one number and cannot be poisoned by a forgery. + let nextCallId = 0 + + const handleFrame = (message: ChildToHost): void => { + /* v8 ignore next -- late frame after settlement; defensive against forged post-settlement traffic. */ + if (settled) return + switch (message.type) { + case 'boot-ack': + return // Presently informational. + case 'log': + if (message.truncated === true) { + // The CHILD ledger hit its cap. Its marker is the last log text + // there will be, so record it and stop host capture at the same + // point: admitting it as ordinary text left the host budget open, + // so later direct `os.write(1, ...)` bytes were retained AFTER the + // marker and a host-side exhaustion could append a second one. + // Both ledgers are keyed to the same `maxLogBytes`, so one marker + // describes the run. + if (!logsTruncated) { + logsTruncated = true + // The host's OWN marker, never the frame's text. `truncated` is + // attacker-reachable, so trusting the text let a program write + // `{"type":"log","truncated":true,"text":<1 MiB>}` and land all + // of it in `logs` under a 64-byte `maxLogBytes` — measured, the + // whole megabyte was retained, bypassing `admit` and its + // ceiling. Both ledgers key off the same `maxLogBytes`, so the + // marker the host generates says the same thing the child's + // would have. + logs.push(logTruncationMarker(this.config.maxLogBytes)) + } + return + } + admit(message.text) + return + case 'done': { + if (message.error) { + finish({ error: { kind: message.error.kind, message: capMessage(message.error.message, this.config.maxValueBytes) } }) + return + } + if (message.value === undefined) { + finish({}) + return + } + // Re-enforce the completion budget and number losslessness + // host-side: a forged done frame bypasses the Python-side + // _done_with_value check, and validateChildFrame no longer scans + // the value (an unbounded scan would push every member of a wide + // forgery before any cap ran). checkDoneValue folds both jobs into + // one bounded, iterative traversal — iterative because the seam's + // CodeJsonValue has no depth limit and an honest deep-but-small + // completion must cross intact rather than dying on stringify + // recursion; bounded because it stops at the cap without + // materializing the encoding, rejecting a forged value anywhere + // below the 256 MiB frame ceiling before it forces host-side copies. + // The seam forbids substituting a rendered/truncated value, so an + // oversized value fails the run as output-limit and a non-lossless + // number as invalid-output. The value is JSON-plain by construction + // (it came from JSON.parse of the frame), the traversal's precondition. + const check = checkDoneValue(message.value, this.config.maxValueBytes) + if (!check.ok) { + finish(check.reason === 'over-budget' + ? { error: { kind: 'output-limit', message: `completion value exceeded ${this.config.maxValueBytes} bytes` } } + : { error: { kind: 'invalid-output', message: 'completion value contained a non-lossless number' } }) + return + } + finish({ value: message.value as CodeJsonValue }) + return + } + case 'call': { + if (message.id !== nextCallId) return + nextCallId += 1 + const record = bindings.get(message.global)?.functions + const fn = record && Object.hasOwn(record, message.name) ? record[message.name] : undefined + if (typeof fn !== 'function') { + // `call.global` and `call.name` are attacker-controlled strings + // with no byte cap of their own — only the 256 MiB fd-3 frame + // ceiling — so each is sliced to `maxValueBytes` CODE UNITS + // BEFORE it reaches the template. Interpolating them whole would + // copy them into the message, `JSON.stringify` would copy the + // escaped form, `encodeJsonPlain` the frame, and the pipe write + // again: four full-size host allocations off one below-ceiling + // forgery, past every hostile-peer bound the log and done-error + // paths apply. Nothing past the first `maxValueBytes` code units + // of either field can survive the byte cap anyway, so the slices + // lose only text `capMessage` would drop, and that final cap + // gives this reply the same budget and marker as a forged done + // error. + const cap = this.config.maxValueBytes + const target = `${message.global.slice(0, cap)}.${message.name.slice(0, cap)}` + sendReply({ type: 'reply', id: message.id, ok: false, message: capMessage(`unknown binding ${JSON.stringify(target)}`, cap) }) + return + } + void (async () => { + try { + const resolved = await fn(message.args) + // The seam requires a lossy resolution to REJECT descriptively, + // not silently coerce: a raw JSON.stringify would turn NaN/ + // Infinity into null and drop undefined fields. Snapshot through + // the same lossless-JSON boundary the worker backend uses (also + // iterative, so a deeply nested value cannot overflow the stack). + const value = snapshotJsonValue(resolved) + if (value === undefined) { + sendReply({ type: 'reply', id: message.id, ok: false, message: 'binding resolution must be lossless JSON' }) + return + } + sendReply({ type: 'reply', id: message.id, ok: true, value }) + } catch (error: unknown) { + sendReply({ type: 'reply', id: message.id, ok: false, message: messageOf(error) }) + } + })() + return + } + } + } + + // Write one reply frame with the iterative encoder: a binding + // resolution has no seam-level depth or byte cap, so a deeply nested + // value must not die on JSON.stringify's recursion. The payload is + // JSON-plain by construction (snapshotJsonValue output, or literal + // strings/numbers), which is encodeJsonPlain's precondition. A closed + // pipe (child already gone) is swallowed since the close path settles + // the run. + const sendReply = (payload: ReplyMessage): void => { + /* v8 ignore next -- `settled` covers a race where the child exits between decision and write. */ + if (settled) return + try { + proto.write(`${encodeJsonPlain(payload)}\n`) + } catch { + // Pipe closed under us (child exited). The close path finishes the run. + } + } + + // Escalate SIGTERM → grace → SIGKILL on the entire process group. Idempotent + // via `killing`. + let killing = false + let graceTimer: NodeJS.Timeout | undefined + // A backstop for the one case `close` cannot cover: model code that starts + // a descendant with `os.setsid()`/`start_new_session=True` moves it into a + // fresh process group, so the SIGTERM/SIGKILL aimed at the child's group + // (`kill(-pid)`) never reaches it. If that orphan inherited stdout/stderr/ + // fd 3 and outlives the run, those pipes stay open and `close` never fires + // — leaving run() (and a teardown awaiting `finished`) hung indefinitely. + // finish() arms this deadline; when it fires we detach our stream handles + // and settle on the already-decided result regardless of the orphan. + let closeDeadline: NodeJS.Timeout | undefined + const killGroup = (sig: NodeJS.Signals): void => { + try { + /* v8 ignore next -- undefined pid means spawn never produced a process; finish() short-circuits before reaching kill(). */ + if (child.pid !== undefined) process.kill(-child.pid, sig) + } catch { + // ESRCH — the process already died. Nothing to do. + } + } + const kill = (): void => { + /* v8 ignore next -- kill() is idempotent; tests do not double-invoke it. */ + if (killing) return + killing = true + killGroup('SIGTERM') + graceTimer = setTimeout(() => { killGroup('SIGKILL') }, this.config.graceMs) + } + + let finishResolve!: () => void + const finished = new Promise((done) => { finishResolve = done }) + let resolved = false + // The decided terminal result for a live child, recorded by finish() and + // read by the `close` handler that settles it once the pipes have drained. + let decided: Omit + + // The single settlement point: resolve run() with the decided result and + // mark the fiber quiescent. Idempotent — the first call wins, so a later + // `close` after done/timeout/abort is absorbed as a no-op. + const settle = (result: Omit): void => { + if (resolved) return + resolved = true + if (graceTimer !== undefined) clearTimeout(graceTimer) + if (closeDeadline !== undefined) clearTimeout(closeDeadline) + // Drop from `live` only at settlement (close / pid-less spawn failure), + // NOT at finish(): between finish() and the child's `close` the child + // may sit in the SIGTERM grace window, and a concurrent teardown() + // snapshot of `this.live` must still see it so disposal awaits its exit + // ("no subprocess outlives the fiber"). teardown's own settle() on an + // already-finished run hits the resolved guard as a no-op. + this.live.delete(live) + // The child has exited by now (settle runs on `close`, or on a spawn + // that produced no pid), so its staging directory is no longer read and + // this run's copy goes away with it. Removed SYNCHRONOUSLY, before + // `resolve` below: a fire-and-forget removal left the directory on disk + // when `run()` resolved, so a caller could not observe the "gone by + // settlement" contract at all. Two files cost nothing to unlink here. + try { + rmSync(bootstrapDir, { recursive: true, force: true }) + } catch { + // Swallows only a failure to remove this run's staging directory — + // `force` already absorbs a missing one, so what remains is a + // filesystem-level refusal. The run's own outcome is already decided + // and must still be delivered, and teardown retries what stays + // tracked; the directory holds no secret, only a copy of two + // checked-in scripts. + } + finishResolve() + resolve({ ...result, logs }) + } + + const finish = (result: Omit): void => { + if (settled) return + settled = true + decided = result + clearTimeout(wallTimer) + request.signal?.removeEventListener('abort', onAbort) + // A spawn failure (ENOENT, EACCES) never produced a pid, so there is no + // process to kill: settle now. Its `close` still fires later and reaches + // the idempotent settle() again as a no-op. + if (child.pid === undefined) { + settle(result) + return + } + // Live child: SIGTERM→grace→SIGKILL, then let `close` (below) settle the + // run so any `done` frame buffered on fd 3 is handled first and the + // process is fully reaped before the fiber goes quiescent. + kill() + // `close` awaits every stdio stream draining, which a setsid-escaped + // orphan holding our inherited pipes can prevent forever. Bound that + // wait: after SIGKILL has had the grace window plus a margin to reap the + // child itself, force settlement on the decided result. Detaching the + // stream handles lets `close` land as a no-op if it ever arrives, and + // stops the orphan's stray output from being accounted against a run + // that already finished. `unref` so the deadline never keeps the host + // process alive on its own. + closeDeadline = setTimeout(() => { + proto.destroy() + child.stdout.destroy() + child.stderr.destroy() + settle(result) + }, this.config.graceMs + CLOSE_REAP_MARGIN_MS) + closeDeadline.unref() + } + + child.on('error', (error: Error) => { + finish({ error: { kind: 'worker-exit', message: `python spawn error: ${error.message}` } }) + }) + // `close` (not `exit`) is the settlement trigger: it fires only after the + // process exits AND every stdio stream — including the fd-3 protocol pipe — + // has drained, so a `done` frame the child wrote just before exiting is + // always handled before we settle. macOS can deliver `exit` before that + // final fd-3 data; keying off `close` makes the ordering irrelevant. + child.on('close', (code: number | null, signal: NodeJS.Signals | null) => { + // If done/timeout/abort already decided the result, finish() is a no-op + // and `decided` holds it — a SIGXCPU that arrives after a decision does + // not override it. Otherwise the child closed before completing: a + // SIGXCPU close is the kernel's own CPU meter firing — the RLIMIT_CPU + // soft limit, or the bootstrap's post-settlement getrusage check + // re-delivering SIGXCPU when a program trapped the soft limit and + // returned inside the soft-to-hard gap. That kernel-authoritative + // signal is the ONLY basis for the timeout classification: wall time + // is not evidence of CPU burn (a sleeping child SIGKILLed by a cgroup + // OOM killer, an operator, or itself consumed none), so every other + // signal or code — including an unsolicited SIGKILL, even the + // hard-limit one — reports as an opaque worker exit. + finish(signal === 'SIGXCPU' + ? { error: { kind: 'timeout', message: `CPU budget (${this.config.cpuSeconds}s) exhausted` } } + : { error: { kind: 'worker-exit', message: `python exited (code=${String(code)}, signal=${String(signal)}) before completing` } }) + settle(decided) + }) + + // Fd-3 and the stdout/stderr pipes emit `error` on early child death + // (ECONNRESET/EPIPE); swallow them so they do not become uncaught. The + // authoritative failure signal is `child.on('close')` above. + const silenceStreamError = (): void => {} + proto.on('error', silenceStreamError) + child.stdout.on('error', silenceStreamError) + child.stderr.on('error', silenceStreamError) + + /* jscpd:ignore-start -- wall-timer/abort/live-run wiring deliberately parallels code-runtime-worker; see the constructor note. */ + const wallTimer = setTimeout(() => { + finish({ error: { kind: 'timeout', message: `wall-clock ceiling reached (${this.config.maxWallMs}ms)` } }) + }, this.config.maxWallMs) + + const onAbort = (): void => { + finish({ error: { kind: 'abort', message: messageOf(request.signal?.reason) } }) + } + request.signal?.addEventListener('abort', onAbort, { once: true }) + + const live: LiveRun = { + kill, + finished, + settle: (failure: CodeRunFailure) => { finish({ error: failure }) }, + } + this.live.add(live) + /* jscpd:ignore-end */ + + // Send the boot frame once fd 3 is writable. This runs LAST in run()'s + // synchronous setup: its failure path calls finish(), which reads + // wallTimer/onAbort and (through settle) live, so those bindings must + // already be initialized — issuing the write earlier hit their + // temporal dead zone and threw a ReferenceError that rejected run() + // instead of resolving the worker-exit it constructs here. + const boot: BootMessage = { + type: 'boot', + cpuSeconds: this.config.cpuSeconds, + addressSpaceBytes: this.config.addressSpaceMb * 1024 * 1024, + maxLogBytes: this.config.maxLogBytes, + maxValueBytes: this.config.maxValueBytes, + namespaces: [...bindings].map(([global, namespace]) => ({ + global, + names: Object.keys(namespace.functions), + ...namespace.errorClass ? { errorClass: namespace.errorClass } : {}, + })), + } + try { + proto.write(`${JSON.stringify(boot)}\n`) + proto.write(`${JSON.stringify({ type: 'run', program: request.program })}\n`) + } catch (error: unknown) { + finish({ error: { kind: 'worker-exit', message: `failed to boot python subprocess: ${messageOf(error)}` } }) + return + } + }) + } +} + +export default PythonCodeRuntime diff --git a/packages/code-runtime/code-runtime-python/tests/boot-write-failure.spec.ts b/packages/code-runtime/code-runtime-python/tests/boot-write-failure.spec.ts new file mode 100644 index 0000000000..2a36fffa17 --- /dev/null +++ b/packages/code-runtime/code-runtime-python/tests/boot-write-failure.spec.ts @@ -0,0 +1,67 @@ +import { EventEmitter } from 'node:events' +import { PassThrough } from 'node:stream' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { Context } from 'cordis' + +/** + * A synchronous `proto.write` throw on the fd-3 pipe is the one boot path a real + * subprocess cannot be coerced into from a test: the pipe accepts queued bytes + * until the kernel buffer fills, and a same-tick EPIPE needs fd 3 already closed + * before the first write. `spawn` is mocked so fd 3 throws on the boot frame, + * which is exactly the branch that regressed. The mock is confined to this file + * so the real-subprocess suite in runtime.spec.ts is untouched. + */ +const { spawnMock } = vi.hoisted(() => ({ spawnMock: vi.fn() })) +vi.mock('node:child_process', async importOriginal => ({ + ...(await importOriginal()), + spawn: spawnMock, +})) + +const { PythonCodeRuntime } = await import('../src/index.ts') + +/** A `child_process.ChildProcess` stand-in whose fd-3 pipe rejects every write. */ +function fakeChildWithThrowingFd3(): EventEmitter { + const child = new EventEmitter() as EventEmitter & { + pid?: number + stdout: PassThrough + stderr: PassThrough + stdio: unknown[] + } + // Leave `pid` absent: `finish()` still runs its `clearTimeout(wallTimer)` / + // `removeEventListener(onAbort)` prologue (the TDZ site) before short- + // circuiting on `child.pid === undefined` to `settle` instead of waiting on a + // `close` this fake never emits, so the run resolves promptly. + child.stdout = new PassThrough() + child.stderr = new PassThrough() + // A duplex whose `write` throws synchronously, standing in for an fd-3 pipe + // that fails the moment the boot frame is issued. + const proto = new PassThrough() + proto.write = () => { throw Object.assign(new Error('EPIPE: broken pipe, write'), { code: 'EPIPE' }) } + child.stdio = [new PassThrough(), child.stdout, child.stderr, proto] + return child +} + +afterEach(() => { + spawnMock.mockReset() +}) + +describe('PythonCodeRuntime — boot-write failure', () => { + it('resolves a worker-exit when the fd-3 boot write throws (no TDZ ReferenceError)', async () => { + // Before the fix, the boot-write block ran BEFORE `wallTimer`, `onAbort`, + // and `live` were initialized, so its `finish()` (which clears `wallTimer`, + // removes `onAbort`, and — through `settle` — deletes `live`) hit the + // temporal dead zone and threw a ReferenceError. That escaped the Promise + // executor and REJECTED run() instead of resolving the worker-exit the catch + // constructs. This test would see that rejection; the fix makes it resolve. + spawnMock.mockImplementation(() => fakeChildWithThrowingFd3()) + const ctx = new Context() + const fiber = await ctx.plugin(PythonCodeRuntime) + const runtime = ctx.codeRuntime as InstanceType + + const result = await runtime.run({ program: 'return 1', bindings: [] }) + + expect(result.error?.kind).toBe('worker-exit') + expect(result.error?.message).toContain('failed to boot python subprocess') + await fiber.dispose() + }) +}) diff --git a/packages/code-runtime/code-runtime-python/tests/residual-detach.spec.ts b/packages/code-runtime/code-runtime-python/tests/residual-detach.spec.ts new file mode 100644 index 0000000000..87c040eb00 --- /dev/null +++ b/packages/code-runtime/code-runtime-python/tests/residual-detach.spec.ts @@ -0,0 +1,29 @@ +import { describe, expect, it } from 'vitest' +import { detachResidual } from '../src/index.ts' + +describe('detachResidual — fd-3 residual detachment', () => { + it('returns a copy that does NOT share the source frame allocation', () => { + // Simulate the data handler's state: one large joined frame from + // Buffer.concat, sliced past its newline to leave a small residual VIEW. + const joined = Buffer.alloc(1024 * 1024, 0x61) // 1 MiB backing allocation + joined[512] = 0x0a // a newline partway through + const residual = joined.subarray(513) // a view onto `joined`'s backing store + + // Before the fix the handler carried this view forward verbatim, pinning the + // whole 1 MiB `joined` allocation behind a residual that reports far fewer + // bytes. A right-sized copy must not point back into `joined`. + const [carried] = detachResidual(residual) + + expect(carried).toBeDefined() + expect(carried!.length).toBe(residual.length) + expect(carried!.equals(residual)).toBe(true) + // The copy's backing store is its own, sized to its content — not the 1 MiB + // frame. A subarray view would report the source's full byteLength here. + expect(carried!.buffer.byteLength).toBe(carried!.length) + expect(carried!.buffer).not.toBe(joined.buffer) + }) + + it('carries nothing forward for an empty residual', () => { + expect(detachResidual(Buffer.alloc(0))).toEqual([]) + }) +}) diff --git a/packages/code-runtime/code-runtime-python/tests/runtime.spec.ts b/packages/code-runtime/code-runtime-python/tests/runtime.spec.ts new file mode 100644 index 0000000000..20567ef5a1 --- /dev/null +++ b/packages/code-runtime/code-runtime-python/tests/runtime.spec.ts @@ -0,0 +1,3287 @@ +import { existsSync, readdirSync, realpathSync } from 'node:fs' +import { mkdtemp, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { basename, dirname, join } from 'node:path' +import { describe, expect, it, vi } from 'vitest' +import { Context } from 'cordis' +import { PythonCodeRuntime } from '../src/index.ts' +import { logTruncationMarker } from '../src/protocol.ts' +import type { Config } from '../src/index.ts' +import type { CodeBindingFunction, CodeJsonValue, CodeRunResult } from '@deepseek-ai/dsh-code-runtime' + +/** + * Names one `py/` script whose `copyFileSync` must fail, for the partial-staging + * case. A real disk-full or missing-asset failure mid-copy cannot be produced + * from a test, and the leak only shows when `mkdtempSync` has already succeeded. + */ +const { failNextCopyOf } = vi.hoisted(() => ({ failNextCopyOf: { value: undefined as string | undefined } })) +vi.mock('node:fs', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + copyFileSync(source: string, destination: string): void { + if (failNextCopyOf.value !== undefined && basename(source) === failNextCopyOf.value) { + failNextCopyOf.value = undefined + throw Object.assign(new Error('simulated ENOSPC on copy'), { code: 'ENOSPC' }) + } + actual.copyFileSync(source, destination) + }, + } +}) + +/** + * Integration suite over REAL python3 subprocesses (no mocks — subprocess is + * cheap and local, per docs/testing.md's real-over-mock policy). Each test + * builds a fresh runtime so budgets can be tuned per case. + */ +async function setup(config: Config = {}) { + const ctx = new Context() + const fiber = await ctx.plugin(PythonCodeRuntime, config) + const runtime = ctx.codeRuntime as PythonCodeRuntime + return { ctx, fiber, runtime } +} + +/** Convenience: one namespace `tools` with the given functions. */ +function tools(functions: Record) { + return [{ global: 'tools', functions }] +} + +describe('PythonCodeRuntime — seam descriptors and misuse', () => { + it('registers the seam descriptors', async () => { + const { runtime } = await setup() + expect(runtime.language).toBe('python') + expect(runtime.isolation).toBe('process') + }) + + it('rejects non-positive config as seam misuse', async () => { + const ctx = new Context() + await expect(ctx.plugin(PythonCodeRuntime, { cpuSeconds: 0 })) + .rejects.toThrow(/cpuSeconds must be a positive number/) + await expect(ctx.plugin(PythonCodeRuntime, { maxWallMs: -1 })) + .rejects.toThrow(/maxWallMs must be a positive number/) + }) + + it('rejects a non-integer cpuSeconds at load (setrlimit needs an int)', async () => { + const ctx = new Context() + await expect(ctx.plugin(PythonCodeRuntime, { cpuSeconds: 1.5 })) + .rejects.toThrow(/cpuSeconds must be a positive integer, got 1.5/) + }) + + it('rejects finite numeric config that cannot cross as an exact rlimit integer', async () => { + // `Number.isFinite` and `Number.isInteger` both admit values that cannot + // round-trip. `addressSpaceMb: 1e308` overflows to `Infinity` once multiplied + // by 1 MiB, and `encodeJsonPlain` renders that as `null`, so the child gets no + // limit at all; `cpuSeconds: 1e100` clears `Number.isInteger` while sitting + // far past the safe range, so `setrlimit` receives a different number than was + // configured. Both used to end every run in a bootstrap exception instead of + // failing at load, where a self-contained configuration error belongs. + const ctx = new Context() + await expect(ctx.plugin(PythonCodeRuntime, { addressSpaceMb: 1e308 })) + .rejects.toThrow(/addressSpaceMb must be at most \d+ .*exact integer/) + await expect(ctx.plugin(PythonCodeRuntime, { cpuSeconds: 1e100 })) + .rejects.toThrow(/cpuSeconds must be at most \d+ .*exact integers/) + // The boundary values still load: the bound rejects what cannot be encoded, + // not everything large. + const okMb = await ctx.plugin(PythonCodeRuntime, { addressSpaceMb: Math.floor(Number.MAX_SAFE_INTEGER / (1024 * 1024)) }) + await okMb.dispose() + const okCpu = await ctx.plugin(PythonCodeRuntime, { cpuSeconds: Number.MAX_SAFE_INTEGER - 1 }) + await okCpu.dispose() + }) + + it('rejects an output cap whose payload could not cross the frame ceiling', async () => { + // The caps budget a payload that must arrive inside ONE fd-3 frame, and the + // 256 MiB framing ceiling is fixed. A larger cap is unsatisfiable rather + // than generous: a completion the cap admits arrives as an over-ceiling + // frame and fails the run as `worker-exit`, inverting the `output-limit` + // the cap describes. The bound is `(ceiling - envelope) / 6`, since a + // control character escapes to six bytes. + const admissible = Math.floor((256 * 1024 * 1024 - 64) / 6) + const ctx = new Context() + await expect(ctx.plugin(PythonCodeRuntime, { maxLogBytes: admissible + 1 })) + .rejects.toThrow(/maxLogBytes must not exceed 44739232 .*fd-3 frame ceiling/) + await expect(ctx.plugin(PythonCodeRuntime, { maxValueBytes: admissible + 1 })) + .rejects.toThrow(/maxValueBytes must not exceed 44739232 .*fd-3 frame ceiling/) + // The boundary value itself loads: the bound is the largest cap a frame can + // still carry, not one below it. + const boundary = await ctx.plugin(PythonCodeRuntime, { maxValueBytes: admissible }) + await boundary.dispose() + }) + + it('rejects a pythonBin that spawn() would throw on, at load', async () => { + // Both values pass the string schema and both make `spawn` throw + // SYNCHRONOUSLY from inside run() — ERR_INVALID_ARG_VALUE for the empty + // path, ERR_INVALID_ARG_TYPE for the NUL — so run() would REJECT instead of + // resolving the worker-exit the seam promises for a child that cannot + // start. Both are self-contained configuration errors, so they fail here. + const ctx = new Context() + await expect(ctx.plugin(PythonCodeRuntime, { pythonBin: '' })) + .rejects.toThrow(/pythonBin must be a non-empty path without NUL bytes/) + await expect(ctx.plugin(PythonCodeRuntime, { pythonBin: 'py\u0000thon3' })) + .rejects.toThrow(/pythonBin must be a non-empty path without NUL bytes/) + }) + + it('rejects a timer budget setTimeout would silently clamp to 1 ms', async () => { + // Node stores a setTimeout delay as a signed 32-bit value and substitutes + // 1 ms for anything larger, inverting the knob's meaning: a huge maxWallMs + // would time every run out at once, and a huge graceMs would SIGKILL one + // millisecond after SIGTERM. Both must fail at load instead. + const ctx = new Context() + await expect(ctx.plugin(PythonCodeRuntime, { maxWallMs: 2_147_483_648 })) + .rejects.toThrow(/maxWallMs must not exceed 2147483647/) + // graceMs is bounded by the close deadline's added margin, not by the raw + // timer maximum, because that sum is what gets armed. + await expect(ctx.plugin(PythonCodeRuntime, { graceMs: 2_147_481_648 })) + .rejects.toThrow(/graceMs must not exceed 2147481647/) + // The exact maxima still load. + await expect(ctx.plugin(PythonCodeRuntime, { maxWallMs: 2_147_483_647, graceMs: 2_147_481_647 })) + .resolves.toBeDefined() + }) + + it('rejects loading this Unix-only backend on Windows', async () => { + // The bootstrap needs the POSIX `resource` module, a positional fd 3, and + // negative-PID process-group signals — none on Windows. The constructor + // must throw at load rather than register ctx.codeRuntime and defer the + // failure to the first run. + const original = process.platform + Object.defineProperty(process, 'platform', { value: 'win32', configurable: true }) + try { + const ctx = new Context() + await expect(ctx.plugin(PythonCodeRuntime, {})).rejects.toThrow(/requires a Unix platform/) + } finally { + Object.defineProperty(process, 'platform', { value: original, configurable: true }) + } + }) + + it('rejects a binding global that is not a Python identifier or is reserved', async () => { + const { runtime } = await setup() + await expect(runtime.run({ + program: 'return 1', + bindings: [{ global: '1bad', functions: {} }], + })).rejects.toThrow(/is not a usable Python identifier/) + await expect(runtime.run({ + program: 'return 1', + bindings: [{ global: 'class', functions: {} }], + })).rejects.toThrow(/is not a usable Python identifier/) + }) + + it('rejects duplicate binding namespaces', async () => { + const { runtime } = await setup() + await expect(runtime.run({ + program: 'return 1', + bindings: [ + { global: 'tools', functions: {} }, + { global: 'tools', functions: {} }, + ], + })).rejects.toThrow(/duplicate binding global/) + }) + + it('rejects run() after disposal, and unregisters ctx.codeRuntime', async () => { + const { ctx, fiber, runtime } = await setup() + await fiber.dispose() + await expect(runtime.run({ program: 'return 1', bindings: [] })) + .rejects.toThrow(/after disposal/) + expect(ctx.get('codeRuntime')).toBeUndefined() + }) + + it('short-circuits when the request signal is already aborted', async () => { + const { runtime } = await setup() + const signal = AbortSignal.abort('already-cancelled') + const result = await runtime.run({ program: 'return 1', bindings: [], signal }) + expect(result.error?.kind).toBe('abort') + expect(result.error?.message).toContain('already-cancelled') + expect(result.logs).toEqual([]) + }) + + it('short-circuits on an already-aborted signal whose reason cannot be converted', async () => { + // The pre-flight arm converted the reason with a bare `String()`, so a + // hostile reason threw out of `run()` — the seam promises to reject only for + // misuse, and a caller's cancellation token is not misuse. + const { runtime } = await setup() + const signal = AbortSignal.abort({ + [Symbol.toPrimitive]() { throw new Error('reason blew up') }, + }) + const result = await runtime.run({ program: 'return 1', bindings: [], signal }) + expect(result.error?.kind).toBe('abort') + expect(result.error?.message).toBe('') + expect(result.logs).toEqual([]) + }) + + it('runs the interpreter from materialized scripts outside the package, and removes them per run', async () => { + // The interpreter is an EXTERNAL process, so it can only open paths the OS + // resolves. Inside the single-file Python-SDK executable the packaged `py/` + // directory lives in pkg's virtual filesystem, which Node reads through its + // patched `fs` but `python3` cannot see, so spawning from that path fails + // with ENOENT. The scripts are therefore copied to a real directory first. + // + // The path is read from the child's own `__main__` module, so it proves + // where the interpreter actually loaded the entry script — asserting on a + // host-side constant would only restate the source. The program namespace + // seeds `__name__` but no `__file__`, hence the module lookup. + // `protocol.py` must land in the SAME directory, since `bootstrap.py` puts + // its own directory on `sys.path` to import it; the run completing at all + // already exercises that import. + const { runtime } = await setup() + const entryOf = async (): Promise => { + const result = await runtime.run({ program: 'import sys\nreturn sys.modules["__main__"].__file__', bindings: [] }) + expect(result.error).toBeUndefined() + return result.value as string + } + const entry = await entryOf() + expect(entry.endsWith('/bootstrap.py')).toBe(true) + const dir = dirname(entry) + expect(dir.startsWith(realpathSync(tmpdir()))).toBe(true) + expect(dir).not.toContain('/packages/') + // Staging is per RUN and removed at settlement, so by the time `run()` + // resolved the directory is already gone — nothing survives to be rewritten + // by a later run. `protocol.py` had to be beside the entry script for the run + // to complete at all, since `bootstrap.py` imports it off `sys.path`. + expect(existsSync(dir)).toBe(false) + // A second run stages its own copy rather than reusing the first. + expect(dirname(await entryOf())).not.toBe(dir) + }) + + it('contains a program that rewrites its own bootstrap to the run that did it', async () => { + // The child runs as the same UID as the host, so `0o700` does not stop model + // code from rewriting the scripts it was started from — + // `sys.modules['__main__'].__file__` names them. While all runs shared one + // staged copy, a program that overwrote `bootstrap.py` broke the NEXT run + // (measured: it settled as `worker-exit`), and substituted code would have + // run before the resource limits were applied. + const { runtime } = await setup({ maxWallMs: 10_000 }) + const sabotage = await runtime.run({ + program: [ + 'import sys', + 'path = sys.modules["__main__"].__file__', + 'open(path, "w").write("raise SystemExit(1)\\n")', + 'return path', + ].join('\n'), + bindings: [], + }) + expect(sabotage.error).toBeUndefined() + // The damage stayed inside the run that caused it. + const after = await runtime.run({ program: 'return 1 + 1', bindings: [] }) + expect(after.error).toBeUndefined() + expect(after.value).toBe(2) + }, 20_000) + + it('leaves no subprocess or scripts behind when disposal races the first run', async () => { + // Staging runs SYNCHRONOUSLY so no async boundary opens between `run()` and + // the point where `execute` registers the run in `live` and installs the + // abort listener. With an `await` there, a disposal landing in that window + // saw an empty `live`, returned, removed the script directory, and let the + // continuation spawn a subprocess after the fiber was gone. + // + // `dispose()` is called in the same synchronous turn as `run()`, with no + // `await` between them, so it lands exactly in that window. + // + // The leak assertion compares before and after rather than requiring an + // empty tmpdir: other tests in this file build runtimes they never dispose, + // so only the directories this test adds are its own evidence. + const staged = (): string[] => + readdirSync(realpathSync(tmpdir())).filter(name => name.startsWith('dsh-code-runtime-python-')) + const before = new Set(staged()) + const { fiber, runtime } = await setup({ maxWallMs: 8_000 }) + const pending = runtime.run({ program: 'import time\nwhile True: time.sleep(0.1)', bindings: [] }) + const disposed = fiber.dispose() + const result = await pending + await disposed + // Whatever the run reports, it must be terminal and must not be a success. + expect(result.value).toBeUndefined() + expect(['abort', 'worker-exit', 'timeout']).toContain(result.error?.kind) + // Disposal is to quiescence, so this run's directory is gone once it + // resolves, and nothing recreated it afterwards. + expect(staged().filter(name => !before.has(name))).toEqual([]) + }, 15_000) + + it('settles as abort when the signal fires in the same turn as the first run', async () => { + // Same window, the other listener. `addEventListener('abort')` does not + // replay an event that already fired, so an abort landing before the + // listener was installed used to be missed entirely and the program ran to + // success or the wall ceiling instead of resolving as `abort`. Synchronous + // staging keeps the pre-flight check and the listener in one turn, leaving + // no gap for the signal to slip through. + const { runtime } = await setup({ maxWallMs: 4_000, graceMs: 200 }) + const controller = new AbortController() + const pending = runtime.run({ + program: 'import time\nwhile True: time.sleep(0.1)', + bindings: [], + signal: controller.signal, + }) + controller.abort('same-turn-abort') + const result = await pending + expect(result.error?.kind).toBe('abort') + expect(result.error?.message).toContain('same-turn-abort') + }, 15_000) + + it('reports a staging failure as worker-exit instead of rejecting run()', async () => { + // Staging touches the filesystem, so it can fail for reasons that are not + // the caller's doing: a full or read-only temp filesystem, or a deployment + // that failed to ship the packaged scripts. Those are SUBSTRATE failures, + // the same class as a child that cannot start, and the seam reserves + // rejection for misuse — so `run()` must resolve, not throw. + // + // `TMPDIR` is the honest lever: `mkdtempSync` builds its path from + // `os.tmpdir()`, so pointing it at a path that is not a directory makes the + // real call fail without stubbing the module under test. + const previous = process.env.TMPDIR + const notADirectory = join(await mkdtemp(join(tmpdir(), 'dsh-staging-')), 'file') + await writeFile(notADirectory, '') + process.env.TMPDIR = notADirectory + try { + const { runtime } = await setup() + const result = await runtime.run({ program: 'return 1', bindings: [] }) + expect(result.error?.kind).toBe('worker-exit') + expect(result.error?.message).toContain('failed to stage the python bootstrap') + expect(result.logs).toEqual([]) + } finally { + if (previous === undefined) delete process.env.TMPDIR + else process.env.TMPDIR = previous + } + }) + + it('leaves no staging directory behind when a script copy fails', async () => { + // `mkdtempSync` succeeding and a later `copyFileSync` failing is its own + // case: the directory exists but is only partially populated. Recording it + // before the copies would leak it, because `run` retries staging on the next + // call and overwrites the single recorded path — teardown could then remove + // only the newest attempt. Staging must clean up its own partial directory. + // + // Only `copyFileSync` is stubbed, and only for the second script, so + // `mkdtempSync` really runs and the directory under assertion is real. + const staged = (): string[] => + readdirSync(realpathSync(tmpdir())).filter(name => name.startsWith('dsh-code-runtime-python-')) + const before = new Set(staged()) + failNextCopyOf.value = 'protocol.py' + try { + const { runtime } = await setup() + const result = await runtime.run({ program: 'return 1', bindings: [] }) + expect(result.error?.kind).toBe('worker-exit') + expect(result.error?.message).toContain('failed to stage the python bootstrap') + // The partial directory is gone, so nothing accumulates across retries. + expect(staged().filter(name => !before.has(name))).toEqual([]) + } finally { + failNextCopyOf.value = undefined + } + }, 15_000) +}) + +describe('PythonCodeRuntime — inherited resource limits', () => { + it('runs under an inherited hard limit tighter than addressSpaceMb', async () => { + // An unprivileged process may lower a hard rlimit but never raise it. Under + // a harness started with `ulimit -v` below `addressSpaceBytes`, requesting + // the configured cap made `setrlimit` raise `ValueError` and every run + // returned a bootstrap exception — even though the inherited limit is + // STRONGER than the one asked for. The bootstrap clamps to the inherited + // hard limit instead, so the run proceeds under the stricter bound. + // + // `pythonBin` is the honest lever: a wrapper that lowers RLIMIT_AS and then + // execs the real interpreter reproduces the inherited-limit condition + // without touching this test process's own limits. + const dir = await mkdtemp(join(tmpdir(), 'dsh-rlimit-')) + const wrapper = join(dir, 'python3-capped') + // 256 MiB, half the 512 MiB addressSpaceMb default, so the requested cap is + // unambiguously above the inherited ceiling. + await writeFile(wrapper, '#!/bin/sh\nulimit -v 262144\nexec python3 "$@"\n', { mode: 0o755 }) + const { runtime } = await setup({ pythonBin: wrapper }) + const result = await runtime.run({ + program: 'import resource\nreturn resource.getrlimit(resource.RLIMIT_AS)[1]', + bindings: [], + }) + expect(result.error).toBeUndefined() + // The applied hard limit is the inherited one, not the configured 512 MiB. + expect(result.value).toBe(256 * 1024 * 1024) + }, 15_000) + + it('applies the configured limits when nothing tighter is inherited', async () => { + // The clamp must not weaken the normal path: with an infinite inherited hard + // limit there is nothing to clamp against, and RLIM_INFINITY compares as -1, + // so treating it as a numeric bound would collapse every limit to -1. + const { runtime } = await setup({ cpuSeconds: 42, addressSpaceMb: 400 }) + const result = await runtime.run({ + // `getrlimit` returns a tuple, which the lossless-JSON completion check + // rejects; the pair is listed explicitly rather than converted. + program: 'import resource\ncpu = resource.getrlimit(resource.RLIMIT_CPU)\nreturn [cpu[0], cpu[1], resource.getrlimit(resource.RLIMIT_AS)[1]]', + bindings: [], + }) + expect(result.error).toBeUndefined() + // Soft at cpuSeconds, hard at +1 (the SIGKILL backstop), address space at + // the configured megabytes — exactly what the unclamped path applied. + expect(result.value).toEqual([42, 43, 400 * 1024 * 1024]) + }, 15_000) +}) + +describe('PythonCodeRuntime — programs and bindings', () => { + it('runs a top-level script, captures print output, and returns `result`', async () => { + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'x = 40', + 'y = 2', + 'print("hello", x + y)', + 'return {"answer": x + y}', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toEqual({ answer: 42 }) + // `print` in Python emits: text, ' ', text, '\n'. Concat the captured + // fragments and assert the model-visible message survives. + expect(result.logs.join('')).toContain('hello 42') + // 15s: this is usually the suite's first real subprocess — a cold python3 + // start (interpreter + asyncio import) on a loaded CI runner can exceed + // the 5s default alone; later tests reuse the warm page cache. + }, 15_000) + + it('bridges binding calls both ways and rejects the program-side call on a host rejection', async () => { + const { runtime } = await setup() + const calls: unknown[] = [] + const result = await runtime.run({ + program: [ + 'first = await tools.echo({"n": 1})', + 'caught = ""', + 'try:', + ' await tools.fail({})', + 'except RuntimeError as e:', + ' caught = str(e)', + 'return {"first": first, "caught": caught}', + ].join('\n'), + bindings: tools({ + echo: async (args) => { calls.push(args); return { echoed: args as CodeJsonValue } }, + fail: async () => { throw new Error('nope') }, + }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toEqual({ first: { echoed: { n: 1 } }, caught: 'nope' }) + expect(calls).toEqual([{ n: 1 }]) + }) + + it('still answers the call when the rejection value cannot be converted to a string', async () => { + // `messageOf` calls `String(error)`, which runs the value's own conversion, + // and this call site is a DETACHED async reply callback. A rejection whose + // `Symbol.toPrimitive` throws therefore escaped as an unhandled rejection: + // the reply frame was never written, the program stayed blocked on `await`, + // and the run degraded to a `maxWallMs` timeout (observed) — a host with no + // `unhandledRejection` listener would exit instead. The rejection must reach + // the program as an ordinary error carrying a fixed placeholder. + const { runtime } = await setup({ maxWallMs: 8_000 }) + const result = await runtime.run({ + program: [ + 'try:', + ' await tools.hostile({})', + 'except RuntimeError as e:', + ' return "rejected: " + str(e)', + 'return "no rejection"', + ].join('\n'), + bindings: tools({ + hostile: async () => { + throw { [Symbol.toPrimitive]() { throw new Error('toPrimitive blew up') } } + }, + }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('rejected: ') + }, 15_000) + + it('still answers the call when an Error carries a cyclic value in place of its message', async () => { + // `Error.message` is typed `string` but is a plain writable property, so a + // rejection can carry any value there. Returning it verbatim handed a + // non-string to `sendReply`, breaching `encodeJsonPlain`'s JSON-plain + // precondition: a cyclic object grew the encoder stack until the host threw + // RangeError from the detached reply callback, so no reply frame was written + // and the run degraded to a `maxWallMs` timeout (observed). The conversion + // must contain it — `String()` on a cycle throws inside the guard and lands + // on the placeholder, so the program sees an ordinary error. + const { runtime } = await setup({ maxWallMs: 8_000 }) + const result = await runtime.run({ + program: [ + 'try:', + ' await tools.hostile({})', + 'except RuntimeError as e:', + ' return "rejected: " + str(e)', + 'return "no rejection"', + ].join('\n'), + bindings: tools({ + hostile: async () => { + const cyclic: { self?: unknown; [Symbol.toPrimitive]: () => string } = { + // A cycle alone is inert for `String()`; the throwing conversion is + // what proves the guard runs rather than the encoder. + [Symbol.toPrimitive]: () => { throw new Error('cyclic message') }, + } + cyclic.self = cyclic + const error = new Error('placeholder') + // Writable per spec, so no cast is needed to install a non-string. + ;(error as unknown as { message: unknown }).message = cyclic + throw error + }, + }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('rejected: ') + }, 15_000) + + it('renders an Error whose message is a value with no JSON form', async () => { + // The non-cyclic arm. A number would not discriminate: `scalarJson` renders + // it as digits and the child `str()`s the field back, so it survives the + // wire either way. `undefined` is the value that separates the two orders — + // `scalarJson` emits a bare `undefined` token, so the reply line is not JSON + // at all, the child's parse drops the frame, and the program stays blocked + // on `await` until the wall ceiling (observed). Converting first sends the + // string "undefined", which the program receives as an ordinary rejection. + const { runtime } = await setup({ maxWallMs: 8_000 }) + const result = await runtime.run({ + program: [ + 'try:', + ' await tools.absent({})', + 'except RuntimeError as e:', + ' return "rejected: " + str(e)', + 'return "no rejection"', + ].join('\n'), + bindings: tools({ + absent: async () => { + const error = new Error('placeholder') + ;(error as unknown as { message: unknown }).message = undefined + throw error + }, + }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('rejected: undefined') + }, 15_000) + + it('runs a program with no await', async () => { + const { runtime } = await setup() + const result = await runtime.run({ + program: 'return 2 + 2', + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe(4) + }) + + it('returns JSON null whether the program returns None or falls off the end', async () => { + // Python has no `undefined`: an async body that returns None and one that + // never returns both yield None, so both complete as an exact JSON null. + // (The worker/TS backend can tell `return undefined` from `return null`; + // Python cannot, and reporting null for both is the honest rendering.) + const { runtime } = await setup() + const explicit = await runtime.run({ program: 'return None', bindings: [] }) + expect(explicit.error).toBeUndefined() + expect(explicit.value).toBeNull() + const noReturn = await runtime.run({ program: 'x = 1', bindings: [] }) + expect(noReturn.error).toBeUndefined() + expect(noReturn.value).toBeNull() + }) + + it('settles with no value on a forged valueless done frame', async () => { + // The child always sends a value now (return None → JSON null), so a done + // frame with no value key can only be forged; the host settles it as a + // value-less completion rather than crashing on the absent field. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'import os', + 'os.write(3, b\'{"type":"done"}\\n\')', + 'import time', + 'time.sleep(5)', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBeUndefined() + }) + + it('coalesces print arguments into one log line, not per-write fragments', async () => { + // print("a","b") calls write() per arg/sep/newline; the stream must emit + // one logical line "a b" so Code Mode's join(newline) does not insert + // spurious blank lines. Two prints → exactly two entries, no empties. + const { runtime } = await setup() + const result = await runtime.run({ + program: ['print("a", "b")', 'print("c")', 'return None'].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.logs).toEqual(['a b', 'c']) + }) + + it('flushes a print with no trailing newline', async () => { + const { runtime } = await setup() + const result = await runtime.run({ + program: ['print("partial", end="")', 'return None'].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.logs).toEqual(['partial']) + }) + + it('fails a completion dict with a non-string key as invalid-output (no key coercion)', async () => { + // json.dumps would coerce {1: "a", "1": "b"} to a single "1" key, silently + // dropping data. The shape validator rejects it before encoding. + const { runtime } = await setup() + const result = await runtime.run({ + program: 'return {1: "first", "1": "second"}', + bindings: [], + }) + expect(result.value).toBeUndefined() + expect(result.error?.kind).toBe('invalid-output') + expect(result.error?.message).toContain('non-string dict key') + }) + + it('rejects a binding argument with a non-string dict key before dispatch', async () => { + const { runtime } = await setup() + let called = false + const result = await runtime.run({ + program: [ + 'caught = ""', + 'try:', + ' await tools.sink({1: "x"})', + 'except RuntimeError as e:', + ' caught = str(e)', + 'return caught', + ].join('\n'), + bindings: tools({ sink: async () => { called = true; return null } }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toContain('lossless JSON') + expect(called).toBe(false) + }) + + it('fails a non-JSON completion value as invalid-output (no repr substitution)', async () => { + // A set is not lossless JSON. The old draft substituted repr(); the seam + // now requires refusing the run instead. + const { runtime } = await setup() + const result = await runtime.run({ + program: 'return {1, 2, 3}', + bindings: [], + }) + expect(result.value).toBeUndefined() + expect(result.error?.kind).toBe('invalid-output') + expect(result.error?.message).toContain('lossless JSON') + expect(result.error?.message).toContain('set') + }) + + it('fails a negative-zero completion value as invalid-output (sign bit is lossy over JSON)', async () => { + // JSON serialization turns -0.0 into 0 (or JS -0), silently changing the + // sign bit; the canonical lossless-JSON boundary rejects it, so the + // Python side must too — as a completion and as a binding argument. + const { runtime } = await setup() + const completion = await runtime.run({ + program: 'return -0.0', + bindings: [], + }) + expect(completion.error?.kind).toBe('invalid-output') + expect(completion.error?.message).toContain('negative zero') + const argument = await runtime.run({ + program: [ + 'try:', + ' await tools.echo(-0.0)', + ' return "accepted"', + 'except RuntimeError as e:', + ' return str(e)', + ].join('\n'), + bindings: tools({ echo: async args => args as never }), + }) + expect(argument.error).toBeUndefined() + expect(argument.value).toContain('negative zero') + }) + + it('fails a NaN completion value as invalid-output (allow_nan=False)', async () => { + // json.dumps would happily emit NaN by default, but NaN is not JSON; the + // bootstrap passes allow_nan=False so it fails as invalid-output. + const { runtime } = await setup() + const result = await runtime.run({ + program: 'return float("nan")', + bindings: [], + }) + expect(result.error?.kind).toBe('invalid-output') + }) + + it('fails an over-budget completion value as output-limit (child-side check)', async () => { + const { runtime } = await setup({ maxValueBytes: 64 }) + const result = await runtime.run({ + program: 'return "V" * 5000', + bindings: [], + }) + expect(result.value).toBeUndefined() + expect(result.error?.kind).toBe('output-limit') + expect(result.error?.message).toContain('exceeded 64 bytes') + }) + + it('rejects a wide completion as output-limit before materializing its traversal state', async () => { + // `[0] * 2000000` sits far above maxValueBytes but well below the frame + // ceiling. The folded checker must reject it via the pre-enqueue bound — + // BEFORE pushing two million elements onto the walk — so a small + // addressSpaceMb does not turn the check itself into an RLIMIT_AS death. + const { runtime } = await setup({ maxValueBytes: 64, addressSpaceMb: 256, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: 'return [0] * 2000000', + bindings: [], + }) + expect(result.value).toBeUndefined() + expect(result.error?.kind).toBe('output-limit') + expect(result.error?.message).toContain('exceeded 64 bytes') + }, 20_000) + + it('rejects a wide dict as output-limit without materializing its items list', async () => { + // Same pre-enqueue bound on the dict branch: `len(current)` replaces + // `list(current.items())`, which allocated one tuple per member before the + // bound could reject the value. Two million entries under a 64-byte cap + // fits the 256 MiB address space as a dict but not as a dict PLUS a + // two-million-tuple list. + const { runtime } = await setup({ maxValueBytes: 64, addressSpaceMb: 256, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: 'return {str(i): 0 for i in range(2000000)}', + bindings: [], + }) + expect(result.value).toBeUndefined() + expect(result.error?.kind).toBe('output-limit') + expect(result.error?.message).toContain('exceeded 64 bytes') + }, 20_000) + + it('meters a float completion in the host\'s number spelling', async () => { + // CPython's repr disagrees with the host's String(number): `1.0` is three + // bytes here and one there, `1e-07` pads the exponent the host writes as + // `1e-7`. Both sides meter the SAME budget, so the child must count the + // bytes the host will receive — otherwise a boundary-sized value is + // falsely reported as output-limit. + const { runtime } = await setup({ maxValueBytes: 1 }) + const integral = await runtime.run({ program: 'return 1.0', bindings: [] }) + expect(integral.error).toBeUndefined() + expect(integral.value).toBe(1) + + const exponent = await setup({ maxValueBytes: 4 }) + const small = await exponent.runtime.run({ program: 'return 1e-7', bindings: [] }) + expect(small.error).toBeUndefined() + expect(small.value).toBe(1e-7) + + // The spelling is a meter input, not a licence to overshoot: `1.5` is three + // bytes on both sides and still fails a two-byte budget. + const tight = await setup({ maxValueBytes: 2 }) + const over = await tight.runtime.run({ program: 'return 1.5', bindings: [] }) + expect(over.error?.kind).toBe('output-limit') + }) + + it('carries floats across the wire in the host\'s number spelling', async () => { + // The child ENCODES with the same speller it meters with, so the frame the + // host parses must reproduce every double exactly — including the branches + // where CPython and ECMAScript disagree (integral floats, sub-1e-6 + // exponents, >= 1e21, and beyond-safe-range integral doubles whose exact + // digits differ from the shortest round-trip form). + const { runtime } = await setup() + const result = await runtime.run({ + program: 'return [1.0, 100.0, 1.5, 0.1, 1e-7, 1e-6, 1e-5, 123.456, -2.5e-8, 1e21, float(2**60), 5e-324, 1.7976931348623157e308]', + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toEqual([1, 100, 1.5, 0.1, 1e-7, 1e-6, 1e-5, 123.456, -2.5e-8, 1e21, 2 ** 60, 5e-324, 1.7976931348623157e308]) + }) + + it('rejects a forged non-lossless done value host-side as invalid-output', async () => { + // A forged done frame bypasses the child's _check_done_value. JSON.parse + // turns 1e400 into Infinity; validateChildFrame no longer scans done.value, + // so the host's own checkDoneValue must catch the non-lossless number. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'import os', + String.raw`os.write(3, b'{"type":"done","value":1e400}' + b'\n')`, + 'import time', + 'time.sleep(5)', + ].join('\n'), + bindings: [], + }) + expect(result.value).toBeUndefined() + expect(result.error?.kind).toBe('invalid-output') + expect(result.error?.message).toContain('non-lossless number') + }) + + it('reports a syntax error as an exception without settling with a value', async () => { + const { runtime } = await setup() + const result = await runtime.run({ + program: '$$invalid python$$', + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('SyntaxError') + expect(result.value).toBeUndefined() + }) + + it('reports a runtime raise as an exception with the traceback', async () => { + const { runtime } = await setup() + const result = await runtime.run({ + program: 'raise ValueError("intentional")', + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('ValueError') + expect(result.error?.message).toContain('intentional') + }) + + it('bounds a deep exception cause chain instead of burning the wall budget formatting it', async () => { + // A chain thousands of links deep would make the rendering walk and + // format() linear in its length, consuming maxWallMs. Rendering is capped + // at 100 links with a marker; the run reports the exception well within + // budget rather than timing out. + const { runtime } = await setup({ maxValueBytes: 1024 * 1024, maxWallMs: 20_000 }) + const start = Date.now() + const result = await runtime.run({ + program: [ + 'err = None', + 'for i in range(3000):', + ' try:', + ' raise ValueError(i) from err', + ' except ValueError as e:', + ' err = e', + 'raise err', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('exception chain truncated at 100 links') + expect(Date.now() - start).toBeLessThan(15_000) + }, 25_000) + + it('bounds an over-cap chain without assigning to the live exception', async () => { + // The cap used to be applied by severing the over-cap link ON the live + // exception. An exception class overriding __setattr__ to raise turned that + // assignment into model code running inside the bootstrap's failure + // handler; the throw skipped the `done` send that sits after the handler, + // so the host blocked on fd 3 and reported a maxWallMs timeout instead of + // the model's own exception. Cutting the chain on the TracebackException + // COPY touches no model hook, so the marker still appears and the run + // reports `exception`. + const { runtime } = await setup({ maxValueBytes: 1024 * 1024, maxWallMs: 15_000 }) + const start = Date.now() + const result = await runtime.run({ + program: [ + 'class Sealed(Exception):', + ' def __setattr__(self, name, value):', + ' raise RuntimeError("live mutation refused")', + 'err = None', + 'for i in range(150):', + ' try:', + ' raise Sealed(i) from err', + ' except Sealed as e:', + ' err = e', + 'raise err', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('Sealed') + expect(result.error?.message).toContain('exception chain truncated at 100 links') + // The sever attempt is what used to leak: its message must not appear, and + // the run must settle well inside the wall budget rather than timing out. + expect(result.error?.message).not.toContain('live mutation refused') + expect(Date.now() - start).toBeLessThan(10_000) + }, 20_000) + + it('still sends done when rendering the diagnostic itself raises', async () => { + // format() reaches the exception's own __str__, so a model class whose + // __str__ raises can throw from inside the failure handler. CPython's + // _safe_string absorbs a raising __str__ during formatting, but the + // fallback must hold for any throw on that path (a raising __repr__ of an + // argument, a MemoryError under RLIMIT_AS), so the assertion is the + // invariant that matters: a `done` frame carrying `exception`, never a + // timeout, and never the failing renderer's own message. + const { runtime } = await setup({ maxWallMs: 10_000 }) + const result = await runtime.run({ + program: [ + 'class Unprintable(Exception):', + ' def __str__(self):', + ' raise RuntimeError("str refused")', + ' def __repr__(self):', + ' raise RuntimeError("repr refused")', + 'raise Unprintable()', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('Unprintable') + expect(result.error?.message).not.toContain('str refused') + expect(result.error?.message).not.toContain('repr refused') + }, 15_000) + + it('sends done with an inert diagnostic when the whole rendering path raises', async () => { + // Drive the fallback itself. `TracebackException.format` reads the + // exception class's `__module__` to decide whether to qualify the name, and + // a metaclass property can raise there — a throw INSIDE the formatter, + // reached with no rebinding of anything the bootstrap owns. Without the + // wrapper it escapes the handler, the `done` send never runs, and the host + // times out at maxWallMs. + const { runtime } = await setup({ maxWallMs: 10_000 }) + const result = await runtime.run({ + program: [ + 'class Meta(type):', + ' @property', + ' def __module__(cls):', + ' raise RuntimeError("renderer refused")', + 'class Hostile(ValueError, metaclass=Meta):', + ' pass', + 'raise Hostile("original failure")', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + // The inert fallback names the class and a fixed literal; it must not carry + // the renderer's message, and must not have become a timeout. `__name__` is + // still a plain str here, so the class name survives. + expect(result.error?.message).toBe('Hostile: ') + }, 15_000) + + it('falls back to a placeholder class name when __name__ itself raises', async () => { + // The fallback reads type(exc).__name__, which a metaclass property can + // hijack. It must neither run that override's failure into the handler nor + // format a non-str __name__ into the message. The hostile `__module__` is + // what drives execution into the fallback in the first place. + const { runtime } = await setup({ maxWallMs: 10_000 }) + const result = await runtime.run({ + program: [ + 'class Meta(type):', + ' @property', + ' def __module__(cls):', + ' raise RuntimeError("renderer refused")', + ' @property', + ' def __name__(cls):', + ' raise RuntimeError("name refused")', + 'class Nameless(Exception, metaclass=Meta):', + ' pass', + 'raise Nameless()', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toBe(': ') + }, 15_000) + + it('reports the real exception when the program rebinds every name the failure path uses', async () => { + // The bootstrap IS __main__, so `import __main__; __main__._X = ...` reaches + // any module global a call-time lookup would read. The failure path is the + // worst place for that: the reporter, the byte cap, the traceback formatter, + // the settlement flush and the `done` send all run AFTER the `except` block, + // so a replacement that raises skips the send, leaves the host blocked on + // fd 3, and the run reports a maxWallMs timeout instead of the model's own + // exception. Rebind all of them at once; the run must still carry the real + // ValueError. + const { runtime } = await setup({ maxWallMs: 10_000 }) + const result = await runtime.run({ + program: [ + 'import __main__', + 'def boom(*a, **k):', + ' raise RuntimeError("hijacked")', + '__main__._SAFE_MODEL_TRACEBACK = boom', + '__main__._cap_message = boom', + '__main__._model_traceback = boom', + '__main__._UNRENDERABLE_DIAGNOSTIC = boom', + '__main__._LogStream.flush_line = boom', + '__main__.ProtocolChannel.send_sync = boom', + 'raise ValueError("real failure")', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('ValueError: real failure') + expect(result.error?.message).not.toContain('hijacked') + }, 15_000) + + it('bounds an over-cap exception-group nesting on the copy', async () => { + // Exception groups link through `exceptions`, not the cause/context + // dunders, so the cap has to count that edge too — otherwise a deeply + // nested group walks past the bound the marker claims to enforce. + const { runtime } = await setup({ maxValueBytes: 1024 * 1024, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: [ + 'group = ValueError("leaf")', + 'for i in range(150):', + ' group = ExceptionGroup(f"g{i}", [group])', + 'raise group', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('exception chain truncated at 100 links') + }, 20_000) + + it('filters every bootstrap frame from the traceback of an uncaught binding rejection', async () => { + // A rejection re-raised by the bootstrap's dispatch adds bootstrap frames + // AFTER the model's own; only frames may reach model-visible, + // durable output — a bootstrap.py path would leak host absolutes and make + // transcripts machine-dependent. + const { runtime } = await setup() + const result = await runtime.run({ + program: 'await tools.boom({})', + bindings: tools({ boom: async () => { throw new Error('exploded') } }), + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('exploded') + expect(result.error?.message).toContain('') + expect(result.error?.message).not.toContain('bootstrap.py') + }) + + it('renders a non-Error thrown value from a host binding as its String form', async () => { + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'caught = ""', + 'try:', + ' await tools.failRaw({})', + 'except RuntimeError as e:', + ' caught = str(e)', + 'return caught', + ].join('\n'), + bindings: tools({ + failRaw: async () => { throw 'raw-nope' }, + }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toContain('raw-nope') + }) + + it('reassembles a frame split across writes behind a completed one', async () => { + // One os.write carrying "\n" leaves a non-empty + // residual after the newline loop; the tail must survive until its own + // newline arrives and then parse as a normal frame. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'import os, json', + 'head = json.dumps({"type":"log","text":"first"}).encode()', + 'tail = json.dumps({"type":"log","text":"second"}).encode()', + 'import time', + 'os.write(3, head + b"\\n" + tail[:5])', + 'time.sleep(0.2)', + 'os.write(3, tail[5:] + b"\\n")', + 'return "ok"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('ok') + expect(result.logs).toContain('first') + expect(result.logs).toContain('second') + }) + + it('raises the declared errorClass with the member name on rejection', async () => { + // Code Mode declares { name: ToolCallError, memberNameProperty: toolName }; + // a host rejection must surface as that class, carrying the failed tool. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'caught = ""', + 'try:', + ' await tools.fail({})', + 'except ToolCallError as e:', + ' caught = f"{type(e).__name__}:{e.toolName}:{e}"', + 'return caught', + ].join('\n'), + bindings: [{ + global: 'tools', + functions: { fail: async () => { throw new Error('typed-nope') } }, + errorClass: { name: 'ToolCallError', memberNameProperty: 'toolName' }, + }], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('ToolCallError:fail:typed-nope') + }) + + it('rejects an errorClass name colliding with its namespace global at the seam', async () => { + const { runtime } = await setup() + await expect(runtime.run({ + program: 'return 1', + bindings: [{ global: 'tools', functions: {}, errorClass: { name: 'tools', memberNameProperty: 'toolName' } }], + })).rejects.toThrow(/collides with another injected global/) + }) + + it('rejects a namespace global colliding with a runtime-owned name at the seam', async () => { + // `__dsh_main__` passes the identifier check, but exec()ing the generated + // wrapper would silently overwrite the binding after injection. `console` + // is the WORKER backend's slot — refused here too so a namespace list + // valid on one backend is valid on all. + const { runtime } = await setup() + // `__debug__` is refused for a different reason than a collision: CPython + // compiles a bare `__debug__` reference to the constant True and refuses to + // assign the name at compile time, so an injected global under it is + // unreachable from the program — accepted by the seam, unusable here. + for (const global of ['__dsh_main__', 'console', '__debug__']) { + await expect(runtime.run({ + program: 'x = 1', + bindings: [{ global, functions: {} }], + })).rejects.toThrow(/collides with a runtime-owned global/) + } + }) + + it('accepts a non-identifier memberNameProperty and rejects only an empty one', async () => { + // The seam permits any non-empty own property except the reserved + // members; Python setattr/getattr carry exotic names like `tool-name`, + // and the worker backend accepts them, so this backend must too. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'try:', + ' await tools.boom({})', + 'except ToolCallError as e:', + ' return getattr(e, "tool-name")', + ].join('\n'), + bindings: [{ + global: 'tools', + functions: { boom: async () => { throw new Error('nope') } }, + errorClass: { name: 'ToolCallError', memberNameProperty: 'tool-name' }, + }], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('boom') + await expect(runtime.run({ + program: 'return 1', + bindings: [{ global: 'tools', functions: {}, errorClass: { name: 'ToolCallError', memberNameProperty: '' } }], + })).rejects.toThrow(/memberNameProperty must be a non-empty attribute name/) + }) + + it('resolves a basename pythonBin to an absolute path (runs a real program)', async () => { + // A bare `python3` basename must resolve against PATH and actually launch + // under the empty-env spawn — exercises the accessSync success branch. + const { runtime } = await setup({ pythonBin: 'python3' }) + const result = await runtime.run({ program: 'return 7', bindings: [] }) + expect(result.error).toBeUndefined() + expect(result.value).toBe(7) + }) + + it('spawns via an absolute python path resolved from a basename against PATH', async () => { + // resolvePythonBin turns the default basename into an absolute path before + // the empty-env spawn; a basename with no PATH match falls through to the + // normal ENOENT worker-exit rather than throwing. + const { runtime } = await setup({ pythonBin: 'definitely-no-such-python-xyz' }) + const result = await runtime.run({ program: 'return 1', bindings: [] }) + expect(result.error?.kind).toBe('worker-exit') + }) + + it('rejects a memberNameProperty naming a constrained BaseException attribute', async () => { + // `__dict__`/`__class__` are constrained descriptors alongside + // `__traceback__` — setattr of a string raises TypeError while + // constructing the rejection — so every dunder is refused at the seam. + const { runtime } = await setup() + // name/message/stack are the seam's own exclusions (CodeBindingErrorClass + // forbids replacing them; the worker backend rejects them identically). + for (const member of ['__traceback__', '__dict__', '__class__', 'args', 'name', 'message', 'stack']) { + await expect(runtime.run({ + program: 'return 1', + bindings: [{ global: 'tools', functions: {}, errorClass: { name: 'ToolCallError', memberNameProperty: member } }], + })).rejects.toThrow(/reserved error member/) + } + }) + + it('rejects a lossy binding resolution (NaN) instead of coercing it to null', async () => { + // JSON.stringify would turn NaN into null and drop undefined fields; the + // seam requires a descriptive rejection so data cannot silently corrupt. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'caught = ""', + 'try:', + ' await tools.bad({})', + 'except RuntimeError as e:', + ' caught = str(e)', + 'return caught', + ].join('\n'), + bindings: tools({ bad: async () => Number.NaN }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toContain('lossless JSON') + }) + + it('contains a forged pathological done value without crashing the host', async () => { + // A ~20k-deep nested array forged onto fd 3 would overflow a recursive + // JSON.stringify; the host's iterative encoder measures it stack-safely + // and fails it deterministically on the byte budget (40 kB > 32 KiB). + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'import os, json', + 'depth = 20000', + 'payload = "[" * depth + "]" * depth', + 'os.write(3, b\'{"type":"done","value":\' + payload.encode() + b\'}\\n\')', + 'import time', + 'time.sleep(5)', + ].join('\n'), + bindings: [], + }) + expect(result.value).toBeUndefined() + expect(result.error?.kind).toBe('output-limit') + }) + + it('preserves a deeply nested completion value below the byte budget', async () => { + // CodeJsonValue has no depth limit: a 10000-deep nested list is only + // ~20 kB — under maxValueBytes — and must cross intact. That depth + // overflows BOTH recursive serializers the pipeline used to rely on + // (CPython's json.dumps recursion limit ~1000s, V8's JSON.stringify), so + // it proves the child-side _encode_json_plain and the host-side + // encodeJsonPlain together. The host JSON.parse of the frame is iterative + // in V8 for arrays, so only the two encoders were at risk. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'v = None', + 'for _ in range(10000):', + ' v = [v]', + 'return v', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + // Walk down iteratively (a recursive toEqual would itself overflow). + let depth = 0 + let cursor: unknown = result.value + while (Array.isArray(cursor)) { + expect(cursor).toHaveLength(1) + cursor = cursor[0] + depth++ + } + expect(depth).toBe(10000) + expect(cursor).toBeNull() + }) + + it('bridges a deeply nested binding resolution back into the program stack-safely', async () => { + // A binding resolution has no seam-level depth or byte cap; neither the + // host's reply serialization nor the CHILD's reply decode may die on + // recursion (json.loads raises RecursionError ~10k levels deep; the + // bootstrap decodes frames iteratively). 12000 levels sits past that + // limit while staying tiny in bytes. + const { runtime } = await setup() + const deep = ((): unknown => { + let v: unknown = null + for (let i = 0; i < 12000; i++) v = [v] + return v + })() + const result = await runtime.run({ + program: [ + 'v = await tools.deep({})', + 'depth = 0', + 'while isinstance(v, list):', + ' v = v[0]', + ' depth += 1', + 'return depth', + ].join('\n'), + bindings: tools({ deep: async () => deep as never }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe(12000) + }) + + it('rejects a reserved errorClass name at the seam', async () => { + const { runtime } = await setup() + await expect(runtime.run({ + program: 'return 1', + bindings: [{ + global: 'tools', + functions: {}, + errorClass: { name: 'class', memberNameProperty: 'toolName' }, + }], + })).rejects.toThrow(/errorClass.name "class" is not a usable Python identifier/) + }) + + it('routes a declared inherited-attribute name through the bridge via subscript', async () => { + // __class__ resolves on `object` before any fallback hook; the proxy's + // __getattribute__ intercepts declared names first, and subscript access + // is the SDK-advertised route for underscore names. + const { runtime } = await setup() + const seen: string[] = [] + const result = await runtime.run({ + program: [ + 'a = await tools["__class__"]({"via": "subscript"})', + 'b = await tools.__class__({"via": "dot"})', + 'return [a, b]', + ].join('\n'), + bindings: tools({ + '__class__': async () => { seen.push('called'); return 'bridged' }, + }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toEqual(['bridged', 'bridged']) + expect(seen).toEqual(['called', 'called']) + }) + + it('rejects NaN binding arguments immediately instead of hanging', async () => { + // Default json.dumps would emit a non-standard NaN token that the host + // JSON.parse drops silently, hanging the call until the wall clock; + // allow_nan=False raises in-program right away. + const { runtime } = await setup({ maxWallMs: 8000 }) + const start = Date.now() + const result = await runtime.run({ + program: [ + 'caught = ""', + 'try:', + ' await tools.echo({"x": float("nan")})', + 'except RuntimeError as e:', + ' caught = str(e)', + 'return caught', + ].join('\n'), + bindings: tools({ echo: async args => args as CodeJsonValue }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toContain('lossless JSON') + expect(Date.now() - start).toBeLessThan(5000) + }) + + it('carries large binding arguments well past maxValueBytes', async () => { + // Binding traffic has no seam byte cap: a call frame far larger than the + // completion budget must reach the host intact (the fd-3 ceiling is a + // fixed memory-safety bound, not an output budget). + const maxValueBytes = 4096 + const { runtime } = await setup({ maxValueBytes }) + let receivedLength = 0 + const result = await runtime.run({ + program: [ + `big = "B" * ${maxValueBytes * 50}`, + 'r = await tools.measure({"payload": big})', + 'return r', + ].join('\n'), + bindings: tools({ + measure: async (args) => { + receivedLength = ((args as { payload: string }).payload).length + return receivedLength + }, + }), + }) + expect(result.error).toBeUndefined() + expect(receivedLength).toBe(maxValueBytes * 50) + expect(result.value).toBe(maxValueBytes * 50) + }) + + it('rejects an unknown binding name inside the program with a matching error', async () => { + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'caught = ""', + 'try:', + ' await tools.nope({})', + 'except (AttributeError, RuntimeError) as e:', + ' caught = str(e)', + 'return caught', + ].join('\n'), + bindings: tools({ known: async () => 'ok' }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toContain('nope') + }) + + it('bounds an unknown-binding diagnostic built from a forged call frame', async () => { + // `call.global` and `call.name` carry no byte cap of their own, only the + // 256 MiB fd-3 frame ceiling, and the reply interpolated them raw: one copy + // into the template result, one into the `JSON.stringify` escape, one into + // the `encodeJsonPlain` frame, one into the pipe write. Slicing each field + // to `maxValueBytes` code units first makes an 8 MiB forged name a + // 128-byte reply. The observable effect is the reply the child then has to + // READ: its fd-3 reader is unbuffered, so `readline` consumes an oversized + // reply one `read(2)` per byte and the run's own legitimate call never gets + // answered — measured under a 60 s ceiling, the 8 MiB case timed out and a + // 64 MiB case cost the host 509.9 MiB of heap against 120.3 MiB with the + // slices in place. The child's address space stays generous enough to BUILD + // the forgery, which is not what is under test. + const { runtime } = await setup({ maxValueBytes: 128, addressSpaceMb: 1024, maxWallMs: 20_000 }) + const result = await runtime.run({ + program: [ + 'import os', + 'frame = b\'{"type":"call","id":9001,"global":"tools","name":"\' + b"n" * (8 * 1024 * 1024) + b\'","args":{}}\\n\'', + // One os.write returns short past the pipe buffer, and a partial frame + // would glue itself to the next one and be dropped as malformed, so the + // forgery goes out through a drain loop. + 'view = memoryview(frame)', + 'while view:', + ' view = view[os.write(3, view):]', + // A legitimate call after the forgery: its reply can only arrive once + // the child has read past whatever the forged frame was answered with. + 'await tools.known({})', + 'return "settled"', + ].join('\n'), + bindings: tools({ known: async () => 'ok' }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('settled') + }, 40_000) + + it('bridges a binding call reached via subscript access (tools["name"])', async () => { + // The SDK tells the model `await tools["my-tool"](args)` works for exotic + // names; the proxy's __getitem__ must route it through the bridge. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'r = await tools["my-tool"]({"n": 7})', + 'return r', + ].join('\n'), + bindings: tools({ 'my-tool': async args => ({ got: args as CodeJsonValue }) }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toEqual({ got: { n: 7 } }) + }) + + it('raises KeyError for an undeclared subscript name', async () => { + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'caught = ""', + 'try:', + ' await tools["absent"]({})', + 'except KeyError as e:', + ' caught = str(e)', + 'return caught', + ].join('\n'), + bindings: tools({ known: async () => 'ok' }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toContain('absent') + }) +}) + +describe('PythonCodeRuntime — budgets, termination, disposal', () => { + it('kills a wall-clock runaway program via SIGTERM/SIGKILL and reports timeout', async () => { + const { runtime } = await setup({ maxWallMs: 500, graceMs: 200 }) + const start = Date.now() + const result = await runtime.run({ + program: 'import time\nwhile True: time.sleep(1)', + bindings: [], + }) + const elapsed = Date.now() - start + // The wall timer may fire first or the exit-after-signal may resolve; both are ok. + expect(['timeout', 'worker-exit']).toContain(result.error?.kind) + // We got somewhere in the neighborhood of maxWallMs, not the underlying `sleep(1)`. + expect(elapsed).toBeLessThan(2000) + }, 5000) + + it('aborts a run when the outer signal fires mid-flight', async () => { + const { runtime } = await setup({ maxWallMs: 10_000 }) + const controller = new AbortController() + const settled: Promise = runtime.run({ + program: 'import time\nwhile True: time.sleep(0.1)', + bindings: [], + signal: controller.signal, + }) + setTimeout(() => { controller.abort('outer-abort') }, 200) + const result = await settled + expect(['abort', 'worker-exit']).toContain(result.error?.kind) + }, 5000) + + it('settles the run when a mid-flight abort reason cannot be converted', async () => { + // The listener converted the reason before calling `finish()`, so a hostile + // reason threw from inside an `AbortSignal` listener. Node reports that as an + // uncaught exception — it can terminate the host — and `finish()` never ran, + // so the run stayed live until the wall ceiling and misreported as `timeout` + // (observed) instead of the caller's cancellation. `maxWallMs` is short so + // that misreport is a fast assertion failure rather than a suite timeout. + const uncaught: unknown[] = [] + const record = (error: unknown): void => { uncaught.push(error) } + process.on('uncaughtException', record) + try { + const { runtime } = await setup({ maxWallMs: 4_000, graceMs: 200 }) + const controller = new AbortController() + const settled: Promise = runtime.run({ + program: 'import time\nwhile True: time.sleep(0.1)', + bindings: [], + signal: controller.signal, + }) + setTimeout(() => { + controller.abort({ [Symbol.toPrimitive]() { throw new Error('reason blew up') } }) + }, 200) + const result = await settled + expect(result.error?.kind).toBe('abort') + expect(result.error?.message).toBe('') + expect(uncaught).toEqual([]) + } finally { + process.off('uncaughtException', record) + } + }, 15_000) + + it('disposes to quiescence: an in-flight run resolves as abort and the child exits', async () => { + const { fiber, runtime } = await setup({ maxWallMs: 10_000 }) + const pending = runtime.run({ + program: 'import time\nwhile True: time.sleep(0.1)', + bindings: [], + }) + // Give the process time to spawn and start running. + await new Promise(resolve => setTimeout(resolve, 200)) + await fiber.dispose() + const result = await pending + expect(['abort', 'worker-exit']).toContain(result.error?.kind) + }, 5000) + + it('reports a spawn failure via a bogus python binary as worker-exit', async () => { + const { runtime } = await setup({ pythonBin: '/nonexistent/python-binary', maxWallMs: 3000 }) + const result = await runtime.run({ + program: 'return 1', + bindings: [], + }) + expect(result.error?.kind).toBe('worker-exit') + }, 8000) + + it('applies the strictest of the configured and inherited resource limits', async () => { + // This case used to drive the bootstrap's `applying resource limits failed` + // handler with `cpuSeconds: 2 ** 63`, asserting that a cap the child cannot + // apply fails the run rather than running it uncapped. That premise no longer + // holds, for two independent reasons, so the test now pins what is actually + // guaranteed instead of a path no admissible input reaches. + // + // First, `2 ** 63` is not a safe integer, so it is now rejected at LOAD as a + // configuration error — it can never reach the child at all. Second, even the + // largest admissible values are applied successfully, because `_clamped` + // bounds every requested pair by the inherited hard limit: an unprivileged + // process may lower a hard limit but never raise one, so the child keeps the + // stricter of the two rather than asking for something `setrlimit` refuses. + // The failure handler remains as a substrate guard (a platform whose kernel + // refuses the call for its own reasons), but it is no longer reachable from + // configuration, and a test that pretends otherwise documents a contract the + // code does not have. + // + // What is observable: a very large cap still yields a working run, and the + // containment it promises is met by the inherited ceiling. + const { runtime } = await setup({ cpuSeconds: Number.MAX_SAFE_INTEGER - 1, maxWallMs: 10_000 }) + const result = await runtime.run({ program: 'return 1', bindings: [] }) + expect(result.error).toBeUndefined() + expect(result.value).toBe(1) + }, 20_000) + + it('settles as worker-exit when the child exits before sending done (no hang)', async () => { + // Regression: settlement must key off `close` (process reaped AND stdio + // drained), not `exit`. With `exit`, finish() re-armed a second exit + // listener that never fired — run() hung forever whenever the exit event + // beat the final fd-3 data (deterministic on macOS, a lost race elsewhere). + const { runtime } = await setup({ maxWallMs: 10_000 }) + const result = await runtime.run({ + program: 'import os\nos._exit(7)', + bindings: [], + }) + expect(result.error?.kind).toBe('worker-exit') + expect(result.error?.message).toContain('code=7') + }, 5000) + + it('classifies RLIMIT_CPU soft-limit expiry (SIGXCPU) as a timeout', async () => { + // A CPU hot loop burns the soft limit; the kernel delivers SIGXCPU, whose + // close signal the host maps to `timeout`. macOS re-delivers SIGXCPU + // differently, so we assert only kind/message here — CI's darwin leg + // validates real delivery. cpuSeconds must be an integer for setrlimit. + const { runtime } = await setup({ cpuSeconds: 1, maxWallMs: 20_000 }) + const result = await runtime.run({ + program: 'while True: pass', + bindings: [], + }) + expect(result.error?.kind).toBe('timeout') + expect(result.error?.message).toContain('CPU budget') + }, 8000) + + it('keeps an early self-inflicted SIGKILL a worker-exit, not a CPU timeout', async () => { + // The unsolicited-SIGKILL-as-timeout classification applies only when the + // CPU budget could have expired (wall time >= cpuSeconds). A SIGKILL + // seconds before that (cgroup OOM, an operator, os.kill) is substrate + // death and stays worker-exit per the orthogonal taxonomy. + const { runtime } = await setup({ cpuSeconds: 60, maxWallMs: 10_000 }) + const result = await runtime.run({ + program: [ + 'import os, signal', + 'os.kill(os.getpid(), signal.SIGKILL)', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('worker-exit') + expect(result.error?.message).toContain('SIGKILL') + }) + + it('charges a forked descendant against the run CPU budget', async () => { + // RLIMIT_CPU is per-process and every child inherits a FRESH budget, so a + // program that shells out multiplies `cpuSeconds` by the number of + // descendants it starts. Measured before the aggregate meter existed: with + // cpuSeconds 1, two sequential busy children burned 2.0 CPU-seconds + // (RUSAGE_CHILDREN) and the run still returned a SUCCESS completion. The + // settle-time check meters RUSAGE_SELF + RUSAGE_CHILDREN and converts the + // overrun into the same SIGXCPU the untrapped soft limit sends. + const { runtime } = await setup({ cpuSeconds: 1, maxWallMs: 30_000 }) + const result = await runtime.run({ + program: [ + 'import subprocess, sys', + 'for _ in range(2):', + ' subprocess.run([sys.executable, "-c", "import time\\nt=time.time()\\nwhile time.time()-t<1.2: pass"])', + 'return "escaped the cpu budget"', + ].join('\n'), + bindings: [], + }) + // Darwin's SIGXCPU re-delivery differs, so accept either terminal + // classification; what must NOT happen is the completion crossing. + expect(['timeout', 'worker-exit']).toContain(result.error?.kind) + expect(result.value).toBeUndefined() + }, 40_000) + + it('does not charge wall time or a cheap descendant against the CPU budget', async () => { + // The meter is CPU, not wall clock, and it must not fire on a child that + // burns almost nothing: a sleeping program and a trivial subprocess both + // have to complete normally, or the check would reject every program that + // shells out. + const { runtime } = await setup({ cpuSeconds: 1, maxWallMs: 30_000 }) + const slept = await runtime.run({ + program: 'import time\ntime.sleep(1.5)\nreturn "slept"', + bindings: [], + }) + expect(slept.error).toBeUndefined() + expect(slept.value).toBe('slept') + const cheap = await runtime.run({ + program: [ + 'import subprocess, sys', + 'subprocess.run([sys.executable, "-c", "pass"])', + 'return "cheap child"', + ].join('\n'), + bindings: [], + }) + expect(cheap.error).toBeUndefined() + expect(cheap.value).toBe('cheap child') + }, 40_000) + + it('spends no part of addressSpaceMb on bootstrap machinery', async () => { + // RLIMIT_AS counts RESERVED address space, so anything the bootstrap maps + // for its own accounting is subtracted from the program's `addressSpaceMb`. + // A sampling thread for the descendant-CPU meter cost 72 MiB here (an 8 MiB + // stack plus a 64 MiB glibc per-thread malloc arena reservation) and turned + // the 2-million-entry dict rejection below into a MemoryError under a + // 256 MiB cap on a slower runner. Assert the child's own mappings directly + // rather than inferring the budget from a near-cap allocation, so the bound + // is read from /proc instead of from how much headroom one machine happens + // to have; 48 MiB is well above the ~30 MiB a bare interpreter maps and + // well below the 102 MiB the thread produced. `addressSpaceMb` itself is + // skipped on darwin (the dyld shared cache makes any practical cap + // unsettable) and /proc/self/maps does not exist there, so the mapping + // assertion is Linux-only; the completion path is checked everywhere. + const { runtime } = await setup({ maxValueBytes: 4096, addressSpaceMb: 256 }) + const mapped = await runtime.run({ + program: [ + 'import sys', + 'if sys.platform != "linux":', + ' return 0', + 'total = 0', + 'with open("/proc/self/maps") as handle:', + ' for line in handle:', + ' low, high = (int(part, 16) for part in line.split(" ", 1)[0].split("-"))', + ' total += high - low', + 'return total // (1024 * 1024)', + ].join('\n'), + bindings: [], + }) + expect(mapped.error).toBeUndefined() + expect(mapped.value).toBeLessThan(48) + }, 20_000) + + it('spends no part of addressSpaceMb on the reply pump, across a binding await', async () => { + // The test above measures BEFORE the program yields, so it could not see the + // reply pump's cost: `loop.run_in_executor(None, read_frame)` created the + // default executor's first thread on the first `await tools.*`, and that + // thread's 8 MiB stack plus a 64 MiB glibc per-thread malloc arena are + // charged to RLIMIT_AS while the limit is already in force — measured, the + // child went from 30.34 MiB to 102.39 MiB across one binding call. Under a + // small `addressSpaceMb` the thread cannot start and a legitimate call hangs + // to `maxWallMs`; under a larger one an allocation that should have fit dies + // as MemoryError. `loop.add_reader` watches the fd with no thread at all. + // + // Measuring both sides inside one run is what discriminates: a single + // after-the-fact number cannot separate the pump's cost from the + // interpreter's own footprint. Linux-only for the same reason as above. + const { runtime } = await setup({ addressSpaceMb: 256, maxWallMs: 20_000 }) + const result = await runtime.run({ + program: [ + 'import sys', + 'def mapped():', + ' if sys.platform != "linux":', + ' return 0', + ' total = 0', + ' with open("/proc/self/maps") as handle:', + ' for line in handle:', + ' low, high = (int(part, 16) for part in line.split(" ", 1)[0].split("-"))', + ' total += high - low', + ' return total // (1024 * 1024)', + 'before = mapped()', + 'echoed = await tools.echo({"ping": True})', + 'return {"before": before, "after": mapped(), "echoed": echoed}', + ].join('\n'), + bindings: tools({ echo: async args => args as CodeJsonValue }), + }) + expect(result.error).toBeUndefined() + const value = result.value as { before: number; after: number; echoed: unknown } + // The binding call really happened, so the pump really ran. + expect(value.echoed).toEqual({ ping: true }) + // Awaiting a binding maps nothing extra. The 8 MiB allowance absorbs ordinary + // heap growth while staying far below the 72 MiB a pump thread cost. + expect(value.after - value.before).toBeLessThan(8) + }, 30_000) + + it('still terminates a program that ignores SIGXCPU (hard-limit backstop)', async () => { + // A hot loop under SIG_IGN burns through the soft limit; the kernel's + // hard limit (cpuSeconds + 1) SIGKILLs it. Only a kernel-authoritative + // SIGXCPU close classifies as the CPU timeout — a bare SIGKILL is + // indistinguishable from a cgroup OOM kill, so it reports worker-exit + // (Darwin re-delivers SIGXCPU instead, where the wall clock settles it + // as timeout). Either way the run TERMINATES within the budget — the + // backstop holds even when the classification is the opaque one. + const { runtime } = await setup({ cpuSeconds: 1, maxWallMs: 6_000 }) + const result = await runtime.run({ + program: [ + 'import signal', + 'signal.signal(signal.SIGXCPU, signal.SIG_IGN)', + 'while True: pass', + ].join('\n'), + bindings: [], + }) + expect(['timeout', 'worker-exit']).toContain(result.error?.kind) + }, 12_000) + + it('enforces the CPU budget even when the program monkeypatches the enforcement primitives', async () => { + // The check uses import-time-captured references, so replacing + // resource.getrusage / signal.signal / os.kill on the modules cannot + // defang it: a trapping program that also swaps the callables and burns + // past the budget still dies by the authoritative SIGXCPU. + const { runtime } = await setup({ cpuSeconds: 1, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: [ + 'import signal, os, resource, time', + 'signal.signal(signal.SIGXCPU, lambda *a: None)', + 'resource.getrusage = lambda *a: (_ for _ in ()).throw(RuntimeError("nope"))', + 'os.kill = lambda *a: None', + 'signal.signal = lambda *a: None', + 'deadline = time.process_time() + 1.05', + 'while time.process_time() < deadline: pass', + 'return "escaped"', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('timeout') + expect(result.value).toBeUndefined() + }, 15_000) + + it('re-delivers SIGXCPU when a trapping program returns inside the soft-to-hard gap', async () => { + // A program can trap SIGXCPU and settle during the one-second gap; the + // bootstrap re-checks the kernel CPU meter (getrusage) after settlement + // and dies by SIGXCPU with the default disposition restored, so the host + // still classifies the exhausted budget as a timeout instead of success. + const { runtime } = await setup({ cpuSeconds: 1, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: [ + 'import signal, time', + 'fired = []', + 'signal.signal(signal.SIGXCPU, lambda *a: fired.append(1))', + 'deadline = time.process_time() + 1.05', + 'while time.process_time() < deadline: pass', + 'return "escaped"', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('timeout') + expect(result.error?.message).toContain('CPU budget') + expect(result.value).toBeUndefined() + }, 15_000) + + it('enforces the CPU budget when the program rebinds the enforcer on __main__', async () => { + // The bootstrap IS `__main__`, so `import __main__` reaches its globals. + // The enforcement callable holds its primitives in closure cells (not + // module attributes) and `_run` reads the callable into a frame local + // before the program starts, so neither replacing the global nor swapping + // the module's captured names changes what runs after settlement. + const { runtime } = await setup({ cpuSeconds: 1, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: [ + 'import signal, time, __main__', + 'signal.signal(signal.SIGXCPU, lambda *a: None)', + '__main__._DIE_IF_CPU_EXHAUSTED = lambda *_: None', + 'deadline = time.process_time() + 1.05', + 'while time.process_time() < deadline: pass', + 'return "escaped"', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('timeout') + expect(result.value).toBeUndefined() + }, 15_000) + + it('bounds a program that defeats the post-check by writing its closure cell', async () => { + // The closure-cell capture raises the cost of defeating the post-check; it + // does NOT make it unreachable, and nothing in-process could: a cell is + // writable through `fn.__closure__[i].cell_contents`, and `sys._getframe` + // reads _run's frame locals. This program does exactly that — walks to + // _run's frame, takes the enforcement callable, and replaces its captured + // `getrusage` with one reporting zero CPU used — then burns past cpuSeconds + // with SIGXCPU trapped. The run must still fail, because the bound that + // model code cannot forge is outside the interpreter: the RLIMIT_CPU HARD + // limit at cpuSeconds + 1, whose SIGKILL admits no handler. No success is + // reportable either way. + const { runtime } = await setup({ cpuSeconds: 1, maxWallMs: 20_000 }) + const start = Date.now() + const result = await runtime.run({ + program: [ + 'import signal, sys, time', + 'signal.signal(signal.SIGXCPU, lambda *a: None)', + // Walk out of __dsh_main__ to _run's frame and take its local. + 'die = None', + 'depth = 1', + 'while depth < 12:', + ' frame = sys._getframe(depth)', + ' if "die_if_cpu_exhausted" in frame.f_locals:', + ' die = frame.f_locals["die_if_cpu_exhausted"]', + ' break', + ' depth += 1', + 'assert die is not None, "enforcer not reachable from the frame chain"', + 'class Zero:', + ' ru_utime = 0.0', + ' ru_stime = 0.0', + 'names = die.__code__.co_freevars', + 'die.__closure__[names.index("getrusage")].cell_contents = lambda *a: Zero()', + // Burn well past the soft limit into the hard limit's SIGKILL. + 'while True: pass', + ].join('\n'), + bindings: [], + }) + // What holds on EVERY platform: the tampering bought no success. The run + // failed, carried no value, and the reported kind is one of the two + // kernel-level outcomes — never a completion. + expect(result.value).toBeUndefined() + expect(result.error?.kind === 'worker-exit' || result.error?.kind === 'timeout').toBe(true) + if (process.platform === 'linux') { + // Linux enforces the RLIMIT_CPU HARD limit at cpuSeconds + 1 promptly, so + // the CPU bound — not the 20 s wall ceiling — is what stops the program. + // Its SIGKILL is not SIGXCPU, so the orthogonal-failure taxonomy reports + // `worker-exit`: a bare SIGKILL is not evidence of CPU burn. + expect(result.error?.kind).toBe('worker-exit') + expect(Date.now() - start).toBeLessThan(15_000) + } else { + // Darwin does not deliver the hard limit's SIGKILL on the same schedule; + // observed on the macOS lane, a program that patches the post-check runs + // to the WALL ceiling instead. The CPU budget is therefore not the + // binding constraint against a tampering program there — the wall clock + // is. Asserted rather than skipped so the difference stays visible. + expect(result.error?.kind).toBe('timeout') + } + }, 30_000) + + it('keeps a finished-but-not-closed run live so dispose awaits the child\'s death', async () => { + // finish() no longer drops the run from `live`; settle() (at close) does. + // A SIGTERM-trapping program with a small graceMs sits in the grace window + // after finish() fires — dispose() must not resolve until the SIGKILL + // backstop actually reaps the child. The program prints its pid (captured + // as a log even on abort); once dispose() resolves, that pid must be dead + // (process.kill(pid, 0) throws ESRCH). + const { fiber, runtime } = await setup({ maxWallMs: 10_000, graceMs: 400 }) + // Deterministic readiness: the program reports its pid through a binding + // AFTER installing the trap, so dispose cannot race the spawn (a fixed + // sleep lost that race on slow CI runners — SIGTERM landed pre-trap). + let reportedPid!: (pid: number) => void + const trapReady = new Promise((resolve) => { reportedPid = resolve }) + const pending = runtime.run({ + program: [ + 'import signal, time, os', + 'signal.signal(signal.SIGTERM, lambda *a: None)', + 'await tools.ready({"pid": os.getpid()})', + 'while True: time.sleep(0.05)', + ].join('\n'), + bindings: tools({ + ready: async (args) => { + reportedPid((args as { pid: number }).pid) + return 'ok' + }, + }), + }) + const pid = await trapReady + const start = Date.now() + await fiber.dispose() + const elapsed = Date.now() - start + const result = await pending + expect(['abort', 'worker-exit', 'timeout']).toContain(result.error?.kind) + // dispose() returned only after the grace window elapsed (the SIGTERM trap + // forces the SIGKILL backstop path), proving the run stayed live past finish(). + expect(elapsed).toBeGreaterThanOrEqual(300) + expect(Number.isInteger(pid) && pid > 0).toBe(true) + // The child is fully reaped by the time dispose() resolved. + expect(() => process.kill(pid, 0)).toThrow(/ESRCH/) + }, 8000) + + it('settles on the decided result even when a setsid-escaped orphan holds stdio open past close', async () => { + // `close` only fires once every inherited stdio stream drains. A descendant + // started with start_new_session=True escapes the child's process group, so + // the SIGTERM/SIGKILL aimed at that group never reaches it; if it inherited + // our stdout/stderr/fd 3 and outlives the run, `close` would never fire and + // run() would hang forever. The close-deadline backstop (graceMs + margin) + // must force settlement on the value the `done` frame already decided. + const { runtime } = await setup({ graceMs: 100 }) + const start = Date.now() + const result = await runtime.run({ + program: [ + 'import subprocess, sys', + // Orphan in a fresh session, inheriting our stdout/stderr/fd 3, alive + // well past the close-deadline so `close` cannot fire on its own. + 'subprocess.Popen([sys.executable, "-c", "import time; time.sleep(10)"],', + ' start_new_session=True)', + 'return "escaped"', + ].join('\n'), + bindings: [], + }) + const elapsed = Date.now() - start + // The done frame decided the value; the deadline settled it despite the + // orphan pinning the pipes open. + expect(result.error).toBeUndefined() + expect(result.value).toBe('escaped') + // Settlement waited for the backstop (graceMs + CLOSE_REAP_MARGIN_MS ≈ 2.1s), + // not the wall-clock ceiling — proving the deadline, not the ceiling, fired. + expect(elapsed).toBeGreaterThanOrEqual(1_500) + expect(elapsed).toBeLessThan(5_000) + }, 8000) +}) + +describe('PythonCodeRuntime — hostile peer', () => { + it('drops garbage bytes and unknown-shape frames posted directly to fd 3', async () => { + // The model program can reach fd 3 and write anything. We inject a + // non-JSON line, a valid JSON but unknown-shape frame, and a broken done + // frame; the host must not crash, and the real `done` still settles the run. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'import os', + 'os.write(3, b"not-json\\n")', + 'os.write(3, b\'{"type":"unknown"}\\n\')', + 'os.write(3, b\'{"type":"done","error":{"message":42}}\\n\')', + 'return "survived"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('survived') + }) + + it('answers a forged call frame for an unknown binding and never crashes', async () => { + // The unknown-binding reply path, driven through the id the host expects: + // the program lets its own first call claim id 0 and forges id 1, which the + // host answers with the `unknown binding` rejection the honest call would + // have received. A forged id out of sequence is dropped instead — that is + // the id-bound test below, not this one. + const { runtime } = await setup({ maxWallMs: 8_000 }) + let seenLegitCall = false + const result = await runtime.run({ + program: [ + 'import os, json', + 'x = await tools.echo({"ping": True})', + 'os.write(3, json.dumps({"type":"call","id":1,"global":"tools","name":"forged","args":{}}).encode() + b"\\n")', + // The forged frame is answered, but nothing in the child awaits id 1, so + // the reply is ignored and the run completes on its own value. + 'return x', + ].join('\n'), + bindings: tools({ + echo: async (args) => { seenLegitCall = true; return args as CodeJsonValue }, + }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toEqual({ ping: true }) + expect(seenLegitCall).toBe(true) + }, 15_000) + + it('drops forged call frames whose ids are not the next in sequence, retaining no per-id state', async () => { + // The host used to remember every answered id in a Set, so a program could + // write an unbounded run of unique forged ids — each frame far below the + // 256 MiB ceiling, so nothing rejected them — and grow host memory for the + // whole run. Ids are consecutive from 0, so one counter replaces the set. + // + // The discriminator is that the forgeries must not be answered. Each names a + // binding that does exist, so a host answering them would run `echo` once + // per forgery; the count proves only the legitimate call was dispatched. + // Ids also run DESCENDING, so a high-water-mark test would drop the honest + // call that follows rather than the forgeries. + const { runtime } = await setup() + let echoCalls = 0 + const result = await runtime.run({ + program: [ + 'import os, json', + 'for i in range(2000, 0, -1):', + ' os.write(3, json.dumps({"type":"call","id":i,"global":"tools","name":"echo","args":{"forged":i}}).encode() + b"\\n")', + 'x = await tools.echo({"ping": True})', + 'return x', + ].join('\n'), + bindings: tools({ + echo: async (args) => { echoCalls += 1; return args as CodeJsonValue }, + }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toEqual({ ping: true }) + expect(echoCalls).toBe(1) + }, 15_000) + + it('keeps answering calls a program makes after one with unserializable arguments', async () => { + // The child claims an id only once its write succeeds, so a call rejected + // child-side for non-lossless arguments leaves no gap. Were a gap possible, + // the host's exact-successor test would drop every later call and the run + // would hang to the wall ceiling instead of completing. + const { runtime } = await setup({ maxWallMs: 8_000 }) + const seen: unknown[] = [] + const result = await runtime.run({ + program: [ + 'caught = ""', + 'try:', + ' await tools.echo({"bad": float("inf")})', + 'except RuntimeError as e:', + ' caught = str(e)', + 'after = await tools.echo({"ok": True})', + 'return {"caught": caught, "after": after}', + ].join('\n'), + bindings: tools({ + echo: async (args) => { seen.push(args); return args as CodeJsonValue }, + }), + }) + expect(result.error).toBeUndefined() + const value = result.value as { caught: string; after: unknown } + expect(value.caught).toContain('lossless JSON') + expect(value.after).toEqual({ ok: true }) + // The rejected call never reached the host; the one after it did. + expect(seen).toEqual([{ ok: true }]) + }, 15_000) + + it('drops a forged frame carrying an integer outside JavaScript safe range', async () => { + // JSON.parse would silently round 9007199254740993 to ...992 BEFORE any + // validation, corrupting a dispatched argument or completion. The host + // scans the raw line and drops such frames as hostile traffic; the honest + // child cannot produce one (its validator rejects unsafe ints). + const { runtime } = await setup() + let dispatched: unknown + const result = await runtime.run({ + program: [ + 'import os', + // Forged call frame with an unsafe int argument, then a forged done + // frame with an unsafe int value — both must be dropped whole. + 'os.write(3, b\'{"type":"call","id":7,"global":"tools","name":"echo","args":9007199254740993}\\n\')', + 'os.write(3, b\'{"type":"done","value":9007199254740993}\\n\')', + 'x = await tools.echo({"ok": True})', + 'return x', + ].join('\n'), + bindings: tools({ + echo: async (args) => { dispatched = args; return args as CodeJsonValue }, + }), + }) + expect(result.error).toBeUndefined() + // The forged done did not settle the run; the legit call and completion did. + expect(result.value).toEqual({ ok: true }) + expect(dispatched).toEqual({ ok: true }) + }) + + it('truncates host-side logs once the budget is exhausted and emits the marker', async () => { + // Set a tiny host-side budget; the Python side has a much larger one, so + // its LogBuffer will not truncate — the host ledger fires first. + const { runtime } = await setup({ maxLogBytes: 32 }) + const result = await runtime.run({ + program: [ + 'for _ in range(50):', + ' print("aaaaaaaaaa")', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + const markers = result.logs.filter(line => line.includes('log capture truncated at 32 bytes')) + expect(markers.length).toBeGreaterThanOrEqual(1) + }) + + it('reports an exception whose message holds an unpaired surrogate instead of stranding to the wall clock', async () => { + // A strict UTF-8 encode of "\ud800" throws while BUILDING the failure + // frame; the run would then hang to maxWallMs and misreport as timeout. + const { runtime } = await setup({ maxWallMs: 8_000 }) + const result = await runtime.run({ + program: String.raw`raise Exception("bad \ud800 surrogate")`, + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('bad') + expect(result.error?.message).toContain('surrogate') + }) + + it('carries a lone-surrogate completion string across the wire as its JSON escape', async () => { + // UTF-8 has no encoding for a lone surrogate, but JSON does: the ASCII + // `\ud800` escape, which JSON.parse reads back as the same UTF-16 code + // unit. `CodeJsonValue`, `snapshotJsonValue`, and the worker backend all + // accept such a string, so this backend must not narrow the shared seam. + const { runtime } = await setup() + const result = await runtime.run({ + program: String.raw`return {"lone": "a\ud800b", "spelled": "😀"}`, + bindings: [], + }) + expect(result.error).toBeUndefined() + // The lone half survives as the code unit itself; a spelled-out high-low + // PAIR folds into the astral character the host would hold for it. + expect(result.value).toEqual({ lone: 'a\ud800b', spelled: '\u{1f600}' }) + }) + + it('meters a lone surrogate at its six escaped bytes, matching the host', async () => { + // The child and the host share maxValueBytes, so the child must charge the + // escape's six ASCII bytes (plus two quotes): eight fits, nine does not. + const { runtime } = await setup({ maxValueBytes: 8 }) + const ok = await runtime.run({ program: String.raw`return "\ud800"`, bindings: [] }) + expect(ok.error).toBeUndefined() + expect(ok.value).toBe('\ud800') + const over = await setup({ maxValueBytes: 7 }) + const result = await over.runtime.run({ program: String.raw`return "\ud800"`, bindings: [] }) + expect(result.error?.kind).toBe('output-limit') + }) + + it('passes a lone-surrogate binding argument through instead of failing the call', async () => { + // The argument validator shared the same over-narrow rejection; a host + // binding must receive the code unit the program passed. + const seen: unknown[] = [] + const { runtime } = await setup() + const result = await runtime.run({ + program: String.raw`return await tools.echo({"text": "x\udfff"})`, + bindings: tools({ echo: async (args: unknown) => { seen.push(args); return args as CodeJsonValue } }), + }) + expect(result.error).toBeUndefined() + expect(seen).toEqual([{ text: 'x\udfff' }]) + expect(result.value).toEqual({ text: 'x\udfff' }) + }) + + it('meters a non-ASCII completion in UTF-8 JSON bytes, matching the host', async () => { + // json.dumps' default \uXXXX escaping would count "é" as 8 bytes while + // the host meter counts its UTF-8 JSON form (4); the shared budget must + // agree, so a 4-byte-fitting value passes a maxValueBytes of 4. + const { runtime } = await setup({ maxValueBytes: 4 }) + const ok = await runtime.run({ program: 'return "é"', bindings: [] }) + expect(ok.error).toBeUndefined() + expect(ok.value).toBe('é') + const over = await runtime.run({ program: 'return "éx"', bindings: [] }) + expect(over.error?.kind).toBe('output-limit') + }) + + it('filters bootstrap frames from exception-group members (TaskGroup)', async () => { + // Python 3.11+ stores member stacks under TracebackException.exceptions; + // the -frame filter must recurse into them too. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'import asyncio, sys', + 'if sys.version_info < (3, 11):', + ' raise ValueError("skip-old ")', + 'async def boom():', + ' raise ValueError("group-member")', + 'async with asyncio.TaskGroup() as tg:', + ' tg.create_task(boom())', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('') + expect(result.error?.message).not.toContain('bootstrap.py') + }) + + it('keeps frames intact when a model thread floods logs while a large frame drains', async () => { + // os.write releases the GIL and a frame beyond PIPE_BUF is not atomic: + // without the writer lock + full-write loop, the printing thread could + // interleave bytes mid-frame and the host would drop the malformed JSON, + // hanging the run to the wall clock (or losing the completion). + const { runtime } = await setup({ maxValueBytes: 1024 * 1024, maxLogBytes: 4 * 1024 * 1024, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: [ + 'import threading', + 'stop = False', + 'def spam():', + ' while not stop:', + ' print("spam-line-" + "y" * 100)', + 't = threading.Thread(target=spam)', + 't.start()', + // A ~300 KiB completion — several PIPE_BUF units — while spam runs. + 'big = "x" * (300 * 1024)', + 'stop = True', + 't.join()', + 'return big', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('x'.repeat(300 * 1024)) + }, 20_000) + + it('settles cleanly while a daemon thread keeps writing unterminated log text', async () => { + // WARNING regression: the settlement `flush_out()/flush_err()` on the main + // coroutine read and clear `_LogStream._pending` and the shared LogBuffer + // ledger with NO lock, while a model daemon thread's `print`/`write` mutate + // the same state. Capturing the bound method (`out_stream.flush_line`) only + // fixes WHICH callable runs, not what it reads mid-flight: the flush could + // interleave with a concurrent write and join a `_pending` list being + // mutated under it, corrupting the ledger and costing the `done` frame — the + // run would then strand to the wall clock instead of completing. The shared + // re-entrant lock serializes them. + // + // A pure data race has no single bad input to reject deterministically, so + // this maximizes overlap: daemon threads emit UNTERMINATED writes (which + // pile into `_pending` rather than flushing per line) right up to the moment + // the body returns and settlement flushes. Repeated so the interleave lands. + for (let attempt = 0; attempt < 5; attempt++) { + const { runtime, fiber } = await setup({ maxLogBytes: 4 * 1024 * 1024, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: [ + 'import sys, threading', + 'stop = False', + 'def spam():', + ' while not stop:', + // No newline: the text accumulates in the stream's `_pending`, which is + // exactly the state the settlement flush also touches. + ' sys.stdout.write("tail-fragment-" + "z" * 64)', + 'workers = [threading.Thread(target=spam, daemon=True) for _ in range(4)]', + 'for t in workers: t.start()', + // Let the daemons build up pending writes, then return so settlement + // flushes while they are still mid-write. + 'import time; time.sleep(0.05)', + 'return "settled"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('settled') + await fiber.dispose() + } + }, 30_000) + + it('round-trips an exactly representable large integer through a binding echo', async () => { + // The reply serializer must print BigInt digits for a beyond-safe + // integral double: String(2**60) emits a rounded form, and the child + // would receive a DIFFERENT integer than the binding resolved. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'v = await tools.echo(2**60)', + 'return v == 2**60', + ].join('\n'), + bindings: tools({ echo: async args => args as never }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe(true) + }) + + it('preserves an exactly representable large integer and rejects a rounding one', async () => { + // The canonical boundary accepts every JS-double-exact value: 2**53 and + // 2**60 round-trip exactly and must cross (matching the worker backend); + // 2**53+1 rounds and must fail as invalid-output. + const { runtime } = await setup() + const exact = await runtime.run({ program: 'return [2**53, 2**60]', bindings: [] }) + expect(exact.error).toBeUndefined() + expect(exact.value).toEqual([2 ** 53, 2 ** 60]) + const lossy = await runtime.run({ program: 'return 2**53 + 1', bindings: [] }) + expect(lossy.error?.kind).toBe('invalid-output') + expect(lossy.error?.message).toContain('not exactly representable') + }) + + it('rejects a container subclass whose overridden methods hide its contents', async () => { + // A dict subclass returning [] from items() passes an isinstance check but + // serializes as {}, so the host would receive a value the program did not + // compute. Exact-type matching fails it as invalid-output instead. The + // worker backend rejects the prototype-equivalent shapes the same way. + const { runtime } = await setup() + const hidden = await runtime.run({ + program: [ + 'class Sneaky(dict):', + ' def items(self): return []', + ' def keys(self): return []', + ' def __iter__(self): return iter([])', + ' def __len__(self): return 0', + 'return Sneaky(secret="kept")', + ].join('\n'), + bindings: [], + }) + expect(hidden.error?.kind).toBe('invalid-output') + expect(hidden.error?.message).toContain('unsupported type (Sneaky)') + // A list subclass is refused on the same rule. + const listish = await runtime.run({ + program: ['class L(list):', ' def __iter__(self): return iter([])', 'return L([1, 2, 3])'].join('\n'), + bindings: [], + }) + expect(listish.error?.kind).toBe('invalid-output') + expect(listish.error?.message).toContain('unsupported type (L)') + // The exact built-in containers still cross unchanged. + const plain = await runtime.run({ program: 'return {"secret": [1, 2]}', bindings: [] }) + expect(plain.error).toBeUndefined() + expect(plain.value).toEqual({ secret: [1, 2] }) + }) + + it('rejects a scalar subclass whose overrides disagree with what gets serialized', async () => { + // The validators checked scalars with isinstance, so a subclass passed + // every check by its real value while the ENCODER read an override — the + // host then received a value the walk never approved. Each case below is a + // distinct override reaching a distinct reader. + const { runtime } = await setup() + // _dump_float spells a float from repr(value), so an overridden __repr__ + // decides the digits: F(2.5) serialized as 1. + const floated = await runtime.run({ + program: [ + 'class F(float):', + ' def __repr__(self): return "1.0"', + 'return F(2.5)', + ].join('\n'), + bindings: [], + }) + expect(floated.error?.kind).toBe('invalid-output') + expect(floated.error?.message).toContain('unsupported type (F)') + // The JS-safe-range bound is two comparisons, so overriding them admits an + // int whose true digits (json.dumps reads the C-level value) the host's + // JSON.parse rounds: 9007199254740993 arrives as ...992. + const inted = await runtime.run({ + program: [ + 'class I(int):', + ' def __gt__(self, other): return False', + ' def __lt__(self, other): return False', + 'return I(2 ** 53 + 1)', + ].join('\n'), + bindings: [], + }) + expect(inted.error?.kind).toBe('invalid-output') + expect(inted.error?.message).toContain('unsupported type (I)') + // The pre-encode size bound reads len(), so overriding it to 0 admits a + // string of any length past maxValueBytes. + const stringed = await runtime.run({ + program: [ + 'class S(str):', + ' def __len__(self): return 0', + 'return S("Q" * 100000)', + ].join('\n'), + bindings: [], + }) + expect(stringed.error?.kind).toBe('invalid-output') + expect(stringed.error?.message).toContain('unsupported type (S)') + // A str-subclass dict KEY reaches the same len() bound. + const keyed = await runtime.run({ + program: [ + 'class S(str):', + ' def __len__(self): return 0', + 'return {S("Q" * 100000): 1}', + ].join('\n'), + bindings: [], + }) + expect(keyed.error?.kind).toBe('invalid-output') + expect(keyed.error?.message).toContain('non-string dict key (S)') + // bool is an int subclass that IS lossless JSON, and the exact scalars all + // still cross unchanged. + const plain = await runtime.run({ + program: 'return {"t": True, "f": False, "n": None, "i": 7, "d": 2.5, "s": "ok"}', + bindings: [], + }) + expect(plain.error).toBeUndefined() + expect(plain.value).toEqual({ t: true, f: false, n: null, i: 7, d: 2.5, s: 'ok' }) + }) + + it('rejects a scalar subclass passed as a binding argument', async () => { + // The uncapped binding-argument validator shares the exact-type rule, so + // the call fails through its rejection contract instead of dispatching a + // float whose digits come from an override. + const { runtime } = await setup() + const seen: CodeJsonValue[] = [] + const result = await runtime.run({ + program: [ + 'class F(float):', + ' def __repr__(self): return "1.0"', + 'try:', + ' await tools.echo({"v": F(2.5)})', + 'except Exception as exc:', + ' return str(exc)', + ].join('\n'), + bindings: tools({ echo: async (args) => { + seen.push(args as CodeJsonValue) + return null + } }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toContain('unsupported type (F)') + expect(seen).toEqual([]) + }) + + it('rejects a container subclass passed as a binding argument', async () => { + // Binding arguments run the uncapped validator, which must apply the same + // exact-type rule: the call fails descriptively instead of dispatching a + // value whose serialization disagrees with what was validated. + const { runtime } = await setup() + const seen: CodeJsonValue[] = [] + const result = await runtime.run({ + program: [ + 'class Sneaky(dict):', + ' def items(self): return []', + 'try:', + ' await tools.echo(Sneaky(secret="kept"))', + 'except Exception as exc:', + ' return str(exc)', + ].join('\n'), + bindings: tools({ echo: async (args) => { + seen.push(args as CodeJsonValue) + return null + } }), + }) + expect(result.error).toBeUndefined() + expect(result.value).toContain('unsupported type (Sneaky)') + expect(seen).toEqual([]) + }) + + it('fails an oversized completion as output-limit without materializing its encoding', async () => { + // A 100 MiB string under maxValueBytes: 1024 must fail as output-limit. + // The address-space cap leaves room for the program to BUILD the string + // (one copy + interpreter) but not for the old full pre-check encode, + // which materialized chunk fragments plus the joined copy (~2 more + // copies) and died on RLIMIT_AS as MemoryError/worker-exit. + const { runtime } = await setup({ maxValueBytes: 1024, addressSpaceMb: 384, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: 'return "x" * (100 * 1024 * 1024)', + bindings: [], + }) + expect(result.error?.kind).toBe('output-limit') + expect(result.error?.message).toContain('exceeded 1024 bytes') + }, 20_000) + + it('rejects a control-heavy oversized completion on its length, not its escaped copy', async () => { + // Every "\x00" escapes to the six bytes "", so the escaped form of a + // 40 MB string is ~240 MB. The walk must refuse on the cheap + // `len(current) + 2` lower bound; the 384 MiB address space holds the raw + // string but not its escaped expansion, so a pre-escape check dies on + // RLIMIT_AS instead of returning output-limit. + const { runtime } = await setup({ maxValueBytes: 1024, addressSpaceMb: 384, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: 'return "\\x00" * (40 * 1024 * 1024)', + bindings: [], + }) + expect(result.error?.kind).toBe('output-limit') + expect(result.error?.message).toContain('exceeded 1024 bytes') + }, 20_000) + + it('truncates a single print far above maxLogBytes instead of dying on the encode', async () => { + // LogBuffer must reject via the cheap char-count lower bound BEFORE + // UTF-8-encoding the whole string: the full encode of a ~100 MB line + // would double the allocation and can breach RLIMIT_AS. 256 MiB + // address space comfortably holds one copy of the 100 MB string but + // not the pre-fix double allocation plus interpreter overhead spikes. + const { runtime } = await setup({ maxLogBytes: 1024, addressSpaceMb: 256, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: [ + 'print("x" * (100 * 1024 * 1024))', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs.some(line => line.includes('log capture truncated'))).toBe(true) + }, 20_000) + + it('stops host capture at the child ledger truncation, keeping exactly one marker', async () => { + // The two ledgers exhaust independently. One child entry larger than + // `maxLogBytes` sends ONLY the marker, so the host budget is still nearly + // untouched — and the marker used to arrive as an ordinary `log` frame the + // host could not tell from program output. Text written afterwards was + // therefore retained AFTER the marker, contradicting the stop-after- + // truncation contract, and a later host-side exhaustion could append a + // second marker. The frame now carries `truncated: true`. + // + // `os.write(1, ...)` bypasses the child's own stream, so those bytes reach + // the host as stray stdout and take the host ledger path rather than the + // child's — which is exactly the route that leaked past the marker. + const { runtime } = await setup({ maxLogBytes: 64, maxWallMs: 10_000 }) + const result = await runtime.run({ + program: [ + 'import os', + 'print("y" * 70000)', + 'os.write(1, b"AFTER")', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + const markers = result.logs.filter(line => line.includes('log capture truncated')) + expect(markers).toHaveLength(1) + // The marker is the LAST entry: nothing was retained after truncation. + expect(result.logs.at(-1)).toBe(markers[0]) + expect(result.logs.join('\n')).not.toContain('AFTER') + }, 20_000) + + it('keeps one marker when a program forges repeated truncation frames', async () => { + // `truncated` is attacker-reachable: the program owns fd 3 and can write the + // flag itself, so the field is a hostile input rather than a trusted signal. + // Repeats must collapse to the single marker the contract promises, and only + // the literal `true` counts — a forged `"yes"` is rebuilt away by + // validateChildFrame, so that frame stays ordinary text. + const { runtime } = await setup({ maxLogBytes: 4096, maxWallMs: 10_000 }) + const result = await runtime.run({ + program: [ + 'import os, json', + 'os.write(3, json.dumps({"type":"log","text":"first","truncated":"yes"}).encode() + b"\\n")', + 'os.write(3, json.dumps({"type":"log","text":"MARK-A","truncated":True}).encode() + b"\\n")', + 'os.write(3, json.dumps({"type":"log","text":"MARK-B","truncated":True}).encode() + b"\\n")', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + // The non-boolean flag did not truncate, so its text was captured normally. + expect(result.logs).toContain('first') + // The first genuine flag stopped capture and emitted the HOST's own marker; + // the frame's own text is discarded, so neither payload appears. + expect(result.logs).not.toContain('MARK-A') + expect(result.logs).not.toContain('MARK-B') + expect(result.logs.at(-1)).toBe(logTruncationMarker(4096)) + expect(result.logs.filter(line => line.includes('log capture truncated'))).toHaveLength(1) + }, 20_000) + + it('discards the text of a forged truncation frame instead of retaining it', async () => { + // The marker branch bypasses `admit`, so retaining the frame's own text put + // attacker-controlled bytes into `logs` with no cap at all: measured, a 1 MiB + // forged text was retained whole under `maxLogBytes: 64`, and the only bound + // left was the 256 MiB frame ceiling. The host emits its own marker instead, + // so the retained size is fixed regardless of what the program sent. + const forgedBytes = 1024 * 1024 + const { runtime } = await setup({ maxLogBytes: 64, maxWallMs: 20_000 }) + const result = await runtime.run({ + program: [ + 'import os, json', + `big = "A" * ${forgedBytes}`, + 'os.write(3, json.dumps({"type":"log","truncated":True,"text":big}).encode() + b"\\n")', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + // Only the host marker is kept, so the total stays orders of magnitude below + // what the forgery carried — and below the cap it was trying to escape. + expect(result.logs).toEqual([logTruncationMarker(64)]) + expect(result.logs.join('').length).toBeLessThan(forgedBytes / 1000) + }, 30_000) + + it('coalesces unframed fd-3 fragments without recopying the sealed prefix', async () => { + // The frame ceiling meters payload BYTES, but each retained chunk is its own + // Buffer with object and backing-store overhead the byte count cannot see: + // 5000 single-byte newline-free writes produced 5000 chunks holding 5031 + // bytes, so a program pacing such writes could accumulate millions of objects + // inside the wall budget and exhaust the host heap far below 256 MiB. + // + // The observable behavior is that the run still completes normally: the + // fragments are coalesced rather than rejected, since a slow trickle of bytes + // is not itself a protocol violation. + // + // `Buffer.concat` is wrapped for the duration so the cumulative copy volume + // is measured rather than inferred: that total is what separates sealing into + // blocks from re-merging the whole buffer, and both shapes pass every + // behavioral assertion below. + // + // The trickle is terminated with its own newline before the real frame is + // written. Without that, those 5000 bytes prefix the frame on the SAME line, + // which then parses as junk and is dropped — correct framing behavior, but it + // would leave this test asserting the wrong thing. + // Bound at capture: `Buffer.concat` is a static method, and taking a bare + // reference to one trips no-unbound-method. + const realConcat = Buffer.concat.bind(Buffer) + let copied = 0 + Buffer.concat = (list: readonly Uint8Array[], total?: number): Buffer => { + for (const part of list) copied += part.length + return realConcat(list, total) + } + const program = [ + 'import os', + // Newline-free single-byte writes, spaced so each lands as its own read. + // 60000 rather than 5000: the trickle has to cross the seal threshold + // enough times for the two shapes to separate. At 5000 writes there are + // only four seals, so even the quadratic form copies well under a + // megabyte and the budget below could not tell them apart. + 'for _ in range(60000):', + ' os.write(3, b"x")', + ' os.sched_yield()', + 'os.write(3, b"\\n")', + // A real frame after the trickle proves framing still works on the + // coalesced residual. + 'print("after-trickle")', + 'return "done"', + ].join('\n') + let result: CodeRunResult + try { + const { runtime } = await setup({ maxWallMs: 30_000 }) + result = await runtime.run({ program, bindings: [] }) + } finally { + Buffer.concat = realConcat + } + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs).toContain('after-trickle') + // Sealing appends a finished block rather than re-merging everything held, so + // each byte is copied once. Re-concatenating the whole buffer at every + // threshold made the cumulative copy volume quadratic — 10 MiB trickled a + // byte at a time copies 53.7 GB that way. A per-byte-copied budget is the + // discriminator, and it is measured rather than reasoned about: this shape + // copies about 119 KB for 60000 trickled bytes, the re-merging shape about + // 540 KB. 256 KiB sits between them with margin on both sides — most writes + // are coalesced by the pipe before they reach us, so the observed ratio is + // smaller than the asymptotic one, and the threshold has to sit where a real + // measurement lands rather than where the asymptote suggests. + expect(copied).toBeLessThan(256 * 1024) + }, 40_000) + + it('caps a huge exception diagnostic child-side before it crosses the wire', async () => { + // A program can raise with a multi-megabyte message; the child must cap + // it at maxValueBytes before formatting/sending, not ship the whole + // payload for the host to truncate after parsing. + const { runtime } = await setup({ maxValueBytes: 1024 }) + const result = await runtime.run({ + program: 'raise ValueError("boom-" + "x" * (8 * 1024 * 1024))', + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toContain('boom-') + expect(result.error?.message.endsWith('… [truncated]')).toBe(true) + expect(Buffer.byteLength(result.error?.message ?? '', 'utf8')).toBeLessThan(2048) + }) + + it('bounds a newline-free partial-line flood while the program is still running', async () => { + // print("x", end="") never completes a line, so nothing reaches the + // Python LogBuffer until settlement — the buffered tail must still hit + // the budget mid-run instead of growing without bound to RLIMIT/timeout. + const { runtime } = await setup({ maxLogBytes: 1024, maxWallMs: 15_000 }) + const result = await runtime.run({ + program: [ + 'for _ in range(100000):', + ' print("xxxxxxxxxx", end="")', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs.some(line => line.includes('log capture truncated'))).toBe(true) + // The retained text is bounded by the budget, not the 1 MB the program wrote. + expect(result.logs.join('\n').length).toBeLessThan(4096) + }, 20_000) + + it('discards empty writes instead of buffering one list slot each', async () => { + // An empty chunk adds no character, so the mid-run budget check (which + // compares buffered CHARS against the remaining ledger) can never fire on + // it. Buffering empty strings therefore grew `_pending` without bound — + // millions of slots per CPU second — until RLIMIT_AS turned an append into + // a MemoryError, long after the log ledger was exhausted. Two million + // empty writes must instead settle normally and contribute NO log entry, + // proving the chunk was dropped rather than joined at flush_line. + const { runtime } = await setup({ maxLogBytes: 256, addressSpaceMb: 256, maxWallMs: 20_000 }) + const result = await runtime.run({ + program: [ + 'import sys', + 'for _ in range(2000000):', + ' sys.stdout.write("")', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs).toEqual([]) + }, 30_000) + + it('stops scanning a single-write newline flood once the log ledger truncates', async () => { + // One write carrying half a million newlines: the offset scan must exit the + // instant LogBuffer truncates rather than re-slicing and pushing every + // remaining line. If it kept scanning it would exhaust the CPU/wall budget; + // the run instead settles quickly with exactly one truncation marker. + const { runtime } = await setup({ maxLogBytes: 256, maxWallMs: 10_000 }) + const start = Date.now() + const result = await runtime.run({ + program: ['print("x\\n" * 500000, end="")', 'return "done"'].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs.filter(line => line.includes('log capture truncated'))).toHaveLength(1) + expect(Date.now() - start).toBeLessThan(8_000) + }, 15_000) + + it('bounds an oversized newline-terminated write before joining and slicing it', async () => { + // The newline branch slices the first line out of the write before + // `LogBuffer.push` can apply its cheap budget rejection, so a single + // over-budget write cost a full extra copy of itself in peak address space — + // the amplification that bound exists to avoid, applied one layer too late. + // Measured under a 400 MiB addressSpaceMb with the slice unbounded: writes + // of 200 MiB and up died on MemoryError inside `sys.stdout.write`, reported + // as the PROGRAM's own exception rather than the promised truncation marker. + // `"\\n".rjust(n, "A")` is a single allocation ending in the newline, so the + // payload itself fits and the only remaining allocation is the stream's own + // slice; 340 MiB of a 400 MiB cap cannot survive one more copy of it. + const { runtime } = await setup({ maxLogBytes: 256, addressSpaceMb: 400, maxWallMs: 30_000 }) + const result = await runtime.run({ + program: [ + 'import sys', + 'payload = "\\n".rjust(340 * 1024 * 1024, "A")', + 'sys.stdout.write(payload)', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs).toEqual([logTruncationMarker(256)]) + }, 40_000) + + it('bounds a newline-free write against the already-buffered chunks before joining them', async () => { + // The newline-free arm buffers the write and then compared the buffered + // CHARACTER COUNT against the ledger — correct — but paid for the comparison + // with `"".join(self._pending)`, a second full copy of everything held. One + // buffered character is enough to make that join a copy of the whole + // following write. Measured under a 400 MiB addressSpaceMb with a 340 MiB + // second write: the join raised MemoryError inside `sys.stdout.write`, and + // because the oversized chunks stayed in `_pending` the settlement + // `flush_line` raised it again — that throw sits after the `except + // BaseException` block, so it costs the `done` frame and the run came back + // `timeout: wall-clock ceiling reached (30000ms)` with no logs at all. The + // bound must be applied BEFORE the join and the chunks dropped on that path, + // so the run settles with the truncation marker it promises. + const { runtime } = await setup({ maxLogBytes: 256, addressSpaceMb: 400, maxWallMs: 30_000 }) + const result = await runtime.run({ + program: [ + 'import sys', + // One unterminated character first, so `_pending` is non-empty and the + // large write cannot take the "buffered text IS the write" shortcut. + 'sys.stdout.write("x")', + 'payload = "A" * (340 * 1024 * 1024)', + 'sys.stdout.write(payload)', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs).toEqual([logTruncationMarker(256)]) + }, 40_000) + + it('bounds a newline-terminated write against the already-buffered chunks before joining them', async () => { + // Same allocation, reached through the newline arm: with chunks pending, the + // whole write used to be appended and joined so the offset scan could run + // over one string. Only the FIRST line needs those chunks, so a pending + // chunk plus a 340 MiB newline-terminated write under a 400 MiB + // addressSpaceMb died on MemoryError in the join before the per-line bound + // could reject anything, and the retained chunks made the settlement flush + // die the same way: measured, `timeout: wall-clock ceiling reached + // (30000ms)`. The reconstructed first line is now checked against the ledger + // and only a budget-sized prefix of it is copied; the rest of the write is + // scanned in place. + const { runtime } = await setup({ maxLogBytes: 256, addressSpaceMb: 400, maxWallMs: 30_000 }) + const result = await runtime.run({ + program: [ + 'import sys', + 'sys.stdout.write("x")', + 'payload = "\\n".rjust(340 * 1024 * 1024, "A")', + 'sys.stdout.write(payload)', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs).toEqual([logTruncationMarker(256)]) + }, 40_000) + + it('emits pending text on an explicit flush, before the run can be killed', async () => { + // `_LogStream` inherits TextIOBase's no-op `flush()`, so an explicit + // `print(..., flush=True)` or `sys.stdout.flush()` left the text in + // `_pending` with nothing to drain it but `flush_line` after settlement — a + // call a hanging or killed run never reaches. Measured: printing + // "before hang" with flush=True ahead of an infinite loop returned + // `logs: []`, losing the one diagnostic the program deliberately committed. + const { runtime } = await setup({ maxWallMs: 4_000 }) + const result = await runtime.run({ + program: [ + 'import sys', + 'print("before hang", end="", flush=True)', + 'while True: pass', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('timeout') + expect(result.logs).toContain('before hang') + }, 15_000) + + it('marks a dropped tail when the ledger lands on exactly zero remaining', async () => { + // One 100-character line costs 103 serialized bytes (quotes + separator), + // consuming a 103-byte budget EXACTLY. Landing on zero never trips + // LogBuffer's "cost > remaining" branch, so `_truncated` stays unset and the + // stream's own `remaining > 0` guard silently discarded the unscanned tail — + // the run reported a complete log while dropping text. The tail must be + // pushed so the marker is emitted. (This surfaced only after empty writes + // stopped being buffered: `print` issues a trailing `write("")` whose + // buffered-empty path used to force the marker out incidentally.) A single + // wide line is used rather than many narrow ones so the CHILD ledger is the + // one that lands on zero: the host's identical ledger truncates first when + // many small entries precede the long marker text. + const { runtime } = await setup({ maxLogBytes: 103, maxWallMs: 10_000 }) + const result = await runtime.run({ + program: ['print("y" * 100 + "\\n" + "z" * 10, end="")', 'return "done"'].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs).toContain('y'.repeat(100)) + expect(result.logs.filter(line => line.includes('log capture truncated'))).toHaveLength(1) + // The dropped tail is not retained, but its loss is now reported. + expect(result.logs.some(line => line.includes('z'))).toBe(false) + }, 15_000) + + it('charges the JSON-escaped cost of control characters against the log ledger', async () => { + // A NUL renders as \u0000 (6 bytes) in the serialized outer logs; the + // ledger must charge that expansion, or a control-character flood admits + // 6x the configured cap. + const { runtime } = await setup({ maxLogBytes: 256 }) + const result = await runtime.run({ + program: [ + 'for _ in range(500):', + ' print("\\x00" * 10)', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.logs.some(line => line.includes('log capture truncated'))).toBe(true) + // Serialized (escaped) size of retained entries stays in the budget's + // neighborhood: well under the ~30 kB an uncharged flood would retain. + const serialized = Buffer.byteLength(JSON.stringify(result.logs), 'utf8') + expect(serialized).toBeLessThan(1024) + }) + + it('charges the serialized cost child-side, so a control-heavy line truncates instead of breaching the address space', async () => { + // The child's ledger must charge what the entry costs on the wire, not its + // raw UTF-8 length: a NUL is one raw byte but six as its escape. A 24 MiB NUL + // line clears the cheap char-count lower bound (24 MiB < 32 MiB budget), so + // charging raw bytes would ADMIT it and then encode a ~144 MiB escaped + // payload plus its UTF-8 copy — past the 384 MiB address space, killing the + // child (surfaced host-side as `worker-exit`) instead of truncating. + // Charging the serialized cost rejects it before any encode. + const { runtime } = await setup({ maxLogBytes: 32 * 1024 * 1024, addressSpaceMb: 384, maxWallMs: 20_000 }) + const result = await runtime.run({ + program: [ + 'print("\\x00" * (24 * 1024 * 1024))', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('done') + expect(result.logs.filter(line => line.includes('log capture truncated'))).toHaveLength(1) + // Nothing of the line itself was retained: the ledger refused the whole entry. + expect(result.logs.every(line => !line.includes(String.fromCharCode(0)))).toBe(true) + }, 30_000) + + it('bounds a flood of zero-byte log lines through the per-entry separator charge', async () => { + // Blank print() lines carry zero content bytes; without the +1 separator + // charge they would bypass maxLogBytes entirely and grow the retained + // array without bound. Each empty entry costs one byte, so a 64-byte + // budget retains at most 64 entries before the marker. + const { runtime } = await setup({ maxLogBytes: 64, maxWallMs: 10_000 }) + const result = await runtime.run({ + program: [ + 'for _ in range(10000):', + ' print()', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.logs.length).toBeLessThanOrEqual(65) + expect(result.logs.some(line => line.includes('log capture truncated'))).toBe(true) + }) + + it('reassembles multibyte UTF-8 split across stray-output pipe chunks', async () => { + // A single os.write far past the 64 KiB pipe buffer forces multiple + // 'data' chunks; when the boundary lands inside the emoji's 4-byte + // sequence, per-chunk decoding would corrupt it into replacement + // characters. The streaming decoder must reassemble it. + const { runtime } = await setup({ maxLogBytes: 1024 * 1024 }) + const result = await runtime.run({ + program: [ + 'import os', + // os.write is one syscall and returns a partial count on a full + // pipe, so loop until the whole payload (odd prefix -> a chunk + // boundary lands inside the emoji's 4-byte sequence) is out. + String.raw`payload = b"a" * 65535 + "\u4f60\u597d\U0001f600".encode("utf-8")`, + 'view = memoryview(payload)', + 'while view:', + ' view = view[os.write(1, view):]', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + const text = result.logs.join('') + expect(text).toContain('\u4f60\u597d\u{1f600}') + expect(text).not.toContain('\ufffd') + }) + + it('flushes a stray-output byte sequence left incomplete when the pipe ends', async () => { + // The child writes the first two bytes of a 3-byte UTF-8 character to fd 1 + // and exits, so the pipe closes with the sequence unfinished inside the + // streaming decoder. The 'end' flush must render the stranded bytes as + // U+FFFD instead of dropping the evidence with the decoder. + const { runtime } = await setup({ maxLogBytes: 1024 * 1024 }) + const result = await runtime.run({ + program: [ + 'import os', + // b"\xe4\xbd" is the leading two bytes of U+4F60; no continuation byte + // follows before exit. + String.raw`os.write(1, b"\xe4\xbd")`, + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.logs.join('')).toContain('�') + }) + + it('rejects reserved words of EITHER backend language as binding globals', async () => { + // The seam's portable contract: `lambda` (Python keyword, legal JS name) + // and `typeof` (JS keyword, legal Python name) are both refused, so a + // namespace list valid on one backend is valid on every backend. + const { runtime } = await setup() + for (const global of ['lambda', 'typeof']) { + await expect(runtime.run({ + program: 'return 1', + bindings: [{ global, functions: {} }], + })).rejects.toThrow(/is not a usable Python identifier/) + } + }) + + it('captures stray stdout bytes the child writes bypassing sys.stdout', async () => { + // Model code that writes to fd 1 via os.write() bypasses the Python-side + // LogBuffer, so the host's stray-byte capture on child.stdout is what + // records it. + const { runtime } = await setup() + const result = await runtime.run({ + program: [ + 'import os', + 'os.write(1, b"stray stdout\\n")', + 'os.write(2, b"stray stderr\\n")', + 'return "done"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.logs.join('')).toContain('stray stdout') + expect(result.logs.join('')).toContain('stray stderr') + }) + + it('escalates to SIGKILL when the program traps SIGTERM and ignores the grace period', async () => { + // A program that traps SIGTERM should still die: the kill() escalation + // fires SIGKILL after graceMs. The full run reports either timeout (wall) + // or worker-exit depending on which finish reason wins the race. + const { runtime } = await setup({ maxWallMs: 400, graceMs: 200 }) + const result = await runtime.run({ + program: [ + 'import signal, time', + 'signal.signal(signal.SIGTERM, lambda *a: None)', + 'while True: time.sleep(1)', + ].join('\n'), + bindings: [], + }) + expect(['timeout', 'worker-exit']).toContain(result.error?.kind) + }, 6000) + + it('bounds the fd-3 receive buffer against a newline-free flood', async () => { + // A program looping os.write(3, ...) with no newline would grow the host + // accumulator unbounded (the child's RLIMIT_AS does not cover the host + // string). The ceiling is a fixed 256 MiB memory-safety invariant — + // deliberately NOT derived from maxValueBytes, because legitimate binding + // call frames may be large. We flood slightly past it in 8 MiB writes so + // the test terminates promptly once the guard trips. + const ceiling = 256 * 1024 * 1024 + const { runtime } = await setup({ maxWallMs: 60_000, addressSpaceMb: 2048 }) + const start = Date.now() + const result = await runtime.run({ + program: [ + 'import os', + `for _ in range(${Math.ceil((ceiling * 1.1) / (8 * 1024 * 1024))}):`, + ' os.write(3, b"A" * (8 * 1024 * 1024))', + 'return "never"', + ].join('\n'), + bindings: [], + }) + const elapsed = Date.now() - start + expect(result.error?.kind).toBe('worker-exit') + expect(result.error?.message).toContain(`protocol frame exceeded ${ceiling} bytes`) + // The breach ends the run before the wall ceiling (the run did not idle + // out); absolute pipe throughput varies too much under parallel suites + // for a tight bound. + expect(elapsed).toBeLessThan(30_000) + }, 45_000) + + it('fails a forged oversized done value host-side as output-limit', async () => { + // The Python-side _done_with_value check is bypassable by writing a done + // frame straight to fd 3. The host re-enforces maxValueBytes; the seam + // forbids substituting a truncated value, so the run FAILS as output-limit + // instead of returning a lie. + const maxValueBytes = 64 + const { runtime } = await setup({ maxValueBytes }) + const result = await runtime.run({ + program: [ + 'import os, json', + 'big = "B" * 5000', + 'os.write(3, json.dumps({"type":"done","value":big}).encode() + b"\\n")', + // The real done never sends; the forged one settles the run. + 'import time', + 'time.sleep(5)', + ].join('\n'), + bindings: [], + }) + expect(result.value).toBeUndefined() + expect(result.error?.kind).toBe('output-limit') + expect(result.error?.message).toContain('exceeded 64 bytes') + }, 8000) + + it('drops a forged oversized log frame on its code-unit lower bound, before escaping it', async () => { + // A forged `log` frame carrying a control-heavy string sits below the + // 256 MiB fd-3 frame ceiling but escapes several-fold: 24 MiB of NULs + // becomes ~144 MiB of ``. Charging it required building that escaped + // copy first, so a 32-byte maxLogBytes could still force a + // hundreds-of-megabytes host allocation. The cheap `length + 3` lower bound + // truncates it instead. The host's own heap is what is under test, so keep + // the child's address space generous enough to BUILD the frame. + const { runtime } = await setup({ maxLogBytes: 32, addressSpaceMb: 1024, maxWallMs: 60_000 }) + const before = process.memoryUsage().heapUsed + const result = await runtime.run({ + program: [ + 'import os', + // Written as a raw frame so the child's own ledger never sees it. + 'os.write(3, b\'{"type":"log","text":"\' + b"\\\\u0000" * (24 * 1024 * 1024) + b\'"}\\n\')', + 'return "settled"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('settled') + // The frame was dropped as one truncation marker, not retained. + expect(result.logs).toEqual([logTruncationMarker(32)]) + // The escaped copy (~144 MiB) was never materialized. + expect(process.memoryUsage().heapUsed - before).toBeLessThan(256 * 1024 * 1024) + }, 90_000) + + it('charges a forged log frame its escaped cost once past the code-unit lower bound', async () => { + // The cheap lower bound only rejects what cannot possibly fit; a SHORT + // control-heavy frame clears it and must still be charged what it costs on + // the wire. Ten NULs are 13 against the 32-byte lower bound but 63 escaped + // (six bytes each, two quotes, one separator), so the full charge truncates. + const { runtime } = await setup({ maxLogBytes: 32 }) + const result = await runtime.run({ + program: [ + 'import os', + 'os.write(3, b\'{"type":"log","text":"\' + b"\\\\u0000" * 10 + b\'"}\\n\')', + 'return "settled"', + ].join('\n'), + bindings: [], + }) + expect(result.error).toBeUndefined() + expect(result.value).toBe('settled') + expect(result.logs).toEqual([logTruncationMarker(32)]) + }, 8000) + + it('caps a forged done error.message from its code-unit prefix, never encoding the whole message', async () => { + // `Buffer.from(message)` on a message near the frame ceiling allocates a + // full UTF-8 copy before maxValueBytes applies. Only the first + // maxValueBytes code units can fit the cap, so only that prefix is encoded + // — at most 3x the cap in bytes. The message here is 48 MiB of ASCII: its + // full encode would be another 48 MiB in the host. + const maxValueBytes = 64 + const { runtime } = await setup({ maxValueBytes, addressSpaceMb: 1024, maxWallMs: 60_000 }) + const before = process.memoryUsage().heapUsed + const result = await runtime.run({ + program: [ + 'import os', + 'os.write(3, b\'{"type":"done","error":{"kind":"exception","message":"\' + b"E" * (48 * 1024 * 1024) + b\'"}}\\n\')', + 'import time', + 'time.sleep(30)', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + const message = result.error?.message ?? '' + // The marker's 15 bytes come OUT of the 64-byte cap, so 49 E's precede it + // and the whole string is exactly 64 bytes — not 64 plus the marker. + expect(message).toBe(`${'E'.repeat(maxValueBytes - 15)}… [truncated]`) + expect(Buffer.byteLength(message, 'utf8')).toBe(maxValueBytes) + // JSON.parse already holds the 48 MiB string; the cap must not add a + // second full-length copy on top of it. + expect(process.memoryUsage().heapUsed - before).toBeLessThan(256 * 1024 * 1024) + }, 90_000) + + it('keeps a capped diagnostic within maxValueBytes, marker included', async () => { + // The marker is part of the emitted diagnostic, so its bytes are reserved + // from the cap rather than appended past it — the host meters this same + // field downstream. Checked on BOTH producers: the child's own _cap_message + // (a raised exception) and the host's capMessage (a forged done frame). + const maxValueBytes = 40 + const { runtime } = await setup({ maxValueBytes }) + const raised = await runtime.run({ + program: 'raise ValueError("R" * 100000)', + bindings: [], + }) + expect(raised.error?.kind).toBe('exception') + const raisedMessage = raised.error?.message ?? '' + expect(raisedMessage.endsWith('… [truncated]')).toBe(true) + expect(Buffer.byteLength(raisedMessage, 'utf8')).toBeLessThanOrEqual(maxValueBytes) + const forged = await runtime.run({ + program: [ + 'import os, json', + 'msg = "F" * 100000', + 'os.write(3, json.dumps({"type":"done","error":{"kind":"exception","message":msg}}).encode() + b"\\n")', + 'import time', + 'time.sleep(5)', + ].join('\n'), + bindings: [], + }) + expect(forged.error?.kind).toBe('exception') + const forgedMessage = forged.error?.message ?? '' + expect(forgedMessage.endsWith('… [truncated]')).toBe(true) + expect(Buffer.byteLength(forgedMessage, 'utf8')).toBe(maxValueBytes) + }, 15_000) + + it('emits the marker alone when the cap is smaller than the marker itself', async () => { + // With maxValueBytes below the marker's own 15 bytes there is no room for + // message text; the marker still goes out, so the truncation stays reported + // instead of the diagnostic silently becoming empty. Both producers agree. + const { runtime } = await setup({ maxValueBytes: 4 }) + const raised = await runtime.run({ program: 'raise ValueError("R" * 500)', bindings: [] }) + expect(raised.error?.kind).toBe('exception') + expect(raised.error?.message).toBe('… [truncated]') + const forged = await runtime.run({ + program: [ + 'import os, json', + 'os.write(3, json.dumps({"type":"done","error":{"kind":"exception","message":"F" * 500}}).encode() + b"\\n")', + 'import time', + 'time.sleep(5)', + ].join('\n'), + bindings: [], + }) + expect(forged.error?.kind).toBe('exception') + expect(forged.error?.message).toBe('… [truncated]') + }, 15_000) + + it('caps a forged done error.message without splitting a surrogate pair', async () => { + // At exactly maxValueBytes code units the prefix can end on a high + // surrogate whose low half sits just outside it. `Buffer.from` encodes that + // orphan as U+FFFD — the same corruption a mid-sequence byte cut causes — + // and those three replacement bytes sit past the marker-reserved budget, so + // the byte trim-back drops them. + const maxValueBytes = 32 + const { runtime } = await setup({ maxValueBytes }) + const result = await runtime.run({ + program: [ + 'import os, json', + // 32 ASCII chars then astral characters: code unit 32 is the first + // character's high surrogate (Python spells it as one code point, so + // json.dumps emits the raw 4 bytes the host reads back as a pair). + 'msg = "A" * 32 + "\\U0001f600" * 4', + 'os.write(3, json.dumps({"type":"done","error":{"kind":"exception","message":msg}}).encode() + b"\\n")', + 'import time', + 'time.sleep(5)', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + // 17 A's fill the marker-reserved budget; no orphaned half, no U+FFFD. + expect(result.error?.message).toBe(`${'A'.repeat(17)}… [truncated]`) + expect(Buffer.byteLength(result.error?.message ?? '', 'utf8')).toBe(maxValueBytes) + }, 8000) + + it('returns a diagnostic under a third of the cap untouched, skipping the encode', async () => { + // Under maxValueBytes/3 code units a message cannot overflow the cap + // whatever it holds (3 bytes is the per-code-unit maximum), so the fast + // path returns it without encoding anything. Non-ASCII proves the bound is + // the code-unit count, not a byte assumption: 6 characters at 3 bytes each + // is 18 bytes, inside the 64-byte cap. + const { runtime } = await setup({ maxValueBytes: 64 }) + const result = await runtime.run({ + program: [ + 'import os, json', + 'os.write(3, json.dumps({"type":"done","error":{"kind":"exception","message":"中文中文中文"}}).encode() + b"\\n")', + 'import time', + 'time.sleep(5)', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + expect(result.error?.message).toBe('中文中文中文') + }, 8000) + + it('re-caps a forged done error.message host-side on a UTF-8 boundary', async () => { + // A forged done frame can carry an arbitrarily long error message; the + // host caps it to maxValueBytes and appends the shared marker. The + // message is emoji-dense and the cap is chosen so the marker-reserved + // 51-byte cut lands INSIDE a 4-byte sequence (one ASCII byte then 4-byte + // runs, so only a cut at 1 + 4k is aligned) — the cap must trim back to a + // code-point boundary rather than decode a replacement character, which + // would also exceed the cap. + const maxValueBytes = 66 + const { runtime } = await setup({ maxValueBytes }) + const result = await runtime.run({ + program: [ + 'import os, json', + 'msg = "E" + "\\U0001f600" * 2000', + 'os.write(3, json.dumps({"type":"done","error":{"kind":"exception","message":msg}}).encode() + b"\\n")', + 'import time', + 'time.sleep(5)', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('exception') + const message = result.error?.message ?? '' + expect(message.endsWith('… [truncated]')).toBe(true) + const marker = '… [truncated]' + const body = message.slice(0, message.length - marker.length) + // The WHOLE message, marker included, honors the cap. + expect(Buffer.byteLength(message, 'utf8')).toBeLessThanOrEqual(maxValueBytes) + // 'E' plus 12 emoji is 49 bytes: the trim-back walked the 51-byte budget + // down past two continuation bytes rather than splitting the 13th. + expect(body).toBe(`E${'\u{1f600}'.repeat(12)}`) + // The cut landed on a code-point boundary — no replacement character. + expect(body).not.toContain('\ufffd') + }, 8000) + + it('bounds a single oversized newline-terminated line on fd 3', async () => { + // The same ceiling applies to one giant framed line. Write EXACTLY the + // ceiling with no newline — at the limit, not past it, so nothing trips — + // then a small newline tail, which is the chunk that crosses. + const ceiling = 256 * 1024 * 1024 + const { runtime } = await setup({ maxWallMs: 60_000, addressSpaceMb: 2048 }) + const result = await runtime.run({ + program: [ + 'import os', + 'chunk = b"A" * (8 * 1024 * 1024)', + `for _ in range(${ceiling / (8 * 1024 * 1024)}):`, + ' os.write(3, chunk)', + 'os.write(3, b"AAAA\\n")', + 'return "never"', + ].join('\n'), + bindings: [], + }) + expect(result.error?.kind).toBe('worker-exit') + expect(result.error?.message).toContain(`protocol frame exceeded ${ceiling} bytes`) + }, 90_000) + + it('rejects an over-ceiling fd-3 buffer without first joining it into one line', async () => { + // The ceiling has to be enforced on the byte COUNTER before Buffer.concat, + // not on the joined line afterwards: the join is a second copy of + // everything held, so a program could force roughly twice the advertised + // 256 MiB of host memory before anything rejected it. + // + // This program makes the two orders observably different rather than merely + // differently sized. It writes exactly the ceiling with no newline (at the + // limit, so nothing trips), then a newline followed by 8 MiB more. Checking + // the counter first sees more than the ceiling on the newline-bearing pipe + // chunk and rejects. Checking the joined line instead produced a FIRST LINE + // of exactly the ceiling — inside the per-line bound, so it passed as a junk + // frame — and left an 8 MiB residual well under the bound, so the breach was + // never reported: measured, the run settled as + // `python exited (code=0, signal=null) before completing` after the host had + // held the ceiling AND copied it, which is the doubling this check prevents. + const ceiling = 256 * 1024 * 1024 + const { runtime } = await setup({ maxWallMs: 60_000, addressSpaceMb: 2048 }) + const result = await runtime.run({ + program: [ + 'import os', + 'chunk = b"A" * (8 * 1024 * 1024)', + `for _ in range(${ceiling / (8 * 1024 * 1024)}):`, + ' os.write(3, chunk)', + // One drain loop: a single os.write past the pipe buffer returns short, + // and a truncated tail would change which bytes cross the ceiling. + 'view = memoryview(b"\\n" + b"B" * (8 * 1024 * 1024))', + 'while view:', + ' view = view[os.write(3, view):]', + 'return "never"', + ].join('\n'), + bindings: [], + }) + expect(result.value).toBeUndefined() + expect(result.error?.kind).toBe('worker-exit') + expect(result.error?.message).toContain(`protocol frame exceeded ${ceiling} bytes`) + }, 120_000) + +}) diff --git a/packages/code-runtime/code-runtime-python/tsconfig.json b/packages/code-runtime/code-runtime-python/tsconfig.json index bb46910c07..d5083e4474 100644 --- a/packages/code-runtime/code-runtime-python/tsconfig.json +++ b/packages/code-runtime/code-runtime-python/tsconfig.json @@ -14,8 +14,20 @@ { "path": "../../../vendor/cordis" }, + { + "path": "../../../vendor/schemastery" + }, + { + "path": "../code-runtime" + }, + { + "path": "../../core/session" + }, { "path": "../../runtime-diagnostics/invariants" + }, + { + "path": "../../util/timeout" } ] } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 326595dfff..1db20cbea3 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -3954,6 +3954,10 @@ importers: version: link:../../runtime-diagnostics/invariants packages/code-runtime/code-runtime-python: + dependencies: + '@deepseek-ai/schemastery': + specifier: link:../../../vendor/schemastery + version: link:../../../vendor/schemastery devDependencies: '@deepseek-ai/cordis': specifier: workspace:^ @@ -3961,6 +3965,12 @@ importers: '@deepseek-ai/dsh-invariants': specifier: workspace:^ version: link:../../runtime-diagnostics/invariants + '@deepseek-ai/dsh-session': + specifier: workspace:^ + version: link:../../core/session + '@deepseek-ai/dsh-timeout': + specifier: workspace:^ + version: link:../../util/timeout packages/code-runtime/code-runtime-worker-thread: dependencies: @@ -10331,6 +10341,9 @@ importers: '@deepseek-ai/dsh-code-runtime': specifier: workspace:^ version: link:../../packages/code-runtime/code-runtime + '@deepseek-ai/dsh-code-runtime-python': + specifier: workspace:^ + version: link:../../packages/code-runtime/code-runtime-python '@deepseek-ai/dsh-code-runtime-worker-thread': specifier: workspace:^ version: link:../../packages/code-runtime/code-runtime-worker-thread diff --git a/python/sdk-runtime/package.json b/python/sdk-runtime/package.json index 38184d656e..e20548309f 100644 --- a/python/sdk-runtime/package.json +++ b/python/sdk-runtime/package.json @@ -23,6 +23,7 @@ "@deepseek-ai/dsh-bash-local": "workspace:^", "@deepseek-ai/dsh-brand": "workspace:^", "@deepseek-ai/dsh-code-runtime": "workspace:^", + "@deepseek-ai/dsh-code-runtime-python": "workspace:^", "@deepseek-ai/dsh-code-runtime-worker-thread": "workspace:^", "@deepseek-ai/dsh-command-compact": "workspace:^", "@deepseek-ai/dsh-command-goal": "workspace:^", diff --git a/scripts/build-exe-for-python-sdk.ts b/scripts/build-exe-for-python-sdk.ts index c7c8cfed66..5977a4906a 100644 --- a/scripts/build-exe-for-python-sdk.ts +++ b/scripts/build-exe-for-python-sdk.ts @@ -38,7 +38,10 @@ const DEPLOY_ONLY_DOCS = ['README.md', 'README.zh.md', 'README.i18n.yaml'] /** * Whole-tree assets cover Cordis's runtime bare-package imports, which pkg's * static analysis cannot see. Package manifests are explicit because bare-name - * resolution depends on them. + * resolution depends on them. `*.py` carries the CPython code-runtime backend's + * bootstrap and protocol scripts into the executable; the backend copies them + * out to a real filesystem path before spawning, since the interpreter is an + * external process that cannot read pkg's virtual filesystem. */ const ASSET_GLOBS = [ 'package.json', @@ -55,6 +58,7 @@ const ASSET_GLOBS = [ 'node_modules/**/*.so', 'node_modules/**/*.so.*', 'node_modules/**/*.wasm', + 'node_modules/**/*.py', 'node_modules/**/*.yaml', 'node_modules/**/*.yml', // web-app builds this path dynamically, so pkg cannot discover the static frontend. diff --git a/vitest.config.ts b/vitest.config.ts index ecfd00ded4..7a05447605 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -31,6 +31,7 @@ const windowsUnsupportedPackages = process.platform === 'win32' 'packages/shell/tool-bash', 'packages/hooks/*', 'packages/terminal/terminal-bash', + 'packages/code-runtime/code-runtime-python', 'packages/sandbox/sandbox-local', ] : []