mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-09-11 04:00:38 +00:00
refactor(code-runtime-python): move the package into packages/experimental
The CPython code runtime's complete public contract is experimental, so it moves to packages/experimental per the experimental-packages rules: npm name @deepseek-ai/dsh-experimental-code-runtime-python, private: true, no publishConfig. All references updated (code-runtime READMEs, config-catalog and module-graph regenerated with zh alignment, tsconfig paths, doc-standard and workspace-constraints scripts, the fd-3 and settlement Agent Notes, and the package README links); md-links and translation pairing pass, and the suite still runs green.
This commit is contained in:
Chinesezjc
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/experimental/code-runtime-python/README.md
|
||||
README.md: 2fc78c0a8066886114600e4f3bab6a56521563e5
|
||||
README.zh.md: c2ec5ca81e5b8a7ee50658663484981699aa9687
|
||||
@@ -0,0 +1,133 @@
|
||||
---
|
||||
description: "CPython-subprocess code runtime: the dsh-code-runtime seam implementation for Python model code, with the fd-3 wire protocol it speaks."
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-code-runtime-python
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
`dsh-experimental-code-runtime-python` ships `PythonCodeRuntime`, the CPython-subprocess implementation of the [`dsh-code-runtime`](../../code-runtime/code-runtime/README.md) seam: it registers as `codeRuntime` with `language: 'python'` and `isolation: 'process'`, spawning a fresh `python3 -I` child per `run()` and executing the program as an async function body over a versionless JSON-lines protocol on the child's fd 3 (stdout/stderr stay free for the program's own output). The host side (`src/protocol.ts`) treats every inbound frame as hostile and rebuilds it before reading; the Python side (`py/protocol.py`) mirrors the message vocabulary. Containment — not a security boundary, model code has bash-equivalent trust — comes from an empty environment, `RLIMIT_CPU`/`RLIMIT_AS`, a wall-clock ceiling, and `SIGTERM`→grace→`SIGKILL` process-group teardown, with all caps validated at plugin load.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Use this package](#use-this-package)
|
||||
- [Understand the implementation](#understand-the-implementation)
|
||||
- [Further Exploration](#further-exploration)
|
||||
- [Model Experience](#model-experience)
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
- [Dev Note](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## Use this package
|
||||
|
||||
Choose this package to run Python model code through the code-runtime seam: register `PythonCodeRuntime` with `dsh-tools` and `run()` executes each program in a fresh `python3 -I` subprocess, resolving with `result.value` on success and `result.error` on failure (the orthogonal `CodeRunFailure.kind` taxonomy classifies parse failures, thrown exceptions, invalid completions, output overflows, budget expiry, aborts, and substrate death). It rejects only for seam misuse — a malformed binding namespace, or a call after disposal. Configuration is rejected at load: a non-Unix platform, a non-positive or non-integer budget, a `maxLogBytes` below the truncation-marker floor (64), a timer value `setTimeout` would clamp, a budget larger than one fd-3 frame can carry, and an `addressSpaceMb`/output-budget pair whose worst-case peak would breach `RLIMIT_AS`.
|
||||
|
||||
### What you get
|
||||
|
||||
The package's default export is the `PythonCodeRuntime` plugin. Its public surface also re-exports the host-side protocol vocabulary: `validateChildFrame` (rebuilds every inbound frame), the lossless-JSON codec and meters (`encodeJsonPlain`, `checkDoneValue`, `hasUnsafeIntegerToken`, `hasNonLosslessNumber`), `logTruncationMarker` (the shared truncation-marker text), plus `resolvePythonBin` (interpreter lookup against the current `PATH`), `readProcessStart` (process-start statistics for tests), and `detachResidual` (a test seam for the settled run's resource cleanup). Every cap is a validated `Config` field with a default: `cpuSeconds` (60), `maxWallMs` (600000), `addressSpaceMb` (512, not applied on Darwin), `maxLogBytes` (65536), `maxValueBytes` (32768), `graceMs` (3000), and `pythonBin` (`python3`, resolved against `PATH` before the child spawns with an empty environment; a basename with no `PATH` match is rejected at load rather than silently falling to the platform default `PATH`).
|
||||
|
||||
### The wire
|
||||
|
||||
Frames travel on the child's fd 3 as JSON-lines — one object per line — so stdout/stderr stay clear for the program's own output. Child → host: `boot-ack`, `call`, `log`, `done`. Host → child: `boot` (first frame, carrying every cap and the namespace declarations), `run` (after `boot-ack`, carrying only the program body), and one `reply` per `call`. A forged frame can carry both `value` and `error` on `done`, so a consumer must check `error` first and ignore `value` when it is set. A `log` frame's `open` flag marks an unterminated line committed by an explicit flush: the host appends the next log frame to the same entry, so `print('a', end='', flush=True); print('b')` reads back as one `'ab'` entry rather than a fake newline (the split-billing arithmetic lives in the fd-3 protocol Agent Note's wire-contract section). The one exception to merging is truncation: when a later over-budget frame trips the ledger, the already-billed prefix is committed as its own entry and the truncation marker follows it (the marker stays last, with no re-charge).
|
||||
|
||||
### What can go wrong
|
||||
|
||||
Host-side validation drops junk without throwing, so a malformed or forged frame never crashes the host process: `validateChildFrame` returns `undefined` for anything that does not rebuild cleanly, a non-number call id can never be echoed into a reply, and forged extra fields never ride along. A completion value that is not lossless JSON, or that exceeds the configured byte budget, is rejected explicitly (`non-lossless` / `over-budget`) rather than silently rounded or truncated. An fd-3 frame whose raw length exceeds 64 MiB settles the run as a `worker-exit` (the receive path caps raw frames before `toString`/`JSON.parse` so a compact wide frame cannot decode to far more host memory than its wire bytes admitted).
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## Understand the implementation
|
||||
|
||||
<details>
|
||||
<summary>Implementation internals — click to expand</summary>
|
||||
|
||||
This section explains the design behind the backend; observable behavior is fully covered in [Use this package](#use-this-package).
|
||||
|
||||
### Design concept
|
||||
|
||||
One direction of trust: the host treats every inbound frame as hostile (model code can forge anything on fd 3) and REBUILDS it field by field before reading; the Python side trusts host replies. The bootstrap (`py/bootstrap.py`) runs the program as the body of an async function, so top-level `await` and `return` work; binding calls travel over fd 3 as JSON-lines and replies are paced across the pump so a flood of large replies cannot pin the host's fd-3 write buffer.
|
||||
|
||||
### Wire contract
|
||||
|
||||
The frames are `boot` / `run` (host → child) and `boot-ack` / `call` / `log` / `done` plus one `reply` per call (child → host). The `log` frame's `truncated` flag marks the frame that IS the child ledger's truncation marker, so the host stops capturing at the same point the child did instead of inferring it from its own budget. The `log` frame's `open` flag marks an unterminated line committed by an explicit flush: the host merges the next log frame into the same entry, so `print('a', end='', flush=True); print('b')` reads back as one `'ab'` entry rather than a fake newline (the split-billing arithmetic lives in the fd-3 protocol Agent Note's wire-contract section). The one exception to merging is truncation: the already-billed prefix is committed as its own entry and the truncation marker follows it (marker last, no re-charge). `done.error.kind` is one of `exception`, `invalid-output`, `output-limit`; wall/CPU budgets, aborts, and substrate death are observed host-side, not carried as frames.
|
||||
|
||||
### Lossless JSON crossing
|
||||
|
||||
Completion values and binding arguments cross as exact JSON: values serialize without recursion, so a deep payload below the byte budget survives instead of dying on `JSON.stringify`'s stack limit, and integral doubles beyond the safe range cross as exact digits rather than silently rounded tokens; the meters in `src/protocol.ts` enforce byte budgets and number losslessness before anything else reads the payload.
|
||||
|
||||
### Mirror alignment
|
||||
|
||||
`tests/protocol-mirror.e2e.ts` spawns a real `python3` and asserts, against `src/protocol.ts`, both `PROTOCOL_FD` / the truncation-marker text and each `TypedDict`'s required/optional wire field set in `py/protocol.py`, so a renamed or dropped field — or one side making a field optional the other requires — fails the test. Field *types* are not compared across the language boundary; that residue stays with review plus the backend's real-subprocess suite (`tests/runtime.spec.ts`).
|
||||
|
||||
### Source map
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| [`src/index.ts`](src/index.ts) | Plugin entry: `PythonCodeRuntime` — spawn, frame pump, budgets, containment, teardown; re-exports the protocol vocabulary |
|
||||
| [`src/protocol.ts`](src/protocol.ts) | Host side: frame codec, hostile-frame validators, lossless-JSON meters, shared marker text |
|
||||
| [`py/bootstrap.py`](py/bootstrap.py) | Child side: fd-3 channel, program execution, binding dispatch, ledger and settlement |
|
||||
| [`py/protocol.py`](py/protocol.py) | Python side: `PROTOCOL_FD`, `TypedDict` frame mirrors, `log_truncation_marker` |
|
||||
| [`tests/runtime.spec.ts`](tests/runtime.spec.ts) | Real-subprocess suite: budgets, containment, hostile frames, name rebinding |
|
||||
| [`tests/protocol-mirror.e2e.ts`](tests/protocol-mirror.e2e.ts) | Cross-language mirror test against a real `python3` |
|
||||
| [`src/invariant.ts`](src/invariant.ts) | Invariant companion (no runtime invariant; the package registers no mutable data relation) |
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## Further Exploration
|
||||
|
||||
Read these when the runtime contract is not enough. They move from the seam definition to the design record and the companion backend.
|
||||
|
||||
- [Code runtime seam](../../code-runtime/code-runtime/README.md) — the abstract contract this backend implements.
|
||||
- [fd-3 protocol Agent Note](../../../.agents/notes/implemented/architecture/2026-07-31-code-runtime-python-fd3-protocol.md) — design rationale and wire contract.
|
||||
- [Settlement-fixes Agent Note](../../../.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.md) — settlement, metering, and containment fixes and their regression cases.
|
||||
- [Worker-thread backend](../../code-runtime/code-runtime-worker-thread/README.md) — the shipped TypeScript sibling.
|
||||
- [Code runtime subsystem reference](../../../docs/subsystems/code-runtime.md) — request/result vocabulary, bindings, and failure taxonomy.
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## Model Experience
|
||||
|
||||
Indirectly, through Code Mode in `dsh-tools`, which renders the program's completion value or failure into a retained `run_code` result.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct invalidation; the named consumer owns any request-prefix changes.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
|
||||
These limits define what the package does and does not cover; they are current package constraints, not a task backlog.
|
||||
|
||||
- **The cross-language guard covers the executed surfaces and the frame field shapes, not the field types** — the mirror e2e compares required/optional field sets, not that `cpuSeconds` is an `int` on both sides; a type-level drift is caught by review plus the backend's real-subprocess suite.
|
||||
- **A descendant that escapes the child's process group with `setsid()` is not reaped by the group teardown** — `kill(-pid)` cannot reach it; the run still settles on the value the done frame decided, and the close-deadline backstop forces settlement if the orphan holds the pipes open, but the orphan itself outlives the fiber until it exits on its own.
|
||||
- **A `log` frame that arrives after settlement is dropped** — once the run has settled, host-side capture is closed; a late fd-3 `log` frame (from a thread that outlived the done frame) is discarded rather than appended to `logs`.
|
||||
- **A binding REPLY value has no seam-level byte or depth cap** — `maxValueBytes` meters only the done frame's completion value; a wide binding reply is rebuilt host-side (`snapshotJsonValue` traversal) and encoded whole, bounded on both sides only by process memory (like a binding argument, which has no child-side budget either).
|
||||
- **A real-Loader assembly snapshot is deferred to issue #1182 layer 5** — this package is exercised through `ctx.plugin(...)` and real-subprocess tests; the full dsh application composition (codeRuntime registered through a real Loader) is covered by a tracked assembly test in that layer, not by this package's suite.
|
||||
- **`run()` is one-shot** — `logs` become available only after `CodeRunResult` resolves; there is no streaming-log or progress interface for output produced by a running program.
|
||||
- **No state persists across runs** — every request executes in a fresh subprocess; a persistent REPL-style kernel stays deferred until a backend brings its own logging scheme.
|
||||
- **An fd-3 frame whose raw length exceeds 64 MiB settles the run as a worker-exit** — `maxLogBytes`/`maxValueBytes` are load-bounded to the same parser cap so an honest child's frames always fit; a model-constructed binding ARGUMENT above 64 MiB (a value with no seam-level budget) trips the same cap — an accepted residual of the OOM guard.
|
||||
- **A combined log-and-value peak is not modelled by the load gate** — a model daemon thread that keeps writing while the completion value is metered and framed can add the two peaks in a way no gate admits or rejects; the run dies as `worker-exit`, containment holds, and only the failure classification is degraded.
|
||||
- **A 1-second dual-limit `ulimit -t 1` CPU overrun is reported as `worker-exit`, not a timeout** — when the host starts under a hard CPU limit equal to the soft and that limit is 1, `_clamped` cannot lower the soft, so the kernel SIGKILLs the busy loop and SIGXCPU is never delivered; containment holds, only the classification is degraded.
|
||||
- **No byte cap on intermediate binding values** — the implementation remains bounded by the lossless-JSON serialization cost and process memory, and a provider or executor may apply its own fetch cap.
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### Dev Note
|
||||
|
||||
<details>
|
||||
<summary>Working context for maintainers — click to expand</summary>
|
||||
|
||||
None.
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,133 @@
|
||||
---
|
||||
description: "CPython 子进程代码 runtime：为 Python 模型代码实现 dsh-code-runtime seam，及其使用的 fd-3 wire 协议。"
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-code-runtime-python
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
## 概述
|
||||
|
||||
`dsh-experimental-code-runtime-python` 交付 `PythonCodeRuntime`——[`dsh-code-runtime`](../../code-runtime/code-runtime/README.zh.md) seam 的 CPython 子进程实现：它以 `language: 'python'`、`isolation: 'process'` 注册为 `codeRuntime`，每次 `run()` 启动一个全新的 `python3 -I` 子进程，把程序作为 async 函数体执行，通过子进程 fd 3 上的无版本 JSON-lines 协议通信（stdout/stderr 留给程序自己的输出）。宿主侧（`src/protocol.ts`）把每条入站帧都视为敌意并逐字段重建后才读取；Python 侧（`py/protocol.py`）镜像消息词汇。隔离（不是安全边界——模型代码与 bash 同等的信任）来自空环境、`RLIMIT_CPU`/`RLIMIT_AS`、墙钟上限与 `SIGTERM`→宽限→`SIGKILL` 进程组拆卸，所有上限都在插件加载期校验。
|
||||
|
||||
## 目录
|
||||
|
||||
- [使用本包](#use-this-package)
|
||||
- [理解实现](#understand-the-implementation)
|
||||
- [进一步探索](#further-exploration)
|
||||
- [模型体验](#model-experience)
|
||||
- [已知限制与延期工作](#known-limitations-and-deferred-work)
|
||||
- [开发备注](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## 使用本包
|
||||
|
||||
在需要通过 code-runtime seam 运行 Python 模型代码时选择本包：向 `dsh-tools` 注册 `PythonCodeRuntime`，`run()` 就在全新的 `python3 -I` 子进程中执行每个程序，成功时以 `result.value` resolve、失败时以 `result.error` resolve（正交的 `CodeRunFailure.kind` 分类涵盖解析失败、抛出异常、无效完成值、输出溢出、预算到期、中止与执行基底终止）；只有 seam 误用才 reject——绑定命名空间畸形，或已释放后仍调用。配置在加载期被拒绝：非 Unix 平台、非正或非整数的预算、低于截断标记下限（64）的 `maxLogBytes`、`setTimeout` 会收敛的定时器值、超过单个 fd-3 帧可承载的预算，以及最坏峰值会突破 `RLIMIT_AS` 的 `addressSpaceMb`/输出预算组合。
|
||||
|
||||
### 你得到什么
|
||||
|
||||
包的默认导出是 `PythonCodeRuntime` 插件。其公开面还重新导出宿主侧协议词汇：`validateChildFrame`（重建每条入站帧）、无损 JSON codec 与计量器（`encodeJsonPlain`、`checkDoneValue`、`hasUnsafeIntegerToken`、`hasNonLosslessNumber`）、`logTruncationMarker`（共享截断标记文本），以及 `resolvePythonBin`（对照当前 `PATH` 的解释器查找）、`readProcessStart`（供测试用的进程启动统计）和 `detachResidual`（已结算运行的资源清理测试 seam）。每个上限都是带默认值并经校验的 `Config` 字段：`cpuSeconds`（60）、`maxWallMs`（600000）、`addressSpaceMb`（512，Darwin 上不生效）、`maxLogBytes`（65536）、`maxValueBytes`（32768）、`graceMs`（3000）与 `pythonBin`（`python3`，在子进程以空环境启动前对照 `PATH` 解析；在 `PATH` 上无命中的裸名会在加载期被拒绝，而不是静默回退到平台默认 `PATH`）。
|
||||
|
||||
### wire
|
||||
|
||||
帧在子进程 fd 3 上以 JSON-lines 传输——每行一个对象——因此 stdout/stderr 留给程序自己的输出。子进程 → 宿主：`boot-ack`、`call`、`log`、`done`。宿主 → 子进程：`boot`（首帧，携带全部上限与命名空间声明）、`run`（`boot-ack` 之后，只携带程序体）与每个 `call` 一个 `reply`。伪造帧可在 `done` 上同时携带 `value` 与 `error`，因此消费方必须先检查 `error`，在它存在时忽略 `value`。`log` 帧的 `open` 标志标记由显式 flush 提交的未结束行：宿主把下一个 log 帧追加到同一条目，因此 `print('a', end='', flush=True); print('b')` 读回为一条 `'ab'` 条目而不是假换行。合并的唯一例外是截断：当后续超预算帧触发账本时，已计费的前缀作为独立条目先提交，截断 marker 跟在后面（marker 保持末位，无重复计费）。
|
||||
|
||||
### 可能出错的地方
|
||||
|
||||
宿主侧校验在不抛异常的情况下丢弃垃圾，因此畸形或伪造帧永远不会让宿主进程崩溃：`validateChildFrame` 对任何不能干净重建的内容返回 `undefined`，非数字的 call id 永远不会被回显进 reply，伪造的额外字段永远不会被带走。非无损 JSON 或超过配置字节预算的完成值会被显式拒绝（`non-lossless`／`over-budget`），而不是被静默取整或截断。原始长度超过 64 MiB 的 fd-3 帧会让本次运行以 `worker-exit` 结算（接收路径在 `toString`/`JSON.parse` 之前限制原始帧，紧凑宽帧不能解码出远超其线上字节的宿主内存）。
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## 理解实现
|
||||
|
||||
<details>
|
||||
<summary>实现内部——点击展开</summary>
|
||||
|
||||
本节解释后端背后的设计；可观察行为在[使用本包](#use-this-package)中完整覆盖。
|
||||
|
||||
### 设计概念
|
||||
|
||||
单向信任：宿主把每条入站帧都视为敌意（模型代码可以在 fd 3 上伪造任何内容）并逐字段重建后才读取；Python 侧信任宿主回复。bootstrap（`py/bootstrap.py`）把程序作为 async 函数体执行，因此顶层 `await` 与 `return` 都可用；binding 调用经 fd 3 以 JSON-lines 往返，回复在 pump 中限速，以免大量大回复钉住宿主的 fd-3 可写缓冲。
|
||||
|
||||
### wire 契约
|
||||
|
||||
帧为 `boot`／`run`（宿主 → 子进程）与 `boot-ack`／`call`／`log`／`done` 加每个 call 一个 `reply`（子进程 → 宿主）。`log` 帧的 `truncated` 标志标记的就是子进程账本自己的截断标记帧，因此宿主在与子进程相同的点停止捕获，而不是从自己的预算推断。`log` 帧的 `open` 标志标记由显式 flush 提交的未结束行：宿主把下一个 log 帧合并进同一条目，因此 `print('a', end='', flush=True); print('b')` 读回为一条 `'ab'` 条目而不是假换行（拆分计费算术在 fd-3 协议 Agent Note 的 wire-contract 段）。合并的唯一例外是截断：当后续超预算帧触发账本时，已计费的前缀作为独立条目先提交，截断 marker 跟在后面（marker 保持末位，无重复计费）。`done.error.kind` 为 `exception`、`invalid-output`、`output-limit` 之一；墙钟／CPU 预算、中止与基底死亡在宿主侧观察，不以帧形式携带。
|
||||
|
||||
### 无损 JSON 跨越
|
||||
|
||||
完成值与 binding 实参以精确 JSON 跨越：值无递归序列化，因此低于字节预算的深层载荷存活，而不会死在 `JSON.stringify` 的栈上限；超出安全范围的整型 double 以精确数字跨越，而不是被静默取整的 token；`src/protocol.ts` 中的计量器在任何其他代码读取载荷之前强制字节预算与数字无损性。
|
||||
|
||||
### 镜像对齐
|
||||
|
||||
`tests/protocol-mirror.e2e.ts` 启动真实 `python3`，对照 `src/protocol.ts` 断言 `PROTOCOL_FD`／截断标记文本以及 `py/protocol.py` 中每个 `TypedDict` 的必填／可选 wire 字段集，因此字段改名、删除或一侧把另一侧必填的字段变成可选都会使测试失败。字段*类型*不跨语言边界比较；该残留由评审加后端的真实子进程套件（`tests/runtime.spec.ts`）负责。
|
||||
|
||||
### 源码地图
|
||||
|
||||
| 文件 | 职责 |
|
||||
|---|---|
|
||||
| [`src/index.ts`](src/index.ts) | 插件入口：`PythonCodeRuntime`——spawn、帧 pump、预算、隔离、拆卸；重新导出协议词汇 |
|
||||
| [`src/protocol.ts`](src/protocol.ts) | 宿主侧：帧 codec、敌意帧校验器、无损 JSON 计量器、共享标记文本 |
|
||||
| [`py/bootstrap.py`](py/bootstrap.py) | 子进程侧：fd-3 通道、程序执行、binding 分发、账本与结算 |
|
||||
| [`py/protocol.py`](py/protocol.py) | Python 侧：`PROTOCOL_FD`、`TypedDict` 帧镜像、`log_truncation_marker` |
|
||||
| [`tests/runtime.spec.ts`](tests/runtime.spec.ts) | 真实子进程套件：预算、隔离、敌意帧、名称重绑 |
|
||||
| [`tests/protocol-mirror.e2e.ts`](tests/protocol-mirror.e2e.ts) | 对照真实 `python3` 的跨语言镜像测试 |
|
||||
| [`src/invariant.ts`](src/invariant.ts) | 不变式伴生（无运行时不变式；本包不注册可变数据关系） |
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## 进一步探索
|
||||
|
||||
当 runtime 契约不够时阅读这些。它们从 seam 定义走向设计记录与配套后端。
|
||||
|
||||
- [Code runtime seam](../../code-runtime/code-runtime/README.zh.md) — 本后端实现的抽象契约。
|
||||
- [fd-3 协议 Agent Note](../../../.agents/notes/implemented/architecture/2026-07-31-code-runtime-python-fd3-protocol.zh.md) — 设计理由与 wire 契约。
|
||||
- [结算修复 Agent Note](../../../.agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.zh.md) — 结算、计量与隔离修复及其回归用例。
|
||||
- [Worker 线程后端](../../code-runtime/code-runtime-worker-thread/README.zh.md) — 已发布的 TypeScript 兄弟。
|
||||
- [Code runtime 子系统参考](../../../docs/subsystems/code-runtime.zh.md) — 请求／结果词汇、binding 与失败分类。
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## 模型体验
|
||||
|
||||
间接地，通过 `dsh-tools` 中的 Code Mode，它把程序的完成值或失败渲染成保留的 `run_code` 结果。
|
||||
|
||||
#### KV Cache 效应
|
||||
|
||||
无直接失效；指定的消费方拥有任何请求前缀变化。
|
||||
|
||||
## 已知限制与延期工作
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
|
||||
这些限制定义本包覆盖与不覆盖的内容；它们是当前包约束，不是任务积压。
|
||||
|
||||
- **跨语言 guard 覆盖执行的表面与帧字段形状，而非字段类型**——mirror e2e 比较必填／可选字段集，而非 `cpuSeconds` 在两侧是否都是 `int`；类型级漂移由评审加后端的真实子进程套件捕获。
|
||||
- **以 `setsid()` 逃出子进程组后代不被组拆卸回收**——`kill(-pid)` 够不到它；运行仍按 done 帧决定的值结算，若该孤儿持有管道，close 截止兜底会强制结算，但孤儿本身在自行退出前一直存活到 fiber 之外。
|
||||
- **结算后到达的 `log` 帧被丢弃**——运行一旦结算，宿主侧捕获即关闭；迟到的 fd-3 `log` 帧（来自比 done 帧存活更久的线程）会被丢弃，而不是追加到 `logs`。
|
||||
- **binding 回复值没有 seam 级字节或深度上限**——`maxValueBytes` 只计量 done 帧的完成值；宽 binding 回复在宿主侧重建（`snapshotJsonValue` 遍历）并整帧编码，两侧都只受进程内存约束（与没有子进程侧预算的 binding 实参一样）。
|
||||
- **`run()` 是一次性的**——`logs` 只有在 `CodeRunResult` resolve 后才能获得；没有为运行中程序产生的输出提供流式日志或进度接口。
|
||||
- **运行之间不保留状态**——每次请求都在全新子进程中执行；持久 REPL 风格内核在某个后端带来自己的日志方案之前保持延期。
|
||||
- **原始长度超过 64 MiB 的 fd-3 帧会让本次运行以 worker-exit 结算**——`maxLogBytes`/`maxValueBytes` 在加载期被限制到同一解析器上限，因此诚实子进程的帧总能放得下；模型构造的超过 64 MiB 的 binding 实参（一个在 seam 层没有预算的值）会触发同一上限——这是该 OOM 防护的已接受残余。
|
||||
- **组合日志与值的峰值不被加载门建模**——持续写入的模型 daemon 线程与完成值计量、分帧相加的峰值没有任何门会放行或拒绝；运行以 `worker-exit` 告终，隔离成立，只有失败分类降级。
|
||||
- **1 秒双限 `ulimit -t 1` CPU 超限被报告为 `worker-exit` 而非 timeout**——当宿主在一个与软限相等的硬 CPU 限下启动且该限为 1 时，`_clamped` 无法下调软限，内核在同一 tick SIGKILL 忙循环，SIGXCPU 永远不会送达；隔离成立，只有分类降级。
|
||||
- **中间 binding 值没有字节上限**——实现仍受无损 JSON 序列化成本与进程内存约束，提供方或执行器可能应用自己的获取上限。
|
||||
- **真实 Loader 装配态快照推迟到 issue #1182 layer 5**——本包通过 `ctx.plugin(...)` 与真实子进程测试得到验证；完整的 dsh 应用组合（codeRuntime 经真实 Loader 注册）由该层一个受跟踪的装配测试覆盖，不由本包的测试套件承担。
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### 开发备注
|
||||
|
||||
<details>
|
||||
<summary>维护者的工作上下文——点击展开</summary>
|
||||
|
||||
无。
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-experimental-code-runtime-python",
|
||||
"description": "CPython subprocess implementation of the DeepSeek Harness code-execution seam",
|
||||
"version": "0.1.2-alpha.2",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
||||
"directory": "packages/experimental/code-runtime-python"
|
||||
},
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"py/**/*.py",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-code-runtime": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"@deepseek-ai/cordis": "workspace:^"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-code-runtime": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"@deepseek-ai/cordis": "workspace:^"
|
||||
},
|
||||
"dependencies": {
|
||||
"@deepseek-ai/schemastery": "workspace:^"
|
||||
},
|
||||
"private": true
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,139 @@
|
||||
"""Wire protocol vocabulary for the Python side of dsh-code-runtime-python.
|
||||
|
||||
Mirrors ``src/protocol.ts``. Frames travel on fd 3 as JSON-lines (one JSON
|
||||
object per line). The host validates every inbound frame; this side trusts
|
||||
host replies.
|
||||
|
||||
The wire uses the JSON key ``global`` (a Python keyword), so the frame
|
||||
``TypedDict``s that carry it are declared with the functional syntax rather than
|
||||
class bodies: a class attribute cannot be named ``global``, and a ``global_``
|
||||
attribute would describe a key the wire never sends. Optional-field messages
|
||||
pair a required base with a ``total=False`` subclass so a required field such as
|
||||
``type`` cannot be dropped while ``value``/``error``/``truncated`` stay optional.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Literal, TypedDict, Union
|
||||
|
||||
# The protocol fd from the child's perspective. Node passes
|
||||
# ``stdio: [pipe, pipe, pipe, pipe]`` so the fourth entry (fd 3) is the
|
||||
# framed-JSON channel; stdout/stderr stay clear for the program's own output.
|
||||
PROTOCOL_FD = 3
|
||||
|
||||
|
||||
class ErrorClass(TypedDict):
|
||||
"""A namespace's program-visible exception class: rejected calls raise its
|
||||
instances carrying the failed member name on ``memberNameProperty``."""
|
||||
|
||||
name: str
|
||||
memberNameProperty: str
|
||||
|
||||
|
||||
# ``global`` is a Python keyword, so the required part is declared functionally
|
||||
# to hold the real wire key; ``errorClass`` is optional per the TS `errorClass?`.
|
||||
_NamespaceRequired = TypedDict("_NamespaceRequired", {"global": str, "names": "list[str]"})
|
||||
|
||||
|
||||
class Namespace(_NamespaceRequired, total=False):
|
||||
"""One binding namespace declaration: the ``global`` name, its function
|
||||
``names``, and an optional program-visible ``errorClass`` for rejected calls."""
|
||||
|
||||
errorClass: ErrorClass
|
||||
|
||||
|
||||
class BootMessage(TypedDict):
|
||||
"""Host → child, first frame on fd 3. Carries every cap and the namespaces."""
|
||||
|
||||
type: Literal["boot"]
|
||||
cpuSeconds: int
|
||||
addressSpaceBytes: int
|
||||
maxLogBytes: int
|
||||
maxValueBytes: int
|
||||
namespaces: "list[Namespace]"
|
||||
|
||||
|
||||
class RunMessage(TypedDict):
|
||||
"""Host → child, sent after ``boot-ack``. Carries only the program body."""
|
||||
|
||||
type: Literal["run"]
|
||||
program: str
|
||||
|
||||
|
||||
class BootAckMessage(TypedDict):
|
||||
"""Child → host: resource limits applied, ready for the run message."""
|
||||
|
||||
type: Literal["boot-ack"]
|
||||
|
||||
|
||||
# ``global`` wire key: whole message declared functionally, all fields required.
|
||||
CallMessage = TypedDict(
|
||||
"CallMessage",
|
||||
{"type": Literal["call"], "id": int, "global": str, "name": str, "args": Any},
|
||||
)
|
||||
|
||||
|
||||
_LogMessageRequired = TypedDict("_LogMessageRequired", {"type": Literal["log"], "text": str})
|
||||
|
||||
|
||||
class LogMessage(_LogMessageRequired, total=False):
|
||||
"""Child → host: one captured text chunk, streamed eagerly.
|
||||
|
||||
``truncated`` is set only on the frame that IS the child ledger's truncation
|
||||
marker (not program output), so the host stops capturing at the same point
|
||||
the child did — mirrors the TS `truncated?`. ``open`` is set on a flushed unterminated line the host appends the next frame to (mirrors `open?`).
|
||||
"""
|
||||
|
||||
truncated: bool
|
||||
open: bool
|
||||
|
||||
|
||||
class DoneErrorField(TypedDict):
|
||||
"""Child → host: the failure carried on a ``done`` frame. ``kind`` is one of
|
||||
the three the host validates; ``message`` is the traceback or diagnostic."""
|
||||
|
||||
kind: Literal["exception", "invalid-output", "output-limit"]
|
||||
message: str
|
||||
|
||||
|
||||
_DoneMessageRequired = TypedDict("_DoneMessageRequired", {"type": Literal["done"]})
|
||||
|
||||
|
||||
class DoneMessage(_DoneMessageRequired, total=False):
|
||||
"""Child → host: the program settled. ``value`` and ``error`` are optional per the TS mirror."""
|
||||
|
||||
value: Any
|
||||
error: DoneErrorField
|
||||
|
||||
|
||||
ChildToHost = Union[BootAckMessage, CallMessage, LogMessage, DoneMessage]
|
||||
|
||||
|
||||
class ReplyOk(TypedDict):
|
||||
type: Literal["reply"]
|
||||
id: int
|
||||
ok: Literal[True]
|
||||
value: Any
|
||||
|
||||
|
||||
class ReplyErr(TypedDict):
|
||||
type: Literal["reply"]
|
||||
id: int
|
||||
ok: Literal[False]
|
||||
message: str
|
||||
|
||||
|
||||
ReplyMessage = Union[ReplyOk, ReplyErr]
|
||||
# The host sends ``boot`` and ``run`` before any ``reply``, so the child-facing
|
||||
# inbound union covers all three, not replies alone.
|
||||
HostToChild = Union[BootMessage, RunMessage, ReplyMessage]
|
||||
|
||||
|
||||
def log_truncation_marker(max_bytes: int) -> str:
|
||||
"""Return the in-band marker for a log ledger that exhausted its budget.
|
||||
|
||||
Byte-identical text on both sides of the wire so a truncated run reads the
|
||||
same however the cap was hit.
|
||||
"""
|
||||
|
||||
return f"[dsh-code-runtime-python] log capture truncated at {max_bytes} bytes"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,33 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-code-runtime-python`.
|
||||
* @module @deepseek-ai/dsh-code-runtime-python/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-code-runtime-python'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'code-runtime-python-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: every relation this backend maintains — frame ordering, budget accounting,
|
||||
* and process teardown — lives in the CPython subprocess or on the fd-3 wire, so no same-process
|
||||
* event sequence or mutable data relation is observable from a Cordis listener. `protocol.spec.ts`,
|
||||
* `protocol-mirror.e2e.ts`, and the real-subprocess `runtime.spec.ts` cover that behavior, matching
|
||||
* the sibling process-boundary backend `@deepseek-ai/dsh-code-runtime-worker-thread`.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
@@ -0,0 +1,668 @@
|
||||
/**
|
||||
* Versionless, JSON-lines wire protocol between the Node host and the CPython subprocess. Frames
|
||||
* travel on the child's fd 3 (one JSON object per line), leaving stdout/stderr free for the
|
||||
* program's own output. Host treats every inbound frame as hostile because model code can post
|
||||
* anything through the same fd; the Python bootstrap trusts host replies.
|
||||
* @module @deepseek-ai/dsh-code-runtime-python/src/protocol
|
||||
*/
|
||||
|
||||
/**
|
||||
* The framed-JSON channel's file descriptor from the child's perspective. The
|
||||
* host pins it positionally when it spawns the child (`stdio` index 3, i.e.
|
||||
* `['pipe','pipe','pipe','pipe']`), and the Python bootstrap reads the same
|
||||
* number from its own `protocol.py`. Exported as the single TS-side source of
|
||||
* truth: the host wiring uses it, and the cross-language mirror test asserts the
|
||||
* Python constant equals it, so a drift on either side breaks the boot channel
|
||||
* loudly rather than silently.
|
||||
*/
|
||||
export const PROTOCOL_FD = 3
|
||||
|
||||
/**
|
||||
* One binding namespace declaration inside a {@link BootMessage}. `global` is
|
||||
* the program-visible name the namespace is materialized under; `errorClass`,
|
||||
* when present, asks the bootstrap to mint a program-visible exception class.
|
||||
*/
|
||||
interface Namespace {
|
||||
global: string
|
||||
names: string[]
|
||||
errorClass?: ErrorClass
|
||||
}
|
||||
|
||||
/**
|
||||
* A namespace's program-visible exception class: rejected calls raise its
|
||||
* instances carrying the failed member name on `memberNameProperty`.
|
||||
*/
|
||||
interface ErrorClass {
|
||||
name: string
|
||||
memberNameProperty: string
|
||||
}
|
||||
|
||||
/**
|
||||
* What the host sends immediately after spawn, as the first line on fd 3. The
|
||||
* Python bootstrap reads this, applies resource limits, then waits for the
|
||||
* subsequent run frame. Separated from the run so the run message stays
|
||||
* pure model input.
|
||||
*/
|
||||
export interface BootMessage {
|
||||
type: 'boot'
|
||||
/** RLIMIT_CPU seconds; the Python bootstrap sets this on itself before executing model code. */
|
||||
cpuSeconds: number
|
||||
/** RLIMIT_AS bytes; caps address space so a runaway allocation fails cleanly. */
|
||||
addressSpaceBytes: number
|
||||
/** Shared byte budget for captured log text (Python-side ledger). */
|
||||
maxLogBytes: number
|
||||
/** Byte cap for the rendered completion value. */
|
||||
maxValueBytes: number
|
||||
/**
|
||||
* The namespaces to materialize inside the program (globals + names;
|
||||
* functions stay host-side). See {@link Namespace}.
|
||||
*/
|
||||
namespaces: Namespace[]
|
||||
}
|
||||
|
||||
/** Host → Python: sent after `boot-ack`; carries only the model's program body. */
|
||||
interface RunMessage {
|
||||
type: 'run'
|
||||
program: string
|
||||
}
|
||||
|
||||
/** Python → host: acknowledges boot completed and resource limits are in place. */
|
||||
interface BootAckMessage {
|
||||
type: 'boot-ack'
|
||||
}
|
||||
|
||||
/** Python → host: one bridged binding call (`await tools.name(args)` inside the program). */
|
||||
interface CallMessage {
|
||||
type: 'call'
|
||||
/** Python-issued correlation id; the host answers each id at most once and ignores duplicates. */
|
||||
id: number
|
||||
/** The namespace global the call targets. */
|
||||
global: string
|
||||
/** The function name within the namespace. */
|
||||
name: string
|
||||
/** The JSON-safe argument the model program passed. */
|
||||
args: unknown
|
||||
}
|
||||
|
||||
/**
|
||||
* Python → host: captured text, streamed eagerly so output survives a
|
||||
* mid-run termination (RLIMIT_CPU, SIGTERM/SIGKILL, host wall-timeout).
|
||||
*/
|
||||
interface LogMessage {
|
||||
type: 'log'
|
||||
text: string
|
||||
/**
|
||||
* Set when this frame IS the child ledger's truncation marker rather than
|
||||
* program output. The two ledgers can exhaust at different points — one
|
||||
* child entry larger than `maxLogBytes` sends only the marker while the host
|
||||
* ledger is still nearly empty — so the host cannot infer the child's state
|
||||
* from its own budget, and comparing the text against the marker string
|
||||
* would also honour a program that printed that string itself. Carrying it
|
||||
* as a field lets the host stop capturing at the same point the child did
|
||||
* and keeps exactly one marker in `logs`.
|
||||
*/
|
||||
truncated?: boolean
|
||||
/**
|
||||
* Set on the frame an explicit `flush()` (or the settlement flush) pushes for
|
||||
* an UNTERMINATED line: the host holds it and appends the next log frame to
|
||||
* the same entry, so `print('a', end='', flush=True); print('b')` reads back
|
||||
* as one `'ab'` entry rather than a fake newline between two entries.
|
||||
*/
|
||||
open?: boolean
|
||||
}
|
||||
|
||||
/** The failure carried on a {@link DoneMessage}: one of three kinds plus text. */
|
||||
interface DoneErrorField {
|
||||
kind: 'exception' | 'invalid-output' | 'output-limit'
|
||||
message: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Python → host: the program settled. `error` carries a program exception
|
||||
* (traceback text), an `invalid-output` (completion value was not lossless
|
||||
* JSON), or an `output-limit` (serialized completion exceeded the configured
|
||||
* cap); wall/CPU budgets, aborts, and substrate death are observed host-side.
|
||||
* From the honest child `value` is present only on a clean completion that
|
||||
* produced one, and crosses as exact lossless JSON — never substituted or
|
||||
* truncated. A forged frame CAN carry both `value` and `error`;
|
||||
* {@link validateChildFrame} preserves both rather than guessing which to drop,
|
||||
* so a consumer MUST check `error` first and ignore `value` when it is set.
|
||||
*/
|
||||
interface DoneMessage {
|
||||
type: 'done'
|
||||
value?: unknown
|
||||
error?: DoneErrorField
|
||||
}
|
||||
|
||||
/**
|
||||
* Every message the Python side sends. The member interfaces stay module-
|
||||
* private: consumers match on the union's discriminant; the host sends the
|
||||
* boot and run frames as inline literals.
|
||||
*/
|
||||
export type ChildToHost = BootAckMessage | CallMessage | LogMessage | DoneMessage
|
||||
|
||||
/** Host → Python: successful answer to one {@link CallMessage}. */
|
||||
interface ReplyOk {
|
||||
type: 'reply'
|
||||
id: number
|
||||
ok: true
|
||||
value: unknown
|
||||
}
|
||||
|
||||
/** Host → Python: failed answer to one {@link CallMessage}. */
|
||||
interface ReplyErr {
|
||||
type: 'reply'
|
||||
id: number
|
||||
ok: false
|
||||
message: string
|
||||
}
|
||||
|
||||
/** Host → Python: the answer to one {@link CallMessage}. */
|
||||
export type ReplyMessage = ReplyOk | ReplyErr
|
||||
|
||||
/** The required (non-optional) keys of `T`, as string literals. */
|
||||
type RequiredKeys<T> = { [K in keyof T]-?: object extends Pick<T, K> ? never : K }[keyof T] & string
|
||||
/** The optional keys of `T`, as string literals. */
|
||||
type OptionalKeys<T> = { [K in keyof T]-?: object extends Pick<T, K> ? K : never }[keyof T] & string
|
||||
|
||||
/**
|
||||
* Whether each key of frame `T` is a `'required'` or `'optional'` wire field.
|
||||
* Because it is `Record<keyof T, …>`, an entry MUST list every key — a field
|
||||
* added to the interface without a corresponding entry fails typecheck — and
|
||||
* `keyof T`-typed keys reject a name no frame declares. The `'required'` /
|
||||
* `'optional'` tag must match the field's actual optionality (checked by the
|
||||
* `satisfies FrameFieldRoles<…>` clause on {@link WIRE_FRAME_FIELD_ROLES}), so
|
||||
* an optionality flip is caught too. This is the exhaustive counterpart the
|
||||
* array form could not express (a subset array satisfied it silently).
|
||||
*/
|
||||
type FrameFieldRoles<T> = Record<RequiredKeys<T>, 'required'> & Record<OptionalKeys<T>, 'optional'>
|
||||
|
||||
interface WireFrameShapes {
|
||||
BootMessage: BootMessage
|
||||
Namespace: Namespace
|
||||
RunMessage: RunMessage
|
||||
BootAckMessage: BootAckMessage
|
||||
CallMessage: CallMessage
|
||||
LogMessage: LogMessage
|
||||
DoneErrorField: DoneErrorField
|
||||
DoneMessage: DoneMessage
|
||||
ErrorClass: ErrorClass
|
||||
ReplyOk: ReplyOk
|
||||
ReplyErr: ReplyErr
|
||||
}
|
||||
|
||||
/**
|
||||
* The frames carried on a message union: everything the host and child send as
|
||||
* a top-level frame (`ChildToHost`, the two reply variants, and the host→child
|
||||
* boot/run frames). The nested shapes `Namespace`, `ErrorClass`, and
|
||||
* `DoneErrorField` are fields of other frames, not frames themselves, so they
|
||||
* are excluded here and covered only by the roles `satisfies` and the mirror e2e.
|
||||
*/
|
||||
type MessageFrames = ChildToHost | ReplyMessage | BootMessage | RunMessage
|
||||
/** The roster's value types minus the three nested (non-frame) shapes. */
|
||||
type RosterMessageFrames = Exclude<WireFrameShapes[keyof WireFrameShapes], Namespace | ErrorClass | DoneErrorField>
|
||||
|
||||
/**
|
||||
* Compile-time proof that {@link WireFrameShapes}'s message-frame entries are
|
||||
* EXACTLY the frames on the message unions — checked BOTH directions. Forward
|
||||
* (`MessageFrames extends RosterMessageFrames`) catches a frame added to a union
|
||||
* without a roster entry; reverse (`RosterMessageFrames extends MessageFrames`)
|
||||
* catches a frame removed from a union while the roster still lists it (e.g.
|
||||
* dropping `ReplyErr` from `ReplyMessage`). Either divergence makes an alias
|
||||
* `false`, failing the assignment below. Type-only; the `const`s emit nothing
|
||||
* meaningful at runtime.
|
||||
*/
|
||||
type UnionSubsetOfRoster = [MessageFrames] extends [RosterMessageFrames] ? true : false
|
||||
type RosterSubsetOfUnion = [RosterMessageFrames] extends [MessageFrames] ? true : false
|
||||
const _unionSubsetOfRoster: UnionSubsetOfRoster = true
|
||||
const _rosterSubsetOfUnion: RosterSubsetOfUnion = true
|
||||
void _unionSubsetOfRoster
|
||||
void _rosterSubsetOfUnion
|
||||
|
||||
/**
|
||||
* Each frame's wire fields tagged by required/optional, keyed by field name so
|
||||
* the mapping is exhaustive over the frame interface (see {@link FrameFieldRoles})
|
||||
* across the whole {@link WireFrameShapes} roster. Bound to the interfaces by
|
||||
* `satisfies` below; {@link WIRE_FRAME_FIELDS} projects it to sorted
|
||||
* required/optional arrays for the cross-language mirror comparison. `global` is
|
||||
* the JSON key {@link CallMessage} and {@link Namespace} send (a reserved word
|
||||
* the Python side carries via a functional `TypedDict`).
|
||||
*/
|
||||
const WIRE_FRAME_FIELD_ROLES = {
|
||||
BootMessage: { type: 'required', cpuSeconds: 'required', addressSpaceBytes: 'required', maxLogBytes: 'required', maxValueBytes: 'required', namespaces: 'required' },
|
||||
Namespace: { global: 'required', names: 'required', errorClass: 'optional' },
|
||||
RunMessage: { type: 'required', program: 'required' },
|
||||
BootAckMessage: { type: 'required' },
|
||||
CallMessage: { type: 'required', id: 'required', global: 'required', name: 'required', args: 'required' },
|
||||
LogMessage: { type: 'required', text: 'required', truncated: 'optional', open: 'optional' },
|
||||
DoneErrorField: { kind: 'required', message: 'required' },
|
||||
DoneMessage: { type: 'required', value: 'optional', error: 'optional' },
|
||||
ErrorClass: { name: 'required', memberNameProperty: 'required' },
|
||||
ReplyOk: { type: 'required', id: 'required', ok: 'required', value: 'required' },
|
||||
ReplyErr: { type: 'required', id: 'required', ok: 'required', message: 'required' },
|
||||
} as const satisfies { [K in keyof WireFrameShapes]: FrameFieldRoles<WireFrameShapes[K]> }
|
||||
|
||||
/**
|
||||
* The wire field names of each frame, split into sorted required and optional
|
||||
* key arrays — the shape the cross-language mirror test compares against
|
||||
* `py/protocol.py`'s `TypedDict` `__required_keys__`/`__optional_keys__`.
|
||||
* Projected from {@link WIRE_FRAME_FIELD_ROLES}, so it inherits that mapping's
|
||||
* exhaustive, optionality-checked binding to the frame interfaces: a TS-side
|
||||
* field add, remove, rename, or optionality flip fails typecheck at the roles
|
||||
* map, and a Python-side divergence fails the mirror test at runtime.
|
||||
*/
|
||||
export const WIRE_FRAME_FIELDS =
|
||||
Object.fromEntries(
|
||||
Object.entries(WIRE_FRAME_FIELD_ROLES).map(([frame, roles]) => {
|
||||
const required = Object.keys(roles).filter(key => (roles as Record<string, string>)[key] === 'required').sort()
|
||||
const optional = Object.keys(roles).filter(key => (roles as Record<string, string>)[key] === 'optional').sort()
|
||||
return [frame, { required, optional }]
|
||||
}),
|
||||
) as Record<keyof typeof WIRE_FRAME_FIELD_ROLES, { required: string[]; optional: string[] }>
|
||||
|
||||
|
||||
/**
|
||||
* The in-band marker text announcing that log capture stopped at the byte
|
||||
* budget. Shared wire vocabulary: the Python-side LogBuffer emits it when ITS
|
||||
* ledger exhausts, and the host emits identical text when its own ledger drops
|
||||
* a frame first (forged fd-3 traffic, stray stdout bytes) — a truncated run
|
||||
* reads the same however the cap was hit.
|
||||
* @param maxBytes - the configured `maxLogBytes` the marker names.
|
||||
* @returns the marker line.
|
||||
*/
|
||||
export function logTruncationMarker(maxBytes: number): string {
|
||||
return `[dsh-code-runtime-python] log capture truncated at ${maxBytes} bytes`
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize one JSON-parse-produced value without recursion. `JSON.stringify`
|
||||
* recurses per nesting level and throws `RangeError` a few thousand levels
|
||||
* deep, but the seam's `CodeJsonValue` has no depth limit — an honest deep
|
||||
* completion or binding resolution below the byte budget must cross intact
|
||||
* (the worker backend's wire is equally stack-safe). Callers must pass a value
|
||||
* produced by `JSON.parse` (or equally JSON-plain): only `null`, finite
|
||||
* numbers, booleans, strings, dense arrays, and plain objects — this encoder
|
||||
* validates nothing. Output matches compact `JSON.stringify` byte for byte
|
||||
* EXCEPT on an integral double beyond the safe range, where {@link scalarJson}
|
||||
* emits the exact integer's BigInt digits rather than `JSON.stringify`'s rounded
|
||||
* spelling (`1152921504606846976`, not `...847000`) so the seam's lossless-JSON
|
||||
* promise holds across the wire.
|
||||
* @param value - a JSON-plain value (e.g. straight from `JSON.parse`).
|
||||
* @returns the compact JSON encoding.
|
||||
*/
|
||||
export function encodeJsonPlain(value: unknown): string {
|
||||
type Task = { text: string } | { value: unknown }
|
||||
const chunks: string[] = []
|
||||
const tasks: Task[] = [{ value }]
|
||||
for (let task = tasks.pop(); task !== undefined; task = tasks.pop()) {
|
||||
if ('text' in task) {
|
||||
chunks.push(task.text)
|
||||
continue
|
||||
}
|
||||
const current = task.value
|
||||
if (typeof current === 'string') {
|
||||
chunks.push(JSON.stringify(current))
|
||||
} else if (Array.isArray(current)) {
|
||||
chunks.push('[')
|
||||
tasks.push({ text: ']' })
|
||||
for (let index = current.length - 1; index >= 0; index--) {
|
||||
if (index < current.length - 1) tasks.push({ text: ',' })
|
||||
tasks.push({ value: current[index] })
|
||||
}
|
||||
} else if (typeof current === 'object' && current !== null) {
|
||||
const record = current as Record<string, unknown>
|
||||
chunks.push('{')
|
||||
tasks.push({ text: '}' })
|
||||
const keys = Object.keys(record)
|
||||
for (let index = keys.length - 1; index >= 0; index--) {
|
||||
const key = keys[index] as string
|
||||
if (index < keys.length - 1) tasks.push({ text: ',' })
|
||||
tasks.push({ value: record[key] })
|
||||
tasks.push({ text: `${JSON.stringify(key)}:` })
|
||||
}
|
||||
} else {
|
||||
chunks.push(scalarJson(current))
|
||||
}
|
||||
}
|
||||
return chunks.join('')
|
||||
}
|
||||
|
||||
/**
|
||||
* One scalar (null, boolean, finite number) as JSON text. A beyond-safe-range
|
||||
* integral double needs BigInt digits: `String(2 ** 60)` emits the ROUNDED
|
||||
* `...847000` form, and echoing that to the child would silently change the
|
||||
* integer the seam promised to carry losslessly — `BigInt(2 ** 60)` prints the
|
||||
* exact `...846976` the double actually holds.
|
||||
* @param current - a JSON-plain scalar (JSON.parse emits nothing else).
|
||||
* @returns its JSON encoding.
|
||||
*/
|
||||
function scalarJson(current: unknown): string {
|
||||
if (typeof current === 'number' && Number.isInteger(current) && !Number.isSafeInteger(current)) {
|
||||
return BigInt(current).toString()
|
||||
}
|
||||
return String(current)
|
||||
}
|
||||
|
||||
/**
|
||||
* Exact UTF-8 byte length of one string's compact JSON form (quotes + escapes),
|
||||
* computed by a single non-allocating scan that stops the instant the running
|
||||
* total exceeds `maxBytes`. Used instead of `Buffer.byteLength(JSON.stringify(s))`
|
||||
* so a control-heavy forged string — whose escaped copy expands up to ~6x — is
|
||||
* rejected BEFORE that copy is materialized: `JSON.stringify` would allocate the
|
||||
* full escaped form first, the very hundreds-of-MB spike the metered traversal
|
||||
* exists to avoid. Mirrors `JSON.stringify`'s escaping byte-for-byte: `"` and
|
||||
* `\` and the five short C0 escapes cost 2, other C0 controls `\uXXXX` cost 6, a
|
||||
* valid surrogate pair is one astral code point emitted as raw 4-byte UTF-8, a
|
||||
* LONE surrogate becomes `\uXXXX` at 6, and any other code point costs its raw
|
||||
* UTF-8 width.
|
||||
* @param text - the string to meter.
|
||||
* @param maxBytes - largest serialized size the caller can still admit.
|
||||
* @returns the exact serialized byte length, or `undefined` once it exceeds `maxBytes`.
|
||||
*/
|
||||
function jsonStringBytesUpTo(text: string, maxBytes: number): number | undefined {
|
||||
let bytes = 2 // the two quotes
|
||||
if (bytes > maxBytes) return undefined
|
||||
for (let index = 0; index < text.length; index++) {
|
||||
const code = text.charCodeAt(index)
|
||||
if (code === 0x22 || code === 0x5c || code === 0x08 || code === 0x09 || code === 0x0a || code === 0x0c || code === 0x0d) {
|
||||
bytes += 2 // `\"` `\\` `\b` `\t` `\n` `\f` `\r`
|
||||
} else if (code < 0x20) {
|
||||
bytes += 6 // other C0 controls: `\uXXXX`
|
||||
} else if (code < 0x80) {
|
||||
bytes += 1
|
||||
} else if (code < 0x800) {
|
||||
bytes += 2
|
||||
} else if (code >= 0xd800 && code <= 0xdbff && index + 1 < text.length) {
|
||||
const next = text.charCodeAt(index + 1)
|
||||
if (next >= 0xdc00 && next <= 0xdfff) {
|
||||
bytes += 4 // valid high+low pair: one astral code point, raw 4-byte UTF-8
|
||||
index++
|
||||
} else {
|
||||
bytes += 6 // lone high surrogate: `\uXXXX`
|
||||
}
|
||||
} else if (code >= 0xd800 && code <= 0xdfff) {
|
||||
bytes += 6 // lone surrogate (unpaired high at end, or any low): `\uXXXX`
|
||||
} else {
|
||||
bytes += 3 // other BMP code point
|
||||
}
|
||||
if (bytes > maxBytes) return undefined
|
||||
}
|
||||
return bytes
|
||||
}
|
||||
|
||||
/**
|
||||
* Meter a `JSON.parse`-produced done value's compact-JSON byte length AND its
|
||||
* number losslessness in one traversal, stopping the instant `maxBytes` is
|
||||
* crossed. This bounds the INCREMENTAL allocation the check itself would add on
|
||||
* top of the already-parsed value — the enqueued children; strings and keys are
|
||||
* metered by {@link jsonStringBytesUpTo} without allocating an escaped copy —
|
||||
* not the parse that produced `value`.
|
||||
* That upstream width is bounded separately, by the host-side cap on inbound
|
||||
* fd-3 frame size before `JSON.parse` runs (owned by the runtime that reads the
|
||||
* channel), so `value` cannot be arbitrarily large when it reaches here. The
|
||||
* budget is the `maxValueBytes` the boot frame carries — a required wire field
|
||||
* with no default at this layer. The traversal rejects over-budget BEFORE
|
||||
* materializing a string's escaped form or enqueuing an array's/object's
|
||||
* children, so a forgery within that frame cap cannot force those secondary
|
||||
* allocations. Object key COUNTING is
|
||||
* unavoidably O(keys) — JS has no lazy own-key iterator, and the parse already
|
||||
* built the key set — but the check still refuses the per-entry work before the
|
||||
* enqueue loop. A non-lossless number (non-finite, negative zero) is caught only
|
||||
* when the value fits the budget — an over-budget value is rejected regardless,
|
||||
* so the distinction is moot. Same JSON-plain precondition and traversal shape
|
||||
* as {@link encodeJsonPlain}; a number's byte length is measured through
|
||||
* {@link scalarJson} (matching the encoder, so a beyond-safe-range integer
|
||||
* meters its exact BigInt digits, not `JSON.stringify`'s rounded spelling) and
|
||||
* a string's/key's through {@link jsonStringBytesUpTo} (the exact escaped size,
|
||||
* scanned without allocating the escaped copy).
|
||||
* @param value - a JSON-plain value (e.g. straight from `JSON.parse`).
|
||||
* @param maxBytes - the completion-value budget in bytes.
|
||||
* @returns `{ ok: true, bytes }` with the exact serialized size, or
|
||||
* `{ ok: false, reason }` — `over-budget` once the size exceeds `maxBytes`,
|
||||
* `non-lossless` on a non-finite or negative-zero number.
|
||||
*/
|
||||
export function checkDoneValue(value: unknown, maxBytes: number): { ok: true; bytes: number } | { ok: false; reason: 'over-budget' | 'non-lossless' } {
|
||||
let bytes = 0
|
||||
// A non-lossless number is recorded, not returned on sight: over-budget must
|
||||
// win regardless of where in the value each violation sits, so the whole
|
||||
// metering finishes first. Otherwise `["<huge>", 1e400]` and `[1e400,
|
||||
// "<huge>"]` — the same over-budget value in two member orders — would
|
||||
// classify differently (non-lossless vs over-budget), and the JSDoc promises
|
||||
// an over-budget value is rejected as over-budget regardless.
|
||||
let nonLossless = false
|
||||
const stack: unknown[] = [value]
|
||||
while (stack.length > 0) {
|
||||
const current = stack.pop()
|
||||
if (typeof current === 'number') {
|
||||
// Flag a non-lossless number but keep counting its encoded bytes: a value
|
||||
// that is BOTH non-lossless and over-budget must classify as over-budget
|
||||
// (the loop's byte check below wins), so the byte count cannot skip the
|
||||
// offending number. `scalarJson` gives the same spelling a legit scalar
|
||||
// would meter.
|
||||
if (!Number.isFinite(current) || Object.is(current, -0)) nonLossless = true
|
||||
bytes += Buffer.byteLength(scalarJson(current), 'utf8')
|
||||
} else if (typeof current === 'string') {
|
||||
// Meter the escaped form WITHOUT allocating it: jsonStringBytesUpTo scans
|
||||
// and bails the instant the running cost crosses the remaining budget, so
|
||||
// a control-heavy forgery (escaped copy up to ~6x) never materializes that
|
||||
// copy the way `JSON.stringify` would.
|
||||
const stringBytes = jsonStringBytesUpTo(current, maxBytes - bytes)
|
||||
if (stringBytes === undefined) return { ok: false, reason: 'over-budget' }
|
||||
bytes += stringBytes
|
||||
} else if (Array.isArray(current)) {
|
||||
// Brackets plus one comma per gap; elements add themselves. Reject
|
||||
// BEFORE enqueuing children: every element serializes to at least one
|
||||
// byte, so a forged flat array far above the budget fails here without
|
||||
// pushing its elements onto the host stack. (The array itself is already
|
||||
// materialized by the upstream parse; this only bounds the extra stack.)
|
||||
bytes += 2 + (current.length > 1 ? current.length - 1 : 0)
|
||||
if (bytes + current.length > maxBytes) return { ok: false, reason: 'over-budget' }
|
||||
for (const item of current) stack.push(item)
|
||||
} else if (typeof current === 'object' && current !== null) {
|
||||
const record = current as Record<string, unknown>
|
||||
// Count own keys with for...in + hasOwn. This IS O(keys) — JS has no lazy
|
||||
// own-key iterator and the parse already built the key set — so the count
|
||||
// cannot be sublinear; what the bound below buys is refusing the per-entry
|
||||
// work (key escaping, value enqueue) before it runs. Each entry costs at
|
||||
// least a quoted key (>= 2 bytes) + colon + >= 1-byte value.
|
||||
let count = 0
|
||||
for (const key in record) if (Object.hasOwn(record, key)) count += 1
|
||||
bytes += 2 + (count > 1 ? count - 1 : 0)
|
||||
if (bytes + count * 4 > maxBytes) return { ok: false, reason: 'over-budget' }
|
||||
for (const key in record) {
|
||||
if (!Object.hasOwn(record, key)) continue
|
||||
// Meter the key's escaped form without allocating it (same reason as the
|
||||
// string branch), then add the colon separator. `+ 1` for the `:`.
|
||||
const keyBytes = jsonStringBytesUpTo(key, maxBytes - bytes)
|
||||
if (keyBytes === undefined) return { ok: false, reason: 'over-budget' }
|
||||
bytes += keyBytes + 1
|
||||
stack.push(record[key])
|
||||
}
|
||||
} else {
|
||||
bytes += Buffer.byteLength(scalarJson(current), 'utf8')
|
||||
}
|
||||
if (bytes > maxBytes) return { ok: false, reason: 'over-budget' }
|
||||
}
|
||||
// The whole value fit the budget; a recorded number violation is the verdict.
|
||||
if (nonLossless) return { ok: false, reason: 'non-lossless' }
|
||||
return { ok: true, bytes }
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a raw JSON line contains an integer token that would lose precision
|
||||
* as a JavaScript number. `JSON.parse` silently rounds such a token
|
||||
* (`9007199254740993` becomes `...992`) BEFORE any validation can see it, so
|
||||
* the check must read the source text; a beyond-safe-range token whose double
|
||||
* parse round-trips exactly (`2**53`, `2**60`) is lossless and passes. The scan walks the line skipping string literals (a digit run
|
||||
* inside a string is data, not a number token) and tests every number token
|
||||
* in plain integer form — no fraction or exponent, which parse as doubles by
|
||||
* intent. A reviver cannot do this job: the reviver walk recurses per nesting
|
||||
* level and would reintroduce the depth limit `encodeJsonPlain` removes.
|
||||
* @param line - the raw UTF-8 text of one JSON-lines frame.
|
||||
* @returns true when an unsafe integer token is present outside strings.
|
||||
*/
|
||||
export function hasUnsafeIntegerToken(line: string): boolean {
|
||||
for (let index = 0; index < line.length; index++) {
|
||||
const char = line[index]
|
||||
if (char === '"') {
|
||||
// Skip the string literal, honoring backslash escapes.
|
||||
for (index++; index < line.length; index++) {
|
||||
if (line[index] === '\\') index++
|
||||
else if (line[index] === '"') break
|
||||
}
|
||||
continue
|
||||
}
|
||||
if (char === '-' || (char !== undefined && char >= '0' && char <= '9')) {
|
||||
let end = index + 1
|
||||
while (end < line.length) {
|
||||
const c = line[end] as string
|
||||
if ((c >= '0' && c <= '9') || c === '.' || c === 'e' || c === 'E' || c === '+' || c === '-') end++
|
||||
else break
|
||||
}
|
||||
const token = line.slice(index, end)
|
||||
// Beyond the safe range an integer token is still lossless IFF the
|
||||
// double parse round-trips exactly (2**53 does; 2**53+1 rounds) — the
|
||||
// canonical boundary accepts every JS-double-exact value, so only a
|
||||
// genuinely rounding token marks the frame as forged.
|
||||
if (/^-?\d+$/.test(token)) {
|
||||
const parsed = Number(token)
|
||||
// A token that parses to Infinity is trivially lossy; a finite
|
||||
// beyond-safe-range one is lossy only when the BigInt round-trip
|
||||
// disagrees.
|
||||
if (!Number.isFinite(parsed)) return true
|
||||
if (!Number.isSafeInteger(parsed) && BigInt(token) !== BigInt(parsed)) return true
|
||||
}
|
||||
index = end - 1
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Lazily yield one plain object's own enumerable property values. A generator
|
||||
* (not `Object.values`/`Object.entries`) because {@link hasNonLosslessNumber}
|
||||
* walks breadth it cannot bound: those helpers copy the whole VALUE (or
|
||||
* key/value pair) list into a fresh array up front, so a wide object would cost
|
||||
* that second full-breadth allocation before a single value is examined. The
|
||||
* `for...in` here does not make the walk sublinear — V8 still materializes the
|
||||
* key-name enumeration when the loop starts — but it avoids the extra value
|
||||
* array, yielding each value straight off the already-parsed object.
|
||||
* @param record - a JSON-parse-produced object.
|
||||
* @yields each own enumerable property value, in key order.
|
||||
*/
|
||||
function* ownValues(record: object): Generator {
|
||||
for (const key in record) {
|
||||
if (Object.hasOwn(record, key)) yield (record as Record<string, unknown>)[key]
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a JSON.parse-produced value contains a number outside lossless
|
||||
* JSON: non-finite (`1e400` parses to `Infinity`) or negative zero (`-0.0`
|
||||
* parses to JS `-0`, whose sign bit a re-serialization drops). The honest
|
||||
* child's validator rejects these before sending, so a frame carrying one is
|
||||
* forged.
|
||||
*
|
||||
* Runs on `call.args`, which — unlike a completion value — has NO seam byte
|
||||
* cap, so there is no budget to reject a wide payload against the way
|
||||
* {@link checkDoneValue} does. The traversal therefore holds ONE cursor per
|
||||
* NESTING LEVEL (an array or {@link ownValues} iterator) instead of one entry
|
||||
* per member: a forged flat `args` at the top of the host's inbound frame-size
|
||||
* cap would
|
||||
* otherwise push tens of millions of stack entries — and `Object.values` would
|
||||
* copy each object's full breadth — allocating hundreds of megabytes beyond
|
||||
* what `JSON.parse` already holds. Iterative either way, so a deep frame
|
||||
* cannot overflow the host stack.
|
||||
* @param value - a JSON-parse-produced value from an fd-3 frame.
|
||||
* @returns true when any contained number is non-finite or negative zero.
|
||||
*/
|
||||
export function hasNonLosslessNumber(value: unknown): boolean {
|
||||
const cursors: Iterator<unknown>[] = [[value].values()]
|
||||
while (cursors.length > 0) {
|
||||
// The loop condition guarantees a top cursor.
|
||||
const cursor = cursors.at(-1) as Iterator<unknown>
|
||||
const step = cursor.next()
|
||||
if (step.done === true) {
|
||||
cursors.pop()
|
||||
continue
|
||||
}
|
||||
const current = step.value
|
||||
if (typeof current === 'number') {
|
||||
if (!Number.isFinite(current) || Object.is(current, -0)) return true
|
||||
} else if (Array.isArray(current)) {
|
||||
cursors.push((current as unknown[]).values())
|
||||
} else if (typeof current === 'object' && current !== null) {
|
||||
cursors.push(ownValues(current))
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Runtime shape gate for inbound fd-3 traffic. Model code has full access to
|
||||
* fd 3 and can post anything — `null`, primitives, poisoned fields — so the
|
||||
* compile-time union means nothing here: every field is validated and REBUILT
|
||||
* before the host reads it (forged extras never ride along; a non-number id
|
||||
* can never be echoed into a reply). Junk returns `undefined` and is dropped
|
||||
* so a throw in the host's `message` handler cannot crash the host process.
|
||||
* @param raw - one JSON-parsed frame from fd 3.
|
||||
* @returns the rebuilt frame, or `undefined` to drop it silently.
|
||||
*/
|
||||
export function validateChildFrame(raw: unknown): ChildToHost | undefined {
|
||||
if (typeof raw !== 'object' || raw === null) return undefined
|
||||
const m = raw as Record<string, unknown>
|
||||
switch (m.type) {
|
||||
case 'boot-ack':
|
||||
return { type: 'boot-ack' }
|
||||
case 'log':
|
||||
if (typeof m.text !== 'string') return undefined
|
||||
// Rebuilt, not passed through: a forged `truncated` of any other type
|
||||
// would reach the host as a truthy value and silence capture for the
|
||||
// rest of the run. Only the literal `true` counts; `open` likewise.
|
||||
return {
|
||||
type: 'log',
|
||||
text: m.text,
|
||||
...m.truncated === true ? { truncated: true } : {},
|
||||
...m.open === true ? { open: true } : {},
|
||||
}
|
||||
case 'call': {
|
||||
// The id must be a finite number: it is echoed verbatim into the reply
|
||||
// frame, and a forged `1e400` id (Infinity after JSON.parse) would make
|
||||
// the reply unencodable as strict JSON. Negative zero is rejected too:
|
||||
// it passes `Number.isFinite`, but the reply re-serializes it as `0`
|
||||
// (`JSON.stringify({id:-0})` is `{"id":0}`), colliding with a real call
|
||||
// whose id is `0` — the honest child never issues `-0`.
|
||||
if (typeof m.id !== 'number' || !Number.isFinite(m.id) || Object.is(m.id, -0) || typeof m.global !== 'string' || typeof m.name !== 'string') return undefined
|
||||
// A forged frame can omit `args` entirely; rebuilding it as `undefined`
|
||||
// would invoke the binding with a non-JSON value, bypassing the
|
||||
// lossless-JSON argument boundary. Any PRESENT value is JSON-plain by
|
||||
// construction (the frame came from JSON.parse), so presence is the
|
||||
// whole check.
|
||||
if (!Object.hasOwn(m, 'args')) return undefined
|
||||
// JSON.parse yields Infinity for 1e400 and preserves -0; both are
|
||||
// outside lossless JSON, and the honest child never sends them.
|
||||
if (hasNonLosslessNumber(m.args)) return undefined
|
||||
return { type: 'call', id: m.id, global: m.global, name: m.name, args: m.args }
|
||||
}
|
||||
case 'done': {
|
||||
// The value passes through untouched here: scanning it for non-lossless
|
||||
// numbers would push every member of a wide forged payload before any
|
||||
// byte cap runs. The done handler's bounded `checkDoneValue` folds the
|
||||
// losslessness check into the metered traversal, rejecting over-budget
|
||||
// before it enqueues children.
|
||||
const err = m.error
|
||||
if (err === undefined) {
|
||||
return m.value === undefined ? { type: 'done' } : { type: 'done', value: m.value }
|
||||
}
|
||||
if (typeof err !== 'object' || err === null) return undefined
|
||||
const { kind, message } = err as Record<string, unknown>
|
||||
if (typeof message !== 'string') return undefined
|
||||
if (kind !== 'exception' && kind !== 'invalid-output' && kind !== 'output-limit') return undefined
|
||||
return m.value === undefined
|
||||
? { type: 'done', error: { kind, message } }
|
||||
: { type: 'done', value: m.value, error: { kind, message } }
|
||||
}
|
||||
default:
|
||||
return undefined
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
import { EventEmitter } from 'node:events'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { dirname } from 'node:path'
|
||||
import { PassThrough } from 'node:stream'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
|
||||
/**
|
||||
* A synchronous `proto.write` throw on the fd-3 pipe is the one boot path a real
|
||||
* subprocess cannot be coerced into from a test: the pipe accepts queued bytes
|
||||
* until the kernel buffer fills, and a same-tick EPIPE needs fd 3 already closed
|
||||
* before the first write. `spawn` is mocked so fd 3 throws on the boot frame,
|
||||
* which is exactly the branch that regressed. The mock is confined to this file
|
||||
* so the real-subprocess suite in runtime.spec.ts is untouched.
|
||||
*/
|
||||
const { spawnMock } = vi.hoisted(() => ({ spawnMock: vi.fn() }))
|
||||
vi.mock('node:child_process', async importOriginal => ({
|
||||
...(await importOriginal<typeof import('node:child_process')>()),
|
||||
spawn: spawnMock,
|
||||
}))
|
||||
|
||||
const { PythonCodeRuntime } = await import('../src/index.ts')
|
||||
|
||||
/** A `child_process.ChildProcess` stand-in whose fd-3 pipe rejects every write. */
|
||||
function fakeChildWithThrowingFd3(): EventEmitter {
|
||||
const child = new EventEmitter() as EventEmitter & {
|
||||
pid?: number
|
||||
stdout: PassThrough
|
||||
stderr: PassThrough
|
||||
stdio: unknown[]
|
||||
}
|
||||
// Leave `pid` absent: `finish()` still runs its `clearTimeout(wallTimer)` /
|
||||
// `removeEventListener(onAbort)` prologue (the TDZ site) before short-
|
||||
// circuiting on `child.pid === undefined` to `settle` instead of waiting on a
|
||||
// `close` this fake never emits, so the run resolves promptly.
|
||||
child.stdout = new PassThrough()
|
||||
child.stderr = new PassThrough()
|
||||
// A duplex whose `write` throws synchronously, standing in for an fd-3 pipe
|
||||
// that fails the moment the boot frame is issued.
|
||||
const proto = new PassThrough()
|
||||
proto.write = () => { throw Object.assign(new Error('EPIPE: broken pipe, write'), { code: 'EPIPE' }) }
|
||||
child.stdio = [new PassThrough(), child.stdout, child.stderr, proto]
|
||||
return child
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
spawnMock.mockReset()
|
||||
})
|
||||
|
||||
/** A child whose fd-3 pipe accepts the boot write, then rejects the run write. */
|
||||
function fakeChildWithAckThenThrowingFd3(): EventEmitter {
|
||||
const child = new EventEmitter() as EventEmitter & {
|
||||
pid?: number
|
||||
stdout: PassThrough
|
||||
stderr: PassThrough
|
||||
stdio: unknown[]
|
||||
}
|
||||
child.stdout = new PassThrough()
|
||||
child.stderr = new PassThrough()
|
||||
const proto = new PassThrough()
|
||||
let writes = 0
|
||||
proto.write = () => {
|
||||
writes += 1
|
||||
if (writes === 1) return true // The boot frame goes out.
|
||||
throw Object.assign(new Error('EPIPE: broken pipe, write'), { code: 'EPIPE' })
|
||||
}
|
||||
child.stdio = [new PassThrough(), child.stdout, child.stderr, proto]
|
||||
// Emit the boot-ack after the boot write, so the run-frame write fires and
|
||||
// hits the throwing pipe.
|
||||
setImmediate(() => proto.emit('data', Buffer.from('{"type":"boot-ack"}\n')))
|
||||
return child
|
||||
}
|
||||
|
||||
describe('PythonCodeRuntime — boot-write failure', () => {
|
||||
it('resolves a worker-exit when the fd-3 boot write throws (no TDZ ReferenceError)', async () => {
|
||||
// Before the fix, the boot-write block ran BEFORE `wallTimer`, `onAbort`,
|
||||
// and `live` were initialized, so its `finish()` (which clears `wallTimer`,
|
||||
// removes `onAbort`, and — through `settle` — deletes `live`) hit the
|
||||
// temporal dead zone and threw a ReferenceError. That escaped the Promise
|
||||
// executor and REJECTED run() instead of resolving the worker-exit the catch
|
||||
// constructs. This test would see that rejection; the fix makes it resolve.
|
||||
spawnMock.mockImplementation(() => fakeChildWithThrowingFd3())
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(PythonCodeRuntime)
|
||||
const runtime = ctx.codeRuntime as InstanceType<typeof PythonCodeRuntime>
|
||||
|
||||
const result = await runtime.run({ program: 'return 1', bindings: [] })
|
||||
|
||||
expect(result.error?.kind).toBe('worker-exit')
|
||||
expect(result.error?.message).toContain('failed to boot python subprocess')
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('resolves a worker-exit and removes the staging dir when spawn throws synchronously', async () => {
|
||||
// `spawn` can throw same-tick — EMFILE on a descriptor-exhausted host, or a
|
||||
// libuv-level failure — before the Promise executor and its settlement path
|
||||
// exist. Left uncaught it rejected run() (the seam permits rejection only for
|
||||
// misuse) and stranded the staging directory materializePyScripts had just
|
||||
// written, which only settle() removes. The fix catches it, unlinks the
|
||||
// directory, and resolves the same `worker-exit` class as an async ENOENT.
|
||||
//
|
||||
// Capture THIS run's exact staging dir from the argv the mocked spawn
|
||||
// received (`['-I', <dir>/bootstrap.py]`) and assert only that path is gone.
|
||||
// A tmpdir scan — even a set difference against a pre-run snapshot — would
|
||||
// flake under vitest's forks pool: a sibling worker creating its own
|
||||
// `dsh-code-runtime-python-*` dir in the window reads as a leak here. Keying
|
||||
// off our own argv is fully isolated from concurrent staging.
|
||||
let stagedBootstrap: string | undefined
|
||||
spawnMock.mockImplementation((_bin: string, args: string[]) => {
|
||||
stagedBootstrap = args[args.length - 1]
|
||||
throw Object.assign(new Error('EMFILE: too many open files'), { code: 'EMFILE' })
|
||||
})
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(PythonCodeRuntime)
|
||||
const runtime = ctx.codeRuntime as InstanceType<typeof PythonCodeRuntime>
|
||||
|
||||
const result = await runtime.run({ program: 'return 1', bindings: [] })
|
||||
|
||||
expect(result.error?.kind).toBe('worker-exit')
|
||||
expect(result.error?.message).toContain('python spawn error')
|
||||
expect(stagedBootstrap).toBeDefined()
|
||||
expect(existsSync(dirname(stagedBootstrap as string))).toBe(false)
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('resolves a worker-exit when the run write after boot-ack throws', async () => {
|
||||
// The run frame goes out from the boot-ack handler; a pipe that accepts
|
||||
// the boot frame but rejects the run write must settle the run as a
|
||||
// worker-exit rather than reject run() or leave it hanging.
|
||||
spawnMock.mockImplementation(() => fakeChildWithAckThenThrowingFd3())
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(PythonCodeRuntime)
|
||||
const runtime = ctx.codeRuntime as InstanceType<typeof PythonCodeRuntime>
|
||||
|
||||
const result = await runtime.run({ program: 'return 1', bindings: [] })
|
||||
|
||||
expect(result.error?.kind).toBe('worker-exit')
|
||||
expect(result.error?.message).toContain('failed to boot python subprocess')
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,103 @@
|
||||
import { execFile } from 'node:child_process'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { promisify } from 'node:util'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { logTruncationMarker, PROTOCOL_FD, WIRE_FRAME_FIELDS } from '../src/protocol.ts'
|
||||
|
||||
/**
|
||||
* Cross-language mirror check between `src/protocol.ts` and `py/protocol.py`,
|
||||
* spawning a real `python3` to read the Python side. Two things are asserted:
|
||||
* the runtime surfaces both sides EXECUTE against — `PROTOCOL_FD` and the log
|
||||
* truncation marker text, where a drift silently corrupts a live run — and the
|
||||
* per-frame wire field sets (required/optional keys of each `TypedDict`), which
|
||||
* turns the otherwise review-only shape mirror into an executable check that
|
||||
* catches the round-12 kind of drift (a renamed/dropped field, or one side
|
||||
* making a field optional the other requires). Self-skips when no `python3` is
|
||||
* on PATH — CI provides one; the pure-TS `protocol.spec.ts` covers the host
|
||||
* codec unconditionally.
|
||||
*/
|
||||
|
||||
const execFileAsync = promisify(execFile)
|
||||
const pyDir = fileURLToPath(new URL('../py', import.meta.url))
|
||||
// `-B` blocks bytecode writes into the source tree (`py/__pycache__/*.pyc`);
|
||||
// `-I` isolates the interpreter but does not imply it.
|
||||
const python3Flags = ['-I', '-B']
|
||||
|
||||
async function hasPython3(): Promise<boolean> {
|
||||
try {
|
||||
await execFileAsync('python3', ['--version'])
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
const python3Available = await hasPython3()
|
||||
|
||||
describe.skipIf(!python3Available)('protocol.py mirrors protocol.ts at runtime', () => {
|
||||
it('agrees on PROTOCOL_FD and the log truncation marker across byte budgets', async () => {
|
||||
const budgets = [1, 65536, 1048576]
|
||||
const probe = [
|
||||
'import json, sys',
|
||||
`sys.path.insert(0, ${JSON.stringify(pyDir)})`,
|
||||
'from protocol import PROTOCOL_FD, log_truncation_marker',
|
||||
`budgets = ${JSON.stringify(budgets)}`,
|
||||
'print(json.dumps({',
|
||||
' "fd": PROTOCOL_FD,',
|
||||
' "markers": [log_truncation_marker(b) for b in budgets],',
|
||||
'}))',
|
||||
].join('\n')
|
||||
const { stdout } = await execFileAsync('python3', [...python3Flags, '-c', probe])
|
||||
const seen = JSON.parse(stdout) as { fd: number; markers: string[] }
|
||||
// Assert against the TS-side PROTOCOL_FD export (the value the host wires),
|
||||
// not a bare literal, so a drift on either side of the wire is caught here.
|
||||
expect(seen.fd).toBe(PROTOCOL_FD)
|
||||
expect(seen.markers).toEqual(budgets.map(budget => logTruncationMarker(budget)))
|
||||
})
|
||||
|
||||
it('agrees on every frame type\'s wire field set between the TS and Python declarations', async () => {
|
||||
// Turn the TypedDict mirror from a review-only obligation into an executable
|
||||
// check: enumerate EVERY TypedDict in py/protocol.py (public names carrying
|
||||
// __required_keys__) and assert both the frame roster and each frame's
|
||||
// required/optional key sets against WIRE_FRAME_FIELDS — projected from the
|
||||
// WIRE_FRAME_FIELD_ROLES map that `satisfies` binds exhaustively to the
|
||||
// frame interfaces in protocol.ts. Together this catches drift on EITHER
|
||||
// side of the wire: a TS-side field add, remove, rename, or optionality flip
|
||||
// fails typecheck at the roles map; a Python frame added, removed, or with a
|
||||
// changed field set fails this comparison. `global` is the reserved-keyword
|
||||
// wire key the Python side carries via a functional TypedDict.
|
||||
const probe = [
|
||||
'import json, sys',
|
||||
`sys.path.insert(0, ${JSON.stringify(pyDir)})`,
|
||||
'import protocol as p',
|
||||
'def keys(td): return {"required": sorted(td.__required_keys__), "optional": sorted(td.__optional_keys__)}',
|
||||
// Every public TypedDict in the module — not a name list from the TS side,
|
||||
// so a Python-only extra frame is visible here.
|
||||
'frames = {n: keys(v) for n, v in vars(p).items()'
|
||||
+ ' if not n.startswith("_") and hasattr(v, "__required_keys__")}',
|
||||
'print(json.dumps(frames))',
|
||||
].join('\n')
|
||||
const { stdout } = await execFileAsync('python3', [...python3Flags, '-c', probe])
|
||||
const seen = JSON.parse(stdout) as Record<string, { required: string[]; optional: string[] }>
|
||||
// Normalize the TS source of truth to the same sorted shape Python reports.
|
||||
const expected = Object.fromEntries(
|
||||
Object.entries(WIRE_FRAME_FIELDS).map(([name, sets]) => [
|
||||
name,
|
||||
{ required: [...sets.required].sort(), optional: [...sets.optional].sort() },
|
||||
]),
|
||||
)
|
||||
// Same frame roster on both sides (catches a frame present on only one),
|
||||
// then identical field sets per frame.
|
||||
expect(Object.keys(seen).sort()).toEqual(Object.keys(expected).sort())
|
||||
expect(seen).toEqual(expected)
|
||||
})
|
||||
})
|
||||
|
||||
it('names the py/ directory that ships with the package', () => {
|
||||
// Resolves py/ relative to this test file; the same directory ships in the
|
||||
// package.json `files` whitelist (`py/**/*.py`). The tests/ directory itself
|
||||
// is not published — this asserts the source-tree layout the mirror test
|
||||
// depends on, so it holds even when python3 is absent from the runner.
|
||||
expect(existsSync(pyDir)).toBe(true)
|
||||
})
|
||||
@@ -0,0 +1,304 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { checkDoneValue, encodeJsonPlain, hasNonLosslessNumber, hasUnsafeIntegerToken, logTruncationMarker, validateChildFrame } from '../src/index.ts'
|
||||
|
||||
describe('logTruncationMarker', () => {
|
||||
it('names the configured byte budget', () => {
|
||||
expect(logTruncationMarker(65536)).toBe('[dsh-code-runtime-python] log capture truncated at 65536 bytes')
|
||||
expect(logTruncationMarker(1)).toBe('[dsh-code-runtime-python] log capture truncated at 1 bytes')
|
||||
})
|
||||
})
|
||||
|
||||
describe('validateChildFrame', () => {
|
||||
it('rebuilds boot-ack frames without extra fields', () => {
|
||||
expect(validateChildFrame({ type: 'boot-ack' })).toEqual({ type: 'boot-ack' })
|
||||
// Forged extras never ride along.
|
||||
expect(validateChildFrame({ type: 'boot-ack', extra: 'x' })).toEqual({ type: 'boot-ack' })
|
||||
})
|
||||
|
||||
it('rebuilds log frames when the text field is a string', () => {
|
||||
expect(validateChildFrame({ type: 'log', text: 'hi' })).toEqual({ type: 'log', text: 'hi' })
|
||||
// Non-string text drops.
|
||||
expect(validateChildFrame({ type: 'log', text: 42 })).toBeUndefined()
|
||||
expect(validateChildFrame({ type: 'log' })).toBeUndefined()
|
||||
})
|
||||
|
||||
it('carries a log frame truncation flag only for the literal true', () => {
|
||||
// The child's own ledger marker sets `truncated: true`; the host rebuilds
|
||||
// it so it stops capturing at the same point.
|
||||
expect(validateChildFrame({ type: 'log', text: 'x', truncated: true }))
|
||||
.toEqual({ type: 'log', text: 'x', truncated: true })
|
||||
// Any other truthy or non-boolean value is a forgery and is dropped from
|
||||
// the rebuild — otherwise it would silence capture for the rest of the run.
|
||||
expect(validateChildFrame({ type: 'log', text: 'x', truncated: 1 })).toEqual({ type: 'log', text: 'x' })
|
||||
expect(validateChildFrame({ type: 'log', text: 'x', truncated: 'yes' })).toEqual({ type: 'log', text: 'x' })
|
||||
expect(validateChildFrame({ type: 'log', text: 'x', truncated: false })).toEqual({ type: 'log', text: 'x' })
|
||||
})
|
||||
|
||||
it('rebuilds call frames with a numeric id, string global, and string name', () => {
|
||||
expect(validateChildFrame({ type: 'call', id: 1, global: 'tools', name: 'echo', args: { x: 1 } }))
|
||||
.toEqual({ type: 'call', id: 1, global: 'tools', name: 'echo', args: { x: 1 } })
|
||||
// A frame with NO args key drops whole: rebuilding it as `undefined`
|
||||
// would invoke the binding with a non-JSON value, bypassing the
|
||||
// lossless-JSON argument boundary. Any present value is JSON-plain by
|
||||
// construction (frames arrive via JSON.parse), so null passes.
|
||||
expect(validateChildFrame({ type: 'call', id: 2, global: 'tools', name: 'echo' })).toBeUndefined()
|
||||
expect(validateChildFrame({ type: 'call', id: 2, global: 'tools', name: 'echo', args: null }))
|
||||
.toEqual({ type: 'call', id: 2, global: 'tools', name: 'echo', args: null })
|
||||
// A missing/mistyped required field drops.
|
||||
expect(validateChildFrame({ type: 'call', id: '1', global: 'tools', name: 'echo' })).toBeUndefined()
|
||||
expect(validateChildFrame({ type: 'call', id: 1, global: 7, name: 'echo' })).toBeUndefined()
|
||||
expect(validateChildFrame({ type: 'call', id: 1, global: 'tools' })).toBeUndefined()
|
||||
})
|
||||
|
||||
it('rebuilds done frames with optional value/error', () => {
|
||||
expect(validateChildFrame({ type: 'done' })).toEqual({ type: 'done' })
|
||||
expect(validateChildFrame({ type: 'done', value: 42 })).toEqual({ type: 'done', value: 42 })
|
||||
expect(validateChildFrame({ type: 'done', error: { kind: 'exception', message: 'boom' } }))
|
||||
.toEqual({ type: 'done', error: { kind: 'exception', message: 'boom' } })
|
||||
expect(validateChildFrame({ type: 'done', error: { kind: 'invalid-output', message: 'lossy' } }))
|
||||
.toEqual({ type: 'done', error: { kind: 'invalid-output', message: 'lossy' } })
|
||||
expect(validateChildFrame({ type: 'done', error: { kind: 'output-limit', message: 'big' } }))
|
||||
.toEqual({ type: 'done', error: { kind: 'output-limit', message: 'big' } })
|
||||
expect(validateChildFrame({ type: 'done', value: 1, error: { kind: 'exception', message: 'boom' } }))
|
||||
.toEqual({ type: 'done', value: 1, error: { kind: 'exception', message: 'boom' } })
|
||||
// A `value: undefined` field is dropped (JSON never carries it, but a forged
|
||||
// shape might; the rebuild coalesces to the absent case).
|
||||
expect(validateChildFrame({ type: 'done', value: undefined })).toEqual({ type: 'done' })
|
||||
// A missing or unrecognized kind drops the frame: the child always sends
|
||||
// one of the three, so anything else is a forgery.
|
||||
expect(validateChildFrame({ type: 'done', error: { message: 'boom' } })).toBeUndefined()
|
||||
expect(validateChildFrame({ type: 'done', error: { kind: 'timeout', message: 'x' } })).toBeUndefined()
|
||||
})
|
||||
|
||||
it('rejects malformed done frames', () => {
|
||||
// error must be an object.
|
||||
expect(validateChildFrame({ type: 'done', error: 'boom' })).toBeUndefined()
|
||||
expect(validateChildFrame({ type: 'done', error: null })).toBeUndefined()
|
||||
// error.message must be a string.
|
||||
expect(validateChildFrame({ type: 'done', error: {} })).toBeUndefined()
|
||||
expect(validateChildFrame({ type: 'done', error: { message: 42 } })).toBeUndefined()
|
||||
})
|
||||
|
||||
it('drops non-object inputs and unknown types silently', () => {
|
||||
expect(validateChildFrame(null)).toBeUndefined()
|
||||
expect(validateChildFrame(undefined)).toBeUndefined()
|
||||
expect(validateChildFrame(42)).toBeUndefined()
|
||||
expect(validateChildFrame('str')).toBeUndefined()
|
||||
expect(validateChildFrame({})).toBeUndefined()
|
||||
expect(validateChildFrame({ type: 'unknown' })).toBeUndefined()
|
||||
})
|
||||
|
||||
it('drops CALL frames whose args are non-finite or negative zero', () => {
|
||||
// JSON.parse turns 1e400 into Infinity and preserves -0; the honest child
|
||||
// rejects both before sending, so a call frame carrying one is forged.
|
||||
expect(validateChildFrame({ type: 'call', id: 1, global: 'tools', name: 'x', args: { n: Infinity } })).toBeUndefined()
|
||||
expect(validateChildFrame({ type: 'call', id: Infinity, global: 'tools', name: 'x', args: null })).toBeUndefined()
|
||||
// Plain zero and ordinary floats pass.
|
||||
expect(validateChildFrame({ type: 'call', id: 1, global: 'tools', name: 'x', args: [0, 1.5] }))
|
||||
.toEqual({ type: 'call', id: 1, global: 'tools', name: 'x', args: [0, 1.5] })
|
||||
})
|
||||
|
||||
it('drops a CALL frame whose id is negative zero', () => {
|
||||
// `-0` passes Number.isFinite, but the reply re-serializes it as `0`
|
||||
// (JSON.stringify({id:-0}) === '{"id":0}'), so a forged `-0` id would
|
||||
// collide with a real call whose id is `0`. The honest child never sends it.
|
||||
expect(validateChildFrame({ type: 'call', id: -0, global: 'tools', name: 'x', args: null })).toBeUndefined()
|
||||
// Plain positive zero is a legitimate id and passes.
|
||||
expect(validateChildFrame({ type: 'call', id: 0, global: 'tools', name: 'x', args: null }))
|
||||
.toEqual({ type: 'call', id: 0, global: 'tools', name: 'x', args: null })
|
||||
})
|
||||
|
||||
it('passes DONE values through untouched — losslessness is metered later', () => {
|
||||
// validateChildFrame no longer scans done.value: an unbounded scan would
|
||||
// push every member of a wide forged payload before any byte cap ran. The
|
||||
// done handler's checkDoneValue folds losslessness into the metered walk.
|
||||
expect(validateChildFrame({ type: 'done', value: Infinity })).toEqual({ type: 'done', value: Infinity })
|
||||
expect(validateChildFrame({ type: 'done', value: [{ x: -0 }] })).toEqual({ type: 'done', value: [{ x: -0 }] })
|
||||
expect(validateChildFrame({ type: 'done', value: [0, 1.5] })).toEqual({ type: 'done', value: [0, 1.5] })
|
||||
})
|
||||
})
|
||||
|
||||
describe('lossless-number scan', () => {
|
||||
it('finds non-finite and negative-zero numbers at any depth, iteratively', () => {
|
||||
expect(hasNonLosslessNumber(Infinity)).toBe(true)
|
||||
expect(hasNonLosslessNumber(-Infinity)).toBe(true)
|
||||
expect(hasNonLosslessNumber(NaN)).toBe(true)
|
||||
expect(hasNonLosslessNumber(-0)).toBe(true)
|
||||
expect(hasNonLosslessNumber({ a: [1, { b: -0 }] })).toBe(true)
|
||||
expect(hasNonLosslessNumber({ a: [0, 1.5, 'x', null, true] })).toBe(false)
|
||||
// Deep nesting must not overflow the stack.
|
||||
let deep: unknown = 0
|
||||
for (let i = 0; i < 100000; i++) deep = [deep]
|
||||
expect(hasNonLosslessNumber(deep)).toBe(false)
|
||||
})
|
||||
|
||||
it('walks wide arrays and objects one member at a time', () => {
|
||||
// `call.args` carries no seam byte cap, so a wide forged payload has no
|
||||
// budget to be rejected against — the walk must hold one cursor per
|
||||
// NESTING LEVEL, not one entry per member, or a flat payload at the top of
|
||||
// the host's inbound frame-size cap would allocate tens of millions of stack
|
||||
// entries (and `Object.values` a second full-breadth copy). Observable
|
||||
// through the boundary: a wide payload whose per-member cost the old shape
|
||||
// would have paid still scans, and a violation ANYWHERE in it is found
|
||||
// wherever it sits.
|
||||
const wideArray = new Array(2_000_000).fill(0) as unknown[]
|
||||
expect(hasNonLosslessNumber(wideArray)).toBe(false)
|
||||
// Last element, so the cursor must run the whole breadth lazily.
|
||||
wideArray[wideArray.length - 1] = -0
|
||||
expect(hasNonLosslessNumber(wideArray)).toBe(true)
|
||||
const wideObject: Record<string, unknown> = {}
|
||||
for (let i = 0; i < 200_000; i++) wideObject[`k${i}`] = i
|
||||
expect(hasNonLosslessNumber(wideObject)).toBe(false)
|
||||
wideObject.last = Infinity
|
||||
expect(hasNonLosslessNumber(wideObject)).toBe(true)
|
||||
// Interleaved nesting: a per-level cursor must resume its parent after a
|
||||
// child level ends, so a violation after a nested container is still seen.
|
||||
expect(hasNonLosslessNumber([[1], { a: 2 }, NaN])).toBe(true)
|
||||
})
|
||||
|
||||
it('scans only own enumerable properties', () => {
|
||||
// The per-level cursor filters own keys (a prototype-carrying frame is
|
||||
// impossible off JSON.parse, but the filter is what keeps the walk equal
|
||||
// to what the encoder would serialize).
|
||||
const withProto = Object.create({ inherited: -0 }) as Record<string, unknown>
|
||||
withProto.own = 1
|
||||
expect(hasNonLosslessNumber(withProto)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('unsafe-integer token scan', () => {
|
||||
it('flags integer tokens outside the safe range, skipping strings and float forms', () => {
|
||||
expect(hasUnsafeIntegerToken('{"v":9007199254740993}')).toBe(true)
|
||||
// Exact beyond-safe-range tokens are lossless and pass (2**53, 2**64).
|
||||
expect(hasUnsafeIntegerToken('{"v":9007199254740992}')).toBe(false)
|
||||
expect(hasUnsafeIntegerToken('{"v":18446744073709551616}')).toBe(false)
|
||||
// A token that parses to Infinity is trivially lossy.
|
||||
expect(hasUnsafeIntegerToken(`{"v":${'9'.repeat(400)}}`)).toBe(true)
|
||||
expect(hasUnsafeIntegerToken('{"v":-9007199254740993}')).toBe(true)
|
||||
expect(hasUnsafeIntegerToken('{"v":9007199254740991}')).toBe(false)
|
||||
expect(hasUnsafeIntegerToken('{"v":"9007199254740993"}')).toBe(false)
|
||||
expect(hasUnsafeIntegerToken(String.raw`{"v":"esc\"9007199254740993"}`)).toBe(false)
|
||||
expect(hasUnsafeIntegerToken('{"v":9007199254740993.0}')).toBe(false)
|
||||
expect(hasUnsafeIntegerToken('{"v":9e99}')).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('checkDoneValue', () => {
|
||||
it('matches the exact encoded size and rejects one byte over', () => {
|
||||
const cases: unknown[] = [null, true, false, 0, -1.5, 'a"b\\', [], {}, [1, 'x', null], { a: [1, 2], b: { c: 'd' } }]
|
||||
for (const value of cases) {
|
||||
const exact = Buffer.byteLength(JSON.stringify(value), 'utf8')
|
||||
expect(checkDoneValue(value, exact), JSON.stringify(value)).toEqual({ ok: true, bytes: exact })
|
||||
expect(checkDoneValue(value, exact - 1), JSON.stringify(value)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
expect(encodeJsonPlain(value)).toBe(JSON.stringify(value))
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects an over-budget value before its secondary allocations', () => {
|
||||
// A huge string is refused on the cheap length lower bound, before its
|
||||
// escaped copy is built.
|
||||
const huge = { data: 'x'.repeat(1_000_000), tail: 'y' }
|
||||
expect(checkDoneValue(huge, 1024)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
// A flat array far above the budget fails on the brackets+length bound,
|
||||
// before its elements are pushed onto the traversal stack. (The array is
|
||||
// already materialized by the upstream parse; this only avoids the extra
|
||||
// per-element stack growth.)
|
||||
const flat = new Array(10_000_000).fill(0)
|
||||
expect(checkDoneValue(flat, 1024)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
// A wide object: braces+commas fit the cap, but the per-entry lower bound
|
||||
// (quoted key + colon + value = count*4) does not, so it fails before any
|
||||
// key is escaped or any value enqueued.
|
||||
const wide: Record<string, number> = {}
|
||||
for (let i = 0; i < 10; i++) wide[`k${i}`] = i
|
||||
expect(checkDoneValue(wide, 12)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
})
|
||||
|
||||
it('meters a string\'s exact escaped size without allocating it', () => {
|
||||
// A control-heavy string that fits by DECODED length but not once escaped
|
||||
// must still reject: 200 NULs are 200 UTF-16 units (would pass a naive
|
||||
// length bound against cap 1024) but escape to 200*6 + 2 = 1202 bytes.
|
||||
// jsonStringBytesUpTo scans and bails before the escaped copy is built.
|
||||
expect(checkDoneValue('\0'.repeat(200), 1024)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
// Exact-size acceptance, no false rejection: one NUL serializes to a
|
||||
// 6-char \\uXXXX escape, so with the two quotes = 8 bytes.
|
||||
expect(checkDoneValue('\0', 8)).toEqual({ ok: true, bytes: 8 })
|
||||
expect(checkDoneValue('\0', 7)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
// Multi-byte and astral characters meter at their raw UTF-8 width (a valid
|
||||
// surrogate pair is 4 bytes, matching JSON.stringify), not a 6-byte escape.
|
||||
expect(checkDoneValue('\u00e9', 4)).toEqual({ ok: true, bytes: 4 }) // 2 quotes + 2-byte UTF-8
|
||||
expect(checkDoneValue('\u{1f600}', 6)).toEqual({ ok: true, bytes: 6 }) // 2 quotes + 4-byte UTF-8
|
||||
expect(checkDoneValue('\u{1f600}', 5)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
// A lone surrogate escapes to \\uXXXX = 6, so with quotes = 8.
|
||||
expect(checkDoneValue('\ud800', 8)).toEqual({ ok: true, bytes: 8 })
|
||||
// A high surrogate followed by a NON-low character is a lone surrogate (6-byte
|
||||
// escape) plus that character: `\ud800` + `a` = 2 quotes + 6 + 1 = 9.
|
||||
expect(checkDoneValue('\ud800a', 9)).toEqual({ ok: true, bytes: 9 })
|
||||
// A BMP 3-byte code point (CJK) meters at its raw UTF-8 width: 2 quotes + 3.
|
||||
expect(checkDoneValue('中', 5)).toEqual({ ok: true, bytes: 5 })
|
||||
// Same non-allocating meter for object keys, before the value is enqueued.
|
||||
expect(checkDoneValue({ ['\0'.repeat(200)]: 1 }, 1024)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
// A string reached with less than the two quotes' worth of budget is refused
|
||||
// immediately (even the empty escaped form does not fit).
|
||||
expect(checkDoneValue('x', 1)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
})
|
||||
|
||||
it('meters only own enumerable keys', () => {
|
||||
// The walk counts keys with a `for...in` + hasOwn pass rather than
|
||||
// Object.keys/entries (which allocate per member before the bound). A
|
||||
// prototype-carrying forgery is impossible off JSON.parse, but the own-key
|
||||
// filter is what keeps the count equal to the encoder's.
|
||||
const withProto = Object.create({ inherited: 'x' }) as Record<string, unknown>
|
||||
withProto.own = 1
|
||||
expect(checkDoneValue(withProto, 1024)).toEqual({ ok: true, bytes: Buffer.byteLength('{"own":1}', 'utf8') })
|
||||
})
|
||||
|
||||
it('rejects non-finite and negative-zero numbers at any depth as non-lossless', () => {
|
||||
expect(checkDoneValue(Infinity, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
|
||||
expect(checkDoneValue(-Infinity, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
|
||||
expect(checkDoneValue(NaN, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
|
||||
expect(checkDoneValue(-0, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
|
||||
expect(checkDoneValue({ a: [1, { b: -0 }] }, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
|
||||
// An ordinary finite value within budget passes with its exact byte count.
|
||||
const clean = { a: [0, 1.5, 'x', null, true] }
|
||||
expect(checkDoneValue(clean, 1024)).toEqual({ ok: true, bytes: Buffer.byteLength(JSON.stringify(clean), 'utf8') })
|
||||
})
|
||||
|
||||
it('classifies an over-budget value as over-budget regardless of member order', () => {
|
||||
// A value that is BOTH over-budget and non-lossless must reject as
|
||||
// over-budget whichever member the walk reaches first — the non-lossless
|
||||
// number is recorded and metering finishes, so the two orders below (the
|
||||
// same value) cannot classify differently. Cap 100 with a 1000-char string.
|
||||
const big = 'x'.repeat(1000)
|
||||
expect(checkDoneValue([big, Infinity], 100)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
expect(checkDoneValue([Infinity, big], 100)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
// A non-lossless number that DOES fit the budget still rejects as
|
||||
// non-lossless (the recorded violation is the verdict once the whole value
|
||||
// is confirmed within budget).
|
||||
expect(checkDoneValue([Infinity], 100)).toEqual({ ok: false, reason: 'non-lossless' })
|
||||
// The non-lossless number's OWN encoded bytes still count toward the budget,
|
||||
// so a value whose only over-budget contribution is the non-lossless number
|
||||
// itself is classified over-budget, not non-lossless. `[Infinity]` encodes
|
||||
// as the 10-byte `[Infinity]`; at cap 3 the byte check wins.
|
||||
expect(checkDoneValue([Infinity], 3)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
expect(checkDoneValue(Infinity, 3)).toEqual({ ok: false, reason: 'over-budget' })
|
||||
})
|
||||
|
||||
it('meters and encodes deep nesting iteratively without overflowing the stack', () => {
|
||||
let deep: unknown = 0
|
||||
for (let i = 0; i < 100_000; i++) deep = [deep]
|
||||
// 100000 '[' + '0' + 100000 ']' = 200001 bytes.
|
||||
expect(checkDoneValue(deep, 1_000_000)).toEqual({ ok: true, bytes: 200_001 })
|
||||
// encodeJsonPlain's headline contract is the same stack-safety (JSON.stringify
|
||||
// recurses per level and throws RangeError a few thousand deep), so exercise
|
||||
// it on the same 100k-deep value — JSON.stringify would throw here.
|
||||
expect(encodeJsonPlain(deep)).toBe(`${'['.repeat(100_000)}0${']'.repeat(100_000)}`)
|
||||
})
|
||||
|
||||
it('emits exact digits for beyond-safe integral doubles', () => {
|
||||
// String(2**60) prints the ROUNDED ...847000; echoing that to the child
|
||||
// would change the integer. BigInt digits give the exact ...846976.
|
||||
const v = JSON.parse('[1152921504606846976]') as unknown
|
||||
expect(encodeJsonPlain(v)).toBe('[1152921504606846976]')
|
||||
expect(checkDoneValue(v, 100)).toEqual({ ok: true, bytes: Buffer.byteLength('[1152921504606846976]', 'utf8') })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,38 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { detachResidual } from '../src/index.ts'
|
||||
|
||||
describe('detachResidual — fd-3 residual detachment', () => {
|
||||
it('returns a copy that does NOT share the source frame allocation', () => {
|
||||
// Simulate the data handler's state: one large joined frame from
|
||||
// Buffer.concat, sliced past its newline to leave a small residual VIEW.
|
||||
// The fixture MUST stay larger than Node's Buffer pool threshold
|
||||
// (`Buffer.poolSize / 2`, 4 KiB): above it `Buffer.from` allocates a
|
||||
// dedicated backing store whose `byteLength` equals the copy's length,
|
||||
// which is what the byteLength assertion below pins. A smaller residual
|
||||
// would be pooled into an 8 KiB shared ArrayBuffer, making `byteLength`
|
||||
// report 8192 and the assertion false-fail even though the fix is intact.
|
||||
const joined = Buffer.alloc(1024 * 1024, 0x61) // 1 MiB backing allocation
|
||||
joined[512] = 0x0a // a newline partway through
|
||||
const residual = joined.subarray(513) // a view onto `joined`'s backing store
|
||||
|
||||
// Before the fix the handler carried this view forward verbatim, pinning the
|
||||
// whole 1 MiB `joined` allocation behind a residual that reports far fewer
|
||||
// bytes. A right-sized copy must not point back into `joined`.
|
||||
const [carried] = detachResidual(residual)
|
||||
|
||||
expect(carried).toBeDefined()
|
||||
expect(carried!.length).toBe(residual.length)
|
||||
expect(carried!.equals(residual)).toBe(true)
|
||||
// The core invariant: the copy does NOT share the source frame's backing
|
||||
// store, so retaining it cannot pin the 1 MiB allocation.
|
||||
expect(carried!.buffer).not.toBe(joined.buffer)
|
||||
// And the copy's own backing store is sized to its content — not the whole
|
||||
// frame. Holds because the fixture exceeds the pool threshold (see above);
|
||||
// a subarray view would report the source's full byteLength here.
|
||||
expect(carried!.buffer.byteLength).toBe(carried!.length)
|
||||
})
|
||||
|
||||
it('carries nothing forward for an empty residual', () => {
|
||||
expect(detachResidual(Buffer.alloc(0))).toEqual([])
|
||||
})
|
||||
})
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../code-runtime/code-runtime"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../runtime-diagnostics/invariants"
|
||||
},
|
||||
{
|
||||
"path": "../../util/timeout"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
import { defineConfig } from 'tsdown'
|
||||
|
||||
/**
|
||||
* Single ESM bundle. The Python-side code is not TypeScript and ships verbatim
|
||||
* under `py/` (whitelisted in package.json `files`) — no build step needed.
|
||||
*/
|
||||
export default defineConfig({
|
||||
entry: ['lib/types/index.js', 'lib/types/invariant.js'],
|
||||
outDir: 'lib',
|
||||
format: ['esm'],
|
||||
platform: 'node',
|
||||
target: 'es2024',
|
||||
fixedExtension: false,
|
||||
dts: false,
|
||||
clean: false,
|
||||
})
|
||||
Reference in New Issue
Block a user