Merge pull request #1083 from deepseek-harness/feat/code-runtime-python-protocol

feat(code-runtime-python): add the fd-3 frame protocol
This commit is contained in:
Chinesezjc
2026-08-17 19:07:50 +08:00
committed by GitHub
27 changed files with 1529 additions and 4 deletions
@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/code-runtime/code-runtime-python/README.md
README.md: 1fc19b89b751ce5063d6937d588b96f2f36ae69e
README.zh.md: 0351001308541b41ba519b70a61d25c48de73a4b
@@ -0,0 +1,29 @@
# @deepseek-ai/dsh-code-runtime-python
English | [中文](README.zh.md)
CPython-subprocess implementation of the [`@deepseek-ai/dsh-code-runtime`](../code-runtime/README.md) seam. Companion to [`@deepseek-ai/dsh-code-runtime-worker-thread`](../code-runtime-worker-thread/README.md); trades the Node worker thread for a fresh `python3` subprocess so model code is Python instead of TypeScript.
The package owns the wire protocol for that seam: the host-side frame codec and the Python-side mirror of the same message vocabulary.
## Wire protocol
The host and the CPython subprocess exchange a versionless, JSON-lines protocol on the child's fd 3 — one JSON object per line, leaving stdout/stderr free for the program's own output. `src/protocol.ts` is the host side; `py/protocol.py` mirrors its message shapes and the shared truncation-marker text on the Python side.
- **fd 3, not stdout** — Node pins the channel positionally with `stdio: ['pipe','pipe','pipe','pipe']`; the Python bootstrap reads the same `PROTOCOL_FD` constant. JSON-lines framing.
- **Host treats every inbound frame as hostile** — model code has full access to fd 3 and can post anything through it, so `validateChildFrame` shape-validates and REBUILDS each frame before the host reads it: forged extra fields never ride along, a non-number call id can never be echoed into a reply, and junk drops to `undefined` rather than throwing in the host's message handler. The Python side trusts host replies (the host is not model-controlled).
- **Lossless-JSON crossing** — completion values and binding arguments cross as exact JSON. `encodeJsonPlain` serializes a `JSON.parse`-produced value without recursion, so a deep value below the byte budget crosses intact instead of dying on `JSON.stringify`'s stack limit; `checkDoneValue` meters a forged completion value's byte length AND number losslessness in one traversal that rejects an over-budget payload before the incremental work it would add (the enqueued children; strings and keys are metered by a non-allocating escaped-size scan, so the escaped copy is never materialized) — the frame's own width is already parsed and capped upstream by the host's fd-3 receive buffer, not re-bounded here; `hasUnsafeIntegerToken` reads the raw frame text to catch an integer token that `JSON.parse` would silently round; `hasNonLosslessNumber` rejects a non-finite or negative-zero number in unbounded `call.args`. Beyond-safe-range integral doubles serialize through `BigInt` digits so the exact integer crosses, not the rounded `String()` form.
- **Shared truncation marker** — `logTruncationMarker(maxBytes)` produces byte-identical text on both sides, so a truncated log run reads the same however the cap was hit. The `log` frame's `truncated` flag distinguishes the child ledger's own marker from program output.
## Model Experience
Indirectly, through Code Mode in [`dsh-tools`](../../core/tools/README.md), which renders this backend's exact completion value when it fits (or an explicit `invalid-output` / `output-limit` failure), plus the exact `[dsh-code-runtime-python] log capture truncated at <maxLogBytes> bytes` log marker, into a retained `run_code` result.
#### KV Cache effect
No direct invalidation; the named consumer owns any request-prefix changes.
## Known Limitations and Deferred Work
- **The cross-language guard covers the runtime-executed surfaces and the frame field shapes** — `tests/protocol-mirror.e2e.ts` spawns a real `python3` and asserts, against `src/protocol.ts`, both `PROTOCOL_FD` / the log truncation marker text AND each `TypedDict`'s required/optional wire field set in `py/protocol.py`. What it does not compare is the field *types* (e.g. that `cpuSeconds` is an `int` on both sides): comparing type declarations across TypeScript and Python has no mechanical equivalent here, so a type-level drift is still caught by review plus the backend's real-subprocess suite rather than this package's tests.
- **`src/index.ts` exports the protocol vocabulary only** — the package carries no subprocess execution path and no Python-side JSON codec, so nothing here spawns `python3` outside the mirror test.
@@ -0,0 +1,29 @@
# @deepseek-ai/dsh-code-runtime-python
[English](README.md) | 中文
[`@deepseek-ai/dsh-code-runtime`](../code-runtime/README.md) seam 的 CPython 子进程实现。与 [`@deepseek-ai/dsh-code-runtime-worker-thread`](../code-runtime-worker-thread/README.md) 配套;以全新的 `python3` 子进程取代 Node worker 线程,让模型代码从 TypeScript 换成 Python。
本包持有该 seam 的 wire protocolhost 侧的帧编解码,以及 Python 侧对同一套消息词汇的镜像。
## Wire protocol
host 与 CPython 子进程在子进程的 fd 3 上交换一个无版本号的 JSON-lines 协议——每行一个 JSON 对象,让 stdout/stderr 空出给程序自己的输出。`src/protocol.ts` 是 host 侧;`py/protocol.py` 在 Python 侧镜像其帧词汇与共享的截断标记文本。
- **fd 3,而非 stdout** —— Node 通过 `stdio: ['pipe','pipe','pipe','pipe']` 按位置钉住通道;Python bootstrap 读取相同的 `PROTOCOL_FD` 常量。JSON-lines 帧。
- **host 把每个入站帧当作敌意输入** —— 模型代码对 fd 3 有完全访问权、可通过它发送任意内容,所以 `validateChildFrame` 在 host 读取前对每个帧做形状校验并重建:伪造的额外字段绝不随行,非数字的 call id 绝不会被回显进 reply,垃圾降为 `undefined` 被丢弃,而不是在 host 的 message handler 里抛错。Python 侧信任 host 回复(host 不受模型控制)。
- **lossless-JSON 穿越** —— 完成值与 binding 参数以精确 JSON 穿越。`encodeJsonPlain` 无递归地序列化一个 `JSON.parse` 产出的值,使低于字节预算的深层值能完整穿越,而不是死在 `JSON.stringify` 的栈限制上;`checkDoneValue` 在一次遍历中同时计量伪造完成值的字节长度与数字无损性,在它本会新增的增量工作之前就拒绝超预算 payload(即入栈子节点;字符串与 key 由非分配的转义尺寸扫描计量,从不物化转义副本)——帧自身的宽度已被上游 `JSON.parse` 支付、由 host 的 fd-3 接收缓冲封顶,并非在此重新约束;`hasUnsafeIntegerToken` 读取原始帧文本,捕获 `JSON.parse` 会静默舍入的整数 token`hasNonLosslessNumber` 拒绝无字节上限的 `call.args` 中的非有限数或负零。超出安全范围的整数型 double 通过 `BigInt` 数字序列化,穿越的是精确整数而非 `String()` 的舍入形式。
- **共享截断标记** —— `logTruncationMarker(maxBytes)` 在两侧产出逐字节一致的文本,使被截断的日志运行无论从哪侧触达上限都读起来一致。`log` 帧的 `truncated` 标志把子进程 ledger 自身的标记与程序输出区分开。
## Model Experience
经由 [`dsh-tools`](../../core/tools/README.md) 里的 Code Mode 间接生效:Code Mode 把本后端的精确完成值(放得下时)或一个明确的 `invalid-output` / `output-limit` 失败,连同精确的 `[dsh-code-runtime-python] log capture truncated at <maxLogBytes> bytes` 日志标记,渲染进一个保留的 `run_code` 结果。
#### KV Cache effect
无直接失效;具名消费者拥有任何请求前缀的变更。
## Known Limitations and Deferred Work
- **跨语言 guard 覆盖运行时执行的面与帧字段形状** —— `tests/protocol-mirror.e2e.ts` 启动一个真实 `python3`,对照 `src/protocol.ts` 断言 `PROTOCOL_FD` / 日志截断标记文本,以及 `py/protocol.py` 中每个 `TypedDict` 的必填/可选 wire 字段集。它不比较字段的*类型*(例如 `cpuSeconds` 两侧都是 `int`):跨 TypeScript 与 Python 比较类型声明在此无机械等价物,故类型级漂移仍由 review 加后端真子进程套件捕获,而非本包的测试。
- **`src/index.ts` 只导出协议词汇** —— 本包不含子进程执行路径,也不含 Python 侧的 JSON codec,因此除 mirror 测试之外没有任何地方会启动 `python3`
@@ -0,0 +1,42 @@
{
"name": "@deepseek-ai/dsh-code-runtime-python",
"description": "CPython subprocess implementation of the DeepSeek Harness code-execution seam",
"version": "0.1.0-rc.6",
"publishConfig": {
"access": "public"
},
"repository": {
"type": "git",
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
"directory": "packages/code-runtime/code-runtime-python"
},
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./invariant": {
"types": "./lib/types/invariant.d.ts",
"default": "./lib/invariant.js"
},
"./package.json": "./package.json"
},
"files": [
"lib/index.js",
"lib/invariant.js",
"py/**/*.py",
"lib/types/**/*.d.ts"
],
"license": "MIT",
"peerDependencies": {
"@deepseek-ai/dsh-invariants": "workspace:^",
"@deepseek-ai/cordis": "workspace:^"
},
"devDependencies": {
"@deepseek-ai/dsh-invariants": "workspace:^",
"@deepseek-ai/cordis": "workspace:^"
}
}
@@ -0,0 +1,138 @@
"""Wire protocol vocabulary for the Python side of dsh-code-runtime-python.
Mirrors ``src/protocol.ts``. Frames travel on fd 3 as JSON-lines (one JSON
object per line). The host validates every inbound frame; this side trusts
host replies.
The wire uses the JSON key ``global`` (a Python keyword), so the frame
``TypedDict``s that carry it are declared with the functional syntax rather than
class bodies: a class attribute cannot be named ``global``, and a ``global_``
attribute would describe a key the wire never sends. Optional-field messages
pair a required base with a ``total=False`` subclass so a required field such as
``type`` cannot be dropped while ``value``/``error``/``truncated`` stay optional.
"""
from __future__ import annotations
from typing import Any, Literal, TypedDict, Union
# The protocol fd from the child's perspective. Node passes
# ``stdio: [pipe, pipe, pipe, pipe]`` so the fourth entry (fd 3) is the
# framed-JSON channel; stdout/stderr stay clear for the program's own output.
PROTOCOL_FD = 3
class ErrorClass(TypedDict):
"""A namespace's program-visible exception class: rejected calls raise its
instances carrying the failed member name on ``memberNameProperty``."""
name: str
memberNameProperty: str
# ``global`` is a Python keyword, so the required part is declared functionally
# to hold the real wire key; ``errorClass`` is optional per the TS `errorClass?`.
_NamespaceRequired = TypedDict("_NamespaceRequired", {"global": str, "names": "list[str]"})
class Namespace(_NamespaceRequired, total=False):
"""One binding namespace declaration: the ``global`` name, its function
``names``, and an optional program-visible ``errorClass`` for rejected calls."""
errorClass: ErrorClass
class BootMessage(TypedDict):
"""Host → child, first frame on fd 3. Carries every cap and the namespaces."""
type: Literal["boot"]
cpuSeconds: int
addressSpaceBytes: int
maxLogBytes: int
maxValueBytes: int
namespaces: "list[Namespace]"
class RunMessage(TypedDict):
"""Host → child, sent after ``boot-ack``. Carries only the program body."""
type: Literal["run"]
program: str
class BootAckMessage(TypedDict):
"""Child → host: resource limits applied, ready for the run message."""
type: Literal["boot-ack"]
# ``global`` wire key: whole message declared functionally, all fields required.
CallMessage = TypedDict(
"CallMessage",
{"type": Literal["call"], "id": int, "global": str, "name": str, "args": Any},
)
_LogMessageRequired = TypedDict("_LogMessageRequired", {"type": Literal["log"], "text": str})
class LogMessage(_LogMessageRequired, total=False):
"""Child → host: one captured text chunk, streamed eagerly.
``truncated`` is set only on the frame that IS the child ledger's truncation
marker (not program output), so the host stops capturing at the same point
the child did — mirrors the TS `truncated?`.
"""
truncated: bool
class DoneErrorField(TypedDict):
"""Child → host: the failure carried on a ``done`` frame. ``kind`` is one of
the three the host validates; ``message`` is the traceback or diagnostic."""
kind: Literal["exception", "invalid-output", "output-limit"]
message: str
_DoneMessageRequired = TypedDict("_DoneMessageRequired", {"type": Literal["done"]})
class DoneMessage(_DoneMessageRequired, total=False):
"""Child → host: the program settled. ``value`` and ``error`` are optional per the TS mirror."""
value: Any
error: DoneErrorField
ChildToHost = Union[BootAckMessage, CallMessage, LogMessage, DoneMessage]
class ReplyOk(TypedDict):
type: Literal["reply"]
id: int
ok: Literal[True]
value: Any
class ReplyErr(TypedDict):
type: Literal["reply"]
id: int
ok: Literal[False]
message: str
ReplyMessage = Union[ReplyOk, ReplyErr]
# The host sends ``boot`` and ``run`` before any ``reply``, so the child-facing
# inbound union covers all three, not replies alone.
HostToChild = Union[BootMessage, RunMessage, ReplyMessage]
def log_truncation_marker(max_bytes: int) -> str:
"""Return the in-band marker for a log ledger that exhausted its budget.
Byte-identical text on both sides of the wire so a truncated run reads the
same however the cap was hit.
"""
return f"[dsh-code-runtime-python] log capture truncated at {max_bytes} bytes"
@@ -0,0 +1,19 @@
/**
* CPython subprocess code runtime for the DeepSeek Harness code-execution seam.
*
* The package owns the versionless fd-3 wire protocol between the Node host and
* the CPython subprocess. The protocol's host-side codec and hostile-frame
* validators are re-exported so every consumer of the wire shares one
* vocabulary.
* @module @deepseek-ai/dsh-code-runtime-python
*/
export type { BootMessage, ChildToHost, ReplyMessage } from './protocol.ts'
export {
checkDoneValue,
encodeJsonPlain,
hasNonLosslessNumber,
hasUnsafeIntegerToken,
logTruncationMarker,
validateChildFrame,
} from './protocol.ts'
@@ -0,0 +1,31 @@
/**
* Package-owned invariant companion for `@deepseek-ai/dsh-code-runtime-python`.
* @module @deepseek-ai/dsh-code-runtime-python/invariant
*/
/* jscpd:ignore-start */
import type { Context } from '@deepseek-ai/cordis'
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
const PACKAGE_NAME = '@deepseek-ai/dsh-code-runtime-python'
/** Cordis companion plugin name. */
export const name = 'code-runtime-python-invariant'
/** Service required before the companion can reserve package ownership. */
export const inject = ['invariants']
/**
* No runtime invariant: this package ships only the fd-3 wire-protocol codec and its Python mirror,
* exposing no runtime event sequence or mutable data relation; `protocol.spec.ts` and
* `protocol-mirror.e2e.ts` cover the protocol's behavior.
*/
const install: InvariantInstaller = () => {}
/**
* Register this package's invariant companion.
* @param ctx - Cordis context carrying the invariant service.
* @returns the installed registration's disposer after setup succeeds.
*/
export const apply = (ctx: Context): Promise<() => void> =>
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
/* jscpd:ignore-end */
@@ -0,0 +1,656 @@
/**
* Versionless, JSON-lines wire protocol between the Node host and the CPython subprocess. Frames
* travel on the child's fd 3 (one JSON object per line), leaving stdout/stderr free for the
* program's own output. Host treats every inbound frame as hostile because model code can post
* anything through the same fd; the Python bootstrap trusts host replies.
* @module @deepseek-ai/dsh-code-runtime-python/src/protocol
*/
/**
* The framed-JSON channel's file descriptor from the child's perspective. The
* host pins it positionally when it spawns the child (`stdio` index 3, i.e.
* `['pipe','pipe','pipe','pipe']`), and the Python bootstrap reads the same
* number from its own `protocol.py`. Exported as the single TS-side source of
* truth: the host wiring uses it, and the cross-language mirror test asserts the
* Python constant equals it, so a drift on either side breaks the boot channel
* loudly rather than silently.
*/
export const PROTOCOL_FD = 3
/**
* One binding namespace declaration inside a {@link BootMessage}. `global` is
* the program-visible name the namespace is materialized under; `errorClass`,
* when present, asks the bootstrap to mint a program-visible exception class.
*/
interface Namespace {
global: string
names: string[]
errorClass?: ErrorClass
}
/**
* A namespace's program-visible exception class: rejected calls raise its
* instances carrying the failed member name on `memberNameProperty`.
*/
interface ErrorClass {
name: string
memberNameProperty: string
}
/**
* What the host sends immediately after spawn, as the first line on fd 3. The
* Python bootstrap reads this, applies resource limits, then waits for the
* subsequent run frame. Separated from the run so the run message stays
* pure model input.
*/
export interface BootMessage {
type: 'boot'
/** RLIMIT_CPU seconds; the Python bootstrap sets this on itself before executing model code. */
cpuSeconds: number
/** RLIMIT_AS bytes; caps address space so a runaway allocation fails cleanly. */
addressSpaceBytes: number
/** Shared byte budget for captured log text (Python-side ledger). */
maxLogBytes: number
/** Byte cap for the rendered completion value. */
maxValueBytes: number
/**
* The namespaces to materialize inside the program (globals + names;
* functions stay host-side). See {@link Namespace}.
*/
namespaces: Namespace[]
}
/** Host → Python: sent after `boot-ack`; carries only the model's program body. */
interface RunMessage {
type: 'run'
program: string
}
/** Python → host: acknowledges boot completed and resource limits are in place. */
interface BootAckMessage {
type: 'boot-ack'
}
/** Python → host: one bridged binding call (`await tools.name(args)` inside the program). */
interface CallMessage {
type: 'call'
/** Python-issued correlation id; the host answers each id at most once and ignores duplicates. */
id: number
/** The namespace global the call targets. */
global: string
/** The function name within the namespace. */
name: string
/** The JSON-safe argument the model program passed. */
args: unknown
}
/**
* Python → host: captured text, streamed eagerly so output survives a
* mid-run termination (RLIMIT_CPU, SIGTERM/SIGKILL, host wall-timeout).
*/
interface LogMessage {
type: 'log'
text: string
/**
* Set when this frame IS the child ledger's truncation marker rather than
* program output. The two ledgers can exhaust at different points — one
* child entry larger than `maxLogBytes` sends only the marker while the host
* ledger is still nearly empty — so the host cannot infer the child's state
* from its own budget, and comparing the text against the marker string
* would also honour a program that printed that string itself. Carrying it
* as a field lets the host stop capturing at the same point the child did
* and keeps exactly one marker in `logs`.
*/
truncated?: boolean
}
/** The failure carried on a {@link DoneMessage}: one of three kinds plus text. */
interface DoneErrorField {
kind: 'exception' | 'invalid-output' | 'output-limit'
message: string
}
/**
* Python → host: the program settled. `error` carries a program exception
* (traceback text), an `invalid-output` (completion value was not lossless
* JSON), or an `output-limit` (serialized completion exceeded the configured
* cap); wall/CPU budgets, aborts, and substrate death are observed host-side.
* From the honest child `value` is present only on a clean completion that
* produced one, and crosses as exact lossless JSON — never substituted or
* truncated. A forged frame CAN carry both `value` and `error`;
* {@link validateChildFrame} preserves both rather than guessing which to drop,
* so a consumer MUST check `error` first and ignore `value` when it is set.
*/
interface DoneMessage {
type: 'done'
value?: unknown
error?: DoneErrorField
}
/**
* Every message the Python side sends. The member interfaces stay module-
* private: consumers match on the union's discriminant; the host sends the
* boot and run frames as inline literals.
*/
export type ChildToHost = BootAckMessage | CallMessage | LogMessage | DoneMessage
/** Host → Python: successful answer to one {@link CallMessage}. */
interface ReplyOk {
type: 'reply'
id: number
ok: true
value: unknown
}
/** Host → Python: failed answer to one {@link CallMessage}. */
interface ReplyErr {
type: 'reply'
id: number
ok: false
message: string
}
/** Host → Python: the answer to one {@link CallMessage}. */
export type ReplyMessage = ReplyOk | ReplyErr
/** The required (non-optional) keys of `T`, as string literals. */
type RequiredKeys<T> = { [K in keyof T]-?: object extends Pick<T, K> ? never : K }[keyof T] & string
/** The optional keys of `T`, as string literals. */
type OptionalKeys<T> = { [K in keyof T]-?: object extends Pick<T, K> ? K : never }[keyof T] & string
/**
* Whether each key of frame `T` is a `'required'` or `'optional'` wire field.
* Because it is `Record<keyof T, …>`, an entry MUST list every key — a field
* added to the interface without a corresponding entry fails typecheck — and
* `keyof T`-typed keys reject a name no frame declares. The `'required'` /
* `'optional'` tag must match the field's actual optionality (checked by the
* `satisfies FrameFieldRoles<…>` clause on {@link WIRE_FRAME_FIELD_ROLES}), so
* an optionality flip is caught too. This is the exhaustive counterpart the
* array form could not express (a subset array satisfied it silently).
*/
type FrameFieldRoles<T> = Record<RequiredKeys<T>, 'required'> & Record<OptionalKeys<T>, 'optional'>
interface WireFrameShapes {
BootMessage: BootMessage
Namespace: Namespace
RunMessage: RunMessage
BootAckMessage: BootAckMessage
CallMessage: CallMessage
LogMessage: LogMessage
DoneErrorField: DoneErrorField
DoneMessage: DoneMessage
ErrorClass: ErrorClass
ReplyOk: ReplyOk
ReplyErr: ReplyErr
}
/**
* The frames carried on a message union: everything the host and child send as
* a top-level frame (`ChildToHost`, the two reply variants, and the host→child
* boot/run frames). The nested shapes `Namespace`, `ErrorClass`, and
* `DoneErrorField` are fields of other frames, not frames themselves, so they
* are excluded here and covered only by the roles `satisfies` and the mirror e2e.
*/
type MessageFrames = ChildToHost | ReplyMessage | BootMessage | RunMessage
/** The roster's value types minus the three nested (non-frame) shapes. */
type RosterMessageFrames = Exclude<WireFrameShapes[keyof WireFrameShapes], Namespace | ErrorClass | DoneErrorField>
/**
* Compile-time proof that {@link WireFrameShapes}'s message-frame entries are
* EXACTLY the frames on the message unions — checked BOTH directions. Forward
* (`MessageFrames extends RosterMessageFrames`) catches a frame added to a union
* without a roster entry; reverse (`RosterMessageFrames extends MessageFrames`)
* catches a frame removed from a union while the roster still lists it (e.g.
* dropping `ReplyErr` from `ReplyMessage`). Either divergence makes an alias
* `false`, failing the assignment below. Type-only; the `const`s emit nothing
* meaningful at runtime.
*/
type UnionSubsetOfRoster = [MessageFrames] extends [RosterMessageFrames] ? true : false
type RosterSubsetOfUnion = [RosterMessageFrames] extends [MessageFrames] ? true : false
const _unionSubsetOfRoster: UnionSubsetOfRoster = true
const _rosterSubsetOfUnion: RosterSubsetOfUnion = true
void _unionSubsetOfRoster
void _rosterSubsetOfUnion
/**
* Each frame's wire fields tagged by required/optional, keyed by field name so
* the mapping is exhaustive over the frame interface (see {@link FrameFieldRoles})
* across the whole {@link WireFrameShapes} roster. Bound to the interfaces by
* `satisfies` below; {@link WIRE_FRAME_FIELDS} projects it to sorted
* required/optional arrays for the cross-language mirror comparison. `global` is
* the JSON key {@link CallMessage} and {@link Namespace} send (a reserved word
* the Python side carries via a functional `TypedDict`).
*/
const WIRE_FRAME_FIELD_ROLES = {
BootMessage: { type: 'required', cpuSeconds: 'required', addressSpaceBytes: 'required', maxLogBytes: 'required', maxValueBytes: 'required', namespaces: 'required' },
Namespace: { global: 'required', names: 'required', errorClass: 'optional' },
RunMessage: { type: 'required', program: 'required' },
BootAckMessage: { type: 'required' },
CallMessage: { type: 'required', id: 'required', global: 'required', name: 'required', args: 'required' },
LogMessage: { type: 'required', text: 'required', truncated: 'optional' },
DoneErrorField: { kind: 'required', message: 'required' },
DoneMessage: { type: 'required', value: 'optional', error: 'optional' },
ErrorClass: { name: 'required', memberNameProperty: 'required' },
ReplyOk: { type: 'required', id: 'required', ok: 'required', value: 'required' },
ReplyErr: { type: 'required', id: 'required', ok: 'required', message: 'required' },
} as const satisfies { [K in keyof WireFrameShapes]: FrameFieldRoles<WireFrameShapes[K]> }
/**
* The wire field names of each frame, split into sorted required and optional
* key arrays — the shape the cross-language mirror test compares against
* `py/protocol.py`'s `TypedDict` `__required_keys__`/`__optional_keys__`.
* Projected from {@link WIRE_FRAME_FIELD_ROLES}, so it inherits that mapping's
* exhaustive, optionality-checked binding to the frame interfaces: a TS-side
* field add, remove, rename, or optionality flip fails typecheck at the roles
* map, and a Python-side divergence fails the mirror test at runtime.
*/
export const WIRE_FRAME_FIELDS =
Object.fromEntries(
Object.entries(WIRE_FRAME_FIELD_ROLES).map(([frame, roles]) => {
const required = Object.keys(roles).filter(key => (roles as Record<string, string>)[key] === 'required').sort()
const optional = Object.keys(roles).filter(key => (roles as Record<string, string>)[key] === 'optional').sort()
return [frame, { required, optional }]
}),
) as Record<keyof typeof WIRE_FRAME_FIELD_ROLES, { required: string[]; optional: string[] }>
/**
* The in-band marker text announcing that log capture stopped at the byte
* budget. Shared wire vocabulary: the Python-side LogBuffer emits it when ITS
* ledger exhausts, and the host emits identical text when its own ledger drops
* a frame first (forged fd-3 traffic, stray stdout bytes) — a truncated run
* reads the same however the cap was hit.
* @param maxBytes - the configured `maxLogBytes` the marker names.
* @returns the marker line.
*/
export function logTruncationMarker(maxBytes: number): string {
return `[dsh-code-runtime-python] log capture truncated at ${maxBytes} bytes`
}
/**
* Serialize one JSON-parse-produced value without recursion. `JSON.stringify`
* recurses per nesting level and throws `RangeError` a few thousand levels
* deep, but the seam's `CodeJsonValue` has no depth limit — an honest deep
* completion or binding resolution below the byte budget must cross intact
* (the worker backend's wire is equally stack-safe). Callers must pass a value
* produced by `JSON.parse` (or equally JSON-plain): only `null`, finite
* numbers, booleans, strings, dense arrays, and plain objects — this encoder
* validates nothing. Output matches compact `JSON.stringify` byte for byte
* EXCEPT on an integral double beyond the safe range, where {@link scalarJson}
* emits the exact integer's BigInt digits rather than `JSON.stringify`'s rounded
* spelling (`1152921504606846976`, not `...847000`) so the seam's lossless-JSON
* promise holds across the wire.
* @param value - a JSON-plain value (e.g. straight from `JSON.parse`).
* @returns the compact JSON encoding.
*/
export function encodeJsonPlain(value: unknown): string {
type Task = { text: string } | { value: unknown }
const chunks: string[] = []
const tasks: Task[] = [{ value }]
for (let task = tasks.pop(); task !== undefined; task = tasks.pop()) {
if ('text' in task) {
chunks.push(task.text)
continue
}
const current = task.value
if (typeof current === 'string') {
chunks.push(JSON.stringify(current))
} else if (Array.isArray(current)) {
chunks.push('[')
tasks.push({ text: ']' })
for (let index = current.length - 1; index >= 0; index--) {
if (index < current.length - 1) tasks.push({ text: ',' })
tasks.push({ value: current[index] })
}
} else if (typeof current === 'object' && current !== null) {
const record = current as Record<string, unknown>
chunks.push('{')
tasks.push({ text: '}' })
const keys = Object.keys(record)
for (let index = keys.length - 1; index >= 0; index--) {
const key = keys[index] as string
if (index < keys.length - 1) tasks.push({ text: ',' })
tasks.push({ value: record[key] })
tasks.push({ text: `${JSON.stringify(key)}:` })
}
} else {
chunks.push(scalarJson(current))
}
}
return chunks.join('')
}
/**
* One scalar (null, boolean, finite number) as JSON text. A beyond-safe-range
* integral double needs BigInt digits: `String(2 ** 60)` emits the ROUNDED
* `...847000` form, and echoing that to the child would silently change the
* integer the seam promised to carry losslessly — `BigInt(2 ** 60)` prints the
* exact `...846976` the double actually holds.
* @param current - a JSON-plain scalar (JSON.parse emits nothing else).
* @returns its JSON encoding.
*/
function scalarJson(current: unknown): string {
if (typeof current === 'number' && Number.isInteger(current) && !Number.isSafeInteger(current)) {
return BigInt(current).toString()
}
return String(current)
}
/**
* Exact UTF-8 byte length of one string's compact JSON form (quotes + escapes),
* computed by a single non-allocating scan that stops the instant the running
* total exceeds `maxBytes`. Used instead of `Buffer.byteLength(JSON.stringify(s))`
* so a control-heavy forged string — whose escaped copy expands up to ~6x — is
* rejected BEFORE that copy is materialized: `JSON.stringify` would allocate the
* full escaped form first, the very hundreds-of-MB spike the metered traversal
* exists to avoid. Mirrors `JSON.stringify`'s escaping byte-for-byte: `"` and
* `\` and the five short C0 escapes cost 2, other C0 controls `\uXXXX` cost 6, a
* valid surrogate pair is one astral code point emitted as raw 4-byte UTF-8, a
* LONE surrogate becomes `\uXXXX` at 6, and any other code point costs its raw
* UTF-8 width.
* @param text - the string to meter.
* @param maxBytes - largest serialized size the caller can still admit.
* @returns the exact serialized byte length, or `undefined` once it exceeds `maxBytes`.
*/
function jsonStringBytesUpTo(text: string, maxBytes: number): number | undefined {
let bytes = 2 // the two quotes
if (bytes > maxBytes) return undefined
for (let index = 0; index < text.length; index++) {
const code = text.charCodeAt(index)
if (code === 0x22 || code === 0x5c || code === 0x08 || code === 0x09 || code === 0x0a || code === 0x0c || code === 0x0d) {
bytes += 2 // `\"` `\\` `\b` `\t` `\n` `\f` `\r`
} else if (code < 0x20) {
bytes += 6 // other C0 controls: `\uXXXX`
} else if (code < 0x80) {
bytes += 1
} else if (code < 0x800) {
bytes += 2
} else if (code >= 0xd800 && code <= 0xdbff && index + 1 < text.length) {
const next = text.charCodeAt(index + 1)
if (next >= 0xdc00 && next <= 0xdfff) {
bytes += 4 // valid high+low pair: one astral code point, raw 4-byte UTF-8
index++
} else {
bytes += 6 // lone high surrogate: `\uXXXX`
}
} else if (code >= 0xd800 && code <= 0xdfff) {
bytes += 6 // lone surrogate (unpaired high at end, or any low): `\uXXXX`
} else {
bytes += 3 // other BMP code point
}
if (bytes > maxBytes) return undefined
}
return bytes
}
/**
* Meter a `JSON.parse`-produced done value's compact-JSON byte length AND its
* number losslessness in one traversal, stopping the instant `maxBytes` is
* crossed. This bounds the INCREMENTAL allocation the check itself would add on
* top of the already-parsed value — the enqueued children; strings and keys are
* metered by {@link jsonStringBytesUpTo} without allocating an escaped copy —
* not the parse that produced `value`.
* That upstream width is bounded separately, by the host-side cap on inbound
* fd-3 frame size before `JSON.parse` runs (owned by the runtime that reads the
* channel), so `value` cannot be arbitrarily large when it reaches here. The
* budget is the `maxValueBytes` the boot frame carries — a required wire field
* with no default at this layer. The traversal rejects over-budget BEFORE
* materializing a string's escaped form or enqueuing an array's/object's
* children, so a forgery within that frame cap cannot force those secondary
* allocations. Object key COUNTING is
* unavoidably O(keys) — JS has no lazy own-key iterator, and the parse already
* built the key set — but the check still refuses the per-entry work before the
* enqueue loop. A non-lossless number (non-finite, negative zero) is caught only
* when the value fits the budget — an over-budget value is rejected regardless,
* so the distinction is moot. Same JSON-plain precondition and traversal shape
* as {@link encodeJsonPlain}; a number's byte length is measured through
* {@link scalarJson} (matching the encoder, so a beyond-safe-range integer
* meters its exact BigInt digits, not `JSON.stringify`'s rounded spelling) and
* a string's/key's through {@link jsonStringBytesUpTo} (the exact escaped size,
* scanned without allocating the escaped copy).
* @param value - a JSON-plain value (e.g. straight from `JSON.parse`).
* @param maxBytes - the completion-value budget in bytes.
* @returns `{ ok: true, bytes }` with the exact serialized size, or
* `{ ok: false, reason }` — `over-budget` once the size exceeds `maxBytes`,
* `non-lossless` on a non-finite or negative-zero number.
*/
export function checkDoneValue(value: unknown, maxBytes: number): { ok: true; bytes: number } | { ok: false; reason: 'over-budget' | 'non-lossless' } {
let bytes = 0
// A non-lossless number is recorded, not returned on sight: over-budget must
// win regardless of where in the value each violation sits, so the whole
// metering finishes first. Otherwise `["<huge>", 1e400]` and `[1e400,
// "<huge>"]` — the same over-budget value in two member orders — would
// classify differently (non-lossless vs over-budget), and the JSDoc promises
// an over-budget value is rejected as over-budget regardless.
let nonLossless = false
const stack: unknown[] = [value]
while (stack.length > 0) {
const current = stack.pop()
if (typeof current === 'number') {
// Flag a non-lossless number but keep counting its encoded bytes: a value
// that is BOTH non-lossless and over-budget must classify as over-budget
// (the loop's byte check below wins), so the byte count cannot skip the
// offending number. `scalarJson` gives the same spelling a legit scalar
// would meter.
if (!Number.isFinite(current) || Object.is(current, -0)) nonLossless = true
bytes += Buffer.byteLength(scalarJson(current), 'utf8')
} else if (typeof current === 'string') {
// Meter the escaped form WITHOUT allocating it: jsonStringBytesUpTo scans
// and bails the instant the running cost crosses the remaining budget, so
// a control-heavy forgery (escaped copy up to ~6x) never materializes that
// copy the way `JSON.stringify` would.
const stringBytes = jsonStringBytesUpTo(current, maxBytes - bytes)
if (stringBytes === undefined) return { ok: false, reason: 'over-budget' }
bytes += stringBytes
} else if (Array.isArray(current)) {
// Brackets plus one comma per gap; elements add themselves. Reject
// BEFORE enqueuing children: every element serializes to at least one
// byte, so a forged flat array far above the budget fails here without
// pushing its elements onto the host stack. (The array itself is already
// materialized by the upstream parse; this only bounds the extra stack.)
bytes += 2 + (current.length > 1 ? current.length - 1 : 0)
if (bytes + current.length > maxBytes) return { ok: false, reason: 'over-budget' }
for (const item of current) stack.push(item)
} else if (typeof current === 'object' && current !== null) {
const record = current as Record<string, unknown>
// Count own keys with for...in + hasOwn. This IS O(keys) — JS has no lazy
// own-key iterator and the parse already built the key set — so the count
// cannot be sublinear; what the bound below buys is refusing the per-entry
// work (key escaping, value enqueue) before it runs. Each entry costs at
// least a quoted key (>= 2 bytes) + colon + >= 1-byte value.
let count = 0
for (const key in record) if (Object.hasOwn(record, key)) count += 1
bytes += 2 + (count > 1 ? count - 1 : 0)
if (bytes + count * 4 > maxBytes) return { ok: false, reason: 'over-budget' }
for (const key in record) {
if (!Object.hasOwn(record, key)) continue
// Meter the key's escaped form without allocating it (same reason as the
// string branch), then add the colon separator. `+ 1` for the `:`.
const keyBytes = jsonStringBytesUpTo(key, maxBytes - bytes)
if (keyBytes === undefined) return { ok: false, reason: 'over-budget' }
bytes += keyBytes + 1
stack.push(record[key])
}
} else {
bytes += Buffer.byteLength(scalarJson(current), 'utf8')
}
if (bytes > maxBytes) return { ok: false, reason: 'over-budget' }
}
// The whole value fit the budget; a recorded number violation is the verdict.
if (nonLossless) return { ok: false, reason: 'non-lossless' }
return { ok: true, bytes }
}
/**
* Whether a raw JSON line contains an integer token that would lose precision
* as a JavaScript number. `JSON.parse` silently rounds such a token
* (`9007199254740993` becomes `...992`) BEFORE any validation can see it, so
* the check must read the source text; a beyond-safe-range token whose double
* parse round-trips exactly (`2**53`, `2**60`) is lossless and passes. The scan walks the line skipping string literals (a digit run
* inside a string is data, not a number token) and tests every number token
* in plain integer form — no fraction or exponent, which parse as doubles by
* intent. A reviver cannot do this job: the reviver walk recurses per nesting
* level and would reintroduce the depth limit `encodeJsonPlain` removes.
* @param line - the raw UTF-8 text of one JSON-lines frame.
* @returns true when an unsafe integer token is present outside strings.
*/
export function hasUnsafeIntegerToken(line: string): boolean {
for (let index = 0; index < line.length; index++) {
const char = line[index]
if (char === '"') {
// Skip the string literal, honoring backslash escapes.
for (index++; index < line.length; index++) {
if (line[index] === '\\') index++
else if (line[index] === '"') break
}
continue
}
if (char === '-' || (char !== undefined && char >= '0' && char <= '9')) {
let end = index + 1
while (end < line.length) {
const c = line[end] as string
if ((c >= '0' && c <= '9') || c === '.' || c === 'e' || c === 'E' || c === '+' || c === '-') end++
else break
}
const token = line.slice(index, end)
// Beyond the safe range an integer token is still lossless IFF the
// double parse round-trips exactly (2**53 does; 2**53+1 rounds) — the
// canonical boundary accepts every JS-double-exact value, so only a
// genuinely rounding token marks the frame as forged.
if (/^-?\d+$/.test(token)) {
const parsed = Number(token)
// A token that parses to Infinity is trivially lossy; a finite
// beyond-safe-range one is lossy only when the BigInt round-trip
// disagrees.
if (!Number.isFinite(parsed)) return true
if (!Number.isSafeInteger(parsed) && BigInt(token) !== BigInt(parsed)) return true
}
index = end - 1
}
}
return false
}
/**
* Lazily yield one plain object's own enumerable property values. A generator
* (not `Object.values`/`Object.entries`) because {@link hasNonLosslessNumber}
* walks breadth it cannot bound: those helpers copy the whole VALUE (or
* key/value pair) list into a fresh array up front, so a wide object would cost
* that second full-breadth allocation before a single value is examined. The
* `for...in` here does not make the walk sublinear — V8 still materializes the
* key-name enumeration when the loop starts — but it avoids the extra value
* array, yielding each value straight off the already-parsed object.
* @param record - a JSON-parse-produced object.
* @yields each own enumerable property value, in key order.
*/
function* ownValues(record: object): Generator {
for (const key in record) {
if (Object.hasOwn(record, key)) yield (record as Record<string, unknown>)[key]
}
}
/**
* Whether a JSON.parse-produced value contains a number outside lossless
* JSON: non-finite (`1e400` parses to `Infinity`) or negative zero (`-0.0`
* parses to JS `-0`, whose sign bit a re-serialization drops). The honest
* child's validator rejects these before sending, so a frame carrying one is
* forged.
*
* Runs on `call.args`, which — unlike a completion value — has NO seam byte
* cap, so there is no budget to reject a wide payload against the way
* {@link checkDoneValue} does. The traversal therefore holds ONE cursor per
* NESTING LEVEL (an array or {@link ownValues} iterator) instead of one entry
* per member: a forged flat `args` at the top of the host's inbound frame-size
* cap would
* otherwise push tens of millions of stack entries — and `Object.values` would
* copy each object's full breadth — allocating hundreds of megabytes beyond
* what `JSON.parse` already holds. Iterative either way, so a deep frame
* cannot overflow the host stack.
* @param value - a JSON-parse-produced value from an fd-3 frame.
* @returns true when any contained number is non-finite or negative zero.
*/
export function hasNonLosslessNumber(value: unknown): boolean {
const cursors: Iterator<unknown>[] = [[value].values()]
while (cursors.length > 0) {
// The loop condition guarantees a top cursor.
const cursor = cursors.at(-1) as Iterator<unknown>
const step = cursor.next()
if (step.done === true) {
cursors.pop()
continue
}
const current = step.value
if (typeof current === 'number') {
if (!Number.isFinite(current) || Object.is(current, -0)) return true
} else if (Array.isArray(current)) {
cursors.push((current as unknown[]).values())
} else if (typeof current === 'object' && current !== null) {
cursors.push(ownValues(current))
}
}
return false
}
/**
* Runtime shape gate for inbound fd-3 traffic. Model code has full access to
* fd 3 and can post anything — `null`, primitives, poisoned fields — so the
* compile-time union means nothing here: every field is validated and REBUILT
* before the host reads it (forged extras never ride along; a non-number id
* can never be echoed into a reply). Junk returns `undefined` and is dropped
* so a throw in the host's `message` handler cannot crash the host process.
* @param raw - one JSON-parsed frame from fd 3.
* @returns the rebuilt frame, or `undefined` to drop it silently.
*/
export function validateChildFrame(raw: unknown): ChildToHost | undefined {
if (typeof raw !== 'object' || raw === null) return undefined
const m = raw as Record<string, unknown>
switch (m.type) {
case 'boot-ack':
return { type: 'boot-ack' }
case 'log':
if (typeof m.text !== 'string') return undefined
// Rebuilt, not passed through: a forged `truncated` of any other type
// would reach the host as a truthy value and silence capture for the
// rest of the run. Only the literal `true` counts.
return { type: 'log', text: m.text, ...m.truncated === true ? { truncated: true } : {} }
case 'call': {
// The id must be a finite number: it is echoed verbatim into the reply
// frame, and a forged `1e400` id (Infinity after JSON.parse) would make
// the reply unencodable as strict JSON. Negative zero is rejected too:
// it passes `Number.isFinite`, but the reply re-serializes it as `0`
// (`JSON.stringify({id:-0})` is `{"id":0}`), colliding with a real call
// whose id is `0` — the honest child never issues `-0`.
if (typeof m.id !== 'number' || !Number.isFinite(m.id) || Object.is(m.id, -0) || typeof m.global !== 'string' || typeof m.name !== 'string') return undefined
// A forged frame can omit `args` entirely; rebuilding it as `undefined`
// would invoke the binding with a non-JSON value, bypassing the
// lossless-JSON argument boundary. Any PRESENT value is JSON-plain by
// construction (the frame came from JSON.parse), so presence is the
// whole check.
if (!Object.hasOwn(m, 'args')) return undefined
// JSON.parse yields Infinity for 1e400 and preserves -0; both are
// outside lossless JSON, and the honest child never sends them.
if (hasNonLosslessNumber(m.args)) return undefined
return { type: 'call', id: m.id, global: m.global, name: m.name, args: m.args }
}
case 'done': {
// The value passes through untouched here: scanning it for non-lossless
// numbers would push every member of a wide forged payload before any
// byte cap runs. The done handler's bounded `checkDoneValue` folds the
// losslessness check into the metered traversal, rejecting over-budget
// before it enqueues children.
const err = m.error
if (err === undefined) {
return m.value === undefined ? { type: 'done' } : { type: 'done', value: m.value }
}
if (typeof err !== 'object' || err === null) return undefined
const { kind, message } = err as Record<string, unknown>
if (typeof message !== 'string') return undefined
if (kind !== 'exception' && kind !== 'invalid-output' && kind !== 'output-limit') return undefined
return m.value === undefined
? { type: 'done', error: { kind, message } }
: { type: 'done', value: m.value, error: { kind, message } }
}
default:
return undefined
}
}
@@ -0,0 +1,103 @@
import { execFile } from 'node:child_process'
import { existsSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import { promisify } from 'node:util'
import { describe, expect, it } from 'vitest'
import { logTruncationMarker, PROTOCOL_FD, WIRE_FRAME_FIELDS } from '../src/protocol.ts'
/**
* Cross-language mirror check between `src/protocol.ts` and `py/protocol.py`,
* spawning a real `python3` to read the Python side. Two things are asserted:
* the runtime surfaces both sides EXECUTE against — `PROTOCOL_FD` and the log
* truncation marker text, where a drift silently corrupts a live run — and the
* per-frame wire field sets (required/optional keys of each `TypedDict`), which
* turns the otherwise review-only shape mirror into an executable check that
* catches the round-12 kind of drift (a renamed/dropped field, or one side
* making a field optional the other requires). Self-skips when no `python3` is
* on PATH — CI provides one; the pure-TS `protocol.spec.ts` covers the host
* codec unconditionally.
*/
const execFileAsync = promisify(execFile)
const pyDir = fileURLToPath(new URL('../py', import.meta.url))
// `-B` blocks bytecode writes into the source tree (`py/__pycache__/*.pyc`);
// `-I` isolates the interpreter but does not imply it.
const python3Flags = ['-I', '-B']
async function hasPython3(): Promise<boolean> {
try {
await execFileAsync('python3', ['--version'])
return true
} catch {
return false
}
}
const python3Available = await hasPython3()
describe.skipIf(!python3Available)('protocol.py mirrors protocol.ts at runtime', () => {
it('agrees on PROTOCOL_FD and the log truncation marker across byte budgets', async () => {
const budgets = [1, 65536, 1048576]
const probe = [
'import json, sys',
`sys.path.insert(0, ${JSON.stringify(pyDir)})`,
'from protocol import PROTOCOL_FD, log_truncation_marker',
`budgets = ${JSON.stringify(budgets)}`,
'print(json.dumps({',
' "fd": PROTOCOL_FD,',
' "markers": [log_truncation_marker(b) for b in budgets],',
'}))',
].join('\n')
const { stdout } = await execFileAsync('python3', [...python3Flags, '-c', probe])
const seen = JSON.parse(stdout) as { fd: number; markers: string[] }
// Assert against the TS-side PROTOCOL_FD export (the value the host wires),
// not a bare literal, so a drift on either side of the wire is caught here.
expect(seen.fd).toBe(PROTOCOL_FD)
expect(seen.markers).toEqual(budgets.map(budget => logTruncationMarker(budget)))
})
it('agrees on every frame type\'s wire field set between the TS and Python declarations', async () => {
// Turn the TypedDict mirror from a review-only obligation into an executable
// check: enumerate EVERY TypedDict in py/protocol.py (public names carrying
// __required_keys__) and assert both the frame roster and each frame's
// required/optional key sets against WIRE_FRAME_FIELDS — projected from the
// WIRE_FRAME_FIELD_ROLES map that `satisfies` binds exhaustively to the
// frame interfaces in protocol.ts. Together this catches drift on EITHER
// side of the wire: a TS-side field add, remove, rename, or optionality flip
// fails typecheck at the roles map; a Python frame added, removed, or with a
// changed field set fails this comparison. `global` is the reserved-keyword
// wire key the Python side carries via a functional TypedDict.
const probe = [
'import json, sys',
`sys.path.insert(0, ${JSON.stringify(pyDir)})`,
'import protocol as p',
'def keys(td): return {"required": sorted(td.__required_keys__), "optional": sorted(td.__optional_keys__)}',
// Every public TypedDict in the module — not a name list from the TS side,
// so a Python-only extra frame is visible here.
'frames = {n: keys(v) for n, v in vars(p).items()'
+ ' if not n.startswith("_") and hasattr(v, "__required_keys__")}',
'print(json.dumps(frames))',
].join('\n')
const { stdout } = await execFileAsync('python3', [...python3Flags, '-c', probe])
const seen = JSON.parse(stdout) as Record<string, { required: string[]; optional: string[] }>
// Normalize the TS source of truth to the same sorted shape Python reports.
const expected = Object.fromEntries(
Object.entries(WIRE_FRAME_FIELDS).map(([name, sets]) => [
name,
{ required: [...sets.required].sort(), optional: [...sets.optional].sort() },
]),
)
// Same frame roster on both sides (catches a frame present on only one),
// then identical field sets per frame.
expect(Object.keys(seen).sort()).toEqual(Object.keys(expected).sort())
expect(seen).toEqual(expected)
})
})
it('names the py/ directory that ships with the package', () => {
// Resolves py/ relative to this test file; the same directory ships in the
// package.json `files` whitelist (`py/**/*.py`). The tests/ directory itself
// is not published — this asserts the source-tree layout the mirror test
// depends on, so it holds even when python3 is absent from the runner.
expect(existsSync(pyDir)).toBe(true)
})
@@ -0,0 +1,304 @@
import { describe, expect, it } from 'vitest'
import { checkDoneValue, encodeJsonPlain, hasNonLosslessNumber, hasUnsafeIntegerToken, logTruncationMarker, validateChildFrame } from '../src/index.ts'
describe('logTruncationMarker', () => {
it('names the configured byte budget', () => {
expect(logTruncationMarker(65536)).toBe('[dsh-code-runtime-python] log capture truncated at 65536 bytes')
expect(logTruncationMarker(1)).toBe('[dsh-code-runtime-python] log capture truncated at 1 bytes')
})
})
describe('validateChildFrame', () => {
it('rebuilds boot-ack frames without extra fields', () => {
expect(validateChildFrame({ type: 'boot-ack' })).toEqual({ type: 'boot-ack' })
// Forged extras never ride along.
expect(validateChildFrame({ type: 'boot-ack', extra: 'x' })).toEqual({ type: 'boot-ack' })
})
it('rebuilds log frames when the text field is a string', () => {
expect(validateChildFrame({ type: 'log', text: 'hi' })).toEqual({ type: 'log', text: 'hi' })
// Non-string text drops.
expect(validateChildFrame({ type: 'log', text: 42 })).toBeUndefined()
expect(validateChildFrame({ type: 'log' })).toBeUndefined()
})
it('carries a log frame truncation flag only for the literal true', () => {
// The child's own ledger marker sets `truncated: true`; the host rebuilds
// it so it stops capturing at the same point.
expect(validateChildFrame({ type: 'log', text: 'x', truncated: true }))
.toEqual({ type: 'log', text: 'x', truncated: true })
// Any other truthy or non-boolean value is a forgery and is dropped from
// the rebuild — otherwise it would silence capture for the rest of the run.
expect(validateChildFrame({ type: 'log', text: 'x', truncated: 1 })).toEqual({ type: 'log', text: 'x' })
expect(validateChildFrame({ type: 'log', text: 'x', truncated: 'yes' })).toEqual({ type: 'log', text: 'x' })
expect(validateChildFrame({ type: 'log', text: 'x', truncated: false })).toEqual({ type: 'log', text: 'x' })
})
it('rebuilds call frames with a numeric id, string global, and string name', () => {
expect(validateChildFrame({ type: 'call', id: 1, global: 'tools', name: 'echo', args: { x: 1 } }))
.toEqual({ type: 'call', id: 1, global: 'tools', name: 'echo', args: { x: 1 } })
// A frame with NO args key drops whole: rebuilding it as `undefined`
// would invoke the binding with a non-JSON value, bypassing the
// lossless-JSON argument boundary. Any present value is JSON-plain by
// construction (frames arrive via JSON.parse), so null passes.
expect(validateChildFrame({ type: 'call', id: 2, global: 'tools', name: 'echo' })).toBeUndefined()
expect(validateChildFrame({ type: 'call', id: 2, global: 'tools', name: 'echo', args: null }))
.toEqual({ type: 'call', id: 2, global: 'tools', name: 'echo', args: null })
// A missing/mistyped required field drops.
expect(validateChildFrame({ type: 'call', id: '1', global: 'tools', name: 'echo' })).toBeUndefined()
expect(validateChildFrame({ type: 'call', id: 1, global: 7, name: 'echo' })).toBeUndefined()
expect(validateChildFrame({ type: 'call', id: 1, global: 'tools' })).toBeUndefined()
})
it('rebuilds done frames with optional value/error', () => {
expect(validateChildFrame({ type: 'done' })).toEqual({ type: 'done' })
expect(validateChildFrame({ type: 'done', value: 42 })).toEqual({ type: 'done', value: 42 })
expect(validateChildFrame({ type: 'done', error: { kind: 'exception', message: 'boom' } }))
.toEqual({ type: 'done', error: { kind: 'exception', message: 'boom' } })
expect(validateChildFrame({ type: 'done', error: { kind: 'invalid-output', message: 'lossy' } }))
.toEqual({ type: 'done', error: { kind: 'invalid-output', message: 'lossy' } })
expect(validateChildFrame({ type: 'done', error: { kind: 'output-limit', message: 'big' } }))
.toEqual({ type: 'done', error: { kind: 'output-limit', message: 'big' } })
expect(validateChildFrame({ type: 'done', value: 1, error: { kind: 'exception', message: 'boom' } }))
.toEqual({ type: 'done', value: 1, error: { kind: 'exception', message: 'boom' } })
// A `value: undefined` field is dropped (JSON never carries it, but a forged
// shape might; the rebuild coalesces to the absent case).
expect(validateChildFrame({ type: 'done', value: undefined })).toEqual({ type: 'done' })
// A missing or unrecognized kind drops the frame: the child always sends
// one of the three, so anything else is a forgery.
expect(validateChildFrame({ type: 'done', error: { message: 'boom' } })).toBeUndefined()
expect(validateChildFrame({ type: 'done', error: { kind: 'timeout', message: 'x' } })).toBeUndefined()
})
it('rejects malformed done frames', () => {
// error must be an object.
expect(validateChildFrame({ type: 'done', error: 'boom' })).toBeUndefined()
expect(validateChildFrame({ type: 'done', error: null })).toBeUndefined()
// error.message must be a string.
expect(validateChildFrame({ type: 'done', error: {} })).toBeUndefined()
expect(validateChildFrame({ type: 'done', error: { message: 42 } })).toBeUndefined()
})
it('drops non-object inputs and unknown types silently', () => {
expect(validateChildFrame(null)).toBeUndefined()
expect(validateChildFrame(undefined)).toBeUndefined()
expect(validateChildFrame(42)).toBeUndefined()
expect(validateChildFrame('str')).toBeUndefined()
expect(validateChildFrame({})).toBeUndefined()
expect(validateChildFrame({ type: 'unknown' })).toBeUndefined()
})
it('drops CALL frames whose args are non-finite or negative zero', () => {
// JSON.parse turns 1e400 into Infinity and preserves -0; the honest child
// rejects both before sending, so a call frame carrying one is forged.
expect(validateChildFrame({ type: 'call', id: 1, global: 'tools', name: 'x', args: { n: Infinity } })).toBeUndefined()
expect(validateChildFrame({ type: 'call', id: Infinity, global: 'tools', name: 'x', args: null })).toBeUndefined()
// Plain zero and ordinary floats pass.
expect(validateChildFrame({ type: 'call', id: 1, global: 'tools', name: 'x', args: [0, 1.5] }))
.toEqual({ type: 'call', id: 1, global: 'tools', name: 'x', args: [0, 1.5] })
})
it('drops a CALL frame whose id is negative zero', () => {
// `-0` passes Number.isFinite, but the reply re-serializes it as `0`
// (JSON.stringify({id:-0}) === '{"id":0}'), so a forged `-0` id would
// collide with a real call whose id is `0`. The honest child never sends it.
expect(validateChildFrame({ type: 'call', id: -0, global: 'tools', name: 'x', args: null })).toBeUndefined()
// Plain positive zero is a legitimate id and passes.
expect(validateChildFrame({ type: 'call', id: 0, global: 'tools', name: 'x', args: null }))
.toEqual({ type: 'call', id: 0, global: 'tools', name: 'x', args: null })
})
it('passes DONE values through untouched — losslessness is metered later', () => {
// validateChildFrame no longer scans done.value: an unbounded scan would
// push every member of a wide forged payload before any byte cap ran. The
// done handler's checkDoneValue folds losslessness into the metered walk.
expect(validateChildFrame({ type: 'done', value: Infinity })).toEqual({ type: 'done', value: Infinity })
expect(validateChildFrame({ type: 'done', value: [{ x: -0 }] })).toEqual({ type: 'done', value: [{ x: -0 }] })
expect(validateChildFrame({ type: 'done', value: [0, 1.5] })).toEqual({ type: 'done', value: [0, 1.5] })
})
})
describe('lossless-number scan', () => {
it('finds non-finite and negative-zero numbers at any depth, iteratively', () => {
expect(hasNonLosslessNumber(Infinity)).toBe(true)
expect(hasNonLosslessNumber(-Infinity)).toBe(true)
expect(hasNonLosslessNumber(NaN)).toBe(true)
expect(hasNonLosslessNumber(-0)).toBe(true)
expect(hasNonLosslessNumber({ a: [1, { b: -0 }] })).toBe(true)
expect(hasNonLosslessNumber({ a: [0, 1.5, 'x', null, true] })).toBe(false)
// Deep nesting must not overflow the stack.
let deep: unknown = 0
for (let i = 0; i < 100000; i++) deep = [deep]
expect(hasNonLosslessNumber(deep)).toBe(false)
})
it('walks wide arrays and objects one member at a time', () => {
// `call.args` carries no seam byte cap, so a wide forged payload has no
// budget to be rejected against — the walk must hold one cursor per
// NESTING LEVEL, not one entry per member, or a flat payload at the top of
// the host's inbound frame-size cap would allocate tens of millions of stack
// entries (and `Object.values` a second full-breadth copy). Observable
// through the boundary: a wide payload whose per-member cost the old shape
// would have paid still scans, and a violation ANYWHERE in it is found
// wherever it sits.
const wideArray = new Array(2_000_000).fill(0) as unknown[]
expect(hasNonLosslessNumber(wideArray)).toBe(false)
// Last element, so the cursor must run the whole breadth lazily.
wideArray[wideArray.length - 1] = -0
expect(hasNonLosslessNumber(wideArray)).toBe(true)
const wideObject: Record<string, unknown> = {}
for (let i = 0; i < 200_000; i++) wideObject[`k${i}`] = i
expect(hasNonLosslessNumber(wideObject)).toBe(false)
wideObject.last = Infinity
expect(hasNonLosslessNumber(wideObject)).toBe(true)
// Interleaved nesting: a per-level cursor must resume its parent after a
// child level ends, so a violation after a nested container is still seen.
expect(hasNonLosslessNumber([[1], { a: 2 }, NaN])).toBe(true)
})
it('scans only own enumerable properties', () => {
// The per-level cursor filters own keys (a prototype-carrying frame is
// impossible off JSON.parse, but the filter is what keeps the walk equal
// to what the encoder would serialize).
const withProto = Object.create({ inherited: -0 }) as Record<string, unknown>
withProto.own = 1
expect(hasNonLosslessNumber(withProto)).toBe(false)
})
})
describe('unsafe-integer token scan', () => {
it('flags integer tokens outside the safe range, skipping strings and float forms', () => {
expect(hasUnsafeIntegerToken('{"v":9007199254740993}')).toBe(true)
// Exact beyond-safe-range tokens are lossless and pass (2**53, 2**64).
expect(hasUnsafeIntegerToken('{"v":9007199254740992}')).toBe(false)
expect(hasUnsafeIntegerToken('{"v":18446744073709551616}')).toBe(false)
// A token that parses to Infinity is trivially lossy.
expect(hasUnsafeIntegerToken(`{"v":${'9'.repeat(400)}}`)).toBe(true)
expect(hasUnsafeIntegerToken('{"v":-9007199254740993}')).toBe(true)
expect(hasUnsafeIntegerToken('{"v":9007199254740991}')).toBe(false)
expect(hasUnsafeIntegerToken('{"v":"9007199254740993"}')).toBe(false)
expect(hasUnsafeIntegerToken(String.raw`{"v":"esc\"9007199254740993"}`)).toBe(false)
expect(hasUnsafeIntegerToken('{"v":9007199254740993.0}')).toBe(false)
expect(hasUnsafeIntegerToken('{"v":9e99}')).toBe(false)
})
})
describe('checkDoneValue', () => {
it('matches the exact encoded size and rejects one byte over', () => {
const cases: unknown[] = [null, true, false, 0, -1.5, 'a"b\\', [], {}, [1, 'x', null], { a: [1, 2], b: { c: 'd' } }]
for (const value of cases) {
const exact = Buffer.byteLength(JSON.stringify(value), 'utf8')
expect(checkDoneValue(value, exact), JSON.stringify(value)).toEqual({ ok: true, bytes: exact })
expect(checkDoneValue(value, exact - 1), JSON.stringify(value)).toEqual({ ok: false, reason: 'over-budget' })
expect(encodeJsonPlain(value)).toBe(JSON.stringify(value))
}
})
it('rejects an over-budget value before its secondary allocations', () => {
// A huge string is refused on the cheap length lower bound, before its
// escaped copy is built.
const huge = { data: 'x'.repeat(1_000_000), tail: 'y' }
expect(checkDoneValue(huge, 1024)).toEqual({ ok: false, reason: 'over-budget' })
// A flat array far above the budget fails on the brackets+length bound,
// before its elements are pushed onto the traversal stack. (The array is
// already materialized by the upstream parse; this only avoids the extra
// per-element stack growth.)
const flat = new Array(10_000_000).fill(0)
expect(checkDoneValue(flat, 1024)).toEqual({ ok: false, reason: 'over-budget' })
// A wide object: braces+commas fit the cap, but the per-entry lower bound
// (quoted key + colon + value = count*4) does not, so it fails before any
// key is escaped or any value enqueued.
const wide: Record<string, number> = {}
for (let i = 0; i < 10; i++) wide[`k${i}`] = i
expect(checkDoneValue(wide, 12)).toEqual({ ok: false, reason: 'over-budget' })
})
it('meters a string\'s exact escaped size without allocating it', () => {
// A control-heavy string that fits by DECODED length but not once escaped
// must still reject: 200 NULs are 200 UTF-16 units (would pass a naive
// length bound against cap 1024) but escape to 200*6 + 2 = 1202 bytes.
// jsonStringBytesUpTo scans and bails before the escaped copy is built.
expect(checkDoneValue('\0'.repeat(200), 1024)).toEqual({ ok: false, reason: 'over-budget' })
// Exact-size acceptance, no false rejection: one NUL serializes to a
// 6-char \\uXXXX escape, so with the two quotes = 8 bytes.
expect(checkDoneValue('\0', 8)).toEqual({ ok: true, bytes: 8 })
expect(checkDoneValue('\0', 7)).toEqual({ ok: false, reason: 'over-budget' })
// Multi-byte and astral characters meter at their raw UTF-8 width (a valid
// surrogate pair is 4 bytes, matching JSON.stringify), not a 6-byte escape.
expect(checkDoneValue('\u00e9', 4)).toEqual({ ok: true, bytes: 4 }) // 2 quotes + 2-byte UTF-8
expect(checkDoneValue('\u{1f600}', 6)).toEqual({ ok: true, bytes: 6 }) // 2 quotes + 4-byte UTF-8
expect(checkDoneValue('\u{1f600}', 5)).toEqual({ ok: false, reason: 'over-budget' })
// A lone surrogate escapes to \\uXXXX = 6, so with quotes = 8.
expect(checkDoneValue('\ud800', 8)).toEqual({ ok: true, bytes: 8 })
// A high surrogate followed by a NON-low character is a lone surrogate (6-byte
// escape) plus that character: `\ud800` + `a` = 2 quotes + 6 + 1 = 9.
expect(checkDoneValue('\ud800a', 9)).toEqual({ ok: true, bytes: 9 })
// A BMP 3-byte code point (CJK) meters at its raw UTF-8 width: 2 quotes + 3.
expect(checkDoneValue('中', 5)).toEqual({ ok: true, bytes: 5 })
// Same non-allocating meter for object keys, before the value is enqueued.
expect(checkDoneValue({ ['\0'.repeat(200)]: 1 }, 1024)).toEqual({ ok: false, reason: 'over-budget' })
// A string reached with less than the two quotes' worth of budget is refused
// immediately (even the empty escaped form does not fit).
expect(checkDoneValue('x', 1)).toEqual({ ok: false, reason: 'over-budget' })
})
it('meters only own enumerable keys', () => {
// The walk counts keys with a `for...in` + hasOwn pass rather than
// Object.keys/entries (which allocate per member before the bound). A
// prototype-carrying forgery is impossible off JSON.parse, but the own-key
// filter is what keeps the count equal to the encoder's.
const withProto = Object.create({ inherited: 'x' }) as Record<string, unknown>
withProto.own = 1
expect(checkDoneValue(withProto, 1024)).toEqual({ ok: true, bytes: Buffer.byteLength('{"own":1}', 'utf8') })
})
it('rejects non-finite and negative-zero numbers at any depth as non-lossless', () => {
expect(checkDoneValue(Infinity, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
expect(checkDoneValue(-Infinity, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
expect(checkDoneValue(NaN, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
expect(checkDoneValue(-0, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
expect(checkDoneValue({ a: [1, { b: -0 }] }, 1024)).toEqual({ ok: false, reason: 'non-lossless' })
// An ordinary finite value within budget passes with its exact byte count.
const clean = { a: [0, 1.5, 'x', null, true] }
expect(checkDoneValue(clean, 1024)).toEqual({ ok: true, bytes: Buffer.byteLength(JSON.stringify(clean), 'utf8') })
})
it('classifies an over-budget value as over-budget regardless of member order', () => {
// A value that is BOTH over-budget and non-lossless must reject as
// over-budget whichever member the walk reaches first — the non-lossless
// number is recorded and metering finishes, so the two orders below (the
// same value) cannot classify differently. Cap 100 with a 1000-char string.
const big = 'x'.repeat(1000)
expect(checkDoneValue([big, Infinity], 100)).toEqual({ ok: false, reason: 'over-budget' })
expect(checkDoneValue([Infinity, big], 100)).toEqual({ ok: false, reason: 'over-budget' })
// A non-lossless number that DOES fit the budget still rejects as
// non-lossless (the recorded violation is the verdict once the whole value
// is confirmed within budget).
expect(checkDoneValue([Infinity], 100)).toEqual({ ok: false, reason: 'non-lossless' })
// The non-lossless number's OWN encoded bytes still count toward the budget,
// so a value whose only over-budget contribution is the non-lossless number
// itself is classified over-budget, not non-lossless. `[Infinity]` encodes
// as the 10-byte `[Infinity]`; at cap 3 the byte check wins.
expect(checkDoneValue([Infinity], 3)).toEqual({ ok: false, reason: 'over-budget' })
expect(checkDoneValue(Infinity, 3)).toEqual({ ok: false, reason: 'over-budget' })
})
it('meters and encodes deep nesting iteratively without overflowing the stack', () => {
let deep: unknown = 0
for (let i = 0; i < 100_000; i++) deep = [deep]
// 100000 '[' + '0' + 100000 ']' = 200001 bytes.
expect(checkDoneValue(deep, 1_000_000)).toEqual({ ok: true, bytes: 200_001 })
// encodeJsonPlain's headline contract is the same stack-safety (JSON.stringify
// recurses per level and throws RangeError a few thousand deep), so exercise
// it on the same 100k-deep value — JSON.stringify would throw here.
expect(encodeJsonPlain(deep)).toBe(`${'['.repeat(100_000)}0${']'.repeat(100_000)}`)
})
it('emits exact digits for beyond-safe integral doubles', () => {
// String(2**60) prints the ROUNDED ...847000; echoing that to the child
// would change the integer. BigInt digits give the exact ...846976.
const v = JSON.parse('[1152921504606846976]') as unknown
expect(encodeJsonPlain(v)).toBe('[1152921504606846976]')
expect(checkDoneValue(v, 100)).toEqual({ ok: true, bytes: Buffer.byteLength('[1152921504606846976]', 'utf8') })
})
})
@@ -0,0 +1,21 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": [
{
"path": "../../../vendor/cosmokit"
},
{
"path": "../../../vendor/cordis"
},
{
"path": "../../runtime-diagnostics/invariants"
}
]
}
@@ -0,0 +1,16 @@
import { defineConfig } from 'tsdown'
/**
* Single ESM bundle. The Python-side code is not TypeScript and ships verbatim
* under `py/` (whitelisted in package.json `files`) — no build step needed.
*/
export default defineConfig({
entry: ['lib/types/index.js', 'lib/types/invariant.js'],
outDir: 'lib',
format: ['esm'],
platform: 'node',
target: 'es2024',
fixedExtension: false,
dts: false,
clean: false,
})