diff --git a/.agents/notes/implemented/architecture/2026-08-20-webworker-node-face.i18n.yaml b/.agents/notes/implemented/architecture/2026-08-20-webworker-node-face.i18n.yaml new file mode 100644 index 0000000000..4b5a98a8d5 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-08-20-webworker-node-face.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-08-20-webworker-node-face.md +2026-08-20-webworker-node-face.md: 08119cce96eff244f8e9ada3462ce5d35c1b538d +2026-08-20-webworker-node-face.zh.md: 573c0be055d066d2d6d0db11a2476ba528517727 diff --git a/.agents/notes/implemented/architecture/2026-08-20-webworker-node-face.md b/.agents/notes/implemented/architecture/2026-08-20-webworker-node-face.md new file mode 100644 index 0000000000..08119cce96 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-08-20-webworker-node-face.md @@ -0,0 +1,34 @@ +# Agent Note: the worker's Node face — builtins, VFS, and the shell process layer + +Status: implemented + +English | [中文](2026-08-20-webworker-node-face.zh.md) + +## Problem + +The worker runs the web profile's Cordis configuration byte for byte — no worker-specific rows — so a browser's missing platform must be replaced at the module layer, where a proxied module keeps its identity and changes its implementation. That covers three fronts: the Node builtins the tree imports, the filesystem those builtins answer from, and a process layer for the bash tool, which mounted, advertised itself to the model, and then failed on every call while `node:child_process` was a structural stub. + +## Decision + +**Builtins.** The proxy table replaces Node builtins and external npm packages, never workspace or vendored modules. `./implemented/.ts` carries real semantics over a worker data source; `./mock/.ts` mounts silently and reports the missing capability when a call reaches it. The loader's table holds one memoized thunk per specifier — evaluation happens at first `require`, not at assembly — and each shim's exported face typechecks against Node's own module type, with the narrow, documented exceptions where structural identity (a real class) cannot be satisfied. The worker installs the `process` global itself and fills it into the table at assembly. + +**VFS.** Memory is the truth. `statSync(path, { bigint: true })` returns Node's BigInt shape, and two fields carry real information because `dsh-fs-local`'s stale-write guard depends on them: `ino` is per-path identity from a monotonic counter (a recreated path reports a new identity), and `mtimeMs` is strictly increasing per entry (`max(now, previous + 1)`), because in-memory writes routinely land in one millisecond and an equal timestamp would let a stale overwrite pass. The hunt that produced this also fixed the silence around it: cordis's logger verbosity counts UP, so an exporter that declares no level drops every warning — `startWorkerHost` installs a console exporter with `levels: { default: 2 }` before any entry mounts. + +**Shell.** `node:child_process` is a real implementation over the VFS. The grammar is bought — `@yarnpkg/parsers`' `parseShell` — and the evaluator and command table are owned, because every candidate interpreter brings its own filesystem: pipelines are strings handed along, and each program is a function over the VFS. The table is the machine's whole `/bin`; an absent name reports `command not found` (127). Each `spawn` starts a child Web Worker from this same bundle, its first frame declaring the shell-process role, so the termination ladder is real: `SIGTERM` asks at the next command boundary, `SIGKILL` terminates the worker mid-loop — the preemption an in-thread interpreter can never have. The filesystem face is asynchronous end to end (child frames to the host VFS); `execSync`, `execFileSync`, and `fork` refuse, and `node-pty` stays a stub. + +## Alternatives considered + +**Replacing `dsh-subprocess-local` or the bash executor.** The first would let the proxy table replace a workspace package against its own classification and invert the layering; the second trips `dsh-permission-presets`' boot-time `sandboxMode` validation and drops tested timeout/output behavior. + +**`@yarnpkg/shell`, WASM shells, WebContainer.** The matching interpreter is built on real Node streams (~1.5 MB closure to own); WASM was removed from this deployment by decision and WASI has no `fork`; all of them arrive with their own filesystem, the one part that cannot be reused. + +**`SharedArrayBuffer` + `Atomics.wait` for a synchronous child filesystem.** Measured on the deployment target: without COOP/COEP headers `SharedArrayBuffer` is not defined, and GitHub Pages cannot set response headers. The asynchronous face is a superset; a SAB backend can slot under it later without touching a program. + +**Fabricating stats or widening error predicates instead of honoring `bigint`.** Constant `ino`/wall-clock `mtimeNs` silently disable the stale-write guard; swallowing `FS_IO_ERROR` in skill discovery would have made the same bug a permanently empty catalog with no failure anywhere. + +## Consequences + +- Sandbox modes other than `danger-full-access` fail loud: `SandboxEnforcement` has no "nothing was enforced" value and a browser has no kernel, so `ctx.sandbox.confine` fails closed and the command never starts. Real enforcement at the VFS frame gate is a designed follow-up, not this note. +- The Node-host ladder test (`tests/node/child-process.spec.ts`) is registered windows-unsupported: the ladder's win32 kill rung is taskkill-by-real-pid, undeliverable to a process-table pid, while the worker itself always reports `linux`. +- Output is incremental but not streamed: programs write into sinks forwarded as `data` events, and a pipeline stage completes before the next starts. +- The runtime's tests mirror `src/` (`tests/node/`, `tests/shell/`, `tests/storage/`, …), so each shim family owns its behavior cases beside the oracle-diff suites. diff --git a/.agents/notes/implemented/architecture/2026-08-20-webworker-node-face.zh.md b/.agents/notes/implemented/architecture/2026-08-20-webworker-node-face.zh.md new file mode 100644 index 0000000000..573c0be055 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-08-20-webworker-node-face.zh.md @@ -0,0 +1,34 @@ +# Agent Note：worker 的 Node 面——builtin、VFS 与 shell 进程层 + +状态：已实施 + +[English](2026-08-20-webworker-node-face.md) | 中文 + +## 问题 + +worker 逐字节运行 web profile 的 Cordis 配置——没有 worker 专属行——因此浏览器缺失的平台必须在模块层被替换：被代理的模块保持身份、更换实现。这覆盖三条战线：树所 import 的 Node builtin、这些 builtin 背后应答的文件系统，以及 bash 工具的进程层——在 `node:child_process` 还是结构桩的时期，工具照常挂载、向模型自我宣告，然后每次调用都失败。 + +## 决定 + +**Builtin。** 代理表只替换 Node builtin 与外部 npm 包，绝不替换 workspace 或 vendored 模块。`./implemented/.ts` 在 worker 数据源之上承载真语义；`./mock/.ts` 静默挂载、在调用真正抵达时报告缺失的能力。装载器的表按 specifier 各持一个 memoized thunk——求值发生在首次 `require` 而非装配期——且每个垫片的导出面对 Node 自身的模块类型作类型检查，仅在结构身份（真实类）确不可满足处留最窄的、有说明的例外。`process` 全局由 worker 自装，装配期填入表中。 + +**VFS。** 内存为真相。`statSync(path, { bigint: true })` 返回 Node 的 BigInt 形状，其中两个字段承载真实信息，因为 `dsh-fs-local` 的 stale-write guard 依赖它们：`ino` 是按路径的身份（单调计数器分配，路径重建即新身份），`mtimeMs` 按条目严格递增（`max(now, previous + 1)`）——内存写例行落在同一毫秒内，相等的时间戳会放过陈旧覆写。这场排查同时修掉了它周围的静默：cordis 日志器的详细度数值向上计数，未声明等级的 exporter 会丢掉所有 warning——`startWorkerHost` 在任何 entry 挂载前安装 `levels: { default: 2 }` 的 console exporter。 + +**Shell。** `node:child_process` 是 VFS 之上的真实现。语法是买来的——`@yarnpkg/parsers` 的 `parseShell`——求值器与命令表是自有的，因为每个候选解释器都自带文件系统：管道是逐段传递的字符串，每个程序是 VFS 上的一个函数。命令表就是这台机器的全部 `/bin`；不存在的名字报告 `command not found`（127）。每次 `spawn` 从同一个 bundle 起一个子 Web Worker，首帧声明 shell 进程角色，因此终止梯是真的：`SIGTERM` 在下一命令边界处请求停止，`SIGKILL` 在任意时刻终止 worker——这是线程内解释器永远没有的抢占。文件系统面端到端异步（子进程经帧到宿主 VFS）；`execSync`、`execFileSync`、`fork` 拒绝，`node-pty` 保持桩。 + +## 曾考虑的替代方案 + +**整包替换 `dsh-subprocess-local` 或替换 bash 执行器。** 前者让代理表首次替换 workspace 包、违背其自身分类并倒置分层；后者撞上 `dsh-permission-presets` 对 `sandboxMode` 的 boot 期硬校验，并丢掉执行器已被测试钉住的超时/输出行为。 + +**`@yarnpkg/shell`、WASM shell、WebContainer。** 配套解释器建立在真实 Node streams 之上（约 1.5 MB 闭包要自养）；WASM 已被本部署的决定排除，WASI 没有 `fork`；且它们全都自带文件系统——恰是无法复用的那部分。 + +**`SharedArrayBuffer` + `Atomics.wait` 给子进程同步文件系统。** 在部署目标实测：无 COOP/COEP 头时 `SharedArrayBuffer` 未定义，而 GitHub Pages 无法设置响应头。异步面是超集；SAB 后端将来可垫入其下而不动任何程序。 + +**伪造 stats 或放宽错误谓词，而非如实实现 `bigint`。** 常量 `ino`/纯挂钟 `mtimeNs` 会静默废掉 stale-write guard；让技能发现吞下 `FS_IO_ERROR` 则会把同一个 bug 变成处处无失败的永久空目录。 + +## 后果 + +- `danger-full-access` 之外的沙箱档 fail loud：`SandboxEnforcement` 没有「未执法」值、浏览器没有内核，`ctx.sandbox.confine` 落闭、命令零启动。在 VFS 帧闸口做真执法是设计中的后续，不属本条。 +- Node 宿主的阶梯测试（`tests/node/child-process.spec.ts`）登记为 windows 不支持：阶梯的 win32 kill 梯级是按真 pid 的 taskkill，对进程表 pid 不可投递，而 worker 自身恒报 `linux`。 +- 输出增量但不流式：程序写入的 sink 以 `data` 事件转发，一个管道阶段完成后下一阶段才开始。 +- 运行时的测试镜像 `src/`（`tests/node/`、`tests/shell/`、`tests/storage/`……），每个垫片族在 oracle-diff 套件旁拥有自己的行为用例。 diff --git a/packages/experimental/webworker-runtime/src/module-proxies.ts b/packages/experimental/webworker-runtime/src/module-proxies.ts index 72ee3396c6..e30128d293 100644 --- a/packages/experimental/webworker-runtime/src/module-proxies.ts +++ b/packages/experimental/webworker-runtime/src/module-proxies.ts @@ -51,6 +51,9 @@ export const MODULE_PROXIES: Record = { 'node:perf_hooks': './node/builtin_modules/implemented/perf_hooks.ts', // Real zstd codec: session-log appends compress on every write. 'node:zlib': './node/builtin_modules/implemented/zlib.ts', + // The worker's own process layer: `bash -c` and the command table run against + // the VFS, because a browser worker has no processes to fork. + 'node:child_process': './node/builtin_modules/implemented/child_process.ts', // Structural mocks: every symbol exists, every call throws. 'node:net': './node/builtin_modules/mock/net.ts', 'node:stream': './node/builtin_modules/mock/stream.ts', diff --git a/packages/experimental/webworker-runtime/src/node/builtin_modules/implemented/child_process.ts b/packages/experimental/webworker-runtime/src/node/builtin_modules/implemented/child_process.ts new file mode 100644 index 0000000000..2c863ce6bb --- /dev/null +++ b/packages/experimental/webworker-runtime/src/node/builtin_modules/implemented/child_process.ts @@ -0,0 +1,398 @@ +/** + * `node:child_process` over the worker's own shell. + * + * A browser worker cannot fork, so this module IS the machine's process layer: + * `spawn` starts the argv as a shell process (`src/shell/process/`) — its own + * Web Worker, off this thread — and reports it through the `ChildProcess` + * surface the subprocess service consumes: pipes, `exit`/`close`, pid, and + * signals, with `SIGKILL` terminating the worker for real. The command table + * is the only `/bin` that exists, so a name it does not hold fails with + * `ENOENT`, exactly as a missing binary does on a real host. + * + * What stays impossible is what needs a real process: synchronous execution + * (`execSync`, and `spawnSync` for a known program) and `fork`. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/node/builtin_modules/implemented/child_process + */ + +import { Buffer } from 'buffer' +import { EventEmitter } from './events.ts' +import { notImplementedFail } from '../../notImplementedFail.ts' +import { registerProcess, releaseProcess, signalProcess } from '../../process-table.ts' +import { startProcess } from '../../../shell/process/host.ts' +import { standardPrograms } from '../../../shell/programs/index.ts' +import { DSH_ROOT } from '../../../storage/paths.ts' + +const MODULE = 'node:child_process' + +/** Per-stream disposition, as Node's `stdio` array spells it. */ +type StdioSetting = 'pipe' | 'ignore' | 'inherit' + +/** The spawn options this shim reads; Node accepts more, none of which apply here. */ +export interface WorkerSpawnOptions { + cwd?: string | undefined + env?: Record | undefined + stdio?: StdioSetting | readonly StdioSetting[] | undefined + /** Accepted and ignored: process groups do not exist, so there is no group to detach into. */ + detached?: boolean | undefined +} + +/** + * The readable half of a pipe: `data` events carrying Buffers, `end`, and a + * `destroy` that stops delivery. + * + * The stream-shaping members below are no-ops rather than omissions. A caller + * that configures the pipe before reading it (the browser launcher calls + * `setEncoding`) would otherwise die of a TypeError on the configuration line, + * hiding the real outcome — which for an unknown program is the `ENOENT` this + * shim is about to emit. + */ +class WorkerReadable extends EventEmitter { + private destroyed = false + + /** + * Accept an encoding (chunks are always UTF-8 text carried as Buffers). + * @returns this stream. + */ + setEncoding(): this { + return this + } + + /** + * Accept a flow-control request; delivery is driven by the command, which + * has already produced whatever it produced. + * @returns this stream. + */ + pause(): this { + return this + } + + /** @returns this stream; see {@link pause}. */ + resume(): this { + return this + } + + /** + * Deliver one chunk to the `data` listeners. + * @param text - the text written by the command. + */ + push(text: string): void { + if (this.destroyed || text === '') return + this.emit('data', Buffer.from(text, 'utf8')) + } + + /** Signal end of stream. */ + finish(): void { + if (this.destroyed) return + this.emit('end') + } + + /** Stop delivering; the collector calls this once the process settles. */ + destroy(): void { + this.destroyed = true + this.emit('close') + } +} + +/** The writable half of stdin: the batch write the subprocess service performs. */ +class WorkerWritable extends EventEmitter { + private text = '' + + /** + * Buffer one write. + * @param chunk - text or bytes to add to standard input. + * @returns true, since nothing here applies backpressure. + */ + write(chunk: string | Uint8Array): boolean { + this.text += typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf8') + return true + } + + /** + * Finish standard input. + * @param chunk - optional final write. + */ + end(chunk?: string | Uint8Array): void { + if (chunk !== undefined) this.write(chunk) + this.emit('finish') + } + + /** @returns everything written so far. */ + contents(): string { + return this.text + } +} + +/** + * One running command, wearing the parts of `ChildProcess` its consumers read. + */ +export class WorkerChildProcess extends EventEmitter { + /** The worker's own process id for this command, from the process table. */ + readonly pid: number + /** Standard input, when the caller asked for a pipe; null otherwise. */ + readonly stdin: WorkerWritable | null + /** Standard output, when the caller asked for a pipe; null otherwise. */ + readonly stdout: WorkerReadable | null + /** Standard error, when the caller asked for a pipe; null otherwise. */ + readonly stderr: WorkerReadable | null + /** Exit status once settled; null while running and after a signal. */ + exitCode: number | null = null + /** The signal that ended the command, or null when it exited on its own. */ + signalCode: NodeJS.Signals | null = null + + constructor(pid: number, stdio: readonly StdioSetting[]) { + super() + this.pid = pid + this.stdin = stdio[0] === 'pipe' ? new WorkerWritable() : null + this.stdout = stdio[1] === 'pipe' ? new WorkerReadable() : null + this.stderr = stdio[2] === 'pipe' ? new WorkerReadable() : null + } + + /** + * Deliver a signal to this command. + * @param signal - signal name; every one of them terminates. + * @returns true when the command was still running. + */ + kill(signal: NodeJS.Signals = 'SIGTERM'): boolean { + return signalProcess(this.pid, signal) + } +} + +/** Normalize the `stdio` option into the three-entry form the shim reads. */ +function stdioOf(option: WorkerSpawnOptions['stdio']): StdioSetting[] { + if (typeof option === 'string') return [option, option, option] + if (option === undefined) return ['pipe', 'pipe', 'pipe'] + return [option[0] ?? 'pipe', option[1] ?? 'pipe', option[2] ?? 'pipe'] +} + +/** The environment a command runs with: the caller's map, minus the removals Node allows. */ +function environmentOf(option: WorkerSpawnOptions['env']): Record { + const inherited = (globalThis as { process?: { env?: Record } }).process?.env ?? {} + const source = option ?? inherited + return Object.fromEntries(Object.entries(source).filter(([, value]) => value !== undefined) as [string, string][]) +} + +/** + * A missing program fails the way Node fails a missing binary, so consumers + * that classify spawn errors by `code`, `path`, and `syscall` keep working. + */ +function spawnEnoent(program: string): NodeJS.ErrnoException { + const error = new Error(`spawn ${program} ENOENT`) as NodeJS.ErrnoException + error.code = 'ENOENT' + error.errno = -2 + error.path = program + error.syscall = `spawn ${program}` + return error +} + +/** Whether this argv is a shell invocation whose script the interpreter should parse. */ +function shellScriptOf(argv: readonly string[]): string | undefined { + const [program, flag, script] = argv + if ((program !== 'bash' && program !== 'sh') || flag !== '-c') return undefined + return script ?? '' +} + +/** + * Run one command in the worker. + * + * The call returns immediately with a handle; the command runs in its own + * worker (or inline where no `Worker` exists) and reports back through the + * handle's pipes and events. + * @param program - the program name, as argv[0]. + * @param args - its arguments. + * @param options - working directory, environment, and stdio dispositions. + * @returns the running command's handle. + */ +export function spawn( + program: string, + args: readonly string[] = [], + options: WorkerSpawnOptions = {}, +): WorkerChildProcess { + if (typeof program !== 'string' || program === '') { + // Node refuses a non-string command with this error rather than starting + // anything; a caller whose own lookup produced nothing reads why. + const invalid = new TypeError(`The "file" argument must be a non-empty string. Received ${program as unknown as string}`) as NodeJS.ErrnoException + invalid.code = 'ERR_INVALID_ARG_TYPE' + throw invalid + } + const argv = [program, ...args] + const stdio = stdioOf(options.stdio) + const entry = registerProcess() + const child = new WorkerChildProcess(entry.pid, stdio) + + const script = shellScriptOf(argv) + const known = script !== undefined || standardPrograms().has(program) + + const emit = (stream: 'stdout' | 'stderr', text: string): void => { + if (text === '') return + const pipe = stream === 'stdout' ? child.stdout : child.stderr + if (pipe !== null) { + pipe.push(text) + return + } + // An inherited stream belongs to the host: the worker's console is the + // only place it can go, and an ignored one goes nowhere. + if (stdio[stream === 'stdout' ? 1 : 2] === 'inherit') { + (stream === 'stdout' ? console.log : console.error)(text.replace(/\n$/, '')) + } + } + + const settle = (exitCode: number): void => { + releaseProcess(entry.pid) + // A signalled command reports no exit code, which is what makes the + // subprocess service classify it as killed rather than finished. + const signal = entry.signal ?? null + child.exitCode = signal === null ? exitCode : null + child.signalCode = signal + child.stdout?.finish() + child.stderr?.finish() + child.emit('exit', child.exitCode, signal) + child.emit('close', child.exitCode, signal) + } + + // The command starts on a microtask, so a caller that attaches listeners and + // writes standard input right after `spawn()` — the subprocess service does + // exactly that — is never racing the first output. + queueMicrotask(() => { + if (!known) { + releaseProcess(entry.pid) + child.emit('error', spawnEnoent(program)) + return + } + entry.process = startProcess({ + script, + argv, + cwd: options.cwd ?? DSH_ROOT, + env: environmentOf(options.env), + stdin: child.stdin?.contents() ?? '', + onOutput: emit, + onExit: settle, + }) + // A signal that arrived while the process was still starting has to reach + // it now; the table recorded it but had nothing to deliver it to. + if (entry.signal !== undefined) { + if (entry.signal === 'SIGKILL') entry.process.destroy() + else entry.process.interrupt() + } + }) + + return child +} + +/** The result shape `spawnSync` returns, holding only the members consumers read. */ +export interface WorkerSpawnSyncResult { + pid: number + status: number | null + signal: NodeJS.Signals | null + stdout: Buffer + stderr: Buffer + output: (Buffer | null)[] + /** Why the run did not happen; carries `code` for the callers that classify by it. */ + error?: NodeJS.ErrnoException +} + +/** + * Report that a command cannot run synchronously. + * + * Callers use `spawnSync` to probe for a binary (the sandbox runner probes do) + * and Node answers a missing one with an `error` rather than a throw, so this + * answers in the same shape: absent programs report `ENOENT`, and a program + * this shell *does* have reports that only the asynchronous path can run it. + * @param program - the program name. + * @returns the Node-shaped synchronous result carrying the failure. + */ +export function spawnSync(program: string): WorkerSpawnSyncResult { + const empty = Buffer.alloc(0) + const error = standardPrograms().has(program) + ? new Error(`${MODULE}.spawnSync cannot run ${program} in the worker host: commands run asynchronously`) + : spawnEnoent(program) + return { pid: -1, status: null, signal: null, stdout: empty, stderr: empty, output: [null, empty, empty], error } +} + +/** Callback `exec` and `execFile` report through. */ +type ExecCallback = (error: Error | null, stdout: string, stderr: string) => void + +/** Split the optional options argument from the callback Node allows in either position. */ +function execArguments( + options: WorkerSpawnOptions | ExecCallback | undefined, + callback: ExecCallback | undefined, +): { options: WorkerSpawnOptions; callback: ExecCallback | undefined } { + if (typeof options === 'function') return { options: {}, callback: options } + return { options: options ?? {}, callback } +} + +/** + * Run a command line and report its output through a callback. + * @param command - the shell source to run. + * @param options - working directory and environment, or the callback. + * @param callback - receives the failure (nonzero status included), stdout, and stderr. + * @returns the running command's handle. + */ +export function exec( + command: string, + options?: WorkerSpawnOptions | ExecCallback, + callback?: ExecCallback, +): WorkerChildProcess { + const settled = execArguments(options, callback) + return execute(['bash', '-c', command], settled.options, settled.callback) +} + +/** + * Run one program with an explicit argv and report its output through a callback. + * @param program - the program name. + * @param args - its arguments, or the options, or the callback. + * @param options - working directory and environment, or the callback. + * @param callback - receives the failure (nonzero status included), stdout, and stderr. + * @returns the running command's handle. + */ +export function execFile( + program: string, + args?: readonly string[] | WorkerSpawnOptions | ExecCallback, + options?: WorkerSpawnOptions | ExecCallback, + callback?: ExecCallback, +): WorkerChildProcess { + const argv = Array.isArray(args) ? [program, ...args as string[]] : [program] + const shifted = Array.isArray(args) ? options : args as WorkerSpawnOptions | ExecCallback | undefined + const settled = execArguments(shifted, typeof options === 'function' ? options : callback) + return execute(argv, settled.options, settled.callback) +} + +/** Shared body of `exec` and `execFile`: spawn, collect both streams, then report. */ +function execute(argv: readonly string[], options: WorkerSpawnOptions, callback: ExecCallback | undefined): WorkerChildProcess { + const child = spawn(argv[0] as string, argv.slice(1), { ...options, stdio: 'pipe' }) + let stdout = '' + let stderr = '' + child.stdout?.on('data', (chunk: unknown) => { stdout += String(chunk) }) + child.stderr?.on('data', (chunk: unknown) => { stderr += String(chunk) }) + child.on('error', (error: unknown) => { callback?.(error instanceof Error ? error : new Error(String(error)), stdout, stderr) }) + child.on('close', (code: unknown) => { + const status = typeof code === 'number' ? code : 1 + callback?.(status === 0 ? null : new Error(`Command failed: ${argv.join(' ')}`), stdout, stderr) + }) + return child +} + +/** Run a command line synchronously (unavailable: the interpreter is asynchronous). */ +export const execSync: typeof import('node:child_process').execSync = notImplementedFail(MODULE, 'execSync') + +/** Run one program synchronously (unavailable: the interpreter is asynchronous). */ +export const execFileSync: typeof import('node:child_process').execFileSync = notImplementedFail(MODULE, 'execFileSync') + +/** Start a Node child (unavailable: the worker cannot create another Node runtime). */ +export const fork: typeof import('node:child_process').fork = notImplementedFail(MODULE, 'fork') + +/** CommonJS interop marker: the worker loader hands `default` to default imports (see ../../builtins.ts). */ +export const __esModule = true + +/** + * The `node:child_process` declarations this module stands in for. The four + * process starters keep this module's own types: they answer + * {@link WorkerChildProcess} and {@link WorkerSpawnSyncResult}, the pipes and exit + * facts a shell worker can carry, where Node declares a `ChildProcess` holding OS + * stream objects and, for `exec`/`execFile`, an overload ladder over encodings + * this shell reports as UTF-8 text. + */ +type NodeFace = Partial> + & Record<'spawn' | 'spawnSync' | 'exec' | 'execFile', unknown> + +/** CommonJS default export: the members `require()` hands a caller of this module. */ +export default { spawn, spawnSync, exec, execFile, execFileSync, execSync, fork } satisfies NodeFace diff --git a/packages/experimental/webworker-runtime/src/node/builtins.ts b/packages/experimental/webworker-runtime/src/node/builtins.ts index f7fb1b8072..f7008b518d 100644 --- a/packages/experimental/webworker-runtime/src/node/builtins.ts +++ b/packages/experimental/webworker-runtime/src/node/builtins.ts @@ -37,6 +37,7 @@ import * as nodeUrl from './builtin_modules/implemented/url.ts' import * as nodeUtil from './builtin_modules/implemented/util.ts' import * as nodeUtilTypes from './builtin_modules/implemented/util/types.ts' import * as nodeZlib from './builtin_modules/implemented/zlib.ts' +import * as nodeChildProcess from './builtin_modules/implemented/child_process.ts' import * as nodeNet from './builtin_modules/mock/net.ts' import * as nodeSqlite from './builtin_modules/mock/sqlite.ts' import * as nodeStream from './builtin_modules/mock/stream.ts' @@ -57,6 +58,7 @@ import type { StaticModuleFactory } from '../module-system/module-loader.ts' const BUILTINS: Record = { async_hooks: () => nodeAsyncHooks, buffer: () => nodeBuffer, + child_process: () => nodeChildProcess, crypto: () => nodeCrypto, events: () => nodeEvents, fs: () => nodeFs, diff --git a/packages/experimental/webworker-runtime/src/node/process-table.ts b/packages/experimental/webworker-runtime/src/node/process-table.ts new file mode 100644 index 0000000000..ed8aa957ed --- /dev/null +++ b/packages/experimental/webworker-runtime/src/node/process-table.ts @@ -0,0 +1,76 @@ +/** + * The worker's process table. A browser worker cannot fork, so the + * `node:child_process` shim keeps its own table: one entry per running + * command, with the pid `process.kill` and the subprocess service's tree + * bookkeeping address it by. + * + * Kept apart from both consumers because they need it from opposite sides — + * the shim registers entries, the `process` global signals them. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/node/process-table + */ + +import type { RunningProcess } from '../shell/process/host.ts' + +/** One running command, as the table tracks it. */ +export interface WorkerProcessEntry { + /** Identifier handed out to the host tree; unique for the worker's lifetime. */ + readonly pid: number + /** The first signal delivered, which decides how the process reports its death. */ + signal: NodeJS.Signals | undefined + /** The started command, attached once it exists; signals reach it through this. */ + process: RunningProcess | undefined +} + +const entries = new Map() + +// Pid 1 is the worker host itself (`process.pid`), so commands start above it. +let lastPid = 1 + +/** + * Reserve one pid before its command starts, so a handle can report it + * synchronously. + * @returns the new table entry, still without its process. + */ +export function registerProcess(): WorkerProcessEntry { + lastPid += 1 + const entry: WorkerProcessEntry = { pid: lastPid, signal: undefined, process: undefined } + entries.set(entry.pid, entry) + return entry +} + +/** + * Drop one entry once its command has settled. + * @param pid - the entry's pid. + */ +export function releaseProcess(pid: number): void { + entries.delete(pid) +} + +/** + * Whether a command with this pid is still running. + * @param pid - pid to look up; a negative value addresses the group, which here + * holds exactly the one process that leads it. + * @returns true while the entry is in the table. + */ +export function processAlive(pid: number): boolean { + return entries.has(Math.abs(pid)) +} + +/** + * Deliver a signal to one running command. + * + * `SIGKILL` stops the command whatever it is doing; every other signal asks it + * to stop at its next command boundary. That distinction is real only for a + * worker-backed process — see {@link RunningProcess.destroy}. + * @param pid - pid or negative process-group id. + * @param signal - the signal name to record and deliver. + * @returns true when an entry received it, false when no such process exists. + */ +export function signalProcess(pid: number, signal: NodeJS.Signals): boolean { + const entry = entries.get(Math.abs(pid)) + if (entry === undefined) return false + entry.signal ??= signal + if (signal === 'SIGKILL') entry.process?.destroy() + else entry.process?.interrupt() + return true +} diff --git a/packages/experimental/webworker-runtime/src/shell/ast.ts b/packages/experimental/webworker-runtime/src/shell/ast.ts new file mode 100644 index 0000000000..d9b04e526f --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/ast.ts @@ -0,0 +1,23 @@ +/** + * The parsed command line, as this shell names it. + * + * `@yarnpkg/parsers` re-exports only part of its grammar's type map from the + * package root, and its `exports` field forbids reaching the grammar module + * directly, so the three missing members are derived from the ones it does + * publish. `CommandChain` is `Command` plus an optional pipeline link, which + * makes it usable wherever a command node is expected. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/ast + */ + +import type { Argument, CommandChain } from '@yarnpkg/parsers' + +export type { ArgumentSegment, ArithmeticExpression, CommandChain, CommandLine, ShellLine } from '@yarnpkg/parsers' + +/** One command node: a program call, a subshell, a group, or bare assignments. */ +export type Command = CommandChain + +/** An argument that becomes argv fields. */ +export type ValueArgument = Extract + +/** An argument that rewires a descriptor instead of becoming argv. */ +export type RedirectArgument = Extract diff --git a/packages/experimental/webworker-runtime/src/shell/expand.ts b/packages/experimental/webworker-runtime/src/shell/expand.ts new file mode 100644 index 0000000000..44b6df8d5e --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/expand.ts @@ -0,0 +1,228 @@ +/** + * Word expansion: one parsed argument becomes the zero or more fields a + * program receives in its argv. Covers the segment kinds the grammar produces + * — literal text, variables (with `:-` / `:+` forms), command substitution, + * arithmetic, and globs matched against the VFS. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/expand + */ + +import picomatch from 'picomatch' +import type { ArgumentSegment, ArithmeticExpression, ShellLine, ValueArgument } from './ast.ts' +import { resolve } from '../module-system/posix-path.ts' +import type { ShellFileSystem, ShellState } from './types.ts' + +/** Characters that make the grammar treat a whole word as a glob pattern. */ +const GLOB_PATTERN = /[*?]|\[[^\]]*\]/ + +/** + * Whether one word is a glob the shell should match against the filesystem. + * Handed to `parseShell`, which decides between a `text` and a `glob` segment. + * @param word - the word exactly as it was written. + * @returns true when the word contains a wildcard. + */ +export function isGlobPattern(word: string): boolean { + return GLOB_PATTERN.test(word) +} + +/** + * Read one variable the way `$name` does. + * + * Shell variables shadow the environment (an assignment without `export` is + * only visible to this shell), and the specials report what a shell without + * job control or positional parameters can honestly report. + * @param state - the shell state to read. + * @param name - variable name, or one of `?`, `$`, `#`, `@`, `*`, `0`. + * @returns the value, or undefined when the variable is unset. + */ +export function readVariable(state: ShellState, name: string): string | undefined { + switch (name) { + case '?': return String(state.lastStatus) + // The worker host runs the whole tree as pid 1; `$$` reports it verbatim. + case '$': return '1' + case '0': return 'bash' + // No positional parameters reach a `bash -c` command line here. + case '#': return '0' + case '@': case '*': return '' + default: return state.variables[name] ?? state.environment[name] + } +} + +/** Evaluate `$(( … ))`. */ +function arithmetic(expression: ArithmeticExpression, state: ShellState): number { + switch (expression.type) { + case 'number': return expression.value + case 'variable': return Number.parseInt(readVariable(state, expression.name) ?? '0', 10) || 0 + case 'addition': return arithmetic(expression.left, state) + arithmetic(expression.right, state) + case 'subtraction': return arithmetic(expression.left, state) - arithmetic(expression.right, state) + case 'multiplication': return arithmetic(expression.left, state) * arithmetic(expression.right, state) + case 'division': return Math.trunc(arithmetic(expression.left, state) / arithmetic(expression.right, state)) + } +} + +/** + * Expand one glob against the filesystem, one path segment at a time. + * + * Matches keep the pattern's own spelling: a relative pattern yields relative + * paths, so `ls *.ts` prints what the model typed. + * @param pattern - the glob as written. + * @param cwd - directory a relative pattern starts from. + * @param fs - the filesystem to match against. + * @returns sorted matches, or an empty array when nothing matches. + */ +export async function expandGlob(pattern: string, cwd: string, fs: ShellFileSystem): Promise { + const absolute = pattern.startsWith('/') + const segments = pattern.split('/').filter(segment => segment !== '') + // A glob walks paths that may not exist or may not be directories; both + // simply contribute no matches, so listing failures are absorbed here. + const safeList = async (path: string): Promise<{ name: string; directory: boolean }[]> => { + try { + return await fs.list(path) + } catch { + return [] + } + } + // Each frontier entry pairs the directory to search with the prefix that + // reproduces the caller's spelling for anything found under it. + let frontier: { path: string; display: string }[] = [{ path: absolute ? '/' : cwd, display: absolute ? '/' : '' }] + for (const [index, segment] of segments.entries()) { + const last = index === segments.length - 1 + const next: { path: string; display: string }[] = [] + for (const entry of frontier) { + if (segment === '**') { + // `**` stands for this directory and every directory below it. + const stack = [entry] + while (stack.length > 0) { + const current = stack.pop() as { path: string; display: string } + next.push(current) + for (const child of await safeList(current.path)) { + if (child.directory) { + stack.push({ path: resolve(current.path, child.name), display: `${current.display}${child.name}/` }) + } + } + } + continue + } + if (!isGlobPattern(segment)) { + const path = resolve(entry.path, segment) + if (await fs.stat(path) === undefined) continue + next.push({ path, display: `${entry.display}${segment}${last ? '' : '/'}` }) + continue + } + const matches = picomatch(segment, { dot: segment.startsWith('.') }) + for (const child of await safeList(entry.path)) { + if (!matches(child.name)) continue + if (!last && !child.directory) continue + next.push({ path: resolve(entry.path, child.name), display: `${entry.display}${child.name}${last ? '' : '/'}` }) + } + } + frontier = next + } + // A `**` frontier carries trailing separators from its own expansion; the + // shell reports directory matches without one. + return [...new Set(frontier.map(entry => entry.display.replace(/\/$/, '')))].filter(match => match !== '').sort() +} + +/** + * Everything expansion needs that the argument itself cannot supply: how to + * run a command substitution, and the state variables resolve against. + */ +export interface ExpansionContext { + state: ShellState + /** The filesystem globs match against. */ + fs: ShellFileSystem + /** + * Run one nested command line and return its standard output. + * @param shell - the parsed line inside `$( … )`. + * @returns the captured output, with trailing newlines already stripped. + */ + substitute(shell: ShellLine): Promise +} + +/** + * Expand one argument into fields. + * + * Unquoted expansions split on whitespace the way a shell does, so + * `cat $FILES` with two names runs `cat` with two arguments while + * `cat "$FILES"` runs it with one. + * @param argument - the parsed argument. + * @param context - substitution hook and shell state. + * @returns the fields this argument contributes to argv. + */ +export async function expandArgument(argument: ValueArgument, context: ExpansionContext): Promise { + const fields: string[] = [] + // `undefined` means "no field started yet": an unset unquoted variable must + // contribute nothing rather than an empty argument. + let current: string | undefined + + const append = (text: string): void => { current = (current ?? '') + text } + const appendSplit = (text: string): void => { + const parts = text.split(/\s+/) + for (const [index, part] of parts.entries()) { + if (index > 0) { + if (current !== undefined) fields.push(current) + current = undefined + } + if (part !== '') append(part) + } + } + + for (const segment of argument.segments) { + switch (segment.type) { + case 'text': + append(segment.text) + break + case 'arithmetic': + append(String(arithmetic(segment.arithmetic, context.state))) + break + case 'variable': { + const value = await expandVariable(segment, context) + if (segment.quoted) append(value) + else appendSplit(value) + break + } + case 'shell': { + const output = await context.substitute(segment.shell) + if (segment.quoted) append(output) + else appendSplit(output) + break + } + case 'glob': { + const matches = await expandGlob(segment.pattern, context.state.cwd, context.fs) + if (matches.length === 0) { + // No match: a POSIX shell passes the pattern through unchanged. + append(segment.pattern) + break + } + for (const [index, match] of matches.entries()) { + if (index > 0) { + fields.push(current as string) + current = undefined + } + append(match) + } + break + } + } + } + if (current !== undefined) fields.push(current) + return fields +} + +/** Resolve one `${name}` segment, including its `:-` and `:+` alternatives. */ +async function expandVariable( + segment: Extract, + context: ExpansionContext, +): Promise { + const value = readVariable(context.state, segment.name) + const set = value !== undefined && value !== '' + if (!set && segment.defaultValue !== undefined) return await joinArguments(segment.defaultValue, context) + if (set && segment.alternativeValue !== undefined) return await joinArguments(segment.alternativeValue, context) + return value ?? '' +} + +/** Expand a `:-` / `:+` operand, which is itself a list of arguments. */ +async function joinArguments(operand: ValueArgument[], context: ExpansionContext): Promise { + const parts: string[] = [] + for (const argument of operand) parts.push(...await expandArgument(argument, context)) + return parts.join(' ') +} diff --git a/packages/experimental/webworker-runtime/src/shell/fs-access.ts b/packages/experimental/webworker-runtime/src/shell/fs-access.ts new file mode 100644 index 0000000000..64d2a6a895 --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/fs-access.ts @@ -0,0 +1,120 @@ +/** + * The in-host filesystem for shell runs: {@link ShellFileSystem} straight over + * the mounted VFS, plus the path and diagnostic helpers every program shares. + * + * This implementation answers from memory. A command running in its own + * worker uses the message-backed one (`./process/child.ts`), which this one + * serves from the host side. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/fs-access + */ + +import { resolve } from '../module-system/posix-path.ts' +import { requireActiveVfs } from '../storage/active.ts' +import type { VfsError, VfsStats } from '../storage/types.ts' +import type { ShellDirent, ShellFileSystem, ShellStats } from './types.ts' + +/** + * Resolve one shell word into an absolute VFS path. + * @param cwd - the shell's working directory. + * @param path - absolute or relative path as the command line spelled it. + * @returns the absolute normalized path. + */ +export function resolveIn(cwd: string, path: string): string { + return resolve(cwd, path) +} + +/** + * Restate a filesystem failure the way a shell utility reports it, so the model + * reads `cat: /dsh/none: No such file or directory` instead of a Node error + * string. + * @param program - the utility's name, used as the message prefix. + * @param path - the path the utility was working on. + * @param error - the failure the filesystem raised. + * @returns the single-line diagnostic, without a trailing newline. + */ +export function describeFailure(program: string, path: string, error: unknown): string { + const code = (error as Partial).code + const reason = code === 'ENOENT' + ? 'No such file or directory' + : code === 'ENOTDIR' + ? 'Not a directory' + : code === 'EISDIR' + ? 'Is a directory' + : code === 'ENOTEMPTY' + ? 'Directory not empty' + : code === 'EEXIST' + ? 'File exists' + : error instanceof Error ? error.message : String(error) + return `${program}: ${path}: ${reason}` +} + +/** + * Build a Node-shaped filesystem error, for the conditions this layer detects + * itself and for the worker transport, which can carry a code but not a class. + * @param code - the Node error code (`ENOENT`, `EISDIR`, …). + * @param syscall - the operation that failed. + * @param path - the path it failed on. + * @returns the error to throw. + */ +export function filesystemError(code: string, syscall: string, path: string): VfsError { + const error = new Error(`${code}: ${syscall} failed, ${syscall} '${path}'`) as VfsError + error.code = code + error.path = path + error.syscall = syscall + return error +} + +/** Project VFS stats onto the facts a program reads. */ +function statsOf(stats: VfsStats): ShellStats { + return { directory: stats.isDirectory(), size: stats.size, mtimeMs: stats.mtimeMs } +} + +/** + * The filesystem backed by the VFS mounted in this thread. + * @returns the in-host {@link ShellFileSystem}. + */ +export function hostFileSystem(): ShellFileSystem { + const vfs = (): ReturnType => requireActiveVfs() + // oxlint-disable-next-line typescript/require-await -- async face, in-memory backend; see the note below. + const stat = async (path: string): Promise => { + try { + return statsOf(vfs().statSync(path) as VfsStats) + } catch { + // Absence is the answer callers branch on; every other failure mode of + // the in-memory backend is also "this path holds nothing readable". + return undefined + } + } + // Several members take no await: the face is asynchronous because a process + // worker's filesystem is, while this backend answers from memory. + /* oxlint-disable typescript/require-await -- see the note above. */ + return { + stat, + list: async (path: string): Promise => { + const names = [...vfs().readdirSync(path) as string[]].sort() + const entries: ShellDirent[] = [] + for (const name of names) { + entries.push({ name, directory: (await stat(resolve(path, name)))?.directory ?? false }) + } + return entries + }, + readText: async (path: string): Promise => { + if ((await stat(path))?.directory === true) throw filesystemError('EISDIR', 'read', path) + return vfs().readFileSync(path, 'utf8') as string + }, + writeText: async (path: string, text: string, append = false): Promise => { + if (append) vfs().appendFileSync(path, text) + else vfs().writeFileSync(path, text) + }, + mkdir: async (path: string, recursive: boolean): Promise => { + vfs().mkdirSync(path, { recursive }) + }, + remove: async (path: string, options: { recursive: boolean; force: boolean }): Promise => { + vfs().rmSync(path, options) + }, + rename: async (from: string, to: string): Promise => { + vfs().renameSync(from, to) + }, + } + /* oxlint-enable typescript/require-await */ +} diff --git a/packages/experimental/webworker-runtime/src/shell/interpret.ts b/packages/experimental/webworker-runtime/src/shell/interpret.ts new file mode 100644 index 0000000000..e38d87e5c0 --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/interpret.ts @@ -0,0 +1,389 @@ +/** + * The interpreter: it walks the parsed command line and runs the command table + * against the VFS. Structure (`;` `&` `|` `|&` `&&` `||`, subshells, groups, + * redirections, prefix assignments) is honored here; what a command *does* + * belongs to its program in `./programs/`. + * + * Output is text, not streams: every program is a JavaScript function that + * returns before the next one runs, so a pipeline hands a string along instead + * of plumbing byte streams a browser worker has no way to schedule between. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/interpret + */ + +import { parseShell } from '@yarnpkg/parsers' +import type { Command, CommandChain, CommandLine, RedirectArgument, ShellLine, ValueArgument } from './ast.ts' +import { expandArgument, isGlobPattern } from './expand.ts' +import type { ExpansionContext } from './expand.ts' +import { describeFailure, hostFileSystem, resolveIn } from './fs-access.ts' +import { standardPrograms } from './programs/index.ts' +import type { ShellFileSystem, ShellIo, ShellProgram, ShellRunOutcome, ShellState } from './types.ts' + +/** Status a command line reports once the caller's abort signal has fired. */ +const ABORTED_STATUS = 130 + +/** Status of a command name the table does not hold, as POSIX shells report it. */ +const NOT_FOUND_STATUS = 127 + +/** Nesting limit for `$( … )`; a deeper line is a runaway, not a command. */ +const MAX_SUBSTITUTION_DEPTH = 16 + +/** Everything one `bash -c` invocation needs. */ +export interface ShellRunOptions { + /** Working directory the line starts in. */ + cwd: string + /** Environment the line starts with. */ + env: Record + /** Standard input contents; absent means empty. */ + stdin?: string | undefined + /** Cancellation: an aborted line stops before its next command. */ + signal?: AbortSignal | undefined + /** + * The filesystem this run acts on; defaults to the VFS mounted in this + * thread. A run inside a process worker passes the message-backed one. + */ + fs?: ShellFileSystem | undefined + /** + * Called with each write as it happens, before the run settles. The returned + * outcome still carries the complete text; this only lets a caller that + * reports progress (a background job's incremental reads) see output while + * the line is still running. + */ + onOutput?: ((stream: 'stdout' | 'stderr', text: string) => void) | undefined +} + +/** Accumulates one output stream. */ +interface Sink { + write: (text: string) => void +} + +/** A sink over a string buffer, for pipelines and command substitution. */ +function buffer(): Sink & { text(): string } { + const chunks: string[] = [] + return { + write: (text: string) => { chunks.push(text) }, + text: () => chunks.join(''), + } +} + +/** + * Run one shell command line to completion. + * @param source - the command source, exactly as `bash -c` would receive it. + * @param options - starting directory, environment, standard input, cancellation, filesystem, output callback. + * @returns the exit status and the complete standard output and standard error. + */ +export async function runShellCommand(source: string, options: ShellRunOptions): Promise { + const run = startRun(options) + let line: ShellLine + try { + line = parseShell(source, { isGlobPattern }) + } catch (error) { + run.io.err(`bash: syntax error: ${error instanceof Error ? error.message.split('\n')[0] : String(error)}\n`) + return run.settle(2) + } + const interpreter = new Interpreter(standardPrograms(), options.fs ?? hostFileSystem(), options.signal) + return run.settle(await interpreter.line(line, run.state, run.io)) +} + +/** + * Run one program directly, without a command line to parse. + * + * This is the path for an argv the caller already has in pieces — a spawn that + * names a program instead of handing `bash` a script — so nothing re-quotes + * words that were never quoted in the first place. + * @param argv - the program name at index 0, then its arguments. + * @param options - starting directory, environment, standard input, cancellation, filesystem, output callback. + * @returns the exit status and the complete standard output and standard error. + */ +export async function runShellProgram(argv: readonly string[], options: ShellRunOptions): Promise { + const run = startRun(options) + const name = argv[0] + const program = name === undefined ? undefined : standardPrograms().get(name) + if (name === undefined || program === undefined) { + run.io.err(`bash: ${name ?? ''}: command not found\n`) + return run.settle(NOT_FOUND_STATUS) + } + if (options.signal?.aborted === true) return run.settle(ABORTED_STATUS) + try { + return run.settle(await program(argv, run.io, run.state, options.fs ?? hostFileSystem())) + } catch (error) { + run.io.err(`bash: ${name}: ${error instanceof Error ? error.message : String(error)}\n`) + return run.settle(1) + } +} + +/** Build the state, the sinks, and the settlement one run reports through. */ +function startRun(options: ShellRunOptions): { + state: ShellState + io: ShellIo + settle: (exitCode: number) => ShellRunOutcome +} { + const stdout = buffer() + const stderr = buffer() + const report = options.onOutput + return { + state: { + cwd: options.cwd, + environment: { ...options.env }, + variables: {}, + lastStatus: 0, + exitRequested: undefined, + signal: options.signal, + }, + io: { + stdin: options.stdin ?? '', + out: (text: string) => { + stdout.write(text) + report?.('stdout', text) + }, + err: (text: string) => { + stderr.write(text) + report?.('stderr', text) + }, + }, + settle: (exitCode: number) => ({ exitCode, stdout: stdout.text(), stderr: stderr.text() }), + } +} + +/** One interpretation pass; holds what every nested command shares. */ +class Interpreter { + constructor( + private readonly programs: ReadonlyMap, + private readonly fs: ShellFileSystem, + private readonly signal: AbortSignal | undefined, + private readonly depth = 0, + ) {} + + /** + * Run every command of one line, left to right. + * @param line - the parsed line. + * @param state - shell state the line reads and mutates. + * @param io - standard input and the output sinks. + * @returns the status of the last command that ran. + */ + async line(line: ShellLine, state: ShellState, io: ShellIo): Promise { + let status = state.lastStatus + for (const entry of line) { + if (this.signal?.aborted === true) return ABORTED_STATUS + // `&` starts no background job here: the worker has no scheduler that + // could run one, so a backgrounded command runs to completion in place. + status = await this.commandLine(entry.command, state, io) + state.lastStatus = status + if (state.exitRequested !== undefined) return state.exitRequested + } + return status + } + + /** + * Run one `&&` / `||` chain. + * + * The grammar nests these to the right, while a shell evaluates them left to + * right: `false && a || b` runs `b`. Flattening first is what makes the + * skipped `&&` hand its status to the following `||` instead of taking the + * whole remainder of the line with it. + */ + private async commandLine(commandLine: CommandLine, state: ShellState, io: ShellIo): Promise { + const links: { type: '&&' | '||'; chain: CommandChain }[] = [] + for (let current = commandLine.then; current !== undefined; current = current.line.then) { + links.push({ type: current.type, chain: current.line.chain }) + } + let status = await this.pipeline(commandLine.chain, state, io) + state.lastStatus = status + for (const link of links) { + if (state.exitRequested !== undefined) return status + if (link.type === '&&' ? status !== 0 : status === 0) continue + status = await this.pipeline(link.chain, state, io) + state.lastStatus = status + } + return status + } + + /** Run one `|` / `|&` pipeline; its status is the last stage's. */ + private async pipeline(chain: CommandChain, state: ShellState, io: ShellIo): Promise { + const stages: { command: CommandChain; mergesStderr: boolean }[] = [] + for (let current: CommandChain | undefined = chain; current !== undefined;) { + const link: CommandChain['then'] = current.then + stages.push({ command: current, mergesStderr: link?.type === '|&' }) + current = link?.chain + } + let input = io.stdin + let status = 0 + for (const [index, stage] of stages.entries()) { + if (this.signal?.aborted === true) return ABORTED_STATUS + const last = index === stages.length - 1 + const piped = buffer() + const stageIo: ShellIo = last + ? { stdin: input, out: io.out, err: io.err } + : { stdin: input, out: piped.write, err: stage.mergesStderr ? piped.write : io.err } + status = await this.command(stage.command, state, stageIo) + if (!last) input = piped.text() + if (state.exitRequested !== undefined) return status + } + return status + } + + /** Run one command node: a program call, a subshell, a group, or bare assignments. */ + private async command(command: Command, state: ShellState, io: ShellIo): Promise { + switch (command.type) { + case 'envs': + for (const env of command.envs) assign(state, env.name, await this.assignedValue(env.args[0], state)) + return 0 + case 'subshell': { + // A subshell sees a copy: its `cd` and its assignments die with it. + const nested = { ...state, environment: { ...state.environment }, variables: { ...state.variables } } + return await this.redirected(command.args, state, io, async inner => await this.line(command.subshell, nested, inner)) + } + case 'group': + return await this.redirected(command.args, state, io, async inner => await this.line(command.group, state, inner)) + case 'command': + return await this.program(command, state, io) + } + } + + /** Expand a command's words and run the program they name. */ + private async program(command: Extract, state: ShellState, io: ShellIo): Promise { + const argv: string[] = [] + const redirections: RedirectArgument[] = [] + for (const argument of command.args) { + if (argument.type === 'redirection') { + redirections.push(argument) + continue + } + argv.push(...await expandArgument(argument, this.context(state))) + } + const prefix: Record = {} + for (const env of command.envs) prefix[env.name] = await this.assignedValue(env.args[0], state) + + if (argv.length === 0) { + for (const [name, value] of Object.entries(prefix)) assign(state, name, value) + return 0 + } + // A prefixed command sees the assignments as environment for its run only, + // which also means it cannot change the caller's directory. + const scope = Object.keys(prefix).length === 0 + ? state + : { ...state, environment: { ...state.environment, ...prefix } } + + const name = argv[0] as string + const program = this.programs.get(name) + if (program === undefined) { + io.err(`bash: ${name}: command not found\n`) + return NOT_FOUND_STATUS + } + return await this.redirected(redirections, state, io, async (inner) => { + try { + return await program(argv, inner, scope, this.fs) + } catch (error) { + // A program's own defect must not take the whole worker down with it. + inner.err(`bash: ${name}: ${error instanceof Error ? error.message : String(error)}\n`) + return 1 + } + }) + } + + /** + * Apply redirections around one body, then restore nothing: every sink is a + * value, so the caller's own `io` is untouched by construction. + */ + private async redirected( + redirections: readonly RedirectArgument[], + state: ShellState, + io: ShellIo, + body: (io: ShellIo) => Promise, + ): Promise { + let stdin = io.stdin + let out = io.out + let err = io.err + // Every file write this redirection set started, awaited before the + // command's status is reported: a `> file` must be complete on return. + const writes: Promise[] = [] + for (const redirection of redirections) { + const targets: string[] = [] + for (const argument of redirection.args) targets.push(...await expandArgument(argument, this.context(state))) + const target = targets[0] + if (target === undefined || targets.length > 1) { + io.err('bash: ambiguous redirect\n') + return 1 + } + try { + switch (redirection.subtype) { + case '<': + stdin = await this.fs.readText(resolveIn(state.cwd, target)) + break + case '<<<': + stdin = `${target}\n` + break + case '>': + case '>>': { + const path = resolveIn(state.cwd, target) + // Truncation happens at redirect time, so `> file` empties it even + // when the command writes nothing. + if (redirection.subtype === '>') await this.fs.writeText(path, '') + // Appends are ordered by the queue below: a sink is synchronous to + // its caller, so writes are chained rather than raced. + let pending: Promise = Promise.resolve() + const sink = (text: string): void => { + pending = pending.then(async () => { await this.fs.writeText(path, text, true) }) + writes.push(pending) + } + if (redirection.fd === 2) err = sink + else out = sink + break + } + case '>&': { + // Only descriptor duplication between stdout and stderr is + // meaningful here: those are the only two the shell owns. + if (redirection.fd === 2 && target === '1') err = out + else if ((redirection.fd === null || redirection.fd === 1) && target === '2') out = err + else { + io.err(`bash: ${String(redirection.fd ?? 1)}>&${target}: unsupported descriptor redirection\n`) + return 1 + } + break + } + case '<&': + io.err(`bash: <&${target}: unsupported descriptor redirection\n`) + return 1 + } + } catch (error) { + io.err(`${describeFailure('bash', resolveIn(state.cwd, target), error)}\n`) + return 1 + } + } + const status = await body({ stdin, out, err }) + await Promise.all(writes) + return status + } + + /** The expansion hook: `$( … )` runs on a nested interpreter of the same table. */ + private context(state: ShellState): ExpansionContext { + return { + state, + fs: this.fs, + substitute: async (shell: ShellLine): Promise => { + if (this.depth >= MAX_SUBSTITUTION_DEPTH) { + throw new Error(`command substitution nested deeper than ${String(MAX_SUBSTITUTION_DEPTH)} levels`) + } + const captured = buffer() + const nested = { ...state, environment: { ...state.environment }, variables: { ...state.variables } } + const inner = new Interpreter(this.programs, this.fs, this.signal, this.depth + 1) + await inner.line(shell, nested, { stdin: '', out: captured.write, err: () => {} }) + return captured.text().replace(/\n+$/, '') + }, + } + } + + /** Expand the right-hand side of one `NAME=value` assignment. */ + private async assignedValue(argument: ValueArgument | undefined, state: ShellState): Promise { + if (argument === undefined) return '' + return (await expandArgument(argument, this.context(state))).join(' ') + } +} + +/** + * Record one assignment. An exported name keeps its export (the environment + * copy is what programs read); anything else stays a shell variable. + */ +function assign(state: ShellState, name: string, value: string): void { + if (name in state.environment) state.environment[name] = value + else state.variables[name] = value +} diff --git a/packages/experimental/webworker-runtime/src/shell/process/child.ts b/packages/experimental/webworker-runtime/src/shell/process/child.ts new file mode 100644 index 0000000000..ad3250e229 --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/process/child.ts @@ -0,0 +1,94 @@ +/** + * The process worker's own half: a fresh worker that received a + * {@link ShellStartFrame} runs one command here and then closes. + * + * It mounts no VFS image, boots no Cordis tree, and loads no plugins — the + * only thing it shares with the host worker is the bundle it was started from. + * Its filesystem is the host's, reached by message. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/process/child + */ + +import { runShellCommand, runShellProgram } from '../interpret.ts' +import { filesystemError } from '../fs-access.ts' +import type { ShellDirent, ShellFileSystem, ShellStats } from '../types.ts' +import type { FilesystemOperation, FromProcessFrame, ShellStartFrame, ToProcessFrame } from './protocol.ts' + +/** The messaging face this module needs from a worker scope. */ +export interface ProcessScope { + postMessage(frame: FromProcessFrame): void + addEventListener(type: 'message', listener: (event: MessageEvent) => void): void + close(): void +} + +/** + * Run one command as this worker's whole purpose, then close. + * + * Output is forwarded as it is written, so a caller reading a background job + * sees progress before the command settles. + * @param start - the frame that named the command, its directory, and its input. + * @param scope - the worker scope to message through (`self`). + */ +export function runShellProcess(start: ShellStartFrame, scope: ProcessScope): void { + const pending = new Map void; fail: (error: unknown) => void }>() + const stopping = new AbortController() + let nextCall = 0 + + scope.addEventListener('message', (event: MessageEvent) => { + const frame = event.data as ToProcessFrame + if (frame.t === 'shell-signal') { + // The host's first termination rung: the command stops at its next + // command boundary. A command that ignores it gets terminated instead. + stopping.abort(new Error('killed by signal')) + return + } + if (frame.t !== 'fs-reply') return + const waiting = pending.get(frame.id) + if (waiting === undefined) return + pending.delete(frame.id) + if (frame.failure === undefined) waiting.settle(frame.value) + else waiting.fail(filesystemError(frame.failure.code ?? 'EIO', 'fs', frame.failure.message)) + }) + + const call = async (op: FilesystemOperation, args: readonly unknown[]): Promise => { + nextCall += 1 + const id = nextCall + const reply = new Promise((settle, fail) => { pending.set(id, { settle, fail }) }) + scope.postMessage({ t: 'fs-call', id, op, args }) + return await reply + } + + const fs: ShellFileSystem = { + stat: async (path: string) => await call('stat', [path]) as ShellStats | undefined, + list: async (path: string) => await call('list', [path]) as ShellDirent[], + readText: async (path: string) => await call('readText', [path]) as string, + writeText: async (path: string, text: string, append = false) => { await call('writeText', [path, text, append]) }, + mkdir: async (path: string, recursive: boolean) => { await call('mkdir', [path, recursive]) }, + remove: async (path: string, options: { recursive: boolean; force: boolean }) => { await call('remove', [path, options]) }, + rename: async (from: string, to: string) => { await call('rename', [from, to]) }, + } + + const options = { + cwd: start.cwd, + env: start.env, + stdin: start.stdin, + signal: stopping.signal, + fs, + onOutput: (stream: 'stdout' | 'stderr', text: string) => { scope.postMessage({ t: 'shell-out', stream, text }) }, + } + const run = start.script === undefined + ? runShellProgram(start.argv, options) + : runShellCommand(start.script, options) + void run.then( + (outcome) => { + scope.postMessage({ t: 'shell-exit', code: outcome.exitCode }) + scope.close() + }, + (error: unknown) => { + // The interpreter contains its own failures; reaching here means the + // shell machinery itself broke, which the host reports as a failed spawn. + scope.postMessage({ t: 'shell-out', stream: 'stderr', text: `bash: ${String(error)}\n` }) + scope.postMessage({ t: 'shell-exit', code: 1 }) + scope.close() + }, + ) +} diff --git a/packages/experimental/webworker-runtime/src/shell/process/host.ts b/packages/experimental/webworker-runtime/src/shell/process/host.ts new file mode 100644 index 0000000000..3dbd386a02 --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/process/host.ts @@ -0,0 +1,181 @@ +/** + * Starting and supervising shell processes from the host worker. + * + * A process is a Web Worker started from this same bundle, told by its first + * frame to be a shell process rather than a host. That is what buys real + * process semantics in a browser: the command runs off the host's thread, and + * `terminate()` stops it even mid-loop — the one thing a cooperative in-thread + * interpreter can never do. + * + * Where no `Worker` constructor exists (a Node test host), the same command + * runs inline on this thread. Everything except preemption behaves the same, + * and the difference is named rather than hidden: {@link RunningProcess.destroy} + * can only ask an inline command to stop. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/process/host + */ + +import { runShellCommand, runShellProgram } from '../interpret.ts' +import { hostFileSystem } from '../fs-access.ts' +import type { ShellFileSystem } from '../types.ts' +import type { FilesystemOperation, FromProcessFrame, ShellStartFrame } from './protocol.ts' +import { runShellProcess } from './child.ts' +import type { ProcessScope } from './child.ts' + +/** What the caller must supply to start one process. */ +export interface ProcessStartOptions { + /** Command source for `bash -c`, or undefined when `argv` names a program. */ + script?: string | undefined + /** The program and its arguments. */ + argv: readonly string[] + /** Working directory the command starts in. */ + cwd: string + /** Environment the command starts with. */ + env: Record + /** Everything on standard input. */ + stdin: string + /** Receives output as it is produced. */ + onOutput: (stream: 'stdout' | 'stderr', text: string) => void + /** Receives the settled status exactly once. */ + onExit: (code: number) => void + /** The filesystem the command acts on; defaults to the mounted VFS. */ + fs?: ShellFileSystem | undefined +} + +/** A started command, from the host's side. */ +export interface RunningProcess { + /** Ask the command to stop at its next command boundary (the `SIGTERM` rung). */ + interrupt(): void + /** + * Stop the command now (the `SIGKILL` rung). A worker-backed process dies + * whatever it was doing; an inline one can only be asked, because nothing + * can preempt a synchronous loop on its own thread. + */ + destroy(): void +} + +/** Whether this thread can start a real process worker. */ +function canSpawnWorker(): boolean { + return typeof Worker === 'function' && typeof self !== 'undefined' && typeof self.location.href === 'string' +} + +/** + * Start one command. + * @param options - the command, its environment, and the sinks for its output and status. + * @returns the handle the process table signals through. + */ +export function startProcess(options: ProcessStartOptions): RunningProcess { + return canSpawnWorker() ? startWorkerProcess(options) : startInlineProcess(options) +} + +/** Serve one filesystem call for a process worker. */ +async function serveFilesystemCall(fs: ShellFileSystem, op: FilesystemOperation, args: readonly unknown[]): Promise { + switch (op) { + case 'stat': return await fs.stat(args[0] as string) + case 'list': return await fs.list(args[0] as string) + case 'readText': return await fs.readText(args[0] as string) + case 'writeText': + await fs.writeText(args[0] as string, args[1] as string, args[2] as boolean) + return undefined + case 'mkdir': + await fs.mkdir(args[0] as string, args[1] as boolean) + return undefined + case 'remove': + await fs.remove(args[0] as string, args[1] as { recursive: boolean; force: boolean }) + return undefined + case 'rename': + await fs.rename(args[0] as string, args[1] as string) + return undefined + default: + // The op crossed a worker postMessage: a name the union does not carry + // must fail the call rather than answer `{ value: undefined }`. + throw new Error(`webworker shell: unknown filesystem op ${String(op)}`) + } +} + +/** The worker-backed process: a second copy of this bundle, running one command. */ +function startWorkerProcess(options: ProcessStartOptions): RunningProcess { + const fs = options.fs ?? hostFileSystem() + // Same bundle, different role: the first frame decides. Starting from this + // worker's own URL keeps the deployment free of a second static asset and of + // the build-order trap a sibling artifact would bring. + const worker = new Worker(self.location.href, { type: 'module' }) + let settled = false + const settle = (code: number): void => { + if (settled) return + settled = true + worker.terminate() + options.onExit(code) + } + + worker.addEventListener('message', (event: MessageEvent) => { + const frame = event.data as FromProcessFrame + if (frame.t === 'shell-out') { + options.onOutput(frame.stream, frame.text) + return + } + if (frame.t === 'shell-exit') { + settle(frame.code) + return + } + void serveFilesystemCall(fs, frame.op, frame.args).then( + (value) => { worker.postMessage({ t: 'fs-reply', id: frame.id, value }) }, + (error: unknown) => { + const failure = { + code: (error as { code?: string }).code, + message: error instanceof Error ? error.message : String(error), + } + worker.postMessage({ t: 'fs-reply', id: frame.id, failure }) + }, + ) + }) + worker.addEventListener('error', (event: ErrorEvent) => { + options.onOutput('stderr', `bash: process worker failed: ${event.message}\n`) + settle(1) + }) + + const start: ShellStartFrame = { + t: 'shell-start', + script: options.script, + argv: options.argv, + cwd: options.cwd, + env: options.env, + stdin: options.stdin, + } + worker.postMessage(start) + + return { + interrupt: () => { if (!settled) worker.postMessage({ t: 'shell-signal' }) }, + // The reason this whole module exists: a worker dies on command, even + // mid-loop, so a timeout is enforceable rather than advisory. + destroy: () => { settle(130) }, + } +} + +/** The inline process: the same command on this thread, stoppable only by asking. */ +function startInlineProcess(options: ProcessStartOptions): RunningProcess { + const stopping = new AbortController() + const runOptions = { + cwd: options.cwd, + env: options.env, + stdin: options.stdin, + signal: stopping.signal, + fs: options.fs ?? hostFileSystem(), + onOutput: options.onOutput, + } + const run = options.script === undefined + ? runShellProgram(options.argv, runOptions) + : runShellCommand(options.script, runOptions) + void run.then( + (outcome) => { options.onExit(outcome.exitCode) }, + (error: unknown) => { + options.onOutput('stderr', `bash: ${String(error)}\n`) + options.onExit(1) + }, + ) + const stop = (): void => { stopping.abort(new Error('killed by signal')) } + return { interrupt: stop, destroy: stop } +} + +/** Re-exported for the worker entry, which decides its role from the first frame. */ +export { runShellProcess } +export type { ProcessScope } diff --git a/packages/experimental/webworker-runtime/src/shell/process/protocol.ts b/packages/experimental/webworker-runtime/src/shell/process/protocol.ts new file mode 100644 index 0000000000..187a18af22 --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/process/protocol.ts @@ -0,0 +1,82 @@ +/** + * The frames a shell process and its host exchange. + * + * A command runs in its own Web Worker, which owns no filesystem: the VFS + * stays in the host worker and every read or write is a request on this + * channel. Blocking the child on a reply is impossible here (that would need + * `SharedArrayBuffer`, which requires a cross-origin isolation this deployment + * cannot have), so the filesystem face is asynchronous end to end. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/process/protocol + */ + +/** The first frame a process worker receives; it also selects its role. */ +export interface ShellStartFrame { + t: 'shell-start' + /** Command source for `bash -c`, or undefined when `argv` names a program directly. */ + script?: string | undefined + /** The program and arguments, used when `script` is absent. */ + argv: readonly string[] + /** Working directory the command starts in. */ + cwd: string + /** Environment the command starts with. */ + env: Record + /** Everything on standard input. */ + stdin: string +} + +/** Output produced by the command, forwarded as it is written. */ +export interface ShellOutputFrame { + t: 'shell-out' + stream: 'stdout' | 'stderr' + text: string +} + +/** The command settled; the process worker closes itself right after. */ +export interface ShellExitFrame { + t: 'shell-exit' + code: number +} + +/** A signal the command is asked to honor (the terminate ladder's first rung). */ +export interface ShellSignalFrame { + t: 'shell-signal' +} + +/** Filesystem operations a process worker can ask its host to perform. */ +export type FilesystemOperation = 'stat' | 'list' | 'readText' | 'writeText' | 'mkdir' | 'remove' | 'rename' + +/** One filesystem call, awaiting its reply by `id`. */ +export interface FilesystemCallFrame { + t: 'fs-call' + id: number + op: FilesystemOperation + args: readonly unknown[] +} + +/** + * One filesystem reply. A failure carries the Node error `code` because the + * utilities branch on it (`ENOENT` prints "No such file or directory"), and an + * Error instance does not survive structured cloning with its class. + */ +export interface FilesystemReplyFrame { + t: 'fs-reply' + id: number + value?: unknown + failure?: { code?: string | undefined; message: string } +} + +/** Everything the host sends to a process worker. */ +export type ToProcessFrame = ShellStartFrame | ShellSignalFrame | FilesystemReplyFrame + +/** Everything a process worker sends to its host. */ +export type FromProcessFrame = ShellOutputFrame | ShellExitFrame | FilesystemCallFrame + +/** + * Whether a message is the frame that turns a fresh worker into a shell + * process. The host worker's entry reads this to pick its role. + * @param data - the raw message payload. + * @returns true when the payload starts a shell process. + */ +export function isShellStartFrame(data: unknown): data is ShellStartFrame { + return typeof data === 'object' && data !== null && (data as { t?: unknown }).t === 'shell-start' +} diff --git a/packages/experimental/webworker-runtime/src/shell/programs/builtins.ts b/packages/experimental/webworker-runtime/src/shell/programs/builtins.ts new file mode 100644 index 0000000000..157fb513ba --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/programs/builtins.ts @@ -0,0 +1,192 @@ +/** + * Shell builtins: the programs that read or change the shell's own state + * (directory, environment, exit status) rather than the filesystem. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/programs/builtins + */ + +import { readVariable } from '../expand.ts' +import { resolveIn } from '../fs-access.ts' +import type { ShellProgram, ShellStats } from '../types.ts' +import { parseOptions } from './options.ts' + +/** Status a command reports when a signal ended it, as a shell renders `128 + SIGINT`. */ +const SIGNAL_EXIT_STATUS = 130 + +const cd: ShellProgram = async (argv, io, state, fs) => { + const target = argv[1] ?? state.environment['HOME'] ?? '/' + const path = target === '-' ? state.variables['OLDPWD'] ?? state.cwd : resolveIn(state.cwd, target) + const stats = await fs.stat(path) + if (stats === undefined) { + io.err(`cd: ${target}: No such file or directory\n`) + return 1 + } + if (!stats.directory) { + io.err(`cd: ${target}: Not a directory\n`) + return 1 + } + state.variables['OLDPWD'] = state.cwd + state.cwd = path + // `$PWD` is what scripts read back, so it has to follow the real directory. + if ('PWD' in state.environment) state.environment['PWD'] = path + return 0 +} + +const pwd: ShellProgram = (_argv, io, state) => { + io.out(`${state.cwd}\n`) + return 0 +} + +const exportProgram: ShellProgram = (argv, io, state) => { + const options = parseOptions(argv) + if (options.operands.length === 0) { + for (const [name, value] of Object.entries(state.environment).sort()) io.out(`declare -x ${name}="${value}"\n`) + return 0 + } + for (const operand of options.operands) { + const separator = operand.indexOf('=') + if (separator < 0) { + // Exporting an existing shell variable moves it into the environment. + state.environment[operand] = state.variables[operand] ?? state.environment[operand] ?? '' + continue + } + state.environment[operand.slice(0, separator)] = operand.slice(separator + 1) + } + return 0 +} + +const unset: ShellProgram = (argv, _io, state) => { + const removed = new Set(argv.slice(1)) + const without = (source: Record): Record => + Object.fromEntries(Object.entries(source).filter(([name]) => !removed.has(name))) + state.environment = without(state.environment) + state.variables = without(state.variables) + return 0 +} + +const env: ShellProgram = (_argv, io, state) => { + for (const [name, value] of Object.entries(state.environment).sort()) io.out(`${name}=${value}\n`) + return 0 +} + +const exitProgram: ShellProgram = (argv, _io, state) => { + const status = argv[1] === undefined ? state.lastStatus : Number.parseInt(argv[1], 10) || 0 + state.exitRequested = status + return status +} + +/** `test` / `[`: the file and string predicates a generated command line uses. */ +const test: ShellProgram = async (argv, io, state, fs) => { + const words = argv[0] === '[' ? argv.slice(1, argv[argv.length - 1] === ']' ? -1 : undefined) : argv.slice(1) + const status = (value: boolean): number => value ? 0 : 1 + const statOf = async (operand: string): Promise => await fs.stat(resolveIn(state.cwd, operand)) + if (words.length === 1) return status((words[0] as string) !== '') + if (words.length === 2) { + const operator = words[0] as string + const operand = words[1] ?? '' + switch (operator) { + case '-e': return status(await statOf(operand) !== undefined) + case '-f': return status((await statOf(operand))?.directory === false) + case '-d': return status((await statOf(operand))?.directory === true) + case '-s': return status(((await statOf(operand))?.size ?? 0) > 0) + case '-r': case '-w': return status(await statOf(operand) !== undefined) + case '-z': return status(operand === '') + case '-n': return status(operand !== '') + case '!': return status(operand === '') + default: + io.err(`test: ${operator}: unsupported unary operator\n`) + return 2 + } + } + if (words.length === 3) { + const [left, operator, right] = words as [string, string, string] + switch (operator) { + case '=': case '==': return status(left === right) + case '!=': return status(left !== right) + case '-eq': return status(Number(left) === Number(right)) + case '-ne': return status(Number(left) !== Number(right)) + case '-lt': return status(Number(left) < Number(right)) + case '-le': return status(Number(left) <= Number(right)) + case '-gt': return status(Number(left) > Number(right)) + case '-ge': return status(Number(left) >= Number(right)) + default: + io.err(`test: ${operator}: unsupported binary operator\n`) + return 2 + } + } + io.err('test: unsupported expression\n') + return 2 +} + +const sleep: ShellProgram = async (argv, io, state) => { + const seconds = Number.parseFloat(argv[1] ?? '') + if (!Number.isFinite(seconds) || seconds < 0) { + io.err(`sleep: invalid time interval '${argv[1] ?? ''}'\n`) + return 2 + } + // A killed command must settle at once: waiting out the full interval would + // keep the caller's process handle open long after its signal arrived. + const killed = await new Promise((settle) => { + const timer = setTimeout(() => { + state.signal?.removeEventListener('abort', onAbort) + settle(false) + }, seconds * 1000) + function onAbort(): void { + clearTimeout(timer) + settle(true) + } + if (state.signal?.aborted === true) onAbort() + else state.signal?.addEventListener('abort', onAbort, { once: true }) + }) + return killed ? SIGNAL_EXIT_STATUS : 0 +} + +const date: ShellProgram = (_argv, io) => { + io.out(`${new Date().toISOString()}\n`) + return 0 +} + +const seq: ShellProgram = (argv, io) => { + const numbers = argv.slice(1).map(value => Number.parseInt(value, 10)) + const [first, second, third] = numbers + const from = numbers.length > 1 ? first as number : 1 + const step = numbers.length > 2 ? second as number : 1 + const to = numbers.length > 2 ? third as number : numbers.length > 1 ? second as number : first + if (to === undefined || !Number.isFinite(to) || step === 0) { + io.err('seq: expected numeric bounds\n') + return 2 + } + for (let value = from; step > 0 ? value <= to : value >= to; value += step) io.out(`${String(value)}\n`) + return 0 +} + +/** `printenv NAME`, which scripts prefer over `echo $NAME` when the name is computed. */ +const printenv: ShellProgram = (argv, io, state) => { + const name = argv[1] + if (name === undefined) { + for (const [key, value] of Object.entries(state.environment).sort()) io.out(`${key}=${value}\n`) + return 0 + } + const value = readVariable(state, name) + if (value === undefined) return 1 + io.out(`${value}\n`) + return 0 +} + +/** The state builtins, keyed by the name a command line uses. */ +export const BUILTIN_PROGRAMS: Readonly> = { + cd, + pwd, + export: exportProgram, + unset, + env, + printenv, + exit: exitProgram, + test, + '[': test, + sleep, + date, + seq, + 'true': () => 0, + 'false': () => 1, + ':': () => 0, +} diff --git a/packages/experimental/webworker-runtime/src/shell/programs/files.ts b/packages/experimental/webworker-runtime/src/shell/programs/files.ts new file mode 100644 index 0000000000..e21f3e5308 --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/programs/files.ts @@ -0,0 +1,293 @@ +/** + * File and directory utilities of the command table, all of them over the + * shell's filesystem. Listings print one entry per line: nothing here is ever + * a terminal, so the column layout a real `ls` picks for a tty would only be + * noise in a tool result. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/programs/files + */ + +import picomatch from 'picomatch' +import { basename, dirname, resolve } from '../../module-system/posix-path.ts' +import { describeFailure, resolveIn } from '../fs-access.ts' +import type { ShellFileSystem, ShellProgram, ShellStats } from '../types.ts' +import { parseOptions } from './options.ts' + +/** Format one entry the way `ls -l` does, with the facts the VFS actually holds. */ +function longEntry(stats: ShellStats | undefined, name: string): string { + const size = String(stats?.size ?? 0).padStart(8) + const modified = new Date(stats?.mtimeMs ?? 0).toISOString().replace('T', ' ').slice(0, 16) + return `${stats?.directory === true ? 'drwxr-xr-x' : '-rw-r--r--'} ${size} ${modified} ${name}` +} + +const ls: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + const operands = options.operands.length > 0 ? options.operands : ['.'] + let status = 0 + for (const [index, operand] of operands.entries()) { + const path = resolveIn(state.cwd, operand) + const stats = await fs.stat(path) + if (stats === undefined) { + io.err(`ls: ${operand}: No such file or directory\n`) + status = 2 + continue + } + if (operands.length > 1) io.out(`${index > 0 ? '\n' : ''}${operand}:\n`) + if (!stats.directory) { + io.out(`${options.flags.has('l') ? longEntry(stats, operand) : operand}\n`) + continue + } + const entries = (await fs.list(path)).filter(entry => options.flags.has('a') || !entry.name.startsWith('.')) + for (const entry of entries) { + const shown = options.flags.has('l') + ? longEntry(await fs.stat(resolve(path, entry.name)), entry.name) + : entry.name + io.out(`${shown}\n`) + } + } + return status +} + +const find: ShellProgram = async (argv, io, state, fs) => { + // `find` spells multi-letter predicates with one dash, which the shared + // option parser would read as bundled short flags; this walk reads them. + const roots: string[] = [] + let namePattern: string | undefined + let kind: string | undefined + let maxDepth = Number.POSITIVE_INFINITY + const words = argv.slice(1) + for (let index = 0; index < words.length; index += 1) { + const word = words[index] as string + if (word === '-name') { index += 1; namePattern = words[index]; continue } + if (word === '-type') { index += 1; kind = words[index]; continue } + if (word === '-maxdepth') { index += 1; maxDepth = Number.parseInt(words[index] ?? '', 10); continue } + if (word.startsWith('-')) { + io.err(`find: unsupported predicate ${word}\n`) + return 2 + } + roots.push(word) + } + const matches = namePattern === undefined ? undefined : picomatch(namePattern, { dot: true }) + let status = 0 + const visit = async (path: string, display: string, depth: number): Promise => { + const stats = await fs.stat(path) + if (stats === undefined) { + io.err(`find: ${display}: No such file or directory\n`) + status = 1 + return + } + const selected = (matches === undefined || matches(basename(display))) + && (kind === undefined || (kind === 'd') === stats.directory) + if (selected) io.out(`${display}\n`) + if (!stats.directory || depth >= maxDepth) return + for (const entry of await fs.list(path)) { + await visit(resolve(path, entry.name), `${display === '/' ? '' : display}/${entry.name}`, depth + 1) + } + } + for (const root of roots.length > 0 ? roots : ['.']) await visit(resolveIn(state.cwd, root), root, 0) + return status +} + +const mkdir: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + let status = 0 + for (const operand of options.operands) { + try { + await fs.mkdir(resolveIn(state.cwd, operand), options.flags.has('p')) + } catch (error) { + io.err(`${describeFailure('mkdir', operand, error)}\n`) + status = 1 + } + } + return status +} + +const rmdir: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + let status = 0 + for (const operand of options.operands) { + const path = resolveIn(state.cwd, operand) + if ((await fs.list(path)).length > 0) { + io.err(`rmdir: ${operand}: Directory not empty\n`) + status = 1 + continue + } + try { + await fs.remove(path, { recursive: true, force: false }) + } catch (error) { + io.err(`${describeFailure('rmdir', operand, error)}\n`) + status = 1 + } + } + return status +} + +const rm: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + const recursive = options.flags.has('r') || options.flags.has('R') + const force = options.flags.has('f') + let status = 0 + for (const operand of options.operands) { + const path = resolveIn(state.cwd, operand) + const stats = await fs.stat(path) + if (stats === undefined) { + if (force) continue + io.err(`rm: ${operand}: No such file or directory\n`) + status = 1 + continue + } + if (stats.directory && !recursive) { + io.err(`rm: ${operand}: Is a directory\n`) + status = 1 + continue + } + try { + await fs.remove(path, { recursive, force }) + } catch (error) { + io.err(`${describeFailure('rm', operand, error)}\n`) + status = 1 + } + } + return status +} + +/** Copy one file or one whole subtree. */ +async function copyTree(from: string, to: string, fs: ShellFileSystem): Promise { + const stats = await fs.stat(from) + if (stats?.directory !== true) { + await fs.writeText(to, await fs.readText(from)) + return + } + await fs.mkdir(to, true) + for (const entry of await fs.list(from)) await copyTree(resolve(from, entry.name), resolve(to, entry.name), fs) +} + +/** Resolve the real destination of a copy or move: into a directory, or onto a path. */ +async function destinationFor(target: string, source: string, fs: ShellFileSystem): Promise { + return (await fs.stat(target))?.directory === true ? resolve(target, basename(source)) : target +} + +const cp: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + const sources = options.operands.slice(0, -1) + const target = options.operands[options.operands.length - 1] + if (target === undefined || sources.length === 0) { + io.err('cp: expected a source and a destination\n') + return 2 + } + const targetPath = resolveIn(state.cwd, target) + let status = 0 + for (const source of sources) { + const sourcePath = resolveIn(state.cwd, source) + const stats = await fs.stat(sourcePath) + if (stats === undefined) { + io.err(`cp: ${source}: No such file or directory\n`) + status = 1 + continue + } + if (stats.directory && !(options.flags.has('r') || options.flags.has('R'))) { + io.err(`cp: ${source}: Is a directory\n`) + status = 1 + continue + } + try { + await copyTree(sourcePath, await destinationFor(targetPath, source, fs), fs) + } catch (error) { + io.err(`${describeFailure('cp', source, error)}\n`) + status = 1 + } + } + return status +} + +const mv: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + const sources = options.operands.slice(0, -1) + const target = options.operands[options.operands.length - 1] + if (target === undefined || sources.length === 0) { + io.err('mv: expected a source and a destination\n') + return 2 + } + const targetPath = resolveIn(state.cwd, target) + let status = 0 + for (const source of sources) { + try { + await fs.rename(resolveIn(state.cwd, source), await destinationFor(targetPath, source, fs)) + } catch (error) { + io.err(`${describeFailure('mv', source, error)}\n`) + status = 1 + } + } + return status +} + +const touch: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + let status = 0 + for (const operand of options.operands) { + const path = resolveIn(state.cwd, operand) + try { + // Rewriting the existing bytes is what advances the VFS timestamp. + await fs.writeText(path, await fs.stat(path) === undefined ? '' : await fs.readText(path)) + } catch (error) { + io.err(`${describeFailure('touch', operand, error)}\n`) + status = 1 + } + } + return status +} + +const stat: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + let status = 0 + for (const operand of options.operands) { + const path = resolveIn(state.cwd, operand) + const stats = await fs.stat(path) + if (stats === undefined) { + io.err(`stat: ${operand}: No such file or directory\n`) + status = 1 + continue + } + io.out(`${path} ${stats.directory ? 'directory' : 'file'} ${String(stats.size)} ${new Date(stats.mtimeMs).toISOString()}\n`) + } + return status +} + +const dirnameProgram: ShellProgram = (argv, io) => { + for (const operand of argv.slice(1)) io.out(`${dirname(operand)}\n`) + return argv.length > 1 ? 0 : 2 +} + +const basenameProgram: ShellProgram = (argv, io) => { + const [, path, suffix] = argv + if (path === undefined) { + io.err('basename: expected a path\n') + return 2 + } + io.out(`${basename(path, suffix)}\n`) + return 0 +} + +/** Refuse a utility whose effect the VFS cannot represent at all. */ +const unavailable = (name: string): ShellProgram => (_argv, io) => { + io.err(`${name}: not available in the worker host\n`) + return 127 +} + +/** The file utilities, keyed by the name a command line uses. */ +export const FILE_PROGRAMS: Readonly> = { + ls, + find, + mkdir, + rmdir, + rm, + cp, + mv, + touch, + stat, + dirname: dirnameProgram, + basename: basenameProgram, + // Symbolic links have no representation in the VFS; refusing is honest and + // keeps a script from believing it created one. + ln: unavailable('ln'), + readlink: unavailable('readlink'), +} diff --git a/packages/experimental/webworker-runtime/src/shell/programs/index.ts b/packages/experimental/webworker-runtime/src/shell/programs/index.ts new file mode 100644 index 0000000000..2034af376c --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/programs/index.ts @@ -0,0 +1,45 @@ +/** + * The command table: every program name this shell can run. A browser worker + * spawns no processes, so this table IS the machine's `/bin` — a name that is + * not here reports `command not found`, exactly as a real shell would for a + * binary that is not installed. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/programs + */ + +import type { ShellProgram } from '../types.ts' +import { BUILTIN_PROGRAMS } from './builtins.ts' +import { FILE_PROGRAMS } from './files.ts' +import { TEXT_PROGRAMS } from './text.ts' + +let table: Map | undefined + +/** + * The standard command table, built once and shared by every command line. + * @returns the program table, keyed by command name. + */ +export function standardPrograms(): ReadonlyMap { + table ??= new Map([ + ...Object.entries(BUILTIN_PROGRAMS), + ...Object.entries(FILE_PROGRAMS), + ...Object.entries(TEXT_PROGRAMS), + ['which', which], + ]) + return table +} + +/** Reports which of the requested names this shell can run. */ +const which: ShellProgram = (argv, io) => { + const known = standardPrograms() + let status = 0 + for (const name of argv.slice(1)) { + // Every program is built into the shell, so a known name reports itself + // instead of a path that would not exist in the VFS. + if (known.has(name)) { + io.out(`${name}: shell built-in command\n`) + continue + } + io.err(`which: no ${name} in the worker host command table\n`) + status = 1 + } + return status +} diff --git a/packages/experimental/webworker-runtime/src/shell/programs/options.ts b/packages/experimental/webworker-runtime/src/shell/programs/options.ts new file mode 100644 index 0000000000..14471f2f73 --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/programs/options.ts @@ -0,0 +1,96 @@ +/** + * Argument splitting shared by the command table: short flags (bundled or + * separate), long flags, `--`, and the operands that follow. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/programs/options + */ + +/** One parsed argv: which flags were given, and what is left to act on. */ +export interface ParsedOptions { + /** Every short letter and long name seen, without their dashes. */ + readonly flags: ReadonlySet + /** Values of flags that take one (`-n 5` and `--name=x` both land here). */ + readonly values: ReadonlyMap + /** Everything that is not a flag, in order. */ + readonly operands: readonly string[] +} + +/** + * Split one program's arguments. + * + * A short letter listed in `valued` consumes the rest of its token (`-n5`) or + * the next argument (`-n 5`); every other letter is a plain flag, so `-rn` + * sets both `r` and `n`. + * @param argv - the program's argv, including its name at index 0. + * @param valued - short letters that take a value. + * @returns the flags, their values, and the operands. + */ +export function parseOptions(argv: readonly string[], valued: ReadonlySet = new Set()): ParsedOptions { + const flags = new Set() + const values = new Map() + const operands: string[] = [] + const rest = argv.slice(1) + let literal = false + for (let index = 0; index < rest.length; index += 1) { + const argument = rest[index] as string + if (literal || argument === '-' || !argument.startsWith('-')) { + operands.push(argument) + continue + } + if (argument === '--') { + literal = true + continue + } + if (argument.startsWith('--')) { + const [name, value] = splitLong(argument.slice(2)) + flags.add(name) + if (value !== undefined) values.set(name, value) + continue + } + for (let cursor = 1; cursor < argument.length; cursor += 1) { + const letter = argument[cursor] as string + flags.add(letter) + if (!valued.has(letter)) continue + const inline = argument.slice(cursor + 1) + if (inline !== '') { + values.set(letter, inline) + } else { + index += 1 + values.set(letter, rest[index] ?? '') + } + break + } + } + return { flags, values, operands } +} + +/** Split `name=value`; a long flag without `=` has no value. */ +function splitLong(text: string): [string, string | undefined] { + const separator = text.indexOf('=') + return separator < 0 ? [text, undefined] : [text.slice(0, separator), text.slice(separator + 1)] +} + +/** + * Read a numeric flag value. + * @param options - the parsed options. + * @param flag - the short letter to read. + * @param fallback - value to use when the flag is absent or unparsable. + * @returns the number the caller should use. + */ +export function numberOption(options: ParsedOptions, flag: string, fallback: number): number { + const raw = options.values.get(flag) + if (raw === undefined) return fallback + const parsed = Number.parseInt(raw, 10) + return Number.isFinite(parsed) ? parsed : fallback +} + +/** + * Split text into lines for the line-oriented utilities. + * @param text - the text to split. + * @returns its lines, without the trailing empty line a final newline creates. + */ +export function toLines(text: string): string[] { + if (text === '') return [] + const lines = text.split('\n') + if (lines[lines.length - 1] === '') lines.pop() + return lines +} diff --git a/packages/experimental/webworker-runtime/src/shell/programs/text.ts b/packages/experimental/webworker-runtime/src/shell/programs/text.ts new file mode 100644 index 0000000000..6cde6c173b --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/programs/text.ts @@ -0,0 +1,363 @@ +/** + * Text utilities of the command table. Each one reads its operands as files + * and falls back to standard input, the way its POSIX counterpart does. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/programs/text + */ + +import { describeFailure, resolveIn } from '../fs-access.ts' +import type { ShellFileSystem, ShellIo, ShellProgram, ShellState } from '../types.ts' +import { numberOption, parseOptions, toLines } from './options.ts' + +/** + * Read every operand as a file, reporting the ones that fail. + * @param program - name used in diagnostics. + * @param operands - paths to read; empty means standard input. + * @param io - source of standard input and sink for diagnostics. + * @param state - shell state supplying the working directory. + * @param fs - the filesystem to read from. + * @returns one entry per readable source and the status the program should report. + */ +async function readInputs( + program: string, + operands: readonly string[], + io: ShellIo, + state: ShellState, + fs: ShellFileSystem, +): Promise<{ sources: { name: string; text: string }[]; status: number }> { + if (operands.length === 0) return { sources: [{ name: '-', text: io.stdin }], status: 0 } + const sources: { name: string; text: string }[] = [] + let status = 0 + for (const operand of operands) { + if (operand === '-') { + sources.push({ name: '-', text: io.stdin }) + continue + } + const path = resolveIn(state.cwd, operand) + try { + sources.push({ name: operand, text: await fs.readText(path) }) + } catch (error) { + io.err(`${describeFailure(program, operand, error)}\n`) + status = 1 + } + } + return { sources, status } +} + +/** Append a trailing newline unless the text already ends with one. */ +function terminated(text: string): string { + return text === '' || text.endsWith('\n') ? text : `${text}\n` +} + +const echo: ShellProgram = (argv, io) => { + const suppressNewline = argv[1] === '-n' + const words = argv.slice(suppressNewline ? 2 : 1) + io.out(`${words.join(' ')}${suppressNewline ? '' : '\n'}`) + return 0 +} + +const printf: ShellProgram = (argv, io) => { + const format = argv[1] ?? '' + const operands = argv.slice(2) + let cursor = 0 + // The conversions a shell script realistically uses; anything else is left + // verbatim so the output shows what was not understood. + const rendered = format.replace(/%[sdi%]/g, (match) => { + if (match === '%%') return '%' + const value = operands[cursor] ?? '' + cursor += 1 + if (match === '%s') return value + const parsed = Number.parseInt(value, 10) + return String(Number.isFinite(parsed) ? parsed : 0) + }) + io.out(rendered.replace(/\\n/g, '\n').replace(/\\t/g, '\t')) + return 0 +} + +const cat: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + const { sources, status } = await readInputs('cat', options.operands, io, state, fs) + let line = 1 + for (const source of sources) { + if (!options.flags.has('n')) { + io.out(source.text) + continue + } + for (const content of toLines(source.text)) { + io.out(`${String(line).padStart(6)}\t${content}\n`) + line += 1 + } + } + return status +} + +const head: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv, new Set(['n'])) + const count = numberOption(options, 'n', 10) + const { sources, status } = await readInputs('head', options.operands, io, state, fs) + for (const [index, source] of sources.entries()) { + if (sources.length > 1) io.out(`${index > 0 ? '\n' : ''}==> ${source.name} <==\n`) + io.out(terminated(toLines(source.text).slice(0, count).join('\n'))) + } + return status +} + +const tail: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv, new Set(['n'])) + const count = numberOption(options, 'n', 10) + const { sources, status } = await readInputs('tail', options.operands, io, state, fs) + for (const [index, source] of sources.entries()) { + if (sources.length > 1) io.out(`${index > 0 ? '\n' : ''}==> ${source.name} <==\n`) + io.out(terminated(toLines(source.text).slice(-count).join('\n'))) + } + return status +} + +const wc: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + const { sources, status } = await readInputs('wc', options.operands, io, state, fs) + const selected = ['l', 'w', 'c'].filter(flag => options.flags.has(flag)) + const columns = selected.length > 0 ? selected : ['l', 'w', 'c'] + for (const source of sources) { + const counts: Record = { + l: toLines(source.text).length, + w: source.text.split(/\s+/).filter(word => word !== '').length, + c: source.text.length, + } + const cells = columns.map(column => String(counts[column] ?? 0).padStart(columns.length > 1 ? 8 : 1)) + io.out(`${cells.join(' ')}${source.name === '-' ? '' : ` ${source.name}`}\n`) + } + return status +} + +/** Collect every file under one directory, for `grep -r`. */ +async function walkFiles( + path: string, + display: string, + into: { path: string; display: string }[], + fs: ShellFileSystem, +): Promise { + for (const entry of await fs.list(path)) { + const child = `${path.endsWith('/') ? path : `${path}/`}${entry.name}` + const shown = `${display.endsWith('/') ? display : `${display}/`}${entry.name}` + if (entry.directory) await walkFiles(child, shown, into, fs) + else into.push({ path: child, display: shown }) + } +} + +const grep: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv, new Set(['e'])) + const pattern = options.values.get('e') ?? options.operands[0] + const targets = options.values.get('e') === undefined ? options.operands.slice(1) : options.operands + if (pattern === undefined) { + io.err('grep: no pattern given\n') + return 2 + } + // Patterns are JavaScript regular expressions; `-F` matches them literally. + const source = options.flags.has('F') ? pattern.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') : pattern + let matcher: RegExp + try { + matcher = new RegExp(source, options.flags.has('i') ? 'i' : '') + } catch (error) { + io.err(`grep: invalid pattern: ${error instanceof Error ? error.message : String(error)}\n`) + return 2 + } + + const sources: { name: string; text: string }[] = [] + let status = 0 + if (targets.length === 0) { + sources.push({ name: '', text: io.stdin }) + } else { + for (const target of targets) { + const path = resolveIn(state.cwd, target) + const stats = await fs.stat(path) + if (stats?.directory === true) { + if (!options.flags.has('r') && !options.flags.has('R')) { + io.err(`grep: ${target}: Is a directory\n`) + status = Math.max(status, 2) + continue + } + const files: { path: string; display: string }[] = [] + await walkFiles(path, target, files, fs) + for (const file of files) sources.push({ name: file.display, text: await fs.readText(file.path) }) + continue + } + try { + sources.push({ name: target, text: await fs.readText(path) }) + } catch (error) { + io.err(`${describeFailure('grep', target, error)}\n`) + status = Math.max(status, 2) + } + } + } + + const label = sources.length > 1 || options.flags.has('H') + let matched = false + for (const entry of sources) { + const hits = toLines(entry.text) + .map((text, index) => ({ text, number: index + 1 })) + .filter(line => matcher.test(line.text) !== options.flags.has('v')) + if (hits.length > 0) matched = true + if (options.flags.has('l')) { + if (hits.length > 0) io.out(`${entry.name}\n`) + continue + } + if (options.flags.has('c')) { + io.out(`${label && entry.name !== '' ? `${entry.name}:` : ''}${String(hits.length)}\n`) + continue + } + for (const hit of hits) { + const prefix = `${label && entry.name !== '' ? `${entry.name}:` : ''}${options.flags.has('n') ? `${String(hit.number)}:` : ''}` + io.out(`${prefix}${hit.text}\n`) + } + } + // `grep` reports "nothing matched" as status 1, distinct from an error. + return status !== 0 ? status : matched ? 0 : 1 +} + +const sort: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + const { sources, status } = await readInputs('sort', options.operands, io, state, fs) + let lines = sources.flatMap(source => toLines(source.text)) + lines = options.flags.has('n') + ? [...lines].sort((left, right) => (Number.parseFloat(left) || 0) - (Number.parseFloat(right) || 0)) + : [...lines].sort((left, right) => left < right ? -1 : left > right ? 1 : 0) + if (options.flags.has('r')) lines.reverse() + if (options.flags.has('u')) lines = [...new Set(lines)] + io.out(terminated(lines.join('\n'))) + return status +} + +const uniq: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + const { sources, status } = await readInputs('uniq', options.operands, io, state, fs) + const lines = sources.flatMap(source => toLines(source.text)) + const groups: { text: string; count: number }[] = [] + for (const line of lines) { + const previous = groups[groups.length - 1] + if (previous !== undefined && previous.text === line) previous.count += 1 + else groups.push({ text: line, count: 1 }) + } + const selected = options.flags.has('d') + ? groups.filter(group => group.count > 1) + : options.flags.has('u') ? groups.filter(group => group.count === 1) : groups + for (const group of selected) { + io.out(`${options.flags.has('c') ? `${String(group.count).padStart(7)} ` : ''}${group.text}\n`) + } + return status +} + +const cut: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv, new Set(['d', 'f', 'c'])) + const delimiter = options.values.get('d') ?? '\t' + const fields = (options.values.get('f') ?? '').split(',').map(field => Number.parseInt(field, 10)).filter(Number.isFinite) + const characters = options.values.get('c') + const { sources, status } = await readInputs('cut', options.operands, io, state, fs) + if (fields.length === 0 && characters === undefined) { + io.err('cut: expected -f or -c\n') + return 2 + } + for (const source of sources) { + for (const line of toLines(source.text)) { + if (characters !== undefined) { + const [from, to] = characters.split('-') + const start = Number.parseInt(from ?? '1', 10) || 1 + const end = to === undefined || to === '' ? start : Number.parseInt(to, 10) + io.out(`${line.slice(start - 1, end)}\n`) + continue + } + const parts = line.split(delimiter) + io.out(`${fields.map(field => parts[field - 1] ?? '').join(delimiter)}\n`) + } + } + return status +} + +/** Expand one `tr` set: `a-z` becomes every character in that range. */ +function characterSet(set: string): string[] { + // oxlint-disable-next-line typescript/no-misused-spread -- a `tr` set names characters, and code points are that unit. + const characters = [...set] + const expanded: string[] = [] + for (let index = 0; index < characters.length; index += 1) { + const start = characters[index] as string + const end = characters[index + 2] + if (characters[index + 1] === '-' && end !== undefined) { + for (let code = start.codePointAt(0) as number; code <= (end.codePointAt(0) as number); code += 1) { + expanded.push(String.fromCodePoint(code)) + } + index += 2 + continue + } + expanded.push(start) + } + return expanded +} + +const tr: ShellProgram = (argv, io) => { + const options = parseOptions(argv) + const [fromSet, toSet] = options.operands + const from = fromSet === undefined ? undefined : characterSet(fromSet).join('') + const to = toSet === undefined ? undefined : characterSet(toSet).join('') + if (from === undefined) { + io.err('tr: expected a source set\n') + return 2 + } + if (options.flags.has('d')) { + // oxlint-disable-next-line typescript/no-misused-spread -- `tr` deletes per character, and code points are the unit it deletes. + io.out([...io.stdin].filter(character => !from.includes(character)).join('')) + return 0 + } + if (to === undefined) { + io.err('tr: expected a replacement set\n') + return 2 + } + // oxlint-disable-next-line typescript/no-misused-spread -- `tr` translates per character, and code points are the unit it maps. + io.out([...io.stdin].map((character) => { + const index = from.indexOf(character) + return index < 0 ? character : to[Math.min(index, to.length - 1)] as string + }).join('')) + return 0 +} + +/** `sed` accepts only the substitute command; anything else is reported, not guessed at. */ +const sed: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv, new Set(['e'])) + const script = options.values.get('e') ?? options.operands[0] + const targets = options.values.get('e') === undefined ? options.operands.slice(1) : options.operands + const parsed = /^s(.)(.*?[^\\])?\1(.*?)\1([gi]*)$/.exec(script ?? '') + if (parsed === null) { + io.err('sed: only substitution scripts (s/pattern/replacement/) run in the worker host\n') + return 2 + } + const [, , pattern = '', replacement = '', modifiers = ''] = parsed + let matcher: RegExp + try { + matcher = new RegExp(pattern, modifiers.includes('g') ? `g${modifiers.replace('g', '')}` : modifiers) + } catch (error) { + io.err(`sed: invalid pattern: ${error instanceof Error ? error.message : String(error)}\n`) + return 2 + } + const { sources, status } = await readInputs('sed', targets, io, state, fs) + for (const source of sources) { + for (const line of toLines(source.text)) io.out(`${line.replace(matcher, replacement.replace(/\\(\d)/g, '$$$1'))}\n`) + } + return status +} + +const tee: ShellProgram = async (argv, io, state, fs) => { + const options = parseOptions(argv) + io.out(io.stdin) + for (const operand of options.operands) { + try { + await fs.writeText(resolveIn(state.cwd, operand), io.stdin, options.flags.has('a')) + } catch (error) { + io.err(`${describeFailure('tee', operand, error)}\n`) + return 1 + } + } + return 0 +} + +/** The text utilities, keyed by the name a command line uses. */ +export const TEXT_PROGRAMS: Readonly> = { + echo, printf, cat, head, tail, wc, grep, sort, uniq, cut, tr, sed, tee, +} diff --git a/packages/experimental/webworker-runtime/src/shell/types.ts b/packages/experimental/webworker-runtime/src/shell/types.ts new file mode 100644 index 0000000000..ac258e0853 --- /dev/null +++ b/packages/experimental/webworker-runtime/src/shell/types.ts @@ -0,0 +1,146 @@ +/** + * Types of the in-worker shell: the state one command line mutates, the byte + * face a program reads and writes, and the program signature the command table + * stores. A browser worker has no processes, so a "program" is a JavaScript + * function over the VFS and the state below is the whole machine. + * @module @deepseek-ai/dsh-experimental-webworker-runtime/src/shell/types + */ + +/** + * Mutable state of one shell instance. A subshell copies it; a group and the + * top level share it, which is what makes `cd` visible to later commands of + * the same line and invisible outside `( … )`. + */ +export interface ShellState { + /** Absolute working directory every relative path resolves against. */ + cwd: string + /** Exported variables — the environment a program reads. */ + environment: Record + /** Shell variables that were assigned but never exported. */ + variables: Record + /** Exit status of the last completed command, read back as `$?`. */ + lastStatus: number + /** Set by `exit`; the interpreter stops the line and reports this status. */ + exitRequested: number | undefined + /** + * The caller's cancellation, for programs that would otherwise keep the + * whole line waiting after a kill. The interpreter checks it between + * commands; a program that waits on anything must check it too. + */ + readonly signal: AbortSignal | undefined +} + +/** + * The byte face of one running program. Output is collected as text rather + * than streamed: every program is a JavaScript function that returns before + * the next one starts, so a pipeline is a string handed along, and nothing in + * this shell can observe a partially written stream. + */ +export interface ShellIo { + /** Everything on standard input, already complete. */ + readonly stdin: string + /** Append to standard output. Declared as a value: redirections hand it around. */ + readonly out: (text: string) => void + /** Append to standard error. Declared as a value: redirections hand it around. */ + readonly err: (text: string) => void +} + +/** One directory entry, as the listing, glob, and walk programs read it. */ +export interface ShellDirent { + readonly name: string + readonly directory: boolean +} + +/** What a program can learn about one path. */ +export interface ShellStats { + readonly directory: boolean + readonly size: number + readonly mtimeMs: number +} + +/** + * The filesystem a shell run acts on. + * + * Asynchronous by construction: a command that runs in its own worker reaches + * the VFS by message, and the browser offers no way to block on that (a + * synchronous face would need `SharedArrayBuffer`, which requires + * cross-origin isolation this deployment cannot have). The in-host + * implementation answers immediately from the mounted VFS. + */ +export interface ShellFileSystem { + /** + * Stat one path. + * @param path - absolute VFS path. + * @returns the entry's facts, or undefined when nothing is there. + */ + stat(path: string): Promise + /** + * List one directory. + * @param path - absolute VFS path of the directory. + * @returns its entries, sorted by name. + * @throws a Node-shaped error when the path is absent or is not a directory. + */ + list(path: string): Promise + /** + * Read one file as UTF-8 text. + * @param path - absolute VFS path. + * @returns the file's contents. + * @throws a Node-shaped error when the path is absent or is a directory. + */ + readText(path: string): Promise + /** + * Write text to one file. + * @param path - absolute VFS path; its parent must exist. + * @param text - the text to store. + * @param append - true to keep the existing contents and add after them. + */ + writeText(path: string, text: string, append?: boolean): Promise + /** + * Create one directory. + * @param path - absolute VFS path. + * @param recursive - true to create missing parents and tolerate an existing directory. + */ + mkdir(path: string, recursive: boolean): Promise + /** + * Remove one path. + * @param path - absolute VFS path. + * @param options - `recursive` to take a whole subtree, `force` to tolerate absence. + */ + remove(path: string, options: { recursive: boolean; force: boolean }): Promise + /** + * Move one file or subtree. + * @param from - absolute source path. + * @param to - absolute destination path. + */ + rename(from: string, to: string): Promise +} + +/** + * One executable of the command table. + * + * A program reports its exit status like a POSIX process: 0 for success, and a + * nonzero status it also explains on {@link ShellIo.err}. Throwing is reserved + * for a defect in the program itself — the interpreter turns a throw into + * status 1 plus a diagnostic naming the program. + * @param argv - the program name at index 0, then its arguments, fully expanded. + * @param io - standard input contents and the output sinks. + * @param state - shell state; a program that changes it (`cd`, `export`) mutates in place. + * @param fs - the filesystem this run acts on. + * @returns the exit status. + */ +export type ShellProgram = ( + argv: readonly string[], + io: ShellIo, + state: ShellState, + fs: ShellFileSystem, +) => number | Promise + +/** Outcome of one interpreted command line. */ +export interface ShellRunOutcome { + /** Exit status of the last command the line ran. */ + exitCode: number + /** Everything written to standard output. */ + stdout: string + /** Everything written to standard error. */ + stderr: string +} diff --git a/packages/experimental/webworker-runtime/src/worker.ts b/packages/experimental/webworker-runtime/src/worker.ts index 017e089d98..2940bd360b 100644 --- a/packages/experimental/webworker-runtime/src/worker.ts +++ b/packages/experimental/webworker-runtime/src/worker.ts @@ -20,16 +20,32 @@ import { installAsyncContextHooks } from './polyfill/async-context/async-context import { createNodeBuiltins, REPLACED_PREFIXES } from './node/builtins.ts' import { whenRequestListener } from './node/builtin_modules/implemented/http.ts' import { installTimerGlobals } from './node/globals/timers.ts' +import { installProcessGlobal } from './node/globals/process.ts' +import { isShellStartFrame } from './shell/process/protocol.ts' +import { runShellProcess } from './shell/process/host.ts' // Before the timer globals, so the wrappers close over the patched platform. installAsyncContextHooks() installTimerGlobals() let host: { handleMessage(data: unknown): void } | undefined +let shellRole = false const pending: unknown[] = [] self.addEventListener('message', (event: MessageEvent) => { const data = event.data as Record | null + // Role, decided by the first frame: a worker started by the host's shell + // runs one command and closes. It mounts no image and boots no tree, so the + // whole assembly below never happens in it. + if (host === undefined && isShellStartFrame(data)) { + shellRole = true + // The command's own directory and environment are the only `process` facts + // a shell process needs; bundled code that reads the global (picomatch's + // platform check) must not find it missing. + installProcessGlobal({ cwd: data.cwd, env: data.env }) + runShellProcess(data, self) + return + } if (host === undefined && data !== null && typeof data === 'object' && data.t === 'init') { if (typeof data.image !== 'string') { throw new Error('webworker: init frame needs a string image url') @@ -54,6 +70,10 @@ self.addEventListener('message', (event: MessageEvent) => { return } if (host === undefined) { + // A shell-role worker's later frames (fs replies, signals) belong to + // runShellProcess's own listener; parking them here would hold every + // file body until the worker exits. + if (shellRole) return pending.push(event.data) return } diff --git a/packages/experimental/webworker-runtime/tests/node/child-process.spec.ts b/packages/experimental/webworker-runtime/tests/node/child-process.spec.ts new file mode 100644 index 0000000000..292c8cfd3d --- /dev/null +++ b/packages/experimental/webworker-runtime/tests/node/child-process.spec.ts @@ -0,0 +1,152 @@ +/** + * The `node:child_process` face over the in-worker shell, and the ladder above + * it: the REAL local subprocess service, running unmodified against this + * module instead of a host kernel. That ladder is what the bash tool walks in + * the browser, so proving it here is what makes the browser probe a + * confirmation rather than the only evidence. + * + * A Node test host has no DOM `Worker`, so the commands here run through the + * inline strategy; the worker strategy and its frames are proven in + * `../shell/shell-process.spec.ts`, and both meet again in the preview probe. + * + * `process.kill` is redirected to the worker's process table for the same + * reason the worker does it: the subprocess service polls process-group + * liveness through it, and on a test host those pids belong to real processes. + */ +import { afterEach, beforeEach, expect, it, vi } from 'vitest' +import { MemoryVfs } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/storage/memory.ts' +import { setActiveVfs } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/storage/active.ts' +import { spawn, spawnSync } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/node/builtin_modules/implemented/child_process.ts' +import { processAlive, signalProcess } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/node/process-table.ts' +import { spawnSubprocess } from '@deepseek-ai/dsh-subprocess-local/src/spawn.ts' + +vi.mock('node:child_process', async () => + await import('@deepseek-ai/dsh-experimental-webworker-runtime/src/node/builtin_modules/implemented/child_process.ts')) + +const WORKSPACE = '/dsh/workspace' + +let vfs: MemoryVfs + +beforeEach(() => { + vfs = new MemoryVfs() + setActiveVfs(vfs) + vfs.mkdirSync(WORKSPACE, { recursive: true }) + vi.spyOn(process, 'kill').mockImplementation((pid: number, signal?: string | number): true => { + if (signal === 0) { + if (processAlive(pid)) return true + const error = new Error('kill ESRCH') as NodeJS.ErrnoException + error.code = 'ESRCH' + throw error + } + signalProcess(pid, (signal ?? 'SIGTERM') as NodeJS.Signals) + return true + }) +}) + +afterEach(() => { + vi.restoreAllMocks() +}) + +/** Collect one child's stdout, stderr, and settlement. */ +async function collect(child: ReturnType): Promise<{ stdout: string; stderr: string; code: number | null }> { + let stdout = '' + let stderr = '' + child.stdout?.on('data', (chunk: unknown) => { stdout += String(chunk) }) + child.stderr?.on('data', (chunk: unknown) => { stderr += String(chunk) }) + const code = await new Promise((settle, fail) => { + child.on('close', (value: unknown) => { settle(value as number | null) }) + child.on('error', fail) + }) + return { stdout, stderr, code } +} + +it('runs a bash command line and reports its output through the pipes', async () => { + const child = spawn('bash', ['-c', 'echo hi; echo oops >&2'], { cwd: WORKSPACE }) + expect(child.pid).toBeGreaterThan(1) + expect(await collect(child)).toEqual({ stdout: 'hi\n', stderr: 'oops\n', code: 0 }) +}) + +it('runs an explicit argv without re-parsing it as a command line', async () => { + vfs.writeFileSync(`${WORKSPACE}/spaced name.txt`, 'kept\n') + const child = spawn('cat', ['spaced name.txt'], { cwd: WORKSPACE }) + expect((await collect(child)).stdout).toBe('kept\n') +}) + +it('fails a program the command table does not hold the way a missing binary does', async () => { + const child = spawn('nowhere-binary', [], { cwd: WORKSPACE }) + // A caller that configures the pipes first (the browser launcher does) must + // reach the ENOENT, not a TypeError on the configuration line. + child.stdout?.setEncoding() + child.stderr?.setEncoding() + const error = await new Promise((settle) => { + child.on('error', (value: unknown) => { settle(value as NodeJS.ErrnoException) }) + }) + expect(error.code).toBe('ENOENT') + expect(error.syscall).toBe('spawn nowhere-binary') +}) + +it('refuses a command name that is not a string, as Node does', () => { + expect(() => spawn(undefined as unknown as string)).toThrow(/must be a non-empty string/) +}) + +it('reports that a synchronous run cannot happen, without throwing at the probe', () => { + expect(spawnSync('bwrap').error?.code).toBe('ENOENT') + expect(spawnSync('echo').error?.message).toContain('commands run asynchronously') +}) + +it('carries a command through the real local subprocess service', async () => { + const handle = spawnSubprocess({ + argv: ['bash', '-c', 'echo written > note.txt && cat note.txt'], + cwd: WORKSPACE, + stdio: { + stdin: 'ignore', + stdout: { maxBytes: 64_000 }, + stderr: { maxBytes: 64_000 }, + }, + graceMs: 3_000, + env: {}, + }) + const outcome = await handle.done + expect(outcome).toEqual({ exitCode: 0, signal: null }) + expect(handle.collected.stdout?.readFrom(0).text).toBe('written\n') + expect(vfs.readFileSync(`${WORKSPACE}/note.txt`, 'utf8')).toBe('written\n') +}) + +it('writes the caller-supplied standard input into the command', async () => { + const handle = spawnSubprocess({ + argv: ['bash', '-c', 'grep -c ""'], + cwd: WORKSPACE, + stdio: { + stdin: { data: 'one\ntwo\nthree\n' }, + stdout: { maxBytes: 64_000 }, + stderr: { maxBytes: 64_000 }, + }, + graceMs: 3_000, + env: {}, + }) + await handle.done + expect(handle.collected.stdout?.readFrom(0).text).toBe('3\n') +}) + +it('kills a running command through the service and reports the signal', async () => { + const handle = spawnSubprocess({ + argv: ['bash', '-c', 'sleep 30; echo never'], + cwd: WORKSPACE, + stdio: { + stdin: 'ignore', + stdout: { maxBytes: 64_000 }, + stderr: { maxBytes: 64_000 }, + }, + graceMs: 3_000, + env: {}, + }) + const started = performance.now() + handle.terminate() + const outcome = await handle.done + expect(outcome.signal).toBe('SIGTERM') + expect(outcome.exitCode).toBeNull() + expect(handle.collected.stdout?.readFrom(0).text).toBe('') + // The command settles on the signal, not on the interval it was waiting out: + // a `sleep` that ignored the abort would hold this handle open for 30s. + expect(performance.now() - started).toBeLessThan(5_000) +}) diff --git a/packages/experimental/webworker-runtime/tests/shell/injected-run.spec.ts b/packages/experimental/webworker-runtime/tests/shell/injected-run.spec.ts new file mode 100644 index 0000000000..8ed2035d95 --- /dev/null +++ b/packages/experimental/webworker-runtime/tests/shell/injected-run.spec.ts @@ -0,0 +1,165 @@ +/** + * The two run options a command inside a process worker supplies: the + * filesystem it acts on, and the callback that reports output before the run + * settles. `src/shell/process/child.ts` passes a message-backed filesystem and + * posts a frame per write, so both are load-bearing for every backgrounded + * command the bash tool starts. + * + * No VFS is mounted here, deliberately. The in-host filesystem reads the + * process-wide slot on first use, so a program that reached it instead of the + * injected face fails with `no filesystem is mounted` — a suite that mounted a + * VFS as well would pass either way. + */ +import { describe, expect, it } from 'vitest' +import { runShellCommand } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/shell/interpret.ts' +import { filesystemError } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/shell/fs-access.ts' +import type { + ShellDirent, ShellFileSystem, ShellRunOutcome, ShellStats, +} from '@deepseek-ai/dsh-experimental-webworker-runtime/src/shell/types.ts' + +const WORKSPACE = '/dsh/workspace' + +/** One call a program made on the injected filesystem. */ +interface Call { + readonly op: string + readonly path: string + readonly text?: string + readonly append?: boolean +} + +/** + * A filesystem over a flat map of absolute paths, recording every call. + * + * Directories are the parents of the files it holds, which is all the programs + * below ask about; nothing here reaches the mounted VFS. + */ +function recordingFileSystem(files: Record): { + fs: ShellFileSystem + calls: Call[] + contents: Map +} { + const contents = new Map(Object.entries(files)) + const calls: Call[] = [] + const directories = (): Set => { + const known = new Set() + for (const path of contents.keys()) { + for (let parent = path.slice(0, path.lastIndexOf('/')); parent !== ''; parent = parent.slice(0, parent.lastIndexOf('/'))) { + known.add(parent) + } + } + return known + } + const fs: ShellFileSystem = { + stat: async (path: string): Promise => { + calls.push({ op: 'stat', path }) + const text = contents.get(path) + if (text !== undefined) return { directory: false, size: text.length, mtimeMs: 1 } + return directories().has(path) ? { directory: true, size: 0, mtimeMs: 1 } : undefined + }, + list: async (path: string): Promise => { + calls.push({ op: 'list', path }) + if (!directories().has(path)) throw filesystemError('ENOENT', 'scandir', path) + const prefix = `${path}/` + const names = new Set() + for (const candidate of [...contents.keys(), ...directories()]) { + if (!candidate.startsWith(prefix)) continue + names.add(candidate.slice(prefix.length).split('/')[0] as string) + } + return [...names].sort().map(name => ({ name, directory: directories().has(`${prefix}${name}`) })) + }, + readText: async (path: string): Promise => { + calls.push({ op: 'readText', path }) + const text = contents.get(path) + if (text === undefined) throw filesystemError('ENOENT', 'open', path) + return text + }, + writeText: async (path: string, text: string, append = false): Promise => { + calls.push({ op: 'writeText', path, text, append }) + contents.set(path, append ? `${contents.get(path) ?? ''}${text}` : text) + }, + mkdir: async (path: string, recursive: boolean): Promise => { + calls.push({ op: 'mkdir', path, append: recursive }) + }, + remove: async (path: string): Promise => { + calls.push({ op: 'remove', path }) + contents.delete(path) + }, + rename: async (from: string, to: string): Promise => { + calls.push({ op: 'rename', path: from, text: to }) + const text = contents.get(from) + if (text === undefined) throw filesystemError('ENOENT', 'rename', from) + contents.delete(from) + contents.set(to, text) + }, + } + return { fs, calls, contents } +} + +describe('injected filesystem', () => { + it('reads through the injected face, at the path the shell resolved', async () => { + const { fs, calls } = recordingFileSystem({ [`${WORKSPACE}/notes.txt`]: 'alpha\nbeta\n' }) + const result = await runShellCommand('cat notes.txt', { cwd: WORKSPACE, env: {}, fs }) + expect(result).toEqual({ exitCode: 0, stdout: 'alpha\nbeta\n', stderr: '' }) + // Programs receive the word as written; the absolute path is the shell's work. + expect(calls.filter(call => call.op === 'readText').map(call => call.path)).toEqual([`${WORKSPACE}/notes.txt`]) + }) + + it('performs a redirection as a truncating write followed by appends', async () => { + const { fs, calls, contents } = recordingFileSystem({}) + const result = await runShellCommand('echo one > out.txt; echo two >> out.txt', { cwd: WORKSPACE, env: {}, fs }) + expect(result.exitCode).toBe(0) + expect(contents.get(`${WORKSPACE}/out.txt`)).toBe('one\ntwo\n') + expect(calls.filter(call => call.op === 'writeText').map(call => [call.text, call.append])).toEqual([ + // `> file` empties the file when the redirection is set up, so a command + // that writes nothing still leaves it empty. + ['', false], + ['one\n', true], + ['two\n', true], + ]) + }) + + it('reports an injected failure as the utility does, not as a filesystem error', async () => { + const { fs } = recordingFileSystem({}) + const result = await runShellCommand('cat missing.txt', { cwd: WORKSPACE, env: {}, fs }) + expect(result.exitCode).toBe(1) + expect(result.stderr).toBe('cat: missing.txt: No such file or directory\n') + }) +}) + +describe('incremental output', () => { + /** Run a line, collecting what the callback saw in order. */ + async function reported(command: string): Promise<{ seen: [string, string][]; outcome: ShellRunOutcome }> { + const { fs } = recordingFileSystem({ [`${WORKSPACE}/notes.txt`]: 'alpha\n' }) + const seen: [string, string][] = [] + const outcome = await runShellCommand(command, { + cwd: WORKSPACE, + env: {}, + fs, + onOutput: (stream, text) => { seen.push([stream, text]) }, + }) + return { seen, outcome } + } + + it('reports each write as it happens and still returns the complete text', async () => { + const { seen, outcome } = await reported('echo one; echo two') + expect(seen).toEqual([['stdout', 'one\n'], ['stdout', 'two\n']]) + expect(outcome.stdout).toBe('one\ntwo\n') + }) + + it('tags a diagnostic as standard error', async () => { + const { seen, outcome } = await reported('definitely-not-a-program') + expect(seen).toEqual([['stderr', 'bash: definitely-not-a-program: command not found\n']]) + expect(outcome).toEqual({ exitCode: 127, stdout: '', stderr: 'bash: definitely-not-a-program: command not found\n' }) + }) + + it('reports only what the line writes out, not what it hands along or captures', async () => { + // A pipeline stage writes into the next stage's input and a redirection + // writes into a file: neither is output of the line, so a caller polling for + // progress must not see it. + const piped = await reported('cat notes.txt | cat') + expect(piped.seen).toEqual([['stdout', 'alpha\n']]) + const redirected = await reported('echo captured > out.txt') + expect(redirected.seen).toEqual([]) + expect(redirected.outcome.stdout).toBe('') + }) +}) diff --git a/packages/experimental/webworker-runtime/tests/shell/shell-process.spec.ts b/packages/experimental/webworker-runtime/tests/shell/shell-process.spec.ts new file mode 100644 index 0000000000..dc2ad25dc1 --- /dev/null +++ b/packages/experimental/webworker-runtime/tests/shell/shell-process.spec.ts @@ -0,0 +1,199 @@ +/** + * The process model: a command runs in its own worker, reaches the VFS only by + * message, and dies when the host says so. + * + * The `Worker` here is a loopback that runs the REAL child half + * (`runShellProcess`) against the REAL host half, so the frames, the + * filesystem service, and the termination ladder are the shipped ones — only + * the thread boundary is simulated, because a Node test host has no DOM + * `Worker` to cross. That a browser worker really can start a nested worker + * and terminate it mid-burn is measured separately, in the preview probe. + */ +import { afterEach, beforeEach, expect, it, vi } from 'vitest' +import { MemoryVfs } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/storage/memory.ts' +import { setActiveVfs } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/storage/active.ts' +import { startProcess } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/shell/process/host.ts' +import { runShellProcess } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/shell/process/child.ts' +import { isShellStartFrame } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/shell/process/protocol.ts' +import type { FromProcessFrame, ToProcessFrame } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/shell/process/protocol.ts' + +const WORKSPACE = '/dsh/workspace' +const WORKER_URL = 'https://example.test/assets/worker.js' + +let vfs: MemoryVfs +/** Every loopback worker the code under test constructed. */ +let started: LoopbackWorker[] + +/** + * A `Worker` that keeps the child half on this thread. Delivery is deferred so + * neither half can observe the other's synchronous progress, which is the one + * property of the real boundary that changes behaviour. + */ +class LoopbackWorker { + readonly url: string + terminated = false + private readonly hostListeners: ((event: MessageEvent) => void)[] = [] + private childListener: ((event: MessageEvent) => void) | undefined + private closed = false + + constructor(url: string | URL, options?: { type?: string }) { + this.url = String(url) + expect(options?.type).toBe('module') + started.push(this) + } + + /** Host → child. The first frame starts the real child half. */ + postMessage(frame: ToProcessFrame): void { + if (this.terminated) return + queueMicrotask(() => { + if (this.terminated) return + if (isShellStartFrame(frame)) { + runShellProcess(frame, { + postMessage: (reply: FromProcessFrame) => { this.toHost(reply) }, + addEventListener: (_type: 'message', listener: (event: MessageEvent) => void) => { this.childListener = listener }, + close: () => { this.closed = true }, + }) + return + } + this.childListener?.({ data: frame } as MessageEvent) + }) + } + + /** Child → host. */ + private toHost(frame: FromProcessFrame): void { + if (this.terminated) return + queueMicrotask(() => { + if (this.terminated) return + for (const listener of this.hostListeners) listener({ data: frame } as MessageEvent) + }) + } + + addEventListener(type: 'message' | 'error', listener: (event: MessageEvent) => void): void { + if (type === 'message') this.hostListeners.push(listener) + } + + terminate(): void { + this.terminated = true + } + + /** Whether the child closed itself after reporting its status. */ + get childClosed(): boolean { + return this.closed + } +} + +beforeEach(() => { + vfs = new MemoryVfs() + setActiveVfs(vfs) + vfs.mkdirSync(WORKSPACE, { recursive: true }) + started = [] + // The selection in `startProcess` reads exactly these two globals. + vi.stubGlobal('Worker', LoopbackWorker) + vi.stubGlobal('self', { location: { href: WORKER_URL } }) +}) + +afterEach(() => { + vi.unstubAllGlobals() +}) + +/** Run one command line through the process model and collect everything. */ +async function run(script: string, stdin = ''): Promise<{ code: number; stdout: string; stderr: string }> { + let stdout = '' + let stderr = '' + const code = await new Promise((settle) => { + startProcess({ + script, + argv: ['bash', '-c', script], + cwd: WORKSPACE, + env: { HOME: '/dsh/home' }, + stdin, + onOutput: (stream, text) => { + if (stream === 'stdout') stdout += text + else stderr += text + }, + onExit: settle, + }) + }) + return { code, stdout, stderr } +} + +it('starts the command as a worker from this bundle, not on this thread', async () => { + await run('echo hi') + expect(started).toHaveLength(1) + // The child is this very bundle in another role: no second asset to serve. + expect(started[0]?.url).toBe(WORKER_URL) +}) + +it('runs the command in the child and reports its output and status', async () => { + expect(await run('echo hi; echo oops >&2; exit 3')).toEqual({ code: 3, stdout: 'hi\n', stderr: 'oops\n' }) +}) + +it('reaches the host filesystem by message', async () => { + const written = await run('mkdir -p nested && echo carried > nested/file.txt && cat nested/file.txt') + expect(written).toEqual({ code: 0, stdout: 'carried\n', stderr: '' }) + // The child holds no VFS of its own: the bytes can only have arrived here + // through the filesystem frames. + expect(vfs.readFileSync(`${WORKSPACE}/nested/file.txt`, 'utf8')).toBe('carried\n') +}) + +it('carries a filesystem failure back with its code, not as a lost exception', async () => { + const missing = await run('cat nowhere.txt') + expect(missing.code).toBe(1) + expect(missing.stderr).toBe('cat: nowhere.txt: No such file or directory\n') +}) + +it('delivers standard input to the child', async () => { + expect((await run('grep -c ""', 'a\nb\nc\n')).stdout).toBe('3\n') +}) + +it('closes the child once the command settles', async () => { + await run('true') + expect(started[0]?.childClosed).toBe(true) +}) + +it('asks first and terminates second', async () => { + const events: number[] = [] + const running = startProcess({ + script: 'sleep 30', + argv: ['bash', '-c', 'sleep 30'], + cwd: WORKSPACE, + env: {}, + stdin: '', + onOutput: () => {}, + onExit: code => events.push(code), + }) + // The first rung asks the command to stop; a `sleep` honours it. + running.interrupt() + await vi.waitFor(() => { expect(events).toHaveLength(1) }) + expect(events[0]).toBe(130) + + // The second rung does not ask: the worker is gone whatever it was doing. + const stubborn = startProcess({ + script: 'sleep 30', + argv: ['bash', '-c', 'sleep 30'], + cwd: WORKSPACE, + env: {}, + stdin: '', + onOutput: () => {}, + onExit: code => events.push(code), + }) + stubborn.destroy() + await vi.waitFor(() => { expect(events).toHaveLength(2) }) + expect(started[1]?.terminated).toBe(true) +}) + +it('runs an explicit argv without a command line to parse', async () => { + vfs.writeFileSync(`${WORKSPACE}/spaced name.txt`, 'kept\n') + let stdout = '' + const code = await new Promise((settle) => { + startProcess({ + argv: ['cat', 'spaced name.txt'], + cwd: WORKSPACE, + env: {}, + stdin: '', + onOutput: (_stream, text) => { stdout += text }, + onExit: settle, + }) + }) + expect({ code, stdout }).toEqual({ code: 0, stdout: 'kept\n' }) +}) diff --git a/packages/experimental/webworker-runtime/tests/shell/shell.spec.ts b/packages/experimental/webworker-runtime/tests/shell/shell.spec.ts new file mode 100644 index 0000000000..8278f9cda5 --- /dev/null +++ b/packages/experimental/webworker-runtime/tests/shell/shell.spec.ts @@ -0,0 +1,225 @@ +/** + * The in-worker shell: structure (pipelines, chaining, subshells, redirections, + * expansion) and the command table's effects on a real MemoryVfs. + * + * ONE module instance, like `../node/fs.spec.ts`: the command table reaches the VFS + * through the module-level slot, so the mount here and the programs under test + * must be the same copy of `src/storage/memory.ts`. + */ +import { beforeEach, describe, expect, it } from 'vitest' +import { MemoryVfs } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/storage/memory.ts' +import { setActiveVfs } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/storage/active.ts' +import { runShellCommand } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/shell/interpret.ts' +import type { ShellRunOutcome } from '@deepseek-ai/dsh-experimental-webworker-runtime/src/shell/types.ts' + +const WORKSPACE = '/dsh/workspace' + +let vfs: MemoryVfs + +/** Run one command line in a fresh workspace with a fixed environment. */ +async function run(command: string, options: { stdin?: string; cwd?: string } = {}): Promise { + return await runShellCommand(command, { + cwd: options.cwd ?? WORKSPACE, + env: { HOME: '/dsh/home', PWD: WORKSPACE, GREETING: 'hello world' }, + stdin: options.stdin, + }) +} + +beforeEach(() => { + vfs = new MemoryVfs() + setActiveVfs(vfs) + vfs.mkdirSync(WORKSPACE, { recursive: true }) + vfs.mkdirSync(`${WORKSPACE}/src`, { recursive: true }) + vfs.writeFileSync(`${WORKSPACE}/notes.txt`, 'alpha\nbeta\ngamma\n') + vfs.writeFileSync(`${WORKSPACE}/src/a.ts`, 'export const a = 1\n') + vfs.writeFileSync(`${WORKSPACE}/src/b.ts`, 'export const b = 2\n') +}) + +describe('command execution', () => { + it('runs a program and reports its output and status', async () => { + expect(await run('echo hi')).toEqual({ exitCode: 0, stdout: 'hi\n', stderr: '' }) + }) + + it('reports an unknown command the way a shell does', async () => { + const result = await run('definitely-not-a-program --help') + expect(result.exitCode).toBe(127) + expect(result.stderr).toBe('bash: definitely-not-a-program: command not found\n') + }) + + it('reads a file through the VFS', async () => { + expect((await run('cat notes.txt')).stdout).toBe('alpha\nbeta\ngamma\n') + }) + + it('reports a missing file as the utility does, with a nonzero status', async () => { + const result = await run('cat missing.txt') + expect(result.exitCode).toBe(1) + expect(result.stderr).toBe('cat: missing.txt: No such file or directory\n') + }) +}) + +describe('structure', () => { + it('pipes standard output into the next stage', async () => { + expect((await run('cat notes.txt | grep -n "^[ab]"')).stdout).toBe('1:alpha\n2:beta\n') + }) + + it('takes the pipeline status from its last stage', async () => { + expect((await run('cat notes.txt | grep zeta')).exitCode).toBe(1) + }) + + it('honours && and || on the previous status', async () => { + expect((await run('true && echo yes || echo no')).stdout).toBe('yes\n') + expect((await run('false && echo yes || echo no')).stdout).toBe('no\n') + }) + + it('runs ; separated commands in order', async () => { + expect((await run('echo one; echo two')).stdout).toBe('one\ntwo\n') + }) + + it('keeps a subshell directory change out of the parent', async () => { + expect((await run('(cd src && pwd); pwd')).stdout).toBe(`${WORKSPACE}/src\n${WORKSPACE}\n`) + }) + + it('keeps a directory change made by the line itself', async () => { + expect((await run('cd src; pwd')).stdout).toBe(`${WORKSPACE}/src\n`) + }) + + it('stops the line at exit and reports its status', async () => { + const result = await run('echo before; exit 3; echo after') + expect(result).toEqual({ exitCode: 3, stdout: 'before\n', stderr: '' }) + }) +}) + +describe('redirections', () => { + it('writes standard output to a file and truncates it first', async () => { + await run('echo first > out.txt') + await run('echo second > out.txt') + expect(vfs.readFileSync(`${WORKSPACE}/out.txt`, 'utf8')).toBe('second\n') + }) + + it('creates an empty file when the command writes nothing', async () => { + await run('true > empty.txt') + expect(vfs.readFileSync(`${WORKSPACE}/empty.txt`, 'utf8')).toBe('') + }) + + it('appends with >>', async () => { + await run('echo one > log.txt; echo two >> log.txt') + expect(vfs.readFileSync(`${WORKSPACE}/log.txt`, 'utf8')).toBe('one\ntwo\n') + }) + + it('reads standard input from a file and from a here-string', async () => { + expect((await run('grep beta < notes.txt')).stdout).toBe('beta\n') + expect((await run('cat <<< inline')).stdout).toBe('inline\n') + }) + + it('sends standard error to its own file with 2>', async () => { + const result = await run('cat missing.txt 2> err.txt') + expect(result.stderr).toBe('') + expect(vfs.readFileSync(`${WORKSPACE}/err.txt`, 'utf8')).toBe('cat: missing.txt: No such file or directory\n') + }) + + it('merges standard error into standard output with 2>&1', async () => { + const result = await run('cat missing.txt 2>&1') + expect(result.stderr).toBe('') + expect(result.stdout).toBe('cat: missing.txt: No such file or directory\n') + }) + + it('reports a missing input file itself and never runs the command', async () => { + // Setting up the redirection is the shell's own work, so the diagnostic is + // prefixed `bash` on the resolved path rather than by the utility. + expect(await run('cat < missing.txt')).toEqual({ + exitCode: 1, + stdout: '', + stderr: `bash: ${WORKSPACE}/missing.txt: No such file or directory\n`, + }) + }) + + it('refuses a target that expands to more than one word', async () => { + const result = await run('cat < src/*.ts') + expect(result.exitCode).toBe(1) + expect(result.stderr).toBe('bash: ambiguous redirect\n') + }) + + it('refuses a descriptor duplication other than between stdout and stderr', async () => { + const result = await run('echo hi 3>&1') + expect(result.exitCode).toBe(1) + expect(result.stderr).toBe('bash: 3>&1: unsupported descriptor redirection\n') + }) +}) + +describe('expansion', () => { + it('expands variables, quoted and unquoted', async () => { + expect((await run('echo "$GREETING"')).stdout).toBe('hello world\n') + expect((await run('echo ${MISSING:-fallback}')).stdout).toBe('fallback\n') + }) + + it('reports the previous status as $?', async () => { + expect((await run('false; echo $?')).stdout).toBe('1\n') + }) + + it('substitutes command output', async () => { + expect((await run('echo "[$(head -n 1 notes.txt)]"')).stdout).toBe('[alpha]\n') + }) + + it('evaluates arithmetic', async () => { + expect((await run('echo $((1 + 2 * 3))')).stdout).toBe('7\n') + }) + + it('expands globs against the VFS and keeps an unmatched pattern literal', async () => { + expect((await run('echo src/*.ts')).stdout).toBe('src/a.ts src/b.ts\n') + expect((await run('echo *.missing')).stdout).toBe('*.missing\n') + }) + + it('passes an assignment prefix as environment for that command only', async () => { + expect((await run('MARK=set printenv MARK; echo "[${MARK}]"')).stdout).toBe('set\n[]\n') + }) +}) + +describe('file utilities', () => { + it('lists a directory one entry per line', async () => { + expect((await run('ls src')).stdout).toBe('a.ts\nb.ts\n') + }) + + it('creates, copies, moves, and removes trees', async () => { + const result = await run('mkdir -p deep/nested && cp -r src deep/nested/copy && mv notes.txt deep/ && rm -r src') + expect(result.exitCode).toBe(0) + expect(vfs.existsSync(`${WORKSPACE}/deep/nested/copy/a.ts`)).toBe(true) + expect(vfs.existsSync(`${WORKSPACE}/deep/notes.txt`)).toBe(true) + expect(vfs.existsSync(`${WORKSPACE}/src`)).toBe(false) + }) + + it('finds by name and type', async () => { + expect((await run('find . -name "*.ts"')).stdout).toBe('./src/a.ts\n./src/b.ts\n') + expect((await run('find . -type d')).stdout).toBe('.\n./src\n') + }) + + it('counts, sorts, and deduplicates text', async () => { + expect((await run('wc -l notes.txt')).stdout.trim()).toBe('3 notes.txt') + expect((await run('sort -r notes.txt | head -n 1')).stdout).toBe('gamma\n') + expect((await run('printf "b\\nb\\na\\n" | sort | uniq')).stdout).toBe('a\nb\n') + }) + + it('translates and deletes characters by range', async () => { + expect((await run('echo shell-works | tr a-z A-Z')).stdout).toBe('SHELL-WORKS\n') + expect((await run('echo a1b2c3 | tr -d 0-9')).stdout).toBe('abc\n') + }) + + it('substitutes with sed and refuses any other script', async () => { + expect((await run('sed s/alpha/ALPHA/ notes.txt | head -n 1')).stdout).toBe('ALPHA\n') + const refused = await run('sed 1d notes.txt') + expect(refused.exitCode).toBe(2) + expect(refused.stderr).toContain('only substitution scripts') + }) +}) + +describe('cancellation', () => { + it('stops before the next command once the caller aborts', async () => { + const controller = new AbortController() + controller.abort() + const result = await runShellCommand('echo never', { + cwd: WORKSPACE, + env: {}, + signal: controller.signal, + }) + expect(result).toEqual({ exitCode: 130, stdout: '', stderr: '' }) + }) +}) diff --git a/packages/experimental/webworker-runtime/tsdown.config.ts b/packages/experimental/webworker-runtime/tsdown.config.ts index 7d927e4286..a55f53f1a0 100644 --- a/packages/experimental/webworker-runtime/tsdown.config.ts +++ b/packages/experimental/webworker-runtime/tsdown.config.ts @@ -1,9 +1,36 @@ +import { createRequire } from 'node:module' +import { dirname, join } from 'node:path' import { fileURLToPath } from 'node:url' import { defineConfig } from 'tsdown' import { MODULE_PROXIES, MODULE_PROXY_PREFIXES } from './src/module-proxies.ts' const here = (relative: string): string => fileURLToPath(new URL(relative, import.meta.url)) +const resolveFrom = createRequire(import.meta.url) + +/** + * `@yarnpkg/parsers`' shell face, reached at its own file rather than through the + * package root. The root barrel also re-exports the Syml parser, which pulls in + * that grammar and js-yaml — around 175 kB of this bundle for a format the worker + * never parses, plus their module bodies at worker start. The shell entry holds + * `parseShell` and the stringifiers, requiring only its own grammar. + * + * This pins a path inside the package, which its `exports` field does not + * publish: upgrading `@yarnpkg/parsers` must re-check that `lib/shell.js` is still + * where the shell parser lives and still exports `parseShell`. The path is + * derived from the package manifest, the one subpath the package does publish, so + * a move fails the build here rather than silently reinstating the barrel. + */ +const SHELL_PARSER_ENTRY = join(dirname(resolveFrom.resolve('@yarnpkg/parsers/package.json')), 'lib/shell.js') + +/** Redirects the parsers root specifier onto {@link SHELL_PARSER_ENTRY}. */ +const shellParserOnlyPlugin = { + name: 'dsh-shell-parser-only', + resolveId(source: string): string | null { + return source === '@yarnpkg/parsers' ? SHELL_PARSER_ENTRY : null + }, +} + /** * Resolve the module proxy table at bundle time: every Node builtin or * replaced external the worker graph imports lands on its `./node/` proxy, @@ -61,7 +88,7 @@ export default defineConfig([{ dts: false, clean: false, noExternal: [/.*/], - plugins: [moduleProxyPlugin], + plugins: [moduleProxyPlugin, shellParserOnlyPlugin], outputOptions: { inlineDynamicImports: true }, }, { // Page half: an ordinary browser ES module the deployment's page imports. It diff --git a/vitest.config.ts b/vitest.config.ts index 9394a580d2..7ac8101425 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -48,6 +48,12 @@ const windowsUnsupportedTests = process.platform === 'win32' // Windows. The worker always speaks POSIX; the Linux lanes hold the diff. 'packages/experimental/webworker-runtime/tests/node/path-diff.spec.ts', 'packages/experimental/webworker-runtime/tests/node/shim-diff.spec.ts', + // The subprocess ladder over the worker's child_process face: its kill + // rung reaches the in-worker process table through `process.kill`, + // which the ladder's win32 branch replaces with taskkill-by-real-pid — + // undeliverable to a table pid. The worker host always reports 'linux', + // so the Linux lanes hold the ladder. + 'packages/experimental/webworker-runtime/tests/node/child-process.spec.ts', ] : []