deepseek-harness/.agents/notes/implemented/architecture/2026-07-10-single-file-executable-sdk-runtime-distribution.md

Agent Note: Single-file executable SDK runtime distribution (single-exe)

Status: implemented

English | 中文

Problem

DeepSeek Harness needs a dedicated SDK distribution form for the Python library — no Node installation, runs directly on the target platform: a single-file executable (hereafter "the exe") that exposes a stdio JSON-RPC serving interface (HarnessSdkJsonRpcServer, the Python SDK's peer), where the plugins and configuration actually booted are decided entirely by a cordis.yml supplied from outside the exe.

• The JSONRPC protocol for talking to the Python SDK is already validated
• A standardized way for cordis.yml to load every plugin (ESModule) is needed
• The distribution must carry the Node runtime, and support a locally linked source debugging mode

Decision

Packaging route: @yao-pkg/pkg's --sea mode

The exe is packaged with the --sea (enhanced SEA) mode of @yao-pkg/pkg (the actively maintained fork after vercel/pkg was archived). Relative to Node's native SEA, pkg adds a /snapshot VFS and runtime module hooks on top, hands the ESM entry to Node's default ESM loader unchanged, and depends on no ESM→CJS transpilation.

Measured (macos-arm64, node24 target, pkg 6.21.0): bare-specifier ESM dynamic import inside the VFS (including top-level await), CJS interop, node:sqlite, fail-loud on package names outside the set, and on-disk ESM import outside the VFS all pass; import.meta.url comes through unchanged as file:///snapshot/....

--sea requires target ≥ node22; the exe uniformly targets node24. One pkg invocation packages exactly one target; multi-platform builds invoke it once per platform.

Terminology reminder: pkg's /snapshot VFS has nothing to do with this repo's testing-system "snapshot" (ACP replay expected outputs, $DSH_SNAPSHOT); this document says "VFS" for the former.

The serving interface is a plugin inside the dsh application

The deterministic serving surface is a plugin selected by the packaged dsh application:

• packages/sdk/server (@deepseek-ai/dsh-sdk-jsonrpc-server): the pure protocol plugin; on apply it mounts HarnessSdkJsonRpcServer plus a line-delimited JSON-RPC transport on the process stdio, with disposal through ctx.effect(). Whether to serve is decided by cordis.yml; a yml that does not mount it is a legitimate process that does not serve. Protocol-level exit belongs to the plugin (after answering and flushing the shutdown response it disposes the root runtime so persistence drains, then exit(0); an HMR-style unload only stops the service without exiting the process).
• apps/cli (@deepseek-ai/dsh): the packaged application entry; its sdk profile mounts dsh-sdk-jsonrpc-server, and the CLI owns environment layering, profile composition, stdin/signal shutdown, and process exit.

The Python client supplies an explicit Harness home and selects the sdk profile plus ordered patch files. A missing home, profile, bundle, or server row fails loudly; there is no external complete-config fallback. The Python profile-runtime decision owns this application surface.

Plugin resolution: the VFS holds a real package tree, the closure manifest IS the deploy root

Inside the exe's VFS sits a real package tree in build-artifact form (each package's lib/ plus a real node_modules). The packaged JSON-RPC entry supplies its installed harness base to app-boot's root Include: relative plugin specifiers resolve from the external configuration directory, while bare package names resolve from the VFS, so a configuration inside another Node project cannot shadow the packaged plugin set. The ordinary development bin leaves bare packages configuration-owned. Bare specifiers in the packaged entry resolve upward along node_modules from the entry's position inside the VFS and land inside the VFS naturally. The closed set needs no allowlist code — the set is whatever the VFS has installed, and importing a name outside the set fails.

The deploy root is python/sdk-runtime/package.json (dsh-python-runtime-closure, a pnpm workspace member and a zero-code pure dependency manifest) — the unified source of truth for "which plugins the exe ships" and "what the Python runtime distributes". Adding a plugin to the exe = adding one dependency line to the manifest and repackaging. scripts/verify-runtime-closure.ts reads every shipped packages/preset/agent-presets/presets/*/agent.cordis.yml, evaluates disabled conditions that compare process.platform for every target in python/sdk-runtime/platforms.json, and requires each active workspace plugin at the runtime root through an explicit workspace: dependency. It also traverses every workspace package covered by that manifest and requires every non-optional workspace peer, reporting the complete preset or referencing-package → missing-dependency chain; unknown platform conditions remain active so a plugin cannot be omitted by an unsupported expression. pnpm run hygiene, CI static, and the single-exe build run it before packaging. Deploy also packs by each package's files, so the shared chunks tsdown splits out must be covered by files.

The deploy root includes @deepseek-ai/dsh-mcp-client as an explicitly supported custom-configuration plugin even though no shipped preset mounts it. An external config can therefore connect to user-supplied stdio and Streamable HTTP MCP servers and register their tools; the distribution does not carry those servers or extend the bridge to MCP Resources and Prompts. The executable and installed-wheel smokes start a temporary stdio server, discover its tool, and complete one model-requested call.

Build pipeline and artifacts

scripts/build-exe-for-python-sdk.ts: runtime closure verification → pnpm run build → (after clearing) pnpm --filter dsh-python-runtime-closure deploy --legacy --prod --config.node-linker=hoisted --config.auto-install-peers=false --config.link-workspace-packages=true directly into python/sdk-runtime/src/deepseek_harness_runtime/runtime/node/ → restore direct workspace packages omitted by legacy deploy and reject any remaining manifest gap → replace staged dependency symlinks with their target bytes, remove package-manager .bin links, and fail if any symlink remains → inject pkg configuration whose bin is node_modules/@deepseek-ai/dsh/lib/bin.js and whose assets cover dynamic profile, bundle, frontend, preset, native-library, and configuration reads → stage the target node-pty addon → invoke pkg --sea once per target → write deepseek-harness-sdk-runtime-<platform>-<arch> under dist-exe/ and copy it into the runtime directory. Linux CI rebuilds pty.node inside the matching manylinux 2.28 container because legacy deploy omits that install side effect. Every target copies its native @vscode/ripgrep binary beside the executable as the required -rg sidecar; pkg runtimes select that sidecar through process.pkg, while ordinary Node execution uses @vscode/ripgrep directly. macOS uses its target prebuild and also emits the required -spawn-helper. All four deploy flags are grounded in measurement: --legacy is the mandatory path with inject-workspace-packages off; hoisted gives pkg a stable single-instance layout that the explicit materialization pass makes symlink-free; disabling automatic peer installation prevents undeclared peers from expanding the closure; link-workspace-packages selects direct workspace dependencies. pnpm-workspace.yaml overrides the transitive @deepseek-ai/cosmokit and @deepseek-ai/schemastery semver requests to the pinned vendor sources so legacy deploy never resolves those unpublished names from a registry.

CI: .github/workflows/build-exe-for-python-sdk.yml is called for all four targets by the installed-wheel Python runtime pull-request validation and the public publication workflow; workflow_dispatch can still select a subset. Native builds run on linux-x64 / linux-arm64 (ubuntu-24.04-arm) / macos-arm64 / win-x64 (windows-2025), with ~/.pkg-cache cached where applicable, and pkg handles macOS ad-hoc signing. Each leg installs the release-shaped SDK and runtime wheels into a clean venv outside the checkout, proves their package and executable provenance, then drives the complete keyless scenario set through the public SDK and direct NDJSON JSON-RPC. Trusted pull requests additionally run a real DeepSeek two-turn tool smoke on every target; fork and Dependabot heads receive no key. Linux inspects the executable and native addon's GLIBC requirements and runs an additional manylinux 2.28 smoke, while macOS verifies that the executable's deployment target fits the wheel tag. A full four-target run retains five artifacts, each containing one release file: the platform-independent SDK wheel and four native runtime wheels; a subset dispatch retains the SDK wheel and selected runtime wheels. Bare executables and source bundles remain intermediate test inputs. .gitlab-ci.yml accepts python-v<repository-version> tag pipelines whose version matches the root package.json, builds one SDK wheel and four native runtime wheels, then a single serialized job checks and publishes all five to the project PyPI registry. The Windows x64 runtime decision owns the fourth target and the explicit exclusion of Windows arm64.

Python SDK distribution: two carriers, exe for production, node for development

The Python SDK lives at python/: python/sdk is the client and python/sdk-runtime is the runtime carrier package. The runtime package's data directory holds the build-injected platform executable with its required -rg sidecar and optional macOS helper, plus the build-injected runtime/node/ closure tree for repository development. resolve_bundled_launch_args() selects the executable by default; explicit DSH_RUNTIME_MODE=node runs runtime/node/node_modules/@deepseek-ai/dsh/lib/bin.js on system Node 22.19 or newer. The node carrier never enters wheel distributions, and neither carrier uses a checked-in complete cordis.yml.

scripts/build-python-release.py reads the authoritative X.Y.Z or prerelease version from the repository root package.json, converts prereleases to their PEP 440 spelling, and stages both packages at that wheel version, with deepseek-harness-sdk depending exactly on the matching deepseek-harness-runtime-bin. An optional python-v<repository-version> release tag is a consistency assertion and is rejected when it differs from the repository version; the source pyproject.toml development sentinel never determines a release version. Staging also carries the repository license into both wheels and the third-party notices into the bundled runtime wheel. The SDK is a py3-none-any wheel; each wheel-only runtime package contains one exe and its architecture-matched ripgrep sidecar, and the macOS wheel also contains its architecture-matched spawn helper. Runtime wheels use py3-none-manylinux_2_28_x86_64, py3-none-manylinux_2_28_aarch64, the conservative py3-none-macosx_14_0_arm64 tag for the Node 24 executable's macOS 13.5 deployment target, or py3-none-win_amd64; the Hatch hook rejects sdists, universal tags, mixed-platform payloads, missing or extra sidecars, and unsupported platforms.

The Python client launches the packaged dsh command with the selected profile (sdk by default), ordered patch files, and an explicit Harness home. The profile owns JSON-RPC serving and application composition; missing homes, profiles, bundles, patches, and server rows fail without an external complete-config fallback.

Naming lineage

dsh-python-runtime-closure is the private deploy manifest and deepseek-harness-sdk-runtime-<platform>-<arch> is the executable family. The wire serverInfo.name is deepseek-harness-sdk-runtime; the Python distribution names are deepseek-harness-sdk / deepseek-harness-runtime-bin, while the import modules are deepseek_harness / deepseek_harness_runtime.

Disposition of worker-style plugins

dsh-workflow-worker-thread and dsh-code-runtime-worker-thread are supported inside the exe. Their built hosts convert the sibling lib/worker.cjs URL with fileURLToPath() and pass the resulting filesystem string to Worker, which is the form pkg's Worker hook resolves inside the VFS. The worker entries are CommonJS because that hook compiles VFS worker files as CommonJS. The workflow engine keeps its data-URL bootstrap for unbuilt source execution; only its built sibling entry uses the filesystem string. The custom-config executable smoke loads both backends, invokes a real run_code call and a zero-agent workflow call, and requires each worker to return 42 from inside pkg's VFS.

Testing

The verification surface has three tiers. Mechanism tier: the measured conclusions for the --sea chain are embedded in the Decision sections (ESM dynamic import inside the VFS, single cordis instance, fail-loud config chain, node:sqlite, macOS ad-hoc signing runs). SDK tier: the complete keyless pytest suite covers the client protocol against a fake runtime peer, subprocess cleanup, absolute cwd propagation, dual-carrier launch, and carrier resolution; root CI runs it on Python 3.10. End-to-end tier: every platform build installs both wheels into a clean venv outside the checkout, proves matching versions and installed module/executable locations, then completes turns against a mock endpoint through the default SDK path, a custom config, the checked-in standalone minimal composition, and the direct binary protocol, with final text and JSONL checked. The minimal run asserts its exact system prompt and two-tool catalog, retains Bash state across calls, and invokes the editor. The custom config additionally drives run_code and a zero-agent workflow through their real worker files inside the packaged VFS. The filesystem-search scenario requires the model to call both glob and grep through the target-native -rg sidecar. The MCP scenario starts a temporary external stdio server, deliberately delays its initial tools/list response, then immediately starts the first SDK prompt; the prompt must see and call the discovered tool, proving that initialize is a real Loader-settlement readiness boundary rather than a timing sleep. The same installed run compares a committed executable-specific snapshot through the Python SDK: a keyless scripted model mounts a Cordis plugin that registers a tool, invokes that tool from run_code, runs a direct spawn subagent and a workflow that starts a second spawn child, then unmounts the plugin. The fixture explicitly disables its unused bundled Bash and local skill discovery so its tool set does not depend on repository-external state, and the comparison normalizes opaque message, agent, workflow-run, and session IDs across the SDK result and notification stream plus the parent and two child JSONL logs. Trusted pull requests add a real-provider two-turn file write/read whose external bytes, tool calls, completed reasons, and persisted log must agree. This harness stays separate from ACP's pnpm run test:snapshot because the protocols and build artifacts differ.

Manual-driving caveat: the bin treats stdin EOF as "the client is gone" and disposes immediately, so a short-lived pipe aborts an in-flight turn — pipe-driven runs must keep stdin open until the turn ends.

Alternatives considered

Bare Node native SEA. The injected main script must be a single CJS file, and the blob carries no filesystem and no module resolution, so a dynamic import of a bare specifier has nothing to resolve against; the only option is compiling plugins statically into the main script and registering them by hand — bypassing standard module resolution and hardcoding the plugin set, contrary to "configuration decides everything". The final route is in fact "the official SEA foundation + pkg's VFS/module-hook layer"; what was rejected is the bare use, not SEA itself.

pkg standard mode. Killed by the PoC, not a trade-off: it turns ESM into CJS + V8 bytecode via esbuild, the runtime vm compilation wires up no dynamic-import callback, every import() throws ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING, and --options experimental-require-module has no effect; it also depends on community-patched Node binaries (no macos-arm64 prebuilt; compiling from source on the spot takes about 10 minutes). Zero viability for this repo's architecture.

Pre-bundling each package ESM→CJS into the VFS. The compromise that keeps real resolution semantics and only downgrades the module format; --sea passed measurement outright, so this layer of build complexity never needed introducing.

jsonrpc-agent carrying the full closure dependencies. The app bin would declare 53+ dependencies it never imports — a "packaging manifest" masquerading as real dependency relationships — and would force constraints to open two exceptions for it, cordis-in-dependencies and a files wildcard. With the closure manifest landing on the python-side manifest package, constraints needs no exception at all and the bin keeps the normal package shape isomorphic to acp-agent.

An open plugin set (loading user plugins from disk). The shipped set is closed; the PoC incidentally confirmed that on-disk ESM import outside the VFS works (through the ctx.baseUrl relative-path channel). It is listed as a future evolution, which must separately solve sharing the cordis instance inside the exe with external plugins.

Consequences

Bought: zero-dependency single-file distribution on target platforms; plugin semantics strictly identical to running from source (the same real package tree, no transpilation, no registry); the serving interface, the plugin set, and the configuration all converge on two sources of truth — cordis.yml plus one dependency manifest; the exe and node carriers share one tree and one semantics, so development verification never waits for packaging; official Node binaries remove the patched-binary supply-chain concern.

Paid: artifacts on the order of 174MB with source entering the blob as-is (no bytecode obfuscation; a closed-source distribution requirement needs a separate evaluation); pkg's VFS/module-hook layer remains community-maintained (the build script pins @yao-pkg/pkg@6.21.0; upgrading is an explicit change); --sea is one invocation per target (matching CI's one leg per platform; local multi-platform builds are serial).