mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-09-11 04:00:38 +00:00
feat(native): add prebuilt Node-API flock support
This commit is contained in:
imccyu
@@ -1,34 +1,25 @@
|
||||
# Architecture
|
||||
|
||||
This repository owns confinement *mechanism*, not policy: consumers (agent harnesses and sandbox capabilities) decide which paths a run may read or write; this package family provides the launcher that enforces those grants and the JavaScript API that resolves and speaks to it. The packaging follows the per-platform-package model of [`node-addon-require-builtin`](https://www.npmjs.com/package/@esplus/node-addon-require-builtin) (and esbuild), adapted from Node addons to standalone static executables.
|
||||
The system package family supplies native mechanisms to Node callers: a Linux confinement executable and a POSIX file-lock binding. Consumers own sandbox policy and Session lifecycle.
|
||||
|
||||
## Two-layer package family
|
||||
## Package family
|
||||
|
||||
The family is one entry package plus per-platform binary packages:
|
||||
The ESM entry `@deepseek-ai/node-addon-system` and its optional platform packages share one version. Platform metadata chooses the operating system and CPU; each package's `prebuilds.json` declares the files it must contain.
|
||||
|
||||
- **Entry package** (`@deepseek-ai/node-addon-system`): ESM JavaScript. Owns the tool's CLI contract — path resolution (`launcherPath`), the functional probe (`probe`), grant-argv construction (`grantArgs`), and the contract constants. Ships the C source in its tarball for auditability. Lists every platform package as an `optionalDependency`.
|
||||
- **Platform packages** (`@deepseek-ai/node-addon-system-linux-{x64,arm64}`): one prebuilt static binary under `bin/`, a `prebuilds.json` declaring it, and no JavaScript at all. npm's `os`/`cpu` fields select the matching one at install time; the entry package resolves it to a file path — there is nothing to import.
|
||||
The root entry owns the existing Landlock path resolution, grant argv, and functional probe. It does not load native addons. The `./flock` entry lazily loads `system.node` only when `tryLockExclusive(fd)` is called. Importing either JavaScript entry therefore works without a matching native payload.
|
||||
|
||||
Because the CLI parser and binary are versioned together in one package family, the parser cannot fall behind that binary version. Preventing that mismatch is why the package split exists.
|
||||
## Separate mechanisms
|
||||
|
||||
There is no shared loader package: platform packages have nothing to load. If a second tool ever needs shared JS, extract it then, not preemptively.
|
||||
`landlock-run` remains a static musl executable with the [existing CLI contract](cli-contract.md). It installs confinement on itself before exec, and refuses to exec if enforcement is unavailable. A missing launcher or unsupported kernel produces an unusable probe.
|
||||
|
||||
## Resolution and availability
|
||||
`system.node` uses stable Node-API v8. Its flock operation follows [fs-ext's asynchronous callback model](https://github.com/baudehlo/node-fs-ext/blob/v2.1.1/fs-ext.cc): it runs `flock(fd, LOCK_EX | LOCK_NB)` in asynchronous work and records errno on that worker. The native callback receives zero or positive errno; JavaScript owns the promise and syscall error construction. Setup errors throw into that promise. Callback exceptions are reported through Node's uncaught-exception handler; unexpected Node-API failures terminate the process. A terminating environment may suppress JavaScript completion, but its cleanup waits for queued or running native work before freeing storage.
|
||||
|
||||
`launcherPath()` resolves `@deepseek-ai/node-addon-system-<platform>-<arch>` and returns `<package>/bin/landlock-run`. When the package is not resolvable it returns a deterministic fallback path inside the entry package's own `node_modules` that simply never exists. Existence is deliberately unchecked either way: `probe()` is the single availability signal, and a missing binary probes `unusable` exactly like an unenforcing kernel. Consumers get one degradation path, not two.
|
||||
The descriptor belongs to the caller and must stay open through completion. The binding neither opens nor closes it; closing the final descriptor for its open file description releases the lock.
|
||||
|
||||
The probe is functional — the launcher builds and enforces a real maximal ruleset in a short-lived child — because version checks would miss a kernel that has the syscalls but refuses enforcement.
|
||||
The JSONL backend retains its inode check, materialization timing, and close lifecycle. Windows uses its existing koffi semaphore and never calls this POSIX binding. The browser worker supplies a single-process replacement for the flock entry, while running the Landlock JavaScript API unchanged.
|
||||
|
||||
## Fail-closed everywhere
|
||||
## Builds and release
|
||||
|
||||
The launcher exits `125` without exec'ing the command on any launcher-level failure: usage error, unenforcing kernel, unopenable grant root, failed exec. Partial enforcement (an older Landlock ABI governing only a subset of accesses) is accepted, reported on stderr, and surfaced by the probe as `partial` — the consumer decides what its mode vocabulary promises at each level. Neither the binary nor the entry package reads environment variables: which binary confines a process is never decidable by the ambient environment.
|
||||
Repository builds and `build:bench` explicitly build the host addon before running consumers. Each platform builds natively on its CI runner. Landlock is static-musl; Linux addons are separately built for glibc and musl, and macOS uses a Mach-O bundle. Stable Node-API removes the Node-major build dimension, not OS, CPU, or libc differences. CI exercises identical addon bytes under Node 20, 22, 24, and 26; Linux also runs the musl addon in Alpine containers.
|
||||
|
||||
## Build and release model
|
||||
|
||||
Builds are native-only. `scripts/build.ts` compiles the running architecture's binaries with the distro `musl-gcc` (static: no loader or libc expectations on consumers, one binary for glibc and musl distros); CI's per-architecture runners are the builders of record, and no cross toolchain exists in the repo. Review covers the C source and the CI job that built each binary, enforced by three gates: platform prepack refuses missing/wrong-ELF binaries, entry prepack refuses unbuilt `lib/`, and the release pipeline byte-pins installed binaries against the workspace builds they were packed from.
|
||||
|
||||
The package matrix is checked-in metadata (`prebuilds.json` + `os`/`cpu` fields); `scripts/github-matrix.mjs` derives the CI and Release matrices from it, so adding a platform extends automation without editing workflows.
|
||||
|
||||
## Adding a platform
|
||||
|
||||
A new platform adds one `packages/<platform>/` package (`package.json` with `os`/`cpu`, `prebuilds.json`, README, LICENSE), a runner entry in `scripts/github-matrix.mjs`, and a row in [support-matrix.md](support-matrix.md) — added only together with a native GitHub runner that builds and proves it (the no-cross-toolchain rule). Sibling launchers for other confinement mechanisms belong in their own repositories on this same template, not as second tools here.
|
||||
Platform prepack validates file formats, architecture, payload completeness, and Node-API exports. The packed-install rehearsal installs local tarballs, checks their bytes, and exercises the installed mechanisms. Missing capabilities fail explicitly; no consumer install runs a compiler. [Packaging](packaging.md) and [release](release.md) own the operational details.
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# POSIX flock behavior
|
||||
|
||||
`tryLockExclusive(fd)` returns a promise for one `flock(fd, LOCK_EX | LOCK_NB)` attempt. The syscall runs off the JavaScript thread. The caller keeps the descriptor open through completion; the binding does not open, duplicate, or close it. It exposes neither a blocking-wait API nor a shared-lock API.
|
||||
|
||||
## Behavior tests
|
||||
|
||||
The [native tests](../test/flock.test.js) exercise real descriptors and independent processes. The [C oracle](../test/fixtures/flock-oracle.c) calls the operating system directly, independently of `system.node`.
|
||||
|
||||
| Condition | Required observation |
|
||||
|---|---|
|
||||
| No conflicting lock | Acquisition resolves to void |
|
||||
| Same open file description acquires again | Acquisition succeeds without a second ownership record |
|
||||
| Separate opens of the same file | Exactly one exclusive holder; the contender rejects with EAGAIN/EWOULDBLOCK |
|
||||
| Different files | Both can be locked |
|
||||
| Independent C flock holder | The addon cannot acquire, and the C oracle cannot acquire while the addon holds the lock |
|
||||
| A shared flock holder | The addon's exclusive attempt conflicts |
|
||||
| Holder remains live | A nonblocking attempt reports contention before the holder unlocks |
|
||||
| Ordinary read/write by another process | Access is allowed: flock is advisory, not an I/O permission mechanism |
|
||||
| One unrelated descriptor closes | The actual holder keeps its lock |
|
||||
| A descriptor inherited by a child remains open | Closing the parent's descriptor does not release the shared open file description's lock |
|
||||
| Last owning descriptor closes | An already-open contender can acquire |
|
||||
| Holder process exits or is killed | Acquisition succeeds after process exit, without a stale-lock timeout |
|
||||
| Invalid descriptor | The promise rejects with EBADF and positive errno |
|
||||
| Native argument validation fails | The JavaScript entry returns a rejected promise without throwing synchronously |
|
||||
| Native completion callback | It receives zero or the request's positive errno asynchronously |
|
||||
| Native completion callback throws | The exception reaches Node's uncaught-exception handler |
|
||||
| Concurrent success/failure calls | Each completion receives its own syscall errno |
|
||||
| Worker environment terminates before or during its callback | Native work and cleanup reach completion without taking ownership of the caller's descriptor |
|
||||
|
||||
Tests synchronize through IPC or flushed line protocols and await process exit before asserting crash recovery. They do not use fixed sleeps or a millisecond performance threshold to prove nonblocking behavior. The syscall oracle is built only for tests and never included in a published platform package.
|
||||
|
||||
## Limits
|
||||
|
||||
Locks belong to open file descriptions and follow the host filesystem's flock semantics. Removing or replacing a pathname does not transfer a lock to the replacement inode; the JSONL backend separately checks inode identity. Network filesystems can have different or unsupported lock semantics. Windows does not use this API and retains its existing semaphore implementation.
|
||||
|
||||
Node-API compatibility tests reuse the same platform addon under different Node versions. They complement these syscall tests; loading a binary alone does not prove correct locking behavior.
|
||||
@@ -9,11 +9,11 @@ The public package family belongs to the `@deepseek-ai` scope and uses the `node
|
||||
@deepseek-ai/node-addon-system-<platform>
|
||||
```
|
||||
|
||||
Platform suffixes carry no libc component (binaries are static musl) and no variant component — variants stay inside `prebuilds.json` and binary filenames.
|
||||
Platform suffixes carry OS and CPU. Linux libc variants live inside the same platform package and are declared in `prebuilds.json`.
|
||||
|
||||
## Binaries
|
||||
|
||||
The launcher executable is `landlock-run`, shipped at `bin/landlock-run` inside each platform package.
|
||||
The Linux launcher remains `bin/landlock-run`. The Node-API addon is `system.node`: `bin/glibc/system.node` and `bin/musl/system.node` on Linux, `bin/system.node` on macOS.
|
||||
|
||||
## Environment variables
|
||||
|
||||
|
||||
@@ -1,45 +1,24 @@
|
||||
# Packaging
|
||||
|
||||
The package family uses the same layout as native packages such as esbuild: one JS entry package plus platform optional packages. Unlike Node addons there is no ABI or backend division — each platform package carries exactly the static executables its `prebuilds.json` declares.
|
||||
The family publishes one ESM entry package plus OS/CPU-selected optional platform packages. All share one version; platform packages publish before the entry.
|
||||
|
||||
## Published packages
|
||||
## Payloads
|
||||
|
||||
```text
|
||||
@deepseek-ai/node-addon-system
|
||||
@deepseek-ai/node-addon-system-linux-x64
|
||||
@deepseek-ai/node-addon-system-linux-arm64
|
||||
```
|
||||
The entry exports the Landlock API at its root and the asynchronous lock API at `./flock`, with C sources included for auditability. Platform packages contain no JavaScript.
|
||||
|
||||
Unsupported platforms are intentionally absent from `optionalDependencies` — see [support-matrix.md](support-matrix.md).
|
||||
- Linux: `bin/landlock-run`, `bin/glibc/system.node`, and `bin/musl/system.node`.
|
||||
- macOS: `bin/system.node`.
|
||||
|
||||
## Package matrix
|
||||
`package.json` supplies OS/CPU metadata; `prebuilds.json` supplies tool, binary kind, path, and addon Node-API/libc metadata. CI matrices and release assembly derive from those files. Nested paths remain intact in uploaded artifacts and tarballs.
|
||||
|
||||
The matrix is explicit in checked-in metadata:
|
||||
## Installation and use
|
||||
|
||||
- `packages/entry/package.json` lists the platform packages as `optionalDependencies`.
|
||||
- `packages/<name>/package.json` declares `os` and `cpu`. There is no `libc` field on purpose: the binaries are statically linked against musl and run on glibc and musl distros alike.
|
||||
- `packages/<name>/prebuilds.json` declares the binaries that may exist in that package (`tool`, `kind`, `path`).
|
||||
- [support-matrix.md](support-matrix.md) explains why unsupported platform packages are not published.
|
||||
Neither entry nor platform packages have installation lifecycle scripts. The entry resolves its matching optional package when a native operation needs it. Optional means that the package manager selects a platform, not that a requested lock can succeed without its binding.
|
||||
|
||||
`scripts/github-matrix.mjs` derives the CI and Release matrices from these files. `scripts/build.ts` builds only the current host's targets, into `packages/<name>/bin/`; it is not a matrix generator. When changing the matrix, update package metadata, `prebuilds.json`, the lockfile, and the support/release docs in the same change.
|
||||
The root Landlock API stays importable without native payloads and reports unavailable enforcement through its probe. The flock entry is also lazy at import; acquisition reports a missing or unloadable addon instead of compiling or granting an unprotected lock.
|
||||
|
||||
## Runtime selection
|
||||
## Pack verification
|
||||
|
||||
1. npm's `os`/`cpu` fields make installers fetch only the matching platform package.
|
||||
2. The entry package's `launcherPath()` resolves it to `<package>/bin/landlock-run`; unresolvable packages yield a deterministic, never-existing fallback path.
|
||||
3. `probe()` is the single availability signal: missing binary and unenforcing kernel are deliberately indistinguishable (`unusable`), so consumers have one fail-closed path.
|
||||
Platform tarballs use npm pack to preserve the launcher's executable bit. The entry uses pnpm pack to convert workspace dependency versions. Prepack rejects missing or undeclared payloads, invalid ELF/Mach-O architecture or type, addons without Node-API exports, and launchers without executable permission.
|
||||
|
||||
## No install fallback
|
||||
|
||||
The entry package has NO install script and never compiles on the consumer host. A compile fallback would require a musl toolchain everywhere and turn a clean fail-closed degradation into an environment-dependent maybe. The packed-manifest check in `verify-packed-install.mjs` enforces the absence of install lifecycle scripts.
|
||||
|
||||
## Pack gates
|
||||
|
||||
Platform tarballs are produced by `npm pack`, entry tarballs by `pnpm pack` — deliberately split: `pnpm pack` (observed on 11.7.0) normalizes file modes and strips the executable bit, which would ship a launcher no consumer can spawn, while platform packages have no dependencies and so need none of pnpm's workspace-protocol conversion; entry packages need that conversion and carry no executables. `scripts/pack-release.mjs` encodes the split — never hand-pack a platform package with pnpm.
|
||||
|
||||
Both pack paths produce the exact publish bytes behind a `prepack` gate:
|
||||
|
||||
- Platform packages: `scripts/verify-launcher-binary.mjs` — every declared binary present, executable, ELF `e_machine` matching the declared `cpu`, nothing undeclared in `bin/`.
|
||||
- Entry packages: `scripts/verify-entry-lib.mjs` — built `lib/` present.
|
||||
|
||||
`scripts/verify-packed-install.mjs` then rehearses the consumer path from the packed tarballs: payload checks, a throwaway install, a byte-pin of the installed binary against the workspace build, an executability check on the installed copy, and a real confinement world-proof through the installed launcher. A non-executable or missing binary fails loudly here instead of masquerading as a non-enforcing kernel.
|
||||
The installed-artifact rehearsal verifies concrete dependency versions and absence of installation hooks, performs an offline npm install from local tarballs, and compares installed bytes with build outputs. It then proves flock contention/close release and probes the installed Landlock launcher; real confinement remains required on enforcing CI kernels.
|
||||
|
||||
@@ -4,7 +4,7 @@ Pre-1.0: treat this as a release checklist, not a stability policy.
|
||||
|
||||
## Versioning
|
||||
|
||||
The launcher workspace root and its three public packages share one version. Run the bump helper from the repository root:
|
||||
The native workspace root and every platform/entry package share one version. Run the bump helper from the repository root:
|
||||
|
||||
```sh
|
||||
pnpm --dir native/system release:bump patch # or minor / major / x.y.z
|
||||
@@ -28,11 +28,14 @@ pnpm --dir native/system typecheck
|
||||
pnpm --dir native/system test:entry
|
||||
```
|
||||
|
||||
On a Linux host, also rehearse the pack path locally:
|
||||
On a supported Linux or macOS host, also rehearse the pack path locally:
|
||||
|
||||
```sh
|
||||
pnpm --dir native/system build:native
|
||||
pnpm --dir native/system build:test-oracle
|
||||
pnpm --dir native/system test:launcher
|
||||
pnpm --dir native/system test:flock
|
||||
pnpm --dir native/system test:packaging
|
||||
node native/system/scripts/pack-release.mjs native/system/.release/npm --current-platform-only
|
||||
node native/system/scripts/verify-packed-install.mjs native/system/.release/npm --current-platform-only
|
||||
```
|
||||
@@ -45,9 +48,9 @@ Use the main repository's `Node Addon System Release` workflow so every binary i
|
||||
2. Create and push the `node-addon-system-vX.Y.Z` tag matching the package versions.
|
||||
3. Run the same workflow from that tag with `publish=true`.
|
||||
|
||||
The workflow publishes only from the final packed tarballs, in `publish-order.txt` order (platform packages before the entry that optionally depends on them). A current-platform rehearsal can still query npm for metadata about an incompatible optional platform package; that package cannot supply the host launcher, which comes from the matching local tarball. Publishing every platform package before the entry ensures a public entry version never points ahead of its platform packages. The workflow supports npm trusted publishing through GitHub OIDC; without it, provide an `NPM_TOKEN` secret in the `npm-publish` environment. Packages publish with `--access public`.
|
||||
The workflow publishes only from the final packed tarballs, in `publish-order.txt` order (platform packages before the entry that optionally depends on them). The current-platform rehearsal uses offline npm installation; the current entry and platform package come from local tarballs. Publishing every platform package before the entry ensures a public entry version never points ahead of its platform packages. The workflow supports npm trusted publishing through GitHub OIDC; without it, provide an `NPM_TOKEN` secret in the `npm-publish` environment. Packages publish with `--access public`.
|
||||
|
||||
The three scoped package names must be bootstrapped with an `@deepseek-ai` organization token through the `NPM_TOKEN` fallback: npm [requires a package to exist before a trusted publisher can be configured](https://docs.npmjs.com/cli/v11/commands/npm-trust/). After the first release creates all three packages, configure each package to trust `node-addon-system-release.yml` in this repository with the `npm-publish` environment, then remove the fallback token when organization policy permits it.
|
||||
New scoped package names must be bootstrapped with an `@deepseek-ai` organization token through the `NPM_TOKEN` fallback: npm [requires a package to exist before a trusted publisher can be configured](https://docs.npmjs.com/cli/v11/commands/npm-trust/). After the first release creates the packages, configure each package to trust `node-addon-system-release.yml` in this repository with the `npm-publish` environment, then remove the fallback token when organization policy permits it.
|
||||
|
||||
Manual local fallback (current platform's packages only) — always through `pack-release.mjs`, never `pnpm publish` directly (pnpm's pack path strips the launcher's executable bit; see [packaging.md](packaging.md)):
|
||||
|
||||
|
||||
@@ -1,18 +1,14 @@
|
||||
# Support matrix
|
||||
|
||||
## Supported
|
||||
|
||||
| Platform package | GitHub runner (builder of record) | Notes |
|
||||
| Platform package suffix | Builder | Payload |
|
||||
|---|---|---|
|
||||
| `@deepseek-ai/node-addon-system-linux-x64` | `ubuntu-24.04` | static musl — glibc and musl distros alike |
|
||||
| `@deepseek-ai/node-addon-system-linux-arm64` | `ubuntu-24.04-arm` | static musl — glibc and musl distros alike |
|
||||
| linux-x64 | ubuntu-24.04 | static Landlock executable; glibc and musl system.node |
|
||||
| linux-arm64 | ubuntu-24.04-arm | static Landlock executable; glibc and musl system.node |
|
||||
| darwin-x64 | macos-15-intel | system.node |
|
||||
| darwin-arm64 | macos-latest | system.node |
|
||||
|
||||
Enforcement additionally requires a kernel with Landlock enabled (5.13+). The negotiated ABI level decides the probe verdict: every access this build knows governed → `full`; an older ABI governing a subset → `partial` (still confined for everything it supports); Landlock absent or disabled → `unusable`, and the launcher refuses to run commands at all. The probe — not the kernel version — is the authority: a kernel built without Landlock, or with the LSM disabled, probes `unusable` regardless of its version.
|
||||
The stable Node-API v8 addon is built once per platform/libc and exercised by CI under Node 20, 22, 24, and 26. macOS builds target 11.0 or later. Linux binding selection uses the running Node process's libc; the static launcher serves both libc variants.
|
||||
|
||||
## Deliberately unsupported
|
||||
Landlock additionally requires an enforcing Linux kernel. The functional probe determines full, partial, or unusable enforcement; kernel version alone is not an availability guarantee.
|
||||
|
||||
- **darwin**: macOS consumers typically confine through `sandbox-exec`/Seatbelt, which ships with the OS — there is no binary to distribute.
|
||||
- **win32**: a Windows confinement launcher would be a different mechanism in its own repository, not a port of this one.
|
||||
- **Other Linux architectures** (riscv64, s390x, …): no native CI builder of record yet. The no-cross-toolchain rule means a platform package is added only together with a native runner that builds and proves it.
|
||||
|
||||
A consumer on an unsupported platform resolves a nonexistent launcher path, probes `unusable`, and falls closed — the documented degradation, exercised by CI's darwin leg.
|
||||
Windows has neither a Landlock launcher nor this POSIX addon. The Harness retains its existing Windows semaphore implementation. Other CPU/OS combinations have no published platform package: Landlock probes unusable, and flock acquisition rejects. New platform support requires a native builder and installed-artifact verification.
|
||||
|
||||
Reference in New Issue
Block a user