mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-08-29 04:26:38 +00:00
fix(spill-local): make startup cleanup race-safe
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-07-08-tool-output-spill-files.md
|
||||
2026-07-08-tool-output-spill-files.md: 14667b74ca877622d05196e9bf83945a842fe366
|
||||
2026-07-08-tool-output-spill-files.zh.md: db297fa6bee707a1d5a10d20260ce6b8a660d207
|
||||
2026-07-08-tool-output-spill-files.md: 81a292a00af63b0aa9145a0c556a428ab8c49d64
|
||||
2026-07-08-tool-output-spill-files.zh.md: 8d9e60d461297fb11ff2252e91f98a0cfad33f62
|
||||
|
||||
@@ -161,7 +161,7 @@ Those cases can consume `ctx.spillStore` directly in later work. They are not pa
|
||||
- Per-tool opt-out or per-tool policy declarations if the built-in `read` skip is insufficient.
|
||||
- Remote or database storage backends for ACP or remote environments where a local path is not meaningful.
|
||||
|
||||
Cleanup shipped for the local backend as a one-shot startup sweep, not tied to session deletion — see the [startup-cleanup RFC](./2026-07-17-local-spill-startup-cleanup.md). The seam still defines no per-session cleanup policy; retention is a backend concern.
|
||||
Cleanup shipped for the local backend as a one-shot startup sweep, not tied to session deletion — see the [startup-cleanup Agent Note](./2026-07-17-local-spill-startup-cleanup.md). The seam still defines no per-session cleanup policy; retention is a backend concern.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
@@ -160,7 +160,8 @@ ctx.tools.register(defineTool({
|
||||
- 由工具负责的 subagent 执行轨迹 spill(`await run.result`,在 `run.dispose()` 前读取进程内子会话,保存 JSONL)。
|
||||
- 如果内置的 `read` 跳过规则不足,再增加逐工具选择退出或逐工具策略声明。
|
||||
- 面向 ACP(Agent Client Protocol)或远程环境的远程/数据库存储后端,因为本地路径在这些环境中没有意义。
|
||||
- 旧 spill 文件的清理和保留策略,很可能与会话清理绑定。
|
||||
|
||||
本地后端通过一次性启动扫描清理旧文件,而不是绑定到会话删除——参见[启动清理 Agent Note](./2026-07-17-local-spill-startup-cleanup.zh.md)。seam 仍未定义逐会话清理策略;保留策略属于后端。
|
||||
|
||||
## 测试
|
||||
|
||||
|
||||
+3
-3
@@ -1,6 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-17-local-spill-startup-cleanup.md: ca4931776f89e641f127072f665e238ca2a1600d
|
||||
2026-07-17-local-spill-startup-cleanup.zh.md: b90923844ab71e1ce570e8adb66f81aed7bc3488
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-07-17-local-spill-startup-cleanup.md
|
||||
2026-07-17-local-spill-startup-cleanup.md: 96378d6ea785d90385f517c1b9a01073ade47fa2
|
||||
2026-07-17-local-spill-startup-cleanup.zh.md: a154cfb824d3a747c2c9acc2b707eb14d703ce2e
|
||||
|
||||
@@ -12,9 +12,9 @@ The local spill backend never deleted the full tool results it wrote. Every over
|
||||
|
||||
`dsh-spill-local` runs one best-effort cleanup sweep after activation. It does not delay service availability, is owned by the plugin fiber (a single `ctx.effect` whose generator launches the sweep and yields an async disposer that awaits it), and is awaited during disposal so no sweep I/O outlives the fiber. There is no recurring timer and no separate process.
|
||||
|
||||
A `cleanupPeriodDays` config defaults to `30`; `0` disables cleanup. An invalid value (negative or fractional) throws at load. The sweep scans the configured/active root plus any prior default `dsh-spill-*` temp roots discovered under the OS temp dir, deletes regular files whose `mtime` is strictly older than `now − cleanupPeriodDays`, and prunes directories left empty. It uses `lstat`, so a symlink is never followed or deleted; unrelated entries (non-`session-` directories, special files) are skipped. Every filesystem failure is caught and logged through `ctx.logger.warn` — the sweep never throws, so it cannot reject activation or a concurrent spill write. Discovery excludes symlinks and non-directories, returning only real `dsh-spill-*` directories the backend could have created.
|
||||
A `cleanupPeriodDays` config defaults to `30`; `0` disables cleanup. An invalid value (negative or fractional) throws at load. The sweep scans the configured/active root plus any prior default `dsh-spill-*` temp roots discovered under the OS temp dir and deletes regular files whose `mtime` is strictly older than `now − cleanupPeriodDays`. It prunes empty session directories and roots only for discovered prior-default roots; the active root keeps its session directories so pruning cannot race a local write, while writes recreate a session directory if another process prunes a discovered root that is still active. It uses `lstat`, so a symlink is never followed or deleted; unrelated entries (non-`session-` directories, special files) are skipped. Every filesystem failure is caught and logged through `ctx.logger.warn`, and a warning-sink exception is also contained — the sweep never throws, so it cannot reject activation or a concurrent spill write. Discovery excludes symlinks and non-directories, returning only real `dsh-spill-*` directories the backend could have created.
|
||||
|
||||
The ctx-free mechanics live in `packages/spill/spill-local/src/store.ts` (`sweepSpillRoots`, `discoverDefaultRoots`, `DEFAULT_ROOT_PREFIX`, `isErrno`), unit-testable without a `ctx`; the service in `src/index.ts` owns the config, the cutoff, and the fiber-owned launch/await.
|
||||
The ctx-free sweep mechanics live in `packages/spill/spill-local/src/cleanup.ts` (`sweepSpillRoots`, `discoverDefaultRoots`), unit-testable without a `ctx`; `store.ts` owns root naming, path derivation, and writes, while the service in `src/index.ts` owns the config, cutoff, and fiber-owned launch/await.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
@@ -32,4 +32,4 @@ Cleanup cost the backend a startup sweep and a config knob, and bought a bounded
|
||||
|
||||
## Testing
|
||||
|
||||
`dsh-spill-local` unit tests cover the age boundary (strictly-older expires, boundary kept), `cleanupPeriodDays: 0` disabling, empty-directory pruning, symlink/unrelated-entry skipping, configured-plus-discovered-root coverage through the real `gatherRoots`/`discoverDefaultRoots` path, active-root de-duplication, load-time validation of a bad `cleanupPeriodDays`, filesystem-failure containment (logged, not thrown) both directly and through the service's `ctx.logger.warn` wiring, and the quiescence contract — activation is available while a barrier-held sweep is parked, and disposal only settles after the sweep finishes.
|
||||
`dsh-spill-local` unit tests cover the age boundary (strictly-older expires, boundary kept), `cleanupPeriodDays: 0` disabling, discovered-root pruning, active-directory preservation, symlink/unrelated-entry skipping, configured-plus-discovered-root coverage through the real `gatherRoots`/`discoverDefaultRoots` path, active-root de-duplication, load-time validation of a bad `cleanupPeriodDays`, filesystem- and warning-sink-failure containment both directly and through the service's `ctx.logger.warn` wiring, and the quiescence contract — activation is available while a barrier-held sweep is parked, and disposal only settles after the sweep finishes.
|
||||
|
||||
@@ -6,15 +6,15 @@ Status: implemented
|
||||
|
||||
## 问题
|
||||
|
||||
本地 spill 后端从不删除它写下的完整工具结果。每个超限结果都会新增一个文件,因此配置的根目录会无限增长,而每进程默认的 `dsh-spill-*` 根目录也会跨多次运行不断累积。立即删除是错误的,因为已持久化、已恢复和已 fork 的会话仍可能引用某个 locator。[工具输出 spill 策略](./2026-07-08-tool-output-spill-files.md)需要一个有界的本地存储生命周期。
|
||||
本地 spill 后端从不删除它写下的完整工具结果。每个超限结果都会新增一个文件,因此配置的根目录会无限增长,而每进程默认的 `dsh-spill-*` 根目录也会跨多次运行不断累积。立即删除是错误的,因为已持久化、已恢复和已 fork 的会话仍可能引用某个 locator。[工具输出 spill 策略](./2026-07-08-tool-output-spill-files.zh.md)需要一个有界的本地存储生命周期。
|
||||
|
||||
## 决策
|
||||
|
||||
`dsh-spill-local` 在激活后运行一次尽力而为的清理扫描。它不延迟服务可用性,由插件 fiber 拥有(一个 `ctx.effect`,其生成器启动该扫描并让出一个等待它的异步 disposer),并在 dispose 期间被等待,因此没有扫描 I/O 会存活到 fiber 之后。既没有周期性定时器,也没有独立进程。
|
||||
|
||||
`cleanupPeriodDays` 配置默认为 `30`;`0` 会禁用清理。无效值(负数或小数)在加载时抛出。扫描会遍历配置的/活动的根目录,以及在 OS 临时目录下发现的任何先前默认 `dsh-spill-*` 临时根目录,删除 `mtime` 严格早于 `now − cleanupPeriodDays` 的常规文件,并修剪清空后的目录。它使用 `lstat`,因此符号链接绝不会被跟随或删除;无关条目(非 `session-` 目录、特殊文件)会被跳过。每一次文件系统失败都会被捕获并通过 `ctx.logger.warn` 记录——扫描绝不抛出,因此它无法让激活失败,也无法影响并发的 spill 写入。发现过程排除符号链接与非目录,只返回后端可能创建过的真实 `dsh-spill-*` 目录。
|
||||
`cleanupPeriodDays` 配置默认为 `30`;`0` 会禁用清理。无效值(负数或小数)在加载时抛出。扫描会遍历配置的/活动的根目录,以及在 OS 临时目录下发现的任何先前默认 `dsh-spill-*` 临时根目录,并删除 `mtime` 严格早于 `now − cleanupPeriodDays` 的常规文件。它只修剪发现的先前默认根目录中的空会话目录和空根目录;活动根目录会保留其会话目录,避免修剪操作与本地写入竞争,而当其他进程修剪了一个仍在使用的发现根目录时,写入操作会重新创建会话目录。扫描使用 `lstat`,因此符号链接绝不会被跟随或删除;无关条目(非 `session-` 目录、特殊文件)会被跳过。每一次文件系统失败都会被捕获并通过 `ctx.logger.warn` 记录,警告接收方抛出的异常也会被兜底——扫描绝不抛出,因此它无法让激活失败,也无法影响并发的 spill 写入。发现过程排除符号链接与非目录,只返回后端可能创建过的真实 `dsh-spill-*` 目录。
|
||||
|
||||
无 ctx 依赖的机制位于 `packages/spill/spill-local/src/store.ts`(`sweepSpillRoots`、`discoverDefaultRoots`、`DEFAULT_ROOT_PREFIX`、`isErrno`),无需 `ctx` 即可做单元测试;`src/index.ts` 中的服务负责配置、截止时间以及 fiber 拥有的启动/等待。
|
||||
无 ctx 依赖的扫描机制位于 `packages/spill/spill-local/src/cleanup.ts`(`sweepSpillRoots`、`discoverDefaultRoots`),无需 `ctx` 即可做单元测试;`store.ts` 负责根目录命名、路径推导与写入,而 `src/index.ts` 中的服务负责配置、截止时间以及 fiber 拥有的启动/等待。
|
||||
|
||||
## 考虑过的替代方案
|
||||
|
||||
@@ -32,4 +32,4 @@ Status: implemented
|
||||
|
||||
## 验证
|
||||
|
||||
`dsh-spill-local` 单元测试覆盖了年龄边界(严格更旧者过期,边界值保留)、`cleanupPeriodDays: 0` 的禁用、空目录修剪、符号链接/无关条目的跳过、通过真实 `gatherRoots`/`discoverDefaultRoots` 路径对配置根加发现根的覆盖、活动根去重、对错误 `cleanupPeriodDays` 的加载期校验、文件系统失败的兜底(记录而非抛出)——既直接测试,也经由服务的 `ctx.logger.warn` 接线测试——以及静止契约:在一个被屏障挂起的扫描停驻期间激活仍然可用,而 dispose 只有在扫描结束后才会完成。
|
||||
`dsh-spill-local` 单元测试覆盖了年龄边界(严格更旧者过期,边界值保留)、`cleanupPeriodDays: 0` 的禁用、发现根目录的修剪、活动目录的保留、符号链接/无关条目的跳过、通过真实 `gatherRoots`/`discoverDefaultRoots` 路径对配置根加发现根的覆盖、活动根去重、对错误 `cleanupPeriodDays` 的加载期校验、直接测试以及经由服务的 `ctx.logger.warn` 接线测试所覆盖的文件系统与警告接收方失败兜底,以及静止契约:在一个被屏障挂起的扫描停驻期间激活仍然可用,而 dispose 只有在扫描结束后才会完成。
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/config-catalog.md
|
||||
config-catalog.md: a845fe22e13ed085765668c7ec8d54d6bbdf129a
|
||||
config-catalog.zh.md: 39ba9d48368f99483733292f997609ba3a8aa43e
|
||||
config-catalog.md: 9fc8c333510c568686ce43f4aa1f6d0ae6bc3615
|
||||
config-catalog.zh.md: 4fb689f63631740d485a854e10a12c6d92c6f4ac
|
||||
|
||||
@@ -2074,7 +2074,7 @@ export interface Config {
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/spill/spill-local/src/index.ts:28`](../packages/spill/spill-local/src/index.ts)
|
||||
Source: [`packages/spill/spill-local/src/index.ts:31`](../packages/spill/spill-local/src/index.ts)
|
||||
|
||||
<a id="deepseek-aidsh-spill-policy"></a>
|
||||
|
||||
|
||||
@@ -2064,10 +2064,19 @@ export interface Config {
|
||||
* a local deployment. Set it to keep spill files under a known location.
|
||||
*/
|
||||
root?: string
|
||||
/**
|
||||
* Age in days after which a spill file is eligible for the one-shot startup
|
||||
* cleanup sweep. Defaults to `30`; `0` disables cleanup entirely. Files whose
|
||||
* `mtime` is strictly older than the cutoff are deleted and emptied
|
||||
* directories are pruned; fresh files, symlinks, and unrelated entries are
|
||||
* left untouched. Retention is deliberate — a resumed or forked session may
|
||||
* still reference an older locator until it ages out.
|
||||
*/
|
||||
cleanupPeriodDays?: number
|
||||
}
|
||||
```
|
||||
|
||||
来源:[`packages/spill/spill-local/src/index.ts:22`](../packages/spill/spill-local/src/index.ts)
|
||||
来源:[`packages/spill/spill-local/src/index.ts:31`](../packages/spill/spill-local/src/index.ts)
|
||||
|
||||
<a id="deepseek-aidsh-spill-policy"></a>
|
||||
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/spill/spill-local/README.md
|
||||
README.md: 2270a65d9270e1549a9e48d6a36b821e48c29070
|
||||
README.zh.md: b3e4999d8f2d982ef01637199299f79d06e12c4b
|
||||
README.md: 75bde20c423e1d2aa4bba49201b5bb0369d34fd0
|
||||
README.zh.md: 0539969cc4bd4da8dad4f0ac00436476555fd08c
|
||||
|
||||
@@ -23,7 +23,7 @@ Files land at `<root>/session-<hash>/<random>-<safeName>`:
|
||||
|
||||
The backend never deletes a spill on the write path — a persisted, resumed, or forked session may still reference an older locator, so immediate deletion would break retrieval. Instead, one best-effort sweep runs **once after activation**: it does not delay service availability, is owned by the plugin fiber, and is awaited on disposal (no sweep I/O outlives the fiber). There is no recurring timer and no separate process, so a long-lived deployment is not swept again until its next restart.
|
||||
|
||||
The sweep scans the configured `root` **and** any earlier default `dsh-spill-*` temp roots that prior default-root runs left under the OS temp dir. Within each, it deletes regular files whose `mtime` is strictly older than `now − cleanupPeriodDays` and prunes any directory left empty. It never follows or deletes a symlink, skips unrelated entries, and contains every filesystem failure (logged, never thrown) so it cannot fail activation or a concurrent spill write. Retention is deliberate: an old model-visible locator goes stale only once it ages past the cutoff.
|
||||
The sweep scans the configured `root` **and** any earlier default `dsh-spill-*` temp roots that prior default-root runs left under the OS temp dir. Within each, it deletes regular files whose `mtime` is strictly older than `now − cleanupPeriodDays`; it prunes empty session directories and roots only for discovered prior-default roots, while the active root keeps its session directories to avoid racing a write. A write recreates its session directory if another process prunes a discovered root that is still active. The sweep never follows or deletes a symlink, skips unrelated entries, and contains every filesystem or warning-sink failure so it cannot fail activation or a concurrent spill write. Retention is deliberate: an old model-visible locator goes stale only once it ages past the cutoff.
|
||||
|
||||
`saveText` rejects on a real storage failure (permissions, ENOSPC); the spill policy treats a rejection as best-effort and keeps the inline result. See the seam README for the vocabulary and the [tool output spill Agent Note](../../../.agents/notes/implemented/architecture/2026-07-08-tool-output-spill-files.md) for the design, and the [startup-cleanup Agent Note](../../../.agents/notes/implemented/architecture/2026-07-17-local-spill-startup-cleanup.md) for the sweep.
|
||||
|
||||
|
||||
@@ -17,8 +17,15 @@
|
||||
| 键 | 默认值 | 含义 |
|
||||
|---|---|---|
|
||||
| `root` | 私有 0700 临时目录 | spill 文件的根目录。设置后可将这些文件保存在已知位置。 |
|
||||
| `cleanupPeriodDays` | `30` | spill 文件在一次性启动清理扫描中符合删除条件前需经过的天数。`0` 禁用清理。 |
|
||||
|
||||
`saveText` 在发生真实存储故障(权限、ENOSPC)时返回拒绝;spill 策略会按尽力而为原则处理该拒绝,并保留内联结果。词汇见 seam README,设计见[工具输出 spill Agent Note](../../../.agents/notes/implemented/architecture/2026-07-08-tool-output-spill-files.zh.md)。
|
||||
## 启动清理
|
||||
|
||||
后端不会在写入路径上删除 spill,因为已持久化、已恢复或 fork 后的会话仍可能引用较旧的定位信息,立即删除会使其无法取回。后端会改为在激活后**仅运行一次**尽力而为的扫描:扫描不延迟服务可用性,由插件 fiber 拥有,并在 dispose 期间被等待(不会有扫描 I/O 存活至 fiber 之后)。它既不使用周期性定时器,也不运行独立进程,因此长期运行的部署要到下次重启才会再次扫描。
|
||||
|
||||
扫描会检查配置的 `root` **以及**先前使用默认根目录的运行在操作系统临时目录下留下的所有 `dsh-spill-*` 临时根目录。在每个根目录中,扫描会删除 `mtime` 严格早于 `now − cleanupPeriodDays` 的常规文件;它只修剪发现的先前默认根目录中的空会话目录和空根目录,而活动根目录会保留其会话目录,以避免与写入操作竞争。如果另一个进程修剪了一个仍在使用的发现根目录,写入操作会重新创建其会话目录。扫描绝不会跟随或删除符号链接,会跳过无关条目,并兜底每一次文件系统或警告接收方失败,因此无法使激活或并发 spill 写入失败。保留是刻意的:旧的模型可见定位信息只有超过截止时间后才会失效。
|
||||
|
||||
`saveText` 在发生真实存储故障(权限、ENOSPC)时返回拒绝;spill 策略会按尽力而为原则处理该拒绝,并保留内联结果。词汇见 seam README,设计见[工具输出 spill Agent Note](../../../.agents/notes/implemented/architecture/2026-07-08-tool-output-spill-files.zh.md),扫描机制见[启动清理 Agent Note](../../../.agents/notes/implemented/architecture/2026-07-17-local-spill-startup-cleanup.zh.md)。
|
||||
|
||||
## 模型体验
|
||||
|
||||
@@ -30,5 +37,5 @@
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **本地 spill 文件会持续存在,直到外部清理为止**:该后端不提供会话生命周期删除或按时间保留的策略,因为已持久化、已恢复和 fork 后的会话可能仍在引用某个路径。
|
||||
- **长期运行的部署需等到重启才会被扫描**:一次性扫描仅在激活后运行一次,因此运行期间达到 `cleanupPeriodDays` 的文件要到下次启动才会被回收;没有周期性定时器。
|
||||
- **定位信息需要与其位于同一文件系统的消费方**:远程或虚拟部署需要另一个 `SpillStore` 后端,其定位信息和取回指引在该环境中有明确含义。
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
/** Startup cleanup mechanics for local spill roots. */
|
||||
import { lstat, readdir, rmdir, unlink } from 'node:fs/promises'
|
||||
import { join } from 'node:path'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { DEFAULT_ROOT_PREFIX, isErrno } from './store.ts'
|
||||
|
||||
/**
|
||||
* A backend-generated default root name: `dsh-spill-` plus the 6-character
|
||||
* suffix `mkdtemp` appends. Discovery matches this
|
||||
* EXACT shape, not the bare prefix, so an unrelated `dsh-spill-test-*` fixture
|
||||
* or a foreign tool's differently-shaped `dsh-spill-…` directory is never
|
||||
* mistaken for a backend root to sweep.
|
||||
*/
|
||||
const DEFAULT_ROOT_RE = new RegExp(`^${DEFAULT_ROOT_PREFIX}[A-Za-z0-9]{6}$`)
|
||||
|
||||
/**
|
||||
* A backend-generated session directory name: `session-` plus the 12 lowercase
|
||||
* hex characters {@link sessionDir} derives from `sha256(sessionId)`. The sweep
|
||||
* only descends into entries of this EXACT shape, so an unrelated
|
||||
* `session-backup` directory under a shared configured root is never swept.
|
||||
*/
|
||||
const SESSION_DIR_RE = /^session-[0-9a-f]{12}$/
|
||||
|
||||
/** A one-argument warning sink — the sweep's only side effect on failure (never throws). */
|
||||
export type WarnFn = (message: string) => void
|
||||
|
||||
/** Report a best-effort sweep failure without allowing the warning sink to reject cleanup. */
|
||||
function warnSafely(warn: WarnFn, message: string): void {
|
||||
try {
|
||||
warn(message)
|
||||
} catch {
|
||||
// Warning sinks are observational callbacks; cleanup must remain best-effort
|
||||
// even when a logger implementation throws.
|
||||
}
|
||||
}
|
||||
|
||||
/** One root to sweep, plus whether its empty session directories and root may be pruned. */
|
||||
export interface SweepRoot {
|
||||
/** Absolute spill root to sweep. */
|
||||
path: string
|
||||
/**
|
||||
* When `true`, prune empty `session-*` children and then remove the root once
|
||||
* empty. Set for DISCOVERED prior-default `dsh-spill-*` roots (one per past
|
||||
* process — otherwise they accumulate empty forever), never for the
|
||||
* active/configured root the live process is still writing into. Writes retry
|
||||
* if another process still using a discovered root races its pruning.
|
||||
*/
|
||||
pruneWhenEmpty: boolean
|
||||
}
|
||||
|
||||
/** Options for {@link sweepSpillRoots} — the roots to scan, the age cutoff, and a failure sink. */
|
||||
export interface SweepOptions {
|
||||
/** Roots to sweep (configured/active root and/or discovered prior-default roots). */
|
||||
roots: SweepRoot[]
|
||||
/**
|
||||
* Epoch-millis cutoff: a regular file is deleted when its `mtime` is strictly
|
||||
* older than this. The caller derives it from `now - cleanupPeriodDays`, so a
|
||||
* file written exactly at the boundary is kept (only strictly-older expires).
|
||||
*/
|
||||
cutoffMs: number
|
||||
/** Where a contained filesystem failure is reported; the sweep itself never throws. */
|
||||
warn: WarnFn
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a single path, treating a concurrent-race disappearance as success.
|
||||
* A parallel process (or another sweep) may `unlink` the same file between our
|
||||
* scan and our own `unlink` — ENOENT then means the goal (file gone) already
|
||||
* holds, so it is not a failure. Any other error is reported and swallowed.
|
||||
*
|
||||
* @param path The absolute file path to remove.
|
||||
* @param warn Sink for a non-ENOENT failure message.
|
||||
* @returns Resolves once the removal was attempted (never rejects).
|
||||
*/
|
||||
async function unlinkIdempotent(path: string, warn: WarnFn): Promise<void> {
|
||||
try {
|
||||
await unlink(path)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- reached only when a file selected for deletion (a
|
||||
regular file that passed lstat) then fails to unlink: either it raced away
|
||||
(ENOENT) or a permission/IO fault struck between the stat and the unlink.
|
||||
Neither is deterministically reproducible in-process. */
|
||||
if (isErrno(error, 'ENOENT')) return
|
||||
warnSafely(warn, `spill-local: failed to delete ${path}: ${String(error)}`)
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sweep one spill session directory: delete expired regular files, skip
|
||||
* everything else, and report the directory empty afterward so the caller can
|
||||
* prune it. The `dir` entry MUST be a real directory — the caller `lstat`s it
|
||||
* first and skips a symlink, so this never follows a `session-*` symlink into a
|
||||
* foreign tree. Inside, a symlink or any non-regular entry (socket, fifo, nested
|
||||
* dir) is left untouched — `lstat` never follows a link, so a planted symlink
|
||||
* can neither be deleted nor redirect the age check. Every per-entry failure is
|
||||
* contained: one unreadable file does not abort the directory.
|
||||
*
|
||||
* @param dir The absolute session directory to scan (already confirmed a real dir).
|
||||
* @param cutoffMs Files with `mtime` strictly older than this are deleted.
|
||||
* @param warn Sink for contained filesystem failures.
|
||||
* @returns `true` when the directory holds no entries after the sweep (a prune candidate).
|
||||
*/
|
||||
async function sweepSessionDir(dir: string, cutoffMs: number, warn: WarnFn): Promise<boolean> {
|
||||
let names: string[]
|
||||
try {
|
||||
names = await readdir(dir)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- the caller lstat'd this entry and confirmed a real
|
||||
directory just before the call, so readdir fails only when the dir races
|
||||
away (ENOENT) or a permission/IO fault strikes in that window; not
|
||||
deterministically reproducible. False keeps it out of the prune step. */
|
||||
warnSafely(warn, `spill-local: failed to read ${dir}: ${String(error)}`)
|
||||
return false
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
let remaining = names.length
|
||||
for (const name of names) {
|
||||
const path = join(dir, name)
|
||||
let stats
|
||||
try {
|
||||
stats = await lstat(path)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- an entry that readdir just returned then fails to
|
||||
lstat only by racing away (ENOENT) or a permission/IO fault; keep it out
|
||||
of the deterministic test surface. */
|
||||
if (isErrno(error, 'ENOENT')) { remaining--; continue }
|
||||
warnSafely(warn, `spill-local: failed to stat ${path}: ${String(error)}`)
|
||||
continue
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
// Only regular files expire. Symlinks and other special entries are skipped
|
||||
// (never followed) so the sweep cannot be redirected or delete a link.
|
||||
if (!stats.isFile()) continue
|
||||
if (stats.mtimeMs >= cutoffMs) continue
|
||||
await unlinkIdempotent(path, warn)
|
||||
remaining--
|
||||
}
|
||||
return remaining === 0
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort one-shot cleanup: across each root, delete expired regular files
|
||||
* under its `session-*` directories, pruning empty directories only in
|
||||
* discovered prior-default roots. The active root keeps its session directories
|
||||
* to avoid racing a local write; writes recreate a directory pruned by another
|
||||
* process. Every filesystem and warning-sink failure is contained, so a caller
|
||||
* can await this during activation/disposal without it ever rejecting.
|
||||
*
|
||||
* @param options The roots to sweep, the age cutoff, and the failure sink.
|
||||
* @returns Resolves when the sweep finishes (never rejects).
|
||||
*/
|
||||
export async function sweepSpillRoots(options: SweepOptions): Promise<void> {
|
||||
const { roots, cutoffMs, warn } = options
|
||||
for (const root of roots) {
|
||||
let entries: string[]
|
||||
try {
|
||||
entries = await readdir(root.path)
|
||||
} catch (error: unknown) {
|
||||
// A root that does not exist yet (no spill ever written) is the common
|
||||
// case, not an error: ENOENT is silent, anything else is reported.
|
||||
if (!isErrno(error, 'ENOENT')) warnSafely(warn, `spill-local: failed to read root ${root.path}: ${String(error)}`)
|
||||
continue
|
||||
}
|
||||
// Track whether the root holds ANY entry the sweep did not fully reclaim, so
|
||||
// a discovered prior-default root can be pruned only when nothing remains.
|
||||
let rootEmptiable = true
|
||||
for (const name of entries) {
|
||||
// Only the backend's own `session-<12 hex>` directories are swept; an
|
||||
// unrelated sibling (`session-backup`, a stray file) is left untouched and
|
||||
// blocks pruning the root.
|
||||
if (!SESSION_DIR_RE.test(name)) { rootEmptiable = false; continue }
|
||||
const dir = join(root.path, name)
|
||||
let stats
|
||||
try {
|
||||
// lstat the session entry itself: a `session-*` SYMLINK must never be
|
||||
// followed (readdir/unlink through it would delete files in a foreign
|
||||
// target). Only a real directory is swept.
|
||||
stats = await lstat(dir)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- an entry readdir just returned fails to lstat only
|
||||
by racing away (ENOENT) or a permission/IO fault; not deterministically
|
||||
reproducible. */
|
||||
if (!isErrno(error, 'ENOENT')) warnSafely(warn, `spill-local: failed to stat ${dir}: ${String(error)}`)
|
||||
continue
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
if (!stats.isDirectory()) { rootEmptiable = false; continue }
|
||||
const empty = await sweepSessionDir(dir, cutoffMs, warn)
|
||||
if (!empty) { rootEmptiable = false; continue }
|
||||
if (!root.pruneWhenEmpty) {
|
||||
// The active root remains writable while cleanup runs. Leaving its empty
|
||||
// session directories in place closes the mkdir/rmdir race with saveText.
|
||||
rootEmptiable = false
|
||||
continue
|
||||
}
|
||||
try {
|
||||
await rmdir(dir)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- prune runs only on a dir observed empty; a failure
|
||||
here means a concurrent writer added a file (ENOTEMPTY) or a
|
||||
permission/IO fault struck — both are races outside deterministic
|
||||
in-process testing. */
|
||||
rootEmptiable = false
|
||||
if (!isErrno(error, 'ENOENT') && !isErrno(error, 'ENOTEMPTY')) {
|
||||
warnSafely(warn, `spill-local: failed to prune ${dir}: ${String(error)}`)
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
// A discovered prior-default root (one per past process) is removed once its
|
||||
// last session dir is gone — otherwise empty roots accumulate forever and
|
||||
// every future startup rescans them. The active/configured root is never
|
||||
// pruned (the live process is still writing into it).
|
||||
if (root.pruneWhenEmpty && rootEmptiable) {
|
||||
try {
|
||||
await rmdir(root.path)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- prune runs only on a root whose every child was
|
||||
reclaimed; a failure here means a concurrent writer added a fresh
|
||||
spill after our scan (ENOTEMPTY) or removed the root already (ENOENT)
|
||||
or a permission/IO fault struck — all races outside deterministic
|
||||
in-process testing. */
|
||||
if (!isErrno(error, 'ENOENT') && !isErrno(error, 'ENOTEMPTY')) {
|
||||
warnSafely(warn, `spill-local: failed to prune root ${root.path}: ${String(error)}`)
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Discover prior default spill roots: the `dsh-spill-<6 chars>` directories
|
||||
* directly under `base` (the OS tmpdir) that earlier default-root runs created.
|
||||
* A long-lived deployment
|
||||
* with a configured root will find none; a series of default-root runs
|
||||
* accumulates one per process, so the startup sweep reclaims them all. Matching
|
||||
* is the EXACT `mkdtemp` shape (see {@link DEFAULT_ROOT_RE}), not the bare
|
||||
* prefix, so an unrelated `dsh-spill-test-*` fixture or a foreign
|
||||
* differently-shaped directory is never swept; symlinks and non-directories are
|
||||
* excluded too — only real directories the backend could have created.
|
||||
*
|
||||
* @param warn Sink for a failure reading `base` (returns `[]` on failure).
|
||||
* @param base The directory to scan; defaults to the OS tmpdir (a test seam).
|
||||
* @returns Absolute paths of the discovered default roots (possibly empty).
|
||||
*/
|
||||
export async function discoverDefaultRoots(warn: WarnFn, base: string = tmpdir()): Promise<string[]> {
|
||||
let entries: string[]
|
||||
try {
|
||||
entries = await readdir(base)
|
||||
} catch (error: unknown) {
|
||||
warnSafely(warn, `spill-local: failed to scan ${base} for default roots: ${String(error)}`)
|
||||
return []
|
||||
}
|
||||
const roots: string[] = []
|
||||
for (const name of entries) {
|
||||
if (!DEFAULT_ROOT_RE.test(name)) continue
|
||||
const path = join(base, name)
|
||||
let stats
|
||||
try {
|
||||
// lstat, not stat: a symlink named `dsh-spill-*` must not be treated as a
|
||||
// root we then sweep (it could point anywhere).
|
||||
stats = await lstat(path)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- an entry readdir just returned fails to lstat only by
|
||||
racing away (ENOENT) or a permission/IO fault; not deterministically
|
||||
reproducible. */
|
||||
if (!isErrno(error, 'ENOENT')) warnSafely(warn, `spill-local: failed to stat default root ${path}: ${String(error)}`)
|
||||
continue
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
if (stats.isDirectory()) roots.push(path)
|
||||
}
|
||||
return roots
|
||||
}
|
||||
@@ -15,11 +15,14 @@ import { tmpdir } from 'node:os'
|
||||
import z from '@deepseek-ai/schemastery'
|
||||
import { SpillLocator, SpillStore } from '@deepseek-ai/dsh-spill'
|
||||
import type { SaveTextSpill, SpillRef } from '@deepseek-ai/dsh-spill'
|
||||
import { discoverDefaultRoots, privateRoot, saveTextFile, sweepSpillRoots } from './store.ts'
|
||||
import type { SweepRoot, WarnFn } from './store.ts'
|
||||
import { discoverDefaultRoots, sweepSpillRoots } from './cleanup.ts'
|
||||
import type { SweepRoot, WarnFn } from './cleanup.ts'
|
||||
import { privateRoot, saveTextFile } from './store.ts'
|
||||
|
||||
export { discoverDefaultRoots, encodeSegment, isErrno, privateRoot, saveTextFile, sessionDir, sweepSpillRoots, DEFAULT_ROOT_PREFIX } from './store.ts'
|
||||
export type { SavedText, SaveTextOptions, SweepOptions, SweepRoot, WarnFn } from './store.ts'
|
||||
export { discoverDefaultRoots, sweepSpillRoots } from './cleanup.ts'
|
||||
export type { SweepOptions, SweepRoot, WarnFn } from './cleanup.ts'
|
||||
export { DEFAULT_ROOT_PREFIX, encodeSegment, isErrno, privateRoot, saveTextFile, sessionDir } from './store.ts'
|
||||
export type { SavedText, SaveTextOptions } from './store.ts'
|
||||
|
||||
/** Milliseconds in one day — converts the `cleanupPeriodDays` config to the sweep cutoff. */
|
||||
const MS_PER_DAY = 24 * 60 * 60 * 1000
|
||||
@@ -117,9 +120,10 @@ export class LocalSpillStore extends SpillStore {
|
||||
/**
|
||||
* The roots the startup sweep covers: each discovered prior-default
|
||||
* `dsh-spill-*` temp root (see {@link discoverDefaultRoots}), pruned when
|
||||
* emptied, plus the active/configured root, which is swept but NEVER pruned
|
||||
* (the live process is still writing into it). The active root is de-duped out
|
||||
* of the discovered set so it is not swept twice or marked prunable. A test
|
||||
* emptied, plus the active/configured root, whose root and session directories
|
||||
* are NEVER pruned (the live process is still writing into them). The active
|
||||
* root is de-duped out of the discovered set so it is not swept twice or
|
||||
* marked prunable. A test
|
||||
* overrides this to inject an isolated root set — and, being the sweep's one
|
||||
* async gather point, to hold the sweep open across a disposal for the
|
||||
* quiescence check; it is a test seam, not a deployment knob.
|
||||
|
||||
@@ -8,44 +8,30 @@
|
||||
|
||||
import { createHash, randomBytes } from 'node:crypto'
|
||||
import { mkdtempSync } from 'node:fs'
|
||||
import { lstat, mkdir, open, readdir, rmdir, unlink } from 'node:fs/promises'
|
||||
import { mkdir, open } from 'node:fs/promises'
|
||||
import { join } from 'node:path'
|
||||
import { tmpdir } from 'node:os'
|
||||
|
||||
/**
|
||||
* Filename prefix for the lazily-created private default spill roots
|
||||
* (`mkdtemp(tmpdir()/dsh-spill-)`). Startup cleanup rediscovers these
|
||||
* per-process roots (from prior runs that used no configured `root`) by this
|
||||
* prefix — see {@link discoverDefaultRoots}.
|
||||
*/
|
||||
/** Prefix shared by default-root creation and startup discovery. */
|
||||
export const DEFAULT_ROOT_PREFIX = 'dsh-spill-'
|
||||
|
||||
/**
|
||||
* A backend-generated default root name: `dsh-spill-` plus the 6-character
|
||||
* suffix `mkdtemp` appends (see {@link privateRoot}). Discovery matches this
|
||||
* EXACT shape, not the bare prefix, so an unrelated `dsh-spill-test-*` fixture
|
||||
* or a foreign tool's differently-shaped `dsh-spill-…` directory is never
|
||||
* mistaken for a backend root to sweep.
|
||||
* Test a caught value for a Node system error code.
|
||||
*
|
||||
* @param error The caught value.
|
||||
* @param code The expected system error code.
|
||||
* @returns Whether the code matches.
|
||||
*/
|
||||
const DEFAULT_ROOT_RE = /^dsh-spill-[A-Za-z0-9]{6}$/
|
||||
|
||||
/**
|
||||
* A backend-generated session directory name: `session-` plus the 12 lowercase
|
||||
* hex characters {@link sessionDir} derives from `sha256(sessionId)`. The sweep
|
||||
* only descends into entries of this EXACT shape, so an unrelated
|
||||
* `session-backup` directory under a shared configured root is never swept.
|
||||
*/
|
||||
const SESSION_DIR_RE = /^session-[0-9a-f]{12}$/
|
||||
export function isErrno(error: unknown, code: string): boolean {
|
||||
return error instanceof Error && (error as NodeJS.ErrnoException).code === code
|
||||
}
|
||||
|
||||
let defaultRoot: string | undefined
|
||||
|
||||
/**
|
||||
* The default spill root: a private (0700) per-process directory under the OS
|
||||
* tmpdir, created lazily. Predictable world-readable paths would let other
|
||||
* local users read spilled tool output or pre-create symlinks; `mkdtemp` gives
|
||||
* an unpredictable suffix and 0700 semantics.
|
||||
* Return the lazily-created private per-process spill root.
|
||||
*
|
||||
* @returns The lazily-created private spill root.
|
||||
* @returns The private root path.
|
||||
*/
|
||||
export function privateRoot(): string {
|
||||
defaultRoot ??= mkdtempSync(join(tmpdir(), DEFAULT_ROOT_PREFIX))
|
||||
@@ -63,8 +49,8 @@ export function privateRoot(): string {
|
||||
* inputs never collide. The whole-segment tokens `.`/`..` are escaped so they
|
||||
* can never traverse. An empty string encodes to `~` (never an empty segment).
|
||||
*
|
||||
* @param raw The untrusted string to encode as one safe path segment.
|
||||
* @returns An injective, filesystem-safe single path segment.
|
||||
* @param raw Untrusted text.
|
||||
* @returns One injective filesystem-safe path segment.
|
||||
*/
|
||||
export function encodeSegment(raw: string): string {
|
||||
if (raw.length === 0) return '~'
|
||||
@@ -74,317 +60,72 @@ export function encodeSegment(raw: string): string {
|
||||
for (let i = 0; i < raw.length; i++) {
|
||||
const code = raw.charCodeAt(i)
|
||||
const ch = String.fromCharCode(code)
|
||||
if (ch !== '~' && /^[A-Za-z0-9._-]$/.test(ch)) {
|
||||
out += ch
|
||||
} else {
|
||||
out += '~' + code.toString(16).toUpperCase().padStart(4, '0')
|
||||
}
|
||||
out += ch !== '~' && /^[A-Za-z0-9._-]$/.test(ch)
|
||||
? ch
|
||||
: '~' + code.toString(16).toUpperCase().padStart(4, '0')
|
||||
}
|
||||
return out
|
||||
}
|
||||
/* jscpd:ignore-end */
|
||||
|
||||
/**
|
||||
* The session-scoped directory: `<root>/session-<hash(sessionId)>`, a short stable hash.
|
||||
* Derive the stable session-scoped directory under a spill root.
|
||||
*
|
||||
* @param root The spill root directory.
|
||||
* @param sessionId The owning session id to hash into a stable directory name.
|
||||
* @returns The absolute session-scoped spill directory path.
|
||||
* @param root The spill root.
|
||||
* @param sessionId The owning session id.
|
||||
* @returns The stable session-scoped directory.
|
||||
*/
|
||||
export function sessionDir(root: string, sessionId: string): string {
|
||||
const hash = createHash('sha256').update(sessionId).digest('hex').slice(0, 12)
|
||||
return join(root, `session-${hash}`)
|
||||
}
|
||||
|
||||
/** Options for {@link saveTextFile} — the resolved root and the request fields the store needs. */
|
||||
/** Inputs needed to save a local spill file. */
|
||||
export interface SaveTextOptions {
|
||||
/** The spill root directory (configured or the lazy private default). */
|
||||
/** Spill root. */
|
||||
root: string
|
||||
/** The owning session id (scopes the directory). */
|
||||
/** Owning session id. */
|
||||
sessionId: string
|
||||
/** Caller-suggested base name; sanitized to one safe segment before use. */
|
||||
/** Caller-suggested filename. */
|
||||
suggestedName: string
|
||||
/** The full text to persist. */
|
||||
/** Full text to persist. */
|
||||
content: string
|
||||
}
|
||||
|
||||
/** A written spill file. */
|
||||
export interface SavedText {
|
||||
/** Absolute saved path. */
|
||||
path: string
|
||||
/** UTF-8 content length. */
|
||||
bytes: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Write `content` to a fresh file under the session-scoped directory and return
|
||||
* its path + byte length. The filename is a random hex prefix plus the
|
||||
* sanitized `suggestedName`, so it is unpredictable (defeats symlink planting in
|
||||
* a shared root) AND stays readable. The open is exclusive + owner-only
|
||||
* (`'wx', 0o600`): it fails on any existing path — symlink or not — so a
|
||||
* pre-planted target cannot redirect the write.
|
||||
*
|
||||
* @param options The resolved root and request fields required to save the file.
|
||||
* @returns The written file path and UTF-8 byte length.
|
||||
* Write text to a fresh 0600 file below its private session directory.
|
||||
* @param options The save request.
|
||||
* @returns The saved path and UTF-8 byte length.
|
||||
*/
|
||||
export async function saveTextFile(options: SaveTextOptions): Promise<SavedText> {
|
||||
const dir = sessionDir(options.root, options.sessionId)
|
||||
await mkdir(dir, { recursive: true, mode: 0o700 })
|
||||
const safeName = encodeSegment(options.suggestedName)
|
||||
const path = join(dir, `${randomBytes(6).toString('hex')}-${safeName}`)
|
||||
const bytes = Buffer.byteLength(options.content, 'utf8')
|
||||
const handle = await open(path, 'wx', 0o600)
|
||||
const path = join(dir, `${randomBytes(6).toString('hex')}-${encodeSegment(options.suggestedName)}`)
|
||||
let handle
|
||||
for (;;) {
|
||||
await mkdir(dir, { recursive: true, mode: 0o700 })
|
||||
try {
|
||||
handle = await open(path, 'wx', 0o600)
|
||||
break
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- requires another process to remove the directory
|
||||
between mkdir and open, or an external permission/IO race. */
|
||||
if (isErrno(error, 'ENOENT')) continue
|
||||
throw error
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
try {
|
||||
await handle.writeFile(options.content)
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
return { path, bytes }
|
||||
}
|
||||
|
||||
/** A one-argument warning sink — the sweep's only side effect on failure (never throws). */
|
||||
export type WarnFn = (message: string) => void
|
||||
|
||||
/** One root to sweep, plus whether an emptied root directory should itself be pruned. */
|
||||
export interface SweepRoot {
|
||||
/** Absolute spill root to sweep. */
|
||||
path: string
|
||||
/**
|
||||
* When `true`, remove the root directory itself once its last `session-*`
|
||||
* child is pruned. Set for DISCOVERED prior-default `dsh-spill-*` roots (one
|
||||
* per past process — otherwise they accumulate empty forever), never for the
|
||||
* active/configured root the live process is still writing into.
|
||||
*/
|
||||
pruneWhenEmpty: boolean
|
||||
}
|
||||
|
||||
/** Options for {@link sweepSpillRoots} — the roots to scan, the age cutoff, and a failure sink. */
|
||||
export interface SweepOptions {
|
||||
/** Roots to sweep (configured/active root and/or discovered prior-default roots). */
|
||||
roots: SweepRoot[]
|
||||
/**
|
||||
* Epoch-millis cutoff: a regular file is deleted when its `mtime` is strictly
|
||||
* older than this. The caller derives it from `now - cleanupPeriodDays`, so a
|
||||
* file written exactly at the boundary is kept (only strictly-older expires).
|
||||
*/
|
||||
cutoffMs: number
|
||||
/** Where a contained filesystem failure is reported; the sweep itself never throws. */
|
||||
warn: WarnFn
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a single path, treating a concurrent-race disappearance as success.
|
||||
* A parallel process (or another sweep) may `unlink` the same file between our
|
||||
* scan and our own `unlink` — ENOENT then means the goal (file gone) already
|
||||
* holds, so it is not a failure. Any other error is reported and swallowed.
|
||||
*
|
||||
* @param path The absolute file path to remove.
|
||||
* @param warn Sink for a non-ENOENT failure message.
|
||||
* @returns Resolves once the removal was attempted (never rejects).
|
||||
*/
|
||||
async function unlinkIdempotent(path: string, warn: WarnFn): Promise<void> {
|
||||
try {
|
||||
await unlink(path)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- reached only when a file selected for deletion (a
|
||||
regular file that passed lstat) then fails to unlink: either it raced away
|
||||
(ENOENT) or a permission/IO fault struck between the stat and the unlink.
|
||||
Neither is deterministically reproducible in-process. */
|
||||
if (isErrno(error, 'ENOENT')) return
|
||||
warn(`spill-local: failed to delete ${path}: ${String(error)}`)
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* True when `error` is a Node system error carrying the given `code`.
|
||||
*
|
||||
* @param error The caught value to test.
|
||||
* @param code The `NodeJS.ErrnoException` code to match (e.g. `'ENOENT'`).
|
||||
* @returns `true` when `error` is an `Error` whose `code` equals `code`.
|
||||
*/
|
||||
export function isErrno(error: unknown, code: string): boolean {
|
||||
return error instanceof Error && (error as NodeJS.ErrnoException).code === code
|
||||
}
|
||||
|
||||
/**
|
||||
* Sweep one spill session directory: delete expired regular files, skip
|
||||
* everything else, and report the directory empty afterward so the caller can
|
||||
* prune it. The `dir` entry MUST be a real directory — the caller `lstat`s it
|
||||
* first and skips a symlink, so this never follows a `session-*` symlink into a
|
||||
* foreign tree. Inside, a symlink or any non-regular entry (socket, fifo, nested
|
||||
* dir) is left untouched — `lstat` never follows a link, so a planted symlink
|
||||
* can neither be deleted nor redirect the age check. Every per-entry failure is
|
||||
* contained: one unreadable file does not abort the directory.
|
||||
*
|
||||
* @param dir The absolute session directory to scan (already confirmed a real dir).
|
||||
* @param cutoffMs Files with `mtime` strictly older than this are deleted.
|
||||
* @param warn Sink for contained filesystem failures.
|
||||
* @returns `true` when the directory holds no entries after the sweep (a prune candidate).
|
||||
*/
|
||||
async function sweepSessionDir(dir: string, cutoffMs: number, warn: WarnFn): Promise<boolean> {
|
||||
let names: string[]
|
||||
try {
|
||||
names = await readdir(dir)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- the caller lstat'd this entry and confirmed a real
|
||||
directory just before the call, so readdir fails only when the dir races
|
||||
away (ENOENT) or a permission/IO fault strikes in that window; not
|
||||
deterministically reproducible. False keeps it out of the prune step. */
|
||||
warn(`spill-local: failed to read ${dir}: ${String(error)}`)
|
||||
return false
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
let remaining = names.length
|
||||
for (const name of names) {
|
||||
const path = join(dir, name)
|
||||
let stats
|
||||
try {
|
||||
stats = await lstat(path)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- an entry that readdir just returned then fails to
|
||||
lstat only by racing away (ENOENT) or a permission/IO fault; keep it out
|
||||
of the deterministic test surface. */
|
||||
if (isErrno(error, 'ENOENT')) { remaining--; continue }
|
||||
warn(`spill-local: failed to stat ${path}: ${String(error)}`)
|
||||
continue
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
// Only regular files expire. Symlinks and other special entries are skipped
|
||||
// (never followed) so the sweep cannot be redirected or delete a link.
|
||||
if (!stats.isFile()) continue
|
||||
if (stats.mtimeMs >= cutoffMs) continue
|
||||
await unlinkIdempotent(path, warn)
|
||||
remaining--
|
||||
}
|
||||
return remaining === 0
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort one-shot cleanup: across each root, delete expired regular files
|
||||
* under its `session-*` directories and prune any directory left empty. The
|
||||
* sweep is idempotent and safe to run concurrently with live spill writes and
|
||||
* with another process's sweep — per-file expiry preserves a fresh write even
|
||||
* if it lands mid-sweep, and every filesystem failure is caught and reported
|
||||
* rather than thrown, so a caller can await this during activation/disposal
|
||||
* without it ever rejecting.
|
||||
*
|
||||
* @param options The roots to sweep, the age cutoff, and the failure sink.
|
||||
* @returns Resolves when the sweep finishes (never rejects).
|
||||
*/
|
||||
export async function sweepSpillRoots(options: SweepOptions): Promise<void> {
|
||||
const { roots, cutoffMs, warn } = options
|
||||
for (const root of roots) {
|
||||
let entries: string[]
|
||||
try {
|
||||
entries = await readdir(root.path)
|
||||
} catch (error: unknown) {
|
||||
// A root that does not exist yet (no spill ever written) is the common
|
||||
// case, not an error: ENOENT is silent, anything else is reported.
|
||||
if (!isErrno(error, 'ENOENT')) warn(`spill-local: failed to read root ${root.path}: ${String(error)}`)
|
||||
continue
|
||||
}
|
||||
// Track whether the root holds ANY entry the sweep did not fully reclaim, so
|
||||
// a discovered prior-default root can be pruned only when nothing remains.
|
||||
let rootEmptiable = true
|
||||
for (const name of entries) {
|
||||
// Only the backend's own `session-<12 hex>` directories are swept; an
|
||||
// unrelated sibling (`session-backup`, a stray file) is left untouched and
|
||||
// blocks pruning the root.
|
||||
if (!SESSION_DIR_RE.test(name)) { rootEmptiable = false; continue }
|
||||
const dir = join(root.path, name)
|
||||
let stats
|
||||
try {
|
||||
// lstat the session entry itself: a `session-*` SYMLINK must never be
|
||||
// followed (readdir/unlink through it would delete files in a foreign
|
||||
// target). Only a real directory is swept.
|
||||
stats = await lstat(dir)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- an entry readdir just returned fails to lstat only
|
||||
by racing away (ENOENT) or a permission/IO fault; not deterministically
|
||||
reproducible. */
|
||||
if (!isErrno(error, 'ENOENT')) warn(`spill-local: failed to stat ${dir}: ${String(error)}`)
|
||||
continue
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
if (!stats.isDirectory()) { rootEmptiable = false; continue }
|
||||
const empty = await sweepSessionDir(dir, cutoffMs, warn)
|
||||
if (!empty) { rootEmptiable = false; continue }
|
||||
try {
|
||||
await rmdir(dir)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- prune runs only on a dir observed empty; a failure
|
||||
here means a concurrent writer added a file (ENOTEMPTY) or a
|
||||
permission/IO fault struck — both are races outside deterministic
|
||||
in-process testing. */
|
||||
rootEmptiable = false
|
||||
if (!isErrno(error, 'ENOENT') && !isErrno(error, 'ENOTEMPTY')) {
|
||||
warn(`spill-local: failed to prune ${dir}: ${String(error)}`)
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
// A discovered prior-default root (one per past process) is removed once its
|
||||
// last session dir is gone — otherwise empty roots accumulate forever and
|
||||
// every future startup rescans them. The active/configured root is never
|
||||
// pruned (the live process is still writing into it).
|
||||
if (root.pruneWhenEmpty && rootEmptiable) {
|
||||
try {
|
||||
await rmdir(root.path)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- prune runs only on a root whose every child was
|
||||
reclaimed; a failure here means a concurrent writer added a fresh
|
||||
spill after our scan (ENOTEMPTY) or removed the root already (ENOENT)
|
||||
or a permission/IO fault struck — all races outside deterministic
|
||||
in-process testing. */
|
||||
if (!isErrno(error, 'ENOENT') && !isErrno(error, 'ENOTEMPTY')) {
|
||||
warn(`spill-local: failed to prune root ${root.path}: ${String(error)}`)
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Discover prior default spill roots: the `dsh-spill-<6 chars>` directories
|
||||
* directly under `base` (the OS tmpdir) that earlier runs created via
|
||||
* {@link privateRoot} when no `root` was configured. A long-lived deployment
|
||||
* with a configured root will find none; a series of default-root runs
|
||||
* accumulates one per process, so the startup sweep reclaims them all. Matching
|
||||
* is the EXACT `mkdtemp` shape (see {@link DEFAULT_ROOT_RE}), not the bare
|
||||
* prefix, so an unrelated `dsh-spill-test-*` fixture or a foreign
|
||||
* differently-shaped directory is never swept; symlinks and non-directories are
|
||||
* excluded too — only real directories the backend could have created.
|
||||
*
|
||||
* @param warn Sink for a failure reading `base` (returns `[]` on failure).
|
||||
* @param base The directory to scan; defaults to the OS tmpdir (a test seam).
|
||||
* @returns Absolute paths of the discovered default roots (possibly empty).
|
||||
*/
|
||||
export async function discoverDefaultRoots(warn: WarnFn, base: string = tmpdir()): Promise<string[]> {
|
||||
let entries: string[]
|
||||
try {
|
||||
entries = await readdir(base)
|
||||
} catch (error: unknown) {
|
||||
warn(`spill-local: failed to scan ${base} for default roots: ${String(error)}`)
|
||||
return []
|
||||
}
|
||||
const roots: string[] = []
|
||||
for (const name of entries) {
|
||||
if (!DEFAULT_ROOT_RE.test(name)) continue
|
||||
const path = join(base, name)
|
||||
let stats
|
||||
try {
|
||||
// lstat, not stat: a symlink named `dsh-spill-*` must not be treated as a
|
||||
// root we then sweep (it could point anywhere).
|
||||
stats = await lstat(path)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- an entry readdir just returned fails to lstat only by
|
||||
racing away (ENOENT) or a permission/IO fault; not deterministically
|
||||
reproducible. */
|
||||
if (!isErrno(error, 'ENOENT')) warn(`spill-local: failed to stat default root ${path}: ${String(error)}`)
|
||||
continue
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
if (stats.isDirectory()) roots.push(path)
|
||||
}
|
||||
return roots
|
||||
return { path, bytes: Buffer.byteLength(options.content, 'utf8') }
|
||||
}
|
||||
|
||||
@@ -3,10 +3,10 @@
|
||||
* returns a locator + byte length + retrieval hint, filename sanitization
|
||||
* neutralizes traversal, the configured `root` is honored (and the private
|
||||
* default when omitted), and a storage failure rejects. The startup cleanup
|
||||
* sweep expires old files, prunes empty dirs, skips symlinks/unknown entries,
|
||||
* sweep expires old files, prunes stale roots, skips symlinks/unknown entries,
|
||||
* discovers prior default roots, contains filesystem failures, and is awaited on
|
||||
* disposal without blocking activation. The Cordis-free `store.ts` helpers are
|
||||
* exercised directly for the naming/encoding and sweep edge cases.
|
||||
* disposal without blocking activation. The Cordis-free store and cleanup
|
||||
* helpers are exercised directly for their edge cases.
|
||||
*/
|
||||
|
||||
import { describe, expect, it, beforeEach, afterEach, vi } from 'vitest'
|
||||
@@ -280,7 +280,7 @@ describe('startup cleanup sweep', () => {
|
||||
expect(existsSync(old)).toBe(true)
|
||||
})
|
||||
|
||||
it('prunes a session directory left empty, keeps one with a surviving file', async () => {
|
||||
it('keeps active session directories after deleting expired files', async () => {
|
||||
const emptied = sessionDir(root, 'emptied')
|
||||
const kept = sessionDir(root, 'kept')
|
||||
mkdirSync(emptied, { recursive: true })
|
||||
@@ -288,7 +288,7 @@ describe('startup cleanup sweep', () => {
|
||||
writeAged(join(emptied, 'a.txt'), 'x', 40)
|
||||
writeAged(join(kept, 'fresh.txt'), 'y', 1)
|
||||
await runSweep([active(root)])
|
||||
expect(existsSync(emptied)).toBe(false)
|
||||
expect(existsSync(emptied)).toBe(true)
|
||||
expect(existsSync(kept)).toBe(true)
|
||||
})
|
||||
|
||||
@@ -351,7 +351,7 @@ describe('startup cleanup sweep', () => {
|
||||
await runSweep([{ path: prior, pruneWhenEmpty: true }, active(root)])
|
||||
expect(existsSync(prior)).toBe(false) // discovered root pruned
|
||||
expect(existsSync(root)).toBe(true) // active root kept
|
||||
expect(existsSync(activeDir)).toBe(false) // its emptied session dir still pruned
|
||||
expect(existsSync(activeDir)).toBe(true) // active session dirs remain writable
|
||||
} finally {
|
||||
rmSync(prior, { recursive: true, force: true })
|
||||
}
|
||||
@@ -458,6 +458,13 @@ describe('startup cleanup sweep', () => {
|
||||
expect(warn).toHaveBeenCalledWith(expect.stringContaining('failed to read root'))
|
||||
})
|
||||
|
||||
it('contains an exception from the warning sink', async () => {
|
||||
const filePath = join(root, 'not-a-dir'); writeFileSync(filePath, 'x')
|
||||
const warn = vi.fn(() => { throw new Error('logger failed') })
|
||||
await expect(sweepSpillRoots({ roots: [active(filePath)], cutoffMs: Date.now(), warn })).resolves.toBeUndefined()
|
||||
expect(warn).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('a nonexistent root is silent (the common no-spill-yet case)', async () => {
|
||||
const warn = vi.fn()
|
||||
await sweepSpillRoots({ roots: [active(join(root, 'never-created'))], cutoffMs: Date.now(), warn })
|
||||
@@ -500,4 +507,3 @@ describe('isErrno', () => {
|
||||
expect(isErrno(new Error('no code'), 'ENOENT')).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
Reference in New Issue
Block a user