diff --git a/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.i18n.yaml b/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.i18n.yaml index 46aafdd528..7409f997a1 100644 --- a/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.md -2026-08-19-projection-cache-per-session-files.md: e32a74de3beea2fbe2dfff8835c8d1ef86983358 -2026-08-19-projection-cache-per-session-files.zh.md: 9c6c9b250282de059b649db1723a5f1f2e9b3945 +2026-08-19-projection-cache-per-session-files.md: 9e102e786a6c06d82d1a0f45cc2f96a50c8abcd8 +2026-08-19-projection-cache-per-session-files.zh.md: d875c3f57800936f66fbf65233637df9bf300e2d diff --git a/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.md b/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.md index e32a74de3b..9e102e786a 100644 --- a/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.md +++ b/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.md @@ -20,7 +20,7 @@ Reads and writes share ONE coherent state: every read (`cachedSnapshot`) is a sy - Listing is a synchronous in-memory read; a session without a record document simply lacks the projection column. - ACP, headless, SDK, and Web sessions publish cache rows for later consumers. The log-leading durability barrier may flush a covered prefix at the cache cadence and split otherwise coalesced physical JSONL runs; recorded profile snapshots re-pack the logical event stream so cache timing does not define fixture layout. - The per-record contract scopes failure: a malformed or stale-version document reads as an absent record at open, so one bad file never bricks the cache, and a checkpoint schema bump discards stale sessions per record instead of rejecting the whole domain. -- No migration: the cache is derived data, never an authority. An obsolete cache (any earlier format) is never read — the first cold read refolds from the log and writes the current format. +- The json backend bootstraps the per-record tree from the legacy whole-unit cache only when enumeration finds no new-layout document path. Any new document path, including an unreadable or stale file, suppresses the bootstrap for the whole unit; missing session rows refold from the log. The legacy file remains untouched. - The cache record is bound to the same log lifecycle as before: the stored `{createdAt, cwd}` identity guards against a recreated id. ## Alternatives considered diff --git a/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.zh.md b/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.zh.md index 9c6c9b2502..d875c3f578 100644 --- a/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.zh.md +++ b/.agents/notes/implemented/architecture/2026-08-19-projection-cache-per-session-files.zh.md @@ -20,7 +20,7 @@ Status: implemented - 列表读取是同步内存读;没有记录文档的会话只是缺少投影列。 - ACP、headless、SDK 与 Web 会话都会发布缓存行,供后续消费方使用。确保日志领先的持久性屏障可能按缓存节奏 flush 已覆盖的前缀,并拆分原本会合并的物理 JSONL 行;各 profile 的录制快照会重新 pack 逻辑事件流,因此缓存时序不会决定 fixture 布局。 - per-record 契约把故障范围缩小到单记录:畸形或过期版本的文档在打开时读作"无此记录",单个坏文件不会拖垮整个缓存;检查点 schema 升级按会话丢弃过期行,而不是拒绝整个域。 -- 无需迁移:缓存是派生数据,绝非权威。过时的缓存(任何更早格式)从不被读取——首次冷读从日志重折叠并写出当前格式。 +- json 后端仅在枚举时没有发现任何新布局文档路径,才从旧整单元缓存引导 per-record 目录树。只要存在任意新文档路径,即使文件不可读或版本陈旧,也会对整个单元禁用引导;缺失的会话行从日志重折叠。旧文件保持不变。 - 缓存记录仍绑定同一日志生命周期:存储的 `{createdAt, cwd}` 身份防止被重建的 id 误导。 ## Alternatives considered diff --git a/packages/storage/storage-json/README.i18n.yaml b/packages/storage/storage-json/README.i18n.yaml index 8f40753305..69853e8521 100644 --- a/packages/storage/storage-json/README.i18n.yaml +++ b/packages/storage/storage-json/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/storage/storage-json/README.md -README.md: 1a9873d2a3300998de67efa9be566ebc1250b0bf -README.zh.md: e01ed29c5464e2f36472936db4a88db1bc4e7612 +README.md: d23362be7ad9d884a60b941663dfdd33668c300d +README.zh.md: b90c78fa41302ce5f14a897b07cf1eb548e1843b diff --git a/packages/storage/storage-json/README.md b/packages/storage/storage-json/README.md index 1a9873d2a3..d23362be7a 100644 --- a/packages/storage/storage-json/README.md +++ b/packages/storage/storage-json/README.md @@ -13,6 +13,7 @@ Design: [domain KV storage Agent Note](../../../.agents/notes/proposed/architect - `single` layout: the in-memory unit state is authoritative; every write primitive republishes the whole file via temp-write + fsync + atomic `rename()` replace. A unit file is always the complete current net state — legibility is this backend's reason to exist; scale is the SQLite backend's job. - A missing file (or unit directory, for `per-record`) opens as an empty unit and materializes on the first write. In `single` layout a foreign or unparsable file rejects with `malformed-medium`, and a stored version differing from the descriptor rejects with `version-mismatch` (no migration, pre-release stance). In `per-record` layout the contract is per record instead: a document that is malformed or stamped with another version reads as an absent record, so one bad or stale file never bricks the unit, and a version bump discards stale records rather than rejecting the whole unit. +- An empty `per-record` tree bootstraps its declared-table records from a legacy `.json` whole-unit file and retains that file unchanged. Any new-layout document path in a declared table, or `global.json` for a declared global, suppresses the bootstrap for the whole unit even when the document is unreadable or stale; absent records remain absent instead of being filled from legacy state. - Write ordering across calls belongs to the caller (the domain layer's write chain); each single call is atomic and durable once resolved. ## Config diff --git a/packages/storage/storage-json/README.zh.md b/packages/storage/storage-json/README.zh.md index e01ed29c54..b90c78fa41 100644 --- a/packages/storage/storage-json/README.zh.md +++ b/packages/storage/storage-json/README.zh.md @@ -13,6 +13,7 @@ - `single` 布局:内存中的单元状态具有最终决定权;每个写入原语都会通过临时文件写入 + fsync + 原子 `rename()` 替换重新发布整个文件。单元文件始终是完整的当前状态:可读性是该后端存在的理由,规模问题则属于 SQLite 后端。 - 缺失文件(`per-record` 时是缺失单元目录)会作为空单元打开,并在第一次写入时物化。`single` 布局下外来或无法解析的文件以 `malformed-medium` 拒绝;已存版本与描述符不同时以 `version-mismatch` 拒绝(预发布立场,不迁移)。`per-record` 布局的契约改为按记录:畸形或版本戳不符的文档读作"无此记录",单个坏文件或过期文件不会拖垮整个单元,版本升级按记录丢弃过期行而不是拒绝整个单元。 +- 空的 `per-record` 目录树会从旧 `.json` 整单元文件引导其已声明表的记录,并原样保留旧文件。已声明表中只要存在任意新布局文档路径,或已声明 global 对应的 `global.json` 存在,就会对整个单元禁用引导,即使该文档不可读或版本陈旧;缺失记录仍保持缺失,不从旧状态补入。 - 跨调用的写入顺序属于调用方(领域层的写入链);每次调用都具备原子性,并在完成时已达到持久状态。 ## 配置 diff --git a/packages/storage/storage-json/src/per-record-unit.ts b/packages/storage/storage-json/src/per-record-unit.ts index e05cec8c9c..b75f9f9452 100644 --- a/packages/storage/storage-json/src/per-record-unit.ts +++ b/packages/storage/storage-json/src/per-record-unit.ts @@ -15,9 +15,11 @@ * of migrating them. Record keys become path segments, so they must be * path-safe (`[a-zA-Z0-9_-]+`); an unsafe key rejects at write. * - * One-time migration: a legacy whole-unit file `/.json` (the - * pre-per-record layout) is split into per-record documents on first open, - * new records winning, and deleted once every record migrated. + * Legacy bootstrap: when the new tree has no document path, a legacy + * whole-unit file `/.json` (the pre-per-record layout) seeds + * per-record documents. Any new document path, including one whose contents + * are unreadable or stale, suppresses the bootstrap for the whole unit. The + * legacy file is never changed or deleted. * @module @deepseek-ai/dsh-storage-json/src/per-record-unit */ @@ -72,39 +74,40 @@ async function loadPerRecordState(descriptor: KvUnitDescriptor, dir: string): Pr entries = await readdir(dir, { withFileTypes: true }) } catch (error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error - // Missing directory = empty unit; the legacy migration below still runs + // Missing directory = empty unit; the legacy bootstrap below still runs // (the fresh-upgrade shape is exactly an absent new tree). } - if (entries !== undefined) { - await Promise.all(entries.map(async (entry) => { + const hasNewDocuments = entries === undefined + ? false + : (await Promise.all(entries.map(async (entry) => { if (entry.isDirectory()) { const records = state.tables.get(entry.name) if (records !== undefined) { - await loadTableRecords(records, descriptor.version, join(dir, entry.name)) + return loadTableRecords(records, descriptor.version, join(dir, entry.name)) } - } else if (entry.isFile() && entry.name === 'global.json' && descriptor.hasGlobal) { + } + if (entry.name === 'global.json' && descriptor.hasGlobal) { const global = await readRecord(join(dir, entry.name), descriptor.version) if (global !== undefined) state.global = global + return true } - })) - } - await migrateLegacyUnit(descriptor, dir, state) + return false + }))).some(Boolean) + if (!hasNewDocuments) await bootstrapLegacyUnit(descriptor, dir, state) return state } /** - * One-time migration of a legacy whole-unit file (`/.json`, the - * pre-per-record layout). Every record it holds that the new tree lacks is - * written as a per-record document — an already-present new record wins — - * and the legacy file is deleted only after all records migrated. A missing - * legacy file, or one that is unreadable, foreign (another unit's name), or - * not a unit document, is left alone: the migration is idempotent, and the - * legacy file's absence is the "migrated" marker. + * Bootstrap an empty per-record tree from a legacy whole-unit file + * (`/.json`, the pre-per-record layout). Every declared-table + * record is copied into a current-version document, while the legacy file is + * retained unchanged. A missing, foreign (another unit's name), malformed, + * or non-unit legacy file is left alone; other read failures propagate. * @param descriptor - Static identity and shape of the unit. * @param dir - The per-record unit directory (`/`). - * @param state - The tree state loaded so far; migrated records are added. + * @param state - The empty tree state; bootstrapped records are added. */ -async function migrateLegacyUnit(descriptor: KvUnitDescriptor, dir: string, state: UnitState): Promise { +async function bootstrapLegacyUnit(descriptor: KvUnitDescriptor, dir: string, state: UnitState): Promise { const legacyPath = join(dirname(dir), `${descriptor.name}.json`) let text: string | undefined try { @@ -130,19 +133,22 @@ async function migrateLegacyUnit(descriptor: KvUnitDescriptor, dir: string, stat const target = state.tables.get(table) if (target === undefined) continue for (const [key, value] of Object.entries(records)) { - if (target.has(key)) continue // An existing new record wins. const path = join(dir, table, `${key}.json`) await mkdir(dirname(path), { recursive: true, mode: 0o700 }) await writeAtomic(path, serializeRecord(descriptor.version, value)) target.set(key, value) } } - await rm(legacyPath, { force: true }) // Every record migrated: drop the legacy file. } -/** Read one declared table's record documents into `records`. */ -async function loadTableRecords(records: Map, version: number, dir: string): Promise { +/** + * Read one declared table's record documents into `records`. + * @returns whether the directory contains any `.json` document path, + * independently of key safety, readability, or stored version. + */ +async function loadTableRecords(records: Map, version: number, dir: string): Promise { const files = await readdir(dir, { withFileTypes: true }) + const hasDocuments = files.some(file => file.name.endsWith('.json')) const loaded = await Promise.all(files.map(async (file) => { if (!file.name.endsWith('.json')) return const key = file.name.slice(0, -'.json'.length) @@ -153,6 +159,7 @@ async function loadTableRecords(records: Map, version: number, for (const record of loaded) { if (record !== undefined) records.set(...record) } + return hasDocuments } /** Read one record document; a foreign (unreadable or stale) one reads as absent. */ diff --git a/packages/storage/storage-json/tests/json-backend.spec.ts b/packages/storage/storage-json/tests/json-backend.spec.ts index 59260b8c03..41ad071ea0 100644 --- a/packages/storage/storage-json/tests/json-backend.spec.ts +++ b/packages/storage/storage-json/tests/json-backend.spec.ts @@ -338,26 +338,41 @@ describe('per-record layout', () => { await backend.close() }) - it('migrates a legacy whole-unit file once, new records win, then deletes it', async () => { + it('bootstraps an empty per-record tree from a legacy whole-unit file and preserves it', async () => { const root = await freshRoot() // A legacy single-layout file for the same unit (any older version); // the extra table is not declared and must be skipped. - await writeFile(join(root, 'recs.json'), JSON.stringify({ + const legacy = JSON.stringify({ unit: { name: 'recs', version: 3 }, global: null, tables: { t: { old1: { v: 1 }, old2: { v: 2 } }, undeclared: { k: { v: 0 } } }, - }), 'utf8') - // A new per-record row that must win over its legacy namesake. - await mkdir(join(root, 'recs', 't'), { recursive: true }) - await writeFile(join(root, 'recs', 't', 'old1.json'), JSON.stringify({ version: 2, record: { v: 9 } }), 'utf8') + }) + await writeFile(join(root, 'recs.json'), legacy, 'utf8') const backend = new JsonStorageBackend(root) const unit = await backend.kv.open(descriptor) - expect(await unit.loadAll()).toEqual({ tables: { t: { old1: { v: 9 }, old2: { v: 2 } } }, global: null }) - // The legacy file was deleted; a reopen reads the migrated tree. - await expect(readFile(join(root, 'recs.json'), 'utf8')).rejects.toMatchObject({ code: 'ENOENT' }) + expect(await unit.loadAll()).toEqual({ tables: { t: { old1: { v: 1 }, old2: { v: 2 } } }, global: null }) + await expect(readFile(join(root, 'recs.json'), 'utf8')).resolves.toBe(legacy) await unit.close() const unit2 = await backend.kv.open(descriptor) - expect(await unit2.loadAll()).toEqual({ tables: { t: { old1: { v: 9 }, old2: { v: 2 } } }, global: null }) + expect(await unit2.loadAll()).toEqual({ tables: { t: { old1: { v: 1 }, old2: { v: 2 } } }, global: null }) + await backend.close() + }) + + it('ignores the legacy whole-unit file when any new document path exists', async () => { + const root = await freshRoot() + const legacy = JSON.stringify({ + unit: { name: 'recs', version: 1 }, + global: null, + tables: { t: { old: { v: 1 } } }, + }) + await writeFile(join(root, 'recs.json'), legacy, 'utf8') + await mkdir(join(root, 'recs', 't'), { recursive: true }) + await writeFile(recordPath(root, 'broken'), '{oops', 'utf8') + const backend = new JsonStorageBackend(root) + const unit = await backend.kv.open(descriptor) + expect(await unit.loadAll()).toEqual({ tables: { t: {} }, global: null }) + await expect(readFile(recordPath(root, 'old'), 'utf8')).rejects.toMatchObject({ code: 'ENOENT' }) + await expect(readFile(join(root, 'recs.json'), 'utf8')).resolves.toBe(legacy) await backend.close() })