deepseek-harness/packages/test-support/session-snapshot/README.md

description, kind

description	kind
Session-log snapshot support for keyless profile tests: manifests, identity redaction, normalization, workspace checks, and protocol adapters.	package-library

@deepseek-ai/dsh-session-snapshot

English | 中文

Summary

dsh-session-snapshot provides the shared support behind keyless recorded-session tests (pnpm run test:snapshot): closed manifests, typed identity redaction, normalization, workspace comparison, fixture guards, and protocol adapters for headless, SDK, ACP, and Web owners. The ACP adapter launches the tested profile as a real subprocess, drives a deterministic input script, and registers the complete record, replay, and refresh suite. Every scenario owns enough committed evidence to prove model-visible output and filesystem effects without trusting the agent's report. The package entry imports vitest and is therefore available only inside a vitest run.

Table of Contents

• Use this package
• Understand the implementation
• Further Exploration
• Model Experience
• Known Limitations and Deferred Work
• Dev Note

---

Use this package

This package turns a shipped profile scenario into a keyless snapshot suite: write a scenario table and a fixtures directory, call the matching adapter once, and the kit owns launching or composing the profile, driving the scenario, comparing normalized output, and guarding the committed fixtures.

Writing a snapshot suite

A consuming *.snapshot.ts is the scenario table plus one factory call. AgentUnderTest supplies absolute binScript, optional libBinScript, configPath, and tsconfigPath paths, because the subprocess cwd sits outside the repository:

import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
import {
  defineAcpSnapshotSuite,
  type Scenario,
  type SnapshotSuiteOptions,
} from '@deepseek-ai/dsh-session-snapshot'

function snapshotMode(value: string | undefined): SnapshotSuiteOptions['mode'] {
  switch (value) {
    case undefined:
    case '':
    case 'replay': return 'replay'
    case 'record': return 'record'
    case 'refresh': return 'refresh'
    default: throw new Error(`unknown DSH_SNAPSHOT mode: ${value}`)
  }
}

const SCENARIOS: Scenario[] = [
  { name: 'text-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
]

defineAcpSnapshotSuite({
  agent: { // absolute paths, resolved from the suite's own location
    binScript: fileURLToPath(new URL('../../../apps/cli/src/bin.ts', import.meta.url)),
    configPath: fileURLToPath(new URL('../cordis.yml', import.meta.url)),
    profile: 'acp',
    tsconfigPath: fileURLToPath(new URL('../../../tsconfig.json', import.meta.url)),
  },
  snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'),
  scenarios: SCENARIOS, // exactly one entry per header class sets pinsHeader
  mode: snapshotMode(process.env.DSH_SNAPSHOT),
})

Each recorded-session directory carries a closed snapshot.yml manifest plus its session.jsonl and contiguous session.<n>.jsonl child logs. The manifest names the scenario, shipped profile, composition/header class, recording source, and only the replay, platform, permission, environment, workspace, or input facts the completed session cannot reconstruct. The adapter registers expected-output, session-log, and optional workspace.expected/ comparisons; guards reject orphan directories, missing files, absolute paths, malformed manifests, and platform-specific separators.

normalizeSessionSnapshot retains the complete session header and event payloads but omits ordinary seq/time and packed-row seq0/time0 envelopes from committed fixtures after normalizing paths and scrubbing request headers. Replay synthesizes the envelopes in memory, while runtime persistence continues to write complete logs. Fixtures use canonical packed rows; the temporary repository migrator (pnpm run migrate:packed-session-fixtures) rewrites older layouts, and its removal proposal owns its deletion.

Record, replay, and refresh

pnpm run test:snapshot:record calls the live LLM and rewrites recorded model fixtures; pnpm run test:snapshot:refresh stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and owned prompt and tool-schema sidecars from committed model scripts. Each composition owner keeps its replay patch beside its live patch; top-level snapshots/ owns session-driven scenarios, while other expected outputs stay beside their owning package. dsh-llm-replay serves the recorded streams selected through DSH_SNAPSHOT_* environment values.

Pinning request headers

A pin owns its generated system-prompt.expected.md or tool-schemas.expected.json sidecar by default; systemPromptSource and toolSchemasSource name another pin when the complete corresponding sequence is identical, so each distinct version is committed once. The pin's session.jsonl stores "system":"{{system}}","tools":"{{tools}}" while retaining config, reason, and any model-visible prefix. A child session whose own scope composes a different request declares it per fixture index with pinsChildToolSchemas and pinsChildSystemPrompts. A scenario that changes the request header mid-run declares expectedHeaderChanges.

Platform and composition variants

A scenario requiring a non-Windows host declares posixOnly, which skips its run test on Windows while the fixture guards keep covering its committed files everywhere; a scenario whose composition needs a usable pwsh declares pwshOnly. workspaceParent moves the generated child cwd outside the platform temp directory when temporary-directory grants are themselves under test; a scenario's committed workspace/ is copied into that child first, then prepareWorkspace runs against the generated cwd before the agent starts. Default generated workspaces are stored in session fixtures as {{cwd}}, so platform temp roots and random basenames do not affect recordings.

What can go wrong

• A fixture guard rejects the committed files — orphan scenario dirs, missing files, multiple pins for one header class, duplicate sidecar content, unscrubbed JSONL headers, and malformed pinning headers all fail the suite before comparisons run.
• The session harvest needs raw JSONL mode — snapshot configs set the JSONL backend's compression: 'none'; compressed JSONL and SQLite compositions have no snapshot-harvest path.
• Built mode needs current artifacts — run pnpm run build before selecting DSH_EXAMPLE_MODE=lib; source mode remains the zero-build path.

---

Understand the implementation

Implementation internals — click to expand

This section explains the design of the kit; the observable behavior is fully covered in Use this package.

Design

The shared core owns manifests, workspace setup/comparison, typed identity mapping, normalizers, and fixture invariants. The ACP adapter adds four composable layers: launcher, scenario harness, normalizers, and suite factory. launchAcpTestAgent boots a source profile under tsx or a built lib profile under plain Node, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, fails closed on unhandled permission requests, and owns shutdown. runScenario drives ACP JSON-RPC stdio and harvests every persisted raw JSONL session log. The pure normalizers replace cwd paths and typed identities with stable tokens, zero times, expand physical provenance ranges, and scrub request-header bulk. defineAcpSnapshotSuite registers comparisons, fixture write-back, and the live uniformity guard.

Source map

File	Role
src/launcher.ts	Subprocess/client launcher and shutdown ownership
src/harness.ts	Scripted scenario driver and session-log harvest
src/manifest.ts	Closed snapshot.yml schema, collection, and ownership rules
src/identity.ts	Typed first-seen identity tokenization across parent and child logs
src/normalize.ts	Pure normalizers and scrubbing helpers
src/workspace.ts	Scenario workspace setup and complete expected-state comparison
src/suite.ts	Scenario-table suite factory, fixture guards, record/refresh write-back
src/index.ts	Package entry re-exporting the four layers
src/invariant.ts	Invariant companion (no runtime invariant; consuming test suites exercise the kit)

Data flow

A scenario runs the agent under the launcher, feeds it the input script through the harness, and captures stdout plus the persisted logs. The normalizers canonicalize those captures — ids to first-seen sequence, generated cwd to {{cwd}}, header bulk to {{system}}/{{tools}} — so recorded and fresh runs compare structurally. The factory then compares normalized stdout and re-persisted logs against committed fixtures, or writes them back in record/refresh mode, and its guards reject malformed or drifting fixtures before any comparison result is trusted.

---

Further Exploration

Read these pages when the package-level contract is not enough. They move from the snapshot kit to the model fixture source, the launch mechanics, and the policy that requires the tier.

• llm-replay — the keyless model fixture source replay mode consumes.
• loader-smoke — the mode-aware subprocess launch mechanics the launcher builds on.
• Testing policy — the keyless snapshot tier, when it is required, and the fixture ownership rules.
• Test-support group map — sibling harnesses and support packages.

---

Model Experience

None, as this test-only support records, normalizes, and compares profile sessions without changing the agent's assembled model request.

KV Cache effect

None; this package neither assembles nor sends a provider request.

Known Limitations and Deferred Work

These limits define when the kit needs special care. They are current package constraints, not a task backlog.

• Session harvest requires raw JSONL mode — runScenario collects persisted .jsonl logs, so snapshot configs set the JSONL backend's compression: 'none'; compressed JSONL and SQLite compositions have no snapshot-harvest path.
• Built mode requires current artifacts — run pnpm run build before selecting DSH_EXAMPLE_MODE=lib; source mode remains the zero-build path.
• ACP remains for protocol behavior — cancellation and permission round trips whose stimulus is the ACP client stay on that adapter; assembled one-shot and persistent-control behavior uses headless and SDK adapters.

Dev Note

Working context for maintainers — click to expand

None.