Merge remote-tracking branch 'origin/master' into simpl-a1-drop-image

# Conflicts:
#	docs/architecture.md
#	packages/compact/compact-basic/README.md

packages/compact/compact-basic/README.md

@@ -1,6 +1,6 @@
 # @deepseek-ai/dsh-compact-basic
 
-The **basic compaction backend**: a `BasicCompactService` implementing the `@deepseek-ai/dsh-compact` seam with a char/4 token heuristic, token-budget retention, and summarization routed through the agent request pipeline.
+The **basic compaction backend**: a `BasicCompactService` implementing the `@deepseek-ai/dsh-compact` seam with a chars-per-token heuristic (the `charsPerToken` config, default 4), token-budget retention, and summarization routed through the agent request pipeline.
 
 This is the implementation tier of the compaction capability — see the [interface package](../compact/README.md) for the seam and the [capability-seam RFC](../../../docs/rfc/implemented/feature/2026-06-18-compaction-capability-seam.md) for the design.
 
@@ -8,7 +8,7 @@ This is the implementation tier of the compaction capability — see the [interf
 
 The abstract contract states only WHAT compaction does; this backend owns every HOW decision:
 
-- **Token estimation** — `estimateContentTokens()`: char/4 with per-block structural overhead (`text`/`reasoning` = `ceil(len/4) + 4`, `tool-call` from name + arguments, `tool-result` recursive, unknown blocks via JSON length).
+- **Token estimation** — `estimateContentTokens()`: chars divided by the `charsPerToken` config (default 4) with per-block structural overhead (`text`/`reasoning` = `ceil(len/charsPerToken) + 4`, `tool-call` from name + arguments, `tool-result` recursive, unknown blocks via JSON length).
 - **Retention policy** — `compactIfNeeded()` walks the surface nodes tail→head summing per-node token estimates, and retains the smallest tail-run of WHOLE units (a closed step, or a single no-step node such as a pre-step `user/message` or inter-step `steering/message`) whose total reaches `retainTokens`; everything older is compacted. Retention is **turn-agnostic** — turn boundaries play no role, so a single runaway turn that alone exceeds the window compacts its OWN early closed steps rather than being retained verbatim (the failure mode that motivated dropping turn-protection: a tool-heavy turn must stay compactable or the harness dies exactly when compaction is needed). The only structural guard is **tool-pairing balance**: the compacted region's edges are balanced cuts on the surface (no unanswered tool-call crosses either edge), so it never splits a step's `assistant/message` tool-calls from their `tool/result`s. When the only compactable content left is an un-splittable open tail step, it declines (returns `null`) and retries once an older step closes. **Single-unit overflow is out of scope, by design**: if one retained unit (a single closed step, or a large pasted `user/message`) ALONE exceeds the budget, compaction cannot help and the call may go out over-budget — bounding an individual unit's size is a separate concern. `compactRegion()` enforces tool-pairing balance strictly, throwing on a boundary that would split a step. `dsh-session` exports `isToolPairingBalanced` for the check.
 - **Dynamic convergence** — no static summary-length config pretends to bound what the model will write. If framing/estimator/system overhead leaves the compacted surface above threshold, `compactIfNeeded()` re-compacts the head checkpoint up to `compactionRetries` extra times; if it still cannot get below threshold, it throws. A summary whose estimated stored size is not smaller than the shadowed content fails closed before it mutates the surface.
 - **Summarization** — `summarize()`: a `GenerateOptions` request assembled via `BlockAssembler` with a fixed system prompt that asks for a structured checkpoint (Primary Request and Intent · Key Technical Concepts · Files and Code · Errors and Fixes · Pending Tasks · Current Work · Next Step · Critical Context), every section mandatory, exact paths/commands/identifiers preserved. The request runs through the `agent/request` waterfall before `ctx.llm.stream()`, so router agents that choose the concrete model there also route compaction summaries. `maxTokens` is the provider-side generation cap; only text blocks from the model's reply are kept before the checkpoint is stored (reasoning is dropped so private chain-of-thought never leaks into the durable summary, and a stray `tool-call` is dropped so the synthesized `user/message` summary cannot land an orphaned call with no matching `tool-result`). The compacted region is flattened to a plain-text transcript first: text and reasoning contribute their text, and every non-text block (tool-call, tool-result, plugin-added types) contributes a type-tagged placeholder (`[tool-call: name(args)]`, `[tool-result: …]`, …) so the summarizer is told what existed rather than silently dropping it.
@@ -32,6 +32,7 @@ Every knob is **required** except `auto` — there is no concrete data yet to ju
 | `maxTokens` | yes | Provider generation cap for the summarization call; may include reasoning tokens. |
 | `compactionRetries` | yes | Extra compaction attempts after the first if the compacted surface remains over threshold. |
 | `auto` | no (default `true`) | Register the `agent/pre-step` auto-compaction listener. Set `false` for manual-only. |
+| `charsPerToken` | no (default `4`) | Token-estimator text density (estimated tokens = chars / `charsPerToken`; may be fractional). The default suits English text; CJK-heavy deployments should set ~1-2 or the estimate undershoots several-fold and compaction fires too late. |
 
 ## Usage
 

packages/compact/compact-basic/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@deepseek-ai/dsh-compact-basic",
-  "description": "Basic compaction backend (char/4 token estimation + token-budget retention + llm.generate() summarization) for the DeepSeek Harness",
+  "description": "Basic compaction backend (chars-per-token estimation + token-budget retention + llm.generate() summarization) for the DeepSeek Harness",
   "version": "0.0.1",
   "private": true,
   "type": "module",

packages/compact/compact-basic/src/index.ts

@@ -2,7 +2,8 @@
  * `BasicCompactService`: the first implementation of the
  * `@deepseek-ai/dsh-compact` seam. It owns the entire compaction strategy:
  *
- * - **Token estimation** — char/4 heuristic with per-block structural overhead.
+ * - **Token estimation** — chars/`charsPerToken` heuristic (config, default 4)
+ *   with per-block structural overhead.
  * - **Retention policy** — walk surface nodes tail→head, keep recent nodes up
  *   to a token budget, compact everything older. The cutoff is snapped forward
  *   to the next balanced tool-pairing boundary so a compacted region never
@@ -145,9 +146,11 @@ function finishError(finish: FinishReason): Error | undefined {
 }
 
 /**
- * Basic, dependency-light compaction backend. Defaults target a 128K context
- * window, compacting at 80% utilization and retaining ~20K tokens of recent
- * context.
+ * Basic, dependency-light compaction backend: estimates the surface's token
+ * footprint, summarizes the stale prefix through the model, and shadows it
+ * behind a durable checkpoint. Every threshold/budget knob is required config
+ * ({@link BasicCompactConfig}); the estimator's text density is the
+ * `charsPerToken` knob.
  */
 export class BasicCompactService extends CompactService {
   static inject = ['llm']
@@ -204,24 +207,27 @@ export class BasicCompactService extends CompactService {
 
   // ---- Token estimation (overridable hooks) ----
 
-  // TODO: char/4 is a coarse heuristic. Replace with an exact count — a real
-  // tokenizer, or the provider's post-response `usage` (input tokens) fed back
-  // as a correction — so threshold decisions match the model's actual budget.
+  // TODO: chars/charsPerToken is a coarse heuristic. Replace with an exact
+  // count — a real tokenizer, or the provider's post-response `usage` (input
+  // tokens) fed back as a correction — so threshold decisions match the
+  // model's actual budget.
   /**
-   * Estimate the token count of content blocks — char/4 with per-block
-   * overhead. Override in a subclass to plug in a real tokenizer.
+   * Estimate the token count of content blocks — chars divided by the
+   * `charsPerToken` config, with per-block overhead. Override in a subclass to
+   * plug in a real tokenizer.
    */
   estimateContentTokens(blocks: readonly ContentBlock[]): number {
+    const { charsPerToken } = this.config
     let tokens = 0
     for (const block of blocks) {
       switch (block.type) {
         case 'text':
         case 'reasoning':
-          tokens += Math.ceil(block.text.length / 4) + BLOCK_OVERHEAD
+          tokens += Math.ceil(block.text.length / charsPerToken) + BLOCK_OVERHEAD
           break
         case 'tool-call':
-          tokens += Math.ceil(block.name.length / 4)
-            + Math.ceil(block.arguments.length / 4)
+          tokens += Math.ceil(block.name.length / charsPerToken)
+            + Math.ceil(block.arguments.length / charsPerToken)
             + BLOCK_OVERHEAD
           break
         case 'tool-result':
@@ -230,7 +236,7 @@ export class BasicCompactService extends CompactService {
         default:
           // Unknown block types (merge-extensible ContentBlockMap):
           // estimate conservatively via JSON stringify.
-          tokens += BLOCK_OVERHEAD + Math.ceil(JSON.stringify(block).length / 4)
+          tokens += BLOCK_OVERHEAD + Math.ceil(JSON.stringify(block).length / charsPerToken)
       }
     }
     return tokens
@@ -260,7 +266,7 @@ export class BasicCompactService extends CompactService {
       total += this.estimateContentTokens(msg.content)
       total += ROLE_OVERHEAD
     }
-    if (systemPrompt) total += Math.ceil(systemPrompt.length / 4)
+    if (systemPrompt) total += Math.ceil(systemPrompt.length / this.config.charsPerToken)
     return total
   }
 

packages/compact/compact-basic/src/types.ts

@@ -10,10 +10,12 @@
  */
 
 /**
- * Backend configuration. Every knob is REQUIRED except `auto`: there is no
- * concrete data yet to justify default thresholds/budgets, so a consumer must
- * state each value explicitly rather than inherit a guessed default. `auto`
- * alone defaults to `true` (auto-compaction is the intended posture).
+ * Backend configuration. Every knob is REQUIRED except `auto` and
+ * `charsPerToken`: there is no concrete data yet to justify default
+ * thresholds/budgets, so a consumer must state each value explicitly rather
+ * than inherit a guessed default. `auto` alone defaults to `true`
+ * (auto-compaction is the intended posture), and `charsPerToken` defaults to
+ * the English-text heuristic its estimator was calibrated on.
  */
 export interface BasicCompactConfig {
   /** Context window size in tokens. */
@@ -30,13 +32,21 @@ export interface BasicCompactConfig {
   compactionRetries: number
   /** Enable automatic compaction on the `agent/pre-step` seam (default true). */
   auto?: boolean
+  /**
+   * Text density for the token estimator: estimated tokens = chars /
+   * `charsPerToken`. Defaults to 4 (typical English text). A CJK-heavy
+   * deployment should set ~1-2 — CJK runs at roughly 1-2 chars per token, so
+   * the default UNDERestimates several-fold and compaction fires far too late.
+   * May be fractional.
+   */
+  charsPerToken?: number
 }
 
-/** Resolved config with `auto` defaulted. */
+/** Resolved config with `auto` and `charsPerToken` defaulted. */
 export type ResolvedConfig = Required<BasicCompactConfig>
 
 /**
- * Default `auto` when unset and reject nonsensical numeric knobs.
+ * Default `auto`/`charsPerToken` when unset and reject nonsensical numeric knobs.
  *
  * Convergence is not a static config invariant: provider generation caps can be
  * spent on hidden or surfaced reasoning tokens, and the model may emit a summary
@@ -46,13 +56,14 @@ export type ResolvedConfig = Required<BasicCompactConfig>
  * throwing if the surface still exceeds the threshold.
  */
 export function resolveConfig(config: BasicCompactConfig): ResolvedConfig {
-  const resolved: ResolvedConfig = { auto: true, ...config }
+  const resolved: ResolvedConfig = { auto: true, charsPerToken: 4, ...config }
 
   assertPositiveInteger('contextWindow', resolved.contextWindow)
   assertRatio('thresholdRatio', resolved.thresholdRatio)
   assertNonNegativeInteger('retainTokens', resolved.retainTokens)
   assertPositiveInteger('maxTokens', resolved.maxTokens)
   assertNonNegativeInteger('compactionRetries', resolved.compactionRetries)
+  assertPositiveFinite('charsPerToken', resolved.charsPerToken)
   if (typeof resolved.summarizationModel !== 'string') {
     throw new Error('BasicCompactConfig: summarizationModel must be a string.')
   }
@@ -74,6 +85,12 @@ function assertNonNegativeInteger(name: string, value: number): void {
   }
 }
 
+function assertPositiveFinite(name: string, value: number): void {
+  if (typeof value !== 'number' || !Number.isFinite(value) || value <= 0) {
+    throw new Error(`BasicCompactConfig: ${name} (${value}) must be a positive finite number.`)
+  }
+}
+
 function assertRatio(name: string, value: number): void {
   if (typeof value !== 'number' || !Number.isFinite(value) || value <= 0 || value > 1) {
     throw new Error(`BasicCompactConfig: ${name} (${value}) must be a number in (0, 1].`)

packages/compact/compact-basic/tests/compact-basic.spec.ts

@@ -815,6 +815,19 @@ describe('BasicCompactService token estimation (char/4 heuristic)', () => {
     const svc = new BasicCompactService(new Context(), cfg({ auto: false }))
     expect(svc.estimateContentTokens([])).toBe(0)
   })
+
+  it('honors a configured charsPerToken (fractional densities included)', () => {
+    // 'this is a somewhat longer text block' = 36 chars.
+    const blocks: ContentBlock[] = [{ type: 'text', text: 'this is a somewhat longer text block' }]
+    // charsPerToken 2: ceil(36/2)+4 = 22 — a CJK-density config doubles the estimate.
+    const dense = new BasicCompactService(new Context(), cfg({ auto: false, charsPerToken: 2 }))
+    expect(dense.estimateContentTokens(blocks)).toBe(22)
+    // Fractional density is legal: ceil(36/1.5)+4 = 28.
+    const fractional = new BasicCompactService(new Context(), cfg({ auto: false, charsPerToken: 1.5 }))
+    expect(fractional.estimateContentTokens(blocks)).toBe(28)
+    // The system-prompt term scales with the same knob: 36-char prompt at density 2 → ceil(36/2) = 18.
+    expect(dense.estimateTokens([], 'this is a somewhat longer text block')).toBe(18)
+  })
 })
 
 describe('BasicCompactService HMR safety', () => {
@@ -857,6 +870,10 @@ describe('BasicCompactService config validation', () => {
     )).toThrow(/summarizationModel must be a string/)
     expect(() => new BasicCompactService(new Context(), cfg({ auto: 'no' } as unknown as Partial<BasicCompactConfig>)))
       .toThrow(/auto must be a boolean/)
+    expect(() => new BasicCompactService(new Context(), cfg({ auto: false, charsPerToken: 0 })))
+      .toThrow(/charsPerToken .* positive finite number/)
+    expect(() => new BasicCompactService(new Context(), cfg({ auto: false, charsPerToken: Number.NaN })))
+      .toThrow(/charsPerToken .* positive finite number/)
   })
 
   it('accepts a large retain budget because convergence is enforced dynamically', () => {