diff --git a/.agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.i18n.yaml b/.agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.i18n.yaml new file mode 100644 index 0000000000..8f4d10d9e8 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.md +2026-08-27-steer-followup-image-delivery.md: fb93612254381b589c8adfe44e46c51dfe232c3e +2026-08-27-steer-followup-image-delivery.zh.md: 35284aba89a7df5da5d472f078d2c2d3ea2e307d diff --git a/.agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.md b/.agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.md new file mode 100644 index 0000000000..fb93612254 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.md @@ -0,0 +1,41 @@ +# Agent Note: Steer and follow-up image delivery + +Status: implemented + +English | [中文](2026-08-27-steer-followup-image-delivery.zh.md) + +## Problem + +Images submitted while an agent is running did not reliably reach the model context (#3186), for three independent reasons. + +First, a steer or follow-up spliced into a live driver latched no wake: the live driver was expected to claim it, but a turn that finished or failed between the splice and the claim exited without re-checking, stranding the accepted message until an unrelated waking send. Image admission widens this window because the Host awaits attachment normalization before `agent.steer()`/`agent.followup()` runs. + +Second, continuable-subagent follow-ups rejected images in the Client (`SUBAGENT_IMAGE_UNSUPPORTED`) before any RPC, and stripped image parts from the text-only call. The Host route had no admission at all, and its wire content was `ContentBlock[]`, so lifting the Client rejection alone would have let a browser cite any `attachmentId` it never uploaded. + +Third, the browser queue projection reduced a queued image to the text `[image]` even though the durable reference was already present and readable through the session attachment authorization. + +## Decision + +**Closing-turn wake delivery.** `ReactLoopAgent` tracks the identities of waking sends still awaiting a claim (`pendingWakes`); claim and discard notifications prune the set. At a driver exit whose turn loop returned without throwing, a non-empty set re-wakes the driver, so a steer or follow-up that lost the race with a normally closing turn is claimed by a fresh turn. Cancellation and `agent/pre-step` rejection instead clear the set: accepted-but-unclaimed input parks until the next waking send, preserving the tested `cancel({ keepInbox: true })` semantics and keeping rejected claims from being re-offered to the rejecting policy. Injected context never enters the set. The turn-flow section of [docs/architecture.md](../../../../docs/architecture.md) records the delivery/parking rule. + +**Host-side subagent image admission.** `SubagentPromptRequest.content` is now upload-shaped `PromptContentPart[]`, whose single home moved from `dsh-api-session-controller` to `dsh-attachment`; the shared `durablePromptContent()` conversion lives in `dsh-llm/content` and is used by both the Session prompt endpoint and `SubagentRuntime.prompt`. The subagent route admits and persists image batches through `ctx.attachments` before `followup()`, and the continuation manager refuses delivery inside the per-child lock when the child's `agent.options` route resolves to a model without image input (`MODEL_DOES_NOT_SUPPORT_IMAGES`, surfaced as `attachment-error` with the same reason vocabulary as the Session route). A child without a fixed options route, or a deployment without the LLM registry, delivers and relies on the LLM layer's text-only projection. The Client forwards image parts unchanged and the `SUBAGENT_IMAGE_UNSUPPORTED` copy is gone. + +**Queue presentation.** The queue mirror's text preview excludes image blocks, and the queue dock renders each durable image part as a thumbnail resolved through `ctx.uiConversation.imageUrl` — the same session-authorized read the transcript uses. Editing queued image messages stays refused (#3072). + +## Alternatives considered + +**Re-wake on every driver exit with pending input.** Rejected: it breaks the deliberate parking semantics of `cancel({ keepInbox: true })` and pre-step rejection, and a pre-commit `turn/start` failure would re-enter a hot loop because the failing turn never claims the message. + +**Latch `wakeRequested` for sends to a live driver.** Rejected: the latch is not pruned on claim, so a claimed steer plus leftover injected context would open a context-only turn at exit, violating the rule that injected context waits for a waking message. + +**Keep the wire content `ContentBlock[]` and admit refs on the Host.** Rejected: a reference-shaped wire lets a Client fabricate `attachmentId` citations; an upload-shaped wire makes Host admission the only way an attachment reference can exist in a child message. + +**Check child image capability in `SubagentRuntime.prompt`.** Rejected: the route may address a cold child whose agent does not exist yet; the continuation manager sees the live or freshly materialized agent in both arms and inside the per-child delivery lock, so the check cannot race a concurrent delivery. + +## Testing + +Agent-loop tests pin the closing-turn window deterministically (a `turn/end` listener queues the send as a microtask ahead of the driver's exit continuation) for steer, follow-up, and the inject non-delivery case. Host tests cover `mode: 'steer'` image admission; subagent control tests cover ordered admission, batch refusal, non-canonical base64, and the capability refusal mapping; continuation tests cover refusal without a partial message, capable delivery, and the routeless deferral. Client tests cover unstripped forwarding, queue thumbnails (load, failure placeholder, unmount), and the image-free preview. + +## Consequences + +A steer or follow-up accepted during a turn's final microtasks is now delivered by a fresh turn instead of hanging in the inbox, while user cancellation still parks pending work — delivery after a stop remains an explicit next waking send. The subagent package now depends on `dsh-attachment` and reads `ctx.llm` optionally. Images persisted by a batch whose delivery is later refused stay as unreachable content-addressed objects under the existing retention rules. Queue thumbnails add one authorized attachment read per queued image, shared with the transcript cache. diff --git a/.agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.zh.md b/.agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.zh.md new file mode 100644 index 0000000000..35284aba89 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-08-27-steer-followup-image-delivery.zh.md @@ -0,0 +1,41 @@ +# Agent Note: steer 与 follow-up 的图片投递 + +Status: implemented + +[English](2026-08-27-steer-followup-image-delivery.md) | 中文 + +## Problem + +agent 运行期间提交的图片没有可靠进入模型上下文（#3186），原因有三个，彼此独立。 + +第一，splice 进在线 driver 的 steer 或 follow-up 不会锁存唤醒：预期由在线 driver 自行认领，但轮次在 splice 与认领之间正常结束或失败时，退出路径不再复查，已接受的消息就滞留到下一次无关的唤醒发送。图片准入放大了这个窗口，因为 Host 在执行 `agent.steer()`/`agent.followup()` 之前要先等待附件规范化完成。 + +第二，可继续子代理的 follow-up 在客户端就拒绝图片（`SUBAGENT_IMAGE_UNSUPPORTED`），并把图片部分从纯文本调用中剥掉。Host 路由完全没有准入，wire 内容又是 `ContentBlock[]`，单独放开客户端拒绝会允许浏览器引用任何它从未上传过的 `attachmentId`。 + +第三，浏览器队列投影把已排队的图片折叠成文本 `[image]`，尽管持久化引用已经存在，并且可以通过会话附件授权读取。 + +## Decision + +**轮次收尾期的唤醒投递。** `ReactLoopAgent` 用 `pendingWakes` 记录尚未被认领的唤醒发送的身份；认领与丢弃通知会移除对应条目。当 driver 的轮次循环无异常返回并退出时，集合非空就重新拉起 driver，输掉与正常收尾轮次竞态的 steer 或 follow-up 由新轮次认领。取消与 `agent/pre-step` 拒绝则清空该集合：已接受但未认领的输入停放到下一次唤醒发送，既保留了有测试保护的 `cancel({ keepInbox: true })` 语义，也避免把被拒绝的认领重新塞给同一个拒绝策略。注入的上下文从不进入该集合。投递与停放规则记录在 [docs/architecture.md](../../../../docs/architecture.zh.md) 的 turn-flow 一节。 + +**Host 侧子代理图片准入。** `SubagentPromptRequest.content` 改为上传形态的 `PromptContentPart[]`，该类型的唯一定义处从 `dsh-api-session-controller` 移到 `dsh-attachment`；共享的 `durablePromptContent()` 转换位于 `dsh-llm/content`，Session prompt 端点与 `SubagentRuntime.prompt` 共用。子代理路由在 `followup()` 之前经 `ctx.attachments` 完成整批图片的准入与持久化；continuation 管理器在逐子级锁内，当子级 `agent.options` 路由解析到不接受图片输入的模型时拒绝投递（`MODEL_DOES_NOT_SUPPORT_IMAGES`，以与 Session 路由一致的 `attachment-error` 词汇表上抛）。子级没有固定 options 路由，或部署未挂载 LLM 注册表时照常投递，交给 LLM 层的纯文本投影。客户端原样转发图片部分，`SUBAGENT_IMAGE_UNSUPPORTED` 文案删除。 + +**队列展示。** 队列镜像的文本预览不再包含图片块，queue dock 把每个持久化图片部分渲染为缩略图，经 `ctx.uiConversation.imageUrl` 解析，与会话记录使用同一个会话授权读取。已排队图片消息的编辑仍然拒绝（#3072）。 + +## Alternatives considered + +**任何 driver 退出都在有滞留输入时重新拉起。** 拒绝：这破坏 `cancel({ keepInbox: true })` 与 pre-step 拒绝的刻意停放语义，而且 `turn/start` 提交前失败的轮次永远不会认领消息，会进入热循环。 + +**对发给在线 driver 的发送也锁存 `wakeRequested`。** 拒绝：锁存不随认领清除，被认领的 steer 加上剩余的注入上下文会在退出时开出一个只有上下文的轮次，违反注入上下文必须等待唤醒消息的规则。 + +**wire 内容保持 `ContentBlock[]`，由 Host 准入引用。** 拒绝：引用形态的 wire 允许客户端伪造 `attachmentId`；上传形态的 wire 使 Host 准入成为子级消息里附件引用的唯一来源。 + +**在 `SubagentRuntime.prompt` 里做子级图片能力检查。** 拒绝：该路由可能寻址冷的子级，其 agent 尚不存在；continuation 管理器在两条分支里都拿得到在线或刚物化的 agent，并且处于逐子级投递锁内，检查不会与并发投递竞态。 + +## Testing + +agent-loop 测试确定性地钉住收尾窗口（`turn/end` 监听器把发送排为微任务，先于 driver 的退出续体执行），覆盖 steer、follow-up 与注入不投递。Host 测试覆盖 `mode: 'steer'` 的图片准入；subagent control 测试覆盖有序准入、整批拒绝、非规范 base64 与能力拒绝映射；continuation 测试覆盖拒绝时不留半条消息、能力通过时投递、无路由时的顺延。客户端测试覆盖不剥离的转发、队列缩略图（加载、失败占位、卸载）与不含图片的预览。 + +## Consequences + +在轮次最后几个微任务里被接受的 steer 或 follow-up 现在由新轮次投递，而不是挂在 inbox 里；用户取消仍然停放待处理工作，停止之后的投递依旧需要一次显式的唤醒发送。subagent 包新增对 `dsh-attachment` 的依赖，并可选读取 `ctx.llm`。整批持久化后投递被拒绝的图片按现有保留规则保持为不可达的内容寻址对象。队列缩略图对每张排队图片增加一次授权附件读取，与会话记录缓存共享。 diff --git a/apps/web/tests/queue-image.e2e.ts b/apps/web/tests/queue-image.e2e.ts new file mode 100644 index 0000000000..e5a4794d82 --- /dev/null +++ b/apps/web/tests/queue-image.e2e.ts @@ -0,0 +1,147 @@ +// Keyless browser coverage for image attachments submitted while a turn is +// running, through the shipped Web composition and real HTTP/SSE wire. A +// text-plus-image submission queues as one occurrence whose dock row renders +// the durable thumbnail, survives a stop as parked work, and delivers as the +// next turn's user message with its image intact — while the session log holds +// only durable attachment references, never base64. +import { existsSync } from 'node:fs' +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { fileURLToPath } from 'node:url' +import { join } from 'node:path' +import type { Browser, Page } from 'playwright' +import { chromium } from 'playwright' +import { afterEach, describe, expect, it, onTestFailed } from 'vitest' +import { deriveReplayScript, parseSessionLog, type ReplayEntry } from '@deepseek-ai/dsh-llm-replay' +import type { SessionEvent } from '@deepseek-ai/dsh-session' +import { + captureStableAria, compareOrRefreshGolden, + launchWebScaffold, watchConsole, webSnapshotMode, type WebScaffold, +} from './scaffold.ts' +import { connectFreshWorkspace, newEnglishPage, saveFailureShot } from './support.ts' + +const SNAPSHOT_DIR = fileURLToPath(new URL('../../../snapshots/web/queued-image', import.meta.url)) +const FIXTURE = fileURLToPath(new URL('../../../snapshots/web/live-interactions/session.jsonl', import.meta.url)) +const PNG = fileURLToPath(new URL('../../../snapshots/session/read-image/workspace/red.png', import.meta.url)) +const QUEUED_EXPECTED = join(SNAPSHOT_DIR, 'queued.expected.md') +const DELIVERED_EXPECTED = join(SNAPSHOT_DIR, 'delivered.expected.md') +const MODE = webSnapshotMode() + +const ACTIVE_PROMPT = 'Reply with a one-sentence description of event sourcing, then stop.' +const QUEUED_TEXT = 'Compare with this screenshot' + +/** Paste one real PNG into the composer through a genuine clipboard event. */ +async function pasteImage(page: Page, bytes: Uint8Array): Promise { + await page.locator('[data-composer-input]').first().evaluate((surface, data) => { + const transfer = new DataTransfer() + transfer.items.add(new File([new Uint8Array(data)], 'queued.png', { type: 'image/png' })) + surface.dispatchEvent(new ClipboardEvent('paste', { + clipboardData: transfer, bubbles: true, cancelable: true, + })) + }, [...bytes]) +} + +describe('web e2e: queued image submission', () => { + let scaffold: WebScaffold | undefined + let browser: Browser | undefined + let page: Page + let overrideDir: string | undefined + + afterEach(async () => { + const failures: unknown[] = [] + await browser?.close().catch((error: unknown) => failures.push(error)) + browser = undefined + const closing = scaffold + scaffold = undefined + await closing?.close().catch((error: unknown) => failures.push(error)) + if (overrideDir !== undefined) { + await rm(overrideDir, { recursive: true, force: true }) + .catch((error: unknown) => failures.push(error)) + } + overrideDir = undefined + if (failures.length === 1) throw failures[0] + if (failures.length > 1) throw new AggregateError(failures, 'queued-image teardown failed') + }) + + it.skipIf(MODE === 'record')('queues a text-plus-image submission with a thumbnail and delivers it as the next turn', async () => { + overrideDir = await mkdtemp(join(tmpdir(), 'dsh-web-queued-image-')) + const readyFile = join(overrideDir, '.hang-ready') + const overridePath = join(overrideDir, 'replay.override.json') + const recorded = deriveReplayScript(parseSessionLog(await readFile(FIXTURE, 'utf8'))) + expect(recorded).toHaveLength(1) + const replay: ReplayEntry[] = [ + { kind: 'hang', readyFile }, + recorded[0]!, + recorded[0]!, + ] + await writeFile(overridePath, JSON.stringify(replay)) + + const sessionEvents: SessionEvent[] = [] + scaffold = await launchWebScaffold({ replayFixture: FIXTURE, replayOverride: overridePath, compareReplaySession: false }) + scaffold.ctx.on('session/event', (_session, event: SessionEvent) => { sessionEvents.push(event) }) + browser = await chromium.launch() + page = await newEnglishPage(browser) + const tripwire = watchConsole(page) + await page.goto(scaffold.authenticatedUrl, { waitUntil: 'load' }) + await page.waitForSelector('[class*="frame"]', { timeout: 30_000 }) + await connectFreshWorkspace(page, scaffold.workspaceCwd) + onTestFailed(() => saveFailureShot(page, 'web-e2e-queued-image')) + + const input = page.locator('[data-composer-input]').first() + const firstSettled = scaffold.whenTurnSettled() + await input.fill(ACTIVE_PROMPT) + await input.press('Enter') + await expect.poll(() => existsSync(readyFile), { timeout: 15_000 }).toBe(true) + + // A just-submitted composer is read-only for the prompt round-trip. + await page.locator('[data-composer-input][contenteditable="true"]').first().waitFor({ timeout: 10_000 }) + await pasteImage(page, await readFile(PNG)) + await page.getByRole('img', { name: 'queued.png' }).waitFor({ timeout: 10_000 }) + await input.fill(QUEUED_TEXT) + await input.press('Enter') + + // The queued row renders the durable thumbnail beside the text preview. + const dockThumb = page.locator('[data-queue-dock] img[alt="Queued message image"]') + await dockThumb.waitFor({ timeout: 15_000 }) + await expect.poll(() => dockThumb.getAttribute('src')).toMatch(/^blob:/) + await page.getByText(QUEUED_TEXT, { exact: true }).waitFor() + const queuedSnapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd) + await compareOrRefreshGolden(QUEUED_EXPECTED, queuedSnapshot, MODE) + + // Stop parks the accepted queue; the next waking send delivers the image + // message first (FIFO), then its own text as the following turn. + await page.getByRole('button', { name: 'Stop generating' }).click() + await firstSettled + await expect.poll(() => page.getByRole('button', { name: 'Stop generating' }).count()).toBe(0) + await dockThumb.waitFor({ timeout: 10_000 }) + + const settled = scaffold.whenTurnSettled() + await input.fill('Continue with the queued comparison') + await input.press('Enter') + await settled + + // The delivered user message renders its image in Chat from the durable + // reference, and the dock row is gone. + await expect.poll( + () => page.locator('[data-queue-dock]').count(), + { timeout: 15_000 }, + ).toBe(0) + const chatImage = page.locator('[class*="userRow"] img') + await chatImage.first().waitFor({ timeout: 15_000 }) + const deliveredSnapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd) + await compareOrRefreshGolden(DELIVERED_EXPECTED, deliveredSnapshot, MODE) + + // Model-visible means logged: the delivered message carries the durable + // reference (never base64), in the composer's canonical images-then-text order. + const delivered = sessionEvents.find(event => event.type === 'user/message' + && event.data.content.some(block => block.type === 'image')) + expect(delivered?.type === 'user/message' && delivered.data.content.map(block => block.type)).toEqual(['image', 'text']) + const imageBlock = delivered?.type === 'user/message' + ? delivered.data.content.find(block => block.type === 'image') + : undefined + expect(imageBlock?.type === 'image' && imageBlock.attachment.name).toBe('queued.png') + expect(JSON.stringify(sessionEvents)).not.toContain('base64') + expect(tripwire.pageErrors).toEqual([]) + expect(tripwire.warnings).toEqual([]) + }, 120_000) +}) diff --git a/apps/web/tsconfig.json b/apps/web/tsconfig.json index f261db5614..03f5a08a75 100644 --- a/apps/web/tsconfig.json +++ b/apps/web/tsconfig.json @@ -73,6 +73,7 @@ "tests/markdown-cjk-strong.e2e.ts", "tests/markdown-inline-code-links.e2e.ts", "tests/queue-actions.e2e.ts", + "tests/queue-image.e2e.ts", "tests/skill-invocation-policy.e2e.ts", "tests/skill-user-invoke.e2e.ts", "tests/permission-policy-context.e2e.ts", diff --git a/docs/architecture.i18n.yaml b/docs/architecture.i18n.yaml index 5169a54f8b..9c98358b90 100644 --- a/docs/architecture.i18n.yaml +++ b/docs/architecture.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/architecture.md -architecture.md: 20d03c079fa8e1f73f733992b6e938f0f539a60f -architecture.zh.md: 5448036ad6e11902e32f89d0a65be99230e27e02 +architecture.md: 5717bf62b0967ff860dee9db5265d8dd56242a1d +architecture.zh.md: 279d8c4cc040bc18f7adfd1ae1b782447c3333d6 diff --git a/docs/architecture.md b/docs/architecture.md index 20d03c079f..5717bf62b0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -94,7 +94,7 @@ turn/end `turn/*`, `step/*`, `user/message`, `assistant/*`, and `tool/*` are durable session events; the rest are live extension points across three domains. `agent/pre-step`, `agent/request`, `llm/stream`, and the three `tools/*` events are waterfalls, whose listeners must call `next()` to delegate; `agent/turn-stopping` is serial and has no `next()`. -Input reaches the driver through one inbox. Some messages wake it immediately; injected context waits in the inbox until another message does. +Input reaches the driver through one inbox. Some messages wake it immediately; injected context waits in the inbox until another message does. A waking message that arrives while a turn is closing is claimed by a fresh turn at the driver's clean exit, while cancellation and step rejection park accepted-but-unclaimed input until the next waking send. `agent/pre-step` decides what the model sees. Listeners may rewrite the claimed messages or reject them outright; a rejected or empty first claim still closes a durable turn that spent no step, so the log records the attempt. An enter decision may also set `startsRequestSeries` to begin a distinct model-message series: the loop then logs a fresh `request/header` (reason `series`, or `change` carrying `startsSeries: true` when the envelope changed too). A listener that rebuilds a downstream enter decision must spread it (`{ ...decision, messages }`) so the declaration survives. Each step reads the prompt sections and tool schemas that plugins registered. diff --git a/docs/architecture.zh.md b/docs/architecture.zh.md index 5448036ad6..279d8c4cc0 100644 --- a/docs/architecture.zh.md +++ b/docs/architecture.zh.md @@ -98,7 +98,7 @@ turn/end `turn/*`、`step/*`、`user/message`、`assistant/*` 和 `tool/*` 是持久会话事件；其余是分属三个事件域的实时扩展点。`agent/pre-step`、`agent/request`、`llm/stream` 和三个 `tools/*` 事件是 waterfall（瀑布式事件），其监听器必须调用 `next()` 才能委托下去；`agent/turn-stopping` 是 serial 事件，没有 `next()`。 -输入通过同一个 inbox 到达驱动器。有些消息会立即唤醒它；注入的上下文会留在 inbox 中，直到另一条消息将其唤醒。 +输入通过同一个 inbox 到达驱动器。有些消息会立即唤醒它；注入的上下文会留在 inbox 中，直到另一条消息将其唤醒。在轮次收尾期间到达的唤醒消息会在驱动器干净退出时由新轮次认领；取消与步骤拒绝则把已接受但未认领的输入停放到下一次唤醒发送。 `agent/pre-step` 决定模型看到什么。监听器可以改写已领取的消息，也可以直接拒绝它们；首次领取被拒绝或被改写为空时，仍会关闭一个不含步骤的持久轮次，因此日志会记录这次尝试。enter 决策还可以设置 `startsRequestSeries` 来开启独立的模型消息序列：loop 会随之记录一个新的 `request/header`（原因为 `series`，或在封装同时变化时为携带 `startsSeries: true` 的 `change`）。重建下游 enter 决策的监听器必须展开它（`{ ...decision, messages }`），该声明才能存活。每个步骤读取插件注册的提示词片段和工具 schema。 diff --git a/docs/subsystems/subagent.i18n.yaml b/docs/subsystems/subagent.i18n.yaml index f4414983b1..8aa69ce9a2 100644 --- a/docs/subsystems/subagent.i18n.yaml +++ b/docs/subsystems/subagent.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/subsystems/subagent.md -subagent.md: f854711f1161ba1c533fdbc43d7d6c35681f2f7f -subagent.zh.md: 960d9b099d915fcf5b1e321ab74374664bb8e468 +subagent.md: 03d32139157b2f2c2bf9b58ce8f0ea36d2187a66 +subagent.zh.md: 61bb2f32d1eb85a894dfbd40fc259fbdab8924de diff --git a/docs/subsystems/subagent.md b/docs/subsystems/subagent.md index f854711f11..03d3213915 100644 --- a/docs/subsystems/subagent.md +++ b/docs/subsystems/subagent.md @@ -669,13 +669,15 @@ listDescendants(rootSessionId: SessionId, signal?: AbortSignal): Promise diff --git a/docs/subsystems/subagent.zh.md b/docs/subsystems/subagent.zh.md index 960d9b099d..61bb2f32d1 100644 --- a/docs/subsystems/subagent.zh.md +++ b/docs/subsystems/subagent.zh.md @@ -673,13 +673,15 @@ listDescendants(rootSessionId: SessionId, signal?: AbortSignal): Promise diff --git a/packages/api/session-controller/src/client/contract/session.ts b/packages/api/session-controller/src/client/contract/session.ts index 6ac4182bb9..fc97dbff8e 100644 --- a/packages/api/session-controller/src/client/contract/session.ts +++ b/packages/api/session-controller/src/client/contract/session.ts @@ -7,12 +7,12 @@ * must stub); implementation-internal entry points (history staging, wire-frame * dispatch) stay on the class, invisible out here. */ -import type { AttachmentIdType, ImageAttachmentRef } from '@deepseek-ai/dsh-attachment' +import type { AttachmentIdType, ImageAttachmentRef, PromptContentPart } from '@deepseek-ai/dsh-attachment' import type { MessageId } from '@deepseek-ai/dsh-llm/brand' import type { SessionId } from '@deepseek-ai/dsh-session/types' import type { RemoteResult } from '@deepseek-ai/dsh-typert-protocol' import type { ObservableSnapshot } from '@deepseek-ai/dsh-client-store' -import type { PromptContentPart, QueueAction } from '../../types.ts' +import type { QueueAction } from '../../types.ts' import type { ClientResult } from './result.ts' import type { SessionSnapshot } from './snapshot.ts' diff --git a/packages/api/session-controller/src/client/sessions/queue-mirror.ts b/packages/api/session-controller/src/client/sessions/queue-mirror.ts index 209349af2c..8ae4539607 100644 --- a/packages/api/session-controller/src/client/sessions/queue-mirror.ts +++ b/packages/api/session-controller/src/client/sessions/queue-mirror.ts @@ -5,8 +5,11 @@ import type { QueuedMessage } from '../contract/snapshot.ts' const QUEUE_PREVIEW_CHARS = 200 +// Image blocks are excluded: queue presentation renders them as thumbnails +// from `content`, so the text preview covers only what has no visual form. function previewOf(content: readonly ContentBlock[]): string { const flat = content + .filter(block => block.type !== 'image') .map(block => (block.type === 'text' ? block.text : `[${block.type}]`)) .join(' ').replace(/\s+/g, ' ').trim() const chars = Array.from(flat) diff --git a/packages/api/session-controller/src/client/sessions/session.ts b/packages/api/session-controller/src/client/sessions/session.ts index db7cf9a997..4645d7d334 100644 --- a/packages/api/session-controller/src/client/sessions/session.ts +++ b/packages/api/session-controller/src/client/sessions/session.ts @@ -208,28 +208,15 @@ export class Session implements SessionFace { }, } } else { - if (content.some(part => part.type === 'image')) { - result = { - ok: false, - error: { - code: 'attachment-error', - message: 'Image input is unavailable for subagent continuations.', - details: { reason: 'SUBAGENT_IMAGE_UNSUPPORTED' }, - }, - } - } else { - const routed = toSessionResult(await this.remote.subagents.prompt({ - requestId: randomUUID() as SessionRequestId, - parentSessionId: this.address.parentSessionId, - childSessionId: this.address.childSessionId, - mode: this.address.mode, - content: content.flatMap(part => part.type === 'text' - ? [{ type: 'text' as const, text: part.text }] - : []), - clientTimeZone: resolvedClientTimeZone(), - }, signal)) - result = routed.ok ? { ok: true, value: { accepted: true } } : routed - } + const routed = toSessionResult(await this.remote.subagents.prompt({ + requestId: randomUUID() as SessionRequestId, + parentSessionId: this.address.parentSessionId, + childSessionId: this.address.childSessionId, + mode: this.address.mode, + content, + clientTimeZone: resolvedClientTimeZone(), + }, signal)) + result = routed.ok ? { ok: true, value: { accepted: true } } : routed } } catch (error) { result = transportResult(error) diff --git a/packages/api/session-controller/src/commands.ts b/packages/api/session-controller/src/commands.ts index 48c41f3aa1..be2b8304e2 100644 --- a/packages/api/session-controller/src/commands.ts +++ b/packages/api/session-controller/src/commands.ts @@ -4,12 +4,12 @@ import { randomUUID } from 'node:crypto' import type { Context } from '@deepseek-ai/cordis' import type { Agent, ModelSelection as AgentModelSelection } from '@deepseek-ai/dsh-agent' import { PresetMountError, UnknownPresetError } from '@deepseek-ai/dsh-agent-presets' -import { AttachmentError, admitEncodedImages } from '@deepseek-ai/dsh-attachment' +import { AttachmentError } from '@deepseek-ai/dsh-attachment' import type { ImageAttachmentRef } from '@deepseek-ai/dsh-attachment' import { - ReasoningEffortId, createUserMessage, freezeMessage, + ReasoningEffortId, createUserMessage, durablePromptContent, freezeMessage, } from '@deepseek-ai/dsh-llm' -import type { ContentBlock, MessageSource } from '@deepseek-ai/dsh-llm' +import type { MessageSource } from '@deepseek-ai/dsh-llm' import { SessionId } from '@deepseek-ai/dsh-session' import type { SessionEvent, SessionHeader, UserMessage } from '@deepseek-ai/dsh-session' import { SessionQueryError, type SessionObservation } from '@deepseek-ai/dsh-session-query' @@ -319,7 +319,7 @@ export class SessionCommandController { ) } } - const content = await durablePromptContent(this.ctx, request.content) + const content = await durablePromptContent(this.ctx.attachments, request.content) const message: UserMessage = createUserMessage({ content, source }) if (request.mode === 'steer') agent.steer(message) else agent.followup(message) @@ -511,21 +511,6 @@ function reject(code: string, message: string, details: object): never { throw new TypertRemoteFailure({ code, message, details }) } -async function durablePromptContent( - ctx: Context, - content: readonly SessionPromptRequest['content'][number][], -): Promise { - if (content.every(part => part.type === 'text')) { - return content.map(part => ({ type: 'text', text: part.text })) - } - const refs = await admitEncodedImages(ctx.attachments, content.filter(part => part.type === 'image')) - let next = 0 - return content.map(part => part.type === 'text' - ? { type: 'text', text: part.text } - // admitEncodedImages returns one reference per image part in order. - : { type: 'image', attachment: refs[next++] as ImageAttachmentRef }) -} - function imageBlockIn( content: unknown, match: (ref: ImageAttachmentRef) => boolean, diff --git a/packages/api/session-controller/src/types.ts b/packages/api/session-controller/src/types.ts index c045e00707..c58eb419ca 100644 --- a/packages/api/session-controller/src/types.ts +++ b/packages/api/session-controller/src/types.ts @@ -1,7 +1,7 @@ /** Browser-safe request, result, and lifecycle vocabulary for the Session Remote service. */ import type { - AttachmentIdType, ImageAttachmentLimits, ImageAttachmentRef, ImageMediaType, + AttachmentIdType, ImageAttachmentLimits, ImageAttachmentRef, PromptContentPart, } from '@deepseek-ai/dsh-attachment' import type { Branded } from '@deepseek-ai/dsh-brand' import type { MessageId } from '@deepseek-ai/dsh-llm/brand' @@ -67,15 +67,7 @@ export interface SessionProjectionBaseline { export type SessionProjectionValues = Partial & Readonly> -/** Browser-submitted prompt content; the Host promotes image bytes to durable references. */ -export type PromptContentPart = - | { readonly type: 'text'; readonly text: string } - | { - readonly type: 'image' - readonly mediaType: ImageMediaType - readonly data: string - readonly name?: string - } +export type { PromptContentPart } from '@deepseek-ai/dsh-attachment' /** Complete model selection for one Session. */ export interface ModelSelection { diff --git a/packages/api/session-controller/tests/queue-store.client.spec.ts b/packages/api/session-controller/tests/queue-store.client.spec.ts index 6fc045cf70..856a861d5a 100644 --- a/packages/api/session-controller/tests/queue-store.client.spec.ts +++ b/packages/api/session-controller/tests/queue-store.client.spec.ts @@ -73,7 +73,7 @@ describe('Session queue snapshot intake', () => { ]) }) - it('marks mixed-content messages non-editable while retaining their preview', () => { + it('marks mixed-content messages non-editable and keeps image blocks out of the text preview', () => { const session = makeSession() session.handleControlFrame(queueFrame([{ id: 'q-image', @@ -86,7 +86,9 @@ describe('Session queue snapshot intake', () => { { id: 'q-image', placement: 'queued', content: [{ type: 'text', text: 'hi' }, { type: 'image', data: 'x' }], - preview: 'hi [image]', text: null, + // Image blocks render as thumbnails from `content`, so the preview + // carries only the text; non-image foreign blocks keep their marker. + preview: 'hi', text: null, }, ]) }) diff --git a/packages/api/session-controller/tests/session-models.host.spec.ts b/packages/api/session-controller/tests/session-models.host.spec.ts index d203585459..b38ee0f94c 100644 --- a/packages/api/session-controller/tests/session-models.host.spec.ts +++ b/packages/api/session-controller/tests/session-models.host.spec.ts @@ -231,6 +231,58 @@ describe('Web session model selection', () => { await ctx.fiber.dispose() }) + it('delivers an admitted image batch through steer with the same ordered content as queue', async () => { + const { ctx, agent, sessionId } = await harness() + const attachments = { + imageLimits: { + maxImageBytes: 4, + maxImagesPerMessage: 2, + maxMessageImageBytes: 4, + maxImagePixels: 4, + maxImageDimension: 2000, + mediaTypes: ['image/png'], + }, + validateImage: vi.fn(() => Promise.resolve()), + saveImage: vi.fn((input: { data: Uint8Array; mediaType: 'image/png'; name?: string }) => Promise.resolve({ + attachmentId: `att-${String(input.data[0])}`, + mediaType: input.mediaType, + bytes: input.data.byteLength, + width: 1, + height: 1, + ...input.name === undefined ? {} : { name: input.name }, + })), + } + ctx.provide('attachments', Object.setPrototypeOf(attachments, AttachmentStore.prototype) as never) + const steer = vi.fn() + const followup = vi.fn() + Object.assign(agent, { steer, followup }) + const remote = createSessionTestRemote(ctx, { + defaultModelSelection: () => ({ provider: 'deepseek-official', model: 'deepseek-chat' }), + cwd: '/tmp', + }) + + const result = await remote.prompt(promptRequest({ + sessionId, + mode: 'steer' as const, + content: [ + { type: 'text' as const, text: 'look at this' }, + { type: 'image' as const, mediaType: 'image/png' as const, data: 'AQ==', name: 'mid-turn.png' }, + ], + })) + expect(result.ok).toBe(true) + expect(followup).not.toHaveBeenCalled() + expect((steer.mock.calls[0]?.[0] as UserMessage).content).toEqual([ + { type: 'text', text: 'look at this' }, + { + type: 'image', + attachment: { + attachmentId: 'att-1', mediaType: 'image/png', bytes: 1, width: 1, height: 1, name: 'mid-turn.png', + }, + }, + ]) + await ctx.fiber.dispose() + }) + it('allows a text-only selection while durable or pending images remain available for later models', async () => { const { ctx, agent, sessionId } = await harness() registerTextOnly(ctx) diff --git a/packages/api/session-controller/tests/session.client.spec.ts b/packages/api/session-controller/tests/session.client.spec.ts index 150e6813d1..ad436779e1 100644 --- a/packages/api/session-controller/tests/session.client.spec.ts +++ b/packages/api/session-controller/tests/session.client.spec.ts @@ -281,6 +281,32 @@ describe('prompt and cancel errors', () => { }) }) + it('forwards continuation image parts to the subagent prompt Remote unstripped', async () => { + const api = new FakeApiClient() + const session = new Session(SID, fakeRemote(api), { + address: { parentSessionId: PARENT, childSessionId: SID, mode: 'continuable' }, + parentAvailable: true, + }) + await session.open() + const content = [ + { type: 'text' as const, text: '看这张图' }, + { type: 'image' as const, mediaType: 'image/png' as const, data: 'aGk=', name: 'shot.png' }, + ] + const prompted = await session.prompt(content, 'queue') + + expect(prompted).toEqual({ ok: true, value: { accepted: true } }) + expect(api.callsOf('subagents.prompt')).toEqual([ + { + requestId: expect.any(String) as unknown as string, + parentSessionId: PARENT, childSessionId: SID, + mode: 'continuable', + content, + clientTimeZone: new Intl.DateTimeFormat().resolvedOptions().timeZone, + }, + ]) + expect(session.getSnapshot().promptError).toBeNull() + }) + it('lands an interrupt business failure in promptError with op=stop', async () => { const api = new FakeApiClient() api.onSubagentInterrupt = () => Promise.resolve(remoteErr({ diff --git a/packages/attachment/attachment/src/index.ts b/packages/attachment/attachment/src/index.ts index 4ee001b86c..1c0f76b9e0 100644 --- a/packages/attachment/attachment/src/index.ts +++ b/packages/attachment/attachment/src/index.ts @@ -23,6 +23,7 @@ export type { ImageAttachmentRef, ImageRequestPolicy, ImageMediaType, + PromptContentPart, RequestImageAttachment, SaveImageAttachment, StoredImageAttachment, diff --git a/packages/attachment/attachment/src/types.ts b/packages/attachment/attachment/src/types.ts index 046444cd76..64c199eaf1 100644 --- a/packages/attachment/attachment/src/types.ts +++ b/packages/attachment/attachment/src/types.ts @@ -52,6 +52,21 @@ export interface EncodedImageAttachment { name?: string } +/** + * Browser-submitted prompt content accepted by Host prompt endpoints; the + * accepting Host promotes image parts to durable references through + * `admitEncodedImages` before any message is created, so a wire caller can + * never cite an attachment it did not upload. + */ +export type PromptContentPart = + | { readonly type: 'text'; readonly text: string } + | { + readonly type: 'image' + readonly mediaType: ImageMediaType + readonly data: string + readonly name?: string + } + /** Request to validate and durably commit one image. */ export interface SaveImageAttachment { data: Uint8Array diff --git a/packages/client/ui-conversation/README.i18n.yaml b/packages/client/ui-conversation/README.i18n.yaml index d7a0420bec..63e2a89061 100644 --- a/packages/client/ui-conversation/README.i18n.yaml +++ b/packages/client/ui-conversation/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/client/ui-conversation/README.md -README.md: 188a4461c1ab46e8be4f96c402c4707b1c66c415 -README.zh.md: 63356b04ef07ddd8c037d46d0ad105ed01afda80 +README.md: 3a51c20caaa78f1e638d5117a9153112ce530fe2 +README.zh.md: be5b5396620566b356cc51aedc3fed21779cccd9 diff --git a/packages/client/ui-conversation/README.md b/packages/client/ui-conversation/README.md index 188a4461c1..3a51c20caa 100644 --- a/packages/client/ui-conversation/README.md +++ b/packages/client/ui-conversation/README.md @@ -38,7 +38,7 @@ The package registers the optional-Session `conversation` shell, strict Session View selection is deterministic: a registered persisted selection wins, otherwise registered `chat` wins, otherwise no View renders. It never chooses the first registered View. Shell phase combines Session lifecycle with the active-target set; no target-specific snapshot is read by the shell. -The resident composer survives no-Session and Session transitions. The no-Session state keeps the same composer surface mounted but inert while the Workspace picker connects a blank Session. The surface is a shell-owned Lexical editor: reference chips are atomic decorator nodes carrying the owner's serialization identity (submission expands them through the owner codec), claimed slash commands stay styled leading text, folder text references carry the folder glyph as an icon prefix, and the draft's clipboard projection is mirrored into the per-Session Conversation store. Queue operations address exact queue occurrences through the scoped `ctx.conversation` service; queue previews render sent text through the shared inline reference projection from `ui-primitives` (wire session forms fold to their label), while an edit exposes the literal sent text. Busy Enter behavior is stored in the Host-backed `ui-conversation` settings namespace. +The resident composer survives no-Session and Session transitions. The no-Session state keeps the same composer surface mounted but inert while the Workspace picker connects a blank Session. The surface is a shell-owned Lexical editor: reference chips are atomic decorator nodes carrying the owner's serialization identity (submission expands them through the owner codec), claimed slash commands stay styled leading text, folder text references carry the folder glyph as an icon prefix, and the draft's clipboard projection is mirrored into the per-Session Conversation store. Queue operations address exact queue occurrences through the scoped `ctx.conversation` service; queue previews render sent text through the shared inline reference projection from `ui-primitives` (wire session forms fold to their label) and show each durable image part as a thumbnail resolved through the session image URL cache, while an edit exposes the literal sent text. Busy Enter behavior is stored in the Host-backed `ui-conversation` settings namespace. While a normal composer is running, its primary pointer action remains Stop when the draft is empty or input is unavailable. Actionable text or attachments switch the same seat to Queue Send; clearing or successfully submitting the draft restores Stop. The busy-Enter setting continues to select the Queue or Steer keyboard action. Continuable subagents keep separate Send and Stop actions ([decision](../../../.agents/notes/implemented/bug-fix/2026-08-20-running-draft-primary-send.md)). diff --git a/packages/client/ui-conversation/README.zh.md b/packages/client/ui-conversation/README.zh.md index 63356b04ef..be5b539662 100644 --- a/packages/client/ui-conversation/README.zh.md +++ b/packages/client/ui-conversation/README.zh.md @@ -38,7 +38,7 @@ target package 通过 declaration merge 扩展 snapshot 与 Location data map， View 选择规则固定：有效且已注册的持久化选择优先，其次是已注册的 `chat`，否则不渲染 View；绝不选择第一个已注册 View。Shell phase 只组合 Session lifecycle 与 active-target set，不读取任何 target-specific snapshot。 -常驻 composer 在无 Session 与有 Session 之间保持挂载。无 Session 时，同一个编辑器表面保持 inert，Workspace picker 连接 blank Session。该表面是 shell 所有的 Lexical 编辑器：引用 chip 是携带 owner 序列化身份的原子 decorator 节点（提交时经 owner codec 展开），已认领的 slash command 保持为带样式的行首文本，文件夹文本引用以图标前缀携带文件夹图形，草稿的剪贴板投影镜像到逐 Session Conversation store。Queue 操作通过 scoped `ctx.conversation` service 寻址准确的 queue occurrence；queue 预览经 `ui-primitives` 的共享行内引用投影渲染已发送文本（wire 会话形式折叠为其标签），编辑态则展示字面发送文本。繁忙时 Enter 行为保存在 Host-backed `ui-conversation` settings namespace。 +常驻 composer 在无 Session 与有 Session 之间保持挂载。无 Session 时，同一个编辑器表面保持 inert，Workspace picker 连接 blank Session。该表面是 shell 所有的 Lexical 编辑器：引用 chip 是携带 owner 序列化身份的原子 decorator 节点（提交时经 owner codec 展开），已认领的 slash command 保持为带样式的行首文本，文件夹文本引用以图标前缀携带文件夹图形，草稿的剪贴板投影镜像到逐 Session Conversation store。Queue 操作通过 scoped `ctx.conversation` service 寻址准确的 queue occurrence；queue 预览经 `ui-primitives` 的共享行内引用投影渲染已发送文本（wire 会话形式折叠为其标签），并把每个持久化图片部分经会话图片 URL 缓存解析为缩略图展示，编辑态则展示字面发送文本。繁忙时 Enter 行为保存在 Host-backed `ui-conversation` settings namespace。 普通 composer 运行时，如果草稿为空或输入不可用，主指针操作保持为 Stop。可提交的文字或附件会把同一位置切换为 Queue Send；清空或成功提交草稿后恢复 Stop。繁忙态 Enter 设置继续选择 Queue 或 Steer 键盘操作。可继续 subagent 保留独立的 Send 与 Stop 操作（[决策](../../../.agents/notes/implemented/bug-fix/2026-08-20-running-draft-primary-send.zh.md)）。 diff --git a/packages/client/ui-conversation/src/client/image-labels.ts b/packages/client/ui-conversation/src/client/image-labels.ts index 6d4ee7a130..d2333c324c 100644 --- a/packages/client/ui-conversation/src/client/image-labels.ts +++ b/packages/client/ui-conversation/src/client/image-labels.ts @@ -31,7 +31,6 @@ export function attachmentErrorText( ): string { switch (reason) { case 'MODEL_DOES_NOT_SUPPORT_IMAGES': return t('image.modelUnsupported') - case 'SUBAGENT_IMAGE_UNSUPPORTED': return t('image.subagentUnsupported') case 'IMAGE_TOO_MANY_PIXELS': return t('image.tooManyPixels') case 'IMAGE_DIMENSION_TOO_LARGE': if (limits !== undefined) return t('image.dimensionTooLarge', { size: limits.maxImageDimension }) diff --git a/packages/client/ui-conversation/src/client/locales.ts b/packages/client/ui-conversation/src/client/locales.ts index 480f73764b..e8509b2c6f 100644 --- a/packages/client/ui-conversation/src/client/locales.ts +++ b/packages/client/ui-conversation/src/client/locales.ts @@ -45,7 +45,6 @@ export const zh = { 'image.tooManyPixels': '图片分辨率过大，请压缩后重试', 'image.dimensionTooLarge': '图片宽高不能超过 {size}px，请缩小后重试', 'image.modelUnsupported': '当前模型不支持图片，请切换支持图片的模型', - 'image.subagentUnsupported': '子智能体会话暂不支持图片', 'image.sendFailed': '图片发送失败（{reason}），请重新添加图片后再试', 'context.aria': '上下文已用 {percent}', 'context.used': '上下文已用', @@ -126,6 +125,7 @@ export const zh = { 'web.contentTruncated': '内容已截断', 'details.running': '运行中…', 'queue.count': '{n} 条排队消息', + 'queue.image': '排队消息图片', 'queue.edit': '编辑排队消息', 'queue.edit.unsupported': '包含非文本内容，暂不支持编辑', 'queue.save': '保存排队消息', @@ -190,7 +190,6 @@ export const en = { 'image.tooManyPixels': 'Image resolution is too high; compress it and try again', 'image.dimensionTooLarge': 'Image sides must be at most {size}px; downscale it and try again', 'image.modelUnsupported': 'The current model does not support images; switch to a model that does', - 'image.subagentUnsupported': 'Subagent sessions do not support images yet', 'image.sendFailed': 'Sending images failed ({reason}); re-add them and try again', 'context.aria': '{percent} of context used', 'context.used': 'of context used', @@ -271,6 +270,7 @@ export const en = { 'web.contentTruncated': 'Content truncated', 'details.running': 'Running…', 'queue.count': '{n} queued messages', + 'queue.image': 'Queued message image', 'queue.edit': 'Edit queued message', 'queue.edit.unsupported': 'Contains non-text content; editing is not supported yet', 'queue.save': 'Save queued message', diff --git a/packages/client/ui-conversation/src/client/queue/QueueDock.module.css b/packages/client/ui-conversation/src/client/queue/QueueDock.module.css index eca51941ca..81ea205b5e 100644 --- a/packages/client/ui-conversation/src/client/queue/QueueDock.module.css +++ b/packages/client/ui-conversation/src/client/queue/QueueDock.module.css @@ -122,6 +122,21 @@ box-shadow: inset 0 1px 0 var(--dsw-alias-border-l1); } +.thumbs { + display: flex; + flex: none; + gap: 4px; +} + +.thumb { + width: 24px; + height: 24px; + border: 1px solid var(--dsw-alias-border-l1); + border-radius: 4px; + background: var(--dsw-alias-bg-base); + object-fit: cover; +} + .preview, .editor { flex: 1 1 auto; diff --git a/packages/client/ui-conversation/src/client/queue/QueueDock.tsx b/packages/client/ui-conversation/src/client/queue/QueueDock.tsx index aadedd7d29..a42b8cea68 100644 --- a/packages/client/ui-conversation/src/client/queue/QueueDock.tsx +++ b/packages/client/ui-conversation/src/client/queue/QueueDock.tsx @@ -1,12 +1,13 @@ import type { Context } from '@deepseek-ai/cordis' import { useEffect, useId, useMemo, useState } from 'react' +import type { ImageAttachmentRef } from '@deepseek-ai/dsh-attachment' import type { PropsLocale, PropsRuntime } from '@deepseek-ai/dsh-client-ui-slots' import type { SessionId } from '@deepseek-ai/dsh-session/types' import { IconCheckOutline16, IconChevronDownOutline14, IconChevronUpOutline14, IconCloseOutline16, IconEditOutline16, IconQueueOutline14, IconSendOutline14, IconTrashOutline16, projectUserText, Tooltip, } from '@deepseek-ai/dsh-client-ui-primitives' -import type { QueueAction, QueueItemId } from '../contract/queue.ts' +import type { QueueAction, QueueItemId, QueueRow } from '../contract/queue.ts' import { NS } from '../locales.ts' import css from './QueueDock.module.css' @@ -14,6 +15,43 @@ import css from './QueueDock.module.css' export interface QueueDockInjected { updateQueue: (itemId: QueueItemId, action: QueueAction) => Promise notify: (level: 'info' | 'error', text: string) => void + /** Resolve one durable queued image into a session-scoped browser URL. */ + loadImage: (attachment: ImageAttachmentRef) => Promise +} + +/** + * Durable references carried by one queued row. Queue frames are wire data + * despite their typed face, so an image block without a reference is skipped + * rather than trusted. + * @param content - the row's wire content blocks. + * @returns the row's durable image references in block order. + */ +function queueImageRefs(content: QueueRow['content']): ImageAttachmentRef[] { + return content.flatMap((block) => { + if (block.type !== 'image') return [] + const { attachment } = block as { attachment?: ImageAttachmentRef } + return attachment === undefined ? [] : [attachment] + }) +} + +/** One durable queued image as a fixed-size thumbnail; a load failure keeps the empty placeholder. */ +function QueueThumb({ attachment, loadImage, label }: { + attachment: ImageAttachmentRef + loadImage: QueueDockInjected['loadImage'] + label: string +}) { + const [url, setUrl] = useState(null) + useEffect(() => { + let alive = true + loadImage(attachment).then( + (resolved) => { if (alive) setUrl(resolved) }, + () => { /* placeholder retained; the durable transcript surfaces read errors */ }, + ) + return () => { alive = false } + }, [attachment, loadImage]) + return url === null + ? + : {label} } /** Full props of a dock entry: InputZone owner share + session standard kit + global seat + the locale seat. */ @@ -23,7 +61,7 @@ export type QueueDockProps = PropsRuntime<'conversation.input.dock'> & QueueDock * Queue strip: one item renders directly; multiple items default to a * collapsible count header; an empty queue renders nothing. */ -export function QueueDock({ useSession, updateQueue, notify, t }: QueueDockProps) { +export function QueueDock({ useSession, updateQueue, notify, loadImage, t }: QueueDockProps) { const inbox = useSession(s => s.queue) const queue = useMemo(() => inbox.filter(row => row.placement === 'queued'), [inbox]) const running = useSession(s => s.running) @@ -114,7 +152,23 @@ export function QueueDock({ useSession, updateQueue, notify, t }: QueueDockProps }} /> ) - : {projectUserText(row.preview, [])}} + : ( + <> + {queueImageRefs(row.content).length > 0 && ( + + {queueImageRefs(row.content).map((attachment, index) => ( + + ))} + + )} + {projectUserText(row.preview, [])} + + )} {queueMutable &&
{editing?.id === row.id ? ( @@ -210,7 +264,7 @@ export function QueueDock({ useSession, updateQueue, notify, t }: QueueDockProps /** Registers queue actions backed by the session-scoped conversation service. */ export const queueDockEntry = { name: 'conversation-queue-dock', - inject: ['slots', 'conversation', 'sessions'], + inject: ['slots', 'conversation', 'sessions', 'uiConversation'], apply(ctx: Context): void { ctx.slots.inject('conversation.input.dock', () => ctx.slots.register({ name: 'conversation.input.dock', @@ -225,6 +279,7 @@ export const queueDockEntry = { return { updateQueue: (itemId, action) => conversation.updateQueue(itemId, action), notify: (level, text) => { conversation.input.for(actx).notify(level, text) }, + loadImage: attachment => ctx.uiConversation.imageUrl(sessionId, attachment), } }, }, QueueDock)) diff --git a/packages/client/ui-conversation/tests/image-labels.client.spec.ts b/packages/client/ui-conversation/tests/image-labels.client.spec.ts index 8e1ff4409d..0244c134db 100644 --- a/packages/client/ui-conversation/tests/image-labels.client.spec.ts +++ b/packages/client/ui-conversation/tests/image-labels.client.spec.ts @@ -24,7 +24,6 @@ describe('attachment rejection copy', () => { it('maps user-solvable reasons to limit-naming copy', () => { expect(attachmentErrorText(t, 'MODEL_DOES_NOT_SUPPORT_IMAGES')).toBe('当前模型不支持图片，请切换支持图片的模型') - expect(attachmentErrorText(t, 'SUBAGENT_IMAGE_UNSUPPORTED')).toBe('子智能体会话暂不支持图片') expect(attachmentErrorText(t, 'IMAGE_TOO_MANY_PIXELS')).toBe('图片分辨率过大，请压缩后重试') expect(attachmentErrorText(t, 'INVALID_IMAGE')).toBe('仅支持 PNG、JPG、WebP、GIF 格式的图片') expect(attachmentErrorText(t, 'IMAGE_TYPE_MISMATCH')).toBe('仅支持 PNG、JPG、WebP、GIF 格式的图片') diff --git a/packages/client/ui-conversation/tests/queue-dock.client.spec.tsx b/packages/client/ui-conversation/tests/queue-dock.client.spec.tsx index 6a30ed4d8a..75701f7076 100644 --- a/packages/client/ui-conversation/tests/queue-dock.client.spec.tsx +++ b/packages/client/ui-conversation/tests/queue-dock.client.spec.tsx @@ -87,10 +87,26 @@ function kitFor(snapshot: SessionSnapshot, injected: Partial input: INPUT_STATE, updateQueue: vi.fn(() => Promise.resolve()), notify: vi.fn(), + loadImage: vi.fn(() => Promise.resolve('blob:unused')), ...injected, } } +/** One queued row carrying a durable image reference (plus optional leading text). */ +function imageRow(id: string, refId: string, text = ''): QueuedMessage { + return { + id: iid(id), messageId: `message-${id}` as never, placement: 'queued', + content: [ + ...text === '' ? [] : [{ type: 'text' as const, text }], + { + type: 'image', + attachment: { attachmentId: refId, mediaType: 'image/png', bytes: 1, width: 1, height: 1 }, + } as never, + ], + preview: text, text: null, + } +} + describe('QueueDock', () => { it('renders null while the queue is empty', () => { const snap = snapshotWith([]) @@ -223,6 +239,52 @@ describe('QueueDock', () => { .toBe('包含非文本内容，暂不支持编辑') }) + it('renders queued image thumbnails from durable references beside the text preview', async () => { + const loadImage = vi.fn(() => Promise.resolve('blob:thumb-1')) + const snap = snapshotWith([imageRow('i-img', 'att-9', '带图消息')]) + const source = liveSession(snap) + const { container } = render( + , + ) + + await waitFor(() => { + expect(container.querySelector('img')?.getAttribute('src')).toBe('blob:thumb-1') + }) + expect(loadImage).toHaveBeenCalledWith(expect.objectContaining({ attachmentId: 'att-9' })) + expect(container.querySelector('img')?.getAttribute('alt')).toBe('排队消息图片') + expect(container.querySelector('li')?.textContent).toBe('带图消息') + }) + + it('keeps the empty thumbnail placeholder when the image read fails', async () => { + const loadImage = vi.fn(() => Promise.reject(new Error('read denied'))) + const snap = snapshotWith([imageRow('i-broken', 'att-x')]) + const source = liveSession(snap) + const { container } = render( + , + ) + + await act(async () => { await Promise.resolve() }) + expect(loadImage).toHaveBeenCalled() + expect(container.querySelector('img')).toBeNull() + }) + + it('ignores a thumbnail resolution landing after unmount', async () => { + let resolveUrl: ((url: string) => void) | undefined + const loadImage = vi.fn(() => new Promise((resolve) => { resolveUrl = resolve })) + const snap = snapshotWith([imageRow('i-late', 'att-late')]) + const source = liveSession(snap) + const { unmount } = render( + , + ) + + unmount() + await act(async () => { + resolveUrl?.('blob:late') + await Promise.resolve() + }) + expect(loadImage).toHaveBeenCalledTimes(1) + }) + it('edits text inline with save and cancel controls, then saves with the same item identity', async () => { const snap = snapshotWith([row('i-edit', 'before')]) const source = liveSession(snap) @@ -388,7 +450,7 @@ describe('QueueDock', () => { it('registers as the terminal composer-context entry', () => { expect(queueDockEntry.name).toBe('conversation-queue-dock') - expect(queueDockEntry.inject).toEqual(['slots', 'conversation', 'sessions']) + expect(queueDockEntry.inject).toEqual(['slots', 'conversation', 'sessions', 'uiConversation']) const register = vi.fn(() => () => undefined) const inject = vi.fn((_name: string, callback: () => () => void) => callback()) queueDockEntry.apply({ slots: { inject, register } } as never) diff --git a/packages/core/agent-loop/src/agent.ts b/packages/core/agent-loop/src/agent.ts index 0d3af9663b..9bac17bff0 100644 --- a/packages/core/agent-loop/src/agent.ts +++ b/packages/core/agent-loop/src/agent.ts @@ -70,6 +70,17 @@ export class ReactLoopAgent implements Agent { readonly inbox: Inbox private phase: Phase private activityDone: Promise = Promise.resolve() + /** + * Identities of waking sends still awaiting a claim. Claim and discard + * notifications prune the set, and {@link cancel} clears it because a + * cancellation parks accepted-but-unclaimed input for a later waking send. + * A non-empty set at driver exit therefore means a steer or follow-up lost + * the race with a normally or erroneously closing turn, and the exit must + * start a fresh driver to deliver it. Injected context never enters the + * set, so it keeps waiting for a waking message instead of opening a turn + * by itself. + */ + private readonly pendingWakes = new Set() /** The agent-scoped registration boundary; the lifecycle owner unwinds it after the driver exits. */ readonly scope: Scope @@ -93,8 +104,14 @@ export class ReactLoopAgent implements Agent { this.dispatch = agentEvents(loopCtx, this) this.inbox = new Inbox(session, { inserted: (message) => { this.dispatch.emit('agent/inbox/inserted', { message }) }, - discarded: (message) => { this.dispatch.emit('agent/inbox/discarded', { message }) }, - claimed: (message, turn) => { this.dispatch.emit('agent/inbox/claimed', { message, turn }) }, + discarded: (message) => { + this.pendingWakes.delete(message.id) + this.dispatch.emit('agent/inbox/discarded', { message }) + }, + claimed: (message, turn) => { + this.pendingWakes.delete(message.id) + this.dispatch.emit('agent/inbox/claimed', { message, turn }) + }, }) const lastTurn = session.events.findLast(event => event.type === 'turn/start')?.data.turn ?? 0 this.phase = { kind: 'idle', lastTurn } @@ -122,7 +139,15 @@ export class ReactLoopAgent implements Agent { // Captured before the insertion so a reentrant cancel from a splice observer cannot reclassify it. const wakingAfterAbort = wakeup && this.phase.kind !== 'idle' && this.phase.abort.signal.aborted const resolvedTarget = wakingAfterAbort ? 'next-turn' : target - this.inbox.splice(resolvedTarget, Infinity, 0, [message]) + // Registered before the splice so a reentrant discard inside the splice + // dispatch still prunes it; a refused splice never leaves an entry behind. + if (wakeup) this.pendingWakes.add(message.id) + try { + this.inbox.splice(resolvedTarget, Infinity, 0, [message]) + } catch (error: unknown) { + this.pendingWakes.delete(message.id) + throw error + } if (wakeup) this.wakeDriver(wakingAfterAbort) } @@ -143,6 +168,10 @@ export class ReactLoopAgent implements Agent { this.inbox.clear() if (this.phase.kind !== 'idle') this.phase.wakeRequested = false } + // Cancellation consumes outstanding wakes: kept inbox work parks until + // the next waking send resumes the queue, and a cleared inbox has nothing + // left to deliver. + this.pendingWakes.clear() if (this.phase.kind !== 'idle') this.phase.abort.abort(cause) } @@ -215,8 +244,14 @@ export class ReactLoopAgent implements Agent { } private async kick(): Promise { + // Set only when the turn loop returns without throwing: an abort or driver + // failure parks unclaimed waking input for the next waking send, while a + // clean exit must deliver a steer or follow-up that lost the race with the + // closing turn (its send saw a live driver, so no wake was latched). + let cleanExit = false try { while (await this.turn()) {} + cleanExit = true } catch (_error) { // Reported failures and cancellation are contained at the driver boundary. } finally { @@ -224,7 +259,9 @@ export class ReactLoopAgent implements Agent { if (this.phase.kind === 'running') { const { turn, wakeRequested } = this.phase this.setPhase({ kind: 'idle', lastTurn: turn }) - if (wakeRequested && this.inbox.hasPending) this.wakeDriver() + if ((wakeRequested || (cleanExit && this.pendingWakes.size > 0)) && this.inbox.hasPending) { + this.wakeDriver() + } } } } @@ -272,6 +309,10 @@ export class ReactLoopAgent implements Agent { const step = phase.step + 1 const decision = await this.preStep(target, { turn, step }) if (decision.kind === 'reject') { + // The rejecting listener owns resumption: input staged behind the + // rejected claim parks until the next waking send, exactly like a + // cancellation, instead of being re-offered to the same policy. + this.pendingWakes.clear() turnEnds = { kind: 'blocked' } return false } diff --git a/packages/core/agent-loop/tests/loop.spec.ts b/packages/core/agent-loop/tests/loop.spec.ts index 487fc60181..3ca29d6450 100644 --- a/packages/core/agent-loop/tests/loop.spec.ts +++ b/packages/core/agent-loop/tests/loop.spec.ts @@ -1530,3 +1530,83 @@ describe('agent loop', () => { expect(replayed.events.at(-1)?.type).toBe('session/end-seed') }) }) + +describe('closing-turn wake races', () => { + /** + * Schedule one send in the closing turn's final microtask window: the + * synchronous turn/end dispatch queues the microtask before the driver's + * exit continuation, so it lands after the final inbox check and before the + * driver boundary — the race a live send cannot latch a wake for. + */ + function sendOnTurnEnd(ctx: Context, agent: Agent, deliver: () => void): void { + const dispose = ctx.on('session/event', (session, event) => { + if (session !== agent.session || event.type !== 'turn/end') return + dispose() + queueMicrotask(deliver) + }) + } + + it('delivers a steer that lands while a clean turn is closing', async () => { + const adapter = new MockAdapter([textResponse('first'), textResponse('second')]) + const ctx = await harness(adapter) + const agent = ctx.agentLoop.create(SessionId('closing-steer'), { provider: 'mock', model: 'mock' }) + sendOnTurnEnd(ctx, agent, () => { + agent.steer(createUserMessage({ content: [{ type: 'text', text: 'late steer' }], source: { kind: 'user' } })) + }) + + send(agent, 'first') + await waitForIdle(ctx, agent) + await agent.whenIdle() + + expect(userTexts(agent)).toEqual(['first', 'late steer']) + expect(adapter.requests).toHaveLength(2) + expect(agent.inbox.hasPending).toBe(false) + }) + + it('delivers a follow-up that lands while a clean turn is closing', async () => { + const adapter = new MockAdapter([textResponse('first'), textResponse('second')]) + const ctx = await harness(adapter) + const agent = ctx.agentLoop.create(SessionId('closing-followup'), { provider: 'mock', model: 'mock' }) + sendOnTurnEnd(ctx, agent, () => { send(agent, 'late follow-up') }) + + send(agent, 'first') + await waitForIdle(ctx, agent) + await agent.whenIdle() + + expect(userTexts(agent)).toEqual(['first', 'late follow-up']) + expect(adapter.requests).toHaveLength(2) + expect(agent.inbox.hasPending).toBe(false) + }) + + it('leaves injected context parked when a clean turn closes without a waking message', async () => { + const adapter = new MockAdapter([textResponse('only')]) + const ctx = await harness(adapter) + const agent = ctx.agentLoop.create(SessionId('closing-inject'), { provider: 'mock', model: 'mock' }) + sendOnTurnEnd(ctx, agent, () => { + agent.inject(createUserMessage({ content: [{ type: 'text', text: 'context' }], source: { kind: 'user' } })) + }) + + send(agent, 'only') + await waitForIdle(ctx, agent) + await agent.whenIdle() + + expect(userTexts(agent)).toEqual(['only']) + expect(adapter.requests).toHaveLength(1) + expect(agent.inbox.nextStep).toHaveLength(1) + }) + + it('a refused duplicate splice keeps the parked injection and registers no wake', async () => { + const adapter = new MockAdapter([]) + const ctx = await harness(adapter) + const agent = ctx.agentLoop.create(SessionId('duplicate-splice'), { provider: 'mock', model: 'mock' }) + const message = createUserMessage({ content: [{ type: 'text', text: 'context' }], source: { kind: 'user' } }) + agent.inject(message) + + expect(() => { agent.steer(message) }).toThrow(`message "${message.id}" is already pending`) + await agent.whenIdle() + + expect(agent.inbox.nextStep).toHaveLength(1) + expect(agent.status).toBe('idle') + expect(adapter.requests).toHaveLength(0) + }) +}) diff --git a/packages/extensions/cordis-client-runner/src/client/api-catalog.ts b/packages/extensions/cordis-client-runner/src/client/api-catalog.ts index 7dd28e586b..c5317882b4 100644 --- a/packages/extensions/cordis-client-runner/src/client/api-catalog.ts +++ b/packages/extensions/cordis-client-runner/src/client/api-catalog.ts @@ -606,10 +606,6 @@ export const TYPE_API: readonly TypeApiEntry[] = [ name: 'ProjectionsFace', declaration: 'export interface ProjectionsFace {\n faceOf(key: string): ObservableSnapshot;\n}', }, - { - name: 'PromptContentPart', - declaration: 'export type PromptContentPart = {\n readonly type: \'text\';\n readonly text: string;\n} | {\n readonly type: \'image\';\n readonly mediaType: ImageMediaType;\n readonly data: string;\n readonly name?: string;\n};', - }, { name: 'PromptError', declaration: 'export interface PromptError {\n readonly op: \'send\' | \'stop\';\n readonly error: ClientFailure;\n}', diff --git a/packages/extensions/tool-cordis/src/api-catalog.ts b/packages/extensions/tool-cordis/src/api-catalog.ts index 5412e025cd..cdd4de1213 100644 --- a/packages/extensions/tool-cordis/src/api-catalog.ts +++ b/packages/extensions/tool-cordis/src/api-catalog.ts @@ -2168,10 +2168,10 @@ export const SERVICE_API: readonly ServiceApiEntry[] = [ }, { signature: '@Remote(\'prompt\') async prompt(request: SubagentPromptRequest, signal: AbortSignal): Promise', - description: 'Deliver one browser-authored message to a continuable child through the exact live direct parent, retaining the caller-minted request identity and validated browser zone on the accepted message. Success identifies the message the child\'s FIFO inbox accepted; later execution is independent of this call.', + description: 'Deliver one browser-authored message to a continuable child through the exact live direct parent, retaining the caller-minted request identity and validated browser zone on the accepted message. Success identifies the message the child\'s FIFO inbox accepted; later execution is independent of this call. Image parts are admitted and persisted through the attachment store before delivery, and the child\'s model must accept image input.', parameters: [{ name: 'request', description: 'durable address, minted identity, content, and optional browser zone.' }, { name: 'signal', description: 'carrier cancellation, owning the call until inbox acceptance.' }], returns: 'the accepted message\'s inbox identity.', - throws: ['{TypertRemoteFailure} `bad-request`, `invalid-time-zone`, `subagent-parent-unavailable`, `subagent-not-resumable`, `subagent-unauthorized`, `subagent-delivery-unavailable`, `cancelled`, or `internal`.'], + throws: ['{TypertRemoteFailure} `bad-request`, `invalid-time-zone`, `attachment-error`, `subagent-parent-unavailable`, `subagent-not-resumable`, `subagent-unauthorized`, `subagent-delivery-unavailable`, `cancelled`, or `internal`.'], }, { signature: '@Remote(\'interruptByParent\') interruptByParent( childSessionId: SessionId, parentSessionId: SessionId, mode: \'continuable\', ): SubagentInterruptReceipt', @@ -5228,7 +5228,7 @@ export const TYPE_API: readonly TypeApiEntry[] = [ }, { name: 'SubagentPromptRequest', - declaration: 'export interface SubagentPromptRequest {\n readonly requestId: SubagentPromptRequestId;\n readonly parentSessionId: SessionId;\n readonly childSessionId: SessionId;\n readonly mode: \'continuable\';\n readonly content: ContentBlock[];\n readonly clientTimeZone?: string;\n}', + declaration: 'export interface SubagentPromptRequest {\n readonly requestId: SubagentPromptRequestId;\n readonly parentSessionId: SessionId;\n readonly childSessionId: SessionId;\n readonly mode: \'continuable\';\n readonly content: readonly PromptContentPart[];\n readonly clientTimeZone?: string;\n}', }, { name: 'SubagentPromptRequestId', diff --git a/packages/llm/llm/src/content.ts b/packages/llm/llm/src/content.ts index 43c392a641..faac6249af 100644 --- a/packages/llm/llm/src/content.ts +++ b/packages/llm/llm/src/content.ts @@ -2,7 +2,8 @@ import type { ContentBlock } from './types.ts' import type { Message } from './message.ts' -import type { AttachmentStore, ImageAttachmentRef, ImageMediaType, RequestImageAttachment } from '@deepseek-ai/dsh-attachment' +import type { AttachmentStore, ImageAttachmentRef, ImageMediaType, PromptContentPart, RequestImageAttachment } from '@deepseek-ai/dsh-attachment' +import { admitEncodedImages } from '@deepseek-ai/dsh-attachment' import { assertNever } from './never.ts' /** Execution-world path that model tools can use to read one normalized attachment. */ @@ -39,6 +40,31 @@ export function resolveImageAttachmentAccess( return readonlyPath === undefined ? undefined : { readonlyPath } } +/** + * Promote one browser-submitted prompt into durable model content: every image + * part is admitted through the attachment store before any block exists, so a + * caller-supplied part can never cite an attachment it did not upload here. + * The shared conversion for every Host prompt endpoint accepting uploads. + * @param attachments - the deployment attachment store owning admission policy. + * @param content - ordered browser prompt parts. + * @returns content blocks in part order, image parts replaced by durable references. + * @throws AttachmentError when the image batch is refused. + */ +export async function durablePromptContent( + attachments: AttachmentStore, + content: readonly PromptContentPart[], +): Promise { + if (content.every(part => part.type === 'text')) { + return content.map(part => ({ type: 'text', text: part.text })) + } + const refs = await admitEncodedImages(attachments, content.filter(part => part.type === 'image')) + let next = 0 + return content.map(part => part.type === 'text' + ? { type: 'text', text: part.text } + // admitEncodedImages returns one reference per image part in order. + : { type: 'image', attachment: refs[next++] as ImageAttachmentRef }) +} + function quoted(value: string): string { return JSON.stringify(value) } diff --git a/packages/llm/llm/tests/content.spec.ts b/packages/llm/llm/tests/content.spec.ts index 748e20f412..bb0492b0af 100644 --- a/packages/llm/llm/tests/content.spec.ts +++ b/packages/llm/llm/tests/content.spec.ts @@ -4,6 +4,7 @@ import type { AttachmentStore, ImageMediaType } from '@deepseek-ai/dsh-attachmen import { ToolCallId, createUserMessage, + durablePromptContent, offloadedImageText, offloadedImagePrefixCount, offloadRequestImagesWithPolicy, @@ -360,3 +361,33 @@ describe('projectImagesForTextModel', () => { ]) }) }) + +describe('durablePromptContent', () => { + it('converts text-only prompts without touching the attachment store', async () => { + const store = { saveImages: () => { throw new Error('text-only prompts must not reach the store') } } + await expect(durablePromptContent(store as unknown as AttachmentStore, [ + { type: 'text', text: 'hello' }, + ])).resolves.toEqual([{ type: 'text', text: 'hello' }]) + }) + + it('replaces image parts with admitted references in part order', async () => { + const store = { + saveImages: (inputs: readonly { data: Uint8Array }[]) => Promise.resolve(inputs.map((input, index) => ({ + attachmentId: AttachmentId(`att-${index}`), + mediaType: 'image/png' as ImageMediaType, + bytes: input.data.byteLength, + width: 1, + height: 1, + }))), + } + await expect(durablePromptContent(store as unknown as AttachmentStore, [ + { type: 'image', mediaType: 'image/png', data: 'AQ==' }, + { type: 'text', text: 'between' }, + { type: 'image', mediaType: 'image/png', data: 'Ag==' }, + ])).resolves.toEqual([ + { type: 'image', attachment: { attachmentId: 'att-0', mediaType: 'image/png', bytes: 1, width: 1, height: 1 } }, + { type: 'text', text: 'between' }, + { type: 'image', attachment: { attachmentId: 'att-1', mediaType: 'image/png', bytes: 1, width: 1, height: 1 } }, + ]) + }) +}) diff --git a/packages/subagent/subagent/README.i18n.yaml b/packages/subagent/subagent/README.i18n.yaml index e807931e2c..9277bc689f 100644 --- a/packages/subagent/subagent/README.i18n.yaml +++ b/packages/subagent/subagent/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/subagent/subagent/README.md -README.md: 76df70351d711d580d3ab1a89d0f929b85eab172 -README.zh.md: c4deb001c47435d29a5ca18cc0f8c0d26e48941e +README.md: f13de661015f3aa59b776273376f07653f6bbba5 +README.zh.md: 42923504ed11b8eff7c0bdcd9d0c9505d741f42e diff --git a/packages/subagent/subagent/README.md b/packages/subagent/subagent/README.md index 76df70351d..f13de66101 100644 --- a/packages/subagent/subagent/README.md +++ b/packages/subagent/subagent/README.md @@ -48,7 +48,7 @@ One-shot children run once and settle with a single result, plus an optional str ### Following up, interrupting, and discovering -Continuable children answer follow-up messages as their next turns, and the parent can interrupt a running turn or list its children at any time. Discovery covers both shapes: the service lists direct children and the full descendant tree — mode, activity, and lineage — reading live session state and optional persistence, without loading any child. +Continuable children answer follow-up messages as their next turns, and the parent can interrupt a running turn or list its children at any time. A browser continuation prompt may carry image parts: the Host admits and persists each image batch through the attachment store before the child inbox accepts the message, and refuses delivery when the child's declared model does not accept image input. Discovery covers both shapes: the service lists direct children and the full descendant tree — mode, activity, and lineage — reading live session state and optional persistence, without loading any child. ### Failure and recovery diff --git a/packages/subagent/subagent/README.zh.md b/packages/subagent/subagent/README.zh.md index c4deb001c4..42923504ed 100644 --- a/packages/subagent/subagent/README.zh.md +++ b/packages/subagent/subagent/README.zh.md @@ -48,7 +48,7 @@ kind: "package-reference" ### 后续消息、中断与发现 -可继续子 agent 把后续消息作为下一个轮次回答，父级随时可以中断运行中的轮次或列举自己的子级。发现覆盖两种形态：服务列举直接子级与完整后代树——模式、活动状态与血缘——直接读取在线会话状态与可选持久化，不加载任何子 agent。 +可继续子 agent 把后续消息作为下一个轮次回答，父级随时可以中断运行中的轮次或列举自己的子级。浏览器发出的继续执行 prompt 可以携带图片部分：Host 先通过附件存储完成整批图片的准入与持久化，子级 inbox 才接受这条消息；当子级声明的模型不接受图片输入时拒绝投递。发现覆盖两种形态：服务列举直接子级与完整后代树——模式、活动状态与血缘——直接读取在线会话状态与可选持久化，不加载任何子 agent。 ### 失败与恢复 diff --git a/packages/subagent/subagent/package.json b/packages/subagent/subagent/package.json index c89f11eee7..49f28fb095 100644 --- a/packages/subagent/subagent/package.json +++ b/packages/subagent/subagent/package.json @@ -54,6 +54,7 @@ "peerDependencies": { "@deepseek-ai/dsh-agent": "workspace:^", "@deepseek-ai/dsh-agent-presets": "workspace:^", + "@deepseek-ai/dsh-attachment": "workspace:^", "@deepseek-ai/dsh-brand": "workspace:^", "@deepseek-ai/dsh-invariants": "workspace:^", "@deepseek-ai/dsh-llm": "workspace:^", @@ -104,6 +105,7 @@ "devDependencies": { "@deepseek-ai/dsh-agent": "workspace:^", "@deepseek-ai/dsh-agent-presets": "workspace:^", + "@deepseek-ai/dsh-attachment": "workspace:^", "@deepseek-ai/dsh-brand": "workspace:^", "@deepseek-ai/dsh-invariants": "workspace:^", "@deepseek-ai/dsh-llm": "workspace:^", diff --git a/packages/subagent/subagent/src/continuation.ts b/packages/subagent/subagent/src/continuation.ts index 6b50947060..f3b90a8075 100644 --- a/packages/subagent/subagent/src/continuation.ts +++ b/packages/subagent/subagent/src/continuation.ts @@ -30,7 +30,7 @@ import type { AgentSetupCommit, CreateAgentOptions, } from '@deepseek-ai/dsh-agent' -import { ReasoningEffortId, boundContextSummary, createUserMessage, errorChain } from '@deepseek-ai/dsh-llm' +import { ReasoningEffortId, boundContextSummary, contentHasImage, createUserMessage, errorChain } from '@deepseek-ai/dsh-llm' import type { ContentBlock, MessageId, MessageSource } from '@deepseek-ai/dsh-llm' import { SessionId } from '@deepseek-ai/dsh-session' import type { SessionEvent } from '@deepseek-ai/dsh-session' @@ -523,6 +523,11 @@ export class SubagentContinuationManager { if (activation.disposal !== undefined) { return activation.disposal.then(() => undefined, () => undefined) } + // Guarded call: text-only delivery must not gain an await hop inside + // the per-child lock, where it would reorder against drain admission. + if (contentHasImage(content)) { + await this.assertImageCapable(activation.handle.agent, options.signal) + } return this.submitAdmitted(activation, content, options.source, parent, options.signal) }) /* v8 ignore start -- only the lost-cutoff arm above returns undefined, so only that @@ -1021,6 +1026,9 @@ export class SubagentContinuationManager { signal: AbortSignal, ): Promise { try { + if (contentHasImage(content)) { + await this.assertImageCapable(activation.handle.agent, signal) + } return this.submitAdmitted(activation, content, source, parent, signal) } catch (error: unknown) { /* v8 ignore next -- rollback disposal failures must not mask the @@ -1030,6 +1038,37 @@ export class SubagentContinuationManager { } } + /** + * Refuse image content addressed to a child whose model accepts text only. + * Callers guard with `contentHasImage`, so text-only delivery never awaits. + * The check runs inside the per-child delivery lock, before the message + * exists, so a rejection leaves no partial user message. When the child's + * route is not fixed by its options (a request-waterfall listener owns it) + * or no LLM registry is composed, delivery proceeds and the LLM layer's + * text-only projection replaces each image with its stable placeholder. + * @param agent - the live or freshly materialized child agent. + * @param signal - caller cancellation bounding the model-info read. + * @throws {SubagentError} `MODEL_DOES_NOT_SUPPORT_IMAGES` when the child's resolved model declines image input. + */ + private async assertImageCapable( + agent: Agent, + signal: AbortSignal, + ): Promise { + const { provider, model } = agent.options + if (provider === undefined || model === undefined) return + const llm = this.ctx.get('llm') + /* v8 ignore next -- a deployment without the LLM registry serves no model + * to refuse against; delivery then defers to the text-only projection. */ + if (llm === undefined) return + const info = await llm.resolveModelInfo(provider, model, signal) + if (info.inputModalities !== undefined && !info.inputModalities.includes('image')) { + throw new SubagentError( + `Model "${model}" does not support image input.`, + 'MODEL_DOES_NOT_SUPPORT_IMAGES', + ) + } + } + /** * Create or resume the child Agent through the private activation-owner * scope, install the handle in a fresh Activation, and register ownership on diff --git a/packages/subagent/subagent/src/control-types.ts b/packages/subagent/subagent/src/control-types.ts index ea02674413..e8707cc13e 100644 --- a/packages/subagent/subagent/src/control-types.ts +++ b/packages/subagent/subagent/src/control-types.ts @@ -6,9 +6,9 @@ * @module @deepseek-ai/dsh-subagent/control-types */ +import type { PromptContentPart } from '@deepseek-ai/dsh-attachment/types' import type { Branded } from '@deepseek-ai/dsh-brand' import type { MessageId } from '@deepseek-ai/dsh-llm/brand' -import type { ContentBlock } from '@deepseek-ai/dsh-llm/types' import type { SessionId } from '@deepseek-ai/dsh-session/types' import type { z as zCore } from 'zod' @@ -103,8 +103,12 @@ export interface SubagentPromptRequest { readonly childSessionId: SessionId /** Required discriminator retained from the browser control address. */ readonly mode: 'continuable' - /** Content delivered as the child's user message. */ - readonly content: ContentBlock[] + /** + * Browser prompt parts delivered as the child's user message. The Host + * admits and persists image parts before delivery, so the wire never + * carries a durable attachment reference the caller could fabricate. + */ + readonly content: readonly PromptContentPart[] /** Optional browser zone sampled for this exact human prompt. */ readonly clientTimeZone?: string } @@ -127,6 +131,8 @@ export interface SubagentInterruptReceipt { */ export interface SubagentControlErrorDetailsMap { 'bad-request': { readonly issues: zCore.core.$ZodIssue[] } + /** Image admission or model image-capability refusal; `reason` carries the stable admission code. */ + 'attachment-error': { readonly reason: string } cancelled: Record 'invalid-time-zone': { readonly value: string } 'subagent-parent-unavailable': { readonly parentSessionId: SessionId } diff --git a/packages/subagent/subagent/src/control.ts b/packages/subagent/subagent/src/control.ts index 661a43153a..0c6bd9d7a4 100644 --- a/packages/subagent/subagent/src/control.ts +++ b/packages/subagent/subagent/src/control.ts @@ -7,6 +7,7 @@ */ import type { Context } from '@deepseek-ai/cordis' +import { AttachmentError } from '@deepseek-ai/dsh-attachment' import type { SessionId } from '@deepseek-ai/dsh-session' import { TypertRemoteFailure } from '@deepseek-ai/dsh-typert-protocol' import { z } from 'zod' @@ -148,8 +149,13 @@ export function rejectPrompt(error: unknown, childSessionId: SessionId, signal: if (isCancellation(error, signal)) { return rejectControl('cancelled', 'subagent prompt was cancelled', {}) } + if (error instanceof AttachmentError) { + return rejectControl('attachment-error', error.message, { reason: error.code }) + } if (error instanceof SubagentError) { switch (error.code) { + case 'MODEL_DOES_NOT_SUPPORT_IMAGES': + return rejectControl('attachment-error', error.message, { reason: error.code }) case 'NOT_RESUMABLE': return rejectControl('subagent-not-resumable', 'subagent cannot be resumed', { childSessionId }) case 'UNAUTHORIZED': diff --git a/packages/subagent/subagent/src/index.ts b/packages/subagent/subagent/src/index.ts index 444f4ee857..f1a8689131 100644 --- a/packages/subagent/subagent/src/index.ts +++ b/packages/subagent/subagent/src/index.ts @@ -34,6 +34,7 @@ import { scopeTarget } from '@deepseek-ai/dsh-scope' import type { Scoped } from '@deepseek-ai/dsh-scope' import { assertObjectJsonSchema } from '@deepseek-ai/dsh-tools' import type { ContentBlock, MessageId } from '@deepseek-ai/dsh-llm' +import { durablePromptContent } from '@deepseek-ai/dsh-llm' import type { Agent } from '@deepseek-ai/dsh-agent' import type { SessionId } from '@deepseek-ai/dsh-session' import { Remote, TypertRemoteService } from '@deepseek-ai/dsh-typert-protocol' @@ -418,13 +419,15 @@ export class SubagentRuntime extends TypertRemoteService { * validated browser zone on the accepted message. Success identifies the * message the child's FIFO inbox accepted; later execution is independent of * this call. + * Image parts are admitted and persisted through the attachment store + * before delivery, and the child's model must accept image input. * @param request - durable address, minted identity, content, and optional browser zone. * @param signal - carrier cancellation, owning the call until inbox acceptance. * @returns the accepted message's inbox identity. * @throws {TypertRemoteFailure} `bad-request`, `invalid-time-zone`, - * `subagent-parent-unavailable`, `subagent-not-resumable`, - * `subagent-unauthorized`, `subagent-delivery-unavailable`, `cancelled`, or - * `internal`. + * `attachment-error`, `subagent-parent-unavailable`, + * `subagent-not-resumable`, `subagent-unauthorized`, + * `subagent-delivery-unavailable`, `cancelled`, or `internal`. */ @Remote('prompt') async prompt(request: SubagentPromptRequest, signal: AbortSignal): Promise { @@ -453,8 +456,10 @@ export class SubagentRuntime extends TypertRemoteService { rpcId: request.requestId, ...(canonicalTimeZone === undefined ? {} : { clientTimeZone: canonicalTimeZone }), } - const content: ContentBlock[] = [...request.content] try { + // Admission precedes delivery: image parts become durable references + // here, so the child inbox only ever accepts Host-persisted attachments. + const content: ContentBlock[] = await durablePromptContent(this.ctx.attachments, request.content) return { messageId: await this.followup(parent, childSessionId, content, { source, signal }) } } catch (error: unknown) { return rejectPrompt(error, childSessionId, signal) diff --git a/packages/subagent/subagent/tests/continuation.spec.ts b/packages/subagent/subagent/tests/continuation.spec.ts index 9508065197..505da10ff2 100644 --- a/packages/subagent/subagent/tests/continuation.spec.ts +++ b/packages/subagent/subagent/tests/continuation.spec.ts @@ -523,6 +523,74 @@ describe('SubagentRuntime.startContinuable', () => { }) }) +describe('continuable image follow-ups', () => { + const imageBlock = { + type: 'image' as const, + attachment: { + attachmentId: 'att-1' as never, mediaType: 'image/png' as const, bytes: 1, width: 1, height: 1, + }, + } + + it('refuses an image follow-up when the child model declines image input, leaving no partial message', async () => { + const { ctx, parent } = await setup([textResponse('child work')]) + const started = await ctx.subagents.startContinuable(startSpec(parent)) + await waitNoActivation(ctx, started.childId) + const resolve = vi.spyOn(ctx.llm, 'resolveModelInfo') + .mockResolvedValue({ inputModalities: ['text'] } as never) + + await expect(ctx.subagents.followup(parent, started.childId, [ + { type: 'text' as const, text: 'see this' }, + imageBlock, + ], { source: { kind: 'user' }, signal: testSignal })) + .rejects.toMatchObject({ code: 'MODEL_DOES_NOT_SUPPORT_IMAGES' }) + + expect(resolve).toHaveBeenCalledWith('mock', 'mock', testSignal) + const loaded = await ctx.sessionPersistence.load(started.childId) + expect(hasUserText(loaded.events, 'see this')).toBe(false) + await drainManager(ctx) + }) + + it('delivers an image follow-up when the child model accepts image input', async () => { + const { ctx, parent } = await setup([textResponse('child work'), textResponse('image reply')]) + const started = await ctx.subagents.startContinuable(startSpec(parent)) + await waitNoActivation(ctx, started.childId) + vi.spyOn(ctx.llm, 'resolveModelInfo') + .mockResolvedValue({ inputModalities: ['text', 'image'] } as never) + + await ctx.subagents.followup(parent, started.childId, [ + { type: 'text' as const, text: 'compare' }, + imageBlock, + ], { source: { kind: 'user' }, signal: testSignal }) + await waitNoActivation(ctx, started.childId) + + const loaded = await ctx.sessionPersistence.load(started.childId) + const delivered = loaded.events.find(event => event.type === 'user/message' + && event.data.content.some(block => block.type === 'image')) + expect(delivered?.type === 'user/message' && delivered.data.content).toEqual([ + { type: 'text', text: 'compare' }, + imageBlock, + ]) + await drainManager(ctx) + }) + + it('defers to the text-only projection when the descriptor declares no model route', async () => { + const { ctx } = await setup([]) + const routeless = ctx.agentLoop.create(SessionId('routeless-image'), {}) + const started = await ctx.subagents.startContinuable(startSpec(routeless)) + await waitNoActivation(ctx, started.childId) + const resolve = vi.spyOn(ctx.llm, 'resolveModelInfo') + + // Acceptance is the success boundary: with no declared route there is no + // model to refuse against, so the image message enters the child inbox. + await ctx.subagents.followup(routeless, started.childId, [imageBlock], { + source: { kind: 'user' }, signal: testSignal, + }) + + expect(resolve).not.toHaveBeenCalled() + await drainManager(ctx) + }) +}) + describe('SubagentRuntime.followup residency routing', () => { it('fails a cold follow-up when Session query is unavailable', async () => { const { ctx, parent } = await setupWith(new MockAdapter([]), { diff --git a/packages/subagent/subagent/tests/control.spec.ts b/packages/subagent/subagent/tests/control.spec.ts index dd04d549b1..99c534dd49 100644 --- a/packages/subagent/subagent/tests/control.spec.ts +++ b/packages/subagent/subagent/tests/control.spec.ts @@ -5,6 +5,7 @@ import { describe, expect, it, vi } from 'vitest' import { Context } from '@deepseek-ai/cordis' +import { AttachmentError } from '@deepseek-ai/dsh-attachment' import type { MessageId } from '@deepseek-ai/dsh-llm' import { SessionId } from '@deepseek-ai/dsh-session' import SubagentRuntime, { @@ -19,6 +20,8 @@ const OTHER = SessionId('other') const BROKEN = SessionId('broken') const REQUEST_ID = 'req-1' as SubagentPromptRequestId const signal = new AbortController().signal +/** Durable-reference base for the fake store; per-test ids and media types override. */ +const IMAGE_REF = { attachmentId: 'att', mediaType: 'image/png', bytes: 2, width: 1, height: 1 } /** The runtime plus a programmable live-Agent registry, omitted to compose none. */ async function bench(live?: Record) { @@ -158,14 +161,70 @@ describe('subagent prompt Remote', () => { expect(followup).not.toHaveBeenCalled() }) - it('forwards non-text content blocks without narrowing them', async () => { - const { subagents } = await bench({ [PARENT]: { status: 'idle' } }) + it('admits ordered image parts into durable references before delivery', async () => { + const { ctx, subagents } = await bench({ [PARENT]: { status: 'idle' } }) + const saveImages = vi.fn(async (inputs: readonly { mediaType: string }[]) => + inputs.map((input, index) => ({ ...IMAGE_REF, attachmentId: `att-${index}`, mediaType: input.mediaType }))) + ctx.provide('attachments', { saveImages } as never) const followup = vi.spyOn(subagents, 'followup').mockResolvedValue('m-content' as MessageId) - const content = [{ type: 'reasoning' as const, text: 'retain this block' }] + const content = [ + { type: 'text' as const, text: 'before' }, + { type: 'image' as const, mediaType: 'image/png' as const, data: 'aGk=' }, + { type: 'text' as const, text: 'after' }, + ] await expect(subagents.prompt({ ...promptRequest(), content }, signal)) .resolves.toEqual({ messageId: 'm-content' }) - expect(followup.mock.calls[0]?.[2]).toEqual(content) + expect(followup.mock.calls[0]?.[2]).toEqual([ + { type: 'text', text: 'before' }, + { type: 'image', attachment: { ...IMAGE_REF, attachmentId: 'att-0', mediaType: 'image/png' } }, + { type: 'text', text: 'after' }, + ]) + }) + + it('maps a refused image batch to attachment-error and delivers nothing', async () => { + const { ctx, subagents } = await bench({ [PARENT]: { status: 'idle' } }) + ctx.provide('attachments', { + saveImages: async () => { + throw new AttachmentError('Image batch exceeds the configured image-count limit.', 'TOO_MANY_IMAGES') + }, + } as never) + const followup = vi.spyOn(subagents, 'followup') + + await expect(subagents.prompt({ + ...promptRequest(), + content: [{ type: 'image' as const, mediaType: 'image/png' as const, data: 'aGk=' }], + }, signal)).rejects.toMatchObject({ + failure: { code: 'attachment-error', details: { reason: 'TOO_MANY_IMAGES' } }, + }) + expect(followup).not.toHaveBeenCalled() + }) + + it('maps non-canonical base64 to attachment-error without touching the store', async () => { + const { ctx, subagents } = await bench({ [PARENT]: { status: 'idle' } }) + const saveImages = vi.fn() + ctx.provide('attachments', { saveImages } as never) + const followup = vi.spyOn(subagents, 'followup') + + await expect(subagents.prompt({ + ...promptRequest(), + content: [{ type: 'image' as const, mediaType: 'image/png' as const, data: 'not base64!' }], + }, signal)).rejects.toMatchObject({ + failure: { code: 'attachment-error', details: { reason: 'INVALID_IMAGE_BASE64' } }, + }) + expect(saveImages).not.toHaveBeenCalled() + expect(followup).not.toHaveBeenCalled() + }) + + it('maps a text-only child model refusal to attachment-error', async () => { + const { subagents } = await bench({ [PARENT]: { status: 'idle' } }) + vi.spyOn(subagents, 'followup').mockRejectedValue( + new SubagentError('Model "text-only" does not support image input.', 'MODEL_DOES_NOT_SUPPORT_IMAGES'), + ) + + await expect(subagents.prompt(promptRequest(), signal)).rejects.toMatchObject({ + failure: { code: 'attachment-error', details: { reason: 'MODEL_DOES_NOT_SUPPORT_IMAGES' } }, + }) }) it('delivers the content under the caller-minted identity and canonical browser zone', async () => { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index f271832de3..432036f26a 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -8239,6 +8239,9 @@ importers: '@deepseek-ai/dsh-agent-presets': specifier: workspace:^ version: link:../../preset/agent-presets + '@deepseek-ai/dsh-attachment': + specifier: workspace:^ + version: link:../../attachment/attachment '@deepseek-ai/dsh-brand': specifier: workspace:^ version: link:../../util/brand diff --git a/snapshots/web/queued-image/delivered.expected.md b/snapshots/web/queued-image/delivered.expected.md new file mode 100644 index 0000000000..332f611048 --- /dev/null +++ b/snapshots/web/queued-image/delivered.expected.md @@ -0,0 +1,82 @@ +- banner: + - navigation "Session hierarchy": + - button "Reply with a one-sentence description" [disabled] + - img + - text: Standard mode + - button "Session log": + - text: Session log + - img + - tablist: + - tab "Chat" [selected] + - tab "Trajectory" +- navigation "Turn navigation": + - button "Jump to turn 1" + - button "Jump to turn 2" + - button "Jump to turn 3" +- button "System prompt": + - img + - img + - text: System prompt +- text: Reply with a one-sentence description of event sourcing, then stop. {{clock}} +- button "Copy": + - img +- button "Context injection @deepseek-ai/dsh-system-prompt": + - img + - img + - text: Context injection @deepseek-ai/dsh-system-prompt +- paragraph: partial +- text: Stopped +- button "Copy": + - img +- button "Good response": + - img +- button "Bad response": + - img +- button "Branch into a new conversation": + - img +- text: {{clock}} Ran for {{duration}} TTFT {{duration}} +- button "queued.png, click to view original": + - img "queued.png" +- text: Compare with this screenshot {{clock}} +- button "Copy": + - img +- button "Think The user is asking for a one-sentence description of event sourcing. This is a straightforward knowledge question that doesn't require any skill loading or tool calls.": + - img + - img + - text: Think The user is asking for a one-sentence description of event sourcing. This is a straightforward knowledge question that doesn't require any skill loading or tool calls. +- paragraph: Event sourcing is a pattern where all changes to an application's state are stored as an immutable, append-only sequence of events, rather than persisting only the current state, enabling full auditability, temporal queries, and event-driven architectures. +- button "Copy": + - img +- button "Good response": + - img +- button "Bad response": + - img +- button "Branch into a new conversation": + - img +- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s Continue with the queued comparison {{clock}} +- button "Copy": + - img +- button "Think The user is asking for a one-sentence description of event sourcing. This is a straightforward knowledge question that doesn't require any skill loading or tool calls.": + - img + - img + - text: Think The user is asking for a one-sentence description of event sourcing. This is a straightforward knowledge question that doesn't require any skill loading or tool calls. +- paragraph: Event sourcing is a pattern where all changes to an application's state are stored as an immutable, append-only sequence of events, rather than persisting only the current state, enabling full auditability, temporal queries, and event-driven architectures. +- button "Copy": + - img +- button "Good response": + - img +- button "Bad response": + - img +- button "Branch into a new conversation": + - img +- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s +- textbox "Message or run a task... / commands, @ files or sessions" +- button "Commands": + - img +- 'button "Access mode, current: Workspace Write"': Workspace Write +- button "Select model, current DeepSeek-V4-Flash": + - text: DeepSeek-V4-Flash + - img +- button "6% of context used" +- button "Send message" [disabled] +- text: 3 turns · 3 steps LLM {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 99% Input 15.6K tok · Output 158 tok diff --git a/snapshots/web/queued-image/queued.expected.md b/snapshots/web/queued-image/queued.expected.md new file mode 100644 index 0000000000..462b6d03d1 --- /dev/null +++ b/snapshots/web/queued-image/queued.expected.md @@ -0,0 +1,42 @@ +- banner: + - navigation "Session hierarchy": + - button "Reply with a one-sentence description" [disabled] + - img + - text: Standard mode + - button "Session log": + - text: Session log + - img + - tablist: + - tab "Chat" [selected] + - tab "Trajectory" +- button "System prompt": + - img + - img + - text: System prompt +- text: Reply with a one-sentence description of event sourcing, then stop. {{clock}} +- button "Copy": + - img +- button "Context injection @deepseek-ai/dsh-system-prompt": + - img + - img + - text: Context injection @deepseek-ai/dsh-system-prompt +- paragraph: partial +- status: Deep diving... +- list: + - listitem: + - img "Queued message image" + - text: Compare with this screenshot + - button "Edit queued message" [disabled]: + - img + - button "Remove queued message": + - img + - button "Steer queued message": + - img +- textbox "Cmd/Ctrl+Enter steers all queued messages" +- button "Commands": + - img +- 'button "Access mode, current: Workspace Write"': Workspace Write +- button "Select model, current DeepSeek-V4-Flash": + - text: DeepSeek-V4-Flash + - img +- button "Stop generating" diff --git a/snapshots/web/queued-image/snapshot.yml b/snapshots/web/queued-image/snapshot.yml new file mode 100644 index 0000000000..69c492d2b2 --- /dev/null +++ b/snapshots/web/queued-image/snapshot.yml @@ -0,0 +1,9 @@ +version: 1 +scenario: queued-image +profile: web +composition: web-default +recording: authored +header: + class: web-default +session: + source: ../live-interactions/session.jsonl diff --git a/tsconfig.host.json b/tsconfig.host.json index b91be8e6ca..9b1f3cde4e 100644 --- a/tsconfig.host.json +++ b/tsconfig.host.json @@ -60,6 +60,7 @@ "apps/web/tests/markdown-cjk-strong.e2e.ts", "apps/web/tests/markdown-inline-code-links.e2e.ts", "apps/web/tests/queue-actions.e2e.ts", + "apps/web/tests/queue-image.e2e.ts", "apps/web/tests/skill-invocation-policy.e2e.ts", "apps/web/tests/skill-user-invoke.e2e.ts", "apps/web/tests/permission-policy-context.e2e.ts",