fix(llm-deepseek): fall back when Files resolution fails

This commit is contained in:
creatixchu
2026-08-21 18:14:46 +08:00
parent e30d92a03e
commit 1b389798dc
19 changed files with 695 additions and 87 deletions
@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-08-21-deepseek-files-inline-fallback.md
2026-08-21-deepseek-files-inline-fallback.md: c58b3e2257b426f1b5df8a4d6952e890a2bd2982
2026-08-21-deepseek-files-inline-fallback.zh.md: 34625c6250d52a73ccaac3e33adbd2ed099aab5b
@@ -0,0 +1,37 @@
# Agent Note: Recover DeepSeek image requests from Files resolution failures
Status: implemented
English | [中文](2026-08-21-deepseek-files-inline-fallback.zh.md)
## Problem
The direct DeepSeek vision route uses provider file ids so repeated requests do not resend image bytes. An unavailable, unsupported, or stalled Files endpoint can prevent chat before the model request begins even though the same endpoint still accepts inline image data. A fallback that retains the 128MiB Files budget would exceed the inline request-body limit, while a fallback that independently transforms images could send different pixels from the failed file-id attempt.
## Decision
Files remains the preferred transport. Each request-image file resolution has the configurable `filesApiTimeoutMs` deadline, one minute by default and always below `streamIdleTimeoutMs`. Successful resolutions refresh the outer idle watchdog. Caller cancellation and the outer stream deadline remain terminal outcomes.
A file resolution failure discards the transient file parts assembled for that chat attempt and rebuilds the complete image request with base64 data URLs. Every retained image uses the already prepared deterministic `RequestImageAttachment`; the fallback performs no additional decode, resize, or encode, and a chat request never mixes file ids with inline images. Upload mappings committed before a later image fails remain available to later requests. The next request tries Files again, so recovery requires no process-wide outage state.
Inline fallback has a separate base64-expanded high watermark, `maxInlineRequestImageBytes`, of 20MiB by default. `inlineImageOffloadByteQuantum` defaults to 10MiB, so crossing the high watermark advances the deterministic oldest-image prefix to the next 10MiB removal boundary. The existing 600-image bound and count quantum still apply. File mode retains its 128MiB high watermark and 64MiB removal quantum.
Provider chat errors keep their existing classifications. A stale file id is invalidated, re-uploaded, and retried once. If that replacement resolution fails, the permitted retry uses the inline representation. A generic chat failure does not switch transports because it does not establish that Files resolution failed.
## Alternatives considered
**Send inline images first.** Rejected because successful Files uploads allow deterministic request bytes to be reused across turns without repeating base64 in every request.
**Mix resolved file ids with inline images after one upload fails.** Rejected because the request would still depend on the failing Files service and would have two independent image budgets.
**Apply the 128MiB Files bound to inline fallback.** Rejected because base64 expands the payload and can exceed the chat request-body limit. The 20MiB budget leaves space for JSON, text history, and tools.
**Remember an outage and bypass Files on later requests.** Rejected because a process-local circuit state introduces recovery timing and shared failure state. Retrying Files on the next request detects service recovery without another timer.
## Verification
Serializer tests cover file and data-URL representations over the same request versions, all supported media types, tool-result placement, and 20-to-10 base64 offload. Adapter tests cover immediate resolution failure, failure after a partial set of file ids, deadline-triggered fallback, stale-id replacement failure, all-inline request bodies, caller cancellation without fallback, and generic chat failure without a transport switch. Configuration tests cover both inline bounds and the Files deadline relationship.
## Consequences
A Files outage no longer prevents an image chat that fits the inline budget. Fallback repeats image bytes and may omit more history than file mode because its limit is lower. A request can leave successful uploads behind when a later image fails, but their indexed mappings are reusable and do not change the chat body sent by the fallback. Explicit file-management operations continue to expose their own failures.
@@ -0,0 +1,37 @@
# Agent Note: DeepSeek Files 解析失败时恢复图片请求
Status: implemented
[English](2026-08-21-deepseek-files-inline-fallback.md) | 中文
## Problem
DeepSeek 官方视觉路由使用提供方文件 ID,使重复请求不必再次发送图片字节。如果 Files 端点不可用、不受支持或一直不返回,chat 会在模型请求开始前失败,即使同一端点仍接受内联图片数据。沿用 128MiB Files 预算的回退会超过内联请求体上限,独立转换图片的回退则可能发送与失败 file ID 尝试不同的像素。
## Decision
Files 仍是首选传输方式。每张请求图片的文件解析都有可配置的 `filesApiTimeoutMs` 时限,默认一分钟,且始终小于 `streamIdleTimeoutMs`。每次成功解析都会刷新外层 idle watchdog。调用方取消和外层流时限仍直接终止请求。
文件解析失败后,适配器会丢弃为该次 chat 尝试组装的临时文件块,并用 base64 data URL 重新组装完整图片请求。每张保留图片都复用已经准备好的确定性 `RequestImageAttachment`;回退不会再次解码、缩放或编码,同一个 chat 请求也不会混用 file ID 和内联图片。较早图片在后续图片失败前已经提交的上传映射会保留,供之后请求使用。下一次请求会重新尝试 Files,因此不需要保存进程级故障状态。
内联回退使用独立的 base64 膨胀后高水位,`maxInlineRequestImageBytes` 默认为 20MiB。`inlineImageOffloadByteQuantum` 默认为 10MiB,因此越过高水位时,确定性的最旧图片前缀会推进到下一个 10MiB 移除边界。现有 600 张图片上限和数量步长继续生效。文件模式继续使用 128MiB 高水位和 64MiB 移除步长。
提供方 chat 错误继续使用现有分类。失效 file ID 会被清除、重新上传并重试一次。如果替换解析失败,这次允许的重试会使用内联表示。普通 chat 错误不能证明 Files 解析失败,因此不会切换传输方式。
## Alternatives considered
**优先发送内联图片。** 不采用,因为 Files 上传成功后可以跨轮次复用确定性的请求字节,不必在每次请求中重复 base64。
**某次上传失败后混用已解析 file ID 和内联图片。** 不采用,因为请求仍依赖发生故障的 Files 服务,而且需要同时处理两套图片预算。
**把 128MiB Files 上限用于内联回退。** 不采用,因为 base64 会扩大负载,并可能超过 chat 请求体上限。20MiB 预算会为 JSON、文本历史和工具留下空间。
**记住故障,并在后续请求中跳过 Files。** 不采用,因为进程级状态会引入恢复时间和共享故障状态。下一次请求重新尝试 Files,可以在无需新增计时器的情况下发现服务恢复。
## Verification
序列化测试覆盖相同请求版本的文件和 data URL 表示、全部支持的媒体类型、工具结果位置,以及 20MiB 到 10MiB 的 base64 offload。适配器测试覆盖立即解析失败、部分 file ID 成功后的失败、时限触发的回退、失效 ID 替换失败、全内联请求体、调用方取消时不回退,以及普通 chat 错误不切换传输方式。配置测试覆盖两项内联预算和 Files 时限关系。
## Consequences
符合内联预算的图片 chat 不会再因 Files 故障而失败。回退会重复发送图片字节,而且由于上限更低,可能比文件模式省略更多历史。后续图片失败时,请求可能留下较早图片的成功上传,但这些索引映射可以复用,也不会改变回退发送的 chat 请求体。显式文件管理操作继续暴露自身错误。
@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-08-20-unified-image-request-pipeline.md
2026-08-20-unified-image-request-pipeline.md: 6a3bae8a970677c32bbfb7966d2bc13d4e504804
2026-08-20-unified-image-request-pipeline.zh.md: 10a4aed0b5ca9168c6a6ee4ec0258a210b50d531
2026-08-20-unified-image-request-pipeline.md: ada15d540539977c631e359ffdc7baa4fa84c78e
2026-08-20-unified-image-request-pipeline.zh.md: 85c9a1f837d82cba2bc62b30402433f50c873cbe
@@ -34,7 +34,7 @@ Every retained request image is preceded by its complete attachment id and actua
### DeepSeek Files lifecycle
The direct `deepseek-official` adapter uploads every retained request version through the OpenAI-compatible Files API and sends only `file_id` content blocks. There is no inline fallback. The default catalog advertises `deepseek-v4-flash-vision-exp` as image-capable. Uploaded ids are indexed by endpoint and API-key scope plus `variantId`. Uploads request seven days by default and record the returned `expires_at`; a mapping with no more than one hour remaining is replaced without a preceding retrieve call. The index never stores the API key.
The direct `deepseek-official` adapter normally uploads every retained request version through the OpenAI-compatible Files API and sends `file_id` content blocks. A [bounded inline fallback](../bug-fix/2026-08-21-deepseek-files-inline-fallback.md) sends the same deterministic request versions when file resolution fails. The default catalog advertises `deepseek-v4-flash-vision-exp` as image-capable. Uploaded ids are indexed by endpoint and API-key scope plus `variantId`. Uploads request seven days by default and record the returned `expires_at`; a mapping with no more than one hour remaining is replaced without a preceding retrieve call. The index never stores the API key.
An upload is indexed only after the response returns a complete file object, matching byte count, and `expires_at`. A missing or inconsistent response leaves no local mapping, so a later request uploads again. Concurrent upload resolution for one scoped `variantId` shares one provider operation; one waiter cannot cancel another, and the upload stops when every waiter has cancelled. A malformed upload index is an empty cache and is replaced on the next successful upload; filesystem I/O failures remain errors. If chat reports expired, deleted, missing, or invalid ids and names one or more ids used by the request, only those mappings are removed. A stale-file response without a specific id removes every mapping used by that chat attempt. The affected request bytes are uploaded again and chat is retried once. A second stale rejection clears the mappings identified by its response and returns the error without a third chat attempt. One upload quota error first lists the configured number of oldest harness-owned `dsh-` files, then deletes that collected set and retries once; deleting after pagination keeps provider cursors valid. Public file operations expose list, retrieve, delete, one-variant release, and namespace-wide release. Every Files request carries the shared Harness `User-Agent`. The client enforces the documented 128MiB upload limit, 32MiB chat-image limit, 10,000-file and 25GiB quotas, and one-hour to 30-day expiry range.
@@ -52,7 +52,7 @@ Historical attachment objects that later disappear or fail integrity verificatio
**Treat PNG as a screenshot and reject 16-bit PNG.** File format does not reveal pixel complexity, and 16-bit RGB/RGBA is a convertible sample depth rather than an unsupported image type. Pixel sampling and post-conversion probes give the required facts.
**Keep DeepSeek data URLs.** Inline base64 repeats bytes on every request and caps usable image history by request-body size. Files API references reuse uploaded deterministic request bytes and provide explicit expiry and deletion.
**Keep DeepSeek data URLs as the primary transport.** Inline base64 repeats bytes on every request and caps usable image history by request-body size. Files API references reuse uploaded deterministic request bytes and provide explicit expiry and deletion; the bounded fallback uses data URLs only when file resolution fails.
**Trust a locally indexed file id indefinitely.** Remote expiry, deletion, and lost upload responses make local and provider state diverge. Response-directed invalidation and one re-upload recover without an unbounded retry loop; an ambiguous stale-file response must invalidate every file used by that attempt because it provides no safe exact target.
@@ -62,8 +62,8 @@ Historical attachment objects that later disappear or fail integrity verificatio
## Verification
Package tests generate 16-bit RGB and RGBA PNG fixtures, prove 8-bit conversion and clean 8-bit passthrough, retain alpha under byte pressure, distinguish high-frequency and ordinary photos from low-color graphics, stop lazy encoding after the first fitting candidate, cover square and wide 640,000-pixel projections, enforce 1MiB request bytes, singleflight equal variants and uploads without shared-cancellation leaks, bound transform concurrency, preserve cache and upload identity, skip attachment reads for conservatively offloaded history, prepare batches once, reject inconsistent Files responses, refresh near-expiry ids without retrieve, recover once from single-id, multiple-id, and ambiguous stale responses, paginate before quota deletion, normalize provider diagnostics, project text-only history, and share normal/compaction request bytes. Keyless assembled snapshots cover the real tool schemas and image request path. A credentialed test uses the built-in `deepseek-official` route and its configured endpoint, never a custom provider entry.
Package tests generate 16-bit RGB and RGBA PNG fixtures, prove 8-bit conversion and clean 8-bit passthrough, retain alpha under byte pressure, distinguish high-frequency and ordinary photos from low-color graphics, stop lazy encoding after the first fitting candidate, cover square and wide 640,000-pixel projections, enforce 1MiB request bytes, singleflight equal variants and uploads without shared-cancellation leaks, bound transform concurrency, preserve cache and upload identity, skip attachment reads for conservatively offloaded history, prepare batches once, reject inconsistent Files responses, refresh near-expiry ids without retrieve, recover once from single-id, multiple-id, and ambiguous stale responses, fall back to bounded all-inline requests after file resolution failure, paginate before quota deletion, normalize provider diagnostics, project text-only history, and share normal/compaction request bytes. Keyless assembled snapshots cover the real tool schemas and image request path. A credentialed test uses the built-in `deepseek-official` route and its configured endpoint, never a custom provider entry.
## Consequences
Normalized attachments consume up to the independent local safety cap, while request caches and remote Files consume additional derived storage. Deterministic identities and singleflight make that work reusable across turns and sessions sharing the same DSH home. Two simultaneous transforms reduce batch latency while increasing peak RSS relative to serial execution; deployments with tighter memory can set the limit to one. Encoder or transform-version changes create new future identities without rewriting existing history. DeepSeek image requests now depend on Files API availability; bounded stale-id recovery handles inconsistent remote state, while a general Files outage remains a visible request failure. Missing or corrupt durable attachments still require the separate quarantine design.
Normalized attachments consume up to the independent local safety cap, while request caches and remote Files consume additional derived storage. Deterministic identities and singleflight make that work reusable across turns and sessions sharing the same DSH home. Two simultaneous transforms reduce batch latency while increasing peak RSS relative to serial execution; deployments with tighter memory can set the limit to one. Encoder or transform-version changes create new future identities without rewriting existing history. DeepSeek image requests prefer Files reuse; bounded stale-id recovery handles inconsistent remote state, while file-resolution failures use the smaller inline budget. Missing or corrupt durable attachments still require the separate quarantine design.
@@ -34,7 +34,7 @@ Status: implemented
### DeepSeek Files 生命周期
直接 `deepseek-official` 适配器通过 OpenAI 兼容 Files API 上传每张保留的请求版本,发送 `file_id` 内容块,不提供内联回退。默认 catalog 把 `deepseek-v4-flash-vision-exp` 公布为支持图片。上传 ID 按端点和 API key 作用域以及 `variantId` 写入索引。上传默认请求 7 天有效期,并记录返回的 `expires_at`;本地映射剩余时间不超过一小时时会直接替换,不会先查询远端文件。索引绝不存储 API key。
直接 `deepseek-official` 适配器通常通过 OpenAI 兼容 Files API 上传每张保留的请求版本,发送 `file_id` 内容块。文件解析失败时,[有界内联回退](../bug-fix/2026-08-21-deepseek-files-inline-fallback.zh.md)会发送相同的确定性请求版本。默认 catalog 把 `deepseek-v4-flash-vision-exp` 公布为支持图片。上传 ID 按端点和 API key 作用域以及 `variantId` 写入索引。上传默认请求 7 天有效期,并记录返回的 `expires_at`;本地映射剩余时间不超过一小时时会直接替换,不会先查询远端文件。索引绝不存储 API key。
只有上传响应返回完整文件对象、匹配的字节数和 `expires_at` 时,上传结果才会写入索引。缺失或不一致的响应不会留下本地映射,后续请求会重新上传。同一作用域和 `variantId` 的并发解析共享一次提供方上传;单个等待方无法取消其他等待方,全部等待方取消时才会停止上传。格式损坏的上传索引按空缓存处理,并在下一次成功上传时替换;文件系统 I/O 失败仍是错误。如果 chat 报告 ID 已过期、删除、缺失或无效,并指出本次请求使用的一个或多个 ID,适配器只删除这些映射。如果响应只说明文件状态失效而没有指出具体 ID,适配器会删除该次 chat 使用的全部映射。受影响的请求字节会重新上传,chat 只重试一次。第二次仍报告文件失效时,适配器会按响应清理映射并返回错误,不会发起第三次 chat。一次上传配额错误会先列出配置数量的最旧 `dsh-` 文件,再删除收集到的文件并重试一次;分页完成后才删除,避免游标失效。公开文件操作提供列表、查询、删除、单个变体释放和整个作用域释放。每个 Files 请求都携带 Harness 的共享 `User-Agent`。客户端执行文档规定的 Files 单次上传 128MiB、chat 单图 32MiB、10,000 个文件、25GiB,以及一小时到 30 天有效期限制。
@@ -52,7 +52,7 @@ Status: implemented
**把 PNG 当作截图,并拒绝 16-bit PNG。** 文件格式不能说明像素复杂度,16-bit RGB/RGBA 是可转换位深,不是不支持的图片类型。像素采样和转换后探测能提供所需事实。
**继续向 DeepSeek 发送 data URL。** 内联 base64 会在每次请求中重复字节,并按请求正文大小限制可用图片历史。Files API 引用会复用上传后的确定性请求字节,并提供显式有效期和删除操作。
** DeepSeek data URL 作为首选传输方式** 内联 base64 会在每次请求中重复字节,并按请求正文大小限制可用图片历史。Files API 引用会复用上传后的确定性请求字节,并提供显式有效期和删除操作;有界回退只在文件解析失败时使用 data URL
**永久信任本地索引中的文件 ID。** 远端过期、删除和上传响应丢失会使本地与提供方状态不一致。按响应失效和一次重新上传可以恢复,同时避免无界重试;响应没有给出可安全使用的精确目标时,必须使该次请求使用的全部文件失效。
@@ -62,8 +62,8 @@ Status: implemented
## Verification
包测试会生成 16-bit RGB 和 RGBA PNG fixture,验证 8-bit 转换与干净 8-bit 字节直通、字节压力下保留透明通道、区分高频和普通照片与低色数图形、首个候选合规后停止编码、正方形和宽屏 640,000 像素投影、请求字节不超过 1MiB、相同变体与上传 singleflight 且不会共享取消、变换并发上限、缓存与上传身份、跳过已保守 offload 的历史附件读取、批量只准备一次、Files 响应不一致、进入刷新余量时不查询远端并更新 ID、单个 ID、多个 ID 和模糊失效响应只恢复一次、删除配额文件前完成分页、规范化提供方诊断、纯文本投影,以及普通请求与压缩共享请求字节。无需密钥的组装快照覆盖真实工具 schema 和图片请求路径。使用凭据的测试只使用内置 `deepseek-official` 路由及其已配置端点,不使用自定义提供方条目。
包测试会生成 16-bit RGB 和 RGBA PNG fixture,验证 8-bit 转换与干净 8-bit 字节直通、字节压力下保留透明通道、区分高频和普通照片与低色数图形、首个候选合规后停止编码、正方形和宽屏 640,000 像素投影、请求字节不超过 1MiB、相同变体与上传 singleflight 且不会共享取消、变换并发上限、缓存与上传身份、跳过已保守 offload 的历史附件读取、批量只准备一次、Files 响应不一致、进入刷新余量时不查询远端并更新 ID、单个 ID、多个 ID 和模糊失效响应只恢复一次、文件解析失败后回退到有界全内联请求、删除配额文件前完成分页、规范化提供方诊断、纯文本投影,以及普通请求与压缩共享请求字节。无需密钥的组装快照覆盖真实工具 schema 和图片请求路径。使用凭据的测试只使用内置 `deepseek-official` 路由及其已配置端点,不使用自定义提供方条目。
## Consequences
持久规范化附件最多占用独立的本地安全上限,请求缓存和远端 Files 还会占用额外派生存储。确定性身份和 singleflight 使这些成本可以被共享同一 DSH home 的轮次和会话复用。同时执行两个变换会降低批次延迟,但峰值 RSS 高于串行执行;内存更紧张的部署可以把上限设为 1。编码器或变换策略版本变化会为未来内容产生新身份,不会改写已有历史。DeepSeek 图片请求现在依赖 Files API 可用性;有界的陈旧 ID 恢复会处理远端状态不一致,一般 Files 故障仍会成为可见请求失败。缺失或损坏的持久附件仍需要单独的隔离设计。
持久规范化附件最多占用独立的本地安全上限,请求缓存和远端 Files 还会占用额外派生存储。确定性身份和 singleflight 使这些成本可以被共享同一 DSH home 的轮次和会话复用。同时执行两个变换会降低批次延迟,但峰值 RSS 高于串行执行;内存更紧张的部署可以把上限设为 1。编码器或变换策略版本变化会为未来内容产生新身份,不会改写已有历史。DeepSeek 图片请求优先复用 Files;有界的陈旧 ID 恢复会处理远端状态不一致,文件解析失败则使用较小的内联预算。缺失或损坏的持久附件仍需要单独的隔离设计。
+2 -2
View File
@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/config-catalog.md
config-catalog.md: 552c09c08abef2cab957d2a8caab9412cb4522e5
config-catalog.zh.md: fc1993bf4c4f21ec6ec85341f4ce09a04b6a8b66
config-catalog.md: de340b7ffade528301b4538b0553bc11ec969985
config-catalog.zh.md: eb17ee89fd7860cc0774073bea542aca315ce652
+8 -2
View File
@@ -940,12 +940,18 @@ export interface Config {
streamIdleTimeoutMs?: number
/** Maximum accumulated file-referenced image bytes per chat request (default 128 MiB). */
maxRequestFilesBytes?: number
/** Maximum number of file-referenced images per chat request (default 600). */
/** Maximum accumulated base64 image payload after Files API fallback (default 20 MiB). */
maxInlineRequestImageBytes?: number
/** Maximum number of represented images per chat request (default 600). */
maxImagesPerRequest?: number
/** Raw-byte removal step after the request exceeds its file bound (default 64 MiB). */
imageOffloadByteQuantum?: number
/** Base64-byte removal step after inline fallback exceeds its bound (default 10 MiB). */
inlineImageOffloadByteQuantum?: number
/** Image-count removal step after the request exceeds its count bound (default 20). */
imageOffloadCountQuantum?: number
/** Maximum duration of one request-image Files API resolution (default one minute). */
filesApiTimeoutMs?: number
/** Explicit lifetime assigned to each uploaded image (default seven days). */
fileExpiresAfterSeconds?: number
/** Remaining lifetime below which an indexed file is replaced (default one hour). */
@@ -981,7 +987,7 @@ export interface DeepSeekCatalogModel {
Depends on: [`ModelModality`](../packages/llm/llm/src/index.ts) · [`RetryPolicyConfig`](../packages/llm/llm/src/index.ts)
Source: [`packages/llm/llm-deepseek/src/index.ts:100`](../packages/llm/llm-deepseek/src/index.ts)
Source: [`packages/llm/llm-deepseek/src/index.ts:106`](../packages/llm/llm-deepseek/src/index.ts)
<a id="deepseek-aidsh-llm-pi-ai"></a>
+8 -2
View File
@@ -942,12 +942,18 @@ export interface Config {
streamIdleTimeoutMs?: number
/** Maximum accumulated file-referenced image bytes per chat request (default 128 MiB). */
maxRequestFilesBytes?: number
/** Maximum number of file-referenced images per chat request (default 600). */
/** Maximum accumulated base64 image payload after Files API fallback (default 20 MiB). */
maxInlineRequestImageBytes?: number
/** Maximum number of represented images per chat request (default 600). */
maxImagesPerRequest?: number
/** Raw-byte removal step after the request exceeds its file bound (default 64 MiB). */
imageOffloadByteQuantum?: number
/** Base64-byte removal step after inline fallback exceeds its bound (default 10 MiB). */
inlineImageOffloadByteQuantum?: number
/** Image-count removal step after the request exceeds its count bound (default 20). */
imageOffloadCountQuantum?: number
/** Maximum duration of one request-image Files API resolution (default one minute). */
filesApiTimeoutMs?: number
/** Explicit lifetime assigned to each uploaded image (default seven days). */
fileExpiresAfterSeconds?: number
/** Remaining lifetime below which an indexed file is replaced (default one hour). */
@@ -983,7 +989,7 @@ export interface DeepSeekCatalogModel {
依赖:[`ModelModality`](../packages/llm/llm/src/index.ts) · [`RetryPolicyConfig`](../packages/llm/llm/src/index.ts)
来源:[`packages/llm/llm-deepseek/src/index.ts:100`](../packages/llm/llm-deepseek/src/index.ts)
来源:[`packages/llm/llm-deepseek/src/index.ts:106`](../packages/llm/llm-deepseek/src/index.ts)
<a id="deepseek-aidsh-llm-pi-ai"></a>
+41 -1
View File
@@ -702,9 +702,10 @@ defineAcpSnapshotSuite({
hasPwsh,
})
it('pins native DeepSeek Files image offload in the request sent by the assembled app', async () => {
it('pins native DeepSeek Files offload and inline fallback in assembled requests', async () => {
const requests: Record<string, unknown>[] = []
const fileRequests: Array<{ method: string; path: string; bytes: number }> = []
let rejectFiles = false
const server = createServer((request: IncomingMessage, response: ServerResponse) => {
const chunks: Buffer[] = []
request.on('data', (chunk: Buffer) => { chunks.push(chunk) })
@@ -723,6 +724,12 @@ it('pins native DeepSeek Files image offload in the request sent by the assemble
const file = form.get('file')
if (!(file instanceof Blob)) throw new Error('snapshot Files upload omitted file')
fileRequests.push({ method: 'POST', path: url.pathname, bytes: file.size })
if (rejectFiles) {
response.writeHead(503, { 'content-type': 'application/json' }).end(JSON.stringify({
error: { message: 'Files temporarily unavailable' },
}))
return
}
const createdAt = Math.floor(Date.now() / 1_000)
response.writeHead(200, { 'content-type': 'application/json' }).end(JSON.stringify({
id: 'file-api-snapshot-1',
@@ -861,6 +868,39 @@ it('pins native DeepSeek Files image offload in the request sent by the assemble
],
},
])
rejectFiles = true
const fallback = await runScenario(input, {
agent: AGENT,
mode: 'record',
configPath: IMAGE_OFFLOAD_CONFIG,
fixtureFile: join(SNAPSHOTS_DIR, 'image-offload-request', 'session.jsonl'),
workspaceDir: join(SNAPSHOTS_DIR, 'read-image', 'workspace'),
env: {
DSH_SNAPSHOT_API_KEY: 'snapshot-fallback-key',
DSH_SNAPSHOT_BASE_URL: `http://127.0.0.1:${address.port}`,
},
})
expect(fallback.stderr).toBe('')
expect(fileRequests).toEqual([
{ method: 'POST', path: '/files', bytes: 69 },
{ method: 'POST', path: '/files', bytes: 69 },
])
expect(requests).toHaveLength(3)
const fallbackMessages = requests[2]?.messages as { content?: unknown }[] | undefined
const fallbackInput = fallbackMessages?.find(message => JSON.stringify(message.content).includes('[image omitted'))
expect(fallbackInput?.content).toEqual([
{ type: 'text', text: 'Compare the older image ' },
{ type: 'text', text: OFFLOADED_IMAGE_TEXT },
{ type: 'text', text: ' with the newer image ' },
{
type: 'text',
text: '\nImage sha256:b1ff9c8ea3a780bad09b346c423d2d0e46815926879b18e841d928376a946640; '
+ 'request image 1x1px.',
},
{ type: 'image_url', image_url: { url: `data:image/png;base64,${image}` } },
{ type: 'text', text: ', then use read_image on red.png and reply with DONE.' },
])
} finally {
await new Promise<void>(resolve => server.close(() => { resolve() }))
}
+2 -2
View File
@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/llm/llm-deepseek/README.md
README.md: d17d520c2444d8a0195d997f4df4ff5e0f05befd
README.zh.md: cc823897894102df0dc1da17478eee6ba7ebd21d
README.md: 7a22955565027b30677e46a80a8b719bc7e61917
README.zh.md: db1669509956d651dcb8948e1191a17cf9a0bfee
+10 -5
View File
@@ -21,9 +21,12 @@ The package root exposes the Cordis plugin contract and `DeepSeekAdapter`; wire
maxTokens: 256000 # optional positive per-request output cap; this is the default
streamIdleTimeoutMs: 300000 # optional; positive finite Node timer delay; five-minute default
maxRequestFilesBytes: 134217728 # optional positive integer; 128 MiB raw request-image default
maxInlineRequestImageBytes: 20971520 # base64 fallback high watermark; 20 MiB default
maxImagesPerRequest: 600 # provider request image-count limit
imageOffloadByteQuantum: 67108864 # oldest-image removal advances in 64 MiB steps
inlineImageOffloadByteQuantum: 10485760 # fallback removal advances in 10 MiB steps
imageOffloadCountQuantum: 20 # count overflow advances in 20-image steps
filesApiTimeoutMs: 60000 # per-image Files resolution deadline; below streamIdleTimeoutMs
fileExpiresAfterSeconds: 604800 # uploaded image lifetime; 1 hour to 30 days
fileRefreshMarginSeconds: 3600 # replace ids with less lifetime remaining
fileQuotaCleanupBatch: 100 # oldest harness-owned files deleted before one quota retry
@@ -49,11 +52,13 @@ The package root exposes the Cordis plugin contract and `DeepSeekAdapter`; wire
The plugin registers the single provider route `deepseek-official` together with its resolved `retryPolicy`; omission resolves to normal mode with five retries. A request selects it with `provider: deepseek-official`; its `model` is passed through as the wire `model` string, so changing DeepSeek models does not require lifecycle-time registration. Omitting `models` advertises `deepseek-v4-flash`, `deepseek-v4-pro`, and the image-capable `deepseek-v4-flash-vision-exp`, each with a 1,000,000-token context window; an explicit list replaces those defaults, while `models: []` advertises none. Catalog entries are exposed through `ctx.llm.listModels('deepseek-official')` for clients such as ACP editors and the Web selector, but remain advisory: unlisted model ids still pass through unchanged as text-only routes. An omitted entry name defaults to its id, and omitted `inputModalities` means `text` only.
An image-capable catalog entry declares `inputModalities: [text, image]` and may set `imagePixelBudget`, `imageMaxBytes`, or `imageDetail: low`. The ordinary default is 640,000 total pixels and 1MiB encoded bytes; low detail defaults to 512 by 512 total pixels. The attachment store scales by `min(1, sqrt(pixelBudget / (width * height)))` and rounds inward to keep the pixel count at or below the hard cap, so a 2048 by 1024 normalized attachment becomes about 1130 by 565 instead of a forced square. Request encoders run lazily: low-color images try PNG (palette only without alpha) then WebP 85 and 80, other alpha images try WebP 85 then 80, and other opaque images try JPEG 85 then 80; dimensions shrink only when both quality attempts exceed 1MiB. Concurrent generation of one `variantId` shares one transform. A caller can cancel its own wait without interrupting other waiters; the transform stops when no waiter remains. The adapter uploads the exact derived request bytes through `POST /files` and sends `{type: "file", file_id}` blocks. It never falls back to an inline data URL. Every retained image is preceded by stable text naming the complete attachment id and actual request dimensions. User, tool-result, agent-loop, compaction, and direct `ctx.llm.stream` requests all use this projection. Text-only routes receive stable attachment placeholders while durable history keeps its image references.
An image-capable catalog entry declares `inputModalities: [text, image]` and may set `imagePixelBudget`, `imageMaxBytes`, or `imageDetail: low`. The ordinary default is 640,000 total pixels and 1MiB encoded bytes; low detail defaults to 512 by 512 total pixels. The attachment store scales by `min(1, sqrt(pixelBudget / (width * height)))` and rounds inward to keep the pixel count at or below the hard cap, so a 2048 by 1024 normalized attachment becomes about 1130 by 565 instead of a forced square. Request encoders run lazily: low-color images try PNG (palette only without alpha) then WebP 85 and 80, other alpha images try WebP 85 then 80, and other opaque images try JPEG 85 then 80; dimensions shrink only when both quality attempts exceed 1MiB. Concurrent generation of one `variantId` shares one transform. A caller can cancel its own wait without interrupting other waiters; the transform stops when no waiter remains. The adapter normally uploads the exact derived request bytes through `POST /files` and sends `{type: "file", file_id}` blocks. A failed or timed-out file-id resolution rebuilds the whole chat request with those same request versions as base64 data URLs; one request never mixes file ids and inline images. Every retained image is preceded by stable text naming the complete attachment id and actual request dimensions. User, tool-result, agent-loop, compaction, and direct `ctx.llm.stream` requests all use this projection. Text-only routes receive stable attachment placeholders while durable history keeps its image references.
`maxRequestFilesBytes` and `maxImagesPerRequest` bound the retained request versions at 128MiB and 600 images by default. The byte and count quanta must not exceed their corresponding bounds. Before attachment reads, the adapter uses each route's request-version byte cap as a conservative upper bound and removes the oldest over-budget prefix; only retained normalized attachments are read and transformed. Exact derived lengths are checked again without restoring omitted images. When the byte bound is crossed, the oldest prefix advances past the next 64MiB boundary; 129 one-megabyte images remove the oldest 65 and retain 64MiB, and that prefix stays unchanged until durable history exceeds 192MiB. Count overflow advances independently in `imageOffloadCountQuantum` steps. Removed images become the fixed model-visible placeholder `[image omitted to keep the request within its image limit; older images are omitted first. If this image is still needed, read its file again when a path is available; otherwise ask the user to attach it again.]`. This high-watermark projection avoids changing an old request prefix after every new image.
Uploaded ids are indexed below `DSH_HOME` by endpoint/API-key scope and request `variantId`. The variant covers the normalized attachment id, transform version, route pixel and byte budgets, and encoder parameters, so Files API and inline-capable adapters refer to the same deterministic bytes. Uploads request a seven-day lifetime by default and store the server's `expires_at`. A local mapping with no more than one hour remaining is replaced before use; the adapter does not retrieve every remote file before chat. If chat reports expired, deleted, missing, or invalid file ids and names one or more ids used by the request, the adapter removes exactly those mappings. If the provider identifies stale file state without naming an id, it removes every file mapping used by that chat attempt. It then uploads the affected request versions again and retries chat once. A second stale-file rejection clears the mappings identified by that response and is returned without a third chat attempt. An upload response without a complete file object, matching byte count, and `expires_at` is never indexed; a later request therefore uploads again instead of trusting inconsistent local state. A malformed local upload index is treated as an empty cache and replaced by the next successful upload; permission and filesystem I/O failures still fail the request.
Inline fallback has an independent base64 budget. `maxInlineRequestImageBytes` defaults to 20MiB and `inlineImageOffloadByteQuantum` to 10MiB, so a history of 21 one-megabyte base64 payloads removes the oldest 11 and retains 10MiB. The calculation uses base64-expanded lengths. The prepared request versions are reused byte-for-byte; fallback does not decode or compress an image again. Successful mappings created before a later image fails remain indexed for future requests.
Uploaded ids are indexed below `DSH_HOME` by endpoint/API-key scope and request `variantId`. The variant covers the normalized attachment id, transform version, route pixel and byte budgets, and encoder parameters, so Files API and inline fallback refer to the same deterministic bytes. Uploads request a seven-day lifetime by default and store the server's `expires_at`. A local mapping with no more than one hour remaining is replaced before use; the adapter does not retrieve every remote file before chat. If chat reports expired, deleted, missing, or invalid file ids and names one or more ids used by the request, the adapter removes exactly those mappings. If the provider identifies stale file state without naming an id, it removes every file mapping used by that chat attempt. It then uploads the affected request versions again and retries chat once. A second stale-file rejection clears the mappings identified by that response and is returned without a third chat attempt. An upload response without a complete file object, matching byte count, and `expires_at` is never indexed; a later request therefore uploads again instead of trusting inconsistent local state. A malformed local upload index is treated as an empty cache and replaced by the next successful upload. File resolution, including local index access and remote upload, has a per-image one-minute deadline by default; it must remain below `streamIdleTimeoutMs`. Each successful resolution refreshes the outer idle watchdog. Any resolution failure switches that request to inline mode, while explicit public file-management operations continue to report their own failures.
Concurrent resolution of one scoped `variantId` shares one Files upload with waiter-local cancellation. One quota upload failure first paginates and collects the configured number of oldest `dsh-` files, then deletes that set before one upload retry. `DeepSeekFilesClient.delete`, `DeepSeekFileStore.release`, and `releaseAll` expose explicit remote-space reclamation. The current provider limits represented by this package are 128MiB per Files upload, 32MiB per chat-referenced image, 10,000 stored files, and 25GiB per API key; the default 1MiB request version remains below the two per-file limits.
@@ -65,7 +70,7 @@ The same exact-model result exposes ordered `off`, `low`, `high`, and `max` effo
`thinking: disabled` is a deployment lock that publishes only `off` with `off` as its default. Omitting `reasoningEffort` or configuring it as `off` is valid; configuring `low`, `high`, or `max` fails plugin loading, and a direct per-request attempt to enable thinking fails before network I/O. A request with `GenerateOptions.purpose: 'session-title'` also forces thinking disabled and omits the already-resolved effort, reserving its bounded output for visible title text without changing conversation or compaction defaults.
`streamIdleTimeoutMs` bounds each outstanding provider read, including the initial `fetch`, without counting time the consumer spends between chunks. DeepSeek SSE comments rearm an outstanding read as transport activity but never become `StreamChunk` values or session-log events. One stable abort signal reaches the request and body reader for the whole call; expiry stops the transport and throws `LlmError('TIMEOUT')`, while an earlier caller abort throws `LlmError('ABORTED')`. The adapter normally makes one chat request per `stream()` call and makes a second only for the stale-file recovery described above. It registers the configured retry policy as provider metadata, and `dsh-llm-retry` separately executes that policy at durable agent-step boundaries.
`streamIdleTimeoutMs` bounds each outstanding provider read, including the initial `fetch`, without counting time the consumer spends between chunks. DeepSeek SSE comments and successful file resolutions rearm an outstanding read as transport activity but never become `StreamChunk` values or session-log events. One stable abort signal reaches the request and body reader for the whole call; expiry stops the transport and throws `LlmError('TIMEOUT')`, while an earlier caller abort throws `LlmError('ABORTED')`. The adapter normally makes one chat request per `stream()` call and makes a second only for stale-file recovery. A file-resolution failure before the first chat sends one inline request. If replacement resolution fails after a stale-file response, the inline request is the one permitted retry. It registers the configured retry policy as provider metadata, and `dsh-llm-retry` separately executes that policy at durable agent-step boundaries.
## Dynamic configuration (settings + credentials)
@@ -104,7 +109,7 @@ Non-2xx responses throw `LlmError` with stable codes: `AUTH` (401/403), `QUOTA`
#### What the model sees
The selected DeepSeek model receives the harness system prompt, message history, tool schemas, stop sequences, and call config. The vision model receives retained user and tool-result images as Files API references beside stable attachment handles and request-image dimensions; an over-budget older image is represented by the documented placeholder. Reasoning content from a prior assistant turn is passed back verbatim, whether or not that turn called a tool.
The selected DeepSeek model receives the harness system prompt, message history, tool schemas, stop sequences, and call config. The vision model normally receives retained user and tool-result images as Files API references beside stable attachment handles and request-image dimensions; a Files resolution failure sends all retained images as inline data URLs instead. An over-budget older image is represented by the documented placeholder. Reasoning content from a prior assistant turn is passed back verbatim, whether or not that turn called a tool.
#### Token effect
@@ -134,4 +139,4 @@ Loop-retained response blocks append to the next request and preserve its earlie
- **`tool_choice` is not mapped** — not part of the core vocabulary (MVP cut, shared with the pi-ai twin).
- **Requests use raw `fetch`, not `@cordisjs/plugin-http`** — no shared proxy/interception configuration; adoption is deferred until a second adapter wants it (`TODO(http)`).
- **Plugin-added content block types are skipped** — core text and supported image blocks are serialized, and empty tool output crosses the wire as the literal `(no output)`.
- **Images are input-only durable attachments** — direct external URLs and assistant image output are not supported; DeepSeek input uses the Files API.
- **Images are input-only durable attachments** — direct external URLs and assistant image output are not supported; DeepSeek input normally uses the Files API and uses inline base64 only for per-request recovery.
+10 -5
View File
@@ -21,9 +21,12 @@ harness LLM(大语言模型)seam 的 DeepSeek chat-completions 适配器:
maxTokens: 256000 # optional positive per-request output cap; this is the default
streamIdleTimeoutMs: 300000 # optional; positive finite Node timer delay; five-minute default
maxRequestFilesBytes: 134217728 # optional positive integer; 128 MiB raw request-image default
maxInlineRequestImageBytes: 20971520 # base64 fallback high watermark; 20 MiB default
maxImagesPerRequest: 600 # provider request image-count limit
imageOffloadByteQuantum: 67108864 # oldest-image removal advances in 64 MiB steps
inlineImageOffloadByteQuantum: 10485760 # fallback removal advances in 10 MiB steps
imageOffloadCountQuantum: 20 # count overflow advances in 20-image steps
filesApiTimeoutMs: 60000 # per-image Files resolution deadline; below streamIdleTimeoutMs
fileExpiresAfterSeconds: 604800 # uploaded image lifetime; 1 hour to 30 days
fileRefreshMarginSeconds: 3600 # replace ids with less lifetime remaining
fileQuotaCleanupBatch: 100 # oldest harness-owned files deleted before one quota retry
@@ -49,11 +52,13 @@ harness LLM(大语言模型)seam 的 DeepSeek chat-completions 适配器:
该插件注册唯一提供方路由 `deepseek-official`,并一同注册解析后的 `retryPolicy`;省略时会解析为 normal 模式并重试五次。请求使用 `provider: deepseek-official` 选择该路由;其 `model` 会作为协议 `model` 字符串原样传递,因此更改 DeepSeek 模型不需要生命周期时注册。省略 `models` 会公布 `deepseek-v4-flash``deepseek-v4-pro` 与支持图片输入的 `deepseek-v4-flash-vision-exp`,三者的上下文窗口均为 1,000,000 token;显式列表会替换这些默认值,`models: []` 则不公布任何模型。Catalog 配置项通过 `ctx.llm.listModels('deepseek-official')` 公开给 ACPAgent Client Protocol)编辑器和 Web 选择器等客户端,但仍只提供建议:未列出模型 id 仍原样传递,并按纯文本路由处理。省略配置项 name 默认为其 id,省略 `inputModalities` 则表示仅支持 `text`
支持图片的 catalog 配置项声明 `inputModalities: [text, image]`,并可设置 `imagePixelBudget``imageMaxBytes``imageDetail: low`。普通默认值为总像素 640,000、编码字节 1MiBlow detail 的默认总像素为 512×512。附件存储按 `min(1, sqrt(pixelBudget / (width * height)))` 缩放,并向预算内取整,确保总像素不超过硬上限。因此 2048×1024 规范化附件会得到约 1130×565 的请求版本,而不会被强制变成正方形。请求编码按需执行:低色数图片先尝试 PNG,只有不带 alpha 通道时才使用 palette,再尝试质量 85 和 80 的 WebP;其他透明图片依次尝试质量 85 和 80 的 WebP;其他非透明图片依次尝试质量 85 和 80 的 JPEG。两个质量档均超过 1MiB 时才缩小尺寸。同一 `variantId` 的并发生成共享一次变换。调用方可以单独取消等待,不会中断其他等待方;没有等待方时才会停止变换。适配器通过 `POST /files` 上传确切的派生请求字节,再发送 `{type: "file", file_id}`,不会回退到内联 data URL。每张保留图片前都有稳定文本,写明完整附件 ID 和实际请求尺寸。User、工具结果、agent loop、压缩和直接 `ctx.llm.stream` 请求都使用该投影。纯文本路由会收到稳定的附件占位文本,持久历史继续保留图片引用。
支持图片的 catalog 配置项声明 `inputModalities: [text, image]`,并可设置 `imagePixelBudget``imageMaxBytes``imageDetail: low`。普通默认值为总像素 640,000、编码字节 1MiBlow detail 的默认总像素为 512×512。附件存储按 `min(1, sqrt(pixelBudget / (width * height)))` 缩放,并向预算内取整,确保总像素不超过硬上限。因此 2048×1024 规范化附件会得到约 1130×565 的请求版本,而不会被强制变成正方形。请求编码按需执行:低色数图片先尝试 PNG,只有不带 alpha 通道时才使用 palette,再尝试质量 85 和 80 的 WebP;其他透明图片依次尝试质量 85 和 80 的 WebP;其他非透明图片依次尝试质量 85 和 80 的 JPEG。两个质量档均超过 1MiB 时才缩小尺寸。同一 `variantId` 的并发生成共享一次变换。调用方可以单独取消等待,不会中断其他等待方;没有等待方时才会停止变换。适配器通常通`POST /files` 上传确切的派生请求字节,再发送 `{type: "file", file_id}`。File ID 解析失败或超时后,适配器会用相同请求版本的 base64 data URL 重新组装整个 chat 请求;同一请求不会混用 file ID 和内联图片。每张保留图片前都有稳定文本,写明完整附件 ID 和实际请求尺寸。User、工具结果、agent loop、压缩和直接 `ctx.llm.stream` 请求都使用该投影。纯文本路由会收到稳定的附件占位文本,持久历史继续保留图片引用。
`maxRequestFilesBytes``maxImagesPerRequest` 限制请求中保留的请求版本,默认值分别为 128MiB 和 600 张。字节和数量步长不得超过对应上限。读取附件前,适配器以路由的请求版本字节上限作为保守上界,移除超预算的最旧前缀,只读取并转换保留的规范化附件。系统随后用确切派生长度再次检查,但不会重新加入已省略图片。字节数越过上限时,被移除的最旧前缀会越过下一个 64MiB 边界。由 1MiB 图片组成的历史达到 129MiB 时会移除最旧的 65 张并保留 64MiB;直到持久历史超过 192MiB,这个前缀才再次变化。图片数量超限时则按 `imageOffloadCountQuantum` 独立递增。移除的图片会变成固定模型可见占位文本 `[image omitted to keep the request within its image limit; older images are omitted first. If this image is still needed, read its file again when a path is available; otherwise ask the user to attach it again.]`。这种定量投影不会因每新增一张图片就改写较早的请求前缀。
上传 ID 按端点和 API key 作用域以及请求 `variantId` 记录在 `DSH_HOME` 下。变体身份覆盖规范化附件 ID、变换策略版本、路由像素和字节预算及编码参数,因此 Files API 和支持内联的适配器引用同一份确定性字节。上传默认请求 7 天有效期,并保存服务端返回的 `expires_at`。本地映射剩余时间不超过一小时时会在使用前替换;适配器不会在每次 chat 前查询远端文件。如果 chat 报告文件 ID 已过期、删除、缺失或无效,并指出本次请求使用的一个或多个 ID,适配器只删除这些映射。如果响应只说明文件状态失效而没有指出 ID,适配器会删除该次 chat 使用的全部文件映射。随后重新上传受影响的请求版本,并重试一次 chat。第二次 chat 仍报告文件失效时,适配器会按该响应清理映射并返回错误,不会发起第三次 chat。上传响应若没有完整文件对象、匹配的字节数和 `expires_at`,就不会写入索引;后续请求会再次上传,而不是信任不一致的本地状态。本地上传索引格式损坏时按空缓存处理,并由下一次成功上传替换;权限和文件系统 I/O 错误仍使请求失败
内联回退使用独立的 base64 预算。`maxInlineRequestImageBytes` 默认为 20MiB`inlineImageOffloadByteQuantum` 默认为 10MiB,因此由 21 个 1MiB base64 负载组成的历史会移除最旧的 11 个并保留 10MiB。计算使用 base64 膨胀后的长度。系统逐字节复用已经准备好的请求版本;回退不会再次解码或压缩图片。前面图片已经成功写入的上传映射会保留,供后续请求复用
上传 ID 按端点和 API key 作用域以及请求 `variantId` 记录在 `DSH_HOME` 下。变体身份覆盖规范化附件 ID、变换策略版本、路由像素和字节预算及编码参数,因此 Files API 和内联回退引用同一份确定性字节。上传默认请求 7 天有效期,并保存服务端返回的 `expires_at`。本地映射剩余时间不超过一小时时会在使用前替换;适配器不会在每次 chat 前查询远端文件。如果 chat 报告文件 ID 已过期、删除、缺失或无效,并指出本次请求使用的一个或多个 ID,适配器只删除这些映射。如果响应只说明文件状态失效而没有指出 ID,适配器会删除该次 chat 使用的全部文件映射。随后重新上传受影响的请求版本,并重试一次 chat。第二次 chat 仍报告文件失效时,适配器会按该响应清理映射并返回错误,不会发起第三次 chat。上传响应若没有完整文件对象、匹配的字节数和 `expires_at`,就不会写入索引;后续请求会再次上传,而不是信任不一致的本地状态。本地上传索引格式损坏时按空缓存处理,并由下一次成功上传替换。文件解析包括本地索引访问和远端上传,默认每张图片的时限为一分钟,且必须小于 `streamIdleTimeoutMs`。每次成功解析都会刷新外层 idle watchdog。任何解析失败都会把该请求切换到内联模式;显式公共文件管理操作仍会报告自身错误。
同一作用域和 `variantId` 的并发解析共享一次 Files 上传,每个等待方可以单独取消。一次上传配额错误会先分页收集配置数量的最旧 `dsh-` 文件,再删除这些文件并重试一次上传。`DeepSeekFilesClient.delete``DeepSeekFileStore.release``releaseAll` 提供主动远端空间回收。本包记录的当前提供方限制为 Files 单次上传 128MiB、chat 单图引用 32MiB、每个 API key 最多 10,000 个文件和 25GiB;默认 1MiB 请求版本低于两个单文件上限。
@@ -65,7 +70,7 @@ harness LLM(大语言模型)seam 的 DeepSeek chat-completions 适配器:
`thinking: disabled` 是部署锁定:它只公布 `off`,并以 `off` 为默认值。省略 `reasoningEffort` 或将其配置为 `off` 均有效;配置 `low``high``max` 会使插件加载失败,直接按请求启用思考也会在网络 I/O 前失败。携带 `GenerateOptions.purpose: 'session-title'` 的请求也会强制禁用思考并省略已解析的推理强度,将有界输出保留给可见标题文本,不改变会话或压缩(compaction)默认值。
`streamIdleTimeoutMs` 会限制每次未完成提供方读取,包括初始 `fetch`,但不计入消费方在分片间花费的时间。DeepSeek SSE 注释会作为传输活动使尚未完成的读取重新布防,但绝不会成为 `StreamChunk` 值或会话日志事件。同一个稳定的 abort 信号会在整个调用期间传递给请求与 body reader;过期会停止传输并抛出 `LlmError('TIMEOUT')`,较早的调用方 abort 则抛出 `LlmError('ABORTED')`。适配器通常每次 `stream()` 调用发起一次 chat 请求,只有上述失效文件恢复会发起第二次。适配器把已配置重试策略注册为提供方元数据,再由 `dsh-llm-retry` 在持久化的 agent(智能体)步骤边界单独执行该策略。
`streamIdleTimeoutMs` 会限制每次未完成提供方读取,包括初始 `fetch`,但不计入消费方在分片间花费的时间。DeepSeek SSE 注释和成功的文件解析会作为传输活动使尚未完成的读取重新计时,但绝不会成为 `StreamChunk` 值或会话日志事件。同一个稳定的 abort 信号会在整个调用期间传递给请求与 body reader;过期会停止传输并抛出 `LlmError('TIMEOUT')`,较早的调用方 abort 则抛出 `LlmError('ABORTED')`。适配器通常每次 `stream()` 调用发起一次 chat 请求,只有失效文件恢复会发起第二次。首次 chat 前的文件解析失败会发送一次内联请求。如果失效文件响应后的替换解析失败,该内联请求就是唯一允许的重试。适配器把已配置重试策略注册为提供方元数据,再由 `dsh-llm-retry` 在持久化的 agent(智能体)步骤边界单独执行该策略。
## 动态配置(settings + credentials
@@ -104,7 +109,7 @@ DeepSeek 请求身份独立于应用归因。凭据解析成功后,每个提
#### 模型看到的内容
所选 DeepSeek 模型会收到 harness 系统提示词、消息历史、工具 schema、stop sequence 和调用配置。视觉模型通过 Files API 引用收到保留的 user 与工具结果图片,旁边带有稳定附件句柄和请求图片尺寸;超出上限的较旧图片由已记录的占位文本表示。之前 assistant 轮次的推理内容会原文回传,无论该轮次是否调用了工具。
所选 DeepSeek 模型会收到 harness 系统提示词、消息历史、工具 schema、stop sequence 和调用配置。视觉模型通常通过 Files API 引用收到保留的 user 与工具结果图片,旁边带有稳定附件句柄和请求图片尺寸;Files 解析失败时,所有保留图片改用内联 data URL。超出上限的较旧图片由已记录的占位文本表示。之前 assistant 轮次的推理内容会原文回传,无论该轮次是否调用了工具。
#### Token 影响
@@ -134,4 +139,4 @@ loop 保留的响应块会追加到下一个请求,并保留其较早可复用
- **未映射 `tool_choice`**:它不属于核心词汇(MVP 取舍,与 pi-ai twin 共享)。
- **请求使用原始 `fetch`,而非 `@cordisjs/plugin-http`**:没有共享 proxy/拦截配置;采用暂缓到第二个适配器需要该功能时(`TODO(http)`)。
- **会跳过插件添加的内容块类型**:核心文本与支持的图片块会被序列化,空工具输出会以字面 `(no output)` 通过协议发送。
- **图片是仅输入的持久附件**:不支持直接外部 URL 和 assistant 图片输出;DeepSeek 图片输入使用 Files API。
- **图片是仅输入的持久附件**:不支持直接外部 URL 和 assistant 图片输出;DeepSeek 图片输入通常使用 Files API,仅在单次请求恢复时使用内联 base64
+77 -22
View File
@@ -28,7 +28,7 @@ import type {
RequestImageAttachment,
} from '@deepseek-ai/dsh-attachment'
import type { CredentialRef } from '@deepseek-ai/dsh-credentials'
import { idleWatchdog, timeoutOf } from '@deepseek-ai/dsh-timeout'
import { deadline, idleWatchdog, timeoutOf } from '@deepseek-ai/dsh-timeout'
import type { AnonymousUserId } from '@deepseek-ai/dsh-anonymous-user-id'
import { serializeRequest, serializeRequestWithImages } from './serialize.ts'
import type { ImageWireLocation, RequestDefaults } from './serialize.ts'
@@ -37,7 +37,7 @@ import type { DeepSeekFilePolicy } from './file-store.ts'
import type { DeepSeekFileId } from './file-id.ts'
import { parseSse } from './sse.ts'
import { translate } from './translate.ts'
import type { WireError } from './types.ts'
import type { WireError, WireRequest } from './types.ts'
/** One optional model entry advertised by the direct-fetch adapter. */
export interface DeepSeekCatalogModel {
@@ -89,12 +89,18 @@ export interface DeepSeekConnectionOptions {
streamIdleTimeoutMs: number
/** Maximum accumulated file-referenced image bytes in one request. */
maxRequestFilesBytes: number
/** Maximum number of file-referenced images in one request. */
/** Maximum accumulated base64 image payload after Files API fallback. */
maxInlineRequestImageBytes: number
/** Maximum number of represented images in one request. */
maxImagesPerRequest: number
/** Raw-byte removal step after the file-reference bound is exceeded. */
imageOffloadByteQuantum: number
/** Base64-byte removal step after the inline fallback bound is exceeded. */
inlineImageOffloadByteQuantum: number
/** Image-count removal step after the count bound is exceeded. */
imageOffloadCountQuantum: number
/** Maximum duration of one request-image Files API resolution. */
filesApiTimeoutMs: number
/** Upload expiry, refresh, and quota-recovery policy. */
filePolicy: DeepSeekFilePolicy
/** Provider-owned model-request retry policy, already resolved. */
@@ -128,6 +134,8 @@ export const DEFAULT_CONTEXT_WINDOW = 1_000_000
export const DEFAULT_MAX_TOKENS = 256_000
/** Default bound on accumulated file-referenced image bytes per request. */
export const DEFAULT_MAX_REQUEST_FILES_BYTES = 128 * 1024 * 1024
/** Default bound on accumulated base64 image payload after Files API fallback. */
export const DEFAULT_MAX_INLINE_REQUEST_IMAGE_BYTES = 20 * 1024 * 1024
/** Provider request image-count limit. */
export const DEFAULT_MAX_IMAGES_PER_REQUEST = 600
/** Total-pixel budget matching DeepSeek's normal vision projection. */
@@ -138,6 +146,8 @@ export const DEFAULT_LOW_DETAIL_IMAGE_PIXEL_BUDGET = 512 * 512
export const DEFAULT_REQUEST_IMAGE_MAX_BYTES = 1024 * 1024
/** Deterministic raw-byte removal step. */
export const DEFAULT_IMAGE_OFFLOAD_BYTE_QUANTUM = 64 * 1024 * 1024
/** Deterministic base64-byte removal step after Files API fallback. */
export const DEFAULT_INLINE_IMAGE_OFFLOAD_BYTE_QUANTUM = 10 * 1024 * 1024
/** Deterministic image-count removal step. */
export const DEFAULT_IMAGE_OFFLOAD_COUNT_QUANTUM = 20
/** Default explicit lifetime for uploaded images. */
@@ -146,7 +156,10 @@ export const DEFAULT_FILE_EXPIRY_SECONDS = 7 * 24 * 60 * 60
export const DEFAULT_FILE_REFRESH_MARGIN_SECONDS = 60 * 60
/** Default number of oldest harness-owned files removed on quota recovery. */
export const DEFAULT_FILE_QUOTA_CLEANUP_BATCH = 100
/** Default deadline for resolving one request image through the Files API. */
export const DEFAULT_FILES_API_TIMEOUT_MS = 60_000
const STREAM_IDLE_TIMEOUT_CODE = 'LLM_STREAM_IDLE_TIMEOUT'
const FILES_API_TIMEOUT_CODE = 'DEEPSEEK_FILES_API_TIMEOUT'
const OFF_REASONING_EFFORT = ReasoningEffortId('off')
const LOW_REASONING_EFFORT = ReasoningEffortId('low')
const HIGH_REASONING_EFFORT = ReasoningEffortId('high')
@@ -161,6 +174,14 @@ const OFF_ONLY_REASONING_EFFORTS = [
{ id: OFF_REASONING_EFFORT, name: 'Off' },
] as const
/** Marks a failed file-id resolution that may be retried as an inline request. */
class FileResolutionFailure extends Error {
constructor(cause: unknown) {
super('DeepSeek Files API could not resolve a request image.', { cause })
this.name = 'FileResolutionFailure'
}
}
function collectImageRefs(
content: readonly ContentBlock[],
refs: Map<AttachmentId, ImageAttachmentRef>,
@@ -494,7 +515,7 @@ export class DeepSeekAdapter extends LlmAdapter {
apiKey: string,
userId: AnonymousUserId,
attachments: AttachmentStore | undefined,
onComment: () => void,
onActivity: () => void,
): AsyncIterable<StreamChunk> {
const headers = {
'authorization': `Bearer ${apiKey}`,
@@ -525,27 +546,58 @@ export class DeepSeekAdapter extends LlmAdapter {
const requestImages = attachments === undefined || model === undefined
? new Map<AttachmentId, RequestImageAttachment>()
: await prepareRequestImages(requestOptions, attachments, model, signal)
for (let fileAttempt = 0; fileAttempt < 2; fileAttempt += 1) {
let representation: 'file' | 'base64' = 'file'
let fileAttempt = 0
while (true) {
const usedFiles: UsedRequestFile[] = []
const body = attachments === undefined
? serializeRequest(requestOptions, connection.defaults)
: await serializeRequestWithImages(requestOptions, {
let body: WireRequest
if (attachments === undefined) {
body = serializeRequest(requestOptions, connection.defaults)
} else if (representation === 'base64') {
body = await serializeRequestWithImages(requestOptions, {
representation: { kind: 'base64' },
requestImages,
resolveFileId: async (version, _block, location) => {
const resolved = await this.files.ensureUploaded(
version,
fileConnection,
connection.filePolicy,
signal,
)
usedFiles.push({ version, fileId: resolved.record.fileId, location })
return resolved.record.fileId
},
maxRequestFilesBytes: connection.maxRequestFilesBytes,
maxRequestImageBytes: connection.maxInlineRequestImageBytes,
maxImagesPerRequest: connection.maxImagesPerRequest,
byteQuantum: connection.imageOffloadByteQuantum,
byteQuantum: connection.inlineImageOffloadByteQuantum,
countQuantum: connection.imageOffloadCountQuantum,
}, connection.defaults)
} else {
try {
body = await serializeRequestWithImages(requestOptions, {
representation: {
kind: 'file',
resolveFileId: async (version, _block, location) => {
using filesDeadline = deadline(signal, connection.filesApiTimeoutMs, FILES_API_TIMEOUT_CODE)
let resolved: Awaited<ReturnType<DeepSeekFileStore['ensureUploaded']>>
try {
resolved = await this.files.ensureUploaded(
version,
fileConnection,
connection.filePolicy,
filesDeadline.signal,
)
} catch (error: unknown) {
if (signal.aborted) throw error
throw new FileResolutionFailure(error)
}
onActivity()
usedFiles.push({ version, fileId: resolved.record.fileId, location })
return resolved.record.fileId
},
},
requestImages,
maxRequestImageBytes: connection.maxRequestFilesBytes,
maxImagesPerRequest: connection.maxImagesPerRequest,
byteQuantum: connection.imageOffloadByteQuantum,
countQuantum: connection.imageOffloadCountQuantum,
}, connection.defaults)
} catch (error: unknown) {
if (!(error instanceof FileResolutionFailure)) throw error
representation = 'base64'
continue
}
}
const payload = JSON.stringify(body)
// TODO(http): adopt the Cordis HTTP service when shared transport configuration
@@ -586,7 +638,10 @@ export class DeepSeekAdapter extends LlmAdapter {
await Promise.all(staleMappings(usedFiles, detail).map(file => (
this.files.invalidate(file.version, file.fileId, fileConnection)
)))
if (fileAttempt === 0) continue
if (fileAttempt === 0) {
fileAttempt += 1
continue
}
}
if (response.status === 400 && usedFiles.length > 0 && providerRejectedNormalizedImage(detail)) {
message = normalizedImageDiagnostic(usedFiles, message, detail)
@@ -604,7 +659,7 @@ export class DeepSeekAdapter extends LlmAdapter {
throw new LlmError('DeepSeek API returned no response body', 'EMPTY_RESPONSE')
}
yield* translate(parseSse(response.body, onComment))
yield* translate(parseSse(response.body, onActivity))
return
}
}
+42 -1
View File
@@ -25,9 +25,12 @@ import {
DEFAULT_FILE_EXPIRY_SECONDS,
DEFAULT_FILE_QUOTA_CLEANUP_BATCH,
DEFAULT_FILE_REFRESH_MARGIN_SECONDS,
DEFAULT_FILES_API_TIMEOUT_MS,
DEFAULT_IMAGE_OFFLOAD_BYTE_QUANTUM,
DEFAULT_IMAGE_OFFLOAD_COUNT_QUANTUM,
DEFAULT_INLINE_IMAGE_OFFLOAD_BYTE_QUANTUM,
DEFAULT_LOW_DETAIL_IMAGE_PIXEL_BUDGET,
DEFAULT_MAX_INLINE_REQUEST_IMAGE_BYTES,
DEFAULT_MAX_IMAGES_PER_REQUEST,
DEFAULT_MAX_REQUEST_FILES_BYTES,
DEFAULT_MAX_TOKENS,
@@ -43,9 +46,12 @@ export {
DEFAULT_FILE_EXPIRY_SECONDS,
DEFAULT_FILE_QUOTA_CLEANUP_BATCH,
DEFAULT_FILE_REFRESH_MARGIN_SECONDS,
DEFAULT_FILES_API_TIMEOUT_MS,
DEFAULT_IMAGE_OFFLOAD_BYTE_QUANTUM,
DEFAULT_IMAGE_OFFLOAD_COUNT_QUANTUM,
DEFAULT_INLINE_IMAGE_OFFLOAD_BYTE_QUANTUM,
DEFAULT_LOW_DETAIL_IMAGE_PIXEL_BUDGET,
DEFAULT_MAX_INLINE_REQUEST_IMAGE_BYTES,
DEFAULT_MAX_IMAGES_PER_REQUEST,
DEFAULT_MAX_REQUEST_FILES_BYTES,
DEFAULT_MAX_TOKENS,
@@ -116,12 +122,18 @@ export interface Config {
streamIdleTimeoutMs?: number
/** Maximum accumulated file-referenced image bytes per chat request (default 128 MiB). */
maxRequestFilesBytes?: number
/** Maximum number of file-referenced images per chat request (default 600). */
/** Maximum accumulated base64 image payload after Files API fallback (default 20 MiB). */
maxInlineRequestImageBytes?: number
/** Maximum number of represented images per chat request (default 600). */
maxImagesPerRequest?: number
/** Raw-byte removal step after the request exceeds its file bound (default 64 MiB). */
imageOffloadByteQuantum?: number
/** Base64-byte removal step after inline fallback exceeds its bound (default 10 MiB). */
inlineImageOffloadByteQuantum?: number
/** Image-count removal step after the request exceeds its count bound (default 20). */
imageOffloadCountQuantum?: number
/** Maximum duration of one request-image Files API resolution (default one minute). */
filesApiTimeoutMs?: number
/** Explicit lifetime assigned to each uploaded image (default seven days). */
fileExpiresAfterSeconds?: number
/** Remaining lifetime below which an indexed file is replaced (default one hour). */
@@ -154,9 +166,12 @@ export const Config: z<Config> = z.object({
models: z.array(catalogModel).default(DEFAULT_MODELS),
streamIdleTimeoutMs: z.number().min(Number.MIN_VALUE).max(MAX_TIMER_DELAY_MS).default(DEFAULT_STREAM_IDLE_TIMEOUT_MS),
maxRequestFilesBytes: z.number().step(1).min(1).default(DEFAULT_MAX_REQUEST_FILES_BYTES),
maxInlineRequestImageBytes: z.number().step(1).min(1).default(DEFAULT_MAX_INLINE_REQUEST_IMAGE_BYTES),
maxImagesPerRequest: z.number().step(1).min(1).default(DEFAULT_MAX_IMAGES_PER_REQUEST),
imageOffloadByteQuantum: z.number().step(1).min(1).default(DEFAULT_IMAGE_OFFLOAD_BYTE_QUANTUM),
inlineImageOffloadByteQuantum: z.number().step(1).min(1).default(DEFAULT_INLINE_IMAGE_OFFLOAD_BYTE_QUANTUM),
imageOffloadCountQuantum: z.number().step(1).min(1).default(DEFAULT_IMAGE_OFFLOAD_COUNT_QUANTUM),
filesApiTimeoutMs: z.number().min(Number.MIN_VALUE).max(MAX_TIMER_DELAY_MS).default(DEFAULT_FILES_API_TIMEOUT_MS),
fileExpiresAfterSeconds: z.number().step(1).min(3_600).max(2_592_000).default(DEFAULT_FILE_EXPIRY_SECONDS),
fileRefreshMarginSeconds: z.number().step(1).min(0).default(DEFAULT_FILE_REFRESH_MARGIN_SECONDS),
fileQuotaCleanupBatch: z.number().step(1).min(1).max(1_000).default(DEFAULT_FILE_QUOTA_CLEANUP_BATCH),
@@ -283,6 +298,10 @@ export function resolveAdapterOptions(config: Config, environment?: LaunchEnviro
if (!Number.isSafeInteger(maxRequestFilesBytes) || maxRequestFilesBytes <= 0) {
throw new Error('llm-deepseek: maxRequestFilesBytes must be a positive safe integer')
}
const maxInlineRequestImageBytes = config.maxInlineRequestImageBytes ?? DEFAULT_MAX_INLINE_REQUEST_IMAGE_BYTES
if (!Number.isSafeInteger(maxInlineRequestImageBytes) || maxInlineRequestImageBytes <= 0) {
throw new Error('llm-deepseek: maxInlineRequestImageBytes must be a positive safe integer')
}
const maxImagesPerRequest = config.maxImagesPerRequest ?? DEFAULT_MAX_IMAGES_PER_REQUEST
if (!Number.isSafeInteger(maxImagesPerRequest) || maxImagesPerRequest <= 0) {
throw new Error('llm-deepseek: maxImagesPerRequest must be a positive safe integer')
@@ -294,6 +313,14 @@ export function resolveAdapterOptions(config: Config, environment?: LaunchEnviro
if (imageOffloadByteQuantum > maxRequestFilesBytes) {
throw new Error('llm-deepseek: imageOffloadByteQuantum must not exceed maxRequestFilesBytes')
}
const inlineImageOffloadByteQuantum = config.inlineImageOffloadByteQuantum
?? DEFAULT_INLINE_IMAGE_OFFLOAD_BYTE_QUANTUM
if (!Number.isSafeInteger(inlineImageOffloadByteQuantum) || inlineImageOffloadByteQuantum <= 0) {
throw new Error('llm-deepseek: inlineImageOffloadByteQuantum must be a positive safe integer')
}
if (inlineImageOffloadByteQuantum > maxInlineRequestImageBytes) {
throw new Error('llm-deepseek: inlineImageOffloadByteQuantum must not exceed maxInlineRequestImageBytes')
}
const imageOffloadCountQuantum = config.imageOffloadCountQuantum ?? DEFAULT_IMAGE_OFFLOAD_COUNT_QUANTUM
if (!Number.isSafeInteger(imageOffloadCountQuantum) || imageOffloadCountQuantum <= 0) {
throw new Error('llm-deepseek: imageOffloadCountQuantum must be a positive safe integer')
@@ -301,6 +328,17 @@ export function resolveAdapterOptions(config: Config, environment?: LaunchEnviro
if (imageOffloadCountQuantum > maxImagesPerRequest) {
throw new Error('llm-deepseek: imageOffloadCountQuantum must not exceed maxImagesPerRequest')
}
const filesApiTimeoutMs = config.filesApiTimeoutMs ?? DEFAULT_FILES_API_TIMEOUT_MS
if (!Number.isFinite(filesApiTimeoutMs)
|| filesApiTimeoutMs <= 0
|| filesApiTimeoutMs > MAX_TIMER_DELAY_MS) {
throw new Error(
`llm-deepseek: filesApiTimeoutMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`,
)
}
if (filesApiTimeoutMs >= streamIdleTimeoutMs) {
throw new Error('llm-deepseek: filesApiTimeoutMs must be below streamIdleTimeoutMs')
}
const fileExpiresAfterSeconds = config.fileExpiresAfterSeconds ?? DEFAULT_FILE_EXPIRY_SECONDS
if (!Number.isSafeInteger(fileExpiresAfterSeconds)
|| fileExpiresAfterSeconds < 3_600
@@ -333,9 +371,12 @@ export function resolveAdapterOptions(config: Config, environment?: LaunchEnviro
models: resolveModels(config.models),
streamIdleTimeoutMs,
maxRequestFilesBytes,
maxInlineRequestImageBytes,
maxImagesPerRequest,
imageOffloadByteQuantum,
inlineImageOffloadByteQuantum,
imageOffloadCountQuantum,
filesApiTimeoutMs,
filePolicy: {
expiresAfterSeconds: fileExpiresAfterSeconds,
refreshMarginSeconds: fileRefreshMarginSeconds,
+39 -27
View File
@@ -1,7 +1,7 @@
/**
* Serialize harness messages into DeepSeek chat completions. Text-only
* requests retain string user content; the image path resolves durable
* attachments into ordered Files API parts. Tool-result images follow their
* attachments into ordered file-id or inline parts. Tool-result images follow their
* string-only tool messages in a separate user message.
* @module dsh-llm-deepseek/serialize
*/
@@ -10,7 +10,7 @@ import { contentHasImage, LlmError, offloadRequestImagesWithPolicy, requestImage
import type { ContentBlock, GenerateOptions, Message } from '@deepseek-ai/dsh-llm'
import type { ImageAttachmentRef, RequestImageAttachment } from '@deepseek-ai/dsh-attachment'
import type {
WireFileContentPart,
WireImageContentPart,
WireMessage,
WireRequest,
WireTextContentPart,
@@ -29,21 +29,30 @@ interface ResolvedThinking {
reasoningEffort?: 'low' | 'high' | 'max'
}
/** Provider representation for every retained image in one request. */
export type ImageRequestRepresentation =
| {
kind: 'file'
/** Resolve a retained request version to a reusable DeepSeek file id. */
resolveFileId: (
version: RequestImageAttachment,
block: Extract<ContentBlock, { type: 'image' }>,
location: ImageWireLocation,
) => Promise<string>
}
| { kind: 'base64' }
/** Dependencies required only when the request contains image input. */
export interface ImageSerializationOptions {
/** Resolve a retained request version to a reusable DeepSeek file id. */
resolveFileId: (
version: RequestImageAttachment,
block: Extract<ContentBlock, { type: 'image' }>,
location: ImageWireLocation,
) => Promise<string>
/** Request versions prepared for the conservatively retained masters, keyed by attachment id. */
/** One representation used for every retained image in this request. */
representation: ImageRequestRepresentation
/** Request versions prepared for the conservatively retained normalized attachments, keyed by attachment id. */
requestImages: ReadonlyMap<ImageAttachmentRef['attachmentId'], RequestImageAttachment>
/** Positive bound on accumulated referenced image bytes. */
maxRequestFilesBytes: number
/** Maximum referenced images in one request. */
/** Positive bound on accumulated represented image bytes. */
maxRequestImageBytes: number
/** Maximum represented images in one request. */
maxImagesPerRequest?: number
/** Raw-byte removal step applied after the request exceeds its byte bound. */
/** Represented-byte removal step applied after the request exceeds its byte bound. */
byteQuantum?: number
/** Image-count removal step applied after the request exceeds its count bound. */
countQuantum?: number
@@ -125,13 +134,13 @@ function imageHandle(
}
}
/** Resolve one durable image into its descriptor and transient DeepSeek file-id part. */
/** Resolve one durable image into its descriptor and transient DeepSeek image part. */
async function imageParts(
block: Extract<ContentBlock, { type: 'image' }>,
images: ImageSerializationOptions,
location: ImageWireLocation,
precededByContent: boolean,
): Promise<[WireTextContentPart, WireFileContentPart]> {
): Promise<[WireTextContentPart, WireImageContentPart]> {
const version = images.requestImages.get(block.attachment.attachmentId)
if (version === undefined) {
throw new LlmError(
@@ -139,10 +148,13 @@ async function imageParts(
'INVALID_REQUEST',
)
}
return [
imageHandle(version, precededByContent),
{ type: 'file', file_id: await images.resolveFileId(version, block, location) },
]
const image: WireImageContentPart = images.representation.kind === 'file'
? { type: 'file', file_id: await images.representation.resolveFileId(version, block, location) }
: {
type: 'image_url',
image_url: { url: `data:${version.mediaType};base64,${Buffer.from(version.data).toString('base64')}` },
}
return [imageHandle(version, precededByContent), image]
}
/** Convert user or nested tool-result blocks into ordered wire parts. */
@@ -177,7 +189,7 @@ async function contentParts(
function userContent(parts: readonly WireUserContentPart[]): string | WireUserContentPart[] {
const text: string[] = []
for (const part of parts) {
if (part.type === 'file') return [...parts]
if (part.type !== 'text') return [...parts]
text.push(part.text)
}
return text.join('')
@@ -263,7 +275,7 @@ export function serializeMessages(messages: Message[]): WireMessage[] {
* Consecutive tool results keep string `tool` messages and share one following
* user message containing their images.
* @param messages - transient request history after request-size offloading.
* @param images - prepared request versions and reusable provider file-id resolver.
* @param images - prepared request versions, one provider representation, and its budget.
* @returns ordered DeepSeek wire messages.
*/
export async function serializeMessagesWithImages(
@@ -272,7 +284,7 @@ export async function serializeMessagesWithImages(
): Promise<WireMessage[]> {
assertSupportedImageRoles(messages)
const wire: WireMessage[] = []
let pendingToolImages: WireFileContentPart[] = []
let pendingToolImages: WireImageContentPart[] = []
const flushToolImages = (): void => {
if (pendingToolImages.length === 0) return
wire.push({
@@ -309,14 +321,14 @@ export async function serializeMessagesWithImages(
}
for (const result of toolResults) {
const parts = await contentParts(result.content, images, messageIndex + 1, nextImage)
const fileParts = parts.filter((part): part is WireFileContentPart => part.type === 'file')
const imageParts = parts.filter((part): part is WireImageContentPart => part.type !== 'text')
const text = parts.filter(part => part.type === 'text').map(part => part.text).join('')
wire.push({
role: 'tool',
tool_call_id: result.toolCallId,
content: text || '(no output)',
})
pendingToolImages.push(...fileParts)
pendingToolImages.push(...imageParts)
}
}
flushToolImages()
@@ -378,7 +390,7 @@ export function serializeRequest(
/**
* Build one image-capable request while keeping durable bytes out of session
* messages. Oversized oldest images become deterministic text after their
* exact request-version byte lengths are known and before provider upload.
* exact request-version byte lengths are known and before provider serialization.
* @param options - harness request containing image-capable user content.
* @param images - attachment resolver, request bound, and cancellation.
* @param defaults - adapter-level thinking defaults.
@@ -391,7 +403,7 @@ export async function serializeRequestWithImages(
): Promise<WireRequest> {
assertSupportedImageRoles(options.messages)
const requestMessages = offloadRequestImagesWithPolicy(options.messages, {
representation: 'raw',
representation: images.representation.kind === 'file' ? 'raw' : 'base64',
byteLength: (ref) => {
const version = images.requestImages.get(ref.attachmentId)
if (version === undefined) {
@@ -399,7 +411,7 @@ export async function serializeRequestWithImages(
}
return version.bytes
},
maxBytes: images.maxRequestFilesBytes,
maxBytes: images.maxRequestImageBytes,
...images.maxImagesPerRequest === undefined ? {} : { maxImages: images.maxImagesPerRequest },
...images.byteQuantum === undefined ? {} : { byteQuantum: images.byteQuantum },
...images.countQuantum === undefined ? {} : { countQuantum: images.countQuantum },
+10 -1
View File
@@ -47,8 +47,17 @@ export interface WireFileContentPart {
file_id: string
}
/** Inline base64 data URL inside a multimodal user message. */
export interface WireImageUrlContentPart {
type: 'image_url'
image_url: { url: string }
}
/** One image representation accepted by a multimodal user message. */
export type WireImageContentPart = WireFileContentPart | WireImageUrlContentPart
/** Ordered input part accepted by a multimodal user message. */
export type WireUserContentPart = WireTextContentPart | WireFileContentPart
export type WireUserContentPart = WireTextContentPart | WireImageContentPart
/** User-role message: text-only string or ordered multimodal input. */
export interface WireUserMessage {
+297 -2
View File
@@ -8,6 +8,7 @@ import type { AttachmentStore, ImageAttachmentRef, RequestImageAttachment } from
import { createLaunchEnvironmentSnapshot } from '@deepseek-ai/dsh-launch-environment'
import LlmRuntime, { CallId, createUserMessage,
CONTEXT_WINDOW_EXCEEDED_CODE,
LlmError,
ProviderRequestId,
QUOTA_EXCEEDED_CODE,
ReasoningEffortId,
@@ -52,6 +53,7 @@ async function harness(baseURL: string, config: object = {}) {
function adapterOf(
config: Partial<LlmDeepSeek.Config> & { apiKey?: string } = {},
attachments?: AttachmentStore,
files?: LlmDeepSeek.DeepSeekFileStore,
): DeepSeekAdapter {
const { apiKey, ...rest } = config
return new DeepSeekAdapter({
@@ -59,6 +61,7 @@ function adapterOf(
resolveApiKey: () => Promise.resolve(apiKey ?? 'k'),
resolveUserId: () => TEST_USER_ID,
resolveAttachments: () => attachments,
...files === undefined ? {} : { resolveFiles: () => files },
})
}
@@ -102,6 +105,32 @@ function attachmentStoreOf(
}
}
function fileStoreOf(
implementation: (...args: Parameters<LlmDeepSeek.DeepSeekFileStore['ensureUploaded']>) => ReturnType<LlmDeepSeek.DeepSeekFileStore['ensureUploaded']>,
) {
const ensureUploaded = vi.fn(implementation)
const invalidate = vi.fn(() => Promise.resolve())
return {
store: { ensureUploaded, invalidate } as unknown as LlmDeepSeek.DeepSeekFileStore,
ensureUploaded,
invalidate,
}
}
function fileReference(fileId: string): Awaited<ReturnType<LlmDeepSeek.DeepSeekFileStore['ensureUploaded']>> {
return {
record: { fileId: LlmDeepSeek.DeepSeekFileId(fileId) },
uploaded: true,
} as Awaited<ReturnType<LlmDeepSeek.DeepSeekFileStore['ensureUploaded']>>
}
function successfulSseResponse(): Response {
return new Response(textEvents.map(event => `data: ${event}\n\n`).join(''), {
status: 200,
headers: { 'content-type': 'text/event-stream' },
})
}
describe('request image policy', () => {
it.each([
[
@@ -199,6 +228,188 @@ describe('DeepSeekAdapter against a mock server', () => {
expect(policies).toEqual([{ maxPixels: 640_000, maxBytes: 1024 * 1024 }])
})
it('falls back to one all-base64 request when Files API resolution fails', async () => {
const server = await mockServer([{ kind: 'sse', events: textEvents }])
const secondRef = { ...imageRef, attachmentId: AttachmentId(`sha256:${'c'.repeat(64)}`) }
const attachments = attachmentStoreOf(ref => Promise.resolve({
...requestImage(ref),
variantId: ImageVariantId(`sha256:${(ref.attachmentId === imageRef.attachmentId ? 'b' : 'd').repeat(64)}`),
})).store
const files = fileStoreOf(() => Promise.reject(new LlmError('Files unavailable', 'SERVER')))
const adapter = adapterOf({
baseURL: server.url,
models: [{ id: 'deepseek-v4-flash-vision-exp', inputModalities: ['text', 'image'] }],
}, attachments, files.store)
await drain(adapter.stream({
provider: 'deepseek-official',
model: 'deepseek-v4-flash-vision-exp',
messages: [createUserMessage({
content: [
{ type: 'image', attachment: imageRef },
{ type: 'image', attachment: secondRef },
],
source: { kind: 'plugin', plugin: 'test' },
})],
}))
const body = server.requests[0] as { messages: Array<{ content: unknown }> }
expect(JSON.stringify(body.messages[0]?.content).match(/"type":"image_url"/g)).toHaveLength(2)
expect(JSON.stringify(body)).not.toContain('file_id')
expect(files.ensureUploaded).toHaveBeenCalledTimes(1)
})
it('reduces base64 fallback history from the configured high watermark to its half-size quantum', async () => {
const server = await mockServer([{ kind: 'sse', events: textEvents }])
const attachments = attachmentStoreOf(ref => Promise.resolve(requestImage(ref))).store
const files = fileStoreOf(() => Promise.reject(new LlmError('Files unavailable', 'SERVER')))
const adapter = adapterOf({
baseURL: server.url,
maxInlineRequestImageBytes: 80,
inlineImageOffloadByteQuantum: 40,
models: [{ id: 'deepseek-v4-flash-vision-exp', inputModalities: ['text', 'image'] }],
}, attachments, files.store)
await drain(adapter.stream({
provider: 'deepseek-official',
model: 'deepseek-v4-flash-vision-exp',
messages: [createUserMessage({
content: Array.from({ length: 21 }, () => ({ type: 'image' as const, attachment: imageRef })),
source: { kind: 'plugin', plugin: 'test' },
})],
}))
const body = JSON.stringify(server.requests[0])
expect(body.match(/older images are omitted first/g)).toHaveLength(11)
expect(body.match(/"type":"image_url"/g)).toHaveLength(10)
})
it('discards partially resolved file ids and falls back with every retained image inline', async () => {
const server = await mockServer([{ kind: 'sse', events: textEvents }])
const secondRef = { ...imageRef, attachmentId: AttachmentId(`sha256:${'c'.repeat(64)}`) }
const attachments = attachmentStoreOf(ref => Promise.resolve({
...requestImage(ref),
variantId: ImageVariantId(`sha256:${(ref.attachmentId === imageRef.attachmentId ? 'b' : 'd').repeat(64)}`),
})).store
const files = fileStoreOf(() => Promise.reject(new Error('unused')))
files.ensureUploaded
.mockResolvedValueOnce(fileReference('file-api-partial'))
.mockRejectedValueOnce(new LlmError('Files unavailable', 'TRANSPORT'))
const adapter = adapterOf({
baseURL: server.url,
models: [{ id: 'deepseek-v4-flash-vision-exp', inputModalities: ['text', 'image'] }],
}, attachments, files.store)
await drain(adapter.stream({
provider: 'deepseek-official',
model: 'deepseek-v4-flash-vision-exp',
messages: [createUserMessage({
content: [
{ type: 'image', attachment: imageRef },
{ type: 'image', attachment: secondRef },
],
source: { kind: 'plugin', plugin: 'test' },
})],
}))
const body = server.requests[0] as { messages: Array<{ content: unknown }> }
expect(JSON.stringify(body.messages[0]?.content).match(/"type":"image_url"/g)).toHaveLength(2)
expect(JSON.stringify(body)).not.toContain('file-api-partial')
})
it('falls back after the configured Files API deadline without aborting chat', async () => {
vi.useFakeTimers()
const started = Promise.withResolvers<undefined>()
const files = fileStoreOf((_version, _connection, _policy, signal) => new Promise((_resolve, reject) => {
started.resolve(undefined)
signal?.addEventListener('abort', () => {
const reason: unknown = signal.reason
reject(reason instanceof Error ? reason : new Error('files operation aborted'))
}, { once: true })
}))
const attachments = attachmentStoreOf(ref => Promise.resolve(requestImage(ref))).store
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue(successfulSseResponse())
const adapter = adapterOf({
baseURL: 'https://deepseek.invalid',
filesApiTimeoutMs: 50,
models: [{ id: 'deepseek-v4-flash-vision-exp', inputModalities: ['text', 'image'] }],
}, attachments, files.store)
const pending = drain(adapter.stream({
provider: 'deepseek-official',
model: 'deepseek-v4-flash-vision-exp',
messages: [createUserMessage({
content: [{ type: 'image', attachment: imageRef }],
source: { kind: 'plugin', plugin: 'test' },
})],
}))
await started.promise
await vi.advanceTimersByTimeAsync(50)
await pending
expect(fetchSpy).toHaveBeenCalledTimes(1)
expect(String(fetchSpy.mock.calls[0]?.[1]?.body)).toContain('image_url')
fetchSpy.mockRestore()
})
it('does not turn caller cancellation during file resolution into base64 fallback', async () => {
const started = Promise.withResolvers<undefined>()
const files = fileStoreOf((_version, _connection, _policy, signal) => new Promise((_resolve, reject) => {
started.resolve(undefined)
signal?.addEventListener('abort', () => { reject(new Error('cancelled')) }, { once: true })
}))
const attachments = attachmentStoreOf(ref => Promise.resolve(requestImage(ref))).store
const fetchSpy = vi.spyOn(globalThis, 'fetch')
const controller = new AbortController()
const adapter = adapterOf({
baseURL: 'https://deepseek.invalid',
models: [{ id: 'deepseek-v4-flash-vision-exp', inputModalities: ['text', 'image'] }],
}, attachments, files.store)
const pending = drain(adapter.stream({
provider: 'deepseek-official',
model: 'deepseek-v4-flash-vision-exp',
signal: controller.signal,
messages: [createUserMessage({
content: [{ type: 'image', attachment: imageRef }],
source: { kind: 'plugin', plugin: 'test' },
})],
}))
await started.promise
controller.abort()
await expect(pending).rejects.toMatchObject({ code: 'ABORTED' })
expect(fetchSpy).not.toHaveBeenCalled()
fetchSpy.mockRestore()
})
it('does not retry a generic chat failure through base64 fallback', async () => {
const server = await mockServer([{
kind: 'http-error',
status: 503,
body: JSON.stringify({ error: { message: 'chat unavailable' } }),
}])
const attachments = attachmentStoreOf(ref => Promise.resolve(requestImage(ref))).store
const files = fileStoreOf(() => Promise.resolve(fileReference('file-api-ready')))
const adapter = adapterOf({
baseURL: server.url,
models: [{ id: 'deepseek-v4-flash-vision-exp', inputModalities: ['text', 'image'] }],
}, attachments, files.store)
await expect(drain(adapter.stream({
provider: 'deepseek-official',
model: 'deepseek-v4-flash-vision-exp',
messages: [createUserMessage({
content: [{ type: 'image', attachment: imageRef }],
source: { kind: 'plugin', plugin: 'test' },
})],
}))).rejects.toMatchObject({ code: 'SERVER', message: 'chat unavailable' })
expect(server.requests).toHaveLength(1)
expect(JSON.stringify(server.requests[0])).toContain('file-api-ready')
expect(JSON.stringify(server.requests[0])).not.toContain('image_url')
})
it('does not prepare an old image removed by request offload', async () => {
const server = await mockServer([{ kind: 'sse', events: textEvents }])
const old = { ...imageRef, attachmentId: AttachmentId(`sha256:${'c'.repeat(64)}`), bytes: 3 }
@@ -454,6 +665,40 @@ describe('DeepSeekAdapter against a mock server', () => {
expect(attachmentMocks.readImageRequest).toHaveBeenCalledTimes(1)
})
it('uses inline fallback when stale-id recovery cannot resolve a replacement file', async () => {
const server = await mockServer([
{
kind: 'http-error',
status: 400,
body: JSON.stringify({ error: { message: 'file_id file-api-stale expired' } }),
},
{ kind: 'sse', events: textEvents },
])
const attachments = attachmentStoreOf(ref => Promise.resolve(requestImage(ref))).store
const files = fileStoreOf(() => Promise.reject(new Error('unused')))
files.ensureUploaded
.mockResolvedValueOnce(fileReference('file-api-stale'))
.mockRejectedValueOnce(new LlmError('Files unavailable', 'SERVER'))
const adapter = adapterOf({
baseURL: server.url,
models: [{ id: 'deepseek-v4-flash-vision-exp', inputModalities: ['text', 'image'] }],
}, attachments, files.store)
await drain(adapter.stream({
provider: 'deepseek-official',
model: 'deepseek-v4-flash-vision-exp',
messages: [createUserMessage({
content: [{ type: 'image', attachment: imageRef }],
source: { kind: 'plugin', plugin: 'test' },
})],
}))
expect(files.invalidate).toHaveBeenCalledTimes(1)
expect(server.requests).toHaveLength(2)
expect(JSON.stringify(server.requests[0])).toContain('file-api-stale')
expect(JSON.stringify(server.requests[1])).toContain('image_url')
})
it('invalidates only the identified mapping when a multi-image request names one stale file id', async () => {
const secondRef: ImageAttachmentRef = {
...imageRef,
@@ -1150,7 +1395,11 @@ describe('DeepSeekAdapter against a mock server', () => {
})
return Promise.resolve(new Response(body, { status: 200 }))
})
const adapter = adapterOf({ baseURL: 'https://example.invalid', streamIdleTimeoutMs: 100 })
const adapter = adapterOf({
baseURL: 'https://example.invalid',
filesApiTimeoutMs: 50,
streamIdleTimeoutMs: 100,
})
try {
const drain = (async () => {
for await (const _chunk of adapter.stream({ provider: 'deepseek-official', model: 'm', messages: [] })) { /* drain */ }
@@ -1181,7 +1430,11 @@ describe('DeepSeekAdapter against a mock server', () => {
})
return Promise.resolve(new Response(body, { status: 200 }))
})
const adapter = adapterOf({ baseURL: 'https://example.invalid', streamIdleTimeoutMs: 100 })
const adapter = adapterOf({
baseURL: 'https://example.invalid',
filesApiTimeoutMs: 50,
streamIdleTimeoutMs: 100,
})
try {
const chunks: string[] = []
const drain = (async () => {
@@ -1573,6 +1826,10 @@ describe('plugin registration and config', () => {
maxRequestFilesBytes: 10,
imageOffloadByteQuantum: 11,
})).toThrow(/imageOffloadByteQuantum must not exceed maxRequestFilesBytes/)
expect(() => resolveAdapterOptions({
maxInlineRequestImageBytes: 10,
inlineImageOffloadByteQuantum: 11,
})).toThrow(/inlineImageOffloadByteQuantum must not exceed maxInlineRequestImageBytes/)
expect(() => resolveAdapterOptions({
maxImagesPerRequest: 10,
imageOffloadCountQuantum: 11,
@@ -1584,6 +1841,8 @@ describe('plugin registration and config', () => {
['maxImagesPerRequest', 1.5, /maxImagesPerRequest must be a positive safe integer/],
['imageOffloadByteQuantum', 0, /imageOffloadByteQuantum must be a positive safe integer/],
['imageOffloadByteQuantum', Number.MAX_SAFE_INTEGER + 1, /imageOffloadByteQuantum must be a positive safe integer/],
['inlineImageOffloadByteQuantum', 0, /inlineImageOffloadByteQuantum must be a positive safe integer/],
['inlineImageOffloadByteQuantum', Number.MAX_SAFE_INTEGER + 1, /inlineImageOffloadByteQuantum must be a positive safe integer/],
['imageOffloadCountQuantum', 0, /imageOffloadCountQuantum must be a positive safe integer/],
['imageOffloadCountQuantum', 1.5, /imageOffloadCountQuantum must be a positive safe integer/],
['fileExpiresAfterSeconds', 3_599, /fileExpiresAfterSeconds must be an integer from 3600 through 2592000/],
@@ -1612,6 +1871,22 @@ describe('plugin registration and config', () => {
},
)
it.each([0, 1.5, Number.MAX_SAFE_INTEGER + 1])(
'rejects invalid inline request image bound %s',
async (maxInlineRequestImageBytes) => {
expect(() => resolveAdapterOptions({ maxInlineRequestImageBytes }))
.toThrow(/maxInlineRequestImageBytes must be a positive safe integer/)
const ctx = new Context()
await ctx.plugin(LlmRuntime)
await expect(ctx.plugin(LlmDeepSeek, {
baseURL: 'http://127.0.0.1:1',
maxInlineRequestImageBytes,
})).rejects.toThrow(/maxInlineRequestImageBytes/)
expect(ctx.llm.listProviders()).toEqual([])
},
)
it('falls back to DEEPSEEK_API_KEY and DEEPSEEK_BASE_URL env vars', async () => {
vi.stubEnv('DEEPSEEK_API_KEY', 'env-key')
vi.stubEnv('DEEPSEEK_BASE_URL', 'http://127.0.0.1:1')
@@ -1753,6 +2028,26 @@ describe('plugin registration and config', () => {
})).rejects.toThrow(/streamIdleTimeoutMs/)
})
it('rejects invalid Files API timeout bounds for direct and plugin composition', async () => {
expect(() => resolveAdapterOptions({ filesApiTimeoutMs: Number.POSITIVE_INFINITY }))
.toThrow(/filesApiTimeoutMs.*positive finite/)
expect(() => resolveAdapterOptions({ filesApiTimeoutMs: MAX_TIMER_DELAY_MS + 1 }))
.toThrow(/filesApiTimeoutMs.*no greater/)
const ctx = new Context()
await ctx.plugin(LlmRuntime)
await expect(ctx.plugin(LlmDeepSeek, {
baseURL: 'http://127.0.0.1:1',
filesApiTimeoutMs: 0,
})).rejects.toThrow(/filesApiTimeoutMs/)
await expect(ctx.plugin(LlmDeepSeek, {
baseURL: 'http://127.0.0.1:1',
filesApiTimeoutMs: MAX_TIMER_DELAY_MS + 1,
})).rejects.toThrow(/filesApiTimeoutMs/)
expect(() => resolveAdapterOptions({ filesApiTimeoutMs: 100, streamIdleTimeoutMs: 100 }))
.toThrow(/filesApiTimeoutMs must be below streamIdleTimeoutMs/)
})
it('rejects invalid nested retryPolicy before registering the provider', async () => {
const ctx = new Context()
await ctx.plugin(LlmRuntime)
@@ -11,6 +11,8 @@ import {
} from '../src/serialize.ts'
import type { ImageSerializationOptions } from '../src/serialize.ts'
type FileResolver = Extract<ImageSerializationOptions['representation'], { kind: 'file' }>['resolveFileId']
function request(overrides: Partial<GenerateOptions> = {}): GenerateOptions {
return { provider: 'deepseek-official', model: 'deepseek-v4-flash', messages: [], ...overrides }
}
@@ -32,7 +34,7 @@ function imageRef(mediaType: ImageMediaType = 'image/png', bytes = 3): ImageAtta
}
function fileResolver(id = 'file-api-image') {
return vi.fn<ImageSerializationOptions['resolveFileId']>(() => Promise.resolve(id))
return vi.fn<FileResolver>(() => Promise.resolve(id))
}
function requestVersion(ref: ImageAttachmentRef): RequestImageAttachment {
@@ -53,13 +55,26 @@ function requestVersion(ref: ImageAttachmentRef): RequestImageAttachment {
function imageOptions(
refs: readonly ImageAttachmentRef[],
resolveFileId: ImageSerializationOptions['resolveFileId'] = fileResolver(),
maxRequestFilesBytes = 20 * 1024 * 1024,
resolveFileId: FileResolver = fileResolver(),
maxRequestImageBytes = 20 * 1024 * 1024,
) {
return {
resolveFileId,
representation: { kind: 'file' as const, resolveFileId },
requestImages: new Map(refs.map(ref => [ref.attachmentId, requestVersion(ref)])),
maxRequestFilesBytes,
maxRequestImageBytes,
}
}
function inlineImageOptions(
refs: readonly ImageAttachmentRef[],
maxRequestImageBytes = 20 * 1024 * 1024,
byteQuantum = 10 * 1024 * 1024,
): ImageSerializationOptions {
return {
representation: { kind: 'base64' },
requestImages: new Map(refs.map(ref => [ref.attachmentId, requestVersion(ref)])),
maxRequestImageBytes,
byteQuantum,
}
}
@@ -359,6 +374,30 @@ describe('image serialization', () => {
}])
})
it.each([
['image/png', 'data:image/png;base64,AAAA'],
['image/jpeg', 'data:image/jpeg;base64,AAAA'],
['image/webp', 'data:image/webp;base64,AAAA'],
['image/gif', 'data:image/gif;base64,AAAA'],
] as const)('serializes every retained %s request version as an inline data URL', async (mediaType, url) => {
const ref = imageRef(mediaType)
const wire = await serializeRequestWithImages(request({
model: 'deepseek-v4-flash-vision-exp',
messages: [createUserMessage({
content: [{ type: 'image', attachment: ref }],
source: { kind: 'plugin', plugin: 'test' },
})],
}), inlineImageOptions([ref]))
expect(wire.messages).toEqual([{
role: 'user',
content: [
{ type: 'text', text: `Image ${ref.attachmentId}; request image 1x1px.` },
{ type: 'image_url', image_url: { url } },
],
}])
})
it('gives image-only input a stable handle and request dimensions', async () => {
const ref = imageRef()
const wire = await serializeRequestWithImages(request({
@@ -548,6 +587,21 @@ describe('image serialization', () => {
expect(resolveFileId.mock.calls[0]?.[0]).toMatchObject({ attachment: { mediaType: 'image/jpeg' } })
})
it('drops base64 history from a 20-unit high watermark to a 10-unit low watermark', async () => {
const ref = imageRef('image/png', 3)
const wire = await serializeRequestWithImages(request({
model: 'deepseek-v4-flash-vision-exp',
messages: [createUserMessage({
content: Array.from({ length: 21 }, () => ({ type: 'image' as const, attachment: ref })),
source: { kind: 'plugin', plugin: 'test' },
})],
}), inlineImageOptions([ref], 80, 40))
const content = wire.messages[0]?.content
expect(JSON.stringify(content).match(/older images are omitted first/g)).toHaveLength(11)
expect(JSON.stringify(content).match(/"type":"image_url"/g)).toHaveLength(10)
})
it('rejects an unprepared image while computing exact request bytes', async () => {
const ref = imageRef()
await expect(serializeRequestWithImages(request({