diff --git a/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.i18n.yaml b/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.i18n.yaml index f0e60bbc20..71c06fbd35 100644 --- a/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md -2026-06-24-web-capability-seam.md: 7e7b09f19864bd2ad8ad9d69579c1d5c79600cde -2026-06-24-web-capability-seam.zh.md: dbb41ee42d2c7503955ead2df32abe80b3a4f641 +2026-06-24-web-capability-seam.md: 5c8ca698386392f87e60e5dc543c6478316338ed +2026-06-24-web-capability-seam.zh.md: 1946748e2fef7db72c7450f2bfc44c46aed51ee2 diff --git a/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md b/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md index 7e7b09f198..5c8ca69838 100644 --- a/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md +++ b/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md @@ -199,7 +199,7 @@ Full page retrieval remains the job of `web_fetch(url)`. Search snippets are dis ## Fetch request and result schema -The `web_fetch` implementation is an anonymous public HTTP(S) fetch provider, `http`. It fetches bytes from a concrete URL, applies the basic transport hygiene below (http/https-only, credential rejection, byte/time caps, cross-origin redirect blocking), decodes textual content, and returns only the minimal model-useful result: final URL, status code, body, and truncation. It carries no browser cookies, editor credentials, git credentials, internal auth tokens, or implicit access to private services. (Full SSRF / private-network blocking is deferred — see [Deferred work](#deferred-work).) +The `web_fetch` implementation is an anonymous public HTTP(S) fetch provider, `http`. It fetches bytes from a concrete URL, resolves and pins public destinations, applies the transport hygiene below, decodes textual content, and returns only the minimal model-useful result: final URL, status code, body, and truncation. It carries no browser cookies, editor credentials, git credentials, internal auth tokens, or implicit access to private services. The seam request stays smaller than OpenCode's model-facing tool: @@ -235,12 +235,14 @@ The provider owns safe resource retrieval: URL validation, HTTP transport, redir The fetch provider's resource controls: - Only `http:` and `https:` URLs are accepted; credentials in URLs are rejected. +- A literal address or the complete result of one hostname lookup must contain only globally reachable unicast IPv4 or IPv6 destinations. Loopback, private, link-local, carrier-grade NAT, multicast, reserved, transition, translation, and private IPv4-mapped IPv6 addresses are rejected. +- The request retains that validated address set in an Undici lookup callback instead of resolving the hostname again. The original hostname remains the HTTP Host and TLS SNI value, while DNS rebinding cannot replace the connection destination after validation. - Maximum URL length, response byte cap, decoded body character cap, timeout, and redirect hop cap are enforced. - Abort signals propagate through network fetches and expensive decoding. -- Only same-origin redirects are followed automatically; a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call and therefore a fresh provider/permission decision. (Claude Code's WebFetch uses this same model — it does not auto-follow a cross-host redirect; it returns the redirect target to the model for a fresh call.) +- Only same-origin redirects are followed automatically; each followed hop performs a fresh public-address lookup and pins its own connection. A cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call and therefore a fresh provider/permission decision. (Claude Code's WebFetch uses this same model — it does not auto-follow a cross-host redirect; it returns the redirect target to the model for a fresh call.) - Requests carry an explicit product user agent rather than silently impersonating a browser. -SSRF / private-network protection (blocking private, loopback, link-local, multicast, and otherwise non-public destinations, with DNS-resolve-then-validate to defeat rebinding and per-hop re-validation on redirects) is **deferred** — see [Deferred work](#deferred-work). Until it lands, `web_fetch` is an SSRF primitive and must not be enabled in a deployment that can reach sensitive internal network targets. +The provider rejects an entire DNS answer set when any address is not public instead of silently filtering the unsafe members. This fail-closed rule prevents connection-family selection or fallback from reaching an address that did not satisfy the public-network policy. ## Tool consumer behavior @@ -308,6 +310,14 @@ Rejected for the first version. Those providers often return extracted or summar Rejected for the seam. `prompt` turns fetch into LLM summarization and couples public-web retrieval to a model provider. The harness seam should fetch and decode deterministically; `dsh-tool-web` can later offer summaries as a presentation mode without making `ctx.web` depend on `ctx.llm`. +### Validate DNS and then call an ordinary fetch + +Rejected because an ordinary fetch resolves the hostname again when it opens the connection. An attacker can return a public address during validation and a private address during the second lookup. Passing the validated answer set through the connection's lookup callback closes that rebinding interval while preserving hostname-based HTTP and TLS behavior. + +### Block private-looking hostname strings without pinning resolved addresses + +Rejected because hostname syntax does not establish the connection destination: an arbitrary public-looking name can resolve to loopback, a private range, or a cloud metadata address. Address classification belongs after resolution, and every address available to connection fallback must pass it. + ## Consequences **The search schema is deliberately thin.** Exa and Perplexity both expose useful provider-specific controls; a control is added only once it can be defined provider-neutrally and enforced honestly by both tool registration and provider execution. @@ -318,13 +328,12 @@ Rejected for the seam. `prompt` turns fetch into LLM summarization and couples p **Provider state can change after startup.** A tool can be visible in the request assembled at step start and lose its provider before execution. The execution path resolves again and fails with a structured error. -**Fetch is a network boundary, not just a read-only tool.** `web_fetch` can reach sensitive network targets or exfiltrate data through URLs. Only the basic transport hygiene ships (http/https-only, credential rejection, byte/time caps, cross-origin redirect blocking); SSRF / private-network blocking is deferred (see [Deferred work](#deferred-work)), so until it lands `web_fetch` must not be enabled where it can reach internal targets. +**Fetch is a network boundary, not just a read-only tool.** Public-address validation and connection pinning prevent `web_fetch` from reaching non-public destinations, but a model can still disclose data through a public URL and fetched text remains untrusted model input. Product enablement therefore still needs a deliberate permission policy rather than treating fetch as equivalent to local read-only observation. **Large web content can damage context quality.** Providers enforce byte/character caps and report `truncated`; `tool-web` formats bounded model output with clear continuation or follow-up guidance. ## Deferred work -- SSRF / private-network protection for `web_fetch`: block private, loopback, link-local, multicast, and otherwise non-public destinations so `web_fetch` is not an SSRF primitive. Doing it correctly is more than a URL-string check — it needs DNS-resolve-then-connect-to-the-validated-IP (to defeat DNS rebinding / TOCTOU), per-hop re-validation across redirects, and IPv6 edge handling (private ranges, IPv4-mapped addresses). Neither reference implementation surveyed does IP-level blocking (OpenCode does a prefix check then fetches; Claude Code relies on a centralized hostname blocklist plus a "private URLs will fail" prompt), so there is no implementation to copy and this is the harness's only SSRF defense — it warrants its own focused design/spike. Until it lands, `web_fetch` must only be enabled in deployments that cannot reach sensitive internal targets. - A `pdf` `WebFetchBody` kind: the `http` provider decodes text-extractable PDFs (best-effort, capped, `truncated`) into a `{ kind: 'pdf'; content; pageCount? }` arm, and `tool-web` renders it. This is fetch, not `web_extract` — PDF retrieval is a concrete HTTP 200 plus deterministic local decoding, not provider-side extraction of a non-HTTP resource. Adding it is a coordinated change across `dsh-web` (declare the arm), the provider (decode + narrow "binary rejection" to "reject binary except text-extractable PDF"; scanned/image PDFs needing OCR stay out of scope), and `tool-web` (render). The closed `WebFetchBody` union makes the consumer side fail to compile until the new arm is handled. - Provider-backed extraction as a separate `web_extract` capability, rather than widening `web_fetch` silently. - Permission policy integration: the permission system now exists ([sandbox and approval](../feature/2026-07-06-sandbox.md), [web permission presets](../feature/2026-07-23-web-permission-and-approval.md)) but bundles only sandbox mode and approval policy; web permission policy remains unintegrated. diff --git a/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.zh.md b/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.zh.md index dbb41ee42d..1946748e2f 100644 --- a/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.zh.md +++ b/.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.zh.md @@ -199,7 +199,7 @@ Exa 搜索将提供方扁平 `results[]` 的每一项映射为 `WebSearchSource` ## Fetch 请求与结果 schema -`web_fetch` 的实现是一个匿名公开 HTTP(S) fetch 提供方 `http`。它从具体 URL 获取字节，应用下述基本传输卫生措施（仅 http/https、拒绝 URL 中的凭证、字节/时间上限、跨源重定向阻断），解码文本内容，并仅返回最小的模型可用结果：最终 URL、状态码、正文和截断标志。它不携带浏览器 cookie、编辑器凭证、git 凭证、内部认证令牌，也不隐式访问私有服务。（完整的 SSRF/私有网络阻断推迟——见[推迟工作](#deferred-work)。） +`web_fetch` 的实现是一个匿名公开 HTTP(S) fetch 提供方 `http`。它从具体 URL 获取字节，解析并固定公开目的地址，应用下述传输卫生措施，解码文本内容，并仅返回最小的模型可用结果：最终 URL、状态码、正文和截断标志。它不携带浏览器 cookie、编辑器凭证、git 凭证、内部认证令牌，也不隐式访问私有服务。 seam 请求比 OpenCode 的面向模型工具更小： @@ -235,12 +235,14 @@ export type WebFetchBody = fetch 提供方的资源控制： - 仅接受 `http:` 和 `https:` URL；拒绝 URL 中的凭证。 +- 字面 IP 地址或 hostname 一次解析得到的完整结果只能包含全球可达的单播 IPv4 或 IPv6 目的地址。loopback、私有、link-local、运营商级 NAT、多播、保留、过渡、转换和映射到私有 IPv4 的 IPv6 地址都会被拒绝。 +- 请求通过 Undici lookup 回调保留这一组已验证地址，不会再次解析 hostname。原 hostname 仍作为 HTTP Host 与 TLS SNI 值，而 DNS rebinding 无法在验证后替换连接目的地址。 - 强制执行最大 URL 长度、响应字节上限、解码正文字符上限、超时和重定向跳数上限。 - Abort 信号传播到网络获取和高开销解码。 -- 仅自动跟随同源重定向；跨源重定向以 `WEB_REDIRECT_BLOCKED` 失败，要求一次新的工具调用，从而触发新的提供方/权限决策。（Claude Code 的 WebFetch 使用同样的模型——它不自动跟随跨主机重定向，而是将重定向目标返回给模型以发起新调用。） +- 仅自动跟随同源重定向；每个跟随的跳转都会重新解析公开地址，并把自己的连接固定到解析结果。跨源重定向以 `WEB_REDIRECT_BLOCKED` 失败，要求一次新的工具调用，从而触发新的提供方/权限决策。（Claude Code 的 WebFetch 使用同样的模型——它不自动跟随跨主机重定向，而是将重定向目标返回给模型以发起新调用。） - 请求携带显式的产品 User-Agent，而非静默伪装浏览器。 -SSRF/私有网络防护（阻断私有、回环、链路本地、多播及其他非公开目的地，通过先 DNS 解析再验证 IP 来防御 rebinding，并在重定向的每一跳重新验证）**推迟**——见[推迟工作](#deferred-work)。在其落地之前，`web_fetch` 是一个 SSRF 原语，不得在能触达敏感内部网络目标的部署中启用。 +只要 DNS 完整解析结果中存在任一非公开地址，提供方就会拒绝整个结果，而不是静默过滤不安全成员。该 fail-closed 规则可防止连接的地址族选择或回退触及未满足公开网络策略的地址。 ## 工具消费方行为 @@ -308,6 +310,14 @@ SSRF/私有网络防护（阻断私有、回环、链路本地、多播及其他 在 seam 层面否决。`prompt` 将 fetch 变成 LLM 摘要，并将公开 web 获取耦合到模型提供方。harness seam 应当确定性地获取和解码；`dsh-tool-web` 日后可以将摘要作为展示模式提供，而无需让 `ctx.web` 依赖 `ctx.llm`。 +### 验证 DNS 后调用普通 fetch + +否决，因为普通 fetch 在打开连接时会再次解析 hostname。攻击者可以在验证时返回公开地址，在第二次解析时返回私有地址。把已验证解析结果通过连接的 lookup 回调传入，可以在保留基于 hostname 的 HTTP 与 TLS 行为的同时关闭这一 rebinding 时间窗口。 + +### 只阻断看起来像私网的 hostname 字符串，不固定解析地址 + +否决，因为 hostname 语法无法确定连接目的地址：任意看似公开的名称都可能解析到 loopback、私有网段或云 metadata 地址。地址分类必须在解析后执行，连接回退可使用的每个地址都必须通过校验。 + ## 后果 **搜索 schema 刻意精简。** Exa 和 Perplexity 都暴露了有用的提供方特有控制；只有当某个控制能以提供方无关的方式定义、且工具注册和提供方执行都能诚实遵守时，才会添加。 @@ -318,7 +328,7 @@ SSRF/私有网络防护（阻断私有、回环、链路本地、多播及其他 **提供方状态可能在启动后变化。** 一个工具可能在步骤开始时组装的请求中可见，但在执行前失去其提供方。执行路径重新解析并以结构化错误失败。 -**Fetch 是网络边界，不仅仅是只读工具。** `web_fetch` 能触达敏感网络目标或通过 URL 外泄数据。仅交付基本传输卫生措施（仅 http/https、拒绝凭证、字节/时间上限、跨源重定向阻断）；SSRF/私有网络阻断推迟（见[推迟工作](#deferred-work)），因此在其落地之前，`web_fetch` 不得在能触达内部目标的环境中启用。 +**Fetch 是网络边界，不仅仅是只读工具。** 公开地址校验与连接固定可防止 `web_fetch` 触达非公开目的地址，但模型仍可通过公开 URL 泄露数据，抓取文本也仍是不受信任的模型输入。因此，产品启用 fetch 仍需要明确的权限策略，不能把它等同于本地只读观察。 **大量 web 内容可能损害上下文质量。** 提供方强制执行字节/字符上限并报告 `truncated`；`tool-web` 格式化有界的模型输出，附带清晰的继续或后续引导。 @@ -326,7 +336,6 @@ SSRF/私有网络防护（阻断私有、回环、链路本地、多播及其他 ## 推迟工作 -- `web_fetch` 的 SSRF/私有网络防护：阻断私有、回环、链路本地、多播及其他非公开目的地，使 `web_fetch` 不再是 SSRF 原语。正确实现不仅仅是 URL 字符串检查——需要先 DNS 解析再连接到已验证的 IP（防御 DNS rebinding/TOCTOU）、跨重定向的每跳重新验证，以及 IPv6 边缘处理（私有范围、IPv4 映射地址）。所调研的参考实现均未做 IP 级阻断（OpenCode 做前缀检查后直接 fetch；Claude Code 依赖集中式主机名黑名单加「私有 URL 会失败」的提示词），因此没有可复制的实现，且这是 harness 唯一的 SSRF 防线——值得一次专门的设计/spike。在其落地之前，`web_fetch` 只能在无法触达敏感内部目标的部署中启用。 - `pdf` `WebFetchBody` 类别：`http` 提供方将可文本提取的 PDF 解码（尽力而为、有上限、`truncated`）为 `{ kind: 'pdf'; content; pageCount? }` 分支，`tool-web` 渲染它。这是 fetch 而非 `web_extract`——PDF 获取是具体的 HTTP 200 加确定性的本地解码，不是提供方侧对非 HTTP 资源的提取。添加它是跨 `dsh-web`（声明分支）、提供方（解码 + 将「二进制拒绝」收窄为「拒绝二进制，但可文本提取的 PDF 除外」；需要 OCR 的扫描/图片 PDF 不在范围内）和 `tool-web`（渲染）的协调变更。封闭的 `WebFetchBody` 联合类型使消费方在新分支被处理之前编译失败。 - 提供方支撑的提取作为独立的 `web_extract` 能力，而非静默扩展 `web_fetch`。 - 权限策略集成：权限系统现已存在（[沙箱与审批](../feature/2026-07-06-sandbox.zh.md)、[web 权限预设](../feature/2026-07-23-web-permission-and-approval.zh.md)），但只捆绑了沙箱模式与审批策略；web 权限策略仍未集成。 diff --git a/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.i18n.yaml b/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.i18n.yaml index 651c076892..3c65879fdb 100644 --- a/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.i18n.yaml +++ b/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.md -2026-07-31-even-out-shipped-tool-rosters.md: 97a9fdaedb97de77c195c319f14f972aac850726 -2026-07-31-even-out-shipped-tool-rosters.zh.md: 130573f0ddf0b94b4dcb017f58f1e0935e53e844 +2026-07-31-even-out-shipped-tool-rosters.md: 20ffda551899826971fbaa1d5d4576b2b10b1362 +2026-07-31-even-out-shipped-tool-rosters.zh.md: 79f1bb569a20e2f87052c35c7dd41dc1ce93d8bf diff --git a/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.md b/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.md index 97a9fdaedb..20ffda5518 100644 --- a/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.md +++ b/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.md @@ -24,7 +24,7 @@ Three capabilities stay out on the evidence their own packages record, and are l **`dsh-tool-cordis`** lets the model write JavaScript and mount it as a temporary plugin. Its README states the limit: "The sandbox is containment for honest code, not a security boundary — host-realm helpers on the sandbox global are reachable, so mount code can reach Node" ([Known limitations](../../../../packages/extensions/tool-cordis/README.md)). The `node:vm` realm lives inside the harness process while `dsh-sandbox-local` confines only the argv it spawns, so on the Web surface both the sandbox and the approval seam are bypassed rather than enforced. -**`dsh-web-fetch-http`** stays unmounted and `dsh-tool-web` keeps `fetch: false`. SSRF protection is deferred in the implementation ([`policy.ts`](../../../../packages/web/web-fetch-http/src/policy.ts) validates protocol, credentials, and length only) and the package says so: "this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets" ([README](../../../../packages/web/web-fetch-http/README.md)). The model chooses the target, which includes the harness's own gateway on loopback, private ranges, and cloud metadata endpoints. +**`dsh-web-fetch-http`** stays unmounted and `dsh-tool-web` keeps `fetch: false`. The provider restricts connections to validated public IP destinations, but `dsh-tool-web` has no web-specific permission policy and executes without asking `ctx.approval` ([README](../../../../packages/web/tool-web/README.md)). The shipped permission presets therefore do not silently broaden from sandboxed file access to model-selected public network requests. Withholding it narrows the surface without removing the reach: `bash` is mounted, so `curl` gets the same page, as a live run confirmed. What the absence buys is the removal of an argument-shaped request primitive that needs no shell — and with it the accidental path where a summarization request quietly reaches loopback. A deployment that must contain outbound traffic needs a network-level control. diff --git a/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.zh.md b/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.zh.md index 130573f0dd..79f1bb569a 100644 --- a/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.zh.md +++ b/.agents/notes/implemented/feature/2026-07-31-even-out-shipped-tool-rosters.zh.md @@ -24,7 +24,7 @@ Status: implemented **`dsh-tool-cordis`** 让模型写一段 JavaScript 并挂成临时插件。它的 README 写明了这个界限:「The sandbox is containment for honest code, not a security boundary — host-realm helpers on the sandbox global are reachable, so mount code can reach Node」([Known limitations](../../../../packages/extensions/tool-cordis/README.zh.md))。`node:vm` 的 realm 就在 harness 进程内,而 `dsh-sandbox-local` 只约束它 spawn 出去的 argv,因此在 Web surface 上,沙箱与批准接缝是被绕过而非被执行。 -**`dsh-web-fetch-http`** 保持不挂,`dsh-tool-web` 保持 `fetch: false`。SSRF 防护在实现中是 deferred 状态([`policy.ts`](../../../../packages/web/web-fetch-http/src/policy.ts) 只校验协议、凭据与长度),包里也直说了:「this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets」([README](../../../../packages/web/web-fetch-http/README.zh.md))。目标由模型选择,其中包括 harness 自己跑在环回地址上的网关、内网段和云元数据端点。 +**`dsh-web-fetch-http`** 保持不挂，`dsh-tool-web` 保持 `fetch: false`。提供方只允许连接到已验证的公开 IP 目的地址，但 `dsh-tool-web` 没有 web 专用权限策略，执行时也不会询问 `ctx.approval`（[README](../../../../packages/web/tool-web/README.zh.md)）。因此，已交付的权限 preset 不会从受 sandbox 约束的文件访问静默扩展到模型选择的公开网络请求。 不挂载它收窄的是接触面而非可达性：`bash` 是挂着的,`curl` 照样能拿到同一个页面——一次真实运行确认了这点。这个缺席买到的是去掉一个无需 shell、以参数成形的请求原语,以及随之而来的那条意外路径:一次「帮我总结这个页面」悄悄打到环回地址。真要收住出站流量的部署需要的是网络层管控。 diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index f2637fe699..0fe83078b9 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -66,6 +66,7 @@ External packages that a workspace package resolves at runtime. The tier covers | [`eventsource-parser`](https://github.com/rexxars/eventsource-parser) | MIT | | [`fflate`](https://github.com/101arrowz/fflate) | MIT | | [`immer`](https://github.com/immerjs/immer) | MIT | +| [`ipaddr.js`](https://github.com/whitequark/ipaddr.js) | MIT | | [`js-yaml`](https://github.com/nodeca/js-yaml) | MIT | | [`katex`](https://github.com/KaTeX/KaTeX) | MIT | | [`koffi`](https://github.com/Koromix/koffi) | MIT | @@ -94,6 +95,7 @@ External packages that a workspace package resolves at runtime. The tier covers | [`tsx`](https://github.com/privatenumber/tsx) | MIT | | [`turndown`](https://github.com/mixmark-io/turndown) | MIT | | [`typescript`](https://github.com/microsoft/TypeScript) | Apache-2.0 | +| [`undici`](https://github.com/nodejs/undici) | MIT | | [`use-sync-external-store`](https://github.com/facebook/react) | MIT | | [`ws`](https://github.com/websockets/ws) | MIT | | [`yaml`](https://github.com/eemeli/yaml) | ISC | diff --git a/docs/subsystems/web.i18n.yaml b/docs/subsystems/web.i18n.yaml index dd16cb1790..91854e163a 100644 --- a/docs/subsystems/web.i18n.yaml +++ b/docs/subsystems/web.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/subsystems/web.md -web.md: 4acab9273b3b2753409c680bd41e93fb3a627843 -web.zh.md: 0133b78d0080ab16c14ac7f42628cc705bb4bc9c +web.md: 72942a62759ce8a875540d4637a34d1d968632a0 +web.zh.md: 58fb351adf13d818a9c0191d7d869d707233b46e diff --git a/docs/subsystems/web.md b/docs/subsystems/web.md index 4acab9273b..72942a6275 100644 --- a/docs/subsystems/web.md +++ b/docs/subsystems/web.md @@ -130,7 +130,7 @@ Selection never depends on registration, config, or HMR order: a capability has ## The service -`WebRuntime` registers search and fetch providers, rejects duplicate ids with `WEB_DUPLICATE_PROVIDER`, and resolves providers at execution time with structured selection errors. The local fetch backend accepts only HTTP(S), rejects credentials, caps redirects, bytes, characters, and time, revalidates every same-origin redirect hop, and decodes the body; the tool owns presentation. The local backend does not block private-network targets; do not enable `web_fetch` where it can reach sensitive internal ones. +`WebRuntime` registers search and fetch providers, rejects duplicate ids with `WEB_DUPLICATE_PROVIDER`, and resolves providers at execution time with structured selection errors. The local fetch backend accepts only HTTP(S), rejects credentials, resolves each hostname once, rejects any answer set containing a non-public IPv4 or IPv6 destination, pins the request connection to the validated addresses, repeats those checks for every same-origin redirect hop, caps redirects, bytes, characters, and time, and decodes the body; the tool owns presentation. diff --git a/docs/subsystems/web.zh.md b/docs/subsystems/web.zh.md index 0133b78d00..58fb351adf 100644 --- a/docs/subsystems/web.zh.md +++ b/docs/subsystems/web.zh.md @@ -130,7 +130,7 @@ type WebFetchBody = ## 服务 -`WebRuntime` 注册搜索与抓取提供方，以 `WEB_DUPLICATE_PROVIDER` 拒绝重复 id，并在执行时以结构化的选择错误解析提供方。本地抓取后端仅接受 HTTP(S)、拒绝凭证、限制重定向次数、字节数、字符数和时间、对每一次同源重定向跳转重新进行安全校验，并解码正文；展示由工具负责。本地后端不会拦截私有网络目标；在能够触及敏感内部目标的环境中，禁止启用 `web_fetch`。 +`WebRuntime` 注册搜索与抓取提供方，以 `WEB_DUPLICATE_PROVIDER` 拒绝重复 id，并在执行时以结构化的选择错误解析提供方。本地抓取后端仅接受 HTTP(S)、拒绝凭证、对每个 hostname 只解析一次、拒绝包含任一非公开 IPv4 或 IPv6 目的地址的解析结果、把请求连接固定到已验证地址、对每一次同源重定向跳转重复这些校验、限制重定向次数、字节数、字符数和时间，并解码正文；展示由工具负责。 diff --git a/packages/bundle/base/cordis.patch.yml b/packages/bundle/base/cordis.patch.yml index e7e963e59f..5fe58dc5e7 100644 --- a/packages/bundle/base/cordis.patch.yml +++ b/packages/bundle/base/cordis.patch.yml @@ -409,10 +409,11 @@ # resolves the same DEEPSEEK_API_KEY credential the Models page manages for # chat, at each search; its Messages endpoint is separate from the # chat-completions endpoint, so it takes its own base-URL override. Fetch stays - # disabled and no fetch provider is mounted: that provider defers SSRF - # protection and the model would choose the request target. Search is a full - # auxiliary model request with server-side retrieval, so this shipped DeepSeek - # route gets 60s while the provider-neutral tool default remains 30s. + # disabled and no fetch provider is mounted because the shipped permission + # presets do not yet classify public network access; web_fetch otherwise runs + # without approval. Search is a full auxiliary model request with server-side + # retrieval, so this shipped DeepSeek route gets 60s while the provider-neutral + # tool default remains 30s. - id: web name: '@deepseek-ai/dsh-web' config: diff --git a/packages/web/web-fetch-http/README.i18n.yaml b/packages/web/web-fetch-http/README.i18n.yaml index 078606e11b..ae32a21bfa 100644 --- a/packages/web/web-fetch-http/README.i18n.yaml +++ b/packages/web/web-fetch-http/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/web/web-fetch-http/README.md -README.md: 5589a8e8605a64ae9ef5f6d9978a9b63331d5b0d -README.zh.md: b0dff1d992f9f84cc8b9b9747544ef5e6c0fc3eb +README.md: 13ff12861b8573a4d60b3300aa33f9b47d7ab7da +README.zh.md: 1670a8a2855effdd93216e7f1b952a13fa5d0516 diff --git a/packages/web/web-fetch-http/README.md b/packages/web/web-fetch-http/README.md index 5589a8e860..13ff12861b 100644 --- a/packages/web/web-fetch-http/README.md +++ b/packages/web/web-fetch-http/README.md @@ -8,7 +8,7 @@ This is an **implementation** package: it registers a provider into `ctx.web`, i ## Responsibility split -The provider owns **safe resource retrieval**: URL validation, HTTP transport, redirect policy, a resource-backstop timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. `@deepseek-ai/dsh-tool-web` owns **presentation** (HTML→markdown, truncation formatting). A non-2xx HTTP response is a *result* (status code + decoded body), not an error; `WebError` is reserved for failures to safely retrieve or represent the resource. +The provider owns **safe resource retrieval**: URL validation, public-address resolution and connection pinning, HTTP transport, redirect policy, a resource-backstop timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. `@deepseek-ai/dsh-tool-web` owns **presentation** (HTML→markdown, truncation formatting). A non-2xx HTTP response is a *result* (status code + decoded body), not an error; `WebError` is reserved for failures to safely retrieve or represent the resource. The provider's `timeoutMs` is a resource backstop for direct `ctx.web.fetch()` callers and misconfigured deployments, not the model-facing tool-call budget. [`dsh-tool-call-timeout-policy`](../../guard/timeout-policy/README.md) owns the `web_fetch` tool-call budget by arming `exec.signal`. @@ -17,9 +17,10 @@ A shipping web-tool deployment sets the provider backstop above the tool budget, ## Transport hygiene - Accepts only `http:` and `https:` URLs; rejects credentials in URLs (`WEB_BLOCKED_URL`) and over-long/malformed URLs (`WEB_INVALID_URL`). +- Resolves each hostname once, rejects the complete answer set if any IPv4 or IPv6 destination is not public unicast (`WEB_BLOCKED_URL`), and pins the connection to that validated set. This blocks loopback, private, link-local, carrier-grade NAT, multicast, reserved, transition, translation, and private IPv4-mapped IPv6 destinations without a second DNS lookup. - Enforces a max URL length, response byte cap (`WEB_FETCH_TOO_LARGE`), decoded body character cap, timeout (`WEB_FETCH_TIMEOUT`), and redirect hop cap. - Propagates the caller's abort signal (`WEB_ABORTED`) into the network request and the streaming read. -- Follows only **same-origin** redirects; a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (the model of Claude Code's WebFetch). +- Follows only **same-origin** redirects; each followed hop repeats public-address resolution and pinning, while a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED` and requires a fresh tool call (the model of Claude Code's WebFetch). - Sends an explicit product `User-Agent`, never a browser disguise. - Rejects unsupported (e.g. binary) content types with `WEB_UNSUPPORTED_CONTENT_TYPE`. @@ -46,6 +47,5 @@ No direct invalidation; the named consumer owns any request-prefix changes. ## Known Limitations and Deferred Work -- **SSRF / private-network protection is deferred** — no blocking of private, loopback, link-local, multicast, or otherwise non-public destinations, no DNS-resolve-then-validate, no per-hop re-validation (see [the web capability seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md)). Until it lands, this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets. - **Only textual content decodes** — html/xhtml and `text/*`-plus-JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work. - **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back. diff --git a/packages/web/web-fetch-http/README.zh.md b/packages/web/web-fetch-http/README.zh.md index b0dff1d992..1670a8a285 100644 --- a/packages/web/web-fetch-http/README.zh.md +++ b/packages/web/web-fetch-http/README.zh.md @@ -8,7 +8,7 @@ ## 职责拆分 -提供方拥有**安全资源获取**：URL 验证、HTTP 传输、重定向策略、资源兜底超时、中止传播、字节上限、charset 解码、内容类型分类与二进制拒绝。`@deepseek-ai/dsh-tool-web` 拥有**呈现**（HTML→markdown、截断格式）。非 2xx HTTP 响应是*结果*（状态码 + 解码主体），不是错误；`WebError` 只用于无法安全获取或表示资源的失败。 +提供方拥有**安全资源获取**：URL 验证、公开地址解析与连接固定、HTTP 传输、重定向策略、资源兜底超时、中止传播、字节上限、charset 解码、内容类型分类与二进制拒绝。`@deepseek-ai/dsh-tool-web` 拥有**呈现**（HTML→markdown、截断格式）。非 2xx HTTP 响应是*结果*（状态码 + 解码主体），不是错误；`WebError` 只用于无法安全获取或表示资源的失败。 提供方的 `timeoutMs` 是直接 `ctx.web.fetch()` 调用方和配置有误的部署所用的资源兜底，不是面向模型的工具调用预算。[`dsh-tool-call-timeout-policy`](../../guard/timeout-policy/README.zh.md) 拥有 `web_fetch` 工具调用预算，并让 `exec.signal` 在超时时触发，以强制执行该预算。 @@ -17,9 +17,10 @@ ## 传输卫生 - 只接受 `http:` 和 `https:` URL；拒绝 URL 中的凭据（`WEB_BLOCKED_URL`）以及过长／格式错误的 URL（`WEB_INVALID_URL`）。 +- 每个 hostname 只解析一次；如果完整解析结果中任一 IPv4 或 IPv6 目的地址不是公开单播地址，则以 `WEB_BLOCKED_URL` 拒绝；连接只使用这一组已验证地址。该策略会阻断 loopback、私有、link-local、运营商级 NAT、多播、保留、过渡、转换和映射到私有 IPv4 的 IPv6 地址，且不会进行第二次 DNS 解析。 - 强制执行 URL 最大长度、响应字节上限（`WEB_FETCH_TOO_LARGE`）、解码主体字符上限、超时（`WEB_FETCH_TIMEOUT`）和重定向跳数上限。 - 把调用方的中止信号（`WEB_ABORTED`）传播到网络请求与流式读取。 -- 只跟随**同源**重定向；跨源重定向以 `WEB_REDIRECT_BLOCKED` 失败，要求发起新的工具调用（沿用 Claude Code 的 WebFetch 模式）。 +- 只跟随**同源**重定向；每个跟随的跳转都会再次执行公开地址解析与连接固定，跨源重定向则以 `WEB_REDIRECT_BLOCKED` 失败并要求发起新的工具调用（沿用 Claude Code 的 WebFetch 模式）。 - 发送显式的产品 `User-Agent`，绝不伪装成浏览器。 - 不受支持的内容类型（例如二进制）以 `WEB_UNSUPPORTED_CONTENT_TYPE` 拒绝。 @@ -46,6 +47,5 @@ ## 已知限制与暂缓事项 -- **SSRF／私有网络防护暂缓**：不会阻止私有、loopback、link-local、multicast 或其他非公开目标，也不进行 DNS 解析后验证或逐跳重新验证（见 [web 能力 seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.zh.md)）。在此功能落地前，该提供方是 SSRF 原语；能够访问敏感内部网络目标的部署**禁止启用它**。 - **只解码文本内容**：包括 html/xhtml 与 `text/*` 加 JSON/XML 家族；缺少 `Content-Type` 或任何二进制类型都会抛出 `WEB_UNSUPPORTED_CONTENT_TYPE`，可提取文本的 PDF 解码属于明确的暂缓工作。 - **charset 只来自 `Content-Type` 标头**（默认为 UTF-8）：HTML `` 声明会被忽略；声明但无法识别的 charset 标签会抛出异常，而非回退。 diff --git a/packages/web/web-fetch-http/package.json b/packages/web/web-fetch-http/package.json index 3dfee71b40..602ce5d239 100644 --- a/packages/web/web-fetch-http/package.json +++ b/packages/web/web-fetch-http/package.json @@ -32,18 +32,20 @@ ], "license": "MIT", "peerDependencies": { + "@deepseek-ai/cordis": "workspace:^", "@deepseek-ai/dsh-invariants": "workspace:^", "@deepseek-ai/dsh-timeout": "workspace:^", - "@deepseek-ai/dsh-web": "workspace:^", - "@deepseek-ai/cordis": "workspace:^" + "@deepseek-ai/dsh-web": "workspace:^" }, "dependencies": { - "@deepseek-ai/schemastery": "workspace:^" + "@deepseek-ai/schemastery": "workspace:^", + "ipaddr.js": "^2.5.0", + "undici": "^8.10.0" }, "devDependencies": { + "@deepseek-ai/cordis": "workspace:^", "@deepseek-ai/dsh-invariants": "workspace:^", "@deepseek-ai/dsh-timeout": "workspace:^", - "@deepseek-ai/dsh-web": "workspace:^", - "@deepseek-ai/cordis": "workspace:^" + "@deepseek-ai/dsh-web": "workspace:^" } } diff --git a/packages/web/web-fetch-http/src/network.ts b/packages/web/web-fetch-http/src/network.ts new file mode 100644 index 0000000000..5fbc64b6bd --- /dev/null +++ b/packages/web/web-fetch-http/src/network.ts @@ -0,0 +1,181 @@ +/** + * Public-network resolution and address-pinned HTTP transport for `web-fetch-http`. + * One DNS answer set is validated before Undici receives it through a custom lookup, + * so the connection cannot resolve the hostname again to a private address. + * + * @module @deepseek-ai/dsh-web-fetch-http/network + */ + +import { lookup as systemLookup } from 'node:dns/promises' +import type { LookupAddress, LookupOptions } from 'node:dns' +import { isIP } from 'node:net' +import { Agent, fetch } from 'undici' +import type { Response } from 'undici' +import ipaddr from 'ipaddr.js' +import { WebError } from '@deepseek-ai/dsh-web' + +/** One address resolved and retained for the subsequent pinned connection. */ +export interface PublicAddress { + /** Canonical textual IPv4 or IPv6 address. */ + readonly address: string + /** Address family accepted by Node's connection lookup callback. */ + readonly family: 4 | 6 +} + +/** The result of one address-pinned request; closing releases its private pool. */ +export interface PinnedResponse { + /** HTTP response whose body remains readable until `close()` is called. */ + readonly response: Response + /** Release the request's dispatcher after the response body is consumed or cancelled. */ + close(): Promise +} + +/** Resolver signature used to test public-address policy without process DNS changes. */ +export type AddressResolver = (hostname: string, options: { all: true; order: 'verbatim' }) => Promise + +/** + * Return whether an address is globally reachable unicast. IPv4-mapped IPv6 is + * classified by its embedded IPv4 address; transition and translation prefixes + * remain blocked because their eventual IPv4 destination cannot be pinned here. + * + * @param input - textual IPv4 or IPv6 address. + * @returns true only for a public unicast destination. + */ +export function isPublicIpAddress(input: string): boolean { + let parsed: ipaddr.IPv4 | ipaddr.IPv6 + try { + parsed = ipaddr.parse(stripIpv6Brackets(input)) + } catch { + return false + } + if (parsed instanceof ipaddr.IPv4) return parsed.range() === 'unicast' + if (parsed.isIPv4MappedAddress()) return parsed.toIPv4Address().range() === 'unicast' + return parsed.range() === 'unicast' +} + +/** + * Resolve a hostname once and reject the complete answer set if any destination + * is not public. The returned addresses are the only ones the transport may use. + * + * @param hostname - URL hostname, including brackets when it is an IPv6 literal. + * @param signal - aborts the wait for system resolution; an in-flight OS lookup may finish unused. + * @param resolver - lookup implementation, overridden only by focused tests. + * @returns the validated, non-empty address set. + */ +export async function resolvePublicAddresses( + hostname: string, + signal: AbortSignal, + resolver: AddressResolver = systemLookup, +): Promise { + const unbracketed = stripIpv6Brackets(hostname) + const literalFamily = isIP(unbracketed) + const resolved = literalFamily === 0 + ? await raceWithSignal(resolver(unbracketed, { all: true, order: 'verbatim' }), signal) + : [{ address: unbracketed, family: literalFamily }] + + if (resolved.length === 0) { + throw new WebError(`hostname "${hostname}" resolved to no addresses`, 'WEB_PROVIDER_ERROR') + } + + const addresses: PublicAddress[] = [] + for (const entry of resolved) { + if ((entry.family !== 4 && entry.family !== 6) || isIP(entry.address) !== entry.family) { + throw new WebError(`hostname "${hostname}" resolved to an invalid IP address`, 'WEB_PROVIDER_ERROR') + } + if (!isPublicIpAddress(entry.address)) { + throw new WebError(`URL hostname "${hostname}" resolves to a non-public IP address`, 'WEB_BLOCKED_URL') + } + addresses.push({ address: entry.address, family: entry.family }) + } + return addresses +} + +/** + * Fetch through an Undici agent whose lookup callback returns only the already + * validated address set. The URL hostname remains intact for HTTP Host and TLS SNI. + * + * @param url - validated HTTP(S) URL. + * @param addresses - public addresses returned by {@link resolvePublicAddresses}. + * @param headers - request headers. + * @param signal - request and body-read cancellation signal. + * @returns a response plus the dispatcher disposer its consumer must call. + */ +export async function requestPinned( + url: URL, + addresses: readonly PublicAddress[], + headers: Record, + signal: AbortSignal, +): Promise { + const dispatcher = new Agent({ + autoSelectFamily: true, + connect: { lookup: createPinnedLookup(addresses) }, + }) + try { + const response = await fetch(url, { method: 'GET', redirect: 'manual', headers, signal, dispatcher }) + return { response, close: async () => { await dispatcher.close() } } + } catch (error: unknown) { + await dispatcher.close() + throw error + } +} + +/** Production network operations kept as an object so provider tests can replace resolution only. */ +export const publicHttpNetwork = { + resolve: resolvePublicAddresses, + request: requestPinned, +} + +type LookupCallback = ( + error: NodeJS.ErrnoException | null, + address: string | LookupAddress[], + family?: number, +) => void + +/** + * Build the connector lookup that serves a fixed validated answer set. + * + * @param addresses - public addresses retained from the preceding resolution. + * @returns a Node-compatible lookup callback that performs no network resolution. + */ +export function createPinnedLookup(addresses: readonly PublicAddress[]): ( + hostname: string, + options: LookupOptions, + callback: LookupCallback, +) => void { + return (hostname: string, options: LookupOptions, callback: LookupCallback): void => { + const family = typeof options.family === 'number' + ? options.family + : options.family === 'IPv4' ? 4 : options.family === 'IPv6' ? 6 : 0 + const eligible = family === 0 ? addresses : addresses.filter(address => address.family === family) + const selected = eligible[0] + if (selected === undefined) { + const error = Object.assign(new Error(`no validated address for ${hostname} in family ${family}`), { + code: 'ENOTFOUND', + hostname, + }) + callback(error, options.all === true ? [] : '', family) + return + } + if (options.all === true) { + callback(null, eligible.map(address => ({ ...address }))) + return + } + callback(null, selected.address, selected.family) + } +} + +/** Race a non-cancellable OS lookup without letting it delay tool cancellation. */ +function raceWithSignal(promise: Promise, signal: AbortSignal): Promise { + const abortError = () => new Error('web fetch aborted during hostname resolution', { cause: signal.reason }) + if (signal.aborted) return Promise.reject(abortError()) + return new Promise((resolve, reject) => { + const abort = () => { reject(abortError()) } + signal.addEventListener('abort', abort, { once: true }) + promise.then(resolve, reject).finally(() => { signal.removeEventListener('abort', abort) }) + }) +} + +/** WHATWG URL retains brackets around IPv6 hostnames; IP parsers do not. */ +function stripIpv6Brackets(hostname: string): string { + return hostname.startsWith('[') && hostname.endsWith(']') ? hostname.slice(1, -1) : hostname +} diff --git a/packages/web/web-fetch-http/src/policy.ts b/packages/web/web-fetch-http/src/policy.ts index d45c28f58d..dcd5239f88 100644 --- a/packages/web/web-fetch-http/src/policy.ts +++ b/packages/web/web-fetch-http/src/policy.ts @@ -15,7 +15,7 @@ export type FetchableKind = 'html' | 'text' * Validate a request URL against the basic transport hygiene the provider * enforces before any network access: http(s) only, no embedded credentials, * bounded length. Returns the parsed `URL`. Throws {@link WebError} otherwise. - * (SSRF / private-network blocking is deferred — see the package Agent Note.) + * Public-address resolution and connection pinning run after this syntax check. * * @param input - the raw URL string from the fetch request. * @param maxUrlLength - inclusive upper bound on `input`'s length. diff --git a/packages/web/web-fetch-http/src/provider.ts b/packages/web/web-fetch-http/src/provider.ts index c3b461d2ca..7ec2a6bb94 100644 --- a/packages/web/web-fetch-http/src/provider.ts +++ b/packages/web/web-fetch-http/src/provider.ts @@ -1,16 +1,16 @@ /** - * Safe HTTP(S) retrieval for `ctx.web`: validates URLs, follows only same-origin redirects, - * enforces time and size limits, classifies and decodes text, and leaves presentation to - * `@deepseek-ai/dsh-tool-web`. Requests carry no browser cookies or ambient credentials. - * - * Private-network and SSRF protection is not implemented; do not enable this provider where - * it can reach sensitive internal targets. + * Safe HTTP(S) retrieval for `ctx.web`: validates and pins public IP destinations, follows + * only same-origin redirects, enforces time and size limits, classifies and decodes text, + * and leaves presentation to `@deepseek-ai/dsh-tool-web`. Requests carry no browser cookies + * or ambient credentials. * @module @deepseek-ai/dsh-web-fetch-http/provider */ import { WebError } from '@deepseek-ai/dsh-web' import type { WebFetchBody, WebFetchProvider, WebFetchRequest, WebFetchResult } from '@deepseek-ai/dsh-web' import { deadline, timeoutOf } from '@deepseek-ai/dsh-timeout' +import type { Response } from 'undici' +import { publicHttpNetwork } from './network.ts' import { classifyContentType, decoderForCharset, isSameOrigin, parseCharset, validateFetchUrl } from './policy.ts' /** Resolved provider limits (the plugin's schemastery Config supplies defaults). */ @@ -58,57 +58,61 @@ export class HttpFetchProvider implements WebFetchProvider { let redirectsFollowed = 0 for (;;) { - const response = await this.requestOnce(currentUrl, signal) - - if (isRedirectStatus(response.status)) { - // Enforce the redirect budget before resolving or validating the next hop. - if (redirectsFollowed >= this.limits.maxRedirects) { - await response.body?.cancel() - throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, 'WEB_REDIRECT_BLOCKED') - } - const location = response.headers.get('location') - if (location === null) { - // A redirect status with no Location is not a usable resource. Cancel - // the (possibly streaming) body before throwing so no socket leaks. - await response.body?.cancel() - throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, 'WEB_PROVIDER_ERROR') - } - const target = resolveRedirect(location, currentUrl) - // Re-validate the target against the same transport hygiene a direct request gets: a - // redirect must not be a back door to a credentialed, non-http(s), or over-long URL - // that validateFetchUrl would reject. - let validatedTarget: URL - try { - validatedTarget = validateFetchUrl(target.toString(), this.limits.maxUrlLength) - if (!isSameOrigin(validatedTarget, currentUrl)) { - throw new WebError( - `cross-origin redirect to ${validatedTarget.origin} is not followed automatically; retry against that URL directly`, - 'WEB_REDIRECT_BLOCKED', - ) + const request = await this.requestOnce(currentUrl, signal) + const { response } = request + try { + if (isRedirectStatus(response.status)) { + // Enforce the redirect budget before resolving or validating the next hop. + if (redirectsFollowed >= this.limits.maxRedirects) { + await response.body?.cancel() + throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, 'WEB_REDIRECT_BLOCKED') + } + const location = response.headers.get('location') + if (location === null) { + // A redirect status with no Location is not a usable resource. Cancel + // the (possibly streaming) body before throwing so no socket leaks. + await response.body?.cancel() + throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, 'WEB_PROVIDER_ERROR') + } + const target = resolveRedirect(location, currentUrl) + // Re-validate the target against the same transport hygiene a direct request gets: a + // redirect must not be a back door to a credentialed, non-http(s), or over-long URL + // that validateFetchUrl would reject. + let validatedTarget: URL + try { + validatedTarget = validateFetchUrl(target.toString(), this.limits.maxUrlLength) + if (!isSameOrigin(validatedTarget, currentUrl)) { + throw new WebError( + `cross-origin redirect to ${validatedTarget.origin} is not followed automatically; retry against that URL directly`, + 'WEB_REDIRECT_BLOCKED', + ) + } + } catch (error: unknown) { + await response.body?.cancel() + throw error } - } catch (error: unknown) { await response.body?.cancel() - throw error + currentUrl = validatedTarget + redirectsFollowed++ + continue } - await response.body?.cancel() - currentUrl = validatedTarget - redirectsFollowed++ - continue - } - return await this.readBody(response, currentUrl, signal) + return await this.readBody(response, currentUrl, signal) + } finally { + await request.close() + } } } - private async requestOnce(url: URL, signal: AbortSignal): Promise { + private async requestOnce(url: URL, signal: AbortSignal) { try { - return await fetch(url, { - method: 'GET', - redirect: 'manual', - headers: { 'user-agent': this.limits.userAgent, 'accept': 'text/html,application/xhtml+xml,text/*;q=0.9,application/json;q=0.8' }, - signal, - }) + const addresses = await publicHttpNetwork.resolve(url.hostname, signal) + return await publicHttpNetwork.request(url, addresses, { + 'user-agent': this.limits.userAgent, + 'accept': 'text/html,application/xhtml+xml,text/*;q=0.9,application/json;q=0.8', + }, signal) } catch (error: unknown) { + if (error instanceof WebError) throw error throw translateAbortOrNetwork(error, signal) } } @@ -168,7 +172,8 @@ export class HttpFetchProvider implements WebFetchProvider { const chunks: Uint8Array[] = [] let total = 0 let truncatedByBytes = false - const reader = response.body.getReader() + // Undici exposes response chunks as `any`; Fetch guarantees body chunks are Uint8Array. + const reader = response.body.getReader() as ReadableStreamDefaultReader try { for (;;) { const { done, value } = await reader.read() diff --git a/packages/web/web-fetch-http/tests/fetch-http.spec.ts b/packages/web/web-fetch-http/tests/fetch-http.spec.ts index 8b3ceac62b..284ea7456a 100644 --- a/packages/web/web-fetch-http/tests/fetch-http.spec.ts +++ b/packages/web/web-fetch-http/tests/fetch-http.spec.ts @@ -6,6 +6,7 @@ import WebRuntime from '@deepseek-ai/dsh-web' import { HttpFetchProvider, LOCAL_FETCH_PROVIDER_ID } from '@deepseek-ai/dsh-web-fetch-http' import type { HttpFetchLimits } from '@deepseek-ai/dsh-web-fetch-http' import * as fetchPlugin from '@deepseek-ai/dsh-web-fetch-http' +import { createPinnedLookup, isPublicIpAddress, publicHttpNetwork, requestPinned, resolvePublicAddresses } from '../src/network.ts' import { classifyContentType, decoderForCharset, isSameOrigin, parseCharset, validateFetchUrl } from '../src/policy.ts' const limits: HttpFetchLimits = { @@ -22,6 +23,7 @@ type Handler = (req: IncomingMessage, res: ServerResponse) => void let server: Server let base: string let handler: Handler +let restoreResolution: () => void beforeEach(async () => { handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('default') } @@ -29,10 +31,13 @@ beforeEach(async () => { await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)) const { port } = server.address() as AddressInfo base = `http://127.0.0.1:${port}` + const spy = vi.spyOn(publicHttpNetwork, 'resolve').mockResolvedValue([{ address: '127.0.0.1', family: 4 }]) + restoreResolution = () => { spy.mockRestore() } }) afterEach(async () => { vi.unstubAllGlobals() + vi.restoreAllMocks() await new Promise(resolve => server.close(() => { resolve() })) }) @@ -78,6 +83,131 @@ describe('policy helpers', () => { }) }) +describe('public-network policy', () => { + it('accepts only globally reachable unicast addresses', () => { + for (const address of ['8.8.8.8', '2001:4860:4860::8888', '::ffff:8.8.8.8']) { + expect(isPublicIpAddress(address), address).toBe(true) + } + for (const address of [ + '0.0.0.0', + '10.0.0.1', + '100.64.0.1', + '127.0.0.1', + '169.254.169.254', + '192.0.2.1', + '224.0.0.1', + '255.255.255.255', + '::', + '::1', + 'fe80::1', + 'fc00::1', + 'ff02::1', + '::ffff:127.0.0.1', + '64:ff9b::808:808', + 'not-an-ip', + ]) { + expect(isPublicIpAddress(address), address).toBe(false) + } + }) + + it('retains one fully public DNS answer set', async () => { + const resolver = vi.fn(async () => [ + { address: '8.8.4.4', family: 4 }, + { address: '2001:4860:4860::8888', family: 6 }, + ]) + await expect(resolvePublicAddresses('example.test', new AbortController().signal, resolver)) + .resolves.toEqual([ + { address: '8.8.4.4', family: 4 }, + { address: '2001:4860:4860::8888', family: 6 }, + ]) + }) + + it('rejects the whole DNS answer set when one address is not public', async () => { + const resolver = vi.fn(async () => [ + { address: '8.8.8.8', family: 4 }, + { address: '127.0.0.1', family: 4 }, + ]) + await expect(resolvePublicAddresses('rebinding.test', new AbortController().signal, resolver)) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' })) + }) + + it('rejects empty and invalid resolver results', async () => { + await expect(resolvePublicAddresses('empty.test', new AbortController().signal, async () => [])) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) + await expect(resolvePublicAddresses('family.test', new AbortController().signal, async () => [{ address: '8.8.8.8', family: 0 }])) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) + await expect(resolvePublicAddresses('mismatch.test', new AbortController().signal, async () => [{ address: '::1', family: 4 }])) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) + }) + + it('validates bracketed IPv6 literals without invoking DNS', async () => { + const resolver = vi.fn(async () => []) + await expect(resolvePublicAddresses('[2001:4860:4860::8888]', new AbortController().signal, resolver)) + .resolves.toEqual([{ address: '2001:4860:4860::8888', family: 6 }]) + expect(resolver).not.toHaveBeenCalled() + }) + + it('stops waiting for DNS when the request is aborted', async () => { + let finish!: (value: never[]) => void + const resolver = vi.fn(() => new Promise((resolve) => { finish = resolve })) + const controller = new AbortController() + const pending = resolvePublicAddresses('slow.test', controller.signal, resolver) + controller.abort(new Error('stop')) + await expect(pending).rejects.toThrow('web fetch aborted during hostname resolution') + finish([]) + + const alreadyAborted = new AbortController() + alreadyAborted.abort(new Error('already stopped')) + await expect(resolvePublicAddresses('slow.test', alreadyAborted.signal, resolver)) + .rejects.toThrow('web fetch aborted during hostname resolution') + }) + + it('propagates resolver failures', async () => { + await expect(resolvePublicAddresses('broken.test', new AbortController().signal, async () => { throw new Error('dns failed') })) + .rejects.toThrow('dns failed') + }) + + it('serves only the retained addresses through the connector lookup', async () => { + const lookup = createPinnedLookup([ + { address: '8.8.8.8', family: 4 }, + { address: '2001:4860:4860::8888', family: 6 }, + ]) + const call = (options: Parameters[1]) => new Promise<{ + error: NodeJS.ErrnoException | null + address: string | import('node:dns').LookupAddress[] + family: number | undefined + }>((resolve) => { + lookup('fixed.test', options, (error, address, family) => { resolve({ error, address, family }) }) + }) + + await expect(call({ all: true })).resolves.toMatchObject({ + error: null, + address: [{ address: '8.8.8.8', family: 4 }, { address: '2001:4860:4860::8888', family: 6 }], + }) + await expect(call({ family: 4 })).resolves.toMatchObject({ error: null, address: '8.8.8.8', family: 4 }) + await expect(call({ family: 'IPv6' })).resolves.toMatchObject({ error: null, address: '2001:4860:4860::8888', family: 6 }) + await expect(call({ family: 'IPv4' })).resolves.toMatchObject({ error: null, address: '8.8.8.8', family: 4 }) + await expect(call({ family: 7 })).resolves.toMatchObject({ error: { code: 'ENOTFOUND' }, address: '', family: 7 }) + await expect(call({ family: 7, all: true })).resolves.toMatchObject({ error: { code: 'ENOTFOUND' }, address: [], family: 7 }) + }) + + it('pins the connection to the validated address without resolving the URL hostname again', async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('pinned') } + const { port } = server.address() as AddressInfo + const request = await requestPinned( + new URL(`http://does-not-resolve.invalid:${port}/`), + [{ address: '127.0.0.1', family: 4 }], + {}, + new AbortController().signal, + ) + try { + await expect(request.response.text()).resolves.toBe('pinned') + } finally { + await request.close() + } + }) +}) + describe('HttpFetchProvider success', () => { it('fetches a text body', async () => { handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('hello world') } @@ -274,6 +404,12 @@ describe('HttpFetchProvider redirects', () => { }) describe('HttpFetchProvider invalid URLs and abort', () => { + it('blocks a loopback destination before opening a connection', async () => { + restoreResolution() + await expect(provider().fetch({ url: base })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' })) + }) + it('rejects a non-http scheme before any network access', async () => { await expect(provider().fetch({ url: 'ftp://example.com' })) .rejects.toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' })) @@ -342,9 +478,16 @@ describe('HttpFetchProvider body cancellation on error paths', () => { return { response, cancelled: () => cancelled } } + function stubRequest(response: Response): void { + vi.spyOn(publicHttpNetwork, 'request').mockResolvedValue({ + response: response as never, + close: async () => {}, + }) + } + it('cancels the body when a cross-origin redirect is blocked', async () => { const { response, cancelled } = fakeResponse({ status: 302, headers: {}, location: 'https://elsewhere.test/' }) - vi.stubGlobal('fetch', vi.fn(async () => response)) + stubRequest(response) await expect(provider().fetch({ url: 'http://127.0.0.1:9/' })) .rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' })) expect(cancelled()).toBe(true) @@ -352,7 +495,7 @@ describe('HttpFetchProvider body cancellation on error paths', () => { it('cancels the body when an unsupported charset is rejected', async () => { const { response, cancelled } = fakeResponse({ status: 200, headers: { 'content-type': 'text/plain; charset=not-a-charset' } }) - vi.stubGlobal('fetch', vi.fn(async () => response)) + stubRequest(response) await expect(provider().fetch({ url: 'http://127.0.0.1:9/' })) .rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' })) expect(cancelled()).toBe(true) @@ -360,7 +503,7 @@ describe('HttpFetchProvider body cancellation on error paths', () => { it('cancels the body when a redirect has no Location header', async () => { const { response, cancelled } = fakeResponse({ status: 302, headers: {} }) - vi.stubGlobal('fetch', vi.fn(async () => response)) + stubRequest(response) await expect(provider().fetch({ url: 'http://127.0.0.1:9/' })) .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) expect(cancelled()).toBe(true) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index e763188fd0..6e60469e14 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9243,6 +9243,12 @@ importers: '@deepseek-ai/schemastery': specifier: link:../../../vendor/schemastery version: link:../../../vendor/schemastery + ipaddr.js: + specifier: ^2.5.0 + version: 2.5.0 + undici: + specifier: ^8.10.0 + version: 8.10.0 devDependencies: '@deepseek-ai/cordis': specifier: workspace:^ @@ -13963,6 +13969,10 @@ packages: resolution: {integrity: sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==} engines: {node: '>= 0.10'} + ipaddr.js@2.5.0: + resolution: {integrity: sha512-aq+t5NAc+cS6rZQQVWC2x98CPqGtKKTMDd4Gaodv0wShnItdKg/51djkGJ1hqH+Oy0ivDftCbSLCQob8zso01w==} + engines: {node: '>= 10'} + is-docker@3.0.0: resolution: {integrity: sha512-eljcgEDlEns/7AXFosB5K/2nCM4P7FQPkGc/DWLy5rmFEWvZayGrik1d9/QIY5nJ4f9YsVvBkA6kJpHn9rISdQ==} engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} @@ -15377,6 +15387,10 @@ packages: resolution: {integrity: sha512-cRZYrTDwWznlnRiPjggAGxZXanty6M8RV1ff8Wm4LWXBp7/IG8v5DnOm74DtUBp9OONpK75YlPnIjQqX0dBDtA==} engines: {node: '>=20.18.1'} + undici@8.10.0: + resolution: {integrity: sha512-HvltHd7avK13QIw/oLe4qoOLyoVSoafqJ2jYOrtMRBkbYT31eiBQ8O0ehRKZiEZCMEyLFQNIADpgCWC5fALvYQ==} + engines: {node: '>=22.19.0'} + unicorn-magic@0.3.0: resolution: {integrity: sha512-+QBBXBCvifc56fsbuxZQ6Sic3wqqc3WWaqxs58gvJrcOuN83HGTCwz3oS5phzU9LthRNE9VrJCFCLUgHeeFnfA==} engines: {node: '>=18'} @@ -19411,6 +19425,8 @@ snapshots: ipaddr.js@1.9.1: {} + ipaddr.js@2.5.0: {} + is-docker@3.0.0: {} is-extglob@2.1.1: {} @@ -21083,6 +21099,8 @@ snapshots: undici@7.28.0: {} + undici@8.10.0: {} + unicorn-magic@0.3.0: {} union@0.5.0: