diff --git a/.gitignore b/.gitignore index b52f86cd61..2788817b23 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,9 @@ examples/*/.sessions/ coverage/ .doc-typecheck-*/ .humanize/ +tmp/ +.claude/commands/ +.claude/settings.json .vscode/ .DS_Store .idea diff --git a/docs/architecture.md b/docs/architecture.md index c76d8f7ba4..dbaf76cd3f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -24,6 +24,9 @@ For a catalog of the **data structures** this architecture moves around — the │ @deepseek-ai/dsh-agent-loop (the ONE concrete plugin) │ │ @deepseek-ai/dsh-bash-local (bash impl) │ │ @deepseek-ai/dsh-tool-bash (bash tool schemas) │ +│ @deepseek-ai/dsh-web-search-exa (web search impl) │ +│ @deepseek-ai/dsh-web-fetch-local (web fetch impl) │ +│ @deepseek-ai/dsh-tool-web (web tool schemas) │ │ @deepseek-ai/dsh-session-persistence-jsonl (persistence impl)│ ├─────────────────────────────────────────────────────────────┤ │ @deepseek-ai/dsh-agent (vocabulary + registry) │ @@ -33,6 +36,7 @@ For a catalog of the **data structures** this architecture moves around — the │ @deepseek-ai/dsh-session-persistence (persistence seam) │ │ @deepseek-ai/dsh-llm (abstract model service) │ │ @deepseek-ai/dsh-bash (abstract bash executor) │ +│ @deepseek-ai/dsh-web (abstract web access) │ ├─────────────────────────────────────────────────────────────┤ │ vendor/: cordis, loader, include, group, timer, hmr, │ │ logger-console, cosmokit, schemastery │ @@ -54,6 +58,7 @@ Dependency rule: **extension** plugins depend on interface packages, never on `d | `ctx.agentLoop` | `AgentLoop` | dsh-agent-loop | creates `ReactLoopAgent`s and drives their loops | | `ctx.bash` | `BashExecutor` (abstract) | dsh-bash | bash execution seam: foreground runs + background tasks | | `ctx.compact` | `CompactService` (abstract) | dsh-compact | compaction seam: decide when history is too large, summarize an older range into a single surface node | +| `ctx.web` | `WebService` | dsh-web | web access seam: search/fetch provider registries, registration-order-independent selection, the `WebError` taxonomy | All registrations (`registerAdapter`, `section`, `tools`, `register`, …) go through `ctx.effect()` and return disposers, so plugin hot-reload (vendored HMR) and fiber disposal clean up automatically. @@ -69,6 +74,8 @@ Swappable capabilities are split into **three packages** so each part evolves in The LLM seam has the same topology folded differently: `dsh-llm` carries the interface (`LlmAdapter`) AND the consumer surface (`ctx.llm.stream()`), with adapters as implementation packages — there the consumer is the loop itself, not a swappable schema surface. Use the full three-package split when the consumer is independently replaceable; keep interface + consumer together when they are one concern. Don't split preemptively: a capability with one conceivable implementation and one consumer stays one package until proven otherwise. +The web capability uses the same three-package split but folds two capabilities onto one seam: `dsh-web` owns the abstract `ctx.web` service, which is a provider REGISTRY (`registerSearchProvider`/`registerFetchProvider`, registration-order-independent selection, the `WebError` taxonomy) rather than a single backend. Providers register capabilities, not tools — `dsh-web-search-exa`, `dsh-web-search-perplexity`, and `dsh-web-fetch-local` each register into `ctx.web` the way an `LlmAdapter` registers into `ctx.llm`, so they are namespace plugins (`inject: ['web']`), not key-owning services. `dsh-tool-web` is the single consumer that owns the model-facing `web_search`/`web_fetch` schemas, prompt sections, and presentation; it reads only the aggregated `ctx.web.searchStatus()`/`fetchStatus()` and executes through `ctx.web.search()`/`fetch()`, so provider selection has one owner. Search and fetch are deliberately one seam (one thing to inject and configure, one selection policy, one abort/error vocabulary) despite sharing no request schema — see the [web capability seam RFC](rfc/implemented/architecture/2026-06-24-web-capability-seam.md). + > **"Capability" — two unrelated meanings.** (1) The *seam pattern* above ("one plugin provides a capability, another needs it") is realized by plain Cordis **services + `inject`**: a provider registers a service (`ctx.bash`, declared in `interface Context`); a consumer declares `inject: ['bash']` and its fiber stays pending until the service exists, tearing down via HMR if it later vanishes. No extra library is needed. (2) `@cordisjs/plugin-capability` is a different axis entirely — a **permission/capability-security** service (named permissions with inheritance/dependency, tested against a session via `ctx.capability.test`). It is a candidate for the deferred permissions/sandbox work (the `tools/execute` veto seam), NOT a mechanism for swapping implementations. ## The vocabulary (dsh-llm) diff --git a/docs/rfc/README.md b/docs/rfc/README.md index 73a2675caa..ab3e97e4bc 100644 --- a/docs/rfc/README.md +++ b/docs/rfc/README.md @@ -59,7 +59,6 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r |---|---| | [Runtime schemas for the event vocabulary (Zod vs the merge-extensible-map pattern)](proposed/architecture/2026-06-16-typed-event-schemas.md) | 2026-06-16 | | [Extract a generic long-running tool runtime](proposed/architecture/2026-06-20-generic-long-running-tool-runtime.md) | 2026-06-20 | -| [Web capability seam - provider registry and model-facing web tools](proposed/architecture/2026-06-24-web-capability-seam.md) | 2026-06-24 | ### Process @@ -120,6 +119,7 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r | [Reorganize packages into a modular hierarchy](implemented/architecture/2026-06-20-package-hierarchy.md) | 2026-06-20 | | [Branded IDs everywhere they belong](implemented/architecture/2026-06-20-branded-ids.md) | 2026-06-20 | | [Extract example apps into packages](implemented/architecture/2026-06-20-extract-example-app-packages.md) | 2026-06-20 | +| [Web capability seam — provider registry and model-facing web tools](implemented/architecture/2026-06-24-web-capability-seam.md) | 2026-06-24 | ### Process diff --git a/docs/rfc/proposed/architecture/2026-06-24-web-capability-seam.md b/docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md similarity index 96% rename from docs/rfc/proposed/architecture/2026-06-24-web-capability-seam.md rename to docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md index e97e8380da..3b9fb166d0 100644 --- a/docs/rfc/proposed/architecture/2026-06-24-web-capability-seam.md +++ b/docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md @@ -1,6 +1,6 @@ # RFC: Web capability seam - stable tools over multiple providers -Status: proposed +Status: implemented ## Problem @@ -64,7 +64,7 @@ flowchart LR `@deepseek-ai/dsh-web` depends only on Cordis and low-level harness support. It declares `ctx.web`, provider interfaces, request/result types, status types, and error codes. It does not import tool, agent, session, LLM, or provider packages. -Provider packages depend on `@deepseek-ai/dsh-web` and Cordis. They own credentials, endpoint config, provider-specific request mapping, provider-specific response parsing, and provider-specific error translation into `WebError`. They issue network requests with the platform-native `fetch` (Node 24), mirroring `@deepseek-ai/dsh-llm-deepseek`'s adapter, NOT a cordis HTTP-client service (`ctx.http`/`@cordisjs/plugin-http`) — even where a Perplexity provider's request is shaped like an OpenAI-compatible chat completion, that wire shape is a provider-private detail and does not make the provider depend on `ctx.llm`. +Provider packages depend on `@deepseek-ai/dsh-web` and Cordis. They own credentials, endpoint config, provider-specific request mapping, provider-specific response parsing, and provider-specific error translation into `WebError`. They issue network requests with the platform-native `fetch` (Node 24), mirroring `@deepseek-ai/dsh-llm-deepseek`'s adapter, NOT a cordis HTTP-client service (`ctx.http`/`@cordisjs/plugin-http`) — even where a Perplexity provider's request is shaped like an OpenAI-compatible chat completion, that wire shape is a provider-private detail and does not make the provider depend on `ctx.llm`. A provider does NOT own the `ctx.web` key (two search providers cannot both own it): like `dsh-llm-deepseek`, each provider package is a function/namespace plugin (`inject: ['web']`) whose `apply` constructs the backend and calls `ctx.web.registerSearchProvider` / `registerFetchProvider`. `@deepseek-ai/dsh-web` is the `export default` service that owns the key. `@deepseek-ai/dsh-tool-web` depends on `@deepseek-ai/dsh-web`, `@deepseek-ai/dsh-tools`, `@deepseek-ai/dsh-system-prompt`, and Cordis. It never imports concrete provider packages. @@ -267,11 +267,11 @@ SSRF / private-network protection (blocking private, loopback, link-local, multi Tool registration in the first version is a minimal stable sync: -1. On plugin startup, read the product/app config that enables or disables web search and web fetch. -2. If web search is enabled, register `web_search` and keep that tool's disposer. -3. If web fetch is enabled, register `web_fetch` and keep that tool's disposer. +1. On plugin startup, read the `dsh-tool-web` `Config` (`search?: boolean`, `fetch?: boolean`, both default `true`) that enables or disables each web tool. +2. If web search is enabled, register `web_search` (its disposer is fiber-scoped via the effect-based registry). +3. If web fetch is enabled, register `web_fetch` (likewise fiber-scoped). 4. Do not dispose either tool merely because `ctx.web.searchStatus()` or `ctx.web.fetchStatus()` is unavailable. -5. Dispose registered tools when the `tool-web` fiber is disposed. +5. Disposing the `tool-web` fiber tears down its registrations automatically. Provider status changes affect execution results and diagnostics, not whether the model-facing schema exists. If a product wants no web tools at all, it disables `dsh-tool-web` or the individual web tool in config; if it wants web tools but the backend is misconfigured, the model sees a structured tool error at execution time. @@ -315,7 +315,7 @@ Search provider tests cover request mapping, response parsing into `content` plu Integration tests should load the real seam, provider, and tool packages together and execute through `ctx.tools.execute()` rather than calling providers directly. If wiring the tools into an ACP-facing example changes editor-visible transcripts, add or update the relevant snapshot scenario in the same change. -At least one test must drive these packages through their REAL cordis Loader/export path, not a hand-built `ctx.plugin({...})` mount, so a broken export shape is caught (see [docs/postmortem/0001](../../../postmortem/0001-acp-default-export-drops-inject.md) and `packages/AGENTS.md` § plugin-export-shape). The two shapes need different guards: `dsh-web` and the provider packages are **services** (`export default` the class) and a stray extra export would surface as a missing service; `dsh-tool-web` is a **namespace plugin** (named `name`/`inject`/`apply`, NO default), and because it has `inject`, a stray `export default apply` makes `unwrapExports` drop the `inject` and the plugin throws `cannot get property … without inject` the moment it loads — so a Loader smoke that boots tool-web over `ctx.web` catches it. Prove the guard bites: add `export default apply` to `tool-web`, watch the smoke go red, revert. +At least one test must drive these packages through their REAL cordis Loader/export path, not a hand-built `ctx.plugin({...})` mount, so a broken export shape is caught (see [docs/postmortem/0001](../../../postmortem/0001-acp-default-export-drops-inject.md) and `packages/AGENTS.md` § plugin-export-shape). The two shapes need different guards: `dsh-web` is a **service** (`export default` the class) and a stray extra export would surface as a missing service; the provider packages and `dsh-tool-web` are **namespace plugins** (named `name`/`inject`/`apply`, NO default), and because each has `inject`, a stray `export default apply` makes `unwrapExports` drop the `inject` and the plugin throws `cannot get property … without inject` the moment it loads — so a Loader smoke that boots tool-web over `ctx.web` catches it (and each provider's registration test mounts it the real way and asserts no default export). Prove the guard bites: add `export default apply` to `tool-web`, watch the smoke go red, revert. ## Migration plan diff --git a/knip.json b/knip.json index 67d99a861d..3f0a56097c 100644 --- a/knip.json +++ b/knip.json @@ -29,6 +29,14 @@ "entry": ["tests/**/*.spec.ts", "tests/**/*.e2e.ts"], "project": ["src/**/*.ts", "tests/**/*.ts"] }, + "packages/web/web-search-exa": { + "entry": ["tests/**/*.spec.ts", "tests/**/*.e2e.ts"], + "project": ["src/**/*.ts", "tests/**/*.ts"] + }, + "packages/web/web-search-perplexity": { + "entry": ["tests/**/*.spec.ts", "tests/**/*.e2e.ts"], + "project": ["src/**/*.ts", "tests/**/*.ts"] + }, "packages/ui/acp-agent": { "entry": ["tests/**/*.spec.ts", "tests/**/*.e2e.ts"], "project": ["src/**/*.ts", "tests/**/*.ts"] diff --git a/packages/README.md b/packages/README.md index 11cace9017..ab81df0a66 100644 --- a/packages/README.md +++ b/packages/README.md @@ -13,6 +13,7 @@ Packages are grouped by modular role at `packages///`. The group dir | [`bash/`](bash/README.md) | Bash capability family: the executor seam, a local impl, and the model-facing tool | Product — stable surface | | [`compact/`](compact/README.md) | Compaction capability family: the abstract seam (backend + tool deferred) | Product — stable surface | | [`subagent/`](subagent/README.md) | Subagent capability family: the provider-registry seam and the model-facing delegation tool | Product — stable surface | +| [`web/`](web/README.md) | Web capability family: the abstract seam, search/fetch provider impls, and the model-facing web tools | Product — stable surface | | [`session-persistence/`](session-persistence/README.md) | Persistence capability family: the seam + JSONL/SQLite backends | Product — stable surface | | [`ui/`](ui/README.md) | Editor/client integration surfaces (the ACP bridge) | Product — stable surface | | [`support/`](support/README.md) | Dev/test/example infrastructure (invariants, stdio UI, replay adapter) | Support — lower compatibility expectations | @@ -33,6 +34,11 @@ dsh-compact ← dsh-session, dsh-llm (abstract compaction s dsh-tools ← dsh-llm, dsh-system-prompt, dsh-agent dsh-bash-local ← dsh-bash (BashExecutor impl) dsh-tool-bash ← dsh-bash, dsh-tools (bash tool schemas) +dsh-web ← dsh-llm (abstract web seam; search/fetch registries, WebError) +dsh-web-search-exa ← dsh-web (Exa WebSearchProvider) +dsh-web-search-perplexity ← dsh-web (Perplexity WebSearchProvider) +dsh-web-fetch-local ← dsh-web (anonymous public HTTP(S) WebFetchProvider) +dsh-tool-web ← dsh-web, dsh-tools, dsh-system-prompt (web tool schemas) dsh-llm-deepseek ← dsh-llm (DeepSeek adapter) dsh-llm-pi-ai ← dsh-llm (pi-ai-backed adapter) dsh-agent-loop ← dsh-llm, dsh-session, dsh-system-prompt, dsh-tools, dsh-agent @@ -68,6 +74,11 @@ The rule: **extension** plugins depend on interfaces, never on the concrete loop | `bash-local/` | `bash` | Local-subprocess `BashExecutor` implementation | (registers `ctx.bash`) | | `tool-bash/` | `bash` | Model-facing `bash`/`bash_output`/`bash_kill` tool schemas | (registers on `ctx.tools`) | | `compact/` | `compact` | Abstract compaction seam + `compact/*` events + `CompactionResult` | `ctx.compact` | +| `web/` | `web` | Abstract web seam (search/fetch provider registries + selection + vocabulary + `WebError`) | `ctx.web` | +| `web-search-exa/` | `web` | Exa-backed `WebSearchProvider` | (registers on `ctx.web`) | +| `web-search-perplexity/` | `web` | Perplexity-backed `WebSearchProvider` | (registers on `ctx.web`) | +| `web-fetch-local/` | `web` | Anonymous public HTTP(S) `WebFetchProvider` | (registers on `ctx.web`) | +| `tool-web/` | `web` | Model-facing `web_search`/`web_fetch` tool schemas | (registers on `ctx.tools`) | | `llm-deepseek/` | `llm` | DeepSeek API adapter (hand-rolled fetch/SSE) | (registers on `ctx.llm`) | | `llm-pi-ai/` | `llm` | DeepSeek adapter via `@earendil-works/pi-ai` (design twin) | (registers on `ctx.llm`) | | `session-persistence/` | `session-persistence` | Persistence seam + write coordinator | `ctx.sessionPersistence` | diff --git a/packages/web/README.md b/packages/web/README.md new file mode 100644 index 0000000000..0742d9c2cc --- /dev/null +++ b/packages/web/README.md @@ -0,0 +1,15 @@ +# web/ - web capability family + +The web access capability seam: an abstract web interface, search/fetch provider implementations, and the model-facing web tools. All **product** packages. + +| Package | Role | ctx key | +|---|---|---| +| `web/` | Abstract web seam (search/fetch provider registries + selection + vocabulary + `WebError`) | `ctx.web` | +| `web-search-exa/` | Exa-backed `WebSearchProvider` | (registers on `ctx.web`) | +| `web-search-perplexity/` | Perplexity-backed `WebSearchProvider` | (registers on `ctx.web`) | +| `web-fetch-local/` | Anonymous public HTTP(S) `WebFetchProvider` | (registers on `ctx.web`) | +| `tool-web/` | Model-facing `web_search`/`web_fetch` tool schemas | (registers on `ctx.tools`) | + +The interface lives at `web/web/`. Unlike bash/fs, the seam spans **two capabilities** (search and fetch) with potentially multiple providers each: `ctx.web` is one web-access middle layer with one provider-selection policy, one abort/error vocabulary, and one product-facing "how this harness reaches the web" config surface. Providers register **capabilities**, not tools; `tool-web` is the only owner of model-facing names, schemas, prompt guidance, and presentation. A search provider swap does not change how the model asks for a query, and a fetch implementation swap does not change how the model asks for a URL. + +See the [web capability seam RFC](../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md) for the design rationale, including why search and fetch are deliberately one seam and why `web_fetch`'s SSRF protection is deferred. diff --git a/packages/web/tool-web/README.md b/packages/web/tool-web/README.md new file mode 100644 index 0000000000..762bfe0189 --- /dev/null +++ b/packages/web/tool-web/README.md @@ -0,0 +1,30 @@ +# @deepseek-ai/dsh-tool-web + +The model-facing web tool suite — `web_search` and `web_fetch` — over the [web capability seam](../web/README.md) (`ctx.web`). It owns model-facing concerns only: tool names, JSON schemas, snake_case argument names, prompt sections, the result-count bound, result formatting, HTML→markdown presentation, and `presentCall`. All web access goes through `ctx.web`; this package never imports a concrete provider. + +Each tool is also a subpath plugin (`@deepseek-ai/dsh-tool-web/search`, `/fetch`) for focused deployments. + +## Tools + +| Tool | Args | Behavior | +|---|---|---| +| `web_search` | `query` (string) | Discovery. Returns an optional answer plus source URLs. `max_results` is **not** model-facing — the tool sets the bound (`WEB_SEARCH_MAX_RESULTS = 8`) and passes it to the seam. | +| `web_fetch` | `url` (string), `timeout_ms` (number, optional) | Retrieves a specific URL. HTML bodies are rendered to markdown-ish text; text bodies pass through. A non-2xx status is reported, not an error. | + +## Config + +| Key | Default | Meaning | +|---|---|---| +| `search` | `true` | Register `web_search`. | +| `fetch` | `true` | Register `web_fetch`. | + +```yaml +- id: tool-web + name: '@deepseek-ai/dsh-tool-web' +``` + +## Stable registration + +Tool registration follows product **enablement**, not backend availability. A tool stays visible even when its selected provider is missing, misconfigured, ambiguous, or temporarily unavailable; the seam resolves the provider at execution time and execution fails with a structured `WebError` (e.g. `WEB_PROVIDER_UNAVAILABLE`, `WEB_PROVIDER_AMBIGUOUS`), which `ToolRegistry.execute()` turns into an error tool result the model can read and hooks/UI can route on. This keeps the model schema stable without making plugin load order, credential state, or HMR timing part of the model-facing contract. To remove a web tool entirely, disable it here in config. + +The tool reads only the aggregated `ctx.web.searchStatus()` / `fetchStatus()` for diagnostics — never each provider's `status()` directly — so provider selection has one owner. diff --git a/packages/web/tool-web/package.json b/packages/web/tool-web/package.json new file mode 100644 index 0000000000..8d46faa157 --- /dev/null +++ b/packages/web/tool-web/package.json @@ -0,0 +1,42 @@ +{ + "name": "@deepseek-ai/dsh-tool-web", + "description": "Model-facing web tools (web_search, web_fetch) over the DeepSeek Harness web capability seam (ctx.web)", + "version": "0.0.1", + "private": true, + "type": "module", + "main": "lib/index.js", + "types": "lib/index.d.ts", + "exports": { + ".": { "types": "./lib/index.d.ts", "default": "./lib/index.js" }, + "./search": { "types": "./lib/search.d.ts", "default": "./lib/search.js" }, + "./fetch": { "types": "./lib/fetch.d.ts", "default": "./lib/fetch.js" }, + "./src/*": "./src/*", + "./package.json": "./package.json" + }, + "files": [ + "lib", + "src" + ], + "license": "BSD-3-Clause", + "peerDependencies": { + "@deepseek-ai/dsh-llm": "^0.0.1", + "@deepseek-ai/dsh-system-prompt": "^0.0.1", + "@deepseek-ai/dsh-tools": "^0.0.1", + "@deepseek-ai/dsh-web": "^0.0.1", + "cordis": "^4.0.0-rc.6" + }, + "dependencies": { + "schemastery": "^3.18.0" + }, + "devDependencies": { + "@deepseek-ai/dsh-agent": "workspace:^", + "@deepseek-ai/dsh-llm": "workspace:^", + "@deepseek-ai/dsh-session": "workspace:^", + "@deepseek-ai/dsh-system-prompt": "workspace:^", + "@deepseek-ai/dsh-tools": "workspace:^", + "@deepseek-ai/dsh-web": "workspace:^", + "@deepseek-ai/dsh-web-fetch-local": "workspace:^", + "@deepseek-ai/dsh-web-search-exa": "workspace:^", + "cordis": "^4.0.0-rc.6" + } +} diff --git a/packages/web/tool-web/src/fetch.ts b/packages/web/tool-web/src/fetch.ts new file mode 100644 index 0000000000..a48ad41414 --- /dev/null +++ b/packages/web/tool-web/src/fetch.ts @@ -0,0 +1,87 @@ +/** + * The model-facing `web_fetch` tool: retrieve the content of a specific URL. + * Execution goes through `ctx.web` — this module owns the model-facing schema, + * argument validation, and PRESENTATION (HTML→markdown, truncation formatting), + * while the fetch provider owns safe retrieval (transport, redirects, caps). + * + * @module @deepseek-ai/dsh-tool-web/fetch + */ + +import type { Context } from 'cordis' +import { defineTool } from '@deepseek-ai/dsh-tools' +import type { ToolCallPresentation } from '@deepseek-ai/dsh-tools' +import type { ContentBlock } from '@deepseek-ai/dsh-llm' +import type { WebFetchBody, WebFetchResult } from '@deepseek-ai/dsh-web' +import { assertNever } from '@deepseek-ai/dsh-llm' +import type {} from '@deepseek-ai/dsh-system-prompt' +import { htmlToMarkdown } from './html.ts' + +/** Validate value constraints the schema DSL can't express. */ +export function parseFetchArgs(args: { url: string; timeout_ms?: number }): { url: string; timeoutMs?: number } { + if (args.url.trim().length === 0) throw new Error('url must be a non-empty string') + if (args.timeout_ms !== undefined && (!Number.isFinite(args.timeout_ms) || args.timeout_ms <= 0)) { + throw new Error('timeout_ms must be a positive number') + } + return { url: args.url, ...args.timeout_ms !== undefined ? { timeoutMs: args.timeout_ms } : {} } +} + +/** Render a fetched body to model-facing markdown text. */ +export function renderBody(body: WebFetchBody): string { + switch (body.kind) { + case 'html': + return htmlToMarkdown(body.content) + case 'text': + return body.content + /* v8 ignore next 2 -- WebFetchBody is a closed union; this arm is unreachable and only makes adding a kind a compile error. */ + default: + return assertNever(body, 'unhandled web fetch body kind') + } +} + +/** Format a fetch result as one model-facing text block. */ +export function formatFetchOutput(result: WebFetchResult): string { + const header = `Fetched ${result.url} (HTTP ${result.statusCode})` + const footer = result.truncated ? '\n\n(Content truncated. Fetch a more specific URL or section for the full text.)' : '' + return `${header}\n\n${renderBody(result.body)}${footer}` +} + +/** Pending-call presentation: a fetch card titled by the URL. */ +export function presentFetchCall(args: { url: string; timeout_ms?: number }): ToolCallPresentation { + return { title: args.url, kind: 'fetch', rawInput: args.url } +} + +/** Register the `web_fetch` tool and its system-prompt guidance. */ +export function apply(ctx: Context): void { + ctx.systemPrompt.section({ + name: 'tool:web_fetch', + order: 111, + text: 'Use the web_fetch tool to retrieve the content of a specific HTTP(S) URL (for example a result from web_search). It returns the page content decoded to text. Cite the URL as a markdown link when you use its content.', + }) + + ctx.tools.register(defineTool({ + name: 'web_fetch', + description: 'Fetch the content of a specific HTTP(S) URL and return it decoded to text.', + parameters: { + url: { type: 'string', required: true, description: 'The HTTP(S) URL to fetch.' }, + timeout_ms: { type: 'number', description: 'Optional fetch timeout in milliseconds (capped by the provider).' }, + }, + async execute(args, exec): Promise { + const input = parseFetchArgs(args) + const result = await ctx.web.fetch( + { url: input.url, ...input.timeoutMs !== undefined ? { timeoutMs: input.timeoutMs } : {} }, + exec.signal ? { signal: exec.signal } : undefined, + ) + return [{ type: 'text', text: formatFetchOutput(result) }] + }, + presentCall: presentFetchCall, + })) +} + +/** Cordis plugin name used by loader diagnostics. */ +export const name = 'web-fetch' + +/** Services required by the `web_fetch` tool plugin. */ +export const inject = ['tools', 'web', 'systemPrompt'] + +/** Named helper for direct registration in the root plugin and tests. */ +export const applyWebFetchTool = apply diff --git a/packages/web/tool-web/src/html.ts b/packages/web/tool-web/src/html.ts new file mode 100644 index 0000000000..622be86fd5 --- /dev/null +++ b/packages/web/tool-web/src/html.ts @@ -0,0 +1,85 @@ +/** + * Minimal, dependency-free HTML→markdown-ish text conversion for `web_fetch` + * presentation. This is intentionally NOT a full HTML parser: it strips + * script/style/noscript, drops tags, decodes the common named/numeric entities, + * and collapses whitespace into a readable plain-text approximation with a few + * markdown affordances (headings, list bullets, links). A heavier converter can + * replace this without touching the seam or the tool schema. + * + * @module @deepseek-ai/dsh-tool-web/html + */ + +/** Decode the handful of HTML entities common in textual content. */ +function decodeEntities(text: string): string { + return text + .replace(/&(#[xX][0-9a-fA-F]+|#[0-9]+|[a-zA-Z]+);/g, (match, entity: string) => { + if (entity.startsWith('#x') || entity.startsWith('#X')) { + const code = Number.parseInt(entity.slice(2), 16) + return safeFromCodePoint(code, match) + } + if (entity.startsWith('#')) { + const code = Number.parseInt(entity.slice(1), 10) + return safeFromCodePoint(code, match) + } + return NAMED_ENTITIES[entity] ?? match + }) +} + +const NAMED_ENTITIES: Record = { + amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', + copy: '©', reg: '®', trade: '™', hellip: '…', mdash: '—', ndash: '–', +} + +function safeFromCodePoint(code: number, fallback: string): string { + try { + return String.fromCodePoint(code) + } catch { + // An out-of-range code point (RangeError) is the only failure here; keep the + // original entity text rather than throwing out of pure presentation. + return fallback + } +} + +/** + * Convert an HTML document to a readable markdown-ish text approximation. + * Best-effort and lossy by design — fidelity is the job of a future heavier + * converter, not this fallback. + */ +export function htmlToMarkdown(html: string): string { + let text = html + // Drop non-content elements entirely (including their contents). + .replace(/]*>[\s\S]*?<\/script>/gi, '') + .replace(/]*>[\s\S]*?<\/style>/gi, '') + .replace(/]*>[\s\S]*?<\/noscript>/gi, '') + .replace(//g, '') + + // Convert links to markdown before stripping tags. + text = text.replace(/]*\bhref\s*=\s*["']([^"']*)["'][^>]*>([\s\S]*?)<\/a>/gi, (_match, href: string, label: string) => { + const cleanLabel = label.replace(/<[^>]+>/g, '').trim() + return cleanLabel.length > 0 ? `[${cleanLabel}](${href})` : href + }) + + // Headings → markdown hashes. + text = text.replace(/]*>([\s\S]*?)<\/h\1>/gi, (_match, level: string, body: string) => { + const hashes = '#'.repeat(Number(level)) + return `\n\n${hashes} ${body.replace(/<[^>]+>/g, '').trim()}\n\n` + }) + + // List items → bullets. + text = text.replace(/]*>([\s\S]*?)<\/li>/gi, (_match, body: string) => `\n- ${body.replace(/<[^>]+>/g, '').trim()}`) + + // Block-level breaks become paragraph breaks. + text = text + .replace(/<\/(p|div|section|article|header|footer|tr|table|ul|ol|blockquote)>/gi, '\n\n') + .replace(//gi, '\n') + + // Drop all remaining tags, decode entities, collapse whitespace. + text = text.replace(/<[^>]+>/g, '') + text = decodeEntities(text) + text = text + .replace(/[ \t\f\v]+/g, ' ') + .replace(/ *\n */g, '\n') + .replace(/\n{3,}/g, '\n\n') + .trim() + return text +} diff --git a/packages/web/tool-web/src/index.ts b/packages/web/tool-web/src/index.ts new file mode 100644 index 0000000000..031d7fe4cb --- /dev/null +++ b/packages/web/tool-web/src/index.ts @@ -0,0 +1,59 @@ +/** + * The model-facing web tool suite (`web_search`, `web_fetch`) over the `ctx.web` + * seam. This root plugin registers the tools the product has ENABLED, composing + * the per-tool registration helpers; each tool is also exposed as a subpath + * plugin (`@deepseek-ai/dsh-tool-web/search`, `/fetch`) for focused deployments. + * + * The package owns model-facing concerns only — tool names, JSON schemas, + * argument validation, prompt sections, result-cap constants, result formatting, + * HTML→markdown presentation. All web access goes through `ctx.web`; this + * package never imports a concrete provider package. + * + * Tool registration follows product/app ENABLEMENT, not backend availability: a + * tool stays visible even when its selected provider is missing/misconfigured, + * and execution fails with a structured `WebError` (resolved by the seam at call + * time). That keeps the model schema stable without making plugin load order, + * credential state, or HMR timing part of the model-facing contract. + * + * @module @deepseek-ai/dsh-tool-web + */ + +import type { Context } from 'cordis' +import z from 'schemastery' +import type {} from '@deepseek-ai/dsh-web' +import { applyWebSearchTool } from './search.ts' +import { applyWebFetchTool } from './fetch.ts' + +export { WEB_SEARCH_MAX_RESULTS, applyWebSearchTool, formatSearchOutput, parseSearchArgs, presentSearchCall } from './search.ts' +export { applyWebFetchTool, formatFetchOutput, parseFetchArgs, presentFetchCall, renderBody } from './fetch.ts' +export { htmlToMarkdown } from './html.ts' + +/** Cordis plugin name used by loader diagnostics. */ +export const name = 'tool-web' + +/** Services required by the web tool suite. */ +export const inject = ['tools', 'web', 'systemPrompt'] + +export interface Config { + /** Register `web_search`. Defaults to true. */ + search?: boolean + /** Register `web_fetch`. Defaults to true. */ + fetch?: boolean +} + +export const Config: z = z.object({ + search: z.boolean().default(true), + fetch: z.boolean().default(true), +}) + +/** + * Register the enabled web tools. `search`/`fetch` default to true; a product + * that wants only one disables the other in config. The tools' disposers are + * fiber-scoped (the effect-based registries clean up on dispose), so no manual + * teardown is needed. + */ +export function apply(ctx: Context, config: Config): void { + if (config.search !== false) applyWebSearchTool(ctx) + if (config.fetch !== false) applyWebFetchTool(ctx) +} + diff --git a/packages/web/tool-web/src/search.ts b/packages/web/tool-web/src/search.ts new file mode 100644 index 0000000000..6ed9991903 --- /dev/null +++ b/packages/web/tool-web/src/search.ts @@ -0,0 +1,105 @@ +/** + * The model-facing `web_search` tool: discover current information on the web. + * Execution goes through `ctx.web` — this module owns only the model-facing + * schema, argument validation, the result-count bound, and result formatting, + * never provider selection or network access. + * + * @module @deepseek-ai/dsh-tool-web/search + */ + +import type { Context } from 'cordis' +import { defineTool } from '@deepseek-ai/dsh-tools' +import type { ToolCallPresentation } from '@deepseek-ai/dsh-tools' +import type { ContentBlock } from '@deepseek-ai/dsh-llm' +import type { WebSearchResult } from '@deepseek-ai/dsh-web' +import type {} from '@deepseek-ai/dsh-system-prompt' + +/** + * Default upper bound on returned sources. Owned by the consumer (not the + * provider or model), mirroring `dsh-tool-fs`'s `READ_LIMIT`/`GREP_LIMIT`. The + * model just asks a question; the product controls how much context returns. + * The default `8` aligns with OpenCode's Exa default. + */ +export const WEB_SEARCH_MAX_RESULTS = 8 + +/** Validate value constraints the schema DSL can't express. */ +export function parseSearchArgs(args: { query: string }): { query: string } { + if (args.query.trim().length === 0) throw new Error('query must be a non-empty string') + return { query: args.query } +} + +/** Display label for a source: its title, else its hostname. */ +function sourceLabel(url: string, title: string | undefined): string { + if (title !== undefined && title.length > 0) return title + try { + return new URL(url).hostname + } catch { + // A provider should return a valid URL, but never let a malformed one throw + // out of pure formatting — fall back to the raw string. + return url + } +} + +/** Format a search result as one model-facing text block. */ +export function formatSearchOutput(result: WebSearchResult): string { + const parts: string[] = [] + if (result.content !== undefined && result.content.length > 0) parts.push(result.content) + + if (result.sources.length > 0) { + const lines = result.sources.map((source) => { + const label = sourceLabel(source.url, source.title) + const meta: string[] = [] + if (source.snippet !== undefined && source.snippet.length > 0) meta.push(source.snippet) + if (source.publishedAt !== undefined && source.publishedAt.length > 0) meta.push(`(${source.publishedAt})`) + const suffix = meta.length > 0 ? ` — ${meta.join(' ')}` : '' + return `- [${label}](${source.url})${suffix}` + }) + parts.push(`Sources:\n${lines.join('\n')}`) + } else if (result.content === undefined || result.content.length === 0) { + parts.push('No results found.') + } + + if (result.truncated) parts.push(`(Showing the first ${result.sources.length} sources. Refine the query for more.)`) + parts.push('Cite the relevant URLs above as markdown links in your answer.') + return parts.join('\n\n') +} + +/** Pending-call presentation: a search card titled by the query. */ +export function presentSearchCall(args: { query: string }): ToolCallPresentation { + return { title: args.query, kind: 'search', rawInput: args.query } +} + +/** Register the `web_search` tool and its system-prompt guidance. */ +export function apply(ctx: Context): void { + ctx.systemPrompt.section({ + name: 'tool:web_search', + order: 110, + text: 'Use the web_search tool to discover current information on the web. It returns an optional answer plus a list of source URLs. Follow up with web_fetch when you need the full content of a specific result, and cite the relevant URLs as markdown links.', + }) + + ctx.tools.register(defineTool({ + name: 'web_search', + description: 'Search the web for current information. Returns an optional summary answer and a list of source URLs.', + parameters: { + query: { type: 'string', required: true, description: 'The search query.' }, + }, + async execute(args, exec): Promise { + const input = parseSearchArgs(args) + const result = await ctx.web.search( + { query: input.query, maxResults: WEB_SEARCH_MAX_RESULTS }, + exec.signal ? { signal: exec.signal } : undefined, + ) + return [{ type: 'text', text: formatSearchOutput(result) }] + }, + presentCall: presentSearchCall, + })) +} + +/** Cordis plugin name used by loader diagnostics. */ +export const name = 'web-search' + +/** Services required by the `web_search` tool plugin. */ +export const inject = ['tools', 'web', 'systemPrompt'] + +/** Named helper for direct registration in the root plugin and tests. */ +export const applyWebSearchTool = apply diff --git a/packages/web/tool-web/tests/integration.spec.ts b/packages/web/tool-web/tests/integration.spec.ts new file mode 100644 index 0000000000..18604190c6 --- /dev/null +++ b/packages/web/tool-web/tests/integration.spec.ts @@ -0,0 +1,99 @@ +/** + * Integration: the real fetch backend (`dsh-web-fetch-local`) + a real search + * provider (`dsh-web-search-exa`) + the real seam (`dsh-web`) + the model tool + * (`dsh-tool-web`), exercised through `ctx.tools.execute()` — nothing bypasses + * the tool registry. Fetch hits a real loopback HTTP server (verifying the + * WORLD); search runs the real Exa provider over a stubbed global `fetch` (the + * network is the one boundary we mock). + */ + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http' +import { AddressInfo } from 'node:net' +import { Context } from 'cordis' +import { CallId } from '@deepseek-ai/dsh-llm' +import SystemPrompt from '@deepseek-ai/dsh-system-prompt' +import ToolRegistry from '@deepseek-ai/dsh-tools' +import WebService from '@deepseek-ai/dsh-web' +import * as WebFetchLocal from '@deepseek-ai/dsh-web-fetch-local' +import * as WebSearchExa from '@deepseek-ai/dsh-web-search-exa' +import * as ToolWeb from '@deepseek-ai/dsh-tool-web' + +type Handler = (req: IncomingMessage, res: ServerResponse) => void + +let server: Server +let base: string +let handler: Handler +let ctx: Context +let fiber: Awaited> + +beforeEach(async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/html' }); res.end('

Hello

World

') } + server = createServer((req, res) => { handler(req, res) }) + await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)) + base = `http://127.0.0.1:${(server.address() as AddressInfo).port}` + + ctx = new Context() + await ctx.plugin(SystemPrompt) + await ctx.plugin(ToolRegistry) + await ctx.plugin(WebService, { searchProvider: WebSearchExa.EXA_PROVIDER_ID, fetchProvider: WebFetchLocal.LOCAL_FETCH_PROVIDER_ID }) + await ctx.plugin(WebFetchLocal, {}) + await ctx.plugin(WebSearchExa, { apiKey: 'exa-key', baseURL: 'https://api.exa.test' }) + fiber = await ctx.plugin(ToolWeb) +}) + +afterEach(async () => { + await fiber.dispose() + vi.unstubAllGlobals() + await new Promise(resolve => server.close(() => { resolve() })) +}) + +let counter = 0 +type ToolResult = { isError: boolean; content: { type: string; text?: string }[]; error?: { code: string } } +function call(name: string, args: unknown): Promise { + return ctx.tools.execute({ callId: CallId(`call-${++counter}`), name, arguments: args }) +} + +describe('web_fetch integration over the real backend', () => { + it('fetches an html page and renders it to markdown', async () => { + const out = await call('web_fetch', { url: base }) + expect(out.isError).toBe(false) + const text = out.content.map(b => b.text).join('') + expect(text).toContain(`Fetched ${base}`) + expect(text).toContain('# Hello') + expect(text).toContain('World') + }) + + it('reports a 404 as a result, not an error', async () => { + handler = (_req, res) => { res.writeHead(404, { 'content-type': 'text/plain' }); res.end('missing') } + const out = await call('web_fetch', { url: base }) + expect(out.isError).toBe(false) + expect(out.content.map(b => b.text).join('')).toContain('HTTP 404') + }) + + it('surfaces WEB_INVALID_URL as a structured tool error', async () => { + const out = await call('web_fetch', { url: 'ftp://example.com' }) + expect(out.isError).toBe(true) + expect(out.error?.code).toBe('WEB_INVALID_URL') + }) + + it('surfaces a blocked cross-origin redirect as WEB_REDIRECT_BLOCKED', async () => { + handler = (_req, res) => { res.writeHead(302, { location: 'https://example.com/' }); res.end() } + const out = await call('web_fetch', { url: base }) + expect(out.isError).toBe(true) + expect(out.error?.code).toBe('WEB_REDIRECT_BLOCKED') + }) +}) + +describe('web_search integration over the real Exa provider', () => { + it('runs web_search end-to-end and formats the provider result', async () => { + vi.stubGlobal('fetch', vi.fn(async () => new Response( + JSON.stringify({ results: [{ url: 'https://result.test', title: 'Result', highlights: ['a highlight'] }] }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ))) + const out = await call('web_search', { query: 'deepseek' }) + expect(out.isError).toBe(false) + expect(out.content.map(b => b.text).join('')).toContain('[Result](https://result.test)') + }) +}) + diff --git a/packages/web/tool-web/tests/load-path.spec.ts b/packages/web/tool-web/tests/load-path.spec.ts new file mode 100644 index 0000000000..5c47f3ce59 --- /dev/null +++ b/packages/web/tool-web/tests/load-path.spec.ts @@ -0,0 +1,49 @@ +/** + * Real-load-path guard for @deepseek-ai/dsh-tool-web. `tool-web` is a NAMESPACE + * plugin with `inject` — so a stray `export default apply` would make the cordis + * Loader's `unwrapExports` (`exports.default ?? exports`) collapse the module to + * the bare `apply` function, DROPPING `inject`. The plugin would then read + * `ctx.web` without having injected it and throw `cannot get property … without + * inject` the moment it loads (postmortem 0001). + * + * A hand-built `ctx.plugin({ apply, inject })` mount CANNOT catch that — it + * bypasses `unwrapExports`. So this test unwraps the module through the REAL + * `Loader.prototype.unwrapExports` and mounts the result over `ctx.web`, + * exercising the exact path the Loader uses. Prove the guard bites: add + * `export default apply` to `src/index.ts`, watch this go red, revert. + */ + +import { describe, expect, it } from 'vitest' +import { Context } from 'cordis' +import Loader from '@cordisjs/plugin-loader' +import SystemPrompt from '@deepseek-ai/dsh-system-prompt' +import ToolRegistry from '@deepseek-ai/dsh-tools' +import WebService from '@deepseek-ai/dsh-web' +import * as toolWeb from '@deepseek-ai/dsh-tool-web' + +describe('dsh-tool-web real-load-path guard', () => { + it('has no default export and keeps name/inject/Config through unwrapExports', () => { + expect('default' in toolWeb).toBe(false) + + const loader = Object.create(Loader.prototype) as Loader + const unwrapped = loader.unwrapExports(toolWeb) as Record + expect(unwrapped).toBe(toolWeb) + expect(unwrapped.name).toBe('tool-web') + expect(unwrapped.inject).toEqual(['tools', 'web', 'systemPrompt']) + expect(typeof unwrapped.apply).toBe('function') + }) + + it('boots over ctx.web through the unwrapped module without an inject error', async () => { + const ctx = new Context() + await ctx.plugin(SystemPrompt) + await ctx.plugin(ToolRegistry) + await ctx.plugin(WebService, {}) + + const loader = Object.create(Loader.prototype) as Loader + const unwrapped = loader.unwrapExports(toolWeb) as Parameters[0] + // A collapsed export shape (dropped inject) would throw "without inject" here. + const fiber = await ctx.plugin(unwrapped) + expect(ctx.tools.schemas().map(s => s.name)).toEqual(expect.arrayContaining(['web_search', 'web_fetch'])) + await fiber.dispose() + }) +}) diff --git a/packages/web/tool-web/tests/tool-web.spec.ts b/packages/web/tool-web/tests/tool-web.spec.ts new file mode 100644 index 0000000000..2422c32ce3 --- /dev/null +++ b/packages/web/tool-web/tests/tool-web.spec.ts @@ -0,0 +1,281 @@ +import { describe, expect, it } from 'vitest' +import { Context } from 'cordis' +import { CallId } from '@deepseek-ai/dsh-llm' +import SystemPrompt from '@deepseek-ai/dsh-system-prompt' +import ToolRegistry from '@deepseek-ai/dsh-tools' +import WebService from '@deepseek-ai/dsh-web' +import type { WebSearchProvider, WebSearchResult, WebProviderStatus } from '@deepseek-ai/dsh-web' +import * as ToolWeb from '@deepseek-ai/dsh-tool-web' +import { + formatSearchOutput, + formatFetchOutput, + parseSearchArgs, + parseFetchArgs, + presentSearchCall, + presentFetchCall, + renderBody, + htmlToMarkdown, +} from '@deepseek-ai/dsh-tool-web' + +const available: WebProviderStatus = { available: true } + +function searchProvider(result: WebSearchResult, status: WebProviderStatus = available): WebSearchProvider { + return { id: 'stub-search', status: () => status, search: () => Promise.resolve(result) } +} + +/** Mount the real registry, seam, and tool-web; return an executor helper. */ +async function mountTools(opts: { + config?: ToolWeb.Config + webConfig?: ConstructorParameters[1] + search?: WebSearchProvider + fetchProvider?: import('@deepseek-ai/dsh-web').WebFetchProvider +} = {}): Promise<{ ctx: Context; fiber: Awaited>; call: (name: string, args: unknown) => Promise<{ isError: boolean; content: { type: string; text?: string }[]; error?: { code: string } }> }> { + const ctx = new Context() + await ctx.plugin(SystemPrompt) + await ctx.plugin(ToolRegistry) + await ctx.plugin(WebService, opts.webConfig ?? {}) + if (opts.search) ctx.web.registerSearchProvider(opts.search) + if (opts.fetchProvider) ctx.web.registerFetchProvider(opts.fetchProvider) + const fiber = await ctx.plugin(ToolWeb, opts.config ?? {}) + let counter = 0 + const call = (name: string, args: unknown) => ctx.tools.execute({ callId: CallId(`call-${++counter}`), name, arguments: args }) as never + return { ctx, fiber, call } +} + +describe('search formatting', () => { + it('renders content, sources with titles/hostnames, snippets, and a citation reminder', () => { + const out = formatSearchOutput({ + providerId: 'p', query: 'q', content: 'an answer', truncated: false, + sources: [ + { url: 'https://a.test/x', title: 'A', snippet: 'about a', publishedAt: '2026-01-01' }, + { url: 'https://b.test/y' }, + ], + }) + expect(out).toContain('an answer') + expect(out).toContain('[A](https://a.test/x) — about a (2026-01-01)') + expect(out).toContain('[b.test](https://b.test/y)') + expect(out).toContain('Cite the relevant URLs') + }) + + it('reports no results when there is neither content nor sources', () => { + expect(formatSearchOutput({ providerId: 'p', query: 'q', sources: [], truncated: false })) + .toContain('No results found.') + }) + + it('renders content alone when there are no sources', () => { + const out = formatSearchOutput({ providerId: 'p', query: 'q', content: 'just an answer', sources: [], truncated: false }) + expect(out).toContain('just an answer') + expect(out).not.toContain('No results found.') + expect(out).not.toContain('Sources:') + }) + + it('notes truncation', () => { + const out = formatSearchOutput({ providerId: 'p', query: 'q', sources: [{ url: 'https://a.test' }], truncated: true }) + expect(out).toContain('Showing the first 1 sources') + }) + + it('validates the query', () => { + expect(() => parseSearchArgs({ query: ' ' })).toThrow('non-empty') + expect(parseSearchArgs({ query: 'hi' })).toEqual({ query: 'hi' }) + }) + + it('presents a search call as a search-kind card titled by the query', () => { + expect(presentSearchCall({ query: 'find me' })).toEqual({ title: 'find me', kind: 'search', rawInput: 'find me' }) + }) +}) + +describe('fetch formatting', () => { + it('renders an html body to markdown text with a status header', () => { + const out = formatFetchOutput({ + providerId: 'p', url: 'https://a.test', statusCode: 200, truncated: false, + body: { kind: 'html', content: '

Title

Body text

' }, + }) + expect(out).toContain('Fetched https://a.test (HTTP 200)') + expect(out).toContain('# Title') + expect(out).toContain('Body text') + }) + + it('passes a text body through and notes truncation', () => { + const out = formatFetchOutput({ + providerId: 'p', url: 'https://a.test', statusCode: 200, truncated: true, + body: { kind: 'text', content: 'plain' }, + }) + expect(out).toContain('plain') + expect(out).toContain('Content truncated') + }) + + it('renderBody dispatches on kind', () => { + expect(renderBody({ kind: 'text', content: 'x' })).toBe('x') + expect(renderBody({ kind: 'html', content: '

y

' })).toBe('y') + }) + + it('validates url and timeout', () => { + expect(() => parseFetchArgs({ url: ' ' })).toThrow('non-empty') + expect(() => parseFetchArgs({ url: 'https://a.test', timeout_ms: -1 })).toThrow('positive') + expect(parseFetchArgs({ url: 'https://a.test', timeout_ms: 5 })).toEqual({ url: 'https://a.test', timeoutMs: 5 }) + }) + + it('presents a fetch call as a fetch-kind card titled by the url', () => { + expect(presentFetchCall({ url: 'https://a.test' })).toEqual({ title: 'https://a.test', kind: 'fetch', rawInput: 'https://a.test' }) + }) +}) + +describe('htmlToMarkdown', () => { + it('drops scripts/styles, keeps text, decodes entities, converts links', () => { + const md = htmlToMarkdown('

Tom & Jerry

link') + expect(md).not.toContain('bad()') + expect(md).not.toContain('.x{}') + expect(md).toContain('Tom & Jerry') + expect(md).toContain('[link](https://a.test)') + }) + + it('decodes numeric entities and collapses whitespace', () => { + expect(htmlToMarkdown('

a'b

')).toBe("a'b") + expect(htmlToMarkdown('
x
\n\n\n
y
')).toBe('x\n\ny') + }) + + it('decodes hex entities and named entities, and leaves unknown/out-of-range ones intact', () => { + expect(htmlToMarkdown('

AB

')).toBe('AB') + expect(htmlToMarkdown('

© —

')).toBe('© —') + expect(htmlToMarkdown('

¬areal;

')).toBe('¬areal;') + // An out-of-range code point keeps the original entity text (fromCodePoint fallback). + expect(htmlToMarkdown('

�

')).toBe('�') + expect(htmlToMarkdown('

�

')).toBe('�') + }) + + it('renders a link with an empty label as its bare href', () => { + expect(htmlToMarkdown('')).toBe('https://a.test') + }) + + it('converts headings and list items to markdown', () => { + expect(htmlToMarkdown('

Heading

after

')).toContain('## Heading') + const list = htmlToMarkdown('
  • one
  • two
') + expect(list).toContain('- one') + expect(list).toContain('- two') + }) + + it('falls back to the raw URL as a source label when the URL is unparseable', () => { + const out = formatSearchOutput({ providerId: 'p', query: 'q', truncated: false, sources: [{ url: 'not a url' }] }) + expect(out).toContain('[not a url](not a url)') + }) +}) + +describe('tool-web registration', () => { + it('registers both tools by default', async () => { + const { fiber, ctx } = await mountTools() + const names = ctx.tools.schemas().map(s => s.name) + expect(names).toContain('web_search') + expect(names).toContain('web_fetch') + await fiber.dispose() + expect(ctx.tools.schemas().map(s => s.name)).not.toContain('web_search') + }) + + it('registers only enabled tools', async () => { + const { fiber, ctx } = await mountTools({ config: { search: true, fetch: false } }) + const names = ctx.tools.schemas().map(s => s.name) + expect(names).toContain('web_search') + expect(names).not.toContain('web_fetch') + await fiber.dispose() + }) + + it('registers only web_fetch when search is disabled', async () => { + const { fiber, ctx } = await mountTools({ config: { search: false, fetch: true } }) + const names = ctx.tools.schemas().map(s => s.name) + expect(names).not.toContain('web_search') + expect(names).toContain('web_fetch') + await fiber.dispose() + }) + + it('registers web_search even when no provider is available (schema follows enablement, not availability)', async () => { + const { fiber, ctx } = await mountTools() + expect(ctx.tools.schemas().map(s => s.name)).toContain('web_search') + expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'none' }) + await fiber.dispose() + }) + + it('contributes prompt sections for the enabled tools', async () => { + const { fiber, ctx } = await mountTools() + const prompt = await ctx.systemPrompt.assemble() + const text = prompt.sections.map(s => (typeof s.text === 'function' ? s.text() : s.text)).join('\n') + expect(text).toContain('web_search') + expect(text).toContain('web_fetch') + await fiber.dispose() + }) +}) + +describe('tool-web execution through the real registry', () => { + it('executes web_search and formats the result', async () => { + const result: WebSearchResult = { + providerId: 'stub-search', query: 'q', content: 'answer', truncated: false, + sources: [{ url: 'https://a.test', title: 'A', snippet: 'snip' }], + } + const { fiber, call } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: searchProvider(result) }) + const out = await call('web_search', { query: 'q' }) + expect(out.isError).toBe(false) + expect(out.content.map(b => b.text).join('')).toContain('[A](https://a.test)') + await fiber.dispose() + }) + + it('surfaces a structured WebError when no provider is available', async () => { + const { fiber, call } = await mountTools() + const out = await call('web_search', { query: 'q' }) + expect(out.isError).toBe(true) + expect(out.error?.code).toBe('WEB_PROVIDER_UNAVAILABLE') + await fiber.dispose() + }) + + it('surfaces WEB_PROVIDER_AMBIGUOUS for multiple unconfigured providers', async () => { + const { ctx, fiber, call } = await mountTools({ search: searchProvider({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) }) + ctx.web.registerSearchProvider({ id: 'other', status: () => available, search: () => Promise.resolve({ providerId: 'other', query: 'q', sources: [], truncated: false }) }) + const out = await call('web_search', { query: 'q' }) + expect(out.isError).toBe(true) + expect(out.error?.code).toBe('WEB_PROVIDER_AMBIGUOUS') + await fiber.dispose() + }) + + it('rejects invalid arguments with a structured INVALID_ARGS error', async () => { + const { fiber, call } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: searchProvider({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) }) + const out = await call('web_search', { query: 123 }) + expect(out.isError).toBe(true) + expect(out.error?.code).toBe('INVALID_ARGS') + await fiber.dispose() + }) + + it('has no default export (namespace plugin export shape)', () => { + expect('default' in ToolWeb).toBe(false) + }) + + it('executes web_fetch, forwarding timeout_ms and the abort signal to the seam', async () => { + const seen: { request?: { url: string; timeoutMs?: number }; signal?: AbortSignal | undefined } = {} + const fetchProvider = { + id: 'stub-fetch', + status: () => available, + fetch: (request: { url: string; timeoutMs?: number }, exec?: { signal?: AbortSignal }) => { + seen.request = request + seen.signal = exec?.signal + return Promise.resolve({ providerId: 'stub-fetch', url: request.url, statusCode: 200, body: { kind: 'text' as const, content: 'ok' }, truncated: false }) + }, + } + const { ctx, fiber } = await mountTools({ webConfig: { fetchProvider: 'stub-fetch' }, fetchProvider }) + const controller = new AbortController() + const out = await ctx.tools.execute({ callId: CallId('fetch-1'), name: 'web_fetch', arguments: { url: 'https://a.test', timeout_ms: 1234 }, signal: controller.signal }) + expect(out.isError).toBe(false) + expect(seen.request).toEqual({ url: 'https://a.test', timeoutMs: 1234 }) + expect(seen.signal).toBe(controller.signal) + await fiber.dispose() + }) + + it('executes web_search, forwarding the abort signal to the seam', async () => { + const seen: { signal?: AbortSignal | undefined } = {} + const provider: WebSearchProvider = { + id: 'stub-search', + status: () => available, + search: (_request, exec) => { seen.signal = exec?.signal; return Promise.resolve({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) }, + } + const { ctx, fiber } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: provider }) + const controller = new AbortController() + await ctx.tools.execute({ callId: CallId('search-1'), name: 'web_search', arguments: { query: 'q' }, signal: controller.signal }) + expect(seen.signal).toBe(controller.signal) + await fiber.dispose() + }) +}) diff --git a/packages/web/tool-web/tsconfig.json b/packages/web/tool-web/tsconfig.json new file mode 100644 index 0000000000..b4121a6c14 --- /dev/null +++ b/packages/web/tool-web/tsconfig.json @@ -0,0 +1,17 @@ +{ + "extends": "../../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib" + }, + "include": ["src"], + "references": [ + { "path": "../../../vendor/cosmokit" }, + { "path": "../../../vendor/cordis" }, + { "path": "../../../vendor/schemastery" }, + { "path": "../../llm/llm" }, + { "path": "../../core/tools" }, + { "path": "../../core/system-prompt" }, + { "path": "../web" } + ] +} diff --git a/packages/web/tool-web/tsdown.config.ts b/packages/web/tool-web/tsdown.config.ts new file mode 100644 index 0000000000..c4849939db --- /dev/null +++ b/packages/web/tool-web/tsdown.config.ts @@ -0,0 +1,18 @@ +import { defineConfig } from 'tsdown' + +/** + * tool-web exposes one package root plus one entry per tool plugin, so each tool + * can be loaded or replaced independently as a subpath plugin + * (`@deepseek-ai/dsh-tool-web/search`, `/fetch`). The root tsdown config only + * auto-discovers `src/index.ts`, so the subpath entries are declared here. + */ +export default defineConfig({ + entry: ['src/index.ts', 'src/search.ts', 'src/fetch.ts'], + outDir: 'lib', + format: ['esm'], + platform: 'node', + target: 'es2024', + fixedExtension: false, + dts: false, + clean: false, +}) diff --git a/packages/web/web-fetch-local/README.md b/packages/web/web-fetch-local/README.md new file mode 100644 index 0000000000..fd9150e46f --- /dev/null +++ b/packages/web/web-fetch-local/README.md @@ -0,0 +1,34 @@ +# @deepseek-ai/dsh-web-fetch-local + +An anonymous public HTTP(S) `WebFetchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It retrieves a concrete URL and returns a status code plus bounded decoded content. + +This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the key and it does not register a model-facing tool. It is a function/namespace plugin (`inject: ['web']`). + +## Responsibility split + +The provider owns **safe resource retrieval**: URL validation, HTTP transport, redirect policy, timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. `@deepseek-ai/dsh-tool-web` owns **presentation** (HTML→markdown, truncation formatting). A non-2xx HTTP response is a *result* (status code + decoded body), not an error; `WebError` is reserved for failures to safely retrieve or represent the resource. + +## Transport hygiene + +- Accepts only `http:` and `https:` URLs; rejects credentials in URLs (`WEB_BLOCKED_URL`) and over-long/malformed URLs (`WEB_INVALID_URL`). +- Enforces a max URL length, response byte cap (`WEB_FETCH_TOO_LARGE`), decoded body character cap, timeout (`WEB_FETCH_TIMEOUT`), and redirect hop cap. +- Propagates the caller's abort signal (`WEB_ABORTED`) into the network request and the streaming read. +- Follows only **same-origin** redirects; a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (the model of Claude Code's WebFetch). +- Sends an explicit product `User-Agent`, never a browser disguise. +- Rejects unsupported (e.g. binary) content types with `WEB_UNSUPPORTED_CONTENT_TYPE`. + +## Config + +| Key | Default | Meaning | +|---|---|---| +| `maxUrlLength` | `2048` | Maximum accepted request URL length. | +| `maxResponseBytes` | `5_000_000` | Maximum response body size in bytes. | +| `maxBodyChars` | `100_000` | Maximum decoded body length in characters. | +| `timeoutMs` | `30_000` | Default fetch timeout. | +| `maxTimeoutMs` | `120_000` | Upper bound for a per-request timeout override. | +| `maxRedirects` | `5` | Maximum same-origin redirect hops. | +| `userAgent` | `deepseek-harness/…` | `User-Agent` header. | + +## Security note + +SSRF / private-network protection (blocking private, loopback, link-local, multicast, and otherwise non-public destinations, with DNS-resolve-then-validate and per-hop re-validation) is **deferred** — see the [web capability seam RFC](../../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md). Until it lands, this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets. diff --git a/packages/web/web-fetch-local/package.json b/packages/web/web-fetch-local/package.json new file mode 100644 index 0000000000..7249697ec3 --- /dev/null +++ b/packages/web/web-fetch-local/package.json @@ -0,0 +1,33 @@ +{ + "name": "@deepseek-ai/dsh-web-fetch-local", + "description": "Anonymous public HTTP(S) fetch provider for the DeepSeek Harness web capability seam (ctx.web)", + "version": "0.0.1", + "private": true, + "type": "module", + "main": "lib/index.js", + "types": "lib/index.d.ts", + "exports": { + ".": { + "types": "./lib/index.d.ts", + "default": "./lib/index.js" + }, + "./src/*": "./src/*", + "./package.json": "./package.json" + }, + "files": [ + "lib", + "src" + ], + "license": "BSD-3-Clause", + "peerDependencies": { + "@deepseek-ai/dsh-web": "^0.0.1", + "cordis": "^4.0.0-rc.6" + }, + "dependencies": { + "schemastery": "^3.18.0" + }, + "devDependencies": { + "@deepseek-ai/dsh-web": "workspace:^", + "cordis": "^4.0.0-rc.6" + } +} diff --git a/packages/web/web-fetch-local/src/index.ts b/packages/web/web-fetch-local/src/index.ts new file mode 100644 index 0000000000..1d57410d68 --- /dev/null +++ b/packages/web/web-fetch-local/src/index.ts @@ -0,0 +1,77 @@ +/** + * `@deepseek-ai/dsh-web-fetch-local`: registers an anonymous public HTTP(S) + * `WebFetchProvider` with `ctx.web`. A function/namespace plugin (NOT a + * default-export service): it registers INTO the seam's fetch registry, like the + * search providers register into the search registry. + * + * @module @deepseek-ai/dsh-web-fetch-local + */ + +import type { Context } from 'cordis' +import z from 'schemastery' +import type {} from '@deepseek-ai/dsh-web' +import { LocalFetchProvider } from './provider.ts' +import type { LocalFetchLimits } from './provider.ts' + +export { + LOCAL_FETCH_PROVIDER_ID, + LocalFetchProvider, +} from './provider.ts' +export type { LocalFetchLimits } from './provider.ts' +export { classifyContentType, isSameOrigin, validateFetchUrl } from './policy.ts' +export type { FetchableKind } from './policy.ts' + +/** Default `User-Agent`: an explicit product agent, never a browser disguise. */ +export const DEFAULT_USER_AGENT = 'deepseek-harness/0.0.1 (+https://github.com/deepseek-ai)' + +/** Cordis plugin name used by loader diagnostics. */ +export const name = 'web-fetch-local' + +/** The web seam this provider registers into. */ +export const inject = ['web'] + +export interface Config { + /** Maximum accepted request URL length. */ + maxUrlLength?: number + /** Maximum response body size in bytes. */ + maxResponseBytes?: number + /** Maximum decoded body length in characters. */ + maxBodyChars?: number + /** Default fetch timeout in milliseconds. */ + timeoutMs?: number + /** Upper bound for a per-request timeout override. */ + maxTimeoutMs?: number + /** Maximum number of same-origin redirect hops to follow. */ + maxRedirects?: number + /** `User-Agent` header sent on every request. */ + userAgent?: string +} + +export const Config: z = z.object({ + maxUrlLength: z.number().default(2048), + maxResponseBytes: z.number().default(5_000_000), + maxBodyChars: z.number().default(100_000), + timeoutMs: z.number().default(30_000), + maxTimeoutMs: z.number().default(120_000), + maxRedirects: z.number().default(5), + userAgent: z.string().default(DEFAULT_USER_AGENT), +}) + +/** The shape after schemastery applies its defaults to every field. */ +type ResolvedConfig = Required + +/** Register the local HTTP(S) fetch provider with `ctx.web`. */ +export function apply(ctx: Context, config: Config): void { + // schemastery (Config) has already filled every defaulted field. + const resolved = config as ResolvedConfig + const limits: LocalFetchLimits = { + maxUrlLength: resolved.maxUrlLength, + maxResponseBytes: resolved.maxResponseBytes, + maxBodyChars: resolved.maxBodyChars, + timeoutMs: resolved.timeoutMs, + maxTimeoutMs: resolved.maxTimeoutMs, + maxRedirects: resolved.maxRedirects, + userAgent: resolved.userAgent, + } + ctx.web.registerFetchProvider(new LocalFetchProvider(limits)) +} diff --git a/packages/web/web-fetch-local/src/policy.ts b/packages/web/web-fetch-local/src/policy.ts new file mode 100644 index 0000000000..8261c5c1ed --- /dev/null +++ b/packages/web/web-fetch-local/src/policy.ts @@ -0,0 +1,59 @@ +/** + * URL validation and content-type classification for the local HTTP(S) fetch + * provider — the pure, network-free half. The provider's `fetch()` composes + * these with transport (redirect following, byte caps, decoding). + * + * @module @deepseek-ai/dsh-web-fetch-local/policy + */ + +import { WebError } from '@deepseek-ai/dsh-web' + +/** The body kinds this provider decodes. */ +export type FetchableKind = 'html' | 'text' + +/** + * Validate a request URL against the basic transport hygiene the provider + * enforces before any network access: http(s) only, no embedded credentials, + * bounded length. Returns the parsed `URL`. Throws {@link WebError} otherwise. + * (SSRF / private-network blocking is deferred — see the package RFC.) + */ +export function validateFetchUrl(input: string, maxUrlLength: number): URL { + if (input.length > maxUrlLength) { + throw new WebError(`URL exceeds the maximum length of ${maxUrlLength}`, 'WEB_INVALID_URL') + } + let url: URL + try { + url = new URL(input) + } catch (error: unknown) { + throw new WebError(`invalid URL: ${input}`, 'WEB_INVALID_URL', { cause: error }) + } + if (url.protocol !== 'http:' && url.protocol !== 'https:') { + throw new WebError(`unsupported URL scheme "${url.protocol}" (only http and https are allowed)`, 'WEB_INVALID_URL') + } + if (url.username.length > 0 || url.password.length > 0) { + throw new WebError('credentials in URLs are not allowed', 'WEB_BLOCKED_URL') + } + return url +} + +/** + * Two URLs are same-origin when scheme, hostname, and port match. A redirect + * that crosses origins is refused so each new origin requires a fresh tool call + * (and thus a fresh provider/permission decision). + */ +export function isSameOrigin(a: URL, b: URL): boolean { + return a.protocol === b.protocol && a.hostname === b.hostname && a.port === b.port +} + +/** + * Classify a response `Content-Type` into a decodable body kind, or `undefined` + * for an unsupported (e.g. binary) type. `text/html` and `application/xhtml+xml` + * are `html`; other `text/*` plus a few structured text types are `text`. + */ +export function classifyContentType(contentType: string | null): FetchableKind | undefined { + const mime = (contentType ?? '').replace(/;.*$/s, '').trim().toLowerCase() + if (mime === 'text/html' || mime === 'application/xhtml+xml') return 'html' + if (mime.startsWith('text/')) return 'text' + if (mime === 'application/json' || mime === 'application/xml' || mime.endsWith('+json') || mime.endsWith('+xml')) return 'text' + return undefined +} diff --git a/packages/web/web-fetch-local/src/provider.ts b/packages/web/web-fetch-local/src/provider.ts new file mode 100644 index 0000000000..0622030af0 --- /dev/null +++ b/packages/web/web-fetch-local/src/provider.ts @@ -0,0 +1,233 @@ +/** + * `LocalFetchProvider`: a `WebFetchProvider` that retrieves a concrete public + * HTTP(S) URL with the platform-native `fetch` (Node 24) and returns a status + * code plus bounded decoded content. It owns SAFE RESOURCE RETRIEVAL — URL + * validation, redirect policy, timeout, abort, byte caps, charset decoding, + * content-type classification, binary rejection — but NOT presentation + * (HTML→markdown lives in `@deepseek-ai/dsh-tool-web`). + * + * Redirects are followed manually (`redirect: 'manual'`) so the provider can + * enforce a same-origin-only policy: a cross-origin redirect is refused with + * `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (Claude Code's WebFetch + * uses the same model). It does NOT carry browser cookies, editor/git + * credentials, or implicit access to private services. + * + * SSRF / private-network protection is DEFERRED (see the package RFC); until it + * lands this provider is an SSRF primitive and must not be enabled where it can + * reach sensitive internal targets. + * + * @module @deepseek-ai/dsh-web-fetch-local/provider + */ + +import { WebError } from '@deepseek-ai/dsh-web' +import type { WebFetchBody, WebFetchProvider, WebFetchRequest, WebFetchResult, WebProviderStatus } from '@deepseek-ai/dsh-web' +import { classifyContentType, isSameOrigin, validateFetchUrl } from './policy.ts' + +/** Resolved provider limits (the plugin's schemastery Config supplies defaults). */ +export interface LocalFetchLimits { + /** Maximum accepted request URL length. */ + maxUrlLength: number + /** Maximum response body size in bytes (read is aborted past this). */ + maxResponseBytes: number + /** Maximum decoded body length in characters (truncated past this). */ + maxBodyChars: number + /** Default fetch timeout in milliseconds. */ + timeoutMs: number + /** Upper bound for a per-request timeout override. */ + maxTimeoutMs: number + /** Maximum number of (same-origin) redirect hops to follow. */ + maxRedirects: number + /** `User-Agent` header sent on every request. */ + userAgent: string +} + +/** Stable id this provider registers under. */ +export const LOCAL_FETCH_PROVIDER_ID = 'local-http' + +/** The anonymous public HTTP(S) fetch provider. */ +export class LocalFetchProvider implements WebFetchProvider { + readonly id = LOCAL_FETCH_PROVIDER_ID + + constructor(private readonly limits: LocalFetchLimits) {} + + /** No credentials to check — an anonymous public fetcher is always usable. */ + status(): WebProviderStatus { + return { available: true } + } + + async fetch(request: WebFetchRequest, exec?: { readonly signal?: AbortSignal }): Promise { + const timeoutMs = request.timeoutMs !== undefined + ? Math.min(request.timeoutMs, this.limits.maxTimeoutMs) + : this.limits.timeoutMs + + // One controller drives both the caller's abort and our own timeout, so the + // network request and the streaming read both stop on either. + const controller = new AbortController() + const onAbort = (): void => { controller.abort() } + if (exec?.signal !== undefined) { + if (exec.signal.aborted) throw new WebError('web fetch aborted', 'WEB_ABORTED') + exec.signal.addEventListener('abort', onAbort, { once: true }) + } + const timer = setTimeout(() => { controller.abort(new WebError('web fetch timed out', 'WEB_FETCH_TIMEOUT')) }, timeoutMs) + + try { + return await this.followAndRead(request.url, controller, timeoutMs) + } finally { + clearTimeout(timer) + if (exec?.signal !== undefined) exec.signal.removeEventListener('abort', onAbort) + } + } + + /** Follow same-origin redirects up to the hop cap, then read the final response. */ + private async followAndRead(initialUrl: string, controller: AbortController, timeoutMs: number): Promise { + let currentUrl = validateFetchUrl(initialUrl, this.limits.maxUrlLength) + + for (let hop = 0; hop <= this.limits.maxRedirects; hop++) { + const response = await this.requestOnce(currentUrl, controller, timeoutMs) + + if (isRedirectStatus(response.status)) { + const location = response.headers.get('location') + if (location === null) { + // A redirect status with no Location is not a usable resource. + throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, 'WEB_PROVIDER_ERROR') + } + const target = resolveRedirect(location, currentUrl) + if (!isSameOrigin(target, currentUrl)) { + throw new WebError( + `cross-origin redirect to ${target.origin} is not followed automatically; retry against that URL directly`, + 'WEB_REDIRECT_BLOCKED', + ) + } + await response.body?.cancel() + currentUrl = target + continue + } + + return await this.readBody(response, currentUrl) + } + + throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, 'WEB_REDIRECT_BLOCKED') + } + + private async requestOnce(url: URL, controller: AbortController, _timeoutMs: number): Promise { + try { + return await fetch(url, { + method: 'GET', + redirect: 'manual', + headers: { 'user-agent': this.limits.userAgent, 'accept': 'text/html,application/xhtml+xml,text/*;q=0.9,application/json;q=0.8' }, + signal: controller.signal, + }) + } catch (error: unknown) { + throw translateAbortOrNetwork(error) + } + } + + /** Read, byte-cap, classify, and decode the final response body. */ + private async readBody(response: Response, finalUrl: URL): Promise { + const kind = classifyContentType(response.headers.get('content-type')) + if (kind === undefined) { + await response.body?.cancel() + throw new WebError(`unsupported content type "${response.headers.get('content-type') ?? 'unknown'}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE') + } + + const { bytes, truncatedByBytes } = await this.readCapped(response) + const decoded = new TextDecoder('utf-8').decode(bytes) + const truncatedByChars = decoded.length > this.limits.maxBodyChars + const content = truncatedByChars ? decoded.slice(0, this.limits.maxBodyChars) : decoded + const body: WebFetchBody = kind === 'html' ? { kind: 'html', content } : { kind: 'text', content } + + return { + providerId: this.id, + url: finalUrl.toString(), + statusCode: response.status, + body, + truncated: truncatedByBytes || truncatedByChars, + } + } + + /** + * Read the response stream up to `maxResponseBytes`. A `Content-Length` over + * the cap rejects immediately with `WEB_FETCH_TOO_LARGE`; a stream that grows + * past the cap is cut short (`truncatedByBytes`) rather than rejected, so a + * server that under-reports still yields a bounded usable body. + */ + private async readCapped(response: Response): Promise<{ bytes: Uint8Array; truncatedByBytes: boolean }> { + const declared = response.headers.get('content-length') + if (declared !== null) { + const length = Number(declared) + if (Number.isFinite(length) && length > this.limits.maxResponseBytes) { + await response.body?.cancel() + throw new WebError(`response exceeds the maximum of ${this.limits.maxResponseBytes} bytes`, 'WEB_FETCH_TOO_LARGE') + } + } + + /* v8 ignore next -- a 2xx Response from fetch always exposes a body stream; the null guard is defensive. */ + if (response.body === null) return { bytes: new Uint8Array(0), truncatedByBytes: false } + + const chunks: Uint8Array[] = [] + let total = 0 + let truncatedByBytes = false + const reader = response.body.getReader() + try { + for (;;) { + const { done, value } = await reader.read() + if (done) break + const remaining = this.limits.maxResponseBytes - total + if (value.byteLength >= remaining) { + chunks.push(value.subarray(0, remaining)) + total += remaining + truncatedByBytes = true + break + } + chunks.push(value) + total += value.byteLength + } + } catch (error: unknown) { + /* v8 ignore next -- mid-stream read fault needs a network drop after headers; translate path covered by request-phase tests. */ + throw translateAbortOrNetwork(error) + } finally { + /* v8 ignore next 4 -- cancel() after a completed/broken read settles without rejecting; unobserved best-effort cleanup. */ + await reader.cancel().catch(() => { + // Cancel after a successful read (or after we broke past the cap) is + // best-effort cleanup; the bytes we need are already collected. + }) + } + + const bytes = new Uint8Array(total) + let offset = 0 + for (const chunk of chunks) { + bytes.set(chunk, offset) + offset += chunk.byteLength + } + return { bytes, truncatedByBytes } + } +} + +/** HTTP redirect status codes that carry a `Location`. */ +function isRedirectStatus(status: number): boolean { + return status === 301 || status === 302 || status === 303 || status === 307 || status === 308 +} + +/** Resolve a (possibly relative) `Location` against the current URL. */ +function resolveRedirect(location: string, base: URL): URL { + try { + return new URL(location, base) + } catch (error: unknown) { + /* v8 ignore next 2 -- URL resolution against a valid absolute base effectively never throws; defensive guard. */ + throw new WebError(`invalid redirect Location "${location}"`, 'WEB_PROVIDER_ERROR', { cause: error }) + } +} + +/** + * Translate a thrown fetch/stream error into a `WebError`. Our own + * `WEB_FETCH_TIMEOUT` (passed to `controller.abort(reason)`) and any other + * already-typed `WebError` pass through; an `AbortError` becomes `WEB_ABORTED`; + * anything else is a transport/network failure (`WEB_PROVIDER_ERROR`). + */ +function translateAbortOrNetwork(error: unknown): WebError { + if (error instanceof WebError) return error + if (error instanceof DOMException && error.name === 'AbortError') { + return new WebError('web fetch aborted', 'WEB_ABORTED', { cause: error }) + } + return new WebError(`web fetch failed: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error }) +} diff --git a/packages/web/web-fetch-local/tests/fetch-local.spec.ts b/packages/web/web-fetch-local/tests/fetch-local.spec.ts new file mode 100644 index 0000000000..3ca48150e1 --- /dev/null +++ b/packages/web/web-fetch-local/tests/fetch-local.spec.ts @@ -0,0 +1,239 @@ +import { afterEach, beforeEach, describe, expect, it } from 'vitest' +import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http' +import { AddressInfo } from 'node:net' +import { Context } from 'cordis' +import WebService from '@deepseek-ai/dsh-web' +import { LocalFetchProvider, LOCAL_FETCH_PROVIDER_ID, classifyContentType, isSameOrigin, validateFetchUrl } from '@deepseek-ai/dsh-web-fetch-local' +import type { LocalFetchLimits } from '@deepseek-ai/dsh-web-fetch-local' +import * as fetchPlugin from '@deepseek-ai/dsh-web-fetch-local' + +const limits: LocalFetchLimits = { + maxUrlLength: 2048, + maxResponseBytes: 5_000_000, + maxBodyChars: 100_000, + timeoutMs: 5_000, + maxTimeoutMs: 10_000, + maxRedirects: 5, + userAgent: 'test-agent/1.0', +} + +type Handler = (req: IncomingMessage, res: ServerResponse) => void + +let server: Server +let base: string +let handler: Handler + +beforeEach(async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('default') } + server = createServer((req, res) => { handler(req, res) }) + await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)) + const { port } = server.address() as AddressInfo + base = `http://127.0.0.1:${port}` +}) + +afterEach(async () => { + await new Promise(resolve => server.close(() => { resolve() })) +}) + +function provider(overrides: Partial = {}): LocalFetchProvider { + return new LocalFetchProvider({ ...limits, ...overrides }) +} + +describe('policy helpers', () => { + it('validates scheme, credentials, and length', () => { + expect(validateFetchUrl('https://example.com/x', 2048).hostname).toBe('example.com') + expect(() => validateFetchUrl('ftp://example.com', 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' })) + expect(() => validateFetchUrl('not a url', 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' })) + expect(() => validateFetchUrl('https://user:pass@example.com', 2048)).toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' })) + expect(() => validateFetchUrl(`https://example.com/${'a'.repeat(3000)}`, 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' })) + }) + + it('classifies content types', () => { + expect(classifyContentType('text/html; charset=utf-8')).toBe('html') + expect(classifyContentType('application/xhtml+xml')).toBe('html') + expect(classifyContentType('text/plain')).toBe('text') + expect(classifyContentType('application/json')).toBe('text') + expect(classifyContentType('image/png')).toBeUndefined() + expect(classifyContentType(null)).toBeUndefined() + }) + + it('compares origins', () => { + expect(isSameOrigin(new URL('https://a.com/x'), new URL('https://a.com/y'))).toBe(true) + expect(isSameOrigin(new URL('https://a.com'), new URL('https://b.com'))).toBe(false) + expect(isSameOrigin(new URL('http://a.com'), new URL('https://a.com'))).toBe(false) + }) +}) + +describe('LocalFetchProvider success', () => { + it('fetches a text body', async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('hello world') } + const result = await provider().fetch({ url: base }) + expect(result.providerId).toBe(LOCAL_FETCH_PROVIDER_ID) + expect(result.statusCode).toBe(200) + expect(result.body).toEqual({ kind: 'text', content: 'hello world' }) + expect(result.truncated).toBe(false) + }) + + it('fetches an html body and classifies it as html', async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/html' }); res.end('

hi

') } + const result = await provider().fetch({ url: base }) + expect(result.body).toEqual({ kind: 'html', content: '

hi

' }) + }) + + it('sends the configured user agent', async () => { + let seen: string | undefined + handler = (req, res) => { seen = req.headers['user-agent']; res.writeHead(200, { 'content-type': 'text/plain' }); res.end('ok') } + await provider().fetch({ url: base }) + expect(seen).toBe('test-agent/1.0') + }) + + it('returns a non-2xx response as a result, not an error', async () => { + handler = (_req, res) => { res.writeHead(404, { 'content-type': 'text/plain' }); res.end('nope') } + const result = await provider().fetch({ url: base }) + expect(result.statusCode).toBe(404) + expect(result.body).toEqual({ kind: 'text', content: 'nope' }) + }) +}) + +describe('LocalFetchProvider caps', () => { + it('rejects an over-cap Content-Length with WEB_FETCH_TOO_LARGE', async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain', 'content-length': '999999' }); res.end('x'.repeat(999999)) } + await expect(provider({ maxResponseBytes: 10 }).fetch({ url: base })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_FETCH_TOO_LARGE' })) + }) + + it('truncates a stream that grows past the byte cap', async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('abcdefghij') } + const result = await provider({ maxResponseBytes: 4 }).fetch({ url: base }) + expect(result.body.content).toBe('abcd') + expect(result.truncated).toBe(true) + }) + + it('truncates a decoded body past the character cap', async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('abcdefghij') } + const result = await provider({ maxBodyChars: 3 }).fetch({ url: base }) + expect(result.body.content).toBe('abc') + expect(result.truncated).toBe(true) + }) + + it('rejects an unsupported content type', async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'image/png' }); res.end('binary') } + await expect(provider().fetch({ url: base })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' })) + }) + + it('rejects a response with no content type at all', async () => { + handler = (_req, res) => { res.writeHead(200); res.end('no type') } + await expect(provider().fetch({ url: base })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' })) + }) + + it('accepts a declared content-length within the cap', async () => { + handler = (_req, res) => { const body = 'sized'; res.writeHead(200, { 'content-type': 'text/plain', 'content-length': String(body.length) }); res.end(body) } + const result = await provider().fetch({ url: base }) + expect(result.body.content).toBe('sized') + }) +}) + +describe('LocalFetchProvider redirects', () => { + it('follows a same-origin redirect and reports the final URL', async () => { + handler = (req, res) => { + if (req.url === '/start') { res.writeHead(302, { location: '/end' }); res.end() } + else { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('arrived') } + } + const result = await provider().fetch({ url: `${base}/start` }) + expect(result.body.content).toBe('arrived') + expect(result.url).toBe(`${base}/end`) + }) + + it('blocks a cross-origin redirect with WEB_REDIRECT_BLOCKED', async () => { + handler = (_req, res) => { res.writeHead(302, { location: 'https://example.com/' }); res.end() } + await expect(provider().fetch({ url: base })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' })) + }) + + it('rejects exceeding the redirect hop cap', async () => { + handler = (req, res) => { + const n = Number(new URL(req.url ?? '/', base).searchParams.get('n') ?? '0') + res.writeHead(302, { location: `/?n=${n + 1}` }) + res.end() + } + await expect(provider({ maxRedirects: 2 }).fetch({ url: `${base}/?n=0` })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' })) + }) + + it('treats a redirect without a Location header as a provider error', async () => { + handler = (_req, res) => { res.writeHead(302); res.end() } + await expect(provider().fetch({ url: base })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) + }) + + it('follows a relative same-origin redirect', async () => { + handler = (req, res) => { + if (req.url === '/a') { res.writeHead(301, { location: 'b' }); res.end() } + else { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('landed') } + } + const result = await provider().fetch({ url: `${base}/a` }) + expect(result.body.content).toBe('landed') + }) +}) + +describe('LocalFetchProvider invalid URLs and abort', () => { + it('rejects a non-http scheme before any network access', async () => { + await expect(provider().fetch({ url: 'ftp://example.com' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' })) + }) + + it('rejects credentials in the URL', async () => { + await expect(provider().fetch({ url: 'http://user:pass@127.0.0.1/' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' })) + }) + + it('honors a pre-aborted signal', async () => { + const controller = new AbortController() + controller.abort() + await expect(provider().fetch({ url: base }, { signal: controller.signal })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' })) + }) + + it('aborts an in-flight fetch via the signal', async () => { + handler = (_req, _res) => { /* never responds */ } + const controller = new AbortController() + const promise = provider().fetch({ url: base }, { signal: controller.signal }) + controller.abort() + await expect(promise).rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' })) + }) + + it('times out a slow response with WEB_FETCH_TIMEOUT', async () => { + handler = (_req, _res) => { /* never responds */ } + await expect(provider({ timeoutMs: 50 }).fetch({ url: base })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_FETCH_TIMEOUT' })) + }) + + it('maps a connection failure to WEB_PROVIDER_ERROR', async () => { + // Port 1 on loopback is not listening: a real connection failure (not abort). + await expect(provider().fetch({ url: 'http://127.0.0.1:1/' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) + }) + + it('caps the per-request timeout at maxTimeoutMs', async () => { + handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('ok') } + const result = await provider({ maxTimeoutMs: 10_000 }).fetch({ url: base, timeoutMs: 999_999 }) + expect(result.statusCode).toBe(200) + }) +}) + +describe('web-fetch-local plugin registration', () => { + it('registers the provider into ctx.web (HMR-safe)', async () => { + const ctx = new Context() + await ctx.plugin(WebService, { fetchProvider: LOCAL_FETCH_PROVIDER_ID }) + const fiber = await ctx.plugin(fetchPlugin, {}) + expect(ctx.web.fetchStatus()).toEqual({ available: true, providerId: LOCAL_FETCH_PROVIDER_ID }) + await fiber.dispose() + expect(ctx.web.fetchStatus()).toEqual({ available: false, reason: 'configured-missing' }) + }) + + it('has no default export (namespace plugin export shape)', () => { + expect('default' in fetchPlugin).toBe(false) + }) +}) diff --git a/packages/web/web-fetch-local/tsconfig.json b/packages/web/web-fetch-local/tsconfig.json new file mode 100644 index 0000000000..cb9eb44552 --- /dev/null +++ b/packages/web/web-fetch-local/tsconfig.json @@ -0,0 +1,24 @@ +{ + "extends": "../../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib" + }, + "include": [ + "src" + ], + "references": [ + { + "path": "../../../vendor/cosmokit" + }, + { + "path": "../../../vendor/cordis" + }, + { + "path": "../../../vendor/schemastery" + }, + { + "path": "../web" + } + ] +} diff --git a/packages/web/web-search-exa/README.md b/packages/web/web-search-exa/README.md new file mode 100644 index 0000000000..7f39356e81 --- /dev/null +++ b/packages/web/web-search-exa/README.md @@ -0,0 +1,23 @@ +# @deepseek-ai/dsh-web-search-exa + +An [Exa](https://exa.ai)-backed `WebSearchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It calls Exa's `POST /search` endpoint with highlight contents and maps the flat `results[]` into the seam's normalized `WebSearchResult`. + +This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the `ctx.web` key and it does not register a model-facing tool (that is `@deepseek-ai/dsh-tool-web`). Like `@deepseek-ai/dsh-llm-deepseek`, it is a function/namespace plugin (`inject: ['web']`) that registers its backend, not a default-export service. + +## Config + +| Key | Default | Meaning | +|---|---|---| +| `apiKey` | `$EXA_API_KEY` | Exa API key. Empty/absent → provider `status()` reports `missing-credential` (the seam reports `configured-unavailable`/`none`). | +| `baseURL` | `https://api.exa.ai` | Endpoint base; `/search` is appended. | + +```yaml +- id: web-search-exa + name: '@deepseek-ai/dsh-web-search-exa' + config: + apiKey: !!js process.env.EXA_API_KEY +``` + +## Mapping + +Exa returns a flat `results[]` and no generated answer, so `content` is omitted. Each result maps to a `WebSearchSource`: `url` ← `url`, `title` ← `title`, `snippet` ← the first non-empty `highlights[]` entry (a result with no highlight has no portable snippet and is dropped), `publishedAt` ← `publishedDate`. The provider passes `maxResults` through as Exa's `numResults` for a cost/latency optimization; the final bound is enforced by the seam. Provider failures (HTTP errors, network failure, unparseable bodies) surface as `WebError` `WEB_PROVIDER_ERROR`; an aborted request surfaces as `WEB_ABORTED`. diff --git a/packages/web/web-search-exa/package.json b/packages/web/web-search-exa/package.json new file mode 100644 index 0000000000..fe79af8ba8 --- /dev/null +++ b/packages/web/web-search-exa/package.json @@ -0,0 +1,33 @@ +{ + "name": "@deepseek-ai/dsh-web-search-exa", + "description": "Exa-backed search provider for the DeepSeek Harness web capability seam (ctx.web)", + "version": "0.0.1", + "private": true, + "type": "module", + "main": "lib/index.js", + "types": "lib/index.d.ts", + "exports": { + ".": { + "types": "./lib/index.d.ts", + "default": "./lib/index.js" + }, + "./src/*": "./src/*", + "./package.json": "./package.json" + }, + "files": [ + "lib", + "src" + ], + "license": "BSD-3-Clause", + "peerDependencies": { + "@deepseek-ai/dsh-web": "^0.0.1", + "cordis": "^4.0.0-rc.6" + }, + "dependencies": { + "schemastery": "^3.18.0" + }, + "devDependencies": { + "@deepseek-ai/dsh-web": "workspace:^", + "cordis": "^4.0.0-rc.6" + } +} diff --git a/packages/web/web-search-exa/src/index.ts b/packages/web/web-search-exa/src/index.ts new file mode 100644 index 0000000000..f266474708 --- /dev/null +++ b/packages/web/web-search-exa/src/index.ts @@ -0,0 +1,48 @@ +/** + * `@deepseek-ai/dsh-web-search-exa`: registers an Exa-backed `WebSearchProvider` + * with `ctx.web`. A function/namespace plugin (NOT a default-export service): + * a search provider does not own the `ctx.web` key — it registers INTO the + * seam's provider registry, exactly as `@deepseek-ai/dsh-llm-deepseek` + * registers an adapter into `ctx.llm`. The key is owned by `@deepseek-ai/dsh-web`. + * + * @module @deepseek-ai/dsh-web-search-exa + */ + +import type { Context } from 'cordis' +import z from 'schemastery' +import type {} from '@deepseek-ai/dsh-web' +import { ExaSearchProvider, EXA_DEFAULT_BASE_URL } from './provider.ts' + +export { + EXA_DEFAULT_BASE_URL, + EXA_PROVIDER_ID, + ExaSearchProvider, + mapExaResponse, + mapExaResult, +} from './provider.ts' +export type { ExaSearchProviderOptions } from './provider.ts' + +/** Cordis plugin name used by loader diagnostics. */ +export const name = 'web-search-exa' + +/** The web seam this provider registers into. */ +export const inject = ['web'] + +export interface Config { + /** Exa API key. Falls back to `$EXA_API_KEY`. Empty → provider unavailable. */ + apiKey?: string + /** Endpoint base; `/search` is appended. Defaults to the public API. */ + baseURL?: string +} + +export const Config: z = z.object({ + apiKey: z.string(), + baseURL: z.string(), +}) + +/** Register the Exa search provider with `ctx.web`. */ +export function apply(ctx: Context, config: Config): void { + const apiKey = config.apiKey ?? process.env.EXA_API_KEY ?? '' + const baseURL = config.baseURL ?? EXA_DEFAULT_BASE_URL + ctx.web.registerSearchProvider(new ExaSearchProvider({ apiKey, baseURL })) +} diff --git a/packages/web/web-search-exa/src/provider.ts b/packages/web/web-search-exa/src/provider.ts new file mode 100644 index 0000000000..d7c464059a --- /dev/null +++ b/packages/web/web-search-exa/src/provider.ts @@ -0,0 +1,130 @@ +/** + * `ExaSearchProvider`: a `WebSearchProvider` backed by the Exa search API + * (`POST /search` with highlight contents). Maps Exa's flat `results[]` into the + * seam's normalized `WebSearchResult`. Exa returns no provider-generated answer, + * so `content` is omitted; each result maps to a `WebSearchSource` with `url`, + * `title`, the first highlight as `snippet`, and `publishedDate` as + * `publishedAt`. + * + * Network requests use platform-native `fetch` (Node 24), mirroring + * `@deepseek-ai/dsh-llm-deepseek`'s adapter — not a cordis HTTP-client service. + * + * @module @deepseek-ai/dsh-web-search-exa/provider + */ + +import { WebError } from '@deepseek-ai/dsh-web' +import type { + WebProviderStatus, + WebSearchProvider, + WebSearchRequest, + WebSearchResult, + WebSearchSource, +} from '@deepseek-ai/dsh-web' +import type { ExaError, ExaResult, ExaSearchResponse } from './types.ts' + +/** Stable id this provider registers under. */ +export const EXA_PROVIDER_ID = 'exa' + +/** Default Exa search endpoint; `/search` is the operation. */ +export const EXA_DEFAULT_BASE_URL = 'https://api.exa.ai' + +/** Attribution header sent on every request. Bump with the package version. */ +const USER_AGENT = 'deepseek-harness/0.0.1' + +export interface ExaSearchProviderOptions { + /** Exa API key. Empty/absent → `status()` reports `missing-credential`. */ + apiKey: string + /** Endpoint base; `/search` is appended. */ + baseURL: string +} + +/** + * Map one Exa result to a normalized source, or `undefined` when it carries no + * portable snippet (an entry with no highlight is dropped — the seam has no + * other field to derive a snippet from, and inventing one would lie). + */ +export function mapExaResult(result: ExaResult): WebSearchSource | undefined { + const snippet = result.highlights?.find(highlight => highlight.trim().length > 0) + if (snippet === undefined) return undefined + return { + url: result.url, + ...result.title != null && result.title.length > 0 ? { title: result.title } : {}, + snippet, + ...result.publishedDate != null && result.publishedDate.length > 0 ? { publishedAt: result.publishedDate } : {}, + } +} + +/** Map an Exa response envelope to a normalized search result. */ +export function mapExaResponse(query: string, response: ExaSearchResponse): WebSearchResult { + const sources = (response.results ?? []) + .map(mapExaResult) + .filter((source): source is WebSearchSource => source !== undefined) + // Exa returns no generated answer, so `content` is omitted. The seam owns the + // final `maxResults` truncation, so this provider reports `truncated: false`. + return { providerId: EXA_PROVIDER_ID, query, sources, truncated: false } +} + +/** The Exa-backed search provider. */ +export class ExaSearchProvider implements WebSearchProvider { + readonly id = EXA_PROVIDER_ID + + constructor(private readonly options: ExaSearchProviderOptions) {} + + status(): WebProviderStatus { + if (this.options.apiKey.length === 0) return { available: false, reason: 'missing-credential' } + return { available: true } + } + + async search(request: WebSearchRequest, exec?: { readonly signal?: AbortSignal }): Promise { + let response: Response + try { + response = await fetch(`${this.options.baseURL}/search`, { + method: 'POST', + headers: { + 'authorization': `Bearer ${this.options.apiKey}`, + 'content-type': 'application/json', + 'accept': 'application/json', + 'user-agent': USER_AGENT, + }, + body: JSON.stringify({ + query: request.query, + contents: { highlights: true }, + ...request.maxResults !== undefined ? { numResults: request.maxResults } : {}, + }), + ...exec?.signal ? { signal: exec.signal } : {}, + }) + } catch (error: unknown) { + if (isAbortError(error)) throw new WebError('Exa search aborted', 'WEB_ABORTED', { cause: error }) + throw new WebError(`Exa search request failed: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error }) + } + + if (!response.ok) { + const status = response.status + let message = `Exa API error (HTTP ${status})` + try { + const parsed = await response.json() as ExaError + const detail = parsed.error ?? parsed.message + if (detail !== undefined && detail.length > 0) message = detail + } catch { + // The HTTP status is already captured in `message` above; a malformed or + // non-JSON error body (normal for gateway 5xx/429s) can only cost a + // richer provider message, never the real error. `response.json()` is + // the sole statement and nothing else of consequence reaches here. + } + throw new WebError(message, 'WEB_PROVIDER_ERROR') + } + + let payload: ExaSearchResponse + try { + payload = await response.json() as ExaSearchResponse + } catch (error: unknown) { + throw new WebError(`Exa returned an unparseable response body: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error }) + } + return mapExaResponse(request.query, payload) + } +} + +/** True for a fetch/`AbortSignal` abort, surfaced as `WEB_ABORTED`. */ +function isAbortError(error: unknown): boolean { + return error instanceof DOMException && error.name === 'AbortError' +} diff --git a/packages/web/web-search-exa/src/types.ts b/packages/web/web-search-exa/src/types.ts new file mode 100644 index 0000000000..a0bda5f768 --- /dev/null +++ b/packages/web/web-search-exa/src/types.ts @@ -0,0 +1,36 @@ +/** + * Wire types for the Exa search API (`POST https://api.exa.ai/search`). Types + * only — no runtime code. Exa returns a flat `results[]`; each entry carries a + * URL, optional title, optional `publishedDate`, and (when highlights are + * requested) a `highlights[]` array of salient sentences. + * + * @module @deepseek-ai/dsh-web-search-exa/types + */ + +/** Request body sent to Exa's search endpoint. */ +export interface ExaSearchRequest { + query: string + /** Exa's result-count control; the seam still enforces the bound on return. */ + numResults?: number + /** Ask Exa to return highlight sentences per result. */ + contents: { highlights: true } +} + +/** One entry of Exa's flat `results[]`. */ +export interface ExaResult { + url: string + title?: string | null + publishedDate?: string | null + highlights?: string[] +} + +/** Exa's search response envelope. */ +export interface ExaSearchResponse { + results?: ExaResult[] +} + +/** Exa's error response envelope (best-effort; fields vary by failure). */ +export interface ExaError { + error?: string + message?: string +} diff --git a/packages/web/web-search-exa/tests/exa.e2e.ts b/packages/web/web-search-exa/tests/exa.e2e.ts new file mode 100644 index 0000000000..78f11940e5 --- /dev/null +++ b/packages/web/web-search-exa/tests/exa.e2e.ts @@ -0,0 +1,19 @@ +import { describe, expect, it } from 'vitest' +import { ExaSearchProvider, EXA_DEFAULT_BASE_URL } from '@deepseek-ai/dsh-web-search-exa' + +/** + * Real-API smoke for the Exa search provider. Self-skips without `$EXA_API_KEY` + * (CI has no secrets), per the with-key e2e policy in AGENTS.md § Secrets. + */ +const apiKey = process.env.EXA_API_KEY +const maybe = apiKey !== undefined && apiKey.length > 0 ? describe : describe.skip + +maybe('ExaSearchProvider real API', () => { + it('returns sources for a live query', async () => { + const provider = new ExaSearchProvider({ apiKey: apiKey!, baseURL: process.env.EXA_BASE_URL ?? EXA_DEFAULT_BASE_URL }) + const result = await provider.search({ query: 'DeepSeek coding agent', maxResults: 5 }) + expect(result.providerId).toBe('exa') + expect(result.sources.length).toBeGreaterThan(0) + for (const source of result.sources) expect(source.url).toMatch(/^https?:\/\//) + }, 30_000) +}) diff --git a/packages/web/web-search-exa/tests/exa.spec.ts b/packages/web/web-search-exa/tests/exa.spec.ts new file mode 100644 index 0000000000..3fdd878180 --- /dev/null +++ b/packages/web/web-search-exa/tests/exa.spec.ts @@ -0,0 +1,193 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { Context } from 'cordis' +import WebService from '@deepseek-ai/dsh-web' +import { ExaSearchProvider, mapExaResponse, mapExaResult, EXA_PROVIDER_ID } from '@deepseek-ai/dsh-web-search-exa' +import * as exaPlugin from '@deepseek-ai/dsh-web-search-exa' + +const options = { apiKey: 'exa-key', baseURL: 'https://api.exa.test' } + +function jsonResponse(body: unknown, init: ResponseInit = {}): Response { + return new Response(JSON.stringify(body), { status: 200, headers: { 'content-type': 'application/json' }, ...init }) +} + +afterEach(() => { + vi.unstubAllGlobals() +}) + +describe('Exa result mapping', () => { + it('maps a full result entry', () => { + expect(mapExaResult({ + url: 'https://a.test', + title: 'A', + publishedDate: '2026-01-01', + highlights: ['salient sentence', 'second'], + })).toEqual({ url: 'https://a.test', title: 'A', snippet: 'salient sentence', publishedAt: '2026-01-01' }) + }) + + it('drops a result with no usable highlight', () => { + expect(mapExaResult({ url: 'https://a.test', highlights: [] })).toBeUndefined() + expect(mapExaResult({ url: 'https://a.test' })).toBeUndefined() + expect(mapExaResult({ url: 'https://a.test', highlights: [' '] })).toBeUndefined() + }) + + it('omits null/empty optional fields rather than emitting them', () => { + expect(mapExaResult({ url: 'https://a.test', title: null, publishedDate: null, highlights: ['hi'] })) + .toEqual({ url: 'https://a.test', snippet: 'hi' }) + expect(mapExaResult({ url: 'https://a.test', title: '', publishedDate: '', highlights: ['hi'] })) + .toEqual({ url: 'https://a.test', snippet: 'hi' }) + }) + + it('maps a response to a result with no content and filtered sources', () => { + const result = mapExaResponse('q', { + results: [ + { url: 'https://a.test', highlights: ['one'] }, + { url: 'https://b.test' }, + { url: 'https://c.test', title: 'C', highlights: ['three'] }, + ], + }) + expect(result).toEqual({ + providerId: EXA_PROVIDER_ID, + query: 'q', + sources: [ + { url: 'https://a.test', snippet: 'one' }, + { url: 'https://c.test', title: 'C', snippet: 'three' }, + ], + truncated: false, + }) + expect(result.content).toBeUndefined() + }) + + it('tolerates a missing results array', () => { + expect(mapExaResponse('q', {}).sources).toEqual([]) + }) +}) + +describe('ExaSearchProvider status', () => { + it('is unavailable without a key', () => { + expect(new ExaSearchProvider({ apiKey: '', baseURL: options.baseURL }).status()) + .toEqual({ available: false, reason: 'missing-credential' }) + }) + + it('is available with a key', () => { + expect(new ExaSearchProvider(options).status()).toEqual({ available: true }) + }) +}) + +describe('ExaSearchProvider request mapping', () => { + it('sends query, highlights, numResults and bearer auth', async () => { + const fetchMock = vi.fn(async () => jsonResponse({ results: [{ url: 'https://a.test', highlights: ['hi'] }] })) + vi.stubGlobal('fetch', fetchMock) + + const provider = new ExaSearchProvider(options) + await provider.search({ query: 'hello', maxResults: 5 }) + + expect(fetchMock).toHaveBeenCalledOnce() + const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + expect(url).toBe('https://api.exa.test/search') + expect((init.headers as Record)['authorization']).toBe('Bearer exa-key') + expect(JSON.parse(init.body as string)).toEqual({ query: 'hello', contents: { highlights: true }, numResults: 5 }) + }) + + it('omits numResults when maxResults is absent', async () => { + const fetchMock = vi.fn(async () => jsonResponse({ results: [] })) + vi.stubGlobal('fetch', fetchMock) + await new ExaSearchProvider(options).search({ query: 'q' }) + const [, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + expect(JSON.parse(init.body as string)).not.toHaveProperty('numResults') + }) + + it('forwards the abort signal', async () => { + const fetchMock = vi.fn(async () => jsonResponse({ results: [] })) + vi.stubGlobal('fetch', fetchMock) + const controller = new AbortController() + await new ExaSearchProvider(options).search({ query: 'q' }, { signal: controller.signal }) + const [, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + expect(init.signal).toBe(controller.signal) + }) +}) + +describe('ExaSearchProvider error handling', () => { + it('maps an HTTP error to WEB_PROVIDER_ERROR with the provider message', async () => { + vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({ error: 'bad key' }, { status: 401 }))) + await expect(new ExaSearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR', message: 'bad key' })) + }) + + it('keeps a status-line message when the error body is not JSON', async () => { + vi.stubGlobal('fetch', vi.fn(async () => new Response('gateway down', { status: 502 }))) + await expect(new ExaSearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR', message: 'Exa API error (HTTP 502)' })) + }) + + it('keeps the status-line message when the JSON error body carries no detail', async () => { + vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({}, { status: 500 }))) + await expect(new ExaSearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ message: 'Exa API error (HTTP 500)' })) + }) + + it('maps a network failure to WEB_PROVIDER_ERROR', async () => { + vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new TypeError('connection refused')))) + await expect(new ExaSearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) + }) + + it('maps an abort to WEB_ABORTED', async () => { + vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new DOMException('aborted', 'AbortError')))) + await expect(new ExaSearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' })) + }) + + it('maps an unparseable success body to WEB_PROVIDER_ERROR', async () => { + vi.stubGlobal('fetch', vi.fn(async () => new Response('not json', { status: 200 }))) + await expect(new ExaSearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) + }) +}) + +describe('web-search-exa plugin registration', () => { + it('registers the provider into ctx.web (HMR-safe)', async () => { + const ctx = new Context() + await ctx.plugin(WebService, { searchProvider: EXA_PROVIDER_ID }) + const fiber = await ctx.plugin(exaPlugin, { apiKey: 'exa-key' }) + expect(ctx.web.searchStatus()).toEqual({ available: true, providerId: EXA_PROVIDER_ID }) + await fiber.dispose() + expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'configured-missing' }) + }) + + it('has no default export (namespace plugin export shape)', () => { + expect('default' in exaPlugin).toBe(false) + }) + + it('falls back to $EXA_API_KEY and the default base URL when config omits them', async () => { + const prev = process.env.EXA_API_KEY + process.env.EXA_API_KEY = 'env-key' + try { + const fetchMock = vi.fn(async () => jsonResponse({ results: [] })) + vi.stubGlobal('fetch', fetchMock) + const ctx = new Context() + await ctx.plugin(WebService, { searchProvider: EXA_PROVIDER_ID }) + const fiber = await ctx.plugin(exaPlugin, {}) + expect(ctx.web.searchStatus()).toEqual({ available: true, providerId: EXA_PROVIDER_ID }) + await ctx.web.search({ query: 'q' }) + const [url] = fetchMock.mock.calls[0] as unknown as [string] + expect(url).toBe('https://api.exa.ai/search') + await fiber.dispose() + } finally { + if (prev === undefined) delete process.env.EXA_API_KEY + else process.env.EXA_API_KEY = prev + } + }) + + it('is unavailable when neither config nor env supplies a key', async () => { + const prev = process.env.EXA_API_KEY + delete process.env.EXA_API_KEY + try { + const ctx = new Context() + await ctx.plugin(WebService, { searchProvider: EXA_PROVIDER_ID }) + await ctx.plugin(exaPlugin, {}) + expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'configured-unavailable' }) + } finally { + if (prev !== undefined) process.env.EXA_API_KEY = prev + } + }) +}) diff --git a/packages/web/web-search-exa/tsconfig.json b/packages/web/web-search-exa/tsconfig.json new file mode 100644 index 0000000000..cb9eb44552 --- /dev/null +++ b/packages/web/web-search-exa/tsconfig.json @@ -0,0 +1,24 @@ +{ + "extends": "../../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib" + }, + "include": [ + "src" + ], + "references": [ + { + "path": "../../../vendor/cosmokit" + }, + { + "path": "../../../vendor/cordis" + }, + { + "path": "../../../vendor/schemastery" + }, + { + "path": "../web" + } + ] +} diff --git a/packages/web/web-search-perplexity/README.md b/packages/web/web-search-perplexity/README.md new file mode 100644 index 0000000000..3850e5e9c1 --- /dev/null +++ b/packages/web/web-search-perplexity/README.md @@ -0,0 +1,24 @@ +# @deepseek-ai/dsh-web-search-perplexity + +A [Perplexity](https://perplexity.ai)-backed `WebSearchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It calls Perplexity's OpenAI-compatible `POST /chat/completions` endpoint and maps the generated answer plus citations into the seam's normalized `WebSearchResult`. + +This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the key and it does not register a model-facing tool. Like `@deepseek-ai/dsh-llm-deepseek`, it is a function/namespace plugin (`inject: ['web']`). The OpenAI-compatible wire shape is a provider-private detail — it does **not** make this provider depend on `ctx.llm`. + +## Config + +| Key | Default | Meaning | +|---|---|---| +| `apiKey` | `$PERPLEXITY_API_KEY` | Perplexity API key. Empty/absent → provider `status()` reports `missing-credential`. | +| `baseURL` | `https://api.perplexity.ai` | Endpoint base; `/chat/completions` is appended. | +| `model` | `sonar` | Search model name. | + +```yaml +- id: web-search-perplexity + name: '@deepseek-ai/dsh-web-search-perplexity' + config: + apiKey: !!js process.env.PERPLEXITY_API_KEY +``` + +## Mapping + +`content` ← `choices[0].message.content` (the generated answer). `sources[]` prefers the structured `search_results[]` (`url`, `title`, `snippet`, `publishedAt` ← `date`), falling back to the URL-only `citations[]` array only when `search_results` is absent — those sources carry just a `url`, which is why `title`/`snippet`/`publishedAt` are optional on the seam. Provider failures surface as `WebError` `WEB_PROVIDER_ERROR`; an aborted request surfaces as `WEB_ABORTED`. Perplexity has no result-count control, so `maxResults` is enforced by the seam (truncating `sources[]` and setting `truncated`). diff --git a/packages/web/web-search-perplexity/package.json b/packages/web/web-search-perplexity/package.json new file mode 100644 index 0000000000..1d6229eae4 --- /dev/null +++ b/packages/web/web-search-perplexity/package.json @@ -0,0 +1,33 @@ +{ + "name": "@deepseek-ai/dsh-web-search-perplexity", + "description": "Perplexity-backed search provider for the DeepSeek Harness web capability seam (ctx.web)", + "version": "0.0.1", + "private": true, + "type": "module", + "main": "lib/index.js", + "types": "lib/index.d.ts", + "exports": { + ".": { + "types": "./lib/index.d.ts", + "default": "./lib/index.js" + }, + "./src/*": "./src/*", + "./package.json": "./package.json" + }, + "files": [ + "lib", + "src" + ], + "license": "BSD-3-Clause", + "peerDependencies": { + "@deepseek-ai/dsh-web": "^0.0.1", + "cordis": "^4.0.0-rc.6" + }, + "dependencies": { + "schemastery": "^3.18.0" + }, + "devDependencies": { + "@deepseek-ai/dsh-web": "workspace:^", + "cordis": "^4.0.0-rc.6" + } +} diff --git a/packages/web/web-search-perplexity/src/index.ts b/packages/web/web-search-perplexity/src/index.ts new file mode 100644 index 0000000000..0fd46ffb71 --- /dev/null +++ b/packages/web/web-search-perplexity/src/index.ts @@ -0,0 +1,52 @@ +/** + * `@deepseek-ai/dsh-web-search-perplexity`: registers a Perplexity-backed + * `WebSearchProvider` with `ctx.web`. A function/namespace plugin (NOT a + * default-export service): it registers INTO the seam's provider registry, like + * `@deepseek-ai/dsh-llm-deepseek` registers an adapter into `ctx.llm`. + * + * @module @deepseek-ai/dsh-web-search-perplexity + */ + +import type { Context } from 'cordis' +import z from 'schemastery' +import type {} from '@deepseek-ai/dsh-web' +import { PerplexitySearchProvider, PERPLEXITY_DEFAULT_BASE_URL, PERPLEXITY_DEFAULT_MODEL } from './provider.ts' + +export { + PERPLEXITY_DEFAULT_BASE_URL, + PERPLEXITY_DEFAULT_MODEL, + PERPLEXITY_PROVIDER_ID, + PerplexitySearchProvider, + mapPerplexityResponse, + mapPerplexityResult, +} from './provider.ts' +export type { PerplexitySearchProviderOptions } from './provider.ts' + +/** Cordis plugin name used by loader diagnostics. */ +export const name = 'web-search-perplexity' + +/** The web seam this provider registers into. */ +export const inject = ['web'] + +export interface Config { + /** Perplexity API key. Falls back to `$PERPLEXITY_API_KEY`. Empty → unavailable. */ + apiKey?: string + /** Endpoint base; `/chat/completions` is appended. Defaults to the public API. */ + baseURL?: string + /** Search model name. Defaults to `sonar`. */ + model?: string +} + +export const Config: z = z.object({ + apiKey: z.string(), + baseURL: z.string(), + model: z.string(), +}) + +/** Register the Perplexity search provider with `ctx.web`. */ +export function apply(ctx: Context, config: Config): void { + const apiKey = config.apiKey ?? process.env.PERPLEXITY_API_KEY ?? '' + const baseURL = config.baseURL ?? PERPLEXITY_DEFAULT_BASE_URL + const model = config.model ?? PERPLEXITY_DEFAULT_MODEL + ctx.web.registerSearchProvider(new PerplexitySearchProvider({ apiKey, baseURL, model })) +} diff --git a/packages/web/web-search-perplexity/src/provider.ts b/packages/web/web-search-perplexity/src/provider.ts new file mode 100644 index 0000000000..2b596414dd --- /dev/null +++ b/packages/web/web-search-perplexity/src/provider.ts @@ -0,0 +1,138 @@ +/** + * `PerplexitySearchProvider`: a `WebSearchProvider` backed by the Perplexity + * search API (an OpenAI-compatible `POST /chat/completions`). Maps the generated + * answer (`choices[0].message.content`) into `content`, and prefers the + * structured `search_results[]` for `sources[]`, falling back to the URL-only + * `citations[]` when `search_results` is absent. + * + * Network requests use platform-native `fetch` (Node 24), mirroring + * `@deepseek-ai/dsh-llm-deepseek`'s adapter. The OpenAI-compatible request shape + * is a provider-private detail and does NOT make this provider depend on + * `ctx.llm`. + * + * @module @deepseek-ai/dsh-web-search-perplexity/provider + */ + +import { WebError } from '@deepseek-ai/dsh-web' +import type { + WebProviderStatus, + WebSearchProvider, + WebSearchRequest, + WebSearchResult, + WebSearchSource, +} from '@deepseek-ai/dsh-web' +import type { PerplexityError, PerplexityResponse, PerplexitySearchResult } from './types.ts' + +/** Stable id this provider registers under. */ +export const PERPLEXITY_PROVIDER_ID = 'perplexity' + +/** Default Perplexity endpoint; `/chat/completions` is the operation. */ +export const PERPLEXITY_DEFAULT_BASE_URL = 'https://api.perplexity.ai' + +/** Default search model. */ +export const PERPLEXITY_DEFAULT_MODEL = 'sonar' + +/** Attribution header sent on every request. Bump with the package version. */ +const USER_AGENT = 'deepseek-harness/0.0.1' + +export interface PerplexitySearchProviderOptions { + /** Perplexity API key. Empty/absent → `status()` reports `missing-credential`. */ + apiKey: string + /** Endpoint base; `/chat/completions` is appended. */ + baseURL: string + /** Search model name. */ + model: string +} + +/** Map one structured Perplexity search result to a normalized source. */ +export function mapPerplexityResult(result: PerplexitySearchResult): WebSearchSource { + return { + url: result.url, + ...result.title != null && result.title.length > 0 ? { title: result.title } : {}, + ...result.snippet != null && result.snippet.length > 0 ? { snippet: result.snippet } : {}, + ...result.date != null && result.date.length > 0 ? { publishedAt: result.date } : {}, + } +} + +/** + * Map a Perplexity response envelope to a normalized search result. Prefers + * structured `search_results[]`; falls back to URL-only `citations[]` (those + * sources carry just a `url`) only when `search_results` is absent. + */ +export function mapPerplexityResponse(query: string, response: PerplexityResponse): WebSearchResult { + const content = response.choices?.[0]?.message?.content + const sources: WebSearchSource[] = response.search_results !== undefined + ? response.search_results.map(mapPerplexityResult) + : (response.citations ?? []).map(url => ({ url })) + return { + providerId: PERPLEXITY_PROVIDER_ID, + query, + ...content != null && content.length > 0 ? { content } : {}, + sources, + truncated: false, + } +} + +/** The Perplexity-backed search provider. */ +export class PerplexitySearchProvider implements WebSearchProvider { + readonly id = PERPLEXITY_PROVIDER_ID + + constructor(private readonly options: PerplexitySearchProviderOptions) {} + + status(): WebProviderStatus { + if (this.options.apiKey.length === 0) return { available: false, reason: 'missing-credential' } + return { available: true } + } + + async search(request: WebSearchRequest, exec?: { readonly signal?: AbortSignal }): Promise { + let response: Response + try { + response = await fetch(`${this.options.baseURL}/chat/completions`, { + method: 'POST', + headers: { + 'authorization': `Bearer ${this.options.apiKey}`, + 'content-type': 'application/json', + 'accept': 'application/json', + 'user-agent': USER_AGENT, + }, + body: JSON.stringify({ + model: this.options.model, + messages: [{ role: 'user', content: request.query }], + }), + ...exec?.signal ? { signal: exec.signal } : {}, + }) + } catch (error: unknown) { + if (isAbortError(error)) throw new WebError('Perplexity search aborted', 'WEB_ABORTED', { cause: error }) + throw new WebError(`Perplexity search request failed: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error }) + } + + if (!response.ok) { + const status = response.status + let message = `Perplexity API error (HTTP ${status})` + try { + const parsed = await response.json() as PerplexityError + const detail = typeof parsed.error === 'string' ? parsed.error : parsed.error?.message ?? parsed.message + if (detail !== undefined && detail.length > 0) message = detail + } catch { + // The HTTP status is already captured in `message` above; a malformed or + // non-JSON error body (normal for gateway 5xx/429s) can only cost a + // richer provider message, never the real error. `response.json()` is + // the sole statement and nothing else of consequence reaches here. + } + throw new WebError(message, 'WEB_PROVIDER_ERROR') + } + + let payload: PerplexityResponse + try { + payload = await response.json() as PerplexityResponse + } catch (error: unknown) { + throw new WebError(`Perplexity returned an unparseable response body: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error }) + } + return mapPerplexityResponse(request.query, payload) + } +} + +/** True for a fetch/`AbortSignal` abort, surfaced as `WEB_ABORTED`. */ +function isAbortError(error: unknown): boolean { + return error instanceof DOMException && error.name === 'AbortError' +} diff --git a/packages/web/web-search-perplexity/src/types.ts b/packages/web/web-search-perplexity/src/types.ts new file mode 100644 index 0000000000..7b1f2e32b0 --- /dev/null +++ b/packages/web/web-search-perplexity/src/types.ts @@ -0,0 +1,41 @@ +/** + * Wire types for the Perplexity search API + * (`POST https://api.perplexity.ai/chat/completions`, an OpenAI-compatible chat + * shape). Types only — no runtime code. Perplexity returns a generated answer in + * `choices[0].message.content` plus citation surfaces: a structured + * `search_results[]` (preferred) and a URL-only `citations[]` fallback. + * + * The OpenAI-compatible wire shape is a provider-private detail; it does not make + * this provider depend on `ctx.llm`. + * + * @module @deepseek-ai/dsh-web-search-perplexity/types + */ + +/** Request body sent to Perplexity's chat-completions endpoint. */ +export interface PerplexityRequest { + model: string + messages: { role: 'user'; content: string }[] +} + +/** One structured search result (the preferred citation surface). */ +export interface PerplexitySearchResult { + url: string + title?: string | null + snippet?: string | null + date?: string | null +} + +/** Perplexity's response envelope. */ +export interface PerplexityResponse { + choices?: { message?: { content?: string | null } }[] + /** Structured citation surface (preferred). */ + search_results?: PerplexitySearchResult[] + /** URL-only citation fallback. */ + citations?: string[] +} + +/** Perplexity's error response envelope (best-effort; fields vary). */ +export interface PerplexityError { + error?: { message?: string } | string + message?: string +} diff --git a/packages/web/web-search-perplexity/tests/perplexity.e2e.ts b/packages/web/web-search-perplexity/tests/perplexity.e2e.ts new file mode 100644 index 0000000000..a546acab70 --- /dev/null +++ b/packages/web/web-search-perplexity/tests/perplexity.e2e.ts @@ -0,0 +1,23 @@ +import { describe, expect, it } from 'vitest' +import { PerplexitySearchProvider, PERPLEXITY_DEFAULT_BASE_URL, PERPLEXITY_DEFAULT_MODEL } from '@deepseek-ai/dsh-web-search-perplexity' + +/** + * Real-API smoke for the Perplexity search provider. Self-skips without + * `$PERPLEXITY_API_KEY`, per the with-key e2e policy in AGENTS.md § Secrets. + */ +const apiKey = process.env.PERPLEXITY_API_KEY +const maybe = apiKey !== undefined && apiKey.length > 0 ? describe : describe.skip + +maybe('PerplexitySearchProvider real API', () => { + it('returns a generated answer and sources for a live query', async () => { + const provider = new PerplexitySearchProvider({ + apiKey: apiKey!, + baseURL: process.env.PERPLEXITY_BASE_URL ?? PERPLEXITY_DEFAULT_BASE_URL, + model: process.env.PERPLEXITY_MODEL ?? PERPLEXITY_DEFAULT_MODEL, + }) + const result = await provider.search({ query: 'What is the DeepSeek coding agent?', maxResults: 5 }) + expect(result.providerId).toBe('perplexity') + expect(result.content ?? '').not.toBe('') + for (const source of result.sources) expect(source.url).toMatch(/^https?:\/\//) + }, 30_000) +}) diff --git a/packages/web/web-search-perplexity/tests/perplexity.spec.ts b/packages/web/web-search-perplexity/tests/perplexity.spec.ts new file mode 100644 index 0000000000..16557af888 --- /dev/null +++ b/packages/web/web-search-perplexity/tests/perplexity.spec.ts @@ -0,0 +1,192 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { Context } from 'cordis' +import WebService from '@deepseek-ai/dsh-web' +import { + PerplexitySearchProvider, + mapPerplexityResponse, + PERPLEXITY_PROVIDER_ID, +} from '@deepseek-ai/dsh-web-search-perplexity' +import * as perplexityPlugin from '@deepseek-ai/dsh-web-search-perplexity' + +const options = { apiKey: 'pplx-key', baseURL: 'https://api.perplexity.test', model: 'sonar' } + +function jsonResponse(body: unknown, init: ResponseInit = {}): Response { + return new Response(JSON.stringify(body), { status: 200, headers: { 'content-type': 'application/json' }, ...init }) +} + +afterEach(() => { + vi.unstubAllGlobals() +}) + +describe('Perplexity response mapping', () => { + it('maps the answer and prefers structured search_results', () => { + const result = mapPerplexityResponse('q', { + choices: [{ message: { content: 'the answer' } }], + search_results: [ + { url: 'https://a.test', title: 'A', snippet: 'snip', date: '2026-02-02' }, + { url: 'https://b.test' }, + ], + citations: ['https://ignored.test'], + }) + expect(result).toEqual({ + providerId: PERPLEXITY_PROVIDER_ID, + query: 'q', + content: 'the answer', + sources: [ + { url: 'https://a.test', title: 'A', snippet: 'snip', publishedAt: '2026-02-02' }, + { url: 'https://b.test' }, + ], + truncated: false, + }) + }) + + it('falls back to URL-only citations when search_results is absent', () => { + const result = mapPerplexityResponse('q', { + choices: [{ message: { content: 'answer' } }], + citations: ['https://a.test', 'https://b.test'], + }) + expect(result.sources).toEqual([{ url: 'https://a.test' }, { url: 'https://b.test' }]) + }) + + it('omits content when the answer is empty or missing', () => { + expect(mapPerplexityResponse('q', { citations: [] }).content).toBeUndefined() + expect(mapPerplexityResponse('q', { choices: [{ message: { content: '' } }] }).content).toBeUndefined() + expect(mapPerplexityResponse('q', { choices: [{ message: { content: null } }] }).content).toBeUndefined() + }) + + it('omits null/empty optional source fields', () => { + const result = mapPerplexityResponse('q', { + search_results: [{ url: 'https://a.test', title: null, snippet: '', date: null }], + }) + expect(result.sources).toEqual([{ url: 'https://a.test' }]) + }) + + it('yields no sources when neither search_results nor citations are present', () => { + expect(mapPerplexityResponse('q', { choices: [{ message: { content: 'a' } }] }).sources).toEqual([]) + }) +}) + +describe('PerplexitySearchProvider status', () => { + it('is unavailable without a key', () => { + expect(new PerplexitySearchProvider({ ...options, apiKey: '' }).status()) + .toEqual({ available: false, reason: 'missing-credential' }) + }) + + it('is available with a key', () => { + expect(new PerplexitySearchProvider(options).status()).toEqual({ available: true }) + }) +}) + +describe('PerplexitySearchProvider request mapping', () => { + it('sends a chat-completions request with the query as a user message', async () => { + const fetchMock = vi.fn(async () => jsonResponse({ choices: [{ message: { content: 'a' } }], citations: [] })) + vi.stubGlobal('fetch', fetchMock) + await new PerplexitySearchProvider(options).search({ query: 'hello' }) + const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + expect(url).toBe('https://api.perplexity.test/chat/completions') + expect((init.headers as Record)['authorization']).toBe('Bearer pplx-key') + expect(JSON.parse(init.body as string)).toEqual({ model: 'sonar', messages: [{ role: 'user', content: 'hello' }] }) + }) + + it('forwards the abort signal', async () => { + const fetchMock = vi.fn(async () => jsonResponse({ citations: [] })) + vi.stubGlobal('fetch', fetchMock) + const controller = new AbortController() + await new PerplexitySearchProvider(options).search({ query: 'q' }, { signal: controller.signal }) + const [, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + expect(init.signal).toBe(controller.signal) + }) +}) + +describe('PerplexitySearchProvider error handling', () => { + it('maps an HTTP error to WEB_PROVIDER_ERROR with the provider message', async () => { + vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({ error: { message: 'rate limited' } }, { status: 429 }))) + await expect(new PerplexitySearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR', message: 'rate limited' })) + }) + + it('handles a string-form error body', async () => { + vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({ error: 'bad request' }, { status: 400 }))) + await expect(new PerplexitySearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ message: 'bad request' })) + }) + + it('keeps a status-line message when the error body is not JSON', async () => { + vi.stubGlobal('fetch', vi.fn(async () => new Response('upstream error', { status: 503 }))) + await expect(new PerplexitySearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ message: 'Perplexity API error (HTTP 503)' })) + }) + + it('keeps the status-line message when the JSON error body carries no detail', async () => { + vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({}, { status: 500 }))) + await expect(new PerplexitySearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ message: 'Perplexity API error (HTTP 500)' })) + }) + + it('maps an abort to WEB_ABORTED', async () => { + vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new DOMException('aborted', 'AbortError')))) + await expect(new PerplexitySearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' })) + }) + + it('maps an unparseable success body to WEB_PROVIDER_ERROR', async () => { + vi.stubGlobal('fetch', vi.fn(async () => new Response('not json', { status: 200 }))) + await expect(new PerplexitySearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) + }) + + it('maps a network failure to WEB_PROVIDER_ERROR', async () => { + vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new TypeError('connection refused')))) + await expect(new PerplexitySearchProvider(options).search({ query: 'q' })) + .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) + }) +}) + +describe('web-search-perplexity plugin registration', () => { + it('registers the provider into ctx.web (HMR-safe)', async () => { + const ctx = new Context() + await ctx.plugin(WebService, { searchProvider: PERPLEXITY_PROVIDER_ID }) + const fiber = await ctx.plugin(perplexityPlugin, { apiKey: 'pplx-key' }) + expect(ctx.web.searchStatus()).toEqual({ available: true, providerId: PERPLEXITY_PROVIDER_ID }) + await fiber.dispose() + expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'configured-missing' }) + }) + + it('has no default export (namespace plugin export shape)', () => { + expect('default' in perplexityPlugin).toBe(false) + }) + + it('falls back to env key and defaults for base URL and model when config omits them', async () => { + const prev = process.env.PERPLEXITY_API_KEY + process.env.PERPLEXITY_API_KEY = 'env-key' + try { + const fetchMock = vi.fn(async () => jsonResponse({ choices: [{ message: { content: 'a' } }], citations: [] })) + vi.stubGlobal('fetch', fetchMock) + const ctx = new Context() + await ctx.plugin(WebService, { searchProvider: PERPLEXITY_PROVIDER_ID }) + const fiber = await ctx.plugin(perplexityPlugin, {}) + expect(ctx.web.searchStatus()).toEqual({ available: true, providerId: PERPLEXITY_PROVIDER_ID }) + await ctx.web.search({ query: 'q' }) + const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + expect(url).toBe('https://api.perplexity.ai/chat/completions') + expect(JSON.parse(init.body as string)).toMatchObject({ model: 'sonar' }) + await fiber.dispose() + } finally { + if (prev === undefined) delete process.env.PERPLEXITY_API_KEY + else process.env.PERPLEXITY_API_KEY = prev + } + }) + + it('is unavailable when neither config nor env supplies a key', async () => { + const prev = process.env.PERPLEXITY_API_KEY + delete process.env.PERPLEXITY_API_KEY + try { + const ctx = new Context() + await ctx.plugin(WebService, { searchProvider: PERPLEXITY_PROVIDER_ID }) + await ctx.plugin(perplexityPlugin, {}) + expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'configured-unavailable' }) + } finally { + if (prev !== undefined) process.env.PERPLEXITY_API_KEY = prev + } + }) +}) diff --git a/packages/web/web-search-perplexity/tsconfig.json b/packages/web/web-search-perplexity/tsconfig.json new file mode 100644 index 0000000000..cb9eb44552 --- /dev/null +++ b/packages/web/web-search-perplexity/tsconfig.json @@ -0,0 +1,24 @@ +{ + "extends": "../../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib" + }, + "include": [ + "src" + ], + "references": [ + { + "path": "../../../vendor/cosmokit" + }, + { + "path": "../../../vendor/cordis" + }, + { + "path": "../../../vendor/schemastery" + }, + { + "path": "../web" + } + ] +} diff --git a/packages/web/web/README.md b/packages/web/web/README.md new file mode 100644 index 0000000000..9b9e2ca333 --- /dev/null +++ b/packages/web/web/README.md @@ -0,0 +1,45 @@ +# @deepseek-ai/dsh-web + +The **web access seam**: an abstract `WebService` (`ctx.web`) defining WHAT web access the harness has — search the web, fetch a URL — over multiple providers, without binding the model contract to one vendor's API shape. + +This package is the interface third of the web capability. Unlike bash/fs it spans two capabilities (search and fetch) on one seam, with potentially multiple providers each: + +| Package | Role | +|---|---| +| `@deepseek-ai/dsh-web` (this) | the interface: the service, provider registries, selection policy, request/result vocabulary, the `WebError` taxonomy | +| `@deepseek-ai/dsh-web-search-exa` | a search implementation: Exa | +| `@deepseek-ai/dsh-web-search-perplexity` | a search implementation: Perplexity | +| `@deepseek-ai/dsh-web-fetch-local` | a fetch implementation: anonymous public HTTP(S) | +| `@deepseek-ai/dsh-tool-web` | the model-facing `web_search` / `web_fetch` tool schemas over `ctx.web` | + +Search and fetch share no request schema and no business logic, but they are deliberately one seam: `ctx.web` is a single web-access middle layer with one provider-selection policy owner, one abort/error vocabulary, and one product-facing "how this harness reaches the web" config surface. The cost is the parallel `Search`/`Fetch` method pairs; that parallelism is intentional, not a missed extraction. + +## Service API (`ctx.web`) + +| Member | Semantics | +|---|---| +| `registerSearchProvider(provider)` / `registerFetchProvider(provider)` | Register a backend. Throws `WebError` `WEB_DUPLICATE_PROVIDER` on a duplicate id within that capability kind. Returns a disposer; emits `web/providers-change` on register and on dispose. Disposed with the calling fiber. | +| `searchStatus()` / `fetchStatus()` | Derived (never stored) `WebCapabilityStatus`: whether the capability has a selected usable provider, or the broad category it fails in. Diagnostics + execution-resolution input. | +| `search(request, exec?)` | Resolve the search provider and run one search. Enforces `request.maxResults` on the result (truncates `sources[]`, sets `truncated`). Throws `WebError` when the capability cannot run. | +| `fetch(request, exec?)` | Resolve the fetch provider and retrieve one URL. A non-2xx response is a result, not a throw. Throws `WebError` for failures to safely retrieve or represent the resource. | + +Providers register **capabilities**, not tools. `dsh-tool-web` is the only owner of model-facing names, descriptions, prompt guidance, JSON schemas, and presentation. + +## Selection + +Selection never depends on registration, config, or HMR order. A capability has an explicit provider id (config `searchProvider`/`fetchProvider`, or env `$DSH_WEB_SEARCH_PROVIDER`/`$DSH_WEB_FETCH_PROVIDER` feeding the same fields), or auto-selects when exactly one usable provider is registered: + +| Situation | `WebCapabilityStatus` | Execution | +|---|---|---| +| configured id registered and `status().available` | `available` for it | runs | +| configured id not registered | `configured-missing` | `WEB_PROVIDER_CONFIGURED_MISSING` | +| configured id registered but unavailable | `configured-unavailable` | `WEB_PROVIDER_CONFIGURED_UNAVAILABLE` | +| no id, exactly one registered usable provider | `available` for it | runs | +| no id, no usable provider | `none` | `WEB_PROVIDER_UNAVAILABLE` | +| no id, multiple usable providers | `ambiguous` | `WEB_PROVIDER_AMBIGUOUS` | + +`WebCapabilityStatus` carries only `available` + a `reason` discriminant (plus the winning `providerId` on the available branch). The branchable per-reason detail lives in the thrown `WebError`, which is the surface callers route on — so the same fact never gets two homes that can disagree. A provider's own `status()` is a cheap local check (credential presence, parseable config) and **must not make network calls**; `dsh-tool-web` reads only the aggregated `searchStatus()`/`fetchStatus()`, never each provider's `status()` directly. + +## Vocabulary + +`WebSearchRequest` (`query`, `maxResults?`) → `WebSearchResult` (`providerId`, `query`, `content?`, `sources[]`, `truncated`); each `WebSearchSource` has a required `url` and optional `title`/`snippet`/`publishedAt` (Perplexity citations may be URL-only). `WebFetchRequest` (`url`, `timeoutMs?`) → `WebFetchResult` (`providerId`, final `url`, `statusCode`, `body`, `truncated`); `WebFetchBody` is a CLOSED discriminated union (`html` | `text`) owned here — consumers `switch` to exhaustiveness so a new kind breaks their compilation until handled. See `src/types.ts` for the full contracts and the `WebError` code taxonomy. diff --git a/packages/web/web/package.json b/packages/web/web/package.json new file mode 100644 index 0000000000..40ac10b715 --- /dev/null +++ b/packages/web/web/package.json @@ -0,0 +1,33 @@ +{ + "name": "@deepseek-ai/dsh-web", + "description": "Abstract web access capability seam (ctx.web) for the DeepSeek Harness — search/fetch provider registry, registration-order-independent selection, request/result vocabulary, and the WebError taxonomy", + "version": "0.0.1", + "private": true, + "type": "module", + "main": "lib/index.js", + "types": "lib/index.d.ts", + "exports": { + ".": { + "types": "./lib/index.d.ts", + "default": "./lib/index.js" + }, + "./src/*": "./src/*", + "./package.json": "./package.json" + }, + "files": [ + "lib", + "src" + ], + "license": "BSD-3-Clause", + "peerDependencies": { + "@deepseek-ai/dsh-llm": "^0.0.1", + "cordis": "^4.0.0-rc.6" + }, + "dependencies": { + "schemastery": "^3.18.0" + }, + "devDependencies": { + "@deepseek-ai/dsh-llm": "workspace:^", + "cordis": "^4.0.0-rc.6" + } +} diff --git a/packages/web/web/src/index.ts b/packages/web/web/src/index.ts new file mode 100644 index 0000000000..172c1a0ecb --- /dev/null +++ b/packages/web/web/src/index.ts @@ -0,0 +1,270 @@ +/** + * The web access seam (`ctx.web`): a provider registry plus a provider-selecting + * execution surface for two capabilities — search and fetch. Provider packages + * register concrete backends with `registerSearchProvider` / + * `registerFetchProvider`; the model-facing consumer + * (`@deepseek-ai/dsh-tool-web`) reads capability status and executes through + * `search()` / `fetch()`. + * + * The registry half stays close to `LlmService`: a `Map` per + * capability kind, register methods that return disposers, duplicate ids that + * throw, and execution-time resolution that throws when the selected provider is + * absent or unusable. On top of that sits one small selection-status layer so + * diagnostics and execution can explain why a capability can or cannot run, + * independent of registration order. + * + * @module @deepseek-ai/dsh-web + */ + +import { Context, Service } from 'cordis' +import z from 'schemastery' +import type { + WebCapabilityStatus, + WebExecContext, + WebFetchProvider, + WebFetchRequest, + WebFetchResult, + WebProviderStatus, + WebSearchProvider, + WebSearchRequest, + WebSearchResult, +} from './types.ts' +import { WebError } from './types.ts' + +export { + WebError, +} from './types.ts' +export type { + WebCapabilityStatus, + WebErrorCode, + WebExecContext, + WebFetchBody, + WebFetchProvider, + WebFetchRequest, + WebFetchResult, + WebProviderStatus, + WebSearchProvider, + WebSearchRequest, + WebSearchResult, + WebSearchSource, +} from './types.ts' + +declare module 'cordis' { + interface Context { + web: WebService + } + + interface Events { + /** + * Fired after the provider registry changes — a search or fetch provider was + * registered or disposed. Carries no payload and no capability graph: it + * means only "the provider registry changed; observers may recompute status + * from `ctx.web`". `searchStatus()` / `fetchStatus()` stay derived, not + * stored. + * @mode emit + */ + 'web/providers-change'(this: WebService): void + } +} + +/** Selection inputs shared by the status query and execution resolution. */ +interface Selection

{ + /** The configured provider id for this capability, if any. */ + readonly configuredId?: string + /** Providers registered for this capability kind. */ + readonly providers: ReadonlyMap +} + +/** + * Config for the web seam. `searchProvider` / `fetchProvider` pin which provider + * wins for each capability; both are optional (a single registered usable + * provider auto-selects). Operational overrides such as environment variables + * must feed these same fields rather than introduce a hidden priority chain. + */ +export interface WebServiceConfig { + /** Explicit search provider id. Omitted = auto-select when exactly one usable. */ + readonly searchProvider?: string + /** Explicit fetch provider id. Omitted = auto-select when exactly one usable. */ + readonly fetchProvider?: string +} + +/** + * The web access service. Registered as `ctx.web` (one instance per context). + * + * Selection semantics (identical for status and execution, never order- + * dependent): + * - A configured id that is registered and `status().available` → that provider. + * - A configured id not registered → `configured-missing` / + * `WEB_PROVIDER_CONFIGURED_MISSING`. + * - A configured id registered but unavailable → `configured-unavailable` / + * `WEB_PROVIDER_CONFIGURED_UNAVAILABLE`. + * - No id configured, exactly one registered usable provider → that provider. + * - No id configured, multiple usable providers → `ambiguous` / + * `WEB_PROVIDER_AMBIGUOUS`. + * - No id configured, no usable provider → `none` / `WEB_PROVIDER_UNAVAILABLE`. + */ +export class WebService extends Service { + /** + * Provider selection config. Operational env overrides feed the SAME fields: + * `$DSH_WEB_SEARCH_PROVIDER` / `$DSH_WEB_FETCH_PROVIDER` are equivalent to + * `searchProvider` / `fetchProvider` and are NOT a hidden priority chain. + */ + static Config: z = z.object({ + searchProvider: z.string(), + fetchProvider: z.string(), + }) + + private searchProviders = new Map() + private fetchProviders = new Map() + private readonly searchProviderId: string | undefined + private readonly fetchProviderId: string | undefined + + constructor(ctx: Context, config: WebServiceConfig = {}) { + super(ctx, 'web') + this.searchProviderId = config.searchProvider ?? process.env.DSH_WEB_SEARCH_PROVIDER + this.fetchProviderId = config.fetchProvider ?? process.env.DSH_WEB_FETCH_PROVIDER + } + + /** + * Register a search provider. Throws {@link WebError} `WEB_DUPLICATE_PROVIDER` + * if its id is already registered for search. Returns a disposer; emits + * `web/providers-change` after a successful register and again on dispose. + * Disposed with the calling fiber. + */ + registerSearchProvider(provider: WebSearchProvider): () => void { + return this.registerProvider(this.searchProviders, provider) + } + + /** + * Register a fetch provider. Throws {@link WebError} `WEB_DUPLICATE_PROVIDER` + * if its id is already registered for fetch. Returns a disposer; emits + * `web/providers-change` after a successful register and again on dispose. + * Disposed with the calling fiber. + */ + registerFetchProvider(provider: WebFetchProvider): () => void { + return this.registerProvider(this.fetchProviders, provider) + } + + private registerProvider

(store: Map, provider: P): () => void { + if (store.has(provider.id)) { + throw new WebError(`a web provider with id "${provider.id}" is already registered`, 'WEB_DUPLICATE_PROVIDER') + } + const dispose = this.ctx.effect(function* (this: WebService) { + store.set(provider.id, provider) + // Yield the rollback BEFORE emitting `web/providers-change`: the generator + // effect collects each yielded disposer before the next step runs, so a + // throwing change listener removes the just-added provider instead of + // leaking it into the registry. + yield () => { + store.delete(provider.id) + this.ctx.emit('web/providers-change') + } + this.ctx.emit('web/providers-change') + }.bind(this), 'web.registerProvider()') + // ctx.effect's disposer returns Promise; our disposer API is + // synchronous fire-and-forget — discard the (always-resolved) promise. + return () => void dispose() + } + + /** Search-capability selection status, derived live (never stored). */ + searchStatus(): WebCapabilityStatus { + return resolveStatus({ + providers: this.searchProviders, + ...this.searchProviderId !== undefined ? { configuredId: this.searchProviderId } : {}, + }) + } + + /** Fetch-capability selection status, derived live (never stored). */ + fetchStatus(): WebCapabilityStatus { + return resolveStatus({ + providers: this.fetchProviders, + ...this.fetchProviderId !== undefined ? { configuredId: this.fetchProviderId } : {}, + }) + } + + /** + * Run one search through the selected provider. Resolves the provider at call + * time with the selection rules above; throws {@link WebError} when the + * capability cannot run. The seam enforces `request.maxResults` on the result: + * if the provider over-returns, `sources[]` is truncated and `truncated` set. + */ + async search(request: WebSearchRequest, exec?: WebExecContext): Promise { + const provider = resolveProvider({ + providers: this.searchProviders, + ...this.searchProviderId !== undefined ? { configuredId: this.searchProviderId } : {}, + }) + const result = await provider.search(request, exec) + return capSources(result, request.maxResults) + } + + /** + * Retrieve one URL through the selected provider. Resolves the provider at + * call time with the selection rules above; throws {@link WebError} when the + * capability cannot run. A non-2xx response is a result, not a throw. + */ + async fetch(request: WebFetchRequest, exec?: WebExecContext): Promise { + const provider = resolveProvider({ + providers: this.fetchProviders, + ...this.fetchProviderId !== undefined ? { configuredId: this.fetchProviderId } : {}, + }) + return provider.fetch(request, exec) + } +} + +interface ResolvableProvider { + readonly id: string + status(): WebProviderStatus +} + +/** Compute the capability status from configured id + registered providers. */ +function resolveStatus

(selection: Selection

): WebCapabilityStatus { + const { configuredId, providers } = selection + if (configuredId !== undefined) { + const provider = providers.get(configuredId) + if (!provider) return { available: false, reason: 'configured-missing' } + if (!provider.status().available) return { available: false, reason: 'configured-unavailable' } + return { available: true, providerId: configuredId } + } + const usable = [...providers.values()].filter(provider => provider.status().available) + const [single] = usable + if (single === undefined) return { available: false, reason: 'none' } + if (usable.length > 1) return { available: false, reason: 'ambiguous' } + return { available: true, providerId: single.id } +} + +/** + * Resolve the selected provider or throw the matching {@link WebError}. Shares + * the selection rules with {@link resolveStatus} so status and execution can + * never disagree. + */ +function resolveProvider

(selection: Selection

): P { + const { configuredId, providers } = selection + if (configuredId !== undefined) { + const provider = providers.get(configuredId) + if (!provider) { + throw new WebError(`configured web provider "${configuredId}" is not registered`, 'WEB_PROVIDER_CONFIGURED_MISSING') + } + if (!provider.status().available) { + throw new WebError(`configured web provider "${configuredId}" is registered but unavailable`, 'WEB_PROVIDER_CONFIGURED_UNAVAILABLE') + } + return provider + } + const usable = [...providers.values()].filter(provider => provider.status().available) + const [single] = usable + if (single === undefined) { + throw new WebError('no usable web provider is registered', 'WEB_PROVIDER_UNAVAILABLE') + } + if (usable.length > 1) { + const ids = usable.map(provider => provider.id).join(', ') + throw new WebError(`multiple usable web providers are registered (${ids}); configure one explicitly`, 'WEB_PROVIDER_AMBIGUOUS') + } + return single +} + +/** Enforce `maxResults` on a search result: truncate `sources[]` and flag it. */ +function capSources(result: WebSearchResult, maxResults: number | undefined): WebSearchResult { + if (maxResults === undefined || result.sources.length <= maxResults) return result + return { ...result, sources: result.sources.slice(0, maxResults), truncated: true } +} + +export default WebService diff --git a/packages/web/web/src/types.ts b/packages/web/web/src/types.ts new file mode 100644 index 0000000000..ec97101ae2 --- /dev/null +++ b/packages/web/web/src/types.ts @@ -0,0 +1,225 @@ +/** + * Vocabulary for the web capability seam (`ctx.web`): the search/fetch + * request/result shapes providers produce and consumers format, the provider + * and capability status discriminants selection reports, the execution-control + * context, and the typed error taxonomy. + * + * These types are shared by every provider backend + * (`@deepseek-ai/dsh-web-search-exa`, `@deepseek-ai/dsh-web-search-perplexity`, + * `@deepseek-ai/dsh-web-fetch-local`, and future backends) and by the + * model-facing consumer (`@deepseek-ai/dsh-tool-web`). Search and fetch share no + * request schema and no business logic, but they are deliberately one seam: + * `ctx.web` is a single web-access middle layer with one provider-selection + * policy, one abort/error vocabulary, and one product-facing configuration + * point. The cost is the parallel `Search`/`Fetch` shapes below; that + * parallelism is intentional. + * + * @module @deepseek-ai/dsh-web/types + */ + +import { HarnessError } from '@deepseek-ai/dsh-llm' + +/** + * Execution control threaded from the tool layer through the seam into a + * provider's network requests, stream readers, and expensive decoding. It is + * NOT business input: the first version carries only `signal` so `tool-web` can + * propagate turn cancellation, tool timeout, and agent disposal. It deliberately + * does NOT carry `ToolExecution`, which would make `dsh-web` depend on + * `dsh-tools`. + */ +export interface WebExecContext { + /** Abort signal a provider must honor for its network/decoding work. */ + readonly signal?: AbortSignal +} + +/** + * What one search-capable backend can return. The model-facing argument is just + * a query; `maxResults` is a `dsh-tool-web`-layer bound passed through unchanged + * and enforced on the way back by the seam (see {@link WebSearchResult}). + */ +export interface WebSearchRequest { + readonly query: string + /** + * Upper bound on returned sources; the seam truncates to it. Omitted = no + * bound. `dsh-tool-web` always sets it. A provider whose API supports a + * result-count control (Exa's `numResults`) should apply it at the request + * layer as a cost/latency optimization; the seam enforces the bound + * regardless. + */ + readonly maxResults?: number +} + +/** + * Normalized search outcome. `content` is optional provider-generated answer + * text or summary (Exa returns none; Perplexity returns a generated answer). + * `sources[]` is the portable citation surface. `truncated` is set by the seam + * when it cut `sources[]` down to `maxResults`. + */ +export interface WebSearchResult { + /** Id of the provider that produced this result. */ + readonly providerId: string + /** Echo of the query the provider answered. */ + readonly query: string + /** Optional provider-generated answer text, search context, or summary. */ + readonly content?: string + /** Citeable sources, already truncated to the request's `maxResults`. */ + readonly sources: readonly WebSearchSource[] + /** True when the seam dropped sources to honor `maxResults`. */ + readonly truncated: boolean +} + +/** + * One citeable source. A source always has a URL; `title`, `snippet`, and + * `publishedAt` are optional because not every provider returns them — forcing + * adapters to invent them would make the seam lie (Perplexity citations may be + * URL-only). `dsh-tool-web` renders `title ?? hostname(url)` for display. + */ +export interface WebSearchSource { + readonly url: string + readonly title?: string + readonly snippet?: string + /** Publication/crawl timestamp as a provider-supplied ISO-8601 string. */ + readonly publishedAt?: string +} + +/** + * What one fetch-capable backend is asked to retrieve. `timeoutMs` is an + * optional positive hint the provider caps. The request deliberately omits + * `format`, `prompt`, and extraction controls — those are presentation or + * higher-level LLM concerns, not safe-retrieval inputs. + */ +export interface WebFetchRequest { + readonly url: string + readonly timeoutMs?: number +} + +/** + * Normalized fetch outcome. A successful network fetch of a non-2xx response is + * a result, not an error: the status code is part of the fetched resource + * state. {@link WebError} is reserved for failures to safely retrieve or + * represent the resource. + */ +export interface WebFetchResult { + /** Id of the provider that produced this result. */ + readonly providerId: string + /** The final URL after allowed redirects (the request URL is in the request). */ + readonly url: string + /** HTTP status code of the fetched response. */ + readonly statusCode: number + /** Decoded body, classified by content kind. */ + readonly body: WebFetchBody + /** True when the provider capped the decoded body. */ + readonly truncated: boolean +} + +/** + * The decoded body of a fetched resource. A CLOSED discriminated union owned by + * `dsh-web`: the provider decodes the kind and `dsh-tool-web` renders it, so a + * new kind is a coordinated change across known packages, not a plugin + * extension. Consumers `switch` on `kind` ending in `default: assertNever(...)` + * so adding a kind breaks compilation at every consumer until handled. Each arm + * stays its own object literal even where fields coincide today, leaving room + * for arm-specific fields later (a `pdf` body's `pageCount`). + */ +export type WebFetchBody = + | { readonly kind: 'html'; readonly content: string } + | { readonly kind: 'text'; readonly content: string } + +/** + * Whether one concrete provider implementation is usable, by cheap local checks + * only (credential presence, parseable endpoint config). A provider `status()` + * must NOT make network calls. It is an input to selection, not a health system. + */ +export type WebProviderStatus = + | { readonly available: true } + | { readonly available: false; readonly reason: 'missing-credential' | 'misconfigured' } + +/** + * Whether a capability (search or fetch) has a selected usable provider, or the + * broad category in which selection fails. Intentionally small: it carries the + * winning `providerId` on the available branch (so diagnostics can report which + * provider won) but NOT the per-reason payload (the missing id, the ambiguous + * candidate set). That branchable detail lives in the {@link WebError} thrown at + * execution time — the surface callers route on — so the same fact does not get + * two homes that can disagree. + */ +export type WebCapabilityStatus = + | { readonly available: true; readonly providerId: string } + | { readonly available: false; readonly reason: 'none' | 'configured-missing' | 'configured-unavailable' | 'ambiguous' } + +/** + * A search-capable backend. Registered with `ctx.web.registerSearchProvider`. + * `id` is a stable string, unique within the search capability kind. + */ +export interface WebSearchProvider { + readonly id: string + /** Cheap local usability check; must not make network calls. */ + status(): WebProviderStatus + /** Run one search; honor `exec.signal` for cancellation. */ + search(request: WebSearchRequest, exec?: WebExecContext): Promise +} + +/** + * A fetch-capable backend. Registered with `ctx.web.registerFetchProvider`. + * `id` is a stable string, unique within the fetch capability kind. + */ +export interface WebFetchProvider { + readonly id: string + /** Cheap local usability check; must not make network calls. */ + status(): WebProviderStatus + /** Retrieve one URL; honor `exec.signal` for cancellation. */ + fetch(request: WebFetchRequest, exec?: WebExecContext): Promise +} + +/** + * Stable codes for {@link WebError}. Callers (hooks, tests, UI) route on these. + * + * - `WEB_PROVIDER_UNAVAILABLE`: no provider configured and none usable. + * - `WEB_PROVIDER_CONFIGURED_MISSING`: a configured id is not registered. + * - `WEB_PROVIDER_CONFIGURED_UNAVAILABLE`: a configured id is registered but its + * `status()` reports unavailable. + * - `WEB_PROVIDER_AMBIGUOUS`: no id configured and multiple usable providers + * exist (selection refuses to pick by registration order). + * - `WEB_DUPLICATE_PROVIDER`: a registration-time programming error — an id is + * already registered for that capability kind. + * - `WEB_INVALID_URL`: the fetch URL is malformed or not http(s). + * - `WEB_BLOCKED_URL`: the fetch URL is rejected by policy (credentials in URL). + * - `WEB_REDIRECT_BLOCKED`: a cross-origin redirect was refused. + * - `WEB_FETCH_TOO_LARGE`: the response exceeded the byte/character cap. + * - `WEB_FETCH_TIMEOUT`: the fetch exceeded its timeout. + * - `WEB_ABORTED`: the operation was aborted via `WebExecContext.signal`. + * - `WEB_UNSUPPORTED_CONTENT_TYPE`: the response content type cannot be decoded. + * - `WEB_PROVIDER_ERROR`: catch-all for a provider's own failure surfaced through + * the seam, including network/transport failure (DNS, connection refused, TLS). + */ +export type WebErrorCode = + | 'WEB_PROVIDER_UNAVAILABLE' + | 'WEB_PROVIDER_CONFIGURED_MISSING' + | 'WEB_PROVIDER_CONFIGURED_UNAVAILABLE' + | 'WEB_PROVIDER_AMBIGUOUS' + | 'WEB_DUPLICATE_PROVIDER' + | 'WEB_INVALID_URL' + | 'WEB_BLOCKED_URL' + | 'WEB_REDIRECT_BLOCKED' + | 'WEB_FETCH_TOO_LARGE' + | 'WEB_FETCH_TIMEOUT' + | 'WEB_ABORTED' + | 'WEB_UNSUPPORTED_CONTENT_TYPE' + | 'WEB_PROVIDER_ERROR' + +/** + * Typed web error. Extends {@link HarnessError} so it carries a stable + * {@link WebErrorCode} and chains `cause`. `dsh-web` owns this vocabulary so + * providers, the seam, and the tool layer raise the same codes instead of each + * inventing message strings. `ToolRegistry.execute()` converts a thrown + * `WebError` into an error tool result whose structured metadata exposes the + * code. + */ +export class WebError extends HarnessError { + override readonly code: WebErrorCode + + constructor(message: string, code: WebErrorCode, options?: ErrorOptions) { + super(message, code, options) + this.code = code + } +} diff --git a/packages/web/web/tests/web.spec.ts b/packages/web/web/tests/web.spec.ts new file mode 100644 index 0000000000..e97630ebab --- /dev/null +++ b/packages/web/web/tests/web.spec.ts @@ -0,0 +1,263 @@ +import { describe, expect, it, vi } from 'vitest' +import { Context } from 'cordis' +import WebService, { + WebError, + type WebFetchProvider, + type WebFetchResult, + type WebProviderStatus, + type WebSearchProvider, + type WebSearchRequest, + type WebSearchResult, +} from '@deepseek-ai/dsh-web' + +/** A scripted search provider for contract tests. */ +function makeSearchProvider( + id: string, + status: WebProviderStatus, + search: (request: WebSearchRequest) => Promise, +): WebSearchProvider { + return { id, status: () => status, search: request => search(request) } +} + +function makeFetchProvider(id: string, status: WebProviderStatus, result: WebFetchResult): WebFetchProvider { + return { id, status: () => status, fetch: () => Promise.resolve(result) } +} + +const available: WebProviderStatus = { available: true } +const unavailable: WebProviderStatus = { available: false, reason: 'missing-credential' } + +function searchResult(providerId: string, overrides: Partial = {}): WebSearchResult { + return { providerId, query: 'q', sources: [], truncated: false, ...overrides } +} + +function fetchResult(providerId: string): WebFetchResult { + return { providerId, url: 'https://example.com', statusCode: 200, body: { kind: 'text', content: 'hi' }, truncated: false } +} + +/** Mount a WebService on a fresh root context with the given config. */ +async function mountWeb(config: ConstructorParameters[1] = {}): Promise<{ ctx: Context; web: WebService }> { + const ctx = new Context() + await ctx.plugin(WebService, config) + return { ctx, web: ctx.web } +} + +describe('WebService registration', () => { + it('registers and disposes a search provider, emitting providers-change each way', async () => { + const { ctx, web } = await mountWeb() + const changed = vi.fn() + ctx.on('web/providers-change', changed) + + const dispose = web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + expect(changed).toHaveBeenCalledTimes(1) + expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' }) + + dispose() + expect(changed).toHaveBeenCalledTimes(2) + expect(web.searchStatus()).toEqual({ available: false, reason: 'none' }) + }) + + it('throws WEB_DUPLICATE_PROVIDER on a duplicate search id', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + expect(() => web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))) + .toThrow(expect.objectContaining({ code: 'WEB_DUPLICATE_PROVIDER' })) + }) + + it('keeps search and fetch id namespaces independent', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('shared', available, () => Promise.resolve(searchResult('shared')))) + expect(() => web.registerFetchProvider(makeFetchProvider('shared', available, fetchResult('shared')))).not.toThrow() + }) + + it('rolls back a registration when a providers-change listener throws', async () => { + const { ctx, web } = await mountWeb() + ctx.on('web/providers-change', () => { throw new Error('listener boom') }) + expect(() => web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))) + .toThrow('listener boom') + // The throwing listener must not leave the provider in the registry. + expect(web.searchStatus()).toEqual({ available: false, reason: 'none' }) + }) + + it('disposes provider registrations when the contributing fiber is disposed (HMR safety)', async () => { + const { ctx, web } = await mountWeb() + const fiber = await ctx.plugin(Object.assign((inner: Context) => { + inner.web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + }, { inject: ['web'] })) + expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' }) + await fiber.dispose() + expect(web.searchStatus()).toEqual({ available: false, reason: 'none' }) + }) +}) + +describe('WebService selection status', () => { + it('reports none when nothing is registered', async () => { + const { web } = await mountWeb() + expect(web.searchStatus()).toEqual({ available: false, reason: 'none' }) + expect(web.fetchStatus()).toEqual({ available: false, reason: 'none' }) + }) + + it('auto-selects the single usable provider when no id is configured', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' }) + }) + + it('reports ambiguous when multiple usable providers exist and none is configured', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity')))) + expect(web.searchStatus()).toEqual({ available: false, reason: 'ambiguous' }) + }) + + it('ignores unusable providers when auto-selecting', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + web.registerSearchProvider(makeSearchProvider('perplexity', unavailable, () => Promise.resolve(searchResult('perplexity')))) + expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' }) + }) + + it('reports none when providers exist but none are usable', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa')))) + expect(web.searchStatus()).toEqual({ available: false, reason: 'none' }) + }) + + it('honors a configured id over a different registered provider', async () => { + const { web } = await mountWeb({ searchProvider: 'perplexity' }) + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity')))) + expect(web.searchStatus()).toEqual({ available: true, providerId: 'perplexity' }) + }) + + it('reports configured-missing when the configured id is not registered', async () => { + const { web } = await mountWeb({ searchProvider: 'perplexity' }) + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + expect(web.searchStatus()).toEqual({ available: false, reason: 'configured-missing' }) + }) + + it('reports configured-unavailable when the configured id is registered but unusable', async () => { + const { web } = await mountWeb({ searchProvider: 'exa' }) + web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa')))) + expect(web.searchStatus()).toEqual({ available: false, reason: 'configured-unavailable' }) + }) + + it('does not let registration order change auto-selection', async () => { + const a = await mountWeb() + a.web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa')))) + a.web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity')))) + expect(a.web.searchStatus()).toEqual({ available: true, providerId: 'perplexity' }) + + const b = await mountWeb() + b.web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity')))) + b.web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa')))) + expect(b.web.searchStatus()).toEqual({ available: true, providerId: 'perplexity' }) + }) +}) + +describe('WebService execution resolution', () => { + it('throws WEB_PROVIDER_UNAVAILABLE when nothing is registered', async () => { + const { web } = await mountWeb() + await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_UNAVAILABLE' })) + }) + + it('throws WEB_PROVIDER_CONFIGURED_MISSING for an unregistered configured id', async () => { + const { web } = await mountWeb({ searchProvider: 'perplexity' }) + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_CONFIGURED_MISSING' })) + }) + + it('throws WEB_PROVIDER_CONFIGURED_UNAVAILABLE for an unusable configured id', async () => { + const { web } = await mountWeb({ searchProvider: 'exa' }) + web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa')))) + await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_CONFIGURED_UNAVAILABLE' })) + }) + + it('throws WEB_PROVIDER_AMBIGUOUS rather than picking by order', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity')))) + await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_AMBIGUOUS' })) + }) + + it('runs the selected provider and returns its result', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve( + searchResult('exa', { content: 'answer', sources: [{ url: 'https://a' }] }), + ))) + const result = await web.search({ query: 'q' }) + expect(result.providerId).toBe('exa') + expect(result.content).toBe('answer') + expect(result.sources).toEqual([{ url: 'https://a' }]) + }) + + it('propagates the abort signal to the provider', async () => { + const { web } = await mountWeb() + const seen: (AbortSignal | undefined)[] = [] + web.registerSearchProvider({ + id: 'exa', + status: () => available, + search: (_request, exec) => { seen.push(exec?.signal); return Promise.resolve(searchResult('exa')) }, + }) + const controller = new AbortController() + await web.search({ query: 'q' }, { signal: controller.signal }) + expect(seen[0]).toBe(controller.signal) + }) +}) + +describe('WebService maxResults enforcement', () => { + it('truncates sources and sets truncated when a provider over-returns', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa', { + sources: [{ url: 'https://1' }, { url: 'https://2' }, { url: 'https://3' }], + })))) + const result = await web.search({ query: 'q', maxResults: 2 }) + expect(result.sources).toHaveLength(2) + expect(result.truncated).toBe(true) + }) + + it('leaves truncated false when within the bound', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa', { + sources: [{ url: 'https://1' }], + })))) + const result = await web.search({ query: 'q', maxResults: 8 }) + expect(result.sources).toHaveLength(1) + expect(result.truncated).toBe(false) + }) + + it('does not bound when maxResults is omitted', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa', { + sources: [{ url: 'https://1' }, { url: 'https://2' }], + })))) + const result = await web.search({ query: 'q' }) + expect(result.sources).toHaveLength(2) + expect(result.truncated).toBe(false) + }) +}) + +describe('WebService fetch capability', () => { + it('resolves and runs the fetch provider independently of search', async () => { + const { web } = await mountWeb() + web.registerFetchProvider(makeFetchProvider('local-http', available, fetchResult('local-http'))) + const result = await web.fetch({ url: 'https://example.com' }) + expect(result.providerId).toBe('local-http') + expect(result.statusCode).toBe(200) + }) + + it('throws WEB_PROVIDER_UNAVAILABLE for fetch when no fetch provider is registered', async () => { + const { web } = await mountWeb() + web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))) + await expect(web.fetch({ url: 'https://example.com' })).rejects.toThrow( + expect.objectContaining({ code: 'WEB_PROVIDER_UNAVAILABLE' }), + ) + }) +}) + +describe('WebError', () => { + it('is a HarnessError carrying its code', () => { + const error = new WebError('boom', 'WEB_INVALID_URL') + expect(error.code).toBe('WEB_INVALID_URL') + expect(error.name).toBe('WebError') + }) +}) diff --git a/packages/web/web/tsconfig.json b/packages/web/web/tsconfig.json new file mode 100644 index 0000000000..b187cddf35 --- /dev/null +++ b/packages/web/web/tsconfig.json @@ -0,0 +1,24 @@ +{ + "extends": "../../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib" + }, + "include": [ + "src" + ], + "references": [ + { + "path": "../../../vendor/cosmokit" + }, + { + "path": "../../../vendor/cordis" + }, + { + "path": "../../../vendor/schemastery" + }, + { + "path": "../../llm/llm" + } + ] +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 2afc331514..a393b6c4b9 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -703,6 +703,92 @@ importers: specifier: ^4.0.0-rc.6 version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4) + packages/web/tool-web: + dependencies: + schemastery: + specifier: ^3.18.0 + version: 3.18.0 + devDependencies: + '@deepseek-ai/dsh-agent': + specifier: workspace:^ + version: link:../../core/agent + '@deepseek-ai/dsh-llm': + specifier: workspace:^ + version: link:../../llm/llm + '@deepseek-ai/dsh-session': + specifier: workspace:^ + version: link:../../core/session + '@deepseek-ai/dsh-system-prompt': + specifier: workspace:^ + version: link:../../core/system-prompt + '@deepseek-ai/dsh-tools': + specifier: workspace:^ + version: link:../../core/tools + '@deepseek-ai/dsh-web': + specifier: workspace:^ + version: link:../web + '@deepseek-ai/dsh-web-fetch-local': + specifier: workspace:^ + version: link:../web-fetch-local + '@deepseek-ai/dsh-web-search-exa': + specifier: workspace:^ + version: link:../web-search-exa + cordis: + specifier: ^4.0.0-rc.6 + version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4) + + packages/web/web: + dependencies: + schemastery: + specifier: ^3.18.0 + version: 3.18.0 + devDependencies: + '@deepseek-ai/dsh-llm': + specifier: workspace:^ + version: link:../../llm/llm + cordis: + specifier: ^4.0.0-rc.6 + version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4) + + packages/web/web-fetch-local: + dependencies: + schemastery: + specifier: ^3.18.0 + version: 3.18.0 + devDependencies: + '@deepseek-ai/dsh-web': + specifier: workspace:^ + version: link:../web + cordis: + specifier: ^4.0.0-rc.6 + version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4) + + packages/web/web-search-exa: + dependencies: + schemastery: + specifier: ^3.18.0 + version: 3.18.0 + devDependencies: + '@deepseek-ai/dsh-web': + specifier: workspace:^ + version: link:../web + cordis: + specifier: ^4.0.0-rc.6 + version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4) + + packages/web/web-search-perplexity: + dependencies: + schemastery: + specifier: ^3.18.0 + version: 3.18.0 + devDependencies: + '@deepseek-ai/dsh-web': + specifier: workspace:^ + version: link:../web + cordis: + specifier: ^4.0.0-rc.6 + version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4) + vendor/cordis: dependencies: '@cordisjs/plugin-include': diff --git a/tsconfig.base.json b/tsconfig.base.json index 7f46a9105a..bc4f13bdd5 100644 --- a/tsconfig.base.json +++ b/tsconfig.base.json @@ -34,6 +34,8 @@ "@cordisjs/plugin-timer": ["./vendor/timer/src"], "@cordisjs/plugin-hmr": ["./vendor/hmr/src"], "@cordisjs/plugin-logger-console": ["./vendor/logger-console/src"], + "@deepseek-ai/dsh-tool-web/search": ["./packages/web/tool-web/src/search.ts"], + "@deepseek-ai/dsh-tool-web/fetch": ["./packages/web/tool-web/src/fetch.ts"], // One wildcard maps every @deepseek-ai/dsh- to its source. Package // dir names are unique across groups, so first-on-disk-wins resolution is // unambiguous; adding a package under an existing group needs no edit @@ -45,6 +47,7 @@ "./packages/bash/*/src", "./packages/compact/*/src", "./packages/subagent/*/src", + "./packages/web/*/src", "./packages/session-persistence/*/src", "./packages/ui/*/src", "./packages/util/*/src", diff --git a/tsconfig.build.json b/tsconfig.build.json index 9d76a33385..fb7a058ea8 100644 --- a/tsconfig.build.json +++ b/tsconfig.build.json @@ -27,6 +27,11 @@ { "path": "./packages/llm/llm-pi-ai" }, { "path": "./packages/bash/bash-local" }, { "path": "./packages/bash/tool-bash" }, + { "path": "./packages/web/web" }, + { "path": "./packages/web/web-search-exa" }, + { "path": "./packages/web/web-search-perplexity" }, + { "path": "./packages/web/web-fetch-local" }, + { "path": "./packages/web/tool-web" }, { "path": "./packages/support/invariants" }, { "path": "./packages/ui/acp" }, { "path": "./packages/ui/acp-agent" },