mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-08-29 04:26:38 +00:00
Add web capability seam: ctx.web, search/fetch providers, web tools
Introduce web access as a first-class capability seam so the model-facing web tools stay stable while backends change. dsh-web owns ctx.web as a provider registry with registration-order-independent selection and the WebError taxonomy; dsh-web-search-exa, dsh-web-search-perplexity, and dsh-web-fetch-local register capabilities into it; dsh-tool-web is the sole owner of the model-facing web_search/web_fetch schemas, prompt sections, and HTML-to-markdown presentation. Search and fetch are deliberately one seam. Providers ship as namespace plugins that register into ctx.web (like an LlmAdapter into ctx.llm), not key-owning services, since multiple search providers cannot each own the key. Tool registration follows product enablement, not backend availability, so load order/credentials never enter the model contract; the seam resolves the provider at execution time and surfaces a structured WebError otherwise. Moves the RFC to implemented/ amended to match what shipped. Example/app configs are intentionally not wired yet (RFC migration step 6).
This commit is contained in:
Dudu-0223
@@ -11,6 +11,9 @@ examples/*/.sessions/
|
||||
coverage/
|
||||
.doc-typecheck-*/
|
||||
.humanize/
|
||||
tmp/
|
||||
.claude/commands/
|
||||
.claude/settings.json
|
||||
.vscode/
|
||||
.DS_Store
|
||||
.idea
|
||||
|
||||
@@ -24,6 +24,9 @@ For a catalog of the **data structures** this architecture moves around — the
|
||||
│ @deepseek-ai/dsh-agent-loop (the ONE concrete plugin) │
|
||||
│ @deepseek-ai/dsh-bash-local (bash impl) │
|
||||
│ @deepseek-ai/dsh-tool-bash (bash tool schemas) │
|
||||
│ @deepseek-ai/dsh-web-search-exa (web search impl) │
|
||||
│ @deepseek-ai/dsh-web-fetch-local (web fetch impl) │
|
||||
│ @deepseek-ai/dsh-tool-web (web tool schemas) │
|
||||
│ @deepseek-ai/dsh-session-persistence-jsonl (persistence impl)│
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ @deepseek-ai/dsh-agent (vocabulary + registry) │
|
||||
@@ -33,6 +36,7 @@ For a catalog of the **data structures** this architecture moves around — the
|
||||
│ @deepseek-ai/dsh-session-persistence (persistence seam) │
|
||||
│ @deepseek-ai/dsh-llm (abstract model service) │
|
||||
│ @deepseek-ai/dsh-bash (abstract bash executor) │
|
||||
│ @deepseek-ai/dsh-web (abstract web access) │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ vendor/: cordis, loader, include, group, timer, hmr, │
|
||||
│ logger-console, cosmokit, schemastery │
|
||||
@@ -54,6 +58,7 @@ Dependency rule: **extension** plugins depend on interface packages, never on `d
|
||||
| `ctx.agentLoop` | `AgentLoop` | dsh-agent-loop | creates `ReactLoopAgent`s and drives their loops |
|
||||
| `ctx.bash` | `BashExecutor` (abstract) | dsh-bash | bash execution seam: foreground runs + background tasks |
|
||||
| `ctx.compact` | `CompactService` (abstract) | dsh-compact | compaction seam: decide when history is too large, summarize an older range into a single surface node |
|
||||
| `ctx.web` | `WebService` | dsh-web | web access seam: search/fetch provider registries, registration-order-independent selection, the `WebError` taxonomy |
|
||||
|
||||
All registrations (`registerAdapter`, `section`, `tools`, `register`, …) go through `ctx.effect()` and return disposers, so plugin hot-reload (vendored HMR) and fiber disposal clean up automatically.
|
||||
|
||||
@@ -69,6 +74,8 @@ Swappable capabilities are split into **three packages** so each part evolves in
|
||||
|
||||
The LLM seam has the same topology folded differently: `dsh-llm` carries the interface (`LlmAdapter`) AND the consumer surface (`ctx.llm.stream()`), with adapters as implementation packages — there the consumer is the loop itself, not a swappable schema surface. Use the full three-package split when the consumer is independently replaceable; keep interface + consumer together when they are one concern. Don't split preemptively: a capability with one conceivable implementation and one consumer stays one package until proven otherwise.
|
||||
|
||||
The web capability uses the same three-package split but folds two capabilities onto one seam: `dsh-web` owns the abstract `ctx.web` service, which is a provider REGISTRY (`registerSearchProvider`/`registerFetchProvider`, registration-order-independent selection, the `WebError` taxonomy) rather than a single backend. Providers register capabilities, not tools — `dsh-web-search-exa`, `dsh-web-search-perplexity`, and `dsh-web-fetch-local` each register into `ctx.web` the way an `LlmAdapter` registers into `ctx.llm`, so they are namespace plugins (`inject: ['web']`), not key-owning services. `dsh-tool-web` is the single consumer that owns the model-facing `web_search`/`web_fetch` schemas, prompt sections, and presentation; it reads only the aggregated `ctx.web.searchStatus()`/`fetchStatus()` and executes through `ctx.web.search()`/`fetch()`, so provider selection has one owner. Search and fetch are deliberately one seam (one thing to inject and configure, one selection policy, one abort/error vocabulary) despite sharing no request schema — see the [web capability seam RFC](rfc/implemented/architecture/2026-06-24-web-capability-seam.md).
|
||||
|
||||
> **"Capability" — two unrelated meanings.** (1) The *seam pattern* above ("one plugin provides a capability, another needs it") is realized by plain Cordis **services + `inject`**: a provider registers a service (`ctx.bash`, declared in `interface Context`); a consumer declares `inject: ['bash']` and its fiber stays pending until the service exists, tearing down via HMR if it later vanishes. No extra library is needed. (2) `@cordisjs/plugin-capability` is a different axis entirely — a **permission/capability-security** service (named permissions with inheritance/dependency, tested against a session via `ctx.capability.test`). It is a candidate for the deferred permissions/sandbox work (the `tools/execute` veto seam), NOT a mechanism for swapping implementations.
|
||||
|
||||
## The vocabulary (dsh-llm)
|
||||
|
||||
+1
-1
@@ -59,7 +59,6 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r
|
||||
|---|---|
|
||||
| [Runtime schemas for the event vocabulary (Zod vs the merge-extensible-map pattern)](proposed/architecture/2026-06-16-typed-event-schemas.md) | 2026-06-16 |
|
||||
| [Extract a generic long-running tool runtime](proposed/architecture/2026-06-20-generic-long-running-tool-runtime.md) | 2026-06-20 |
|
||||
| [Web capability seam - provider registry and model-facing web tools](proposed/architecture/2026-06-24-web-capability-seam.md) | 2026-06-24 |
|
||||
|
||||
### Process
|
||||
|
||||
@@ -120,6 +119,7 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r
|
||||
| [Reorganize packages into a modular hierarchy](implemented/architecture/2026-06-20-package-hierarchy.md) | 2026-06-20 |
|
||||
| [Branded IDs everywhere they belong](implemented/architecture/2026-06-20-branded-ids.md) | 2026-06-20 |
|
||||
| [Extract example apps into packages](implemented/architecture/2026-06-20-extract-example-app-packages.md) | 2026-06-20 |
|
||||
| [Web capability seam — provider registry and model-facing web tools](implemented/architecture/2026-06-24-web-capability-seam.md) | 2026-06-24 |
|
||||
|
||||
### Process
|
||||
|
||||
|
||||
+7
-7
@@ -1,6 +1,6 @@
|
||||
# RFC: Web capability seam - stable tools over multiple providers
|
||||
|
||||
Status: proposed
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
@@ -64,7 +64,7 @@ flowchart LR
|
||||
|
||||
`@deepseek-ai/dsh-web` depends only on Cordis and low-level harness support. It declares `ctx.web`, provider interfaces, request/result types, status types, and error codes. It does not import tool, agent, session, LLM, or provider packages.
|
||||
|
||||
Provider packages depend on `@deepseek-ai/dsh-web` and Cordis. They own credentials, endpoint config, provider-specific request mapping, provider-specific response parsing, and provider-specific error translation into `WebError`. They issue network requests with the platform-native `fetch` (Node 24), mirroring `@deepseek-ai/dsh-llm-deepseek`'s adapter, NOT a cordis HTTP-client service (`ctx.http`/`@cordisjs/plugin-http`) — even where a Perplexity provider's request is shaped like an OpenAI-compatible chat completion, that wire shape is a provider-private detail and does not make the provider depend on `ctx.llm`.
|
||||
Provider packages depend on `@deepseek-ai/dsh-web` and Cordis. They own credentials, endpoint config, provider-specific request mapping, provider-specific response parsing, and provider-specific error translation into `WebError`. They issue network requests with the platform-native `fetch` (Node 24), mirroring `@deepseek-ai/dsh-llm-deepseek`'s adapter, NOT a cordis HTTP-client service (`ctx.http`/`@cordisjs/plugin-http`) — even where a Perplexity provider's request is shaped like an OpenAI-compatible chat completion, that wire shape is a provider-private detail and does not make the provider depend on `ctx.llm`. A provider does NOT own the `ctx.web` key (two search providers cannot both own it): like `dsh-llm-deepseek`, each provider package is a function/namespace plugin (`inject: ['web']`) whose `apply` constructs the backend and calls `ctx.web.registerSearchProvider` / `registerFetchProvider`. `@deepseek-ai/dsh-web` is the `export default` service that owns the key.
|
||||
|
||||
`@deepseek-ai/dsh-tool-web` depends on `@deepseek-ai/dsh-web`, `@deepseek-ai/dsh-tools`, `@deepseek-ai/dsh-system-prompt`, and Cordis. It never imports concrete provider packages.
|
||||
|
||||
@@ -267,11 +267,11 @@ SSRF / private-network protection (blocking private, loopback, link-local, multi
|
||||
|
||||
Tool registration in the first version is a minimal stable sync:
|
||||
|
||||
1. On plugin startup, read the product/app config that enables or disables web search and web fetch.
|
||||
2. If web search is enabled, register `web_search` and keep that tool's disposer.
|
||||
3. If web fetch is enabled, register `web_fetch` and keep that tool's disposer.
|
||||
1. On plugin startup, read the `dsh-tool-web` `Config` (`search?: boolean`, `fetch?: boolean`, both default `true`) that enables or disables each web tool.
|
||||
2. If web search is enabled, register `web_search` (its disposer is fiber-scoped via the effect-based registry).
|
||||
3. If web fetch is enabled, register `web_fetch` (likewise fiber-scoped).
|
||||
4. Do not dispose either tool merely because `ctx.web.searchStatus()` or `ctx.web.fetchStatus()` is unavailable.
|
||||
5. Dispose registered tools when the `tool-web` fiber is disposed.
|
||||
5. Disposing the `tool-web` fiber tears down its registrations automatically.
|
||||
|
||||
Provider status changes affect execution results and diagnostics, not whether the model-facing schema exists. If a product wants no web tools at all, it disables `dsh-tool-web` or the individual web tool in config; if it wants web tools but the backend is misconfigured, the model sees a structured tool error at execution time.
|
||||
|
||||
@@ -315,7 +315,7 @@ Search provider tests cover request mapping, response parsing into `content` plu
|
||||
|
||||
Integration tests should load the real seam, provider, and tool packages together and execute through `ctx.tools.execute()` rather than calling providers directly. If wiring the tools into an ACP-facing example changes editor-visible transcripts, add or update the relevant snapshot scenario in the same change.
|
||||
|
||||
At least one test must drive these packages through their REAL cordis Loader/export path, not a hand-built `ctx.plugin({...})` mount, so a broken export shape is caught (see [docs/postmortem/0001](../../../postmortem/0001-acp-default-export-drops-inject.md) and `packages/AGENTS.md` § plugin-export-shape). The two shapes need different guards: `dsh-web` and the provider packages are **services** (`export default` the class) and a stray extra export would surface as a missing service; `dsh-tool-web` is a **namespace plugin** (named `name`/`inject`/`apply`, NO default), and because it has `inject`, a stray `export default apply` makes `unwrapExports` drop the `inject` and the plugin throws `cannot get property … without inject` the moment it loads — so a Loader smoke that boots tool-web over `ctx.web` catches it. Prove the guard bites: add `export default apply` to `tool-web`, watch the smoke go red, revert.
|
||||
At least one test must drive these packages through their REAL cordis Loader/export path, not a hand-built `ctx.plugin({...})` mount, so a broken export shape is caught (see [docs/postmortem/0001](../../../postmortem/0001-acp-default-export-drops-inject.md) and `packages/AGENTS.md` § plugin-export-shape). The two shapes need different guards: `dsh-web` is a **service** (`export default` the class) and a stray extra export would surface as a missing service; the provider packages and `dsh-tool-web` are **namespace plugins** (named `name`/`inject`/`apply`, NO default), and because each has `inject`, a stray `export default apply` makes `unwrapExports` drop the `inject` and the plugin throws `cannot get property … without inject` the moment it loads — so a Loader smoke that boots tool-web over `ctx.web` catches it (and each provider's registration test mounts it the real way and asserts no default export). Prove the guard bites: add `export default apply` to `tool-web`, watch the smoke go red, revert.
|
||||
|
||||
## Migration plan
|
||||
|
||||
@@ -29,6 +29,14 @@
|
||||
"entry": ["tests/**/*.spec.ts", "tests/**/*.e2e.ts"],
|
||||
"project": ["src/**/*.ts", "tests/**/*.ts"]
|
||||
},
|
||||
"packages/web/web-search-exa": {
|
||||
"entry": ["tests/**/*.spec.ts", "tests/**/*.e2e.ts"],
|
||||
"project": ["src/**/*.ts", "tests/**/*.ts"]
|
||||
},
|
||||
"packages/web/web-search-perplexity": {
|
||||
"entry": ["tests/**/*.spec.ts", "tests/**/*.e2e.ts"],
|
||||
"project": ["src/**/*.ts", "tests/**/*.ts"]
|
||||
},
|
||||
"packages/ui/acp-agent": {
|
||||
"entry": ["tests/**/*.spec.ts", "tests/**/*.e2e.ts"],
|
||||
"project": ["src/**/*.ts", "tests/**/*.ts"]
|
||||
|
||||
@@ -13,6 +13,7 @@ Packages are grouped by modular role at `packages/<group>/<pkg>/`. The group dir
|
||||
| [`bash/`](bash/README.md) | Bash capability family: the executor seam, a local impl, and the model-facing tool | Product — stable surface |
|
||||
| [`compact/`](compact/README.md) | Compaction capability family: the abstract seam (backend + tool deferred) | Product — stable surface |
|
||||
| [`subagent/`](subagent/README.md) | Subagent capability family: the provider-registry seam and the model-facing delegation tool | Product — stable surface |
|
||||
| [`web/`](web/README.md) | Web capability family: the abstract seam, search/fetch provider impls, and the model-facing web tools | Product — stable surface |
|
||||
| [`session-persistence/`](session-persistence/README.md) | Persistence capability family: the seam + JSONL/SQLite backends | Product — stable surface |
|
||||
| [`ui/`](ui/README.md) | Editor/client integration surfaces (the ACP bridge) | Product — stable surface |
|
||||
| [`support/`](support/README.md) | Dev/test/example infrastructure (invariants, stdio UI, replay adapter) | Support — lower compatibility expectations |
|
||||
@@ -33,6 +34,11 @@ dsh-compact ← dsh-session, dsh-llm (abstract compaction s
|
||||
dsh-tools ← dsh-llm, dsh-system-prompt, dsh-agent
|
||||
dsh-bash-local ← dsh-bash (BashExecutor impl)
|
||||
dsh-tool-bash ← dsh-bash, dsh-tools (bash tool schemas)
|
||||
dsh-web ← dsh-llm (abstract web seam; search/fetch registries, WebError)
|
||||
dsh-web-search-exa ← dsh-web (Exa WebSearchProvider)
|
||||
dsh-web-search-perplexity ← dsh-web (Perplexity WebSearchProvider)
|
||||
dsh-web-fetch-local ← dsh-web (anonymous public HTTP(S) WebFetchProvider)
|
||||
dsh-tool-web ← dsh-web, dsh-tools, dsh-system-prompt (web tool schemas)
|
||||
dsh-llm-deepseek ← dsh-llm (DeepSeek adapter)
|
||||
dsh-llm-pi-ai ← dsh-llm (pi-ai-backed adapter)
|
||||
dsh-agent-loop ← dsh-llm, dsh-session, dsh-system-prompt, dsh-tools, dsh-agent
|
||||
@@ -68,6 +74,11 @@ The rule: **extension** plugins depend on interfaces, never on the concrete loop
|
||||
| `bash-local/` | `bash` | Local-subprocess `BashExecutor` implementation | (registers `ctx.bash`) |
|
||||
| `tool-bash/` | `bash` | Model-facing `bash`/`bash_output`/`bash_kill` tool schemas | (registers on `ctx.tools`) |
|
||||
| `compact/` | `compact` | Abstract compaction seam + `compact/*` events + `CompactionResult` | `ctx.compact` |
|
||||
| `web/` | `web` | Abstract web seam (search/fetch provider registries + selection + vocabulary + `WebError`) | `ctx.web` |
|
||||
| `web-search-exa/` | `web` | Exa-backed `WebSearchProvider` | (registers on `ctx.web`) |
|
||||
| `web-search-perplexity/` | `web` | Perplexity-backed `WebSearchProvider` | (registers on `ctx.web`) |
|
||||
| `web-fetch-local/` | `web` | Anonymous public HTTP(S) `WebFetchProvider` | (registers on `ctx.web`) |
|
||||
| `tool-web/` | `web` | Model-facing `web_search`/`web_fetch` tool schemas | (registers on `ctx.tools`) |
|
||||
| `llm-deepseek/` | `llm` | DeepSeek API adapter (hand-rolled fetch/SSE) | (registers on `ctx.llm`) |
|
||||
| `llm-pi-ai/` | `llm` | DeepSeek adapter via `@earendil-works/pi-ai` (design twin) | (registers on `ctx.llm`) |
|
||||
| `session-persistence/` | `session-persistence` | Persistence seam + write coordinator | `ctx.sessionPersistence` |
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
# web/ - web capability family
|
||||
|
||||
The web access capability seam: an abstract web interface, search/fetch provider implementations, and the model-facing web tools. All **product** packages.
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| `web/` | Abstract web seam (search/fetch provider registries + selection + vocabulary + `WebError`) | `ctx.web` |
|
||||
| `web-search-exa/` | Exa-backed `WebSearchProvider` | (registers on `ctx.web`) |
|
||||
| `web-search-perplexity/` | Perplexity-backed `WebSearchProvider` | (registers on `ctx.web`) |
|
||||
| `web-fetch-local/` | Anonymous public HTTP(S) `WebFetchProvider` | (registers on `ctx.web`) |
|
||||
| `tool-web/` | Model-facing `web_search`/`web_fetch` tool schemas | (registers on `ctx.tools`) |
|
||||
|
||||
The interface lives at `web/web/`. Unlike bash/fs, the seam spans **two capabilities** (search and fetch) with potentially multiple providers each: `ctx.web` is one web-access middle layer with one provider-selection policy, one abort/error vocabulary, and one product-facing "how this harness reaches the web" config surface. Providers register **capabilities**, not tools; `tool-web` is the only owner of model-facing names, schemas, prompt guidance, and presentation. A search provider swap does not change how the model asks for a query, and a fetch implementation swap does not change how the model asks for a URL.
|
||||
|
||||
See the [web capability seam RFC](../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md) for the design rationale, including why search and fetch are deliberately one seam and why `web_fetch`'s SSRF protection is deferred.
|
||||
@@ -0,0 +1,30 @@
|
||||
# @deepseek-ai/dsh-tool-web
|
||||
|
||||
The model-facing web tool suite — `web_search` and `web_fetch` — over the [web capability seam](../web/README.md) (`ctx.web`). It owns model-facing concerns only: tool names, JSON schemas, snake_case argument names, prompt sections, the result-count bound, result formatting, HTML→markdown presentation, and `presentCall`. All web access goes through `ctx.web`; this package never imports a concrete provider.
|
||||
|
||||
Each tool is also a subpath plugin (`@deepseek-ai/dsh-tool-web/search`, `/fetch`) for focused deployments.
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Args | Behavior |
|
||||
|---|---|---|
|
||||
| `web_search` | `query` (string) | Discovery. Returns an optional answer plus source URLs. `max_results` is **not** model-facing — the tool sets the bound (`WEB_SEARCH_MAX_RESULTS = 8`) and passes it to the seam. |
|
||||
| `web_fetch` | `url` (string), `timeout_ms` (number, optional) | Retrieves a specific URL. HTML bodies are rendered to markdown-ish text; text bodies pass through. A non-2xx status is reported, not an error. |
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `search` | `true` | Register `web_search`. |
|
||||
| `fetch` | `true` | Register `web_fetch`. |
|
||||
|
||||
```yaml
|
||||
- id: tool-web
|
||||
name: '@deepseek-ai/dsh-tool-web'
|
||||
```
|
||||
|
||||
## Stable registration
|
||||
|
||||
Tool registration follows product **enablement**, not backend availability. A tool stays visible even when its selected provider is missing, misconfigured, ambiguous, or temporarily unavailable; the seam resolves the provider at execution time and execution fails with a structured `WebError` (e.g. `WEB_PROVIDER_UNAVAILABLE`, `WEB_PROVIDER_AMBIGUOUS`), which `ToolRegistry.execute()` turns into an error tool result the model can read and hooks/UI can route on. This keeps the model schema stable without making plugin load order, credential state, or HMR timing part of the model-facing contract. To remove a web tool entirely, disable it here in config.
|
||||
|
||||
The tool reads only the aggregated `ctx.web.searchStatus()` / `fetchStatus()` for diagnostics — never each provider's `status()` directly — so provider selection has one owner.
|
||||
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-tool-web",
|
||||
"description": "Model-facing web tools (web_search, web_fetch) over the DeepSeek Harness web capability seam (ctx.web)",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/index.d.ts",
|
||||
"exports": {
|
||||
".": { "types": "./lib/index.d.ts", "default": "./lib/index.js" },
|
||||
"./search": { "types": "./lib/search.d.ts", "default": "./lib/search.js" },
|
||||
"./fetch": { "types": "./lib/fetch.d.ts", "default": "./lib/fetch.js" },
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-system-prompt": "^0.0.1",
|
||||
"@deepseek-ai/dsh-tools": "^0.0.1",
|
||||
"@deepseek-ai/dsh-web": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-system-prompt": "workspace:^",
|
||||
"@deepseek-ai/dsh-tools": "workspace:^",
|
||||
"@deepseek-ai/dsh-web": "workspace:^",
|
||||
"@deepseek-ai/dsh-web-fetch-local": "workspace:^",
|
||||
"@deepseek-ai/dsh-web-search-exa": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* The model-facing `web_fetch` tool: retrieve the content of a specific URL.
|
||||
* Execution goes through `ctx.web` — this module owns the model-facing schema,
|
||||
* argument validation, and PRESENTATION (HTML→markdown, truncation formatting),
|
||||
* while the fetch provider owns safe retrieval (transport, redirects, caps).
|
||||
*
|
||||
* @module @deepseek-ai/dsh-tool-web/fetch
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import { defineTool } from '@deepseek-ai/dsh-tools'
|
||||
import type { ToolCallPresentation } from '@deepseek-ai/dsh-tools'
|
||||
import type { ContentBlock } from '@deepseek-ai/dsh-llm'
|
||||
import type { WebFetchBody, WebFetchResult } from '@deepseek-ai/dsh-web'
|
||||
import { assertNever } from '@deepseek-ai/dsh-llm'
|
||||
import type {} from '@deepseek-ai/dsh-system-prompt'
|
||||
import { htmlToMarkdown } from './html.ts'
|
||||
|
||||
/** Validate value constraints the schema DSL can't express. */
|
||||
export function parseFetchArgs(args: { url: string; timeout_ms?: number }): { url: string; timeoutMs?: number } {
|
||||
if (args.url.trim().length === 0) throw new Error('url must be a non-empty string')
|
||||
if (args.timeout_ms !== undefined && (!Number.isFinite(args.timeout_ms) || args.timeout_ms <= 0)) {
|
||||
throw new Error('timeout_ms must be a positive number')
|
||||
}
|
||||
return { url: args.url, ...args.timeout_ms !== undefined ? { timeoutMs: args.timeout_ms } : {} }
|
||||
}
|
||||
|
||||
/** Render a fetched body to model-facing markdown text. */
|
||||
export function renderBody(body: WebFetchBody): string {
|
||||
switch (body.kind) {
|
||||
case 'html':
|
||||
return htmlToMarkdown(body.content)
|
||||
case 'text':
|
||||
return body.content
|
||||
/* v8 ignore next 2 -- WebFetchBody is a closed union; this arm is unreachable and only makes adding a kind a compile error. */
|
||||
default:
|
||||
return assertNever(body, 'unhandled web fetch body kind')
|
||||
}
|
||||
}
|
||||
|
||||
/** Format a fetch result as one model-facing text block. */
|
||||
export function formatFetchOutput(result: WebFetchResult): string {
|
||||
const header = `Fetched ${result.url} (HTTP ${result.statusCode})`
|
||||
const footer = result.truncated ? '\n\n(Content truncated. Fetch a more specific URL or section for the full text.)' : ''
|
||||
return `${header}\n\n${renderBody(result.body)}${footer}`
|
||||
}
|
||||
|
||||
/** Pending-call presentation: a fetch card titled by the URL. */
|
||||
export function presentFetchCall(args: { url: string; timeout_ms?: number }): ToolCallPresentation {
|
||||
return { title: args.url, kind: 'fetch', rawInput: args.url }
|
||||
}
|
||||
|
||||
/** Register the `web_fetch` tool and its system-prompt guidance. */
|
||||
export function apply(ctx: Context): void {
|
||||
ctx.systemPrompt.section({
|
||||
name: 'tool:web_fetch',
|
||||
order: 111,
|
||||
text: 'Use the web_fetch tool to retrieve the content of a specific HTTP(S) URL (for example a result from web_search). It returns the page content decoded to text. Cite the URL as a markdown link when you use its content.',
|
||||
})
|
||||
|
||||
ctx.tools.register(defineTool({
|
||||
name: 'web_fetch',
|
||||
description: 'Fetch the content of a specific HTTP(S) URL and return it decoded to text.',
|
||||
parameters: {
|
||||
url: { type: 'string', required: true, description: 'The HTTP(S) URL to fetch.' },
|
||||
timeout_ms: { type: 'number', description: 'Optional fetch timeout in milliseconds (capped by the provider).' },
|
||||
},
|
||||
async execute(args, exec): Promise<ContentBlock[]> {
|
||||
const input = parseFetchArgs(args)
|
||||
const result = await ctx.web.fetch(
|
||||
{ url: input.url, ...input.timeoutMs !== undefined ? { timeoutMs: input.timeoutMs } : {} },
|
||||
exec.signal ? { signal: exec.signal } : undefined,
|
||||
)
|
||||
return [{ type: 'text', text: formatFetchOutput(result) }]
|
||||
},
|
||||
presentCall: presentFetchCall,
|
||||
}))
|
||||
}
|
||||
|
||||
/** Cordis plugin name used by loader diagnostics. */
|
||||
export const name = 'web-fetch'
|
||||
|
||||
/** Services required by the `web_fetch` tool plugin. */
|
||||
export const inject = ['tools', 'web', 'systemPrompt']
|
||||
|
||||
/** Named helper for direct registration in the root plugin and tests. */
|
||||
export const applyWebFetchTool = apply
|
||||
@@ -0,0 +1,85 @@
|
||||
/**
|
||||
* Minimal, dependency-free HTML→markdown-ish text conversion for `web_fetch`
|
||||
* presentation. This is intentionally NOT a full HTML parser: it strips
|
||||
* script/style/noscript, drops tags, decodes the common named/numeric entities,
|
||||
* and collapses whitespace into a readable plain-text approximation with a few
|
||||
* markdown affordances (headings, list bullets, links). A heavier converter can
|
||||
* replace this without touching the seam or the tool schema.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-tool-web/html
|
||||
*/
|
||||
|
||||
/** Decode the handful of HTML entities common in textual content. */
|
||||
function decodeEntities(text: string): string {
|
||||
return text
|
||||
.replace(/&(#[xX][0-9a-fA-F]+|#[0-9]+|[a-zA-Z]+);/g, (match, entity: string) => {
|
||||
if (entity.startsWith('#x') || entity.startsWith('#X')) {
|
||||
const code = Number.parseInt(entity.slice(2), 16)
|
||||
return safeFromCodePoint(code, match)
|
||||
}
|
||||
if (entity.startsWith('#')) {
|
||||
const code = Number.parseInt(entity.slice(1), 10)
|
||||
return safeFromCodePoint(code, match)
|
||||
}
|
||||
return NAMED_ENTITIES[entity] ?? match
|
||||
})
|
||||
}
|
||||
|
||||
const NAMED_ENTITIES: Record<string, string> = {
|
||||
amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ',
|
||||
copy: '©', reg: '®', trade: '™', hellip: '…', mdash: '—', ndash: '–',
|
||||
}
|
||||
|
||||
function safeFromCodePoint(code: number, fallback: string): string {
|
||||
try {
|
||||
return String.fromCodePoint(code)
|
||||
} catch {
|
||||
// An out-of-range code point (RangeError) is the only failure here; keep the
|
||||
// original entity text rather than throwing out of pure presentation.
|
||||
return fallback
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert an HTML document to a readable markdown-ish text approximation.
|
||||
* Best-effort and lossy by design — fidelity is the job of a future heavier
|
||||
* converter, not this fallback.
|
||||
*/
|
||||
export function htmlToMarkdown(html: string): string {
|
||||
let text = html
|
||||
// Drop non-content elements entirely (including their contents).
|
||||
.replace(/<script\b[^>]*>[\s\S]*?<\/script>/gi, '')
|
||||
.replace(/<style\b[^>]*>[\s\S]*?<\/style>/gi, '')
|
||||
.replace(/<noscript\b[^>]*>[\s\S]*?<\/noscript>/gi, '')
|
||||
.replace(/<!--[\s\S]*?-->/g, '')
|
||||
|
||||
// Convert links to markdown before stripping tags.
|
||||
text = text.replace(/<a\b[^>]*\bhref\s*=\s*["']([^"']*)["'][^>]*>([\s\S]*?)<\/a>/gi, (_match, href: string, label: string) => {
|
||||
const cleanLabel = label.replace(/<[^>]+>/g, '').trim()
|
||||
return cleanLabel.length > 0 ? `[${cleanLabel}](${href})` : href
|
||||
})
|
||||
|
||||
// Headings → markdown hashes.
|
||||
text = text.replace(/<h([1-6])\b[^>]*>([\s\S]*?)<\/h\1>/gi, (_match, level: string, body: string) => {
|
||||
const hashes = '#'.repeat(Number(level))
|
||||
return `\n\n${hashes} ${body.replace(/<[^>]+>/g, '').trim()}\n\n`
|
||||
})
|
||||
|
||||
// List items → bullets.
|
||||
text = text.replace(/<li\b[^>]*>([\s\S]*?)<\/li>/gi, (_match, body: string) => `\n- ${body.replace(/<[^>]+>/g, '').trim()}`)
|
||||
|
||||
// Block-level breaks become paragraph breaks.
|
||||
text = text
|
||||
.replace(/<\/(p|div|section|article|header|footer|tr|table|ul|ol|blockquote)>/gi, '\n\n')
|
||||
.replace(/<br\s*\/?>/gi, '\n')
|
||||
|
||||
// Drop all remaining tags, decode entities, collapse whitespace.
|
||||
text = text.replace(/<[^>]+>/g, '')
|
||||
text = decodeEntities(text)
|
||||
text = text
|
||||
.replace(/[ \t\f\v]+/g, ' ')
|
||||
.replace(/ *\n */g, '\n')
|
||||
.replace(/\n{3,}/g, '\n\n')
|
||||
.trim()
|
||||
return text
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
/**
|
||||
* The model-facing web tool suite (`web_search`, `web_fetch`) over the `ctx.web`
|
||||
* seam. This root plugin registers the tools the product has ENABLED, composing
|
||||
* the per-tool registration helpers; each tool is also exposed as a subpath
|
||||
* plugin (`@deepseek-ai/dsh-tool-web/search`, `/fetch`) for focused deployments.
|
||||
*
|
||||
* The package owns model-facing concerns only — tool names, JSON schemas,
|
||||
* argument validation, prompt sections, result-cap constants, result formatting,
|
||||
* HTML→markdown presentation. All web access goes through `ctx.web`; this
|
||||
* package never imports a concrete provider package.
|
||||
*
|
||||
* Tool registration follows product/app ENABLEMENT, not backend availability: a
|
||||
* tool stays visible even when its selected provider is missing/misconfigured,
|
||||
* and execution fails with a structured `WebError` (resolved by the seam at call
|
||||
* time). That keeps the model schema stable without making plugin load order,
|
||||
* credential state, or HMR timing part of the model-facing contract.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-tool-web
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {} from '@deepseek-ai/dsh-web'
|
||||
import { applyWebSearchTool } from './search.ts'
|
||||
import { applyWebFetchTool } from './fetch.ts'
|
||||
|
||||
export { WEB_SEARCH_MAX_RESULTS, applyWebSearchTool, formatSearchOutput, parseSearchArgs, presentSearchCall } from './search.ts'
|
||||
export { applyWebFetchTool, formatFetchOutput, parseFetchArgs, presentFetchCall, renderBody } from './fetch.ts'
|
||||
export { htmlToMarkdown } from './html.ts'
|
||||
|
||||
/** Cordis plugin name used by loader diagnostics. */
|
||||
export const name = 'tool-web'
|
||||
|
||||
/** Services required by the web tool suite. */
|
||||
export const inject = ['tools', 'web', 'systemPrompt']
|
||||
|
||||
export interface Config {
|
||||
/** Register `web_search`. Defaults to true. */
|
||||
search?: boolean
|
||||
/** Register `web_fetch`. Defaults to true. */
|
||||
fetch?: boolean
|
||||
}
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
search: z.boolean().default(true),
|
||||
fetch: z.boolean().default(true),
|
||||
})
|
||||
|
||||
/**
|
||||
* Register the enabled web tools. `search`/`fetch` default to true; a product
|
||||
* that wants only one disables the other in config. The tools' disposers are
|
||||
* fiber-scoped (the effect-based registries clean up on dispose), so no manual
|
||||
* teardown is needed.
|
||||
*/
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
if (config.search !== false) applyWebSearchTool(ctx)
|
||||
if (config.fetch !== false) applyWebFetchTool(ctx)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* The model-facing `web_search` tool: discover current information on the web.
|
||||
* Execution goes through `ctx.web` — this module owns only the model-facing
|
||||
* schema, argument validation, the result-count bound, and result formatting,
|
||||
* never provider selection or network access.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-tool-web/search
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import { defineTool } from '@deepseek-ai/dsh-tools'
|
||||
import type { ToolCallPresentation } from '@deepseek-ai/dsh-tools'
|
||||
import type { ContentBlock } from '@deepseek-ai/dsh-llm'
|
||||
import type { WebSearchResult } from '@deepseek-ai/dsh-web'
|
||||
import type {} from '@deepseek-ai/dsh-system-prompt'
|
||||
|
||||
/**
|
||||
* Default upper bound on returned sources. Owned by the consumer (not the
|
||||
* provider or model), mirroring `dsh-tool-fs`'s `READ_LIMIT`/`GREP_LIMIT`. The
|
||||
* model just asks a question; the product controls how much context returns.
|
||||
* The default `8` aligns with OpenCode's Exa default.
|
||||
*/
|
||||
export const WEB_SEARCH_MAX_RESULTS = 8
|
||||
|
||||
/** Validate value constraints the schema DSL can't express. */
|
||||
export function parseSearchArgs(args: { query: string }): { query: string } {
|
||||
if (args.query.trim().length === 0) throw new Error('query must be a non-empty string')
|
||||
return { query: args.query }
|
||||
}
|
||||
|
||||
/** Display label for a source: its title, else its hostname. */
|
||||
function sourceLabel(url: string, title: string | undefined): string {
|
||||
if (title !== undefined && title.length > 0) return title
|
||||
try {
|
||||
return new URL(url).hostname
|
||||
} catch {
|
||||
// A provider should return a valid URL, but never let a malformed one throw
|
||||
// out of pure formatting — fall back to the raw string.
|
||||
return url
|
||||
}
|
||||
}
|
||||
|
||||
/** Format a search result as one model-facing text block. */
|
||||
export function formatSearchOutput(result: WebSearchResult): string {
|
||||
const parts: string[] = []
|
||||
if (result.content !== undefined && result.content.length > 0) parts.push(result.content)
|
||||
|
||||
if (result.sources.length > 0) {
|
||||
const lines = result.sources.map((source) => {
|
||||
const label = sourceLabel(source.url, source.title)
|
||||
const meta: string[] = []
|
||||
if (source.snippet !== undefined && source.snippet.length > 0) meta.push(source.snippet)
|
||||
if (source.publishedAt !== undefined && source.publishedAt.length > 0) meta.push(`(${source.publishedAt})`)
|
||||
const suffix = meta.length > 0 ? ` — ${meta.join(' ')}` : ''
|
||||
return `- [${label}](${source.url})${suffix}`
|
||||
})
|
||||
parts.push(`Sources:\n${lines.join('\n')}`)
|
||||
} else if (result.content === undefined || result.content.length === 0) {
|
||||
parts.push('No results found.')
|
||||
}
|
||||
|
||||
if (result.truncated) parts.push(`(Showing the first ${result.sources.length} sources. Refine the query for more.)`)
|
||||
parts.push('Cite the relevant URLs above as markdown links in your answer.')
|
||||
return parts.join('\n\n')
|
||||
}
|
||||
|
||||
/** Pending-call presentation: a search card titled by the query. */
|
||||
export function presentSearchCall(args: { query: string }): ToolCallPresentation {
|
||||
return { title: args.query, kind: 'search', rawInput: args.query }
|
||||
}
|
||||
|
||||
/** Register the `web_search` tool and its system-prompt guidance. */
|
||||
export function apply(ctx: Context): void {
|
||||
ctx.systemPrompt.section({
|
||||
name: 'tool:web_search',
|
||||
order: 110,
|
||||
text: 'Use the web_search tool to discover current information on the web. It returns an optional answer plus a list of source URLs. Follow up with web_fetch when you need the full content of a specific result, and cite the relevant URLs as markdown links.',
|
||||
})
|
||||
|
||||
ctx.tools.register(defineTool({
|
||||
name: 'web_search',
|
||||
description: 'Search the web for current information. Returns an optional summary answer and a list of source URLs.',
|
||||
parameters: {
|
||||
query: { type: 'string', required: true, description: 'The search query.' },
|
||||
},
|
||||
async execute(args, exec): Promise<ContentBlock[]> {
|
||||
const input = parseSearchArgs(args)
|
||||
const result = await ctx.web.search(
|
||||
{ query: input.query, maxResults: WEB_SEARCH_MAX_RESULTS },
|
||||
exec.signal ? { signal: exec.signal } : undefined,
|
||||
)
|
||||
return [{ type: 'text', text: formatSearchOutput(result) }]
|
||||
},
|
||||
presentCall: presentSearchCall,
|
||||
}))
|
||||
}
|
||||
|
||||
/** Cordis plugin name used by loader diagnostics. */
|
||||
export const name = 'web-search'
|
||||
|
||||
/** Services required by the `web_search` tool plugin. */
|
||||
export const inject = ['tools', 'web', 'systemPrompt']
|
||||
|
||||
/** Named helper for direct registration in the root plugin and tests. */
|
||||
export const applyWebSearchTool = apply
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* Integration: the real fetch backend (`dsh-web-fetch-local`) + a real search
|
||||
* provider (`dsh-web-search-exa`) + the real seam (`dsh-web`) + the model tool
|
||||
* (`dsh-tool-web`), exercised through `ctx.tools.execute()` — nothing bypasses
|
||||
* the tool registry. Fetch hits a real loopback HTTP server (verifying the
|
||||
* WORLD); search runs the real Exa provider over a stubbed global `fetch` (the
|
||||
* network is the one boundary we mock).
|
||||
*/
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'
|
||||
import { AddressInfo } from 'node:net'
|
||||
import { Context } from 'cordis'
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry from '@deepseek-ai/dsh-tools'
|
||||
import WebService from '@deepseek-ai/dsh-web'
|
||||
import * as WebFetchLocal from '@deepseek-ai/dsh-web-fetch-local'
|
||||
import * as WebSearchExa from '@deepseek-ai/dsh-web-search-exa'
|
||||
import * as ToolWeb from '@deepseek-ai/dsh-tool-web'
|
||||
|
||||
type Handler = (req: IncomingMessage, res: ServerResponse) => void
|
||||
|
||||
let server: Server
|
||||
let base: string
|
||||
let handler: Handler
|
||||
let ctx: Context
|
||||
let fiber: Awaited<ReturnType<Context['plugin']>>
|
||||
|
||||
beforeEach(async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/html' }); res.end('<h1>Hello</h1><p>World</p>') }
|
||||
server = createServer((req, res) => { handler(req, res) })
|
||||
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
|
||||
base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`
|
||||
|
||||
ctx = new Context()
|
||||
await ctx.plugin(SystemPrompt)
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(WebService, { searchProvider: WebSearchExa.EXA_PROVIDER_ID, fetchProvider: WebFetchLocal.LOCAL_FETCH_PROVIDER_ID })
|
||||
await ctx.plugin(WebFetchLocal, {})
|
||||
await ctx.plugin(WebSearchExa, { apiKey: 'exa-key', baseURL: 'https://api.exa.test' })
|
||||
fiber = await ctx.plugin(ToolWeb)
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await fiber.dispose()
|
||||
vi.unstubAllGlobals()
|
||||
await new Promise<void>(resolve => server.close(() => { resolve() }))
|
||||
})
|
||||
|
||||
let counter = 0
|
||||
type ToolResult = { isError: boolean; content: { type: string; text?: string }[]; error?: { code: string } }
|
||||
function call(name: string, args: unknown): Promise<ToolResult> {
|
||||
return ctx.tools.execute({ callId: CallId(`call-${++counter}`), name, arguments: args })
|
||||
}
|
||||
|
||||
describe('web_fetch integration over the real backend', () => {
|
||||
it('fetches an html page and renders it to markdown', async () => {
|
||||
const out = await call('web_fetch', { url: base })
|
||||
expect(out.isError).toBe(false)
|
||||
const text = out.content.map(b => b.text).join('')
|
||||
expect(text).toContain(`Fetched ${base}`)
|
||||
expect(text).toContain('# Hello')
|
||||
expect(text).toContain('World')
|
||||
})
|
||||
|
||||
it('reports a 404 as a result, not an error', async () => {
|
||||
handler = (_req, res) => { res.writeHead(404, { 'content-type': 'text/plain' }); res.end('missing') }
|
||||
const out = await call('web_fetch', { url: base })
|
||||
expect(out.isError).toBe(false)
|
||||
expect(out.content.map(b => b.text).join('')).toContain('HTTP 404')
|
||||
})
|
||||
|
||||
it('surfaces WEB_INVALID_URL as a structured tool error', async () => {
|
||||
const out = await call('web_fetch', { url: 'ftp://example.com' })
|
||||
expect(out.isError).toBe(true)
|
||||
expect(out.error?.code).toBe('WEB_INVALID_URL')
|
||||
})
|
||||
|
||||
it('surfaces a blocked cross-origin redirect as WEB_REDIRECT_BLOCKED', async () => {
|
||||
handler = (_req, res) => { res.writeHead(302, { location: 'https://example.com/' }); res.end() }
|
||||
const out = await call('web_fetch', { url: base })
|
||||
expect(out.isError).toBe(true)
|
||||
expect(out.error?.code).toBe('WEB_REDIRECT_BLOCKED')
|
||||
})
|
||||
})
|
||||
|
||||
describe('web_search integration over the real Exa provider', () => {
|
||||
it('runs web_search end-to-end and formats the provider result', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => new Response(
|
||||
JSON.stringify({ results: [{ url: 'https://result.test', title: 'Result', highlights: ['a highlight'] }] }),
|
||||
{ status: 200, headers: { 'content-type': 'application/json' } },
|
||||
)))
|
||||
const out = await call('web_search', { query: 'deepseek' })
|
||||
expect(out.isError).toBe(false)
|
||||
expect(out.content.map(b => b.text).join('')).toContain('[Result](https://result.test)')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
/**
|
||||
* Real-load-path guard for @deepseek-ai/dsh-tool-web. `tool-web` is a NAMESPACE
|
||||
* plugin with `inject` — so a stray `export default apply` would make the cordis
|
||||
* Loader's `unwrapExports` (`exports.default ?? exports`) collapse the module to
|
||||
* the bare `apply` function, DROPPING `inject`. The plugin would then read
|
||||
* `ctx.web` without having injected it and throw `cannot get property … without
|
||||
* inject` the moment it loads (postmortem 0001).
|
||||
*
|
||||
* A hand-built `ctx.plugin({ apply, inject })` mount CANNOT catch that — it
|
||||
* bypasses `unwrapExports`. So this test unwraps the module through the REAL
|
||||
* `Loader.prototype.unwrapExports` and mounts the result over `ctx.web`,
|
||||
* exercising the exact path the Loader uses. Prove the guard bites: add
|
||||
* `export default apply` to `src/index.ts`, watch this go red, revert.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import Loader from '@cordisjs/plugin-loader'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry from '@deepseek-ai/dsh-tools'
|
||||
import WebService from '@deepseek-ai/dsh-web'
|
||||
import * as toolWeb from '@deepseek-ai/dsh-tool-web'
|
||||
|
||||
describe('dsh-tool-web real-load-path guard', () => {
|
||||
it('has no default export and keeps name/inject/Config through unwrapExports', () => {
|
||||
expect('default' in toolWeb).toBe(false)
|
||||
|
||||
const loader = Object.create(Loader.prototype) as Loader
|
||||
const unwrapped = loader.unwrapExports(toolWeb) as Record<string, unknown>
|
||||
expect(unwrapped).toBe(toolWeb)
|
||||
expect(unwrapped.name).toBe('tool-web')
|
||||
expect(unwrapped.inject).toEqual(['tools', 'web', 'systemPrompt'])
|
||||
expect(typeof unwrapped.apply).toBe('function')
|
||||
})
|
||||
|
||||
it('boots over ctx.web through the unwrapped module without an inject error', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SystemPrompt)
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(WebService, {})
|
||||
|
||||
const loader = Object.create(Loader.prototype) as Loader
|
||||
const unwrapped = loader.unwrapExports(toolWeb) as Parameters<Context['plugin']>[0]
|
||||
// A collapsed export shape (dropped inject) would throw "without inject" here.
|
||||
const fiber = await ctx.plugin(unwrapped)
|
||||
expect(ctx.tools.schemas().map(s => s.name)).toEqual(expect.arrayContaining(['web_search', 'web_fetch']))
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,281 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry from '@deepseek-ai/dsh-tools'
|
||||
import WebService from '@deepseek-ai/dsh-web'
|
||||
import type { WebSearchProvider, WebSearchResult, WebProviderStatus } from '@deepseek-ai/dsh-web'
|
||||
import * as ToolWeb from '@deepseek-ai/dsh-tool-web'
|
||||
import {
|
||||
formatSearchOutput,
|
||||
formatFetchOutput,
|
||||
parseSearchArgs,
|
||||
parseFetchArgs,
|
||||
presentSearchCall,
|
||||
presentFetchCall,
|
||||
renderBody,
|
||||
htmlToMarkdown,
|
||||
} from '@deepseek-ai/dsh-tool-web'
|
||||
|
||||
const available: WebProviderStatus = { available: true }
|
||||
|
||||
function searchProvider(result: WebSearchResult, status: WebProviderStatus = available): WebSearchProvider {
|
||||
return { id: 'stub-search', status: () => status, search: () => Promise.resolve(result) }
|
||||
}
|
||||
|
||||
/** Mount the real registry, seam, and tool-web; return an executor helper. */
|
||||
async function mountTools(opts: {
|
||||
config?: ToolWeb.Config
|
||||
webConfig?: ConstructorParameters<typeof WebService>[1]
|
||||
search?: WebSearchProvider
|
||||
fetchProvider?: import('@deepseek-ai/dsh-web').WebFetchProvider
|
||||
} = {}): Promise<{ ctx: Context; fiber: Awaited<ReturnType<Context['plugin']>>; call: (name: string, args: unknown) => Promise<{ isError: boolean; content: { type: string; text?: string }[]; error?: { code: string } }> }> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SystemPrompt)
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(WebService, opts.webConfig ?? {})
|
||||
if (opts.search) ctx.web.registerSearchProvider(opts.search)
|
||||
if (opts.fetchProvider) ctx.web.registerFetchProvider(opts.fetchProvider)
|
||||
const fiber = await ctx.plugin(ToolWeb, opts.config ?? {})
|
||||
let counter = 0
|
||||
const call = (name: string, args: unknown) => ctx.tools.execute({ callId: CallId(`call-${++counter}`), name, arguments: args }) as never
|
||||
return { ctx, fiber, call }
|
||||
}
|
||||
|
||||
describe('search formatting', () => {
|
||||
it('renders content, sources with titles/hostnames, snippets, and a citation reminder', () => {
|
||||
const out = formatSearchOutput({
|
||||
providerId: 'p', query: 'q', content: 'an answer', truncated: false,
|
||||
sources: [
|
||||
{ url: 'https://a.test/x', title: 'A', snippet: 'about a', publishedAt: '2026-01-01' },
|
||||
{ url: 'https://b.test/y' },
|
||||
],
|
||||
})
|
||||
expect(out).toContain('an answer')
|
||||
expect(out).toContain('[A](https://a.test/x) — about a (2026-01-01)')
|
||||
expect(out).toContain('[b.test](https://b.test/y)')
|
||||
expect(out).toContain('Cite the relevant URLs')
|
||||
})
|
||||
|
||||
it('reports no results when there is neither content nor sources', () => {
|
||||
expect(formatSearchOutput({ providerId: 'p', query: 'q', sources: [], truncated: false }))
|
||||
.toContain('No results found.')
|
||||
})
|
||||
|
||||
it('renders content alone when there are no sources', () => {
|
||||
const out = formatSearchOutput({ providerId: 'p', query: 'q', content: 'just an answer', sources: [], truncated: false })
|
||||
expect(out).toContain('just an answer')
|
||||
expect(out).not.toContain('No results found.')
|
||||
expect(out).not.toContain('Sources:')
|
||||
})
|
||||
|
||||
it('notes truncation', () => {
|
||||
const out = formatSearchOutput({ providerId: 'p', query: 'q', sources: [{ url: 'https://a.test' }], truncated: true })
|
||||
expect(out).toContain('Showing the first 1 sources')
|
||||
})
|
||||
|
||||
it('validates the query', () => {
|
||||
expect(() => parseSearchArgs({ query: ' ' })).toThrow('non-empty')
|
||||
expect(parseSearchArgs({ query: 'hi' })).toEqual({ query: 'hi' })
|
||||
})
|
||||
|
||||
it('presents a search call as a search-kind card titled by the query', () => {
|
||||
expect(presentSearchCall({ query: 'find me' })).toEqual({ title: 'find me', kind: 'search', rawInput: 'find me' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('fetch formatting', () => {
|
||||
it('renders an html body to markdown text with a status header', () => {
|
||||
const out = formatFetchOutput({
|
||||
providerId: 'p', url: 'https://a.test', statusCode: 200, truncated: false,
|
||||
body: { kind: 'html', content: '<h1>Title</h1><p>Body text</p>' },
|
||||
})
|
||||
expect(out).toContain('Fetched https://a.test (HTTP 200)')
|
||||
expect(out).toContain('# Title')
|
||||
expect(out).toContain('Body text')
|
||||
})
|
||||
|
||||
it('passes a text body through and notes truncation', () => {
|
||||
const out = formatFetchOutput({
|
||||
providerId: 'p', url: 'https://a.test', statusCode: 200, truncated: true,
|
||||
body: { kind: 'text', content: 'plain' },
|
||||
})
|
||||
expect(out).toContain('plain')
|
||||
expect(out).toContain('Content truncated')
|
||||
})
|
||||
|
||||
it('renderBody dispatches on kind', () => {
|
||||
expect(renderBody({ kind: 'text', content: 'x' })).toBe('x')
|
||||
expect(renderBody({ kind: 'html', content: '<p>y</p>' })).toBe('y')
|
||||
})
|
||||
|
||||
it('validates url and timeout', () => {
|
||||
expect(() => parseFetchArgs({ url: ' ' })).toThrow('non-empty')
|
||||
expect(() => parseFetchArgs({ url: 'https://a.test', timeout_ms: -1 })).toThrow('positive')
|
||||
expect(parseFetchArgs({ url: 'https://a.test', timeout_ms: 5 })).toEqual({ url: 'https://a.test', timeoutMs: 5 })
|
||||
})
|
||||
|
||||
it('presents a fetch call as a fetch-kind card titled by the url', () => {
|
||||
expect(presentFetchCall({ url: 'https://a.test' })).toEqual({ title: 'https://a.test', kind: 'fetch', rawInput: 'https://a.test' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('htmlToMarkdown', () => {
|
||||
it('drops scripts/styles, keeps text, decodes entities, converts links', () => {
|
||||
const md = htmlToMarkdown('<style>.x{}</style><script>bad()</script><p>Tom & Jerry</p><a href="https://a.test">link</a>')
|
||||
expect(md).not.toContain('bad()')
|
||||
expect(md).not.toContain('.x{}')
|
||||
expect(md).toContain('Tom & Jerry')
|
||||
expect(md).toContain('[link](https://a.test)')
|
||||
})
|
||||
|
||||
it('decodes numeric entities and collapses whitespace', () => {
|
||||
expect(htmlToMarkdown('<p>a'b</p>')).toBe("a'b")
|
||||
expect(htmlToMarkdown('<div>x</div>\n\n\n<div>y</div>')).toBe('x\n\ny')
|
||||
})
|
||||
|
||||
it('decodes hex entities and named entities, and leaves unknown/out-of-range ones intact', () => {
|
||||
expect(htmlToMarkdown('<p>AB</p>')).toBe('AB')
|
||||
expect(htmlToMarkdown('<p>© —</p>')).toBe('© —')
|
||||
expect(htmlToMarkdown('<p>¬areal;</p>')).toBe('¬areal;')
|
||||
// An out-of-range code point keeps the original entity text (fromCodePoint fallback).
|
||||
expect(htmlToMarkdown('<p>�</p>')).toBe('�')
|
||||
expect(htmlToMarkdown('<p>�</p>')).toBe('�')
|
||||
})
|
||||
|
||||
it('renders a link with an empty label as its bare href', () => {
|
||||
expect(htmlToMarkdown('<a href="https://a.test"></a>')).toBe('https://a.test')
|
||||
})
|
||||
|
||||
it('converts headings and list items to markdown', () => {
|
||||
expect(htmlToMarkdown('<h2>Heading</h2><p>after</p>')).toContain('## Heading')
|
||||
const list = htmlToMarkdown('<ul><li>one</li><li>two</li></ul>')
|
||||
expect(list).toContain('- one')
|
||||
expect(list).toContain('- two')
|
||||
})
|
||||
|
||||
it('falls back to the raw URL as a source label when the URL is unparseable', () => {
|
||||
const out = formatSearchOutput({ providerId: 'p', query: 'q', truncated: false, sources: [{ url: 'not a url' }] })
|
||||
expect(out).toContain('[not a url](not a url)')
|
||||
})
|
||||
})
|
||||
|
||||
describe('tool-web registration', () => {
|
||||
it('registers both tools by default', async () => {
|
||||
const { fiber, ctx } = await mountTools()
|
||||
const names = ctx.tools.schemas().map(s => s.name)
|
||||
expect(names).toContain('web_search')
|
||||
expect(names).toContain('web_fetch')
|
||||
await fiber.dispose()
|
||||
expect(ctx.tools.schemas().map(s => s.name)).not.toContain('web_search')
|
||||
})
|
||||
|
||||
it('registers only enabled tools', async () => {
|
||||
const { fiber, ctx } = await mountTools({ config: { search: true, fetch: false } })
|
||||
const names = ctx.tools.schemas().map(s => s.name)
|
||||
expect(names).toContain('web_search')
|
||||
expect(names).not.toContain('web_fetch')
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('registers only web_fetch when search is disabled', async () => {
|
||||
const { fiber, ctx } = await mountTools({ config: { search: false, fetch: true } })
|
||||
const names = ctx.tools.schemas().map(s => s.name)
|
||||
expect(names).not.toContain('web_search')
|
||||
expect(names).toContain('web_fetch')
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('registers web_search even when no provider is available (schema follows enablement, not availability)', async () => {
|
||||
const { fiber, ctx } = await mountTools()
|
||||
expect(ctx.tools.schemas().map(s => s.name)).toContain('web_search')
|
||||
expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('contributes prompt sections for the enabled tools', async () => {
|
||||
const { fiber, ctx } = await mountTools()
|
||||
const prompt = await ctx.systemPrompt.assemble()
|
||||
const text = prompt.sections.map(s => (typeof s.text === 'function' ? s.text() : s.text)).join('\n')
|
||||
expect(text).toContain('web_search')
|
||||
expect(text).toContain('web_fetch')
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
|
||||
describe('tool-web execution through the real registry', () => {
|
||||
it('executes web_search and formats the result', async () => {
|
||||
const result: WebSearchResult = {
|
||||
providerId: 'stub-search', query: 'q', content: 'answer', truncated: false,
|
||||
sources: [{ url: 'https://a.test', title: 'A', snippet: 'snip' }],
|
||||
}
|
||||
const { fiber, call } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: searchProvider(result) })
|
||||
const out = await call('web_search', { query: 'q' })
|
||||
expect(out.isError).toBe(false)
|
||||
expect(out.content.map(b => b.text).join('')).toContain('[A](https://a.test)')
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('surfaces a structured WebError when no provider is available', async () => {
|
||||
const { fiber, call } = await mountTools()
|
||||
const out = await call('web_search', { query: 'q' })
|
||||
expect(out.isError).toBe(true)
|
||||
expect(out.error?.code).toBe('WEB_PROVIDER_UNAVAILABLE')
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('surfaces WEB_PROVIDER_AMBIGUOUS for multiple unconfigured providers', async () => {
|
||||
const { ctx, fiber, call } = await mountTools({ search: searchProvider({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) })
|
||||
ctx.web.registerSearchProvider({ id: 'other', status: () => available, search: () => Promise.resolve({ providerId: 'other', query: 'q', sources: [], truncated: false }) })
|
||||
const out = await call('web_search', { query: 'q' })
|
||||
expect(out.isError).toBe(true)
|
||||
expect(out.error?.code).toBe('WEB_PROVIDER_AMBIGUOUS')
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('rejects invalid arguments with a structured INVALID_ARGS error', async () => {
|
||||
const { fiber, call } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: searchProvider({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) })
|
||||
const out = await call('web_search', { query: 123 })
|
||||
expect(out.isError).toBe(true)
|
||||
expect(out.error?.code).toBe('INVALID_ARGS')
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('has no default export (namespace plugin export shape)', () => {
|
||||
expect('default' in ToolWeb).toBe(false)
|
||||
})
|
||||
|
||||
it('executes web_fetch, forwarding timeout_ms and the abort signal to the seam', async () => {
|
||||
const seen: { request?: { url: string; timeoutMs?: number }; signal?: AbortSignal | undefined } = {}
|
||||
const fetchProvider = {
|
||||
id: 'stub-fetch',
|
||||
status: () => available,
|
||||
fetch: (request: { url: string; timeoutMs?: number }, exec?: { signal?: AbortSignal }) => {
|
||||
seen.request = request
|
||||
seen.signal = exec?.signal
|
||||
return Promise.resolve({ providerId: 'stub-fetch', url: request.url, statusCode: 200, body: { kind: 'text' as const, content: 'ok' }, truncated: false })
|
||||
},
|
||||
}
|
||||
const { ctx, fiber } = await mountTools({ webConfig: { fetchProvider: 'stub-fetch' }, fetchProvider })
|
||||
const controller = new AbortController()
|
||||
const out = await ctx.tools.execute({ callId: CallId('fetch-1'), name: 'web_fetch', arguments: { url: 'https://a.test', timeout_ms: 1234 }, signal: controller.signal })
|
||||
expect(out.isError).toBe(false)
|
||||
expect(seen.request).toEqual({ url: 'https://a.test', timeoutMs: 1234 })
|
||||
expect(seen.signal).toBe(controller.signal)
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('executes web_search, forwarding the abort signal to the seam', async () => {
|
||||
const seen: { signal?: AbortSignal | undefined } = {}
|
||||
const provider: WebSearchProvider = {
|
||||
id: 'stub-search',
|
||||
status: () => available,
|
||||
search: (_request, exec) => { seen.signal = exec?.signal; return Promise.resolve({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) },
|
||||
}
|
||||
const { ctx, fiber } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: provider })
|
||||
const controller = new AbortController()
|
||||
await ctx.tools.execute({ callId: CallId('search-1'), name: 'web_search', arguments: { query: 'q' }, signal: controller.signal })
|
||||
expect(seen.signal).toBe(controller.signal)
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": ["src"],
|
||||
"references": [
|
||||
{ "path": "../../../vendor/cosmokit" },
|
||||
{ "path": "../../../vendor/cordis" },
|
||||
{ "path": "../../../vendor/schemastery" },
|
||||
{ "path": "../../llm/llm" },
|
||||
{ "path": "../../core/tools" },
|
||||
{ "path": "../../core/system-prompt" },
|
||||
{ "path": "../web" }
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
import { defineConfig } from 'tsdown'
|
||||
|
||||
/**
|
||||
* tool-web exposes one package root plus one entry per tool plugin, so each tool
|
||||
* can be loaded or replaced independently as a subpath plugin
|
||||
* (`@deepseek-ai/dsh-tool-web/search`, `/fetch`). The root tsdown config only
|
||||
* auto-discovers `src/index.ts`, so the subpath entries are declared here.
|
||||
*/
|
||||
export default defineConfig({
|
||||
entry: ['src/index.ts', 'src/search.ts', 'src/fetch.ts'],
|
||||
outDir: 'lib',
|
||||
format: ['esm'],
|
||||
platform: 'node',
|
||||
target: 'es2024',
|
||||
fixedExtension: false,
|
||||
dts: false,
|
||||
clean: false,
|
||||
})
|
||||
@@ -0,0 +1,34 @@
|
||||
# @deepseek-ai/dsh-web-fetch-local
|
||||
|
||||
An anonymous public HTTP(S) `WebFetchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It retrieves a concrete URL and returns a status code plus bounded decoded content.
|
||||
|
||||
This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the key and it does not register a model-facing tool. It is a function/namespace plugin (`inject: ['web']`).
|
||||
|
||||
## Responsibility split
|
||||
|
||||
The provider owns **safe resource retrieval**: URL validation, HTTP transport, redirect policy, timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. `@deepseek-ai/dsh-tool-web` owns **presentation** (HTML→markdown, truncation formatting). A non-2xx HTTP response is a *result* (status code + decoded body), not an error; `WebError` is reserved for failures to safely retrieve or represent the resource.
|
||||
|
||||
## Transport hygiene
|
||||
|
||||
- Accepts only `http:` and `https:` URLs; rejects credentials in URLs (`WEB_BLOCKED_URL`) and over-long/malformed URLs (`WEB_INVALID_URL`).
|
||||
- Enforces a max URL length, response byte cap (`WEB_FETCH_TOO_LARGE`), decoded body character cap, timeout (`WEB_FETCH_TIMEOUT`), and redirect hop cap.
|
||||
- Propagates the caller's abort signal (`WEB_ABORTED`) into the network request and the streaming read.
|
||||
- Follows only **same-origin** redirects; a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (the model of Claude Code's WebFetch).
|
||||
- Sends an explicit product `User-Agent`, never a browser disguise.
|
||||
- Rejects unsupported (e.g. binary) content types with `WEB_UNSUPPORTED_CONTENT_TYPE`.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `maxUrlLength` | `2048` | Maximum accepted request URL length. |
|
||||
| `maxResponseBytes` | `5_000_000` | Maximum response body size in bytes. |
|
||||
| `maxBodyChars` | `100_000` | Maximum decoded body length in characters. |
|
||||
| `timeoutMs` | `30_000` | Default fetch timeout. |
|
||||
| `maxTimeoutMs` | `120_000` | Upper bound for a per-request timeout override. |
|
||||
| `maxRedirects` | `5` | Maximum same-origin redirect hops. |
|
||||
| `userAgent` | `deepseek-harness/…` | `User-Agent` header. |
|
||||
|
||||
## Security note
|
||||
|
||||
SSRF / private-network protection (blocking private, loopback, link-local, multicast, and otherwise non-public destinations, with DNS-resolve-then-validate and per-hop re-validation) is **deferred** — see the [web capability seam RFC](../../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md). Until it lands, this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets.
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-web-fetch-local",
|
||||
"description": "Anonymous public HTTP(S) fetch provider for the DeepSeek Harness web capability seam (ctx.web)",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-web": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-web": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
/**
|
||||
* `@deepseek-ai/dsh-web-fetch-local`: registers an anonymous public HTTP(S)
|
||||
* `WebFetchProvider` with `ctx.web`. A function/namespace plugin (NOT a
|
||||
* default-export service): it registers INTO the seam's fetch registry, like the
|
||||
* search providers register into the search registry.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-fetch-local
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {} from '@deepseek-ai/dsh-web'
|
||||
import { LocalFetchProvider } from './provider.ts'
|
||||
import type { LocalFetchLimits } from './provider.ts'
|
||||
|
||||
export {
|
||||
LOCAL_FETCH_PROVIDER_ID,
|
||||
LocalFetchProvider,
|
||||
} from './provider.ts'
|
||||
export type { LocalFetchLimits } from './provider.ts'
|
||||
export { classifyContentType, isSameOrigin, validateFetchUrl } from './policy.ts'
|
||||
export type { FetchableKind } from './policy.ts'
|
||||
|
||||
/** Default `User-Agent`: an explicit product agent, never a browser disguise. */
|
||||
export const DEFAULT_USER_AGENT = 'deepseek-harness/0.0.1 (+https://github.com/deepseek-ai)'
|
||||
|
||||
/** Cordis plugin name used by loader diagnostics. */
|
||||
export const name = 'web-fetch-local'
|
||||
|
||||
/** The web seam this provider registers into. */
|
||||
export const inject = ['web']
|
||||
|
||||
export interface Config {
|
||||
/** Maximum accepted request URL length. */
|
||||
maxUrlLength?: number
|
||||
/** Maximum response body size in bytes. */
|
||||
maxResponseBytes?: number
|
||||
/** Maximum decoded body length in characters. */
|
||||
maxBodyChars?: number
|
||||
/** Default fetch timeout in milliseconds. */
|
||||
timeoutMs?: number
|
||||
/** Upper bound for a per-request timeout override. */
|
||||
maxTimeoutMs?: number
|
||||
/** Maximum number of same-origin redirect hops to follow. */
|
||||
maxRedirects?: number
|
||||
/** `User-Agent` header sent on every request. */
|
||||
userAgent?: string
|
||||
}
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
maxUrlLength: z.number().default(2048),
|
||||
maxResponseBytes: z.number().default(5_000_000),
|
||||
maxBodyChars: z.number().default(100_000),
|
||||
timeoutMs: z.number().default(30_000),
|
||||
maxTimeoutMs: z.number().default(120_000),
|
||||
maxRedirects: z.number().default(5),
|
||||
userAgent: z.string().default(DEFAULT_USER_AGENT),
|
||||
})
|
||||
|
||||
/** The shape after schemastery applies its defaults to every field. */
|
||||
type ResolvedConfig = Required<Config>
|
||||
|
||||
/** Register the local HTTP(S) fetch provider with `ctx.web`. */
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
// schemastery (Config) has already filled every defaulted field.
|
||||
const resolved = config as ResolvedConfig
|
||||
const limits: LocalFetchLimits = {
|
||||
maxUrlLength: resolved.maxUrlLength,
|
||||
maxResponseBytes: resolved.maxResponseBytes,
|
||||
maxBodyChars: resolved.maxBodyChars,
|
||||
timeoutMs: resolved.timeoutMs,
|
||||
maxTimeoutMs: resolved.maxTimeoutMs,
|
||||
maxRedirects: resolved.maxRedirects,
|
||||
userAgent: resolved.userAgent,
|
||||
}
|
||||
ctx.web.registerFetchProvider(new LocalFetchProvider(limits))
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
/**
|
||||
* URL validation and content-type classification for the local HTTP(S) fetch
|
||||
* provider — the pure, network-free half. The provider's `fetch()` composes
|
||||
* these with transport (redirect following, byte caps, decoding).
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-fetch-local/policy
|
||||
*/
|
||||
|
||||
import { WebError } from '@deepseek-ai/dsh-web'
|
||||
|
||||
/** The body kinds this provider decodes. */
|
||||
export type FetchableKind = 'html' | 'text'
|
||||
|
||||
/**
|
||||
* Validate a request URL against the basic transport hygiene the provider
|
||||
* enforces before any network access: http(s) only, no embedded credentials,
|
||||
* bounded length. Returns the parsed `URL`. Throws {@link WebError} otherwise.
|
||||
* (SSRF / private-network blocking is deferred — see the package RFC.)
|
||||
*/
|
||||
export function validateFetchUrl(input: string, maxUrlLength: number): URL {
|
||||
if (input.length > maxUrlLength) {
|
||||
throw new WebError(`URL exceeds the maximum length of ${maxUrlLength}`, 'WEB_INVALID_URL')
|
||||
}
|
||||
let url: URL
|
||||
try {
|
||||
url = new URL(input)
|
||||
} catch (error: unknown) {
|
||||
throw new WebError(`invalid URL: ${input}`, 'WEB_INVALID_URL', { cause: error })
|
||||
}
|
||||
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
|
||||
throw new WebError(`unsupported URL scheme "${url.protocol}" (only http and https are allowed)`, 'WEB_INVALID_URL')
|
||||
}
|
||||
if (url.username.length > 0 || url.password.length > 0) {
|
||||
throw new WebError('credentials in URLs are not allowed', 'WEB_BLOCKED_URL')
|
||||
}
|
||||
return url
|
||||
}
|
||||
|
||||
/**
|
||||
* Two URLs are same-origin when scheme, hostname, and port match. A redirect
|
||||
* that crosses origins is refused so each new origin requires a fresh tool call
|
||||
* (and thus a fresh provider/permission decision).
|
||||
*/
|
||||
export function isSameOrigin(a: URL, b: URL): boolean {
|
||||
return a.protocol === b.protocol && a.hostname === b.hostname && a.port === b.port
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a response `Content-Type` into a decodable body kind, or `undefined`
|
||||
* for an unsupported (e.g. binary) type. `text/html` and `application/xhtml+xml`
|
||||
* are `html`; other `text/*` plus a few structured text types are `text`.
|
||||
*/
|
||||
export function classifyContentType(contentType: string | null): FetchableKind | undefined {
|
||||
const mime = (contentType ?? '').replace(/;.*$/s, '').trim().toLowerCase()
|
||||
if (mime === 'text/html' || mime === 'application/xhtml+xml') return 'html'
|
||||
if (mime.startsWith('text/')) return 'text'
|
||||
if (mime === 'application/json' || mime === 'application/xml' || mime.endsWith('+json') || mime.endsWith('+xml')) return 'text'
|
||||
return undefined
|
||||
}
|
||||
@@ -0,0 +1,233 @@
|
||||
/**
|
||||
* `LocalFetchProvider`: a `WebFetchProvider` that retrieves a concrete public
|
||||
* HTTP(S) URL with the platform-native `fetch` (Node 24) and returns a status
|
||||
* code plus bounded decoded content. It owns SAFE RESOURCE RETRIEVAL — URL
|
||||
* validation, redirect policy, timeout, abort, byte caps, charset decoding,
|
||||
* content-type classification, binary rejection — but NOT presentation
|
||||
* (HTML→markdown lives in `@deepseek-ai/dsh-tool-web`).
|
||||
*
|
||||
* Redirects are followed manually (`redirect: 'manual'`) so the provider can
|
||||
* enforce a same-origin-only policy: a cross-origin redirect is refused with
|
||||
* `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (Claude Code's WebFetch
|
||||
* uses the same model). It does NOT carry browser cookies, editor/git
|
||||
* credentials, or implicit access to private services.
|
||||
*
|
||||
* SSRF / private-network protection is DEFERRED (see the package RFC); until it
|
||||
* lands this provider is an SSRF primitive and must not be enabled where it can
|
||||
* reach sensitive internal targets.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-fetch-local/provider
|
||||
*/
|
||||
|
||||
import { WebError } from '@deepseek-ai/dsh-web'
|
||||
import type { WebFetchBody, WebFetchProvider, WebFetchRequest, WebFetchResult, WebProviderStatus } from '@deepseek-ai/dsh-web'
|
||||
import { classifyContentType, isSameOrigin, validateFetchUrl } from './policy.ts'
|
||||
|
||||
/** Resolved provider limits (the plugin's schemastery Config supplies defaults). */
|
||||
export interface LocalFetchLimits {
|
||||
/** Maximum accepted request URL length. */
|
||||
maxUrlLength: number
|
||||
/** Maximum response body size in bytes (read is aborted past this). */
|
||||
maxResponseBytes: number
|
||||
/** Maximum decoded body length in characters (truncated past this). */
|
||||
maxBodyChars: number
|
||||
/** Default fetch timeout in milliseconds. */
|
||||
timeoutMs: number
|
||||
/** Upper bound for a per-request timeout override. */
|
||||
maxTimeoutMs: number
|
||||
/** Maximum number of (same-origin) redirect hops to follow. */
|
||||
maxRedirects: number
|
||||
/** `User-Agent` header sent on every request. */
|
||||
userAgent: string
|
||||
}
|
||||
|
||||
/** Stable id this provider registers under. */
|
||||
export const LOCAL_FETCH_PROVIDER_ID = 'local-http'
|
||||
|
||||
/** The anonymous public HTTP(S) fetch provider. */
|
||||
export class LocalFetchProvider implements WebFetchProvider {
|
||||
readonly id = LOCAL_FETCH_PROVIDER_ID
|
||||
|
||||
constructor(private readonly limits: LocalFetchLimits) {}
|
||||
|
||||
/** No credentials to check — an anonymous public fetcher is always usable. */
|
||||
status(): WebProviderStatus {
|
||||
return { available: true }
|
||||
}
|
||||
|
||||
async fetch(request: WebFetchRequest, exec?: { readonly signal?: AbortSignal }): Promise<WebFetchResult> {
|
||||
const timeoutMs = request.timeoutMs !== undefined
|
||||
? Math.min(request.timeoutMs, this.limits.maxTimeoutMs)
|
||||
: this.limits.timeoutMs
|
||||
|
||||
// One controller drives both the caller's abort and our own timeout, so the
|
||||
// network request and the streaming read both stop on either.
|
||||
const controller = new AbortController()
|
||||
const onAbort = (): void => { controller.abort() }
|
||||
if (exec?.signal !== undefined) {
|
||||
if (exec.signal.aborted) throw new WebError('web fetch aborted', 'WEB_ABORTED')
|
||||
exec.signal.addEventListener('abort', onAbort, { once: true })
|
||||
}
|
||||
const timer = setTimeout(() => { controller.abort(new WebError('web fetch timed out', 'WEB_FETCH_TIMEOUT')) }, timeoutMs)
|
||||
|
||||
try {
|
||||
return await this.followAndRead(request.url, controller, timeoutMs)
|
||||
} finally {
|
||||
clearTimeout(timer)
|
||||
if (exec?.signal !== undefined) exec.signal.removeEventListener('abort', onAbort)
|
||||
}
|
||||
}
|
||||
|
||||
/** Follow same-origin redirects up to the hop cap, then read the final response. */
|
||||
private async followAndRead(initialUrl: string, controller: AbortController, timeoutMs: number): Promise<WebFetchResult> {
|
||||
let currentUrl = validateFetchUrl(initialUrl, this.limits.maxUrlLength)
|
||||
|
||||
for (let hop = 0; hop <= this.limits.maxRedirects; hop++) {
|
||||
const response = await this.requestOnce(currentUrl, controller, timeoutMs)
|
||||
|
||||
if (isRedirectStatus(response.status)) {
|
||||
const location = response.headers.get('location')
|
||||
if (location === null) {
|
||||
// A redirect status with no Location is not a usable resource.
|
||||
throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, 'WEB_PROVIDER_ERROR')
|
||||
}
|
||||
const target = resolveRedirect(location, currentUrl)
|
||||
if (!isSameOrigin(target, currentUrl)) {
|
||||
throw new WebError(
|
||||
`cross-origin redirect to ${target.origin} is not followed automatically; retry against that URL directly`,
|
||||
'WEB_REDIRECT_BLOCKED',
|
||||
)
|
||||
}
|
||||
await response.body?.cancel()
|
||||
currentUrl = target
|
||||
continue
|
||||
}
|
||||
|
||||
return await this.readBody(response, currentUrl)
|
||||
}
|
||||
|
||||
throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, 'WEB_REDIRECT_BLOCKED')
|
||||
}
|
||||
|
||||
private async requestOnce(url: URL, controller: AbortController, _timeoutMs: number): Promise<Response> {
|
||||
try {
|
||||
return await fetch(url, {
|
||||
method: 'GET',
|
||||
redirect: 'manual',
|
||||
headers: { 'user-agent': this.limits.userAgent, 'accept': 'text/html,application/xhtml+xml,text/*;q=0.9,application/json;q=0.8' },
|
||||
signal: controller.signal,
|
||||
})
|
||||
} catch (error: unknown) {
|
||||
throw translateAbortOrNetwork(error)
|
||||
}
|
||||
}
|
||||
|
||||
/** Read, byte-cap, classify, and decode the final response body. */
|
||||
private async readBody(response: Response, finalUrl: URL): Promise<WebFetchResult> {
|
||||
const kind = classifyContentType(response.headers.get('content-type'))
|
||||
if (kind === undefined) {
|
||||
await response.body?.cancel()
|
||||
throw new WebError(`unsupported content type "${response.headers.get('content-type') ?? 'unknown'}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE')
|
||||
}
|
||||
|
||||
const { bytes, truncatedByBytes } = await this.readCapped(response)
|
||||
const decoded = new TextDecoder('utf-8').decode(bytes)
|
||||
const truncatedByChars = decoded.length > this.limits.maxBodyChars
|
||||
const content = truncatedByChars ? decoded.slice(0, this.limits.maxBodyChars) : decoded
|
||||
const body: WebFetchBody = kind === 'html' ? { kind: 'html', content } : { kind: 'text', content }
|
||||
|
||||
return {
|
||||
providerId: this.id,
|
||||
url: finalUrl.toString(),
|
||||
statusCode: response.status,
|
||||
body,
|
||||
truncated: truncatedByBytes || truncatedByChars,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the response stream up to `maxResponseBytes`. A `Content-Length` over
|
||||
* the cap rejects immediately with `WEB_FETCH_TOO_LARGE`; a stream that grows
|
||||
* past the cap is cut short (`truncatedByBytes`) rather than rejected, so a
|
||||
* server that under-reports still yields a bounded usable body.
|
||||
*/
|
||||
private async readCapped(response: Response): Promise<{ bytes: Uint8Array; truncatedByBytes: boolean }> {
|
||||
const declared = response.headers.get('content-length')
|
||||
if (declared !== null) {
|
||||
const length = Number(declared)
|
||||
if (Number.isFinite(length) && length > this.limits.maxResponseBytes) {
|
||||
await response.body?.cancel()
|
||||
throw new WebError(`response exceeds the maximum of ${this.limits.maxResponseBytes} bytes`, 'WEB_FETCH_TOO_LARGE')
|
||||
}
|
||||
}
|
||||
|
||||
/* v8 ignore next -- a 2xx Response from fetch always exposes a body stream; the null guard is defensive. */
|
||||
if (response.body === null) return { bytes: new Uint8Array(0), truncatedByBytes: false }
|
||||
|
||||
const chunks: Uint8Array[] = []
|
||||
let total = 0
|
||||
let truncatedByBytes = false
|
||||
const reader = response.body.getReader()
|
||||
try {
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read()
|
||||
if (done) break
|
||||
const remaining = this.limits.maxResponseBytes - total
|
||||
if (value.byteLength >= remaining) {
|
||||
chunks.push(value.subarray(0, remaining))
|
||||
total += remaining
|
||||
truncatedByBytes = true
|
||||
break
|
||||
}
|
||||
chunks.push(value)
|
||||
total += value.byteLength
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore next -- mid-stream read fault needs a network drop after headers; translate path covered by request-phase tests. */
|
||||
throw translateAbortOrNetwork(error)
|
||||
} finally {
|
||||
/* v8 ignore next 4 -- cancel() after a completed/broken read settles without rejecting; unobserved best-effort cleanup. */
|
||||
await reader.cancel().catch(() => {
|
||||
// Cancel after a successful read (or after we broke past the cap) is
|
||||
// best-effort cleanup; the bytes we need are already collected.
|
||||
})
|
||||
}
|
||||
|
||||
const bytes = new Uint8Array(total)
|
||||
let offset = 0
|
||||
for (const chunk of chunks) {
|
||||
bytes.set(chunk, offset)
|
||||
offset += chunk.byteLength
|
||||
}
|
||||
return { bytes, truncatedByBytes }
|
||||
}
|
||||
}
|
||||
|
||||
/** HTTP redirect status codes that carry a `Location`. */
|
||||
function isRedirectStatus(status: number): boolean {
|
||||
return status === 301 || status === 302 || status === 303 || status === 307 || status === 308
|
||||
}
|
||||
|
||||
/** Resolve a (possibly relative) `Location` against the current URL. */
|
||||
function resolveRedirect(location: string, base: URL): URL {
|
||||
try {
|
||||
return new URL(location, base)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore next 2 -- URL resolution against a valid absolute base effectively never throws; defensive guard. */
|
||||
throw new WebError(`invalid redirect Location "${location}"`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate a thrown fetch/stream error into a `WebError`. Our own
|
||||
* `WEB_FETCH_TIMEOUT` (passed to `controller.abort(reason)`) and any other
|
||||
* already-typed `WebError` pass through; an `AbortError` becomes `WEB_ABORTED`;
|
||||
* anything else is a transport/network failure (`WEB_PROVIDER_ERROR`).
|
||||
*/
|
||||
function translateAbortOrNetwork(error: unknown): WebError {
|
||||
if (error instanceof WebError) return error
|
||||
if (error instanceof DOMException && error.name === 'AbortError') {
|
||||
return new WebError('web fetch aborted', 'WEB_ABORTED', { cause: error })
|
||||
}
|
||||
return new WebError(`web fetch failed: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
|
||||
import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'
|
||||
import { AddressInfo } from 'node:net'
|
||||
import { Context } from 'cordis'
|
||||
import WebService from '@deepseek-ai/dsh-web'
|
||||
import { LocalFetchProvider, LOCAL_FETCH_PROVIDER_ID, classifyContentType, isSameOrigin, validateFetchUrl } from '@deepseek-ai/dsh-web-fetch-local'
|
||||
import type { LocalFetchLimits } from '@deepseek-ai/dsh-web-fetch-local'
|
||||
import * as fetchPlugin from '@deepseek-ai/dsh-web-fetch-local'
|
||||
|
||||
const limits: LocalFetchLimits = {
|
||||
maxUrlLength: 2048,
|
||||
maxResponseBytes: 5_000_000,
|
||||
maxBodyChars: 100_000,
|
||||
timeoutMs: 5_000,
|
||||
maxTimeoutMs: 10_000,
|
||||
maxRedirects: 5,
|
||||
userAgent: 'test-agent/1.0',
|
||||
}
|
||||
|
||||
type Handler = (req: IncomingMessage, res: ServerResponse) => void
|
||||
|
||||
let server: Server
|
||||
let base: string
|
||||
let handler: Handler
|
||||
|
||||
beforeEach(async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('default') }
|
||||
server = createServer((req, res) => { handler(req, res) })
|
||||
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
|
||||
const { port } = server.address() as AddressInfo
|
||||
base = `http://127.0.0.1:${port}`
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await new Promise<void>(resolve => server.close(() => { resolve() }))
|
||||
})
|
||||
|
||||
function provider(overrides: Partial<LocalFetchLimits> = {}): LocalFetchProvider {
|
||||
return new LocalFetchProvider({ ...limits, ...overrides })
|
||||
}
|
||||
|
||||
describe('policy helpers', () => {
|
||||
it('validates scheme, credentials, and length', () => {
|
||||
expect(validateFetchUrl('https://example.com/x', 2048).hostname).toBe('example.com')
|
||||
expect(() => validateFetchUrl('ftp://example.com', 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
expect(() => validateFetchUrl('not a url', 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
expect(() => validateFetchUrl('https://user:pass@example.com', 2048)).toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
|
||||
expect(() => validateFetchUrl(`https://example.com/${'a'.repeat(3000)}`, 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
})
|
||||
|
||||
it('classifies content types', () => {
|
||||
expect(classifyContentType('text/html; charset=utf-8')).toBe('html')
|
||||
expect(classifyContentType('application/xhtml+xml')).toBe('html')
|
||||
expect(classifyContentType('text/plain')).toBe('text')
|
||||
expect(classifyContentType('application/json')).toBe('text')
|
||||
expect(classifyContentType('image/png')).toBeUndefined()
|
||||
expect(classifyContentType(null)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('compares origins', () => {
|
||||
expect(isSameOrigin(new URL('https://a.com/x'), new URL('https://a.com/y'))).toBe(true)
|
||||
expect(isSameOrigin(new URL('https://a.com'), new URL('https://b.com'))).toBe(false)
|
||||
expect(isSameOrigin(new URL('http://a.com'), new URL('https://a.com'))).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('LocalFetchProvider success', () => {
|
||||
it('fetches a text body', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('hello world') }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.providerId).toBe(LOCAL_FETCH_PROVIDER_ID)
|
||||
expect(result.statusCode).toBe(200)
|
||||
expect(result.body).toEqual({ kind: 'text', content: 'hello world' })
|
||||
expect(result.truncated).toBe(false)
|
||||
})
|
||||
|
||||
it('fetches an html body and classifies it as html', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/html' }); res.end('<h1>hi</h1>') }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.body).toEqual({ kind: 'html', content: '<h1>hi</h1>' })
|
||||
})
|
||||
|
||||
it('sends the configured user agent', async () => {
|
||||
let seen: string | undefined
|
||||
handler = (req, res) => { seen = req.headers['user-agent']; res.writeHead(200, { 'content-type': 'text/plain' }); res.end('ok') }
|
||||
await provider().fetch({ url: base })
|
||||
expect(seen).toBe('test-agent/1.0')
|
||||
})
|
||||
|
||||
it('returns a non-2xx response as a result, not an error', async () => {
|
||||
handler = (_req, res) => { res.writeHead(404, { 'content-type': 'text/plain' }); res.end('nope') }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.statusCode).toBe(404)
|
||||
expect(result.body).toEqual({ kind: 'text', content: 'nope' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('LocalFetchProvider caps', () => {
|
||||
it('rejects an over-cap Content-Length with WEB_FETCH_TOO_LARGE', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain', 'content-length': '999999' }); res.end('x'.repeat(999999)) }
|
||||
await expect(provider({ maxResponseBytes: 10 }).fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_FETCH_TOO_LARGE' }))
|
||||
})
|
||||
|
||||
it('truncates a stream that grows past the byte cap', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('abcdefghij') }
|
||||
const result = await provider({ maxResponseBytes: 4 }).fetch({ url: base })
|
||||
expect(result.body.content).toBe('abcd')
|
||||
expect(result.truncated).toBe(true)
|
||||
})
|
||||
|
||||
it('truncates a decoded body past the character cap', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('abcdefghij') }
|
||||
const result = await provider({ maxBodyChars: 3 }).fetch({ url: base })
|
||||
expect(result.body.content).toBe('abc')
|
||||
expect(result.truncated).toBe(true)
|
||||
})
|
||||
|
||||
it('rejects an unsupported content type', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'image/png' }); res.end('binary') }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
})
|
||||
|
||||
it('rejects a response with no content type at all', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200); res.end('no type') }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
})
|
||||
|
||||
it('accepts a declared content-length within the cap', async () => {
|
||||
handler = (_req, res) => { const body = 'sized'; res.writeHead(200, { 'content-type': 'text/plain', 'content-length': String(body.length) }); res.end(body) }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.body.content).toBe('sized')
|
||||
})
|
||||
})
|
||||
|
||||
describe('LocalFetchProvider redirects', () => {
|
||||
it('follows a same-origin redirect and reports the final URL', async () => {
|
||||
handler = (req, res) => {
|
||||
if (req.url === '/start') { res.writeHead(302, { location: '/end' }); res.end() }
|
||||
else { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('arrived') }
|
||||
}
|
||||
const result = await provider().fetch({ url: `${base}/start` })
|
||||
expect(result.body.content).toBe('arrived')
|
||||
expect(result.url).toBe(`${base}/end`)
|
||||
})
|
||||
|
||||
it('blocks a cross-origin redirect with WEB_REDIRECT_BLOCKED', async () => {
|
||||
handler = (_req, res) => { res.writeHead(302, { location: 'https://example.com/' }); res.end() }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' }))
|
||||
})
|
||||
|
||||
it('rejects exceeding the redirect hop cap', async () => {
|
||||
handler = (req, res) => {
|
||||
const n = Number(new URL(req.url ?? '/', base).searchParams.get('n') ?? '0')
|
||||
res.writeHead(302, { location: `/?n=${n + 1}` })
|
||||
res.end()
|
||||
}
|
||||
await expect(provider({ maxRedirects: 2 }).fetch({ url: `${base}/?n=0` }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' }))
|
||||
})
|
||||
|
||||
it('treats a redirect without a Location header as a provider error', async () => {
|
||||
handler = (_req, res) => { res.writeHead(302); res.end() }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
|
||||
it('follows a relative same-origin redirect', async () => {
|
||||
handler = (req, res) => {
|
||||
if (req.url === '/a') { res.writeHead(301, { location: 'b' }); res.end() }
|
||||
else { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('landed') }
|
||||
}
|
||||
const result = await provider().fetch({ url: `${base}/a` })
|
||||
expect(result.body.content).toBe('landed')
|
||||
})
|
||||
})
|
||||
|
||||
describe('LocalFetchProvider invalid URLs and abort', () => {
|
||||
it('rejects a non-http scheme before any network access', async () => {
|
||||
await expect(provider().fetch({ url: 'ftp://example.com' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
})
|
||||
|
||||
it('rejects credentials in the URL', async () => {
|
||||
await expect(provider().fetch({ url: 'http://user:pass@127.0.0.1/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
|
||||
})
|
||||
|
||||
it('honors a pre-aborted signal', async () => {
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
await expect(provider().fetch({ url: base }, { signal: controller.signal }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' }))
|
||||
})
|
||||
|
||||
it('aborts an in-flight fetch via the signal', async () => {
|
||||
handler = (_req, _res) => { /* never responds */ }
|
||||
const controller = new AbortController()
|
||||
const promise = provider().fetch({ url: base }, { signal: controller.signal })
|
||||
controller.abort()
|
||||
await expect(promise).rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' }))
|
||||
})
|
||||
|
||||
it('times out a slow response with WEB_FETCH_TIMEOUT', async () => {
|
||||
handler = (_req, _res) => { /* never responds */ }
|
||||
await expect(provider({ timeoutMs: 50 }).fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_FETCH_TIMEOUT' }))
|
||||
})
|
||||
|
||||
it('maps a connection failure to WEB_PROVIDER_ERROR', async () => {
|
||||
// Port 1 on loopback is not listening: a real connection failure (not abort).
|
||||
await expect(provider().fetch({ url: 'http://127.0.0.1:1/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
|
||||
it('caps the per-request timeout at maxTimeoutMs', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('ok') }
|
||||
const result = await provider({ maxTimeoutMs: 10_000 }).fetch({ url: base, timeoutMs: 999_999 })
|
||||
expect(result.statusCode).toBe(200)
|
||||
})
|
||||
})
|
||||
|
||||
describe('web-fetch-local plugin registration', () => {
|
||||
it('registers the provider into ctx.web (HMR-safe)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, { fetchProvider: LOCAL_FETCH_PROVIDER_ID })
|
||||
const fiber = await ctx.plugin(fetchPlugin, {})
|
||||
expect(ctx.web.fetchStatus()).toEqual({ available: true, providerId: LOCAL_FETCH_PROVIDER_ID })
|
||||
await fiber.dispose()
|
||||
expect(ctx.web.fetchStatus()).toEqual({ available: false, reason: 'configured-missing' })
|
||||
})
|
||||
|
||||
it('has no default export (namespace plugin export shape)', () => {
|
||||
expect('default' in fetchPlugin).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../web"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# @deepseek-ai/dsh-web-search-exa
|
||||
|
||||
An [Exa](https://exa.ai)-backed `WebSearchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It calls Exa's `POST /search` endpoint with highlight contents and maps the flat `results[]` into the seam's normalized `WebSearchResult`.
|
||||
|
||||
This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the `ctx.web` key and it does not register a model-facing tool (that is `@deepseek-ai/dsh-tool-web`). Like `@deepseek-ai/dsh-llm-deepseek`, it is a function/namespace plugin (`inject: ['web']`) that registers its backend, not a default-export service.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `apiKey` | `$EXA_API_KEY` | Exa API key. Empty/absent → provider `status()` reports `missing-credential` (the seam reports `configured-unavailable`/`none`). |
|
||||
| `baseURL` | `https://api.exa.ai` | Endpoint base; `/search` is appended. |
|
||||
|
||||
```yaml
|
||||
- id: web-search-exa
|
||||
name: '@deepseek-ai/dsh-web-search-exa'
|
||||
config:
|
||||
apiKey: !!js process.env.EXA_API_KEY
|
||||
```
|
||||
|
||||
## Mapping
|
||||
|
||||
Exa returns a flat `results[]` and no generated answer, so `content` is omitted. Each result maps to a `WebSearchSource`: `url` ← `url`, `title` ← `title`, `snippet` ← the first non-empty `highlights[]` entry (a result with no highlight has no portable snippet and is dropped), `publishedAt` ← `publishedDate`. The provider passes `maxResults` through as Exa's `numResults` for a cost/latency optimization; the final bound is enforced by the seam. Provider failures (HTTP errors, network failure, unparseable bodies) surface as `WebError` `WEB_PROVIDER_ERROR`; an aborted request surfaces as `WEB_ABORTED`.
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-web-search-exa",
|
||||
"description": "Exa-backed search provider for the DeepSeek Harness web capability seam (ctx.web)",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-web": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-web": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
/**
|
||||
* `@deepseek-ai/dsh-web-search-exa`: registers an Exa-backed `WebSearchProvider`
|
||||
* with `ctx.web`. A function/namespace plugin (NOT a default-export service):
|
||||
* a search provider does not own the `ctx.web` key — it registers INTO the
|
||||
* seam's provider registry, exactly as `@deepseek-ai/dsh-llm-deepseek`
|
||||
* registers an adapter into `ctx.llm`. The key is owned by `@deepseek-ai/dsh-web`.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-search-exa
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {} from '@deepseek-ai/dsh-web'
|
||||
import { ExaSearchProvider, EXA_DEFAULT_BASE_URL } from './provider.ts'
|
||||
|
||||
export {
|
||||
EXA_DEFAULT_BASE_URL,
|
||||
EXA_PROVIDER_ID,
|
||||
ExaSearchProvider,
|
||||
mapExaResponse,
|
||||
mapExaResult,
|
||||
} from './provider.ts'
|
||||
export type { ExaSearchProviderOptions } from './provider.ts'
|
||||
|
||||
/** Cordis plugin name used by loader diagnostics. */
|
||||
export const name = 'web-search-exa'
|
||||
|
||||
/** The web seam this provider registers into. */
|
||||
export const inject = ['web']
|
||||
|
||||
export interface Config {
|
||||
/** Exa API key. Falls back to `$EXA_API_KEY`. Empty → provider unavailable. */
|
||||
apiKey?: string
|
||||
/** Endpoint base; `/search` is appended. Defaults to the public API. */
|
||||
baseURL?: string
|
||||
}
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
apiKey: z.string(),
|
||||
baseURL: z.string(),
|
||||
})
|
||||
|
||||
/** Register the Exa search provider with `ctx.web`. */
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
const apiKey = config.apiKey ?? process.env.EXA_API_KEY ?? ''
|
||||
const baseURL = config.baseURL ?? EXA_DEFAULT_BASE_URL
|
||||
ctx.web.registerSearchProvider(new ExaSearchProvider({ apiKey, baseURL }))
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
/**
|
||||
* `ExaSearchProvider`: a `WebSearchProvider` backed by the Exa search API
|
||||
* (`POST /search` with highlight contents). Maps Exa's flat `results[]` into the
|
||||
* seam's normalized `WebSearchResult`. Exa returns no provider-generated answer,
|
||||
* so `content` is omitted; each result maps to a `WebSearchSource` with `url`,
|
||||
* `title`, the first highlight as `snippet`, and `publishedDate` as
|
||||
* `publishedAt`.
|
||||
*
|
||||
* Network requests use platform-native `fetch` (Node 24), mirroring
|
||||
* `@deepseek-ai/dsh-llm-deepseek`'s adapter — not a cordis HTTP-client service.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-search-exa/provider
|
||||
*/
|
||||
|
||||
import { WebError } from '@deepseek-ai/dsh-web'
|
||||
import type {
|
||||
WebProviderStatus,
|
||||
WebSearchProvider,
|
||||
WebSearchRequest,
|
||||
WebSearchResult,
|
||||
WebSearchSource,
|
||||
} from '@deepseek-ai/dsh-web'
|
||||
import type { ExaError, ExaResult, ExaSearchResponse } from './types.ts'
|
||||
|
||||
/** Stable id this provider registers under. */
|
||||
export const EXA_PROVIDER_ID = 'exa'
|
||||
|
||||
/** Default Exa search endpoint; `/search` is the operation. */
|
||||
export const EXA_DEFAULT_BASE_URL = 'https://api.exa.ai'
|
||||
|
||||
/** Attribution header sent on every request. Bump with the package version. */
|
||||
const USER_AGENT = 'deepseek-harness/0.0.1'
|
||||
|
||||
export interface ExaSearchProviderOptions {
|
||||
/** Exa API key. Empty/absent → `status()` reports `missing-credential`. */
|
||||
apiKey: string
|
||||
/** Endpoint base; `/search` is appended. */
|
||||
baseURL: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Map one Exa result to a normalized source, or `undefined` when it carries no
|
||||
* portable snippet (an entry with no highlight is dropped — the seam has no
|
||||
* other field to derive a snippet from, and inventing one would lie).
|
||||
*/
|
||||
export function mapExaResult(result: ExaResult): WebSearchSource | undefined {
|
||||
const snippet = result.highlights?.find(highlight => highlight.trim().length > 0)
|
||||
if (snippet === undefined) return undefined
|
||||
return {
|
||||
url: result.url,
|
||||
...result.title != null && result.title.length > 0 ? { title: result.title } : {},
|
||||
snippet,
|
||||
...result.publishedDate != null && result.publishedDate.length > 0 ? { publishedAt: result.publishedDate } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/** Map an Exa response envelope to a normalized search result. */
|
||||
export function mapExaResponse(query: string, response: ExaSearchResponse): WebSearchResult {
|
||||
const sources = (response.results ?? [])
|
||||
.map(mapExaResult)
|
||||
.filter((source): source is WebSearchSource => source !== undefined)
|
||||
// Exa returns no generated answer, so `content` is omitted. The seam owns the
|
||||
// final `maxResults` truncation, so this provider reports `truncated: false`.
|
||||
return { providerId: EXA_PROVIDER_ID, query, sources, truncated: false }
|
||||
}
|
||||
|
||||
/** The Exa-backed search provider. */
|
||||
export class ExaSearchProvider implements WebSearchProvider {
|
||||
readonly id = EXA_PROVIDER_ID
|
||||
|
||||
constructor(private readonly options: ExaSearchProviderOptions) {}
|
||||
|
||||
status(): WebProviderStatus {
|
||||
if (this.options.apiKey.length === 0) return { available: false, reason: 'missing-credential' }
|
||||
return { available: true }
|
||||
}
|
||||
|
||||
async search(request: WebSearchRequest, exec?: { readonly signal?: AbortSignal }): Promise<WebSearchResult> {
|
||||
let response: Response
|
||||
try {
|
||||
response = await fetch(`${this.options.baseURL}/search`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'authorization': `Bearer ${this.options.apiKey}`,
|
||||
'content-type': 'application/json',
|
||||
'accept': 'application/json',
|
||||
'user-agent': USER_AGENT,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
query: request.query,
|
||||
contents: { highlights: true },
|
||||
...request.maxResults !== undefined ? { numResults: request.maxResults } : {},
|
||||
}),
|
||||
...exec?.signal ? { signal: exec.signal } : {},
|
||||
})
|
||||
} catch (error: unknown) {
|
||||
if (isAbortError(error)) throw new WebError('Exa search aborted', 'WEB_ABORTED', { cause: error })
|
||||
throw new WebError(`Exa search request failed: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const status = response.status
|
||||
let message = `Exa API error (HTTP ${status})`
|
||||
try {
|
||||
const parsed = await response.json() as ExaError
|
||||
const detail = parsed.error ?? parsed.message
|
||||
if (detail !== undefined && detail.length > 0) message = detail
|
||||
} catch {
|
||||
// The HTTP status is already captured in `message` above; a malformed or
|
||||
// non-JSON error body (normal for gateway 5xx/429s) can only cost a
|
||||
// richer provider message, never the real error. `response.json()` is
|
||||
// the sole statement and nothing else of consequence reaches here.
|
||||
}
|
||||
throw new WebError(message, 'WEB_PROVIDER_ERROR')
|
||||
}
|
||||
|
||||
let payload: ExaSearchResponse
|
||||
try {
|
||||
payload = await response.json() as ExaSearchResponse
|
||||
} catch (error: unknown) {
|
||||
throw new WebError(`Exa returned an unparseable response body: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
return mapExaResponse(request.query, payload)
|
||||
}
|
||||
}
|
||||
|
||||
/** True for a fetch/`AbortSignal` abort, surfaced as `WEB_ABORTED`. */
|
||||
function isAbortError(error: unknown): boolean {
|
||||
return error instanceof DOMException && error.name === 'AbortError'
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
/**
|
||||
* Wire types for the Exa search API (`POST https://api.exa.ai/search`). Types
|
||||
* only — no runtime code. Exa returns a flat `results[]`; each entry carries a
|
||||
* URL, optional title, optional `publishedDate`, and (when highlights are
|
||||
* requested) a `highlights[]` array of salient sentences.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-search-exa/types
|
||||
*/
|
||||
|
||||
/** Request body sent to Exa's search endpoint. */
|
||||
export interface ExaSearchRequest {
|
||||
query: string
|
||||
/** Exa's result-count control; the seam still enforces the bound on return. */
|
||||
numResults?: number
|
||||
/** Ask Exa to return highlight sentences per result. */
|
||||
contents: { highlights: true }
|
||||
}
|
||||
|
||||
/** One entry of Exa's flat `results[]`. */
|
||||
export interface ExaResult {
|
||||
url: string
|
||||
title?: string | null
|
||||
publishedDate?: string | null
|
||||
highlights?: string[]
|
||||
}
|
||||
|
||||
/** Exa's search response envelope. */
|
||||
export interface ExaSearchResponse {
|
||||
results?: ExaResult[]
|
||||
}
|
||||
|
||||
/** Exa's error response envelope (best-effort; fields vary by failure). */
|
||||
export interface ExaError {
|
||||
error?: string
|
||||
message?: string
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { ExaSearchProvider, EXA_DEFAULT_BASE_URL } from '@deepseek-ai/dsh-web-search-exa'
|
||||
|
||||
/**
|
||||
* Real-API smoke for the Exa search provider. Self-skips without `$EXA_API_KEY`
|
||||
* (CI has no secrets), per the with-key e2e policy in AGENTS.md § Secrets.
|
||||
*/
|
||||
const apiKey = process.env.EXA_API_KEY
|
||||
const maybe = apiKey !== undefined && apiKey.length > 0 ? describe : describe.skip
|
||||
|
||||
maybe('ExaSearchProvider real API', () => {
|
||||
it('returns sources for a live query', async () => {
|
||||
const provider = new ExaSearchProvider({ apiKey: apiKey!, baseURL: process.env.EXA_BASE_URL ?? EXA_DEFAULT_BASE_URL })
|
||||
const result = await provider.search({ query: 'DeepSeek coding agent', maxResults: 5 })
|
||||
expect(result.providerId).toBe('exa')
|
||||
expect(result.sources.length).toBeGreaterThan(0)
|
||||
for (const source of result.sources) expect(source.url).toMatch(/^https?:\/\//)
|
||||
}, 30_000)
|
||||
})
|
||||
@@ -0,0 +1,193 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import WebService from '@deepseek-ai/dsh-web'
|
||||
import { ExaSearchProvider, mapExaResponse, mapExaResult, EXA_PROVIDER_ID } from '@deepseek-ai/dsh-web-search-exa'
|
||||
import * as exaPlugin from '@deepseek-ai/dsh-web-search-exa'
|
||||
|
||||
const options = { apiKey: 'exa-key', baseURL: 'https://api.exa.test' }
|
||||
|
||||
function jsonResponse(body: unknown, init: ResponseInit = {}): Response {
|
||||
return new Response(JSON.stringify(body), { status: 200, headers: { 'content-type': 'application/json' }, ...init })
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
describe('Exa result mapping', () => {
|
||||
it('maps a full result entry', () => {
|
||||
expect(mapExaResult({
|
||||
url: 'https://a.test',
|
||||
title: 'A',
|
||||
publishedDate: '2026-01-01',
|
||||
highlights: ['salient sentence', 'second'],
|
||||
})).toEqual({ url: 'https://a.test', title: 'A', snippet: 'salient sentence', publishedAt: '2026-01-01' })
|
||||
})
|
||||
|
||||
it('drops a result with no usable highlight', () => {
|
||||
expect(mapExaResult({ url: 'https://a.test', highlights: [] })).toBeUndefined()
|
||||
expect(mapExaResult({ url: 'https://a.test' })).toBeUndefined()
|
||||
expect(mapExaResult({ url: 'https://a.test', highlights: [' '] })).toBeUndefined()
|
||||
})
|
||||
|
||||
it('omits null/empty optional fields rather than emitting them', () => {
|
||||
expect(mapExaResult({ url: 'https://a.test', title: null, publishedDate: null, highlights: ['hi'] }))
|
||||
.toEqual({ url: 'https://a.test', snippet: 'hi' })
|
||||
expect(mapExaResult({ url: 'https://a.test', title: '', publishedDate: '', highlights: ['hi'] }))
|
||||
.toEqual({ url: 'https://a.test', snippet: 'hi' })
|
||||
})
|
||||
|
||||
it('maps a response to a result with no content and filtered sources', () => {
|
||||
const result = mapExaResponse('q', {
|
||||
results: [
|
||||
{ url: 'https://a.test', highlights: ['one'] },
|
||||
{ url: 'https://b.test' },
|
||||
{ url: 'https://c.test', title: 'C', highlights: ['three'] },
|
||||
],
|
||||
})
|
||||
expect(result).toEqual({
|
||||
providerId: EXA_PROVIDER_ID,
|
||||
query: 'q',
|
||||
sources: [
|
||||
{ url: 'https://a.test', snippet: 'one' },
|
||||
{ url: 'https://c.test', title: 'C', snippet: 'three' },
|
||||
],
|
||||
truncated: false,
|
||||
})
|
||||
expect(result.content).toBeUndefined()
|
||||
})
|
||||
|
||||
it('tolerates a missing results array', () => {
|
||||
expect(mapExaResponse('q', {}).sources).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('ExaSearchProvider status', () => {
|
||||
it('is unavailable without a key', () => {
|
||||
expect(new ExaSearchProvider({ apiKey: '', baseURL: options.baseURL }).status())
|
||||
.toEqual({ available: false, reason: 'missing-credential' })
|
||||
})
|
||||
|
||||
it('is available with a key', () => {
|
||||
expect(new ExaSearchProvider(options).status()).toEqual({ available: true })
|
||||
})
|
||||
})
|
||||
|
||||
describe('ExaSearchProvider request mapping', () => {
|
||||
it('sends query, highlights, numResults and bearer auth', async () => {
|
||||
const fetchMock = vi.fn(async () => jsonResponse({ results: [{ url: 'https://a.test', highlights: ['hi'] }] }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
const provider = new ExaSearchProvider(options)
|
||||
await provider.search({ query: 'hello', maxResults: 5 })
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledOnce()
|
||||
const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit]
|
||||
expect(url).toBe('https://api.exa.test/search')
|
||||
expect((init.headers as Record<string, string>)['authorization']).toBe('Bearer exa-key')
|
||||
expect(JSON.parse(init.body as string)).toEqual({ query: 'hello', contents: { highlights: true }, numResults: 5 })
|
||||
})
|
||||
|
||||
it('omits numResults when maxResults is absent', async () => {
|
||||
const fetchMock = vi.fn(async () => jsonResponse({ results: [] }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
await new ExaSearchProvider(options).search({ query: 'q' })
|
||||
const [, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit]
|
||||
expect(JSON.parse(init.body as string)).not.toHaveProperty('numResults')
|
||||
})
|
||||
|
||||
it('forwards the abort signal', async () => {
|
||||
const fetchMock = vi.fn(async () => jsonResponse({ results: [] }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
const controller = new AbortController()
|
||||
await new ExaSearchProvider(options).search({ query: 'q' }, { signal: controller.signal })
|
||||
const [, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit]
|
||||
expect(init.signal).toBe(controller.signal)
|
||||
})
|
||||
})
|
||||
|
||||
describe('ExaSearchProvider error handling', () => {
|
||||
it('maps an HTTP error to WEB_PROVIDER_ERROR with the provider message', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({ error: 'bad key' }, { status: 401 })))
|
||||
await expect(new ExaSearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR', message: 'bad key' }))
|
||||
})
|
||||
|
||||
it('keeps a status-line message when the error body is not JSON', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => new Response('gateway down', { status: 502 })))
|
||||
await expect(new ExaSearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR', message: 'Exa API error (HTTP 502)' }))
|
||||
})
|
||||
|
||||
it('keeps the status-line message when the JSON error body carries no detail', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({}, { status: 500 })))
|
||||
await expect(new ExaSearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ message: 'Exa API error (HTTP 500)' }))
|
||||
})
|
||||
|
||||
it('maps a network failure to WEB_PROVIDER_ERROR', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new TypeError('connection refused'))))
|
||||
await expect(new ExaSearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
|
||||
it('maps an abort to WEB_ABORTED', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new DOMException('aborted', 'AbortError'))))
|
||||
await expect(new ExaSearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' }))
|
||||
})
|
||||
|
||||
it('maps an unparseable success body to WEB_PROVIDER_ERROR', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => new Response('not json', { status: 200 })))
|
||||
await expect(new ExaSearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
})
|
||||
|
||||
describe('web-search-exa plugin registration', () => {
|
||||
it('registers the provider into ctx.web (HMR-safe)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, { searchProvider: EXA_PROVIDER_ID })
|
||||
const fiber = await ctx.plugin(exaPlugin, { apiKey: 'exa-key' })
|
||||
expect(ctx.web.searchStatus()).toEqual({ available: true, providerId: EXA_PROVIDER_ID })
|
||||
await fiber.dispose()
|
||||
expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'configured-missing' })
|
||||
})
|
||||
|
||||
it('has no default export (namespace plugin export shape)', () => {
|
||||
expect('default' in exaPlugin).toBe(false)
|
||||
})
|
||||
|
||||
it('falls back to $EXA_API_KEY and the default base URL when config omits them', async () => {
|
||||
const prev = process.env.EXA_API_KEY
|
||||
process.env.EXA_API_KEY = 'env-key'
|
||||
try {
|
||||
const fetchMock = vi.fn(async () => jsonResponse({ results: [] }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, { searchProvider: EXA_PROVIDER_ID })
|
||||
const fiber = await ctx.plugin(exaPlugin, {})
|
||||
expect(ctx.web.searchStatus()).toEqual({ available: true, providerId: EXA_PROVIDER_ID })
|
||||
await ctx.web.search({ query: 'q' })
|
||||
const [url] = fetchMock.mock.calls[0] as unknown as [string]
|
||||
expect(url).toBe('https://api.exa.ai/search')
|
||||
await fiber.dispose()
|
||||
} finally {
|
||||
if (prev === undefined) delete process.env.EXA_API_KEY
|
||||
else process.env.EXA_API_KEY = prev
|
||||
}
|
||||
})
|
||||
|
||||
it('is unavailable when neither config nor env supplies a key', async () => {
|
||||
const prev = process.env.EXA_API_KEY
|
||||
delete process.env.EXA_API_KEY
|
||||
try {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, { searchProvider: EXA_PROVIDER_ID })
|
||||
await ctx.plugin(exaPlugin, {})
|
||||
expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'configured-unavailable' })
|
||||
} finally {
|
||||
if (prev !== undefined) process.env.EXA_API_KEY = prev
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../web"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
# @deepseek-ai/dsh-web-search-perplexity
|
||||
|
||||
A [Perplexity](https://perplexity.ai)-backed `WebSearchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It calls Perplexity's OpenAI-compatible `POST /chat/completions` endpoint and maps the generated answer plus citations into the seam's normalized `WebSearchResult`.
|
||||
|
||||
This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the key and it does not register a model-facing tool. Like `@deepseek-ai/dsh-llm-deepseek`, it is a function/namespace plugin (`inject: ['web']`). The OpenAI-compatible wire shape is a provider-private detail — it does **not** make this provider depend on `ctx.llm`.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `apiKey` | `$PERPLEXITY_API_KEY` | Perplexity API key. Empty/absent → provider `status()` reports `missing-credential`. |
|
||||
| `baseURL` | `https://api.perplexity.ai` | Endpoint base; `/chat/completions` is appended. |
|
||||
| `model` | `sonar` | Search model name. |
|
||||
|
||||
```yaml
|
||||
- id: web-search-perplexity
|
||||
name: '@deepseek-ai/dsh-web-search-perplexity'
|
||||
config:
|
||||
apiKey: !!js process.env.PERPLEXITY_API_KEY
|
||||
```
|
||||
|
||||
## Mapping
|
||||
|
||||
`content` ← `choices[0].message.content` (the generated answer). `sources[]` prefers the structured `search_results[]` (`url`, `title`, `snippet`, `publishedAt` ← `date`), falling back to the URL-only `citations[]` array only when `search_results` is absent — those sources carry just a `url`, which is why `title`/`snippet`/`publishedAt` are optional on the seam. Provider failures surface as `WebError` `WEB_PROVIDER_ERROR`; an aborted request surfaces as `WEB_ABORTED`. Perplexity has no result-count control, so `maxResults` is enforced by the seam (truncating `sources[]` and setting `truncated`).
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-web-search-perplexity",
|
||||
"description": "Perplexity-backed search provider for the DeepSeek Harness web capability seam (ctx.web)",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-web": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-web": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* `@deepseek-ai/dsh-web-search-perplexity`: registers a Perplexity-backed
|
||||
* `WebSearchProvider` with `ctx.web`. A function/namespace plugin (NOT a
|
||||
* default-export service): it registers INTO the seam's provider registry, like
|
||||
* `@deepseek-ai/dsh-llm-deepseek` registers an adapter into `ctx.llm`.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-search-perplexity
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {} from '@deepseek-ai/dsh-web'
|
||||
import { PerplexitySearchProvider, PERPLEXITY_DEFAULT_BASE_URL, PERPLEXITY_DEFAULT_MODEL } from './provider.ts'
|
||||
|
||||
export {
|
||||
PERPLEXITY_DEFAULT_BASE_URL,
|
||||
PERPLEXITY_DEFAULT_MODEL,
|
||||
PERPLEXITY_PROVIDER_ID,
|
||||
PerplexitySearchProvider,
|
||||
mapPerplexityResponse,
|
||||
mapPerplexityResult,
|
||||
} from './provider.ts'
|
||||
export type { PerplexitySearchProviderOptions } from './provider.ts'
|
||||
|
||||
/** Cordis plugin name used by loader diagnostics. */
|
||||
export const name = 'web-search-perplexity'
|
||||
|
||||
/** The web seam this provider registers into. */
|
||||
export const inject = ['web']
|
||||
|
||||
export interface Config {
|
||||
/** Perplexity API key. Falls back to `$PERPLEXITY_API_KEY`. Empty → unavailable. */
|
||||
apiKey?: string
|
||||
/** Endpoint base; `/chat/completions` is appended. Defaults to the public API. */
|
||||
baseURL?: string
|
||||
/** Search model name. Defaults to `sonar`. */
|
||||
model?: string
|
||||
}
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
apiKey: z.string(),
|
||||
baseURL: z.string(),
|
||||
model: z.string(),
|
||||
})
|
||||
|
||||
/** Register the Perplexity search provider with `ctx.web`. */
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
const apiKey = config.apiKey ?? process.env.PERPLEXITY_API_KEY ?? ''
|
||||
const baseURL = config.baseURL ?? PERPLEXITY_DEFAULT_BASE_URL
|
||||
const model = config.model ?? PERPLEXITY_DEFAULT_MODEL
|
||||
ctx.web.registerSearchProvider(new PerplexitySearchProvider({ apiKey, baseURL, model }))
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
/**
|
||||
* `PerplexitySearchProvider`: a `WebSearchProvider` backed by the Perplexity
|
||||
* search API (an OpenAI-compatible `POST /chat/completions`). Maps the generated
|
||||
* answer (`choices[0].message.content`) into `content`, and prefers the
|
||||
* structured `search_results[]` for `sources[]`, falling back to the URL-only
|
||||
* `citations[]` when `search_results` is absent.
|
||||
*
|
||||
* Network requests use platform-native `fetch` (Node 24), mirroring
|
||||
* `@deepseek-ai/dsh-llm-deepseek`'s adapter. The OpenAI-compatible request shape
|
||||
* is a provider-private detail and does NOT make this provider depend on
|
||||
* `ctx.llm`.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-search-perplexity/provider
|
||||
*/
|
||||
|
||||
import { WebError } from '@deepseek-ai/dsh-web'
|
||||
import type {
|
||||
WebProviderStatus,
|
||||
WebSearchProvider,
|
||||
WebSearchRequest,
|
||||
WebSearchResult,
|
||||
WebSearchSource,
|
||||
} from '@deepseek-ai/dsh-web'
|
||||
import type { PerplexityError, PerplexityResponse, PerplexitySearchResult } from './types.ts'
|
||||
|
||||
/** Stable id this provider registers under. */
|
||||
export const PERPLEXITY_PROVIDER_ID = 'perplexity'
|
||||
|
||||
/** Default Perplexity endpoint; `/chat/completions` is the operation. */
|
||||
export const PERPLEXITY_DEFAULT_BASE_URL = 'https://api.perplexity.ai'
|
||||
|
||||
/** Default search model. */
|
||||
export const PERPLEXITY_DEFAULT_MODEL = 'sonar'
|
||||
|
||||
/** Attribution header sent on every request. Bump with the package version. */
|
||||
const USER_AGENT = 'deepseek-harness/0.0.1'
|
||||
|
||||
export interface PerplexitySearchProviderOptions {
|
||||
/** Perplexity API key. Empty/absent → `status()` reports `missing-credential`. */
|
||||
apiKey: string
|
||||
/** Endpoint base; `/chat/completions` is appended. */
|
||||
baseURL: string
|
||||
/** Search model name. */
|
||||
model: string
|
||||
}
|
||||
|
||||
/** Map one structured Perplexity search result to a normalized source. */
|
||||
export function mapPerplexityResult(result: PerplexitySearchResult): WebSearchSource {
|
||||
return {
|
||||
url: result.url,
|
||||
...result.title != null && result.title.length > 0 ? { title: result.title } : {},
|
||||
...result.snippet != null && result.snippet.length > 0 ? { snippet: result.snippet } : {},
|
||||
...result.date != null && result.date.length > 0 ? { publishedAt: result.date } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Map a Perplexity response envelope to a normalized search result. Prefers
|
||||
* structured `search_results[]`; falls back to URL-only `citations[]` (those
|
||||
* sources carry just a `url`) only when `search_results` is absent.
|
||||
*/
|
||||
export function mapPerplexityResponse(query: string, response: PerplexityResponse): WebSearchResult {
|
||||
const content = response.choices?.[0]?.message?.content
|
||||
const sources: WebSearchSource[] = response.search_results !== undefined
|
||||
? response.search_results.map(mapPerplexityResult)
|
||||
: (response.citations ?? []).map(url => ({ url }))
|
||||
return {
|
||||
providerId: PERPLEXITY_PROVIDER_ID,
|
||||
query,
|
||||
...content != null && content.length > 0 ? { content } : {},
|
||||
sources,
|
||||
truncated: false,
|
||||
}
|
||||
}
|
||||
|
||||
/** The Perplexity-backed search provider. */
|
||||
export class PerplexitySearchProvider implements WebSearchProvider {
|
||||
readonly id = PERPLEXITY_PROVIDER_ID
|
||||
|
||||
constructor(private readonly options: PerplexitySearchProviderOptions) {}
|
||||
|
||||
status(): WebProviderStatus {
|
||||
if (this.options.apiKey.length === 0) return { available: false, reason: 'missing-credential' }
|
||||
return { available: true }
|
||||
}
|
||||
|
||||
async search(request: WebSearchRequest, exec?: { readonly signal?: AbortSignal }): Promise<WebSearchResult> {
|
||||
let response: Response
|
||||
try {
|
||||
response = await fetch(`${this.options.baseURL}/chat/completions`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'authorization': `Bearer ${this.options.apiKey}`,
|
||||
'content-type': 'application/json',
|
||||
'accept': 'application/json',
|
||||
'user-agent': USER_AGENT,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model: this.options.model,
|
||||
messages: [{ role: 'user', content: request.query }],
|
||||
}),
|
||||
...exec?.signal ? { signal: exec.signal } : {},
|
||||
})
|
||||
} catch (error: unknown) {
|
||||
if (isAbortError(error)) throw new WebError('Perplexity search aborted', 'WEB_ABORTED', { cause: error })
|
||||
throw new WebError(`Perplexity search request failed: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const status = response.status
|
||||
let message = `Perplexity API error (HTTP ${status})`
|
||||
try {
|
||||
const parsed = await response.json() as PerplexityError
|
||||
const detail = typeof parsed.error === 'string' ? parsed.error : parsed.error?.message ?? parsed.message
|
||||
if (detail !== undefined && detail.length > 0) message = detail
|
||||
} catch {
|
||||
// The HTTP status is already captured in `message` above; a malformed or
|
||||
// non-JSON error body (normal for gateway 5xx/429s) can only cost a
|
||||
// richer provider message, never the real error. `response.json()` is
|
||||
// the sole statement and nothing else of consequence reaches here.
|
||||
}
|
||||
throw new WebError(message, 'WEB_PROVIDER_ERROR')
|
||||
}
|
||||
|
||||
let payload: PerplexityResponse
|
||||
try {
|
||||
payload = await response.json() as PerplexityResponse
|
||||
} catch (error: unknown) {
|
||||
throw new WebError(`Perplexity returned an unparseable response body: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
return mapPerplexityResponse(request.query, payload)
|
||||
}
|
||||
}
|
||||
|
||||
/** True for a fetch/`AbortSignal` abort, surfaced as `WEB_ABORTED`. */
|
||||
function isAbortError(error: unknown): boolean {
|
||||
return error instanceof DOMException && error.name === 'AbortError'
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
/**
|
||||
* Wire types for the Perplexity search API
|
||||
* (`POST https://api.perplexity.ai/chat/completions`, an OpenAI-compatible chat
|
||||
* shape). Types only — no runtime code. Perplexity returns a generated answer in
|
||||
* `choices[0].message.content` plus citation surfaces: a structured
|
||||
* `search_results[]` (preferred) and a URL-only `citations[]` fallback.
|
||||
*
|
||||
* The OpenAI-compatible wire shape is a provider-private detail; it does not make
|
||||
* this provider depend on `ctx.llm`.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-search-perplexity/types
|
||||
*/
|
||||
|
||||
/** Request body sent to Perplexity's chat-completions endpoint. */
|
||||
export interface PerplexityRequest {
|
||||
model: string
|
||||
messages: { role: 'user'; content: string }[]
|
||||
}
|
||||
|
||||
/** One structured search result (the preferred citation surface). */
|
||||
export interface PerplexitySearchResult {
|
||||
url: string
|
||||
title?: string | null
|
||||
snippet?: string | null
|
||||
date?: string | null
|
||||
}
|
||||
|
||||
/** Perplexity's response envelope. */
|
||||
export interface PerplexityResponse {
|
||||
choices?: { message?: { content?: string | null } }[]
|
||||
/** Structured citation surface (preferred). */
|
||||
search_results?: PerplexitySearchResult[]
|
||||
/** URL-only citation fallback. */
|
||||
citations?: string[]
|
||||
}
|
||||
|
||||
/** Perplexity's error response envelope (best-effort; fields vary). */
|
||||
export interface PerplexityError {
|
||||
error?: { message?: string } | string
|
||||
message?: string
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { PerplexitySearchProvider, PERPLEXITY_DEFAULT_BASE_URL, PERPLEXITY_DEFAULT_MODEL } from '@deepseek-ai/dsh-web-search-perplexity'
|
||||
|
||||
/**
|
||||
* Real-API smoke for the Perplexity search provider. Self-skips without
|
||||
* `$PERPLEXITY_API_KEY`, per the with-key e2e policy in AGENTS.md § Secrets.
|
||||
*/
|
||||
const apiKey = process.env.PERPLEXITY_API_KEY
|
||||
const maybe = apiKey !== undefined && apiKey.length > 0 ? describe : describe.skip
|
||||
|
||||
maybe('PerplexitySearchProvider real API', () => {
|
||||
it('returns a generated answer and sources for a live query', async () => {
|
||||
const provider = new PerplexitySearchProvider({
|
||||
apiKey: apiKey!,
|
||||
baseURL: process.env.PERPLEXITY_BASE_URL ?? PERPLEXITY_DEFAULT_BASE_URL,
|
||||
model: process.env.PERPLEXITY_MODEL ?? PERPLEXITY_DEFAULT_MODEL,
|
||||
})
|
||||
const result = await provider.search({ query: 'What is the DeepSeek coding agent?', maxResults: 5 })
|
||||
expect(result.providerId).toBe('perplexity')
|
||||
expect(result.content ?? '').not.toBe('')
|
||||
for (const source of result.sources) expect(source.url).toMatch(/^https?:\/\//)
|
||||
}, 30_000)
|
||||
})
|
||||
@@ -0,0 +1,192 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import WebService from '@deepseek-ai/dsh-web'
|
||||
import {
|
||||
PerplexitySearchProvider,
|
||||
mapPerplexityResponse,
|
||||
PERPLEXITY_PROVIDER_ID,
|
||||
} from '@deepseek-ai/dsh-web-search-perplexity'
|
||||
import * as perplexityPlugin from '@deepseek-ai/dsh-web-search-perplexity'
|
||||
|
||||
const options = { apiKey: 'pplx-key', baseURL: 'https://api.perplexity.test', model: 'sonar' }
|
||||
|
||||
function jsonResponse(body: unknown, init: ResponseInit = {}): Response {
|
||||
return new Response(JSON.stringify(body), { status: 200, headers: { 'content-type': 'application/json' }, ...init })
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
describe('Perplexity response mapping', () => {
|
||||
it('maps the answer and prefers structured search_results', () => {
|
||||
const result = mapPerplexityResponse('q', {
|
||||
choices: [{ message: { content: 'the answer' } }],
|
||||
search_results: [
|
||||
{ url: 'https://a.test', title: 'A', snippet: 'snip', date: '2026-02-02' },
|
||||
{ url: 'https://b.test' },
|
||||
],
|
||||
citations: ['https://ignored.test'],
|
||||
})
|
||||
expect(result).toEqual({
|
||||
providerId: PERPLEXITY_PROVIDER_ID,
|
||||
query: 'q',
|
||||
content: 'the answer',
|
||||
sources: [
|
||||
{ url: 'https://a.test', title: 'A', snippet: 'snip', publishedAt: '2026-02-02' },
|
||||
{ url: 'https://b.test' },
|
||||
],
|
||||
truncated: false,
|
||||
})
|
||||
})
|
||||
|
||||
it('falls back to URL-only citations when search_results is absent', () => {
|
||||
const result = mapPerplexityResponse('q', {
|
||||
choices: [{ message: { content: 'answer' } }],
|
||||
citations: ['https://a.test', 'https://b.test'],
|
||||
})
|
||||
expect(result.sources).toEqual([{ url: 'https://a.test' }, { url: 'https://b.test' }])
|
||||
})
|
||||
|
||||
it('omits content when the answer is empty or missing', () => {
|
||||
expect(mapPerplexityResponse('q', { citations: [] }).content).toBeUndefined()
|
||||
expect(mapPerplexityResponse('q', { choices: [{ message: { content: '' } }] }).content).toBeUndefined()
|
||||
expect(mapPerplexityResponse('q', { choices: [{ message: { content: null } }] }).content).toBeUndefined()
|
||||
})
|
||||
|
||||
it('omits null/empty optional source fields', () => {
|
||||
const result = mapPerplexityResponse('q', {
|
||||
search_results: [{ url: 'https://a.test', title: null, snippet: '', date: null }],
|
||||
})
|
||||
expect(result.sources).toEqual([{ url: 'https://a.test' }])
|
||||
})
|
||||
|
||||
it('yields no sources when neither search_results nor citations are present', () => {
|
||||
expect(mapPerplexityResponse('q', { choices: [{ message: { content: 'a' } }] }).sources).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('PerplexitySearchProvider status', () => {
|
||||
it('is unavailable without a key', () => {
|
||||
expect(new PerplexitySearchProvider({ ...options, apiKey: '' }).status())
|
||||
.toEqual({ available: false, reason: 'missing-credential' })
|
||||
})
|
||||
|
||||
it('is available with a key', () => {
|
||||
expect(new PerplexitySearchProvider(options).status()).toEqual({ available: true })
|
||||
})
|
||||
})
|
||||
|
||||
describe('PerplexitySearchProvider request mapping', () => {
|
||||
it('sends a chat-completions request with the query as a user message', async () => {
|
||||
const fetchMock = vi.fn(async () => jsonResponse({ choices: [{ message: { content: 'a' } }], citations: [] }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
await new PerplexitySearchProvider(options).search({ query: 'hello' })
|
||||
const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit]
|
||||
expect(url).toBe('https://api.perplexity.test/chat/completions')
|
||||
expect((init.headers as Record<string, string>)['authorization']).toBe('Bearer pplx-key')
|
||||
expect(JSON.parse(init.body as string)).toEqual({ model: 'sonar', messages: [{ role: 'user', content: 'hello' }] })
|
||||
})
|
||||
|
||||
it('forwards the abort signal', async () => {
|
||||
const fetchMock = vi.fn(async () => jsonResponse({ citations: [] }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
const controller = new AbortController()
|
||||
await new PerplexitySearchProvider(options).search({ query: 'q' }, { signal: controller.signal })
|
||||
const [, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit]
|
||||
expect(init.signal).toBe(controller.signal)
|
||||
})
|
||||
})
|
||||
|
||||
describe('PerplexitySearchProvider error handling', () => {
|
||||
it('maps an HTTP error to WEB_PROVIDER_ERROR with the provider message', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({ error: { message: 'rate limited' } }, { status: 429 })))
|
||||
await expect(new PerplexitySearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR', message: 'rate limited' }))
|
||||
})
|
||||
|
||||
it('handles a string-form error body', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({ error: 'bad request' }, { status: 400 })))
|
||||
await expect(new PerplexitySearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ message: 'bad request' }))
|
||||
})
|
||||
|
||||
it('keeps a status-line message when the error body is not JSON', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => new Response('upstream error', { status: 503 })))
|
||||
await expect(new PerplexitySearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ message: 'Perplexity API error (HTTP 503)' }))
|
||||
})
|
||||
|
||||
it('keeps the status-line message when the JSON error body carries no detail', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => jsonResponse({}, { status: 500 })))
|
||||
await expect(new PerplexitySearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ message: 'Perplexity API error (HTTP 500)' }))
|
||||
})
|
||||
|
||||
it('maps an abort to WEB_ABORTED', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new DOMException('aborted', 'AbortError'))))
|
||||
await expect(new PerplexitySearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' }))
|
||||
})
|
||||
|
||||
it('maps an unparseable success body to WEB_PROVIDER_ERROR', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(async () => new Response('not json', { status: 200 })))
|
||||
await expect(new PerplexitySearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
|
||||
it('maps a network failure to WEB_PROVIDER_ERROR', async () => {
|
||||
vi.stubGlobal('fetch', vi.fn(() => Promise.reject(new TypeError('connection refused'))))
|
||||
await expect(new PerplexitySearchProvider(options).search({ query: 'q' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
})
|
||||
|
||||
describe('web-search-perplexity plugin registration', () => {
|
||||
it('registers the provider into ctx.web (HMR-safe)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, { searchProvider: PERPLEXITY_PROVIDER_ID })
|
||||
const fiber = await ctx.plugin(perplexityPlugin, { apiKey: 'pplx-key' })
|
||||
expect(ctx.web.searchStatus()).toEqual({ available: true, providerId: PERPLEXITY_PROVIDER_ID })
|
||||
await fiber.dispose()
|
||||
expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'configured-missing' })
|
||||
})
|
||||
|
||||
it('has no default export (namespace plugin export shape)', () => {
|
||||
expect('default' in perplexityPlugin).toBe(false)
|
||||
})
|
||||
|
||||
it('falls back to env key and defaults for base URL and model when config omits them', async () => {
|
||||
const prev = process.env.PERPLEXITY_API_KEY
|
||||
process.env.PERPLEXITY_API_KEY = 'env-key'
|
||||
try {
|
||||
const fetchMock = vi.fn(async () => jsonResponse({ choices: [{ message: { content: 'a' } }], citations: [] }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, { searchProvider: PERPLEXITY_PROVIDER_ID })
|
||||
const fiber = await ctx.plugin(perplexityPlugin, {})
|
||||
expect(ctx.web.searchStatus()).toEqual({ available: true, providerId: PERPLEXITY_PROVIDER_ID })
|
||||
await ctx.web.search({ query: 'q' })
|
||||
const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit]
|
||||
expect(url).toBe('https://api.perplexity.ai/chat/completions')
|
||||
expect(JSON.parse(init.body as string)).toMatchObject({ model: 'sonar' })
|
||||
await fiber.dispose()
|
||||
} finally {
|
||||
if (prev === undefined) delete process.env.PERPLEXITY_API_KEY
|
||||
else process.env.PERPLEXITY_API_KEY = prev
|
||||
}
|
||||
})
|
||||
|
||||
it('is unavailable when neither config nor env supplies a key', async () => {
|
||||
const prev = process.env.PERPLEXITY_API_KEY
|
||||
delete process.env.PERPLEXITY_API_KEY
|
||||
try {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, { searchProvider: PERPLEXITY_PROVIDER_ID })
|
||||
await ctx.plugin(perplexityPlugin, {})
|
||||
expect(ctx.web.searchStatus()).toEqual({ available: false, reason: 'configured-unavailable' })
|
||||
} finally {
|
||||
if (prev !== undefined) process.env.PERPLEXITY_API_KEY = prev
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../web"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
# @deepseek-ai/dsh-web
|
||||
|
||||
The **web access seam**: an abstract `WebService` (`ctx.web`) defining WHAT web access the harness has — search the web, fetch a URL — over multiple providers, without binding the model contract to one vendor's API shape.
|
||||
|
||||
This package is the interface third of the web capability. Unlike bash/fs it spans two capabilities (search and fetch) on one seam, with potentially multiple providers each:
|
||||
|
||||
| Package | Role |
|
||||
|---|---|
|
||||
| `@deepseek-ai/dsh-web` (this) | the interface: the service, provider registries, selection policy, request/result vocabulary, the `WebError` taxonomy |
|
||||
| `@deepseek-ai/dsh-web-search-exa` | a search implementation: Exa |
|
||||
| `@deepseek-ai/dsh-web-search-perplexity` | a search implementation: Perplexity |
|
||||
| `@deepseek-ai/dsh-web-fetch-local` | a fetch implementation: anonymous public HTTP(S) |
|
||||
| `@deepseek-ai/dsh-tool-web` | the model-facing `web_search` / `web_fetch` tool schemas over `ctx.web` |
|
||||
|
||||
Search and fetch share no request schema and no business logic, but they are deliberately one seam: `ctx.web` is a single web-access middle layer with one provider-selection policy owner, one abort/error vocabulary, and one product-facing "how this harness reaches the web" config surface. The cost is the parallel `Search`/`Fetch` method pairs; that parallelism is intentional, not a missed extraction.
|
||||
|
||||
## Service API (`ctx.web`)
|
||||
|
||||
| Member | Semantics |
|
||||
|---|---|
|
||||
| `registerSearchProvider(provider)` / `registerFetchProvider(provider)` | Register a backend. Throws `WebError` `WEB_DUPLICATE_PROVIDER` on a duplicate id within that capability kind. Returns a disposer; emits `web/providers-change` on register and on dispose. Disposed with the calling fiber. |
|
||||
| `searchStatus()` / `fetchStatus()` | Derived (never stored) `WebCapabilityStatus`: whether the capability has a selected usable provider, or the broad category it fails in. Diagnostics + execution-resolution input. |
|
||||
| `search(request, exec?)` | Resolve the search provider and run one search. Enforces `request.maxResults` on the result (truncates `sources[]`, sets `truncated`). Throws `WebError` when the capability cannot run. |
|
||||
| `fetch(request, exec?)` | Resolve the fetch provider and retrieve one URL. A non-2xx response is a result, not a throw. Throws `WebError` for failures to safely retrieve or represent the resource. |
|
||||
|
||||
Providers register **capabilities**, not tools. `dsh-tool-web` is the only owner of model-facing names, descriptions, prompt guidance, JSON schemas, and presentation.
|
||||
|
||||
## Selection
|
||||
|
||||
Selection never depends on registration, config, or HMR order. A capability has an explicit provider id (config `searchProvider`/`fetchProvider`, or env `$DSH_WEB_SEARCH_PROVIDER`/`$DSH_WEB_FETCH_PROVIDER` feeding the same fields), or auto-selects when exactly one usable provider is registered:
|
||||
|
||||
| Situation | `WebCapabilityStatus` | Execution |
|
||||
|---|---|---|
|
||||
| configured id registered and `status().available` | `available` for it | runs |
|
||||
| configured id not registered | `configured-missing` | `WEB_PROVIDER_CONFIGURED_MISSING` |
|
||||
| configured id registered but unavailable | `configured-unavailable` | `WEB_PROVIDER_CONFIGURED_UNAVAILABLE` |
|
||||
| no id, exactly one registered usable provider | `available` for it | runs |
|
||||
| no id, no usable provider | `none` | `WEB_PROVIDER_UNAVAILABLE` |
|
||||
| no id, multiple usable providers | `ambiguous` | `WEB_PROVIDER_AMBIGUOUS` |
|
||||
|
||||
`WebCapabilityStatus` carries only `available` + a `reason` discriminant (plus the winning `providerId` on the available branch). The branchable per-reason detail lives in the thrown `WebError`, which is the surface callers route on — so the same fact never gets two homes that can disagree. A provider's own `status()` is a cheap local check (credential presence, parseable config) and **must not make network calls**; `dsh-tool-web` reads only the aggregated `searchStatus()`/`fetchStatus()`, never each provider's `status()` directly.
|
||||
|
||||
## Vocabulary
|
||||
|
||||
`WebSearchRequest` (`query`, `maxResults?`) → `WebSearchResult` (`providerId`, `query`, `content?`, `sources[]`, `truncated`); each `WebSearchSource` has a required `url` and optional `title`/`snippet`/`publishedAt` (Perplexity citations may be URL-only). `WebFetchRequest` (`url`, `timeoutMs?`) → `WebFetchResult` (`providerId`, final `url`, `statusCode`, `body`, `truncated`); `WebFetchBody` is a CLOSED discriminated union (`html` | `text`) owned here — consumers `switch` to exhaustiveness so a new kind breaks their compilation until handled. See `src/types.ts` for the full contracts and the `WebError` code taxonomy.
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-web",
|
||||
"description": "Abstract web access capability seam (ctx.web) for the DeepSeek Harness — search/fetch provider registry, registration-order-independent selection, request/result vocabulary, and the WebError taxonomy",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,270 @@
|
||||
/**
|
||||
* The web access seam (`ctx.web`): a provider registry plus a provider-selecting
|
||||
* execution surface for two capabilities — search and fetch. Provider packages
|
||||
* register concrete backends with `registerSearchProvider` /
|
||||
* `registerFetchProvider`; the model-facing consumer
|
||||
* (`@deepseek-ai/dsh-tool-web`) reads capability status and executes through
|
||||
* `search()` / `fetch()`.
|
||||
*
|
||||
* The registry half stays close to `LlmService`: a `Map<id, provider>` per
|
||||
* capability kind, register methods that return disposers, duplicate ids that
|
||||
* throw, and execution-time resolution that throws when the selected provider is
|
||||
* absent or unusable. On top of that sits one small selection-status layer so
|
||||
* diagnostics and execution can explain why a capability can or cannot run,
|
||||
* independent of registration order.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web
|
||||
*/
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {
|
||||
WebCapabilityStatus,
|
||||
WebExecContext,
|
||||
WebFetchProvider,
|
||||
WebFetchRequest,
|
||||
WebFetchResult,
|
||||
WebProviderStatus,
|
||||
WebSearchProvider,
|
||||
WebSearchRequest,
|
||||
WebSearchResult,
|
||||
} from './types.ts'
|
||||
import { WebError } from './types.ts'
|
||||
|
||||
export {
|
||||
WebError,
|
||||
} from './types.ts'
|
||||
export type {
|
||||
WebCapabilityStatus,
|
||||
WebErrorCode,
|
||||
WebExecContext,
|
||||
WebFetchBody,
|
||||
WebFetchProvider,
|
||||
WebFetchRequest,
|
||||
WebFetchResult,
|
||||
WebProviderStatus,
|
||||
WebSearchProvider,
|
||||
WebSearchRequest,
|
||||
WebSearchResult,
|
||||
WebSearchSource,
|
||||
} from './types.ts'
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
web: WebService
|
||||
}
|
||||
|
||||
interface Events {
|
||||
/**
|
||||
* Fired after the provider registry changes — a search or fetch provider was
|
||||
* registered or disposed. Carries no payload and no capability graph: it
|
||||
* means only "the provider registry changed; observers may recompute status
|
||||
* from `ctx.web`". `searchStatus()` / `fetchStatus()` stay derived, not
|
||||
* stored.
|
||||
* @mode emit
|
||||
*/
|
||||
'web/providers-change'(this: WebService): void
|
||||
}
|
||||
}
|
||||
|
||||
/** Selection inputs shared by the status query and execution resolution. */
|
||||
interface Selection<P> {
|
||||
/** The configured provider id for this capability, if any. */
|
||||
readonly configuredId?: string
|
||||
/** Providers registered for this capability kind. */
|
||||
readonly providers: ReadonlyMap<string, P>
|
||||
}
|
||||
|
||||
/**
|
||||
* Config for the web seam. `searchProvider` / `fetchProvider` pin which provider
|
||||
* wins for each capability; both are optional (a single registered usable
|
||||
* provider auto-selects). Operational overrides such as environment variables
|
||||
* must feed these same fields rather than introduce a hidden priority chain.
|
||||
*/
|
||||
export interface WebServiceConfig {
|
||||
/** Explicit search provider id. Omitted = auto-select when exactly one usable. */
|
||||
readonly searchProvider?: string
|
||||
/** Explicit fetch provider id. Omitted = auto-select when exactly one usable. */
|
||||
readonly fetchProvider?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* The web access service. Registered as `ctx.web` (one instance per context).
|
||||
*
|
||||
* Selection semantics (identical for status and execution, never order-
|
||||
* dependent):
|
||||
* - A configured id that is registered and `status().available` → that provider.
|
||||
* - A configured id not registered → `configured-missing` /
|
||||
* `WEB_PROVIDER_CONFIGURED_MISSING`.
|
||||
* - A configured id registered but unavailable → `configured-unavailable` /
|
||||
* `WEB_PROVIDER_CONFIGURED_UNAVAILABLE`.
|
||||
* - No id configured, exactly one registered usable provider → that provider.
|
||||
* - No id configured, multiple usable providers → `ambiguous` /
|
||||
* `WEB_PROVIDER_AMBIGUOUS`.
|
||||
* - No id configured, no usable provider → `none` / `WEB_PROVIDER_UNAVAILABLE`.
|
||||
*/
|
||||
export class WebService extends Service {
|
||||
/**
|
||||
* Provider selection config. Operational env overrides feed the SAME fields:
|
||||
* `$DSH_WEB_SEARCH_PROVIDER` / `$DSH_WEB_FETCH_PROVIDER` are equivalent to
|
||||
* `searchProvider` / `fetchProvider` and are NOT a hidden priority chain.
|
||||
*/
|
||||
static Config: z<WebServiceConfig> = z.object({
|
||||
searchProvider: z.string(),
|
||||
fetchProvider: z.string(),
|
||||
})
|
||||
|
||||
private searchProviders = new Map<string, WebSearchProvider>()
|
||||
private fetchProviders = new Map<string, WebFetchProvider>()
|
||||
private readonly searchProviderId: string | undefined
|
||||
private readonly fetchProviderId: string | undefined
|
||||
|
||||
constructor(ctx: Context, config: WebServiceConfig = {}) {
|
||||
super(ctx, 'web')
|
||||
this.searchProviderId = config.searchProvider ?? process.env.DSH_WEB_SEARCH_PROVIDER
|
||||
this.fetchProviderId = config.fetchProvider ?? process.env.DSH_WEB_FETCH_PROVIDER
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a search provider. Throws {@link WebError} `WEB_DUPLICATE_PROVIDER`
|
||||
* if its id is already registered for search. Returns a disposer; emits
|
||||
* `web/providers-change` after a successful register and again on dispose.
|
||||
* Disposed with the calling fiber.
|
||||
*/
|
||||
registerSearchProvider(provider: WebSearchProvider): () => void {
|
||||
return this.registerProvider(this.searchProviders, provider)
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a fetch provider. Throws {@link WebError} `WEB_DUPLICATE_PROVIDER`
|
||||
* if its id is already registered for fetch. Returns a disposer; emits
|
||||
* `web/providers-change` after a successful register and again on dispose.
|
||||
* Disposed with the calling fiber.
|
||||
*/
|
||||
registerFetchProvider(provider: WebFetchProvider): () => void {
|
||||
return this.registerProvider(this.fetchProviders, provider)
|
||||
}
|
||||
|
||||
private registerProvider<P extends { readonly id: string }>(store: Map<string, P>, provider: P): () => void {
|
||||
if (store.has(provider.id)) {
|
||||
throw new WebError(`a web provider with id "${provider.id}" is already registered`, 'WEB_DUPLICATE_PROVIDER')
|
||||
}
|
||||
const dispose = this.ctx.effect(function* (this: WebService) {
|
||||
store.set(provider.id, provider)
|
||||
// Yield the rollback BEFORE emitting `web/providers-change`: the generator
|
||||
// effect collects each yielded disposer before the next step runs, so a
|
||||
// throwing change listener removes the just-added provider instead of
|
||||
// leaking it into the registry.
|
||||
yield () => {
|
||||
store.delete(provider.id)
|
||||
this.ctx.emit('web/providers-change')
|
||||
}
|
||||
this.ctx.emit('web/providers-change')
|
||||
}.bind(this), 'web.registerProvider()')
|
||||
// ctx.effect's disposer returns Promise<void>; our disposer API is
|
||||
// synchronous fire-and-forget — discard the (always-resolved) promise.
|
||||
return () => void dispose()
|
||||
}
|
||||
|
||||
/** Search-capability selection status, derived live (never stored). */
|
||||
searchStatus(): WebCapabilityStatus {
|
||||
return resolveStatus({
|
||||
providers: this.searchProviders,
|
||||
...this.searchProviderId !== undefined ? { configuredId: this.searchProviderId } : {},
|
||||
})
|
||||
}
|
||||
|
||||
/** Fetch-capability selection status, derived live (never stored). */
|
||||
fetchStatus(): WebCapabilityStatus {
|
||||
return resolveStatus({
|
||||
providers: this.fetchProviders,
|
||||
...this.fetchProviderId !== undefined ? { configuredId: this.fetchProviderId } : {},
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Run one search through the selected provider. Resolves the provider at call
|
||||
* time with the selection rules above; throws {@link WebError} when the
|
||||
* capability cannot run. The seam enforces `request.maxResults` on the result:
|
||||
* if the provider over-returns, `sources[]` is truncated and `truncated` set.
|
||||
*/
|
||||
async search(request: WebSearchRequest, exec?: WebExecContext): Promise<WebSearchResult> {
|
||||
const provider = resolveProvider({
|
||||
providers: this.searchProviders,
|
||||
...this.searchProviderId !== undefined ? { configuredId: this.searchProviderId } : {},
|
||||
})
|
||||
const result = await provider.search(request, exec)
|
||||
return capSources(result, request.maxResults)
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieve one URL through the selected provider. Resolves the provider at
|
||||
* call time with the selection rules above; throws {@link WebError} when the
|
||||
* capability cannot run. A non-2xx response is a result, not a throw.
|
||||
*/
|
||||
async fetch(request: WebFetchRequest, exec?: WebExecContext): Promise<WebFetchResult> {
|
||||
const provider = resolveProvider({
|
||||
providers: this.fetchProviders,
|
||||
...this.fetchProviderId !== undefined ? { configuredId: this.fetchProviderId } : {},
|
||||
})
|
||||
return provider.fetch(request, exec)
|
||||
}
|
||||
}
|
||||
|
||||
interface ResolvableProvider {
|
||||
readonly id: string
|
||||
status(): WebProviderStatus
|
||||
}
|
||||
|
||||
/** Compute the capability status from configured id + registered providers. */
|
||||
function resolveStatus<P extends ResolvableProvider>(selection: Selection<P>): WebCapabilityStatus {
|
||||
const { configuredId, providers } = selection
|
||||
if (configuredId !== undefined) {
|
||||
const provider = providers.get(configuredId)
|
||||
if (!provider) return { available: false, reason: 'configured-missing' }
|
||||
if (!provider.status().available) return { available: false, reason: 'configured-unavailable' }
|
||||
return { available: true, providerId: configuredId }
|
||||
}
|
||||
const usable = [...providers.values()].filter(provider => provider.status().available)
|
||||
const [single] = usable
|
||||
if (single === undefined) return { available: false, reason: 'none' }
|
||||
if (usable.length > 1) return { available: false, reason: 'ambiguous' }
|
||||
return { available: true, providerId: single.id }
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the selected provider or throw the matching {@link WebError}. Shares
|
||||
* the selection rules with {@link resolveStatus} so status and execution can
|
||||
* never disagree.
|
||||
*/
|
||||
function resolveProvider<P extends ResolvableProvider>(selection: Selection<P>): P {
|
||||
const { configuredId, providers } = selection
|
||||
if (configuredId !== undefined) {
|
||||
const provider = providers.get(configuredId)
|
||||
if (!provider) {
|
||||
throw new WebError(`configured web provider "${configuredId}" is not registered`, 'WEB_PROVIDER_CONFIGURED_MISSING')
|
||||
}
|
||||
if (!provider.status().available) {
|
||||
throw new WebError(`configured web provider "${configuredId}" is registered but unavailable`, 'WEB_PROVIDER_CONFIGURED_UNAVAILABLE')
|
||||
}
|
||||
return provider
|
||||
}
|
||||
const usable = [...providers.values()].filter(provider => provider.status().available)
|
||||
const [single] = usable
|
||||
if (single === undefined) {
|
||||
throw new WebError('no usable web provider is registered', 'WEB_PROVIDER_UNAVAILABLE')
|
||||
}
|
||||
if (usable.length > 1) {
|
||||
const ids = usable.map(provider => provider.id).join(', ')
|
||||
throw new WebError(`multiple usable web providers are registered (${ids}); configure one explicitly`, 'WEB_PROVIDER_AMBIGUOUS')
|
||||
}
|
||||
return single
|
||||
}
|
||||
|
||||
/** Enforce `maxResults` on a search result: truncate `sources[]` and flag it. */
|
||||
function capSources(result: WebSearchResult, maxResults: number | undefined): WebSearchResult {
|
||||
if (maxResults === undefined || result.sources.length <= maxResults) return result
|
||||
return { ...result, sources: result.sources.slice(0, maxResults), truncated: true }
|
||||
}
|
||||
|
||||
export default WebService
|
||||
@@ -0,0 +1,225 @@
|
||||
/**
|
||||
* Vocabulary for the web capability seam (`ctx.web`): the search/fetch
|
||||
* request/result shapes providers produce and consumers format, the provider
|
||||
* and capability status discriminants selection reports, the execution-control
|
||||
* context, and the typed error taxonomy.
|
||||
*
|
||||
* These types are shared by every provider backend
|
||||
* (`@deepseek-ai/dsh-web-search-exa`, `@deepseek-ai/dsh-web-search-perplexity`,
|
||||
* `@deepseek-ai/dsh-web-fetch-local`, and future backends) and by the
|
||||
* model-facing consumer (`@deepseek-ai/dsh-tool-web`). Search and fetch share no
|
||||
* request schema and no business logic, but they are deliberately one seam:
|
||||
* `ctx.web` is a single web-access middle layer with one provider-selection
|
||||
* policy, one abort/error vocabulary, and one product-facing configuration
|
||||
* point. The cost is the parallel `Search`/`Fetch` shapes below; that
|
||||
* parallelism is intentional.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web/types
|
||||
*/
|
||||
|
||||
import { HarnessError } from '@deepseek-ai/dsh-llm'
|
||||
|
||||
/**
|
||||
* Execution control threaded from the tool layer through the seam into a
|
||||
* provider's network requests, stream readers, and expensive decoding. It is
|
||||
* NOT business input: the first version carries only `signal` so `tool-web` can
|
||||
* propagate turn cancellation, tool timeout, and agent disposal. It deliberately
|
||||
* does NOT carry `ToolExecution`, which would make `dsh-web` depend on
|
||||
* `dsh-tools`.
|
||||
*/
|
||||
export interface WebExecContext {
|
||||
/** Abort signal a provider must honor for its network/decoding work. */
|
||||
readonly signal?: AbortSignal
|
||||
}
|
||||
|
||||
/**
|
||||
* What one search-capable backend can return. The model-facing argument is just
|
||||
* a query; `maxResults` is a `dsh-tool-web`-layer bound passed through unchanged
|
||||
* and enforced on the way back by the seam (see {@link WebSearchResult}).
|
||||
*/
|
||||
export interface WebSearchRequest {
|
||||
readonly query: string
|
||||
/**
|
||||
* Upper bound on returned sources; the seam truncates to it. Omitted = no
|
||||
* bound. `dsh-tool-web` always sets it. A provider whose API supports a
|
||||
* result-count control (Exa's `numResults`) should apply it at the request
|
||||
* layer as a cost/latency optimization; the seam enforces the bound
|
||||
* regardless.
|
||||
*/
|
||||
readonly maxResults?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalized search outcome. `content` is optional provider-generated answer
|
||||
* text or summary (Exa returns none; Perplexity returns a generated answer).
|
||||
* `sources[]` is the portable citation surface. `truncated` is set by the seam
|
||||
* when it cut `sources[]` down to `maxResults`.
|
||||
*/
|
||||
export interface WebSearchResult {
|
||||
/** Id of the provider that produced this result. */
|
||||
readonly providerId: string
|
||||
/** Echo of the query the provider answered. */
|
||||
readonly query: string
|
||||
/** Optional provider-generated answer text, search context, or summary. */
|
||||
readonly content?: string
|
||||
/** Citeable sources, already truncated to the request's `maxResults`. */
|
||||
readonly sources: readonly WebSearchSource[]
|
||||
/** True when the seam dropped sources to honor `maxResults`. */
|
||||
readonly truncated: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* One citeable source. A source always has a URL; `title`, `snippet`, and
|
||||
* `publishedAt` are optional because not every provider returns them — forcing
|
||||
* adapters to invent them would make the seam lie (Perplexity citations may be
|
||||
* URL-only). `dsh-tool-web` renders `title ?? hostname(url)` for display.
|
||||
*/
|
||||
export interface WebSearchSource {
|
||||
readonly url: string
|
||||
readonly title?: string
|
||||
readonly snippet?: string
|
||||
/** Publication/crawl timestamp as a provider-supplied ISO-8601 string. */
|
||||
readonly publishedAt?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* What one fetch-capable backend is asked to retrieve. `timeoutMs` is an
|
||||
* optional positive hint the provider caps. The request deliberately omits
|
||||
* `format`, `prompt`, and extraction controls — those are presentation or
|
||||
* higher-level LLM concerns, not safe-retrieval inputs.
|
||||
*/
|
||||
export interface WebFetchRequest {
|
||||
readonly url: string
|
||||
readonly timeoutMs?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalized fetch outcome. A successful network fetch of a non-2xx response is
|
||||
* a result, not an error: the status code is part of the fetched resource
|
||||
* state. {@link WebError} is reserved for failures to safely retrieve or
|
||||
* represent the resource.
|
||||
*/
|
||||
export interface WebFetchResult {
|
||||
/** Id of the provider that produced this result. */
|
||||
readonly providerId: string
|
||||
/** The final URL after allowed redirects (the request URL is in the request). */
|
||||
readonly url: string
|
||||
/** HTTP status code of the fetched response. */
|
||||
readonly statusCode: number
|
||||
/** Decoded body, classified by content kind. */
|
||||
readonly body: WebFetchBody
|
||||
/** True when the provider capped the decoded body. */
|
||||
readonly truncated: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* The decoded body of a fetched resource. A CLOSED discriminated union owned by
|
||||
* `dsh-web`: the provider decodes the kind and `dsh-tool-web` renders it, so a
|
||||
* new kind is a coordinated change across known packages, not a plugin
|
||||
* extension. Consumers `switch` on `kind` ending in `default: assertNever(...)`
|
||||
* so adding a kind breaks compilation at every consumer until handled. Each arm
|
||||
* stays its own object literal even where fields coincide today, leaving room
|
||||
* for arm-specific fields later (a `pdf` body's `pageCount`).
|
||||
*/
|
||||
export type WebFetchBody =
|
||||
| { readonly kind: 'html'; readonly content: string }
|
||||
| { readonly kind: 'text'; readonly content: string }
|
||||
|
||||
/**
|
||||
* Whether one concrete provider implementation is usable, by cheap local checks
|
||||
* only (credential presence, parseable endpoint config). A provider `status()`
|
||||
* must NOT make network calls. It is an input to selection, not a health system.
|
||||
*/
|
||||
export type WebProviderStatus =
|
||||
| { readonly available: true }
|
||||
| { readonly available: false; readonly reason: 'missing-credential' | 'misconfigured' }
|
||||
|
||||
/**
|
||||
* Whether a capability (search or fetch) has a selected usable provider, or the
|
||||
* broad category in which selection fails. Intentionally small: it carries the
|
||||
* winning `providerId` on the available branch (so diagnostics can report which
|
||||
* provider won) but NOT the per-reason payload (the missing id, the ambiguous
|
||||
* candidate set). That branchable detail lives in the {@link WebError} thrown at
|
||||
* execution time — the surface callers route on — so the same fact does not get
|
||||
* two homes that can disagree.
|
||||
*/
|
||||
export type WebCapabilityStatus =
|
||||
| { readonly available: true; readonly providerId: string }
|
||||
| { readonly available: false; readonly reason: 'none' | 'configured-missing' | 'configured-unavailable' | 'ambiguous' }
|
||||
|
||||
/**
|
||||
* A search-capable backend. Registered with `ctx.web.registerSearchProvider`.
|
||||
* `id` is a stable string, unique within the search capability kind.
|
||||
*/
|
||||
export interface WebSearchProvider {
|
||||
readonly id: string
|
||||
/** Cheap local usability check; must not make network calls. */
|
||||
status(): WebProviderStatus
|
||||
/** Run one search; honor `exec.signal` for cancellation. */
|
||||
search(request: WebSearchRequest, exec?: WebExecContext): Promise<WebSearchResult>
|
||||
}
|
||||
|
||||
/**
|
||||
* A fetch-capable backend. Registered with `ctx.web.registerFetchProvider`.
|
||||
* `id` is a stable string, unique within the fetch capability kind.
|
||||
*/
|
||||
export interface WebFetchProvider {
|
||||
readonly id: string
|
||||
/** Cheap local usability check; must not make network calls. */
|
||||
status(): WebProviderStatus
|
||||
/** Retrieve one URL; honor `exec.signal` for cancellation. */
|
||||
fetch(request: WebFetchRequest, exec?: WebExecContext): Promise<WebFetchResult>
|
||||
}
|
||||
|
||||
/**
|
||||
* Stable codes for {@link WebError}. Callers (hooks, tests, UI) route on these.
|
||||
*
|
||||
* - `WEB_PROVIDER_UNAVAILABLE`: no provider configured and none usable.
|
||||
* - `WEB_PROVIDER_CONFIGURED_MISSING`: a configured id is not registered.
|
||||
* - `WEB_PROVIDER_CONFIGURED_UNAVAILABLE`: a configured id is registered but its
|
||||
* `status()` reports unavailable.
|
||||
* - `WEB_PROVIDER_AMBIGUOUS`: no id configured and multiple usable providers
|
||||
* exist (selection refuses to pick by registration order).
|
||||
* - `WEB_DUPLICATE_PROVIDER`: a registration-time programming error — an id is
|
||||
* already registered for that capability kind.
|
||||
* - `WEB_INVALID_URL`: the fetch URL is malformed or not http(s).
|
||||
* - `WEB_BLOCKED_URL`: the fetch URL is rejected by policy (credentials in URL).
|
||||
* - `WEB_REDIRECT_BLOCKED`: a cross-origin redirect was refused.
|
||||
* - `WEB_FETCH_TOO_LARGE`: the response exceeded the byte/character cap.
|
||||
* - `WEB_FETCH_TIMEOUT`: the fetch exceeded its timeout.
|
||||
* - `WEB_ABORTED`: the operation was aborted via `WebExecContext.signal`.
|
||||
* - `WEB_UNSUPPORTED_CONTENT_TYPE`: the response content type cannot be decoded.
|
||||
* - `WEB_PROVIDER_ERROR`: catch-all for a provider's own failure surfaced through
|
||||
* the seam, including network/transport failure (DNS, connection refused, TLS).
|
||||
*/
|
||||
export type WebErrorCode =
|
||||
| 'WEB_PROVIDER_UNAVAILABLE'
|
||||
| 'WEB_PROVIDER_CONFIGURED_MISSING'
|
||||
| 'WEB_PROVIDER_CONFIGURED_UNAVAILABLE'
|
||||
| 'WEB_PROVIDER_AMBIGUOUS'
|
||||
| 'WEB_DUPLICATE_PROVIDER'
|
||||
| 'WEB_INVALID_URL'
|
||||
| 'WEB_BLOCKED_URL'
|
||||
| 'WEB_REDIRECT_BLOCKED'
|
||||
| 'WEB_FETCH_TOO_LARGE'
|
||||
| 'WEB_FETCH_TIMEOUT'
|
||||
| 'WEB_ABORTED'
|
||||
| 'WEB_UNSUPPORTED_CONTENT_TYPE'
|
||||
| 'WEB_PROVIDER_ERROR'
|
||||
|
||||
/**
|
||||
* Typed web error. Extends {@link HarnessError} so it carries a stable
|
||||
* {@link WebErrorCode} and chains `cause`. `dsh-web` owns this vocabulary so
|
||||
* providers, the seam, and the tool layer raise the same codes instead of each
|
||||
* inventing message strings. `ToolRegistry.execute()` converts a thrown
|
||||
* `WebError` into an error tool result whose structured metadata exposes the
|
||||
* code.
|
||||
*/
|
||||
export class WebError extends HarnessError {
|
||||
override readonly code: WebErrorCode
|
||||
|
||||
constructor(message: string, code: WebErrorCode, options?: ErrorOptions) {
|
||||
super(message, code, options)
|
||||
this.code = code
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,263 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import WebService, {
|
||||
WebError,
|
||||
type WebFetchProvider,
|
||||
type WebFetchResult,
|
||||
type WebProviderStatus,
|
||||
type WebSearchProvider,
|
||||
type WebSearchRequest,
|
||||
type WebSearchResult,
|
||||
} from '@deepseek-ai/dsh-web'
|
||||
|
||||
/** A scripted search provider for contract tests. */
|
||||
function makeSearchProvider(
|
||||
id: string,
|
||||
status: WebProviderStatus,
|
||||
search: (request: WebSearchRequest) => Promise<WebSearchResult>,
|
||||
): WebSearchProvider {
|
||||
return { id, status: () => status, search: request => search(request) }
|
||||
}
|
||||
|
||||
function makeFetchProvider(id: string, status: WebProviderStatus, result: WebFetchResult): WebFetchProvider {
|
||||
return { id, status: () => status, fetch: () => Promise.resolve(result) }
|
||||
}
|
||||
|
||||
const available: WebProviderStatus = { available: true }
|
||||
const unavailable: WebProviderStatus = { available: false, reason: 'missing-credential' }
|
||||
|
||||
function searchResult(providerId: string, overrides: Partial<WebSearchResult> = {}): WebSearchResult {
|
||||
return { providerId, query: 'q', sources: [], truncated: false, ...overrides }
|
||||
}
|
||||
|
||||
function fetchResult(providerId: string): WebFetchResult {
|
||||
return { providerId, url: 'https://example.com', statusCode: 200, body: { kind: 'text', content: 'hi' }, truncated: false }
|
||||
}
|
||||
|
||||
/** Mount a WebService on a fresh root context with the given config. */
|
||||
async function mountWeb(config: ConstructorParameters<typeof WebService>[1] = {}): Promise<{ ctx: Context; web: WebService }> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, config)
|
||||
return { ctx, web: ctx.web }
|
||||
}
|
||||
|
||||
describe('WebService registration', () => {
|
||||
it('registers and disposes a search provider, emitting providers-change each way', async () => {
|
||||
const { ctx, web } = await mountWeb()
|
||||
const changed = vi.fn()
|
||||
ctx.on('web/providers-change', changed)
|
||||
|
||||
const dispose = web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(changed).toHaveBeenCalledTimes(1)
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' })
|
||||
|
||||
dispose()
|
||||
expect(changed).toHaveBeenCalledTimes(2)
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
|
||||
it('throws WEB_DUPLICATE_PROVIDER on a duplicate search id', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(() => web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))))
|
||||
.toThrow(expect.objectContaining({ code: 'WEB_DUPLICATE_PROVIDER' }))
|
||||
})
|
||||
|
||||
it('keeps search and fetch id namespaces independent', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('shared', available, () => Promise.resolve(searchResult('shared'))))
|
||||
expect(() => web.registerFetchProvider(makeFetchProvider('shared', available, fetchResult('shared')))).not.toThrow()
|
||||
})
|
||||
|
||||
it('rolls back a registration when a providers-change listener throws', async () => {
|
||||
const { ctx, web } = await mountWeb()
|
||||
ctx.on('web/providers-change', () => { throw new Error('listener boom') })
|
||||
expect(() => web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))))
|
||||
.toThrow('listener boom')
|
||||
// The throwing listener must not leave the provider in the registry.
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
|
||||
it('disposes provider registrations when the contributing fiber is disposed (HMR safety)', async () => {
|
||||
const { ctx, web } = await mountWeb()
|
||||
const fiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
inner.web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
}, { inject: ['web'] }))
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' })
|
||||
await fiber.dispose()
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebService selection status', () => {
|
||||
it('reports none when nothing is registered', async () => {
|
||||
const { web } = await mountWeb()
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
expect(web.fetchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
|
||||
it('auto-selects the single usable provider when no id is configured', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' })
|
||||
})
|
||||
|
||||
it('reports ambiguous when multiple usable providers exist and none is configured', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'ambiguous' })
|
||||
})
|
||||
|
||||
it('ignores unusable providers when auto-selecting', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
web.registerSearchProvider(makeSearchProvider('perplexity', unavailable, () => Promise.resolve(searchResult('perplexity'))))
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' })
|
||||
})
|
||||
|
||||
it('reports none when providers exist but none are usable', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
|
||||
it('honors a configured id over a different registered provider', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'perplexity' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'perplexity' })
|
||||
})
|
||||
|
||||
it('reports configured-missing when the configured id is not registered', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'perplexity' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'configured-missing' })
|
||||
})
|
||||
|
||||
it('reports configured-unavailable when the configured id is registered but unusable', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'exa' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'configured-unavailable' })
|
||||
})
|
||||
|
||||
it('does not let registration order change auto-selection', async () => {
|
||||
const a = await mountWeb()
|
||||
a.web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
a.web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
expect(a.web.searchStatus()).toEqual({ available: true, providerId: 'perplexity' })
|
||||
|
||||
const b = await mountWeb()
|
||||
b.web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
b.web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(b.web.searchStatus()).toEqual({ available: true, providerId: 'perplexity' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebService execution resolution', () => {
|
||||
it('throws WEB_PROVIDER_UNAVAILABLE when nothing is registered', async () => {
|
||||
const { web } = await mountWeb()
|
||||
await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_UNAVAILABLE' }))
|
||||
})
|
||||
|
||||
it('throws WEB_PROVIDER_CONFIGURED_MISSING for an unregistered configured id', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'perplexity' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_CONFIGURED_MISSING' }))
|
||||
})
|
||||
|
||||
it('throws WEB_PROVIDER_CONFIGURED_UNAVAILABLE for an unusable configured id', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'exa' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_CONFIGURED_UNAVAILABLE' }))
|
||||
})
|
||||
|
||||
it('throws WEB_PROVIDER_AMBIGUOUS rather than picking by order', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_AMBIGUOUS' }))
|
||||
})
|
||||
|
||||
it('runs the selected provider and returns its result', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(
|
||||
searchResult('exa', { content: 'answer', sources: [{ url: 'https://a' }] }),
|
||||
)))
|
||||
const result = await web.search({ query: 'q' })
|
||||
expect(result.providerId).toBe('exa')
|
||||
expect(result.content).toBe('answer')
|
||||
expect(result.sources).toEqual([{ url: 'https://a' }])
|
||||
})
|
||||
|
||||
it('propagates the abort signal to the provider', async () => {
|
||||
const { web } = await mountWeb()
|
||||
const seen: (AbortSignal | undefined)[] = []
|
||||
web.registerSearchProvider({
|
||||
id: 'exa',
|
||||
status: () => available,
|
||||
search: (_request, exec) => { seen.push(exec?.signal); return Promise.resolve(searchResult('exa')) },
|
||||
})
|
||||
const controller = new AbortController()
|
||||
await web.search({ query: 'q' }, { signal: controller.signal })
|
||||
expect(seen[0]).toBe(controller.signal)
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebService maxResults enforcement', () => {
|
||||
it('truncates sources and sets truncated when a provider over-returns', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa', {
|
||||
sources: [{ url: 'https://1' }, { url: 'https://2' }, { url: 'https://3' }],
|
||||
}))))
|
||||
const result = await web.search({ query: 'q', maxResults: 2 })
|
||||
expect(result.sources).toHaveLength(2)
|
||||
expect(result.truncated).toBe(true)
|
||||
})
|
||||
|
||||
it('leaves truncated false when within the bound', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa', {
|
||||
sources: [{ url: 'https://1' }],
|
||||
}))))
|
||||
const result = await web.search({ query: 'q', maxResults: 8 })
|
||||
expect(result.sources).toHaveLength(1)
|
||||
expect(result.truncated).toBe(false)
|
||||
})
|
||||
|
||||
it('does not bound when maxResults is omitted', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa', {
|
||||
sources: [{ url: 'https://1' }, { url: 'https://2' }],
|
||||
}))))
|
||||
const result = await web.search({ query: 'q' })
|
||||
expect(result.sources).toHaveLength(2)
|
||||
expect(result.truncated).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebService fetch capability', () => {
|
||||
it('resolves and runs the fetch provider independently of search', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerFetchProvider(makeFetchProvider('local-http', available, fetchResult('local-http')))
|
||||
const result = await web.fetch({ url: 'https://example.com' })
|
||||
expect(result.providerId).toBe('local-http')
|
||||
expect(result.statusCode).toBe(200)
|
||||
})
|
||||
|
||||
it('throws WEB_PROVIDER_UNAVAILABLE for fetch when no fetch provider is registered', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
await expect(web.fetch({ url: 'https://example.com' })).rejects.toThrow(
|
||||
expect.objectContaining({ code: 'WEB_PROVIDER_UNAVAILABLE' }),
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebError', () => {
|
||||
it('is a HarnessError carrying its code', () => {
|
||||
const error = new WebError('boom', 'WEB_INVALID_URL')
|
||||
expect(error.code).toBe('WEB_INVALID_URL')
|
||||
expect(error.name).toBe('WebError')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
}
|
||||
]
|
||||
}
|
||||
Generated
+86
@@ -703,6 +703,92 @@ importers:
|
||||
specifier: ^4.0.0-rc.6
|
||||
version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4)
|
||||
|
||||
packages/web/tool-web:
|
||||
dependencies:
|
||||
schemastery:
|
||||
specifier: ^3.18.0
|
||||
version: 3.18.0
|
||||
devDependencies:
|
||||
'@deepseek-ai/dsh-agent':
|
||||
specifier: workspace:^
|
||||
version: link:../../core/agent
|
||||
'@deepseek-ai/dsh-llm':
|
||||
specifier: workspace:^
|
||||
version: link:../../llm/llm
|
||||
'@deepseek-ai/dsh-session':
|
||||
specifier: workspace:^
|
||||
version: link:../../core/session
|
||||
'@deepseek-ai/dsh-system-prompt':
|
||||
specifier: workspace:^
|
||||
version: link:../../core/system-prompt
|
||||
'@deepseek-ai/dsh-tools':
|
||||
specifier: workspace:^
|
||||
version: link:../../core/tools
|
||||
'@deepseek-ai/dsh-web':
|
||||
specifier: workspace:^
|
||||
version: link:../web
|
||||
'@deepseek-ai/dsh-web-fetch-local':
|
||||
specifier: workspace:^
|
||||
version: link:../web-fetch-local
|
||||
'@deepseek-ai/dsh-web-search-exa':
|
||||
specifier: workspace:^
|
||||
version: link:../web-search-exa
|
||||
cordis:
|
||||
specifier: ^4.0.0-rc.6
|
||||
version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4)
|
||||
|
||||
packages/web/web:
|
||||
dependencies:
|
||||
schemastery:
|
||||
specifier: ^3.18.0
|
||||
version: 3.18.0
|
||||
devDependencies:
|
||||
'@deepseek-ai/dsh-llm':
|
||||
specifier: workspace:^
|
||||
version: link:../../llm/llm
|
||||
cordis:
|
||||
specifier: ^4.0.0-rc.6
|
||||
version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4)
|
||||
|
||||
packages/web/web-fetch-local:
|
||||
dependencies:
|
||||
schemastery:
|
||||
specifier: ^3.18.0
|
||||
version: 3.18.0
|
||||
devDependencies:
|
||||
'@deepseek-ai/dsh-web':
|
||||
specifier: workspace:^
|
||||
version: link:../web
|
||||
cordis:
|
||||
specifier: ^4.0.0-rc.6
|
||||
version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4)
|
||||
|
||||
packages/web/web-search-exa:
|
||||
dependencies:
|
||||
schemastery:
|
||||
specifier: ^3.18.0
|
||||
version: 3.18.0
|
||||
devDependencies:
|
||||
'@deepseek-ai/dsh-web':
|
||||
specifier: workspace:^
|
||||
version: link:../web
|
||||
cordis:
|
||||
specifier: ^4.0.0-rc.6
|
||||
version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4)
|
||||
|
||||
packages/web/web-search-perplexity:
|
||||
dependencies:
|
||||
schemastery:
|
||||
specifier: ^3.18.0
|
||||
version: 3.18.0
|
||||
devDependencies:
|
||||
'@deepseek-ai/dsh-web':
|
||||
specifier: workspace:^
|
||||
version: link:../web
|
||||
cordis:
|
||||
specifier: ^4.0.0-rc.6
|
||||
version: 4.0.0-rc.6(@cordisjs/plugin-include@1.0.4)(@cordisjs/plugin-loader@1.0.0-rc.4)
|
||||
|
||||
vendor/cordis:
|
||||
dependencies:
|
||||
'@cordisjs/plugin-include':
|
||||
|
||||
@@ -34,6 +34,8 @@
|
||||
"@cordisjs/plugin-timer": ["./vendor/timer/src"],
|
||||
"@cordisjs/plugin-hmr": ["./vendor/hmr/src"],
|
||||
"@cordisjs/plugin-logger-console": ["./vendor/logger-console/src"],
|
||||
"@deepseek-ai/dsh-tool-web/search": ["./packages/web/tool-web/src/search.ts"],
|
||||
"@deepseek-ai/dsh-tool-web/fetch": ["./packages/web/tool-web/src/fetch.ts"],
|
||||
// One wildcard maps every @deepseek-ai/dsh-<name> to its source. Package
|
||||
// dir names are unique across groups, so first-on-disk-wins resolution is
|
||||
// unambiguous; adding a package under an existing group needs no edit
|
||||
@@ -45,6 +47,7 @@
|
||||
"./packages/bash/*/src",
|
||||
"./packages/compact/*/src",
|
||||
"./packages/subagent/*/src",
|
||||
"./packages/web/*/src",
|
||||
"./packages/session-persistence/*/src",
|
||||
"./packages/ui/*/src",
|
||||
"./packages/util/*/src",
|
||||
|
||||
@@ -27,6 +27,11 @@
|
||||
{ "path": "./packages/llm/llm-pi-ai" },
|
||||
{ "path": "./packages/bash/bash-local" },
|
||||
{ "path": "./packages/bash/tool-bash" },
|
||||
{ "path": "./packages/web/web" },
|
||||
{ "path": "./packages/web/web-search-exa" },
|
||||
{ "path": "./packages/web/web-search-perplexity" },
|
||||
{ "path": "./packages/web/web-fetch-local" },
|
||||
{ "path": "./packages/web/tool-web" },
|
||||
{ "path": "./packages/support/invariants" },
|
||||
{ "path": "./packages/ui/acp" },
|
||||
{ "path": "./packages/ui/acp-agent" },
|
||||
|
||||
Reference in New Issue
Block a user