ci: add integration workflow with a live pgvector service

Adds a live-PostgreSQL integration workflow (.github/workflows/integration.yml)
that runs the existing tests/vector_store/test_pgvector_store.py suite against
a real pgvector/pgvector:pg16 service container, on pull_request/weekly
cron/workflow_dispatch. Kept separate from ci.yml (a required check) so an
image pull, Postgres startup, or occasional service flake can't block
unrelated merges, matching install-matrix.yml's existing precedent.

Design notes:
- Service image pinned by digest (verify-action-pins.sh only checks `uses:`
  entries, not service images, so that's called out explicitly in-workflow).
- The postgres service uses POSTGRES_HOST_AUTH_METHOD: trust rather than a
  password: it's a throwaway container reachable only from this job, so
  trust auth avoids putting any credential in the workflow at all.
- vector extension is created in its own step (PgVectorStore intentionally
  refuses to auto-create it) which doubles as an explicit connectivity gate.
- pg_available() now raises instead of skipping when TEST_PGVECTOR_URL was
  set explicitly (which CI always does), so a genuinely broken service fails
  the job instead of the suite quietly reporting green having run nothing.
- Installs are hash-verified throughout (pep517-build.txt + --no-build-isolation
  + a new pgvector-extra.txt lockfile), matching ci.yml's existing convention
  for the OpenSSF Scorecard Pinned-Dependencies check.
- Two pre-existing test issues the first live run exposed are fixed: a dead
  leftover cleanup block in test_search_empty_store that targeted the wrong
  table through an already-closed connection, and a vacuous
  assert all(uuid.UUID(...)) that could never evaluate False.

History note: this replaces several earlier commits on this branch, squashed
to drop an early revision that briefly hardcoded a throwaway
POSTGRES_PASSWORD for the ephemeral CI-only service container before this
was reworked to trust auth. That value was never reachable outside the
job and protected no real data, but GitGuardian correctly flags any
committed secret-shaped string regardless of real-world risk, so it's
removed from history rather than just superseded.

.github/requirements/README.md

@@ -30,7 +30,8 @@ each file's own autogenerated header comment for its exact command).
 | `pep517-build.txt` | ci.yml, benchmark.yml, Dockerfile | exact `[build-system] requires` from `pyproject.toml` (setuptools, wheel) - installed with `--no-build-isolation` before any `pip install -e .` / `pip install .`, since `--no-deps` alone doesn't stop pip's PEP 517 build isolation from fetching those two *unhashed* |
 | `explorer-extra-py311.txt` | ci.yml | semantica's base deps + the `explorer` extra, resolved for python 3.11 |
 | `explorer-extra-py313.txt` | Dockerfile | the same, resolved for python 3.13 (the image's actual interpreter) |
-| `pytest-tool.txt` | ci.yml | pytest, for the pre-all-extras deterministic test |
+| `pgvector-extra.txt` | integration.yml | semantica's base deps + the `vectorstore-pgvector` extra, resolved for python 3.11 |
+| `pytest-tool.txt` | ci.yml, integration.yml | pytest, for the pre-all-extras deterministic test |
 | `uv-tool.txt` | ci.yml | uv, to verify requirements-ci.txt is current |
 | `build-tools.txt` | ci.yml, release.yml | build, wheel |
 | `twine.txt` | release.yml | twine |

.github/workflows/integration.yml

@@ -0,0 +1,91 @@
+name: Integration Tests
+
+# Separate from ci.yml, which is a required check: a slow image pull or a
+# container flake must not block unrelated merges.
+
+permissions:
+  contents: read
+
+on:
+  pull_request:
+    branches: [main]
+    paths-ignore:
+      - 'docs/**'
+      - 'docs_check.py'
+      - '**/*.md'
+  schedule:
+    - cron: '0 5 * * 1'
+  workflow_dispatch:
+
+jobs:
+  pgvector:
+    name: pgvector (live PostgreSQL)
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+
+    services:
+      postgres:
+        # pgvector/pgvector:pg16 as published 2026-08-13. Pinned by digest like
+        # the action pins, though verify-action-pins.sh does not check images.
+        image: pgvector/pgvector@sha256:ccc6e83d6e35e931dc7c5def2022729d5a6c370318d099181995567ff1fb4d6b
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_DB: test
+          # Throwaway container reachable only from this job, so trust auth
+          # avoids putting a credential in the workflow at all.
+          POSTGRES_HOST_AUTH_METHOD: trust
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U postgres -d test"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 10
+
+    env:
+      TEST_PGVECTOR_URL: postgresql://postgres@localhost:5432/test
+
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+
+      - name: Install semantica with the pgvector extra
+        # Hash-verified installs throughout, matching ci.yml/security.yml/etc
+        # (OpenSSF Scorecard's Pinned-Dependencies check). --no-deps here
+        # skips runtime dependency resolution for the editable install itself
+        # (nothing to hash); pep517-build.txt + --no-build-isolation stops
+        # its PEP 517 build from separately fetching an unhashed
+        # setuptools/wheel via build isolation.
+        run: |
+          pip install -r .github/requirements/bootstrap.txt --require-hashes
+          pip install -r .github/requirements/pep517-build.txt --require-hashes
+          pip install --no-deps --no-build-isolation -e .
+          pip install -r .github/requirements/pgvector-extra.txt --require-hashes
+          pip install -r .github/requirements/pytest-tool.txt --require-hashes
+
+      - name: Create the vector extension
+        # PgVectorStore._verify_pgvector_extension() requires it and refuses to
+        # create it. Doubles as the connectivity gate.
+        run: |
+          python - <<'PY'
+          import os
+
+          import psycopg
+
+          with psycopg.connect(os.environ["TEST_PGVECTOR_URL"]) as conn:
+              conn.execute("CREATE EXTENSION IF NOT EXISTS vector")
+              conn.commit()
+          print("vector extension ready")
+          PY
+
+      - name: Run the live pgvector suite
+        # pg_available raises rather than skipping when TEST_PGVECTOR_URL was
+        # set explicitly (which this job always does), so a service that's
+        # actually unreachable fails this step instead of the suite quietly
+        # reporting green having run nothing.
+        run: |
+          pytest tests/vector_store/test_pgvector_store.py -v -rs

CHANGELOG.md

@@ -151,6 +151,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - **Also fixed, on the JSON-LD paths**: the first fix covered the Turtle, N-Triples and RDF/XML serializers, and left both JSON-LD writers interpolating the entity's own text into `f"semantica:entity/{text}"` and the endpoints into `f"semantica:rel/{source}_{target}"`. Three consequences, all live in 0.6.5: an entity whose text contained a space produced an invalid IRI, and a JSON-LD parser dropped that node in full rather than reporting it, so the entity disappeared from the export; every relationship carrying `source`/`target` rather than `source_id`/`target_id` minted the identical `semantica:rel/_`, collapsing all of them onto one node whose types and endpoints merged; and the JSON-LD `@id` disagreed with the Turtle IRI for the same entity, so the two serializations of one knowledge graph were two different graphs. Both JSON-LD writers now use `mint_entity_iri`/`mint_relationship_iri`, and `JSONExporter.export_entities`/`export_relationships` declare the `semantica` prefix their `@context` was already writing `semantica:entities` against — without it a processor reads that as an IRI in the scheme `semantica`, which is the original #1101 defect on a third path
   - `tests/export/test_jsonld_iri_minting.py` parses each export with a real JSON-LD processor and asserts the entity survives, the relationships stay distinct, no term expands into the `semantica` scheme, and the JSON-LD `@id` equals the Turtle IRI
   - 236 export and ontology tests pass
+- **`semantica.evals` runner gains per-metric objectives** (#1091)
+  - `evaluate()` now accepts `config={"<evaluator>": {"objective": {"direction": "maximize"|"minimize", "threshold": X}}}` to override the evaluator's default pass verdict with a threshold; `{"objective": {"expect": bool}}` expresses a Boolean expectation
+  - `minimize` requires a `threshold` — omitting it or setting it to `None` raises `ValueError`; `maximize` without a threshold is a no-op (the evaluator's own verdict stands); `expect` cannot be combined with `direction`/`threshold`; invalid config raises `ValueError` before any evaluator runs
+  - Error metrics are never affected by objectives (error wins over fail)
+  - Backward compatible: no `objective` key → existing behavior unchanged
+  - New tests in `tests/evals/test_runner.py::TestObjective`
+- **`semantica.evals` is now a fully implemented evaluation module** (was a "Coming Soon" stub in the package layout)
+  - `evaluate(cases, evaluators, config=None, target_fn=None)` runner with per-case `pass`/`fail`/`error` status and an aggregate `pass_rate`, using a registry of named evaluators (`list_evaluators()`)
+  - 10 built-in evaluators: `exact_match`, `regex_match`, `numeric_range`, `temporal_range`, `length_range`, `keyword_check`, `levenshtein` (edit-distance similarity), `rouge` (in-house token F1, no new dependencies), `llm_as_judge` (lazy: caller-supplied `judge_fn`), and `decision_scores` (composite over `semantica.context.Decision`)
+  - `decision_scores` validates field-level (expected outcome, confidence bounds, non-empty maker/reasoning/scenario) and governance-level (provenance record presence; opt-in `PolicyEngine.check_compliance`) checks, coercing dict inputs via `Decision(**actual)` and never crashing on malformed input; an interface slot for causal-chain/embedding checks is reserved and raises `NotImplementedError` (V2)
+  - `__version__` is `0.1.0`, and the module ships a usage guide at `semantica/evals/usage.md` with worked import/run/interpret examples
+  - `semantica.evals` is reachable through the root package lazy module proxy (`semantica.evals`)
+  - 99 unit tests in `tests/evals/` covering every evaluator, registry errors, runner aggregation, decision coercion, and per-metric objectives; `python -m pytest tests/evals -q` → 99 passed
 
 - **First-class CrewAI integration** (#988, closes #962) by @Shindevrp
   - New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)

docs/superpowers/plans/2026-08-19-evals-objective.md

@@ -0,0 +1,338 @@
+# Objective Layer for semantica.evals Runner — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add per-metric objective support (direction + threshold, or Boolean expectation) to the `evaluate()` runner, overriding evaluator default pass verdicts, backward-compatible when no objective is configured.
+
+**Architecture:** The runner already iterates evaluators and computes per-case status. Objectives are read from `config["<name>"]["objective"]`, validated up front, and applied to each returned metric's `passed` field (and `details`) before aggregation. Error metrics always win over objectives.
+
+**Tech Stack:** Python 3.8+, stdlib only (typing, dataclasses). pytest for tests.
+
+## Global Constraints
+
+- Python >= 3.8: use `typing.Dict/List/Optional/Union`, never builtin generics or `|`.
+- Zero new dependencies.
+- Do not change the `EvalMetric` shape, the `evaluate()` signature, or the evaluator function signature.
+- Existing behavior with no `objective` configured must be byte-for-byte unchanged (all 62 existing tests keep passing).
+- Error metrics (`meta` contains `"error"`) always classify the case as `error`, regardless of objective.
+- Config errors are programmer errors: raise `ValueError` from `evaluate()` before any evaluator runs (fail-fast).
+- Tests go in `tests/evals/`, pytest class style, no new files outside the listed paths.
+
+---
+
+### Task 1: Objective parsing, validation, and re-decision in the runner
+
+**Files:**
+- Modify: `semantica/evals/runner.py`
+- Test: `tests/evals/test_runner.py`
+
+**Interfaces:**
+- Consumes: `EvalMetric` from `.types` (fields: `score`, `passed`, `meta`); `evaluate(cases, evaluators, config=None, target_fn=None)` existing signature.
+- Produces: private helpers `_parse_objective(name, eval_config) -> Optional[Dict]` (returns `None` when no objective configured, raises `ValueError` on invalid config) and `_apply_objective(metric, objective) -> bool` (returns the re-decided `passed`). Public `evaluate()` behavior extended as specified.
+
+- [ ] **Step 1: Write the failing tests**
+
+Append a new test class to `tests/evals/test_runner.py`:
+
+```python
+class TestObjective:
+    def test_maximize_with_threshold_pass(self):
+        # levenshtein similarity 1.0 for identical, objective demands >= 0.5
+        result = evaluate(
+            [("apple", "apple")],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.5}}},
+        )
+        assert result.cases[0].status == "pass"
+        assert result.cases[0].metrics["levenshtein"].passed is True
+
+    def test_maximize_with_threshold_fail(self):
+        result = evaluate(
+            [("apple", "aple")],  # similarity < 1.0
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.99}}},
+        )
+        assert result.cases[0].status == "fail"
+        assert result.cases[0].metrics["levenshtein"].passed is False
+        assert "levenshtein" in result.cases[0].details
+
+    def test_minimize_with_threshold_pass(self):
+        # edit distance normalized ~0.2; objective: distance <= 0.5
+        result = evaluate(
+            [("night", "nacht")],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.5}}},
+        )
+        assert result.cases[0].status == "pass"
+        assert result.cases[0].metrics["levenshtein"].passed is True
+
+    def test_minimize_with_threshold_fail(self):
+        result = evaluate(
+            [("night", "nacht")],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.1}}},
+        )
+        assert result.cases[0].status == "fail"
+
+    def test_expect_true_on_boolean_metric(self):
+        result = evaluate(
+            [("ok", "ok")],
+            evaluators=["exact_match"],
+            config={"exact_match": {"objective": {"expect": True}}},
+        )
+        assert result.cases[0].status == "pass"
+
+    def test_expect_false_overrides_passing_metric(self):
+        # exact_match passes (score 1.0) but expectation is false -> fail
+        result = evaluate(
+            [("ok", "ok")],
+            evaluators=["exact_match"],
+            config={"exact_match": {"objective": {"expect": False}}},
+        )
+        assert result.cases[0].status == "fail"
+        assert result.cases[0].metrics["exact_match"].passed is False
+        assert "exact_match" in result.cases[0].details
+
+    def test_maximize_without_threshold_is_noop(self):
+        # identical behavior to no objective: evaluator's own verdict stands
+        result = evaluate(
+            [("ok", "no")],
+            evaluators=["exact_match"],
+            config={"exact_match": {"objective": {"direction": "maximize"}}},
+        )
+        assert result.cases[0].status == "fail"
+
+    def test_minimize_without_threshold_raises(self):
+        with pytest.raises(ValueError):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "minimize"}}},
+            )
+
+    def test_bad_direction_raises(self):
+        with pytest.raises(ValueError):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "sideways", "threshold": 0.5}}},
+            )
+
+    def test_expect_with_direction_raises(self):
+        with pytest.raises(ValueError):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"expect": True, "direction": "maximize"}}},
+            )
+
+    def test_error_metric_wins_over_objective(self):
+        result = evaluate(
+            [("[invalid", "x")],
+            evaluators=["regex_match"],
+            config={"regex_match": {"objective": {"direction": "maximize", "threshold": 0.0}}},
+        )
+        assert result.cases[0].status == "error"
+        assert result.errors == 1
+        assert result.failed == 0
+
+    def test_no_objective_unchanged(self):
+        result = evaluate([("ok", "no")], evaluators=["exact_match"])
+        assert result.cases[0].status == "fail"
+```
+
+- [ ] **Step 2: Run tests to verify they fail**
+
+Run: `python3 -m pytest tests/evals/test_runner.py -q`
+Expected: the new `TestObjective` tests fail (objective config ignored → `exact_match` passes under `expect:false` etc.); the pre-existing tests in the file still pass.
+
+- [ ] **Step 3: Implement objective parsing, validation, and re-decision**
+
+In `semantica/evals/runner.py`, add two helpers before `evaluate` and wire them into the evaluator loop.
+
+```python
+def _parse_objective(name, eval_config):
+    """Return the validated objective dict, or None when not configured.
+
+    Raises ValueError for invalid configurations (programmer error).
+    """
+    objective = (eval_config or {}).get("objective")
+    if objective is None:
+        return None
+    direction = objective.get("direction")
+    threshold = objective.get("threshold")
+    expect = objective.get("expect")
+
+    if expect is not None:
+        if direction is not None or threshold is not None:
+            raise ValueError(
+                f"objective for '{name}': 'expect' cannot be combined with "
+                "'direction' or 'threshold'"
+            )
+        return {"expect": bool(expect)}
+    if direction == "minimize":
+        if threshold is None:
+            raise ValueError(
+                f"objective for '{name}': 'minimize' requires a 'threshold'"
+            )
+        return {"direction": "minimize", "threshold": float(threshold)}
+    if direction == "maximize":
+        if threshold is None:
+            # no bar to re-decide against; treat as absent (evaluator default stands)
+            return None
+        return {"direction": "maximize", "threshold": float(threshold)}
+    raise ValueError(
+        f"objective for '{name}': 'direction' must be 'maximize' or 'minimize' "
+        f"(got {direction!r})"
+    )
+
+
+def _apply_objective(metric, objective):
+    """Return the objective-adjusted pass verdict for a non-error metric."""
+    if "expect" in objective:
+        return bool(metric.score) == objective["expect"]
+    if objective["direction"] == "minimize":
+        return metric.score <= objective["threshold"]
+    return metric.score >= objective["threshold"]
+```
+
+Then modify the evaluator loop in `evaluate()` so the parsed objective is computed once per case (outside the evaluator loop, since it only depends on merged config), and applied inside the loop:
+
+```python
+        objective_by_name = {
+            name: _parse_objective(name, merged.get(name) or {})
+            for name in evaluators
+        }
+        metrics: Dict[str, EvalMetric] = {}
+        details: Dict[str, Any] = {}
+        failed, errored = False, False
+        for name in evaluators:
+            eval_config = merged.get(name) or {}
+            try:
+                metric = get_evaluator(name)(actual, expected, config=eval_config)
+                objective = objective_by_name.get(name)
+                if objective is not None and "error" not in metric.meta:
+                    metric = EvalMetric(metric.score, _apply_objective(metric, objective), metric.meta)
+                metrics[name] = metric
+                if "error" in metric.meta:
+                    errored = True
+                    details[name] = metric.meta
+                elif not metric.passed:
+                    failed = True
+                    details[name] = metric.meta
+            except Exception as exc:  # noqa: BLE001
+                errored = True
+                metrics[name] = EvalMetric(0.0, False, {"error": str(exc)})
+                details[name] = {"error": str(exc)}
+```
+
+Note: `objective_by_name` is computed once per case (it depends only on merged config), so invalid config raises `ValueError` at the first case — satisfying the fail-fast requirement. `EvalMetric` is a frozen dataclass, so the re-verdict constructs a new instance preserving score/meta.
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+Run: `python3 -m pytest tests/evals/test_runner.py -q`
+Expected: all `TestObjective` tests pass; pre-existing tests still pass.
+
+- [ ] **Step 5: Run the full evals suite**
+
+Run: `python3 -m pytest tests/evals -q`
+Expected: 62 existing + new tests all pass (no regressions).
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add semantica/evals/runner.py tests/evals/test_runner.py
+git commit -m "feat(evals): add per-metric objective support to runner"
+```
+
+---
+
+### Task 2: Documentation — usage.md and CHANGELOG
+
+**Files:**
+- Modify: `semantica/evals/usage.md`
+- Modify: `CHANGELOG.md`
+
+**Interfaces:**
+- Consumes: the objective config surface implemented in Task 1 (exact keys: `objective.direction`, `objective.threshold`, `objective.expect`; validation rules).
+- Produces: docs only.
+
+- [ ] **Step 1: Add objective section to usage.md**
+
+Append a section after the existing "Run the runner over decision records" section:
+
+```markdown
+## Set per-evaluator objectives
+
+By default each evaluator decides its own pass/fail. To override that
+verdict at the run level, configure an **objective** per evaluator name:
+
+```python
+from semantica.evals import evaluate
+
+# Require a minimum similarity (default direction is maximize):
+evaluate(
+    [("apple", "aple")],
+    evaluators=["levenshtein"],
+    config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.7}}},
+)
+
+# Lower is better — override the direction:
+evaluate(
+    [("night", "nacht")],
+    evaluators=["levenshtein"],
+    config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.5}}},
+)
+
+# Boolean expectation on a 0/1 metric:
+evaluate(
+    [("ok", "ok")],
+    evaluators=["exact_match"],
+    config={"exact_match": {"objective": {"expect": False}}},
+)
+```
+
+Rules:
+
+- `maximize` + `threshold`: pass iff `score >= threshold`. `maximize` without
+  a threshold is a no-op (the evaluator's own verdict stands).
+- `minimize` + `threshold`: pass iff `score <= threshold`. `minimize`
+  **requires** a threshold — omitting it raises `ValueError`.
+- `expect` (`true`/`false`): pass iff `bool(score)` matches; cannot be
+  combined with `direction`/`threshold`.
+- A metric whose `meta` contains `"error"` is always an error, never affected
+  by an objective.
+- Invalid objective config raises `ValueError` before any evaluator runs.
+```
+
+- [ ] **Step 2: Add CHANGELOG entry**
+
+Under `## [Unreleased]` → `### Added`, insert a new bullet at the top (before the `semantica.evals` module entry), following existing style:
+
+```markdown
+- **`semantica.evals` runner gains per-metric objectives** (#1091)
+  - `evaluate()` now accepts `config={"<evaluator>": {"objective": {"direction": "maximize"|"minimize", "threshold": X}}}` to override the evaluator's default pass verdict with a threshold; `{"objective": {"expect": bool}}` expresses a Boolean expectation
+  - `minimize` requires a `threshold`; `maximize` without one is a no-op; `expect` cannot be combined with `direction`/`threshold`; invalid config raises `ValueError` before any evaluator runs
+  - Error metrics are never affected by objectives (error wins over fail)
+  - Backward compatible: no `objective` key → existing behavior unchanged
+  - New tests in `tests/evals/test_runner.py::TestObjective`
+```
+
+- [ ] **Step 3: Verify docs examples run**
+
+Run the three examples from Step 1 as a Python script (import `evaluate`, run each snippet) to confirm they don't raise unexpectedly. No test output assertion needed beyond "no exception" and sensible status values.
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add semantica/evals/usage.md CHANGELOG.md
+git commit -m "docs(evals): document per-metric objectives"
+```
+
+---
+
+## Self-Review Notes
+
+- **Spec coverage:** §3.1 (config surface) → Task 1 helpers + Task 2 docs; §3.2 (semantics: maximize/minimize/expect) → Task 1 `_apply_objective`; §3.3 (error wins) → Task 1 error branch + `test_error_metric_wins_over_objective`; §3.4 rules 1-3 (validation) → Task 1 `_parse_objective` + 4 validation tests; §3.4 rule 4 → error branch; §3.5 (aggregation unchanged, details on final verdict) → Task 1 loop + `test_expect_false_overrides_passing_metric` asserts `details`; §4 (fail-fast ValueError) → `_parse_objective` at case top; §5 (tests) → Task 1 test class; §6 (compat) → `test_no_objective_unchanged` + full-suite green.
+- **Type consistency:** `_parse_objective(name, eval_config) -> Optional[Dict]`, `_apply_objective(metric, objective) -> bool`; `EvalMetric(score, passed, meta)` positional construction preserved everywhere.
+- **Backward compat:** objective parsed to `None` for absent config → loop behavior identical to before.

docs/superpowers/specs/2026-08-19-evals-objective-design.md

@@ -0,0 +1,115 @@
+# Design: Objective layer for `semantica.evals` runner
+
+**Date:** 2026-08-19
+**Issue:** semantica-agi/semantica#1091 (assigned to pkupt)
+**Base:** PR #1090 (`semantica.evals` module)
+
+## 1. Problem
+
+`semantica.evals` runs named evaluators and aggregates per-case pass/fail, but the pass judgement is hard-coded inside each evaluator — a higher score always means "better". There is no way to express an evaluation objective at the run level:
+
+- apply a threshold the evaluator does not encode (e.g. "F1 must be ≥ 0.7");
+- reverse the direction (e.g. "lower edit distance is better");
+- express a Boolean expectation (e.g. "this metric should be `false`").
+
+This blocks the domain-specific benchmark harnesses `docs/community-projects.md` says `semantica.evals` supports. Palantir AIP Evals models exactly this: each metric has an **objective** (Boolean expected value, or numeric `maximize`/`minimize` direction with an optional threshold), and a test case passes when **all** its metrics meet their objectives.
+
+## 2. Scope
+
+In scope:
+
+- A per-metric objective configuration consumed by the `evaluate()` runner.
+- Runner-level pass/fail re-decision for numeric scores and Boolean metrics.
+- Backward-compatible behavior when no objective is configured.
+- Tests and docs.
+
+Out of scope:
+
+- Changing the evaluator signature or the `EvalMetric` shape.
+- Multi-iteration test cases (AIP Evals has them; Semantica's runner is single-iteration per case).
+- Objective-aware aggregation beyond per-case `pass`/`fail` (existing `pass_rate` semantics are kept).
+
+## 3. Design
+
+### 3.1 Configuration surface
+
+Objective is configured per evaluator inside the runner's `config`, under the evaluator name:
+
+```python
+config = {
+    "<evaluator_name>": {
+        "objective": {
+            "direction": "maximize" | "minimize",
+            "threshold": <float>,   # optional
+        }
+    }
+}
+```
+
+Boolean-form objective (shorthand): for metrics whose score is Boolean-like (0.0/1.0) or for semantic clarity, `{"objective": {"expect": true}}` / `{"objective": {"expect": false}}` is also supported.
+
+### 3.2 Evaluation semantics
+
+For each metric produced by an evaluator during a case run, if an objective exists for that evaluator name, the runner recomputes the metric's pass verdict:
+
+- **maximize**: pass iff `score >= threshold`. If no `threshold` is given, the objective is treated as absent (evaluator's own verdict stands) — see 3.4 rule 2.
+- **minimize**: pass iff `score <= threshold` (threshold required, see 3.4 rule 1).
+- **expect**: pass iff `bool(score)` equals `expect` (for Boolean-style metrics).
+
+When an objective is present, the runner **overrides** `metric.passed` with the objective verdict. When absent, `metric.passed` is used unchanged (existing behavior).
+
+The `objective` key is a **reserved runner-level key**: it is consumed by the runner and is passed through to the evaluator function inside `eval_config` (evaluators already ignore unknown config keys via `cfg.get(...)`, so this is harmless); evaluators must not rely on it. The runner re-decision happens on the metric the evaluator returns, so no evaluator change is required.
+
+### 3.3 Interaction with errors
+
+An `EvalMetric` whose `meta` contains `"error"` remains classified as an error regardless of objective (error wins over fail, per the existing contract). Objectives only affect non-error metrics.
+
+### 3.4 Ambiguity rules (explicit decisions)
+
+1. **`minimize` without `threshold`** is rejected at config-validation time with a clear error (`ValueError`), because "lowest is best" has no absolute pass bar without a threshold. (AIP Evals allows direction-only; we require threshold to keep pass/fail well-defined.) — *Chosen for determinism; revisit if a use case demands direction-only minimize.*
+2. **`maximize` without `threshold`** behaves like no objective (pass iff evaluator's own `passed`), because the evaluator's default is already "higher is better".
+3. **`expect` with a numeric `direction`/`threshold`** is a config error (`ValueError`): pick one form.
+4. **Objective on a metric that errors** → the error wins (3.3), objective ignored.
+
+### 3.5 Aggregation
+
+Unchanged:
+
+- Case `status`: `"error"` if any metric errored, else `"fail"` if any failed, else `"pass"`.
+- `pass_rate` = passed / total (1.0 on empty).
+- `metrics` dict holds the (possibly re-verdict'd) `EvalMetric`; the re-verdict is observable via `metric.passed`.
+- `details[name]` is populated when a metric ends up failed **after** objective re-decision (i.e. objective-failed metrics appear in `details`; metrics that pass under objective are not recorded there). This mirrors the existing "record failures in details" behavior applied to the final verdict.
+
+### 3.6 Files
+
+- `semantica/evals/runner.py` — add objective parsing/validation and re-decision inside the evaluator loop.
+- `tests/evals/test_runner.py` — new test class(es) for objective semantics.
+- `semantica/evals/usage.md` — document the objective config and examples.
+- `CHANGELOG.md` — `[Unreleased]` entry.
+
+No new dependencies; Python ≥ 3.8 (stdlib `typing`).
+
+## 4. Error handling
+
+- Invalid objective config (`direction` not in {maximize, minimize}, both `expect` and `direction`, `minimize` without threshold, non-numeric threshold) → `ValueError` raised at runner config parse, before any evaluator runs. Deterministic, fail-fast.
+- These are programmer errors, not per-case data errors — no per-case `error` status involved.
+
+## 5. Testing
+
+New tests in `tests/evals/test_runner.py`:
+
+1. maximize + threshold: score ≥ threshold → pass; below → fail.
+2. minimize + threshold: score ≤ threshold → pass; above → fail (e.g. levenshtein on a close pair).
+3. minimize without threshold → `ValueError`.
+4. expect=true / expect=false on a Boolean metric (exact_match) — pass/fail per expectation.
+5. no objective → existing behavior unchanged (evaluator's own verdict).
+6. objective + error metric → error wins (status=error, not fail).
+7. config error (bad direction) → `ValueError` raised by `evaluate()`.
+8. objective turns a passing metric into failing → `details` records it; case status becomes fail.
+9. backward-compat: all existing 62 tests keep passing.
+
+## 6. Compatibility
+
+- Public API (`evaluate`, `list_evaluators`, `get_evaluator`, types) unchanged in signature.
+- `EvalMetric` shape unchanged (score, passed, meta) — only `passed` may be recomputed by the runner.
+- Existing configs (no `objective` key) behave identically.

semantica/evals/__init__.py

@@ -1,10 +1,21 @@
-"""
-Semantica Evals Module
+"""Semantica Evals — evaluation layer for decision intelligence outputs.
 
-Coming Soon
+Provides a small library of deterministic and model-backed evaluators plus a
+runner for measuring decision records, audit trails, and reasoning output.
 """
 
-__version__ = "0.1.1"
-__status__ = "coming_soon"
-__all__ = []
+from . import decision_evaluators  # noqa: F401 (registers decision_scores)
+from . import evaluators  # noqa: F401 (registers the generic evaluators)
+from .registry import get_evaluator, list_evaluators
+from .runner import evaluate
+from .types import CaseResult, EvalMetric, EvalSummary
 
+__version__ = "0.1.0"
+__all__ = [
+    "evaluate",
+    "get_evaluator",
+    "list_evaluators",
+    "CaseResult",
+    "EvalMetric",
+    "EvalSummary",
+]

semantica/evals/decision_evaluators.py

@@ -0,0 +1,90 @@
+"""Decision-specialized evaluator.
+
+``decision_scores`` validates a ``Decision`` (or dict) against field-level and
+governance-level checks: expected outcome, confidence bounds, non-empty
+required fields, provenance presence, and (when configured) policy compliance
+via ``PolicyEngine.check_compliance``.
+"""
+
+from typing import Any, Dict, Optional
+
+from .registry import register
+from .types import EvalMetric
+
+
+def _coerce_decision(actual: Any):
+    """Return a Decision or None; never raise for dict inputs."""
+    from semantica.context.decision_models import Decision
+
+    if isinstance(actual, Decision):
+        return actual
+    if isinstance(actual, dict):
+        try:
+            return Decision(**actual)
+        except (TypeError, ValueError, KeyError):
+            return None
+    return None
+
+
+@register("decision_scores")
+def decision_scores(actual, expected=None, config=None, **kwargs):
+    """Composite evaluator over a Decision; see module docstring for sub-checks."""
+    cfg = config or {}
+    decision = _coerce_decision(actual)
+    if decision is None:
+        return EvalMetric(0.0, False, {"error": "input is not a valid Decision or dict"})
+
+    checks: Dict[str, bool] = {}
+    reasons: Dict[str, str] = {}
+
+    expected_outcome = cfg.get("expected_outcome", expected)
+    if expected_outcome is not None:
+        checks["decision_outcome"] = decision.outcome == expected_outcome
+        if not checks["decision_outcome"]:
+            reasons["decision_outcome"] = f"expected {expected_outcome!r}, got {decision.outcome!r}"
+
+    lo = cfg.get("min_confidence", 0.0)
+    hi = cfg.get("max_confidence", 1.0)
+    checks["decision_confidence"] = lo <= decision.confidence <= hi
+    if not checks["decision_confidence"]:
+        reasons["decision_confidence"] = f"{decision.confidence} not in [{lo}, {hi}]"
+
+    for field in ("decision_maker", "reasoning", "scenario"):
+        value = getattr(decision, field, None)
+        checks[field] = isinstance(value, str) and bool(value.strip())
+        if not checks[field]:
+            reasons[field] = f"field {field!r} is empty"
+
+    metadata = decision.metadata if isinstance(decision.metadata, dict) else {}
+    prov = metadata.get(cfg.get("provenance_key", "provenance"))
+    checks["provenance"] = bool(prov)
+    if not checks["provenance"]:
+        reasons["provenance"] = "no provenance record found in metadata"
+
+    policy_engine = cfg.get("policy_engine")
+    policy_id = cfg.get("policy_id")
+    if policy_engine is not None and policy_id is not None:
+        try:
+            compliant = bool(policy_engine.check_compliance(decision, policy_id))
+            checks["policy"] = compliant == cfg.get("expected_policy_compliant", True)
+            if not checks["policy"]:
+                reasons["policy"] = f"compliance={compliant}"
+        except Exception as exc:  # noqa: BLE001
+            checks["policy"] = False
+            reasons["policy"] = str(exc)
+
+    if cfg.get("causal_chain_exists"):
+        raise NotImplementedError(
+            "decision_scores causal_chain_exists is an interface slot reserved for V2"
+        )
+
+    passed_count = sum(checks.values())
+    total = len(checks)
+    passed = total > 0 and passed_count == total
+    meta = dict(checks)
+    meta["reasons"] = reasons
+    return EvalMetric(
+        score=passed_count / total if total else 0.0,
+        passed=passed,
+        meta=meta,
+    )

semantica/evals/evaluators.py

@@ -0,0 +1,181 @@
+"""Generic (non-decision) evaluators for the evals module.
+
+Each evaluator takes ``(actual, expected, config=None, **kwargs)`` and returns
+an ``EvalMetric``. Config uses ``min``/``max`` bounds where relevant.
+"""
+
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+from .registry import register
+from .types import EvalMetric
+
+
+def _default_config(config):
+    return config or {}
+
+
+@register("exact_match")
+def exact_match(actual, expected, config=None, **kwargs):
+    """Score 1.0 if ``actual`` equals ``expected`` (scalar or list)."""
+    matched = actual == expected
+    return EvalMetric(
+        score=1.0 if matched else 0.0,
+        passed=matched,
+        meta={} if matched else {"reason": f"expected {expected!r}, got {actual!r}"},
+    )
+
+
+@register("regex_match")
+def regex_match(actual, expected, config=None, **kwargs):
+    """Score 1.0 if string ``actual`` matches regex ``expected``."""
+    import re
+    try:
+        matched = re.search(expected, actual) is not None
+        return EvalMetric(
+            score=1.0 if matched else 0.0,
+            passed=matched,
+            meta={} if matched else {"reason": f"'{actual}' does not match {expected}"},
+        )
+    except re.error as exc:
+        return EvalMetric(0.0, False, {"error": str(exc)})
+
+
+@register("numeric_range")
+def numeric_range(actual, expected=None, config=None, **kwargs):
+    """Score 1.0 if number ``actual`` is within inclusive ``[min, max]``."""
+    cfg = _default_config(config)
+    lo, hi = cfg.get("min"), cfg.get("max")
+    passed = lo is not None and hi is not None and lo <= actual <= hi
+    return EvalMetric(
+        score=1.0 if passed else 0.0,
+        passed=passed,
+        meta={} if passed else {"reason": f"{actual} not in [{lo}, {hi}]"},
+    )
+
+
+@register("temporal_range")
+def temporal_range(actual, expected=None, config=None, **kwargs):
+    """Score 1.0 if datetime ``actual`` is within inclusive ISO-datetime window."""
+    cfg = _default_config(config)
+    try:
+        stamp = datetime.fromisoformat(actual)
+        lo = datetime.fromisoformat(cfg["min"])
+        hi = datetime.fromisoformat(cfg["max"])
+        passed = lo <= stamp <= hi
+        return EvalMetric(
+            score=1.0 if passed else 0.0,
+            passed=passed,
+            meta={} if passed else {"reason": f"{actual} not in [{cfg['min']}, {cfg['max']}]"},
+        )
+    except (KeyError, TypeError, ValueError) as exc:
+        return EvalMetric(0.0, False, {"error": str(exc)})
+
+
+@register("length_range")
+def length_range(actual, expected=None, config=None, **kwargs):
+    """Score 1.0 if length of ``actual`` is within inclusive ``[min, max]``."""
+    cfg = _default_config(config)
+    size = len(actual)
+    lo = cfg.get("min", 0)
+    hi = cfg.get("max")
+    passed = hi is not None and lo <= size <= hi
+    return EvalMetric(
+        score=1.0 if passed else 0.0,
+        passed=passed,
+        meta={} if passed else {"reason": f"length {size} not in [{lo}, {hi}]"},
+    )
+
+
+@register("keyword_check")
+def keyword_check(actual, expected=None, config=None, **kwargs):
+    """Score 1.0 if all required terms appear in ``actual`` (word-boundary matching)."""
+    cfg = _default_config(config)
+    required = cfg.get("required") or (expected or [])
+    import re
+    tokens = set(re.findall(r"\w+", str(actual).lower()))
+    missing = [term for term in required if str(term).lower() not in tokens]
+    passed = not missing
+    return EvalMetric(
+        score=1.0 if passed else 0.0,
+        passed=passed,
+        meta={} if passed else {"missing": missing},
+    )
+
+
+def _levenshtein(a: str, b: str) -> int:
+    """Classic Levenshtein edit distance."""
+    if a == b:
+        return 0
+    if not a:
+        return len(b)
+    if not b:
+        return len(a)
+    prev = list(range(len(b) + 1))
+    for i, ca in enumerate(a, 1):
+        cur = [i]
+        for j, cb in enumerate(b, 1):
+            cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
+        prev = cur
+    return prev[-1]
+
+
+@register("levenshtein")
+def levenshtein(actual, expected, config=None, **kwargs):
+    """Score normalized similarity (1 - distance/max_len) vs ``threshold`` (default 0.8)."""
+    cfg = _default_config(config)
+    threshold = cfg.get("threshold", 0.8)
+    a, b = str(actual), str(expected)
+    max_len = max(len(a), len(b))
+    similarity = 1.0 if max_len == 0 else 1.0 - _levenshtein(a, b) / max_len
+    passed = similarity >= threshold
+    return EvalMetric(
+        score=similarity,
+        passed=passed,
+        meta={"similarity": similarity},
+    )
+
+
+def _tokenize(text: str) -> List[str]:
+    import re
+    return re.findall(r"\w+", str(text).lower())
+
+
+@register("rouge")
+def rouge(actual, expected, config=None, **kwargs):
+    """ROUGE-1 precision/recall/F1 over tokens; pass on F1 >= ``threshold`` (default 0.0)."""
+    cfg = _default_config(config)
+    threshold = cfg.get("threshold", 0.0)
+    hyp, ref = _tokenize(actual), _tokenize(expected)
+    from collections import Counter
+    hyp_c, ref_c = Counter(hyp), Counter(ref)
+    overlap = sum((hyp_c & ref_c).values())
+    precision = overlap / len(hyp) if hyp else 0.0
+    recall = overlap / len(ref) if ref else 0.0
+    f1 = 0.0 if (precision + recall) == 0 else 2 * precision * recall / (precision + recall)
+    passed = f1 > 0 and f1 >= threshold
+    return EvalMetric(
+        score=f1,
+        passed=passed,
+        meta={"precision": precision, "recall": recall, "f1": f1},
+    )
+
+
+@register("llm_as_judge")
+def llm_as_judge(actual, expected, config=None, **kwargs):
+    """Score 1.0 when a caller-supplied ``judge_fn(actual, expected) -> bool`` passes.
+
+    The judge resolver stays lazy: no LLM backend is imported unless the caller
+    provides one in config.
+    """
+    cfg = _default_config(config)
+    judge_fn = cfg.get("judge_fn")
+    if judge_fn is None:
+        return EvalMetric(
+            0.0, False, {"error": "config['judge_fn'] required (callable(actual, expected) -> bool)"}
+        )
+    try:
+        verdict = bool(judge_fn(actual, expected))
+        return EvalMetric(score=1.0 if verdict else 0.0, passed=verdict)
+    except Exception as exc:  # noqa: BLE001
+        return EvalMetric(0.0, False, {"error": str(exc)})

semantica/evals/registry.py

@@ -0,0 +1,34 @@
+"""Evaluator registry for the evals module.
+
+Evaluators are plain functions ``fn(actual, expected, config=None, **kwargs)
+-> EvalMetric`` registered under a stable string name so the runner and users
+can select them by name without importing individual modules.
+"""
+
+from typing import Callable, Dict, List
+
+from .types import EvalMetric
+
+EVALUATORS: Dict[str, Callable] = {}
+
+
+def register(name: str) -> Callable:
+    """Decorator registering an evaluator function under ``name``."""
+    def _register(fn: Callable) -> Callable:
+        if name in EVALUATORS:
+            raise ValueError(f"evaluator already registered: {name}")
+        EVALUATORS[name] = fn
+        return fn
+    return _register
+
+
+def list_evaluators() -> List[str]:
+    """Return sorted names of all registered evaluators."""
+    return sorted(EVALUATORS)
+
+
+def get_evaluator(name: str) -> Callable:
+    """Look up an evaluator by name, raising ValueError with a hint otherwise."""
+    if name not in EVALUATORS:
+        raise ValueError(f"unknown evaluator '{name}'. Available: {list_evaluators()}")
+    return EVALUATORS[name]

semantica/evals/runner.py

@@ -0,0 +1,211 @@
+"""Evaluation runner: orchestrates evaluators over a list of cases."""
+
+import math
+from typing import Any, Callable, Dict, List, Optional, Tuple, Union
+
+from .registry import get_evaluator
+from .types import CaseResult, EvalMetric, EvalSummary
+
+Case = Union[Dict[str, Any], Tuple[Any, Any]]
+
+
+def _coerce_threshold(name, threshold):
+    """Convert ``threshold`` to a finite float, raising ``ValueError`` otherwise.
+
+    Accepts any value that ``float()`` accepts (int, float, bool, numeric
+    strings) as long as the result is finite.  Raises ``ValueError`` — never
+    ``TypeError`` — for non-convertible types, NaN, and infinity so that
+    all invalid objective config produces the same exception type.
+    """
+    try:
+        value = float(threshold)
+    except (TypeError, ValueError) as exc:
+        raise ValueError(
+            f"objective for '{name}': 'threshold' must be a finite number "
+            f"(got {threshold!r})"
+        ) from exc
+    if not math.isfinite(value):
+        raise ValueError(
+            f"objective for '{name}': 'threshold' must be a finite number "
+            f"(got {threshold!r})"
+        )
+    return value
+
+
+def _parse_objective(name, eval_config):
+    """Return the validated objective dict, or None when not configured.
+
+    Raises ValueError for invalid configurations (programmer error).
+    """
+    objective = (eval_config or {}).get("objective")
+    if objective is None:
+        return None
+    if not isinstance(objective, dict):
+        raise ValueError(
+            f"objective for '{name}': expected a dict, got {type(objective).__name__}"
+        )
+    direction = objective.get("direction")
+    threshold = objective.get("threshold")
+    expect = objective.get("expect")
+
+    if expect is not None:
+        if not isinstance(expect, bool):
+            raise ValueError(
+                f"objective for '{name}': 'expect' must be a bool (got {expect!r})"
+            )
+        if direction is not None or threshold is not None:
+            raise ValueError(
+                f"objective for '{name}': 'expect' cannot be combined with "
+                "'direction' or 'threshold'"
+            )
+        return {"expect": expect}
+    if direction == "minimize":
+        if threshold is None:
+            raise ValueError(
+                f"objective for '{name}': 'minimize' requires a 'threshold'"
+            )
+        return {"direction": "minimize", "threshold": _coerce_threshold(name, threshold)}
+    if direction == "maximize":
+        if threshold is None:
+            # no bar to re-decide against; treat as absent (evaluator default stands)
+            return None
+        return {"direction": "maximize", "threshold": _coerce_threshold(name, threshold)}
+    raise ValueError(
+        f"objective for '{name}': 'direction' must be 'maximize' or 'minimize' "
+        f"(got {direction!r})"
+    )
+
+
+def _apply_objective(metric, objective):
+    """Return the objective-adjusted pass verdict for a non-error metric."""
+    if "expect" in objective:
+        return bool(metric.score) == objective["expect"]
+    if objective["direction"] == "minimize":
+        return metric.score <= objective["threshold"]
+    return metric.score >= objective["threshold"]
+
+
+def _extract(case: Case, target_fn: Optional[Callable]):
+    """Return (case_id, expected, actual, config, per_case_target_fn)."""
+    if isinstance(case, tuple):
+        expected, actual = case[0], (case[1] if len(case) > 1 else None)
+        return str(id(case)), expected, actual, {}, None
+    case_id = case.get("id") or f"case-{id(case)}"
+    expected = case.get("expected")
+    actual = case.get("actual")
+    config = case.get("config") or {}
+    per_fn = case.get("target_fn")
+    return case_id, expected, actual, config, per_fn
+
+
+def _merge_config(default_config: Dict[str, Any], case_config: Dict[str, Any]) -> Dict[str, Any]:
+    """Deep-merge per-case config over the global config (two levels deep).
+
+    Level 1 (top-level keys, e.g. evaluator names): merged key-by-key so a
+    per-case override of one evaluator's settings does not erase the whole
+    global evaluator entry.
+
+    Level 2 (evaluator config keys, e.g. ``"objective"``): also merged
+    key-by-key so a per-case override that specifies only some objective fields
+    (e.g. just ``"threshold"``) inherits the rest from the global objective
+    (e.g. ``"direction"``).  Per-case values always take precedence.
+
+    Depth-3+ values are replaced wholesale, consistent with the previous
+    single-level behaviour (no evaluator config currently nests beyond two
+    levels).  Neither the caller's global config nor the case config is
+    mutated.
+    """
+    merged = dict(default_config)
+    for key, value in (case_config or {}).items():
+        if isinstance(value, dict) and isinstance(merged.get(key), dict):
+            # Merge level-1 dict (evaluator config) key-by-key.
+            current = dict(merged[key])
+            for k, v in value.items():
+                if isinstance(v, dict) and isinstance(current.get(k), dict):
+                    # Merge level-2 dict (e.g. objective sub-dict) key-by-key.
+                    inner = dict(current[k])
+                    inner.update(v)
+                    current[k] = inner
+                else:
+                    current[k] = v
+            merged[key] = current
+        else:
+            merged[key] = value
+    return merged
+
+
+def evaluate(
+    cases: List[Case],
+    evaluators: List[str],
+    config: Optional[Dict[str, Any]] = None,
+    target_fn: Optional[Callable] = None,
+) -> EvalSummary:
+    """Run named evaluators over each case and aggregate metrics.
+
+    A per-case or top-level ``target_fn`` produces ``actual`` when the case
+    does not already carry one. Evaluator failures become ``error`` results.
+    """
+    default_config = config or {}
+    case_results: List[CaseResult] = []
+
+    # Validate objective config for every case up front so an invalid objective
+    # rejects the run before any target_fn or evaluator executes (fail-fast),
+    # regardless of which case carries it.
+    pre_resolved = []
+    for case in cases:
+        _, _, _, case_config, _ = _extract(case, target_fn)
+        merged = _merge_config(default_config, case_config)
+        pre_resolved.append(
+            {
+                name: _parse_objective(name, merged.get(name) or {})
+                for name in evaluators
+            }
+        )
+
+    for case, objective_by_name in zip(cases, pre_resolved):
+        case_id, expected, actual, case_config, per_fn = _extract(case, target_fn)
+        merged = _merge_config(default_config, case_config)
+        if expected is None:
+            expected = merged.get("expected")
+        resolver = per_fn or target_fn
+        if actual is None and resolver is not None:
+            try:
+                actual = resolver(case)
+            except Exception as exc:  # noqa: BLE001
+                case_results.append(
+                    CaseResult(case_id, "error", {}, {"target_fn": str(exc)})
+                )
+                continue
+        metrics: Dict[str, EvalMetric] = {}
+        details: Dict[str, Any] = {}
+        failed, errored = False, False
+        for name in evaluators:
+            eval_config = merged.get(name) or {}
+            try:
+                metric = get_evaluator(name)(actual, expected, config=eval_config)
+                objective = objective_by_name.get(name)
+                if objective is not None and "error" not in metric.meta:
+                    metric = EvalMetric(metric.score, _apply_objective(metric, objective), metric.meta)
+                metrics[name] = metric
+                if "error" in metric.meta:
+                    errored = True
+                    details[name] = metric.meta
+                elif not metric.passed:
+                    failed = True
+                    details[name] = metric.meta
+            except Exception as exc:  # noqa: BLE001
+                errored = True
+                metrics[name] = EvalMetric(0.0, False, {"error": str(exc)})
+                details[name] = {"error": str(exc)}
+        status = "error" if errored else ("fail" if failed else "pass")
+        case_results.append(CaseResult(case_id, status, metrics, details))
+
+    total = len(case_results)
+    passed = sum(1 for c in case_results if c.status == "pass")
+    failed = sum(1 for c in case_results if c.status == "fail")
+    errors = sum(1 for c in case_results if c.status == "error")
+    pass_rate = (passed / total) if total else 1.0
+    return EvalSummary(
+        total, passed, failed, errors, pass_rate,
+        cases=case_results,
+    )

semantica/evals/types.py

@@ -0,0 +1,37 @@
+"""Evals result data models.
+
+Defines the metric and result shapes produced by the evals module.
+"""
+
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, NamedTuple
+
+
+@dataclass(frozen=True)
+class EvalMetric:
+    """One evaluator's numeric score plus pass/fail verdict."""
+
+    score: float
+    passed: bool
+    meta: Dict[str, Any] = field(default_factory=dict)
+
+
+class CaseResult(NamedTuple):
+    """Evaluation output for a single case."""
+
+    case_id: str
+    status: str
+    metrics: Dict[str, EvalMetric]
+    details: Dict[str, Any]
+
+
+@dataclass
+class EvalSummary:
+    """Aggregate evaluation output across cases."""
+
+    total: int
+    passed: int
+    failed: int
+    errors: int
+    pass_rate: float
+    cases: List[CaseResult] = field(default_factory=list)

semantica/evals/usage.md

@@ -0,0 +1,176 @@
+# Semantica Evals — Usage
+
+The evals module measures decision intelligence outputs: decision records,
+audit trails, and reasoning output — with deterministic and model-backed
+evaluators plus a small runner.
+
+## Import
+
+```python
+import semantica.evals as evals          # through the root lazy proxy
+from semantica.evals import evaluate, list_evaluators
+```
+
+## Discover evaluators
+
+```python
+>>> evals.list_evaluators()
+['decision_scores', 'exact_match', 'keyword_check', 'length_range',
+ 'levenshtein', 'llm_as_judge', 'numeric_range', 'regex_match', 'rouge',
+ 'temporal_range']
+```
+
+`list_evaluators` returns every name registered by importing the package —
+the import wiring runs each evaluator module's `register()` side effects.
+
+## Run the runner over decision records
+
+`evaluate(cases, evaluators, config=None)` accepts a list of cases; each case is
+a dict with `expected`, `actual`, optional `config`, and optional `id`. The
+`actual` can be a finished `Decision` object or its dict form.
+
+```python
+from datetime import datetime
+from semantica.context.decision_models import Decision
+from semantica.evals import evaluate
+
+decision = Decision(
+    decision_id="d-1",
+    category="loan",
+    scenario="loan-request",
+    reasoning="vetted by policy",
+    outcome="approve",
+    confidence=0.87,
+    timestamp=datetime.now(),
+    decision_maker="approver-a",
+    metadata={"provenance": "workflow:loan/v3"},
+)
+
+cases = [
+    {
+        "id": "loan-001",
+        "actual": decision,
+        "config": {
+            "decision_scores": {
+                "expected_outcome": "approve",
+                "min_confidence": 0.7,
+            }
+        },
+    },
+    {
+        "id": "loan-002",
+        "actual": {
+            "decision_id": "d-2",
+            "category": "loan",
+            "scenario": "loan-request",
+            "reasoning": "auto",
+            "outcome": "reject",
+            "confidence": 0.9,
+            "timestamp": datetime.now().isoformat(),
+            "decision_maker": "system",
+            "metadata": {},
+        },
+        "config": {
+            "decision_scores": {
+                "expected_outcome": "approve",
+                "min_confidence": 0.7,
+            }
+        },
+    },
+]
+
+summary = evaluate(cases, ["decision_scores"])
+```
+
+`evaluate` also runs high-level names like `exact_match`, `keyword_check`, or
+`llm_as_judge`; per-case or top-level `config` may carry per-evaluator settings
+(e.g. `config={"exact_match": {...}}`).
+
+## Set per-evaluator objectives
+
+By default each evaluator decides its own pass/fail. To override that
+verdict at the run level, configure an **objective** per evaluator name:
+
+```python
+from semantica.evals import evaluate
+
+# Require a minimum similarity (levenshtein's default bar is >= 0.8; here we set 0.7):
+evaluate(
+    [("apple", "aple")],
+    evaluators=["levenshtein"],
+    config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.7}}},
+)
+
+# Lower is better — override the direction:
+evaluate(
+    [("night", "nacht")],
+    evaluators=["levenshtein"],
+    config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.7}}},
+)
+
+# Boolean expectation — the metric matches (score 1), but we expect it not to:
+evaluate(
+    [("ok", "ok")],
+    evaluators=["exact_match"],
+    config={"exact_match": {"objective": {"expect": False}}},
+)
+```
+
+Rules:
+
+- `maximize` + `threshold`: pass iff `score >= threshold`. `maximize` without
+  a threshold is a no-op (the evaluator's own verdict stands).
+- `minimize` + `threshold`: pass iff `score <= threshold`. `minimize`
+  **requires** a threshold — omitting it or setting it to `None` raises
+  `ValueError`.
+- `expect` (`true`/`false`): pass iff `bool(score)` matches; cannot be
+  combined with `direction`/`threshold`. `expect` must be a real boolean
+  (a string like `"false"` is rejected).
+- A metric whose `meta` contains `"error"` is always an error, never affected
+  by an objective.
+- Invalid objective config (non-dict objective, bad `direction`, non-bool
+  `expect`, missing `minimize` threshold) raises `ValueError` before any
+  evaluator runs.
+
+## Interpret the summary
+
+```python
+>>> summary.total, summary.passed, summary.failed, summary.errors
+(2, 1, 1, 0)
+>>> summary.pass_rate
+0.5
+
+>>> for case in summary.cases:
+...     print(case.case_id, case.status)
+...     for name, metric in case.metrics.items():
+...         print("  ", name, metric.score, metric.passed)
+...         print("    ", metric.meta.get("reasons"))
+loan-001 pass
+   decision_scores 1.0 True
+    {}
+loan-002 fail
+   decision_scores 0.667 False
+    {'decision_outcome': "expected 'approve', got 'reject'",
+     'provenance': 'no provenance record found in metadata'}
+```
+
+`EvalSummary` fields:
+
+- `total` / `passed` / `failed` / `errors` — case counts by status.
+- `pass_rate` — `passed / total` (1.0 on an empty case list).
+- `cases` — one `CaseResult` per input case: `case_id`, `status`
+  (`pass` | `fail` | `error`), `metrics` (name → `EvalMetric` with `score`,
+  `passed`, `meta`), and `details`.
+
+Evaluator failures do not crash the run; they surface as `status="error"` on
+the affected case with the exception text captured in the metric meta.
+
+## Notes
+
+- **`llm_as_judge` needs `config["judge_fn"]`**: a callable
+  `judge_fn(actual, expected) -> bool` supplied by the caller. Without it the
+  evaluator fails with `config['judge_fn'] required`.
+- **`decision_scores` governance checks are opt-in**: policy compliance is only
+  evaluated when both `config["policy_engine"]` and `config["policy_id"]` are
+  provided; otherwise those checks are skipped. The reserved
+  `causal_chain_exists` slot is not yet implemented.

semantica/vector_store/milvus_store.py

@@ -661,6 +661,15 @@ class MilvusStore:
         except Exception:
             return None
 
+    @staticmethod
+    def _record_to_result(item: Dict[str, Any]) -> Dict[str, Any]:
+        vec = item.get("vector")
+        return {
+            "id": str(item.get("id")),
+            "metadata": item.get("metadata") or {},
+            "vector": np.array(vec) if vec is not None else None,
+        }
+
     def filter_by_metadata(
         self, filters: Dict[str, Any], limit: int = 10
     ) -> List[Dict[str, Any]]:
@@ -705,21 +714,83 @@ class MilvusStore:
                 limit=limit,
                 output_fields=["id", "vector", "metadata"],
             )
-            results = []
-            for item in query_results:
-                vec = item.get("vector")
-                results.append(
-                    {
-                        "id": str(item.get("id")),
-                        "metadata": item.get("metadata") or {},
-                        "vector": np.array(vec) if vec is not None else None,
-                    }
-                )
-            return results
+            return [self._record_to_result(item) for item in query_results]
         except Exception as e:
             self.logger.warning(f"Failed to query Milvus vectors by metadata expression: {e}")
             return []
 
+    def iter_all(self, batch_size: int = 500):
+        """
+        Iterate over every stored entity using Milvus's query iterator.
+
+        Paginates by primary-key cursor rather than row offset, which is why
+        this exists instead of scan_vectors(offset, limit). query(offset=...)
+        is capped by the 16384 result window and would truncate anything
+        larger.
+
+        Assumes the schema create_collection() builds: a VARCHAR `id` primary
+        key plus vector and metadata fields, as get_vector() and
+        filter_by_metadata() already do. get_collection() does not validate
+        schema, so a collection with an integer key or no metadata field fails
+        here.
+
+        Args:
+            batch_size: Entities to request per iterator batch
+
+        Yields:
+            Result dicts with 'id', 'metadata', and 'vector', in cursor order
+
+        Raises:
+            ProcessingError: If the collection is not initialized, or the
+                installed pymilvus does not expose query_iterator().
+        """
+        if self.collection is None:
+            raise ProcessingError(
+                "Collection not initialized. Call create_collection() or get_collection() first."
+            )
+
+        if not MILVUS_AVAILABLE:
+            raise ProcessingError("Milvus not available")
+
+        query_iterator = getattr(self.collection.collection, "query_iterator", None)
+        if not callable(query_iterator):
+            raise ProcessingError(
+                "This pymilvus version does not expose Collection.query_iterator(), "
+                "which full enumeration requires. Falling back to query(offset=...) "
+                "is not safe here: it is capped by the 16384 result window and would "
+                "silently truncate a larger collection."
+            )
+
+        # Query operations need a loaded collection. Idempotent, and once per
+        # scan rather than per batch.
+        self.collection.load()
+
+        # Milvus rejects an empty expression; this match-all form is what
+        # filter_by_metadata() already uses.
+        iterator = query_iterator(
+            batch_size=batch_size,
+            expr="id != ''",
+            output_fields=["id", "vector", "metadata"],
+        )
+
+        try:
+            while True:
+                batch = iterator.next()
+                if not batch:
+                    return
+                for item in batch:
+                    yield self._record_to_result(item)
+        finally:
+            # Release the server-side iterator even if the consumer stops early.
+            # Swallowed so a broken connection at cleanup time doesn't replace
+            # whatever real exception was already propagating out of the try.
+            close = getattr(iterator, "close", None)
+            if callable(close):
+                try:
+                    close()
+                except Exception as e:
+                    self.logger.warning(f"Failed to close Milvus query iterator: {e}")
+
     def get_stats(self, collection_name: Optional[str] = None) -> Dict[str, Any]:
         """Get collection statistics."""
         if self.collection is None and collection_name:

semantica/vector_store/pinecone_store.py

@@ -299,6 +299,44 @@ class PineconeSearch:
         )
 
 
+def _pinecone_listed_ids(response: Any) -> List[str]:
+    """Extract vector IDs from a list_paginated() response.
+
+    Accepts record objects, bare id strings and dicts, since what listing
+    returns has changed across pinecone SDK major versions.
+    """
+    records = getattr(response, "vectors", None)
+    if records is None and isinstance(response, dict):
+        records = response.get("vectors")
+
+    ids: List[str] = []
+    for record in records or []:
+        if isinstance(record, str):
+            ids.append(record)
+        elif isinstance(record, dict):
+            if record.get("id") is not None:
+                ids.append(record["id"])
+        else:
+            record_id = getattr(record, "id", None)
+            if record_id is not None:
+                ids.append(record_id)
+    return ids
+
+
+def _pinecone_next_token(response: Any) -> Optional[str]:
+    """Return the continuation token, or None when the listing is exhausted."""
+    pagination = getattr(response, "pagination", None)
+    if pagination is None and isinstance(response, dict):
+        pagination = response.get("pagination")
+    if pagination is None:
+        return None
+
+    token = getattr(pagination, "next", None)
+    if token is None and isinstance(pagination, dict):
+        token = pagination.get("next")
+    return token or None
+
+
 class PineconeStore:
     """
     Pinecone store for vector storage and similarity search.
@@ -735,6 +773,91 @@ class PineconeStore:
             self.logger.warning(f"Failed to filter Pinecone vectors by metadata: {e}")
             return []
 
+    def iter_all(self, batch_size: int = 500, namespace: str = ""):
+        """
+        Iterate over every stored vector by listing IDs then fetching them.
+
+        Paginates with an opaque continuation token, which is why this exists
+        instead of scan_vectors(offset, limit): the token for page N cannot be
+        constructed without walking there.
+
+        Needs two calls per page, unlike the other backends, because listing
+        returns IDs only. Both calls are namespace scoped and must agree, and
+        listing covers one namespace rather than the whole index.
+
+        Args:
+            batch_size: IDs to request per list_paginated() call
+            namespace: Namespace to enumerate (default: the default namespace)
+
+        Yields:
+            Result dicts with 'id', 'metadata', and 'vector', in listing order
+
+        Raises:
+            ProcessingError: If the index is not initialized, if the installed
+                SDK does not expose list_paginated(), or if the listing stops
+                advancing.
+        """
+        if self.index is None or not PINECONE_AVAILABLE:
+            raise ProcessingError(
+                "Index not initialized. Call create_index() or get_index() first."
+            )
+
+        # list_paginated() rather than list(): list() is an auto-paging
+        # iterator in current SDKs but reads as plain id lists in older
+        # examples. Threading the token explicitly is version-agnostic.
+        list_paginated = getattr(self.index.index, "list_paginated", None)
+        if not callable(list_paginated):
+            raise ProcessingError(
+                "This pinecone SDK version does not expose Index.list_paginated(), "
+                "which full enumeration requires."
+            )
+
+        token = None
+        while True:
+            kwargs: Dict[str, Any] = {"limit": batch_size, "namespace": namespace}
+            if token is not None:
+                kwargs["pagination_token"] = token
+
+            response = list_paginated(**kwargs)
+            vector_ids = _pinecone_listed_ids(response)
+
+            # A page listing zero ids is not necessarily exhaustion: Pinecone's
+            # contract is that a scan ends only when there's no pagination
+            # token, and a page can legitimately come back empty while
+            # pagination.next is still set (sparse/filtered namespaces,
+            # eventual-consistency windows on serverless indexes). Skip the
+            # fetch (nothing to hydrate) but still fall through to the token
+            # check below instead of returning early, or a gap like that
+            # silently truncates the scan with no error.
+            if vector_ids:
+                fetched = self.index.fetch_vectors(vector_ids, namespace=namespace)
+                vectors = fetched.get("vectors") or {}
+
+                for vector_id in vector_ids:
+                    entry = vectors.get(vector_id)
+                    if entry is None:
+                        # fetch() omits ids it cannot find: deleted since listing.
+                        continue
+                    values = entry.get("values")
+                    yield {
+                        "id": vector_id,
+                        "metadata": entry.get("metadata") or {},
+                        "vector": np.array(values) if values is not None else None,
+                    }
+
+            next_token = _pinecone_next_token(response)
+            if not next_token:
+                return
+            if next_token == token:
+                # Distinct from exhaustion above: a partial scan here would be
+                # indistinguishable from a complete one.
+                raise ProcessingError(
+                    "Pinecone returned the same pagination token twice, so the "
+                    "listing is not advancing. Refusing to return a truncated "
+                    "scan."
+                )
+            token = next_token
+
     def fetch_vectors(
         self, vector_ids: List[str], namespace: str = "", **options
     ) -> Dict[str, Any]:

semantica/vector_store/weaviate_store.py

@@ -488,6 +488,28 @@ class WeaviateStore:
             self.logger.debug(f"Could not build native Weaviate filter: {e}")
             return None
 
+    def _fetch_objects_offset_or_plain(self, kwargs: Dict[str, Any], scanned_count: int):
+        """Retry a failed `after`-cursor fetch_objects() call with `offset`, then
+        with no pagination argument at all. Returns (objs, mode)."""
+        kwargs = dict(kwargs)
+        kwargs.pop("after", None)
+        kwargs["offset"] = scanned_count
+        try:
+            return self.collection.query.fetch_objects(**kwargs), "offset"
+        except TypeError:
+            kwargs.pop("offset", None)
+            return self.collection.query.fetch_objects(**kwargs), "single_page"
+
+    @staticmethod
+    def _extract_vector(raw_vector: Any) -> Optional[np.ndarray]:
+        """weaviate-client v4 returns vector as {'default': [...]} rather than a
+        bare list; older clients and mocks may still hand back a bare list."""
+        if isinstance(raw_vector, dict):
+            raw_vector = raw_vector.get("default")
+        if raw_vector is None or len(raw_vector) == 0:
+            return None
+        return np.array(raw_vector)
+
     def filter_by_metadata(
         self, filters: Dict[str, Any], limit: int = 10
     ) -> List[Dict[str, Any]]:
@@ -533,22 +555,11 @@ class WeaviateStore:
                         try:
                             objs = self.collection.query.fetch_objects(**kwargs)
                         except TypeError:
-                            if "after" in kwargs:
-                                kwargs.pop("after", None)
-                                kwargs["offset"] = scanned_count
-                                try:
-                                    objs = self.collection.query.fetch_objects(**kwargs)
-                                except TypeError:
-                                    kwargs.pop("offset", None)
-                                    objs = self.collection.query.fetch_objects(**kwargs)
+                            if "after" not in kwargs:
+                                raise
+                            objs, _ = self._fetch_objects_offset_or_plain(kwargs, scanned_count)
                     elif "after" in kwargs:
-                        kwargs.pop("after", None)
-                        kwargs["offset"] = scanned_count
-                        try:
-                            objs = self.collection.query.fetch_objects(**kwargs)
-                        except TypeError:
-                            kwargs.pop("offset", None)
-                            objs = self.collection.query.fetch_objects(**kwargs)
+                        objs, _ = self._fetch_objects_offset_or_plain(kwargs, scanned_count)
                     else:
                         raise te
                 except Exception as fe:
@@ -611,6 +622,122 @@ class WeaviateStore:
             self.logger.warning(f"Failed to fetch Weaviate objects by metadata filter: {e}")
             return results if results else []
 
+    def iter_all(self, batch_size: int = 500):
+        """
+        Iterate over every stored object using Weaviate's UUID cursor.
+
+        Paginates by the last object's UUID rather than a row offset, which is
+        why this exists instead of scan_vectors(offset, limit). An empty page
+        under that cursor falls back to offset pagination once before ending
+        the scan, since an empty page isn't on its own proof there's nothing
+        left past it (see the inline comment below).
+
+        Assumes a single unnamed vector per object, as get_vector() and
+        filter_by_metadata() already do. Named-vector collections return a
+        mapping and are not handled.
+
+        Args:
+            batch_size: Objects to request per fetch_objects() call
+
+        Yields:
+            Result dicts with 'id', 'metadata', and 'vector', in cursor order
+
+        Raises:
+            ProcessingError: If the collection is not initialized, or if the
+                scan cannot advance past a full page.
+        """
+        if self.collection is None or not WEAVIATE_AVAILABLE:
+            raise ProcessingError(
+                "Collection not initialized. Call get_collection() first."
+            )
+
+        after_cursor = None
+        scanned_count = 0
+        # Degrades cursor -> offset -> single_page as the client rejects each
+        # form. Tracked across iterations, not just inside the except branch,
+        # or later pages go out with no pagination argument at all.
+        mode = "cursor"
+
+        while True:
+            kwargs = {"limit": batch_size, "include_vector": True}
+            if mode == "cursor" and after_cursor is not None:
+                kwargs["after"] = after_cursor
+            elif mode == "offset":
+                kwargs["offset"] = scanned_count
+
+            try:
+                objs = self.collection.query.fetch_objects(**kwargs)
+            except TypeError:
+                if mode == "cursor" and "after" in kwargs:
+                    objs, mode = self._fetch_objects_offset_or_plain(kwargs, scanned_count)
+                elif mode == "offset":
+                    mode = "single_page"
+                    kwargs.pop("offset", None)
+                    objs = self.collection.query.fetch_objects(**kwargs)
+                else:
+                    raise
+
+            batch_objects = getattr(objs, "objects", None) if objs else None
+            if not batch_objects:
+                # An empty page in "cursor" mode isn't necessarily the end.
+                # Unlike an offset, `after` has no server-issued continuation
+                # value of its own - it's derived client-side from the last
+                # object's uuid - so an empty page gives nothing to advance
+                # it with. If Weaviate's cursor walks internal storage
+                # position rather than strict uuid order, a batch can in
+                # principle land entirely on a gap (e.g. tombstoned objects)
+                # with live data past it, the same risk already confirmed for
+                # Qdrant's scroll cursor (#1316). Offset pagination doesn't
+                # have that ambiguity - it addresses live rows by position -
+                # so fall back to it once to confirm before ending the scan.
+                if mode == "cursor":
+                    mode = "offset"
+                    continue
+                return
+
+            page_full = len(batch_objects) >= batch_size
+            next_cursor = after_cursor
+
+            # Checked before yielding: a page that can't advance is truncation,
+            # not completion, and the caller shouldn't see any of it go out
+            # before the error does.
+            if page_full:
+                if mode == "single_page":
+                    raise ProcessingError(
+                        "This Weaviate client accepts neither an `after` cursor nor a "
+                        "numeric offset, so the scan cannot advance past the first "
+                        "page. Refusing to return a truncated scan."
+                    )
+                if mode == "cursor":
+                    last_uuid = getattr(batch_objects[-1], "uuid", None)
+                    if last_uuid is None:
+                        raise ProcessingError(
+                            "The last object of a full Weaviate page has no uuid, so the "
+                            "cursor cannot advance. Refusing to return a truncated scan."
+                        )
+                    next_cursor = str(last_uuid)
+                    if next_cursor == after_cursor:
+                        raise ProcessingError(
+                            "The Weaviate cursor stopped advancing, so the listing is "
+                            "repeating a page. Refusing to return a truncated scan."
+                        )
+
+            for obj in batch_objects:
+                obj_uuid = getattr(obj, "uuid", None)
+                yield {
+                    "id": str(obj_uuid) if obj_uuid is not None else None,
+                    "metadata": getattr(obj, "properties", None) or {},
+                    "vector": self._extract_vector(getattr(obj, "vector", None)),
+                }
+
+            scanned_count += len(batch_objects)
+
+            if not page_full:
+                return
+
+            if mode == "cursor":
+                after_cursor = next_cursor
+
 
     def query_vectors(
         self,

tests/evals/test_decision_evaluators.py

@@ -0,0 +1,136 @@
+"""Tests for the decision_scores composite evaluator."""
+import pytest
+from datetime import datetime
+
+from semantica.context.decision_models import Decision
+from semantica.evals import registry as reg
+
+
+def _decision(**overrides):
+    base = dict(
+        decision_id="d1",
+        category="loan",
+        scenario="mortgage application",
+        reasoning="strong credit history",
+        outcome="approved",
+        confidence=0.95,
+        timestamp=datetime(2026, 1, 1),
+        decision_maker="loan_officer",
+    )
+    base.update(overrides)
+    return Decision(**base)
+
+
+class TestDecisionScores:
+    def test_full_pass(self):
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
+        r = reg.get_evaluator("decision_scores")(
+            d, config={"expected_outcome": "approved"}
+        )
+        assert r.passed
+        assert r.meta["decision_outcome"] is True
+        assert r.meta["provenance"] is True
+
+    def test_outcome_mismatch(self):
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
+        r = reg.get_evaluator("decision_scores")(
+            d, config={"expected_outcome": "denied"}
+        )
+        assert not r.passed
+        assert r.meta["decision_outcome"] is False
+
+    def test_outcome_from_expected_argument(self):
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
+        r = reg.get_evaluator("decision_scores")(d, expected="approved")
+        assert r.passed
+        assert r.meta["decision_outcome"] is True
+
+    def test_outcome_mismatch_via_expected_argument(self):
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
+        r = reg.get_evaluator("decision_scores")(d, expected="denied")
+        assert not r.passed
+        assert r.meta["decision_outcome"] is False
+        assert "decision_outcome" in r.meta["reasons"]
+
+    def test_outcome_check_skipped_when_no_expected(self):
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
+        r = reg.get_evaluator("decision_scores")(d)
+        assert "decision_outcome" not in r.meta
+
+    def test_confidence_out_of_range(self):
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}}, confidence=0.4)
+        r = reg.get_evaluator("decision_scores")(
+            d, config={"expected_outcome": "approved", "min_confidence": 0.8}
+        )
+        assert not r.passed
+        assert r.meta["decision_confidence"] is False
+
+    def test_missing_provenance_fails(self):
+        d = _decision(metadata={})
+        r = reg.get_evaluator("decision_scores")(d, config={"expected_outcome": "approved"})
+        assert not r.passed
+        assert r.meta["provenance"] is False
+
+    def test_missing_required_fields(self):
+        d = _decision(reasoning="")
+        r = reg.get_evaluator("decision_scores")(d, config={"expected_outcome": "approved"})
+        assert not r.passed
+        assert r.meta["reasoning"] is False
+
+    def test_dict_input_coerced(self):
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
+        as_dict = d.to_dict()
+        r = reg.get_evaluator("decision_scores")(
+            as_dict, config={"expected_outcome": "approved"}
+        )
+        assert r.passed
+
+    def test_malformed_dict_is_error_not_crash(self):
+        r = reg.get_evaluator("decision_scores")({"foo": "bar"}, config={})
+        assert not r.passed
+        assert r.meta.get("error")
+
+    def test_non_dict_metadata_is_error_not_crash(self):
+        bad = _decision(metadata="not-a-dict")
+        r = reg.get_evaluator("decision_scores")(bad, config={})
+        assert not r.passed
+        assert r.meta["provenance"] is False
+
+    def test_policy_compliance_check(self):
+        class FakePolicyEngine:
+            def check_compliance(self, decision, policy_id):
+                return True
+
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
+        r = reg.get_evaluator("decision_scores")(
+            d, config={
+                "expected_outcome": "approved",
+                "policy_engine": FakePolicyEngine(),
+                "policy_id": "p1",
+                "expected_policy_compliant": True,
+            }
+        )
+        assert r.meta["policy"] is True
+
+    def test_policy_mismatch_fails(self):
+        class FakePolicyEngine:
+            def check_compliance(self, decision, policy_id):
+                return False
+
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
+        r = reg.get_evaluator("decision_scores")(
+            d, config={
+                "policy_engine": FakePolicyEngine(),
+                "policy_id": "p1",
+                "expected_policy_compliant": True,
+            }
+        )
+        assert not r.passed
+        assert r.meta["policy"] is False
+
+    def test_causal_chain_gate(self):
+        d = _decision(metadata={"provenance": {"prov_record": "rid-1"}}, decision_id="only-decision")
+        with pytest.raises(NotImplementedError):
+            reg.get_evaluator("decision_scores")(
+                d, config={"causal_chain_exists": True, "graph_store": object()}
+            )

tests/evals/test_evaluators_part1.py

@@ -0,0 +1,81 @@
+"""Tests for generic evaluators: exact, regex, ranges, length."""
+import pytest
+
+from semantica.evals import registry as reg
+
+
+class TestExactMatch:
+    def test_exact_str(self):
+        r = reg.get_evaluator("exact_match")("approved", "approved")
+        assert r.passed and r.score == 1.0
+
+    def test_exact_str_negative(self):
+        r = reg.get_evaluator("exact_match")("approved", "denied")
+        assert not r.passed and r.score == 0.0
+
+    def test_exact_number(self):
+        r = reg.get_evaluator("exact_match")(5, 5)
+        assert r.passed
+
+    def test_exact_array(self):
+        r = reg.get_evaluator("exact_match")([1, 2], [1, 2])
+        assert r.passed
+
+
+class TestRegexMatch:
+    def test_matching(self):
+        r = reg.get_evaluator("regex_match")("abc123", r"^[a-z]+\d+$")
+        assert r.passed
+
+    def test_non_matching(self):
+        r = reg.get_evaluator("regex_match")("ABC", r"^[a-z]+$")
+        assert not r.passed
+        assert "ABC" in r.meta.get("reason", "")
+
+    def test_invalid_regex_is_error_metric(self):
+        r = reg.get_evaluator("regex_match")("x", "[invalid")
+        assert not r.passed
+        assert r.meta.get("error")
+
+
+class TestNumericRange:
+    def test_inside(self):
+        r = reg.get_evaluator("numeric_range")(0.9, config={"min": 0.8, "max": 1.0})
+        assert r.passed and r.score == 1.0
+
+    def test_outside(self):
+        r = reg.get_evaluator("numeric_range")(0.5, config={"min": 0.8, "max": 1.0})
+        assert not r.passed and r.score == 0.0
+
+    def test_bounds_inclusive(self):
+        assert reg.get_evaluator("numeric_range")(0.8, config={"min": 0.8, "max": 0.8}).passed
+
+
+class TestTemporalRange:
+    def test_inside_window(self):
+        r = reg.get_evaluator("temporal_range")(
+            "2026-01-15T10:00:00",
+            config={"min": "2026-01-01T00:00:00", "max": "2026-02-01T00:00:00"},
+        )
+        assert r.passed
+
+    def test_outside_window(self):
+        r = reg.get_evaluator("temporal_range")(
+            "2026-03-01T00:00:00",
+            config={"min": "2026-01-01T00:00:00", "max": "2026-02-01T00:00:00"},
+        )
+        assert not r.passed
+
+
+class TestLengthRange:
+    def test_ok(self):
+        r = reg.get_evaluator("length_range")("hello", config={"min": 3, "max": 5})
+        assert r.passed
+
+    def test_too_long(self):
+        r = reg.get_evaluator("length_range")([1, 2, 3], config={"min": 1, "max": 2})
+        assert not r.passed
+
+    def test_min_not_given_defaults_zero(self):
+        r = reg.get_evaluator("length_range")("abc", config={"max": 5})
+        assert r.passed

tests/evals/test_evaluators_part2.py

@@ -0,0 +1,67 @@
+"""Tests for generic evaluators: keyword, levenshtein, rouge, llm-as-judge."""
+import pytest
+
+from semantica.evals import registry as reg
+
+
+class TestKeywordCheck:
+    def test_all_required_present(self):
+        r = reg.get_evaluator("keyword_check")(
+            "the loan was approved", expected=["loan", "approved"]
+        )
+        assert r.passed
+
+    def test_missing_keyword(self):
+        r = reg.get_evaluator("keyword_check")(
+            "the loan was approved", expected=["loan", "denied"]
+        )
+        assert not r.passed
+        assert "denied" in r.meta.get("missing", [])
+
+    def test_short_words_ignored(self):
+        r = reg.get_evaluator("keyword_check")("x and y", expected=["and"])
+        assert r.passed
+
+
+class TestLevenshtein:
+    def test_identical(self):
+        r = reg.get_evaluator("levenshtein")("credit approved", "credit approved")
+        assert r.passed
+
+    def test_close_above_threshold(self):
+        r = reg.get_evaluator("levenshtein")(
+            "credit approved", "credit denied", config={"threshold": 0.8}
+        )
+        assert not r.passed
+
+    def test_default_threshold(self):
+        assert reg.get_evaluator("levenshtein")("a", "a").passed
+
+
+class TestRouge:
+    def test_identical(self):
+        r = reg.get_evaluator("rouge")("loan approved by committee", "loan approved by committee")
+        assert r.passed
+        assert r.meta["f1"] == pytest.approx(1.0)
+
+    def test_no_overlap(self):
+        r = reg.get_evaluator("rouge")("one two three", "four five six")
+        assert not r.passed
+
+    def test_partial_sets_meta(self):
+        r = reg.get_evaluator("rouge")("a b c", "a b d", config={"threshold": 0.5})
+        assert "precision" in r.meta and "recall" in r.meta
+
+
+class TestLlmAsJudge:
+    def test_uses_supplied_judge(self):
+        judge = lambda actual, expected: actual == expected  # noqa: E731
+        r = reg.get_evaluator("llm_as_judge")(
+            "x", "x", config={"judge_fn": judge}
+        )
+        assert r.passed
+
+    def test_missing_judge_is_error(self):
+        r = reg.get_evaluator("llm_as_judge")("x", "y", config={})
+        assert not r.passed
+        assert r.meta.get("error")

tests/evals/test_public_api.py

@@ -0,0 +1,32 @@
+"""Tests for the evals public package API."""
+from semantica import evals
+from semantica.evals import evaluate, get_evaluator, list_evaluators
+
+
+class TestPublicAPI:
+    def test_imports(self):
+        assert callable(evaluate)
+        assert callable(list_evaluators)
+        assert callable(get_evaluator)
+
+    def test_version_present(self):
+        assert hasattr(evals, "__version__")
+
+    def test_module_proxy_via_root(self):
+        # semantica.evals must resolve through the lazy proxy
+        assert hasattr(evals, "evaluate")
+
+    def test_all_populated(self):
+        assert len(evals.__all__) >= 2
+        assert "evaluate" in evals.__all__
+        assert "list_evaluators" in evals.__all__
+        assert "get_evaluator" in evals.__all__
+
+    def test_register_discovery(self):
+        names = evals.list_evaluators()
+        for expected in (
+            "exact_match", "regex_match", "numeric_range", "temporal_range",
+            "length_range", "keyword_check", "levenshtein", "rouge",
+            "llm_as_judge", "decision_scores",
+        ):
+            assert expected in names

tests/evals/test_registry.py

@@ -0,0 +1,36 @@
+"""Tests for the evaluator registry."""
+import pytest
+
+from semantica.evals import registry as reg
+from semantica.evals.types import EvalMetric
+
+# A unique name that will not collide with any production evaluator.
+_TEST_EVAL_NAME = "test_registry_demo_eval"
+
+
+class TestRegistry:
+    def teardown_method(self, method):
+        # Remove the test evaluator after each test that may have registered it,
+        # so re-runs and randomised collection cannot see stale state.
+        reg.EVALUATORS.pop(_TEST_EVAL_NAME, None)
+
+    def test_register_and_get(self):
+        @reg.register(_TEST_EVAL_NAME)
+        def demo(actual, expected, config=None, **kwargs):
+            return EvalMetric(1.0, True)
+
+        assert reg.get_evaluator(_TEST_EVAL_NAME) is demo
+        assert _TEST_EVAL_NAME in reg.list_evaluators()
+
+    def test_registration_is_immutable_after_commit(self):
+        with pytest.raises(ValueError):
+            reg.get_evaluator("does_not_exist")
+
+    def test_unknown_evaluator_failure_message(self):
+        with pytest.raises(ValueError) as exc:
+            reg.get_evaluator("nope")
+        msg = str(exc.value)
+        assert "nope" in msg
+        # The error message lists available evaluators; verify using a name
+        # that is always registered at import time (independent of test order).
+        assert "exact_match" in msg

tests/evals/test_runner.py

@@ -0,0 +1,459 @@
+"""Tests for the evals runner."""
+import pytest
+
+from semantica.evals.runner import evaluate
+
+
+class TestEvaluate:
+    def test_raw_tuple_cases(self):
+        result = evaluate(
+            [("approved", "approved"), ("approved", "denied")],
+            evaluators=["exact_match"],
+        )
+        assert result.total == 2
+        assert result.passed == 1
+        assert result.failed == 1
+        assert result.errors == 0
+        assert result.pass_rate == 0.5
+
+    def test_dict_cases_with_target_fn(self):
+        def fn(case):
+            return "ok" if case["id"] == "good" else "no"
+
+        result = evaluate(
+            [{"id": "good"}, {"id": "bad"}],
+            evaluators=["exact_match"],
+            target_fn=fn,
+            config={"expected": "ok"},
+        )
+        assert result.passed == 1
+        assert result.failed == 1
+
+    def test_error_capture(self):
+        result = evaluate([("x", "y")], evaluators=["does_not_exist"])
+        assert result.errors == 1
+        assert result.failed == 0
+        assert result.pass_rate == 0.0
+
+    def test_error_metric_classified_as_error(self):
+        result = evaluate(
+            [("[invalid", "x")],
+            evaluators=["regex_match"],
+        )
+        assert result.errors == 1
+        assert result.failed == 0
+        assert result.cases[0].status == "error"
+
+    def test_error_metric_and_fail_combine_as_error(self):
+        result = evaluate(
+            [("[invalid", "apple pie")],
+            evaluators=["regex_match", "exact_match"],
+        )
+        assert result.errors == 1
+        assert result.failed == 0
+        assert result.cases[0].status == "error"
+
+    def test_per_case_details(self):
+        result = evaluate([("a", "b")], evaluators=["exact_match"])
+        case = result.cases[0]
+        assert case.status == "fail"
+        assert "exact_match" in case.details
+
+    def test_empty_cases(self):
+        result = evaluate([], evaluators=["exact_match"])
+        assert result.total == 0 and result.pass_rate == 1.0
+
+    def test_multiple_evaluators(self):
+        result = evaluate(
+            [("apple pie", "apple pie")],
+            evaluators=["exact_match", "keyword_check"],
+            config={"keyword_check": {"required": ["apple"]}},
+        )
+        assert result.passed == 1
+        assert "exact_match" in result.cases[0].metrics
+        assert "keyword_check" in result.cases[0].metrics
+
+
+class TestObjective:
+    def test_maximize_with_threshold_pass(self):
+        # levenshtein similarity 1.0 for identical, objective demands >= 0.5
+        result = evaluate(
+            [("apple", "apple")],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.5}}},
+        )
+        assert result.cases[0].status == "pass"
+        assert result.cases[0].metrics["levenshtein"].passed is True
+
+    def test_maximize_with_threshold_fail(self):
+        result = evaluate(
+            [("apple", "aple")],  # similarity < 1.0
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.99}}},
+        )
+        assert result.cases[0].status == "fail"
+        assert result.cases[0].metrics["levenshtein"].passed is False
+        assert "levenshtein" in result.cases[0].details
+
+    def test_minimize_with_threshold_pass(self):
+        # levenshtein similarity 0.6 for ("night", "nacht"); objective: similarity <= 0.7
+        result = evaluate(
+            [("night", "nacht")],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.7}}},
+        )
+        assert result.cases[0].status == "pass"
+        assert result.cases[0].metrics["levenshtein"].passed is True
+
+    def test_minimize_with_threshold_fail(self):
+        result = evaluate(
+            [("night", "nacht")],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.1}}},
+        )
+        assert result.cases[0].status == "fail"
+
+    def test_expect_true_on_boolean_metric(self):
+        result = evaluate(
+            [("ok", "ok")],
+            evaluators=["exact_match"],
+            config={"exact_match": {"objective": {"expect": True}}},
+        )
+        assert result.cases[0].status == "pass"
+
+    def test_expect_false_overrides_passing_metric(self):
+        # exact_match passes (score 1.0) but expectation is false -> fail
+        result = evaluate(
+            [("ok", "ok")],
+            evaluators=["exact_match"],
+            config={"exact_match": {"objective": {"expect": False}}},
+        )
+        assert result.cases[0].status == "fail"
+        assert result.cases[0].metrics["exact_match"].passed is False
+        assert "exact_match" in result.cases[0].details
+
+    def test_maximize_without_threshold_is_noop(self):
+        # identical behavior to no objective: evaluator's own verdict stands
+        result = evaluate(
+            [("ok", "no")],
+            evaluators=["exact_match"],
+            config={"exact_match": {"objective": {"direction": "maximize"}}},
+        )
+        assert result.cases[0].status == "fail"
+
+    def test_minimize_without_threshold_raises(self):
+        # direction-only minimize has no well-defined pass bar; must be rejected
+        with pytest.raises(ValueError, match="'minimize' requires a 'threshold'"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "minimize"}}},
+            )
+
+    def test_minimize_with_explicit_none_threshold_raises(self):
+        # explicit threshold=None is the same as omitting it; must also be rejected
+        with pytest.raises(ValueError, match="'minimize' requires a 'threshold'"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "minimize", "threshold": None}}},
+            )
+
+    def test_bad_direction_raises(self):
+        with pytest.raises(ValueError):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "sideways", "threshold": 0.5}}},
+            )
+
+    def test_expect_with_direction_raises(self):
+        with pytest.raises(ValueError):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"expect": True, "direction": "maximize"}}},
+            )
+
+    def test_error_metric_wins_over_objective(self):
+        result = evaluate(
+            [("[invalid", "x")],
+            evaluators=["regex_match"],
+            config={"regex_match": {"objective": {"direction": "maximize", "threshold": 0.0}}},
+        )
+        assert result.cases[0].status == "error"
+        assert result.errors == 1
+        assert result.failed == 0
+
+    def test_no_objective_unchanged(self):
+        result = evaluate([("ok", "no")], evaluators=["exact_match"])
+        assert result.cases[0].status == "fail"
+
+    def test_non_dict_objective_raises(self):
+        with pytest.raises(ValueError):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": "maximize"}},
+            )
+
+    def test_non_bool_expect_raises(self):
+        with pytest.raises(ValueError):
+            evaluate(
+                [("a", "b")],
+                evaluators=["exact_match"],
+                config={"exact_match": {"objective": {"expect": "false"}}},
+            )
+
+    def test_invalid_per_case_objective_fails_fast_before_target_fn(self):
+        calls = []
+
+        def side_effectful_target_fn(case):
+            calls.append(case)
+            return "line"
+
+        with pytest.raises(ValueError):
+            evaluate(
+                [{"id": "c1"}, {"id": "c2", "config": {"levenshtein": {"objective": {"direction": "diagonal"}}}}],
+                evaluators=["levenshtein"],
+                target_fn=side_effectful_target_fn,
+            )
+        # validation must reject the run before any case is processed
+        assert calls == []
+
+    def test_case_config_keeps_global_objective(self):
+        # global objective on the evaluator must survive a per-case override
+        # that touches other settings for the same evaluator (deep merge)
+        result = evaluate(
+            [{"id": "c1", "expected": "abc", "actual": "abd",
+              "config": {"levenshtein": {"ignore_case": False}}}],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
+        )
+        # levenshtein("abc","abd") == 1 > 0 -> objective fails the case
+        assert result.cases[0].status == "fail"
+
+
+class TestMergeConfig:
+    """Focused tests for _merge_config two-level deep-merge semantics."""
+
+    def test_partial_per_case_objective_inherits_global_direction(self):
+        # Per-case overrides only threshold; direction must come from global.
+        result = evaluate(
+            [{"id": "c1", "expected": "abc", "actual": "abd",
+              "config": {"levenshtein": {"objective": {"threshold": 0.99}}}}],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
+        )
+        # Effective objective: minimize, threshold=0.99.
+        # levenshtein("abc","abd") similarity ~0.667; 0.667 <= 0.99 -> pass.
+        assert result.cases[0].status == "pass"
+        assert result.cases[0].metrics["levenshtein"].passed is True
+
+    def test_partial_per_case_objective_inherits_global_threshold(self):
+        # Per-case overrides only direction; threshold must come from global.
+        result = evaluate(
+            [{"id": "c1", "expected": "abc", "actual": "abd",
+              "config": {"levenshtein": {"objective": {"direction": "maximize"}}}}],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.99}}},
+        )
+        # Effective objective: maximize, threshold=0.99.
+        # levenshtein("abc","abd") similarity ~0.667; 0.667 >= 0.99 -> fail.
+        assert result.cases[0].status == "fail"
+        assert result.cases[0].metrics["levenshtein"].passed is False
+
+    def test_per_case_threshold_overrides_global_threshold(self):
+        # Global: minimize, threshold=0.0 (would fail for any positive score).
+        # Per-case: threshold=0.99 (almost everything passes minimize).
+        result = evaluate(
+            [{"id": "c1", "expected": "abc", "actual": "abd",
+              "config": {"levenshtein": {"objective": {"threshold": 0.99}}}}],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
+        )
+        # Effective: minimize, threshold=0.99 -> ~0.667 <= 0.99 -> pass.
+        assert result.cases[0].status == "pass"
+
+    def test_per_case_direction_overrides_global_direction(self):
+        # Global: maximize, threshold=0.99 (would fail for ~0.667).
+        # Per-case: direction=minimize (with inherited threshold=0.99).
+        result = evaluate(
+            [{"id": "c1", "expected": "abc", "actual": "abd",
+              "config": {"levenshtein": {"objective": {"direction": "minimize"}}}}],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.99}}},
+        )
+        # Effective: minimize, threshold=0.99 -> ~0.667 <= 0.99 -> pass.
+        assert result.cases[0].status == "pass"
+
+    def test_fully_specified_per_case_objective_replaces_global(self):
+        # Both direction and threshold specified per-case; nothing from global.
+        result = evaluate(
+            [{"id": "c1", "expected": "abc", "actual": "abd",
+              "config": {"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.5}}}}],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
+        )
+        # Effective: maximize, threshold=0.5 -> ~0.667 >= 0.5 -> pass.
+        assert result.cases[0].status == "pass"
+
+    def test_per_case_non_objective_keys_do_not_erase_global_objective(self):
+        # Per-case touches only non-objective evaluator keys; global objective intact.
+        result = evaluate(
+            [{"id": "c1", "expected": "abc", "actual": "abd",
+              "config": {"levenshtein": {"threshold": 0.5}}}],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
+        )
+        # Effective: minimize, threshold=0.0 -> ~0.667 > 0.0 -> fail.
+        assert result.cases[0].status == "fail"
+
+    def test_no_objective_anywhere_unchanged(self):
+        # No objectives anywhere; evaluator's own verdict stands throughout.
+        result = evaluate(
+            [{"id": "c1", "expected": "ok", "actual": "ok",
+              "config": {"exact_match": {"some_key": "v"}}}],
+            evaluators=["exact_match"],
+            config={"exact_match": {"other_key": "w"}},
+        )
+        assert result.cases[0].status == "pass"
+
+    def test_global_config_not_mutated(self):
+        import copy
+        global_config = {"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.5}}}
+        case_config = {"levenshtein": {"objective": {"threshold": 0.2}}}
+        original_global = copy.deepcopy(global_config)
+        original_case = copy.deepcopy(case_config)
+        evaluate(
+            [{"id": "c1", "expected": "abc", "actual": "abd", "config": case_config}],
+            evaluators=["levenshtein"],
+            config=global_config,
+        )
+        assert global_config == original_global
+        assert case_config == original_case
+
+
+class TestThresholdValidation:
+    """Threshold coercion and validation: types, NaN, infinity."""
+
+    # --- valid numeric thresholds ---
+
+    def test_maximize_integer_threshold(self):
+        # int is a valid threshold; coerced to float
+        result = evaluate(
+            [("apple", "apple")],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 1}}},
+        )
+        assert result.cases[0].status == "pass"
+        assert result.cases[0].metrics["levenshtein"].passed is True
+
+    def test_minimize_integer_threshold(self):
+        result = evaluate(
+            [("night", "nacht")],
+            evaluators=["levenshtein"],
+            config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 1}}},
+        )
+        # similarity 0.6 <= 1 -> pass
+        assert result.cases[0].status == "pass"
+
+    # --- invalid threshold types ---
+
+    def test_non_numeric_string_threshold_raises(self):
+        with pytest.raises(ValueError, match="'threshold' must be a finite number"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "maximize", "threshold": "high"}}},
+            )
+
+    def test_list_threshold_raises_value_error(self):
+        # Must be ValueError, not TypeError
+        with pytest.raises(ValueError, match="'threshold' must be a finite number"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "maximize", "threshold": [0.5]}}},
+            )
+
+    def test_dict_threshold_raises_value_error(self):
+        with pytest.raises(ValueError, match="'threshold' must be a finite number"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "maximize", "threshold": {"v": 1}}}},
+            )
+
+    # --- NaN and infinity ---
+
+    def test_nan_threshold_raises(self):
+        with pytest.raises(ValueError, match="'threshold' must be a finite number"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "maximize", "threshold": float("nan")}}},
+            )
+
+    def test_positive_infinity_threshold_raises(self):
+        with pytest.raises(ValueError, match="'threshold' must be a finite number"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "maximize", "threshold": float("inf")}}},
+            )
+
+    def test_negative_infinity_threshold_raises(self):
+        with pytest.raises(ValueError, match="'threshold' must be a finite number"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "maximize", "threshold": float("-inf")}}},
+            )
+
+    def test_nan_minimize_threshold_raises(self):
+        with pytest.raises(ValueError, match="'threshold' must be a finite number"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "minimize", "threshold": float("nan")}}},
+            )
+
+    # --- preserved behaviors ---
+
+    def test_maximize_without_threshold_still_noop(self):
+        # maximize without threshold remains a no-op regardless of threshold validation
+        result = evaluate(
+            [("ok", "no")],
+            evaluators=["exact_match"],
+            config={"exact_match": {"objective": {"direction": "maximize"}}},
+        )
+        assert result.cases[0].status == "fail"
+
+    def test_minimize_explicit_none_threshold_still_raises(self):
+        # threshold=None for minimize hits the None check before coercion
+        with pytest.raises(ValueError, match="'minimize' requires a 'threshold'"):
+            evaluate(
+                [("a", "b")],
+                evaluators=["levenshtein"],
+                config={"levenshtein": {"objective": {"direction": "minimize", "threshold": None}}},
+            )
+
+    def test_threshold_errors_are_fail_fast(self):
+        # Invalid threshold on case 2 must reject the whole run before case 1 executes
+        calls = []
+
+        def recording_fn(case):
+            calls.append(case)
+            return "x"
+
+        with pytest.raises(ValueError, match="'threshold' must be a finite number"):
+            evaluate(
+                [
+                    {"id": "c1"},
+                    {"id": "c2", "config": {"levenshtein": {"objective": {"direction": "maximize", "threshold": [0.5]}}}},
+                ],
+                evaluators=["levenshtein"],
+                target_fn=recording_fn,
+            )
+        assert calls == []

tests/evals/test_types.py

@@ -0,0 +1,43 @@
+"""Tests for evals result models."""
+import pytest
+
+from semantica.evals.types import CaseResult, EvalMetric, EvalSummary
+
+
+class TestEvalMetric:
+    def test_construction(self):
+        m = EvalMetric(score=1.0, passed=True, meta={"threshold": 1.0})
+        assert m.score == 1.0 and m.passed and m.meta["threshold"] == 1.0
+
+    def test_default_meta(self):
+        m = EvalMetric(0.0, False)
+        assert m.meta == {}
+
+    def test_default_meta_is_not_shared(self):
+        m1 = EvalMetric(0.0, False)
+        m2 = EvalMetric(0.0, False)
+        m1.meta["mutated"] = True
+        assert "mutated" not in m2.meta
+
+
+class TestCaseResult:
+    def test_status_fail_on_any_failed_metric(self):
+        r = CaseResult(
+            case_id="c1",
+            status="fail",
+            metrics={"exact_match": EvalMetric(0.0, False)},
+            details={},
+        )
+        assert r.status == "fail"
+        assert r.metrics["exact_match"].passed is False
+
+
+class TestEvalSummary:
+    def test_pass_rate(self):
+        s = EvalSummary(total=10, passed=8, failed=1, errors=1, pass_rate=0.8)
+        assert s.pass_rate == 0.8
+
+    def test_cases_are_mutable(self):
+        s = EvalSummary(0, 0, 0, 0, 1.0)
+        s.cases.append(CaseResult("c", "pass", {}, {}))
+        assert len(s.cases) == 1

tests/vector_store/test_milvus_store.py

@@ -0,0 +1,176 @@
+"""Tests for MilvusStore.iter_all() query-iterator enumeration.
+
+pymilvus is not installed in this environment, so these drive the real
+MilvusStore against MagicMocks, following the pattern already used for milvus
+in test_backend_metadata_filtering.py.
+"""
+
+from unittest.mock import MagicMock, patch
+
+import numpy as np
+import pytest
+
+from semantica.utils.exceptions import ProcessingError
+from semantica.vector_store.milvus_store import MilvusStore
+
+
+def _store_with_batches(*batches):
+    """MilvusStore whose query_iterator yields the given batches then stops.
+
+    The attribute path is doubled here: the pymilvus Collection sits at
+    wrapper.collection.
+    """
+    store = MilvusStore()
+    wrapper = MagicMock()
+    inner = MagicMock()
+    iterator = MagicMock()
+    iterator.next.side_effect = list(batches)
+    inner.query_iterator.return_value = iterator
+    wrapper.collection = inner
+    store.collection = wrapper
+    return store, wrapper, inner, iterator
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_yields_batches_until_exhausted():
+    """Exhaustion is an empty list, not StopIteration."""
+    store, _, _, iterator = _store_with_batches(
+        [{"id": 1, "vector": [0.1], "metadata": {}}],
+        [{"id": 2, "vector": [0.2], "metadata": {}}],
+        [],
+    )
+
+    result = list(store.iter_all(batch_size=1))
+
+    assert [item["id"] for item in result] == ["1", "2"]
+    assert iterator.next.call_count == 3
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_requests_the_fields_needed_for_the_result_shape():
+    store, _, inner, _ = _store_with_batches([])
+
+    list(store.iter_all(batch_size=64))
+
+    kwargs = inner.query_iterator.call_args[1]
+    assert kwargs["batch_size"] == 64
+    assert kwargs["output_fields"] == ["id", "vector", "metadata"]
+    # Milvus rejects an empty expression, so a match-all form is required.
+    assert kwargs["expr"] == "id != ''"
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_loads_the_collection_before_querying():
+    """Milvus requires a loaded collection for query operations."""
+    store, wrapper, _, _ = _store_with_batches([])
+
+    list(store.iter_all())
+
+    assert wrapper.load.called
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_closes_the_iterator_on_exhaustion():
+    store, _, _, iterator = _store_with_batches([])
+
+    list(store.iter_all())
+
+    assert iterator.close.called
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_closes_the_iterator_when_consumer_stops_early():
+    """Abandoning the generator early must still release the iterator."""
+    store, _, _, iterator = _store_with_batches(
+        [{"id": 1, "vector": [0.1], "metadata": {}}],
+        [{"id": 2, "vector": [0.2], "metadata": {}}],
+        [],
+    )
+
+    generator = store.iter_all(batch_size=1)
+    next(generator)
+    assert not iterator.close.called
+    generator.close()
+
+    assert iterator.close.called
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_converts_entities_to_the_shared_result_shape():
+    store, _, _, _ = _store_with_batches(
+        [{"id": 7, "vector": [0.1, 0.2, 0.3], "metadata": {"tag": "x"}}], []
+    )
+
+    item = list(store.iter_all())[0]
+
+    assert item["id"] == "7"
+    assert item["metadata"] == {"tag": "x"}
+    np.testing.assert_allclose(item["vector"], np.array([0.1, 0.2, 0.3]))
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_handles_missing_vector_and_metadata():
+    store, _, _, _ = _store_with_batches([{"id": 1, "vector": None, "metadata": None}], [])
+
+    item = list(store.iter_all())[0]
+
+    assert item["metadata"] == {}
+    assert item["vector"] is None
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_empty_collection_yields_nothing():
+    store, _, _, _ = _store_with_batches([])
+
+    assert list(store.iter_all()) == []
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_raises_when_query_iterator_is_unavailable():
+    """Older pymilvus lacks query_iterator; falling back to query(offset=...)
+    would truncate at the 16384 window."""
+    store = MilvusStore()
+    wrapper = MagicMock()
+    wrapper.collection = MagicMock(spec=["query"])
+    store.collection = wrapper
+
+    with pytest.raises(ProcessingError, match="query_iterator"):
+        list(store.iter_all())
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_raises_when_collection_not_initialized():
+    """Must fail loudly: an empty scan reads the same as an empty source."""
+    store = MilvusStore()
+
+    with pytest.raises(ProcessingError, match="Collection not initialized"):
+        list(store.iter_all())
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", False)
+def test_iter_all_raises_when_milvus_unavailable():
+    store = MilvusStore()
+    store.collection = MagicMock()
+
+    with pytest.raises(ProcessingError):
+        list(store.iter_all())
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_propagates_iterator_errors():
+    store, _, _, iterator = _store_with_batches()
+    iterator.next.side_effect = RuntimeError("connection reset")
+
+    with pytest.raises(RuntimeError, match="connection reset"):
+        list(store.iter_all())
+
+
+@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
+def test_iter_all_closes_the_iterator_when_a_batch_fails():
+    store, _, _, iterator = _store_with_batches()
+    iterator.next.side_effect = RuntimeError("connection reset")
+
+    with pytest.raises(RuntimeError):
+        list(store.iter_all())
+
+    assert iterator.close.called

tests/vector_store/test_pgvector_store.py

@@ -10,7 +10,7 @@ To run these tests locally with Docker:
         -e POSTGRES_PASSWORD=postgres \
         -e POSTGRES_DB=test \
         -p 5432:5432 \
-        ankane/pgvector:latest
+        pgvector/pgvector:pg16
 
     pytest tests/vector_store/test_pgvector_store.py -v
 
@@ -63,29 +63,37 @@ TEST_CONNECTION_STRING = os.getenv(
 
 @pytest.fixture(scope="module")
 def pg_available() -> bool:
-    """Check if PostgreSQL with pgvector is available."""
+    """Check if PostgreSQL with pgvector is available.
+
+    A connection failure only means "skip" when TEST_PGVECTOR_URL wasn't set
+    explicitly, i.e. this is a local run falling back to the documented
+    default. CI sets it on purpose, so a failure there means the service is
+    genuinely broken and the suite should fail loudly instead of skipping.
+    """
     if not psycopg_available:
         return False
 
+    explicit_url = "TEST_PGVECTOR_URL" in os.environ
+
     try:
-        if psycopg_available:
-            try:
-                import psycopg
+        try:
+            import psycopg
 
-                conn = psycopg.connect(TEST_CONNECTION_STRING, connect_timeout=5)
-            except ImportError:
-                import psycopg2
+            conn = psycopg.connect(TEST_CONNECTION_STRING, connect_timeout=5)
+        except ImportError:
+            import psycopg2
 
-                conn = psycopg2.connect(TEST_CONNECTION_STRING, connect_timeout=5)
+            conn = psycopg2.connect(TEST_CONNECTION_STRING, connect_timeout=5)
 
-            cur = conn.cursor()
-            cur.execute("SELECT 1")
-            cur.close()
-            conn.close()
-            return True
+        cur = conn.cursor()
+        cur.execute("SELECT 1")
+        cur.close()
+        conn.close()
+        return True
     except Exception:
+        if explicit_url:
+            raise
         return False
-    return False
 
 
 @pytest.fixture
@@ -191,7 +199,13 @@ class TestPgVectorStoreAdd:
         ids = store.add(vectors, metadata)
 
         assert len(ids) == 5
-        assert all(id.startswith("vec_") for id in ids)
+        assert len(set(ids)) == 5
+        # add() assigns uuid4 identifiers, not a "vec_" prefix
+        for vector_id in ids:
+            try:
+                uuid.UUID(vector_id)
+            except ValueError:
+                pytest.fail(f"{vector_id!r} is not a valid uuid4 id")
 
     def test_add_auto_generate_ids(self, store):
         """Test that IDs are auto-generated if not provided."""
@@ -290,38 +304,40 @@ class TestPgVectorStoreSearch:
         if not pg_available:
             pytest.skip("PostgreSQL not available")
 
-        from semantica.vector_store.pgvector_store import PgVectorStore
+        from semantica.vector_store.pgvector_store import PgVectorStore, psycopg_sql
 
+        # setup_vectors is autouse and seeds unique_table_name, and fixtures are
+        # cached per test, so this needs a table of its own to be empty at all.
+        empty_table = f"{unique_table_name}_empty"
         empty_store = PgVectorStore(
             connection_string=TEST_CONNECTION_STRING,
-            table_name=unique_table_name,
+            table_name=empty_table,
             dimension=128,
             distance_metric="cosine",
         )
 
-        query = np.random.rand(128).astype(np.float32)
-        results = empty_store.search(query, top_k=5)
-
-        assert len(results) == 0
-
-        # Cleanup: Drop test table after test completes
-        # Uses best-effort cleanup - failures are silently ignored since
-        # this is teardown of optional test resources
         try:
-            with empty_store._get_connection() as conn:
-                cur = conn.cursor()
-                from semantica.vector_store.pgvector_store import psycopg_sql
-                drop_sql = psycopg_sql.SQL("DROP TABLE IF EXISTS {}").format(
-                    psycopg_sql.Identifier(unique_table_name)
-                )
-                cur.execute(drop_sql)
-                conn.commit()
-                cur.close()
-            empty_store.close()
-        except Exception:
-            # Best-effort cleanup: PostgreSQL may be unavailable during teardown
-            # This is expected when tests are skipped or connection is lost
-            pass
+            query = np.random.rand(128).astype(np.float32)
+            results = empty_store.search(query, top_k=5)
+
+            assert len(results) == 0
+        finally:
+            try:
+                with empty_store._get_connection() as conn:
+                    cur = conn.cursor()
+                    cur.execute(
+                        psycopg_sql.SQL("DROP TABLE IF EXISTS {}").format(
+                            psycopg_sql.Identifier(empty_table)
+                        )
+                    )
+                    conn.commit()
+                    cur.close()
+                empty_store.close()
+            except Exception:
+                # Best-effort cleanup: PostgreSQL may be unavailable during
+                # teardown. This is expected when tests are skipped or the
+                # connection is lost.
+                pass
 
 
 class TestPgVectorStoreGet:

tests/vector_store/test_pinecone_store.py

@@ -249,6 +249,191 @@ class TestPineconeIndex(unittest.TestCase):
         mock_index.query.assert_called_once()
 
 
+class TestPineconeIterAll(unittest.TestCase):
+    """PineconeStore.iter_all() list-then-fetch enumeration."""
+
+    def _page(self, ids, next_token):
+        """Stand-in for a list_paginated() response."""
+        response = MagicMock()
+        response.vectors = [MagicMock(id=vector_id) for vector_id in ids]
+        response.pagination = MagicMock(next=next_token)
+        return response
+
+    def _store(self, pages, fetch_results):
+        store = PineconeStore()
+        wrapper = MagicMock()
+        raw_index = MagicMock()
+        raw_index.list_paginated.side_effect = list(pages)
+        wrapper.index = raw_index
+        wrapper.fetch_vectors.side_effect = list(fetch_results)
+        store.index = wrapper
+        return store, wrapper, raw_index
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_threads_pagination_token_across_pages(self):
+        store, _, raw_index = self._store(
+            [self._page(["a", "b"], "token-1"), self._page(["c"], None)],
+            [
+                {"vectors": {"a": {"values": [0.1], "metadata": {}},
+                             "b": {"values": [0.2], "metadata": {}}}},
+                {"vectors": {"c": {"values": [0.3], "metadata": {}}}},
+            ],
+        )
+
+        result = list(store.iter_all(batch_size=2))
+
+        self.assertEqual([item["id"] for item in result], ["a", "b", "c"])
+        calls = raw_index.list_paginated.call_args_list
+        self.assertNotIn("pagination_token", calls[0][1])
+        self.assertEqual(calls[1][1]["pagination_token"], "token-1")
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_hydrates_listed_ids_with_a_fetch(self):
+        """Listing returns ids only, so each page needs a fetch()."""
+        store, wrapper, _ = self._store(
+            [self._page(["a"], None)],
+            [{"vectors": {"a": {"values": [0.1, 0.2], "metadata": {"tag": "x"}}}}],
+        )
+
+        item = list(store.iter_all())[0]
+
+        self.assertEqual(item["id"], "a")
+        self.assertEqual(item["metadata"], {"tag": "x"})
+        np.testing.assert_allclose(item["vector"], np.array([0.1, 0.2]))
+        wrapper.fetch_vectors.assert_called_once_with(["a"], namespace="")
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_list_and_fetch_use_the_same_namespace(self):
+        store, wrapper, raw_index = self._store(
+            [self._page(["a"], None)],
+            [{"vectors": {"a": {"values": [0.1], "metadata": {}}}}],
+        )
+
+        list(store.iter_all(namespace="prod"))
+
+        self.assertEqual(raw_index.list_paginated.call_args[1]["namespace"], "prod")
+        wrapper.fetch_vectors.assert_called_once_with(["a"], namespace="prod")
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_skips_ids_deleted_between_list_and_fetch(self):
+        """fetch() omits ids it cannot find rather than returning blanks."""
+        store, _, _ = self._store(
+            [self._page(["a", "gone"], None)],
+            [{"vectors": {"a": {"values": [0.1], "metadata": {}}}}],
+        )
+
+        result = list(store.iter_all())
+
+        self.assertEqual([item["id"] for item in result], ["a"])
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_raises_when_pagination_token_repeats(self):
+        """A stalled token must not loop forever, nor quietly return a partial
+        scan that reads as a complete one."""
+        store, _, raw_index = self._store(
+            [self._page(["a"], "same"), self._page(["b"], "same")],
+            [
+                {"vectors": {"a": {"values": [0.1], "metadata": {}}}},
+                {"vectors": {"b": {"values": [0.2], "metadata": {}}}},
+            ],
+        )
+
+        with self.assertRaises(ProcessingError):
+            list(store.iter_all())
+
+        self.assertEqual(raw_index.list_paginated.call_count, 2)
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_empty_listing_yields_nothing_without_fetching(self):
+        store, wrapper, _ = self._store([self._page([], None)], [])
+
+        self.assertEqual(list(store.iter_all()), [])
+        wrapper.fetch_vectors.assert_not_called()
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_continues_past_an_empty_page_with_a_live_token(self):
+        """An empty page is not necessarily the end: Pinecone can legitimately
+        list zero ids for a page while pagination.next is still set (sparse
+        or filtered namespaces, eventual-consistency windows on serverless
+        indexes). Only the absence of a next token means exhaustion."""
+        store, wrapper, raw_index = self._store(
+            [
+                self._page(["a"], "token-1"),
+                self._page([], "token-2"),  # empty page, but the token still advances
+                self._page(["b"], None),
+            ],
+            [
+                {"vectors": {"a": {"values": [0.1], "metadata": {}}}},
+                {"vectors": {"b": {"values": [0.2], "metadata": {}}}},
+            ],
+        )
+
+        result = list(store.iter_all(batch_size=1))
+
+        self.assertEqual([item["id"] for item in result], ["a", "b"])
+        self.assertEqual(raw_index.list_paginated.call_count, 3)
+        # Nothing to hydrate on the empty page, so only two fetches happen.
+        self.assertEqual(wrapper.fetch_vectors.call_count, 2)
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_accepts_plain_string_ids_from_listing(self):
+        """SDK generations differ on what listing yields."""
+        store, _, _ = self._store(
+            [self._page([], None)],
+            [{"vectors": {"a": {"values": [0.1], "metadata": {}}}}],
+        )
+        response = MagicMock()
+        response.vectors = ["a"]
+        response.pagination = MagicMock(next=None)
+        store.index.index.list_paginated.side_effect = [response]
+
+        self.assertEqual([item["id"] for item in store.iter_all()], ["a"])
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_handles_missing_values_and_metadata(self):
+        store, _, _ = self._store(
+            [self._page(["a"], None)],
+            [{"vectors": {"a": {"values": None, "metadata": None}}}],
+        )
+
+        item = list(store.iter_all())[0]
+
+        self.assertIsNone(item["vector"])
+        self.assertEqual(item["metadata"], {})
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_raises_when_list_paginated_unavailable(self):
+        store = PineconeStore()
+        wrapper = MagicMock()
+        wrapper.index = MagicMock(spec=["query", "fetch"])
+        store.index = wrapper
+
+        with self.assertRaises(ProcessingError):
+            list(store.iter_all())
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_raises_when_index_not_initialized(self):
+        """Must fail loudly: an empty scan reads the same as an empty source."""
+        with self.assertRaises(ProcessingError):
+            list(PineconeStore().iter_all())
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', False)
+    def test_raises_when_pinecone_unavailable(self):
+        store = PineconeStore()
+        store.index = MagicMock()
+
+        with self.assertRaises(ProcessingError):
+            list(store.iter_all())
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_propagates_listing_errors(self):
+        store, _, raw_index = self._store([], [])
+        raw_index.list_paginated.side_effect = RuntimeError("connection reset")
+
+        with self.assertRaises(RuntimeError):
+            list(store.iter_all())
+
+
 if __name__ == '__main__':
     print("DEBUG: Starting unittest.main()")
     unittest.main()

tests/vector_store/test_weaviate_store.py

@@ -0,0 +1,260 @@
+"""Tests for WeaviateStore.iter_all() cursor enumeration.
+
+weaviate-client is not installed in this environment, so these drive the real
+WeaviateStore against MagicMocks, following the pattern already used for
+weaviate in test_backend_metadata_filtering.py.
+"""
+
+from unittest.mock import MagicMock, patch
+
+import numpy as np
+import pytest
+
+from semantica.utils.exceptions import ProcessingError
+from semantica.vector_store.weaviate_store import WeaviateStore
+
+
+def _obj(uuid, properties=None, vector=None):
+    """Stand-in for a weaviate v4 returned object."""
+    obj = MagicMock()
+    obj.uuid = uuid
+    obj.properties = properties
+    obj.vector = vector
+    return obj
+
+
+def _page(objects):
+    """Stand-in for a fetch_objects() response."""
+    response = MagicMock()
+    response.objects = objects
+    return response
+
+
+def _store_with_pages(*pages):
+    store = WeaviateStore()
+    store.collection = MagicMock()
+    store.collection.query.fetch_objects.side_effect = list(pages)
+    return store
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_threads_uuid_cursor_across_pages():
+    """The next page must continue after the last object's UUID."""
+    store = _store_with_pages(
+        _page([_obj("uuid-1"), _obj("uuid-2")]),
+        _page([_obj("uuid-3")]),
+    )
+
+    result = list(store.iter_all(batch_size=2))
+
+    assert [item["id"] for item in result] == ["uuid-1", "uuid-2", "uuid-3"]
+    calls = store.collection.query.fetch_objects.call_args_list
+    assert "after" not in calls[0][1]
+    assert calls[1][1]["after"] == "uuid-2"
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_stops_on_short_page():
+    """A page smaller than batch_size means the collection is exhausted."""
+    store = _store_with_pages(_page([_obj("uuid-1")]))
+
+    result = list(store.iter_all(batch_size=5))
+
+    assert [item["id"] for item in result] == ["uuid-1"]
+    assert store.collection.query.fetch_objects.call_count == 1
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_raises_when_cursor_stops_advancing():
+    """A stalled cursor must terminate, but not quietly: a partial scan reads
+    as a complete one."""
+    store = WeaviateStore()
+    store.collection = MagicMock()
+    store.collection.query.fetch_objects.return_value = _page(
+        [_obj("same-uuid"), _obj("same-uuid")]
+    )
+
+    with pytest.raises(ProcessingError, match="stopped advancing"):
+        list(store.iter_all(batch_size=2))
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_continues_past_empty_page_in_cursor_mode():
+    """A full page followed by an empty page must not be read as the end of
+    the collection: the empty page could be a gap (e.g. a window landing on
+    tombstoned objects) with real data past it, the same failure mode
+    already confirmed for Qdrant's scroll cursor (#1316). The `after` cursor
+    has no server-issued value to advance past an empty page with, so this
+    must fall back to offset pagination rather than silently stopping."""
+    store = _store_with_pages(
+        _page([_obj("uuid-1"), _obj("uuid-2")]),  # full page, cursor -> uuid-2
+        _page([]),                                 # empty page: not the end
+        _page([_obj("uuid-3")]),                   # real data past the gap
+    )
+
+    result = [item["id"] for item in store.iter_all(batch_size=2)]
+
+    assert result == ["uuid-1", "uuid-2", "uuid-3"]
+    calls = store.collection.query.fetch_objects.call_args_list
+    assert len(calls) == 3
+    assert calls[1][1]["after"] == "uuid-2"  # the empty page still queried by cursor
+    assert calls[2][1].get("offset") == 2    # then the fallback used position, not the cursor
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_offset_fallback_advances_across_pages():
+    """Regression: the offset was only set inside the except branch, so pages
+    after the fallback went out with no pagination at all and the scan
+    restarted from page one."""
+    store = WeaviateStore()
+    store.collection = MagicMock()
+    calls = []
+
+    def _fetch(**kwargs):
+        calls.append(dict(kwargs))
+        if "after" in kwargs:
+            raise TypeError("unexpected keyword argument 'after'")
+        page_number = len(calls)
+        if page_number < 4:
+            return _page([_obj(f"u{page_number}a"), _obj(f"u{page_number}b")])
+        return _page([_obj("last")])
+
+    store.collection.query.fetch_objects.side_effect = _fetch
+
+    ids = [item["id"] for item in store.iter_all(batch_size=2)]
+
+    assert len(set(ids)) == len(ids), f"duplicate ids means the scan restarted: {ids}"
+    assert [c.get("offset") for c in calls] == [None, None, 2, 4]
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_raises_when_no_pagination_is_supported():
+    """A client rejecting both `after` and `offset` cannot page past the first
+    result."""
+    store = WeaviateStore()
+    store.collection = MagicMock()
+
+    def _fetch(**kwargs):
+        if "after" in kwargs or "offset" in kwargs:
+            raise TypeError("unsupported")
+        return _page([_obj("a"), _obj("b")])
+
+    store.collection.query.fetch_objects.side_effect = _fetch
+
+    with pytest.raises(ProcessingError, match="neither an .after. cursor nor a"):
+        list(store.iter_all(batch_size=2))
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_empty_collection_yields_nothing():
+    """A genuinely empty collection needs two empty pages to confirm: the
+    first (in cursor mode) triggers the offset fallback, and the second
+    (in offset mode, which has no gap ambiguity) is what actually ends the
+    scan. See test_iter_all_continues_past_empty_page_in_cursor_mode for the
+    case where the first empty page is *not* the end."""
+    store = _store_with_pages(_page([]), _page([]))
+
+    assert list(store.iter_all()) == []
+    assert store.collection.query.fetch_objects.call_count == 2
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_converts_objects_to_the_shared_result_shape():
+    store = _store_with_pages(
+        _page([_obj("uuid-7", properties={"tag": "x"}, vector=[0.1, 0.2, 0.3])]),
+    )
+
+    item = list(store.iter_all())[0]
+
+    assert item["id"] == "uuid-7"
+    assert item["metadata"] == {"tag": "x"}
+    np.testing.assert_allclose(item["vector"], np.array([0.1, 0.2, 0.3]))
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_handles_missing_properties_and_vector():
+    store = _store_with_pages(_page([_obj("uuid-1", properties=None, vector=None)]))
+
+    item = list(store.iter_all())[0]
+
+    assert item["metadata"] == {}
+    assert item["vector"] is None
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_treats_empty_vector_as_none():
+    store = _store_with_pages(_page([_obj("uuid-1", vector=[])]))
+
+    assert list(store.iter_all())[0]["vector"] is None
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_requests_vectors():
+    """Weaviate omits vectors unless include_vector is set."""
+    store = _store_with_pages(_page([]), _page([]))
+
+    list(store.iter_all(batch_size=64))
+
+    kwargs = store.collection.query.fetch_objects.call_args[1]
+    assert kwargs["include_vector"] is True
+    assert kwargs["limit"] == 64
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_falls_back_to_offset_when_after_unsupported():
+    """Older clients reject `after`; the scan degrades to numeric offset."""
+    store = WeaviateStore()
+    store.collection = MagicMock()
+    seen = {"calls": 0}
+
+    def _fetch(**kwargs):
+        if "after" in kwargs:
+            raise TypeError("unexpected keyword argument 'after'")
+        seen["calls"] += 1
+        if seen["calls"] == 1:
+            return _page([_obj("uuid-1"), _obj("uuid-2")])
+        return _page([_obj("uuid-3")])
+
+    store.collection.query.fetch_objects.side_effect = _fetch
+
+    result = list(store.iter_all(batch_size=2))
+
+    assert [item["id"] for item in result] == ["uuid-1", "uuid-2", "uuid-3"]
+    offsets = [
+        c[1]["offset"]
+        for c in store.collection.query.fetch_objects.call_args_list
+        if "offset" in c[1]
+    ]
+    assert offsets == [2]
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_raises_when_collection_not_initialized():
+    """Must fail loudly, not yield nothing.
+
+    An empty scan is indistinguishable from an empty source, which would let
+    `store migrate` report success having copied nothing (issue #1083).
+    """
+    store = WeaviateStore()
+
+    with pytest.raises(ProcessingError, match="Collection not initialized"):
+        list(store.iter_all())
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", False)
+def test_iter_all_raises_when_weaviate_unavailable():
+    store = WeaviateStore()
+    store.collection = MagicMock()
+
+    with pytest.raises(ProcessingError):
+        list(store.iter_all())
+
+
+@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
+def test_iter_all_propagates_fetch_errors():
+    store = WeaviateStore()
+    store.collection = MagicMock()
+    store.collection.query.fetch_objects.side_effect = RuntimeError("connection reset")
+
+    with pytest.raises(RuntimeError, match="connection reset"):
+        list(store.iter_all())